Can the Internet Get “Walled”?

What’s the Suez Canal of the Internet?

The Ever Given blocking the Suez Canal
Ever Given

Over the last few days we bore witness to a minor economic disaster, thanks to the Ever Given having firmly planted itself into both walls of the Suez Canal. The Financial Times gives a very good overview of the factors that to this mishap. In that article, Brendan Greeley describes how the Ever Given got “walled” more so than just grounded, because it implanted itself into the canal walls.

For those of us whose life is about providing resilient services, one has to ask: where was the failure? Mr. Greeley goes into some depth about how the sheer height (beam), weight, and width of the ship, the shape of the canal, the water forces and wind all contributed to this mishap. He also pointed out that the economics favor larger vessels. This is an externality- there is no chance that the owners will ever pay for the amount of damage the blocked canal has caused, which is estimated to have been up to $10 billion. Syria was reportedly rationing fuel because of the blockage, and fuel prices across the globe ticked up. Several ships rerouted to go around the horn of Africa, risking hijackings.

The other far bigger failure here is that there is but one canal through which upon which large portions of the world economy depends. One big anything doesn’t make for good resilience. That canal could fail again. Knowing this, Iran has offered to create an alternate shipping lane, adding at least a bit of redundancy into the system. Ultimately, manufacturers throughout the supply chain can re-evaluate how to manage this sort of delivery delay. Should new lanes be formed? Should more production be closer to the end consumer? A new canal would surely cost tens of billions of dollars, and may offer only limited resilience. After all, why wouldn’t the same failure happen in both canals? In all likelihood it won’t be this precise “walling”, the hope being that canal operators and pilots will update their procedures to limit the risk.

We Internet geeks understand this class of problem in great detail, in many dimensions. A major benefit of cloud computing is to spread load across multiple CPUs in multiple locations, so that no single failure would cause disruption.

Taken individually and impacting individual customers, it’s a sure bet that cloud services are far more reliable than people rolling their own, just as it is safer to use a container vessel than trying to carry one’s products across in a dingy. However, the flip side of that coin is the impact those services have when they fail. Some examples:

WhenWhatImpact
2016Mirai BOTNET / DYN attackTwitter, other services out for a day
2020GMail, YouTube, Google DocsServices disrupted for an hour
2020Amazon Web Services East Coast Data CenterLarge numbers of application services failed
2020Cloudflare DNS outageClient resolvers failed for 27 minutes
2021Microsoft Teams and Office 365Services to their customers unavailable for four hours

Can an Internet-wide failure happen? Where’s that “Internet canal” bottleneck? I wrote about that for Cisco not long ago. It could very well be cloud-based DNS resolvers, such as Cloudflare’s 1.1.1.1. What we know is that these services can fail because they have done so in the past. Last year, MIT sage Dan Geer looked at market concentration effects on cybersecurity risk, which opens up a bigger question. This time, The Ever Given failed without any malice. Geer’s major point is that there is an asymmetric attack on large targets, like popular cloud services. The same perhaps can be said about the Suez Canal.

Note that large cloud services are not the only aggregate risk we face. Geer’s earlier work looked at software monocultures. When a large number of systems all use the same software, a single attack can affect all, or at least a great many, of them. This is just another example of a Suez Canal.

The economic drivers are always toward economies of scale, whether that’s a large cloud service or a single supplier, but at the often hidden price of aggregate resiliency. The cost generally amounts to an externality because of the size and scope of the service as well as the impact of an outage on others are not understood until an event happens. Having not considered it a week ago, some producers are considering this question today.


Courtesy of Copernicus Sentinel data 2021, https://commons.wikimedia.org/w/index.php?curid=102251045

The Challenges of CISOs

Are CISOs investing enough in protection? Do they have good visibility to threats?

Image
Aub Persian Zam Zam

Long ago there used to be a bar on Haight St. called Aub Persian Zam Zam, run by a cranky guy named Bruno. Bruno who hated everyone, and he preferred only to serve martinis.  If you walked in before 7:00pm, he told you that table service started at 8:00pm.  And if you walked in after 7:00pm, table service stopped at 6:00pm. As a customer, I felt a little like a Chief Information Security Officer (CISO). 

CISOs constantly face a challenge with their boards: how much to invest in security. If you haven’t been hacked, then you are accused of spending too much on protection (and might be out of a job); and if you have, then you spent too little (and might be out of a job).  But CISOs have to operate in the here and now. They don’t get to have the luxury of hindsight. What CISOs need is an appropriate level of investment to secure their charges and situational awareness to make good decisions.

Much is being made of the lax security that Solar Winds had. As Bruce Schneier pointed out in the New York Times, they had been hacked not just once, but several times. There was the attack on the company and then there was the attack on their customers. The attack on the customers involved the use of a DNS-based command and control (C&C) network, very stealthily crafted code, and the potential for an infected system to probe whatever was available to it at government and industrial installations across the globe. This may have been particularly damaging in the case of Solar Winds because the legitimate software could have stood in a privileged point within an enterprise, requiring access to lots of other core infrastructure. The Russians picked a really juicy target. They were, if you will, an incident waiting to happen, and happen it did. Solar Winds was detectable, but it required an appropriate investment in not only tooling but back-end expert services to provide situational awareness.

Not every target is quite so juicy. Most hackers hit web servers or laptops with various viruses. The soft underbelly of cybersecurity, however, are the control systems, who themselves have access to other infrastructure, as was demonstrated this past month, when a hacker attempted to poison a Florida city with lye. Assuming they have one, the Oldsmar CISO might have some explaining to do. How might that person do so, especially when it is the very system meant to protect the others? It starts by knowing how one compares to one’s peers in terms of expenditures. It’s possible to both under- and overspend.

Gordon Loeb Model

Optimal investment models for cybersecurity has been an ongoing area of research. The seminal Gordon-Loeb Model demonstrates a point of optimality and a point of diminishing returns for risk mitigation. The model doesn’t given you the shape of either curves. That was the next area of research.

For one, some things are easy to do, and some are hard; but the easy things are often not the right things to do. Low level cybersecurity professionals sometimes make the wrong choices, being risk seeking for big ticket items like device policy management, two-factor authentication, training, and auditing; while being risk adverse to matters that are within their control. Back in 2015, Armin Sarabi, Parinaz Naghizadeh, Yang Liu, and Mingyan Liu set out to answer this question. The table below liberally borrowed from their paper shows a risk analysis of different sectors.

Sarabi et al, Prioritizing Security Spending: A Quantitative Analysis of Risk Distributions for Different Business Profiles, Workshop on the Economics of Information Security, 2015.

What this says is that based on reports received, configuration errors were a substantial risk factor pretty much everywhere but accommodation and food services, but they suffered because employees share credentials. It was a limited survey, and surely the model has changed since then. In the intervening time, cloud computing has become far more prevalent, and we have seen numerous state actors take on a much bigger, and nastier, role. It’s useful, however, is for a CISO to have situational awareness of what sorts of common risks are being encountered, and to have some notion as to what best practices are to counter those risks, so that whatever a firm spends is effective.

Expenditures alone don’t guarantee against break-ins. Knowing one’s suppliers and their practices is also critical. Knowing that Verkada had sloppy practices would have both deterred some from using their cameras, and in turn encouraged that provider to clean up their act. Again, situational awareness matters.


Gordon Loeb Diagram by By Luca Rainieri – Own work, CC BY-SA 4.0

Internet Balkanization is here already, Mr. Schmidt.


In the technical community we like to say that the Internet is a network of networks, and that each network is independently operated and controlled. That may be true in some technical sense, but it far from the pragmatic truth.

By ProjectManhattan – Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=39714913

Today’s New York Times contains an editorial that supports former Google CEO Eric Schmidt’s view that the Internet will balkanize into two – one centered around US/Western values and one around values of China, and indeed it goes farther, to state that there will be three large Internets, where Europe has its own center.

The fact is that this is the world in which we already live.  It is well known that China already has its own Internet, in which all applications can be spied by the government.  With the advent of the GDPR, those of us in Europe have been cut off from a number of non-European web sites because they refuse to comply with Europe’s privacy regulations.  For example, I cannot read the Los Angeles Times from Switzerland.  I get this lovely message:

Unfortunately, our website is currently unavailable in most European countries. We are engaged on the issue and committed to looking at options that support our full range of digital offerings to the EU market. We continue to identify technical compliance solutions that will provide all readers with our award-winning journalism.

And then there are other mini-Internets, such as that of Iran, in which they have attempted to establish their own borders, not only to preserve their culture, but also their security, at least in their view, thanks to such attacks as Stuxnet.

If China can make its own rules, and Europe can establish its own rules, and the U.S. has its own rules, and Iran has its own rules, can we really say that there is a single Internet today?  And how many more Internets will there be tomorrow?

The trend is troubling. 

We Internet geeks also like to highlight The Network Effect, in which the value of the network to each individual increases based on the number of network participants, an effect first observed with telephone networks.  There is a risk that it can operate in reverse: each time the network bifurcates, its value to each participant decreases because of the loss of the participants who are now on separate networks.

Ironically, the capabilities found in China’s network may be very appealing to other countries such as Iran and Saudi Arabia, just as shared values around the needs of law enforcement had previously meant that a single set of lawful intercept capabilities exists in most telecommunications equipment.  This latter example reflected shared societal values of the time.

If you believe that the Internet is a good thing on the whole, then a single Internet is therefore preferable to many bifurcated Internets.  But that value is, at least for the moment, losing to the divergent views that we see reflected in the isolationist policies of the United States, the unilateral policies of Europe, BREXIT, and of course China.  Unless and until the economic effects of the Reverse Network Effect are felt, there is no economic incentive for governments to change their direction.

But be careful.  A new consensus may be forming that some might not like: a number of countries seemingly led by Australia are seeking ways to gain access to personal devices such as iPhones for purposes of law enforcement, with or without strong technical protections.  Do you want to be on that Internet, and perhaps as  importantly, will you have a choice?   Perhaps there will eventually be one Internet, and we may not like it.

One thing is certain: I probably won’t be reading the LA Times any time soon.

My views do not necessarily represent those of my employer.


Taxing Bitcoin? IRS gets involved

Once again: is bitcoin a currency, and do currency rules apply? Or is it a capital asset and do those rules apply?

The Wall Street Journal is reporting that a large Bitcoin exchange Coinbase has been served with a so-called “John Doe” warrant in search of those people attempting to evade taxes.  A number of privacy advocates are upset at the breadth of the warrant, because it demands access for an entire broad class of people, and not specific people.

Bitcoin is used for all sorts of nefarious purposes, including online ransoming.  Tax evasion would be the least of its problems.  Were Coinbase a bank, they would be required to inform the federal government of transactions greater than $10,000 or of those individuals believed to be structuring transactions to avoid the $10,000 filing requirement.  These are anti-money laundering provisions that go hand in hand with tax enforcement.

And so my question: if it is wrong for the federal government to make such a demand of Coinbase, is it also wrong of them to make the same demand of banks?  If it is not, then why should Coinbase be treated differently?  And if Coinbase is not treated as a bank, is Bitcoin then not a currency?  If it’s not a currency, should it be treated as a capital asset for taxing purposes?  If that is the case, how would the IRS be able to enforce the reporting requirements associated with assets?

The alternative seems to be to trust people to not launder through Bitcoin.  If history, including recent history, is any measure, that’s a bad idea.  Either way, Bitcoin has already shown that privacy has its downsides.

Why I don’t Eat Beef

Those of you who know me well know that I don’t serve beef at home and do my best to avoid it on the road.  I don’t normally talk about why; most people assume it’s for religious reasons, because I also avoid pork.  But it’s not for religious reasons, nor is it for health reasons.  It’s for the environment.

Back in 1999 the Union of Concerned Scientists came out with a book entitled The Consumer’s Guide to Effective Environmental Choices.  This was based on a report that was roughly entitled, “Paper or Plastic: Who Cares?”  The number 1 thing that UCS said that one could do for the environment was to drive less and buy an efficient car.  The number 2 thing one could do was to eat less meat, and most specifically beef. Well now CNN has an article about just this.  Borrowing a graph:

Carbon footprint of beef

In that article, the author calculates that eating 1.27 lb of beef has the same carbon footprint as a 70 mile drive in a car that gets 21 mpg.  In other words, that number goes up with a more efficient vehicle. He argues that to help arrest the rate of global warming we need to eat less meat.

Even back in 1996, when the first UCS report came out, the one thing I could do for the environment was eat less beef. (I’ve since curtailed my driving, and Christine and I have reduced to 1 vehicle.)  The bad news is that lamb is probably just about as bad (wah!) and I will probably reduce but not eliminate my lamb consumption.

The above graph only looks at carbon footprint, and probably not all of it.  1 lb of beef requires about 1,800 gallons of water.  When I lived in California, this number seemed unsustainably large, even while we were being hit with El Niño after El Niño.  In addition, cattle also cause grazing damage, although it may be possible to mitigate those effects.

Pigs are a different matter.  I stopped eating pork products when several Colorado counties became awash in pig effluvia.  It wasn’t scientific, but I figured I could do my part by simply reducing demand for the animal.

I’ve refrained from writing this sort of article.  This was a personal choice I made, and I really didn’t push it on anyone.  I’m doing so now – just this once – for my daughter, so that she and her generation have just a little less damage from our generation to repair.