A bit off my usual selection of topics, but an article by Bruce Perens about a cyber-attack on Morgan Hill, a small city in northern California caught my attention:
Just after midnight on Thursday, April 9, unidentified attackers climbed down four manholes serving the Northern California city of Morgan Hill and cut eight fiber cables in what appears to have been an organized attack on the electronic infrastructure of an American city. Its implications, though startling, have gone almost un-reported.
That attack demonstrated a severe fault in American infrastructure: its centralization. The city of Morgan Hill and parts of three counties lost 911 service, cellular mobile telephone communications, land-line telephone, DSL internet and private networks, central station fire and burglar alarms, ATMs, credit card terminals, and monitoring of critical utilities. In addition, resources that should not have failed, like the local hospital’s internal computer network, proved to be dependent on external resources, leaving the hospital with a “paper system” for the day.
Read the full article for details. What struck me was the following question: is the vulnerability a sign of our being too connected, or not connected enough?
Perens notes how the attack demonstrated unnecessary dependence on connectivity, e.g., in the hospital’s internal network. But in an era of cloud computing, such dependencies on external services are becoming more common. It’s certainly easy to read a lesson in this experience that our systems should perform better in disconnected mode.
But the other lesson may be that it was too easy to disconnect the city. Should cutting eight cables be enough to disconnect over 50,000 people (not just in Morgan Hill, but also in nearby counties)? Should we instead be trying to achieve the fault tolerance of a mesh network? I’m no networking expert, so I don’t know whether, aside from the fixed costs associated with overhauling network infrastructure, mesh networking is efficient enough to replace our current architecture.
In any case, it was a sobering article. I’d like to believe it would be much harder to perpretrate a similar attack on my somewhat larger home town. But, more importantly, I’d like to think we are building a more reliable network of dependencies that exploits the extensive research on the subject.
The Noisy Channel 
6 replies on “Too Connected, Or Not Connected Enough?”
This event generated considerable discussion on Dave Farber’s Interesting People list (e.g. http://www.interesting-people.org/archives/interesting-people/200904/msg00100.html )
As said there, part of the vulnerability stems from the current Internet billing model and its resultant ISP bias towards internal vs distributed network resources — their own network infrastructure is expensed based on depreciation of the sunk cost, while peering with another carrier is an operating expense payable either in “cash money” or commitments of reciprocal transit (and carriers hate to commit resources to competitors.)
As for cloud computations, individual data centers have very high internal bandwidths and very low latencies, making it relatively easy to migrate computation from one resource to another. But, once that move spans physical locations (e.g. for failure recovery, or just to get a better resource/cost deal) your application’s data and stored state suddenly seems to have a very high inertia. It takes time to move Terabytes and someone’s billing you by the Gbyte for that I/O!
LikeLike
One thing we all (I live in Morgan Hill) learned very quickly is that an emergency stash of *cash* is essential. With the trunk line down, there were no operating ATM’s, or even debit card processing at retailers (grocery, gas stations, etc.). Also interesting is how all forms of communication were affected; landline, wireless, Cable, DSL, etc. I use an AirCard for redundency, but am now forced to rethink that selection.
btw – Morgan Hill (pop. 34,000) was just a sidelight in this outage, that affected around 100,000 people.
LikeLike
Wow, I’d gotten a 52K number from this article, but rereading I see that was just the estimated number of Verizon land lines. At 100K, that feels like disproportionate impact for the size of attack.
LikeLike
crazy stuff. It kind of relates to the discussion we had in Boston though huh. Imagine if they cut off a Google server farm instead of Morgan Hill. We’d all be lost! heh.
Im definately pro re-population rather than archival of digital artefacts.
LikeLike
Daniel, to your point about our home town, don’t forget about this event (http://en.wikipedia.org/wiki/2003_North_America_blackout) that happened only 5 and a half years ago. Although it wasn’t triggered by sabotage, our infrastructure is in desperate need for additional resiliency and modernization.
LikeLike
I’m sympathetic to planners who have to confront the enormous costs of replacing a legacy infrastructure like our power grid. But I worry when we make fresh decisions to build critical functions that depend entirely on external services with no backup plan if someone cuts a handful of wires. You’d think we’d have learned our lesson by now.
LikeLike