Contingency Planning for Stormy Weather

I just spent the last three days snowed in. My firm, Evolve Technologies, is based just across the Potomac River from the National Mall in Washington DC, and for those who missed it we just endured what local bloggers called the “Snowpocalypse.” A weekend of continuous snow left the District and much of the Mid Atlantic region under lockdown as about two feet of snow had us stuck inside. (For those who are reading from farther north and much colder places bemoaning what wimps those south of them are, take away all your plows and snow blowers and tell me how tough you are then. When you only get significant snow once every five years, it doesn’t make much sense to own that stuff. You’d be stuck too if you had no equipment.)

During this time, we enacted our Disaster Recovery plan. The snow started late Friday night, and continued until Sunday morning, and many major services in the area were shut down. Metrorail was only running underground until Monday morning, Metrobus suspended most services through Sunday, and had most bus routes available by Monday.    

However, smaller roads were still a significant problem on Sunday night and into Monday morning, and we were not able to be certain about our staff’s ability to move about, either to get into the office or to our customer sites.   With that in mind, on Sunday night we enacted the part of our DR plan for having our staff work from home, connecting to our network remotely.   We rolled our phones to remote extensions, and were fully operational even without a field presence. We were operational to service our customers Monday morning – those customers who were open themselves, with much of the city closed.

It made me think a lot about our dependence on the connections to the Net. An amazing enabling technology, we are able to work from anywhere, but seemingly unable to work without it. Without the connection, the usefulness of our systems drops to little or nothing. Email doesn’t flow, browsing stops, messaging ends, and VoIP phones have no dial tone.  

For all this dependency, how robust is our planning for this eventuality?   Our servers come with redundancy at many levels: power, network cards, disks, processors.  We can hot swap just about any piece in the hardware – but often times don’t include redundancy in our connection to the outside.   A back hoe in the wrong spot or an outage in a remote central office can bring our operations to their knees. This problem becomes even more profound in a virtualized or cloud world, where resources are only available when connectivity exists.

Connectivity still isn’t as ubiquitous as we often think.  As AT&T and Verizon are reminding us, 3G coverage is not assured in all parts of the US. DSL coverage is still the only option in some areas. While it seems like every Starbucks in the world has Wi-Fi (and McDonald’s is making theirs free!), I still can’t get inexpensive broadband in my office building beyond that served by a telecommunications company. No cable or fiber to our building, within viewing distance of the Lincoln Memorial.
  
As we design systems in the future, the strength of the connectivity will become an ever increasing factor. An afterthought a mere ten years ago, within the next five it will likely be the most critical portion of designing a network infrastructure, with services delivered only from the cloud, public or private. Lack of connectivity will shut down operations ever faster, and multiple connections will become standard. Disaster planning will move from being one of backup procedures and restores to reestablishing connectivity.
 
Have you put Internet connectivity on your list of disaster planning activities, not only for you but for your customers?

I’m going out to get a pizza now. Something, anything, just not stuck at home.  And with my dinner at the local pizzeria – free Wi-Fi.