Back in the days when I was happily spending time on the operations floor in computing centers, we always observed that the greatest security threats to our systems were well-intended operators who make simple mistakes.  No hacker or criminal ever brought down a network like the bored network guy on the late shift who decided to upload a new version of the Cisco IOS on all the routers of a global ISP without testing first.  A bug in the IOS release caused every router go down, one-by-one.  I remember being called into work to fix the problem (had to send people on-site to reload the IOS at each location) and then spending many hours writing code and wrapper scripts to record every keystroke on operational systems by operators, circa 1994.

Over and over we see the unsexy truth of self-inflicted denial-of-service attacks, as we often refer to  these incidents.   The focus by IT security professionals is often on small, almost trivial exploits; while the major problems are always by a well-intended operator we are paying to do the work.  

It was not long ago where Google had the same problem.  If you recall (I think I posted something here), one of Google's employees uploaded a "/" (forward slash) as a malicious site in their "super filter".  This very small error caused the entire Internet to be inaccessible via Google for around a hour (or a little less, as I recall).  With so many companies depending on Google Adsense for revenue (last count Google owned over 70% of the search market), this was a substantial loss for countless businesses (but most of all, Google).

So, it should come as no surprise that in our rush to outsource services to "the clouds" we forget that an operational error in "the cloud we rely on" by a cloud service provider is more-likely to cause a service disruption than a hacker hackin'-the-clouds. Never-the-less we read cautious reports on cloud hacking, not cloud operational issues.

As a case-in-point, one of the "web-sites-under-our-wings" decided to experiment with Amazon CloudFront to deliver static content.   We were focused on speed of delivery, latency and the user experience.  We were "happy campers" and advocating Amazon AWS as the next great coming of technology.  All seemed fine.  Then, the objects stopped raining from our cloud.   Our objects were not served anymore.  The web site was adversely effected because the AWS CloudFront content delivery network (CDN) stop serving content.  Lucky for us, we had only moved over small static graphical objects, not Javascript or other operational web code. We tend to work in baby steps, lessons learned from the freezing operational floors of computing centers.

What happened?

A rule-based system by our (once favorite) cloud provider flagged the account as "suspect" and, without warning, email notice, phone call or SMS message, shut down our cloud services.   No more content.  Service denied.  Our cloud was dry.  There was no hacker, criminal or other troublesome person to cause damage, no fraudster or bad guy, it was the cloud provider we paid to take care of these things - a well-intended series of operational errors.

This short story serves as a reminder to all IT security professionals about the hidden dangers in cloud services and how operational issues by well-intended folks we trust are generally the greatest risk to  IT systems and system security.




More...