Unix/Linux Go Back    


War Stories Tell your work related tech stories and share experiences here. Share your successes and failures and other "war stories" in this forum.

Data Centre meets Vacuum Cleaner

War Stories


Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 03-07-2018   -   Original Discussion by gull04
gull04's Unix or Linux Image
gull04 gull04 is offline Forum Advisor  
Registered User
 
Join Date: Dec 2004
Last Activity: 19 April 2018, 3:37 AM EDT
Location: Isle-of-Skye
Posts: 758
Thanks: 19
Thanked 140 Times in 123 Posts
Data Centre meets Vacuum Cleaner

Hi Folks,

I have just spent a couple of days resolving some problems at the remote DR data centre, sorting out the problems caused by the over zealous use of a Vacuum cleaner of all things.

We have a backup server a SUN V480R with a Storedge 3510 and expansion attached which suffered a significant unexplained failure, all tracked back to an ID selector being touched by the nozzle of said vacuum cleaner - it looks like things went as follows over a period of time.

When the array was installed the setup was disks 0-9 were setup as a 10way stripe with disks 10 and 11 as hot standby disks. Over a period of time, a disk in the expansion (disk 3) failed and the and the first available spare (disk 10) built from the surviving mirror.

At this point the situation that existed left us exposed in a way which wasn't really appreciated, in that the one of the arrays had both mirrors of one part of the stripe. That would be the part that had the exposed ID selector switch, the one that the Vacuum Cleaner nozzle could change causing the failure of one whole stripe and one slice of an other stripe. The result as you can imagine was somewhat unpredictable, which is exactly what the Sun manual for the array says.

To add insult to injury the contents of the 3510 array, was the Legato Networker Backup Catalogue from the 24 drive ATL - making the recovery somewhat awkward.

What's the point of the story - don't let some idiot into a data centre with a Vacuum Cleaner.

Regards

Gull04
Sponsored Links
    #2  
Old Unix and Linux 03-07-2018   -   Original Discussion by gull04
joeyg's Unix or Linux Image
joeyg joeyg is offline Forum Staff  
modérateur
 
Join Date: Dec 2007
Last Activity: 19 April 2018, 10:18 AM EDT
Location: Within two miles of a Dunkin donuts.
Posts: 2,480
Thanks: 142
Thanked 209 Times in 184 Posts
Or, moral of the story, make sure buttons have the cover in place to prevent accidental bumping.
The Following User Says Thank You to joeyg For This Useful Post:
Neo (03-08-2018)
Sponsored Links
    #3  
Old Unix and Linux 03-07-2018   -   Original Discussion by gull04
gull04's Unix or Linux Image
gull04 gull04 is offline Forum Advisor  
Registered User
 
Join Date: Dec 2004
Last Activity: 19 April 2018, 3:37 AM EDT
Location: Isle-of-Skye
Posts: 758
Thanks: 19
Thanked 140 Times in 123 Posts
Hi Joeyg,

You are of course right, however I was advised that this solution was built from disposed of equipment as a short term solution in 2006 and the cover for the ID switch was left in the skip.

Not much help when you are performing the headless chicken dance in the data centre I know, but I'm told that ensuring this won't happen again is now a priority.

I think that they'll probably try and get a cover off aBay, this will probably mean that it will be in service as a temporary solution for an other 12 years.

Regards

Gull04
    #4  
Old Unix and Linux 03-08-2018   -   Original Discussion by gull04
hicksd8's Unix or Linux Image
hicksd8 hicksd8 is offline Forum Staff  
Moderator
 
Join Date: Feb 2012
Last Activity: 21 April 2018, 4:51 PM EDT
Location: Devon, UK
Posts: 1,831
Thanks: 272
Thanked 481 Times in 409 Posts
I had a similar experience a few years ago when an additional storage unit was urgently added to a system. The cleaner entered the highly secure computer room and unplugged the new unit to plug in her vacuum cleaner. When challenged whether she had switched off the new storage she simply said, "Well yes, I've always used that socket".
Sponsored Links
    #5  
Old Unix and Linux 03-08-2018   -   Original Discussion by gull04
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is offline Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 21 April 2018, 6:35 PM EDT
Location: San Jose, CA, USA
Posts: 11,208
Thanks: 633
Thanked 3,903 Times in 3,341 Posts
Many years ago, while I was working at Sun Microsystems, Inc. on adding POSIX-conformance into SunOS 4.1, I was making changes to the OS and utilities during the day and running complete builds of the system overnight (starting a build just before I left work in the evening). This worked fine for several weeks until one Monday night when the system died at about 8:30pm killing the build-in-progress. There was no core dump, no indication of any hardware problems, and restarting the build when I got to work Tuesday morning completed normally (taking a little over three hours to complete).

The same thing happened the next three days in a row, with the system always dying sometime between 8:20pm and 8:40pm.

I decided to stay at work late Friday evening to see if I could figure out what was causing the crashes. I went to the bathroom at about 7pm so I would be sure that I could be at my computer by 7:15pm and would be able to stay there until I found out why my computer was dying every night. When I got back to my office five minutes later, I found that my computer had been unplugged by a member of the cleaning crew so he could plug in the vacuum cleaner they used to clean the hallway and offices where my office was located. (The way my office was arranged left a power strip close to the hallway while other offices in my area had their power strips in less accessible locations.) I unplugged the sweeper, plugged my computer back in, and waited for the cleaning guy to come back to my office.

I found out that:
  1. the cleaning crew comes in an hour and a half earlier on Friday that they did Monday through Thursday,
  2. they got a new vacuum cleaner on Monday with a cord that wouldn't reach from the plugs in the conference room to the other end of the building, and
  3. unplugging a computer is inconsequential to a janitor if doing so allows him to sweep the carpets at the end of the hallway.
Sponsored Links
    #6  
Old Unix and Linux 03-08-2018   -   Original Discussion by gull04
Neo's Unix or Linux Image
Neo Neo is offline Forum Staff  
Administrator
 
Join Date: Sep 2000
Last Activity: 21 April 2018, 7:02 AM EDT
Location: Asia pacific region
Posts: 14,365
Thanks: 989
Thanked 1,365 Times in 650 Posts
Quote:
Originally Posted by gull04 View Post

What's the point of the story - don't let some idiot into a data centre with a Vacuum Cleaner.
It's not a good idea to call cleaners and others "idiots" as their job is not to develop IT policy or data center cleaning procedures. Their job is to follow procedures for how to clean a data center and to be supervised when cleaning, especially if the data center has critical systems.

Hence, the problem is not "the cleaner" but the management and the administrators of the data center who seemingly did not put proper cleaning procedures, controls and supervision (if required) in place to insure that anyone who worked in the data center understood the proper procedures on how to clean and vacuum in a data center.

Even in my home, I don't permit our biweekly housekeeper to clean and vacuum around my key IT devices unless I am there.

So, for a data center with critical data and business systems, the responsibility falls on the data center management and administrators, not the "cleaners," to insure all personnel working in the data center understand how to do their job in a careful and responsible way.

FYI, forum rules do state:

Quote:
(1) No flames, shouting (all caps), sarcasm, bullying, profanity or arrogant posts.

(2) No negative comments about others or impolite remarks. ....
When we founded these forums decades ago, we have always been careful to insure that everyone is treated with respect (unlike many other site, where anything goes and name calling is normal); and although we may disagree, we do not call each other names.

We would never call my housekeeper a bad name, for example. She is nearly blind and cleans the mirrors and glass with her face a few inches away, and if she needs to be supervised, we give her gentle instructions.

I have worked in many data centers, some very critical and highly secure. Data centers should have clear procedures for how to clean, how to vacuum, where to plug in appliances, how close to get to the electrical gear and more.

It is typically the fault of data center managers when these types of incidents happen; as most are simply too distracted or not "very detailed oriented" to create housekeeping procedures that insure safe housekeeping operations in a data center.

Honestly, who in their "right data center management mind," who runs a data center, would have a cleaning person or team just "plug in" wherever they wanted or to clean near wires and racks without clear procedures and supervision if necessary?
The Following User Says Thank You to Neo For This Useful Post:
RudiC (03-08-2018)
Sponsored Links
    #7  
Old Unix and Linux 03-08-2018   -   Original Discussion by gull04
Neo's Unix or Linux Image
Neo Neo is offline Forum Staff  
Administrator
 
Join Date: Sep 2000
Last Activity: 21 April 2018, 7:02 AM EDT
Location: Asia pacific region
Posts: 14,365
Thanks: 989
Thanked 1,365 Times in 650 Posts
Quote:
Originally Posted by joeyg View Post
Or, moral of the story, make sure buttons have the cover in place to prevent accidental bumping.
Yes, and to have proper procedures on how to clean, where to plugin, how close to get to wires, gear, etc. and to have proper supervision if the data center has critical servers that perform critical business tasks and functions.

We call these "controls"..... there are "physical controls" (button covers, LOL), "technical controls" (backup UPS systems, etc) and "administrative controls" (procedures, supervision... )

It's all pretty basic "IT security" Linux
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Maybe a cleaner way to generate a file? crimso Shell Programming and Scripting 2 07-13-2017 06:00 PM
A cleaner way to rearrange column LMHmedchem Shell Programming and Scripting 8 11-30-2016 05:36 AM
Cleaner method for this if-then statement? lupin..the..3rd Shell Programming and Scripting 2 05-25-2012 02:04 PM



All times are GMT -4. The time now is 07:24 PM.