Data Centre meets Vacuum Cleaner


 
Thread Tools Search this Thread
The Lounge War Stories Data Centre meets Vacuum Cleaner
# 1  
Old 03-07-2018
Data Centre meets Vacuum Cleaner

Hi Folks,

I have just spent a couple of days resolving some problems at the remote DR data centre, sorting out the problems caused by the over zealous use of a Vacuum cleaner of all things.

We have a backup server a SUN V480R with a Storedge 3510 and expansion attached which suffered a significant unexplained failure, all tracked back to an ID selector being touched by the nozzle of said vacuum cleaner - it looks like things went as follows over a period of time.

When the array was installed the setup was disks 0-9 were setup as a 10way stripe with disks 10 and 11 as hot standby disks. Over a period of time, a disk in the expansion (disk 3) failed and the and the first available spare (disk 10) built from the surviving mirror.

At this point the situation that existed left us exposed in a way which wasn't really appreciated, in that the one of the arrays had both mirrors of one part of the stripe. That would be the part that had the exposed ID selector switch, the one that the Vacuum Cleaner nozzle could change causing the failure of one whole stripe and one slice of an other stripe. The result as you can imagine was somewhat unpredictable, which is exactly what the Sun manual for the array says.

To add insult to injury the contents of the 3510 array, was the Legato Networker Backup Catalogue from the 24 drive ATL - making the recovery somewhat awkward.

What's the point of the story - don't let some idiot into a data centre with a Vacuum Cleaner.

Regards

Gull04
# 2  
Old 03-07-2018
Or, moral of the story, make sure buttons have the cover in place to prevent accidental bumping.
This User Gave Thanks to joeyg For This Post:
# 3  
Old 03-07-2018
Hi Joeyg,

You are of course right, however I was advised that this solution was built from disposed of equipment as a short term solution in 2006 and the cover for the ID switch was left in the skip.

Not much help when you are performing the headless chicken dance in the data centre I know, but I'm told that ensuring this won't happen again is now a priority.

I think that they'll probably try and get a cover off aBay, this will probably mean that it will be in service as a temporary solution for an other 12 years.

Regards

Gull04
# 4  
Old 03-08-2018
I had a similar experience a few years ago when an additional storage unit was urgently added to a system. The cleaner entered the highly secure computer room and unplugged the new unit to plug in her vacuum cleaner. When challenged whether she had switched off the new storage she simply said, "Well yes, I've always used that socket".
# 5  
Old 03-08-2018
Many years ago, while I was working at Sun Microsystems, Inc. on adding POSIX-conformance into SunOS 4.1, I was making changes to the OS and utilities during the day and running complete builds of the system overnight (starting a build just before I left work in the evening). This worked fine for several weeks until one Monday night when the system died at about 8:30pm killing the build-in-progress. There was no core dump, no indication of any hardware problems, and restarting the build when I got to work Tuesday morning completed normally (taking a little over three hours to complete).

The same thing happened the next three days in a row, with the system always dying sometime between 8:20pm and 8:40pm.

I decided to stay at work late Friday evening to see if I could figure out what was causing the crashes. I went to the bathroom at about 7pm so I would be sure that I could be at my computer by 7:15pm and would be able to stay there until I found out why my computer was dying every night. When I got back to my office five minutes later, I found that my computer had been unplugged by a member of the cleaning crew so he could plug in the vacuum cleaner they used to clean the hallway and offices where my office was located. (The way my office was arranged left a power strip close to the hallway while other offices in my area had their power strips in less accessible locations.) I unplugged the sweeper, plugged my computer back in, and waited for the cleaning guy to come back to my office.

I found out that:
  1. the cleaning crew comes in an hour and a half earlier on Friday that they did Monday through Thursday,
  2. they got a new vacuum cleaner on Monday with a cord that wouldn't reach from the plugs in the conference room to the other end of the building, and
  3. unplugging a computer is inconsequential to a janitor if doing so allows him to sweep the carpets at the end of the hallway.
# 6  
Old 03-08-2018
Quote:
Originally Posted by gull04

What's the point of the story - don't let some idiot into a data centre with a Vacuum Cleaner.
It's not a good idea to call cleaners and others "idiots" as their job is not to develop IT policy or data center cleaning procedures. Their job is to follow procedures for how to clean a data center and to be supervised when cleaning, especially if the data center has critical systems.

Hence, the problem is not "the cleaner" but the management and the administrators of the data center who seemingly did not put proper cleaning procedures, controls and supervision (if required) in place to insure that anyone who worked in the data center understood the proper procedures on how to clean and vacuum in a data center.

Even in my home, I don't permit our biweekly housekeeper to clean and vacuum around my key IT devices unless I am there.

So, for a data center with critical data and business systems, the responsibility falls on the data center management and administrators, not the "cleaners," to insure all personnel working in the data center understand how to do their job in a careful and responsible way.

FYI, forum rules do state:

Quote:
(1) No flames, shouting (all caps), sarcasm, bullying, profanity or arrogant posts.

(2) No negative comments about others or impolite remarks. ....
When we founded these forums decades ago, we have always been careful to insure that everyone is treated with respect (unlike many other site, where anything goes and name calling is normal); and although we may disagree, we do not call each other names.

We would never call my housekeeper a bad name, for example. She is nearly blind and cleans the mirrors and glass with her face a few inches away, and if she needs to be supervised, we give her gentle instructions.

I have worked in many data centers, some very critical and highly secure. Data centers should have clear procedures for how to clean, how to vacuum, where to plug in appliances, how close to get to the electrical gear and more.

It is typically the fault of data center managers when these types of incidents happen; as most are simply too distracted or not "very detailed oriented" to create housekeeping procedures that insure safe housekeeping operations in a data center.

Honestly, who in their "right data center management mind," who runs a data center, would have a cleaning person or team just "plug in" wherever they wanted or to clean near wires and racks without clear procedures and supervision if necessary?
This User Gave Thanks to Neo For This Post:
# 7  
Old 03-08-2018
Quote:
Originally Posted by joeyg
Or, moral of the story, make sure buttons have the cover in place to prevent accidental bumping.
Yes, and to have proper procedures on how to clean, where to plugin, how close to get to wires, gear, etc. and to have proper supervision if the data center has critical servers that perform critical business tasks and functions.

We call these "controls"..... there are "physical controls" (button covers, LOL), "technical controls" (backup UPS systems, etc) and "administrative controls" (procedures, supervision... )

It's all pretty basic "IT security" Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

6 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Maybe a cleaner way to generate a file?

greetings, to be clear, i have a solution but i'm wondering if anyone has a cleaner way to accomplish the following: the variable: LSB_MCPU_HOSTS='t70c7n120 16 t70c7n121 16 t70c7n122 16 t70c7n123 16 t70c7n124 16 t70c7n125 16 t70c7n126 16 t70c7n127 16 t70c7n128 16 t70c7n129 16 t70c7n130 16... (2 Replies)
Discussion started by: crimso
2 Replies

2. Shell Programming and Scripting

A cleaner way to rearrange column

Hello, I have some tab delimited text data, index name chg_p chg_m 1 name,1 1 0 2 name,2 1 1 3 name,3 1 0 4 name,4 1 0 5 name,5 1 1 I need to duplicate the "index" column, call it "id" and insert it after the... (8 Replies)
Discussion started by: LMHmedchem
8 Replies

3. Shell Programming and Scripting

Cleaner way to use shell variable in awk /X/,/Y/ syntax?

$ cat data Do NOT print me START_MARKER Print Me END_MARKER Do NOT print me $ cat awk.sh start=START_MARKER end=END_MARKER echo; echo Is this ugly syntax the only way? awk '/'"$start"'/,/'"$end"'/ { print }' data echo; echo Is there some modification of this that would work? awk... (2 Replies)
Discussion started by: hanson44
2 Replies

4. Shell Programming and Scripting

Cleaner method for this if-then statement?

I have a script that runs once per month. It performs a certain task ONLY if the month is January, April, July, or October. MONTH=`date +%m` if || || || ; then do something else do a different thing fi Is there a neater way of doing it than my four separate "or" comparisons? That... (2 Replies)
Discussion started by: lupin..the..3rd
2 Replies

5. Solaris

Sun Management Centre

Hi Guys, Does anyone ever use Sun MC before? If yes, can u share with me how to do the installation. Thanks (1 Reply)
Discussion started by: raziayub
1 Replies

6. News, Links, Events and Announcements

The HP-UX Porting and Archive Centre

If you're looking for an easy source of free software for HP-UX, look no further. A consortium of major universities and HP user groups have banded together and operate what is simply the finest free software site in existence. The world's best HP-UX software engineers have ported each package... (0 Replies)
Discussion started by: Perderabo
0 Replies
Login or Register to Ask a Question