War Stories

Tell your work related tech stories and share experiences here. Share your successes and failures and other "war stories" in this forum.

Wrong Array...

👤 Login to reply

    #1  
Old 05-30-2018
xenix xenix is offline
Registered User
 
Join Date: Feb 2009
Last Activity: 30 May 2018, 4:43 PM EDT
Posts: 4
Thanks: 1
Thanked 1 Time in 1 Post
Wrong Array...

Back in my early days, just starting at my first job where I was doing 100% UNIX...

We had a mixture of Solaris, AIX, and HPUX systems. Our most critical systems (Logistics) was running on a bunch of large HP 9000 servers. I was tasked with decommisoning one of the servers.

It was one of those arrangements where there was three cabinets in a row. One large HP 9000 on each side, and the middle was used to store the boot disks, tape drives, optical drives, etc.

Well, long story short.. I hit the power button on the wrong storage cabinet. Did a few other things... Casually walked back to my desk. It was a good 20 minutes later.

All of a sudden, people are screaming.. PROD IS DOWN.. PROD IS DOWN. I thought, great.. Another fun day....

A few minutes later, it dawned on me that the drives for the production server was in the same rack that I was decommissioning. So, I walked back down to the data center to verify... Sure enough. I hit the wrong button.

I turned the array back on, walked back to my desk, and could now login again, and everything was fine. To my surprise.. There was no fallout from this. This system just completely hung... once the storage was turned back on, it just kept on going where it left off.
The Following User Says Thank You to xenix For This Useful Post:
dodona (06-02-2018)
Sponsored Links
    #2  
Old 05-31-2018
vbe's Unix or Linux Image
vbe vbe is offline Forum Staff  
Moderator
 
Join Date: Sep 2005
Last Activity: 16 July 2018, 9:56 AM EDT
Location: Switzerland - GE
Posts: 6,550
Thanks: 311
Thanked 564 Times in 526 Posts

These old HP were really great boxes...
Sponsored Links
    #3  
Old 05-31-2018
vbe's Unix or Linux Image
vbe vbe is offline Forum Staff  
Moderator
 
Join Date: Sep 2005
Last Activity: 16 July 2018, 9:56 AM EDT
Location: Switzerland - GE
Posts: 6,550
Thanks: 311
Thanked 564 Times in 526 Posts
I remembered having no other alternative than to press the power of button on a K class, because I was called on a sunday between Xmas and NewYear, and all the expected batches produced nothing, when I asked what output or why only answer I got was the machine only responds to ping, only this was a very sensible machine, something ever happens and heads falls...
I went in the white room just to realise the box was out of control, you cant even connect from the console plugged directly to it.
So crossed my fingers and off..
Then just as you I had lived a long moment of solitude... to finally see the box up and running, even old Oracle was up...
As I knew I would be asked what happened , I went through all the activity logs and after an hour searching the batches crons etc... Found out it was a stupid developper that sent corrupted prints, no seeing them coming to the printer executed a few ( many...) times the his print job without looking the output ( queue down...) decided as it was friday to leave for Xmas holiday...
The sunday I was called was 10 days after...
    #4  
Old 06-13-2018
gull04 gull04 is offline Forum Advisor  
Registered User
 
Join Date: Dec 2004
Last Activity: 10 July 2018, 5:47 AM EDT
Location: Isle-of-Skye
Posts: 765
Thanks: 21
Thanked 144 Times in 126 Posts
Hi Guys,

This also goes back around 25 years when in the space of two weeks I was involved in two power switch incidents in the space of two weeks. The first incident was on a trading floor in the Minories in London, on a Data General Aviion system an 8550 I think. Anyway, while getting ready for a very short outage on the system - I found out that the Grey import that this company had sneaked onto the contract had a power interlock on one of the Zeus fasteners on the rear panel of the system.

The other incident was much more humerous from my perspective at least.

Whilst going past a comms room in a fairly deserted part of the building, my attention was attracted by shouts for help. I went into the comms room to see one of the software developers perched cross legged on a swivel chair infront of a big (for it's time) Data General Clariion Disk Array (60 Bay I think), when I asked what the problem was I did have to struggle to keep a straigh face.

The developer had spun round on his chair and the pointed toe of his shoe had depressed the power button on the array, he had had the common sense to not move so the button hadn't popped out and dropped the array power - but he'd been there for about an hour and was now suffering cramps.

Fortunately there was enough space to get a finger in and hold the button down, whilst we got a clean shutdown of the application and the system. However the developer had to be helped out of the swivel chair by others as we just left him there while verything was brought down.

The Clariion power button was modified with a Domestos bottle top, which had a hole cut in it being Araldited over the offending button.

Ah the good old days.

Regards

Gull04
Sponsored Links
👤 Login to reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Bash arrays: rebin/interpolate smaller array to large array f77hack Shell Programming and Scripting 2 02-13-2016 04:34 PM
Bash 3.2 - Array / Regex - IF 3rd member in array ends in 5 digits then do somthing... briandanielz Shell Programming and Scripting 4 08-26-2013 01:28 PM
Why result is wrong here ? whether break statement is wrong ? Akshay Hegde Shell Programming and Scripting 2 03-10-2013 11:31 PM
PERL : Read an array and write to another array with intial string pattern checks irudayaraj Shell Programming and Scripting 2 11-05-2011 08:11 AM
Creating an array to hold posix thread ids: Only dynamic array works kmehta Programming 4 09-21-2008 08:24 PM



All times are GMT -4. The time now is 07:25 PM.

Unix & Linux Forums Content Copyrightę1993-2018. All Rights Reserved.
×
UNIX.COM Login
Username:
Password:  
Show Password





Not a Forum Member?
Forgot Password?