Wrong Array...


 
Thread Tools Search this Thread
The Lounge War Stories Wrong Array...
# 1  
Old 05-30-2018
Wrong Array...

Back in my early days, just starting at my first job where I was doing 100% UNIX...

We had a mixture of Solaris, AIX, and HPUX systems. Our most critical systems (Logistics) was running on a bunch of large HP 9000 servers. I was tasked with decommisoning one of the servers.

It was one of those arrangements where there was three cabinets in a row. One large HP 9000 on each side, and the middle was used to store the boot disks, tape drives, optical drives, etc.

Well, long story short.. I hit the power button on the wrong storage cabinet. Did a few other things... Casually walked back to my desk. It was a good 20 minutes later.

All of a sudden, people are screaming.. PROD IS DOWN.. PROD IS DOWN. I thought, great.. Another fun day....

A few minutes later, it dawned on me that the drives for the production server was in the same rack that I was decommissioning. So, I walked back down to the data center to verify... Sure enough. I hit the wrong button.

I turned the array back on, walked back to my desk, and could now login again, and everything was fine. To my surprise.. There was no fallout from this. This system just completely hung... once the storage was turned back on, it just kept on going where it left off.
These 3 Users Gave Thanks to xenix For This Post:
# 2  
Old 05-31-2018
Smilie
These old HP were really great boxes...
# 3  
Old 05-31-2018
I remembered having no other alternative than to press the power of button on a K class, because I was called on a sunday between Xmas and NewYear, and all the expected batches produced nothing, when I asked what output or why only answer I got was the machine only responds to ping, only this was a very sensible machine, something ever happens and heads falls...
I went in the white room just to realise the box was out of control, you cant even connect from the console plugged directly to it.
So crossed my fingers and off..
Then just as you I had lived a long moment of solitude... to finally see the box up and running, even old Oracle was up...
As I knew I would be asked what happened , I went through all the activity logs and after an hour searching the batches crons etc... Found out it was a stupid developper that sent corrupted prints, no seeing them coming to the printer executed a few ( many...) times the his print job without looking the output ( queue down...) decided as it was friday to leave for Xmas holiday...
The sunday I was called was 10 days after...
This User Gave Thanks to vbe For This Post:
# 4  
Old 06-13-2018
Hi Guys,

This also goes back around 25 years when in the space of two weeks I was involved in two power switch incidents in the space of two weeks. The first incident was on a trading floor in the Minories in London, on a Data General Aviion system an 8550 I think. Anyway, while getting ready for a very short outage on the system - I found out that the Grey import that this company had sneaked onto the contract had a power interlock on one of the Zeus fasteners on the rear panel of the system.

The other incident was much more humerous from my perspective at least.

Whilst going past a comms room in a fairly deserted part of the building, my attention was attracted by shouts for help. I went into the comms room to see one of the software developers perched cross legged on a swivel chair infront of a big (for it's time) Data General Clariion Disk Array (60 Bay I think), when I asked what the problem was I did have to struggle to keep a straigh face.

The developer had spun round on his chair and the pointed toe of his shoe had depressed the power button on the array, he had had the common sense to not move so the button hadn't popped out and dropped the array power - but he'd been there for about an hour and was now suffering cramps.

Fortunately there was enough space to get a finger in and hold the button down, whilst we got a clean shutdown of the application and the system. However the developer had to be helped out of the swivel chair by others as we just left him there while verything was brought down.

The Clariion power button was modified with a Domestos bottle top, which had a hole cut in it being Araldited over the offending button.

Ah the good old days.

Regards

Gull04
This User Gave Thanks to gull04 For This Post:
# 5  
Old 12-05-2018
Here is my personal "power-loss-story" and it has nothing to do with a power-button:

Back around 2000 i was working for a data center, which - nothing really unusual - had a large ~1000hp diesel generator to back up its UPS. This diesel was tested every year once by switching it on for 2 minutes. Everything worked fine.

Now, what happens if you run such a system for two minutes a year? Disaster struck and showed to the responsible people exactly what happens in such a case: during a heavy thunderstorm the nearby transformer station (or, specifically, its large 6m-diameter capacitor was hit by a lightning. The whole area was suddenly without electrical current and the remains of the capacitor lay across half of the street in pieces not bigger than a few centimeters each.

First, our UPS kicked in - as planned. Then, when the UPS batteries went low, the diesel started. It ran for about 5-6 minutes, then stuck with a piston seizure because running keeps such a machine lubed. When it doesn't run all the bearings eventually run dry of oil. So far, so bad and our data center stood without electrical power.

What makes the story interesting, though, is: at the same time adear friend of mine (shockneck) worked for another data center across the street. So, two days after the incident, we sat in a cafe commiserating about life as an admin. And he told me, they had a UPS, backed by a diesel. And because they knew that engines have to run every now and then to remain properly lubed they switched it on every month and let it run for 10 minutes. Alas, their diesel gave up a few minutes after starting. You know, when letting it run so often you need to refill the tank now and then.....

bakunin
These 4 Users Gave Thanks to bakunin For This Post:
# 6  
Old 12-05-2018
I have similar story regarding UPS, diesel motor and not so automatic power switch Smilie
Needless to say i got a call in 3 in the morning, jumped on my bike, drove to DC to see sweaty engineers with laptops everywhere.
The senior management was also there, breading behind our necks - which did not speed up the process, rather slowed it down.

So i joined the party and started breaking sweat, to find some NFS clustered UFS filesystems needed fsck (big filesystems), but the clusterware timeout happend due to lasting to long.
Then when timeout occurred, the package was switched to another node, started fsck again ... nightmare loop to say.

Was happy since i prepared ZFS some days ago and clustered to do migration to new storage, so i offlined and unmanaged the package in question, mounted the UFS in read only and ran couple of rsyncs.
By the time everything else got up, everything but couple of TB filesystems mostly used for archive, were done.
The NFS was accessible immediately so apps can write, but data was continuously pouring for the rest of the day, and checksuming in the days that followed Smilie

It went pretty well (no major consequence in general, except downtime), for which part i blame enterprise storage batteries which held the data in cache and flushed it to disks when powered back on again.
Among other things, since this was a extremely mixed environment hardware and operating system wise.

Regards
Peasant.
This User Gave Thanks to Peasant For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash arrays: rebin/interpolate smaller array to large array

hello, i need a bit of help on how to do this effectively in bash without a lot of extra looping or massive switch/case i have a long array of M elements and a short array of N elements, so M > N always. M is not a multiple of N. for case 1, I want to stretch N to fit M arrayHuge H = (... (2 Replies)
Discussion started by: f77hack
2 Replies

2. Shell Programming and Scripting

Bash 3.2 - Array / Regex - IF 3rd member in array ends in 5 digits then do somthing...

Trying to do some control flow parsing based on the index postion of an array member. Here is the pseudo code I am trying to write in (preferably in pure bash) where possible. I am thinking regex with do the trick, but need a little help. pesudo code if == ENDSINFIVEINTS ]]; then do... (4 Replies)
Discussion started by: briandanielz
4 Replies

3. Shell Programming and Scripting

Why result is wrong here ? whether break statement is wrong ?

Hi ! all I am just trying to check range in my datafile pls tell me why its resulting wrong admin@IEEE:~/Desktop$ cat test.txt 0 28.4 5 28.4 10 28.4 15 28.5 20 28.5 25 28.6 30 28.6 35 28.7 40 28.7 45 28.7 50 28.8 55 28.8 60 28.8 65 28.1... (2 Replies)
Discussion started by: Akshay Hegde
2 Replies

4. Shell Programming and Scripting

PERL : Read an array and write to another array with intial string pattern checks

I have an array and two variables as below, I need to check if $datevar is present in $filename. If so, i need to replace $filename with the values in the array. I need the output inside an ARRAY How can this be done. Any help will be appreciated. Thanks in advance. (2 Replies)
Discussion started by: irudayaraj
2 Replies

5. Shell Programming and Scripting

Array in Perl - Detect several file to be in one array

Hi everyone I have one question about using array in perl. let say I have several log file in one folder.. example test1.log test2.log test3.log and the list goes on.. how to make an array for this file? It suppose to detect log file in the current directory and all the log file will... (3 Replies)
Discussion started by: sayachop
3 Replies

6. Shell Programming and Scripting

Store all the passed arguments in an array and display the array

Hi I want to write a script which store all the parameters passed to the script into an array. Once it is stored I want scan through the array and and delete those files for last month present inside the directory. The files in directory is appneded with YYYY_MM_DD. I want to know how can I... (3 Replies)
Discussion started by: dgmm
3 Replies

7. Shell Programming and Scripting

perl, put one array into many array when field is equal to sth

Hi Everyone, #!/usr/bin/perl use strict; use warnings; my @test=("a;b;qqq;c;d","a;b;ggg;c;d","a;b;qqq;c;d"); would like to split the @test array into two array: @test1=(("a;b;qqq;c;d","a;b;qqq;c;d"); and @test2=("a;b;ggg;c;d"); means search for 3rd filed. Thanks find the... (0 Replies)
Discussion started by: jimmy_y
0 Replies

8. Shell Programming and Scripting

PHP: Search Multi-Dimensional(nested) array and export values of currenly worked on array.

Hi All, I'm writing a nagios check that will see if our ldap servers are in sync... I got the status data into a nested array, I would like to search key of each array and if "OK" is NOT present, echo other key=>values in the current array to a variable so...eg...let take the single array... (1 Reply)
Discussion started by: zeekblack
1 Replies

9. Programming

Creating an array to hold posix thread ids: Only dynamic array works

I am facing a strange error while creating posix threads: Given below are two snippets of code, the first one works whereas the second one gives a garbage value in the output. Snippet 1 This works: -------------- int *threadids; threadids = (int *) malloc (num_threads * sizeof(int)); ... (4 Replies)
Discussion started by: kmehta
4 Replies

10. Shell Programming and Scripting

create array holding characters from sring then echo array.

Hi, I wish to store $string1 in $string1array a character in each array element. Then i wish to echo the entire array to the screen so that it reads as the normal string again. I have been trying with the code below but does not work. Please help... To put string into array: ... (5 Replies)
Discussion started by: rorey_breaker
5 Replies
Login or Register to Ask a Question