Give us your poor, your tired, your huddled Admins... War Stories


View Poll Results: This thread ...
yes I have 7 77.78%
No I haven't 1 11.11%
wasn't me was a gremlin 1 11.11%
Voters: 9. This poll is closed

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Give us your poor, your tired, your huddled Admins... War Stories
# 1  
Old 10-24-2002
Give us your poor, your tired, your huddled Admins... War Stories

I have been toying with the idea of posting a thread asking for your dumbest mistakes so that we may all learn from each other.

"A smart man learns from his own mistakes,
A wise man learns from others' mistakes"



I think I will start us off. I haven't had too many that were really bad, but here goes. I was playing around on a box as ROOT when I was looking in a certain directory to see what scripts where there.

I was catting them to look but I inadvertently executed one of them when I deleted the cat and left only the command and hit enter by mistake. I didn't notice my handywork right away though. About 15 minutes later, people called me saying they can't login. Anyway, 3 hours later I finally figured out that when I exe that script it changes the permissionsn on your home directory to 444. Well, for me that was /. Smilie Once I fixed that everything was fine.

I had 4 hours of downtime charged to OE, operator error. BTW, this was on a production system at work! Doh!!! Smilie
# 2  
Old 10-24-2002
This is not Unix specific, but the concept is just the same regardless of OS.

Back in my xBase days I was working for a medical software company here in Dallas. I wrote an upgrade install batch file that would pkzip the FoxPro programs, database and back it all up and then delete all the files, re-install the upgrade and then unzip the database, recreate indexes, etc.

I tested the routine time after time and the world was a happy place until I got "The Call".

It seems that while the client had pkzip install on their machine, it wasn't in the path. The backup zip was never created, but the sub-dirs were blown away just the same.

Data was gone. No more, non-existant, poof.

Disaster recovery? Client never even heard of a backup.

The only way that I kept my job was that I spent the entire weekend on the client site manually re-entering data via our application and getting things up and running.

Luckilly for me, it was a slow site and I was able to do all the data entry in a single weekend.

Very, very painful lesson learned. When doing any thing destructive, check, double check, triple check and trust NOTHING.
# 3  
Old 10-24-2002
I'm a no administrator (though I'm trying to become one), but I too have a snippet to share...

Right from my college days I was exposed to UNIX systems and even if I was not from comp sci background was attracted to the philosophy of the system as thought of by its designers - Thompson, Ritchi, McIlroy and all...

So when I went to my first job in a consulting company.. I found an atmosphere where my European client had a production and test UNIX boxes which were connected to IBM SNA / CPI-CC to his SAP R/2 production system. I was very happy that I can make use of my UNIX skills. Yes indeed my first assignment was to write automation shell scripts which post/extract EDI messages to/from SAP - UNIX. These scripts used to call C programs using CPI-C interface.

I also learned CPI-C programming, which involves writing your own C code including the CPI-C interface headers and sources - in all there were 8 .C sources and 4 headers, requiring makefiles to do the compilation. There were production C executables doing daily updates and extracts to SAP.

One day I was making some compilation on the test box, while I was monitoring the production box as a privileged user in another terminal. I inadvertently executed a makefile with the same name on the production box which luckily didn't overwrite any existing executable. But my heart stopped thumping, as I had no idea what the script did.. but I had to spend extra hours that day reading the whole script checking for paths and date/time of modifications...

from that day on I was extra careful ascertaining which terminal I'm using for prod/test boxes...
# 4  
Old 10-25-2002
I'm no REAL ADMIN - but I have a FreeBSD server box for my home LAN:

Anyway : I was new to FreeBSD - and had just about set it up EXACTLY as wanted; after much tinkering! I was then trying to update my fstab to include my floppy drive. Unfortunately, I did the stupid thing of "echo"ing the details in, sorta like:
Code:
echo /dev/fd0.1440 msdos /floppy rw,noauto 0 0 > /etc/fstab

Experts - can you spot the mistake? Yep, only 1 '>' - so when I rebooted, no drives mounted- ARRRGGGHHHH!

Fortunately, I inserted the boot "fixit" cd and edited the file after mounting the correct partition! My heart has NEVER beated so fast!

Smilie

Last edited by WIntellect; 10-29-2002 at 06:12 PM..
# 5  
Old 10-25-2002
I've got two great ones done by the same person - an Engineer working in a hospital.

One - problem with A/C units drain pipe clogging and backing up under the data center floor and shorting out data cables due to corrosion.

Solution - put bricks under the cables.

Problem with solution - he never thought that the water could get so high to go beyond the bricks - and never put bricks under the electrical cables. Problem with draining happened again and fortunately was discovered before major electrical problems. Water reached 3/4 of the way up the bricks. Almost topped the electrial outlets (flat outlets on floor maybe 3 inches high) Engineer was not allowed in D-C without escort.

Two: Same engineer brought back in for water problem. He contacted a company that makes Hog socks. They are suppose to be used for containing oil/hydraulic spills in shops. He figured they would soak up water too. He was escorted into the D-C and started placing these 'socks' around the computer electrical outlets. They were near or in front of the A/C blower and the constant blowing caused the material inside the socks to be blow out. It was ground corn husk which looked like fairy dust in the cartoons. The stuff was all over the place and the two escorts and the Engineer didn't notice it. I walked into the D-C and noticed this gold dust everywhere! I went beserk and threw the guy out, started collecting up the hog socks and tossing them out the door. We had to bring in an Enviromental team from Digital to check out our servers. Remember, this is in a hospital where we are suppose to be up 24X7, 365 days a year. We were down 2 days, lost something like 5 hard drives. There is still some dust under the floor if you look under cables. My old boss has a small glass container of the 'fairy dust' sitting on his desk to remind him to never trust the guy. Water problem was addressed by others (water sensor/alarm installed - second drain installed ) The company who makes those hog socks was contacted - they told us these things are not made for computer environments and that the Engineer never mentioned what he was using them for.
# 6  
Old 10-29-2002
Somebody's been sleeping in my thread!

Okay who changed my poll??? Smilie

Please change it back, okely dokely?

Thanks for your help.

I was just trying to inject a little humor and at the same time allow for people to see what NOT to do on their systems.

Smilie
# 7  
Old 10-29-2002
fzzzz

We moved a big call center system (a NCR unix server and a connected tele switch) into a new server room. After 2 weeks someone called from the sales staff complaning that the system didnt deliver any phone calls to the sale people using it (around 100 persons). We tried to acces the server and it did look OK then we checked the admin console on the big tele switch and it said that it couldnt connect to the switch cards in the cabinet so we went down to the server room.. .. mmmmm... mmm..

We had placed the tele switch directly under a cooling machine that was water driven (you can se were im going with this.. Smilie ).

we hade around 5-6 liter of water in the tele switch cabinet... we even had water in the power supply that was mounted in the cabinet with no shielding cover! Smilie the funny thing was that the admin console card was still working despite the fact that 6 liters of water hade gone trough the power suply Smilie im just glad we didnt get electricuted Smilie

VERY EXPENSIVE MISTAKE !!!!!! the swtich was around 200K $...
 
Login or Register to Ask a Question

Previous Thread | Next Thread

2 More Discussions You Might Find Interesting

1. AIX

Poor Performance of server

Hi, I am new registered user here in this UNIX forums. I am a new system administrator for AIX 6.1. One of our servers performs poorly every time our application (FINACLE) runs many processes/instances. (see below for topas snapshot) I use NMON or Topas to monitor the server utilization. I... (9 Replies)
Discussion started by: guzzelle
9 Replies

2. Shell Programming and Scripting

regexp: really tired, guru help needed...

Hello all! Please help me with the following complex regexp which works in egrep: egrep '\"http:\/\/ccc\.bbb\.com\/documents\/0000\/{4}\/(+\.{3})*\"' my-file.htmlBut silently does not work in sed: sed "s/\"http:\/\/ccc\.bbb\.com\/documents\/0000\/{4}\/(+\.{3})*\"/aaa/g" my-file.htmlor sed... (7 Replies)
Discussion started by: ulrith
7 Replies
Login or Register to Ask a Question