What is on Your Mind?

Curious, what did you do?

I deliberately left that part out because I didn't want to encourage anyone from doing the same thing.

It occurs to me that if I tell you, you probably WON'T do it. So here it is.

I had just had my first intro to unix class. It was a 2 day session, hosted by our software vendor, geared towards the computer illiterate.

I.E. "Ignorance is bliss" became "Just a handful of facts will get you into trouble."

The next day at work I was having some problem (don't remember what), but for whatever reason, I attributed it to bad permissions. So, while logged in as root, I set the permission bits of every file on the system to 777.

It was actually quite interesting how the system responded: The serial printers stopped working. Users on serial terminals were fine as long as they were already logged in. If they logged out, they couldn't get back in. Network printers and users were fine. The Fax server software (or fax modem) stopped running. One modem refused to connect, but the other, which was attached to a console port, was fine.

It was as if the system was trying to keep working, while also doing it's best to lock down the system in response to the problem.

My system backup was well over a year old, so I spend the weekend comparing the permission bits on the backup and manually resetting everything.

P.S. Now I create a mksysb backup tape just about every month.

I only happened to be next to her when it went down, she just got married recently and maybe still on honeymoon mood and probably still dazed and day dreaming ... and did something that should never have been done on a very critical production system. She was very young at that time and very inexperienced, it was a very good but painful lesson for her. She will remember it for the rest of her life I think.

I remember the aftermath was terrible, we had endless conference calls late into the night, meetings after meetings explaining over and over to angry people calling in from all over the globe at different time zones wanting to know what the heck was going on and they were screaming the blame at each other during the calls it was like being at fish market and we're right there in the middle of a shit storm and you will just get the sinking feeling that somebody's head is going to roll if you say the wrong thing.

I will not say how it all started or what the outcome was as you never know who visits unix.com these days.

Anyway the bottom line is this, if you are ever caught in such a situation whether it's your fault or not. In a heated situation where everybody is angry screaming and shouting, DONT EVER tell the truth immediately, delay it a day or two.

if you tell truth somebody's definitely going to get sacked on the spot maybe you, maybe your friend or both of you. Don't give in to pressure just delay it! delay it after a day or two by then the system would have already been restored, service resumed back to normal and the incident will be largely ignored by then due to other new pressing issues of the day starting to come in and there will be very much lesser angry people around.

Though I should tell you that this customer will probably never trust you again in the future and maybe you will even even lose his business when the time for contract renewal comes around so it's 50-50 choice and both are difficult to make.

I actually learned this from another incident that actually happened many years before this incident and the earlier incident was my own fault being young and inexperienced myself once before and a more experienced friend helped me out, we did the same thing and it worked so this counts as paying it forward.

So lesson learned: Never give root access to "just married" newbies(just kidding)