emergency shutdown best practices.


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users emergency shutdown best practices.
# 1  
Old 10-08-2008
emergency shutdown best practices.

Has anyone implemented or have suggestions on how to shutdown many remote unix/linux servers from a single script initiated from 1 server?

I need this to execute in parallel as time is not on my side. Our ups is sadly underrated and will die in approximately 15 minutes. (There is not any money in the budget to upgrade the ups.)
# 2  
Old 10-08-2008
If you can ssh around as root
Code:
for host in `cat hostlist`; do ssh $host shutdown <arguments>;done

(assuming that you put all of the hosts in a file named hostlist)
# 3  
Old 10-08-2008
If you have more than 15 minutes to plan for "emergency shut down" of your servers, I'd recommend:

1. All applications that are running have corresponding startup and shutdown scripts in rc.*
2. Identify the order that your hosts should be shutdown in - NIS or LDAP should shutdown last, NFS servers second last... NTP would go first...
3. Write scripts.
You need to send a wall to all users connected informing them of an impending outage.
You need to ensure that you send the right shutdown options to the right OS types.
You need to create for each command that you are sending - for audit and CYA purposes later.
4. Inform your business/clients/users that these are the "emergency shutdown" procedures. Get them to sign-off and buy into them. If they have special requirements, amend your policy to include those.

Ensure that you have adequate time to shutdown storage devices that may have a great deal of data in cache. Ensure that you have adequate time to shutdown tape storage systems, as the robotics may need more time to get to "home" than you might expect.

There is no shortage of things that you could do, but this should get you started.

Last edited by avronius; 10-08-2008 at 03:17 PM.. Reason: Grammer
# 4  
Old 10-08-2008
Here in lies the problem. Our network security officer will not allow ssh as root. Also, we have many "flavors" of unix/linux which have different shutdown options. I tried something like this but it does not work on all the servers. (They do not like the <<\EOT...EOT construct)
$1 is list of remote servers. mbaker has sudo root privilege

cat $1 | while read X
do
echo "Starting shutdown of $X"
ssh -T ${X} << \EOT >> Emergency_shutdown.log 2>> error.log
name=`uname -n`
echo "name = $name"
type=`uname -a | awk '{print $1}'`
echo "type = $type"
if [ "$type" = "SunOS" ]
then
echo "Emergency shutdown initiated for $name."
# sudo -u root shutdown -y -i5 -g0 "Emergency shutdown started!!!!!" &
fi
if [ "$type" = "Linux" ]
then
echo "Emergency shutdown initiated for $name."
sudo -u root /sbin/shutdown -k now "This is just a test. Not really re-booting." < /dev/null > /dev/null 2>&1 &
fi
EOT
if [ $? -ne 0 ]
then
echo "Host $X connect failed."
fi
done
exit 0
# 5  
Old 10-08-2008
the best way is to create an script per server. called something like
"emergency-shutdown.sh"
and call that one.
and put all the os specific commands on each server
its harder to maintin maybe, but is cleaner, and more flexible.
# 6  
Old 10-08-2008
can you do this as user mbaker:


Code:
for host in `cat hostlist`; do ssh $host sudo -u root ifconfig -a;done

without being asked for a password? (In Solaris, only root can see MAC address - it's a harmless test).

Depending on how your systems are config'd, you may be able to sudo without providing a password. If you DO need a password to do the sudo, you could add some scripting magic to wait and apply the password, but it's not terribly secure...
# 7  
Old 10-08-2008
Good point Broli - since you are already using sudo, create the shutdown scripts (one script for all hosts - perform the OS check locally), and give mbaker the right to run the shutdown script.
Then, your script would simply be:
Code:
for host in `cat $hostlist` do; ssh $host emergencyShutdown <flags/options>;done

Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Linux

Virtualization best practices

Hello admins and gurus I have a controversial topic: now we are investing in a new Linux OS that will hold our Sybase database. The server will virtualized on a VMware server hosted on SAN storage. Now the question is, when we install the database engine is it better - in terms of performance -... (1 Reply)
Discussion started by: abohmeed
1 Replies

2. Shell Programming and Scripting

UNIX emergency

can anyone please tell me that how can i boot unix from a cd? full procedure. it's an emergency. reply asap (1 Reply)
Discussion started by: niket agarwal
1 Replies

3. Solaris

Best practices

Dear all, Kinda lame question but i'd like to hear your experiences and advice. Question in short ----------------- What permission should a mount point "ideally" have - i think it's root. Ex:- /usr/app/ i'd set the app to be owned by root and within /usr/app i would create another... (1 Reply)
Discussion started by: ossupport55
1 Replies

4. Solaris

Emergency !!!Help Please--- NFS

Hi Guru's, I am unable to mount NFS share on unix system (DG/UX) which is nfs client. Error: mount: /nfsshare: Invalid argument mount: giving up on: /mountpoint i tried following commands mount -t nfs remotehost:/nfsshare /mountpoint Error: mount: /nfsshare: Invalid... (1 Reply)
Discussion started by: Justin John
1 Replies

5. Ubuntu

Emergency help with ubuntu

Hi guys i have some question about ubuntu(10.10) 1.what time scheduling & page replacement algorithm have been used in ubuntu 2.how to create a process 3.how to kill a process 4.how to send information to a process 5.how to see a process 6.how to increase priority of a process 7.how to... (1 Reply)
Discussion started by: mortez28
1 Replies

6. UNIX for Dummies Questions & Answers

Script to force Oracle database shutdown when shutdown immediate does not work

I have Oracle 9i R2 on AIX 5.2. My Database is running in shared server mode (MTS). Sometimes when I shutdown the database it shutsdown cleanly in 4-5 mints and sometimes it takes good 15-20 minutes and then I get some ora-600 errors and only way to shutdown is by opening another session and... (7 Replies)
Discussion started by: aixhp
7 Replies

7. Shell Programming and Scripting

Need Emergency help with script!

I'm trying to write a script to push out to all our servers some Veritas add-ons. I want the script to push to all servers and if there server hardware matches the uname -i statement, it will install a additional add-on.. The script keeps giving me a error on line 29. Here is the script.... (3 Replies)
Discussion started by: soupbone38
3 Replies

8. SCO

Emergency boot

I need to boot an OpenServer 5.0.5 server but I don't have emergency boot disks for it. There are some boot disks of other servers. Can I use these disks, changing defbootstr ?. How ?. The fact is that administrator can't login as root and it seems to be a corrupted auth system issue. Other... (2 Replies)
Discussion started by: dags
2 Replies
Login or Register to Ask a Question