Shell Programming and Scripting

Hello all, this is my first post so please go easy on me. I have like 30 redhat servers that I manage now and it’s a requirement to do hourly checks on some processes, yes, even though we have a full time monitoring solution they still want hourly checks so I'm looking for a way to script this so I'm not spending my day logging into servers and running the same commands on all of them.

So it would go something like this;

Code:

ssh servername username and password
ps -ef | grep jboss
ps -ef | grep sendmail
ps -ef | grep auditd
ps -ef | grep symantec
process sendmail status
process jboss status
output data or send results to email by server name
ssh servername username and password
ps -ef | grep jboss
ps -ef | grep sendmail
ps -ef | grep auditd
ps -ef | grep symantec
process sendmail status
process jboss status
output data or send results to email by server name

Ect....

Then I would like it to email me the output from all the process
checks or even just give them to me in a shell. Now there are 30
servers that this needs to be done on so I want to chain this together so I only have to run one script an hour, verify that everything is good and go about getting the rest of my job done. I just got off of amazon and purchased 4 scripting books but any help would be greatly appreciated until they arrive!!

Thanks in advance!

G

edit by bakunin: i threw a few missing code-tags your way. If you don't want to get hit (and hurt) accidentally you bring them with you yourself from now on. Thank you.

First off, before you start scripting lets analyze your requirements: you have some 30 servers and you want every one of them be checked and generate a mail. This is 24*30=720 mails per day and even if you would manage to implement this script (which is entirely possible) i wonder if you really would be happy with it. You might end up reading mails all day.

So probably there should be some sort of log, of course, but there should be a error condition upon which you want to be informed (by mail or whatever) and otherwise the check script should silently write its log and finish. This means it might be wise to set a number (range?) for each of the processes which is deemed "normal" and test against this number or implement some other means of deciding what is "normal" and what is "error".

Furthermore, your script shall run for a long time. This means that it will - sooner or later - have to deal with some disruptions to its normal mode of operation: a system not being reachable by ssh (system shut down, ssh-daemon not working, network down, ....) or an accidentally deleted exchanged ssh-key, whatever. All these possibilities should be taken into account and there should be some reaction built into the script dealing with this: should the script create an error? Should it silently ignore it? Should it silently ignore it for <n> runs and then raise an alarm?

These are questions best asked before you start writing a script. Even if i provide no code (yet - state your requirements and i will gladly help you write it) i

hope this helps.

bakunin

After specifying your requirement (As Bakunin said ) check "/etc/sshd/ssh_config" file if the user is allowed to use ssh or not.

Besides using this is easily readible.

Code:

ps -def -o pcpu,pid,user,args | grep -i jboss >> FILE

Error handling is important if you have too many servers and too many processes to check. Connectivity to all hosts etc..

You have a very valid point that I didn't even consider great catch. The requirement is to ensure that certain processes are running and not hung or crashed. I'm currently using Big Brother to monitor some of the processes but there is also a requirement to hand test them and others hourly, Now in reality I can't actually do that because that would take my whole 9 hours a day. So what I was hoping to do what write a script that would check the status of the processes across all the servers and tell me when there is an issue and tell me nothing at all if all is well. As for handling errors if the script ran once and hour via cron or whatever, I would say to have it ignore one failed connection attempt (say a server is down) but alert on the second attempt. As for ssh this is a private network (air gap) so all the ssh keys are shared all over the network. From any shell logged in as root I can ssh as root, without having to login. I hope this answers your questions. I really didn't think about the amount of email.... Once again great call...

Thanks!

G