UNIX for Advanced & Expert Users

need advise on the ff

a. what would be the best language for script that I can use to create a restart facility of all applications and instances I have on let say more than 20 servers?

b. don't you think its not secure to have a central script residing on one server to remotely create a script to restart remote applications?

c. can you recommend how can this be done w/o sacrificing security? GUI?

You need something like net-snmp or nagios. SNMP is a protocol that is meant to monitor remote systems. SNMP has security built-in. If you are running Oracle, there are Oracle products like datagrid that will do this for you.

I would view creating a script as a poor alternative.

While you could probably write a script, the most straightforward method is to create a single God-like account for the entire set of systems you have. This has negative security implications.

A. I would say the scripting language you are most comfortable. And then weight that against what is required to start the apps. Off hand probably combinations of shell,awk,perl.

B. With logic and keeping in mind what these scripts will be doing. I think you can be secure in the process. Tools like ssh,sudo,sudosh with general user / group permissions. But using all these with a clean logic of access and validations.

C. Your choice for the access and control of the scripts. Any user interface can be used to run and execute the scripts as needed. So if you are giving this ability to say your operations support area. Then the access and control could be different than for a group of support UNIX admins.

The how to do it and stay in SOX compliance depends on your environment and what the applications and or instances that you are trying to bounce. If it is a DB environment that would be different and have other support groups and controls that need to be considered like a oracle grid stop / start sql script. Or a web server would maybe need a maintenance page put in place before the service is stopped. Can it be done I would say Yes. Can it be secure, again I would say yes. Has this been done to meet all the security requirements? That would be a yes also.

You don't have to pay for a 3rd party app to do this and then have all the money paid out for support and maintenance. nagios/bigbro/hobbit freeware system monitors / alerting are a start and can send alerts out about problems. But I don't think I would use it to perform the actual bounce of services and applications. But the alert could be sent to a operations support and then they have a procedure that they follow per the alert.

Many services have a auto restart built into them. And the system could do a restart from inittab for services also. That way if a service stops on the server then the OS sees and performs the restart local to the server. No login required and no outside resources needed. You can build smart scripts that the init service would use to bounce a service or application. These methods have been around for years. And if you test and build a script to perform the actions it should not be a problem. Many of the Linux servers have these scripts in the init.d directory. And they are called and used during server stop / start.

Sounds like a challenge and will keep you busy for a month or so. Maybe longer lol...

thanks for all your replies. All my 20servers do have restart script individually. My company would like though to have a central script for our monitoring officer to restart it instead of logging in EACH to the server. is it possible and secure? any recommendation?

Yes it can be done and be secure. I done a couple and basically here was my approach.
1. Create the central users script. If needed the make menu driven. We created a base user not a super user for this account. In out case the home dir is a NFS mount to all the servers. So a script to send commands to another server wasn't needed. A NAS mounted device would work for this also.
2. So since in most cases root needs to be the user to do the work.
I have a local root script that is looping and looking for a file to show up in a defined directory on the global mount. Now this is were it depends on what you are trying to do on the server. So the dropped file
could contain the objective to be bounce. So if you would bounce oracle and refresh sendmail lets say. the file would be named <server name>_???.trg and contain oracle|sendmail. The local script would see the file then consume the file and create a results file.
3. the master script would look for the files to be consumed and the results being created.

There are a lot more steps to all this...

Security wise - since the user login into the server as the service user. They don't know or have ability to be root. That is all controlled by the scripts and your programs. And the service account when it logs in will only run the menu to do your application. And you will also do the trap command in the login control so they can't ctl c or anything out to a shell.

This may sound worse than what the work really is. You could have the whole thing done in a day or so.

Use some great NMS like NetXMS

Are yo running Oracle (you mentioned instances)? It doesn't matter though. There have been many, many posts about 'remote commands' and 'central administration'.

My approach, which I'm surprised wasn't mentioned, is a wrapper for ssh.

If you can ssh to all your servers, then you can write a wrapper (or loop) to connect to all the servers and run a command.