Servers lacked maintenance, here's my to-do list


 
Thread Tools Search this Thread
Operating Systems Solaris Servers lacked maintenance, here's my to-do list
# 1  
Old 01-06-2013
Servers lacked maintenance, here's my to-do list

I'll be taking over administration of a rack of Solaris machines that haven't had an admin for the last 9 months. Prior to that they had limited maintenance. I understand there are a few tickets that will need addressed, but I won't have the details for a few days on them. Regardless, I'm trying to compile a to-do list. What would you add to this list?
  1. Check for hardware failures, disks, fans, psus, etc... repair as needed
  2. Ensure backups are being taken and are restorable
  3. Snapshot filesystems
  4. Check who has permissions to access these servers, internally and externally. Verify they all should have access.
  5. Reset the root passwords, and check who else may have root access via sudo, powerbroker (if used), or uids.
  6. Check all installed packages for exploits, update as needed
  7. Verify you have account access to the SC/SP/ALOM/ILOM over serial console. If I don't have access, look into resetting the password.
  8. Setup monitoring providing me with immediate access of issues.
  9. Identify critical apps, machines, etc... and prioritize them for support
  10. Acquire Oracle Support agreement details so if\when I need them, I have ready access.
  11. Check the cron tables on each system as well, just to see what the prior admins have tried to automate (system admin related or application related).
  12. Check the messages file on each system as well to catch any other issues that may have been written via syslog.
  13. Review logs specifically with a view to what has happened before\after reboots to return the server to the expected state.
  14. Check /var/crash/<hostname> to see if/when the last time the server may have panicked.
  15. Check if startup and shutdown of applications is implemented well and if it is automatic or manual
  16. Check for possible dependencies on other systems. Track ingoing and outgoing traffic if needed to check dependencies.
  17. Check external hardware, for example NAS / SAN Disk Arrays, Network and SAN-switches, UPS, Airco, etc...
  18. Try to track documentation and if possible reports of past changes and logs, if not available, see if you can interview old admin.
  19. Acquire a test system so I can try stuff out.
  20. Make a runbook.

What would you add\change\remove on this list? Thanks in advance for your help.

Last edited by DustinT; 01-07-2013 at 09:36 AM.. Reason: Added prioritize. Added Scrutinize and Bryan's tips. Attempted to prioritize.
# 2  
Old 01-06-2013
You should also get some information on what kind of applications are running there and which servers are critical - so they should be handled first.
This User Gave Thanks to bartus11 For This Post:
# 3  
Old 01-06-2013
Quote:
Originally Posted by bartus11
You should also get some information on what kind of applications are running there and which servers are critical - so they should be handled first.
Yes, I'll add that to the written list. I believe this is just a single rack of Solaris equipment so I don't expect it will be too hard to hit the priorities.
# 4  
Old 01-06-2013
Also maybe:
  • Try to track documentation and if possible reports of past changes and logs, if not available, see if you can interview past admins. It is really nice to be confident that systems will come back up without problems if rebooted...
  • Check if startup and shutdown of applications is implemented well and if it is automatic or manual..
  • Check for possible dependencies on other systems. Track ingoing and outgoing traffic..
  • Also check external hardware, for example NAS / SAN Disk Arrays, Network and SAN-switches, UPS, Airco, etc...
  • Acquire a test system so you can try stuff out..
This User Gave Thanks to Scrutinizer For This Post:
# 5  
Old 01-06-2013
Brainstorming a bit here..

Check the cron tables on each system as well, just to see what the prior admins have tried to automate (system admin related or application related)..

I'd also verify you have account access to the SC/SP/ALOM/ILOM over serial console, having this information handy will go a long way if a critical server goes down. If you don't have access, look into resetting the password.

You hit on user access, but to expand that, reset the root passwords, and check who else may have root access via sudo, powerbroker (if used), or uids.

Check the messages file on each system as well to catch any other issues that may have been written via syslog.

Oh yea, check /var/crash/<hostname> to see if/when the last time the server may have panic'd..
This User Gave Thanks to bryanNJ For This Post:
# 6  
Old 01-07-2013
Quote:
Originally Posted by Scrutinizer
Also maybe:
  • Try to track documentation and if possible reports of past changes and logs, if not available, see if you can interview past admins. It is really nice to be confident that systems will come back up without problems if rebooted...
  • Check if startup and shutdown of applications is implemented well and if it is automatic or manual..
  • Check for possible dependencies on other systems. Track ingoing and outgoing traffic..
  • Also check external hardware, for example NAS / SAN Disk Arrays, Network and SAN-switches, UPS, Airco, etc...
  • Acquire a test system so you can try stuff out..
These are some excellent tips. I'll be adding all of them to my to-do list. It's starting to seem that I may not get through them as quickly as I hoped. Oh well, job security, I suppose. I honestly don't see how I could skip any of the steps. They're all critical things that could take down the system.

Regarding the test system, there's a large VMware cluster. At a minimum, I'll use that to provide a test environment. Because of the size of the environment, I tend to think there's no unused servers, but I'll look for on.

I will have the chance to interview a former admin. I'll try to find out if there's a log, change log, etc... If not, I'll press for details. Any specific questions or terms I might want to use?

---------- Post updated at 09:38 PM ---------- Previous update was at 09:30 PM ----------

Quote:
Originally Posted by bryanNJ
Brainstorming a bit here..

Check the cron tables on each system as well, just to see what the prior admins have tried to automate (system admin related or application related)..

I'd also verify you have account access to the SC/SP/ALOM/ILOM over serial console, having this information handy will go a long way if a critical server goes down. If you don't have access, look into resetting the password.

You hit on user access, but to expand that, reset the root passwords, and check who else may have root access via sudo, powerbroker (if used), or uids.

Check the messages file on each system as well to catch any other issues that may have been written via syslog.

Oh yea, check /var/crash/<hostname> to see if/when the last time the server may have panic'd..
Great brain storm, man. There's some great security tips in here. I'll have to add all these in too. At some point, I may have to get some help in prioritizing these. It's a good problem to have, I suppose. I want my client to get their money's worth.

---------- Post updated 01-07-13 at 08:47 AM ---------- Previous update was 01-06-13 at 09:38 PM ----------

I had cross posted this on Oracle's forums and got a nice tip for taking a snapshot. Also, to review the logs associated with reboot and looking for anything unusual to return the server to it's expected state.
# 7  
Old 01-08-2013
Well, I'd just like to say thanks for everyone's help. You have been most helpful.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to find Error: rpmdb open failed on list of servers

Hello all, I have a task to patch red hat servers and some servers have a corrupted rpm database and return the error: Error: rpmdb open failed I know how to fix this when it occurs. What I'm hoping to do is scan a list of servers by IP and report back which server have this error. ... (6 Replies)
Discussion started by: greavette
6 Replies

2. Shell Programming and Scripting

Traverse through list of servers using ssh non-interactively.

I have 300 servers listed in servers.txt I motto is to check if my id "user1" has sudo privileges on the 3000 servers. I m using sudo -l to check if I have privileges or not. If wish to check this either non-interactively; if not; interactively. Below is the script I wrote: ... (5 Replies)
Discussion started by: mohtashims
5 Replies

3. Shell Programming and Scripting

How to Find List of MQ and Websphere certificates that are installed on Linux and UNIX servers?

Dear All, I am planning to find the list of certificates(WEBshpere/MQ) on a servers. My certificates are either stored in (.jks) / (.pem)/ (.cer) . But some of the certificates are stored without these file formats. I tried using find command but unless I give the file name its difficult for... (4 Replies)
Discussion started by: sidh_arth85
4 Replies

4. Shell Programming and Scripting

Ping script to list of servers

Hi Friends, I have experience in redhat/ Ubuntu OS, but I am very new to solaries os. my servers OS is Oracle Solaris 10 8/11 s10x_u10wos_17b X86. I have a file contains 200 servers IPs one by one. now I want a script to chaeck which IPs are pinging, not pingning. I... (8 Replies)
Discussion started by: kumar85shiv
8 Replies

5. Shell Programming and Scripting

List the IP address of list of servers

Hi I have a file containing server names and i can ssh to all these servers without password. Could any body suggest me how to list out IP address of all the servers? Now i am manually doing this, like ssh to each server and run "ifcong -a" command and copy the ipaddress to a excel sheet.... (5 Replies)
Discussion started by: kumar85shiv
5 Replies

6. Shell Programming and Scripting

List of servers that are NOT authorized for password-less SSH

Hello friends, We have around 500 servers (HPUX, AIX and linux) and all of them need to be accessed from our management box (linux) via password-less ssh. Out of 500 around 150 servers are setup password-less. We need to setup password-less SSH for remaining servers. First we need to get the... (3 Replies)
Discussion started by: magnus29
3 Replies

7. Shell Programming and Scripting

List and Compare Files accross different servers.

Hi all, This is my situation. First thing is I cannot use rsync to accomplish this. I don't have on my systems and we can't put it on. I run HP-UX 11v3. I have a list of files generated every day which tells me which files are not in sync with the rest of the servers.I want to ls -l the... (1 Reply)
Discussion started by: zixzix01
1 Replies

8. Shell Programming and Scripting

How to find out list of all proccess which are running on unix servers from last two days.

Hi All, I have a requirment, i need to get the list of all the process which are running from last two days on my unix server and also to put this list into an another file. i am giving you a sample example : $ ps -ef UID PID PPID C STIME TTY TIME CMD (1 Reply)
Discussion started by: akshu.agni
1 Replies

9. UNIX for Advanced & Expert Users

want to mail a list of files in different servers

Hi All, I am running my script in crontab and it is generating a file everyday. And I have 10 different servers and i am running the same script in every server in crontab which generates a file in every server per day. I want to retrieve those files from each server everyday. I want to... (1 Reply)
Discussion started by: AshishK
1 Replies

10. Solaris

I will do Maintenance for my Servers ... pls help

Dear All, My manager asked me to make full maintenance for my Sun Servers , to improve the performance. What do suggest ? what is the major things I have to do ? with some details pls because I am new admin for the System. (currently we don't have any errors or any problem in the system... (5 Replies)
Discussion started by: ArabOracle.com
5 Replies
Login or Register to Ask a Question