system getting crashed

Thread Tools Search this Thread
Operating Systems Linux system getting crashed
# 1  
Old 05-23-2009
system getting crashed


Iam a running a apache webserver in CentOS and i get a heavy traffic about 2.5 lac pageviews daily and my db size is about 2GB. Now the problem is after serving some lacs of requests by apache....Both apache and mysql hangouts and the system gets hanged up...using all resources in the CPU..I have Xenon Quad core Processor with 4GB ram....i think it can donno after some time it jus hangs up..I want to find where the problem is...Iam checking the error log but i did'n find a clue for it...Anything i can look at it....the sys got stuck and it vil take 5 - 10 mins to make a reboot..

Waiting for replies....
# 2  
Old 05-25-2009
First, go into the cron folder and update sysstat to run more frequently. On our centos systems, inside /etc/cron.d/sysstat, I use:
*/10 * * * * root /usr/lib/sa/sa1 -d -I 30 20

To have this take effect immediately, you need to delete today's sa file. Otherwise, the change will start taking place tomorrow.
rm -f /var/log/sa/sa`date +%d`

The next thing is to monitor processes. Something like this should work. Add to the sysstat cron file these two lines:
1 * * * * root find /var/log/sa -name "ps-*" -cmin +300 | xargs rm -f  &>/dev/null
* * * * * root ps -N --sort comm,pid -ww 
  -o  tty:1,pid,c,pmem:5,rss:8,sz:8,size:8=TSIZE,vsz:8,nlwp,lstart,wchan,args |
  sed -n 's/^? //p' |
  awk '$4 != "0" && $5 != "0"' 
  &>/var/log/sa/ps-`date +%H%m`

(Note, you must put these on exactly TWO lines. For readability, I've broken up the second entry onto multiple lines.)
Every hour, the first command cleans up after the second command any data that is more than 5 hours old (to prevent the directory from getting too full). You can change that if it's not enough. The second command runs every minute and saves a very details ps-listing to the disk.

If you have a hang, reboot and then run "sar -A", which should now give you very detailed information about everything. You might notice a memory spike followed by IO, or vice-versa. Note the time when the problem occurs, and then go into the appropriate ps-* files to see if you can see the problem process. You might need to look and previous ps outputs to see a change. The processes are ordered by command-name and pid, so you are able to do a "diff" between two ps files to see where a change really occurs.

Last edited by otheus; 05-25-2009 at 08:28 AM.. Reason: a few filters to exclude non-system non-kernel processes
# 3  
Old 05-26-2009
Also, Just an FYI, but Xenon is a noble gas. Xeon is a processor.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

System got crashed.

Hi Admins, In my local Vmware system i have installed solaris but while getting my root disk mirrored in svm I changed the vfstab entries and rebooted the server , the server got crashed, and now the root file systems and other filesystems are crashed. Please help me in recovering this. (2 Replies)
Discussion started by: Laxxi
2 Replies

2. Red Hat

What do you do right after a server crashed.

What do you check???? Thanks! JC (0 Replies)
Discussion started by: 300zxmuro
0 Replies

3. Red Hat

Redhat Linux 4 crashed

Hi, I am having redhat enterprise linux 4 machine with kernel version 2.6.9-39.EL.Whenever I ran some java applications related to mechanical the system got crashed and powered off. last output is as shown below. reboot system boot 2.6.9-39.EL Fri Sep 24 15:23 (01:43)... (0 Replies)
Discussion started by: ktrimu
0 Replies

4. Solaris

Disk crashed

My system is SUN Solaris 5.6 and one of the disks on the server got crashed. Here are the details d23: Mirror Submirror 0: d24 State: Okay Submirror 1: d25 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) ... (1 Reply)
Discussion started by: asalman.qazi
1 Replies

5. Linux

Suse Linux System Crashed

Friends please let me know. How we can check the reason why system restarted automatically or rebooted. Suggest me some way other than messag log. Thankx Bryan (1 Reply)
Discussion started by: bryanabhay
1 Replies

6. SuSE

suse 9 crashed

Hi, Running SLES 9 (update 4) on dell's poweredge 1950 server. Kernel: 2.6.5-7.315-smp #1 SMP Wed Nov 26 13:03:18 UTC 2008 x86_64 x86_64 x86_64 GNU/Linux Yesterday night my monitoring service emailed me system(ssh/smtp) unreachable...I tried connection through ssh, it did not let me through... (0 Replies)
Discussion started by: upengan78
0 Replies

7. AIX

AIX Crashed..

My AIX 5.3 Machine Carshed Can any one tell some way to find out what went wrong.. I mean debug why it got creahed... (3 Replies)
Discussion started by: pbsrinivas
3 Replies

8. UNIX for Dummies Questions & Answers

old server crashed

Hello We had an old system designed in fortran that ran on a IBM RS6000 AIX 3.2 system. The person who designed is long gone. It was replaced with a completely different (non unix) system 6 years ago. We still used it for historical lookups of older information. Well yesterday it died. The... (5 Replies)
Discussion started by: billfaith
5 Replies

9. Post Here to Contact Site Administrators and Moderators

Gollum got crashed

Gollum got crashed, needs Administrator's attention. Check this: (0 Replies)
Discussion started by: tayyabq8
0 Replies

10. Shell Programming and Scripting

kill crashed out users

Hi all, We have a problem where we get a fair few users either exiting incorrectly or crashing. I'm trying to get a script together that runs every hour to kill these processes off. We are running Sco OperServer(TM) Release 5 The command we use to get a list of users who have crashed: ps... (2 Replies)
Discussion started by: tez
2 Replies
Login or Register to Ask a Question

Featured Tech Videos