Process going to 100% CPU


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Old 12-05-2013
Process going to 100% CPU

I have a httpd process which uses up 100% cpu. This makes my application to stop working and it just gets hung.

I tried getting the tusc output looks something like this:
Code:
# /usr/local/bin/tusc -pl 26516
( Attached to process 26516 ("/opt/hpws22/apache/bin/httpd -d /opt/hpws22/apache -k start") [64-bit] )
[26516]{579822} ksleep(PTH_CONDVAR_OBJECT, 0x6000000000177090, 0x6000000000347640, NULL) .............. [sleeping]
[26516]{579821} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579820} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579819} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579818} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579817} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579816} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579815} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579814} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579813} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579812} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579811} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579810} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579809} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579808} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579807} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579806} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579805} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579804} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579803} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579802} In user-mode .......................................................................... [running]
[26516]{579801} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579800} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579799} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579798} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579797} ksleep(PTH_MUTEX_OBJECT, 0x6000000000027df8, 0x6000000000347140, NULL) ................ [sleeping]
[26516]{579795} read(8, 0x9fffffffffff7080, 1) ........................................................ [sleeping]
( Detaching from process 26516 ("/opt/hpws22/apache/bin/httpd -d /opt/hpws22/apache -k start") )

I am using HP-UX B.11.31 and my apache version is B.2.2.15.15.
Any idea how can I proceed further on my analysis?
# 2  
Old 12-05-2013
I have had similar issues and the end result on many are the http server isn't tuned to handle the load being placed on it.

How many total http process are there?

There is a parent and it spawns off a number of child processes. You can elect to have the children start at startup or get started dynamically based on need. I think the default is to have an intial number of children already running.

I never adjust anything until I understand the load being placed on the http server...the bigger picture rather than the detailed look you are doing.

------------------------------------------------------------
netstat -an ....

I look at the state of the send and receive buffers of all the clients (netstat -an | grep 80).

Look at how many are established and if any or in other states. Look at the Recv-q and Send-Q for each connections and see if they are stuck...or what those values are...0 0 means that particluar connection doesn't have any transactions in progress. If you see one of the buffers with a value that isn't changing, that particular connection may be hanging.

------------------------------------------------------------
access.log

I also go through my http logs and look to see how much data and the fequency of of the requests. So you can look in the access log and see the ip addresses of clients, their frequency, and the size of the data involved in a get for example. On one occassion we had someone using the server as a patch server (the clients would connect over http). I grep'd through the logs and counted the bytes over time and eventually found 12 clients hitting the server (http) with high frequency and large data requests and the http server just couldn't handle it.

grep through your access.log for ipaddress, gets and puts, and the amount of the transfer...see the format of the access log to determine the fields of an access log entry

So...after such an analysis I was able to determine the frequency of when the http process got to 100 percent and I exampled the
data accumulated during this frequency and then I stopped some of the clients I saw making frequent requests to test my theory.

---------------------------------------------------------------------------------------------------------
How to solve or provide relief for overworked HTTP server...

So once you understand if this is a load issue, that doesn't mean you automatically start tweaking the httpd server. When I saw the same clients frequently hittng the http server I reconfigured those clients to use something else. I could also have elected to do a hard-temp fix by restarting my http server every so often. These things are easier to do if you're too busy to get to the perfomance adjustments you need to make to resolve an overworked web server.

In short, I counted the amount of data that was sent through the webserver each time it went to 100 percent and I knew how long the webserver would run from startup of the webserver to the time of the 100 precent cpu usage.

Last edited by blackrageous; 12-05-2013 at 11:23 AM..
This User Gave Thanks to blackrageous For This Post:
chacko193 (12-06-2013)
# 3  
Old 12-06-2013
I tried getting the Recv-q and Send-Q for each connections and they are 0.

And this issue is coming even when the we are giving very small load also.

There is nothing in the access.log which suggest there is any heavy data transfer.
# 4  
Old 12-06-2013
What is trying to connect to your Apache?
Windows PC?
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
If a process cpu % is higer than bilboNIX Shell Programming and Scripting 9 11-26-2012 06:09 PM
CPU and memory utilization of a process, by process name ggayathri Emergency UNIX and Linux Support 4 08-13-2012 01:58 PM
process CPU time orange47 Solaris 9 09-22-2011 01:17 PM
Process by CPU ajaincv Red Hat 3 06-06-2011 03:15 AM
list which process using how many cpu utoptas Shell Programming and Scripting 5 08-02-2010 09:06 AM
100% cpu process karlochacon UNIX for Dummies Questions & Answers 3 07-19-2010 02:39 PM
CPU Usage for a particular process jitendriya.dash Shell Programming and Scripting 1 06-01-2009 12:00 PM
Multi CPU Solaris system shows 100% CPU usage. mahive Solaris 4 05-27-2009 03:52 PM
process cpu usage cwsmichigan UNIX for Dummies Questions & Answers 9 11-27-2008 02:49 AM
how to get persistant cpu utilization values per process per cpu in linux (! top,ps) pankajd UNIX for Dummies Questions & Answers 3 10-09-2008 11:26 AM
cpu utilization of a process iam AIX 2 02-20-2008 12:50 AM
CPU usage of a process robot23 AIX 5 09-27-2006 11:36 AM
How to find which process is using up too much CPU 0ktalmagik Solaris 1 06-03-2006 03:48 AM
process not using enough cpu hcclnoodles UNIX for Dummies Questions & Answers 1 05-27-2004 07:22 AM
process -> 1 cpu Solitare UNIX for Advanced & Expert Users 2 03-21-2002 04:28 PM