Well, you've got 20 instances of this "zoombie" thing running, whatever it is. Each instance appears to be threaded since some are using more than one CPU at once -- see the ones going over 100%. If they all decided to get busy at once, you certainly would get a very high load average.
As for why they'd do this, that depends on what the application's doing at any particular moment. It could be processing lots of queries, or it could be bogged down and accepting queries faster than it can process them...
One of them seems to have been running for nearly a year. Is this normal?
Yes, you are correct it is processing lots of queries every moment. What do you think is the best possible solution to solve this problem. And how cam you say that each instance are using more than one cpu at a time....
A single-thread process can't use more than one core -- 100% -- so anything above 100% must have threads. All of them might have threads, just not as busy right now.
I must make this clear: We know nothing of this "zoombie" application. (Searching for it on the net just finds a game for android.) For all I know, this problem might be a bug or misconfiguration, fixed without resorting to anything I suggest next. You should really consult your vendor about this issue before you invest too much time or money.
That said, I see two paths.
1) Configure zoombie to run less threads. People will need to start waiting their turn. How to do this depends on the application in question.
If your disks and memory can't keep up with the load, it doesn't matter how many cores you have. 50 idle cores all waiting for the same disk (which might actually be counted in the load average on Linux) aren't any faster than 30 idle cores.
2) Toughen your system to tolerate such a high load.
A load average of 50 on a 12-core system isn't as horrible as it sounds... It's not great, but it's no worse than 4 on a single-core. Lots of systems (especially leased ones -- getting the most for you money) tolerate quite high loads nonstop.
They deal with it by having lots of memory, beefing up the disks, nice-ing the hungriest processes (so you get priority when you need to administrate), putting swap on a separate hard-disk, and other such things -- this is standard configuration for most large UNIX systems, but not so much for consumer Linux unless they really planned ahead. This reduces the punishment various bottlenecks will cause, so it slows down more or less proportionately instead of bogging down.
You still need to control how fast queries are allowed to happen, though. The total amount of work done per second doesn't improve without more cores, so the more queries you accept, the slower they'll be completed. If your server allows itself to accept more queries per second than it can complete in a second, it's doomed.
Questions for you:
I'd like to see top and ps and iotop sometime your load average is really, really high. If you can't catch it in the act, you might need something like sar to record it.
Last edited by Corona688; 07-31-2014 at 04:53 PM..
Looks like you are running at 98% memory, but not swapping. There isn't much swap used. However you may not be getting very good file cache hits and it could be adding a bit to the load.
I hada experience with a search indexer that had 10+% improvement in performance when all the memory slots were filled v.s the same amount of memory larger dimms.
1st you need to identify your bottleneck. Memory or Disk is likely with indexers.
You get most of what you need our of sar, but you need to "tweek" sar to do disks. There is a nice site to graph it for you, but I can't post URLs.
Code:
sar -Ap
Change your sar to report every 5 minutes, instead of minutes. 10 minutes is not enough and 1 minute can be too much.
Disk I/O - to look at disk I/O, system time and I/O wait, to start to see if it a issue. You can "tweek" sar to show disks usage and it may help identify bottlenecks.
Network - look at for errors on interfaces and over bandwidth limits. Make sure you are running full deplex and the max speed you expect. Look out for things like time_waits in
Code:
netstat -antp
. There are a number or kernel tuning parms.
Memory - look at utilization and swapping. In this time, there isn't much excuse to have critical application swapping. Be carefull how you read sar, free, ... they report file cache which is tempory memory use to improve performace and is a cache so it may not be used actively. There is some tuning that can be done, depending on the application behavior and memory. Like large pages, but you need to know the behavior and the what else you need to adjust.
CPU - look at user vs system time and I/O wait.
Load - wil be shown in sar, but needs to be idenitied. top has a 'H' option to toggle between showing threads and not show them. It could be the application isn't tuned. This is a overall indication of everything
It may just be the application needs tuning. Threads, size, Garbage Collection (GC), ... If java, watch the the GCs and look at tuning it.
Hi,
could someone give me an example for a debian server script? I need to check a process if the process has a high cpu load (top).
If yes the whole server needs to reboot.
Thats it, nothing more. ;)
Hope someone could help me.
Regards
woisch (2 Replies)
Hi ,
I am trying to :wall: my head while scripting ..I am really new to this stuff , never did it before :( .
how to find cpu's system high time and user time high in a script??
thanks , help would be appreciated !
:) (9 Replies)
i have a Intel Quad Core Xeon X3440 (4 x 2.53GHz, 8MB Cache, Hyper Threaded) with 16gig and 1tb harddrive with a 1gb port and my apache is causing my cpu to go up to 100% on all four cores heres my http.config
<IfModule prefork.c>
StartServers 10
MinSpareServers 10
MaxSpareServers 15... (4 Replies)
I'm not sure if this belong in dummies or advanced so I made my best guess. Go easy on me if I get it wrong.
I'm trying to determine what a high load for my system is. I run a php/mysql web server with a dedicated host. The host has a Intel Xeon 3110 (Dual Core) processor.
Our load seems to... (5 Replies)
Hi guys,
I need a script that will generate a core of a process when the process uses high cpu for a sustained period?
So for example if a process is using greater than 80% cpu for more than 30 minutes do "gcore /var/tmp/process.core pid" (1 Reply)
Hi Buddies,
Thanx for reading my first post...
After googling a lot and searching so many forums I am feeling down a bit...
Please don't mind my ignorence, and my grammer ... :)
My server is running RHEL 2.6.9-5.EL. The cpu load is going higher than roof, almost 100 sometimes.
I am... (2 Replies)
Hi All.
In my production server the load is very high.
normally it used to be less than 1,but now it is more than 5.
I am new to unix all together.
I want to know what is the reason behind high load.
and if it is high what is the impact? (4 Replies)
Hi,
I am seeing very high kernel usage and very high load averages on my system (Although we are not loading much data to our database). Here is the output of top...does anyone know what i should be looking at?
Thanks,
Lorraine
last pid: 13144; load averages: 22.32, 19.81, 16.78 ... (4 Replies)
after a long period of running, the network application's CPU load in our syst em increase slowly, the failed at the end. we use "truss" tool to trace the process, found that it processes something like "semop" ,"semctl","thread_waitlock","kread" kernel call . The trace log file looks like the... (0 Replies)
