Visit Our UNIX and Linux User Community


SNMP responses failing under high system load


 
Thread Tools Search this Thread
Special Forums UNIX and Linux Applications Infrastructure Monitoring SNMP responses failing under high system load
# 1  
Old 11-03-2011
SNMP responses failing under high system load

Greetings,

I've got a Zenoss v2.5 server monitoring a large video encoding farm. Needless to say, these systems are under high bandwidth and CPU utilization the majority of the time.

What I'm running into is that, occasionally, these systems will fail to respond to a standard SNMP request, thereby throwing "SNMP agent down" errors in Zenoss, and generating lots of otherwise unnecessary alerts. Then, the next time the system is polled, it works, and a clear message is also sent (generating even more alerts).

Short of nice-ing the snmpd process down so that it doesn't get completely blocked by the video encoding, what would be the best way to handle this, either via configuring Zenoss, SNMP, or the servers themselves? I don't see an obvious solution to this puzzle.. Smilie
# 2  
Old 11-03-2011
How to fix it depends on why it's not responding.

If UDP packets are actually lost due to network overload, I'm not sure you can fix that. Is it possible to get your monitoring system to retry SNMP at least once instead of sending a failure message?

If the SNMP process just isn't responding in time due to CPU overload, then nice-ing your video processes to reduce their priority will do the job. Reducing something's priority is a better idea than increasing something else's since reducing your own privilege doesn't need root privileges. Low-priority jobs still get 100% CPU when nothing else competes with them, so you shouldn't lose throughput on a system that doesn't have other intensive tasks.

If it's not responding in time due to disk thrashing, I'm less sure how to deal with that; the server literally can't respond in time since things need to be loaded from an already-occupied disk first...

Last edited by Corona688; 11-03-2011 at 02:06 PM..

Previous Thread | Next Thread
Test Your Knowledge in Computers #976
Difficulty: Medium
In April 2014, Linus Torvalds banned Kay Sievers from submitting patches to the Linux kernel for failing to deal with bugs that caused systemd to negatively interact with the kernel.
True or False?

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Module load failing

I can use the "module load xyz" command interactively, but when run from a script, it says it can't find the "module" command. Is there any way of loading modules in a script? (2 Replies)
Discussion started by: karan8810
2 Replies

2. UNIX for Advanced & Expert Users

While trying to load .so file manually using command its failing

Hi all, I am newbie to linux environment. I was trying to run an .so file manually which in turn call a method in bin folder. Command given, XXX_MODULES=libxxx.so /opt/servicename/bin/methodname -Le -c /opt/servicename/etc/methodname/methodname.conf -n -C -t -m "" When i tried to execute... (1 Reply)
Discussion started by: sharathpadman
1 Replies

3. UNIX for Dummies Questions & Answers

While trying to load .so file manually using command its failing

Hi all, I am newbie to linux environment. I was trying to run an .so file manually which in turn call a method in bin folder. Command given, XXX_MODULES=libxxx.so /opt/servicename/bin/methodname -Le -c /opt/servicename/etc/methodname/methodname.conf -n -C -t -m "" When i tried to... (1 Reply)
Discussion started by: sharathpadman
1 Replies

4. UNIX for Dummies Questions & Answers

Log files @ high load

Hi, my VPS was overloaded and inaccessible for some time and i want to ask for help in which log files i need to look, or which tools to setup to monitor and find the cause of repeated hig load? watched: /var/log/messages /var/log/secure /var/log/httpd/access_log /var/log/httpd/error_log... (1 Reply)
Discussion started by: postcd
1 Replies

5. Red Hat

apache high cpu load on high traffic

i have a Intel Quad Core Xeon X3440 (4 x 2.53GHz, 8MB Cache, Hyper Threaded) with 16gig and 1tb harddrive with a 1gb port and my apache is causing my cpu to go up to 100% on all four cores heres my http.config <IfModule prefork.c> StartServers 10 MinSpareServers 10 MaxSpareServers 15... (4 Replies)
Discussion started by: awww
4 Replies

6. UNIX for Advanced & Expert Users

High availability/Load balancing

Hi folks, (Sorry I don't know what its technology is termed exactly. High Availability OR load balancing) What I'm going to explore is as follows:- For example, on Physical Servers; Server-1 - LAMP, a working server Server-2 - LAMP, for redundancy While Server-1 is working all... (3 Replies)
Discussion started by: satimis
3 Replies

7. UNIX for Advanced & Expert Users

What's a high load for my system?

I'm not sure if this belong in dummies or advanced so I made my best guess. Go easy on me if I get it wrong. I'm trying to determine what a high load for my system is. I run a php/mysql web server with a dedicated host. The host has a Intel Xeon 3110 (Dual Core) processor. Our load seems to... (5 Replies)
Discussion started by: vanguard
5 Replies

8. HP-UX

HIgh Load

Hi All. In my production server the load is very high. normally it used to be less than 1,but now it is more than 5. I am new to unix all together. I want to know what is the reason behind high load. and if it is high what is the impact? (4 Replies)
Discussion started by: jyoti
4 Replies

9. UNIX for Advanced & Expert Users

Sun: High kernel usage & very high load averages

Hi, I am seeing very high kernel usage and very high load averages on my system (Although we are not loading much data to our database). Here is the output of top...does anyone know what i should be looking at? Thanks, Lorraine last pid: 13144; load averages: 22.32, 19.81, 16.78 ... (4 Replies)
Discussion started by: lorrainenineill
4 Replies

Featured Tech Videos