Login or Register to Ask a Question and Join Our Community


SuSE

sles 9 - sudden high load avg

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Linux SuSE sles 9 - sudden high load avg
# 1  
Old 02-04-2009
sles 9 - sudden high load avg
Hi
Running SLES 9(4) on PE 1950. I saw yesterday that the load average on the machine was 54 and keeping around that number.

Later I found there were 54 /USR/SBIN/CRON processes running in the system. I tried to kill using killall, kill -9 pid but they did not get killed. I also tried stopping cron daemon using init script but that too failed..

Finally I had to reboot the system. I have no idea what caused it and so I am here to know..if this is a known issue...

I have disabled cron, rcd, audit, acct daemons in the system. Why Acct and Audit , because these were the daemons which I had enabled 2 days back.

rcd was already running but saw error messages like below in /var/log/messages so thought of disabling that one too..
Quote:
Feb 3 02:47:24 ada -- MARK --
Feb 3 02:59:01 ada /USR/SBIN/CRON[11134]: (root) CMD ( rm -f /var/spool/cron/lastrun/cron.hourly)
Feb 3 03:26:42 ada rcd[7263]: Running heartbeat at Tue Feb 3 03:26:42 2009
Feb 3 03:26:42 ada rcd[7263]: Memory limit reached, restarting
Feb 3 03:26:42 ada rcd[7263]: Shutting down daemon...
Feb 3 03:26:42 ada rcd[7263]: Shutting down local server
Feb 3 03:26:42 ada rcd[7263]: Shutting down remote server
Feb 3 03:26:43 ada rcd[12268]: Red Carpet Daemon 2.4.9
Feb 3 03:26:43 ada rcd[12268]: Copyright (C) 2000-2003 Ximian Inc.
Feb 3 03:26:43 ada rcd[12268]: Start time: Tue Feb 3 03:26:43 2009
Feb 3 03:26:43 ada rcd[12268]: Initializing RPC system
Feb 3 03:26:43 ada rcd[12268]: Initializing modules
Feb 3 03:26:43 ada rcd[12268]: [rcd.serverpoll] Starting server-poll
Feb 3 03:26:43 ada rcd[12268]: Starting local server
Feb 3 03:26:43 ada rcd[12268]: Starting remote server
Feb 3 03:26:43 ada rcd[12268]: Loading system packages
Feb 3 03:26:44 ada rcd[12268]: Done loading system packages
Feb 3 03:26:53 ada rcd[12268]: id=3 COMPLETE 'Downloading https://update.novell.com/data/channels.php' time=7s (failed)
Feb 3 03:26:53 ada rcd[12268]: Unable to downloaded channel list: IO error - Soup error: Internal Server Error^M Date: Tue, 03 Feb 2009 09:26:53 GMT^M Server: Apache/2.0.49 (Linux/SuSE)^M Vary: accept-language,accept-charset^M Accept-Ranges: bytes^M Connection: close^M Content-Type: text/html; charset=iso-8859-1^M Content-Language: en^M X-Pad: avoid browser bug^M (500)
Feb 3 03:26:53 ada rcd[12268]: Can't find rcd 1.x subscription file '/var/lib/redcarpet/subscriptions.xml'
Feb 3 03:26:53 ada rcd[12268]: Starting heartbeat
Feb 3 03:26:54 ada rcd[12268]: [rcd.rce-privs] Unable to download privileges from https://update.novell.com/data: Authentication failed
Feb 3 03:26:58 ada rcd[12268]: [rcd.supertransaction] Error adding supertransactions to 'Novell_Update_Server': Authentication failed
Feb 3 03:27:03 ada rcd-hardware: starting up
Feb 3 03:27:03 ada rcd-hardware: Could not read audio info
Feb 3 03:27:03 ada rcd-hardware: Could not read video info
Feb 3 03:27:04 ada rcd[12268]: Send harwdware info failed: Authentication failed
Feb 3 03:27:54 ada rcd[12268]: Send package hash failed: Authentication failed
Feb 3 03:30:07 ada rcd[12268]: [rcd.hostname] Send hostname/ip failed: Authentication failed
Feb 3 03:47:24 ada -- MARK --
Feb 3 03:59:01 ada /USR/SBIN/CRON[13580]: (root) CMD ( rm -f /var/spool/cron/lastrun/cron.hourly)
Feb 3 04:08:20 ada sshd[13970]: Did not receive identification string from 218.16.239.244
Feb 3 04:14:01 ada /USR/SBIN/CRON[14199]: (root) CMD ( rm -f /var/spool/cron/lastrun/cron.daily)
Feb 3 04:15:04 ada su: (to cyrus) root on none
Feb 3 04:15:04 ada su: pam_unix2: session started for user cyrus, service su
Feb 3 04:15:04 ada ctl_mboxlist[14545]: DBERROR: reading /var/lib/imap/db/skipstamp, assuming the worst: No such file or directory
Feb 3 04:15:04 ada ctl_mboxlist[14545]: skiplist: recovered /var/lib/imap/mailboxes.db (0 records, 144 bytes) in 0 seconds
Feb 3 04:15:04 ada su: pam_unix2: session finished for user cyrus, service su
Feb 3 04:15:25 ada su: (to nobody) root on none
Feb 3 04:15:25 ada su: pam_unix2: session started for user nobody, service su
Feb 3 04:15:39 ada Server Administrator: Storage Service EventID: 2095 SCSI sense data Sense key: 5 Sense code: 24 Sense qualifier: 0: Enclosure 0:0 Controller 0, Connector 0
Feb 3 04:16:17 ada su: pam_unix2: session finished for user nobody, service su
Feb 3 04:26:53 ada rcd[12268]: [rcd.serverpoll] Error getting task info: Authentication failed
Feb 3 04:26:53 ada rcd[12268]: [rcd.serverpoll] Error polling server: Authentication failed
Feb 3 04:47:25 ada -- MARK --
Feb 3 04:59:01 ada /USR/SBIN/CRON[17518]: (root) CMD ( rm -f /var/spool/cron/lastrun/cron.hourly)
Feb 3 05:26:53 ada rcd[12268]: Running heartbeat at Tue Feb 3 05:26:53 2009
Feb 3 05:26:53 ada rcd[12268]: Memory limit reached, restarting
Feb 3 05:26:53 ada rcd[12268]: Shutting down daemon...
Feb 3 05:26:53 ada rcd[12268]: Shutting down local server
Feb 3 05:26:53 ada rcd[12268]: Shutting down remote server
Feb 3 05:26:53 ada rcd[18660]: Red Carpet Daemon 2.4.9
Feb 3 05:26:53 ada rcd[18660]: Copyright (C) 2000-2003 Ximian Inc.
Feb 3 05:26:53 ada rcd[18660]: Start time: Tue Feb 3 05:26:53 2009
Feb 3 05:26:53 ada rcd[18660]: Initializing RPC system
Feb 3 05:26:53 ada rcd[18660]: Initializing modules
Feb 3 05:26:54 ada rcd[18660]: [rcd.serverpoll] Starting server-poll
Feb 3 05:26:54 ada rcd[18660]: Starting local server
Feb 3 05:26:54 ada rcd[18660]: Starting remote server
Feb 3 05:26:54 ada rcd[18660]: Loading system packages
Feb 3 05:26:55 ada rcd[18660]: Done loading system packages
Feb 3 05:27:04 ada rcd[18660]: id=3 COMPLETE 'Downloading https://update.novell.com/data/channels.php' time=8s (failed)
Feb 3 05:27:04 ada rcd[18660]: Unable to downloaded channel list: IO error - Soup error: Internal Server Error^M Date: Tue, 03 Feb 2009 11:27:04 GMT^M Server: Apache/2.0.49 (Linux/SuSE)^M Vary: accept-language,accept-charset^M Accept-Ranges: bytes^M Connection: close^M Content-Type: text/html; charset=iso-8859-1^M Content-Language: en^M X-Pad: avoid browser bug^M (500)
Feb 3 05:27:04 ada rcd[18660]: Can't find rcd 1.x subscription file '/var/lib/redcarpet/subscriptions.xml'
Feb 3 05:27:04 ada rcd[18660]: Starting heartbeat
Feb 3 05:27:04 ada rcd[18660]: [rcd.rce-privs] Unable to download privileges from https://update.novell.com/data: Authentication failed
Feb 3 05:27:09 ada rcd[18660]: [rcd.supertransaction] Error adding supertransactions to 'Novell_Update_Server': Authentication failed
Feb 3 05:27:14 ada rcd-hardware: starting up
Feb 3 05:27:14 ada rcd-hardware: Could not read audio info
Feb 3 05:27:14 ada rcd-hardware: Could not read video info
Feb 3 05:27:14 ada rcd[18660]: Send harwdware info failed: Authentication failed
Feb 3 05:28:04 ada rcd[18660]: Send package hash failed: Authentication failed
Feb 3 05:35:04 ada rcd[18660]: [rcd.hostname] Send hostname/ip failed: Authentication failed
Feb 3 05:47:26 ada -- MARK --
Feb 3 05:59:01 ada /USR/SBIN/CRON[19967]: (root) CMD ( rm -f /var/spool/cron/lastrun/cron.hourly)
Feb 3 06:00:01 ada /USR/SBIN/CRON[20011]: (root) CMD ( /usr/lib/sa/sa2 -A)
Feb 3 06:27:05 ada rcd[18660]: [rcd.serverpoll] Error getting task info: Authentication failed
Feb 3 06:27:05 ada rcd[18660]: [rcd.serverpoll] Error polling server: Authentication failed
Feb 3 06:30:38 ada submountd: resmgr: server response code 200
Feb 3 06:30:40 ada kernel: ISO 9660 Extensions: Microsoft Joliet Level 3
Feb 3 06:30:40 ada kernel: ISO 9660 Extensions: RRIP_1991A
Feb 3 06:34:56 ada sshd[21481]: Invalid user test from 61.237.15.202
Feb 3 06:47:26 ada -- MARK --
I am wondering what is going..
# 2  
Old 02-04-2009
just tried to start cron daemon on sles 9 for the system I mentioned above,

got following in mesages file:
Quote:

Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - cron.c:129 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:81 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:92 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada last message repeated 4 times
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:103 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:81 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:92 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada last message repeated 2 times
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:103 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:81 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:92 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:92 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:103 - laus_log: (22) laus_log: Invalid argument
logwatch sent me an email (after enabling cron on this machine mentioned above)

Quote:
--------------------- Selinux Audit Begin ------------------------

**Unmatched Entries**
run notify command: /usr/sbin/audbin -S /var/log/audit.d/save.%u -C /var/log/audit.d/bin.0
started notify command, pid=25567 for file /var/log/audit.d/bin.0
Waiting for notify command (pid=25567) to complete (/var/log/audit.d/bin.0). Further processing is blocked for this time!
Notify command (pid=25567) for file '/var/log/audit.d/bin.0' completed, status 256
Notify command /usr/sbin/audbin -S /var/log/audit.d/save.%u -C exited with status 1 for bin-file 0
output error; suspending execution

---------------------- Selinux Audit End -------------------------
Last edited by upengan78; 02-04-2009 at 03:26 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Troubleshooting sudden high memory usage

Hi, This morning there was an app that caused a sudden spike in I/O and memory usage in the server. We found the reason for the I/O, however the memory spike was something new, as it had never happened before. I figured out what caused the memory spike, however, how do I investigate why... (6 Replies)
Discussion started by: anaigini45
6 Replies

2. SuSE

Newbi: High availbilty extenstion for SLES 11 on VMware

Hi Guys , Can some one help me out with the basic requirements and steps required to setting up High availabilty extension in SLES11 sp2 on vmware .:) Iam struggling with the basic installation of SLES and finally completed it after a long trilas and it will be helpful if some one do help me... (0 Replies)
Discussion started by: shiek.kaleem
0 Replies

3. Red Hat

apache high cpu load on high traffic

i have a Intel Quad Core Xeon X3440 (4 x 2.53GHz, 8MB Cache, Hyper Threaded) with 16gig and 1tb harddrive with a 1gb port and my apache is causing my cpu to go up to 100% on all four cores heres my http.config <IfModule prefork.c> StartServers 10 MinSpareServers 10 MaxSpareServers 15... (4 Replies)
Discussion started by: awww
4 Replies

4. UNIX for Advanced & Expert Users

High availability/Load balancing

Hi folks, (Sorry I don't know what its technology is termed exactly. High Availability OR load balancing) What I'm going to explore is as follows:- For example, on Physical Servers; Server-1 - LAMP, a working server Server-2 - LAMP, for redundancy While Server-1 is working all... (3 Replies)
Discussion started by: satimis
3 Replies

5. UNIX for Advanced & Expert Users

What's a high load for my system?

I'm not sure if this belong in dummies or advanced so I made my best guess. Go easy on me if I get it wrong. I'm trying to determine what a high load for my system is. I run a php/mysql web server with a dedicated host. The host has a Intel Xeon 3110 (Dual Core) processor. Our load seems to... (5 Replies)
Discussion started by: vanguard
5 Replies

6. UNIX for Dummies Questions & Answers

Determining cause behind high load average

How to determine what is causing high load average in a system? (3 Replies)
Discussion started by: proactiveaditya
3 Replies

7. HP-UX

HIgh Load

Hi All. In my production server the load is very high. normally it used to be less than 1,but now it is more than 5. I am new to unix all together. I want to know what is the reason behind high load. and if it is high what is the impact? (4 Replies)
Discussion started by: jyoti
4 Replies

8. UNIX for Advanced & Expert Users

Sun: High kernel usage & very high load averages

Hi, I am seeing very high kernel usage and very high load averages on my system (Although we are not loading much data to our database). Here is the output of top...does anyone know what i should be looking at? Thanks, Lorraine last pid: 13144; load averages: 22.32, 19.81, 16.78 ... (4 Replies)
Discussion started by: lorrainenineill
4 Replies

9. AIX

Application high CPU load

after a long period of running, the network application's CPU load in our syst em increase slowly, the failed at the end. we use "truss" tool to trace the process, found that it processes something like "semop" ,"semctl","thread_waitlock","kread" kernel call . The trace log file looks like the... (0 Replies)
Discussion started by: Frank2004
0 Replies
Login or Register to Ask a Question