sles 9 - sudden high load avg

upengan78 · #1 (**permalink**) 02-04-2009

Hi
Running SLES 9(4) on PE 1950. I saw yesterday that the load average on the machine was 54 and keeping around that number.

Later I found there were 54 /USR/SBIN/CRON processes running in the system. I tried to kill using killall, kill -9 pid but they did not get killed. I also tried stopping cron daemon using init script but that too failed..

Finally I had to reboot the system. I have no idea what caused it and so I am here to know..if this is a known issue...

I have disabled cron, rcd, audit, acct daemons in the system. Why Acct and Audit , because these were the daemons which I had enabled 2 days back.

rcd was already running but saw error messages like below in /var/log/messages so thought of disabling that one too..

Quote:

Feb 3 02:47:24 ada -- MARK --
Feb 3 02:59:01 ada /USR/SBIN/CRON[11134]: (root) CMD ( rm -f /var/spool/cron/lastrun/cron.hourly)
Feb 3 03:26:42 ada rcd[7263]: Running heartbeat at Tue Feb 3 03:26:42 2009
Feb 3 03:26:42 ada rcd[7263]: Memory limit reached, restarting
Feb 3 03:26:42 ada rcd[7263]: Shutting down daemon...
Feb 3 03:26:42 ada rcd[7263]: Shutting down local server
Feb 3 03:26:42 ada rcd[7263]: Shutting down remote server
Feb 3 03:26:43 ada rcd[12268]: Red Carpet Daemon 2.4.9
Feb 3 03:26:43 ada rcd[12268]: Copyright (C) 2000-2003 Ximian Inc.
Feb 3 03:26:43 ada rcd[12268]: Start time: Tue Feb 3 03:26:43 2009
Feb 3 03:26:43 ada rcd[12268]: Initializing RPC system
Feb 3 03:26:43 ada rcd[12268]: Initializing modules
Feb 3 03:26:43 ada rcd[12268]: [rcd.serverpoll] Starting server-poll
Feb 3 03:26:43 ada rcd[12268]: Starting local server
Feb 3 03:26:43 ada rcd[12268]: Starting remote server
Feb 3 03:26:43 ada rcd[12268]: Loading system packages
Feb 3 03:26:44 ada rcd[12268]: Done loading system packages
Feb 3 03:26:53 ada rcd[12268]: id=3 COMPLETE 'Downloading https://update.novell.com/data/channels.php' time=7s (failed)
Feb 3 03:26:53 ada rcd[12268]: Unable to downloaded channel list: IO error - Soup error: Internal Server Error^M Date: Tue, 03 Feb 2009 09:26:53 GMT^M Server: Apache/2.0.49 (Linux/SuSE)^M Vary: accept-language,accept-charset^M Accept-Ranges: bytes^M Connection: close^M Content-Type: text/html; charset=iso-8859-1^M Content-Language: en^M X-Pad: avoid browser bug^M (500)
Feb 3 03:26:53 ada rcd[12268]: Can't find rcd 1.x subscription file '/var/lib/redcarpet/subscriptions.xml'
Feb 3 03:26:53 ada rcd[12268]: Starting heartbeat
Feb 3 03:26:54 ada rcd[12268]: [rcd.rce-privs] Unable to download privileges from https://update.novell.com/data: Authentication failed
Feb 3 03:26:58 ada rcd[12268]: [rcd.supertransaction] Error adding supertransactions to 'Novell_Update_Server': Authentication failed
Feb 3 03:27:03 ada rcd-hardware: starting up
Feb 3 03:27:03 ada rcd-hardware: Could not read audio info
Feb 3 03:27:03 ada rcd-hardware: Could not read video info
Feb 3 03:27:04 ada rcd[12268]: Send harwdware info failed: Authentication failed
Feb 3 03:27:54 ada rcd[12268]: Send package hash failed: Authentication failed
Feb 3 03:30:07 ada rcd[12268]: [rcd.hostname] Send hostname/ip failed: Authentication failed
Feb 3 03:47:24 ada -- MARK --
Feb 3 03:59:01 ada /USR/SBIN/CRON[13580]: (root) CMD ( rm -f /var/spool/cron/lastrun/cron.hourly)
Feb 3 04:08:20 ada sshd[13970]: Did not receive identification string from 218.16.239.244
Feb 3 04:14:01 ada /USR/SBIN/CRON[14199]: (root) CMD ( rm -f /var/spool/cron/lastrun/cron.daily)
Feb 3 04:15:04 ada su: (to cyrus) root on none
Feb 3 04:15:04 ada su: pam_unix2: session started for user cyrus, service su
Feb 3 04:15:04 ada ctl_mboxlist[14545]: DBERROR: reading /var/lib/imap/db/skipstamp, assuming the worst: No such file or directory
Feb 3 04:15:04 ada ctl_mboxlist[14545]: skiplist: recovered /var/lib/imap/mailboxes.db (0 records, 144 bytes) in 0 seconds
Feb 3 04:15:04 ada su: pam_unix2: session finished for user cyrus, service su
Feb 3 04:15:25 ada su: (to nobody) root on none
Feb 3 04:15:25 ada su: pam_unix2: session started for user nobody, service su
Feb 3 04:15:39 ada Server Administrator: Storage Service EventID: 2095 SCSI sense data Sense key: 5 Sense code: 24 Sense qualifier: 0: Enclosure 0:0 Controller 0, Connector 0
Feb 3 04:16:17 ada su: pam_unix2: session finished for user nobody, service su
Feb 3 04:26:53 ada rcd[12268]: [rcd.serverpoll] Error getting task info: Authentication failed
Feb 3 04:26:53 ada rcd[12268]: [rcd.serverpoll] Error polling server: Authentication failed
Feb 3 04:47:25 ada -- MARK --
Feb 3 04:59:01 ada /USR/SBIN/CRON[17518]: (root) CMD ( rm -f /var/spool/cron/lastrun/cron.hourly)
Feb 3 05:26:53 ada rcd[12268]: Running heartbeat at Tue Feb 3 05:26:53 2009
Feb 3 05:26:53 ada rcd[12268]: Memory limit reached, restarting
Feb 3 05:26:53 ada rcd[12268]: Shutting down daemon...
Feb 3 05:26:53 ada rcd[12268]: Shutting down local server
Feb 3 05:26:53 ada rcd[12268]: Shutting down remote server
Feb 3 05:26:53 ada rcd[18660]: Red Carpet Daemon 2.4.9
Feb 3 05:26:53 ada rcd[18660]: Copyright (C) 2000-2003 Ximian Inc.
Feb 3 05:26:53 ada rcd[18660]: Start time: Tue Feb 3 05:26:53 2009
Feb 3 05:26:53 ada rcd[18660]: Initializing RPC system
Feb 3 05:26:53 ada rcd[18660]: Initializing modules
Feb 3 05:26:54 ada rcd[18660]: [rcd.serverpoll] Starting server-poll
Feb 3 05:26:54 ada rcd[18660]: Starting local server
Feb 3 05:26:54 ada rcd[18660]: Starting remote server
Feb 3 05:26:54 ada rcd[18660]: Loading system packages
Feb 3 05:26:55 ada rcd[18660]: Done loading system packages
Feb 3 05:27:04 ada rcd[18660]: id=3 COMPLETE 'Downloading https://update.novell.com/data/channels.php' time=8s (failed)
Feb 3 05:27:04 ada rcd[18660]: Unable to downloaded channel list: IO error - Soup error: Internal Server Error^M Date: Tue, 03 Feb 2009 11:27:04 GMT^M Server: Apache/2.0.49 (Linux/SuSE)^M Vary: accept-language,accept-charset^M Accept-Ranges: bytes^M Connection: close^M Content-Type: text/html; charset=iso-8859-1^M Content-Language: en^M X-Pad: avoid browser bug^M (500)
Feb 3 05:27:04 ada rcd[18660]: Can't find rcd 1.x subscription file '/var/lib/redcarpet/subscriptions.xml'
Feb 3 05:27:04 ada rcd[18660]: Starting heartbeat
Feb 3 05:27:04 ada rcd[18660]: [rcd.rce-privs] Unable to download privileges from https://update.novell.com/data: Authentication failed
Feb 3 05:27:09 ada rcd[18660]: [rcd.supertransaction] Error adding supertransactions to 'Novell_Update_Server': Authentication failed
Feb 3 05:27:14 ada rcd-hardware: starting up
Feb 3 05:27:14 ada rcd-hardware: Could not read audio info
Feb 3 05:27:14 ada rcd-hardware: Could not read video info
Feb 3 05:27:14 ada rcd[18660]: Send harwdware info failed: Authentication failed
Feb 3 05:28:04 ada rcd[18660]: Send package hash failed: Authentication failed
Feb 3 05:35:04 ada rcd[18660]: [rcd.hostname] Send hostname/ip failed: Authentication failed
Feb 3 05:47:26 ada -- MARK --
Feb 3 05:59:01 ada /USR/SBIN/CRON[19967]: (root) CMD ( rm -f /var/spool/cron/lastrun/cron.hourly)
Feb 3 06:00:01 ada /USR/SBIN/CRON[20011]: (root) CMD ( /usr/lib/sa/sa2 -A)
Feb 3 06:27:05 ada rcd[18660]: [rcd.serverpoll] Error getting task info: Authentication failed
Feb 3 06:27:05 ada rcd[18660]: [rcd.serverpoll] Error polling server: Authentication failed
Feb 3 06:30:38 ada submountd: resmgr: server response code 200
Feb 3 06:30:40 ada kernel: ISO 9660 Extensions: Microsoft Joliet Level 3
Feb 3 06:30:40 ada kernel: ISO 9660 Extensions: RRIP_1991A
Feb 3 06:34:56 ada sshd[21481]: Invalid user test from 61.237.15.202
Feb 3 06:47:26 ada -- MARK --

I am wondering what is going..

upengan78 · #2 (**permalink**) 02-04-2009

just tried to start cron daemon on sles 9 for the system I mentioned above,

got following in mesages file:

Quote:

Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - cron.c:129 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:81 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:92 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada last message repeated 4 times
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:103 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:81 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:92 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada last message repeated 2 times
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:103 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:81 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:92 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:92 - laus_log: (22) laus_log: Invalid argument
Feb 4 12:43:48 ada /usr/sbin/cron[29279]: LAuS error - user.c:103 - laus_log: (22) laus_log: Invalid argument

logwatch sent me an email (after enabling cron on this machine mentioned above)

Quote:

--------------------- Selinux Audit Begin ------------------------

**Unmatched Entries**
run notify command: /usr/sbin/audbin -S /var/log/audit.d/save.%u -C /var/log/audit.d/bin.0
started notify command, pid=25567 for file /var/log/audit.d/bin.0
Waiting for notify command (pid=25567) to complete (/var/log/audit.d/bin.0). Further processing is blocked for this time!
Notify command (pid=25567) for file '/var/log/audit.d/bin.0' completed, status 256
Notify command /usr/sbin/audbin -S /var/log/audit.d/save.%u -C exited with status 1 for bin-file 0
output error; suspending execution

---------------------- Selinux Audit End -------------------------

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Need help in wrting Load Script for a Load-Resume type of load.	ankitgupta	Shell Programming and Scripting	1	11-10-2006 12:46 AM
High cpu load average	squid04	Red Hat	2	09-27-2006 09:07 AM
HIgh Load	jyoti	HP-UX	4	05-01-2006 03:31 AM
Sun: High kernel usage & very high load averages	lorrainenineill	UNIX for Advanced & Expert Users	4	02-06-2006 12:32 PM
Application high CPU load	Frank2004	AIX	0	04-20-2005 11:41 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by