Solaris 10 - script creating huge spike in Kernel CPU


 
Thread Tools Search this Thread
Operating Systems Solaris Solaris 10 - script creating huge spike in Kernel CPU
# 1  
Old 03-03-2010
Solaris 10 - script creating huge spike in Kernel CPU

I'm running on Solaris 10, and I have a script that's running on several machines. Basically, what it's doing is:

* tail -f | grep one or more log files into a temp file
* Every minute or so, copy that temp file to a second temp and zero the first
* Sed through the 2nd temp to pull out a user ID
* grep through the file for occurences of that ID with some other text (the other text is in a local file that I read from in a loop)
* Output a text record into a log file that basically consists of the User, that text, and a count.

Another server then does a tail -f of the output log file.

The user cpu on this script is small... on the order of 2-4% depending on what box I'm running it on. But the crazy thing is that on my test box, with one input data stream, as soon as I start running the script, my kernal % jumps from like 1% to 60%.

As far as I can see, the issue is NOT with disk i/o wait, or with memory (still half free and very very small paging values).
Code:
# sar -g 5 5

SunOS ssdev01 5.10 138889-03 i86pc    03/03/2010

13:09:01  pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
13:09:06     0.20     0.80     0.60     0.00     0.00
13:09:11     0.40     1.00     0.80     0.00     0.00
13:09:16     0.20     0.80     0.80     0.00     0.00
13:09:21     0.20     0.80     0.60     0.00     0.00
13:09:26     0.60     1.80     1.40     0.00     0.00

Average      0.32     1.04     0.84     0.00     0.00

Pre-run top:
Code:
load averages:  0.05,  0.27,  0.66                                                      13:25:39
69 processes:  68 sleeping, 1 on cpu
CPU states: 96.7% idle,  2.5% user,  0.8% kernel,  0.0% iowait,  0.0% swap
Memory: 2048M real, 1126M free, 486M swap in use, 2652M swap free

During run:
Code:
load averages:  0.68,  0.39,  0.68                                                      13:26:19
78 processes:  75 sleeping, 1 running, 1 zombie, 1 on cpu
CPU states: 20.5% idle, 18.3% user, 61.2% kernel,  0.0% iowait,  0.0% swap
Memory: 2048M real, 1123M free, 491M swap in use, 2647M swap free

here's the script in ps:
Code:
# /usr/ucb/ps -aux |more     
USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
ops      15436  4.7  0.1 1524 1028 ?        S 13:25:51  0:06 /usr/bin/ksh /sscp

Any ideas what I can look at to see what's chewing all the kernel?
# 2  
Old 03-03-2010
That's a good job for dtrace.

Any difference in configuration between that machine and the others ? Like say ZFS compression enabled ?
# 3  
Old 03-04-2010
Does the kernel percentage go down when you run the script a second time?
(The question is about caching).

Does your test system have a lesser number of CPUs than your production system? Does your test system have more than one CPU?

You could always post the script, making sure that we know how big these files are.

Hmm. With only 69 processes running this computer is tiny. Less processes than a Windows PC when it is doing nothing.

In general 100% utilisation of a CPU is only a problem if other processes are waiting for CPU.

What is your output from:

Code:
sar -u

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Extremely high kernel CPU Usage (Solaris 10 SPARC)

I've got a domain running on a few boards of a 25k. I'm seeing very high kernel cpu usage in top and cant' quite explain it. System runs a large number of smallish Oracle 10g2 databases (30), used mainly for development. load average: 36.63, 36.68, 37.42 2489 processes: 2452 sleeping, 21... (0 Replies)
Discussion started by: utopiajoe
0 Replies

2. Solaris

Multi CPU Solaris system shows 100% CPU usage.

Hello Friends, On one of my Solaris 10 box, CPU usage shows 100% using "sar", "vmstat". However, it has 4 CPUs and prstat and glance are not showing enough processes to justify high CPU utilization. ========================================================================= $ prstat -a ... (4 Replies)
Discussion started by: mahive
4 Replies

3. Solaris

How to check CPU spike between certain priod of time

Hi, Does anyone know how to check which process has been used the most CPU between certain time period? say I have noticed CPU spike happend between 1:00 to 1:30 now it is 3:00 o'clock already. Is there anyway to find out what process/PID use the most CPU then? Thanks a lot (3 Replies)
Discussion started by: uuontario
3 Replies

4. UNIX for Dummies Questions & Answers

Apache causing CPU to spike when POSTing

Hello all. I've built a SAMP server on Solaris 10 using Apache 2.2.9 and PHP 5.2.6 For the most part everything seems to be working except when I post form data, Apache causes the CPU to max out. Sometimes the script will complete about 40 seconds later. Other times I have to stop Apache to get... (4 Replies)
Discussion started by: RobertSubnet
4 Replies

5. Programming

Solaris CPU/memory status monitoring (Shell script or c++)

i'm trying to find a way to monitor the CPU/Memory status of a solaris station using vmstat. I like to write a small script to periodically run vmstat and store the output. Can anyone show me how (preferrably in C++ if possible)? Thanks in advance. (6 Replies)
Discussion started by: shingpui
6 Replies

6. Ubuntu

kernel-huge

Hi All, I have a uname -a Linux caws101arop 2.6.9-67.0.20.ELsmp #1 SMP Wed Jun 18 12:35:02 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux And I'm having lots of problem especially when the web server got new two additional website. Before the memory was just 12G but we upgraded it to 65G. Do... (1 Reply)
Discussion started by: itik
1 Replies

7. Programming

Much time in insmod of huge kernel module

I'm using Linux-2.6.14. My application is having one kernel module of large size(approx 8MB), insmoding of that kernel module is taking pretty much time(approx 8Mins). Is there a way to reduce the insmod time? I tried even by modprobe also. (0 Replies)
Discussion started by: ptprabu
0 Replies

8. Solaris

Find any spike in CPU usage a hour ago

Folks, I need to find out from the system logs ( or elsewhere ) if the CPU utilization reached say 90% an hour ago from now. Is there a way out. (2 Replies)
Discussion started by: kavera
2 Replies

9. Linux

Kernel panic - not syncing: cannot execute a PAE-enabled kernel on PAE-less CPU

ok so I just installed fedora core 6 on my dell inspiron 700m and I go to boot into linux and I get this error. Has anyone seen this before? I also had XP Pro and Vista installed on this pc prior to putting fedora core 6 on the machine. I'm trying to setup a triple boot system. Please Help... (2 Replies)
Discussion started by: dave043
2 Replies

10. UNIX for Advanced & Expert Users

prngd consumes huge CPU

I've noticed that the prngd process seems to consume lots of CPU every now and again on our Sparc Solaris 8 machines - I've done a google search and others too have experienced this. I have yet to find a reason or a fix. Does anyone here know why this happens? Once I stop ssh and restart it prngd... (1 Reply)
Discussion started by: ozzmosiz
1 Replies
Login or Register to Ask a Question