Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: vmstat and waiting kernel threads
Operating Systems Solaris vmstat and waiting kernel threads Post 99328 by jabberwocky on Thursday 16th of February 2006 09:58:57 AM
Old 02-16-2006
vmstat and waiting kernel threads
I am running Apache:

# httpd -version
Server version: Apache/1.3.26 (Unix)
Server built: Jul 15 2002 03:27:01

On a V280:

# uname -a
SunOS bsmweb01 5.8 Generic_108528-23 sun4u sparc SUNW,Sun-Fire-280R
#

I am assessing the server for additional applications. I ran vmstat 10 and a high number of waiting processes were reported; this output was typical of the vmstat output over a 30 minute period:

procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s6 sd sd -- in sy cs us sy id
0 0 42 2007552 1646280 0 0 0 0 0 0 0 0 0 0 0 130 47 123 0 0 100
0 0 42 2007552 1646280 0 0 0 0 0 0 0 0 2 0 0 130 42 122 0 1 99
0 0 42 2007552 1646280 0 0 0 0 0 0 0 0 0 0 0 131 46 125 0 1 99
0 0 42 2006280 1644288 106 320 73 0 0 0 0 0 0 8 0 151 1373 179 6 5 89
0 0 42 2007552 1646176 1 2 2 0 0 0 0 0 4 2 0 145 60 135 0 1 99
0 0 42 2006448 1645240 120 259 0 1 1 0 0 0 1 0 0 127 691 145 1 1 98
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 0 0 0 124 49 121 0 0 100
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 7 0 0 144 43 126 0 1 99
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 0 0 0 133 45 121 0 0 100
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 0 0 0 128 44 120 0 0 100
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 0 0 0 129 40 121 0 1 99
0 0 42 2007552 1646168 0 0 0 0 0 0 0 0 0 0 0 132 45 123 0 0 100
0 0 42 2003376 1640672 117 415 111 0 0 0 0 0 0 11 0 162 1606 213 11 1 87
0 0 42 2007552 1646080 0 0 0 0 0 0 0 0 4 2 0 134 39 124 0 1 99

CPU was consistently reporting 98-100% idle, negligible scan rate. Ample free memory was also reported. The is ample disk space on the server. Runing prstat gave no CPU/memory intensive processes over the same time period.
What causes the kernel threads to be waiting?
Will this affect server performance?
I have looked at netstat - a and found:

# netstat -a

UDP: IPv4
Local Address Remote Address State
-------------------- -------------------- -------
*.sunrpc Idle
*.* Unbound
*.32771 Idle
*.lockd Idle
*.* Unbound
*.* Unbound
*.32772 Idle
*.syslog Idle
*.* Unbound
*.630 Idle
*.2024 Idle
*.32782 Idle
*.nfsd Idle
*.177 Idle
*.32783 Idle
*.161 Idle
*.* Unbound
*.name Idle
*.biff Idle
*.talk Idle
*.time Idle
*.echo Idle
*.discard Idle
*.daytime Idle
*.chargen Idle
*.35517 Idle
*.35518 Idle
*.35519 Idle
*.35520 Idle
*.35521 Idle
*.35522 Idle
*.35523 Idle
*.35524 Idle
*.* Unbound

UDP: IPv6
Local Address Remote Address State
If
--------------------------------- --------------------------------- ---------- -
----
*.time Idle
*.echo Idle
*.discard Idle
*.daytime Idle
*.chargen Idle

TCP: IPv4
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- ----- ------ ----- ------ -------
*.* *.* 0 0 24576 0 IDLE
*.sunrpc *.* 0 0 24576 0 LISTEN
*.* *.* 0 0 24576 0 IDLE
*.lockd *.* 0 0 24576 0 LISTEN
*.32771 *.* 0 0 24576 0 LISTEN
*.32773 *.* 0 0 24576 0 LISTEN
*.fs *.* 0 0 24576 0 LISTEN
*.2201 *.* 0 0 24576 0 LISTEN
*.2200 *.* 0 0 24576 0 LISTEN
*.5987 *.* 0 0 24576 0 LISTEN
*.898 *.* 0 0 24576 0 LISTEN
*.32783 *.* 0 0 24576 0 LISTEN
*.wnn6 *.* 0 0 24576 0 LISTEN
*.9010 *.* 0 0 24576 0 LISTEN
*.8888 *.* 0 0 24576 0 LISTEN
*.32784 *.* 0 0 24576 0 LISTEN
*.nfsd *.* 0 0 24576 0 LISTEN
*.32785 *.* 0 0 24576 0 LISTEN
*.32786 *.* 0 0 24576 0 LISTEN
*.smtp *.* 0 0 24576 0 LISTEN
*.smtp *.* 0 0 24576 0 LISTEN
*.submission *.* 0 0 24576 0 LISTEN
*.* *.* 0 0 24576 0 IDLE
*.* *.* 0 0 24576 0 IDLE
*.* *.* 0 0 24576 0 IDLE
*.* *.* 0 0 24576 0 IDLE
*.* *.* 0 0 24576 0 IDLE
*.bpdbm *.* 0 0 24576 0 LISTEN
*.bpjobd *.* 0 0 24576 0 LISTEN
*.* *.* 0 0 24576 0 IDLE
*.* *.* 0 0 24576 0 IDLE
*.vmd *.* 0 0 24576 0 LISTEN
*.tldcd *.* 0 0 24576 0 LISTEN
*.bprd *.* 0 0 24576 0 LISTEN
*.46284 *.* 0 0 24576 0 LISTEN
*.ftp *.* 0 0 24576 0 LISTEN
*.telnet *.* 0 0 24576 0 LISTEN
*.shell *.* 0 0 24576 0 LISTEN
*.shell *.* 0 0 24576 0 LISTEN
*.login *.* 0 0 24576 0 LISTEN
*.exec *.* 0 0 24576 0 LISTEN
*.exec *.* 0 0 24576 0 LISTEN
*.uucp *.* 0 0 24576 0 LISTEN
*.finger *.* 0 0 24576 0 LISTEN
*.time *.* 0 0 24576 0 LISTEN
*.echo *.* 0 0 24576 0 LISTEN
*.discard *.* 0 0 24576 0 LISTEN
*.daytime *.* 0 0 24576 0 LISTEN
*.chargen *.* 0 0 24576 0 LISTEN
*.46654 *.* 0 0 24576 0 LISTEN
*.printer *.* 0 0 24576 0 LISTEN
*.dtspc *.* 0 0 24576 0 LISTEN
*.bpcd *.* 0 0 24576 0 LISTEN
*.vnetd *.* 0 0 24576 0 LISTEN
*.vopied *.* 0 0 24576 0 LISTEN
*.bpjava-msvc *.* 0 0 24576 0 LISTEN
*.hengili-poly *.* 0 0 24576 0 LISTEN
*.hengite-poly *.* 0 0 24576 0 LISTEN
*.midsoli-poly *.* 0 0 24576 0 LISTEN
*.zedali-poly *.* 0 0 24576 0 LISTEN
*.aegonli-poly *.* 0 0 24576 0 LISTEN
*.aegonte-poly *.* 0 0 24576 0 LISTEN
*.aegonde-poly *.* 0 0 24576 0 LISTEN
*.ofgemli-poly *.* 0 0 24576 0 LISTEN
*.ofgemde-poly *.* 0 0 24576 0 LISTEN
*.ofgemte-poly *.* 0 0 24576 0 LISTEN
*.reabcli-poly *.* 0 0 24576 0 LISTEN
*.reabcde-poly *.* 0 0 24576 0 LISTEN
*.midsote-poly *.* 0 0 24576 0 LISTEN
*.zedate-poly *.* 0 0 24576 0 LISTEN
*.56914 *.* 0 0 24576 0 LISTEN
*.56915 *.* 0 0 24576 0 LISTEN
*.56916 *.* 0 0 24576 0 LISTEN
*.56918 *.* 0 0 24576 0 LISTEN
*.56919 *.* 0 0 24576 0 LISTEN
*.80 *.* 0 0 24576 0 LISTEN
bsmweb01.telnet 172.17.100.187.1162 63741 1 24820 0 ESTABLISHED
*.* *.* 0 0 24576 0 IDLE
bsmweb01.52741 172.17.100.187.6000 64512 0 24820 0 ESTABLISHED
bsmweb01.telnet 172.17.100.187.1817 63291 0 24820 0 ESTABLISHED
bsmweb01.80 172.16.50.113.4734 63204 0 24820 0 FIN_WAIT_2
bsmweb01.80 172.16.50.113.4735 63981 0 24820 0 FIN_WAIT_2
bsmweb01.midsoli-poly bsmweb01.53877 32768 0 32768 0 TIME_WAIT
bsmweb01.53878 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.midsoli-poly bsmweb01.53879 32768 0 32768 0 TIME_WAIT
bsmweb01.53880 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.bpjobd bsmweb01.53881 32768 0 32768 0 TIME_WAIT
bsmweb01.951 bsmweb01.bpcd 32768 0 32768 0 TIME_WAIT
bsmweb01.914 bsmweb01.593 32768 0 32768 0 TIME_WAIT
bsmweb01.692 bsmweb01.bpcd 32768 0 32768 0 TIME_WAIT
bsmweb01.625 bsmweb01.532 32768 0 32768 0 TIME_WAIT
bsmweb01.bpcd bsmweb01.591 32768 0 32768 0 TIME_WAIT
bsmweb01.957 bsmweb01.768 32768 0 32768 0 TIME_WAIT
bsmweb01.hengili-poly bsmweb01.53883 32768 0 32768 0 TIME_WAIT
bsmweb01.53884 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.zedali-poly bsmweb01.53885 32768 0 32768 0 TIME_WAIT
bsmweb01.53886 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.hengili-poly bsmweb01.53887 32768 0 32768 0 TIME_WAIT
bsmweb01.53888 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.hengili-poly bsmweb01.53889 32768 0 32768 0 TIME_WAIT
bsmweb01.53890 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.hengili-poly bsmweb01.53891 32768 0 32768 0 TIME_WAIT
bsmweb01.53892 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.hengili-poly bsmweb01.53893 32768 0 32768 0 TIME_WAIT
bsmweb01.53894 bsmdb02.1521 49640 0 24820 0 TIME_WAIT
bsmweb01.bpjobd bsmweb01.53896 32768 0 32768 0 TIME_WAIT
bsmweb01.785 bsmweb01.bpcd 32768 0 32768 0 TIME_WAIT
bsmweb01.576 bsmweb01.995 32768 0 32768 0 TIME_WAIT
bsmweb01.866 bsmweb01.bpcd 32768 0 32768 0 TIME_WAIT
bsmweb01.908 bsmweb01.954 32768 0 32768 0 TIME_WAIT
bsmweb01.bpcd bsmweb01.900 32768 0 32768 0 TIME_WAIT
bsmweb01.nfsd bsmapp01.866 24820 0 24820 0 ESTABLISHED
bsmweb01.828 bsmweb01.670 32768 0 32768 0 TIME_WAIT
bsmweb01.80 192.168.127.9.35795 64929 0 24820 0 ESTABLISHED
*.* *.* 0 0 24576 0 IDLE

TCP: IPv6
Local Address Remote Address Swind Send-Q
Rwind Recv-Q State If
--------------------------------- --------------------------------- ----- ------
----- ------ ----------- -----
*.* *.* 0 0
24576 0 IDLE
*.smtp *.* 0 0
24576 0 LISTEN
*.ftp *.* 0 0
24576 0 LISTEN
*.telnet *.* 0 0
24576 0 LISTEN
*.shell *.* 0 0
24576 0 LISTEN
*.login *.* 0 0
24576 0 LISTEN
*.exec *.* 0 0
24576 0 LISTEN
*.finger *.* 0 0
24576 0 LISTEN
*.time *.* 0 0
24576 0 LISTEN
*.echo *.* 0 0
24576 0 LISTEN
*.discard *.* 0 0
24576 0 LISTEN
*.daytime *.* 0 0
24576 0 LISTEN
*.chargen *.* 0 0
24576 0 LISTEN
*.printer *.* 0 0
24576 0 LISTEN

Active UNIX domain sockets
Address Type Vnode Conn Local Addr Remote Addr
300035a1958 stream-ord 30003589720 00000000 /tmp/jd_sockV6
300035a1b08 stream-ord 30002fff8f0 00000000 /dev/ccv
300035a1cb8 stream-ord 30002fffac8 00000000 /dev/kkcv

The web applications on this server connect to oracle databases on another Sun server, bsmdb02 using port 1521. The status of alot of these connections is TIME_WAIT, although the OS process (from the prstat command) only exists for the duration of the http transaction.
Is the fact that there are connections reported at TIME_WAIT for completed transactions with no evidence from prstat causing waiting kernel threads to be reported?

Thanks,

Jabberwocky
 

9 More Discussions You Might Find Interesting

1. SuSE

max number of slabs per kernel module (kernel 2.6.17, suse)

Hi All, Is there a max number of slabs that can be used per kernel module? I'm having a tough time finding out that kind of information, but the array 'node_zonelists' (mmzone.h) has a size of 5. I just want to avoid buffer overruns and other bad stuff. Cheers, Brendan (4 Replies)
Discussion started by: Brendan Kennedy
4 Replies

2. UNIX for Advanced & Expert Users

Threads and Threads Count ?

Hi all, How can I get the list of all Threads and the Total count of threads under a particular process ? Do suggest !! Awaiting for the replies !! Thanks Varun:b: (2 Replies)
Discussion started by: varungupta
2 Replies

3. Linux

vmstat help

Hi everyone, I need to see some VM manager performance/behavior information on some Linux boxes regarding pages scanned/activation of the paging algorithm in order to get an idea if a given server needs more memory and is actually paging. In Aix servers, by using the vmstat cmd you... (1 Reply)
Discussion started by: jcpetela
1 Replies

4. UNIX for Advanced & Expert Users

Passing socket struct between kernel threads - module programming

I write kernel module with kernel threads using linux/kthread.h on 2.6.* kernel I tried to passing data between two kernel threads with data argument of kthread_run( fun, data , NAME ); but this is not work I dont know why. I tried many possibility and nothing works. So I thought that... (0 Replies)
Discussion started by: marcintom
0 Replies

5. Solaris

Which file is read by kernel to set its default system kernel parameters values?

Hi gurus Could anybody tell me which file is read by kernel to set its default system kernal parameters values in solaris. Here I am not taking about /etc/system file which is used to load kernal modules or to change any default system kernal parameter value Is it /dev/kmem file or something... (1 Reply)
Discussion started by: girish.batra
1 Replies

6. Programming

Number of threads waiting on a pthread mutex/rwlock

Using pthreads is there a way to determine how many threads are waiting on a locked resource? I mean, once a shared resource is protected using e.g. pthread_rwlock_t or pthread_mutex_t one thread grabs the lock and other threads will go to sleep waiting for the resource to be available again. Is... (0 Replies)
Discussion started by: muggu
0 Replies

7. AIX

AIX: mkdev is waiting in kernel at et_wait

Hi All, bash-3.2# ps -eaf | grep mkdev root 4378752 1548526 0 03:27:04 - 0:00 /usr/sbin/mkdev -c disk -s vscsi -t xyz -l virtualdev bash-3.2# Here we are trying to create a virtual device using mkdev and the device is already populated in the namespace and the corresponding ODMs... (1 Reply)
Discussion started by: Muppana Prasad
1 Replies

8. Linux

Unload kernel module at boot time (Debian Wheezy 7.2, 3.2.0-4-686-pae kernel)

Hi everyone, I am trying to prevent the ehci_hcd kernel module to load at boot time. Here's what I've tried so far: 1) Add the following line to /etc/modprobe.d/blacklist.conf (as suggested here): 2) Blacklisted the module by adding the following string to 3) Tried to blacklist the module... (0 Replies)
Discussion started by: gacanepa
0 Replies

9. Linux

Vmstat

I m checking idle time using vmstat, below are the results var=$(ssh wmtmgr@$hostname vmstat | tail -1 | awk '{print $15}') 89 and now im subtracting 89 with 100 & im getting expected results expr 100 - $var 11 Now How can I get the result 11 in one line code? (4 Replies)
Discussion started by: sam@sam
4 Replies
All times are GMT -4. The time now is 01:23 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy