Swap space issue

solaris_1977 · #1 09-11-2012

I have Solaris-10 with mutiple zones running in it. My Big Brother monitoring is complaining for very less swap space available, but I am not able to find, what process has consumed its swap space and how to clear it. All zones including global server have almost blank /tmp with very less data.

HTML Code:

root@apps_serv201:/# df -h /tmp
Filesystem             size   used  avail capacity  Mounted on
swap                   512M    88K   512M     1%    /tmp
root@apps_serv201:/# /usr/sbin/swap -s
total: 75843704k bytes allocated + 39689080k reserved = 115532784k used, 1395416k available
root@apps_serv201:/# /bin/vmstat 1 2 | tail -1
 16 0 0 1394080 43647304 81 135 0 0 0 0  0  0  0  0  0 40221 49765 43543 10 4 86
root@apps_serv201:/# /usr/sbin/swap -l
swapfile             dev  swaplo blocks   free
/dev/zvol/dsk/rpool/swap 256,1      16 25165808 24877088

I downloaded a script called swapinfo and that gave me this info --

PHP Code:


root@apps_serv201:/# /var/tmp/swapinfo
RAM  _____Total 131072.0 Mb
RAM    Unusable  2436.1 Mb
RAM      Kernel 12859.0 Mb
RAM      Locked 98162.2 Mb
RAM        Used -25007.9 Mb
RAM       Avail 42622.5 Mb
Disk _____Total 12288.0 Mb
Disk      Alloc   141.0 Mb
Disk       Free 12147.0 Mb
Swap _____Total 114179.5 Mb
Swap      Alloc 74060.4 Mb
Swap    Unalloc 38733.0 Mb
Swap      Avail  1386.1 Mb
Swap  (MinFree) 16100.7 Mb
root@apps_serv201:/#

PHP Code:


root@apps_serv201:/# echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     995817              7779    6%
ZFS File Data              322707              2521    2%
Anon                      9483981             74093   58%
Exec and libs               10027                78    0%
Page cache                 253496              1980    2%
Free (cachelist)            33201               259    0%
Free (freelist)           5387854             42092   33%
Total                    16487083            128805
Physical                 16465391            128635

Please help

jlliagre · #2 09-11-2012

prstat -Z output would be a good start to investigate.

solaris_1977 · #3 09-12-2012

Here is the output -

PHP Code:


root@apps_serv201:/# prstat -Z
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 29612 55966    1750M 1748M cpu13   43    0 145:47:05 6.7% java/107
 22734 56006    1895M  777M cpu11    0    0  46:52:39 3.1% java/77
 11587 root     3792K 3416K cpu30   59    0   0:01:08 0.7% prstat/1
 25262 24688    7549M 7543M sleep   59    0   3:05:56 0.2% java/119
 20275 afpwebld 1756M 1736M sleep   59    0   1:24:08 0.1% java/145
 16368 bfdeploy  305M  231M sleep   59    0  93:19:46 0.1% java/62
  5775 afpwebld 1782M 1677M sleep   59    0  39:01:56 0.1% java/158
 16517 bfdeploy  313M  240M sleep   59    0  63:37:00 0.1% java/61
 15018 bfdeploy  297M  224M sleep   59    0  92:27:23 0.1% java/61
 22392 24688    3583M 3531M sleep   59    0   0:43:12 0.1% java/42
 15581 bfdeploy  310M  236M sleep   59    0  80:05:05 0.1% java/61
 15951 bfdeploy  297M  223M sleep   59    0  51:41:11 0.1% java/61
 15070 root       24M   17M sleep   59    0  11:03:26 0.1% scopeux/1
 16432 bfdeploy  309M  235M sleep   59    0  89:34:56 0.1% java/62
 14175 bfdeploy  305M  231M sleep   59    0  90:38:38 0.1% java/62
 27816 56755    2503M 2368M sleep   59    0   2:53:30 0.0% java/162
 14078 bfdeploy  349M  275M sleep   59    0  67:15:16 0.0% java/62
 29647 atswebld  854M  531M sleep   59    0  34:47:03 0.0% java/86
 15158 bfdeploy  304M  207M sleep   59    0  31:28:58 0.0% java/61
 20249 56695    2459M 1284M sleep   59    0   0:05:56 0.0% java/160
 25257 24688     559M  555M sleep   59    0   0:16:57 0.0% java/17
 22367 24688     559M  555M sleep   59    0   0:09:51 0.0% java/17
 15162 dwfawlsd 1386M 1031M sleep   59    0  29:14:38 0.0% java/49
 20393 55687     966M  329M sleep   59    0   0:05:19 0.0% java/47
 15163 dwfawlsd 1390M 1011M sleep   59    0  29:14:36 0.0% java/55
 11930 bfdeploy  284M  205M sleep   59    0  29:26:49 0.0% java/66
 16894 bfdeploy  397M  222M sleep   59    0  17:13:26 0.0% java/82
 16907 bfdeploy  353M  252M sleep   59    0  17:08:15 0.0% java/82
  2669 bfdeploy  385M  240M sleep   59    0   9:31:25 0.0% java/77
 16879 bfdeploy  356M  246M sleep   59    0  17:08:26 0.0% java/82
ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE
     8       42 9038M 6008M   4.6% 232:05:53 6.8% apps_serv201-zinct01
    12       48   13G 8618M   6.6% 117:34:39 3.2% apps_serv201-zddoad01
     0      127  429M  524M   0.4% 800:44:24 0.8% global
    26       31   12G   12G   9.4%   6:22:41 0.3% apps_serv201-zttmst01
    16       23 5385M 4992M   3.8%  46:02:35 0.2% apps_serv201-zafpd01
     7       42 9744M 7183M   5.5%  99:14:50 0.1% apps_serv201-zincd01
     2       47 3719M 2142M   1.6% 118:56:06 0.1% apps_serv201-ztipt01
     5       43 3818M 2363M   1.8% 125:51:14 0.1% apps_serv201-zautd01
    18       32 3606M 1866M   1.4% 105:51:10 0.1% apps_serv201-zpscd01
     1       48 3702M 2069M   1.6% 115:06:58 0.1% apps_serv201-ztipd01
     6       45 8121M 3598M   2.7%   5:26:59 0.1% apps_serv201-zbid03
Total: 886 processes, 10843 lwps, load averages: 7.22, 7.85, 6.22
root@apps_serv201:/#

jlliagre · #4 09-12-2012

So you have about 72 GB of virtual memory used by zones, 5 of them use more than 8 GB, I would start by investigating them.

In any case, there doesn't seem to have real paging issue as most of the swap area is free. You might want to add more swap area though to avoid running out of virtual memory.

The swapinfo script you downloaded gives bogus results (-25GB used RAM looks unrealistic

)

Thread Tools	Search this Thread
Show Printable Version Email this Page Subscribe to this Thread	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Swap space issue.	mohtashims	HP-UX	2	09-06-2012 05:08 PM
No Space Left - Memory/Swap issue	aychbee45	Solaris	21	02-18-2011 02:37 AM
swap space issue	sumeet	Solaris	10	12-30-2010 02:12 PM
Issue with insufficient swap or memory space	kavithakuttyk	UNIX for Advanced & Expert Users	3	09-25-2009 01:02 PM
Issue with insufficient swap or memory space	kavithakuttyk	Shell Programming and Scripting	2	09-24-2009 03:59 AM