No Space Left - Memory/Swap issue


 
Thread Tools Search this Thread
Operating Systems Solaris No Space Left - Memory/Swap issue
# 15  
Old 02-17-2011
Quote:
Originally Posted by jlliagre
That depends on what definition of swap you take. When swaps means virtual memory like in the statistics posted, it does.Not necessarily. The rule is there must be enough virtual memory for all reservation to fit. The amount of memory needed is not related to the amount of RAM installed but is related to the number of applications and their requirements.
So in other words, my remaining physical memory and swap is spoken for. It may never be used, but it's spoken for.

Here is a pmap of our one of our processes:
Code:
 Address  Kbytes     RSS    Anon  Locked Mode   Mapped File
00010000   23248   22416       -       - r-x--  frmweb
016D2000     632     600     144       - rwx--  frmweb
01770000     576       -       -       - rwx--  frmweb
01800000     344       8       8       - rwx--  frmweb
01856000    7848    5072    3936       - rwx--    [ heap ]
FE920000      72      72      32       - rw--R  dev:256,65540 ino:33028
FE940000     136     136      48       - rw--R  dev:256,65540 ino:42199
FE970000      72      72      32       - rw--R  dev:256,65540 ino:33028
FE990000     240     240       -       - r-x--  libresolv.so.2
FE9DC000      16      16      16       - rwx--  libresolv.so.2
FE9F0000      16      16       -       - r-x--  nss_dns.so.1
FEA04000       8       8       8       - rwx--  nss_dns.so.1
FEA10000     408     408     120       - rw--R  dev:256,65540 ino:46165
FEA80000    1296     512       -       - r-x--  libX11.so.4
FEBD4000      24      24      16       - rwx--  libX11.so.4
FEBE0000       8       8       8       - rw---  libX11.so.4
FEC00000     336     256       -       - r-x--  libXt.so.4
FEC64000      24      24      24       - rwx--  libXt.so.4
FEC70000       8       8       8       - rw---  libXt.so.4
FEC80000    2096     872       -       - r-x--  libXm.so.4
FEE9C000      88      88      88       - rwx--  libXm.so.4
FEEB2000       8       -       -       - rwx--  libXm.so.4
FEEC0000      64      64      64       - rwx--    [ anon ]
FEEE0000      32      32       -       - r-x--  nss_files.so.1
FEEF8000       8       8       8       - rwx--  nss_files.so.1
FEF00000     584     584       -       - r-x--  libnsl.so.1
FEFA2000      40      40      40       - rwx--  libnsl.so.1
FEFAC000      24      16      16       - rwx--  libnsl.so.1
FEFC0000      64      64      64       - rwx--    [ anon ]
FEFE0000      64      16      16       - rwx--    [ anon ]
FF000000    1416    1120       -       - r-x--  libnnz10.so
FF170000     112     112      88       - rwx--  libnnz10.so
FF190000       8       8       8       - rwx--    [ anon ]
FF1A0000      48      48       -       - r-x--  libsocket.so.1
FF1BC000       8       8       8       - rwx--  libsocket.so.1
FF1D0000      16      16       -       - r-x--  libthread.so.1
FF1E0000       8       8       -       - r-x--  libdl.so.1
FF1F2000       8       8       8       - rwx--  libdl.so.1
FF200000    1216    1216       -       - r-x--  libc.so.1
FF330000      40      40      40       - rwx--  libc.so.1
FF33A000       8       8       8       - rwx--  libc.so.1
FF340000      24      16      16       - rwx--    [ anon ]
FF350000      96      40       -       - r-x--  libXext.so.0
FF370000       8       8       8       - rwx--    [ anon ]
FF378000       8       8       8       - rwx--  libXext.so.0
FF380000       8       -       -       - rw---  libXext.so.0
FF390000       8       8       8       - rwx--    [ anon ]
FF3A0000       8       8       -       - r-x--  libc_psr.so.1
FF3B0000     208     208       -       - r-x--  ld.so.1
FF3E8000       8       8       -       - rwxs-    [ anon ]
FF3F0000       8       8       -       - r--s-  dev:256,65539 ino:46641
FF3F4000       8       8       8       - rwx--  ld.so.1
FF3F6000       8       8       8       - rwx--  ld.so.1
FFBF0000      64      64      64       - rwx--    [ stack ]
-------- ------- ------- ------- -------
total Kb   41736   34664    4976       -

We have about 1090 of these at peak. Is there any way to show how much a process has reserved as well as actually consumed?
# 16  
Old 02-17-2011
Quote:
Originally Posted by aychbee45
So in other words, my remaining physical memory and swap is spoken for. It may never be used, but it's spoken for.
Precisely. Solaris doesn't overcommit memory so make sure any reservation (i.e. malloc) made is backed by free either physical or swap space.

A command the ouput you missed to post is:
Code:
swap -s

It gives the whole virtual memory usage.

Note that a process copying a large file to /tmp might also cause the "no space left" message you see, as tmpfs is backed by virtual memory.
This User Gave Thanks to jlliagre For This Post:
# 17  
Old 02-17-2011
Here is a swap -s from a previous time we received the error.

Code:
total: 15836552k bytes allocated + 6354480k reserved = 22191032k used, 157288k available

I'm still baffled why we wouldn't see sr though.
# 18  
Old 02-17-2011
What "sr" ?
You only have 157MB available here.
# 19  
Old 02-17-2011
And the output of swap -l might be informative as well.
# 20  
Old 02-17-2011
What kind of application are you running. Is it possible that you hit any kind of memory limit in the application (java?) ?
# 21  
Old 02-17-2011
Quote:
Originally Posted by jlliagre
What "sr" ?
You only have 157MB available here.
Well, normally when heavy swapping starts to occur, vmstat will show high numbers in vmstat under the "sr" or scan reads column. Ours is 0 across the board.

Also, "top" shows 8GB of swap as "free". and "swap -l" shows everything but 62MB as free. I'm assuming top and swap -l are showing similar values, but it's odd that both would show that swap is almost untouched.

---------- Post updated at 09:44 PM ---------- Previous update was at 09:43 PM ----------

Here is our output from "top" at the time of one of the occurances.

Code:
load averages:  2.73,  2.14,  2.27;                    up 107+12:52:45 11:55:01
1815 processes: 1812 sleeping, 1 zombie, 2 on cpu
CPU states: 85.8% idle,  9.3% user,  4.9% kernel,  0.0% iowait,  0.0% swap
Memory: 24G phys mem, 1147M free mem, 8192M total swap, 8132M free swap

   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
 19698 root       1  42    0   93M   37M cpu/0   17.0H  4.54% lpd
  7171 oracle     1  53    2 5112K 2880K cpu/10   0:00  0.72% top
  8288 oracle     1  29   10   69M   39M sleep    0:24  0.70% frmweb
 20073 oracle     1  29   10   81M   43M sleep    0:42  0.58% frmweb
  7585 oracle     1  29   10   51M   37M sleep    0:05  0.41% frmweb
   176 daemon    10  59    0 6768K 3968K sleep  372:10  0.36% kcfd
  4086 oracle   999  29   10  819M  518M sleep   37.1H  0.33% java
  3538 oracle     1  29   10   49M   38M sleep    0:03  0.33% frmweb
 24481 oracle     1  25   10   63M   39M sleep    0:14  0.32% frmweb
 17152 oracle     1  29   10   63M   39M sleep    0:15  0.27% frmweb
  3518 oracle     1  29   10   72M   40M sleep    0:26  0.25% frmweb
 12031 oracle     1  29   10   90M   39M sleep    0:35  0.25% frmweb
  8866 oracle     1  29   10   67M   36M sleep    0:22  0.21% frmweb
 12763 oracle     1  29   10   98M   41M sleep    0:52  0.21% frmweb
  4949 applmgr    1  29   10   72M   43M sleep    0:03  0.19% f60webmx

This all just seems off to me.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Linux

No space left on device while there is plenty of space available

Hello all posting here after scanning the net and tried most of the things offered still no solution that worked when I do : $ df -h Filesystem Size Used Avail Use% Mounted on footmpfs 7.9G 60K 7.9G 1% /dev tmpfs 7.9G 0 7.9G 0% /dev/shm /dev/da1 ... (3 Replies)
Discussion started by: umen
3 Replies

2. AIX

AIX swap space, physical memory & cache

Hi, I am new to AIX, Can someone please help me how to know the swap space, total physical memory and system cache? We are using AIX 5.3. Thanks! (3 Replies)
Discussion started by: Phaneendra G
3 Replies

3. Linux

Swap memory issue

Hi, In our production box i can see the Swap space using the below command free total used free shared buffers cached Mem: 65963232 41041084 24922148 0 877160 35936292 -/+ buffers/cache: 4227632 61735600 Swap: 4192880 ... (6 Replies)
Discussion started by: ratheeshjulk
6 Replies

4. UNIX for Advanced & Expert Users

Swap space issue

I have Solaris-10 with mutiple zones running in it. My Big Brother monitoring is complaining for very less swap space available, but I am not able to find, what process has consumed its swap space and how to clear it. All zones including global server have almost blank /tmp with very less data.... (3 Replies)
Discussion started by: solaris_1977
3 Replies

5. HP-UX

Swap space issue.

Hi, I am not sure how many scripts / java processes running on my HP-UX server. I need to calculate the total heap of these processes. I then need to recommend increasing the swap memory to be increase and equal to total heap if that is the right concept. Currently we are facing... (2 Replies)
Discussion started by: mohtashims
2 Replies

6. Solaris

No space left on device but free space and inodes are available...

hi guys, me again ;) i recently opened a thread about physical to zone migration. My zone is mounted over a "bigger" LUN (500GB) and step is now to move the old files, from the physical server, to my zone. We are talking about 22mio of files. i used rsync to do that and every time at... (8 Replies)
Discussion started by: beta17
8 Replies

7. Solaris

swap space issue

HI All, Recently during oracle install I realized that I did not have enough swap space. So I - 1. Created a swap file "swap_fille1" in /rpool using mkfile - # ls -ltr /rpool total 10487121 drwxr-xr-x 3 root root 3 Dec 21 12:09 boot drwxr-xr-x 2 root root ... (10 Replies)
Discussion started by: sumeet
10 Replies

8. UNIX for Advanced & Expert Users

Issue with insufficient swap or memory space

Hi, When I execute one of my shellscript I am getting the below mentioned error message .This application takes 2input files which have the records counts 26463 and 1178046 exec(2): insufficient swap or memory available. exec(2): insufficient swap or memory available. exec(2): insufficient swap... (3 Replies)
Discussion started by: kavithakuttyk
3 Replies

9. Shell Programming and Scripting

Issue with insufficient swap or memory space

Hi, When I execute one of my shellscript I am getting the below mentioned error message .This application takes 2input files which have the records counts 26463 and 1178046 exec(2): insufficient swap or memory available. exec(2): insufficient swap or memory available. exec(2):... (2 Replies)
Discussion started by: kavithakuttyk
2 Replies

10. HP-UX

swap space (lvol2) is not using when physical memory is 98% used

Dear all I have rp7620 sever with hp-ux 11.23 this swap space is not yet used when the physical memory usage of 98 %. through openview it's shows memory bottleneck . how to resolve the problem. Rajesh (1 Reply)
Discussion started by: rajeshtt32
1 Replies
Login or Register to Ask a Question