Login or Register to Ask a Question and Join Our Community


AIX

Memory full

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems AIX Memory full
# 1  
Old 04-18-2012
Memory full
I need second opinion:

I have a AIX system (5.3) and it is at 100% usage of ram
now how to deal with this, without affecting the system, because it's a production system ?
here is the information about the system:

Code:
bash-3.2# svmon
                  size                 inuse        free         pin     virtual
memory      4194304     4191918        2386      571269     1356682
pg space    1572864      420673

               work           pers           clnt           other
pin          343634           0           0           227635
in use      1073927      108493     3009498

MB used: 16380.0 MB
MB free:    4.0 MB

│ Memory ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│
│          Physical  PageSpace |        pages/sec  In     Out | FileSystemCache                                                                                                                                                                │
│% Used      100.0%     26.8%  | to Paging Space   4.0   15.0 | (numperm) 73.7%                                                                                                                                                                │
│% Free        0.0%     73.2%  | to File System  706.0  165.5 | Process   15.7%                                                                                                                                                                │
│MB Used   16379.9MB  1644.6MB | Page Scans     1074.0        | System    10.6%                                                                                                                                                                │
│MB Free       4.1MB  4499.4MB | Page Cycles       0.0        | Free       0.0%                                                                                                                                                                │
│Total(MB) 16384.0MB  6144.0MB | Page Steals     779.5        |           ------                                                                                                                                                               │
│                              | Page Faults     440.0        | Total    100.0%                                                                                                                                                                │
│------------------------------------------------------------ | numclient 71.5%                                                                                                                                                                │
│Min/Maxperm     3044MB( 19%)  12176MB( 74%) <--% of RAM      | maxclient 74.3%                                                                                                                                                                │
│Min/Maxfree     960   1088       Total Virtual   22.0GB      | User      83.9%                                                                                                                                                                │
│Min/Maxpgahead    2      8    Accessed Virtual    5.2GB 23.5%| Pinned    13.6%  

bash-3.2# ps aux | sort -r -nk6
root      671802  0.2  0.0 127040 44064      - A      Apr 15 18:28 /usr/java5_64/j
root      307452  0.1  0.0 54492 43060      - A      Apr 17  2:31 /app/dlc_oe/bin
root      700510  0.1  0.0 36308 37688      - A    01:02:10  0:36 /app/dlc_oe/bin
CUP9999  1032252  0.1  0.0 48348 36208      - A    03:43:01  0:33 /usr/java5_64/j
CUP9999   704648  0.7  0.0 28548 34320      - A    19:43:04  9:31 /app/dlc_oe/bin
root      884892  0.1  0.0 29512 33676      - A    01:02:10  0:37 /app/dlc_oe/bin
root     1015878  0.1  0.0 36008 31760      - A      Apr 16  3:35 /app/dlc_oe/bin
root     1036444  0.1  0.0 35200 31204      - A      Apr 15  8:54 /app/dlc_oe/bin
root     1052774  0.1  0.0 37412 30968      - A      Apr 15  6:19 /app/dlc_oe/bin
root      770188  0.1  0.0 35972 30232      - A      Apr 15  6:15 /app/dlc_oe/bin
root     1040438  0.2  0.0 67012 30128      - A      Apr 15 15:03 /usr/java5_64/j
root      356584  0.1  0.0 35224 29408      - A      Apr 15  9:12 /app/dlc_oe/bin
root      319514  0.1  0.0 55064 26108      - A      Apr 15 11:41 /usr/java5_64/j
CUP9999   585800  0.0  0.0 16512 22416      - A    19:43:03  0:11 /app/dlc_oe/bin
CUP9999   446548  0.0  0.0 14932 21372      - A    07:03:05  0:00 /app/dlc_oe/bin
CUP9999   987360  0.0  0.0 15096 21188      - A    19:43:03  0:01 /app/dlc_oe/bin
CUP9999   741512  0.0  0.0 14672 20760      - A    19:43:04  0:04 /app/dlc_oe/bin
CUP9999   286872  0.0  0.0 14100 20512      - A    19:43:00  0:09 /app/dlc_oe/bin
CUP9999   221350  0.0  0.0 14096 20360      - A    19:43:03  0:09 /app/dlc_oe/bin
CUP9999   159804  0.0  0.0 12616 18048      - A    19:43:05  0:02 /app/dlc_oe/bin
mqm       676022  1.1  0.0 26724 17112      - A      Feb 04 2408:44 /usr/java5_64/b
root      503910  0.0  0.0 17512 15148      - A      Feb 04 68:43 /opt/CA/DSM/Age
root      516252  0.0  0.0 22788 14840      - A      Feb 04 49:59 /opt/CA/DSM/Age
root      495624  0.1  0.0 51704 14528      - A      Feb 04 109:09 /opt/ibm/direct
root      692362  0.0  0.0 29984 10508      - A      Apr 15  2:27 /usr/java5_64/j
root     1048632  0.0  0.0 30632 10356      - A      Apr 15  2:27 /usr/java5_64/j


Moderator's Comments:
Mod Comment Please use code tags, thanks!
Last edited by zaxxon; 04-18-2012 at 08:19 AM.. Reason: code tags
# 2  
Old 04-18-2012
I would suggest some tuning of your box in first instance and than if you still have memory problems - add memory ?
If you have free memory on your frame and your lpar is configured accordingly (like the maximum memory in lparstat -i output being higher than the desired memory) you can do this dynamically.

Regards
zxmaus
This User Gave Thanks to zxmaus For This Post:
misterx12345
# 3  
Old 04-18-2012
I have put your posting into code tags - there is a link how to use them.

To your question:

AIX as well as other UNIX derivates and Linux distributions uses a part of it's RAM as file cache. So it is absolutely normal to have a close to 100% usage of RAM.

What would be a problem is when your box starts to page.
From your posting you can see, that a part of your Paging Space is being used:
Code:
            size         inuse
            ...          ...
pg space    1572864      420673

So for some reason the OS had the need to page out some memory pages to disk as there was a demand for free RAM.
When this happens, you can monitor this by using things like
Code:
vmstat -tw 2 10

for example. There are two columns called pi and po, which stand for pages being paged out to disk and pages being paged in from disk which is somewhat slow in case they are needed again to be loaded into RAM (pi).
If there is any other values but 0, then you know that your box is paging. Even low values can sometimes mean problems on a busy system, as said especially when being paged in.
For long time monitoring it might be a good idea to use nmon which can be downloaded from IBM for free as it didn't come out of the box with AIX 5.3. There is also additional tools (nmon2rrd I think) to create graphs from the data to see at what time which impact happened.

As your AIX 5.3 might not be tuned, it could also be interessting what output the following command produces:
Code:
vmo -o maxclient% -o maxperm% -o minperm% -o lru_file_repage

You can also have a look at this:
https://www.ibm.com/developerworks/w....pdf?version=2
This User Gave Thanks to zaxxon For This Post:
misterx12345
# 4  
Old 04-18-2012
Remember that the OS will keep the memory full. You should concentrate on the paging space usage more. You have little relative to real memory (about 10%) where most people will run with at least double paging space to real memory (or allocated memory for an LPAR). 100% full in itself is not a problem. It doesn't cause a performance loss unless you end up paging. A "full" page in memory can still be counted even if it is not in use. It's just there in case it is re-requested.

A requirement for a new memory page will overwrite an obsolete one. If all the memory pages are marked as in-use, then the last recently referenced page is moved to disk (I/O wait) and the real page re-used.

I doubt you have more that /dev/h6 assigned for paging (issue lsps -a) so you can't easily see what paging space is actually marked as required. Over version 6 of AIX, the paging space is cleared when the owning process ends. For version 5, you can (if you have more than one paging space) issue a swapoff command which will work through the names paging space, moving 'live' pages to any remaining paging space. You can then run swapon and then do the same with the other paging spaces.


The time you need to worry is if you are seeing the server in I/O wait and lots of paging activity. Use vmstat to spot this. You are looking at the pi and po columns. Large numbers tells you that you have a problem. We had this recently and from using ps -el|sort -n +9 we could see the biggest memory allocation process and then had to work out what they were. You may need to check the column being sorted. It is the SZ column.

We determined that the Cobol written application was not releasing memory and indeed increased it every time certain routines ran, and again and again if they ran again in the process's life. A small coding change later helped tremendously.

I hope that this calms your fears. After that, you are into purchasing memory I'm afraid, most often in pairs of cards, e.g. 2x4Gb card.

To add another paging space there are smit panels to help you under "System Storage Management (Physical & Logical Storage)" or you can read the mkps and chps manual pages.


I hoep that this helps, but please write back if you are still stuck.

Robin
Liverpool/Blackburn
UK
This User Gave Thanks to rbatte1 For This Post:
misterx12345
# 5  
Old 04-23-2012
Thx everyone for your replies !!!!
I was off for a couple of days and forgot to reply.
I was thinking that first the Ram will first be used and then the swap files, but I forgot this is not Windows machineSmilie
Anyway, this is all interesting. At the moment I am busy with Solaris systems and have a couple of AIX systems running. Those systems will be now managed by our AIX guy and I can concentrate more on the Solaris systems.
Thx.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. AIX

100% Inode full with only 67% FS full.

AIX Version 6.1 and 7.1. I understand that when the OS initially creates the FS and inodes, its pretty strict, but not always tuned to a 1:1 ratio. I see the same thing when adding a whole disk LV to a separate device. It seems that when we expand a filesystem the inodes don't get tuned... (5 Replies)
Discussion started by: mrmurdock
5 Replies

2. Solaris

[DOUBT] Memory high in idle process on Solaris 10 (Memory Utilization > 90%)

Hi Experts, Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing. Why memory utilization on solaris always looks high? I have statement about memory on solaris, is this true: Memory in solaris is used for... (4 Replies)
Discussion started by: edydsuranta
4 Replies

3. Shell Programming and Scripting

How to extract strings from full path when full path is not fixed

/Path/snowbird9/nrfCompMgrRave1230100920.log.gz:09/20/2010 06:14:51 ERROR Error Message. /Path/snowbird6/nrfCompMgrRave1220100920.log.gz:09/20/2010 06:14:51 ERROR Error Message. /Path/snowbird14/nrfCompMgrRave920100920.log.gz:09/20/2010 06:14:51 ERROR Error Message.... (0 Replies)
Discussion started by: Shirisha
0 Replies

4. Solaris

restrcit physical memory with zone.max-locked-memory

Is it possible to restrict physical memory in solaris zone with zone.max-locked-memory just like we can do with rcapd ? I do not want to used rcapd (1 Reply)
Discussion started by: fugitive
1 Replies

5. Programming

How to deal with lots of data in memory in order not to run out of memory

Hi, I'm trying to learn how to manage memory when I have to deal with lots of data. Basically I'm indexing a huge file (5GB, but it can be bigger), by creating tables that holds offset <-> startOfSomeData information. Currently I'm mapping the whole file at once (yep!) but of course the... (1 Reply)
Discussion started by: emitrax
1 Replies

6. Solaris

How to find Total and Free Physical Memory and Logical Memory in SOLARIS 9

Hi, Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:- 1. Total Physical Memory in the system(total RAM). 2. Available Physical Memory(i.e. RAM Usage) 3. Total (Logical) Memory in the system 4. Available (Logical) Memory. I know... (4 Replies)
Discussion started by: 0ktalmagik
4 Replies

7. Linux

my box can't see full memory

Hi I'be recently installed Virtouzzo on Centos 5 on 16GB box , but the system could only see 4 GB of RAM, I installed the package kernel-PAE, but the virtuozzo kernel still can't see the full memory. even the kernel system can see 16GB of RAM is there any idea bout that ? Thanks (2 Replies)
Discussion started by: Raied
2 Replies

8. Shell Programming and Scripting

memory full warning message

Hi I wrote a script #!/usr/bin/ksh #set -x for fs in `df -k|awk '{print $1}'|sed -n "3,14 p"` do x=`df -kl | grep $fs | awk '{ print $5 }'` y=50% if then message="File System `df -k |grep $fs |awk '{print $6\", \"$5}'`... (1 Reply)
Discussion started by: namishtiwari
1 Replies

9. Filesystems, Disks and Memory

Memory full (waiting)

Hi When I launch topas I have this result I have a memory problem. I don't know how see the process which waiting. How can I do that? Thanks all!!! (0 Replies)
Discussion started by: Castelior
0 Replies
Login or Register to Ask a Question