I have a AIX system (5.3) and it is at 100% usage of ram
now how to deal with this, without affecting the system, because it's a production system ?
here is the information about the system:
Code:
bash-3.2# svmon
size inuse free pin virtual
memory 4194304 4191918 2386 571269 1356682
pg space 1572864 420673
work pers clnt other
pin 343634 0 0 227635
in use 1073927 108493 3009498
MB used: 16380.0 MB
MB free: 4.0 MB
│ Memory ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────│
│ Physical PageSpace | pages/sec In Out | FileSystemCache │
│% Used 100.0% 26.8% | to Paging Space 4.0 15.0 | (numperm) 73.7% │
│% Free 0.0% 73.2% | to File System 706.0 165.5 | Process 15.7% │
│MB Used 16379.9MB 1644.6MB | Page Scans 1074.0 | System 10.6% │
│MB Free 4.1MB 4499.4MB | Page Cycles 0.0 | Free 0.0% │
│Total(MB) 16384.0MB 6144.0MB | Page Steals 779.5 | ------ │
│ | Page Faults 440.0 | Total 100.0% │
│------------------------------------------------------------ | numclient 71.5% │
│Min/Maxperm 3044MB( 19%) 12176MB( 74%) <--% of RAM | maxclient 74.3% │
│Min/Maxfree 960 1088 Total Virtual 22.0GB | User 83.9% │
│Min/Maxpgahead 2 8 Accessed Virtual 5.2GB 23.5%| Pinned 13.6%
bash-3.2# ps aux | sort -r -nk6
root 671802 0.2 0.0 127040 44064 - A Apr 15 18:28 /usr/java5_64/j
root 307452 0.1 0.0 54492 43060 - A Apr 17 2:31 /app/dlc_oe/bin
root 700510 0.1 0.0 36308 37688 - A 01:02:10 0:36 /app/dlc_oe/bin
CUP9999 1032252 0.1 0.0 48348 36208 - A 03:43:01 0:33 /usr/java5_64/j
CUP9999 704648 0.7 0.0 28548 34320 - A 19:43:04 9:31 /app/dlc_oe/bin
root 884892 0.1 0.0 29512 33676 - A 01:02:10 0:37 /app/dlc_oe/bin
root 1015878 0.1 0.0 36008 31760 - A Apr 16 3:35 /app/dlc_oe/bin
root 1036444 0.1 0.0 35200 31204 - A Apr 15 8:54 /app/dlc_oe/bin
root 1052774 0.1 0.0 37412 30968 - A Apr 15 6:19 /app/dlc_oe/bin
root 770188 0.1 0.0 35972 30232 - A Apr 15 6:15 /app/dlc_oe/bin
root 1040438 0.2 0.0 67012 30128 - A Apr 15 15:03 /usr/java5_64/j
root 356584 0.1 0.0 35224 29408 - A Apr 15 9:12 /app/dlc_oe/bin
root 319514 0.1 0.0 55064 26108 - A Apr 15 11:41 /usr/java5_64/j
CUP9999 585800 0.0 0.0 16512 22416 - A 19:43:03 0:11 /app/dlc_oe/bin
CUP9999 446548 0.0 0.0 14932 21372 - A 07:03:05 0:00 /app/dlc_oe/bin
CUP9999 987360 0.0 0.0 15096 21188 - A 19:43:03 0:01 /app/dlc_oe/bin
CUP9999 741512 0.0 0.0 14672 20760 - A 19:43:04 0:04 /app/dlc_oe/bin
CUP9999 286872 0.0 0.0 14100 20512 - A 19:43:00 0:09 /app/dlc_oe/bin
CUP9999 221350 0.0 0.0 14096 20360 - A 19:43:03 0:09 /app/dlc_oe/bin
CUP9999 159804 0.0 0.0 12616 18048 - A 19:43:05 0:02 /app/dlc_oe/bin
mqm 676022 1.1 0.0 26724 17112 - A Feb 04 2408:44 /usr/java5_64/b
root 503910 0.0 0.0 17512 15148 - A Feb 04 68:43 /opt/CA/DSM/Age
root 516252 0.0 0.0 22788 14840 - A Feb 04 49:59 /opt/CA/DSM/Age
root 495624 0.1 0.0 51704 14528 - A Feb 04 109:09 /opt/ibm/direct
root 692362 0.0 0.0 29984 10508 - A Apr 15 2:27 /usr/java5_64/j
root 1048632 0.0 0.0 30632 10356 - A Apr 15 2:27 /usr/java5_64/j
I would suggest some tuning of your box in first instance and than if you still have memory problems - add memory ?
If you have free memory on your frame and your lpar is configured accordingly (like the maximum memory in lparstat -i output being higher than the desired memory) you can do this dynamically.
I have put your posting into code tags - there is a link how to use them.
To your question:
AIX as well as other UNIX derivates and Linux distributions uses a part of it's RAM as file cache. So it is absolutely normal to have a close to 100% usage of RAM.
What would be a problem is when your box starts to page.
From your posting you can see, that a part of your Paging Space is being used:
Code:
size inuse
... ...
pg space 1572864 420673
So for some reason the OS had the need to page out some memory pages to disk as there was a demand for free RAM.
When this happens, you can monitor this by using things like
Code:
vmstat -tw 2 10
for example. There are two columns called pi and po, which stand for pages being paged out to disk and pages being paged in from disk which is somewhat slow in case they are needed again to be loaded into RAM (pi).
If there is any other values but 0, then you know that your box is paging. Even low values can sometimes mean problems on a busy system, as said especially when being paged in.
For long time monitoring it might be a good idea to use nmon which can be downloaded from IBM for free as it didn't come out of the box with AIX 5.3. There is also additional tools (nmon2rrd I think) to create graphs from the data to see at what time which impact happened.
As your AIX 5.3 might not be tuned, it could also be interessting what output the following command produces:
Remember that the OS will keep the memory full. You should concentrate on the paging space usage more. You have little relative to real memory (about 10%) where most people will run with at least double paging space to real memory (or allocated memory for an LPAR). 100% full in itself is not a problem. It doesn't cause a performance loss unless you end up paging. A "full" page in memory can still be counted even if it is not in use. It's just there in case it is re-requested.
A requirement for a new memory page will overwrite an obsolete one. If all the memory pages are marked as in-use, then the last recently referenced page is moved to disk (I/O wait) and the real page re-used.
I doubt you have more that /dev/h6 assigned for paging (issue lsps -a) so you can't easily see what paging space is actually marked as required. Over version 6 of AIX, the paging space is cleared when the owning process ends. For version 5, you can (if you have more than one paging space) issue a swapoff command which will work through the names paging space, moving 'live' pages to any remaining paging space. You can then run swapon and then do the same with the other paging spaces.
The time you need to worry is if you are seeing the server in I/O wait and lots of paging activity. Use vmstat to spot this. You are looking at the pi and po columns. Large numbers tells you that you have a problem. We had this recently and from using ps -el|sort -n +9 we could see the biggest memory allocation process and then had to work out what they were. You may need to check the column being sorted. It is the SZ column.
We determined that the Cobol written application was not releasing memory and indeed increased it every time certain routines ran, and again and again if they ran again in the process's life. A small coding change later helped tremendously.
I hope that this calms your fears. After that, you are into purchasing memory I'm afraid, most often in pairs of cards, e.g. 2x4Gb card.
To add another paging space there are smit panels to help you under "System Storage Management (Physical & Logical Storage)" or you can read the mkps and chps manual pages.
I hoep that this helps, but please write back if you are still stuck.
Thx everyone for your replies !!!!
I was off for a couple of days and forgot to reply.
I was thinking that first the Ram will first be used and then the swap files, but I forgot this is not Windows machine
Anyway, this is all interesting. At the moment I am busy with Solaris systems and have a couple of AIX systems running. Those systems will be now managed by our AIX guy and I can concentrate more on the Solaris systems.
Thx.
AIX Version 6.1 and 7.1.
I understand that when the OS initially creates the FS and inodes, its pretty strict, but not always tuned to a 1:1 ratio. I see the same thing when adding a whole disk LV to a separate device.
It seems that when we expand a filesystem the inodes don't get tuned... (5 Replies)
Hi Experts,
Our servers running Solaris 10 with SAP Application. The memory utilization always >90%, but the process on SAP is too less even nothing.
Why memory utilization on solaris always looks high?
I have statement about memory on solaris, is this true:
Memory in solaris is used for... (4 Replies)
Is it possible to restrict physical memory in solaris zone with zone.max-locked-memory just like we can do with rcapd ? I do not want to used rcapd (1 Reply)
Hi,
I'm trying to learn how to manage memory when I have to deal with lots of data.
Basically I'm indexing a huge file (5GB, but it can be bigger), by creating tables that
holds offset <-> startOfSomeData information. Currently I'm mapping the whole file at
once (yep!) but of course the... (1 Reply)
Hi,
Im working on Solaris 9 on SPARC-32 bit running on an Ultra-80, and I have to find out the following:-
1. Total Physical Memory in the system(total RAM).
2. Available Physical Memory(i.e. RAM Usage)
3. Total (Logical) Memory in the system
4. Available (Logical) Memory.
I know... (4 Replies)
Hi
I'be recently installed Virtouzzo on Centos 5 on 16GB box , but the system could only see 4 GB of RAM, I installed the package kernel-PAE, but the virtuozzo kernel still can't see the full memory.
even the kernel system can see 16GB of RAM
is there any idea bout that ?
Thanks (2 Replies)
Hi I wrote a script
#!/usr/bin/ksh
#set -x
for fs in `df -k|awk '{print $1}'|sed -n "3,14 p"`
do
x=`df -kl | grep $fs | awk '{ print $5 }'`
y=50%
if
then
message="File System `df -k |grep $fs |awk '{print $6\", \"$5}'`... (1 Reply)
Hi
When I launch topas I have this result
I have a memory problem. I don't know how see the process which waiting.
How can I do that?
Thanks all!!! (0 Replies)