Strange Real Memory usage behavior


 
Thread Tools Search this Thread
Operating Systems AIX Strange Real Memory usage behavior
# 1  
Old 05-29-2007
Bug Strange Real Memory usage behavior

We have 2 Rs6000 S-85
Each initially had/have 6 processors and 8 GIG of RAM each
Node 1 we added 14 processors and 32 GIG of RAM On May 19. (11 days ago)
My memory utilization reporting for node 1 showed a spike in available memory 25-30GB for May 19 to 25 . I kind of expected this because I did not think the app would try to take any more memory. Then suddenly since the 26th, available Real memory took a sudden drop down to 2 GB!

Virtual memory up until the 26th was hovering around 80% since then and now it is at 99%.

Application on the node is is Health care System called Cerner. It also Runs MQ Series Middleware.

It is almost like the system realized it had the memory to use Saturday morning and has since then decided to use it!

I have not yet added paging space for this memory, if it is needed. I need to look into that. I believe there is an economy of scale or breaking point for paging. Currently paging space is at 8192MB currently.

The memory and CPU we added were a total surprise. Our vender may not have known what they sent us. We thought we were getting 4 more CPU and anothe 8 GIG of RAM. Instead we got 14 CPU and 32GIG of RAM! Christmas in May!

We are going to give node 2 the same treatment for memory and CPU (which strictly runs a 300 GB Oracle database)

I am just not sure what to make of node 1 behaviour.
Any comments are appreciated. SkyyBugg



Smilie
# 2  
Old 05-30-2007
Can you or have you tried a re-boot. Also set NMON running (If you have not already) to analyse what app is taking most of the memory. Also with 32Gb of memory your paging is now far to small it should be 48 Gb.
# 3  
Old 05-30-2007
It looks like when I list top proecesses on nmon, no single process ever takes more than 1% of the memory. Even then, only one or two processes use use 1% at once. The utilization report that uses nmon says that there is 2.6 GB of 40GB Available. It sounds like that nmon AIX "sees" the memory but the application os is actually not using all of it.

If you are intrested attached is some output from nmon.
The m option seemns to be reporting differently than the t option.

I don't plan to change paging because no paging is taking place and it will take alot of my disk space to build paging at 80GB.
# 4  
Old 05-31-2007
You need to run NMON over a 24 hour period and use the NMON analyser to produce a set of Excel graphs which will show a comprehensive set of statistics for the environment. Using NMON the way you have will give no useful information. I take it the machine has been re-booted?
# 5  
Old 05-31-2007
John thanks for the perspective and questions.
May 19: I shut down the box. IBM Installed the memory, then I booted up the box. Nmon reports show available memory hovered around 33-35GB.

I have CPU/memory monitoring data from nmon each hour 24/7/365. This is how I was able to see that on May 26 through today available memory went from 35GB all the way down to 2.5 GB.
My attachement is currently what is typical as far as how much memory the application is using. You can see that there is no paging. The box is not breaking a sweat.

A caveat. We are still running AIX 5.1 ML04. In a week we will upgrade to AIX 5.3 TL04. This series of downtimes were set to upgrade harware, Oracle, AIX, MQ, CPU/memory.
# 6  
Old 05-31-2007
Tools

Also, I have put in an Excell Sheet the past 30 days of nmon CPU/Memory.
It is intresing to see what the CPU/ Memory, application adjustments did to this box. It is starke improvement to say the least!

SkyyBugg
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Strange Ctrl+C behavior

Hello All, I have a strange issue. I've created a shell script which connects to RMAN (Oracle Recovery Manager) and executes full DB backup. I then executed this script with nohup and in the background: $ nohup my_script.sh > logfile.log 2>&1 &The issue is that when I tried to take a look into... (6 Replies)
Discussion started by: JackK
6 Replies

2. Shell Programming and Scripting

Strange behavior of grep

Hi All, I am facing a strange problem while grepping for a process. Here is the small script that i have written. It will look for any process running with the parameter passed to the script. If no process is running it should print appropriate message. $ cat t.ksh #!/bin/ksh set -x ... (9 Replies)
Discussion started by: veeresh_15
9 Replies

3. AIX

Strange behavior with tar

I am trying to create an archive using tar. I am specifying a list of directories using the -L option. For testing purposes I created a simple directory structure: /backup/test /backup/test/test1 /backup/test/test2 The file specified by the -L option, named files.txt, contains:... (8 Replies)
Discussion started by: judykstra
8 Replies

4. Shell Programming and Scripting

Strange behavior on one of my server

I am not sure what is wrong, but I have some strange behavior when printing things out. I do create a file with only one word test, no space, no new line etc. nano file<enter> test<ctrl x>y<enter> Server 1 gets (fail) awk '{print "+"$0"*"}' file *test Server 2 gets (OK) awk '{print... (9 Replies)
Discussion started by: Jotne
9 Replies

5. AIX

Strange memory behavior

Hello together, i have a strange memory behavior on a AIX 7.1 System, which i cannot explain. The Filesystem-Cache will not be grow up and drops often after few minutes. I know if a file was deleted, that the same segment in the FS-Cache will also be cleared. But i am not sure if this is the... (8 Replies)
Discussion started by: -=XrAy=-
8 Replies

6. Ubuntu

Ubuntu strange behavior

It is so till login screen. I mean that when I boot my computer, Ubuntu shows a splash screen with mouse instead of Ubuntu logo and in the login screen it shows XUbuntu login screen... It began when I upgraded to previous kernel, I suppose, but I'm not sure... I can't say that it annoys me very... (6 Replies)
Discussion started by: Sapfeer
6 Replies

7. Programming

Strange behavior in C++

I have the following program: int main(int argc, char** argv){ unsigned long int mean=0; for(int i=1;i<10;i++){ mean+=poisson(12); cout<<mean<<endl; } cout<<"Sum of poisson: "<< mean; return 0; } when I run it, I get the... (4 Replies)
Discussion started by: santiagorf
4 Replies

8. AIX

How to monitor the IBM AIX server for I/O usage,memory usage,CPU usage,network..?

How to monitor the IBM AIX server for I/O usage, memory usage, CPU usage, network usage, storage usage? (3 Replies)
Discussion started by: laknar
3 Replies

9. AIX

Real memory usage is reaching 99.99%

Hi i am new to aix , we are using aix 5.3 version and model is 510 and 520 model servers. in our servers contains 8 gb ram. but servers average ram usage is reaching 99.9% , can any body help on this , is any impact if memory usage reach 99.9%. Regards Balaji Poola (2 Replies)
Discussion started by: Balajipoola007
2 Replies

10. HP-UX

how can I find cpu usage memory usage swap usage and logical volume usage

how can I find cpu usage memory usage swap usage and I want to know CPU usage above X% and contiue Y times and memory usage above X % and contiue Y times my final destination is monitor process logical volume usage above X % and number of Logical voluage above can I not to... (3 Replies)
Discussion started by: alert0919
3 Replies
Login or Register to Ask a Question