Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

Performance Bottleneck in server, Need help

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Performance Bottleneck in server, Need help
# 1  
Old 02-24-2017
Wrench Performance Bottleneck in server, Need help
We are wondering if we are facing performance issue in our server when running Informatica jobs. Two things to suspect:
  • cache memory never comes down even when Top shows > 99% used.
  • There is some contention io or network related or Cache is clogged
Code:
top - 20:58:20 up 16 days,  4:37, 16 users,  load average: 7.50, 4.85, 3.82
Tasks: 386 total,   2 running, 376 sleeping,   7 stopped,   1 zombie
Cpu(s): 35.1%us, 10.1%sy,  0.0%ni, 32.5%id, 20.9%wa,  0.0%hi,  1.3%si,  0.2%st
Mem:  32877500k total, 32626692k used,   250808k free,   291804k buffers
Swap: 20971516k total,    37056k used, 20934460k free, 18906004k cached

Code:
free -g
             total       used       free     shared    buffers     cached
Mem:            31         31          0          0          0         18
-/+ buffers/cache:         12         18
Swap:           19          0         19


If We kill all Informatica services & jobs... the utilization in Top comes down to

Code:
top - 20:16:17 up 16 days,  3:55, 18 users,  load average: 1.26, 0.94, 0.85
Tasks: 366 total,   1 running, 357 sleeping,   7 stopped,   1 zombie
Cpu(s):  8.0%us,  3.9%sy,  0.0%ni, 88.0%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32877500k total, 19378856k used, 13498644k free,   225320k buffers
Swap: 20971516k total,    37060k used, 20934456k free, 18426624k cached


A) I am not sure why the cache memory is not released ?
B) Even when utilization in Top is going to 32 GB i.e. when Informatica jobs are running, Cache is still at 18GB ... Shouldn't cache be released because Informatica jobs are hanging ?


I strongly feel this is nothing to do with Server memory because I see swap memory not come into play at all but before I go to Informatica and raise a flag... I want to make sure there is nothing from server side... what more can I do to ensure that everything on server side works fine i.e. CPU & IO or Network, as I am not able to think of any other factor affecting this from server side?
# 2  
Old 02-24-2017
Quote:
Originally Posted by smart_guy471
We are wondering if we are facing performance issue in our server when running Informatica jobs. Two things to suspect:
  • cache memory never comes down even when Top shows > 99% used.
  • There is some contention io or network related or Cache is clogged
Code:
top - 20:58:20 up 16 days,  4:37, 16 users,  load average: 7.50, 4.85, 3.82
Tasks: 386 total,   2 running, 376 sleeping,   7 stopped,   1 zombie
Cpu(s): 35.1%us, 10.1%sy,  0.0%ni, 32.5%id, 20.9%wa,  0.0%hi,  1.3%si,  0.2%st
Mem:  32877500k total, 32626692k used,   250808k free,   291804k buffers
Swap: 20971516k total,    37056k used, 20934460k free, 18906004k cached

Code:
free -g
             total       used       free     shared    buffers     cached
Mem:            31         31          0          0          0         18
-/+ buffers/cache:         12         18
Swap:           19          0         19


If We kill all Informatica services & jobs... the utilization in Top comes down to

Code:
top - 20:16:17 up 16 days,  3:55, 18 users,  load average: 1.26, 0.94, 0.85
Tasks: 366 total,   1 running, 357 sleeping,   7 stopped,   1 zombie
Cpu(s):  8.0%us,  3.9%sy,  0.0%ni, 88.0%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32877500k total, 19378856k used, 13498644k free,   225320k buffers
Swap: 20971516k total,    37060k used, 20934456k free, 18426624k cached


A) I am not sure why the cache memory is not released ?
B) Even when utilization in Top is going to 32 GB i.e. when Informatica jobs are running, Cache is still at 18GB ... Shouldn't cache be released because Informatica jobs are hanging ?


I strongly feel this is nothing to do with Server memory because I see swap memory not come into play at all but before I go to Informatica and raise a flag... I want to make sure there is nothing from server side... what more can I do to ensure that everything on server side works fine i.e. CPU & IO or Network, as I am not able to think of any other factor affecting this from server side?
Your bottleneck is IO. Compare between under load and normal load:
20.9%wa -> That's the percentage the CPU had to wait for IO. (Slow)
0.1%wa -> Normal load.

I am going to assume that this is a Linux kernel operating system based on how top looks; you did not say:
cached means memory not used and since it would be wasted, the kernel is using it for caching disk. At any time needed by any application use, it will be taken. You should not be concerned with it. You should considerate it free memory.
# 3  
Old 02-26-2017
Quote:
Originally Posted by Aia
Your bottleneck is IO. Compare between under load and normal load:
20.9%wa -> That's the percentage the CPU had to wait for IO. (Slow)
0.1%wa -> Normal load.

I am going to assume that this is a Linux kernel operating system based on how top looks; you did not say:
cached means memory not used and since it would be wasted, the kernel is using it for caching disk. At any time needed by any application use, it will be taken. You should not be concerned with it. You should considerate it free memory.

Sorry I forgot to mention that its Red Hat Linux & We are running this from Amazon Cloud.. so not sure if the IO problem is due to the box being in cloud. Is there a way to isolate which process is getting affected due to IO problem ?

Regards,
Jey
Last edited by smart_guy471; 02-26-2017 at 08:32 PM.. Reason: typo
# 4  
Old 02-27-2017
Take a look at iotop.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Server performance

Hi Guys, I am running Oracle database on RHEL 2.6.18-164.el5, now I want to check and make sure that my server is performing optimally. I check top: top - 09:45:03 up 2 days, 15:22, 3 users, load average: 2.57, 2.85, 2.77 Tasks: 433 total, 3 running, 430 sleeping, 0 stopped, 0... (1 Reply)
Discussion started by: Phuti
1 Replies

2. UNIX for Advanced & Expert Users

Performance of a server

Hi, how can we define performance of a server (Windows or Unix or Linux) ? If processes waiting for CPU (on queue) are usually more than 3 or 5 can we conclude that CPU is not enough for that usage ? Thank you. (1 Reply)
Discussion started by: big123456
1 Replies

3. Solaris

Server performance bottleneck

hi all, My server box is slow running. I have provide some statistics below: Where is the bottleneck on the server? I guess the bottleneck is disk I/O? bash-3.00# prstat -Z PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 29206 mobi1 334M 264M sleep ... (8 Replies)
Discussion started by: buyantugs
8 Replies

4. Infrastructure Monitoring

99% performance wa, slow server.

There is a big problem with the server (VPS based on OpenVZ, CentOS 5, 3GB RAM). The problem is the following. The first 15-20 minutes after starting the server is operating normally, the load average is less than or about 1.0, but then begins to increase sharply% wa, then hovers around 95-99%.... (2 Replies)
Discussion started by: draiphod
2 Replies

5. Solaris

Server performance

Hi, Please advice .. if server is slowness then what are the steps i have to follow... Thanks, Mani (5 Replies)
Discussion started by: Mani_apr08
5 Replies

6. Solaris

Server performance

Hi, i would like to ask if it is possible to get the server load on a solaris machine, but i don't want something like uptime (load average), iostat and vmstat. I would like to get something in percentage like CPU load in %, Disk usage in %, Ram usage in %. I want to collect this data and to... (3 Replies)
Discussion started by: tafil
3 Replies

7. UNIX for Dummies Questions & Answers

performance of unix server

Hi, I am supposed to run few jobs based on the usage of unix server. How to find out if the server is too busy. what are the commands we can use to find out that. Thanks (1 Reply)
Discussion started by: dnat
1 Replies

8. UNIX and Linux Applications

server/client performance

Ello group, I have general question about how the performance of server/client should be? My server is able to answer about 650times per second. Is it good performance? the apache on the same machine makes 1600/sec BUT there is nine instances of httpd daemon what makes 180/ sec /instance. of... (1 Reply)
Discussion started by: tomjan
1 Replies

9. AIX

monitor aix bottleneck

Hi All, I would like to create a script that will monitor the server if it's heavy on the processing. I have already some scripts for monitoring disk capacity, application monitoring, etc but not on the bottleneck of the server processing. I don't know which one to measure/query. Can you... (2 Replies)
Discussion started by: itik
2 Replies

10. HP-UX

server performance..

Hi, would like to know the server status from the following 'top' out put. Because, the application is giving a bad performance. would like to know whether the load is within the acceptable limit. Murali... System: shpu28 Tue Feb 3 10:03:31... (4 Replies)
Discussion started by: manduva
4 Replies
Login or Register to Ask a Question