Linux

hello,

i'm having some performance problem on one of my linux machines and i hope someone will be able to help me analyzing the problem.

machine info:
Linux fedora, cpu x 4 cores of 1.6Ghz, 8G memory, 8G swap.

i've enabled sar on my machine and created a graph using ksar utility for the last week.

the sar -A command actually monitored everything on my machine.
now, i'm trying to understand what was the problem

my problem was that on 2 days at that week my application hanges and we had to restart the machine in order to solve this.
now, i'm trying to understand its its i/o problem due which related to my disk, or perhaps memory problem.

the problem occured at 15:00PM on one of the days i've monitored. i will write down the resluts:

Code:

CPU monitor: (0.0 - 30.0) 
   cpu used: 17%.
context: (0-31,000)
   cswch/s: 17,000
i/o: 
  transerf/s (0- 5000) : 3000
  block read/write (0-300,000) : 100,000
  read/write/s (0 - 4000) : 1500
memory:
  memused: 7.8G (97%)
memory misc:
   buffers: (0-700m) : 200m
   cached: (0-7.5G): 7G
swap:
   swapfree: 8G
   swapused: 40kb
 
load:
  runq-sz (0-3): 2.5
  plist-sz(0-400): 350
  load average (0-40) : 30
 
page:
  frmpg/s((-)1000 - 1000) : (-)1000
  bugpg/s: ((-)150 - 150): 50
  campg/s ((-)1000 - 1000): 1000
 
paging: 
   pgpgin/pgpgout  /s (0-80,000) : 30,000
   fault/majflt / s (0-55,000) : 1000
 
processes:
  proc/s (0-60.00): 1

i hope it was clear
i can see there is not enough memory cause he used almost all the memory, and the runq-sz is high so a lot of process running/waiting to run i think, so perhaps it might indicate the problem?

please help...

10x

Looking at what you are showing us the CPU is not being overloaded and you are not drastically short of memory, as you say the run queue size is not good, I think you need to look at the figures for CPU I/O wait and I/O service time (svctm) and see what devices are not responding very quickly (disk storage perhaps?)?
Do you consider your application to be doing a lot of I/O?

hi,

what did you mean about "my application hanges"??

do you mean "application crashes" then it's not a performance issue,
maybe it's software issues, OS bug or Application Bug,
or Hardware Failure, Memory or CPU Failure,

hi,
thanks a lot for you answer tony.
sorry for the late response i didnt see you answered me.
anyway, how can i check the cpu i/o. i have all this statistics of ksar output.
i though this is all the statistics i can have.
how can i check if the disk are the problem?

thanks

This may be different for your Unix but for Unbuntu sar(1) you can run:

Code:

sar -d

for today's I/O stats or:

Code:

sar -d -f /var/log/sysstat/sa16

for the I/O stats from the 16th.
The simple thing to look for is if the figures for one volume are worse than the others and may benefit from being moved to a dedicated disk or even onto a striped volume?


Also take a look at:

Code:

sar -n ALL

for network device stats.

The other one to look at is:

Code:

sar -q

and see if the run queue size (runq-sz) gets larger when the I/O stats are at there worse.

hi tony,
thanks a lot for your help!

i've tried to execute sar -d -f <sa file> and it gives me the error:
Requested activities not available in file

i though that this is due to my sysstat configuration. i've added the SA1_OPTIONS="-d" on my /etc/init.d/sysstat file so it will know to check device statistics, restart sysstat service and run the crontab entry "sa1 -d -I 1 1" but i'm still getting the same error on the sar command

perhaps i'm not doing something right.
i've used the ksar to generate a PDF with all the statistics and i am missing i/o wait for disks graphs. so i've added the SA1_OPTIONS="-d" to the file i've told you in order to enable this options. isnt that enough?

thanks

i will

If you've only just configured sar to collect device I/O stats then none of the previous days sar files will contain that information, presumably on the next day you will be able to run:

Code:

sar -d

and see something for that day at least?