ok, i was finally able to enable the disk statistics using sar. ( i have do add -d on /usr/lib64/sa/sa1 script.
no now i've created new PDF with ksar with all the statisics i can.
the problem is that i dont understand and know what to look for in order to understand the problem.
as you can see i'm not on a cpu problem, i have a vary high load average and high numberof runq-sz.
so, i've write down all the statistics from a specific minutes i'ev chosen. its not the specific time our machine hanged but perhaps we can see the problem in a reguler time.
tony, are you see any problem?
first i though i just need to add memory. cause its looks like he uses all the memory, byt it doesnt make any sense cause there is no paging, and i guess linux allocate all the memory but not realy using it.
perhaps its the disk problem?
this is the output for iostat -n -x ALL:
sda and sdb are included LVM partition.
i have a RAID5 in my machine with 5 disks.
perhaps my disks are slow? they are sata 500G 7200rpms.
how it can be he used all the memory and no swap? and he always uses it, not only when he machine is overloaded.
thanks,
i have no idea how to figure it out
Last edited by Scott; 02-12-2010 at 05:09 AM..
Reason: Code tags, please...
The interesting lines are:
The busiest device is dm-0 with 263.15 writes per second but the device with the longest average read queue size is sdb, sda has a higher utilisation and a quite high run queue size.
Run:
to find out what /dev/sdb and /dev/sda is mounted as.
The NFS mounts are evidently very quiet, but running iostat after your machine has done a lot of real work may show something different.
RAID 5 is good for securing your data but for best performance you need striping, the best compromise is a stripe mirrored against a another stripe.
Last edited by TonyFullerMalv; 02-13-2010 at 10:26 AM..
first of all i would likw to thank you a lot for all your help! i'm very appriciate it, so thanks a lot!
sda is my root filesystem. sdb is a disk which has LVM partition so i cant see it on mount command.
i have a rootvg with 3 filesystem which resides on both sdb and additional partition of sda.
i guess md-0 is the LVM as a total. (not sure).
anyway, so we can see the problem is in our disks, this why i have a high i/o wait and load.
in order to change the RAID, or even add more drives to the RAID 5 i need to break the raid, and currenlty its the wrost option, cause its a production machine.
what else can i do in oredr to increase performance?
addidng more cpu is not an option, cause i will still get the i.o wait. the processor will process the requerst much faster and still wait(and much more) to the disks.
i have a 8G RAM. perhaps i need to add more RAM for caching? i'm not sure how do i do that.
i'm not exactly sure i can say loud and clear, the disks are the problem
To be fair RAID 5 is going to be better than a single disc or jut a mirrored pair of discs.
Performance tuning is not an exact art, the figures can look terrible but if the machine is performing the task you want it to do okay then they are not a problem, if the machine is performing the required task then you start looking at the figures to see where the bottle neck is, once you deal with it another bottle neck may also need dealing with.
Suggestions:
Copy the contents of your current disks onto another set of discs in a stripe then reconstruct your RAID 5 set of discs into a stripe and mirror them against the first stripe?
Copy your current RAID 5 configuration onto 10K RPM discs?
Ensure you are using the fastest interface possible for your platform, i.e. SCSI or SATA tend to be faster than IDE?
Last edited by TonyFullerMalv; 02-13-2010 at 05:30 PM..
To be fair RAID 5 is going to be better than a single disc or jut a mirrored pair of discs.
Possibly.
In this case, unless I am mistaken there is a two disk RAID 5 which would me that:
1. It is always running as degraded.
2. Every write has a at least a double overhead.
Also there is a massive IO imbalance between the two disks, possibly due to different workloads hitting the same disk and causing a lot of head shuttle.
No matter how you look at this it appears that you don't have enough disks for what you are doing.
SATA drives are pretty good for sequential I/O so if the workloads are cleanly separated there is a chance that they could be fine. However with a random workload faster disks are definitely preferable.
Hi all,
-------------------------
Linux OS Version/Release:
-------------------------
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
Linux <hostname> 2.6.18-194.8.1.el5 #1 SMP Wed Jun 23 10:52:51 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
I have a server that hosts 30+ Oracle... (1 Reply)
What tools can I use to look "deeper" into a process to see if the job is actually running or just hanging. What is the best method to accomplish this?
SunOS 5.10 Generic_142900-14 sun4v sparc SUNW,T5240 (2 Replies)
As I am new to the Unix field, I would like to get the clarification regarding the Filesystem.
The scenario is.. The filesystem (/drbackup) is getting monitored and if it exceeds the threshold, we will receive an alert from it. The issue is that we receive an alert with the description of... (2 Replies)
Hi all,
I am planning to give a presentation on performance measure. I have decided to focus on the commands which are used to know the performance of the server. I have a idea of prstat,vmstat,netstat, and iostat. Could anybody suggest me any other commands which are used for perforamance... (7 Replies)
This is my first post (yes I'm a newbie).... :D
I'm looking for a list of Linux and Unix commands for performance monitoring and a good sight or area on this site that would have man pages and or information on those commands.....
Thanks if anyone can take the time to post..... :cool: (14 Replies)
Hello,
I am trying to find a way to view current CPU and disk usage. I used to use nmon which worked fine but since an upgrade to our servers this is no longer available. I have tried to get it reinstalled to no avail!
Are there any other commands you can use within unix which will allow me... (4 Replies)
Hi all
The place I work for is about to to place there database server under heavy load for testing and would like the effect recorded as much as possible.
Can anyone point me in the right direction with respect to real time system monitoring. I am aware of of 'sar', vmstat etc and hope to... (2 Replies)
Hi All
I am looking for a script that would collect statistics in a summarised format.
CPU, Memory,Swap, Wait queue, Run queue and disk activity.
Something that would allow me to profile the environment based on a 1 line output that I could run every 15 min.
Thx
Junaid (1 Reply)
hi,
can any one tell me, is there is any way i can check the performance of my solaris 8 os on an Ent 3500. Other than top to check for the top most processes, how to make the calculations with vmstat, iostat, mpstat and nfsstat. Or is there any other tools that i can use?
cheers. (3 Replies)