Linux machine performance issue.


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Linux machine performance issue.
# 1  
Old 08-06-2010
Linux machine performance issue.

One of our database server is suddenly became very slow and i have no clue what to do .Please help. I m sharing the performance inforamtion regarding cpu,harddisk,ram .

Code:
########CPU Information########
Machine Uptime Information:
 uptime
 10:25:06 up 16:50,  1 user,  load average: 5.84, 5.65, 5.54
Command Description:
<current time> <Uptime since> <hour:min>
Number of users logged in to machine
load average <for last one minute>,<for last 5 minutes>,<for last 15 minutes>
________________________________________________________________
Terminal Information:
 w
 10:25:07 up 16:50,  1 user,  load average: 5.84, 5.65, 5.54
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/1    node1           10:18    5.00s  3.61s  0.64s -bash
________________________________________________________________
 cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
stepping        : 10
cpu MHz         : 2660.056
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat clflush dts mmx fxsr sse sse2 ss syscall lm constant_tsc pni ds_cpl cx16 xtpr lahf_lm
bogomips        : 6660.87
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz
stepping        : 10
cpu MHz         : 2660.056
cache size      : 6144 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat clflush dts mmx fxsr sse sse2 ss syscall lm constant_tsc pni ds_cpl cx16 xtpr lahf_lm
bogomips        : 6660.87
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

________________________________________________________________
Top 10 C.P.U Consumer Processes:
 ps -auxf | sort -nr -k 3 | head -10
root     13857 59.5  0.0  65532   972 pts/1    R+   10:25   0:01              \_ ps -auxf
root     13858 27.5  0.0 109112   664 pts/1    S+   10:25   0:00              \_ sort -nr -k 3
oracle   13842 21.7  0.1 1732792 13488 ?       Rs   10:25   0:02 oracleoracle (LOCAL=NO)
oracle   13860 20.0  0.1  74384 13508 ?        R    10:25   0:00      \_ /u01/app/oracle/111/bin/tnslsnr LISTENER -inherit
oracle   13845 18.4  0.1 1732280 10772 ?       Rs   10:25   0:01 oracleoracle (LOCAL=NO)
oracle   13855 14.6  0.0      0     0 ?        R    10:25   0:00  \_ [tnslsnr]
oracle   13840 13.4  0.1 1732280 12084 ?       Rs   10:25   0:01 oracleoracle (LOCAL=NO)
oracle   10080 10.9  3.4 1740824 279276 ?      Ss   09:29   6:10 oracleoracle (LOCAL=NO)
root     13859 10.5  0.0      0     0 pts/1    R+   10:25   0:00              \_ [bash]
root     13847  8.4  0.0   8680   944 ?        Ss   10:25   0:00      \_ /bin/sh -c /usr/sbin/ntpdate -b 10.180.23.51 >/dev/null 2>&1
________________________________________________________________
########Memory Information########
 cat /proc/meminfo
MemTotal:      8126860 kB
MemFree:       4766148 kB
Buffers:        375952 kB
Cached:        2203984 kB
SwapCached:          0 kB
Active:        1376116 kB
Inactive:      1319760 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:      8126860 kB
LowFree:       4766148 kB
SwapTotal:     2555888 kB
SwapFree:      2555888 kB
Dirty:             480 kB
Writeback:           0 kB
AnonPages:      116040 kB
Mapped:         453748 kB
Slab:           387024 kB
PageTables:      23384 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   6619316 kB
Committed_AS:  2439244 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      1516 kB
VmallocChunk: 34359736843 kB
________________________________________________________________
 free -m
             total       used       free     shared    buffers     cached
Mem:          7936       3280       4656          0        367       2152
-/+ buffers/cache:        760       7175
Swap:         2495          0       2495
________________________________________________________________
Top 10 Memory Consumer Processes:
 ps -auxf | sort -nr -k 4 | head -10
oracle   10080 10.9  3.4 1740824 279304 ?      Ss   09:29   6:10 oracleoracle (LOCAL=NO)
oracle   10067  5.8  1.8 1742880 149684 ?      Ss   09:28   3:17 oracleoracle (LOCAL=NO)
oracle    9853  0.2  1.5 1735532 123392 ?      Ss   09:25   0:09 ora_smon_oracle
oracle    9857  0.5  0.9 1738836 79624 ?       Ss   09:26   0:18 ora_mmon_oracle
oracle    9994  4.8  0.8 1741860 66148 ?       Ss   09:27   2:46 oracleoracle (LOCAL=NO)
oracle    9847  0.5  0.5 1737024 42632 ?       Ss   09:25   0:19 ora_dbw0_oracle
oracle    9845  0.1  0.5 1732800 44440 ?       Ss   09:25   0:05 ora_mman_oracle
oracle   10214  0.7  0.5 1734396 41364 ?       Ss   09:31   0:25 ora_cjq0_oracle
oracle   10063  0.1  0.5 1740824 44272 ?       Ss   09:28   0:03 oracleoracle (LOCAL=NO)
oracle   10050  0.1  0.5 1740836 42716 ?       Ss   09:28   0:05 oracleoracle (LOCAL=NO)
________________________________________________________________
########Disk Information########
Disk Utilization
 df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       13G  9.5G  2.4G  80% /
/dev/hda1              99M   20M   75M  21% /boot
tmpfs                 3.9G  1.6M  3.9G   1% /dev/shm
/dev/hdb1             5.0G  414M  4.3G   9% /mnt
/dev/hdd               30G  5.5G   23G  20% /data
10.180.23.194:/dump    29G  1.9G   26G   7% /emdump
________________________________________________________________
Disk Partition Information
 cat /proc/partitions
major minor  #blocks  name

   3     0   15728640 hda
   3     1     104391 hda1
   3     2   15623212 hda2
   3    64    5242880 hdb
   3    65    5237158 hdb1
  22    64   31457280 hdd
 253     0   13565952 dm-0
 253     1    2031616 dm-1
________________________________________________________________

# 2  
Old 08-06-2010
Could you post the output of a
Code:
vmstat 1 10

when it is currently having this issue?
# 3  
Old 08-06-2010
Quote:
Originally Posted by zaxxon
Could you post the output of a
Code:
vmstat 1 10

when it is currently having this issue?
Code:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 5831512  96952 1773232    0    0    35    27   84   43 31 47 21  1  0
 8  1      0 5830232  96956 1773228    0    0     0   200  252  125 35 49 14  1  0
 6  1      0 5829472  96996 1773236    0    0     0   212  366  182 30 68  1  0  0
 5  0      0 5828984  97056 1773176    0    0     0   168  282  164 39 61  0  0  0
 6  0      0 5827488  97056 1773260    0    0     0     0  264   90 35 65  0  0  0
 5  0      0 5826140  97056 1773260    0    0     0    48  263  108 35 65  0  0  0
 4  0      0 5820800  97064 1773252    0    0     0    12  353  139 41 57  2  0  0
 7  0      0 5818164  97064 1773252    0    0     0     0  376  178 41 57  2  0  0
 5  0      0 5820180  97068 1773260    0    0     0   180  325  147 35 63  1  0  0
 1  0      0 5827320  97068 1773260    0    0     0    12  322  146 28 33 39  0  0



---------- Post updated at 04:16 AM ---------- Previous update was at 04:15 AM ----------

I dont have a much information on when this problem occurred but whenever i try to execute any command or take a putty session of the machine it takes too much time to execute it.
# 4  
Old 08-06-2010
This systems is CPU bound:
Code:
 6  1      0 5829472  96996 1773236    0    0     0   212  366  182 30 68  1  0  0
 5  0      0 5828984  97056 1773176    0    0     0   168  282  164 39 61  0  0  0
 6  0      0 5827488  97056 1773260    0    0     0     0  264   90 35 65  0  0  0
 5  0      0 5826140  97056 1773260    0    0     0    48  263  108 35 65  0  0  0
 4  0      0 5820800  97064 1773252    0    0     0    12  353  139 41 57  2  0  0
 7  0      0 5818164  97064 1773252    0    0     0     0  376  178 41 57  2  0  0
 5  0      0 5820180  97068 1773260    0    0     0   180  325  147 35 63  1  0  0

I made the run-queue for kernel threads bold. According to that you can see that there is all CPU-time used up for user and system while at same time idle is zero.
The top ten of your CPU-using processes might not help since there could be loads of small processes, that use only a bit of CPU-time but all together a big amount.
You might want to check top for a longer listing when this ps aux is not running and also you might check with a ps piped into awk for example to count how many processes are using how much CPU, as a summary.

You can try something like
Code:
ps -eo pcpu,comm| awk '{_[$2]+=$1} END{for(a in _)print _[a],a| "sort"}'

It is not very exact but it adds the CPU-time in % of the same commands up.

It could be that the box has so much trouble because there is something uneffectively written/configured in the software or if everything is allright, it might need more CPUs. Just a guess so far - you have to monitor and research more.
# 5  
Old 08-06-2010
What is your uptime shows ?
# 6  
Old 08-17-2010
Code:
uptime
 10:25:06 up 16:50,  1 user,  load average: 5.84, 5.65, 5.54

# 7  
Old 08-17-2010
How many CPUs you have ?

If it is only 1, then you have to do something for process performance, like increasing the processor, or restricting unwanted process in that machine and make it exclusive for your purposes.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Linux shell programming performance issue

Hi All, can any one help me on this please. Replace sting in FILE1.txt with FILE2.txt. FILE1.txt record must have at least one state is repeated once.But need to replace only from second occurrence in record in FILE1.txt Condition: order of searching the records in FILE2.txt is impartent.... (8 Replies)
Discussion started by: ureddy
8 Replies

2. Red Hat

Performance issue in Linux

IN solaris, for network high-availability we are using IPMP concept, can u tell me in REDHAT LINUX what we are using... also pls share good step to read & understand the that concept... Also performance issue in linux what are step & cmd can u tell me??? (2 Replies)
Discussion started by: tiger09
2 Replies

3. Shell Programming and Scripting

shell script to copy files frm a linux machine to a windows machine using SCP

I need a shell script to copy files frm a linux machine to a windows machine using SCP. The files keeps changing day-to-day. I have to copy the latest file to the windows machine frm the linux machine. for example :In Linux, On July 20, the file name will be 20.txt and it should be copied to... (3 Replies)
Discussion started by: nithin6034
3 Replies

4. UNIX for Dummies Questions & Answers

Performance issue

hi I am having a performance issue with the following requirement i have to create a permutation and combination on a set of three files such that each record in each file is picked and the output is redirected in a specific format but it is taking around 70 odd hours to prepare a combination... (7 Replies)
Discussion started by: mad_man12
7 Replies

5. UNIX for Advanced & Expert Users

run win app on Linux -performance issue

We develop software for diagnostic tools for cars. we a use a portable PC(x86) runs Win98 to run our applications. Hence the working environment in the company is Windows, specifically we use BASIC to develop the GUI, communication functions, DLL, etc. and run them on the Win98 PC. We suggested... (1 Reply)
Discussion started by: raedbenz
1 Replies

6. Shell Programming and Scripting

SSH into a linux machine from a windows machine

I basically want to login into different linux machines( on the same network) from a windows machine. I know i can use ssh <machine name>. But i want to automate this process. I dont want to enter the username and password. Is there any way to do it. Can i make some sort of a batch script for it. (4 Replies)
Discussion started by: lassimanji
4 Replies

7. News, Links, Events and Announcements

Announcing collectl - new performance linux performance monitor

About 4 years ago I wrote this tool inspired by Rob Urban's collect tool for DEC's Tru64 Unix. What makes this tool as different as collect was in its day is its ability to run at a low overhead and collect tons of stuff. I've expanded the general concept and even include data not available in... (0 Replies)
Discussion started by: MarkSeger
0 Replies

8. UNIX for Advanced & Expert Users

performance issue

Hi, on a linux server I have the following : vmstat 2 10 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 4 0 675236 39836 206060 1617660 3 3 3 6 8 7 1 1 ... (1 Reply)
Discussion started by: big123456
1 Replies

9. AIX

performance issue

We have a AIX v5.3 on a p5 system with a poor performing Ingres database. We added one CPU to the system to see if this would help. Now there are two CPU's. with sar and topas -P I see good results: CPU usage around 30% with topas I only see good results in the process output screen, the... (1 Reply)
Discussion started by: rein
1 Replies

10. Programming

the performance impaction of TCP communication in same machine?

Two processes at the same machine communicaton with each other use TCP. The communicaton flow between them is very heavy. My question is How the impaction of "processes at the same machine communicaton " .If I mv one process to another machine and send the same data, the impact is less than... (1 Reply)
Discussion started by: chenhao_no1
1 Replies
Login or Register to Ask a Question