Visit Our UNIX and Linux User Community


T4-1/Solaris 11 and sudden 50% performance drop


 
Thread Tools Search this Thread
Operating Systems Solaris T4-1/Solaris 11 and sudden 50% performance drop
# 1  
Old 09-28-2012
Question [solved] T4-1/Solaris 11 and sudden 50% performance drop

We have a SPARC T4-1 server, running Solaris 11, and it's doing some pretty extensive parsing on roughly 100GB data set.

All was well still few weeks ago, when I was testing the performance, I was reaching rougly 50minute calculation times, and it was more or less expected performance.
Now that I started moving the server to it's final location, when testing the tools, I noticed that performance has dropped dramatically. For some reason I can't parse the data set anymore under 2 hours.. I don't understand why.

I haven't done any modifications to the system, except some network configuration - and nobody else has touched the system to my knowledge.Smilie

Where should I start looking, why the parsing performance has dropped more than 50% ? Obviously it's quite HDD intensive, since our dataset is so large, but our zpool seems to be healthy, and without any errors.
I checked /var/adm/messages, and there were no errors either.

prstat doesn't show anything odd, only our parsing process (sed) is taking most of the time.
Code:
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP       
  2096 username 9024K 1880K cpu55    0    0   2:26:36 1.6% sed/1
  2098 username 8960K 1808K cpu28    0    0   2:26:30 1.6% sed/1
  2097 username 9024K 1848K sleep   22    0   0:41:52 0.4% grep/1
  2095 username 9024K 1872K sleep   59    0   0:02:13 0.0% grep/1
  2099 username  132M  122M sleep   59    0   0:01:19 0.0% sort/1
   327 root        0K    0K sleep   99  -20   0:01:30 0.0% zpool-workarea/262
   621 root      702M  226M sleep   59    0   0:02:07 0.0% eptelemon/2
   539 root       17M   13M sleep   59    0   0:01:07 0.0% ldmd/16
    47 netcfg   4944K 2824K sleep   59    0   0:00:00 0.0% netcfgd/4
    81 daemon   7936K 5936K sleep   59    0   0:00:00 0.0% kcfd/3
   102 root     3136K 1128K sleep   59    0   0:00:00 0.0% in.mpathd/1
    44 root     5184K 4064K sleep   59    0   0:00:00 0.0% dlmgmtd/8
    74 netadm   5344K 3032K sleep   59    0   0:00:00 0.0% ipmgmtd/5
   116 root     2392K 1776K sleep   59    0   0:00:00 0.0% pfexecd/3
   793 root     3360K 1944K sleep   59    0   0:00:00 0.0% in.routed/1
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU                             
    11 username  244M  164M   0.5%   5:38:31 3.6%
    62 root     1047M  444M   1.3%   0:06:10 0.0%
     1 netcfg   4944K 2824K   0.0%   0:00:00 0.0%
     2 daemon     11M 6912K   0.0%   0:00:00 0.0%
     1 netadm   5344K 3032K   0.0%   0:00:00 0.0%
Total: 82 processes, 914 lwps, load averages: 2.63, 2.64, 2.64

I tried looking at iostat and vmstat, but I don't really understand them, any pointers on what I should look at ?
The output looks like this (during parsing), iostat:
Code:
   tty        sd1           sd2           sd3           sd4            cpu
 tin tout kps tps serv  kps tps serv  kps tps serv  kps tps serv   us sy wt id
   1   77 1664  14    4  1669  14    4  123  17   14  536   6    7    2  1  0 98
   0    0 3355  26    3  3303  26    3    0   0    0  1119  23    3    4  1  0 96
   0    0 3343  34    3  3414  31    3    0   0    0    0   0    0    4  1  0 96
   0    0 3328  26    2  3226  25    4    0   0    0    0   0    0    4  1  0 96
   0    0 2790  22    2  2893  23    3    0   0    0    0   0    0    4  1  0 96

vmstat:
Code:
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s1 s2 s3 s4   in   sy   cs us sy id
 0 0 0 14679712 16787992 7 16 0 0  0  0  8 14 14 17  6 2114 3539 2189  2  1 98
 0 0 0 1991952 4236944 2  6  0  0  0  0  0 27 26  0  0 1988 4960 2104  4  1 96
 0 0 0 1957032 4202024 0  1  0  0  0  0  0 26 26  0 20 2029 4929 2157  4  1 96
 0 0 0 1923640 4168632 0  1  0  0  0  0  0 32 35  0  0 2029 4940 2163  4  1 96
 0 0 0 1893096 4138088 0  0  0  0  0  0  0 31 32  0  0 2001 4889 2115  4  1 96

I know that I should change our CPU threading mode to max-ipc, as it's currently throughput, but this hasn't changed since last time I run tests.

Thanks for any tips!

---------- Post updated 09-28-12 at 05:23 PM ---------- Previous update was 09-27-12 at 07:12 PM ----------

Ok, looks like I found the problem. The processing times are back to normal, after I changed the encoding back to C from UTF8.
For some reason it had become UTF-8, not sure who/what changed that - but narutally that made sed and grep work extra hard Smilie
This User Gave Thanks to julumme For This Post:

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Solaris

Solaris Performance

Dear All, I have Sun OS 5.10 Solaris OS running. SunOS **** 5.10 Generic_118833-36 sun4v sparc SUNW,Sun-Fire-T200 ++There is a process named nscd utilizing high cpu not 100 though. Due to this the printer are delayed in printing and retreving information from this server last pid: 21848; ... (4 Replies)
Discussion started by: Sunil Koya
4 Replies

2. Solaris

Solaris Performance tuning

Dear all, I have a Local zone , where users feel that performance is not good. Is it wise to collect the inputs from the local zone rather than taking from the global zone. And also Can I tune from Global zone , so that it will reflect in local zone. Rgds rj (2 Replies)
Discussion started by: jegaraman
2 Replies

3. Solaris

solaris performance

Dear all, I have a solaris 10 server , which is running web applications and webspere also is available. All the parameters like Virtual memory , swap , hard disk space and Cpu utilisation and I/o statistics are good , but still the server is slow. It has 2 processores and 128 VPs. ... (3 Replies)
Discussion started by: jegaraman
3 Replies

4. Infrastructure Monitoring

Solaris Performance Monitoring Tools????

Hi, Are there any GUI (preferably web based) Solaris monitoring tools available for the SPARC platform. Just to clarify, when i say GUI, I don't mean buttons to configure the Software, of course that would be a plus, but rather GUI in terms of output, like Graphs. Thanks (6 Replies)
Discussion started by: Mack1982
6 Replies

5. Shell Programming and Scripting

Script to drop oracle DB users on HP, LINUX, SOLARIS

Hi everybody, i need help from all of you. I have many users who are no more work in our Company. Some who get objects are Locked. There Many users and many Database Servers. And it'll take long for me to finished this job. That's why i need your help to provide me anyone Scripts... (2 Replies)
Discussion started by: Sonson
2 Replies

6. Solaris

How I can get System Performance on Solaris

Hi All, Can someone help me out knowing all commands for getting system performance on Solaris machines. Thanks in advance, Yagami Light. (5 Replies)
Discussion started by: Yagami
5 Replies

7. Solaris

Help! Performance Threshold for Solaris 8,9,10

hi, I am trying to determine a 'rule of thumb' threshold for memory usage on different Solaris versions. I know that prior to Solaris 8, "page scan rate > 300" can be used as a general rule of thumb to determine any memory shortages. Since Solaris 9 and 10 have a different memory handling... (2 Replies)
Discussion started by: bwclu
2 Replies

8. UNIX for Dummies Questions & Answers

sudden poweroff ??

Hi I am using unix for last few days. Here is my problem during boot the machine stop giving video signal and I don't know what's happening. When I ping (during boot) it from another machine it comes alive then goes out. The power on the CPU is on all the time. please help. (9 Replies)
Discussion started by: santosh1981
9 Replies

9. UNIX Desktop Questions & Answers

oracle performance on solaris 8

Good evening all, I have a very interesting issue. I have an oracle database on unix and nt. The dba says that his delete process runs in 22 mins on nt. When I run the same delete process on unix, it run in an hour. I downloaded sarcheck to see what was going on. It told me abpout the autoup... (3 Replies)
Discussion started by: niasdad
3 Replies

Featured Tech Videos