Login or Register to Ask a Question and Join Our Community


Solaris

Performance (iops) becomes bad, what is the reason?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Solaris Performance (iops) becomes bad, what is the reason?
# 1  
Old 06-16-2011
Performance (iops) becomes bad, what is the reason?
I have written a virtual HBA driver named "xmp_vhba". A scsi disk is attached on it. as shown below:

Code:
xmp_vhba, instance #0
        disk, instance #11

But the performance became very bad when we read/write the scsi disk using the vdbench(a read/write io tool).
What is the reason? Thanks!!


the performance is shown as below:

Code:
-bash-3.00# ./vdbench -vt -f xxx.txt
Vdbench distribution: vdbench502
For documentation, see 'vdbench.pdf'.
15:23:40.130 input argument scanned: '-vt'
15:23:40.163 input argument scanned: '-fxxx.txt'
15:23:41.635 Starting slave: /export/home/vdbench502/vdbench SlaveJvm -m localhost -n localhost-10-110616-15.23.39.879 -l localhost-0 -p 5570   
15:23:43.481 All slaves are now connected
15:23:48.002 Starting RD=run1; I/O rate: Uncontrolled MAX; elapsed=900000; For loops: threads=30.0
Jun 16, 2011  interval        i/o   MB/sec   bytes   read     resp     resp     resp    cpu%  cpu%
                             rate  1024**2     i/o    pct     time      max   stddev sys+usr   sys
15:23:49.281         1   14827.00     7.24     512 100.00    1.700   22.333    0.607    20.3  15.4
15:23:50.094         2   15736.00     7.68     512 100.00    1.771   18.857    0.308    22.0  17.0
15:23:51.085         3   16376.00     8.00     512 100.00    1.790   17.089    0.238    18.9  15.5
15:23:52.083         4   16797.00     8.20     512 100.00    1.744   17.868    0.198    19.1  15.7
15:23:53.076         5   16635.00     8.12     512 100.00    1.764   18.878    0.260    19.2  15.6
15:23:54.076         6   16769.00     8.19     512 100.00    1.748   17.625    0.224    19.1  15.7
15:23:55.076         7   16752.00     8.18     512 100.00    1.750   18.424    0.266    19.0  15.6

---------- Post updated at 06:38 AM ---------- Previous update was at 03:05 AM ----------

Maybe the properties of the VHBA and the scsi disk can affect the performance. Or the vdbench cause this problem.
I don't know what are the possible reasons. Who can give me some suggestions to resolve this problem. Thanks!!

---------- Post updated at 06:40 AM ---------- Previous update was at 06:38 AM ----------

Maybe the properties of the VHBA and the scsi disk can affect the performance. Or the vdbench cause this problem.
I don't know what are the possible reasons. Who can give me some suggestions to resolve this problem. Thanks!!
# 2  
Old 06-16-2011
What's the output from

Code:
iostat -sndzx 1

while vdbench is running?

FWIW, it looks like it's doing a bunch of very small IO operations (512 bytes)
# 3  
Old 06-16-2011
the output from "iostat -sndzx 1" is shown as below:

Code:
 
-bash-3.00# iostat -sndzx 1
extended device statistics 
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.2 0.8 3.6 3.6 0.0 0.0 0.0 6.3 0 0 c1t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0 0 c1t2d0
11300.2 0.1 5650.1 0.0 0.2 19.1 0.0 1.7 16 69 c0t661B205100A12200000FB02D0000000Dd0
extended device statistics 
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 167.3 0.0 735.4 0.0 1.0 0.0 6.0 0 98 c1t0d0
16684.4 0.0 8342.2 0.0 0.2 27.8 0.0 1.7 22 100 c0t661B205100A12200000FB02D0000000Dd0
extended device statistics 
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 166.1 0.0 740.8 0.0 1.0 0.0 6.0 0 98 c1t0d0
16858.8 0.0 8429.4 0.0 0.2 27.8 0.0 1.6 22 100 c0t661B205100A12200000FB02D0000000Dd0
extended device statistics 
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 192.0 0.0 847.6 0.0 1.0 0.0 5.1 0 97 c1t0d0
16234.7 0.0 8117.4 0.0 0.2 27.8 0.0 1.7 22 100 c0t661B205100A12200000FB02D0000000Dd0

And you are right, it is doing a bunch of small IO with 512 bytes. The parameter setting of vdbench is shown as following:

Code:
 
sd=sd0,lun=/dev/rdsk/c0t661B205100A12200000FB02D0000000Dd0s2
wd=wd1,sd=(sd*),xfersize=512,rdpct=100,seekpct=0
rd=run1,wd=(wd1),iorate=max,elapsed=900000,interval=1,forthreads=30




Quote:
Originally Posted by achenle
What's the output from

Code:
iostat -sndzx 1

while vdbench is running?

FWIW, it looks like it's doing a bunch of very small IO operations (512 bytes)
Last edited by ForgetChen; 06-16-2011 at 10:53 PM..
# 4  
Old 06-17-2011
Can you run vdbench with larger IO sizes? Like 1 MB or larger (but keep in in powers of 2). What happens then?
# 5  
Old 06-22-2011
Quote:
Originally Posted by achenle
Can you run vdbench with larger IO sizes? Like 1 MB or larger (but keep in in powers of 2). What happens then?
Thank you for your replies!

Code:
 
the result with running vdbench with 1MB IO :

Jun 22, 2011 interval i/o MB/sec bytes read resp resp resp cpu% cpu%
rate 1024**2 i/o pct time max stddev sys+usr sys
16:06:21.052 31 782.00 782.00 1048576 100.00 149.715 161.195 0.493 4.5 4.3
16:06:22.051 32 781.00 781.00 1048576 100.00 149.697 161.233 0.475 4.5 4.2
16:06:23.051 33 781.00 781.00 1048576 100.00 148.282 154.836 2.734 4.7 4.3

The io rate is always very bad ,especial with 512 IO size.
Now I doubt the DMA property and buf struct in scsi_init_pkt function. But I didn't understand these fully.
# 6  
Old 06-22-2011
Quote:
Originally Posted by ForgetChen
Thank you for your replies!

Code:
 
the result with running vdbench with 1MB IO :

Jun 22, 2011 interval i/o MB/sec bytes read resp resp resp cpu% cpu%
rate 1024**2 i/o pct time max stddev sys+usr sys
16:06:21.052 31 782.00 782.00 1048576 100.00 149.715 161.195 0.493 4.5 4.3
16:06:22.051 32 781.00 781.00 1048576 100.00 149.697 161.233 0.475 4.5 4.2
16:06:23.051 33 781.00 781.00 1048576 100.00 148.282 154.836 2.734 4.7 4.3

The io rate is always very bad ,especial with 512 IO size.
Now I doubt the DMA property and buf struct in scsi_init_pkt function. But I didn't understand these fully.
If I'm reading that right, you're getting 781 MB/sec with 1 MB IO operations.

How fast do you think it should be going?
# 7  
Old 06-22-2011
Quote:
Originally Posted by achenle
If I'm reading that right, you're getting 781 MB/sec with 1 MB IO operations.

How fast do you think it should be going?
You are right. The performance is normal with 1M IO operations.

While, the problem is that the performance is bad with 512 bytes compared with the one without our virtual HBA dirver.

The performace of same disk without VHBA is shown as below:
Code:
 
Jun 23, 2011 interval i/o MB/sec bytes read resp resp resp cpu% cpu%
rate 1024**2 i/o pct time max stddev sys+usr sys
11:12:20.729 1 73623.00 35.95 512 100.00 0.391 30.889 0.353 31.1 21.0
11:12:21.102 2 34416.00 16.80 512 100.00 0.775 14.747 0.140 39.5 25.3
11:12:22.093 3 66576.00 32.51 512 100.00 1.099 15.570 0.222 36.3 24.9
11:12:23.089 4 66305.00 32.38 512 100.00 1.687 16.420 0.283 36.5 25.1
11:12:24.084 5 66237.00 32.34 512 100.00 1.899 16.832 0.208 35.9 24.5
11:12:25.082 6 66379.00 32.41 512 100.00 1.895 17.151 0.274 36.2 25.0

The max io rate can reach 7k while the io rate with our VHBA is only 1.7k when running vdbench with 512 bytes IO.
That is the problem. But I don't know why.
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. AIX

AIX lpar bad disk I/O performance - 4k per IO limitation ?

Hi Guys, I have fresh new installed VIO 2.2.3.70 on a p710, 3 physical SAS disks, rootvg on hdisk0 and 3 VIO clients through vscsi, AIX7.1tl4 AIX6.1tl9 RHEL6.5ppc, each lpar has its rootvg installed on a LV on datavg (hdisk2) mapped to vhost0,1,2 There is no vg on hdisk1, I use it for my... (1 Reply)
Discussion started by: frenchy59
1 Replies

2. Solaris

How to measure IOPS?

Hi I have a system running solaris 10, and I intend to use a NetApp as its storage system. The application requires a throughput between the server and the storage 7000 disk IOPS (random IO sustained throughput with response time of 20 mili second and 16k block size). How to make sure that I... (6 Replies)
Discussion started by: fretagi
6 Replies

3. Shell Programming and Scripting

Why I get bad bad substitution when using eval?

Why I get bad replace when using eval? $ map0=( "0" "0000" "0") $ i=0 $ eval echo \${map$i} 0000 $ a=`eval echo \${map$i}` !!!error happens!!! bash: ${map$i}: bad substitution How to resolve it ? Thanks! (5 Replies)
Discussion started by: 915086731
5 Replies

4. HP-UX

Bad performance but Low CPU loading?

There might be some problem with my server, because every morning at 7, it's performance become bad with no DB extra deadlock. But I just couldn't figure it out. Please give me some advise, thanks a lot... According to the CPU performace chart, Daily CPU loading Maximum: 42 %, Average:36%. ... (8 Replies)
Discussion started by: GreenShery
8 Replies

5. UNIX for Dummies Questions & Answers

nohup has a terrible bad performance compared with interactive command, why?

I have a strange situation. I'm running a shell script containing several data uploads (using Oracle sqlloader utility). This script is being run on a Red Hat server. I tried to run it in background: $ nohup upload.sh & This script uploads some thousands files. After several hours I... (0 Replies)
Discussion started by: viniciov
0 Replies

6. News, Links, Events and Announcements

Announcing collectl - new performance linux performance monitor

About 4 years ago I wrote this tool inspired by Rob Urban's collect tool for DEC's Tru64 Unix. What makes this tool as different as collect was in its day is its ability to run at a low overhead and collect tons of stuff. I've expanded the general concept and even include data not available in... (0 Replies)
Discussion started by: MarkSeger
0 Replies

7. AIX

Bad performance when log in with putty

Hello guys! I'm n00b in AIX and I'm sticked in a problem. (my English is poor enough, but I hope you can understand me :P). So.. I'm trying to connect to an AIX machine with putty, and .. 'using username xxx' appears after 2 sec (OK), but 'xxx@ip's password' appears after 1:15 min. After... (6 Replies)
Discussion started by: combat2k
6 Replies

8. UNIX for Dummies Questions & Answers

comparing Huge Files - Performance is very bad

Hi All, Can you please help me in resolving the following problem? My requirement is like this: 1) I have two files YESTERDAY_FILE and TODAY_FILE. Each one is having nearly two million data. 2) I need to check each record of TODAY_FILE in YESTERDAY_FILE. If exists we can skip that by... (5 Replies)
Discussion started by: madhukalyan
5 Replies
Login or Register to Ask a Question