Solaris

Thanks but no, the tcp_xmit_hiwat and tcp_recv_hiwat were both set higher than the default (and higher then the limit we were setting on tcp_host_param, with no change in the time it took to transfer the file.

For whatever reason this worked in Solaris 8, it seems to fail to change the transfer rate in Solaris 10. I was hoping to hear that these settings now had to be done with a different parameter. The application owner believes it's SUN's way of making them upgrade all their servers to Solaris 10.

I suggest to use iperf to test bandwidth, iperf will show the theoretical bandwith, The scp result is affected by disk I/O, CPU (encryption).
Please run vmstat to check CPU while copying files, maybe the bottleneck is cpu not network.
As a bonus, Iperf also show the window size detected, so It can verify if the change is effective.

you can get iperf from www.sunfreeware.com

an example.

Code:

# iperf   -n 900M -mc 172.16.1.12
------------------------------------------------------------
Client connecting to 172.16.1.12, TCP port 5001
TCP window size:   192 KByte (default)
------------------------------------------------------------
[  3] local 172.16.1.11 port 36932 connected with 172.16.1.12 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 7.9 sec    900 MBytes    955 Mbits/sec
[  3] MSS size 8948 bytes (MTU 8988 bytes, unknown interface)

Unfortunately, I can't put freeware on the server.

Output from iostat, vmstat, top during a scp transfer (all of these from the new server). You have to remember, if I try a scp transfer from another server in the same location (but not on the same subnet), the transfer rate shown in the scp is 4 to 5 times faster, so the server isn't having an issue when accepting a transfer. Also, showing netstat at the time of transfer shows the increased values are getting used.

netstat -an (cut down to just the scp process and my ssh to the server)
TCP: IPv4
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- ----- ------ ----- ------ -----------
10.140.16.20.22 172.16.10.10.44764 280912 135 280912 0 ESTABLISHED
10.140.16.20.49395 172.16.10.10.22 280912 0 525624 0 ESTABLISHED

As you can see, the scp (first process shown) is at the rate set up with tcp_host_params. My ssh to the old server has the higher rate on Rwind of 525624.


iostat 5 200
tty ramdisk1 sd0 sd1 sd2 cpu
tin tout kps tps serv kps tps serv kps tps serv kps tps serv us sy wt id
0 2 0 0 0 1 0 4 0 0 0 262 48 47 0 0 0 100
0 143 0 0 0 0 0 0 0 0 0 7 1 5 0 0 0 100
0 111 0 0 0 0 0 0 0 0 0 8 2 6 0 0 0 100
0 111 0 0 0 0 0 0 0 0 0 2 0 8 0 0 0 100
0 143 0 0 0 0 0 0 0 0 0 20 11 36 0 0 0 100
0 111 0 0 0 0 0 0 0 0 0 19 4 5 0 0 0 99
0 111 0 0 0 0 0 0 0 0 0 24 6 6 0 0 0 100
0 111 0 0 0 0 0 0 0 0 0 42 10 4 0 0 0 100
0 111 0 0 0 0 0 0 0 0 0 3 1 6 0 0 0 100
3 111 0 0 0 0 0 0 0 0 0 263 57 5 0 0 0 100

vmstat 5 200
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr rm s0 s1 s2 in sy cs us sy id
0 0 0 39429784 31169736 19 94 0 33 33 0 0 0 0 0 52 1665 3742 1331 0 0 100
0 0 0 39425768 31165392 30 128 0 13 13 0 0 0 0 0 8 1926 10266 1915 0 0 100
0 0 0 39425712 31165424 146 226 0 142 142 0 0 0 0 0 17 1633 5231 1350 0 0 100
0 0 0 39423680 31164952 40 262 0 241 241 0 0 0 0 0 37 1838 5427 1584 0 0 100
0 0 0 39421672 31163688 10 59 0 2 2 0 0 0 0 0 0 1541 2021 1164 0 0 100
0 0 0 39419568 31161488 59 289 0 17 17 0 0 0 0 0 12 1971 9501 2057 0 0 100
0 0 0 39417416 31159296 32 143 0 11 11 0 0 0 0 0 2 1498 3184 1147 0 0 100
0 0 0 39415360 31157216 9 59 0 0 0 0 0 0 0 0 0 1514 2051 1126 0 0 100
0 0 0 39413336 31155128 31 120 0 61 61 0 0 0 0 0 8 1658 10844 1217 0 0 99
snoop5.out | 110MB | 413kB/s | ETA: 00:31:16 | 12%


last pid: 6764; load avg: 0.26, 0.22, 0.19; up 0+23:33:41 18:28:48
80 processes: 79 sleeping, 1 on cpu
CPU states: 99.8% idle, 0.1% user, 0.1% kernel, 0.0% iowait, 0.0% swap
Kernel: 1583 ctxsw, 189 trap, 1727 intr, 4811 syscall, 59 flt, 8 pgout
Memory: 32G phys mem, 30G free mem, 12G total swap, 12G free swap

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
5009 AutoMon 1 29 10 32M 30M sleep 104:53 0.05% PatrolAgent
6541 v821383 1 54 0 10M 7976K sleep 0:03 0.04% sshd
2583 root 1 158 -20 63M 61M sleep 29:29 0.03% seosd
6365 root 1 54 0 5128K 4136K sleep 0:02 0.01% vmstat
6550 v821383 1 59 0 3752K 2144K sleep 0:00 0.01% sftp-server
406 root 9 59 0 10M 9504K sleep 0:58 0.00% picld
6188 root 1 59 0 4296K 3480K cpu/104 0:01 0.00% top

Opened a case with SUN - this is their final answer...

Too bad you focused so much on the Solaris release and didn't told in the first place you were using such different server models.

Yea - of course, I also just found out that SUN's version of ssh sucks.

I threw F-secure on the server, replacing SUN ssh, and it flies! Giving the kind of throughput the application folks expected. So, is F-secure so much better (using multi-thread )? I find that hard to believe.

I think SUN just doesn't bother to check their own software/hardware or bother to see what they are giving the customer is worth it...and then just stall, give lame answers and don't bother truly helping the customer. "Yea, it's sun spots that causing those issues...wait until it's dark out and try it again!"

So - final results - setting the tcp_host_param will up the send/receive packets and F-secure will work with that to give better throughput versus using SUN's ssh.

Thanks all who took notice and helped.

Sun version is the same openssh everyone bundles.

Instead of ranting and name calling without a clue, you'd better investigate why it goes faster with F-secure.
Possible reasons would be: