Solaris

Hi, I tried this question on OTN Discussion Forum, but it seems that forum doesn't have as much activity so I thought I'd try here.

I have a big performance problem with Samba server on a fresh install of Solaris Express 11 svn_151a, and a freshly created ZFS pool. I'm getting about 3 to 10 MB/s transfers for large media files (and other large incompressible test files), under controlled testing.

Just prior, I was getting 90 MB/s with OpenSolaris 2009.06 svn_11b (also with minimal tweaking), with same testing and conditions. Nothing changed with hardware or network config...just fresh new Solaris, and fresh new ZPool.

Subjectively too, it absolutely c..r..a..w..l..s, when it used to fly before.

There are many reasons I'm using Samba server rather than the built-in CIFS/SMB, but that is beyond the scope of this question.

Thanks!
-Jim

Specs:

Machine:
- 2*4-core Xeon CPUs
- 16gb ECC RAM
- Onboard e1000g NIC
- 20 hot-swap bays

rpool:
- mirrored 30gb SSDs

Pool "zp3" config:
- LSI SAS1068E-R c9 HBAs
- 6 * 3-way mirrors, 1tb, 7200 RPM enterprise-class SATA
- 2 hot spares
- 2 * 30gb SSD for L2ARC

Pool "zp3" properties:
zp3 version 5 -
zp3 sync disabled local
zp3 com.sun:auto-snapshot true local
zp3 recordsize 128K default
zp3 utf8only on -
zp3 normalization formD -
zp3 casesensitivity insensitive -
zp3 encryption off -
zp3 compression on local
zp3 compressratio 1.07x -
zp3 dedup off local
zp3 logbias throughput local
zp3 used 4.21T -
zp3 usedbysnapshots 311G -
zp3 usedbydataset 3.90T -
zp3 usedbychildren 8.78G -
zp3 usedbyrefreservation 0 -
zp3 available 1.04T -

Type of data and access:
- Mostly large media files (DNG photos, large photoshop project files, high bitrate 1080p video, 32-bit/96khz audio, etc.)
- Plenty of small "office" data files too, but of secondary importance.
- Low usage. Typically accessed by one CIFS/SMB client - and one application - at a time (mix of Windows, Mac, Linux). Needs throughput, not IOPS.

the last time a friend has this kind of problem it was the MTU size. maybe worth a check...

Thanks for the suggestion, DukeNuke2.

I tried increasing the MTU. Hours of Googling how to do it, and trial-and-error later, I couldn't get 'er done. (And I have less hair now...as with so many things Solaris it seems...)

To wit: I have an Intel e1000g NIC, so in /kernel/drv/e1000g.conf, I specified:

Code:

ForceSpeedDuplex=7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7;
AutoNegAdvertised=32,32,32,32,32,32,32,32,32,32,32,32,32,32,32,32;
MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;

(I know I technically only needed to populate the first instance for each setting, but I wanted to be sure!)

Then in /etc/hostname.e1000g0, I specified:

Code:

b13 mtu 9000 up

(The latter one seems to be for older versions of Solaris near as I could figure, but did it just in case.)

However, after rebooting, ifconfig continues to e1000g0 yield:

Code:

e1000g0: flags=1004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4> mtu 1500 index 5

(Same for IPv6.) So, it doesn't seem to be accepting the change. Furthermore, I now recall having the same problem with OpenSolaris 2009.06 with trying (and failing) to set Jumbo Frames. (But still got good performance without it.)

The strange thing is, the performance problems seem to have gone away. I'm now getting about 40 to 95 MB/s on large sequential reads and writes.

I know it had to be something I did, because my previous tests spanned countless hours of fiddling, tweaking and dozens of reboots. The problem didn't just solve itself, nor as it transient.

(But, since it is working now, I'm going to leave it alone. Forever! Seriously. No patches, nothing. This machine will remain with this exact hardware configuration (even though it is expensive overkill for the job by today's standard), and OS/version, FOREVER. Just swapping out ever-larger disks regularly to continuously grow the pool, replacing failed components with exact replacements, until the horsepower just can't keep up with the growing volume of data any more. In hindsight I should have never upgraded from svn_111b. I've kind of had it with Solaris . I love ZFS, but for the love of God these kinds of things are killing me. Every little minor thing turns into hours of research, tweaking, trial-and-error, and mish-mashes of "new" but non-*nix-ey commands, and ancient config files, scripts, and commands. When it comes time, I'll replace it with a new machine, running a Linux OS, and Btrfs which should easily be ready by then.)

Anyway, thanks for the suggestion! Although I wasn't able to make your suggestion work, it was at least responsible, somehow, for indirectly improving the performance somehow...


Edit: I should also point out that I'm using a managed switch with jumbo frames support explicitly enabled, and jumbo frames turned on for the other mix of various Windows versions and Linux machines.