Solaris

Current Setup:

OS: Solaris 10 (5.10) x64
Motherboard: SUPERMICRO MBD-PDSME+-O LGA 775 Intel 3010
SATA Controller: SUPERMICRO AOC-SAT2-MV8 - SATA Controller (x2)
Link Aggregated @ 2Gb: 110MB sustained throughput

Code:

  NAME        STATE     READ WRITE CKSUM
       storagepool  ONLINE       0     0     0
         raidz1    ONLINE       0     0     0 
           c0t4d0  ONLINE       0     0     0        
           c0t5d0  ONLINE       0     0     0
           c0t6d0  ONLINE       0     0     0
           c0t7d0  ONLINE       0     0     0
         raidz1    ONLINE       0     0     0
           c1t0d0  ONLINE       0     0     0
           c1t1d0  ONLINE       0     0     0
           c1t2d0  ONLINE       0     0     0
           c1t3d0  ONLINE       0     0     0
       spares
         c1t5d0    AVAIL

Question:

I've had this setup for about a year now and want to restructure my zpool to optimize throughput.

On my current setup each raidz in located on a different controller and each controller is plugged into a PCI-X slot on the MB. This, I thought, was the best way to load-balance throughput. However I've done more reading and now my thoughts are to combine all HD's on one controller and create one zpool.

Is their merit to having all HD's on one controller or is it better to spit up the zpool over 2 controllers?

Does anyone have experience with this?

Thanks in advance

The more disks you can stripe together the wider the bandwidth until you reach the maximum bandwidth of the controller and/or PCI connection at which point if the there is any PCI bus bandwidth left then you need to use another controller in order to be able to gain more throughtput from striping more disks together.
Two controllers are obviously very advisable for mirrored storage in order to remove another single point of failure...

If im not wrong we dont have a mirrored setup here, but concatenation of 2 raidz. Using two controllers is always better than using one. You can try mixing disks that create one raidz on two controllers so it woul be something like:
raidz1
c0t4d0
c1t1d0
c0t6d0
c1t3d0
raidz1
c1t0d0
c0t5d0
c1t2d0
c0t7d0

That will give you same level of redundancy as before but might increase performance.

Correct, each raidz is on its own controller which is plugged into its own PCI-X slot.

Also, the Sun docs say that it's best to use 2-9 disks per vdev.