The writes from CIFS are probably synchronous to ensure reliable data transfer all the way from the client to disk platters. If so, what you're seeing is a feature of ZFS. Read the "
ZFS Evil Tuning Guide" and pay particular attention to the ARC and ZIL sections. You can try disabling your ZIL and see if your jumpiness disappears.
FWIW, unless you have a need to have deterministic performance at all times, I wouldn't worry too much about what you're seeing. Especially if the jumpiness in IO throughput disappears if you test with the ZIL disabled.
In my opinion, ZFS has a weakness in its ability to guarantee deterministic performance. Similar to what you're seeing, in my experience ZFS tends to be very episodic in the way it writes data to disk. Most of the time it's pretty fast, but every few seconds I've noted that when it is under a continuous write load, ZFS tends to grind to a short halt.
