Solaris

the box is dedicated for cifs storage so nothing apart from serving

the data is all sequential data and not in parallel

compression isnt enabled

and a netapp uses Ontap which is basically ZFS with a different name... sun and netapp had a big lawsuit over it :S

This looks to me to be a typical ZFS IO pattern - buffer all writes, dump them to disk periodically.

This is what I think is a flaw with ZFS - it's impossible to get good high-speed streaming write performance from any file system if that file system doesn't bother to write data to disk for long periods of time. The time where the disks are idle is lost bandwidth. As you're seeing, ZFS leaves disks idle too long to support fast streaming writing for long periods of time with it's "fill up the ARC, stop everything while the ARC is flushed to disk, repeat" manner of writing data. And when the ARC is being flushed to disk, everything else on the server comes to a halt.

Does you're projected real-world workload involve dumping gigabyte+ sized files onto the file system all at once? If not, this behavior shouldn't be a problem.