I think one of the biggest mistakes people make when testing is to time the test from start to finish and not pay attention to anything in between. For example, I always use netperf to measure network performance, or dt to measure disk as well as others, BUT I always run collectl in another window to see what's happening to my cpu, disk, network, memory and other subsystems while the test is in process. If you get an bad end-to-end number, and the intermediate numbers are very erratic it just may be a system or network switch is misconfigured. Looking at the elapsed time or average load will never give you a true picture.

In fact, if you run collectl with a monitoring interval of 0.1 seconds or even less which doing disk tests you can actually watch the cache fill as the tests run faster at first.

-mark