Solaris

I think ufsdump doesn't backup partition table? so it might be a good idea to backup those few first sectors with dd. or simply write down label and slices info.

btw, we should *always* increase blocksize when using dd.
iirc, the default is 1byte so its slooow.
of course keeping in mind blocksize when using count!
(sorry if this all is too obvious)

edit: one advantage of dd would be that you can do the restore on any computer/OS with scsi card. even Windows has dd port.

Just a thought, By any chance if you are using mirroring for your root/app partition, add one more sub-mirror residing on USB disk.
Whenever you want to take backup attach additional mirror, once done detach it to reduce sync I/O.
You can schedule it using cronjob also.
No downtime! No User intervention!!!

That is a good idea, but we need to have multiple images of the servers in different states on each of the USB drives. We are doing application development and need to be able to put the servers back multiple configurations in the development cycle. The idea is that after every software change we back it up and store it along with the rest.

You should migrate to ZFS root. That would allow almost instant rollbacks to your reference installations instead of these painful backup-restore procedures. Depending on your requirements, zones might also help simplifying your process.