Thread synchronisation problem...

[Gnaag]

Hello, hope my english will be sufficient to be clear enough...

I'm in progress on some script that should copy one big source file (200GB average) on one sata drive, to multiple (30+) sata drives.
Hardware is not the problem but copy performance is... If i launch all copy process at the same time, source device is overrun by read IO and performance is (very) bad. If i launch one copy after one copy, copy performance is ok but i have multiply copy time by 30...

Let's summary the story :

i thnak to some script that read via ''dd'' in block mode to a ramdrive, ''1MB per 1MB piece'', then this script launch n writing dd (n = number of destination devices) to destination devices from ramdrive.
Idea is reading from a ramdrive with 30+ thread at the same time do not cause catastrophic perfomrance drop as it does with mecanical source device.

BUT...

I'm just facing a simple but big problem :
in the writing part of my script, i'm launching several ''write dd'' at the same time :
dd if=/mnt/ramdrive/buf.dat of=$destination/stillwriting.tmp bs=$BS seek=$POINTEUR_BLOC oflag=append status=noxfer &
"&" is my only known way to launch parralel process in a script, the problem is that the script continue to run and especially begins to put following 1MB block to ramdrive BEFORE end of all writing process. This result in corrupted files. I'm quite sure of that diagnostic because if i insert some sleep commande in my writing loop, error do not occur (but performance is lower and it's kind of very dirty scripting...).

Would you know a way to pause running of my script until all writing process are finished ?


Thanks a lot for your help from Paris !

[pludi]

Yes there is. It's called 'wait', eg

( sleep 30 ; echo "Sleep ended" ) &
echo "After sleep"
wait

[Gnaag]

Thanks for this answer !
However, pardon me but i'm still in doubt in applying this wait command to my portion of script :

Here is my loop :
$2 is a file containing destination path list. It oculd be as big as 30 or more different paths. So the blue loop is creating as many dd processes as number of different destination paths. Those dd processes have to be completed before going to the read of following block.

for i in `seq 1 $NB_PASS`
do

#copie d'une tranche vers ramdrive. "Read" from source to ramdrive
dd if=$1 of=/mnt/ramdrive/buf.dat bs=$BS count=$COUNT skip=$POINTEUR_BLOC status=noxfer

#copie de cette tranche vers les destinations. "Write" from ramdrive to multi destinations
for destination in $(cat $2)
do
dd if=/mnt/ramdrive/buf.dat of=$destination/stillwriting.tmp bs=$BS seek=$POINTEUR_BLOC oflag=append status=noxfer &
done

POINTEUR_BLOC=`expr $POINTEUR_BLOC + $COUNT`
done

I would like that the blue loop wait for all it's child dd process to finish before go back to red loop.

[fpmurphy]

dd if=/mnt/ramdrive/buf.dat of=$destination/stillwriting.tmp bs=$BS seek=$POINTEUR_BLOC oflag=append status=noxfer &
done

wait 

POINTEUR_BLOC=`expr $POINTEUR_BLOC + $COUNT`

[Gnaag]

seems to work !

Thnak you very much for your help, have a good day !