Thread synchronisation problem...


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Thread synchronisation problem...
# 1  
Old 05-28-2009
Thread synchronisation problem...

Hello, hope my english will be sufficient to be clear enough...

I'm in progress on some script that should copy one big source file (200GB average) on one sata drive, to multiple (30+) sata drives.
Hardware is not the problem but copy performance is... If i launch all copy process at the same time, source device is overrun by read IO and performance is (very) bad. If i launch one copy after one copy, copy performance is ok but i have multiply copy time by 30...

Let's summary the story :

i thnak to some script that read via ''dd'' in block mode to a ramdrive, ''1MB per 1MB piece'', then this script launch n writing dd (n = number of destination devices) to destination devices from ramdrive.
Idea is reading from a ramdrive with 30+ thread at the same time do not cause catastrophic perfomrance drop as it does with mecanical source device.

BUT...

I'm just facing a simple but big problem :
in the writing part of my script, i'm launching several ''write dd'' at the same time :
dd if=/mnt/ramdrive/buf.dat of=$destination/stillwriting.tmp bs=$BS seek=$POINTEUR_BLOC oflag=append status=noxfer &
"&" is my only known way to launch parralel process in a script, the problem is that the script continue to run and especially begins to put following 1MB block to ramdrive BEFORE end of all writing process. This result in corrupted files. I'm quite sure of that diagnostic because if i insert some sleep commande in my writing loop, error do not occur (but performance is lower and it's kind of very dirty scripting...).

Would you know a way to pause running of my script until all writing process are finished ?


Thanks a lot for your help from Paris !
# 2  
Old 05-28-2009
Yes there is. It's called 'wait', eg
Code:
( sleep 30 ; echo "Sleep ended" ) &
echo "After sleep"
wait

# 3  
Old 05-28-2009
Thanks for this answer !
However, pardon me but i'm still in doubt in applying this wait command to my portion of script :

Here is my loop :
$2 is a file containing destination path list. It oculd be as big as 30 or more different paths. So the blue loop is creating as many dd processes as number of different destination paths. Those dd processes have to be completed before going to the read of following block.

for i in `seq 1 $NB_PASS`
do


#copie d'une tranche vers ramdrive. "Read" from source to ramdrive
dd if=$1 of=/mnt/ramdrive/buf.dat bs=$BS count=$COUNT skip=$POINTEUR_BLOC status=noxfer

#copie de cette tranche vers les destinations. "Write" from ramdrive to multi destinations
for destination in $(cat $2)
do

dd if=/mnt/ramdrive/buf.dat of=$destination/stillwriting.tmp bs=$BS seek=$POINTEUR_BLOC oflag=append status=noxfer &
done

POINTEUR_BLOC=`expr $POINTEUR_BLOC + $COUNT`
done

I would like that the blue loop wait for all it's child dd process to finish before go back to red loop.

Last edited by Gnaag; 05-28-2009 at 10:00 AM..
# 4  
Old 05-28-2009
Code:
dd if=/mnt/ramdrive/buf.dat of=$destination/stillwriting.tmp bs=$BS seek=$POINTEUR_BLOC oflag=append status=noxfer &
done

wait 

POINTEUR_BLOC=`expr $POINTEUR_BLOC + $COUNT`

# 5  
Old 05-28-2009
seems to work !

Thnak you very much for your help, have a good day !
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Programming

Thread synchronisation

hi, i have to do a program with following condition. please help me to write the program. conditions-i have to create a thread with handle called first and it should call the member function(may do anything lik print anything) of a class called thread1 and for example let take that this first... (5 Replies)
Discussion started by: senthil.march
5 Replies

2. Shell Programming and Scripting

Synchronisation of 2 arrays

If the element found in array2 doesn't exist in array1 i want i to be deleted. This code doesn't work correctly. What's wrong? Is there a simpler solution? for (( i=0; i<=${#array2}; i++ )) do for (( v=0; v<=${#array1}; v++ )) do if }" = "{$array1}" ] then count=1 break... (3 Replies)
Discussion started by: iums1
3 Replies

3. Shell Programming and Scripting

rsync script for synchronisation and backup

hello, i need to modified my synch/back scripts.... i want that this script only syncro folders in destinationfolder. f.e. when in destination are two folders 1) admin 2) users but in SOURCE are three: 1) admin 2) users 3) antivirus the script should only increnmential sync the... (0 Replies)
Discussion started by: onkeldave
0 Replies

4. UNIX for Dummies Questions & Answers

thread pool problem

hello everyone. I want to implement a thread pool, with 10 threads most. Inside main,I call a function (lets say it foo) wich creates (if it is needed) or uses an existing thread from the pool and sends it to do a job.My problem is that I dont know how to pass the argument from the main to the... (2 Replies)
Discussion started by: garag11
2 Replies

5. UNIX for Dummies Questions & Answers

Need help on file synchronisation in unix

i want to do file synchronisation...its a client-server model..have to do system call 'ls -l' in both client and server ..the server has to keeep track of client files and have to keep a back up of client files..and tracking must be done based on time stamp.. suppose if client contains a file... (2 Replies)
Discussion started by: shilpam,edappal
2 Replies

6. Post Here to Contact Site Administrators and Moderators

posting thread is problem

hi, i am unable to post new thread. pls revert back what i need to do. i already posted 3 threads.... I am getting message like " Message is too short" even though my message is more than 100 characters. (1 Reply)
Discussion started by: spc432
1 Replies

7. Shell Programming and Scripting

sh : URGENT synchronisation insmod in script

Hello, By now in linux 2.4, I have a sh script wich start 2 modules as follow : /sbin/insmod module1.o /sbin/insmod module2.o I added an application in user space named "user_app" which communicate with module1 with a /proc. I now tape the commands myself during code execution on a... (1 Reply)
Discussion started by: crip01
1 Replies

8. UNIX for Advanced & Expert Users

shared memory synchronisation

hello everybody i want to do synchronisation to access a shared memory bu i don't know too much how well i know that i should use semaphore have you any example of synchronisation of a shared memory by use of semaphore because i haven't find any thanks (0 Replies)
Discussion started by: student00
0 Replies
Login or Register to Ask a Question