Synchronization in copying multiple files to a directory


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Synchronization in copying multiple files to a directory
# 1  
Old 03-12-2009
Synchronization in copying multiple files to a directory

Hi Friends,

My goal is to copy 1000 of files of same filesize simultaneously to a directory. i.e i do multiple copies into a same directory. What i feel here is, at some 1000th iteration, few transfers ends in advance before some transfers start. I need to get synchronized file transfers so that all the 1000 copies run simultaneously.

To achieve this, i follow the below procedure.

i stop all the cp processes when they start and continue them after reaching 1000th cp. What is happening here is the cp does not stop at same filesize and continuing them again results in the same synchronization problem. i.e few transfers ends in advance before some transfers continue again. Am not sure whether this could be possible to achieve synchronization or am following wrong procedure.

My script is below.

Code:
declare -a mypid

for((i=0;i<1000;i++))
do
myvar="File$i.txt"
./cp_child.sh $i &
mypid[$i]=`ps -w aux | awk '/'"$myvar"'/ && !/awk/ { print $2 }'`
kill -s SIGSTOP ${mypid[$i]}
done
sleep 50
for((i=0;i<1000;i++))
do
kill -s SIGCONT ${mypid[$i]}
done

Code of cp_child.sh is

Code:
FILENAME="/home/TestFiles/FILE_250MB.txt"
WR_PATH="/home/MyTest/File$1.txt"
RD_PATH="/home/TestFiles/Read/Readfile$1.txt"
cp $FILENAME $WR_PATH
cp $WR_PATH $RD_PATH
cmp $FILENAME $RD_PATH

Also, is it possible to stop the cp process at specific filesize? If so, how to monitor the written file for its changing file size?

Thanks for ur patience.
Awaiting possible solutions.
Amio
# 2  
Old 03-12-2009
You are looping 1000 times and fork/exec'ing a shell script. This alone precludes "synchronized file transfers so that all the 1000 copies run simultaneously."
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Copying files from parent directory only

Hello, Please can someone assist on a issue I am having. I want to find specific files in the parent directory only that have been modified over the last 2 days and copy them to another location. NOTE: The version of AIX I am using does not have MAXDEPTH. I have currently written the... (3 Replies)
Discussion started by: Dolph
3 Replies

2. Shell Programming and Scripting

Copying files to a directory

Hi, I am newbie to unix scripting, need a help in the doubt i have. It is " when files are copied to a directory using cp command, they are arranged by default according to the file name, is there anyway where the files that are copied be arranged with respect to their size, with using the sort... (5 Replies)
Discussion started by: pundalik
5 Replies

3. UNIX for Dummies Questions & Answers

Help in copying multiple files from th directory

Hey Guys, I have directory with thousands of files, I need to copy only march data. can any one please Help.......... Thanks, Uttam N (4 Replies)
Discussion started by: Uttamnsd
4 Replies

4. UNIX for Dummies Questions & Answers

copying multiple files

We want to copy the data from all the files SPR_201103* to AMGEN. We tried the following command cp $AGENT_DATA/amgen/SPR_201103* $AGENT_DATA/AMGEN amgen is directory . AMGEN is file we are trying to creat. for SPR_201103* 10 files are there Please help me to how to achive this.... (2 Replies)
Discussion started by: ajaykumarkona
2 Replies

5. UNIX for Dummies Questions & Answers

No such file or directory error while copying files

Hi, I need to copy files from one dir to another dir. The list of filesnames to be moved are in a file called files2cp.log Script: #!/bin/ksh exec 0</home/amdocs/files2cp.log while read LINE do cp -i /iccs33/attach/"$LINE" /iccs30/attach/"$LINE" done The output is "No such... (6 Replies)
Discussion started by: srinirsr
6 Replies

6. Shell Programming and Scripting

Find a directory and copying files to it

Hi all, wondering if someone would be able to help with a little query I have. I currently have a script which copies everything in a specific directory to a number of other directories depending on a variable. The problem I have is the destination directories vary (in and incoming directories)... (3 Replies)
Discussion started by: JayC89
3 Replies

7. Shell Programming and Scripting

Copying files to a directory

Hi I have a few questions. I am trying to copy a file to a directory. I need to copy files that do not end in numbers, for example, into a directory. This is what I tried so far. cp filename directorytowhereIwannacopy but it says it can't copy to a directory. I need to copy many files into one... (2 Replies)
Discussion started by: #moveon
2 Replies

8. Shell Programming and Scripting

Copying files from one directory into another.

Could someone please tell me if there is a command similar to head or tail that can be used on directories. I want to select a given number of files from a directory and copy them into another directory. But I found out I can't use head as it doesn't (or I don't know how yet!) work on directories.... (4 Replies)
Discussion started by: Krush187
4 Replies

9. UNIX for Dummies Questions & Answers

Help with cron: copying all files in a directory to another...

Here's what I have thus far: cp -r /home/mydom/public_html/products/Widget/ /home/mydom/public_html/ This works fine but suppose the folder in public_html has a different name (Main_Widget). The cron above needs to copy the files within the folder (Widget) instead of the folder itself. How... (1 Reply)
Discussion started by: ashleyquick
1 Replies

10. UNIX for Dummies Questions & Answers

Copying files from one directory to Other

Hi UNIX Gurus, Could please help me out regarding following situation. I am copying some files from one directory to other directotry using following command. cp /var/tmp/*date*.gz /var/tmp/user/ Problem: Once the copy has completed, I need to check whether all the files (including... (3 Replies)
Discussion started by: satishkeshetty
3 Replies
Login or Register to Ask a Question