Sponsored Content
Top Forums Shell Programming and Scripting Synchronization in copying multiple files to a directory Post 302296961 by amio on Thursday 12th of March 2009 08:12:42 AM
Old 03-12-2009
Synchronization in copying multiple files to a directory

Hi Friends,

My goal is to copy 1000 of files of same filesize simultaneously to a directory. i.e i do multiple copies into a same directory. What i feel here is, at some 1000th iteration, few transfers ends in advance before some transfers start. I need to get synchronized file transfers so that all the 1000 copies run simultaneously.

To achieve this, i follow the below procedure.

i stop all the cp processes when they start and continue them after reaching 1000th cp. What is happening here is the cp does not stop at same filesize and continuing them again results in the same synchronization problem. i.e few transfers ends in advance before some transfers continue again. Am not sure whether this could be possible to achieve synchronization or am following wrong procedure.

My script is below.

Code:
declare -a mypid

for((i=0;i<1000;i++))
do
myvar="File$i.txt"
./cp_child.sh $i &
mypid[$i]=`ps -w aux | awk '/'"$myvar"'/ && !/awk/ { print $2 }'`
kill -s SIGSTOP ${mypid[$i]}
done
sleep 50
for((i=0;i<1000;i++))
do
kill -s SIGCONT ${mypid[$i]}
done

Code of cp_child.sh is

Code:
FILENAME="/home/TestFiles/FILE_250MB.txt"
WR_PATH="/home/MyTest/File$1.txt"
RD_PATH="/home/TestFiles/Read/Readfile$1.txt"
cp $FILENAME $WR_PATH
cp $WR_PATH $RD_PATH
cmp $FILENAME $RD_PATH

Also, is it possible to stop the cp process at specific filesize? If so, how to monitor the written file for its changing file size?

Thanks for ur patience.
Awaiting possible solutions.
Amio
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Copying files from one directory to Other

Hi UNIX Gurus, Could please help me out regarding following situation. I am copying some files from one directory to other directotry using following command. cp /var/tmp/*date*.gz /var/tmp/user/ Problem: Once the copy has completed, I need to check whether all the files (including... (3 Replies)
Discussion started by: satishkeshetty
3 Replies

2. UNIX for Dummies Questions & Answers

Help with cron: copying all files in a directory to another...

Here's what I have thus far: cp -r /home/mydom/public_html/products/Widget/ /home/mydom/public_html/ This works fine but suppose the folder in public_html has a different name (Main_Widget). The cron above needs to copy the files within the folder (Widget) instead of the folder itself. How... (1 Reply)
Discussion started by: ashleyquick
1 Replies

3. Shell Programming and Scripting

Copying files from one directory into another.

Could someone please tell me if there is a command similar to head or tail that can be used on directories. I want to select a given number of files from a directory and copy them into another directory. But I found out I can't use head as it doesn't (or I don't know how yet!) work on directories.... (4 Replies)
Discussion started by: Krush187
4 Replies

4. Shell Programming and Scripting

Copying files to a directory

Hi I have a few questions. I am trying to copy a file to a directory. I need to copy files that do not end in numbers, for example, into a directory. This is what I tried so far. cp filename directorytowhereIwannacopy but it says it can't copy to a directory. I need to copy many files into one... (2 Replies)
Discussion started by: #moveon
2 Replies

5. Shell Programming and Scripting

Find a directory and copying files to it

Hi all, wondering if someone would be able to help with a little query I have. I currently have a script which copies everything in a specific directory to a number of other directories depending on a variable. The problem I have is the destination directories vary (in and incoming directories)... (3 Replies)
Discussion started by: JayC89
3 Replies

6. UNIX for Dummies Questions & Answers

No such file or directory error while copying files

Hi, I need to copy files from one dir to another dir. The list of filesnames to be moved are in a file called files2cp.log Script: #!/bin/ksh exec 0</home/amdocs/files2cp.log while read LINE do cp -i /iccs33/attach/"$LINE" /iccs30/attach/"$LINE" done The output is "No such... (6 Replies)
Discussion started by: srinirsr
6 Replies

7. UNIX for Dummies Questions & Answers

copying multiple files

We want to copy the data from all the files SPR_201103* to AMGEN. We tried the following command cp $AGENT_DATA/amgen/SPR_201103* $AGENT_DATA/AMGEN amgen is directory . AMGEN is file we are trying to creat. for SPR_201103* 10 files are there Please help me to how to achive this.... (2 Replies)
Discussion started by: ajaykumarkona
2 Replies

8. UNIX for Dummies Questions & Answers

Help in copying multiple files from th directory

Hey Guys, I have directory with thousands of files, I need to copy only march data. can any one please Help.......... Thanks, Uttam N (4 Replies)
Discussion started by: Uttamnsd
4 Replies

9. Shell Programming and Scripting

Copying files to a directory

Hi, I am newbie to unix scripting, need a help in the doubt i have. It is " when files are copied to a directory using cp command, they are arranged by default according to the file name, is there anyway where the files that are copied be arranged with respect to their size, with using the sort... (5 Replies)
Discussion started by: pundalik
5 Replies

10. Shell Programming and Scripting

Copying files from parent directory only

Hello, Please can someone assist on a issue I am having. I want to find specific files in the parent directory only that have been modified over the last 2 days and copy them to another location. NOTE: The version of AIX I am using does not have MAXDEPTH. I have currently written the... (3 Replies)
Discussion started by: Dolph
3 Replies
libcurl-multi(5)					      libcurl multi interface						  libcurl-multi(5)

NAME
libcurl-multi - how to use the multi interface DESCRIPTION
This is an overview on how to use the libcurl multi interface in your C programs. There are specific man pages for each function mentioned in here. There's also the libcurl-the-guide document for a complete tutorial to programming with libcurl and the libcurl(3) man page for an overview of the libcurl easy interface. All functions in the multi interface are prefixed with curl_multi. PLEASE NOTICE
The multi interface is a rather new member of the libcurl family. It has not yet been very widely used. It may still be a few more bugs lurking in there than we are used to. That said, it might also just work in every aspect you try it. Please report all bugs and oddities you see. OBJECTIVES
The multi interface introduces several new abilities that the easy interface refuses to offer. They are mainly: 1. Enable a "pull" interface. The application that uses libcurl decides where and when to ask libcurl to get/send data. 2. Enable multiple simultaneous transfers in the same thread without making it complicated for the application. 3. Enable the application to select() on its own file descriptors and curl's file descriptors simultaneous easily. ONE MULTI HANDLE MANY EASY HANDLES
To use the multi interface, you must first create a 'multi handle' with curl_multi_init. This handle is then used as input to all further curl_multi_* functions. Each single transfer is built up with an easy handle. You must create them, and setup the appropriate options for each easy handle, as out- lined in the libcurl(3) man page. When the easy handle is setup for a transfer, then instead of using curl_easy_perform (as when using the easy interface for transfers), you should instead add the easy handle to the multi handle using curl_easy_add_handl. The multi handle is sometimes referred to as a 'multi stack' because of the fact that it may hold a large amount of easy handles. Should you change your mind, the easy handle is again removed from the multi stack using curl_multi_remove_handle. Once removed from the multi handle, you can again use other easy interface functions like curl_easy_perform or whatever you think is necessary. Adding the easy handles to the multi handle does not start any transfer. Remember that one of the main ideas with this interface is to let your application drive. You drive the transfers by invoking curl_multi_perform. libcurl will then transfer data if there is anything avail- able to transfer. It'll use the callbacks and everything else you have setup in the individual easy handles. It'll transfer data on all current transfers in the multi stack that are ready to transfer anything. It may be all, it may be none. Your application can acquire knowledge from libcurl when it would like to get invoked to transfer data, so that you don't have to busy-loop and call that curl_multi_perform like a mad man! curl_multi_fdset offers an interface using which you can extract fd_sets from libcurl to use in select() or poll() calls in order to get to know when the transfers in the multi stack might need attention. This also makes it very easy for your program to wait for input on your own private file descriptors at the same time or perhaps timeout every now and then, should you want that. curl_multi_perform stores the number of still running transfers in one of its input arguments, and by reading that you can figure out when all the transfers in the multi handles are done. 'done' does not mean successful. One or more of the transfers may have failed. To get information about completed transfers, to figure out success or not and similar, curl_multi_info_read should be called. It can return a message about a current or previous transfer. Repeated invokes of the function get more messages until the message queue is empty. When all transfers in the multi stack are done, cleanup the multi handle with curl_multi_cleanup. Be careful and please note that you MUST invoke separate curl_easy_cleanup calls on every single easy handle to clean them up properly. libcurl 7.9.5 20 March 2001 libcurl-multi(5)
All times are GMT -4. The time now is 10:11 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy