Synchronization in copying multiple files to a directory
Hi Friends,
My goal is to copy 1000 of files of same filesize simultaneously to a directory. i.e i do multiple copies into a same directory. What i feel here is, at some 1000th iteration, few transfers ends in advance before some transfers start. I need to get synchronized file transfers so that all the 1000 copies run simultaneously.
To achieve this, i follow the below procedure.
i stop all the cp processes when they start and continue them after reaching 1000th cp. What is happening here is the cp does not stop at same filesize and continuing them again results in the same synchronization problem. i.e few transfers ends in advance before some transfers continue again. Am not sure whether this could be possible to achieve synchronization or am following wrong procedure.
My script is below.
Code:
declare -a mypid
for((i=0;i<1000;i++))
do
myvar="File$i.txt"
./cp_child.sh $i &
mypid[$i]=`ps -w aux | awk '/'"$myvar"'/ && !/awk/ { print $2 }'`
kill -s SIGSTOP ${mypid[$i]}
done
sleep 50
for((i=0;i<1000;i++))
do
kill -s SIGCONT ${mypid[$i]}
done
Hi UNIX Gurus,
Could please help me out regarding following situation.
I am copying some files from one directory to other directotry using following command.
cp /var/tmp/*date*.gz /var/tmp/user/
Problem:
Once the copy has completed, I need to check whether all the files (including... (3 Replies)
Here's what I have thus far:
cp -r /home/mydom/public_html/products/Widget/ /home/mydom/public_html/
This works fine but suppose the folder in public_html has a different name (Main_Widget). The cron above needs to copy the files within the folder (Widget) instead of the folder itself. How... (1 Reply)
Could someone please tell me if there is a command similar to head or tail that can be used on directories. I want to select a given number of files from a directory and copy them into another directory. But I found out I can't use head as it doesn't (or I don't know how yet!) work on directories.... (4 Replies)
Hi I have a few questions. I am trying to copy a file to a directory. I need to copy files that do not end in numbers, for example, into a directory. This is what I tried so far. cp filename directorytowhereIwannacopy but it says it can't copy to a directory. I need to copy many files into one... (2 Replies)
Hi all, wondering if someone would be able to help with a little query I have.
I currently have a script which copies everything in a specific directory to a number of other directories depending on a variable. The problem I have is the destination directories vary (in and incoming directories)... (3 Replies)
Hi,
I need to copy files from one dir to another dir.
The list of filesnames to be moved are in a file called files2cp.log
Script:
#!/bin/ksh
exec 0</home/amdocs/files2cp.log
while read LINE
do
cp -i /iccs33/attach/"$LINE" /iccs30/attach/"$LINE"
done
The output is "No such... (6 Replies)
We want to copy the data from all the files SPR_201103* to AMGEN.
We tried the following command
cp $AGENT_DATA/amgen/SPR_201103* $AGENT_DATA/AMGEN
amgen is directory .
AMGEN is file we are trying to creat.
for SPR_201103* 10 files are there
Please help me to how to achive this.... (2 Replies)
Hi,
I am newbie to unix scripting, need a help in the doubt i have. It is " when files are copied to a directory using cp command, they are arranged by default according to the file name, is there anyway where the files that are copied be arranged with respect to their size, with using the sort... (5 Replies)
Hello,
Please can someone assist on a issue I am having.
I want to find specific files in the parent directory only that have been modified over the last 2 days and copy them to another location.
NOTE: The version of AIX I am using does not have MAXDEPTH.
I have currently written the... (3 Replies)
Discussion started by: Dolph
3 Replies
LEARN ABOUT REDHAT
libcurl-multi
libcurl-multi(5) libcurl multi interface libcurl-multi(5)NAME
libcurl-multi - how to use the multi interface
DESCRIPTION
This is an overview on how to use the libcurl multi interface in your C programs. There are specific man pages for each function mentioned
in here. There's also the libcurl-the-guide document for a complete tutorial to programming with libcurl and the libcurl(3) man page for an
overview of the libcurl easy interface.
All functions in the multi interface are prefixed with curl_multi.
PLEASE NOTICE
The multi interface is a rather new member of the libcurl family. It has not yet been very widely used. It may still be a few more bugs
lurking in there than we are used to. That said, it might also just work in every aspect you try it. Please report all bugs and oddities
you see.
OBJECTIVES
The multi interface introduces several new abilities that the easy interface refuses to offer. They are mainly:
1. Enable a "pull" interface. The application that uses libcurl decides where and when to ask libcurl to get/send data.
2. Enable multiple simultaneous transfers in the same thread without making it complicated for the application.
3. Enable the application to select() on its own file descriptors and curl's file descriptors simultaneous easily.
ONE MULTI HANDLE MANY EASY HANDLES
To use the multi interface, you must first create a 'multi handle' with curl_multi_init. This handle is then used as input to all further
curl_multi_* functions.
Each single transfer is built up with an easy handle. You must create them, and setup the appropriate options for each easy handle, as out-
lined in the libcurl(3) man page.
When the easy handle is setup for a transfer, then instead of using curl_easy_perform (as when using the easy interface for transfers), you
should instead add the easy handle to the multi handle using curl_easy_add_handl. The multi handle is sometimes referred to as a 'multi
stack' because of the fact that it may hold a large amount of easy handles.
Should you change your mind, the easy handle is again removed from the multi stack using curl_multi_remove_handle. Once removed from the
multi handle, you can again use other easy interface functions like curl_easy_perform or whatever you think is necessary.
Adding the easy handles to the multi handle does not start any transfer. Remember that one of the main ideas with this interface is to let
your application drive. You drive the transfers by invoking curl_multi_perform. libcurl will then transfer data if there is anything avail-
able to transfer. It'll use the callbacks and everything else you have setup in the individual easy handles. It'll transfer data on all
current transfers in the multi stack that are ready to transfer anything. It may be all, it may be none.
Your application can acquire knowledge from libcurl when it would like to get invoked to transfer data, so that you don't have to busy-loop
and call that curl_multi_perform like a mad man! curl_multi_fdset offers an interface using which you can extract fd_sets from libcurl to
use in select() or poll() calls in order to get to know when the transfers in the multi stack might need attention. This also makes it very
easy for your program to wait for input on your own private file descriptors at the same time or perhaps timeout every now and then, should
you want that.
curl_multi_perform stores the number of still running transfers in one of its input arguments, and by reading that you can figure out when
all the transfers in the multi handles are done. 'done' does not mean successful. One or more of the transfers may have failed.
To get information about completed transfers, to figure out success or not and similar, curl_multi_info_read should be called. It can
return a message about a current or previous transfer. Repeated invokes of the function get more messages until the message queue is empty.
When all transfers in the multi stack are done, cleanup the multi handle with curl_multi_cleanup. Be careful and please note that you MUST
invoke separate curl_easy_cleanup calls on every single easy handle to clean them up properly.
libcurl 7.9.5 20 March 2001 libcurl-multi(5)