Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Synchronization in copying multiple files to a directory
Top Forums Shell Programming and Scripting Synchronization in copying multiple files to a directory Post 302296961 by amio on Thursday 12th of March 2009 08:12:42 AM
Old 03-12-2009
Synchronization in copying multiple files to a directory
Hi Friends,

My goal is to copy 1000 of files of same filesize simultaneously to a directory. i.e i do multiple copies into a same directory. What i feel here is, at some 1000th iteration, few transfers ends in advance before some transfers start. I need to get synchronized file transfers so that all the 1000 copies run simultaneously.

To achieve this, i follow the below procedure.

i stop all the cp processes when they start and continue them after reaching 1000th cp. What is happening here is the cp does not stop at same filesize and continuing them again results in the same synchronization problem. i.e few transfers ends in advance before some transfers continue again. Am not sure whether this could be possible to achieve synchronization or am following wrong procedure.

My script is below.

Code:
declare -a mypid

for((i=0;i<1000;i++))
do
myvar="File$i.txt"
./cp_child.sh $i &
mypid[$i]=`ps -w aux | awk '/'"$myvar"'/ && !/awk/ { print $2 }'`
kill -s SIGSTOP ${mypid[$i]}
done
sleep 50
for((i=0;i<1000;i++))
do
kill -s SIGCONT ${mypid[$i]}
done

Code of cp_child.sh is

Code:
FILENAME="/home/TestFiles/FILE_250MB.txt"
WR_PATH="/home/MyTest/File$1.txt"
RD_PATH="/home/TestFiles/Read/Readfile$1.txt"
cp $FILENAME $WR_PATH
cp $WR_PATH $RD_PATH
cmp $FILENAME $RD_PATH

Also, is it possible to stop the cp process at specific filesize? If so, how to monitor the written file for its changing file size?

Thanks for ur patience.
Awaiting possible solutions.
Amio
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Copying files from one directory to Other

Hi UNIX Gurus, Could please help me out regarding following situation. I am copying some files from one directory to other directotry using following command. cp /var/tmp/*date*.gz /var/tmp/user/ Problem: Once the copy has completed, I need to check whether all the files (including... (3 Replies)
Discussion started by: satishkeshetty
3 Replies

2. UNIX for Dummies Questions & Answers

Help with cron: copying all files in a directory to another...

Here's what I have thus far: cp -r /home/mydom/public_html/products/Widget/ /home/mydom/public_html/ This works fine but suppose the folder in public_html has a different name (Main_Widget). The cron above needs to copy the files within the folder (Widget) instead of the folder itself. How... (1 Reply)
Discussion started by: ashleyquick
1 Replies

3. Shell Programming and Scripting

Copying files from one directory into another.

Could someone please tell me if there is a command similar to head or tail that can be used on directories. I want to select a given number of files from a directory and copy them into another directory. But I found out I can't use head as it doesn't (or I don't know how yet!) work on directories.... (4 Replies)
Discussion started by: Krush187
4 Replies

4. Shell Programming and Scripting

Copying files to a directory

Hi I have a few questions. I am trying to copy a file to a directory. I need to copy files that do not end in numbers, for example, into a directory. This is what I tried so far. cp filename directorytowhereIwannacopy but it says it can't copy to a directory. I need to copy many files into one... (2 Replies)
Discussion started by: #moveon
2 Replies

5. Shell Programming and Scripting

Find a directory and copying files to it

Hi all, wondering if someone would be able to help with a little query I have. I currently have a script which copies everything in a specific directory to a number of other directories depending on a variable. The problem I have is the destination directories vary (in and incoming directories)... (3 Replies)
Discussion started by: JayC89
3 Replies

6. UNIX for Dummies Questions & Answers

No such file or directory error while copying files

Hi, I need to copy files from one dir to another dir. The list of filesnames to be moved are in a file called files2cp.log Script: #!/bin/ksh exec 0</home/amdocs/files2cp.log while read LINE do cp -i /iccs33/attach/"$LINE" /iccs30/attach/"$LINE" done The output is "No such... (6 Replies)
Discussion started by: srinirsr
6 Replies

7. UNIX for Dummies Questions & Answers

copying multiple files

We want to copy the data from all the files SPR_201103* to AMGEN. We tried the following command cp $AGENT_DATA/amgen/SPR_201103* $AGENT_DATA/AMGEN amgen is directory . AMGEN is file we are trying to creat. for SPR_201103* 10 files are there Please help me to how to achive this.... (2 Replies)
Discussion started by: ajaykumarkona
2 Replies

8. UNIX for Dummies Questions & Answers

Help in copying multiple files from th directory

Hey Guys, I have directory with thousands of files, I need to copy only march data. can any one please Help.......... Thanks, Uttam N (4 Replies)
Discussion started by: Uttamnsd
4 Replies

9. Shell Programming and Scripting

Copying files to a directory

Hi, I am newbie to unix scripting, need a help in the doubt i have. It is " when files are copied to a directory using cp command, they are arranged by default according to the file name, is there anyway where the files that are copied be arranged with respect to their size, with using the sort... (5 Replies)
Discussion started by: pundalik
5 Replies

10. Shell Programming and Scripting

Copying files from parent directory only

Hello, Please can someone assist on a issue I am having. I want to find specific files in the parent directory only that have been modified over the last 2 days and copy them to another location. NOTE: The version of AIX I am using does not have MAXDEPTH. I have currently written the... (3 Replies)
Discussion started by: Dolph
3 Replies

LEARN ABOUT BSD

tftp

TFTP(1) 						      General Commands Manual							   TFTP(1)

NAME

       tftp - trivial file transfer program

SYNOPSIS

       tftp [ host ]

DESCRIPTION

       Tftp is the user interface to the Internet TFTP (Trivial File Transfer Protocol), which allows users to transfer files to and from a remote
       machine.  The remote host may be specified on the command line, in which case tftp uses host as the default host for future transfers  (see
       the connect command below).

COMMANDS

       Once tftp is running, it issues the prompt tftp> and recognizes the following commands:

       connect host-name [ port ]
	      Set  the	host (and optionally port) for transfers.  Note that the TFTP protocol, unlike the FTP protocol, does not maintain connec-
	      tions betweeen transfers; thus, the connect command does not actually create a connection, but merely remembers what host is  to	be
	      used  for  transfers.   You  do not have to use the connect command; the remote host can be specified as part of the get or put com-
	      mands.

       mode transfer-mode
	      Set the mode for transfers; transfer-mode may be one of ascii or binary.	The default is ascii.

       put file
       put localfile remotefile
       put file1 file2 ... fileN remote-directory
	      Put a file or set of files to the specified remote file or directory.  The destination can be in one of two forms: a filename on the
	      remote  host,  if  the host has already been specified, or a string of the form host:filename to specify both a host and filename at
	      the same time.  If the latter form is used, the hostname specified becomes the default for future transfers.  If	the  remote-direc-
	      tory form is used, the remote host is assumed to be a UNIX machine.

       get filename
       get remotename localname
       get file1 file2 ... fileN
	      Get  a  file  or	set of files from the specified sources.  Source can be in one of two forms: a filename on the remote host, if the
	      host has already been specified, or a string of the form host:filename to specify both a host and filename at the same time.  If the
	      latter form is used, the last hostname specified becomes the default for future transfers.

       quit   Exit tftp.  An end of file also exits.

       verbose
	      Toggle verbose mode.

       trace  Toggle packet tracing.

       status Show current status.

       rexmt retransmission-timeout
	      Set the per-packet retransmission timeout, in seconds.

       timeout total-transmission-timeout
	      Set the total transmission timeout, in seconds.

       ascii  Shorthand for "mode ascii"

       binary Shorthand for "mode binary"

       ?  [ command-name ... ]
	      Print help information.

BUGS

       Because there is no user-login or validation within the TFTP protocol, the remote site will probably have some sort of file-access restric-
       tions in place.	The exact methods are specific to each site and therefore difficult to document here.

4.3 Berkeley Distribution						1Q								   TFTP(1)
All times are GMT -4. The time now is 01:39 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy