Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Copy files in Parallel
Top Forums UNIX for Dummies Questions & Answers Copy files in Parallel Post 302389411 by simonsimon on Sunday 24th of January 2010 09:05:31 PM
Old 01-24-2010
Scrutinizer,
You are simply great. Thanx for you suggestions. Ah I wish me as a DBA , to learn more about shell scripting and Unix admisnistration.

Ok Few Questions.

I traced your script to see what it does .And I have some Questions.
Directory /data/d1/ds has 4 files

Code:
SOURCEDIR="/data/d1/ds"
TARGETDIR="/data/d1/dt"
MAX_PARALLEL=2
nroffiles=$(ls $SOURCEDIR|wc -w)
setsize=$(( nroffiles/MAX_PARALLEL + 1 ))
ls -1 $SOURCEDIR/* | xargs -n $setsize | while read workset; do
  cp -p $workset $TARGETDIR &
done
wait

Code:
$ ct.sh
+ SOURCEDIR=/data/d1/ds
+ TARGETDIR=/data/d1/dt
+ MAX_PARALLEL=2
+ + ls /data/d1/ds
+ wc -w
nroffiles=4
+ setsize=3
+ read workset
+ xargs -n 3
+ ls -1 /data/d1/ds/f1.txt /data/d1/ds/f2.txt /data/d1/ds/f3.txt /data/d1/ds/f4.txt
+ read workset
+ cp -p /data/d1/ds/f1.txt /data/d1/ds/f2.txt /data/d1/ds/f3.txt /data/d1/dt
+ read workset
+ cp -p /data/d1/ds/f4.txt /data/d1/dt
+ wait

Can You kindly explain what does the following command do :-

Code:
setsize=$(( nroffiles/MAX_PARALLEL + 1 ))  :--  (( nroffiles/MAX_PARALLEL + 1 ))   gives only integer ? 

ls -1 $SOURCEDIR/* | xargs -n $setsize | while read workset; do
  cp -p $workset $TARGETDIR &
done
wait

ls -1 $SOURCEDIR/* :--- Gives file lsiting one below the other

xargs -n $setsize  :--- does what ? $setsize =3

How does it spawn two threads of cp command ?

What does the wait command do ? Does it wait for all parallel processes to finish.
So If I want to send email notification after all the files are copied a mailx command after wait command should do it ?
Last edited by pludi; 01-25-2010 at 02:10 AM.. Reason: code tags, please...
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

split process files in parallel and rejoin

Hi I need to split a huge file into multiple smaller files using split command. After that i need to process each file in the back ground with sql loader .Sql loader is a utlity to load CSV files into oracle . Check the status of each of these sqlloaders and then after sucessfull... (6 Replies)
Discussion started by: xiamin
6 Replies

2. UNIX for Advanced & Expert Users

implementation of copy command in parallel

hey i have to implement copy command in parallel in c language. i dont know how to create a new directory in destination. if anything u know related to this help me (1 Reply)
Discussion started by: rajsekhar28
1 Replies

3. Shell Programming and Scripting

parallel excution for 2000 files.

Hi, I have a function abc(). i am calling the function 9 times. it is working fine and completed the script execution 10 hours. input files: CUSTOMER_INFO_1111_12345.csv CUSTOMER_INFO_1222_12345.csv CUSTOMER_INFO_1322_12345.csv CUSTOMER_INFO_1333_12345.csv CUSTOMER_INFO_1151_12345.csv... (4 Replies)
Discussion started by: onesuri
4 Replies

4. Shell Programming and Scripting

scp or rsync multiple files in parallel from a remote host

Hi. I'm trying to speed up an rsync command by running it in parallel. There's no real option for this other than if the files are in multiple directories (which they're not). And even then there's no way of knowing if rsync has succeeded as the process is running in the background .. and... (4 Replies)
Discussion started by: Big_Jeffrey
4 Replies

5. Shell Programming and Scripting

Need to read two files in parallel

Following is the requirement In FileA I have the content as follows. 1,2,3 111,222,333 1000,2000,3000 In FileB I have the content as follows. 4,5,6 444,555,666 4000,5000,6000 I need to read FileA and FileB parallely and create the FileC as follows. 1,2,3,4,5,6... (1 Reply)
Discussion started by: kmanivan82
1 Replies

6. Shell Programming and Scripting

Comparing list of files in parallel

Hi everyone. I have a list of files like: file001 file002 file003 . . . . file385 file386 file387 There are more files than above, but I hope you understand what I'm trying to do here. Is there a way I can create a loop to compare: file001 with file385 file002 with file386 (9 Replies)
Discussion started by: craigsky
9 Replies

7. Shell Programming and Scripting

Parallel move keeping folder structure along with files in it

The below will move all the files in the directory dir to the destination using parallel and create a log, however will not keep them in the directory. I have tried mkdir -p but that does not seem to work or at least I can not seem to get it (as it deletes others files when I use it). What is the... (2 Replies)
Discussion started by: cmccabe
2 Replies

8. Shell Programming and Scripting

Copy file from different ports in parallel

Hello folks, Can you please help me to solve the below concern. I have a source server with 2 ports and have to copy the files from both the port to destination server simultaneously in my shell script. How can I achieve that? Source : x.x.x.x port -22 X.x.x.x port -2222 ... (7 Replies)
Discussion started by: sadique.manzar
7 Replies

9. Shell Programming and Scripting

Alignment tool to join text files in 2 directories to create a parallel corpus

I have two directories called English and Hindi. Each directory contains the same number of files with the only difference being that in the case of the English Directory the tag is .english and in the Hindi one the tag is .Hindi The file may contain either a single text or more than one text... (7 Replies)
Discussion started by: gimley
7 Replies

10. UNIX for Beginners Questions & Answers

How to paste multiple files in parallel?

Hi all, I am trying to paste thousands of files together into a matrix. Each file has only 1 column and all the files have the same number of rows (~27k rows). I tried paste * > output as well as some other for loops but the output only contains the columns from the 1st and last files. The... (8 Replies)
Discussion started by: notimenocall
8 Replies

LEARN ABOUT MOJAVE

extutils::command

ExtUtils::Command(3pm)					 Perl Programmers Reference Guide				    ExtUtils::Command(3pm)

NAME

       ExtUtils::Command - utilities to replace common UNIX commands in Makefiles etc.

SYNOPSIS

	 perl -MExtUtils::Command -e cat files... > destination
	 perl -MExtUtils::Command -e mv source... destination
	 perl -MExtUtils::Command -e cp source... destination
	 perl -MExtUtils::Command -e touch files...
	 perl -MExtUtils::Command -e rm_f files...
	 perl -MExtUtils::Command -e rm_rf directories...
	 perl -MExtUtils::Command -e mkpath directories...
	 perl -MExtUtils::Command -e eqtime source destination
	 perl -MExtUtils::Command -e test_f file
	 perl -MExtUtils::Command -e test_d directory
	 perl -MExtUtils::Command -e chmod mode files...
	 ...

DESCRIPTION

       The module is used to replace common UNIX commands.  In all cases the functions work from @ARGV rather than taking arguments.  This makes
       them easier to deal with in Makefiles.  Call them like this:

	 perl -MExtUtils::Command -e some_command some files to work on

       and NOT like this:

	 perl -MExtUtils::Command -e 'some_command qw(some files to work on)'

       For that use Shell::Command.

       Filenames with * and ? will be glob expanded.

   FUNCTIONS
       cat
	       cat file ...

	   Concatenates all files mentioned on command line to STDOUT.

       eqtime
	       eqtime source destination

	   Sets modified time of destination to that of source.

       rm_rf
	       rm_rf files or directories ...

	   Removes files and directories - recursively (even if readonly)

       rm_f
	       rm_f file ...

	   Removes files (even if readonly)

       touch
	       touch file ...

	   Makes files exist, with current timestamp

       mv
	       mv source_file destination_file
	       mv source_file source_file destination_dir

	   Moves source to destination.  Multiple sources are allowed if destination is an existing directory.

	   Returns true if all moves succeeded, false otherwise.

       cp
	       cp source_file destination_file
	       cp source_file source_file destination_dir

	   Copies sources to the destination.  Multiple sources are allowed if destination is an existing directory.

	   Returns true if all copies succeeded, false otherwise.

       chmod
	       chmod mode files ...

	   Sets UNIX like permissions 'mode' on all the files.	e.g. 0666

       mkpath
	       mkpath directory ...

	   Creates directories, including any parent directories.

       test_f
	       test_f file

	   Tests if a file exists.  Exits with 0 if it does, 1 if it does not (ie.  shell's idea of true and false).

       test_d
	       test_d directory

	   Tests if a directory exists.  Exits with 0 if it does, 1 if it does not (ie. shell's idea of true and false).

       dos2unix
	       dos2unix files or dirs ...

	   Converts DOS and OS/2 linefeeds to Unix style recursively.

SEE ALSO

       Shell::Command which is these same functions but take arguments normally.

AUTHOR

       Nick Ing-Simmons "ni-s@cpan.org"

       Maintained by Michael G Schwern "schwern@pobox.com" within the ExtUtils-MakeMaker package and, as a separate CPAN package, by Randy Kobes
       "r.kobes@uwinnipeg.ca".

perl v5.18.2							    2014-01-06						    ExtUtils::Command(3pm)
All times are GMT -4. The time now is 09:23 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy