Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Copy files in Parallel
Top Forums UNIX for Dummies Questions & Answers Copy files in Parallel Post 302389393 by Scrutinizer on Sunday 24th of January 2010 05:26:43 PM
Old 01-24-2010
simonsimon, I am not talking about 500 threads. Did you read my answer about the pros and cons?

SAN means the disks are serially attached and probably there are multiple underlying physical disks so there are likely performance benefits for the backup process itself, however you should do a single thread restore..

For the parallell copy process you could probably use this as a basis (you would still have to create checks for directories and such, this is just the working principle):
Code:
SOURCEDIR="$1"
TARGETDIR="$2"
MAX_PARALLEL=4
nroffiles=$(ls $SOURCEDIR|wc -w)
setsize=$(( nroffiles/MAX_PARALLEL + 1 ))
ls -1 $SOURCEDIR/* | xargs -n $setsize | while read workset; do
  cp -p $workset $TARGETDIR &
done
wait

Last edited by Scrutinizer; 01-24-2010 at 06:34 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

split process files in parallel and rejoin

Hi I need to split a huge file into multiple smaller files using split command. After that i need to process each file in the back ground with sql loader .Sql loader is a utlity to load CSV files into oracle . Check the status of each of these sqlloaders and then after sucessfull... (6 Replies)
Discussion started by: xiamin
6 Replies

2. UNIX for Advanced & Expert Users

implementation of copy command in parallel

hey i have to implement copy command in parallel in c language. i dont know how to create a new directory in destination. if anything u know related to this help me (1 Reply)
Discussion started by: rajsekhar28
1 Replies

3. Shell Programming and Scripting

parallel excution for 2000 files.

Hi, I have a function abc(). i am calling the function 9 times. it is working fine and completed the script execution 10 hours. input files: CUSTOMER_INFO_1111_12345.csv CUSTOMER_INFO_1222_12345.csv CUSTOMER_INFO_1322_12345.csv CUSTOMER_INFO_1333_12345.csv CUSTOMER_INFO_1151_12345.csv... (4 Replies)
Discussion started by: onesuri
4 Replies

4. Shell Programming and Scripting

scp or rsync multiple files in parallel from a remote host

Hi. I'm trying to speed up an rsync command by running it in parallel. There's no real option for this other than if the files are in multiple directories (which they're not). And even then there's no way of knowing if rsync has succeeded as the process is running in the background .. and... (4 Replies)
Discussion started by: Big_Jeffrey
4 Replies

5. Shell Programming and Scripting

Need to read two files in parallel

Following is the requirement In FileA I have the content as follows. 1,2,3 111,222,333 1000,2000,3000 In FileB I have the content as follows. 4,5,6 444,555,666 4000,5000,6000 I need to read FileA and FileB parallely and create the FileC as follows. 1,2,3,4,5,6... (1 Reply)
Discussion started by: kmanivan82
1 Replies

6. Shell Programming and Scripting

Comparing list of files in parallel

Hi everyone. I have a list of files like: file001 file002 file003 . . . . file385 file386 file387 There are more files than above, but I hope you understand what I'm trying to do here. Is there a way I can create a loop to compare: file001 with file385 file002 with file386 (9 Replies)
Discussion started by: craigsky
9 Replies

7. Shell Programming and Scripting

Parallel move keeping folder structure along with files in it

The below will move all the files in the directory dir to the destination using parallel and create a log, however will not keep them in the directory. I have tried mkdir -p but that does not seem to work or at least I can not seem to get it (as it deletes others files when I use it). What is the... (2 Replies)
Discussion started by: cmccabe
2 Replies

8. Shell Programming and Scripting

Copy file from different ports in parallel

Hello folks, Can you please help me to solve the below concern. I have a source server with 2 ports and have to copy the files from both the port to destination server simultaneously in my shell script. How can I achieve that? Source : x.x.x.x port -22 X.x.x.x port -2222 ... (7 Replies)
Discussion started by: sadique.manzar
7 Replies

9. Shell Programming and Scripting

Alignment tool to join text files in 2 directories to create a parallel corpus

I have two directories called English and Hindi. Each directory contains the same number of files with the only difference being that in the case of the English Directory the tag is .english and in the Hindi one the tag is .Hindi The file may contain either a single text or more than one text... (7 Replies)
Discussion started by: gimley
7 Replies

10. UNIX for Beginners Questions & Answers

How to paste multiple files in parallel?

Hi all, I am trying to paste thousands of files together into a matrix. Each file has only 1 column and all the files have the same number of rows (~27k rows). I tried paste * > output as well as some other for loops but the output only contains the columns from the 1st and last files. The... (8 Replies)
Discussion started by: notimenocall
8 Replies

LEARN ABOUT DEBIAN

debaux-build

DEBAUX-BUILD(1p)					User Contributed Perl Documentation					  DEBAUX-BUILD(1p)

NAME

       debaux-build - Debian package building script

SYNOPSIS

       debaux-build [OPTIONS] DEBDIR [BUILDDIR SOURCEDIR]

       debaux-build --cpan Net::Google

VERSION

       0.1.10

DESCRIPTION

       debaux-build is a helper script for building Debian packages.  Besides options it takes up to three parameters: DEBDIR, BUILDDIR,
       SOURCEDIR. DEBDIR is a required parameter.

       If DEBDIR doesn't exist, debaux-build tries to download the package with apt-get source.

       debaux-build applies all patches found in the DEBDIR subdirectory patches.

       If no source tarball exist and the DEBDIR subdirectory sources exist, all files in this directory get packed and used as source tarball.

OPTIONS

       "--apt"
	   Forces apt-get source mode.

       "--archive"
	   Specifies file name of source archive.

       "--build-options"
	   Options passed to "dpkg-buildpackage".

       "--cpan"
	   Download Perl module from CPAN and create necessary Debian packaging files. You need to have the dh-make Debian package installed in
	   order to utilize this option.

       "--empty"
	   Empty package (no source code archive). Intended to use for task packages.

       "--epoch"
	   Specifies epoch for CPAN builds.

	   "--extra-sources"
	       Specifies a directory below DEBDIR which is copied to the build directory.

	   "-i, --install, --install=PACKAGE"
	       Install build package.

	   "-l, --lintian"
	       Run lintian on the source file and the generated packages.

	   "--make-archive"
	       Intended to use for native Debian packages. Specify the commands to build the archive here.

	   "--release"
	       The generated packages are targeted for official Debian packages.

	   "--rpm, --alien"
	       Convert generated packages into RPM packages by using "alien".

	   "-s, --sign"
	       Sign source package and changelog.

	   "--skip-patches"
	       Don't apply patches from the patches directory and sources.diff file.

	       "--source-tree"
		   Instructs "debaux-build" to use the specified directory instead of an archive file.

	       "--stop-before-build"
		   Stop script before calling "dpkg-buildpackage".

CONFIGURATION FILE

       "debaux-build" looks for the configuration file ".debauxrc" in the home directory and in the current directory and recognizes the following
       directives:

       "debauxdir" PACKAGE
	   Used as "DEBDIR" for package if it is an existing directory.  Considered before the <debiandir> directive.

       "debiandir"
	   If the directory "DEBDIR" on the commandline doesn't exist, "debaux-build" appends "DEBDIR" to the specified directory and tries again.

       "sourcedir"
	   Directory with the source archives. Can be overridden by the commandline parameter "SOURCEDIR".

       "source-tree" PACKAGE
	   Directory with the source code for PACKAGE.

SEE ALSO

       dpkg(8), dpkg-source(1), apt-get(8)

AUTHOR

       Stefan Hornburg (Racke) <racke@linuxia.de>.

LICENSE

       debaux-build comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute and modify it under the terms of
       the GNU General Public License.

COPYRIGHT

       Copyright 2000,2001,2002,2003,2004,2005,2006,2007 Stefan Hornburg (Racke) <racke@linuxia.de>.

perl v5.8.8							    2007-07-15							  DEBAUX-BUILD(1p)
All times are GMT -4. The time now is 01:21 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy