Sponsored Content
Full Discussion: Copy files in Parallel
Top Forums UNIX for Dummies Questions & Answers Copy files in Parallel Post 302389418 by matrixmadhan on Sunday 24th of January 2010 10:55:12 PM
Old 01-24-2010
Please use
Code:
CODE TAGS

as it makes easier to read the code.

$nroffiles - this number represents the number of files that would be there in each of the set that is to be executed in parallel.

Yes, after wait command - mail | mailx | sendmail to be used.

---------- Post updated at 09:25 AM ---------- Previous update was at 09:20 AM ----------

Am not clear with one thing, if parallelization needs to be achieved with count set to 'n'
Then at any point of time only 'n' cp process should be running and not the other way.
I think in this case, if total number of files is 'm' and parellization count 'k', then m / k process would be running in parallel which could be way too large than 'n' discussed above.
Code:
ls $srcdir | while read file
  cnt=1
  while [ $cnt -le 3 ]
  do
    # do some copy here as background jobs
    cnt=$(($cnt + 1))
  done
  wait
done

Am not sure, why
Code:
wc -w

is needed here.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

split process files in parallel and rejoin

Hi I need to split a huge file into multiple smaller files using split command. After that i need to process each file in the back ground with sql loader .Sql loader is a utlity to load CSV files into oracle . Check the status of each of these sqlloaders and then after sucessfull... (6 Replies)
Discussion started by: xiamin
6 Replies

2. UNIX for Advanced & Expert Users

implementation of copy command in parallel

hey i have to implement copy command in parallel in c language. i dont know how to create a new directory in destination. if anything u know related to this help me (1 Reply)
Discussion started by: rajsekhar28
1 Replies

3. Shell Programming and Scripting

parallel excution for 2000 files.

Hi, I have a function abc(). i am calling the function 9 times. it is working fine and completed the script execution 10 hours. input files: CUSTOMER_INFO_1111_12345.csv CUSTOMER_INFO_1222_12345.csv CUSTOMER_INFO_1322_12345.csv CUSTOMER_INFO_1333_12345.csv CUSTOMER_INFO_1151_12345.csv... (4 Replies)
Discussion started by: onesuri
4 Replies

4. Shell Programming and Scripting

scp or rsync multiple files in parallel from a remote host

Hi. I'm trying to speed up an rsync command by running it in parallel. There's no real option for this other than if the files are in multiple directories (which they're not). And even then there's no way of knowing if rsync has succeeded as the process is running in the background .. and... (4 Replies)
Discussion started by: Big_Jeffrey
4 Replies

5. Shell Programming and Scripting

Need to read two files in parallel

Following is the requirement In FileA I have the content as follows. 1,2,3 111,222,333 1000,2000,3000 In FileB I have the content as follows. 4,5,6 444,555,666 4000,5000,6000 I need to read FileA and FileB parallely and create the FileC as follows. 1,2,3,4,5,6... (1 Reply)
Discussion started by: kmanivan82
1 Replies

6. Shell Programming and Scripting

Comparing list of files in parallel

Hi everyone. I have a list of files like: file001 file002 file003 . . . . file385 file386 file387 There are more files than above, but I hope you understand what I'm trying to do here. Is there a way I can create a loop to compare: file001 with file385 file002 with file386 (9 Replies)
Discussion started by: craigsky
9 Replies

7. Shell Programming and Scripting

Parallel move keeping folder structure along with files in it

The below will move all the files in the directory dir to the destination using parallel and create a log, however will not keep them in the directory. I have tried mkdir -p but that does not seem to work or at least I can not seem to get it (as it deletes others files when I use it). What is the... (2 Replies)
Discussion started by: cmccabe
2 Replies

8. Shell Programming and Scripting

Copy file from different ports in parallel

Hello folks, Can you please help me to solve the below concern. I have a source server with 2 ports and have to copy the files from both the port to destination server simultaneously in my shell script. How can I achieve that? Source : x.x.x.x port -22 X.x.x.x port -2222 ... (7 Replies)
Discussion started by: sadique.manzar
7 Replies

9. Shell Programming and Scripting

Alignment tool to join text files in 2 directories to create a parallel corpus

I have two directories called English and Hindi. Each directory contains the same number of files with the only difference being that in the case of the English Directory the tag is .english and in the Hindi one the tag is .Hindi The file may contain either a single text or more than one text... (7 Replies)
Discussion started by: gimley
7 Replies

10. UNIX for Beginners Questions & Answers

How to paste multiple files in parallel?

Hi all, I am trying to paste thousands of files together into a matrix. Each file has only 1 column and all the files have the same number of rows (~27k rows). I tried paste * > output as well as some other for loops but the output only contains the columns from the 1st and last files. The... (8 Replies)
Discussion started by: notimenocall
8 Replies
UKOPP(1)							  Ukopp man page							  UKOPP(1)

NAME
Ukopp - copy files to USB stick or other disk or disk-like device SYNOPSIS
ukopp [ -nogui ] [ -job | -run ] jobfile DESCRIPTION
Ukopp is a graphical menu-driven program operating in its own window. Ukopp copies the files and directories specified in the job file to disk or disk-like media (e.g. USB stick). OVERVIEW
Ukopp works incrementally: only files that are not already identical on the target media are copied (new and modified files or files newly added to the backup job). Files/directories to include or exclude can be selected from the file system hierarchy, using a GUI navigator. Specifications are saved in a job file which can be re-edited and/or executed. Ukopp can be run in batch mode using the -nogui option. Ukopp can optionally keep old file versions instead of replacing them with updated versions. The retention time and/or the number of old versions to keep can be specified for each selected file or directory. File owners and permissions are retained even if the copy media has a Microsoft file system. Summary of functionality: - Copy files incrementally with optional file version retention. - Three media verification modes: full, incremental, compare. - Report disk:backup differences in detail or summary form. - Select and restore files from backup media or use drag and drop. OPTIONS
Command line options: [ -job ] jobfile open job file for editing [ -nogui ] -run jobfile execute a job file SEE ALSO
The online user manual is available using the menu Help > contents. This manual explains Ukopp operation in great detail. AUTHORS
Written by Michael Cornelison <kornelix2@googlemail.com> Linux 2010-10-01 UKOPP(1)
All times are GMT -4. The time now is 08:00 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy