Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Copy files in Parallel
Top Forums UNIX for Dummies Questions & Answers Copy files in Parallel Post 302389465 by Scrutinizer on Monday 25th of January 2010 03:55:57 AM
Old 01-25-2010
Hi,

setsize is the total number of files divided by the number of parallel processes. I added 1 so that you will have no more than the number op parallel process. In this case the workload is split in 3 and 1, but for larger numbers it works out more evenly.

wc -w tests the number of entries although wc -l is perhaps more appropriate in this case
xargs cuts up the set of files in more or less equal parts
each cp process copies the files in these equal parts to the target directory.

The & puts the copy processes in the background and the wait command waits for them to finish. So the are not multiple threads but multiple processes.

The script breaks with file names that contain spaces. I'll fix that when I have more time.

S.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

split process files in parallel and rejoin

Hi I need to split a huge file into multiple smaller files using split command. After that i need to process each file in the back ground with sql loader .Sql loader is a utlity to load CSV files into oracle . Check the status of each of these sqlloaders and then after sucessfull... (6 Replies)
Discussion started by: xiamin
6 Replies

2. UNIX for Advanced & Expert Users

implementation of copy command in parallel

hey i have to implement copy command in parallel in c language. i dont know how to create a new directory in destination. if anything u know related to this help me (1 Reply)
Discussion started by: rajsekhar28
1 Replies

3. Shell Programming and Scripting

parallel excution for 2000 files.

Hi, I have a function abc(). i am calling the function 9 times. it is working fine and completed the script execution 10 hours. input files: CUSTOMER_INFO_1111_12345.csv CUSTOMER_INFO_1222_12345.csv CUSTOMER_INFO_1322_12345.csv CUSTOMER_INFO_1333_12345.csv CUSTOMER_INFO_1151_12345.csv... (4 Replies)
Discussion started by: onesuri
4 Replies

4. Shell Programming and Scripting

scp or rsync multiple files in parallel from a remote host

Hi. I'm trying to speed up an rsync command by running it in parallel. There's no real option for this other than if the files are in multiple directories (which they're not). And even then there's no way of knowing if rsync has succeeded as the process is running in the background .. and... (4 Replies)
Discussion started by: Big_Jeffrey
4 Replies

5. Shell Programming and Scripting

Need to read two files in parallel

Following is the requirement In FileA I have the content as follows. 1,2,3 111,222,333 1000,2000,3000 In FileB I have the content as follows. 4,5,6 444,555,666 4000,5000,6000 I need to read FileA and FileB parallely and create the FileC as follows. 1,2,3,4,5,6... (1 Reply)
Discussion started by: kmanivan82
1 Replies

6. Shell Programming and Scripting

Comparing list of files in parallel

Hi everyone. I have a list of files like: file001 file002 file003 . . . . file385 file386 file387 There are more files than above, but I hope you understand what I'm trying to do here. Is there a way I can create a loop to compare: file001 with file385 file002 with file386 (9 Replies)
Discussion started by: craigsky
9 Replies

7. Shell Programming and Scripting

Parallel move keeping folder structure along with files in it

The below will move all the files in the directory dir to the destination using parallel and create a log, however will not keep them in the directory. I have tried mkdir -p but that does not seem to work or at least I can not seem to get it (as it deletes others files when I use it). What is the... (2 Replies)
Discussion started by: cmccabe
2 Replies

8. Shell Programming and Scripting

Copy file from different ports in parallel

Hello folks, Can you please help me to solve the below concern. I have a source server with 2 ports and have to copy the files from both the port to destination server simultaneously in my shell script. How can I achieve that? Source : x.x.x.x port -22 X.x.x.x port -2222 ... (7 Replies)
Discussion started by: sadique.manzar
7 Replies

9. Shell Programming and Scripting

Alignment tool to join text files in 2 directories to create a parallel corpus

I have two directories called English and Hindi. Each directory contains the same number of files with the only difference being that in the case of the English Directory the tag is .english and in the Hindi one the tag is .Hindi The file may contain either a single text or more than one text... (7 Replies)
Discussion started by: gimley
7 Replies

10. UNIX for Beginners Questions & Answers

How to paste multiple files in parallel?

Hi all, I am trying to paste thousands of files together into a matrix. Each file has only 1 column and all the files have the same number of rows (~27k rows). I tried paste * > output as well as some other for loops but the output only contains the columns from the 1st and last files. The... (8 Replies)
Discussion started by: notimenocall
8 Replies

LEARN ABOUT OSX

tdbbackup

TDBBACKUP(8)															      TDBBACKUP(8)

NAME

       tdbbackup - tool for backing up and for validating the integrity of samba .tdb files

SYNOPSIS

       tdbbackup [-ssuffix] [-v] [-h]

DESCRIPTION

       This tool is part of the samba(1) suite.

       tdbbackup is a tool that may be used to backup samba .tdb files. This tool may also be used to verify the integrity of the .tdb files prior
       to samba startup or during normal operation. If it finds file damage and it finds a prior backup the backup file will be restored.

OPTIONS

       -h
	  Get help information.

       -s suffix
	  The -s option allows the adminisistrator to specify a file backup extension. This way it is possible to keep a  history  of  tdb  backup
	  files by using a new suffix for each backup.

       -v
	  The -v will check the database for damages (currupt data) which if detected causes the backup to be restored.

COMMANDS

       GENERAL INFORMATION

       The  tdbbackup  utility	can safely be run at any time. It was designed so that it can be used at any time to validate the integrity of tdb
       files, even during Samba operation. Typical usage for the command will be:

       tdbbackup [-s suffix] *.tdb

       Before restarting samba the following command may be run to validate .tdb files:

       tdbbackup -v [-s suffix] *.tdb

       Samba .tdb files are stored in various locations, be sure to run backup all .tdb file on the system. Important files includes:

       o

	  secrets.tdb - usual location is in the /usr/local/samba/private directory, or on some systems in /etc/samba.

       o

	  passdb.tdb - usual location is in the /usr/local/samba/private directory, or on some systems in /etc/samba.

       o

	  *.tdb located in the /usr/local/samba/var directory or on some systems in the /var/cache or /var/lib/samba directories.

VERSION

       This man page is correct for version 3.0 of the Samba suite.

AUTHOR

       The original Samba software and related utilities were created by Andrew Tridgell. Samba is now developed by the  Samba	Team  as  an  Open
       Source project similar to the way the Linux kernel is developed.

       The tdbbackup man page was written by John H Terpstra.

																      TDBBACKUP(8)
All times are GMT -4. The time now is 04:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy