Sponsored Content
Full Discussion: Copy files in Parallel
Top Forums UNIX for Dummies Questions & Answers Copy files in Parallel Post 302389146 by simonsimon on Friday 22nd of January 2010 08:48:29 PM
Old 01-22-2010
Question Copy files in Parallel

Is there any out of box Command in Unix
or is it possible through shell scripting to
copy all the files in a directory in Parallel.

Example. I am doing a COLD database backup. I have the data/dbx directory which has around 1000 data files.

I was thinking is there a way to spawn the copy process so that I would have say n number or say 4 threads which would copy 500 files each.

I am trying to cp the data directory to my backup folder.

right now I am doing

cp -Rf /data4/dbx /backup/dbx

dbx directory has around 1000 data files

If I use cp command , it only does that in sequence.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

split process files in parallel and rejoin

Hi I need to split a huge file into multiple smaller files using split command. After that i need to process each file in the back ground with sql loader .Sql loader is a utlity to load CSV files into oracle . Check the status of each of these sqlloaders and then after sucessfull... (6 Replies)
Discussion started by: xiamin
6 Replies

2. UNIX for Advanced & Expert Users

implementation of copy command in parallel

hey i have to implement copy command in parallel in c language. i dont know how to create a new directory in destination. if anything u know related to this help me (1 Reply)
Discussion started by: rajsekhar28
1 Replies

3. Shell Programming and Scripting

parallel excution for 2000 files.

Hi, I have a function abc(). i am calling the function 9 times. it is working fine and completed the script execution 10 hours. input files: CUSTOMER_INFO_1111_12345.csv CUSTOMER_INFO_1222_12345.csv CUSTOMER_INFO_1322_12345.csv CUSTOMER_INFO_1333_12345.csv CUSTOMER_INFO_1151_12345.csv... (4 Replies)
Discussion started by: onesuri
4 Replies

4. Shell Programming and Scripting

scp or rsync multiple files in parallel from a remote host

Hi. I'm trying to speed up an rsync command by running it in parallel. There's no real option for this other than if the files are in multiple directories (which they're not). And even then there's no way of knowing if rsync has succeeded as the process is running in the background .. and... (4 Replies)
Discussion started by: Big_Jeffrey
4 Replies

5. Shell Programming and Scripting

Need to read two files in parallel

Following is the requirement In FileA I have the content as follows. 1,2,3 111,222,333 1000,2000,3000 In FileB I have the content as follows. 4,5,6 444,555,666 4000,5000,6000 I need to read FileA and FileB parallely and create the FileC as follows. 1,2,3,4,5,6... (1 Reply)
Discussion started by: kmanivan82
1 Replies

6. Shell Programming and Scripting

Comparing list of files in parallel

Hi everyone. I have a list of files like: file001 file002 file003 . . . . file385 file386 file387 There are more files than above, but I hope you understand what I'm trying to do here. Is there a way I can create a loop to compare: file001 with file385 file002 with file386 (9 Replies)
Discussion started by: craigsky
9 Replies

7. Shell Programming and Scripting

Parallel move keeping folder structure along with files in it

The below will move all the files in the directory dir to the destination using parallel and create a log, however will not keep them in the directory. I have tried mkdir -p but that does not seem to work or at least I can not seem to get it (as it deletes others files when I use it). What is the... (2 Replies)
Discussion started by: cmccabe
2 Replies

8. Shell Programming and Scripting

Copy file from different ports in parallel

Hello folks, Can you please help me to solve the below concern. I have a source server with 2 ports and have to copy the files from both the port to destination server simultaneously in my shell script. How can I achieve that? Source : x.x.x.x port -22 X.x.x.x port -2222 ... (7 Replies)
Discussion started by: sadique.manzar
7 Replies

9. Shell Programming and Scripting

Alignment tool to join text files in 2 directories to create a parallel corpus

I have two directories called English and Hindi. Each directory contains the same number of files with the only difference being that in the case of the English Directory the tag is .english and in the Hindi one the tag is .Hindi The file may contain either a single text or more than one text... (7 Replies)
Discussion started by: gimley
7 Replies

10. UNIX for Beginners Questions & Answers

How to paste multiple files in parallel?

Hi all, I am trying to paste thousands of files together into a matrix. Each file has only 1 column and all the files have the same number of rows (~27k rows). I tried paste * > output as well as some other for loops but the output only contains the columns from the 1st and last files. The... (8 Replies)
Discussion started by: notimenocall
8 Replies
nisbackup(1M)						  System Administration Commands					     nisbackup(1M)

NAME
nisbackup - backup NIS+ directories SYNOPSIS
nisbackup [-v] backup-dir directory... nisbackup [-v] -a backup-dir DESCRIPTION
nisbackup backs up a NIS+ directory object on a NIS+ master server. Updates to the NIS+ database will be temporarily disabled while nis- backup is running. The backup-dir is a UNIX directory that must exist prior to running nisbackup. The nisbackup command can be used to backup an individual NIS+ directory object or all ( -a) of the NIS+ directory objects served by a master server. The NIS+ directory objects being backed up will be placed into subdirectories under the backup-dir directory. These subdirectories are named according to the NIS+ directory object they contain. nisbackup operates on individual NIS+ directory objects (for example, org_dir.wiz.com). This allows an administrator to selectively backup specific directories. The rpc.nisd(1M) process must be running on the master server with a stable NIS+ database for nisbackup to complete. nisbackup will not attempt to correct any corruption in the NIS+ database, so it is important that backups be done regularly as part of the NIS+ administra- tion. The first synopsis is used to backup a single NIS+ directory object or a list of NIS+ directory objects. The objects can be partially qual- ified or fully qualified. The machine on which the command is executing must be the master for the NIS+ directory objects specified. The second synopsis will backup all of the NIS+ directory objects that are served by this master. The -a option is the recommended method of backing up a master server, since it will backup all NIS+ directory objects that are served by this master. If this server is a master server for more than one domain, the backup will include NIS+ directories that belong to all of the domains served. Individual NIS+ direc- tory objects can be selected for restoring from a backup-dir created with the -a option. See nisrestore(1M). The -a option only includes directory objects for which this server is the master. It is possible, but not recommended, to configure a master server as a replica for other domains. The objects belonging to those replicated domains will not be backed up with the -a option. The backup of replicated objects must be run on the master server for those objects. Do not use the same backup-dir to backup different master servers. Each master server must have its own backup-dir. nisbackup will set the rpc.nisd(1M) to read only mode, which will disable updates to the NIS+ database. This is neccessary to ensure the consistency of the backup. For this reason, nisbackup should not be run while large numbers of updates are being applied to the NIS+ data- base. Update utilities such as nisaddent(1M) should not be run simultaneously with nisbackup. OPTIONS
-a Creates a backup of all NIS+ directory objects for which this server is a master. -v Verbose option. Additional output will be produced and sent to syslog(3C) upon execution of the command (see syslog.conf(4)). OPERANDS
backup-dir The directory into which the subdirectories containing the backed up objects are placed. This must be created prior to running nisbackup. directory The NIS+ directory object(s) being backed up. EXAMPLES
Example 1: Backup of the org_dir NIS+ directory object of the domain foo.com on a master server to a directory named /backup To backup the org_dir NIS+ directory object of the domain foo.com on a master server to a directory named /backup: master_server# nisbackup /backup org_dir.foo.com. Example 2: Backup of the entire NIS+ domain foo.com to a directory named /backup To backup the entire NIS+ domain foo.com to a directory named /backup: master_server# nisbackup /backup foo.com. org_dir.foo.com. groups_dir.foo.com. ctx_dir.foo.com. Example 3: Backup of an entire NIS+ database to a backup directory named /backup To backup an entire NIS+ database to a backup directory named /backup: master_server# nisbackup -a /backup EXIT STATUS
0 Successful completion. 1 An error occurred. FILES
/backup-dir/backup_list This ascii file contains a list of all the objects contained in this backup-dir directory. /backup-dir/directory-object A subdirectory that is created in the backup-dir that contains the NIS+ directory-object backup. /backup-dir/directory-object/data A subdirectory that contains the data files that are part of the NIS+ directory-object backup. /backup-dir/directory-object/last.upd This data file contains timestamp information about the directory-object. /backup-dir/directory-object/data.dict A NIS+ data dictionary for all of the objects contained in the NIS+ directory-object backup. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWnisu | +-----------------------------+-----------------------------+ SEE ALSO
nis+(1), nisdefaults(1), nisrm(1), nisrestore(1M), rpc.nisd(1M), syslog(3C), nisfiles(4), syslog.conf(4), attributes(5) NOTES
NIS+ might not be supported in future releases of the SolarisTM Operating Environment. Tools to aid the migration from NIS+ to LDAP are available in the Solaris 9 operating environment. For more information, visit http://www.sun.com/directory/nisplus/transition.html. SunOS 5.10 12 Dec 2001 nisbackup(1M)
All times are GMT -4. The time now is 11:07 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy