Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Copy files in Multiple Threads

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Copy files in Multiple Threads
# 1  
Old 02-18-2010
Question Copy files in Multiple Threads
Hello all,

I have a directory of files of varying sizes.

I want to copy all these files in n number of threads to another directory such that each

copy set is more or less the same size.

Example :

Say /mydirA

It has around say 23 files of various sizes.

Number of copy threads say = 3

total directory size of /mydirA = 25 GB

So each thread should copy files whose sizes add up to almost 25/3 ~ 8 GB

So need to gather files based on the size for each thread such that they add upto 8GB

Thread 1 --> 8GB ..could be 11 files which add up to 8 gb

Thread 2 -->8 Gb ... couldbe 5 files which add up to 8 gb

Thread 3 ---> 9GB ...could be 8 files which add up to 8 or 9 gb

Want roughly equal copy set threads. It is also possible that even though I select 3 threads of equal size because of lack of number of files not all 3 threads could satisy the 8gb copy set size. So atleast try to fulfill the copy set thread size as far as possible.

All files need to go from /mydirA to /mydirB in N threads bases on the size of each thread as
(Total size of directory)/N which could have different number of files in each thread based on size to add up to the individual copy thread size
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Copy data at specified location from multiple files

Hello everyone, Im super new to coding but increasingly in need of it at work. Im have task stacked because of this problems, that I cannot figure out how to solve looking on the internet after trying many many things that looked similar to me. I have multiple data files of the form (see below).... (2 Replies)
Discussion started by: Xfiles_fan
2 Replies

2. Shell Programming and Scripting

Copy files matching multiple conditions

Hello How do i copy files matching multiple conditions. Requirement is to search files starting with name abc* and def* and created on a particular date or date range given by the user and copy it to the destination folder. i tried with different commands. below one will give the list ,... (5 Replies)
Discussion started by: NarayanaPrakash
5 Replies

3. Shell Programming and Scripting

Copy multiple files from A to B through passwordless ssh

hi all, I need to write one script to copy multiple imp files like /etc/passwd /etc/group /etc/shadow /etc/printers.conf from system A, System B and system C to system Z and I need to execute this script on System Z. like if system is equal A copy 1 2 3 files to system Z into... (9 Replies)
Discussion started by: manalisharmabe
9 Replies

4. UNIX for Dummies Questions & Answers

copy multiple files

Hi, I am facing this problem, however i am not finding any solution. Kindly help I have the list of files to be search , i need to search for those files and copy the files to a folder. Really its urgent. MG_0281.JPG Tdfa_0077.JPG The%20SirehSet%20Geduing%20KpgGlam%20.jpg... (4 Replies)
Discussion started by: umapearl
4 Replies

5. UNIX for Dummies Questions & Answers

Copy multiple files with space to folder

Please help , I am in an urgent need, Please help nawk '{for(i=1;i<=NF;i++){printf("%s\n",$i)}}' filename | sed 's/.*com//' | nawk '/pdf/ {printf("F:%s\n",$0)}' | while read line; do mv $line /images/; done the above script works for without spaces but,My path is also having some space... (3 Replies)
Discussion started by: umapearl
3 Replies

6. UNIX for Dummies Questions & Answers

Zip multiple files and copy to help

Hi All, I have a set of large files ~ 500_900Mb I have generated and I'd like to quickly zip and copy them to a new folder elsewhere ... Can anyone suggest a quicky ?? Cheers :) (3 Replies)
Discussion started by: pawannoel
3 Replies

7. Shell Programming and Scripting

ksh to copy multiple files

Guys, I've got a list of about 200 files I need to copy from /tmp to /data. I can't use wildcards because the filenames are all very different. What I want to do is cut and paste them into a file and read that as the input to a copy command (line by line). I tried using find and -exec... (4 Replies)
Discussion started by: Grueben
4 Replies

8. UNIX for Dummies Questions & Answers

Copy multiple files

Hi i have 1000 files is a directory, which are serially numbered (file1,file2,file3...). I would like to copy every 200 files to different directories. many thanks in advance. (6 Replies)
Discussion started by: saint2006
6 Replies

9. UNIX for Advanced & Expert Users

Very Challenging :Copy files in Multiple Threads

Hello all, I asked this in the basic Unix forum got no answer since one week. So I believe this is an advanced level question hence posting it here. Any suggestions welcome. I have a directory of files of varying sizes. I want to copy all these files in n number of threads to... (2 Replies)
Discussion started by: samoo
2 Replies

10. UNIX for Dummies Questions & Answers

copy multiple files in different directories

I have a report file that is generated every day by a scheduled process. Each day the file is written to a directory named .../blah_blah/Y07/MM-DD-YY/reportmmddyy.tab I want to copy all of this reports to a separate directory without having to do it one by one. However, if I try cp... (3 Replies)
Discussion started by: ken2834
3 Replies
Login or Register to Ask a Question

LEARN ABOUT HPUX

efi_cp

efi_cp(1M)																efi_cp(1M)

NAME

       efi_cp - copy to or from EFI file

SYNOPSIS

       devicefile] file1 file2

       devicefile] file1 [file2]...  dest-directory

       devicefile] file_or_dir1 [file_or_dir2]...  dest-directory

DESCRIPTION

       copies files between HP-UX and EFI file systems.

       The EFI file system is based on the FAT file system and used by the Itanium-based system BIOS to locate an HP-UX bootloader.  See efi(4).

       Without	the  option,  copies  from an HP-UX file system to the EFI volume specified by devicefile; in this case, the destination should be
       relative to the root of the EFI volume specified by devicefile.	Use the option to copy from an EFI file system to an  HP-UX  file  system;
       with the option, only a single regular file may be copied at a time.

       With two arguments:

       o If file2 does not exist, creates it and copies the contents of file1 into file2.

       o If file2 exists and is a regular file, deletes the the contents of file2 and copies the contents of file1 into file2.

       o If file2 exists and is a directory, creates a copy of file1 in that directory.

       With more than two arguments, the final argument must be an existing directory.	If any of the other arguments is a directory, you must use
       the option, which specifies that the source directory and the subtree rooted there are to be copied to the destination directory.

   Options
       Options can be used singly or combined in any order before the file names.  The space between option and argument is optional.

       devicefile     is the device special file of the EFI file system to be copied.  Without the option, uses the device file specified  in  the
		      environment variable if it is defined.  The option overrides the environment variable.

       Recursively copy the subtree rooted at each source directory to
		      the destination directory.

       Copy	      file1 from an EFI volume to file2 in HP-UX file system.  file1 should be relative to the root of the EFI volume specified by
		      devicefile, not to an HP-UX file system.	With the option, only a single regular file may be copied at a time.

RETURN VALUE

       returns exit code 0 if the copy is successful.  Otherwise it prints a diagnostic message and returns nonzero.

	   0	  Successful completion.
	  >0	  Error condition occurred.

EXAMPLES

       All the following examples copy to or from the EFI file system on device

       Copy into directory in the EFI file system on the specified device:

       Copy all files in current directory into directory in the EFI file system on the specified device

       Copy into directory in the EFI file system on the specified device and rename it

       Copy into the EFI boot directory of the EFI file system on the specified device:

       Copy into of the EFI file system on the device specified in the environment variable and rename it to

       Copy from the EFI file system on the specified device into the current HP-UX directory:

       Using a persistent device special file (dsf) copy from the EFI file system on the specified device into the current  HP-UX  directory  (see
       intro(7)):

AUTHOR

       was developed by HP.

SEE ALSO

       efi_fsinit(1M), efi_ls(1M), efi_mkdir(1M), efi_rm(1M), efi_rmdir(1M), efi(4), intro(7).

							   Itanium(R)-based Systems Only						efi_cp(1M)