Very Challenging :Copy files in Multiple Threads Post: 302399490

Sponsored Content

Top Forums UNIX for Advanced & Expert Users Very Challenging :Copy files in Multiple Threads Post 302399490 by samoo on Sunday 28th of February 2010 11:43:56 AM

02-28-2010

Registered User

Very Challenging :Copy files in Multiple Threads

Hello all,

I asked this in the basic Unix forum got no answer since one week.

So I believe this is an advanced level question hence posting it here.

Any suggestions welcome.

I have a directory of files of varying sizes.

I want to copy all these files in n number of threads to another directory such that each

copy set is more or less the same size.

Example :

Say /mydirA

It has around say 23 files of various sizes.

Number of copy threads say = 3

total directory size of /mydirA = 25 GB

So each thread should copy files whose sizes add up to almost 25/3 ~ 8 GB

So need to gather files based on the size for each thread such that they add upto 8GB

Thread 1 --> 8GB ..could be 11 files which add up to 8 gb

Thread 2 -->8 Gb ... couldbe 5 files which add up to 8 gb

Thread 3 ---> 9GB ...could be 8 files which add up to 8 or 9 gb

Want roughly equal copy set threads. It is also possible that even though I select 3 threads of equal size because of lack of number of files not all 3 threads could satisy the 8gb copy set size. So atleast try to fulfill the copy set thread size as far as possible.

All files need to go from /mydirA to /mydirB in N threads bases on the size of each thread as
(Total size of directory)/N which could have different number of files in each thread based on size to add up to the individual copy thread size

samoo

View Public Profile for samoo

Find all posts by samoo

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

copy multiple files in different directories

I have a report file that is generated every day by a scheduled process. Each day the file is written to a directory named .../blah_blah/Y07/MM-DD-YY/reportmmddyy.tab I want to copy all of this reports to a separate directory without having to do it one by one. However, if I try cp...

2. UNIX for Dummies Questions & Answers

Copy files in Multiple Threads

Hello all, I have a directory of files of varying sizes. I want to copy all these files in n number of threads to another directory such that each copy set is more or less the same size. Example : Say /mydirA It has around say 23 files of various sizes. Number of copy...

3. UNIX for Dummies Questions & Answers

Copy multiple files

Hi i have 1000 files is a directory, which are serially numbered (file1,file2,file3...). I would like to copy every 200 files to different directories. many thanks in advance.

4. Shell Programming and Scripting

ksh to copy multiple files

Guys, I've got a list of about 200 files I need to copy from /tmp to /data. I can't use wildcards because the filenames are all very different. What I want to do is cut and paste them into a file and read that as the input to a copy command (line by line). I tried using find and -exec...

5. UNIX for Dummies Questions & Answers

Zip multiple files and copy to help

Hi All, I have a set of large files ~ 500_900Mb I have generated and I'd like to quickly zip and copy them to a new folder elsewhere ... Can anyone suggest a quicky ?? Cheers :)

6. UNIX for Dummies Questions & Answers

Copy multiple files with space to folder

Please help , I am in an urgent need, Please help nawk '{for(i=1;i<=NF;i++){printf("%s\n",$i)}}' filename | sed 's/.*com//' | nawk '/pdf/ {printf("F:%s\n",$0)}' | while read line; do mv $line /images/; done the above script works for without spaces but,My path is also having some space...

7. UNIX for Dummies Questions & Answers

copy multiple files

Hi, I am facing this problem, however i am not finding any solution. Kindly help I have the list of files to be search , i need to search for those files and copy the files to a folder. Really its urgent. MG_0281.JPG Tdfa_0077.JPG The%20SirehSet%20Geduing%20KpgGlam%20.jpg...

8. Shell Programming and Scripting

Copy multiple files from A to B through passwordless ssh

hi all, I need to write one script to copy multiple imp files like /etc/passwd /etc/group /etc/shadow /etc/printers.conf from system A, System B and system C to system Z and I need to execute this script on System Z. like if system is equal A copy 1 2 3 files to system Z into...

9. Shell Programming and Scripting

Copy files matching multiple conditions

Hello How do i copy files matching multiple conditions. Requirement is to search files starting with name abc* and def* and created on a particular date or date range given by the user and copy it to the destination folder. i tried with different commands. below one will give the list ,...

10. UNIX for Beginners Questions & Answers

Copy data at specified location from multiple files

Hello everyone, Im super new to coding but increasingly in need of it at work. Im have task stacked because of this problems, that I cannot figure out how to solve looking on the internet after trying many many things that looked similar to me. I have multiple data files of the form (see below)....

LEARN ABOUT DEBIAN

tiobench

tiobench(1)															       tiobench(1)

NAME

       tiobench - Threaded I/O bench

SYNOPSIS

       tiobench  [--help]  [--nofrag]  [--size	SizeInMB  [--size  ...]]   [--numruns  NumberOfRuns  [--numruns ...]]  [--dir TestDir [--dir ...]]
       [--block BlkSizeInBytes [--block ...]]  [--random NumberRandOpsPerThread [--random ...]]  [--threads NumberOfThreads [--threads ...]]

DESCRIPTION

       tiobench is a perl wrapper to tiotest calling it multiple times with varying sets of parameters as instructed.

   OPTIONS
       --help  Display a brief help and exit.

       --nofrag
	       Instructs tiobench to pass -W to tiotest so it waits for previous threads to finish before starting a new one in the writing phase.
	       For more info see the -W option in the tiotest(1) manpage.

       --size SizeInMB
	       The  total  size  in MBytes of the files may use together. If this option is not given, tiobench tries to be smart and figure out a
	       size making sense.

       --numruns NumberOfRuns
	       This number specifies over how many runs each test should be averaged.  Defaults to 1.

       --dir TestDir
	       The directory in which to test. Defaults to ., the current directory.

       --block BlkSizeInBytes
	       The blocksize in Bytes to use. Defaults to 4096.

       --random NumberRandOpsPerThread
	       Random I/O operations per thread. Defaults to 1000.

       --threads NumberOfThreads
	       The number of concurrent test threads. Defaults to 4.

       The options --size, --numruns, --dir, --block, --random, and --threads may be given multiple times to cover multiple cases,  for  instance:
       tiobench --block 4096 --block 8192 will first run through with a 4KB block size and then again with a 8KB block size.

       To  get usefull results the used file sizes should be a lot larger than the physical amount of memory you have. A good idea is to boot with
       16 Megs of RAM (Try passing the "mem=16M" option to the kernel to limit Linux to using a very small amount of memory) and into Single  User
       mode only.

SEE ALSO

       tiotest(1), bonnie(1), hdparm(8)

AUTHOR

       tiobench was written by James Manning <jmm@computer.org>.

       This  manual  page  was	written by Peter Palfrader <weasel@debian.org>, for the Debian GNU/Linux system (but may be used by others).

Mar-2001															       tiobench(1)