Shell Programming and Scripting

Hello All,
This is my first post to unix.com.
I have requirement to copy multiple *.dmp files from one server to other server.
I have that other server NFS filesystem mapped to source server.
Dump[*.dmp] files are so huge almost 20TB. There are around 15-20 files each of 1.1TB.
I want to copy these files from one server to other server.
But I do not want to start copying all files at once.
Usually we do it manually copying 3 files at a time to reduce network bandwidth consumption.
Is there any way to write script to copy 3 file at one time and after 3 are done then new 3 files to be picked for file copy.
I have linux operating system.

Thanks
Amit Bansode

Welcome to the forum.

Not sure I grasped your requirement (or problem) in its entirety. If you

Code:

cp *.dmp target

, there will always be just one single file on its way. And, increasing the count of copying processes will not necessarily improve the overall time consumed, as competition conditions on all types of resources will slow down the operation.

Welcome Amit Bansode,

I'm sure we can help, but I have a few to questions pose in response first:-Most importantly, What have you tried so far?

There are probably many ways to achieve most tasks, so giving us an idea of your style and thoughts will help us guide you to an answer most suitable to you so you can adjust it to suit your needs in future.


We're all here to learn and getting the relevant information will help us all.



Kind regards,
Robin

I would have never considered to transfer all the files simultaneously, so I don't quite understand your problem.

If you want to transfer 3 files at the same time, a poor man's solution would be to run 3 cp commands in the background and then wait until all of them are done.

A more clever solution would be to monitor the child processes, and as soon one is finished, starts the next background process.

Without understanding your exact requirements, it is difficult to suggest a concrete solution.

how would I copy 3 files at time ?
How to check the status of previous file copy
Is there any loop which copies first 3 files and then pickup next 3 files until last one.
You are right , I can not start all file copy simultenously it will choke up network.

Thank you for all your time and suggestion..

Thanks
Amit Bansode

If you put a cp command in the background, you can get the process id as $! from which you can then watch the processes running. Perhaps some logic like this might help:-
... or just ...

Code:

cd /source/directory
for source_file in *
do
   cp "$source_file" /target/direcory
done

... will do it sequentially and probably no slower overall.

The time taken for the copy will all be based on the contention between disk access and network access. Trying to squeeze multiple jobs through either of these constraints is even likely so be slower overall unless the bottlenecks are multi-pathed in some way so they can spread the load, e.g. your connection to the SAN is 8x1Gb fibre and they load balance (rather than just failover) or you have network throttling on a per-conversation basis, but that's probably more likely for an FTP rather than a cp.

Why do you believe sending 3 files at a time be beneficial?


Robin

As I said, you do this by copying in the background. From a shell (bash, zsh, ksh,...), you do it by appending an ampersand to the statement, for example:

Code:

cp source target &

The variable $! holds the process ID of the background process. You will use it to monitor this process.

If you want to wait, until all of the jobs are finished, have a look at the wait command in the bash man page.

If you want to check, whether a job is alive (without waiting), send it the signal 0 and evaluate the return code:

Code:

kill -0 $process_id_of_your_background_job