Shell Programming and Scripting

I have a simple script which copies directory from one place to another and deleting the source .
I am facing a situation when new files gets added when the script has started running. Its resulting in data loss

Please suggest a way to avoid data loss. I googled a lot but most are perl solutions . I am looking for something is shell script.

what ratio of files is being seen when you do 'ls -ltr' as changed. whats the frequency of change and what's the phenomena how often these file are updated

cyclic check on files being transferred would be a better idea. Do you expect the files changing every 1 sec, 1 minute or what. calculate that time's mean value and then start transferring them.

2ndly, you may try some register file, where you can write down the files being transferred so far and then in the next run of the script, avoid them.

the files are posted at random intervals . So I would not be sure.

Requirement here is to clean up the source after proper copy .

Don't cp, use mv instead. mv is an atomic command, deleting the source only if copy succeeded.

mv is safe if source and destination are in the same file system.
On different file systems, it must copy the data like cp.
--
It might help to only copy files with ctime greater than 1 hour.

Agreed. But wouldn't it delete the source only if the copy across file systems succeeded?

No offense intended (just honesty), but you're problem statement is useless. Given its utter lack of specificity, I'm surprised anyone invested any of their time in responding to it.

Accurate answers to the following questions will probably lead to a quick resolution:

* What operating system you are using?
* What are the exact commands used to add files to the current directory?
* What are the exact commands used to copy the files to their new location?
* Are these two directories part of the same filesystem?
* What are the exact commands (if any) that are run as part of any subsequent clean up.
* What exactly do you mean by data loss? Are entire files missing? Are you seeing partially complete files? Something else?

For all we know, your problem may be as simple as misusing 'rm -fr' when 'rmdir' is required.

In the future, if you would like accurate, focused assistance, save everyone (yourself included) time and be specific from the start.

Regards,
Alister