Shell Programming and Scripting

As MadeInGermany informs us that the -delete is available in newer versions, then that will probably run better than -exec rm {} \; as the latter will spawn a new process for each file, and that in itself will take a small amount of time. Multiply by perhaps 100,000 hits suddenly becomes a lot of time spent just creating a new process for each delete.

To my embarrassment, most of my servers are rather behind the times (AIX 4.3.3 for some) so I've only got this flag in RHEL 6.3.

Interestingly in the RHEL man page, I found this:- Perhaps a neater way (with examples) of what I was trying to say earlier.



Glad that we could collectively help, and I've learned something too

Robin

Hi.
You mention MFT -- is this filesystem NTFS? ... cheers, drl

Yes, this is windows XP, but I do the heavy lifting with cygwin. Nothing beats a linux tool box for large scale file operations. I suppose there may be a dos equivalent, but I never bothered to learn dos when I could use cygwin and learn a real shell like bash instead.

LMHmedchem

Hi.

I generally advise people to use whatever seems best for them. However, we need to accept drawbacks along with the advantages.

For your rsync / fragmentation issue, I think it's the XP NTFS filesystem, not the application. Some links you may want to look over are:

Defragmentation - Wikipedia, the free encyclopedia

A forum discussion (mostly slanted toward Linux, against MS): Why does Windows suffer from disk fragmentation when Linux doesn't? [Archive] - Ubuntu Forums

A blog that is more MS-centric view: Rants and Raves Linux File System Fragmentation

File system fragmentation - Wikipedia, the free encyclopedia

And many many more from Google searches.

My own experience with rsync (via rsnapshot): my workstation backup is run 6 times per day. I have been running it for more than 1 year. The numbers from fsck are:

Code:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdc1             241G   35G  194G  16% /media/big_disk_1

big_disk_1: 1,285,708/16,007,168 files (0.3% non-contiguous), 8984671/64000944 blocks

So very low fragmentation for a disk that sees a fair amount of activity. This does not necessarily parallel your use, but I have never used a defragmentation code on any Linux filesystem. When I used W2K, I seemed to need a defragmentation run quite often.

Best wishes ... cheers, drl

That's a very inefficient method of deleting a large number of files with Cygwin. The fork required to delete each file performs very, very poorly. Either use -delete or the + version of -exec.

Regards,
Alister

So would this be the preferable version if there is the potential for a large number of files to be involved?

find . -name "*.back" -type f -delete

I'm not familiar with what the "+ version of -exec" would be.

What is the difference between using double and single quotes for the extension? I have seen both, meaning "*.back" or '*.back'?

LMHmedchem

To reduce the calls to programs (like rm), there are two implementations
1. Unix, later defined by Posix

Code:

find . -name "*.back" -type f -exec rm -f {} +

I.e. you replace \; by + and the program (here rm) must accept multiple arguments.
This implementation seems difficult; I have met some buggy ones.
2. GNU find, by means of the xargs program that converts an input stream to multiple arguments:

Code:

find . -name "*.back" -type f -print0 | xargs -0 rm -f

Note the corresponding -print0 and -0; without them there is incorrect handling of filenames with space characters.
--
A demonstration of quoting types:

Code:

echo "this is $HOME"
echo 'this is $HOME'

Concerning *.back there is no difference.