Shell Programming and Scripting

Hi All,

I am having a situation now to delete a huge number of temp files created during run times approx. 16700+ files. We have never imagined that we will get this this much big list of files during run time. It worked fine for lesser no of files in the list. But when list is huge we are getting "List too long error".

Code:

 
 remvfile='/etc/usr/bin/removefilelist'
 rm -f `cat $remvfile`

Please find the code above. The removefilelist is the file which contains the list of files to delete. Please find the sample content of removefilelist below.

Code:

 /etc/usr/workfile/filename1
 /etc/usr/workfile/filename2
 /etc/usr/workfile/filename3
 /etc/usr/workfile/filename4

We also don't know how many number of file we will get in the list. Probably more than 50000+. So I would like to get your help here.

Thanks.

Try:

Code:

while read X; do
    rm -f $X
done < /etc/usr/bin/removefilelist

Thanks Zaxxon,

But I would like to avoid while loops because it would introduce performance issues.

Lets consider this then:-

Code:

xargs rm -f < $remvfile

I tested on 100,000 files in a temporary directory like this:-

Code:

i=0;until [ $i -gt 99999 ]
do
   ((i=$i+1))
   touch $i
done

ls -1 > /tmp/remvfile
xargs rm < /tmp/remvfile

Does this help? There are far fewer rm commands called so it should run faster because they're not spawning a process for each file, however the IO overhead will still be the same. For my test, I actually ran rm five times.



Robin

It would probably be better to ensure that the directory that contains the temporary files has no other files in it so that:

Code:

rm -r temp_directory
mkdir temp_directory

I tend to use for loops for this kind of thing :-

Code:

for file in `cat /etc/usr/bin/removefilelist`
do
  rm -f "$file"
done

...which is a bad habit as this is a useless use of cat, and dangerous use of backticks.