Shell Programming and Scripting

I have a task to move more than 35000 files every two hours, from the same directory to another directory based on a file that has the list of filenames

I tried the following logics

(1)
find . -name \*.dat > list

for i in `cat list` do mv $i test/ done

(2)
cat list|xargs -i mv "{}" test/


logic 1 takes 40mins and logic 2 in 25 mins


Is there any other faster way I can move files based on a file that has the list of filenames.

Are you sure you need the list?

Code:

mv *.dat test

P.S. May be tar/cpio are better for large number of files ...

I don't know that this would be faster, but it should execute "mv" fewer times
(You'd have to figure out an appropriate number for -s)

Code:

xargs -s 1000 echo  < list | sed 's+^+mv +;s+$+ test/+' |sh -

If you have perl and the test directory is in the same partition, you might try

Code:

perl -nle 'rename "$_", "test/$_";' list

Just a suggestion..... Not sure if this is even an option for your environment.

Don't move the files first. Just create links from the old dir to the new dir. run the link command first. then come back and move the files to over write the link. this way the files are available very quickly and your move cmd can take 40 minutes.

Just thinking outside the box.

I doubt this command, because OP had said 35000 files and more.

What if the number of command line arguments to ' mv ' exceeds the permissible limit ?

Its quite possible !

Here is another way,

with "list" as the file with list of filenames

split the list file into n chunks .. list_1, list_2 ... list_n

you should arrive at a bench mark say 10 to 50 or something like that

now operate any of the logic on the n list_* files simulatenously

and this should definitely be faster as mv is executed in multiples.

Didn't think about that

So, with recent zsh:

Code:

setopt nomatch
while set -- *.dat([1]);do
mv -- *.dat([1,1000]) test
done

Adjust 1000 for your limit.