Shell Programming and Scripting

Is it guarantee that mv would move the file?

It is not odd.
Change [ ! -d "$d/processed/$b/$a" ] && mkdir -p $d/processed/$b/$a && mv $d/$line $d/processed/$b/$a/
to

Code:

if [ ! -d "$d/processed/$b/$a" ]; then
     mkdir -p $d/processed/$b/$a || continue
fi
mv $d/$line $d/processed/$b/$a/

---------- Post updated at 10:55 AM ---------- Previous update was at 10:50 AM ----------

No, on that design, unless, further code is added

How about

Code:

ls $d | while read line; do TD="processed/${line:15:6}/${line:0:5}"; TD=${TD//_}; mkdir -p $TD; echo cp $line $TD; done

?

Let it try to mkdir existing directories, because man mkdir:

How can you *guarantee* that "your" mv command will work and kind of doubt that "my" mkdir command will work?
I think you're just trying to find something to grouse about.

I thought you were referring to the fact that by placing the mv under the requirement of `mkdir' success, it would not `mv' any files where the directory tree was already in place. Sorry, I misunderstood you there.
And sure, that was my only initial point, there's nothing guarantee to be successful when issuing a command, that's why is good to check for any success return, if available, or accept the result fully aware that we intent to not care if it fails, or provide a way of recovering.

The real problem with the performance is the number of subprocesses spawned for each loop iteration.

You're spawning something like 8 processes for each iteration.

If you really want to optimize it, process the "ls" listing with awk in one pass to produce the proper commands, sorting the results so you can process multiple files at a time with one "mv" command to move more than one file at a time into the target directory.

Code:

d=/conversion/program/out
cnt=0
background_limit=10ls $d | while read line ; do
        a=`echo $line|cut -c1-5|sed "s/_//g"`
        b=`echo $line|cut -c16-21`
        if [ ! -d "$d/processed/$b/$a" ] ; then
              mkdir -p $d/processed/$b/$a  || continue
        fi
        mv $d/$line $d/processed/$b/$a/ &
        cnt=$(( $cnt + 1 ))
        [ $(( $cnt %  $backgound_limit )) -eq 0 ] && wait
done
wait

My contention is that mv is taking tons of time, for reasons I don't understand.
Is this an NFS mount?

If $d/processed is on the same filesystem as this $d, then the mv operation should be REALLY fast. Unless the directories involved are really munged with way too many files in which case ls would be super slow and so would any directory operation, like mv or [ -d dirname ] --

I ran the modified code on an unloaded system with 1500+ files in a directory, took less than 2 minutes - 1TB 7200rpm sata disk. They were dummy files of 1 block size - 8192.

Without doing a & on the mv call.

IF mv is mv-ing a file around on a single filesystem then the call amounts to a small, quick change to directory entries. And Achenle has a great point - child processes like that use resources and time.

So I suspect filesystem/directory problems.

What OS and filesystem do you have?