Shell Programming and Scripting

G'day all,

I'm have tons of image files I need to process, but I don't need to process all of them and it would take a long time to process them all if I don't have to.

The images are arranged in folders like this...

folder1/RawData
folder2/RawData
folder3/RawData
...
folderN/RawData

Then, within those RawData folders, the images are named...
blahblah.000001.T000.D000.P000.H001.LA.TIF
blahblah.000002.T000.D000.P000.H001.LA.TIF
blahblah.000003.T000.D000.P000.H001.LA.TIF
...
blahblah.00000N.T000.D000.P000.H001.LA.TIF

Sooo, when I created the images, they were created in blocks of 30 images, but I only want, say, the 14th image to the 27th image in the block, and there might be 9 blocks of images in a single RawData folder (so 30*9 = 270 images in a folder). So, in my most recent experiment, I've taken 9 blocks of 30 images per folder, and 17 folders, so 270 in a folder and 17 folder, so 4590 (270*17) images in total.

At the moment, I'm using MATLAB to move the files and it takes agggges to do, since "movefile" in MATLAB isn't as fast as cut/paste file. I'm pretty terrible at programming and really any solution to this would make me happy since my MATLAB script takes upwards of an hour to run each time.

The nice thing about MATLAB is even an idiot like me can use it, at the moment my script looks like this (which works fine, but is painfully slow)...

Code:

%%Load file names in to a variable
files = dir('*.TIF');

%%define the size of each block of images and which images to take
    skipfirst = 13;
    takeimage = 14;
    blocksize = 30;

%%create a vector which holds the indexes for the files to copy
    take_vec = [];
    for kk = 1:length(files)/blocksize

        take_vec(length(take_vec)+1:length(take_vec)+takeimage) = [(blocksize*(kk-1)+skipfirst+1): (takeimage+skipfirst+blocksize*(kk-1))];

    end

%%extract the names of the files to copy from the names of all the files in the folder
    for mm = 1:length(take_vec)

       copyfiles(mm,: ) = files(take_vec(mm)).name;
        
    end

%%loop through the array of copyfile names and move the files
  n = size(copyfiles,1);

    for jj = 1:n

        str = sprintf([copyfiles(jj,: )]);
        movefile(str,['../../sorted_files/' str])

    end

Then I can just nest that script in a loop which cycles through all folders within a given experiment.

But the matlab "movefile" command is painfully slow. In the time it's taken me to write this message, I'm moving files and it's only gotten a quarter of the way through a single experiment worth of data.

Can someone think of a way to do the same thing with a bash script? Or anything that won't take as long as MATLAB, I know if I just cut/paste the files by hand it's almost instant, but then it takes me an hour just to find and select the files!

I tried bash scripting with multiple counters going to tell me when I hit the start and end of blocks, but I couldn't get it working, I also tried a windows batch script and failed miserably. My usual tactic of googling similar questions and adapting the scripts I find online has failed me.

I'm a programmers worst nightmare, an engineer who needs to be able to write simple scripts like this but can't



Thanks.

---------- Post updated 10-17-12 at 01:55 AM ---------- Previous update was 10-16-12 at 10:36 PM ----------

Actually I should add to that and say it's not actually folder1,folder2,folder3 etc, the folders all have different names which relate to the experiment, so to loop the folders I can't just go "folder$i". Sorry I should have clarified that.

Also the actual image names...
blahblah.000001.T000.D000.P000.H001.LA.TIF
blahblah.000002.T000.D000.P000.H001.LA.TIF
blahblah.000003.T000.D000.P000.H001.LA.TIF
...
blahblah.00000N.T000.D000.P000.H001.LA.TIF

The H increments within each block, so two blocks (the first 60 images in a folder) will be

blahblah.000000.T000.D000.P000.H001.LA.TIF
blahblah.000001.T000.D000.P000.H001.LA.TIF
blahblah.000002.T000.D000.P000.H001.LA.TIF
...
blahblah.000029.T000.D000.P000.H001.LA.TIF
blahblah.000030.T000.D000.P000.H002.LA.TIF
blahblah.000031.T000.D000.P000.H002.LA.TIF
blahblah.000032.T000.D000.P000.H002.LA.TIF
...
blahblah.000059.T000.D000.P000.H002.LA.TIF

Brilliant! That works, thanks heaps.

I only had to change a couple of things (because I wasn't specific enough in my question!), there were actually files other than the images in the RawData folder, I only wanted the images though, and also I didn't need to keep the folder structure intact, I just wanted to dump all images in to a single folder (as the processing software we use is retarded, it can only process 1 folder at a time). Also I forgot I had directories in the experiment folder that had RawData subfolders and images in them that I didn't want to copy, but luckily the folders are arranged by date, so I just modified it to this and it works perfectly...

Code:

#!/bin/bash
start=14
end=27
sz=30
cd <experiment folder>
mkdir sorted_files
for dir in $(find 10162012_3d* -type d -name RawData)
do
        b=0
        for file in $(find $dir/*.LA.TIF -type f -print)
        do
           let b=b+1
           if [ $b -ge $start -a $b -le $end ]
           then
               mv $file sorted_files
           fi
           [ $b -ge $sz ] && b=0
        done
done

Thanks for that, I would have struggled for many hours with the syntax since I'm only just learning bash scripting.

That script took 5 minutes per experiment to move all the files (running in cygwin) compared to the MATLAB script which took over an hour per experiment.