I'm have tons of image files I need to process, but I don't need to process all of them and it would take a long time to process them all if I don't have to.
Then, within those RawData folders, the images are named...
blahblah.000001.T000.D000.P000.H001.LA.TIF
blahblah.000002.T000.D000.P000.H001.LA.TIF
blahblah.000003.T000.D000.P000.H001.LA.TIF
...
blahblah.00000N.T000.D000.P000.H001.LA.TIF
Sooo, when I created the images, they were created in blocks of 30 images, but I only want, say, the 14th image to the 27th image in the block, and there might be 9 blocks of images in a single RawData folder (so 30*9 = 270 images in a folder). So, in my most recent experiment, I've taken 9 blocks of 30 images per folder, and 17 folders, so 270 in a folder and 17 folder, so 4590 (270*17) images in total.
At the moment, I'm using MATLAB to move the files and it takes agggges to do, since "movefile" in MATLAB isn't as fast as cut/paste file. I'm pretty terrible at programming and really any solution to this would make me happy since my MATLAB script takes upwards of an hour to run each time.
The nice thing about MATLAB is even an idiot like me can use it, at the moment my script looks like this (which works fine, but is painfully slow)...
Then I can just nest that script in a loop which cycles through all folders within a given experiment.
But the matlab "movefile" command is painfully slow. In the time it's taken me to write this message, I'm moving files and it's only gotten a quarter of the way through a single experiment worth of data.
Can someone think of a way to do the same thing with a bash script? Or anything that won't take as long as MATLAB, I know if I just cut/paste the files by hand it's almost instant, but then it takes me an hour just to find and select the files!
I tried bash scripting with multiple counters going to tell me when I hit the start and end of blocks, but I couldn't get it working, I also tried a windows batch script and failed miserably. My usual tactic of googling similar questions and adapting the scripts I find online has failed me.
I'm a programmers worst nightmare, an engineer who needs to be able to write simple scripts like this but can't
Thanks.
---------- Post updated 10-17-12 at 01:55 AM ---------- Previous update was 10-16-12 at 10:36 PM ----------
Actually I should add to that and say it's not actually folder1,folder2,folder3 etc, the folders all have different names which relate to the experiment, so to loop the folders I can't just go "folder$i". Sorry I should have clarified that.
Also the actual image names...
blahblah.000001.T000.D000.P000.H001.LA.TIF
blahblah.000002.T000.D000.P000.H001.LA.TIF
blahblah.000003.T000.D000.P000.H001.LA.TIF
...
blahblah.00000N.T000.D000.P000.H001.LA.TIF
The H increments within each block, so two blocks (the first 60 images in a folder) will be
I only had to change a couple of things (because I wasn't specific enough in my question!), there were actually files other than the images in the RawData folder, I only wanted the images though, and also I didn't need to keep the folder structure intact, I just wanted to dump all images in to a single folder (as the processing software we use is retarded, it can only process 1 folder at a time). Also I forgot I had directories in the experiment folder that had RawData subfolders and images in them that I didn't want to copy, but luckily the folders are arranged by date, so I just modified it to this and it works perfectly...
Thanks for that, I would have struggled for many hours with the syntax since I'm only just learning bash scripting.
That script took 5 minutes per experiment to move all the files (running in cygwin) compared to the MATLAB script which took over an hour per experiment.
I have three files as an input and I need to rearrange this input to match the rules by which the processing program consumes the data.
My files are:
/tmp$ cat F
# file -1-
FS00|0|zero-zero|
FSTA|0|10|
FSTA|0|12|
FSTA|0|15|
FSTA|0|17|
FS00|3|negative|
FSTA|3|-1|
FS00|5|regular|... (2 Replies)
Hey guys,
I have wrote the following script to apply a module named "trinity" on my files. (it takes two input files and spit a trinity.fasta as output)
#!/bin/bash -l
#SBATCH -p node
#SBATCH -A <projectID>
#SBATCH -n 16
#SBATCH -t 7-00:00:00
#SBATCH --mem=128GB
#SBATCH --mail-type=ALL... (1 Reply)
I want to move all files from one directory to another directory excluding today (sysdate files) on daily basis.
file name is in pattern file_2013031801, file_2013031802 etc (2 Replies)
Hi,
I am just new to scripting but got to write a complex scipt please help. i need a shell script which can check the list of data listed in a txt doc and see if they belong to any of the groups that are listed in other list file.... (5 Replies)
Hi Guys,
First post! I've seen a few options but dont know the most efficient:
I have a directory with a 150,000+ text files in it
I want to merge them into files contain 10,000 files with a carriage return in between.
Thanks
P
The following is an example but doesnt limit the... (2 Replies)
I know I can use an ls -l junk1 command to get a listing of all files in the directory junk1, but I was wondering how I'd go about going through the files in junk1 in a for-in loop and issuing the ls -l command on them one by one.
This is what I have so far:
for file in $(ls -a $1)
do
ls... (1 Reply)
I have five directories, dir1 to dir5
for each directory, I have all same number-named folders. There are four types of folders, {1..10}, {20..30}, { 40..50}, {60..70}
Now for each types of folder, I will do the same thing, here is the code
for i in {1..5}
do
cd dir$i
mkdir temp1
for... (5 Replies)
Hi
i have many sets of files as shown below(here i have shown 2 sets)
basel_aa_20091030.txt
basel_aa_20091130.txt
basel_aa_20091230.txt
basel_bb_20091030.txt
basel_bb_20091130.txt
basel_bb_20091230.txt
from each set of files i need to select the latest updated file(there are... (3 Replies)
Hi Guru's,
I have to write a shell script which groups file names based upon the certain matching string pattern, then creates the Tar file for that particular group of files and then zips the Tar file created for the respective group of files.
For example, In the given directory these files... (3 Replies)
Hi,
I am a newbie to shell scripting. here is my objective:
1)The shell program should take 2 parameters - ie-> DestinationFolder, WebFolder
2)Destination folder contains few files that has to has be verified and deleted.
3)WebFolder is a folder containing a list of master files
4)It... (1 Reply)