Please sit down, we need to talk. More specifically, i need to give you "the talk" - and, no, it is not about flowers and bees.... ;-))

What you do might look to you like some "quick hacks" to make your life easier. In fact it is full-fledged software development and you will never be successful in this endeavour if you do not apply the tenets and procedures of software development. You will never a successful biologist if you, instead of following established good lab practice, do whatever comes to your mind. I take it, you learned your trade as a researcher and acquired all these established best practices. It is now time you do the same with this certain aspect of your research.

Instead of giving you an answer i'd like to show you how to methodically apply procedures to hunt down a bug and find out the answer yourself. Even if your code is only a few lines long you apply the same techniques.

The first, obvious thing that comes to mind is this discrepancy:

Code:

bash /path/to/exon.sh static $file > path/to/${pref}_output.txt

whereas Don Craguns script from post #1 reads:

Code:

#!/bin/ksh

Now, the bash is well capable of strting a Korn shell, so that should work, but it is best practice to minimize every possible source of problems. Because the script states its command processor anyway you can change the line to:

Code:

/path/to/exon.sh static $file > path/to/${pref}_output.txt

which will perhaps not remedy the problem. Let us get on! The next thing is: if you get empty output you may have empty input. We actually see how the involved programs are called and what they are told to do, but there is some uncertainty involved and that are the variable contents: we suppose them to be correct, but better to "be sure" about something is to test it, so let us test it. For this we change the script a little bit. We do that in single steps, like walking is done: you do one step at a time, because if you try to make several steps at once you hop up and down but won't go anywhere.

The first thing to test is the for-llop itself. Does it produce all the files we want it to produce? And, while we are at it:

- does it produce all the files we want?
- does it produce files we don NOT want ("false positives")?
- does it not produce files we do want ("false negatives")?

Code:

for file in path/to/*.txt ; do
     echo "$file"
     # bname=$(basename $file)
     # pref=${bname%%_*.txt}
     # bash /path/to/exon.sh static $file > path/to/${pref}_output.txt
done

What did you find? Often software does not do what it is supposed to do because of small things we easily overlook: i.e. "path/to/*.txt" lacks an introducing "/" to be an absolute path. I understand this is not your real path, but maybe you made the same typo (or a similar one) there as you did here. This makes sure that - if the correct list is produced - this is not the case. This part will be "provenly correct". Let us assume it is and get on. The next thing we test is the variable expansion

Code:

for file in path/to/*.txt ; do
     echo "$file"
     bname=$(basename $file)
     pref=${bname%%_*.txt}
     echo "bname: \"$bname\"   pref:\"$pref\""
     # bash /path/to/exon.sh static $file > path/to/${pref}_output.txt
done

The first thing i notice is the lacking quoting of the variables. Your code will break when a filename will contain a space. A line like:

Code:

variable=something

should, if you are not absolutely 101% sure about what "something" is (and even then, because it doesn't hurt and you should do it habitually right) be quoted:

Code:

variable="something"

Therefore:

Code:

for file in path/to/*.txt ; do
     echo "$file"
     bname=$(basename "$file")
     pref="${bname%%_*.txt}"
     echo "bname: \"$bname\"   pref:\"$pref\""
     # bash /path/to/exon.sh static "$file" > "path/to/${pref}_output.txt"
done

Now, run that. Do the variables all contain the expected values? To be honest, i am suspicious that they do not, for some reason. But ou now have the tools to find out - the first step in correcting it.

The last thing, if the variables do produce the correct values, is to test the command itself: instead of running it we just display it. Notice that we need to escape the redirection:

Code:

for file in path/to/*.txt ; do
     bname=$(basename "$file")
     pref="${bname%%_*.txt}"
     echo /path/to/exon.sh static "$file" \> "path/to/${pref}_output.txt"
done

Now, this should produce a list of commands. Copy and paste one of them to another window and let it run. There might be some diagnostic message (very common are "file not found", "path does not exist" and similar ones, also an attempt to write to some write protected place, full disks, ....) in case the command is what fails.

In one sentence: we took a complex procedure which didn't work as expected and tested one step after the other until we found the culprit. This is how every scientist works and this is how software developer work. Here is a bonus information: you can switch "tracing mode" in the shell on and off so that every command is displayed (to stderr) before it is executed. Try this modification:

Code:

set -xv
for file in path/to/*.txt ; do
     bname=$(basename "$file")
     pref="${bname%%_*.txt}"
     echo /path/to/exon.sh static "$file" \> "path/to/${pref}_output.txt"
done
set +xv

set -xv switches on the trace, set +xv sitches it off again. You could also only trace certain parts:

Code:

for file in path/to/*.txt ; do
     bname=$(basename "$file")
     pref="${bname%%_*.txt}"
     set -xv
     /path/to/exon.sh static "$file" \> "path/to/${pref}_output.txt"
     set +xv
done

This works for Korn shell (ksh) and bash alike.

I hope this helps.

bakunin