Shell Programming and Scripting

i have tried process substitution, but run in some problems.

this works:

Code:

 
samtools view -h $SAMdir/$b.bam | htseq-count -m union -s no -q -t exon -i gene_id - $gtf > $b.count &

but this not:

Code:

 
htseq-count -m union -s no -q -t exon -i gene_id <(samtools view -h $SAMdir/$b.bam) $gtf > $b.count &

the error is not fundamental, and some (few) of the runs worked (i looped over a bunch of files). the error indicated a truncated sam-file, that means that one line of the file (which is in a specific format, the sam format) was malformed (some columns are missing). how can the above commands be different?
or has it to do with the multiple processes i started (using &)?

Maybe it treats stdin - as a pipe but does seeks when there is a file name. Use strace/tusc/truss to verify what calls are made to the kernel.

What shell are you using and on what OS and version ?

@Scrutinizer
OS: Scientific Linux 6.3 (= Red Hat Enterprise Linux 6.3)
shell: GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)


@DGPicket
do you mean that htseq knows that - is a piped stream but is confused from <(..). i thought <(..) simulates a pipe perfectly.

strace is new for me, i will look if i can use it correctly...

Could you check if there is still a difference if you use:

Code:

( samtools view -h $SAMdir/$b.bam | htseq-count -m union -s no -q -t exon -i gene_id - $gtf > $b.count ) &

vs.

Code:

( htseq-count -m union -s no -q -t exon -i gene_id <(samtools view -h $SAMdir/$b.bam) $gtf > $b.count ) &

--
Also, do you use a wait statement further down in your script?

While <() substitutes the name of a named pipe as a 'word' (has virtual spaces around it, cannot be concatenated to quoted strings, says David Korne), the programmer may have assumed (ass U me) that it was a flat file, and that only '-' was suspected of being a pipe, and so incapable of a seek. You can see that on strace. I say this not out of experience with such behaviors, just from knowing the stupid things half-knowing UNIX programmers do. If there is a problem, stat() would tell the deducated programmer before that it was a pipe. Presumably, the proces copies the pipe data into a tmpfile() and uses that in place of the pipe, adding latency. Hard to say. Maybe the programmer likes to rewind() files out of paranoia alone, and only had problems with stdin as he did not have or know of <() and 'mknod <name> p'.

You might dump the data in a temp file and exec redirect stdout in the <() to give it a flat file. I would get back to the writer of the app about this goofiness.

The ksh only supports <() on systems with the open fd in the file tree, like /proc/$$/fd/#, so a simple pipe() call makes ephemeral named pipes that disappear at close or exit. In comparison, bash makes durable named pipes in /var/tmp to use (and never cleans them up, an unfortunate bug with no easy solution).

It doesn't simulate a pipe, it is one -- a different kind of pipe though, since it must have a filename.