Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Redirecting stdout inside a loop

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Redirecting stdout inside a loop
# 1  
Old 08-07-2014
Redirecting stdout inside a loop
hi,
OK. I am writing a bash script, and it is almost working for me.
Problem 1: I currently have stout sent to a file (stout.miRNA.bash.$date_formatted) which I would like to have work inside my loop, but when I move it, it just prints to the screen.
Problem 2: I have a second file (summary.fastqc.$date_formatted), which is basically redirecting the same output file from multiple run output directories.
Is there a way to consolidate these output within the loop so that there is no need for the last cat statement (which is not working) and a third file? So, the script would basically print stout1 summary1; stout2 summary2;... stoutN summaryN.
Problem 3: Is there a way to add in sed/awk to print out specific lines (using search term) of the stout (stout.miRNA.bash.$date_formatted) within the loop, so that I could get the mother load and have edited_stout1 summary1; edited_stout2 summary2;... edited_stoutN summaryN?

HERE IS SOME OF THE CODE, MINUS CALLS TO PROGRAMS INSIDE THE LOOP

Code:
#!/bin/sh -f

date_formatted=$(date +%m_%d_%y)

LOCATION="/illumina/runs/Runs/Project_DefaultProject"

#move into each directory in location and do
for i in "$LOCATION"/Sample_*; do

#pass the filename to $infile
	infile=`ls $i/*.fastq.gz`

####Run several programs#######


#move into each directory from one program in location and send contents of summary.txt to stout
for i in "$LOCATION"/*_fastqc; do
	infile3=`ls $i/summary.txt`
	echo "***testing***" $infile3
cat <$infile3 >>summary.fastqc.$date_formatted
done

done >stout.miRNA.bash.$date_formatted

#cat is not concatenating  both files, seems to only be stout.miRNA.bash that is being written.
#cat summary.fastqc.$date_formatted stout.miRNA.bash.$date_formatted >new.$date_formatted

Thanks for trying to wrap your head around this one:-)
Last edited by Corona688; 08-07-2014 at 06:19 PM.. Reason: needed to say thanks
# 2  
Old 08-07-2014
You have told us a lot about which lines were messing up, but what's less clear is what you'd want the program to be doing, if it were working.

Quote:
Originally Posted by hmortens
Problem 1: I currently have stout sent to a file (stout.miRNA.bash.$date_formatted) which I would like to have work inside my loop, but when I move it, it just prints to the screen.
Both cases do what I'd expect them to do.. What were you expecting?
Quote:
Problem 2: I have a second file (summary.fastqc.$date_formatted), which is basically redirecting the same output file from multiple run output directories.
Is there a way to consolidate these output within the loop so that there is no need for the last cat statement (which is not working) and a third file? So, the script would basically print stout1 summary1; stout2 summary2;... stoutN summaryN.
What do you mean by "redirect" here?

What are stout1 summary1, stout2 summary2, ...?

In what way is the cat "not working"?
Quote:
Problem 3: Is there a way to add in sed/awk to print out specific lines (using search term) of the stout (stout.miRNA.bash.$date_formatted) within the loop, so that I could get the mother load and have edited_stout1 summary1; edited_stout2 summary2;... edited_stoutN summaryN?
Probably, once we know what files you're talking about and what you want us to do to them.

Quote:
...seems to only be stout.miRNA.bash that is being written.
Check the contents of 'summary' then, if they don't appear in the output they probably weren't there in the first place.
Last edited by Corona688; 08-07-2014 at 06:37 PM..
# 3  
Old 08-07-2014
Let me illustrate how redirections work with loops (or any other shell statements):

Code:
# Everything which prints to standard output in this loop ends up in 'output'.
for x in ....
do
        print "something"
        cat $x/file1 $x/file2
done > output

# 4  
Old 08-11-2014
ok I spent ages on my reply and then lost it when trying to post, so here it is again. bah...
You had asked
Quote:
You have told us a lot about which lines were messing up, but what's less clear is what you'd want the program to be doing, if it were working.
Apologies for not being totally clear. I currently have a bash script that is run from a project directory location that contains many Sample directories. The script contains a for loop that is running three third party tools on files in each Sample Directory. I have added a loop within this loop (probably bad form!) to pull a file ('summary.txt') from one program output directory for each sample and print all summary.txt files to a single .txt. Lastly, I direct all stout from the script to a second .txt file.
I would like select specific lines from stout for each sample (or iteration of the loop), and print with each summary.txt, so that I get a report for all Samples for QA purposes. This would need to be in order of Sample processing through the loop to be readable, for Example: sample 1 stout lines, sample1 summary
sample 2 stout lines, sample2 summary.....

This output text file would look something like this (THIS IS OUTPUT FROM CODE, NOT CODE!):
Code:
Sample1
Processed reads: 4378294
Total reads:54783243
Summary Report Sample 1:XXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
SampleN
Processed reads: 213221213
Total reads:33213333
Summary Report Sample 1:XXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Now, I can put all summary files in a single .txt and pull the whole stout to a second .txt, but have not been able to select stout lines within the loop and iterate with printing the summary.
All suggestions welcome!
Moderator's Comments:
Mod Comment CODE tags should be used whenever you are displaying sample input, sample output, and code segments; not just code segments!
Last edited by Don Cragun; 08-11-2014 at 02:03 PM.. Reason: Put CODE tags back in!
# 5  
Old 08-11-2014
Quote:
Originally Posted by hmortens
Apologies for not being totally clear. I currently have a bash script that is run from a project directory location that contains many Sample directories.
Named what? They cannot all be named 'Sample' without some sort of prefix or postfix.
Quote:
The script contains a for loop that is running three third party tools on files in each Sample Directory.
Generating multiple summary.txt's from each 'Sample-XXX'?
Quote:
I have added a loop within this loop (probably bad form!) to pull a file ('summary.txt') from one program output directory for each sample and print all summary.txt files to a single .txt. Lastly, I direct all stout from the script to a second .txt file.
A loop inside a loop is not bad form.
Quote:
I would like select specific lines from stout for each sample (or iteration of the loop)
What is 'stout' here? A filename, a program name, or a misspelling of 'stdout'? If it's stdout, what things stdout? What's it printing? And which bits of it do you wish to keep?
Quote:
...and print with each summary.txt, so that I get a report for all Samples for QA purposes. This would need to be in order of Sample processing through the loop to be readable, for Example: sample 1 stout lines, sample1 summary
sample 2 stout lines, sample2 summary.....
OK, great. You've given us half of what we need instead of none: You've shown the output you want.

Now show the input you have, please, and tell us how you wish to go from one to the other. Preferably enough mock data that we can actually run your test on it, so we have any idea whether what we write does anything remotely like what you want.
# 6  
Old 08-11-2014
Ok. Here is my code
Code:
#!/bin/sh -f

date_formatted=$(date +%m_%d_%y)
#outputDir=$1
speciesBuild=$1
speciesAbbrev=$2


LOCATION="/illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject"

#move into each directory in location and do
for i in "$LOCATION"/Sample_*; do

#test if i is a file or dir, if so change permissions 
	test -f "$i" && chmod 755 "$i"

#pass the filename to $infile
	infile=`ls $i/*.fastq.gz`
	echo "**testing**" $infile

#pass the sample name to $y to append the output directory name 
	x="$i"
	y=${x%.*}
	echo ${y}
	echo ${y#*/Sample_*}


#count .fastq.gz indexes
#zcat $infile | grep '^@H' | cut -d : -f10 | sort | uniq -c | sort -nr | less > ${y#*/Sample_*}.indices.txt

#run CutAdapt to trim Wafergen adapter 
/usr/local/bin/python2.7 /illumina/runs/Runs/cutadapt-1.2.1/bin/cutadapt -a AGATCGGAAGAGCACACGTCT -o $i/${y#*/Sample_*}_cutadapt_AdapterRemoved.fastq.gz $infile 

#define cutadapt output as infile2  
	infile2=`ls $i/*cutadapt_AdapterRemoved.fastq.gz`
	echo "**testing**" $infile2

#run QC with FastQC
/illumina/runs/Runs/FastQC/fastqc --outdir=$LOCATION --quiet $infile2

#run sRNAbench w/o adapter trimming
java -jar /illumina/runs/Runs/miRanalyzer/sRNAbenchDB/sRNAbench.jar input=$infile2 output=${y#*/Sample_*}.sRNAbench.$date_formatted dbPath=/illumina/runs/Runs/miRanalyzer/sRNAbenchDB/ species=$speciesBuild microRNA=$speciesAbbrev p=14 noMM=0 alignType=v minRC=2 

#move into each fastqc directory in location and send contents of summary.txt to stout
for i in "$LOCATION"/*_fastqc; do
	infile3=`ls $i/summary.txt`
	echo "***testing***" $infile3
cat <$infile3 >>summary.fastqc.$date_formatted
done

done >stout.miRNA.bash.$date_formatted

The part that is not working is at the end of this code, where I get two outfiles:
summary.fastqc.$date_formatted

Code:
PASS	Basic Statistics	cutadapt_output_DEHP1_29B.fastq
PASS	Per base sequence quality	cutadapt_output_DEHP1_29B.fastq
PASS	Per sequence quality scores	cutadapt_output_DEHP1_29B.fastq
WARN	Per base sequence content	cutadapt_output_DEHP1_29B.fastq
WARN	Per base GC content	cutadapt_output_DEHP1_29B.fastq
WARN	Per sequence GC content	cutadapt_output_DEHP1_29B.fastq
PASS	Per base N content	cutadapt_output_DEHP1_29B.fastq
FAIL	Sequence Length Distribution	cutadapt_output_DEHP1_29B.fastq
FAIL	Sequence Duplication Levels	cutadapt_output_DEHP1_29B.fastq
FAIL	Overrepresented sequences	cutadapt_output_DEHP1_29B.fastq
FAIL	Kmer Content	cutadapt_output_DEHP1_29B.fastq

And, stout.miRNA.bash.$date_formatted

Code:
**testing** /illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C/GRC270_DEHP2_67C_CGATGT_L001_R1_001.fastq.gz
/illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C
GRC270_DEHP2_67C
cutadapt version 1.2.1
Command line parameters: -a AGATCGGAAGAGCACACGTCT -o /illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C/GRC270_DEHP2_67C_cutadapt_AdapterRemoved.fastq.gz /illumina/runs/Runs/140513_H207_0249_AD2B9LACXX/Unaligned_Lane1/Project_DefaultProject/Sample_GRC270_DEHP2_67C/GRC270_DEHP2_67C_CGATGT_L001_R1_001.fastq.gz
Maximum error rate: 10.00%
   No. of adapters: 1
   Processed reads:      2139267
   Processed bases:    109102617 bp (109.1 Mbp)
     Trimmed reads:      2075206 (97.0%)
     Trimmed bases:     53680380 bp (53.7 Mbp) (49.20% of total)
   Too short reads:            0 (0.0% of processed reads)
    Too long reads:            0 (0.0% of processed reads)
        Total time:    155.78 s
     Time per read:      0.07 ms

=== Adapter 1 ===

Adapter 'AGATCGGAAGAGCACACGTCT', length 21, was trimmed 2075206 times.

No. of allowed errors:
0-9 bp: 0; 10-19 bp: 1; 20-21 bp: 2

Lengths of removed sequences
length	count	expected	max. errors
3	13563	33426.0	0
4	13593	8356.5	0
5	19992	2089.1	0
6	143974	522.3	0
7	48435	130.6	0
8	22001	32.6	0

# 7  
Old 08-11-2014
It writes two files because you tell it to write two files:

Code:
cat <$infile3 >>summary.fastqc.$date_formatted

Code:
done >stout.miRNA.bash.$date_formatted

Which one is correct?
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Redirecting stdion, stdout within an AT command

Hello, I'm strugling with some redirecting and all help is apreciated. The following program is working as expected, but the result of the AT command doesn't go to any file. Thanks in advance for the help. #!/bin/bash modem=/dev/ttyUSB1 file=/root/imsi.txt # print error to stderr and exit... (4 Replies)
Discussion started by: cleitao
4 Replies

2. Shell Programming and Scripting

Redirecting stdout continously to a file

I have a C program that continously outputs info to stdout. The problem is that I am redirecting the stdout and stderr to a file and stdout is written at the end of the problem rather than continously to the file. This could be a problem if for example the program is killed and the stdout output is... (3 Replies)
Discussion started by: igurov
3 Replies

3. Shell Programming and Scripting

Redirecting stdout problem

I have a simple bash script that prints sth every 5 seconds. What I do is the following. I redirect the output of the script to a file, tail the file and see that it works and then from another console I delete the file where the output is redirected to. Even though I have deleted the file, the... (2 Replies)
Discussion started by: igurov
2 Replies

4. Shell Programming and Scripting

Redirecting stdout on background task

Hello, I have a script (videostream.sh) which invokes the GStreamer command-line tool gst-launch with all the correct command line parameters. When I invoke this program, I add the '&' character at the end to make it a background task, so that my script can complete and exit, i.e. gst-launch... (1 Reply)
Discussion started by: salukibob
1 Replies

5. Shell Programming and Scripting

Redirecting stdout to variable while printing it

Hi everybody, I am trying to do the thing you see in the title, and I can't simply do a=$(svn up) echo $a because the program (svn) gives output on lots of lines and in the variable the output is stored on only one line (resulting in a horribly formatted text). Any tips? Thanks,... (2 Replies)
Discussion started by: ocirne94
2 Replies

6. Shell Programming and Scripting

redirecting to stdout in betwen command

can anyone help me in making singleline command for Capital Letters are folders ,small letter are files X,Y,Z are subfolders of A as shown below A - X,Y,Z Folder X has three files a.txt,b.txt,c.txt similarly Y,Z. as shown below X- a.txt,b.txt,c.txt Y- a.txt,b.txt,c.txt Z-... (4 Replies)
Discussion started by: phoenix_nebula
4 Replies

7. UNIX for Dummies Questions & Answers

Redirecting several outputs to /dev/stdout

I have an executable that, depending on its input, outputs to either one file or several. It usually prints nothing on screen. The usual way to call this program is to specify an input and output filenames, like this: ./executable.exe -i inputfile -o outputfileIt will then try to use the output... (1 Reply)
Discussion started by: aplaydoc
1 Replies

8. Shell Programming and Scripting

Redirecting part of output to stdout

Hi, I am trying to execute a command like this: find ./ -name "*.gz" -exec sh -c 'zcat {} | awk -f parse.awk' \; >> output If I want to print the filename, i generally use the -print argument to the find command but when I am redirecting the output to a file, how can I print just the... (2 Replies)
Discussion started by: Legend986
2 Replies

9. Shell Programming and Scripting

implicitly redirecting stdout to a file

Is there a way to redirect all stdout to a file implicitly - like defining stdout=/home/me/process.log - so that all "echo" commands in several scripts/subscripts are written to that file; instead of having to edit all scripts to redirect the "echo" (e.g. echo 'This is a test ' >>... (1 Reply)
Discussion started by: ALTRUNVRSOFLN
1 Replies

10. Shell Programming and Scripting

redirecting STDOUT & STDERR

In bash, I need to send the STDOUT and STDERR from a command to one file, and then just STDERR to another file. Doing one or the other using redirects is easy, but trying to do both at once is a bit tricky. Anyone have any ideas? (9 Replies)
Discussion started by: jshinaman
9 Replies
Login or Register to Ask a Question