Redirecting stdout inside a loop


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Redirecting stdout inside a loop
# 15  
Old 08-11-2014
Quote:
Originally Posted by hmortens
is it possible to organize it real time? like as it is being written to file?
That's your program's output. Once it's written to the file, it's too late to do anything about it inside the loop. Organize it before you put it into the file.


Quote:
I was not specific about this before because I thought it would be the same pattern search, but I need to pull lines containing "No. input reads:", "No. reads in analysis:", "mapped...reads to genomes(s)", and "Detected:"
If you want to pile all three commands together then filter them, do it like this, before it gets to the file:

Code:
for ... in ...
do
        # All commands get piped through egrep, which then prints to stdout
        (
                command1
                command2
                command3
        ) | egrep 'expression1|expression2|expression3'

        # This file gets printed to stdout
        cat restoffile

# Everything printed to stdout gets saved to onesingleoutputfile, in order
done > onesingleoutputfile

The ( ) join them together in a subshell, letting you put a pipe on the end to filter the contents of all three.

Last edited by Corona688; 08-11-2014 at 06:54 PM..
# 16  
Old 08-12-2014
how would you integrate this with summary.txt in this example?
I am guessing I would put the
Code:
for ... in ...
do
        # All commands get piped through egrep, which then prints to stdout
        (
                command1
                command2
                command3
        ) | egrep 'expression1|expression2|expression3'

for i in "$LOCATION"/*_fastqc; do
	infile3=`ls $i/summary.txt`
	echo "***testing***" $infile3
cat <$infile3 >>summary.fastqc.$date_formatted
done
 # This file gets printed to stdout ADD summary.txt HERE???
        cat restoffile summary.fastqc.$date_formatted
# Everything printed to stdout gets saved to onesingleoutputfile, in order
done > onesingleoutputfile

# 17  
Old 08-12-2014
Quote:
Originally Posted by hmortens
how would you integrate this with summary.txt in this example?
I have no idea.

I am completely unable to give details, due to your complete inability to give details. I don't think that "$LOCATION"/*_fastqc is what you want since that will match all fastqc as I understand it, but cannot say what you do want.

I still don't know which things of your code generate what bits of your output. I still don't know where most of your files are stored... I am not even sure if these fastqc files are inside or outside your 'sample' things or not, which would have helped me answer your last question. In four days and three pages of pulling teeth, I have found one eye, one ear, and one toe of the elephant. I suppose these must be proprietary things you're not allowed to divulge.

I do note that you're blindly continuing to redirect >>summary.fastqc.$date_formatted even though you've told me you don't want to put it there. To not redirect into that file, I suggest not redirecting into that file... It will end up in stdout instead, which will be caught by the outer loop.

If that's not exactly when you wanted it, then print it sooner -- or later. If you wanted only part of it, then filter it then and there. If you print exactly what you want, in the order you want, that's what you'll get in your output.

I think I have given you all the techniques you need.

Last edited by Corona688; 08-12-2014 at 12:41 PM..
# 18  
Old 08-12-2014
wow. communication breakdown. Part of the difficulty may be that the input is large, genomic files and the various output of each program is statistical in nature. I did not think this was of interest to the linux forum or consequence for manipulating the stdout and single outfile that I have pasted here. thanks for your help. I will post any solution.
# 19  
Old 08-12-2014
When you have a big problem, a small model of it is sometimes sufficient to find out what you need. But not too small a model. You could also break the problem into halves, to make it simpler to deal with. Right now we either solve all of it or nothing...

You've also been confusing some terms, and I think I sense some confusion on how 'stdout' and redirection in general work, your questions may not mean what you think they do, causing confusion on both sides. Perhaps if you explained what you wanted, instead of the way you wanted to accomplish it.

Perhaps something like:

Quote:
Part 1: I have 3 programs which print <three sets of madeup lines>, of which I need <these subsets of them> as selected by <these madeup strings>, printed in <this order>. There is also a single file which looks like <these pretend contents> which must be printed in its entirety, after <those subsets>, but before <this subset>. How do I get <this exact pretend-output>, given those exact pretend-inputs?
Quote:
Part 2: Same problem, but now there are thousands of these which must be looped over. Each is in its own folder. They have <this string> in common, changing <this last part> for each folder. Additionally, <these contents> are are in <individual different folders>, but the two parts have <this string> in common if that helps you pair them up.
This is what I've been trying to get. This is sufficient to build a working scale-model of your problem, which we can code around and test.

Once we get that working, you can fill in the pretend bits with real bits and see if it still works for the larger problem.

Or you could keep hiding bits and dodging questions and hope we can guess the rest...

Last edited by Corona688; 08-12-2014 at 02:54 PM..
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Redirecting stdion, stdout within an AT command

Hello, I'm strugling with some redirecting and all help is apreciated. The following program is working as expected, but the result of the AT command doesn't go to any file. Thanks in advance for the help. #!/bin/bash modem=/dev/ttyUSB1 file=/root/imsi.txt # print error to stderr and exit... (4 Replies)
Discussion started by: cleitao
4 Replies

2. Shell Programming and Scripting

Redirecting stdout continously to a file

I have a C program that continously outputs info to stdout. The problem is that I am redirecting the stdout and stderr to a file and stdout is written at the end of the problem rather than continously to the file. This could be a problem if for example the program is killed and the stdout output is... (3 Replies)
Discussion started by: igurov
3 Replies

3. Shell Programming and Scripting

Redirecting stdout problem

I have a simple bash script that prints sth every 5 seconds. What I do is the following. I redirect the output of the script to a file, tail the file and see that it works and then from another console I delete the file where the output is redirected to. Even though I have deleted the file, the... (2 Replies)
Discussion started by: igurov
2 Replies

4. Shell Programming and Scripting

Redirecting stdout on background task

Hello, I have a script (videostream.sh) which invokes the GStreamer command-line tool gst-launch with all the correct command line parameters. When I invoke this program, I add the '&' character at the end to make it a background task, so that my script can complete and exit, i.e. gst-launch... (1 Reply)
Discussion started by: salukibob
1 Replies

5. Shell Programming and Scripting

Redirecting stdout to variable while printing it

Hi everybody, I am trying to do the thing you see in the title, and I can't simply do a=$(svn up) echo $a because the program (svn) gives output on lots of lines and in the variable the output is stored on only one line (resulting in a horribly formatted text). Any tips? Thanks,... (2 Replies)
Discussion started by: ocirne94
2 Replies

6. Shell Programming and Scripting

redirecting to stdout in betwen command

can anyone help me in making singleline command for Capital Letters are folders ,small letter are files X,Y,Z are subfolders of A as shown below A - X,Y,Z Folder X has three files a.txt,b.txt,c.txt similarly Y,Z. as shown below X- a.txt,b.txt,c.txt Y- a.txt,b.txt,c.txt Z-... (4 Replies)
Discussion started by: phoenix_nebula
4 Replies

7. UNIX for Dummies Questions & Answers

Redirecting several outputs to /dev/stdout

I have an executable that, depending on its input, outputs to either one file or several. It usually prints nothing on screen. The usual way to call this program is to specify an input and output filenames, like this: ./executable.exe -i inputfile -o outputfileIt will then try to use the output... (1 Reply)
Discussion started by: aplaydoc
1 Replies

8. Shell Programming and Scripting

Redirecting part of output to stdout

Hi, I am trying to execute a command like this: find ./ -name "*.gz" -exec sh -c 'zcat {} | awk -f parse.awk' \; >> output If I want to print the filename, i generally use the -print argument to the find command but when I am redirecting the output to a file, how can I print just the... (2 Replies)
Discussion started by: Legend986
2 Replies

9. Shell Programming and Scripting

implicitly redirecting stdout to a file

Is there a way to redirect all stdout to a file implicitly - like defining stdout=/home/me/process.log - so that all "echo" commands in several scripts/subscripts are written to that file; instead of having to edit all scripts to redirect the "echo" (e.g. echo 'This is a test ' >>... (1 Reply)
Discussion started by: ALTRUNVRSOFLN
1 Replies

10. Shell Programming and Scripting

redirecting STDOUT & STDERR

In bash, I need to send the STDOUT and STDERR from a command to one file, and then just STDERR to another file. Doing one or the other using redirects is easy, but trying to do both at once is a bit tricky. Anyone have any ideas? (9 Replies)
Discussion started by: jshinaman
9 Replies
Login or Register to Ask a Question