Problem with pipes on infinite streams


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Problem with pipes on infinite streams
# 1  
Old 03-27-2010
Problem with pipes on infinite streams

Here is an example code that shows the issue I have:

Code:
#!/bin/bash
counter() {
  seq 1000 | while read NUM; do
    echo $NUM
    echo "debug: $NUM" >&2 
    sleep 0.1 # slow it down so we know when this loop really ends
  done
}

counter | grep --line-buffered "[27]" | head -n1

Code:
debug: 1
debug: 2
2
debug: 3
debug: 4
debug: 5
debug: 6
debug: 7

If I understand it correctly, "head" finishes on the first match (as expected), but "grep" is not aware of it until it tries to write the next line (the second match). When it does, it finds out the pipe is closed so it also finishes.

That's normally not a problem, but if you have an infinite input stream containing only one match, it won't never stop. Any solution?
# 2  
Old 03-27-2010
Hello, tokland:

If using GNU grep:
Code:
counter | grep -m1 '[27]'

If that's not available:
Code:
counter | sed -n '/[27]/{p;q;}'

Regards,
Alister
# 3  
Old 03-27-2010
Hi!

Quote:
Originally Posted by alister
If using GNU grep:
Code:
counter | grep -m1 '[27]'

If that's not available:
Code:
counter | sed -n '/[27]/{p;q;}'

Thanks, those are good solutions. However, the grep in my code was just an example, let's imagine you cannot change how the stream is generated:

stream_generator | head -n1

By the way, using process substitution "works":

head -n1 <(stream_generator)

but it keeps the generator running on the background until the next match.
# 4  
Old 03-28-2010
You can run the stream generator in the background, asynchronously, and use a named pipe to communicate with it:
Code:
mkfifo sg_pipe
stream_generator > sg_pipe &
head -n1 sg_pipe
kill %?stream

Regards,
Alister

---------- Post updated at 10:51 PM ---------- Previous update was at 10:38 PM ----------

For the example code you used in your original post:
Code:
#!/bin/bash
counter() {
  seq 1000 | while read NUM; do
    echo $NUM
    echo "debug: $NUM" >&2 
    sleep 0.1 # slow it down so we know when this loop really ends
  done
}

mkfifo p
counter | grep --line-buffered "[27]" > p &
head -n1 p
kill %?counter

Outputs:
Code:
$ ./tokland.sh 
debug: 1
debug: 2
2

Alister
# 5  
Old 03-29-2010
Quote:
You can run the stream generator in the background, asynchronously, and use a named pipe to communicate with it:
Very nice. Named pipes are not so cool, but now we have full control over the job (I guess that's not so easy with process substitution)

Thanks Alister!
# 6  
Old 03-29-2010
If you make it sleep shorter time or make your machine busier, you will observe your original problem with named pipe (or any other methods). A Unix pipe has at least 4k buffer size and I don't think there is a way to make it smaller. Without a way to reduce the pipe size and not able to modify the streaming code, I see no way to solve your problem.
# 7  
Old 03-30-2010
Quote:
Originally Posted by binlib
If you make it sleep shorter time or make your machine busier, you will observe your original problem with named pipe (or any other methods). A Unix pipe has at least 4k buffer size and I don't think there is a way to make it smaller. Without a way to reduce the pipe size and not able to modify the streaming code, I see no way to solve your problem.
If you're saying that the loop may run a few more times, sure. You are quite correct. The generator will write a few bytes into the pipe's buffer, never filling the buffer, and will loop until its timeslice is exhausted. But a few extra loop iterations is not the same as his original problem, in which the generator would run without end.

Regards,
Alister

Last edited by alister; 03-30-2010 at 02:01 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Homework & Coursework Questions

Help with infinite loop problem

1. The problem statement, all variables and given/known data: My problem is an infinite loop when i press any other key other then Y or y in the while loop. what i want it to do is return to the normal script outside of it if pressing N or n or keep asking the same question if its any other... (4 Replies)
Discussion started by: Ren_kun
4 Replies

2. Programming

Problem with Pipes => Only works first pipe

Hi! I'm having problems with pipes... I need comunnications with childs processes and parents, but only one child can comunnicate with parent (first child), others childs can't. A brief of code: if(pipe(client1r)<0){ perror("pipe"); } ... (1 Reply)
Discussion started by: serpens11
1 Replies

3. Programming

please help a problem in client-server ipc message 2 pipes communication simple example

I want to have a message send & receive through 2 half-duplex pipes Flow of data top half pipe stdin--->parent(client) fd1--->pipe1-->child(server) fd1 bottom half pipe child(server) fd2---->pipe2--->parent(client) fd2--->stdout I need to have boundary structed message... (1 Reply)
Discussion started by: ouou
1 Replies

4. Shell Programming and Scripting

Problem with call of Java Programm & return code handling & output to several streams.

Hello Everybody, thanks in advance for spending some time in my problem. My problem is this: I want to call a java-Programm out of my shell skript, check if die return code is right, and split the output to the normal output and into a file. The following code doesn't work right, because in... (2 Replies)
Discussion started by: danifunny
2 Replies

5. Programming

Problem with pipes

problem solved. (1 Reply)
Discussion started by: superfons
1 Replies

6. UNIX for Advanced & Expert Users

problem using pipes with "ls"

Hi all, I tried the following command $ find / -name xyx | ls -l so logically it should show the listing of directory xyz , assuming there's only one instance of xyz . But the above command shows the listing of current directory instead. I got the desired result using it in the... (4 Replies)
Discussion started by: bijeet_sunny
4 Replies

7. Shell Programming and Scripting

infinite while do loop problem

hi all, this is how my scrip looks like #!/bin/sh bindir='/opt/apps/script/bin' datadir='/opt/apps/script/data' dir='/opt/apps/script' while : ; do ls -1rt /opt/apps/script/data/check.txt*|tail -1 > /dev/null 2>&1 if ;then chmod +rwx $bindir/dummy2.sh ... (8 Replies)
Discussion started by: tententen
8 Replies

8. UNIX for Dummies Questions & Answers

problem with pipes

I have written the following program. The function of this prog is to read data from a file(source.c) and write into another file(dest.c) using pipes. I have just written a line in the source file.Im able to compile and run the program without errors. But the data is not written onto the other... (2 Replies)
Discussion started by: afser
2 Replies

9. Filesystems, Disks and Memory

PIPEs and Named PIPEs (FIFO) Buffer size

Hello! How I can increase or decrease predefined pipe buffer size? System FreeBSD 4.9 and RedHat Linux 9.0 Thanks! (1 Reply)
Discussion started by: Jus
1 Replies

10. UNIX for Dummies Questions & Answers

STREAMS

Hi Everyone I am building some A Class HP boxes as web proxy servers, have just installed HP-UX 11.00 and am starting to configure one according to our standard build policy. However on the A Class I just happened to place the software depot this error message keeps popping up every 2 mins: ... (1 Reply)
Discussion started by: alwayslearningunix
1 Replies
Login or Register to Ask a Question