Problem with pipes on infinite streams | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Problem with pipes on infinite streams

Shell Programming and Scripting


Tags
pipe

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 03-27-2010
tokland tokland is offline
Registered User
 
Join Date: Aug 2008
Last Activity: 22 April 2010, 12:55 PM EDT
Posts: 16
Thanks: 0
Thanked 0 Times in 0 Posts
Problem with pipes on infinite streams

Here is an example code that shows the issue I have:


Code:
#!/bin/bash
counter() {
  seq 1000 | while read NUM; do
    echo $NUM
    echo "debug: $NUM" >&2 
    sleep 0.1 # slow it down so we know when this loop really ends
  done
}

counter | grep --line-buffered "[27]" | head -n1


Code:
debug: 1
debug: 2
2
debug: 3
debug: 4
debug: 5
debug: 6
debug: 7

If I understand it correctly, "head" finishes on the first match (as expected), but "grep" is not aware of it until it tries to write the next line (the second match). When it does, it finds out the pipe is closed so it also finishes.

That's normally not a problem, but if you have an infinite input stream containing only one match, it won't never stop. Any solution?
Sponsored Links
    #2  
Old 03-27-2010
alister alister is offline Forum Advisor  
Registered User
 
Join Date: Dec 2009
Last Activity: 24 April 2014, 11:08 AM EDT
Posts: 3,132
Thanks: 172
Thanked 937 Times in 761 Posts
Hello, tokland:

If using GNU grep:

Code:
counter | grep -m1 '[27]'

If that's not available:

Code:
counter | sed -n '/[27]/{p;q;}'

Regards,
Alister
Sponsored Links
    #3  
Old 03-27-2010
tokland tokland is offline
Registered User
 
Join Date: Aug 2008
Last Activity: 22 April 2010, 12:55 PM EDT
Posts: 16
Thanks: 0
Thanked 0 Times in 0 Posts
Hi!

Quote:
Originally Posted by alister View Post
If using GNU grep:

Code:
counter | grep -m1 '[27]'

If that's not available:

Code:
counter | sed -n '/[27]/{p;q;}'

Thanks, those are good solutions. However, the grep in my code was just an example, let's imagine you cannot change how the stream is generated:

stream_generator | head -n1

By the way, using process substitution "works":

head -n1 <(stream_generator)

but it keeps the generator running on the background until the next match.
    #4  
Old 03-27-2010
alister alister is offline Forum Advisor  
Registered User
 
Join Date: Dec 2009
Last Activity: 24 April 2014, 11:08 AM EDT
Posts: 3,132
Thanks: 172
Thanked 937 Times in 761 Posts
You can run the stream generator in the background, asynchronously, and use a named pipe to communicate with it:

Code:
mkfifo sg_pipe
stream_generator > sg_pipe &
head -n1 sg_pipe
kill %?stream

Regards,
Alister

---------- Post updated at 10:51 PM ---------- Previous update was at 10:38 PM ----------

For the example code you used in your original post:

Code:
#!/bin/bash
counter() {
  seq 1000 | while read NUM; do
    echo $NUM
    echo "debug: $NUM" >&2 
    sleep 0.1 # slow it down so we know when this loop really ends
  done
}

mkfifo p
counter | grep --line-buffered "[27]" > p &
head -n1 p
kill %?counter

Outputs:

Code:
$ ./tokland.sh 
debug: 1
debug: 2
2

Alister
Sponsored Links
    #5  
Old 03-29-2010
tokland tokland is offline
Registered User
 
Join Date: Aug 2008
Last Activity: 22 April 2010, 12:55 PM EDT
Posts: 16
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
You can run the stream generator in the background, asynchronously, and use a named pipe to communicate with it:
Very nice. Named pipes are not so cool, but now we have full control over the job (I guess that's not so easy with process substitution)

Thanks Alister!
Sponsored Links
    #6  
Old 03-29-2010
binlib binlib is offline
Registered User
 
Join Date: Aug 2009
Last Activity: 15 March 2013, 10:40 AM EDT
Location: New Jersey
Posts: 380
Thanks: 7
Thanked 90 Times in 75 Posts
If you make it sleep shorter time or make your machine busier, you will observe your original problem with named pipe (or any other methods). A Unix pipe has at least 4k buffer size and I don't think there is a way to make it smaller. Without a way to reduce the pipe size and not able to modify the streaming code, I see no way to solve your problem.
Sponsored Links
    #7  
Old 03-30-2010
alister alister is offline Forum Advisor  
Registered User
 
Join Date: Dec 2009
Last Activity: 24 April 2014, 11:08 AM EDT
Posts: 3,132
Thanks: 172
Thanked 937 Times in 761 Posts
Quote:
Originally Posted by binlib View Post
If you make it sleep shorter time or make your machine busier, you will observe your original problem with named pipe (or any other methods). A Unix pipe has at least 4k buffer size and I don't think there is a way to make it smaller. Without a way to reduce the pipe size and not able to modify the streaming code, I see no way to solve your problem.
If you're saying that the loop may run a few more times, sure. You are quite correct. The generator will write a few bytes into the pipe's buffer, never filling the buffer, and will loop until its timeslice is exhausted. But a few extra loop iterations is not the same as his original problem, in which the generator would run without end.

Regards,
Alister

Last edited by alister; 03-30-2010 at 01:01 AM..
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
problem using pipes with "ls" bijeet_sunny UNIX for Advanced & Expert Users 4 10-12-2009 04:22 PM
infinite while do loop problem tententen Shell Programming and Scripting 8 07-22-2009 09:20 AM
problem with pipes afser UNIX for Dummies Questions & Answers 2 09-23-2008 04:15 AM
PIPEs and Named PIPEs (FIFO) Buffer size Jus Filesystems, Disks and Memory 1 08-20-2004 10:14 AM
STREAMS alwayslearningunix UNIX for Dummies Questions & Answers 1 04-10-2001 05:11 AM



All times are GMT -4. The time now is 12:30 PM.