Grep causing long delay (batching) whilst piping


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Grep causing long delay (batching) whilst piping
# 1  
Old 07-02-2008
Java Grep causing long delay (batching) whilst piping

Hi all.

I have a problem at work which I have managed to break down into a simple test scenario:

I have written a monitoring script that outputs every second the status of various processes, but for now, lets just print the date

input.sh:
Code:
while true
do
  date
  sleep 1
done

This will produce results ok for logging, but I have also started on a script that takes this input and processes it on the fly. For now, lets use...

output.sh
Code:
while read in
do
   echo "input = $in       output = " `date`
done

Now, make them executable etc and run them, piping input to output:

Quote:
spudtheimpaler@spudslaptop:~$ ./input.sh | ./output.sh
input = Wed Jul 2 21:02:35 BST 2008 output = Wed Jul 2 21:02:35 BST 2008
input = Wed Jul 2 21:02:36 BST 2008 output = Wed Jul 2 21:02:36 BST 2008
input = Wed Jul 2 21:02:37 BST 2008 output = Wed Jul 2 21:02:37 BST 2008
input = Wed Jul 2 21:02:38 BST 2008 output = Wed Jul 2 21:02:38 BST 2008
input = Wed Jul 2 21:02:39 BST 2008 output = Wed Jul 2 21:02:39 BST 2008
input = Wed Jul 2 21:02:40 BST 2008 output = Wed Jul 2 21:02:40 BST 2008
No surprises so far.

Here is the but... say I only wanted certain pieces of the input to work with? lets add an arbitrary grep to do the filtering:

Quote:
spudtheimpaler@spudslaptop:~$ ./input.sh | grep 2008 | ./output.sh
## long pause ensues
input = Wed Jul 2 21:04:20 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:21 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:22 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:23 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:24 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:25 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:26 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:27 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:28 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:29 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:30 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:31 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:32 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:33 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:34 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:35 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:36 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:37 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:38 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:39 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:40 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:41 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:42 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:43 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:44 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:45 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:46 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:47 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:48 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:49 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:50 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:51 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:52 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:53 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
input = Wed Jul 2 21:04:54 BST 2008 output = Wed Jul 2 21:04:55 BST 2008
So that is my problem. Unsurprisingly, if you just grep the results of input.sh they are displayed every second.

I was wondering if, assuming I've been clear, anyone knew a reason for this and how to avoid it? My output.sh is supposed to update every second also, and currently it is displaying the updates in batches after long pauses. Is there a better way than while read to get the input from a pipe? Maybe that's it? I'm not a complete novice, but this is my first dealing with writing scripts that can be piped to.

I've tried with taking the delay out, although the delays are much reduced, they still exist. I've also tried the search here, other places, and google, but haven't found anything (though the search terms are pretty ubiquitous)

Thanks for your time.

Regards,
Mitch.
# 2  
Old 07-02-2008
Hi Mitch,

I had exactly the same problem with a script I wrote to put timestamps on some tail -f output - I kept getting bunches of lines with the same timestamp instead of them being processed in real time.

I ended up replacing grep with a small perl script:

Code:
./input.sh | perl -nwe '
        BEGIN {
                # From the open() section on the perlfunc man page.
                # Makes output unbuffered.
                select(STDOUT); $| = 1;
        }
        /2008/ && print;
' | ./output.sh

Hope that helps.
# 3  
Old 07-03-2008
Question

Quote:
Originally Posted by Annihilannic
Hi Mitch,

[...]
I ended up replacing grep with a small perl script:
[...]
Hope that helps.
Annihilannic,

I appreciate the response, and it's good to know I'm not alone with the problem, but I'm writing this for what could be many boxes, including production servers where perl wont be available. That is of course secondary to the fact that there is already a tool that, as far as I know, should work fine. Apart from your workaround, did you find any information on *why* it might be happening?

Thanks again for your time.

Regards,
Mitch.
# 4  
Old 07-03-2008
why do you need to do things in separate scripts? do your processing in the while loop itself. Its just a program design problem.
# 5  
Old 07-03-2008
Bug

Quote:
Originally Posted by ghostdog74
why do you need to do things in separate scripts? do your processing in the while loop itself. Its just a program design problem.
Because:

the actual 'input' script outputs something like

Quote:
timestamp process_name pid memory_usage cpu_usage
13:00:00 proc1 1000 500M 50%
13:00:00 proc2 1001 500M 10%
13:00:00 proc3 1002 256M 13%
13:00:01 proc1 1000 500M 50%
13:00:01 proc2 1001 500M 10%
13:00:01 proc3 1002 256M 13%
13:00:02 proc1 1000 500M 50%
13:00:02 proc2 1001 500M 10%
13:00:02 proc3 1002 256M 13%
which is primarily for logging purposes. The 'output' script was going to attempt to take the processes as they were output, and display running averages. As there are, in reality, many processes this will monitor and you would only need to actively monitor one t a time, I used a simple grep on it, which is when i noticed the issue.

Can I work around it? Sure. I can log to a file and tail that, for example, but I'm more curious as to why this is happening. This is less of a 'I have a problem and need a workaround' and more of a 'this is unexpected behaviour, does anyone have an explanation?' kind of query.

Cheers,
Mitch.
# 6  
Old 07-03-2008
what i meant is ( if you are able to modify your input script)

Code:
while some condition
do
 # somewhere here produces  
 # the input script output
 # at the same time 
 # store running averages. 
done

# 7  
Old 07-03-2008
Quote:
Originally Posted by ghostdog74
what i meant is ( if you are able to modify your input script)

Code:
while some condition
do
 # somewhere here produces  
 # the input script output
 # at the same time 
 # store running averages. 
done

I see what you mean, and you aren't wrong. Thanks Smilie

To be honest at this point I am just curious as to why it is behaving as such, rather than how to workaround it. It's a learning exercise Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. OS X (Apple)

Piping to grep with pbpaste

cat file 1 aaa 2 bbb 3 ccc 4 ddd In TextEdit, I then copy the characters “ccc” to the clipboard. The problem is that the following command gives no output: bash-3.2$ pbpaste | grep - file Desired output: 3 ccc What should the syntax be for that command? I am using MacOS El... (3 Replies)
Discussion started by: palex
3 Replies

2. Solaris

SSH and telnet long delay to recieve prompt.

Hi guys. You'd have to excuse me a bit, as I'm a noob. I really try to avoid asking questions and do research for whatever linux issues that may arise. I am experiencing a long wait for the shell to come up when I ssh or telnet into a Sunos 5.10 environment. It takes 70 seconds to give me... (12 Replies)
Discussion started by: gpenco
12 Replies

3. UNIX for Dummies Questions & Answers

Piping grep into awk, read the next line using grep

Hi, I have a number of files containing the information below. """"" Fundallinfo 6.3950 14.9715 14.0482 """"" I would like to grep for Fundallinfo and use it to read the next line? I ideally would like to read the three numbers that follow in the next line and... (2 Replies)
Discussion started by: Paul Moghadam
2 Replies

4. Ubuntu

Piping with grep

Hi everybody, I have a big file with blast results (if you know what this means, otherwise look at it just as a text file with a specific form). I am trying to extract some ids from within this file, which have certain parameters. For example, some Of my IDs have the term 'No hit results'... (6 Replies)
Discussion started by: frymor
6 Replies

5. Ubuntu

Piping with grep

Hi everybody, I have a big file with blast results (if you know what this means, otherwise look at it just as a text file with a specific form). I am trying to extract some ids from within this file, which have certain parameters. For example, some Of my IDs have the term 'No hit results'... (1 Reply)
Discussion started by: frymor
1 Replies

6. Shell Programming and Scripting

Piping STDOUT as pattern to grep or sed

$>cat file.txt 123 d3 234 abc 3 zyf 23 124 def 8 ghi kz0 ... ... I have the following output on the screen through <some command>. $> <some command> abc def ghi ... ... I have to search for each of these patterns in the file.txt and print the lines in file.txt matching the... (4 Replies)
Discussion started by: VNR
4 Replies

7. UNIX for Dummies Questions & Answers

Piping GREP

Hi, I need to use a double grep so to speak. I need to grep for a particular item say BOB and then for each successful result I need to grep for another item say SMITH. I tried grep "BOB" filename | grep "SMITH" but it does not seem to work. I can achieve my desired result using an... (12 Replies)
Discussion started by: mojoman
12 Replies

8. Shell Programming and Scripting

question about grep, cut, and piping

Howdy folks, I am fairly new to scripting but have lost of expirience in c++, pascal, and a few other. I am trying to complete a file search script that is sent a file name containing data to search that is arranged like this "id","name","rating" "1","bob","7" etc and an argument to... (1 Reply)
Discussion started by: dyrt
1 Replies

9. UNIX for Advanced & Expert Users

Long Delay if any with network services

While installing a firewall, I was pinging the interface from SCO 5.0.6 Openserver box, while no response, I hit "DEL" to cancel, but no cancel. Then all of a sudden I get BOO-KOO traffic lights on HUB and Switch.... Then a kernel trap error. System froze... Proceeded with a cold boot. Now I have... (8 Replies)
Discussion started by: nashvillek5
8 Replies
Login or Register to Ask a Question