09-07-2003
Question about pipes in solaris (and others) and buffering....
This is weird, so I'm hoping someone here knows solaris and how it handles pipes...
OK... here goes...
Theres this log file, right? I want to tail -f it, grep that, gzip that, then pipe that into more commands. Well thats easy, right?
tail -f file | grep pattern | gzip | otherstuff...
Except...
The output of grep is buffered... Not linebuffered, but 5120 character buffered. As in:
tail -f file | grep pattern | cat
Buffers, and only put stuff to the terminal after grep has outputted 5191 characters.
tail -f file | grep pattern
and
tail -f file | cat
do not.
Using perl, or sed, or tr, or awk, or gzip, or a number of others instead of grep exhibit this behavior as well. It was suggested to me that this might have to do with solaris' bidirectional pipes. I'm not sure that this means.
The buffering doesn't work for me, because with the grep, and further with the gzip, it could take the thing a week to fill once in real use, which isn't really workable for what I need to do.
On another machine, also running solaris, the same thing occurs, except in blocks of 4096. Same on a linux machine.
The question... Is there any way to make a pipe not exhibit this behavior? It is undesirable for my use.
Last edited by sannik; 09-07-2003 at 04:08 AM..
10 More Discussions You Might Find Interesting
1. Windows & DOS: Issues & Discussions
Hello!
Will someone out there pls help in clearifying what is really wrong with my system. I use window 98 as my operating system. I am connected to a proxy server for browsing the net. Whenever l am listening to music online l have the problem of intermitted break in which the playing will... (10 Replies)
Discussion started by: kayode
10 Replies
2. Filesystems, Disks and Memory
Hello!
How I can increase or decrease predefined pipe buffer size?
System FreeBSD 4.9 and RedHat Linux 9.0
Thanks! (1 Reply)
Discussion started by: Jus
1 Replies
3. UNIX for Advanced & Expert Users
I am trying to figure out why when i have the following code
int main( { printf("0\n"); fork(); printf("1\n"); exit(0);}
and type in the shell
a.out | cat
the output of this program is
0
1
0
1
instead of
0
1
1
does anyone know? (3 Replies)
Discussion started by: Phantom12345
3 Replies
4. Shell Programming and Scripting
Hi,
I'm have a tcl application using expect. The application is connecting to a unix box and is running a script. What i need my app to do is show the output of the console as the script runs in realtime. At the moment, it my app is waiting until the script finishes and then dumps what is... (0 Replies)
Discussion started by: Phi01
0 Replies
5. Solaris
hi..
I have a question because there are files on Solaris and will not let me change the root user is the owner.
excuse is that I come from a Linux environment and I like to learn Solaris
thank (5 Replies)
Discussion started by: cejodrake
5 Replies
6. UNIX for Advanced & Expert Users
Hi,
There are three type of buffering: Fully buffered,line buffered,and unbuffered. The goal of the buffering provided by the standard I/O library is to use the minimum number of read and write calls.
I understand that decrease write() calls when use output function output data to a file ,... (4 Replies)
Discussion started by: Edward114
4 Replies
7. Programming
If a code forks 2 childs, what can the values be for the process id's of each of the child? I child pid is supposed to be 0, but what if you fork 2 of them? (5 Replies)
Discussion started by: omega666
5 Replies
8. Shell Programming and Scripting
Hello,
There's a third-party application's command that shows the application's status like "tail -f verybusy.log". When use the command, the output comes every 1-sec. but when it goes in a script below the output comes every 8-sec...What is the problem and how can I fix it?
open(CMD,... (2 Replies)
Discussion started by: Shawn, Lee
2 Replies
9. Solaris
Hello all.
I am new hear and would like to ask a question regarding to the Solaris Crash Analysis Tool.
We are analyzing the results of "thread summary" but not quite sure what the asterisk represents.
Following are the items that asterisk were attached.
50* threads sleeping on a semaphore (49... (1 Reply)
Discussion started by: YuW
1 Replies
10. Shell Programming and Scripting
The old buffering problem again, in a very specific case. On FreeBSD this time, but it's the generic line-buffered vs fully-buffered problem.
I'm trying to pick an available bluetooth speaker (all named audio_N), by pinging all of them en mass and taking the first to respond.
The... (10 Replies)
Discussion started by: Juha Nurmela
10 Replies
LEARN ABOUT NETBSD
zfgrep
ZGREP(1) BSD General Commands Manual ZGREP(1)
NAME
zgrep, zegrep, zfgrep -- print lines matching a pattern in gzip-compressed files
SYNOPSIS
zgrep [grep-flags] [--] pattern [files ...]
zegrep [grep-flags] [--] pattern [file ...]
zfgrep [grep-flags] [--] pattern [file ...]
DESCRIPTION
zgrep runs grep(1) on files or stdin, if no files argument is given, after decompressing them with zcat(1).
The grep-flags and pattern arguments are passed on to grep(1). If an -e flag is found in the grep-flags, zgrep will not look for a pattern
argument.
zegrep calls egrep(1), while zfgrep calls fgrep(1).
EXIT STATUS
In case of missing arguments or missing pattern, 1 will be returned, otherwise 0.
SEE ALSO
egrep(1), fgrep(1), grep(1), gzip(1), zcat(1)
AUTHORS
Thomas Klausner <wiz@NetBSD.org>
BSD
December 28, 2003 BSD