tail | grep lagging badly


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting tail | grep lagging badly
# 1  
Old 04-22-2008
tail | grep lagging badly

I'm trying to use tail/grep to monitor a log file. The command I cooked up is:

tail -n 50 -f output.log | grep 'type:system' | cut -f 5-

A sample line from the log file is:

1208894862 type:system session:0 severity:4 load started

the columns are tab delimited.

this works ok, except that it's inexplicably lagging badly. If I have 2 terminals open, one simply tailing the file and the other tail|greping the file as specified above, it might take 30 seconds or more for the tail|grep terminal to display the next line while the tail terminal displays it instantly.

EDIT: forgot to add this.
Also, sometimes new text does not appear until even more txt is added to the log file.
Example:
'test 1' is written to the log
tail will show this right away, tail | grep will NOT
'test 2' is written to the log
tail will show this right away, tail | grep will show both 'test1' and 'test2' now



Any hints for me? Thanks in advance

Last edited by WasabiVengeance; 04-22-2008 at 05:23 PM..
# 2  
Old 04-22-2008
The pipe buffers I/O so it won't be printed until the output buffer is full. This is a classic FAQ. Google for "grep unbuffered" for example.
# 3  
Old 04-22-2008
Thank you for your reply, it's much appreciated.

I think I understand the issue, but I don't see an obvious solution in your reply. Should I not be using a pipe? Is there some grep option I should be using, or maybe awk or sed instead?
# 4  
Old 04-22-2008
Some grep implementations have a --line-buffered option which would help. I don't think there is a standard, one-size-fits-all solution to this. Maybe a Perl script to do the tail and grep in the same process; that should certainly avoid any pipe buffering.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Trouble with tail and grep

Good Morning, i ran into some trouble this morning while 'improving' my monitoring stuff. i would like to get a warning when the number of mails sent (outbound) by postfix is above a certain number. so far, so easy. to test that i simply put cat /var/log/mail.info | grep 'to=<' | grep -v -e... (1 Reply)
Discussion started by: Mike
1 Replies

2. Debian

LM 17.3 xfce constant lagging

I'm using LM 17.3x LIVE. Have constant and sometimes, severe lagging issues. Problems started when I "upgraded" to 18.3x. I tried 5 other distros all with the same issues. Went back to 17.3 and , alas, the problem followed. Found this: sudo gedit /etc/sysctl. conf vm. swappiness = 15, but all it... (4 Replies)
Discussion started by: 69Rixter
4 Replies

3. UNIX for Dummies Questions & Answers

Tail -f | grep > output.txt

hi guys, I perform a sort of monitoring. I have a server running and with tail -f | grep "Searchstring"I monitor the log-file for recent specific entries. This is ok and works fine. Now, in addition I want to have my search results not posted into the shell but into a file. I tried: tail... (3 Replies)
Discussion started by: LaUs3r
3 Replies

4. Shell Programming and Scripting

How do i use tail & grep in real time here??

Hi I have a file which is updated very frequently. Where in i wanted to use tail -f command in the script and wanted to grep for a particular word. But the issue is when i use tail -f filename|grep "word" ... it will show me blank until the word is found in the real time. if it shows... (13 Replies)
Discussion started by: nikhil jain
13 Replies

5. UNIX for Dummies Questions & Answers

mulitple grep using tail

I have a basic tail/grep question. I have logs that are generated & kept in a directory called alert_audit. I am using "tail" to see the logs that are coming in, but I only need logs that contain the IP address 10.249.185. or 10.247.231. Here is the command I have, but it pulls all IP... (3 Replies)
Discussion started by: robertson1995
3 Replies

6. UNIX for Dummies Questions & Answers

Need help with a tail and a grep

I need to tail -f a file so I can monitor it as it is being written to. However, there is a lot of garbage in the file that I don't care about. So normally I would just pipe and grep for the string that is important to me. However, in this case, there are two things I need to grep for. I can't... (3 Replies)
Discussion started by: Silver11
3 Replies

7. Shell Programming and Scripting

tail | grep

The program that is running on my machine generates log files. I want to be able to know the number of lines that contain "FT" in the most recent log file. I wrote the following, but it always returns zero. And I know the count is not zero. Any ideas? ls -rt *.log | tail -n 1 | grep -c FT (6 Replies)
Discussion started by: sdilucca
6 Replies

8. Shell Programming and Scripting

Grep or Tail in shell script

Hi, I am writing a shell script that checks catalina logs on a production system and mails me if it detects errors. It greps the logs for known errors which i have defined as variables. The problem is the logs are huge, approx 30,000 before they rotate. So I am forced to use grep instead... (3 Replies)
Discussion started by: Moxy
3 Replies

9. Shell Programming and Scripting

tail, grep and cut

Hello all, I have some weird problem that kinda baffles me. Say I have the following test file: claudia:~/tmp$ cat testfile.txt This is a test line This is the second test line And yeah, this is the third test line Then say I want to tail the file, grep for the word "third" then... (7 Replies)
Discussion started by: sylaan
7 Replies

10. UNIX for Dummies Questions & Answers

Help with tail /grep needed

Hello: I'm a very newbee at UNIX/AIX. What i want to do is to tail a file from the bottom until a certain string is found and write all the lines after the found string to another file. I've tried out a lot of combination with tail and grep but doesn't find the good one. Could someone help... (4 Replies)
Discussion started by: Felix2511
4 Replies
Login or Register to Ask a Question