Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: [Solved] Help piping tail to read STDIN
Top Forums Shell Programming and Scripting [Solved] Help piping tail to read STDIN Post 302444274 by Samb95 on Wednesday 11th of August 2010 11:26:58 AM
Old 08-11-2010
[Solved] Help piping tail to read STDIN
Hello everybody,

I need some help here. I have a log file that gets updated every hour approximately.
I want to make some processing on each line which is added in the log file with a program written in PERL.
The problem is that I don't see anything when a line is added in the log file.
I use the tail command on my log file piped to my PERL program :

Code:
tail -f my_log_file | perl my_perl_program.pl arg1 arg2

Here is a sample of the log file :

Code:
[0720 16:00:00.646257] 0x42258940 (Debug) Cache SUMMARY [s1.meta] attrs now/668 min/668 max/668.
[0720 16:00:00.646262] 0x42258940 (Debug) RSVD SUMMARY [s1.meta] reserved space max requested/128 MB accounted now/0 MB
[0720 16:00:00.646273] 0x42258940 (Debug) VOP SUMMARY [s1.meta] Setattr cnt/14 avg/53+54369 min/13+42 max/334+259525.
[0720 16:00:00.646283] 0x42258940 (Debug) VOP SUMMARY [s1.meta] Open cnt/7 avg/24+187 min/19+175 max/34+231.
[0720 16:00:00.646291] 0x42258940 (Debug) VOP SUMMARY [s1.meta] Close cnt/7 avg/20+37 min/16+29 max/28+54.
[0720 16:00:00.646300] 0x42258940 (Debug) VOP SUMMARY [s1.meta] GetResyncAttr cnt/1 avg/20+25 min/20+25 max/20+25.
[0720 16:00:00.646305] 0x42258940 (Debug) Cache SUMMARY [s3.meta] attrs now/4 min/4 max/4.
[0720 16:00:00.646309] 0x42258940 (Debug) RSVD SUMMARY [s3.meta] reserved space max requested/32 MB accounted now/0 MB
[0720 16:00:00.646320] 0x42258940 (Debug) VOP SUMMARY [s3.meta] Setattr cnt/12 avg/31+63182 min/17+39 max/150+334001.
[0720 16:00:00.646329] 0x42258940 (Debug) VOP SUMMARY [s3.meta] Open cnt/6 avg/22+180 min/18+159 max/31+210.
[0720 16:00:00.646337] 0x42258940 (Debug) VOP SUMMARY [s3.meta] Close cnt/6 avg/19+31 min/17+25 max/29+37.
[0720 16:00:00.646346] 0x42258940 (Debug) VOP SUMMARY [s3.meta] GetResyncAttr cnt/1 avg/19+24 min/19+24 max/19+24.
[0720 16:00:00.646351] 0x42258940 (Debug) Cache SUMMARY [s4.meta] attrs now/387 min/85 max/387.
[0720 16:00:00.646355] 0x42258940 (Debug) RSVD SUMMARY [s4.meta] reserved space max requested/128 MB accounted now/128 MB
[0720 16:00:00.646366] 0x42258940 (Debug) VOP SUMMARY [s4.meta] Setattr cnt/7838 avg/23+41116 min/10+22 max/1515+525846.
[0720 16:00:00.646376] 0x42258940 (Debug) VOP SUMMARY [s4.meta] Open cnt/321 avg/20+35 min/11+15 max/297+343.

And here is my program :

Code:
#!/usr/bin/perl

use warnings;
use strict;

if ($#ARGV != 1){
    print "You did not give the two necessary arguments.\nUsage: perl my_prog.pl filename VOP\n";
    exit;
}

#name of the file
my $file_name=$ARGV[0];
#Summary type that I want to analyze
my $sum_type=$ARGV[1];

my $node_name;
my $operation_type;
my $operation_counter;

chomp($sum_type, $file_name);

while (<STDIN>){
    chomp;

    #I use regular expressions here to retrieve the name of the server the type of operation and the counter value only if the summary type is VOP
    if (/\s$sum_type\sSUMMARY\s\[(\w+)\.\w+\]\s(\w+)\scnt\/(\d+)\s/) {
        $server_name=$1;
        $operation_type=$2;
        $operation_counter=$3;
        my $timestamp=time;

        #This is a print so that collectd (a program that I use to monitor my servers) can put a value in a RRD file from the print output
        print "PUTVAL $node_name/CVFS-$file_name/operations-$operation_type $timestamp:$operation_counter";
    }
}

To understand what I want to see for the output let's take an example. If this line was added to the log file nothing should happen because it is not a VOP type :
Code:
[0720 16:00:00.646355] 0x42258940 (Debug) RSVD SUMMARY [s4.meta] reserved space max requested/128 MB accounted now/128 MB

Whereas, if this line was added :
Code:
[0720 16:00:00.646376] 0x42258940 (Debug) VOP SUMMARY [s4.meta] Open cnt/321 avg/20+35 min/11+15 max/297+343.

Here is what I should see as an output if the filename is log1:
Code:
PUTVAL s4/CVFS-log1/operations-Open 1281540375:321

That's it. I don't get any output with the tail command. Does anyone see an error there or has an idea of what I am doing wrong here ?

Thanks by advance.
Last edited by pludi; 08-12-2010 at 06:44 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Piping output to while read

Hi. Im using cat to output the contents of a file, then piping it to my while read loop.In this loop variables get assigned values. However when i try to use the variables outside the loop their values has been reset.I understand about subshells etc. but I have no idea how to "preserve" the... (3 Replies)
Discussion started by: Ultimodiablo
3 Replies

2. Shell Programming and Scripting

piping output of tail running in background

Not sure why this does not work in bash: tail -f err.log |& -bash: syntax error near unexpected token `&' I am attempting to continuously read a file that is being updated by doing a "tail -f" on the file and piping the output to stdin which can then be read by the next shell command Thnx (4 Replies)
Discussion started by: anuramdas
4 Replies

3. Shell Programming and Scripting

Piping tail to awk to parse a log file

Hello all, I've got what I'm pretty sure is a simple problem, but I just can't seem to work past it. I'm trying to use awk to pretty up a log file, and calculate a percentage. The log file looks like this: # tail strtovrUsage 20090531-18:15:45 RSreq - 24, RSsuc - 24, RSrun - 78, RSerr -... (4 Replies)
Discussion started by: DeCoTwc
4 Replies

4. Shell Programming and Scripting

Piping through commands read as variables from file

This is going to be part of a longer script with more features, but I have boiled it down to the one thing that is presently stumping me. The goal is a script which checks for updates to web pages that can be run as a cron job. The script reads (from a tab-delim file) a URL, an MD5 digest, and an... (1 Reply)
Discussion started by: fitzwilliam
1 Replies

5. Programming

read and write stdin/stdout in unix

Hi, i am using the below program to read from the standard input or to write to standard out put. i know that using highlevel functions this can be done better than what i have done here. i just want to know is there any other method by which i find the exact number of characters ( this... (3 Replies)
Discussion started by: MrUser
3 Replies

6. Shell Programming and Scripting

how to read tail -F command output in perl

Hi All, How I will read the output of the tail -F command in perl. I have text file with below contains file1.txt 1 2 3 4 $running=1; sub openLog($) { (my $log) = @_; (1 Reply)
Discussion started by: pravin27
1 Replies

7. UNIX for Dummies Questions & Answers

tar/cpio/pax read patterns from stdin

tar has the -T operand for reading patterns from a file. Is there any way to read patterns from stdin, without creating a temp file? I would like to avoid iterating over the archive repeatedly (e.g. with a loop or xargs) as this is a large archive and we're only extracting a small number of... (2 Replies)
Discussion started by: uiop44
2 Replies

8. UNIX for Dummies Questions & Answers

[SOLVED] Piping Problem

Hey, I want to create a new file (devices) with the 39th and the 40th character of the line wich is in the array line and in the file drivers. But unfortunately my try doesn't work: sed -n '$linep' drivers | cut -c 39-40 | echo >>devices Perhaps one of you can help me. Thank you! emoly ... (0 Replies)
Discussion started by: emoly
0 Replies

9. UNIX for Dummies Questions & Answers

[Solved] Tail command issue in Linux

Hello, When I am trying to use tail +13 filename.csv it is throwing an error. tail: cannot open `+13' for reading: No such file or directory and then prints last 10 lines of the file. (File is present on the path) But when i try tail -13 filename.csv it runs perfectly. Could I have... (5 Replies)
Discussion started by: AnkitSenghani
5 Replies

10. Programming

How to read extended ASCII characters from stdin?

Hi, I want to read extended ASCII characters from keyboard using c language on unix/linux. How to read extended characters from keyboard or by copy-paste in terminal irrespective of locale set in the system. I want to read the input characters from keyboard, store it in an array or some local... (3 Replies)
Discussion started by: sanzee007
3 Replies
All times are GMT -4. The time now is 12:38 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy