grep sed and a loop Post: 302521630

Sponsored Content

Top Forums UNIX for Dummies Questions & Answers grep sed and a loop Post 302521630 by agama on Thursday 12th of May 2011 12:21:22 AM

05-12-2011

Registered User

Here is a simple example of how to process a log file and save the last line for reference on the next run. It uses the entire last line from the log as the reference point, rather than a date/time stamp. Less efficient, but more accurate. I'm sure there are other ways to make this more efficient, but this might help get you started.

It does run in Kshell. Should run in bash, but for several reasons I prefer Kshell and didn't test it under bash, so beware if you want to use this as a bash script.

Code:

#!/usr/bin/env ksh


logf=$1                 # log file to suss; supplied as parameter on cmd line
pat=$2                  # pattern to search log for
this=${0##*/}           # base name of script
lastf=/tmp/$this.data   # file where we tuck our last found line away for next time

if [[ -z $logf || -z $pat ]]        # error if logfile name or pattern isn't supplied
then
    echo "usage: $0 log-file-name pattern"
    exit 1
fi

if [[ ! -s $lastf ]]    # no last run data; must check all lines from log
then
    need_all=1          # this will cause us to check for pattern straight away
fi

# parse the log file writing, to stdout, any matches that haven't been seen yet
# The last line seen is written to the last data file for next time.
#
awk  -v snarf=$need_all -v lastf=$lastf -v pat=$pat '
    BEGIN {
        if( !snarf )                # not in check all mode, must find last data
        {
            getline < lastf;
            last_data = $0;
            close( lastf );
        }
    }

    snarf {                         # need all, or found the last line from prev run
        if( match( $0, pat ) )      # if it contains the pattern, print it to stdout
            print;
        new_last = $0;              # save the last line we saw
        next;                       # go to next input line (skip remaining awk code)
    }

    {       # not snarfing yet, check to see if this is the last line we saw before
        snarf = $0 == last_data;    # start snarfing if it matches
        # tricky way of saying:
        # if( $0 == last_data)
        #     snarf = 1;
    }

    END {
        if( new_last )
            printf( "%s\n", new_last) >lastf;   # save our last observed line in the data file.
    }
' <$logf

exit $?           # return the exit code that awk returned with

The one thing this script doesn't do is to deal with the case where the log file has been rolled off. In this case the data file would be non-empty, but we'd need to start checking for the pattern at the start of the log rather than after the previous marker is encountered.

The easy solution to this is to remove the last data file when the log is rolled. If that's not possible, then additional code will be needed to detect this condition; beyond my few minutes to spend answering questions tonight -- sorry.

If you need some info about awk, this is a decent set of doc:
Awk - A Tutorial and Introduction - by Bruce Barnett
and there are other posts on this forum that link other sites.

Hope this helps.

agama

View Public Profile for agama

Find all posts by agama

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

grep in a loop

Hi , I am trying a script which takes user input userid . I am stuck how to check whether that is a valid user id or not in the audit log files. My code is : cd $CCP_AUDIT cat * > /export/home/$USR/l***/files echo "UserId:\c" read UserId #Date Function echo "DATE : \c" read xxx I...

2. UNIX for Dummies Questions & Answers

grep -v while loop

alist contain: a b c d e blist contain: a b c the code: #!/usr/bin/ksh cat blist | while read line do grep -V "$line" alist > data done

3. Shell Programming and Scripting

How to get lines started with matched strings using sed or grep for loop?

I have a huge file and want to separate it into several subsets. The file looks like: C1 C2 C3 C4 ... (variable names) 1 .... 2 .... 3 .... : 22 .... 23 .... I want to separate the huge file using the column 1, which has numbers from 1 to 23 (but there are different amount of...

4. Shell Programming and Scripting

grep within while read loop

I have a key file $ cat klist 5 N:8855 CASA VERDE ROAD :32827 :ORLAND 5 N:585 MOLLY LANE :30189 :WOODST 5 N:320 NINA ROAD :32304 :TALLAH and a data file, see example of the line below: N:RT 15 & N 7TH STREET :17837 :U SAVE I need to search by key (2nd field) from klist...

5. Shell Programming and Scripting

Using grep within a while loop

Hi all, I have the below script to get input but i cannot get grep to work. input1.txt AAAAAAAAG input2.txt >gi|184009.1| LEAFY-like |AAAAAAAAGSGGGDHLPY However, when i use grep -f input1.txt input2.txt i cannot get any output matches (note that the match is underlined). Is it...

6. Shell Programming and Scripting

Help on grep in a do while loop

So this is what I'm trying to do: I have a file called registry.txt which has a list of registry entries I want to search for. I have another file called inctrl.txt on which I want to perform the search on. Here's the example contents of registry.txt SOFTWARE\Microsoft\Security...

7. Red Hat

using grep in a while loop

Hello everybody, I have been searching it, but it seems I am unable to find the correct information, that s why I am asking you guys, hoping somebody get an idea. Here is my problem : I want a script to loop until a string is identified in a log file. Here is the script : #!/bin/sh...

8. Homework & Coursework Questions

GREP loop

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: I need to search through the users home directories for keywords, display them. The code listed below will show...

9. UNIX for Beginners Questions & Answers

Grep command in a loop

Hello - I am running a script that is outputting to a log. Let call it output.log I would like to monitor that log until the line "Build Successful" is found. I think I would need to use the grep command. How would I do that in a loop? Thanks Marty

10. Shell Programming and Scripting

Ps ax with grep in loop

Hello, I have built the following script to check if processes supplied by the argument are running or not. #!/bin/bash PROCLIST=$1 PROCESS="0" ERROR_PROCS="" IFS='+' read -ra ADDR <<< "$PROCLIST" for PROC in "${ADDR}"; do if ; then PROCESS=1 ...

LEARN ABOUT DEBIAN

quiz

QUIZ(6) 							 BSD Games Manual							   QUIZ(6)

NAME

     quiz -- random knowledge tests

SYNOPSIS

     quiz [-t] [-i file] [question answer]

DESCRIPTION

     The quiz utility tests your knowledge of random facts.  It has a database of subjects from which you can choose.  With no arguments, quiz
     displays the list of available subjects.

     The options are as follows:

     -t      Use tutorial mode, in which questions are repeated later if you didn't get them right the first time, and new questions are presented
	     less frequently to help you learn the older ones.

     -i      Specify an alternative index file.

     Subjects are divided into categories.  You can pick any two categories from the same subject.  quiz will ask questions from the first cate-
     gory and it expects answers from the second category.  For example, the command ``quiz victim killer'' asks questions which are the names of
     victims, and expects you to answer with the cause of their untimely demise, whereas the command ``quiz killer victim'' works the other way
     around.

     If you get the answer wrong, quiz lets you try again.  To see the right answer, enter a blank line.

   Index and Data File Syntax
     The index and data files have a similar syntax.  Lines in them consist of several categories separated by colons.	The categories are regular
     expressions formed using the following meta-characters:

	   pat|pat  alternative patterns
	   {pat}    optional pattern
	   [pat]    delimiters, as in pat[pat|pat]pat

     In an index file, each line represents a subject.	The first category in each subject is the pathname of the data file for the subject.  The
     remaining categories are regular expressions for the titles of each category in the subject.

     In data files, each line represents a question/answer set.  Each category is the information for the question/answer for that category.

     The backslash character (``'') is used to quote syntactically significant characters, or at the end of a line to signify that a continuation
     line follows.

     If either a question or its answer is empty, quiz will refrain from asking it.

FILES

     /usr/share/games/bsdgames/quiz  The default index and data files.

BUGS

     quiz is pretty cynical about certain subjects.

BSD
								   May 31, 1993 							       BSD

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

grep in a loop

Discussion started by: gundu

2. UNIX for Dummies Questions & Answers

grep -v while loop

Discussion started by: bobo

3. Shell Programming and Scripting

How to get lines started with matched strings using sed or grep for loop?

Discussion started by: AMBER

4. Shell Programming and Scripting

grep within while read loop

Discussion started by: migurus