Parsing Log File Based on Date & Error


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parsing Log File Based on Date & Error
# 1  
Old 11-16-2010
Parsing Log File Based on Date & Error

I'm still up trying to figure this out and it is driving me nuts.

I have a log file which has a basic format of this...
Code:
2010-10-10 22:25:42
[Error: 43059] Init block 'UA Deployment Date': Dynamic refresh of repository scope variables has failed.
[Error: 16024] The ODBC function has returned an error. The database may not be available, or the network may be down.
2010-12-10 22:25:42
[Error: 43059] Init block 'Current Usage Accelerator Refresh Date': Dynamic refresh of repository scope variables has failed.
[Error: 16023] The ODBC function has returned an error. The database may not be available, or the network may be down.
2010-12-11 22:25:42
[Error: 43059] Init block 'New Activity Interval for Asset': Dynamic refresh of repository scope variables has failed.
[Error: 16023] The ODBC function has returned an error. The database may not be available, or the network may be down.
2010-12-11 22:25:42
[Error: 43059] Init block 'New Activity Interval for Account': Dynamic refresh of repository scope variables has failed.
[Error: 16024] The ODBC function has returned an error. The database may not be available, or the network may be down.
2010-12-13 22:25:42
[Error: 43059] Init block 'New Activity Interval for Opportunity': Dynamic refresh of repository scope variables has failed.
[Error: 16024] The ODBC function has returned an error. The database may not be available, or the network may be down.

I have a date which I return from Oracle in the exact same format as this log file in a variable called $lastdate. I was able to use grep to store the log date into $logdate.

I have to go through this logfile starting at the point where $logdate > $lastdate and only return the whole error message of error 16024 or 16023(ex: [Error: 16024] The ODBC function has returned an error. The database may not be available, or the network may be down.). I'm going nuts trying to figure out how to do the > operator between dates and then only getting the 16024 or 16023 error.

This is my code thus far which obviously doesnt work very well:

Code:
tail -f $logfile |
   while read LINE; do
    if echo $LINE | grep ^20..'-'..'-'.. >/dev/null; then
        logdate=$LINE
    echo " the log date is: ========== $logdate ============="
        if ['$logdate'<'$lastdate']; then
            if echo $LINE | grep '16024' >/dev/null; then
               echo " ========= I find the error   =================================: $LINE"
            
            #    echo $LINE | cut -d "|" -f > $tempfile
            fi
        fi
#    if echo $LINE | date >/dev/null; then
#        lastrun=$LINE
#        echo " ========= Last run date is=================================: $lastrun"
#        if [$lastrun < $lastdate ]; then
#           echo " Last run date is less than last date"
#               if echo $LINE | grep ^20..'-'..'-'.. >/dev/null; then
#                linecount=$(($linecount + 1))
#                echo " I find the line: $LINE"
#                      echo $LINE | cut -d "|" -f > $tempfile
#                         breakdown=$(echo $LINE | cut -d "]" -f 1)
#                 echo "$LINE" >> $tempfile
#                    echo "The break down is:   $breakdown" 
#            fi
        else
            echo " Last run date is larger than last date"
        fi
#    fi
exit 0
done<$logfile


Last edited by Scott; 11-16-2010 at 03:01 AM..
# 2  
Old 11-16-2010
Try...
Code:
 awk -v l="$lastdate" '/^....-..-../&&$0>l{d=$0}d&&/Error: 1602[34]/{print d,$0}' file1


Last edited by Ygor; 11-16-2010 at 04:00 AM..
# 3  
Old 11-16-2010
Code:
awk -v l="$lastdate" '($0==l){n=1}(n==1)&&(/Error: 1602[34]/||/^....-/)' input


Last edited by ctsgnb; 11-16-2010 at 05:18 AM.. Reason: oops
# 4  
Old 11-16-2010
Thank you so much for these - I am very grateful. One more question. How would I store the date of the matching error in the logfile ($logdate) along with the error message?

After finding a matching error I need to Insert into an oracle table like this:

Code:
inserter=`sqlplus -s <<EOF $USERID/$PASSW@$DB
set heading off
set pagesize 0
set tab off
INSERT INTO DB.ERROR(ERR_DATE, ERR_MSG)
VALUES
($logdate,$errormsg);
exit
EOF`


thank you so much for the help though, lifesavers.

---------- Post updated at 01:12 PM ---------- Previous update was at 10:29 AM ----------

So I got this to work perfectly thanks to your help
Code:
awk -v l="$lastdate" '/^....-..-../&&$0>l{d=$0}d&&/Error: 1602[34]/{print d,$0}' $logfile

I want to put this in a DO WHILE loop though and insert the result into a variable so that I can call SQL PLUS to insert into an oracle table. So heres the syntax which i'm trying to do but doesn't seem to be working too well

Code:
while read LINE; do
DATE=`echo $LINE | awk -v l="$lastdate" '/^....-..-../&&$0>l{d=$0}d&&/Error: 1602[34]/{print d}'`
MSG=`echo $LINE | awk -v l="$lastdate" '/^....-..-../&&$0>l{d=$0}d&&/Error: 1602[34]/{print $0}'`
#here i will call sql plus and insert above variables into database
done<$logfile

# 5  
Old 11-16-2010
Try something like...
Code:
$ export lastdate='2010-12-11 22:25:42'
$ export fmt="INSERT INTO DB.ERROR(ERR_DATE, ERR_MSG) \nVALUES \n('%s','%s');\n"
$ awk -v l="$lastdate" -v f="$fmt" '/^....-..-../&&$0>l{d=$0}d&&/Error: 1602[34]/{printf(f,d,$0)}' file1
INSERT INTO DB.ERROR(ERR_DATE, ERR_MSG)
VALUES
('2010-12-13 22:25:42','[Error: 16024] The ODBC function has returned an error. The database may not be available, or the network may be down.');
$

...redirect to a file and start in sqlplus.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Print Error in Console and both Error & Output in Log file - UNIX

I am writing a shell script with 2 run time arguments. During the execution if i got any error, then it needs to redirected to a error file and in console. Also both error and output to be redirected to a log file. But i am facing the below error. #! /bin/sh errExit () { errMsg=`cat... (1 Reply)
Discussion started by: sarathy_a35
1 Replies

2. UNIX for Dummies Questions & Answers

Log file - Delete duplicate line & keep last date

Hello All ! I need your help on this case, I have a csv file with this: ITEM105;ARI FSR;2016-02-01 08:02;243 ITEM101;ARI FSR;2016-02-01 06:02;240 ITEM032;RNO TLE;2016-02-01 11:03;320 ITEM032;RNO TLE;2016-02-02 05:43;320 ITEM032;RNO TLE;2016-02-01 02:03;320 ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies

3. Shell Programming and Scripting

Need Script to ZIP/SAVE & then DELETE Log file & send a mail conformation for any error

ENVIROMENT Linux: RHEL 6.4 Log Path: /usr/iplanet/servers/https-company/logs Log Format: user.log.03-15-2015 I have log4j log rotation enabled rotating files on a daily basis. The rotated logs are NOT compressed & are taking up too much space. I need a script that will run daily that... (1 Reply)
Discussion started by: admin_job_admin
1 Replies

4. UNIX for Dummies Questions & Answers

Copy log based on from-date and to-date

Hi all, i go a customer support requirement where i need to scan several files based on from/to date like 1-oct to 2-oct please help... (3 Replies)
Discussion started by: AbhiJ
3 Replies

5. Shell Programming and Scripting

List Files Based On Time & Date

Hi All, I am using HP Unix. I want to list files which are created 5 minutes before on the same day as well as before today's date. I checked all the forums and the commands provided there does not work on HP Unix. Can you please help me on this? Your help is highly aprreciated. Thanks and... (3 Replies)
Discussion started by: angshuman
3 Replies

6. Shell Programming and Scripting

Delete log file entries based on the Date/Timestamp within log file

If a log file is in the following format 28-Jul-10 ::: Log message 28-Jul-10 ::: Log message 29-Jul-10 ::: Log message 30-Jul-10 ::: Log message 31-Jul-10 ::: Log message 31-Jul-10 ::: Log message 1-Aug-10 ::: Log message 1-Aug-10 ::: Log message 2-Aug-10 ::: Log message 2-Aug-10 :::... (3 Replies)
Discussion started by: vikram3.r
3 Replies

7. Shell Programming and Scripting

Finding & Moving Oldest File by Parsing/Sorting Date Info in File Names

I'm trying to write a script that will look in an /exports folder for the oldest export file and move it to a /staging folder. "Oldest" in this case is actually determined by date information embedded in the file names themselves. Also, the script should only move a file from /exports to... (6 Replies)
Discussion started by: nikosey
6 Replies

8. Shell Programming and Scripting

Processing a log file based on date/time input and the date/time on the log file

Hi, I'm trying to accomplish the following and would like some suggestions or possible bash script examples that may work I have a directory that has a list of log files that's periodically dumped from a script that is crontab that are rotated 4 generations. There will be a time stamp that is... (4 Replies)
Discussion started by: primp
4 Replies

9. Shell Programming and Scripting

parsing a system log file via the 'date' command

Hello, I'm trying to update some scripts here that parse our system logs daily. They report information just fine... but they just report too much info. Specifically, if there's been some failed login attempts on several different days (say Monday and Tuesday), when I get the report from... (5 Replies)
Discussion started by: cjones
5 Replies

10. UNIX for Dummies Questions & Answers

Inserting Date&Time Stamp In Existing Log File

I am trying to insert a line with a date stamp in a file that is used to monitor activity in one of our directories. By doing this, I want to grep that file each day and go to the last entry for each time a error occurred and pull all errors generated if any exist. If error exists I want that error... (3 Replies)
Discussion started by: shephardfamily
3 Replies
Login or Register to Ask a Question