Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Issue with awk script parsing log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Issue with awk script parsing log file
# 1  
Old 05-23-2014
Issue with awk script parsing log file
Hello All,
I am trying to parse a log file and i got this code from one of the good forum colleagues, However i realised later there is a problem with this awk script, being naive to awk world wanted to see if you guys can help me out.

AWK script:

Code:
awk '$1 ~ "^WRITER_" {p=1;next} p&&/X_fc_Loan/{p++;next}; p==2 && NF>=6 && $6 !~/[^0-9]/{print $6;p=0}' s_GenerateXMLDataFile.log.467.txt


The log file (s_GenerateXMLDataFile.log.467.txt) is an Informatica log which would be populated with count of records being populated in each table as the job progresses which is represented for example here as X_fc_Loan & X_fc_Customer.

This awk script parses the log file and prints the 6th column number below the X_fc_Loan line however when it encounters the line "WRT_8044 No data loaded for this target", it is printing the 6th column number from the line below the X_fc_Customer, which i don't want. Excerpt from the log file of this scenario is below, I just want the count or number below the line where it has X_fc_Loan. How do i restrict this and print only number below X_fc_Loan from the other lines where the counts/numbers are actually printed in log???


Excerpt From Log File:

Code:
WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Loan (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8044 No data loaded for this target


WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Customer (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 501600     Applied: 0          Rejected: 0          Affected: 0


I came up with this below code but its printing nothing, please throw some light appreciate your help.

Code:
awk '$1 ~ "^WRITER_" {p=1;next} p&&/X_fc_Loan/{p++;next}; p==2 && NF>=6 && $1 !~/[^WRT_8044]/ && $6 !~/[^0-9]/{print $6;p=0}' s_GenerateXMLDataFile.log.467


Thank you.
s_GenerateXMLDataFile.log.467.txt (21.9 KB) 
Last edited by Ariean; 05-23-2014 at 10:28 AM.. Reason: attaching log file
# 2  
Old 05-24-2014
Quote:
Originally Posted by Ariean
Hello All,
I am trying to parse a log file and i got this code from one of the good forum colleagues, However i realised later there is a problem with this awk script, being naive to awk world wanted to see if you guys can help me out.

AWK script:

Code:
awk '$1 ~ "^WRITER_" {p=1;next} p&&/X_fc_Loan/{p++;next}; p==2 && NF>=6 && $6 !~/[^0-9]/{print $6;p=0}' s_GenerateXMLDataFile.log.467.txt


The log file (s_GenerateXMLDataFile.log.467.txt) is an Informatica log which would be populated with count of records being populated in each table as the job progresses which is represented for example here as X_fc_Loan & X_fc_Customer.

This awk script parses the log file and prints the 6th column number below the X_fc_Loan line however when it encounters the line "WRT_8044 No data loaded for this target", it is printing the 6th column number from the line below the X_fc_Customer, which i don't want. Excerpt from the log file of this scenario is below, I just want the count or number below the line where it has X_fc_Loan. How do i restrict this and print only number below X_fc_Loan from the other lines where the counts/numbers are actually printed in log???


Excerpt From Log File:

Code:
WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Loan (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8044 No data loaded for this target


WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Customer (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 501600     Applied: 0          Rejected: 0          Affected: 0


I came up with this below code but its printing nothing, please throw some light appreciate your help.

Code:
awk '$1 ~ "^WRITER_" {p=1;next} p&&/X_fc_Loan/{p++;next}; p==2 && NF>=6 && $1 !~/[^WRT_8044]/ && $6 !~/[^0-9]/{print $6;p=0}' s_GenerateXMLDataFile.log.467


Thank you.
The code marked in red above is looking for lines in your input file with a 1st field that starts with the string "WRITER_" before anything can be printed. I don't see any lines in your input file that match that string, so there is no chance that anything will be output.
# 3  
Old 05-24-2014
@Ariean

The issue is that you say:
Quote:
[...] the log file and prints the 6th column number below the X_fc_Loan line [...]
However, in the attached file, X_fc_Loan appears all over in context that it doesn't provide a following line with meaningful data, in fact, if we collect out all the lines right after X_fc_Loan in that sample file, only one line shows a potential match:

Code:
WRT_8044 No data loaded for this target
WRT_8044 No data loaded for this target
WRT_8044 No data loaded for this target
WRT_8044 No data loaded for this target
WRT_8044 No data loaded for this target
WRT_8044 No data loaded for this target
WRT_8044 No data loaded for this target
WRT_8038 Inserted rows - Requested: 100320     Applied: 0          Rejected: 0          Affected: 0

If that's correct test this snippet and confirm that's what you want

Code:
awk '$3 ~ /X_fc_Loan/ && $6 == "[XMLTgt_FCSLOANS_Ver25_Norm])" {R=NR;next}(NR - R) == 1 && $5=="Requested:" {print $6}'

---------- Post updated 05-24-14 at 12:09 AM ---------- Previous update was 05-23-14 at 11:45 PM ----------

If that works let's make it cleaner:

Code:
awk '$3 ~ /X_fc_Loan/ && $6 == "[XMLTgt_FCSLOANS_Ver25_Norm])" {getline; if($5=="Requested:")print $6}'

Last edited by Aia; 05-24-2014 at 02:54 AM..
This User Gave Thanks to Aia For This Post:
Ariean
# 4  
Old 05-28-2014
Thank you it worked. one more additional request would it be possible
Scenario1: to print unique numbers.
Scenario2: print only last instance.

Code:
WRITER_1_*_1> WRT_8161
TARGET BASED COMMIT POINT  Tue May 27 15:35:09 2014
===================================================

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Provider (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 1          Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Institution (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 17         Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Loan (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 1203840    Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Customer (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 583985     Applied: 0          Rejected: 0          Affected: 0

WRITER_1_*_1> WRT_8161
TARGET BASED COMMIT POINT  Tue May 27 15:35:10 2014
===================================================

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Provider (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 1          Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Institution (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 17         Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Loan (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 1203840    Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Customer (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 583985     Applied: 0          Rejected: 0          Affected: 0

WRITER_1_*_1> WRT_8161
TARGET BASED COMMIT POINT  Tue May 27 15:35:14 2014
===================================================

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Provider (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 1          Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Institution (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 17         Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Loan (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 1304160    Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Customer (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 583985     Applied: 0          Rejected: 0          Affected: 0

your awk script would return below output for the pasted excerpt from log file.

Output:
Code:
1203840
1203840
1304160

Expected Output Scenario1:
Code:
1203840
1304160

Expected Output Scenario2:
Code:
1304160

---------- Post updated 05-28-14 at 02:06 PM ---------- Previous update was 05-27-14 at 04:56 PM ----------

Appreciate if somebody can help me here. Thank you.
# 5  
Old 05-28-2014
Only unique:
Code:
awk '$3 ~ /X_fc_Loan/ && $6 == "[XMLTgt_FCSLOANS_Ver25_Norm])" {getline; if($5=="Requested:")a[$6]} END { for (k in a) print k }' file

Last instance as a whole:
Code:
awk '$3 ~ /X_fc_Loan/ && $6 == "[XMLTgt_FCSLOANS_Ver25_Norm])" {getline; if($5=="Requested:")l=$6} END { print l }' file

Last instance for unique:
Code:
awk '$3 ~ /X_fc_Loan/ && $6 == "[XMLTgt_FCSLOANS_Ver25_Norm])" {getline; if($5=="Requested:")a[$6]=$6; l=$6} END { print a[l] }' file

Last edited by Aia; 05-28-2014 at 05:59 PM..
This User Gave Thanks to Aia For This Post:
Ariean
# 6  
Old 05-29-2014
I came across another challenge, how do i keep reading the log file and keep printing the record count using the awk script you suggested and exit from the script when the job process writes the phraseTM_6020 into the log file as shown below.

excerpt from log:
Code:
DIRECTOR> TM_6020 Session [s_GenerateXMLDataFile] completed at [Thu May 29 16:08:03 2014].


I came up with below script but not sure how to embed exit logic.

Code:
#!/bin/bash -x

while true
do
COUNT=`awk '$3 ~ /X_fc_Loan/ && $6 == "[XMLTgt_FCSLOANS_Ver25_Norm])" {getline; if($5=="Requested:")l=$6} END { print l }' s_GenerateXMLDataFile.log`

if [ -z "$COUNT" ]
then
        COUNT="0"
fi

if [ "$COUNT1" != "$COUNT" ]
then
        echo "$COUNT"
fi
COUNT1="$COUNT"

`awk '$2 == "TM_6020" {exit}' s_GenerateXMLDataFile.log`
done

# 7  
Old 05-29-2014
Ariean, I don't get it.

Let's take the obvious first.
while true is an infinite loop which it will never exit unless you stop it with the keyword break and it is independent of the awk invocation

Code:
while true
do
COUNT=`awk '$3 ~ /X_fc_Loan/ && $6 == "[XMLTgt_FCSLOANS_Ver25_Norm])" {getline; if($5=="Requested:")l=$6} END { print l }' s_GenerateXMLDataFile.log`
.
.
.
done

Unless, the content of the file s_GenerateXMLDataFile.log has changed within an iteration cycle of the loop, its return is always the same no matter how many times it loops. Waste of CPU.

The return of awk invocation is an string extraction, I don't know if in the context of the file is a count or not, but that's what you asked previously.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parsing a log file and creating a report script

The log file is huge and lot of information, i would like to parse and make a report . below is the log file looks like: REPORT DATE: Mon Aug 10 04:16:17 CDT 2017 SYSTEN VER: v1.3.0.9 TERMINAL TYPE: prod SYSTEM: nb11cu51 UPTIME: 04:16AM up 182 days 57 mins min MODEL, TYPE, and SN:... (8 Replies)
Discussion started by: amir07
8 Replies

2. Shell Programming and Scripting

Issue in awk parsing under while loop

Hi I am trying to parse a grep output using awk. It works fine individually and not working under the loop with variable name assigned. cat > file.txt dict=/dictr/abcd/d1/wq:/dictr/abcd/d2/wq:/dictr/abcd/d3/wq: sample tried code Nos=`grep -w "dict" file.txt | awk -F"=" '{print... (10 Replies)
Discussion started by: ananan
10 Replies

3. Shell Programming and Scripting

Shell script not parsing complete file using AWK

Hi, I have shell script which will read single edi document and break data between ST & SE to separate files.Below example should create 3 separate files. I have written script with the below command and it is working fine for smaller files. awk -F\| -vt=`date +%m%d%y%H%M%S%s` \ ... (2 Replies)
Discussion started by: prasadm
2 Replies

4. Shell Programming and Scripting

Parsing out access.log with awk and grep

In part of my script I use awk to pull out the urls. awk '{print $8}' then I take them and send them to grep.` Some of them are straight .com/ or .org or whatever (address bar entries), while others are locations of images, js, etc. I'm trying to only pull any line that ends with .com/... (11 Replies)
Discussion started by: druisgod
11 Replies

5. Shell Programming and Scripting

Script for Parsing Log File

Working on a script that inputs an IP, parses and outputs to another file. A Sample of the log is as follows: I need the script to be able to input IP and print the data in an output file in the following format or something similar: Thanks for any help you can give me! (8 Replies)
Discussion started by: Winsarc
8 Replies

6. Shell Programming and Scripting

parsing issue with edi file

Hello, We have edi files we need to do some extra parsing on. There is a line that shows up that looks like this: GE|8,845|000000000 We need to parse the file, find the line ( that begins with GE "^GE" ), and remove the comma(s). What is the easiest way to do that ? I know I can grab... (5 Replies)
Discussion started by: fwellers
5 Replies

7. Shell Programming and Scripting

Log file issue within script

Hi, I have a script where it does several tasks and 3 of them being SQLPLUS activity. Within these SQLPLUS sessions, I have a spool file going but what ever is going on within each SQLPLUS session I would like to write it to my main log file where everything else is running. sqlplus -s <<... (2 Replies)
Discussion started by: ramangill
2 Replies

8. UNIX for Dummies Questions & Answers

Script for parsing details in a log file to a seperate file

Hi Experts, Im a new bee for scripting, I would ned to do the following via linux shell scripting, I have an application which throws a log file, on each action of a particular work with the application, as sson as the action is done, the log file would vanish or stops updating there, the... (2 Replies)
Discussion started by: pingnagan
2 Replies

9. Shell Programming and Scripting

Help with script parsing a log file

I have a large log file, which I want to first use grep to get the specific lines then send it to awk to print out the specific column and if the result is zero, don't do anything. What I have so far is: LOGDIR=/usr/local/oracle/Transcription/log ERRDIR=/home/edixftp/errors #I want to be... (3 Replies)
Discussion started by: mevasquez
3 Replies

10. Shell Programming and Scripting

Shell script for parsing 300mb log file..

am relatively new to Shell scripting. I have written a script for parsing a big file. The logic is: Apart from lot of other useless stuffs, there are many occurances of <abc> and corresponding </abc> tags. (All of them are properly closed) My requirement is to find a particular tag (say... (3 Replies)
Discussion started by: gurpreet470
3 Replies
Login or Register to Ask a Question