Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Parse A Log File

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parse A Log File
# 8  
Old 05-09-2014
Apologies for the confusion, interesting though looks like when i downloaded the log (which i attached) from Informatica client tools it is showing like below with timestamp and some extra info.
Code:
2014-05-08 09:33:14 : INFO : (29223 | WRITER_1_*_1) : (IS | Dev_Int) : Dev_Node_01 : WRT_8161 : 
TARGET BASED COMMIT POINT  Thu May 08 09:33:13 2014
===================================================

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Provider (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 1          Applied: 0          Rejected: 0          Affected: 0         

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Institution (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 17         Applied: 0          Rejected: 0          Affected: 0         

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Loan (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 700728     Applied: 0          Rejected: 0          Affected: 0         

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Customer (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 583985     Applied: 0          Rejected: 0          Affected: 0

but when i vi the log in Linux server it is showing like below.

Code:
WRITER_1_*_1> WRT_8161
TARGET BASED COMMIT POINT  Thu May 08 09:33:13 2014
===================================================

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Provider (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 1          Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Institution (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 17         Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Loan (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 700728     Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Customer (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 583985     Applied: 0          Rejected: 0          Affected: 0

since i would be executing my script in Linux server, we should choose the vi version of the log output. When i execute your peice of code against the log file in Linux, the output i got is as shown in my previous post, we need to get rid of the word 'this' in the output.

I actually winscpied the log file from Linux server and attached it for your reference please take a look.

Thank you.
s_GenerateXMLDataFile.txt (38.0 KB) 
Last edited by Ariean; 05-09-2014 at 01:17 PM..
# 9  
Old 05-09-2014
a simplified version - see if it helps - the samples are kind of small in scope...:
Code:
awk '$1 ~ "^WRITER_" {p=1;next} p&&/X_fc_Loan/{p++;next}; p==2{print $6;p=0}' myFile

# 10  
Old 05-09-2014
I actually winscpied the log file from Linux server and attached it for your reference please take a look.
# 11  
Old 05-09-2014
ok, that's better - a lil bit more of error checking:
Code:
awk '$1 ~ "^WRITER_" {p=1;next} p&&/X_fc_Loan/{p++;next}; p==2 && NF>=6 && $6 !~/[^0-9]/{print $6;p=0}' s_GenerateXMLDataFile.txt

Last edited by vgersh99; 05-09-2014 at 01:36 PM..
This User Gave Thanks to vgersh99 For This Post:
Ariean
# 12  
Old 05-09-2014
Thank you it worked, would you mind explaining splitting your awk script into chunks and how you are achieving the desired output.
# 13  
Old 05-09-2014
Code:
awk '
# if the first field starts with WRITER_, set the flag p to 1 and skip to processing the next line
$1 ~ "^WRITER_" {p=1;next}

# if flag p is not 0 AND there's a string X_fc_Loan on current line, increment flag p and skipp to processing the next line
p&&/X_fc_Loan/{p++;next}

# if flag p is equal to 2 (already saw WRITER_ and X_fv_Loan lines) AND NumberOfFields (NF) is greater than 6 AND the 6-th field contains ONLY numbers (the last 2 conditions filter out non-complete lines AND lines where the 6-th field is not numeric), print the 6-th field and reset flag p to 0.
# Example of the invalid lines:
#WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Loan (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
#WRT_8044 No data loaded for this target

p==2 && NF>=6 && $6 !~/[^0-9]/{print $6;p=0}
' s_GenerateXMLDataFile.txt

not that we have it all squared away, could I get a discount for a new Bimmer? Smilie
This User Gave Thanks to vgersh99 For This Post:
Ariean
# 14  
Old 05-22-2014
Hello,
For example if the log file has below lines, it is printing the value beneath the X_fc_Customer when X_fc_Loan has "WRT_8044 No data loaded for this target" which it shouldn't have printed, how do i restrict to print only the numbers below X_fc_Loan??

FYI...I modified your command little bit as shown below (print $6 instead of print $12)

Code:
awk '$1 ~ "^WRITER_" {p=1;next} p&&/X_fc_Loan/{p++;next}; p==2 && NF>=6 && $6 !~/[^0-9]/{print $6;p=0}' s_GenerateXMLDataFile.log.467


logfile excerpt:
Code:
WRITER_1_*_1> WRT_8168 End loading table [XMLTgt_FCSLoans25::X_fc_Customer] at: Tue May 20 12:35:24 2014
WRITER_1_*_1> WRT_8141
Commit on end-of-data  Tue May 20 12:35:24 2014
===================================================

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Provider (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 1          Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Institution (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 17         Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Loan (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8044 No data loaded for this target


WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Customer (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 583985     Applied: 0          Rejected: 0          Affected: 0

WRITER_1_*_1> WRT_8165 TIMEOUT BASED COMMIT POINT
READER_1_3_1> RR_4050 First row returned from database to reader : (Tue May 20 12:36:54 2014)
WRITER_1_*_1> WRT_8167 Start loading table [XMLTgt_FCSLoans25::X_fc_Loan] at: Tue May 20 12:36:54 2014
WRITER_1_*_1> WRT_8161
TARGET BASED COMMIT POINT  Tue May 20 12:36:56 2014
===================================================

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Provider (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 1          Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Institution (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 17         Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Loan (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 100320     Applied: 0          Rejected: 0          Affected: 0

WRT_8036 Target: XMLTgt_FCSLoans25::X_fc_Customer (Instance Name: [XMLTgt_FCSLOANS_Ver25_Norm])
WRT_8038 Inserted rows - Requested: 583985     Applied: 0          Rejected: 0          Affected: 0
                                                                                                                                           467,1         Bot

Output:
Code:
200640
300960
401280
501600
583985
100320

It should not have printed the numbers above 100320 which belongs to X_fc_Customer, please help
Thank you.

---------- Post updated 05-22-14 at 11:30 AM ---------- Previous update was 05-21-14 at 05:11 PM ----------

Can some one please throw any ideas. i am doing the below but unsuccessfull.

Code:
awk '$1 ~ "^WRITER_" {p=1;next} p&&/X_fc_Loan/{p++;next}; p==2 && NF>=6 && $1 !~/[^WRT_8044]/ && $6 !~/[^0-9]/{print $6;p=0}' s_GenerateXMLDataFile.log.467

Last edited by Ariean; 05-21-2014 at 09:21 PM..
This User Gave Thanks to Ariean For This Post:
vgersh99
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Parse apache log file with three different time formats

Hi, I want to parse below file and Write a function to extract the logs between two given timestamp. Apache (Unix) Log Samples - MonitorWare The challenge here is there are three date and time format. First :- 07/Mar/2004:16:05:49 Second :- Sun Mar 7 16:02:00 2004 Third :- 29-Mar... (6 Replies)
Discussion started by: sahil_shine
6 Replies

2. Shell Programming and Scripting

Shell Script | Parse log file after a given date and time stamp

I am developing one script which will take log file name, output file name, date, hour and minute as an argument and based on these inputs, the script will scan and capture all the error(s) that have been triggered from a given time. Example: script should capture all the error after 13:50 on Jan... (2 Replies)
Discussion started by: ROMA3
2 Replies

3. Shell Programming and Scripting

Parse log file to insert into database

I have a log file that's created daily by this command: sar -u 300 288 >> /var/log/usage/$(date "+%Y-%m-%d")_$(hostname)_cpu.log It that contains data like this: Linux 3.16.0-4-amd64 (myhostname) 08/15/2015 _x86_64_ (1 CPU) 11:34:17 PM CPU %user %nice ... (12 Replies)
Discussion started by: unplugme71
12 Replies

4. Shell Programming and Scripting

Check/Parse log file's lines using time difference/timestamp

I was looking at this script which outputs the two lines which differs less than one sec. #!/usr/bin/perl -w use strict; use warnings; use Time::Local; use constant SEC_MILIC => 1000; my $file='infile'; ## Open for reading argument file. open my $fh, "<", $file or die "Cannot... (1 Reply)
Discussion started by: cele_82
1 Replies

5. Shell Programming and Scripting

Sed to parse log file

Hi all, thanks for reading the post. I'm trying to parse hundreds of log files in a directory. One log file looks similar to below: Investigator : Jim_Foo Custodian : Jim_Foo-HDD1-FOO-1234 Export Path : N:\FOO-1234\Foo_Foo Compute MD5 : No File List Only: No Extensions Selected:... (4 Replies)
Discussion started by: chipperuga
4 Replies

6. Shell Programming and Scripting

Parse the log file

./abc.sh started at Sun Oct 24 06:42:04 PDT 2010 Message: ======= Summary Report of NAME count ----------------------------------------------------------------- Below is the output of the SQL query :- NAME COUNT... (2 Replies)
Discussion started by: sandy1028
2 Replies

7. Shell Programming and Scripting

parse a log file and remember last line

Hi all: I'm working on a HPUX 11.23 system and I am needing to parse a tomcat-jakarta log file for memory use. Getting the desired data is easy, assuming the log file does not grow. This file grows constantly and I want to check it q 5 min. The next check will pick up from where it left off 5... (4 Replies)
Discussion started by: raggmopp
4 Replies

8. Shell Programming and Scripting

Help w/ script to read file and parse log message

Hi, I am working on the script to parsing the specific message like "aaaa" in multiple log files like N1-***,N2-***,N3-***... The script is to find the list of lof files which contains the message "aaaa" and export the list into excel filE. Can anyone give help? Thanks (2 Replies)
Discussion started by: shyork2001
2 Replies

9. Shell Programming and Scripting

Piping tail to awk to parse a log file

Hello all, I've got what I'm pretty sure is a simple problem, but I just can't seem to work past it. I'm trying to use awk to pretty up a log file, and calculate a percentage. The log file looks like this: # tail strtovrUsage 20090531-18:15:45 RSreq - 24, RSsuc - 24, RSrun - 78, RSerr -... (4 Replies)
Discussion started by: DeCoTwc
4 Replies

10. Shell Programming and Scripting

Parse out known messages from a log file

I am looking for a script to do the following. I have a large log file that contains hundreds of warnings, a lot of which can be ignored. The tool doesn't allow me to suppress it, so I like to parse it out from the log file and isolate just the new messages/warnings, based on an exception file. ... (12 Replies)
Discussion started by: cdn2008
12 Replies
Login or Register to Ask a Question