Parsing a log file to cut off some parts


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parsing a log file to cut off some parts
# 1  
Old 01-10-2017
Parsing a log file to cut off some parts

Dear all,

I would like to use SQL's log file to extract information from it.

This file can include four different types of instruction with the number of lines involved for each of them:

-> (1) "INSERT" instruction with the number of lines inserted
-> (2) "UPDATE" instruction with the number of lines updated
-> (3) "DELETE" instruction with the number of lines deleted
-> (4) "COLLECT STATISTICS" instruction with the number of lines involved.

The number of lines involved for "UPDATE" and "COLLECT STATISTICS" has exactly the same appearance in the log file:

-> for "COLLECT STATISTICS" instruction (check "Update completed" below)

Code:
COLLECT STATISTICS           COLUMN (STD_CDH_ID)
                           , COLUMN (PARTITION)
                           , COLUMN (METDVERS_OID_ID)
         -- AXE ITEMGROUP
                           , COLUMN (ITEMGROUP_OID_CD)
                           , COLUMN (STD_CDH_ID, METDVERS_OID_ID, ITEMGROUP_OID_CD)
                           -- AXE VISIT
                           , COLUMN (VISIT_OID_CD)
                           , COLUMN (STD_CDH_ID, METDVERS_OID_ID, VISIT_OID_CD)
         -- AXE FORM
                           , COLUMN (FORM_OID_CD)
                           , COLUMN (STD_CDH_ID, METDVERS_OID_ID, FORM_OID_CD)
         -- AXE SITE
                           , COLUMN (SITE_OID_CD)
                           , COLUMN (STD_CDH_ID, METDVERS_OID_ID, SITE_OID_CD)
       -- AXE SUBJCT
                           , COLUMN (SUBJCT_KEY_ID)
                           , COLUMN (STD_CDH_ID, METDVERS_OID_ID, SUBJCT_KEY_ID)

ON                           CDH_STGN_DEV1.RECRD_SHREDDED
;

 *** Update completed. 14 rows changed.
 *** Total elapsed time was 4 seconds.

-> for "UPDATE" instruction (check "Update completed" below)

Code:
UPDATE
 TGT
FROM
 CDH_REJT_DEV1.RAVE_LNKITEMGROUPITEM TGT
, CDH_STGN_DEV1.LNKITEMGROUPITEM SRC
SET
 LNKITEMGROUPITEM_REJTDELTD_TS = CURRENT_TIMESTAMP(0)
WHERE
 TGT.STD_CDH_ID   = SRC.STD_CDH_ID
AND TGT.METDVERS_OID_ID  = SRC.METDVERS_OID_ID
AND TGT.ITEM_OID_CD   = SRC.ITEM_OID_CD
AND TGT.LNKITEMGROUPITEM_REJTDELTD_TS IS NULL
;

 *** Update completed. No rows changed.
 *** Total elapsed time was 1 second.



The problem I face is the following: I am not interested by the number of lines involved in the "COLLECT STATISTICS" instruction and I am not able to recognize from the log file what kind of expression - "COLLECT STATISTICS" or "UPDATE" - produced "Update completed" in the log file !

So, I thought to cut off from the log file the multi-line record starting with "COLLECT STATISTICS" and ending with "changed." ... but I failed, I failed, I failed !

Thanks in advance for the attention you pay to my request and obviously, any suggestion over welcomed,

Didier.
# 2  
Old 01-10-2017
Not sure I understand your request. Completing it with sample input data and the relevant code snippet from your failing attempts might help to come to a close interpretation.
# 3  
Old 01-10-2017
Thanks a lot for your reply RudiC,

for instance, below is the whole content of the log file MyLogFile:

Code:
UPDATE
 TGT
FROM
 CDH_REJT_DEV1.RAVE_LNKITEMGROUPITEM TGT
, CDH_STGN_DEV1.LNKITEMGROUPITEM SRC
SET
 LNKITEMGROUPITEM_REJTDELTD_TS = CURRENT_TIMESTAMP(0)
WHERE
 TGT.STD_CDH_ID   = SRC.STD_CDH_ID
AND TGT.METDVERS_OID_ID  = SRC.METDVERS_OID_ID
AND TGT.ITEM_OID_CD   = SRC.ITEM_OID_CD
AND TGT.LNKITEMGROUPITEM_REJTDELTD_TS IS NULL
;

 *** Update completed. No rows changed.
 *** Total elapsed time was 1 second.

COLLECT STATISTICS           COLUMN (STD_CDH_ID)
                           , COLUMN (PARTITION)
                           , COLUMN (METDVERS_OID_ID)
         -- AXE ITEMGROUP
                           , COLUMN (ITEMGROUP_OID_CD)
                           , COLUMN (STD_CDH_ID, METDVERS_OID_ID, ITEMGROUP_OID_CD)
                           -- AXE VISIT
                           , COLUMN (VISIT_OID_CD)
                           , COLUMN (STD_CDH_ID, METDVERS_OID_ID, VISIT_OID_CD)
         -- AXE FORM
                           , COLUMN (FORM_OID_CD)
                           , COLUMN (STD_CDH_ID, METDVERS_OID_ID, FORM_OID_CD)
         -- AXE SITE
                           , COLUMN (SITE_OID_CD)
                           , COLUMN (STD_CDH_ID, METDVERS_OID_ID, SITE_OID_CD)
       -- AXE SUBJCT
                           , COLUMN (SUBJCT_KEY_ID)
                           , COLUMN (STD_CDH_ID, METDVERS_OID_ID, SUBJCT_KEY_ID)

ON                           CDH_STGN_DEV1.RECRD_SHREDDED
;

 *** Update completed. 14 rows changed.
 *** Total elapsed time was 2 seconds.

In that log file, we have two kinds of clause:
(1) an UPDATE clause (the first one)
(2) a "COLLECT STATISTICS" clause (the second one).

I want to know how many records were implied in the UPDATE clause only.

Using the following shell command, I got the two records output:

Code:
grep "Update completed" MyLogFile:
 *** Update completed. No rows changed.
 *** Update completed. 14 rows changed.

but only the first record is necessary to me (it allows to know the UPDATE clause did not have any effect: "No rows changed"), the second one is related to the "COLLECT STATISTICS" clause and I do not care about the impact of that clause.

The problem I face is the output is exactly the same for the two clauses: I am not able to link the first record from the grep command to the "UPDATE" clause, and the second record to the "COLLECT STATISTICS" !

So I thought to perform a treatment on "MyLogFile", before issuing the grep command, to delete all the information linked with the "COLLECT STATISTICS" clause in order to obtain only the record linked with UPDATE clause when performing "grep" !

But I do not know how to perform that operation ....
# 4  
Old 01-10-2017
something along these lines:
Code:
awk '
/^UPDATE/ {type=1;next}
type==1 && /Update completed/ { print "Updated rows: " (($4=="No")?0:$4); type=0}
' myFile

This User Gave Thanks to vgersh99 For This Post:
# 5  
Old 01-12-2017
Wrench

Hi - I may be missing the point here, but if it is only the first update line that you are interested in, you could simply do the following :-

grep "Update completed" MyLogFile | head -1
Is this any good to you.
This User Gave Thanks to Chrismcq For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Getting various parts from the log

I am dealing with some app log, see example below: 22:16:13.601 ClientSession(905)--Connection(5)--SELECT GETDATE() 22:16:13.632 ClientSession(158)--Connection(5)--SELECT 1 22:16:13.632 ClientSession(848)--Connection(6735)--SELECT 1 So far I needed to collect certain column from it, such as... (3 Replies)
Discussion started by: migurus
3 Replies

2. Shell Programming and Scripting

HELP on parsing this log file

Hi, I have a log file that looks like below and I am wanting to know if there is a better way of parsing it from how I am doing it right now. I am looking for when an application service is OFFLINE and ONLINE. This log file is getting written into every 30 minutes ... (1 Reply)
Discussion started by: newbie_01
1 Replies

3. Shell Programming and Scripting

Incrementing parts of ten digits number by parts

I have number in file which contains date and serial number: 2013101000. The last two digits are serial number (00). So maximum of serial number is 100. After reaching 100 it becomes 00 with incrementing 10 which is day with max 31. after reaching 31 it becomes 00 and increments 10... (31 Replies)
Discussion started by: Natalie
31 Replies

4. Shell Programming and Scripting

Parsing Log File help

Hi, I am a newbie to scripting. I have multiple log files (saved as .gz) in a directory that looks like this 01-01-2013 10:00 pn: 123 01-01-2013 10:00 sn: 987 01-01-2013 10:00 Test1 01-01-2013 10:00 Result: Pass 01-01-2013 10:00 Time: 5:00 01-01-2013 10:00 Test2 01-01-2013 10:00... (3 Replies)
Discussion started by: linuxnew
3 Replies

5. Shell Programming and Scripting

Cut the path into two parts

Hi, file=/usr/lib I need to cut and put it into two variable like string1=/usr string2=lib I made it for string2 string2=${file#/*/} How to get String1 in the same way which I have get string2. Use even more code tags ;) (4 Replies)
Discussion started by: munna_dude
4 Replies

6. Shell Programming and Scripting

Help Parsing a Log File

Hello all, I am new to scripting and I have written a script that performs an Rsync on my NAS and then moves on to send me an email with the status etc. The problem is that I think Rsync is taking to long to complete and the IF statement is timing out, as it doesn't appear to move on. Here... (1 Reply)
Discussion started by: Mongrel
1 Replies

7. UNIX for Dummies Questions & Answers

How to cut a string in two parts and show the other part

hi everybody.. I have a string like : abcd:efgh xxyy:yyxx ssddf:kjlioi ghtyu:jkksk nhjkk:heuiiue please tell me how i can display only the characters after ":" in the output the output should be : efgh yyxx kjlioi jkksk heuiiue please give quick reply.. its urgent..!! (6 Replies)
Discussion started by: adityamitra
6 Replies

8. UNIX for Dummies Questions & Answers

parsing a log file

I need help in parsing the following log files. 10 Apr 2009 0:16:16 * name: Tuna Belly Format: Well done, Price: 999 only 10 Apr 2009 0:16:16 * name: Roast Beef Format: Raw, Price: 55 c 10 Apr 2009 0:16:16 * name: Pasta Format: Dry, Price: 88.43 only etcetc I need to parse this... (8 Replies)
Discussion started by: izuma
8 Replies

9. Shell Programming and Scripting

Parsing textfile problem with cut

Hi, I have a textfile with several lines like this: text num: USER text (num) num num I need all these stuff. Problem is, how to get these stuff after ":". USER is a username and all chars are possible, even whitespace. So I cant use cut. Any ideas? (3 Replies)
Discussion started by: mcW
3 Replies

10. Shell Programming and Scripting

Parsing a Log file

Hi All, I'm deffently not a Unix specialist so be Gentel. I need to parse a Log file that looks like that: 2006-06-12 01:00:00,463 ERROR {cleanLoggersFiles} General Error comverse.compas.shared.exceptions.SystemParametersException: Error in reading parameter FileLocation at... (4 Replies)
Discussion started by: tbirenzweig
4 Replies
Login or Register to Ask a Question