Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to use awk with multiple pattern?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to use awk with multiple pattern?
# 1  
Old 07-20-2019
How to use awk with multiple pattern?
Hi All,


need your help , i want grouping complicated log below became count data message error per 5 minutes.


log some like below :


Code:
/data/logs/ag/tdr25001.log:2019-07-20 19:49:25,077|57|twallet|{"originatorConversationID":"STRANGE_CHECK_PIN_1563626965017","msisdn":"62859130860152","commandID":"InitTrans_ApplyTokenWithoutNotification"}|{"data":{"res:Body":{"res:ResultCode":-1,"res:ReferenceData":"","res:ResultType":0,"res:ResultDesc":"Mohon maaf, transaksi anda saat ini belum dapat diproses"},"res:Header":{"res:OriginatorConversationID":"STRANGE_CHECK_PIN_1563626965017","res:Version":1,"res:ConversationID":"AG_20190720_000076559a3563bd26fe"}},"message":"Mohon maaf, transaksi anda saat ini belum dapat diproses","status":"-1"}
/data/logs/ag/tdr25001.log:2019-07-20 00:47:17,227|50|twallet|{"uid":"100024215547","remark":"LinkNFCTag","originatorConversationID":"STRANGE_LinkNFCTag_1563558437174","msisdn":"6282226261
100","commandID":"LinkNFCTagSticker"}|{"data":{"res:Body":{"res:ResultCode":-1,"res:ResultType":0,"res:ResultDesc":"System internal error."},"res:Header":{"res:OriginatorConversationID":"STRANGE_LinkNFCTag_1563558437174","res:Version":1,"res:ConversationID":"ag11907200047176110076184"}},"message":"System internal error.","status":"-1"}
/data/logs/ag/tdr25001.log:2019-07-20 00:15:47,343 {"data":{"res:Body":{"res:ResultCode":-1,"res:ResultType":0,"res:ResultDesc":"System internal error."},"res:Header":{"res:OriginatorConve
rsationID":"STR0191062831239189741563556547218","res:Version":1,"res:ConversationID":"ag11907200015471897471631"}},"message":"System internal error.","status":"-1"}
/data/logs/ag/tdr25001.log:2019-07-20 00:15:48,381 {"data":{"res:Body":{"res:ResultCode":-1,"res:ResultType":0,"res:ResultDesc":"System internal error."},"res:Header":{"res:OriginatorConve
rsationID":"STR0226062831239189741563556548192","res:Version":1,"res:ConversationID":"ag11907200015481897471634"}},"message":"System internal error.","status":"-1"}
/data/logs/ag/tdr25001.log:2019-07-20 00:47:17,227 {"data":{"res:Body":{"res:ResultCode":-1,"res:ResultType":0,"res:ResultDesc":"System internal error."},"res:Header":{"res:OriginatorConve
rsationID":"STRANGE_LinkNFCTag_1563558437174","res:Version":1,"res:ConversationID":"ag11907200047176110076184"}},"message":"System internal error.","status":"-1"}
/data/logs/ag/tdr25001.log:2019-07-20 01:00:26,394 {"data":{"res:Body":{"res:ResultCode":-1,"res:ReferenceData":{"res:ReferenceItem":[{"com:Key":"bill_ref","com:Value":"0031005931706050004
1907A              92290 AGUNG DYATMIKA EKA NUGRAHA"},{"com:Key":"amount","com:Value":"000000092290"},{"com:Key":"Requester","com:Value":628110000009},{"com:Key":"OriginatorConversationID"
,"com:Value":"ASSTR0777I06281216666691563559209619"},{"com:Key":"TransactionDateAndTime","com:Value":"0720010009"},{"com:Key":"BillReferenceNumber","com:Value":"0315931706"},{"com:Key":"Sy
stemTraceAuditNumber","com:Value":792566}]},"res:ResultType":0,"res:ResultDesc":"Mohon maaf, transaksi anda saat ini belum dapat diproses"},"res:Header"
:{"res:OriginatorConversationID":"ASSTR0777I06281216666691563559209619","res:Version":1,"res:ConversationID":"ag11907200100256666977667"}},"message":"Mohon maaf, transaksi anda saat ini belum dapat diproses","status":"-1"}




i already test some awk below but still confuse how to get second parameter / value result

Code:
grep "\"res:ResultCode\":-1" /data/logs/ag/tdr2500*.log | awk -F '[|]' '{print $1,$5}'

my expectation :
Code:
2019-07-20 04:29 "Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi"  5
2019-07-20 04:29 "System internal error."  10
2019-07-20 04:34 "Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi"  1

2019-07-20 04:34 "System internal error."  5




need help...
# 2  
Old 07-20-2019
Again, your sample file doesn't match your expectation. Try
Code:
awk '
/"res:ResultCode":-1/   {sub (/:[0-9]*,[0-9]*/, " ", $0)
                         match ($0, /"message":"[^"]*/)
                         CNT[$1 " " $2 " " substr($0, RSTART+11, RLENGTH-11)]++
                        }
END                     {for (c in CNT) print c, CNT[c]
                        }
' file
2019-07-20 00:47 System internal error. 2
2019-07-20 00:15 System internal error. 2
2019-07-20 01:00 Mohon maaf, transaksi anda saat ini belum dapat diproses 1
2019-07-20 19:49 Mohon maaf, transaksi anda saat ini belum dapat diproses 1

This User Gave Thanks to RudiC For This Post:
fajar_3t3
# 3  
Old 07-20-2019
Hi rudy,


thanks for your reply , your awk great.


btw if the output result want to add delimiter "|" how to add for the awk, i try below but dont know how to add delimitter for column 3 and 4 .


Code:
2019-07-21 08:16 Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21 08:18 Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21 08:18 System internal error. 1
2019-07-21 08:19 System internal error. 2
2019-07-21 08:20 Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21 08:21 System internal error. 1


try with :


Code:
cat files | awk '/"res:ResultCode":-1/   {sub (/:[0-9]*,[0-9]*/, " ", $0) match ($0, /"message":"[^"]*/) CNT[$1 "|" $2 "|" substr($0, RSTART+11, RLENGTH-11)]++}END{for (c in CNT) print c, CNT[c]}' | sort -nk1


the result just missed column 3 and 4 :


Code:
2019-07-21|08:16|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21|08:18|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21|08:18|System internal error. 1
2019-07-21|08:19|System internal error. 2
2019-07-21|08:20|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi 1
2019-07-21|08:21|System internal error. 1




expected result :



Code:
2019-07-21|08:16|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|1
2019-07-21|08:18|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|1
2019-07-21|08:18|System internal error.|1
2019-07-21|08:19|System internal error.|2
2019-07-21|08:20|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|1
2019-07-21|08:21|System internal error.|1


many thanks for your effort


Regards

--- Post updated at 10:10 PM ---

once question again , and how to check some threshold when have more than 1 row, sample :


have 3 row, and count more than 300 ( threshold = 300 ), if > 300 will be sent alert.



Code:
2019-07-21|08:37|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|500
2019-07-21|08:37|System internal error.|1
2019-07-21|08:37|belum dapat diproses.|1

maybe if only 1 row i can use this awk to check files :


Code:
data=`cat data.txt | awk -F "|" '{print $4}'`
if [ $data -gt 300 ]
then
echo "sent alert"

but don't know how to check with 3 row...please kind to help


Regards
# 4  
Old 07-21-2019
Try
Code:
awk '
/"res:ResultCode":-1/   {sub (/:[0-9]*,[0-9]*/, " ", $0)
                         match ($0, /"message":"[^"]*/)
                         CNT[$1 OFS $2 OFS substr($0, RSTART+11, RLENGTH-11)]++
                        }
END                     {for (c in CNT) {print c, CNT[c]
                                         if (CNT[c] > 300) {print "alert " c | ("mail user@domain.com") }
                                        }
                        }
' OFS="|" file

This User Gave Thanks to RudiC For This Post:
fajar_3t3
# 5  
Old 07-21-2019
Hi RudyC,


thanks a lot for your reply.


your awk running well...but how print count and sent alert with also "messages error" ( row 3 )



Code:
2019-07-20|21:17|Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi|145
2019-07-20|21:17|System internal error|6
2019-07-20|21:17|Silahkan ulangi beberapa saat lagi|3

expectation alert



Code:
please check have many error "Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi" count 145

Thanks alot


Regards
Last edited by fajar_3t3; 07-21-2019 at 10:58 AM..
# 6  
Old 07-21-2019
Hi rudyC,


i found some awk almost done for this script but dont know how to print result $0 to alert ( i am using curl )


awk it will like below :


Code:
cat datafiles  | strings | awk '/"res:ResultCode":-1/   {sub (/:[0-9]*,[0-9]*/, " ", $0) match ($0, /"message":"[^"]*/) CNT[$1 OFS $2 OFS substr($0, RSTART+11, RLENGTH-11)]++}END{for (c in CNT) print c, CNT[c]}' OFS="|" | sort -nk1 | awk -F"|" '{if ($4 > 100)print ""$1","$2",please check have many error respond MFS",$3,"count :",$4;'}''

result :


Code:
2019-07-20,21:06,please check have many error respond MFS Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi count : 150
2019-07-20,21:07,please check have many error respond MFS Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi count : 151
2019-07-20,21:08,please check have many error respond MFS Mohon maaf, transaksi anda saat ini belum dapat diproses. Silahkan ulangi beberapa saat lagi count : 150

but how to sent alert for this result print $0 if using curl


expectation alert :


Code:
curl -X GET "http://x.x.x.x:10000/telegram/submit_fajar.php?msg=print $0+`hostname`+time_`date +%d%h%y_%H.%M.%S`"

noted :


print $0 mean result count > 100.


Thanks
Fajar
Last edited by fajar_3t3; 07-21-2019 at 10:57 AM..
# 7  
Old 07-21-2019
I'm afraid I don't understand your requirements.


You don't need the second awk invocation. Make the first's END section
Code:
if (CNT[c] > 300) {print "Please check have many error " c " count " CNT[c] | ("mail ...") }
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Match Pattern and print pattern and multiple lines into one line

Hello Experts , require help . See below output: File inputs ------------------------------------------ Server Host = mike id rl images allocated last updated density vimages expiration last read <------- STATUS ------->... (4 Replies)
Discussion started by: tigerhills
4 Replies

2. Shell Programming and Scripting

awk script issue redirecting to multiple files after matching pattern

Hi All I am having one awk and sed requirement for the below problem. I tried multiple options in my sed or awk and right output is not coming out. Problem Description ############################################################### I am having a big file say file having repeated... (4 Replies)
Discussion started by: kshitij
4 Replies

3. Shell Programming and Scripting

sed and awk usage to grep a pattern 1 and with reference to this grep a pattern 2 and pattern 3

Hi , I have a file where i have modifed certain things compared to original file . The difference of the original file and modified file is as follows. # diff mir_lex.c.modified mir_lex.c.orig 3209c3209 < if(yy_current_buffer -> yy_is_our_buffer == 0) { --- >... (5 Replies)
Discussion started by: breezevinay
5 Replies

4. Shell Programming and Scripting

Awk: print lines with one of multiple pattern in the same field (column)

Hi all, I am new to using awk and am quickly discovering what a powerful pattern-recognition tool it is. However, I have what seems like a fairly basic task that I just can't figure out how to perform in one line. I want awk to find and print all the lines in which one of multiple patterns (e.g.... (8 Replies)
Discussion started by: elgo4
8 Replies

5. Shell Programming and Scripting

awk with multiple pattern search

Want to fetch a column with multiple pattern using awk. How to achieve the same. Tried cat test address : 10.63.20.92/24 address : 10.64.22.93/24 address : 10.53.40.91/24 cat test | awk '{print $3}' |awk -F "/" '{print $1}' 10.63.20.92 10.64.22.93 10.53.40.91 Is there any... (2 Replies)
Discussion started by: Manasa Pradeep
2 Replies

6. Shell Programming and Scripting

Sed/awk/perl command to replace pattern in multiple lines

Hi I know sed and awk has options to give range of line numbers, but I need to replace pattern in specific lines Something like sed -e '1s,14s,26s/pattern/new pattern/' file name Can somebody help me in this.... I am fine with see/awk/perl Thank you in advance (9 Replies)
Discussion started by: dani777
9 Replies

7. Shell Programming and Scripting

Multiple pattern matching using awk and getting count of lines

Hi , I have a file which has multiple rows of data, i want to match the pattern for two columns and if both conditions satisfied i have to add the counter by 1 and finally print the count value. How to proceed... I tried in this way... awk -F, 'BEGIN {cnt = 0} {if $6 == "VLY278" &&... (6 Replies)
Discussion started by: aemunathan
6 Replies

8. Shell Programming and Scripting

awk delete/remove rest of line on multiple search pattern

Need to remove rest of line after the equals sign on search pattern from the searchfile. Can anybody help. Couldn't find any similar example in the forum: infile: 64_1535: Delm. = 86 var, aaga 64_1535: Fran. = 57 ex. ccc 64_1639: Feb. = 26 (link). def 64_1817: mar. = 3/4. drz ... (7 Replies)
Discussion started by: sdf
7 Replies

9. Shell Programming and Scripting

Help to search multiple pattern in file with grep/sed/awk

Hello All, I have a file which is having below type of data, Jul 19 2011 | 123456 Jul 19 2011 | 123456 Jul 20 2011 | 123456 Jul 20 2011 | 123456 Here I wanted to grep for date pattern as below, so that it should only grep "Jul 20" OR "Jul ... (9 Replies)
Discussion started by: gr8_usk
9 Replies

10. Shell Programming and Scripting

sed/awk to insert multiple lines before pattern

I'm attempting to insert multiple lines before a line matching a given search pattern. These lines are generated in a separate function and can either be piped in as stdout or read from a temporary file. I've been able to insert the lines from a file after the pattern using: sed -i '/pattern/... (2 Replies)
Discussion started by: zksailor534
2 Replies
Login or Register to Ask a Question