awk script to detect specific string in a log file and count it
Hello, can someone guide me on this?
I don't know what is the best approach, (awk script, shell script)
I am using RedHat Linux version 6.5. There is a third party application deployed on that server. This app by default generates 5 log files and each file is 20MB. These log rollover automatically so at any given time there always be 5 log file like below
When the app runs, it creates the following entries in the log file. for simplicity, i have removed the other information.
Requirement is whenever this string RESTful/BR4/Dispatchers/Drop Down Dispatch Process.process/Log appears in the log file, count it. So at the end of the day, we wanted to know on Oct 20, 2018, how many times that entry appeared in the log file. Basically, everyday, we need to collect statics about this app. The script should run automatically and spit out the stats in a file like this
appnamedatecount
Code:
BW.AMI_Services-Dropdown-AMI_Services-Dropdown-ProcessArchive Oct 20, 2018 3
-----log files entries------
Code:
2017 Oct 20 13:44:03:359 GMT -0600 BW.AMI_Services-Dropdown-AMI_Services-Dropdown-ProcessArchive User [BW-User] - Job-20540 [Services - RESTful/BR4/Dispatchers/Drop Down Dispatch Process.process/Log]:
2017 Oct 20 13:45:02:170 GMT -0600 BW.AMI_Services-Dropdown-AMI_Services-Dropdown-ProcessArchive User [BW-User] - Job-20546 [Services - RESTful/BR4/Dispatchers/Drop Down Dispatch Process.process/Log]:
2017 Oct 20 13:45:10:605 GMT -0600 BW.AMI_Services-Dropdown-AMI_Services-Dropdown-ProcessArchive User [BW-User] - Job-20547 [Services - RESTful/BR4/Dispatchers/Drop Down Dispatch Process.process/Log]:
Moderator's Comments:
Please use CODE tags as required by forum rules!
Last edited by RudiC; 10-29-2017 at 06:24 AM..
Reason: Added CODE tags.
Welcome to forums, thank you for mentioning the details(though your question creates some doubts still), please use code tags for Input_file/sample/output/code/commands which you are using in your posts. Could you please try following and let me know if this helps you.
EDIT: Apologies forum, I had misread the post and posted wrong answer, so adding a answer which could be taken as a starting point, THANKS to Scrutinizer for letting know
@Ravinder, I am unsure how this supposed to work?
The OP is looking for a way to get a daily count of log occurrences in a rotating set of log files into a new log file.
What you propose has the date fixed in both the search string and the print result. So should he create a new script every day?
The closing braces in the FNR==1 section do not align with the if statements, which is confusing.
I do not get why you need to close files when there are 5 max.
Also, he appears to not be looking in which file exactly the data occurred, but how many times, so I do not see why a total needs to be printed per file, so why not only create an END section?
Also the END section is missing, if there are contents are in the last file, then their total will not get printed.
Why do you also include the filename in the result?
--
@OP:
a solution like that will only work if the rotation frequency is less then 5 per day, so maybe a solution should test for this?
You want the result for 2018, I assume that is a typo?
The date format change is that really necessary, or can it be any date format?
The script would need to only record or discover the differences with the last time it was run to avoid double or missing entries in the statistics log..
Last edited by Scrutinizer; 10-29-2017 at 02:39 AM..
This User Gave Thanks to Scrutinizer For This Post:
A simple question: is there a timestamp on each line of the logfile or in the lines of interest. That would eliminate the issues scrutinizer mentioned I think.
e.g.,
First of all thank you for looking into this. Apologies if I was not clear in my post and I will try to follow the forum rules.
Here is the simplified version of the requirements
In real time, a script should monitor appname.log file (no need to monitor .log.1 or .log.2 or .log.3 or .log.4)
On a new line in log file which matches this RESTful/BR4/Dispatchers/Drop Down Dispatch Process.process/Log add the appname date (any format) and count in a new a file. (appnamestats.txt)
For next entry, for same day, we already have appname and date in appnamestats.txt, we just need to increment the count. The same appnamestats.txt file can be used for next day and so.
There is a typo about the date, it should be for current year 2017 and not 2018
The date can be in any format. For example 2017 Oct 20
There always be complete timestamp like below in the lines of interest
2017 Oct 20 13:44:03:359 GMT -0600
btw, i run the provided solution and it only return 16. There are 288 entries for that string in the log file and it should return 288
I will appreciate if you help me here in this script in Solaris Enviroment.
Scenario:
i have 2 files :
1) /tmp/TRANSACTIONS_DAILY_20180730.txt:
201807300000000004
201807300000000005
201807300000000006
201807300000000007
201807300000000008
2)... (10 Replies)
I am trying to use awk to change a specific string in a field, if it is found, to another value. In the tab-delimited file the text in bold in $3 contains the string 23, which is always right before a ., if it is present.
I am trying to change that string to X, keeping the formatting and the... (3 Replies)
In the below awk I am trying output to one file those lines that match between $2,$3,$4 of file1 and file2 with the count in (). I am also trying to output those lines that are missing between $2,$3,$4 of file1 and file2 with the count of in () each. Both input files are tab-delimited, but the... (7 Replies)
Hi,
I have log file which rolls out every second which is as this.
HttpGenRequest - -<!--OXi dbPublish--> <created="2014-03-24 23:45:37" lastMsgId="" requestTime="0.0333"> <response request="getOutcomeDetails" code="114" message="Request found no matching data" debug="" provider="undefined"/>... (3 Replies)
I need a scripting AWK to compare 2 files.
file 1 and 2 are list of keywords
1 is
a
b
c
d
2 is
aa
aaa
b
bb
ccc
d
I want the AWK script to give us the number of times every keyword in file 1 occurs in file 2.
output should be
a 2 (7 Replies)
I'm still beginner and maybe someone can help me.
I have this input:
the great warrior a, b, c
and what i want to know is, with awk, how can i detect the string with 'warrior' string on it and print the a, b, and c seperately, become like this :
Warrior Type
a
b
c
Im still very... (3 Replies)
Hi guys I'm very new to unix and I have to create an awk script that detects webpage addresses from a file/webpage and outputs how many times each webpage was detected.e.g. if my file was:
www.google.com
www.facebook.com
www.google.com
the output should be:
www.google.com x2... (2 Replies)
Hi guys I'm very new to unix and I have to create an awk script that detects webpage addresses from a file/webpage and outputs how many times each webpage was detected.e.g. if my file was:
(Note: The symbol " was added to stop them being created into links)
"www.google.com"... (1 Reply)
Hi,
I have a log like this :
actually i want to get the log like this :
where % can get from :
100 * pmTotNoRrcConnectReqSucc / pmTotNoRrcConnectReq
Thanks in advance.. :) (8 Replies)
Hello my friends, I need to count how many words are into a log file, I'm using:
cat logfile | grep 'word' | wc -l
Cuz the 'word' appears once per line.
But my logfile grow faster and at the end ofthe day is really big, so how i can count the 'word' only from (by example) line 4000 of... (5 Replies)
10-28-2017
awk script to detect specific string in a log file and count it
