Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Request per second script

Shell Programming and Scripting


Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 02-16-2017
Katkota Katkota is offline
Registered User
 
Join Date: Oct 2007
Last Activity: 4 March 2017, 8:50 PM EST
Posts: 151
Thanks: 0
Thanked 0 Times in 0 Posts
Request per second script

Hello;
I'm having about 800 log files and i'm trying to write a script that report the counts of lines per second or "requests per second" in each log file and report the output which includes the timestamp for the highest lines per second count and the log file name and the highest number per second from that file and then go to the second log file and do the same thing then append the results to the same output file

I'm currently using this command to get the results from each log but doing this for 800 log files is not practical

Code:
grep "2017-16-02" logfile1 | cut -c2-18 | uniq -c

This command reports a list of lines count per 10 seconds but it's not very efficient

So the output file should look like this:


Code:
Date/time    "1st LogFileName"    "highest requests per second for this log"
Date/time    "2nd LogFileName"    "highest requests per second for this log"

Any help would be greatly appreciated

Here's the log files i'm working with:


Code:
#Fields: date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method
 rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id x-bl
uecoat-application-name x-bluecoat-application-operation
#Remark: 1412140034 "lofnetsg1" "192.168.13.14" "main"
2017-02-15 17:49:06 8 10.23.53.112 - - - OBSERVED "Technology/Internet" -  404 TCP_NC_MISS GET text/html;%20charset=iso-8859-1 http dci.sophosupd.c
om 80 /osxhe/d/16/d16e39472e867bf7f8407db13d3502b5.dat - dat "Sophos%20AutoUpdate/ CFNetwork/760.6.3 Darwin/15.6.0 (x86_64)" 192.168.13.14 437 243
- "none" "none"
2017-02-15 17:49:06 578 10.4.33.22 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp tpc.googlesyndication.com 443 / - - "Mozill
a/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" 192.168.13.14 21106 2873 - "none" "none"
2017-02-15 17:49:06 17 10.17.87.27 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_NC_MISS GET - http t.lkqd.net 80 /t ?ev=147&tsid=132117&env=3&cb=7
43590466414&format=0&did=2&osid=2&pubtagtype=lkqd-formats&render=html5&apt=auto&uimp=6944564170577988928&svrs=0.45&srvid=10.10.4.214&oip=134.174.21
.158&vrs=197&tsl=13827&ear=0&width=300&height=250&phost=&host=reuters.com&appname=&idfa=&aid=&bundleid=&lsid=&loclat=&loclong=&contentid=&contentti
tle=&contentlength=&contenturl=&appstoreurl=&execution=inbanner&placement=&browserid=1&adplayersize=small&pid=326&sid=150846&spid=18106&psid=&ppid=
&lkqdtagtype=platform-connection&tlr=0&adid=127634&asrc=15817&dealid=117723&c1=&c2=&c3=&tsia=8945&tssa=&rnd=157089284&cs=3668523497 - "Mozilla/5.0
(Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 290 1534 - "none" "none"
2017-02-15 17:49:06 669 10.23.112.63 - - - OBSERVED "Radio/Audio Streams" -  200 TCP_MISS GET audio/x-aac http primary.hls-streaming.production.str
eaming.siriusxm.com 80 /AAC_Data/broadwaysbest/HLS_broadwaysbest_64k_v3/broadwaysbest_64k_1_021560803727_00310913_v3.aac ?consumer=k2&gupId=BE4A750
32FEC2FF7293A7F5C0FAEFEA3 aac "SXMLiveAudioPlayer/1485824608 CFNetwork/808.3 Darwin/16.3.0" 192.168.13.14 79798 1119 - "none" "none"
2017-02-15 17:49:06 32 10.64.5.81 - - - OBSERVED "Web Ads/Analytics" Error
.html  200 TCP_NC_MISS GET text/javascript http beacon.krxd.net 80 /optout_check ?callback=Krux.ns._default.kxjsonp_optOutCheck - "Mozilla/5.0 (Win
dows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 635 451 - "none" "none"


Last edited by Katkota; 02-17-2017 at 09:39 PM.. Reason: Change ICODE tags to CODE tags and add CODE tags.
Sponsored Links
    #2  
Old Unix and Linux 02-17-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is online now Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 24 June 2017, 4:27 AM EDT
Location: San Jose, CA, USA
Posts: 10,396
Thanks: 527
Thanked 3,626 Times in 3,092 Posts
You have given us no indication of where the Date/time is in an input line in your log files. You have given us no indication of where the "requests per second for this log" appears in an input line in your log files, nor what a line in your input log files represents. Your command line seems to only be looking for entries that occur on a certain date, but your description says nothing about looking for a specific date. You talk about maximum requests/second, but nothing in your pipeline seems to be making any attempt to find a maximum value in any of the individual input files nor in the combined aggregation of input files.

I could make lots of guesses about what you might be trying to do and what your data format(s) is(are), but it would be MUCH better if you would clearly describe your input file format(s), show us a couple of sample input files, and show us the exact outputs that should be produced from those sample inputs.
Sponsored Links
    #3  
Old Unix and Linux 02-17-2017
Katkota Katkota is offline
Registered User
 
Join Date: Oct 2007
Last Activity: 4 March 2017, 8:50 PM EST
Posts: 151
Thanks: 0
Thanked 0 Times in 0 Posts
Sorry for not posting some of the log files.. Here it's below.. What i'm looking for is to get the highest requests per seconds "mostly will be lines per second" for each log file and append the output to a file.
The output needs to include the log file name and the highest request per second in that log file


Code:
#Fields: date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method
 rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id x-bl
uecoat-application-name x-bluecoat-application-operation
#Remark: 1412140034 "lofnetsg1" "192.168.13.14" "main"
2017-02-15 17:49:06 8 10.23.53.112 - - - OBSERVED "Technology/Internet" -  404 TCP_NC_MISS GET text/html;%20charset=iso-8859-1 http dci.sophosupd.c
om 80 /osxhe/d/16/d16e39472e867bf7f8407db13d3502b5.dat - dat "Sophos%20AutoUpdate/ CFNetwork/760.6.3 Darwin/15.6.0 (x86_64)" 192.168.13.14 437 243
- "none" "none"
2017-02-15 17:49:06 578 10.4.33.22 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_TUNNELED CONNECT - tcp tpc.googlesyndication.com 443 / - - "Mozill
a/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko" 192.168.13.14 21106 2873 - "none" "none"
2017-02-15 17:49:06 17 10.17.87.27 - - - OBSERVED "Web Ads/Analytics" -  200 TCP_NC_MISS GET - http t.lkqd.net 80 /t ?ev=147&tsid=132117&env=3&cb=7
43590466414&format=0&did=2&osid=2&pubtagtype=lkqd-formats&render=html5&apt=auto&uimp=6944564170577988928&svrs=0.45&srvid=10.10.4.214&oip=134.174.21
.158&vrs=197&tsl=13827&ear=0&width=300&height=250&phost=&host=reuters.com&appname=&idfa=&aid=&bundleid=&lsid=&loclat=&loclong=&contentid=&contentti
tle=&contentlength=&contenturl=&appstoreurl=&execution=inbanner&placement=&browserid=1&adplayersize=small&pid=326&sid=150846&spid=18106&psid=&ppid=
&lkqdtagtype=platform-connection&tlr=0&adid=127634&asrc=15817&dealid=117723&c1=&c2=&c3=&tsia=8945&tssa=&rnd=157089284&cs=3668523497 - "Mozilla/5.0
(Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 290 1534 - "none" "none"
2017-02-15 17:49:06 669 10.23.112.63 - - - OBSERVED "Radio/Audio Streams" -  200 TCP_MISS GET audio/x-aac http primary.hls-streaming.production.str
eaming.siriusxm.com 80 /AAC_Data/broadwaysbest/HLS_broadwaysbest_64k_v3/broadwaysbest_64k_1_021560803727_00310913_v3.aac ?consumer=k2&gupId=BE4A750
32FEC2FF7293A7F5C0FAEFEA3 aac "SXMLiveAudioPlayer/1485824608 CFNetwork/808.3 Darwin/16.3.0" 192.168.13.14 79798 1119 - "none" "none"
2017-02-15 17:49:06 32 10.64.5.81 - - - OBSERVED "Web Ads/Analytics" Error
.html  200 TCP_NC_MISS GET text/javascript http beacon.krxd.net 80 /optout_check ?callback=Krux.ns._default.kxjsonp_optOutCheck - "Mozilla/5.0 (Win
dows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" 192.168.13.14 635 451 - "none" "none"

    #4  
Old Unix and Linux 02-17-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is online now Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 24 June 2017, 4:27 AM EDT
Location: San Jose, CA, USA
Posts: 10,396
Thanks: 527
Thanked 3,626 Times in 3,092 Posts
I am very disappointed in your response. You said:
Quote:
I'm currently using this command to get the results from each log but doing this for 800 log files is not practical
which makes it sound like the results you are getting from your script give you what you want, but is too slow to use to process 800 log files. But, with your sample input, it produces no output. If we convert the date in the grep in your code:

Code:
grep "2017-16-02" logfile1 | cut -c2-18 | uniq -c

which is in the format YYYY-DD-MM to the date found in your sample file:

Code:
2017-02-15 17:49:06 ... ... ...

which is in the format YYYY-MM-DD, we get the output:

Code:
   5 017-02-15 17:49:0

which in addition to truncating the year also truncates the time so it will give you counts of ten second intervals instead of one second intervals, and makes no attempt to find the maximum count and makes no attempt to include the filename in the output.

Are you looking for counts for each one second interval or for 10 second intervals?

Are you looking for counts only on a specific date, or are you looking for the maximum count in each file no matter what the date might be for that count?

You have not answered most of the questions I asked in post #2 in this thread:
  1. You have given us no indication of where the "requests per second for this log" appears in an input line in your log files, nor what a line in your input log files represents.
  2. Your command line seems to only be looking for entries that occur on a certain date, but your description says nothing about looking for a specific date.
  3. You talk about maximum requests/second, but nothing in your pipeline seems to be making any attempt to find a maximum value in any of the individual input files nor in the combined aggregation of input files.
  4. Please clearly describe your input file format(s). (What you have shown us seems to be a bunch of random text that is chopped into lines that are a little less than 150 characters per line, but no guaranteed way to detect the start of a record.)
  5. Show us a couple of sample input files.
  6. And, show us the exact outputs that should be produced from those sample inputs.
If you can't provide a clear specification of what it is that needs to be done, it will be very hard for us to help you find a solution to your problem!
Sponsored Links
    #5  
Old Unix and Linux 02-18-2017
Katkota Katkota is offline
Registered User
 
Join Date: Oct 2007
Last Activity: 4 March 2017, 8:50 PM EST
Posts: 151
Thanks: 0
Thanked 0 Times in 0 Posts
Sorry to disappoint you my friend.
if i get the interval every 10 seconds is fine. the issue is i want this to be done through a script and not manually one at a time. My apology if i'm not explaining it better.
Sponsored Links
    #6  
Old Unix and Linux 02-18-2017
Don Cragun's Unix or Linux Image
Don Cragun Don Cragun is online now Forum Staff  
Administrator
 
Join Date: Jul 2012
Last Activity: 24 June 2017, 4:27 AM EDT
Location: San Jose, CA, USA
Posts: 10,396
Thanks: 527
Thanked 3,626 Times in 3,092 Posts
Repeatedly saying that you want to perform some unspecified task instead of doing it manually is getting us nowhere. Please clearly answer the questions I asked in post #4, or I will close this thread.
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Shell script request udayakumar Shell Programming and Scripting 5 04-02-2012 01:31 PM
Solaris request script potro Shell Programming and Scripting 2 11-18-2008 06:03 AM
Using a request script Tornado UNIX for Advanced & Expert Users 4 04-01-2008 10:19 PM
unix script http request arksal UNIX for Dummies Questions & Answers 1 08-24-2006 10:15 PM



All times are GMT -4. The time now is 04:47 AM.