Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Need Time Stamp Range On Log Files

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need Time Stamp Range On Log Files
# 8  
Old 07-01-2015
Hi Chubler_XL,

This looks really good especially being able to pass the time stamps to the program. However, it seems to only work part way. It is not givng any ip numbers or their counts.

Code:
[root@ip-3.3.3.3 log]# ./gawk.sh "30 June 2015" 02:20 02:30
Examining from Tue Jun 30 02:20:00 EDT 2015 (1435645200)
            to Tue Jun 30 02:30:00 EDT 2015 (1435645800)

Processing /data/log/access_bhp.log file
Processing /data/log/access_hpc.log file
Processing /data/log/access_tfl.log file
Processing /data/log/access_thp.log file

Those are the actual file names but looking at your code I couldn't see that would cause the problem. Am I wrong?
Last edited by rbatte1; 07-02-2015 at 06:03 AM.. Reason: Changed ICODE to just CODE tags
# 9  
Old 07-01-2015
It's probably the format of your input file. This piece of code $NF == "-0400]" { looks for a line ending in "-0400]",

Perhaps your actual file isn't split on multi lines like the posted example?

The following may work better for 1 line per log entry:

Code:
$5 == "-0400]" {
  split($4,v,"[[/: ]")

Also try debug=1 to debug time range mismatches.
# 10  
Old 07-02-2015
I tried your suggested code change but it still isn't producing any output.

Here is the code after the change
Code:
$5 == "-0400]" {
  split($4,v,"[[/: ]")
  mnum=int(index("JanFebMarAprMayJunJulAugSepOctNovDec", v[3])/3)
  tm=mktime(v[4] " " mnum " " v[2] " " v[5] " " v[6] " " v[7])

  if (tm >= F && tm <= T) C[$1]++
  else if(debug) print tm " not between " F " and " T
}
END {for(ip in C) printf "%7d %s\n", C[ip], ip} ' $FILES

If you need more detail on the log file let me know how to do it. I have never used gawk so I don't know where to insert the debug=1 statement. I tried it right under the shebang but no output was generated.
Last edited by Don Cragun; 07-02-2015 at 01:51 PM.. Reason: Change ICODE tags to CODE tags.
# 11  
Old 07-02-2015
Quote:
Originally Posted by sharingsunshine
I tried your suggested code change but it still isn't producing any output.

Here is the code after the change
Code:
$5 == "-0400]" {
  split($4,v,"[[/: ]")
  mnum=int(index("JanFebMarAprMayJunJulAugSepOctNovDec", v[3])/3)
  tm=mktime(v[4] " " mnum " " v[2] " " v[5] " " v[6] " " v[7])

  if (tm >= F && tm <= T) C[$1]++
  else if(debug) print tm " not between " F " and " T
}
END {for(ip in C) printf "%7d %s\n", C[ip], ip} ' $FILES

If you need more detail on the log file let me know how to do it. I have never used gawk so I don't know where to insert the debug=1 statement. I tried it right under the shebang but no output was generated.
If you look in Chubbier_XL's suggested code, the gawk script started with:
Code:
gawk -v F=$FROM -v T=$TO -v debug=0 '

To enable debugging, change the -v debug=0 to -v debug=1.
This User Gave Thanks to Don Cragun For This Post:
Chubler_XL
# 12  
Old 07-02-2015
Well there are 3 possible causes of the records not matching.

1. mktime is not seeing a valid date string and it will set tm to -1
2. The tm value is correct but logfile date/time is not falling in between the F and T range
3. the records are not being processed because field 5 is not "-0400]"

Changing debug=0 to debug=1 will help you identify problem 1 and 2

For problem 3 try examining your logfile awk splits fields up using space as the separator so for the expected logfile format of:

Code:
1.1.1.1 - - [01/Jul/2015:10:59:29 -0400] 
         "GET /themes/warehouse/img/arrow_right_2.png HTTP/1.1" 200 149 
         "http://www.abcd.com/content/152-Tea_Tree_Oil_Uses_sp_153" 
         "Mozilla/5.0 (iPad; CPU OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F69 Safari/600.1.4"

We should get
Code:
$1=1.1.1.1
$2=-
$3=-
$4=01/Jul/2015:10:59:29
$5=-0400]

If you can't spot the problem try and change the code to:
Code:
gawk -v F=$FROM -v T=$TO -v debug=1 '
{for(i=1;i<=NF;i++) printf "$%d=%s\n", i, $i }
FNR==1 {
    for(ip in C) printf "%7d %s\n", C[ip], ip
    delete C
    print "Processing " FILENAME " file"
}
...

This will produce a lot of output so get ready to try CTRL-C or pipe it to head -50
# 13  
Old 07-02-2015
Thanks to Don he pointed me in the right direction of debug=1

It seems that when I run the script I am getting this output

Code:
1432820036 not between 1433169000 and 1433224800
1432820037 not between 1433169000 and 1433224800
1432820037 not between 1433169000 and 1433224800
1432820037 not between 1433169000 and 1433224800
1432820037 not between 1433169000 and 1433224800
1432820037 not between 1433169000 and 1433224800
1432820037 not between 1433169000 and 1433224800
1432820060 not between 1433169000 and 1433224800
1432820061 not between 1433169000 and 1433224800
1432820061 not between 1433169000 and 1433224800
1432820061 not between 1433169000 and 1433224800
1432820061 not between 1433169000 and 1433224800

I just used your sample input of

Code:
./gawk.sh "1 June 2015" 10:30 2:00

I also ran it with out the e on June since my log uses Jun
Code:
./gawk.sh "1 Jun 2015" 10:30 2:00

but I got the same results.

I ran
Code:
./gawk.sh "02 Jul 2015" 01:50 02:30

with debug=0 I get this
Code:
 ./gawk.sh "02 Jul 2015" 01:30 3:20
Examining from Thu Jul  2 01:30:00 EDT 2015 (1435815000)
            to Thu Jul  2 03:20:00 EDT 2015 (1435821600)

Processing /data/log/access_bhp.log file
Processing /data/log/access_hpc.log file
Processing /data/log/access_tfl.log file
Processing /data/log/access_thp.log file

with debug=1 I get

Code:
1432822160 not between 1435816200 and 1435818600
1432822160 not between 1435816200 and 1435818600
1432822160 not between 1435816200 and 1435818600
1432822160 not between 1435816200 and 1435818600
1432822160 not between 1435816200 and 1435818600
1432822160 not between 1435816200 and 1435818600
1432822160 not between 1435816200 and 1435818600
1432822161 not between 1435816200 and 1435818600
1432822161 not between 1435816200 and 1435818600
1432822165 not between 1435816200 and 1435818600
1432822175 not between 1435816200 and 1435818600
1432822178 not between 1435816200 and 1435818600

By the way field 5 is -0400]

I'll reply again with the latest suggested code change output.

---------- Post updated at 05:43 PM ---------- Previous update was at 05:28 PM ----------

You're correct it does produce a lot of output. I will include the actual ip on this one entry because it is bingbot

Code:
$1=157.55.39.187
$2=-
$3=-
$4=[28/Jun/2015:16:27:29
$5=-0400]
$6="GET
$7=/content/10-customer-testimonials
$8=HTTP/1.1"
$9=200
$10=12025
$11="-"
$12="Mozilla/5.0
$13=(compatible;
$14=bingbot/2.0;
$15=+http://www.bing.com/bingbot.htm)"
1432844849 not between 1435816200 and 1435818600

If you want me to include something specific let me know.

Here is the entire script just to make sure I don't have something wrong

Code:
#!/bin/bash

if (( $# < 3 || $# > 4 ))
then
   printf "Usage: $0 from_date from_time [to_date] to_time\n" >&2
   exit 2
fi

FDAY=$1
FTIME=$2

if (( $# == 3 ))
then
    TDAY=$FDAY
    TTIME=$3
else
    TDAY=$4
    TTIME=$3
fi

FROM=$(date -d "$FDAY $FTIME" +%s)
(($? != 0 )) && exit 3
TO=$(date -d "$TDAY $TTIME" +%s)
(($? != 0 )) && exit 4

if (( $# == 3 && TO < FROM ))
then
   #FROM time later that TO time so add a day
   (( TO+=3600*24))
fi

if (( TO < FROM ))
then
    echo "$0: FROM date must be before TO date" >&2
    exit 5
fi


echo "Examining from $(date -d @$FROM) ($FROM)"
echo "            to $(date -d @$TO) ($TO)"
echo
FILES=/data/log/access_*.log

gawk -v F=$FROM -v T=$TO -v debug=1 '
{for(i=1;i<=NF;i++) printf "$%d=%s\n", i, $i }
FNR==1 {
    for(ip in C) printf "%7d %s\n", C[ip], ip
    delete C
    print "Processing " FILENAME " file"
}

$5 == "-0400]" {
  split($4,v,"[[/: ]")
  mnum=int(index("JanFebMarAprMayJunJulAugSepOctNovDec", v[3])/3)
  tm=mktime(v[4] " " mnum " " v[2] " " v[5] " " v[6] " " v[7])

  if (tm >= F && tm <= T) C[$1]++
  else if(debug) print tm " not between " F " and " T
}
END {for(ip in C) printf "%7d %s\n", C[ip], ip} ' $FILES

I noticed this output is including June 28 entries but the code I used was

Code:
./gawk.sh "02 Jul 2015" 01:50 02:30

I picked that time frame because there are over 30 entries with the same IP since that is the time I run my sitemap program. It seems it is ignoring what date I am putting in.
# 14  
Old 07-02-2015
Not much need to add the detailed debugging as debug=1 is revealing the proglem... the times in your logfiles are not within the specified date range:

Code:
$ date -d@1432820036 
Thu, May 28, 2015  9:33:56 AM

$ date -d @1432820061
Thu, May 28, 2015  9:34:21 AM

Why not try a small test file with a few entries that are within the specified date/time range, you are probably being swamped by thousands of entries that are genuinely being skipped.

Looking at the debug ouput everything seems to be working fine, Field 4 has been rendered into the correct julian timestamp:

Code:
date -d @1432844849 
Thu, May 28, 2015  4:27:29 PM

And this in not between 1:50AM and 2:30AM on Jul-2

Just to be sure, you are filtering on 1:50AM to 2:30AM on 2 July, if you want PM use 24-hour notation e.g. 13:40 to 15:30
Last edited by Chubler_XL; 07-02-2015 at 07:01 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Collecting all lines between two time stamp from the log

Can you help me to collect the entire logs between two time stamp. The below awk command collecting the logs only if the line has time stamp. awk '$0>=from && $0<=to' from="150318 23:19:04" to="150318 23:55:04" log file 150318 23:19:04 logentries 150318 23:29:04 logentries 150318... (11 Replies)
Discussion started by: zenkarthi
11 Replies

2. Shell Programming and Scripting

To check time stamp in log file and calculate.

Hi Friends, I have the following logfile. i want to make a script for calculate time by time2 - time1 1600266278|random|1|2014-09-19 02:08:56.024|2014-09-19 02:08:59.398|A|B|ROOM|Num0208559970111101788|1|dog|dos 1600266200|random|4|2014-09-19 02:08:06.572|2014-09-19... (2 Replies)
Discussion started by: ooilinlove
2 Replies

3. Shell Programming and Scripting

Files with date and time stamp

Hi Folks, Need a clarification on files with date and time stamp. Here is my requirement. There is a file created everyday with the following format "file.txt.YYYYMMDDHHMMSS". Now i need to check for this file and if it is available then i need to do some task to the file. I tried... (6 Replies)
Discussion started by: jayadanabalan
6 Replies

4. Shell Programming and Scripting

awk - check time stamp between range or not

I want to check given time stamp is between the given time stamp or not. I am using AIX. YYYYMMDDHHMMSS abc.csv START TIME, END TIME 20130209018000,20130509022000 20120209018000,20130509022000 20120209018000,20130509022000 Script will check given time stamp is between above two range or... (2 Replies)
Discussion started by: vegasluxor
2 Replies

5. Shell Programming and Scripting

Select files by time stamp

Hi, I need help to read file in a directory on basis of time stamp. e.g. If file access in last 2 minutes it should not be copy to remote directory. Below is my script. +++++++++++++++++++++++++ #!/bin/ksh DATE=`date +"%Y-%m-%d_%H%M"` SEPARATER=" " exec < out_interfaces.cfg... (1 Reply)
Discussion started by: qamar.alam
1 Replies

6. Shell Programming and Scripting

Identify log files based on time stamp,zip and then copy..HELP

Hi All, PFB is a requirement. I am new to shell scripting. So plz help. It would be highly appreciated. 1. choose all the log files based on a particular date (files location is '/test/domain')--i.e,we should choose all the files that are modified on 29th November, neither 28th nor 30th 2.... (3 Replies)
Discussion started by: skdas_niladri
3 Replies

7. Shell Programming and Scripting

Old time stamp being updated for new files

Hello Friends I am facing a weird problem :confused:, we receive thousands of files in my system on a daily basis, access time stamp on some of the files are being updated as old time stamp like 1968-01-19, Could some one help me what could be causing this? so that i can narrow down the problem... (4 Replies)
Discussion started by: Prateek007
4 Replies

8. Shell Programming and Scripting

time stamp perl script error out of range 1..31

Hi, while running the perl script i am getting this error message , Day '' out of range 1..31 at rsty.sh line 44 what do iam missing in the script, any suggestion #!/usr/bin/perl use Time::Local; my $wday = $ARGV; my $month = $ARGV; # convert the month shortname into 0-11 number if... (4 Replies)
Discussion started by: saha
4 Replies

9. Solaris

doubt reg time stamp in files.

I copied a file from one host to another using sftp. But after copying the time stamp is not updating . Even though I checked the permission, it looks good. I copied the same file to some temporary location, there it updating the time stamp. Anyone have any idea on this (6 Replies)
Discussion started by: rogerben
6 Replies

10. UNIX for Dummies Questions & Answers

How to search for files based on the time stamp

Hi All, I know the timestamp of a file. Now i would like to list all the files in the with the same time stamp in the same file. Any help would be appreciated. Thanks. sunny (1 Reply)
Discussion started by: sunny_03
1 Replies
Login or Register to Ask a Question