Need Time Stamp Range On Log Files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need Time Stamp Range On Log Files
# 1  
Old 07-01-2015
Need Time Stamp Range On Log Files

I have created this script
Code:
#!/bin/sh
FILES=/data/log/access_*.log
for f in $FILES
do
echo "Processing $f file"
cat $f | awk '{print $1}' | sort | uniq -c | sort -n | tail
done

It produces this output
Code:
Processing /data/log/access_abc.log file
     
    114 1.1.1.1
    167 2.2.2.2
   2258 3.3.3.3
Processing /data/log/access_def.log file
    102 1.1.1.1
    102 2.2.2.2
   2040 3.3.3.3

This is the log file it comes from

Code:
1.1.1.1 - - [01/Jul/2015:10:59:29 -0400] 
         "GET /themes/warehouse/img/arrow_right_2.png HTTP/1.1" 200 149 
         "http://www.abcd.com/content/152-Tea_Tree_Oil_Uses_sp_153" 
         "Mozilla/5.0 (iPad; CPU OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F69 Safari/600.1.4"

(edited to show whole line - was a single long line)

I need to be able to specify the date and time range in the script. For example, I need to know what IP's were active 02:20 -02:30 June 30th, 2015.

Please help me to figure out the script that will permit this range of times reporting.

Thanks,

Randal

Moderator's Comments:
Mod Comment Please use code tags, not icode tags, thanks.

Last edited by jim mcnamara; 07-01-2015 at 05:24 PM.. Reason: Fix several format issues
# 2  
Old 07-01-2015
What operating system and shell are you using? (The tools available to process timestamps vary considerably from OS to OS, and in some cases from shell to shell.)

Are your log files sorted in increasing timestamp order?

Will the timestamp range you want to select ever cross day boundaries or will the start and end points always be on the same date?

Can you specify the timestamp format that the user has to enter for the start and end points (e.g., YYYYMMDDhhmmss, YYYYMMDD:hh:mm:ss, or DD/Mon/YYYY:hh:mm:ss)? If so, have you chosen a format, or can we select one that makes processing easier? If not, what format(s) do(es) your script need to be able to interpret?

Will all of the offsets from GMT (i.e., -0400) be the same for the timestamp ranges you're interested in? (Your script will be much more complex if log entries are coming from various timezones around the world or if your script has to make adjustments for timestamp ranges crossing shifts to and from daylight savings time.)

Is the output supposed to change at all (other than reducing the number of selected entires being processed)? If so, how?
# 3  
Old 07-01-2015
Hi Don,

Thanks for getting back to me. I am using Centos 6 and Apache as an operating system.

My logs are sorted in increasing time stamp order.

1.1.1.1 - - [01/Jul/2015:10:59:29 -0400]
1.1.1.1 - - [01/Jul/2015:10:59:29 -0400]
2.2.2.2 - - [01/Jul/2015:10:59:30 -0400]

Yes, the range can cross day boundaries.

The time stamp format is
[01/Jul/2015:10:59:29 and it needs to be in this format DD/Mon/YYYY:hh:mm:ss

all of the offsets are the same which is GMT -0400

The output is the same as I posted earlier. I just need to find the most active IP's during a specific time stamp range.

Thanks,
# 4  
Old 07-01-2015
I can't even guess how many times similar/identical requests have been posted and answered in these fora. Nevertheless, try (as a starter, NO error checking, day boundary crossing) and adapt
Code:
awk ' 
NR==1           {split (START, T, ":")
                 START=T[1]*60+T[2]
                 split (STOP, T, ":")
                 STOP=T[1]*60+T[2]
                }

FNR==1          {print "processing " FILENAME " file"}

                {sub (/\[/, _, $4)
                 split ($4, T, ":")
                 TIME=60*T[2]+T[3]
                }

DAY=T[1] &&
TIME >= START &&
TIME <= STOP    {print FILENAME, $1}
' DAY="01/Jul/2015" START="10:50" STOP="10:59" $FILES

# 5  
Old 07-01-2015
Hi RudiC,

I appreciate the code but since the file location wildcards aren't specified, I need to know where in your script they should be specified.

I am sure you are correct about the many times this question has been asked but unless you know the keywords that were used they are hard to find. I looked up wildcards and range on the unix.com site as well as on the net and never could get an understandable answer I could use.

So I really appreciate you volunteering this code.
# 6  
Old 07-01-2015
The files are specified in the FILES variable, as you did in post#1.

For other posts on the same topic, look at the bottom of this page.
# 7  
Old 07-01-2015
How about this, using GNU date and gawk as they should be available on Centos 6:

Code:
#!/bin/bash

if (( $# < 3 || $# > 4 ))
then
   printf "Usage: $0 from_date from_time [to_date] to_time\n" >&2
   exit 2
fi

FDAY=$1
FTIME=$2

if (( $# == 3 ))
then
    TDAY=$FDAY
    TTIME=$3
else
    TDAY=$4
    TTIME=$3
fi

FROM=$(date -d "$FDAY $FTIME" +%s)
(($? != 0 )) && exit 3
TO=$(date -d "$TDAY $TTIME" +%s)
(($? != 0 )) && exit 4

if (( $# == 3 && TO < FROM ))
then
   #FROM time later that TO time so add a day
   (( TO+=3600*24))
fi

if (( TO < FROM ))
then
    echo "$0: FROM date must be before TO date" >&2
    exit 5
fi


echo "Examining from $(date -d @$FROM) ($FROM)"
echo "            to $(date -d @$TO) ($TO)"
echo
FILES=/data/log/access_*.log

gawk -v F=$FROM -v T=$TO -v debug=0 '
FNR==1 {
    for(ip in C) printf "%7d %s\n", C[ip], ip
    delete C
    print "Processing " FILENAME " file"
}
$NF == "-0400]" {
  split($(NF-1),v,"[[/: ]")
  mnum=int(index("JanFebMarAprMayJunJulAugSepOctNovDec", v[3])/3)
  tm=mktime(v[4] " " mnum " " v[2] " " v[5] " " v[6] " " v[7])

  if (tm >= F && tm <= T) C[$1]++
  else if(debug) print tm " not between " F " and " T
}
END {for(ip in C) printf "%7d %s\n", C[ip], ip} ' $FILES


You can use it like this:

Code:
$ ./logcount "1 June 2015" 10:30 2:00
Examining FROM Mon, Jun 01, 2015 10:30:00 AM (1433118600)
            TO Tue, Jun 02, 2015  2:00:00 AM (1433174400)

Processing /data/log/access_abc.log file
    114 1.1.1.1
    167 2.2.2.2
   2258 3.3.3.3
Processing /data/log/access_def.log file
    102 1.1.1.1
    102 2.2.2.2
   2040 3.3.3.3

This User Gave Thanks to Chubler_XL For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Collecting all lines between two time stamp from the log

Can you help me to collect the entire logs between two time stamp. The below awk command collecting the logs only if the line has time stamp. awk '$0>=from && $0<=to' from="150318 23:19:04" to="150318 23:55:04" log file 150318 23:19:04 logentries 150318 23:29:04 logentries 150318... (11 Replies)
Discussion started by: zenkarthi
11 Replies

2. Shell Programming and Scripting

To check time stamp in log file and calculate.

Hi Friends, I have the following logfile. i want to make a script for calculate time by time2 - time1 1600266278|random|1|2014-09-19 02:08:56.024|2014-09-19 02:08:59.398|A|B|ROOM|Num0208559970111101788|1|dog|dos 1600266200|random|4|2014-09-19 02:08:06.572|2014-09-19... (2 Replies)
Discussion started by: ooilinlove
2 Replies

3. Shell Programming and Scripting

Files with date and time stamp

Hi Folks, Need a clarification on files with date and time stamp. Here is my requirement. There is a file created everyday with the following format "file.txt.YYYYMMDDHHMMSS". Now i need to check for this file and if it is available then i need to do some task to the file. I tried... (6 Replies)
Discussion started by: jayadanabalan
6 Replies

4. Shell Programming and Scripting

awk - check time stamp between range or not

I want to check given time stamp is between the given time stamp or not. I am using AIX. YYYYMMDDHHMMSS abc.csv START TIME, END TIME 20130209018000,20130509022000 20120209018000,20130509022000 20120209018000,20130509022000 Script will check given time stamp is between above two range or... (2 Replies)
Discussion started by: vegasluxor
2 Replies

5. Shell Programming and Scripting

Select files by time stamp

Hi, I need help to read file in a directory on basis of time stamp. e.g. If file access in last 2 minutes it should not be copy to remote directory. Below is my script. +++++++++++++++++++++++++ #!/bin/ksh DATE=`date +"%Y-%m-%d_%H%M"` SEPARATER=" " exec < out_interfaces.cfg... (1 Reply)
Discussion started by: qamar.alam
1 Replies

6. Shell Programming and Scripting

Identify log files based on time stamp,zip and then copy..HELP

Hi All, PFB is a requirement. I am new to shell scripting. So plz help. It would be highly appreciated. 1. choose all the log files based on a particular date (files location is '/test/domain')--i.e,we should choose all the files that are modified on 29th November, neither 28th nor 30th 2.... (3 Replies)
Discussion started by: skdas_niladri
3 Replies

7. Shell Programming and Scripting

Old time stamp being updated for new files

Hello Friends I am facing a weird problem :confused:, we receive thousands of files in my system on a daily basis, access time stamp on some of the files are being updated as old time stamp like 1968-01-19, Could some one help me what could be causing this? so that i can narrow down the problem... (4 Replies)
Discussion started by: Prateek007
4 Replies

8. Shell Programming and Scripting

time stamp perl script error out of range 1..31

Hi, while running the perl script i am getting this error message , Day '' out of range 1..31 at rsty.sh line 44 what do iam missing in the script, any suggestion #!/usr/bin/perl use Time::Local; my $wday = $ARGV; my $month = $ARGV; # convert the month shortname into 0-11 number if... (4 Replies)
Discussion started by: saha
4 Replies

9. Solaris

doubt reg time stamp in files.

I copied a file from one host to another using sftp. But after copying the time stamp is not updating . Even though I checked the permission, it looks good. I copied the same file to some temporary location, there it updating the time stamp. Anyone have any idea on this (6 Replies)
Discussion started by: rogerben
6 Replies

10. UNIX for Dummies Questions & Answers

How to search for files based on the time stamp

Hi All, I know the timestamp of a file. Now i would like to list all the files in the with the same time stamp in the same file. Any help would be appreciated. Thanks. sunny (1 Reply)
Discussion started by: sunny_03
1 Replies
Login or Register to Ask a Question