Extract data from log file within specified time


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract data from log file within specified time
# 1  
Old 10-14-2014
Question Extract data from log file within specified time

So, we have a script, that is supposed to have a couple of functions like showing number of failed connections, recieved bytes per IP-address, and so on. We are supposed to be able to limit the number of results to either 0-24 hours or X days back from the last data in the log file.

Everything is working, except we dont know how to limit searches within a given timespace.

Our code looks like this:
Code:
#!/bin/sh

#-n: Limit the number of results to N
#-h: Limit the query to the last number of hours (< 24)
#-d: Limit the query to the last number of days (counting from
#midnight)
#-c: Which IP address makes the most number of connection attempts?
#-2: Which address makes the most number of successful attempts?
#-r: What are the most common results codes and where do they come
#from?
#-F: What are the most common result codes that indicate failure (no
#auth, not found etc) and where do they come from?
#-t: Which IP number get the most bytes sent to them?

#<filename> refers to the logfile. If '-' is given as a filename, or
#no filename is given, then standard input should be read. This
#enables the script to be used in a pipeline.

FILENAME=*.log
MAXSHOW=99999
LIMITHOURS=0
LIMITDAYS=0
h=1
c=0
b=0
r=0
F=0
t=0
while getopts :n:h:d:c2rFt option
do    
    case $option in
    n)
        MAXSHOW=$OPTARG
        ;;
    h)
        LIMITHOURS=$OPTARG
        ;;
    d)
        LIMITDAYS=$OPTARG
        ;;
    c)
        c=1
            ;;
    2)
        b=1
        ;;
    r)
        r=1
        ;;
    F)
        F=1
        ;;
    t)
        t=1
        ;;
    esac
done
if [ "$h" -eq "1" ]; then
   #?????
fi

if [ "$d" -eq "1" ]; then
#??????
fi

if [ "$c" -eq "1" ]; then
    cat $FILENAME|awk '{print $1}' |sort|uniq -c|sort -k 1 -n -r|head -$MAXSHOW
fi

if [ "$b" -eq "1" ]; then
    grep -Eo "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}.* 200" $FILENAME|awk '{print $1}'|sort|uniq -c|sort -nr|head -$MAXSHOW
fi

if [ "$r" -eq "1" ]; then
    cat $FILENAME|awk '{print $1" "$9}'|sort|uniq -c|sort -nr|head -$MAXSHOW
fi

if [ "$F" -eq "1" ]; then
    cat $FILENAME | if $9 > "200" ; then
        awk '{print $1" "$9}' |sort|uniq -c|sort -nr|head -$MAXSHOW
    fi
fi

if [ "$t" -eq "1" ]; then
cat $FILENAME |awk '{print $1" "$10}'|awk '{ x[$1]+=$2 } END{for(data in x) print data, x[data]}' | sort -k2,2 -nr|head -$MAXSHOW
fi

And our log file is full of data like this:
Code:
213.46.27.204 - - [01/Jan/2003:12:55:20 +0100] "GET /scripts/..%%35%63../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" "" 
213.46.27.204 - - [01/Jan/2003:12:55:20 +0100] "GET /scripts/..%%35c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" "" 
213.46.27.204 - - [01/Jan/2003:12:55:20 +0100] "GET /scripts/..%25%35%63../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" "" 
213.46.27.204 - - [01/Jan/2003:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""

If anyone knows how we can fix this problem, we would be very thankful!
# 2  
Old 10-14-2014
All those cats are completely useless, see useless use of cat award. They are particularly harmful on Windows, whose scheduler deals badly with too many short-lived programs.

If you rearrange your dates into YYYY/MM/DD HH:MM:SS order they easily compare alphabetically, which is why the rest of the world has converted to this order. You can do this rearranging inside awk, though a lookup table of month names is needed.

Code:
awk -F"[ \t/:\\\[\\\]]+" 'BEGIN { split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", A); for(X in A) M[A[X]]=sprintf("%02d", X) }
# Create timestamp in YYYY/MM/DD:HH:MM:SS so it can be compared to TSTART and TEND, which controls printing of lines
{ T=$6"/"M[$5]"/"$4":"$7":"$8":"$9 } (T >= TSTART) && (T <= TEND)' TSTART="2003/01/01:12:55:20" TEND="2003/01/01:12:55:20" inputfile

If awk doesn't work, try gawk or nawk.

Last edited by Corona688; 10-14-2014 at 12:54 PM..
# 3  
Old 10-14-2014
The thing is we are supposed to input for example: ./script.sh -h 5 -c or ./script.sh -d 4 -c and our output is then supposed to be the IP-addresses most connected during the past 5 hours or past 4 days.

We should be able to use -h or -d together with the other flags.

Last edited by rbatte1; 10-14-2014 at 01:54 PM.. Reason: Added ICODE tags
# 4  
Old 10-14-2014
Thanks for telling me that now.

Do you have GNU date? It can calculate time offsets and print date stamps like I do with awk there.

Code:
awk ... TSTART=$(date -d "- 5 day" +"%Y/%m/%d:%H:%M:%S") TEND=$(date +"%Y/%m/%d:%H:%M:%S") filename

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract data from a log file and put it in a file

Hi, I would like to seek your help for a script that will extract data from log file and put it in a file. Sample log file 2018-10-23 12:33:21 AI ERROR -- tpid: SAMPLE_TH account: 123456789 aiSessionNumber: 660640464 mapName: xxx to yyy errorDesc: Translation Error:ErrorNumber : 993 ... (2 Replies)
Discussion started by: neverwinter112
2 Replies

2. Shell Programming and Scripting

Archiving or removing few data from log file in real time

Hi, I have a log file that gets updated every second. Currently the size has grown to 20+ GB. I need to have a command/script, that will try to get the actual size of the file and will remove 50% of the data that are in the log file. I don't mind removing the data as the size has grown to huge... (8 Replies)
Discussion started by: Souvik Patra
8 Replies

3. UNIX for Dummies Questions & Answers

Extract date ranged data from log file

Hi, I am trying to extract lines of data within a log file on a Redhat 5 Linux system. eg I need all the lines with a particular username over the last 3 minutes. the log file may read like this, and I want a way to search all the lines extracting all the relevant lines over the last 3... (2 Replies)
Discussion started by: mantis100
2 Replies

4. Shell Programming and Scripting

Extract data from log file in specified range of time

I was searching for parsing a log file and found what I need in this link http://stackoverflow.com/questions/7575267/extract-data-from-log-file-in-specified-range-of-time But the most useful answer (posted by @Kent): # this variable you could customize, important is convert to seconds. # e.g... (2 Replies)
Discussion started by: kingk110
2 Replies

5. Shell Programming and Scripting

Data Extract from XML Log File

Please help me out to extract the Data from the XML Log files. So here is the data ERROR|2010-08-26 00:05:52,958|SERIAL_ID=128279996|ST=2010-08-2600:05:52|DEVICE=113.2.21.12:601|TYPE=TransactionLog... (9 Replies)
Discussion started by: raghunsi
9 Replies

6. Shell Programming and Scripting

need a shell script to extract data from a log file.

If I have a log like : Mon Jul 19 05:07:34 2010; TCP; eth3; 52 bytes; from abc to def Mon Jul 19 05:07:35 2010; UDP; eth3; 46 bytes; from aaa to bbb Mon Jul 19 05:07:35 2010; TCP; eth3; 52 bytes; from def to ghi I will need an output like this : Time abc to def... (1 Reply)
Discussion started by: hitha87
1 Replies

7. Shell Programming and Scripting

Extract info from log file and compute using time date stamp

Looking for a shell script or a simple perl script . I am new to scripting and not very good at it . I have 2 directories . One of them holds a text file with list of files in it and the second one is a daily log which shows the file completion time. I need to co-relate both and make a report. ... (0 Replies)
Discussion started by: breez_drew
0 Replies

8. Shell Programming and Scripting

how to make a log file of extract time

Dear All, Please apology to me if this question already posted, because I try to find it but not found. I have make bash script to automatically download data from ftp and this running very well. and after the data downloaded it will automatically extract the data and keep in the specific... (2 Replies)
Discussion started by: chenboly
2 Replies

9. Shell Programming and Scripting

Extract data from log file from or after the specific date

Hi , I am having a script which will start a process and appends the process related logs to a log file. The log file writes logs with every line starting with date in the format of: date +"%Y %b %d %H:%M:%S". So, in the script, before I start the process, I am storing the date as DATE=`date +"%Y... (5 Replies)
Discussion started by: chiru_h
5 Replies

10. Shell Programming and Scripting

shell-script which extract data from log file

give me a shell-script which extract data from log file on a server by giving date and time as input (for both start time and end time) and it will give the logs generated during the given time as output. (4 Replies)
Discussion started by: abhishek27
4 Replies
Login or Register to Ask a Question