awk and log files


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers awk and log files
# 1  
Old 04-25-2011
awk and log files

Hello,

I have a series of logs that I need to analyse.
each looks something like:
234.10.72.175 Mon Mar 02 20:25:00 GMT 2009
226.91.87.86 Thu Mar 05 03:50:26 GMT 2009
226.91.87.86 Thu Mar 05 04:06:07 GMT 2009

Using awk, so far I have been able to count the lines in a specific file that match a patter. For example, those in Mar, 2009, between 10:00:00 and 12:00:00. But a combination of the text month and numeric year have me stumped.

an example of the lines I have been using
Code:
# between two times
awk -v "sTIME=${STARTTIME}" -v "eTIME=${ENDTIME}" '{if ($5 >= sTIME && $5 <= eTIME) print $1 }' home.hits 

#day begins with S, so a weekend
awk '$2 ~ /^S/' test.hits 

# doesnt match above...
awk '$2 !~ /^S/' test.hits

I am now having trouble with the date as they are in the mmm format.

I have thought about using sed to change the months format.
using an array and piping the output of awk into another to check.

change the times to epoch


I am currently not sure what the best way to do this is.
Cheers for looking
# 2  
Old 04-25-2011
What is your ultimate goal? You are doing a report, it seems, so how do you want to group the final results? ie., group ip address by month, for workday activity?

For example:
Code:
awk ' /(Mon|Tue|Wed|Thu|Fri)/  { arr[$1 " " $3]++ }
        END (for i in arr) { print i, arr[i] } ' logfile  | sort

# 3  
Old 04-26-2011
Cheers for the reply Jim,

I would like to be able to apply two patterns to the files.
So that I am able to get the records that are from Feb 2009 to Mar 2010 for example.
# 4  
Old 04-26-2011
Code:
awk '
       /Apr/ && /2010/ {ok=0}
       /Feb/ && /2009/ {ok=1}
      ok { your awk code to format and group the data goes here }
      {next}  ' logfile > report

The {next} is needed if you are using an ancient version of awk.

This awk syntax is:
Code:
    boolean { function }

where boolean evaluates to true or false. By default undeclared values like ok - are zero.
This turns on processing for the first occurrence on Feb && 2009. turns it off for the first occurrence of Apr 2010 - ie. just past Mar 2010
# 5  
Old 05-07-2011
Jim,
sorry for my delayed reply.
I dont understand where you have used ok.

are you able to explain it further?
many thanks
# 6  
Old 05-08-2011
After running this, but with Apr+ 2006 and Feb+2009
I only got results from Feb and Mar 2009
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Redirecting log files to null writing junk into log files

Redirecting log files to null writing junk into log files. i have log files which created from below command exec <processname> >$logfile but when it reaches some size i am redirecting to null while process is running like >$logfile manually but after that it writes some junk into... (7 Replies)
Discussion started by: greenworld123
7 Replies

2. Shell Programming and Scripting

[awk] combine and convert time from log files

dear all, an awk newbie need your help.... i have log files with this format: mylog1a.log: "08/10/2012","5:05 PM" "Hostname","Device Address","Count" "","10.10.10.18","10234" mylog2a.log: "08/11/2012","5:05 PM" "Hostname","Device Address","Count" "","10.10.10.18","12543" ... (18 Replies)
Discussion started by: makan
18 Replies

3. Shell Programming and Scripting

awk command to compare a file with set of files in a directory using 'awk'

Hi, I have a situation to compare one file, say file1.txt with a set of files in directory.The directory contains more than 100 files. To be more precise, the requirement is to compare the first field of file1.txt with the first field in all the files in the directory.The files in the... (10 Replies)
Discussion started by: anandek
10 Replies

4. Shell Programming and Scripting

Comparing the matches in two files using awk when both files have their own field separators

I've two files with data like below: file1.txt: AAA,Apples,123 BBB,Bananas,124 CCC,Carrot,125 file2.txt: Store1|AAA|123|11 Store2|BBB|124|23 Store3|CCC|125|57 Store4|DDD|126|38 So,the field separator in file1.txt is a comma and in file2.txt,it is | Now,the output should be... (2 Replies)
Discussion started by: asyed
2 Replies

5. Shell Programming and Scripting

How can view log messages between two time frame from /var/log/message or any type of log files

How can view log messages between two time frame from /var/log/message or any type of log files. when logfiles are very big and especially many messages with in few minutes, I would like to display log messages between 5 minute interval. Could you pls give me the command? (1 Reply)
Discussion started by: johnveslin
1 Replies

6. Shell Programming and Scripting

Apply 'awk' to all files in a directory or individual files from a command line

Hi All, I am using the awk command to replace ',' by '\t' (tabs) in a csv file. I would like to apply this to all .csv files in a directory and create .txt files with the tabs. How would I do this in a script? I have the following script called "csvtabs": awk 'BEGIN { FS... (4 Replies)
Discussion started by: ScKaSx
4 Replies

7. Shell Programming and Scripting

Comparison and editing of files using awk.(And also a possible bug in awk for loop?)

I have two files which I would like to compare and then manipulate in a way. File1: pictures.txt 1.1 1.3 dance.txt 1.2 1.4 treehouse.txt 1.3 1.5 File2: pictures.txt 1.5 ref2313 1.4 ref2345 1.3 ref5432 1.2 ref4244 dance.txt 1.6 ref2342 1.5 ref2352 1.4 ref0695 1.3 ref5738 1.2... (1 Reply)
Discussion started by: linuxkid
1 Replies

8. Shell Programming and Scripting

Arrange log files with AWK

Hello friends, I have too many log files to arrange. I use a simple script to create log files with below format and i forgot to create daily directory for them at the beginning. Because of this i should move all daily logs into a directory that i need to create. a part of "ls -l" output:... (1 Reply)
Discussion started by: EAGL€
1 Replies

9. Shell Programming and Scripting

Merge files of differrent size with one field common in both files using awk

hi, i am facing a problem in merging two files using awk, the problem is as stated below, file1: A|B|C|D|E|F|G|H|I|1 M|N|O|P|Q|R|S|T|U|2 AA|BB|CC|DD|EE|FF|GG|HH|II|1 .... .... .... file2 : 1|Mn|op|qr (2 Replies)
Discussion started by: shashi1982
2 Replies

10. UNIX for Dummies Questions & Answers

Constantly updating log files (tail -f? grep? awk?)

I have a log file which is continuously added to, called log.file. I'd like to monitor this file, and when certain lines are found, update some totals in another file. I've played around with tail -f, grep, and awk, but can't seem to hit the right note, so to speak. The lines I'm... (0 Replies)
Discussion started by: nortonloaf
0 Replies
Login or Register to Ask a Question