Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Grepping log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Grepping log file
# 15  
Old 10-21-2009
You could just do a grep on a special date and write it to a file which you then check with the little awk line. Not sure if this should be a returning automated mechanism as report for some people that demand it or if it is just for you occasionally.
It can also be included into the awk so grep would not be needed. When I have time again I check it out if nobody comes first Smilie Smilie
# 16  
Old 10-21-2009
Thanks for the feedback,

An automated solution would be the ideal, the files are too big and will be stored in archive. Od course in case of searching a specific range date date, the awk would suit just fine.

---------- Post updated at 05:29 AM ---------- Previous update was at 05:28 AM ----------

Code:
awk '$0>=from&&$0<=to' from="2007/03/20 15:13" to="2007/08/19 14:31" log.dates



---------- Post updated at 05:33 AM ---------- Previous update was at 05:29 AM ----------

As for occasional reports, i think that excel pivot tables give an excelent result for visualization
# 17  
Old 10-21-2009
Defining a pattern range - this has nothing to do with arithmetics or something, just a pattern/string/text matching a range (to keep in mind!):

Code:
$> cat infile
16-SEP-2009 09:11:47 10.65.4.24
16-SEP-2009 09:11:47 10.3.4.11
30-SEP-2009 10:11:47 10.3.4.11
1-OCT-2009 10:11:47 10.65.4.24
6-OCT-2009 10:11:47 10.3.4.11
6-OCT-2009 12:31:01 10.3.4.11
16-OCT-2009 11:11:47 10.65.4.24
17-OCT-2009 11:11:47 10.65.4.24
18-OCT-2009 11:11:47 10.3.4.11
$> awk 'NR==FNR && $1 ~ /30-SEP-2009/,/16-OCT-2009/ {a[$3]+=1; next} END{for(x in a){print x,"->",a[x]}}' infile
10.65.4.24 -> 2
10.3.4.11 -> 3

The range is telling awk to take all from the 1st pattern up to the 1st match of the 2nd pattern. So if there are more entries for 16-OCT-2009 you'd better thake 17-OCT, if there is any entries for it. Or maybe just take a complete month to be sure without a range. Instead of a range there could stand a pattern matching for a day, month etc.
Play arround with awk pattern matching Smilie
# 18  
Old 10-21-2009
Thanks! I changed the pattern to a time range like and got a strange output :

Code:
awk 'NR==FNR && $2 ~ /15:44:13/,/15:44:14/ {a[$3]+=1; next} END{for(x in a){print x,"->",a[x]}}' access1.log


 -> 2.4.24
 -> 1.4.17
 -> 1.4.8

The counting is done, but part of the ip vanished! Smilie

---------- Post updated at 09:30 AM ---------- Previous update was at 08:51 AM ----------

ok, sorry, i have it working, my file had more than one space between each scalar... Smilie

---------- Post updated at 10:13 AM ---------- Previous update was at 09:30 AM ----------

Hi again,

Is it possible to put this in a perl script or a shell unix script? This is powerfull, but it only works changing the ranges each time. Automation of the process would indeed optimize work! Having a file that could be run from the console, and according to its granularity, split the file and data into severel intervals and sub-files of 1 hour, or split the file and data into subfiles of days, or if bigger splitting into weeks. I look foward from hearing your advice...
# 19  
Old 10-21-2009
So far, you can just take this line and put it in a file called dosomething.sh for example, change it's permissions with chown +x dosomething.sh and fire it off with ./dosomething.sh.

Sure, this can be done in perl too. But that's a job for someone else Smilie
# 20  
Old 10-22-2009
Dear All,

I have this code that gives me a list and a count of duplicates in the range specified.
Is there a way of creating a generic pattern for hours/days? Can this range be fixed for a huge file with lots of days?

Code:
 awk ' NR==FNR && $2 ~ /06:00:00/,/15:44:14/{print "---"$1"---" $2"---" $3  } {a[$3]+=1; next }END{for(x in a){print" # Ocurrences --->",  x,"--->",a[x]  }}' right.space



---------- Post updated at 07:14 AM ---------- Previous update was at 06:43 AM ----------

So when it "recieves" one bigger file of a week time frame, it would generate several files, named based on the day...f.e. 10-OCT-2009.file, 11-OCT-2009.file and so on....
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

ksh Script, Reading A File, Grepping A File Contents In Another File

So I'm stumped. First... APOLOGIES... my work is offline in an office that has zero internet connectivity, as required by our client. If need be, I could print out my script attempts and retype them here. But on the off chance... here goes. I have a text file (file_source) of terms, each line... (3 Replies)
Discussion started by: Brusimm
3 Replies

2. Shell Programming and Scripting

Grepping only dates from a log file

Hi All, I have a log file where every line contains a date and some other data, i want to grep only the date from every line to a different file. Please help how to get this. Thanks in advance !! (25 Replies)
Discussion started by: nanz143
25 Replies

3. Shell Programming and Scripting

Grepping file and returning passed variable if the value does not exist in file at all.

I have a list of fields that I want to check a file for, returning that field if it not found at all in the file. Is there a way to do a grep -lc and return the passed variable too rather then just the count? I am doing some crappy work-around now but I was not sure how to regrep this for :0 so... (3 Replies)
Discussion started by: personalt
3 Replies

4. UNIX for Dummies Questions & Answers

grepping log files

I have a log file and I have two unique strings which represent the start and end of the text I want to obtain. How can I get all the text inbetween this start string and the end string? Thanks (2 Replies)
Discussion started by: chrisjones
2 Replies

5. Shell Programming and Scripting

Grepping string from out file

Guys .. Need to pull this highlighted strings irrespective of line numbers & should be echoed . But these strings are from Outfile from different dir. In which way this can be grepped ?? Need an idea http-timeout 120 seconds persistent-timeout 180 seconds host-rewriting on ... (7 Replies)
Discussion started by: raghunsi
7 Replies

6. Shell Programming and Scripting

Grepping the last 30 minutes of a log file...

I need to know if anyone can assist me on how to grab the last (we'll just say "x" minutes) of a log file. How do you tell the grep command without specifying an exact window of time? (So relative instead of absolute.) Thanks, Jon (2 Replies)
Discussion started by: jtelep
2 Replies

7. Shell Programming and Scripting

Grepping Errors in a file

Hey All, I have to grep for an error from a file and get the results of errror in a different file...... But there should be no duplicate entries. Can anyone help me in giving a shell script for this This is file which contains pattern error which I am supposed to grep and put this in a... (4 Replies)
Discussion started by: achararun
4 Replies

8. Shell Programming and Scripting

Loop and grepping into a file

I wrote this script for: 1. Get the Web log for today 2. Give me a list of all the IP addresses that have accessed the web server today 3. Remove a list of known IPs listed in a file (line by line) 4. Mail the final file to selected recipients. I am unable to do part 3. In the script... (3 Replies)
Discussion started by: skotapal
3 Replies

9. UNIX for Dummies Questions & Answers

grepping the first 3 characters from a file

give this a try and let me know if it works grep '^' filename rachael (2 Replies)
Discussion started by: rachael
2 Replies

10. UNIX for Dummies Questions & Answers

grepping the first 3 characters from a file

Hi I was wondering if it's possible to use a command to get the first 3 characters of a line in a text file, I tried grep but it returns the whole line but I am only interested in the first 3 characters. Is this possible with grep or I need any other command? Also is it possible deleting from... (2 Replies)
Discussion started by: g-e-n-o
2 Replies
Login or Register to Ask a Question