Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using SED/AWK to Summarize Log File in 10min Intervals

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using SED/AWK to Summarize Log File in 10min Intervals
# 1  
Old 01-29-2011
Using SED/AWK to Summarize Log File in 10min Intervals
I have this huge log file on my linux box that gets generated every day. I'm able to extract the information I need; however I really would like it to be broken down every 10mins.

Log File Snippet
HTML Code:
01:23:45 MARYHADA  Maryhadalittle.lamb(): fleece as white as snow 1394 for and everywhere that Mary went 0x12 lamb was free to go
One Liner to Extract Info
HTML Code:
 grep -i '0x12' maryhada.log.2011-01-28 |sed 's/:[0-9][0-9] / /g;s/.*-[0-9]\{4\} //' |awk '{print $4, $18}' |sort |uniq -c
Output
HTML Code:
1 00:03:40 1432
      1 00:03:54 1432
      1 00:04:15 1432
      1 00:04:29 1432
      1 00:04:36 1432
      1 00:04:50 1432
      1 00:04:59 1432
      1 00:06:21 1432
      1 00:06:32 1432
      1 00:06:41 1432
      1 00:09:06 1432
      1 00:09:13 1432
      1 00:11:08 1432
      1 00:11:09 1432
      1 00:11:19 1432
      1 00:19:35 1571
      1 00:19:41 1571
      1 00:19:44 1571
      1 00:19:49 1571
      1 00:19:53 1571
      1 00:19:54 1571
      1 00:19:57 1571
      1 00:19:59 1571
      1 00:23:11 1429
Last edited by ravzter; 01-29-2011 at 09:22 AM..
# 2  
Old 01-30-2011
Code:
awk ' /0x12/{   
     split($1,a,":");
     min=int(a[2]/10)*10;
     b[a[1] ":" sprintf("%02d",min) FS $9]++
   }
END{for (i in b) print b[i],i }
' maryhada.log.2011-01-28

12 00:00 1432
3 00:10 1432
1 00:20 1429
8 00:10 1571

Last edited by rdcwayx; 01-30-2011 at 03:08 AM..
This User Gave Thanks to rdcwayx For This Post:
ravzter
# 3  
Old 01-31-2011
Hey Thanks RDC!

The count works flawless however I apologize for not providing the full string in the log. Below I've include the full line. I'm looking to display just the time from the GMT portion. I'm certain it's just an adjustment to the script cause you have the output I desire.

Actual Log File Snippet
HTML Code:
 2011/01/28 05:19:44.187 GMT(01/28 00:19:44 -0500) ERROR  MARY  [HADALITTLELAMB-92]  - ITFLEECWASWHITEASSNOW(): every where that mary went lamb free 1571 to go 001000011100
- MaryLambFleece response 0x12 was not COOL
HTML Code:
 5 2011/01/29:00 1379
3 2011/01/28:00 570
175 2011/01/29:00 1394
6 2011/01/28:00 807
2 2011/01/28:00 1377
55 2011/01/29:00 1430
5 2011/01/28:00 822
3 2011/01/28:00 789
95 2011/01/29:00 1432
3 2011/01/28:00 823
2 2011/01/28:00 824
New Output Desired
HTML Code:
00:10 95 Occurrences found in ID:1432
00:20 3 Occurrences found in ID:823
00:10 175 Occurrences found in ID:1394
# 4  
Old 01-31-2011
The line "- MaryLambFleece response 0x12 was not COOL" is separate line or not? Above sample has two lines or only one line?

Do you need summarize Log File in 10min Intervals or just calculate the daily summary?
# 5  
Old 01-31-2011
Thanks RDC!

Yeah I need the total at10min intervals for the day.
Also this is not a new line it's all part of the string separated by the "-."

"- MaryLambFleece response 0x12 was not COOL"
# 6  
Old 01-31-2011
Code:
awk ' /0x12/{ split($4,a,":");min=int(a[2]/10)*10;date=substr($3,5);b[date FS a[1] ":" sprintf("%02d",min) FS $(NF-10)]++}
END{for (i in b) print i,b[i] }' infile

This User Gave Thanks to rdcwayx For This Post:
ravzter
# 7  
Old 01-31-2011
Thnx RDC!!! Works perfectly!!! Really appreciate your time and effort!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using awk to Summarize Log File in 5min Intervals

I have huge log file that taken every minute and I need the total at 5min intervals. Sample log: #timestamp(yyyymmddhhmm);result;transaction 201703280000;120;6 201703280001;120;3 201703280002;105;3 201703280003;105;5 201703280004;105;5 201703280005;105;4 201703280006;120;2... (2 Replies)
Discussion started by: wwolfking
2 Replies

2. Shell Programming and Scripting

Help on Log File format using sed or awk

Hello Gurus, First, i would like to know is there any way to solve my problem. i have a log file like this: INFO - ABCDRequest :: processing started for the record <0> TransactionNo <Txn#1> recordID <recID#1> INFO - ABCDRequest :: processing started for the record <0> TransactionNo... (9 Replies)
Discussion started by: VasuKukkapalli
9 Replies

3. Shell Programming and Scripting

Compare intervals (columns) from two files (awk, grep, Perl?)

Hi dear users, I need to compare numeric columns in two files. These files have the following structure. K.txt (4 columns) A001 chr21 9805831 9846011 A002 chr21 9806202 9846263 A003 chr21 9887188 9988593 A003 chr21 9887188 ... (2 Replies)
Discussion started by: jcvivar
2 Replies

4. Shell Programming and Scripting

Summarize file with column matching

Guys, Please help me with this code. I have 2GB file to process and shell seems to be the best option. I am a biologist and though I can think of the logic, the commands are beyond me. Any help is greatly appreciated. Please look at the attched file and the requirement will be very clear. I... (6 Replies)
Discussion started by: newbie83
6 Replies

5. Shell Programming and Scripting

Use grep sed or awk to extract string from log file and put into CSV

I'd like to copy strings from a log file and put them into a CSV. The strings could be on different line numbers, depending on size of log. Example Log File: File = foo.bat Date = 11/11/11 User = Foo Bar Size = 1024 ... CSV should look like: "foo.bat","11/11/11","Foo Bar","1024" (7 Replies)
Discussion started by: chipperuga
7 Replies

6. Shell Programming and Scripting

running a 5-10min SW on the server and use the result right after

Hi everyone, thanks in advance for your kind help! Here is what i need to achieve (simplificated example of course): 1-i have a "300-hour-music.mp3" file in my server HD side 2-a page HTML/PHP gets user input : the length of the track he want to listen, let's say 2 hours 3-a BASH script... (0 Replies)
Discussion started by: mygost
0 Replies

7. Shell Programming and Scripting

Shell Script - Copy File at intervals

Hi, I want to copy some files from a Folder say, /usr/X at random intervals to another location. Basically, new files will be dumped at random intervals to location /usr/X and I have to copy those new files to some other location (after copying, I cannot delete those files from source... (2 Replies)
Discussion started by: angshuman_ag
2 Replies

8. Shell Programming and Scripting

Summarize the sed script

Hi folks, I have a situation where i have a raw file like cat file_raw 776 713 111 0776713113 317-713-114 235776713115 776713116 336713117 77 6 713 118 0776713119 235776713120 and would like to replace all leading zeros with 235, remove all spaces and dashes, and make all... (3 Replies)
Discussion started by: jerkesler
3 Replies

9. UNIX for Advanced & Expert Users

WEB Server Log File Analysis using awk/sed/grep

I'm trying to find a way to show large page sizes (page size in K) from multiple web server log files. Essentially I want to show only rows from a file where a specific column is larger than some value. Has anyone ever done this type of log analysis? If so, a snippet of code would be very... (2 Replies)
Discussion started by: mike_cataldo@ad
2 Replies

10. Linux

SED/AWK Script to clear log file using timestamp?

I have a log file on our system which fills up with lines that have been timestamped, as follows.... 03/03/2008 10:56:06:815] (ERROR) balance: continuing session to genapp02 : 18500 03/03/2008 10:56:06:820] (ERROR) balance: continuing session to genapp02 : 18500 03/03/2008 10:56:07:003]... (2 Replies)
Discussion started by: davesimm
2 Replies
Login or Register to Ask a Question