01-29-2011
Using SED/AWK to Summarize Log File in 10min Intervals
I have this huge log file on my linux box that gets generated every day. I'm able to extract the information I need; however I really would like it to be broken down every 10mins.
Log File Snippet
HTML Code:
01:23:45 MARYHADA Maryhadalittle.lamb(): fleece as white as snow 1394 for and everywhere that Mary went 0x12 lamb was free to go
One Liner to Extract Info
HTML Code:
grep -i '0x12' maryhada.log.2011-01-28 |sed 's/:[0-9][0-9] / /g;s/.*-[0-9]\{4\} //' |awk '{print $4, $18}' |sort |uniq -c
Output
HTML Code:
1 00:03:40 1432
1 00:03:54 1432
1 00:04:15 1432
1 00:04:29 1432
1 00:04:36 1432
1 00:04:50 1432
1 00:04:59 1432
1 00:06:21 1432
1 00:06:32 1432
1 00:06:41 1432
1 00:09:06 1432
1 00:09:13 1432
1 00:11:08 1432
1 00:11:09 1432
1 00:11:19 1432
1 00:19:35 1571
1 00:19:41 1571
1 00:19:44 1571
1 00:19:49 1571
1 00:19:53 1571
1 00:19:54 1571
1 00:19:57 1571
1 00:19:59 1571
1 00:23:11 1429
Last edited by ravzter; 01-29-2011 at 09:22 AM..
10 More Discussions You Might Find Interesting
1. Linux
I have a log file on our system which fills up with lines that have been timestamped, as follows....
03/03/2008 10:56:06:815] (ERROR) balance: continuing session to genapp02 : 18500
03/03/2008 10:56:06:820] (ERROR) balance: continuing session to genapp02 : 18500
03/03/2008 10:56:07:003]... (2 Replies)
Discussion started by: davesimm
2 Replies
2. UNIX for Advanced & Expert Users
I'm trying to find a way to show large page sizes (page size in K) from multiple web server log files. Essentially I want to show only rows from a file where a specific column is larger than some value. Has anyone ever done this type of log analysis? If so, a snippet of code would be very... (2 Replies)
Discussion started by: mike_cataldo@ad
2 Replies
3. Shell Programming and Scripting
Hi folks,
I have a situation where i have a raw file like
cat file_raw
776 713 111
0776713113
317-713-114
235776713115
776713116
336713117
77 6 713 118
0776713119
235776713120
and would like to replace all leading zeros with 235, remove all spaces and dashes, and make all... (3 Replies)
Discussion started by: jerkesler
3 Replies
4. Shell Programming and Scripting
Hi,
I want to copy some files from a Folder say, /usr/X at random intervals to another location.
Basically, new files will be dumped at random intervals to location /usr/X and I have to copy those new files to some other location (after copying, I cannot delete those files from source... (2 Replies)
Discussion started by: angshuman_ag
2 Replies
5. Shell Programming and Scripting
Hi everyone, thanks in advance for your kind help!
Here is what i need to achieve (simplificated example of course):
1-i have a "300-hour-music.mp3" file in my server HD side
2-a page HTML/PHP gets user input : the length of the track he want to listen, let's say 2 hours
3-a BASH script... (0 Replies)
Discussion started by: mygost
0 Replies
6. Shell Programming and Scripting
I'd like to copy strings from a log file and put them into a CSV.
The strings could be on different line numbers, depending on size of log.
Example Log File:
File = foo.bat
Date = 11/11/11
User = Foo Bar
Size = 1024
...
CSV should look like:
"foo.bat","11/11/11","Foo Bar","1024" (7 Replies)
Discussion started by: chipperuga
7 Replies
7. Shell Programming and Scripting
Guys,
Please help me with this code. I have 2GB file to process and shell seems to be the best option. I am a biologist and though I can think of the logic, the commands are beyond me. Any help is greatly appreciated. Please look at the attched file and the requirement will be very clear.
I... (6 Replies)
Discussion started by: newbie83
6 Replies
8. Shell Programming and Scripting
Hi dear users,
I need to compare numeric columns in two files. These files have the following structure.
K.txt (4 columns)
A001 chr21 9805831 9846011
A002 chr21 9806202 9846263
A003 chr21 9887188 9988593
A003 chr21 9887188 ... (2 Replies)
Discussion started by: jcvivar
2 Replies
9. Shell Programming and Scripting
Hello Gurus,
First, i would like to know is there any way to solve my problem.
i have a log file like this:
INFO - ABCDRequest :: processing started for the record <0> TransactionNo <Txn#1> recordID <recID#1>
INFO - ABCDRequest :: processing started for the record <0> TransactionNo... (9 Replies)
Discussion started by: VasuKukkapalli
9 Replies
10. Shell Programming and Scripting
I have huge log file that taken every minute
and I need the total at 5min intervals.
Sample log:
#timestamp(yyyymmddhhmm);result;transaction
201703280000;120;6
201703280001;120;3
201703280002;105;3
201703280003;105;5
201703280004;105;5
201703280005;105;4
201703280006;120;2... (2 Replies)
Discussion started by: wwolfking
2 Replies
LEARN ABOUT SUSE
log::dispatch::file
Log::Dispatch::File(3) User Contributed Perl Documentation Log::Dispatch::File(3)
NAME
Log::Dispatch::File - Object for logging to files
SYNOPSIS
use Log::Dispatch;
my $log =
Log::Dispatch->new
( outputs =>
[ [ 'File',
min_level => 'info',
filename => 'Somefile.log',
mode => '>>',
newline => 1 ]
],
);
$log->emerg("I've fallen and I can't get up");
DESCRIPTION
This module provides a simple object for logging to files under the Log::Dispatch::* system.
Note that a newline will not be added automatically at the end of a message by default. To do that, pass "newline => 1".
CONSTRUCTOR
The constructor takes the following parameters in addition to the standard parameters documented in Log::Dispatch::Output:
o filename ($)
The filename to be opened for writing.
o mode ($)
The mode the file should be opened with. Valid options are 'write', '>', 'append', '>>', or the relevant constants from Fcntl. The
default is 'write'.
o binmode ($)
A layer name to be passed to binmode, like ":utf8" or ":raw".
o close_after_write ($)
Whether or not the file should be closed after each write. This defaults to false.
If this is true, then the mode will aways be append, so that the file is not re-written for each new message.
o autoflush ($)
Whether or not the file should be autoflushed. This defaults to true.
o permissions ($)
If the file does not already exist, the permissions that it should be created with. Optional. The argument passed must be a valid
octal value, such as 0600 or the constants available from Fcntl, like S_IRUSR|S_IWUSR.
See "chmod" in perlfunc for more on potential traps when passing octal values around. Most importantly, remember that if you pass a
string that looks like an octal value, like this:
my $mode = '0644';
Then the resulting file will end up with permissions like this:
--w----r-T
which is probably not what you want.
AUTHOR
Dave Rolsky, <autarch@urth.org>
perl v5.12.1 2009-09-22 Log::Dispatch::File(3)