07-27-2009
Generating formatted reports from log files
Given that I have a log file of the format:
DATE ID LOG_LEVEL | EVENT
2009-07-23T14:05:11Z T-4030097550 D | MessX
2009-07-23T14:10:44Z T-4030097550 D | MessY
2009-07-23T14:34:08Z T-7298651656 D | MessX
2009-07-23T14:41:00Z T-7298651656 D | MessY
2009-07-23T15:05:10Z T-4030097550 D | MessZ
etc
How do I produce a report that looks like
ID = T-4030097550
2009-07-23T14:05:11Z MessX
2009-07-23T14:10:44Z MessY
2009-07-23T15:05:10Z MessZ
ID = T-7298651656
2009-07-23T14:34:08Z MessX
2009-07-23T14:41:00Z MessY
etc
Other info:
The IDs are not pre-determined
The messages in the report are actually a subset of the log-entries and some messages may be missing for a given ID.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
My aim is to generate a report using shell script. There are various formats fields coloumns etc.
I want to print in a single line (row) but in different coloumn as given below:
field1 field2 field3 field4
....... ....... ...... .......
The spacing... (1 Reply)
Discussion started by: jisha
1 Replies
2. Shell Programming and Scripting
hi i have data extracted in the following format ranging around 300000 to 800000 records in a text file , the format is of network data .
No. Time Source Destination Protocol
1 1998-06-05 17:20:23.569905 HP_61:aa:c9 HP_61:aa:c9 ... (1 Reply)
Discussion started by: renukaprasadb
1 Replies
3. Shell Programming and Scripting
I/P file name:- 20092008.txt
Check number of entries in i/p file by following command
ChkEnt -infl 20092008.txt -opfl 20092008_test.txt >count.txt
Dear Friends,
Please help me in automating following thing.
If output generated (count.txt) is having value more than 1000 i.e.... (8 Replies)
Discussion started by: anushree.a
8 Replies
4. Shell Programming and Scripting
Hi,
I have a question with sed/awk. When I handle some log files I want to search all reports with specified keyword. For example, in the log below.
abcd
efg
===start
abc
e
===end
xyz
===start
af
f
===end
nf
ga
===start
ab
===end (4 Replies)
Discussion started by: danielnpu
4 Replies
5. Red Hat
I installed sarg from sarg rpm and i am facing issue while generating sarg reports and getting this time different error below
sarg -l /var/log/squid/access.log
SARG: Records in file: 242332, reading: 0.00%
SARG: Records in file: 242332, reading: 2.06%
SARG: Records in file: 242332, reading:... (0 Replies)
Discussion started by: mail4vij
0 Replies
6. Shell Programming and Scripting
Good evening! Trying to make a shell script to parse log file and show only required information.
log file has 44 fields and alot of lines, each columns separated by ":".
log file is like:
first_1:3:4:5:6:1:3:4:5:something:notinterested
second_2:3:4:3:4:2
first_1:3:4:6:6:7:8
I am interested... (3 Replies)
Discussion started by: dummie55
3 Replies
7. Linux
Hello all,
I'm trying to find some tool on generating reports based on apache access_log files (of Common format).
I found some of them (awstats, lire/logreport, weblog expert, apache logs viewer, etc..) but they generate some global and general report about the log file.
Also some perl... (0 Replies)
Discussion started by: enux
0 Replies
8. Shell Programming and Scripting
Hi Experts,
I am having a requirement, where i need to generate a report of the execution time of all the processes. All the processes generate the log files in a log directory and I can get the execution time from the log files. like below is the log file.
/home/vikas/log >ls -l... (2 Replies)
Discussion started by: k_vikash
2 Replies
9. Shell Programming and Scripting
How to split the file and have suffix with formatted numbers
Tried the following code
awk '{filename="split."int((NR-1)/2)".txt"; print >> filename}' split.txt
Current Result
Expected Result (21 Replies)
Discussion started by: bobbygsk
21 Replies
10. UNIX for Beginners Questions & Answers
Team,
Hope you all are doing fine
I have one admin server which is being used dedicately to run cron jobs on hourly basis, fetching the details from Database which is in a different server.These cronjob are run on every hourly/5 minutes basis depending as per end user requirement.The script... (12 Replies)
Discussion started by: whizkidash
12 Replies
LEARN ABOUT DEBIAN
w3c_extended2dlf
W3C_EXTENDED2DLF.IN(1) LogReport's Lire Documentation W3C_EXTENDED2DLF.IN(1)
NAME
w3c_extended2dlf - convert W3C Extended Log file as used by MS IIS to DLF
SYNOPSIS
w3c_extended2dlf file
DESCRIPTION
w3c_extended2dlf converts web server log files in the W3C Extended Log Format to the www DLF. This log format is defined at
http://www.w3.org/TR/WD-logfile.html
It is used by IIS 4.0 and IIS 5.0. This is a customizable format which contains an header specifying which information is present in the
log.
Some documenation on these log file formats is available in the section on "Analyzing Log Files (IIS 6.0)" in the "IIS 6.0 Documentation"
at the "Microsoft Windows Server 2003 TechCenter" at http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/
Library/IIS/610b7d2c-90d6-4e40-be79-aaf88a283f03.mspx?mfr=true . The newer IIS 7.0 product is documented at http://www.iis.net/.
To have the maximum information in you reports, we suggests that you log the following fields :
date, time, c-ip, c-dns, cs-uri, cs-method, sc-bytes, sc-status,
cs(User-Agent), cs(Referer) and cs-username
We also support the cs-uri-stem field.
Other fields will be ignored.
LIMITATIONS
The converter doesn't handle aggregation (record with count field) and will refuse to process those logs. Also it doesn't support changing
the fields in the middle of the log file. It will ignore records that don't have the same schema than the first one defined.
EXAMPLES
To process a log as produced in the W3C Extended Log Format:
$ w3c_extended2dlf < extended.log
w3c_extended2dlf will be rarely used on its own, but is more likely called by lr_log2report:
$ lr_log2report w3c_extended < /var/log/httpd/extended.log
AUTHORS
Francis J. Lacoste <flacoste@logreport.org>
VERSION
$Id: w3c_extended2dlf.in,v 1.17 2009/03/15 08:10:55 vanbaal Exp $
COPYRIGHT
Copyright (C) 2001, 2002 Stichting LogReport Foundation LogReport@LogReport.org
This program is part of Lire.
Lire is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free
Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program (see COPYING); if not, check with
http://www.gnu.org/copyleft/gpl.html.
Lire 2.1.1 2009-03-15 W3C_EXTENDED2DLF.IN(1)