Visit Our UNIX and Linux User Community


Generating formatted reports from log files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Generating formatted reports from log files
# 1  
Old 07-27-2009
Generating formatted reports from log files

Given that I have a log file of the format:

DATE ID LOG_LEVEL | EVENT
2009-07-23T14:05:11Z T-4030097550 D | MessX
2009-07-23T14:10:44Z T-4030097550 D | MessY
2009-07-23T14:34:08Z T-7298651656 D | MessX
2009-07-23T14:41:00Z T-7298651656 D | MessY
2009-07-23T15:05:10Z T-4030097550 D | MessZ
etc

How do I produce a report that looks like

ID = T-4030097550
2009-07-23T14:05:11Z MessX
2009-07-23T14:10:44Z MessY
2009-07-23T15:05:10Z MessZ
ID = T-7298651656
2009-07-23T14:34:08Z MessX
2009-07-23T14:41:00Z MessY
etc

Other info:
The IDs are not pre-determined
The messages in the report are actually a subset of the log-entries and some messages may be missing for a given ID.
# 2  
Old 07-27-2009
Try this,give your file name as argument to script
Code:
#!/bin/sh
for id in `cut -d' ' -f2 $1 |sort -u`
do
      echo 'ID='$id
      grep $id $1 |awk '{print $1 " "  $5}'
done

# 3  
Old 07-27-2009
Almost there

That works well.

One thing I ommitted was that the message elements is actually a text string so each record is more of the format

2009-07-27T08:08:18Z T-6769544753 D | this is some log entry

In this case I wnat to pick up all fields from column 5

The other thing is if I only want to pick up log entries that contain strings of interest is there a simple of way of seeding this from a file with the entries.

My alternative was to run a log of grep statements for each string and then sort the resulting file by timestamp (column 1) before presenting it to the script you provided. - I suspect there is a better way.
# 4  
Old 07-27-2009
replace
Code:
 grep $id $1 |awk '{print $1 " "  $5}'

with
Code:
grep $id $1 |sed 's/ .*|//'

# 5  
Old 07-28-2009
perl:


Code:
while(<DATA>){
	chomp;
	next if $. == 1;
  my @tmp=split("[ |]",$_);
  push @{$hash{$tmp[1]}}, $tmp[0]." ".$tmp[5];
}
foreach my $key(keys %hash){
	print "ID = $key\n";
	print join "\n", @{$hash{$key}};
	print "\n";
}
__DATA__
DATE ID LOG_LEVEL | EVENT
2009-07-23T14:05:11Z T-4030097550 D | MessX
2009-07-23T14:10:44Z T-4030097550 D | MessY
2009-07-23T14:34:08Z T-7298651656 D | MessX
2009-07-23T14:41:00Z T-7298651656 D | MessY
2009-07-23T15:05:10Z T-4030097550 D | MessZ

# 6  
Old 07-28-2009
I think the code by JohnBach is good enough:
Code:
#!/bin/sh
lookfor='(MessX|MessY|MessZ|this|that|blabla)'
for id in `cut -d' ' -f2 $1 |sort -u` ; do
      echo 'ID='$id
      grep $id $1 | grep $lookfor |awk '{print $1 " "  $5}'
done


Previous Thread | Next Thread
Test Your Knowledge in Computers #193
Difficulty: Easy
Python was selected the Programming Language of the Year in 2007, 2010, and 2018.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Cron job scheduled is running once, but reports are generating twice

Team, Hope you all are doing fine I have one admin server which is being used dedicately to run cron jobs on hourly basis, fetching the details from Database which is in a different server.These cronjob are run on every hourly/5 minutes basis depending as per end user requirement.The script... (12 Replies)
Discussion started by: whizkidash
12 Replies

2. Shell Programming and Scripting

Split files with formatted numbers

How to split the file and have suffix with formatted numbers Tried the following code awk '{filename="split."int((NR-1)/2)".txt"; print >> filename}' split.txt Current Result Expected Result (21 Replies)
Discussion started by: bobbygsk
21 Replies

3. Shell Programming and Scripting

Generate a Execution time reports using log files.

Hi Experts, I am having a requirement, where i need to generate a report of the execution time of all the processes. All the processes generate the log files in a log directory and I can get the execution time from the log files. like below is the log file. /home/vikas/log >ls -l... (2 Replies)
Discussion started by: k_vikash
2 Replies

4. Linux

Generating apache log reports

Hello all, I'm trying to find some tool on generating reports based on apache access_log files (of Common format). I found some of them (awstats, lire/logreport, weblog expert, apache logs viewer, etc..) but they generate some global and general report about the log file. Also some perl... (0 Replies)
Discussion started by: enux
0 Replies

5. Shell Programming and Scripting

Extract information from Log file formatted

Good evening! Trying to make a shell script to parse log file and show only required information. log file has 44 fields and alot of lines, each columns separated by ":". log file is like: first_1:3:4:5:6:1:3:4:5:something:notinterested second_2:3:4:3:4:2 first_1:3:4:6:6:7:8 I am interested... (3 Replies)
Discussion started by: dummie55
3 Replies

6. Red Hat

sarg issue while generating reports for squid

I installed sarg from sarg rpm and i am facing issue while generating sarg reports and getting this time different error below sarg -l /var/log/squid/access.log SARG: Records in file: 242332, reading: 0.00% SARG: Records in file: 242332, reading: 2.06% SARG: Records in file: 242332, reading:... (0 Replies)
Discussion started by: mail4vij
0 Replies

7. Shell Programming and Scripting

how to search reports with specified keyword in log

Hi, I have a question with sed/awk. When I handle some log files I want to search all reports with specified keyword. For example, in the log below. abcd efg ===start abc e ===end xyz ===start af f ===end nf ga ===start ab ===end (4 Replies)
Discussion started by: danielnpu
4 Replies

8. Shell Programming and Scripting

Generating files.

I/P file name:- 20092008.txt Check number of entries in i/p file by following command ChkEnt -infl 20092008.txt -opfl 20092008_test.txt >count.txt Dear Friends, Please help me in automating following thing. If output generated (count.txt) is having value more than 1000 i.e.... (8 Replies)
Discussion started by: anushree.a
8 Replies

9. Shell Programming and Scripting

generating reports based on time field of network data

hi i have data extracted in the following format ranging around 300000 to 800000 records in a text file , the format is of network data . No. Time Source Destination Protocol 1 1998-06-05 17:20:23.569905 HP_61:aa:c9 HP_61:aa:c9 ... (1 Reply)
Discussion started by: renukaprasadb
1 Replies

10. Shell Programming and Scripting

Generating a report -Formatted printing -Urgent

Hi, My aim is to generate a report using shell script. There are various formats fields coloumns etc. I want to print in a single line (row) but in different coloumn as given below: field1 field2 field3 field4 ....... ....... ...... ....... The spacing... (1 Reply)
Discussion started by: jisha
1 Replies

Featured Tech Videos