Logfile - extracting certain lines to concatenate into 1 line


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Logfile - extracting certain lines to concatenate into 1 line
# 1  
Old 03-17-2009
Logfile - extracting certain lines to concatenate into 1 line

I've got a log file from automatic diagnostic runs. The log file is appended to each time an automatic log is run.
I'd like to just pull certain lines from each run in the log file, and concatenate them into 1 comma delimited line (for export into excel or an html table).

Each diagnostic run is bracketed by a begin and end comment so I'm able to group it that way but I'm having trouble getting everything into 1 line.

Typical data:

Diags begin.
Tues March 17 18:07:34 EDT 2009

PASS: (123) Power Check
.
.
[more data]
Diags end.
Diags begin.
Tues March 17 19:09:22 EDT 2009

FAIL: (123) Power Check
.
description: Voltage clamp
.
.[more failing data details]
.
Diags end.


I want the timestamp, test result (for the 123 test)


I was able to use awk to get close but I can't quite get it.

awk '/Diags begin/{getline;print};{if ($2=="(123)") print $1,$2,$3,$4}'

This gives me:

Mon Mar 16 11:37:07 EDT 2009
PASS: (123) Power Check
Mon Mar 16 12:31:10 EDT 2009
PASS: (123) Power Check
Tue Mar 17 01:30:54 EDT 2009
**FAIL: (123) Power Check
Tue Mar 17 03:08:16 EDT 2009
PASS: (123) Power Check



What I'm trying to get is:
Mon Mar 16 11:37:07 EDT 2009,PASS: (123) Power Check
Mon Mar 16 12:31:10 EDT 2009,PASS: (123) Power Check
Tue Mar 17 01:30:54 EDT 2009,**FAIL: (123) Power Check
Tue Mar 17 03:08:16 EDT 2009,PASS: (123) Power Check


Also, I'm looking for a way to just pull the information for a certain time frame (for instance the current date - 7 days) but I'll worry about that later.. baby steps... Smilie

The system is Unix (HP-UX) so awk, perl, or sed are options.

Thanks for any help..

Paul
# 2  
Old 03-18-2009
#!/usr/bin/perl -w

use strict;

open (FH,'txt');
my $fb=0;
my @str ;
while (my $line = <FH>){
chomp ($line);
if ($line =~ m/Diags begin/){
$fb =1;
}
if ($fb && !($line =~ m/Diags begin/ || $line =~ m/Diags end/ )){
push @str , $line;
}
if ($line =~ m/Diags end/){
my $x = join " ", @str;
print "$x\n";
@str=();
$fb=0;
}
}

Try this

Cheers

Last edited by daptal; 03-18-2009 at 12:36 AM.. Reason: output
# 3  
Old 03-18-2009
Following on your awk script - use printf:

Code:
awk '/Diags begin/{getline;printf};{if ($2=="(123)") print ","$1,$2,$3,$4}'

# 4  
Old 03-30-2009
ok, thanks for the good ideas.. after some hacking and testing I finally got the output in a workable format using awk.

I've outputted the results to an ascii file in a comma delimited format..
my output file is called tstres.txt
and typical lines in the file look like this:

TSTR01 , Mar 29 21:29:17 EDT 2009 , PASS: , Power Check
TSTR01 , Mar 30 00:54:55 EDT 2009 , PASS: , Power Check
TSTR01 , Mar 30 08:31:31 EDT 2009 , **FAIL: , Power Check
TSTR02 , Mar 07 14:41:08 EST 2009 , PASS: , Power Check
TSTR02 , Mar 07 21:46:33 EST 2009 , PASS: , Power Check

What is the easiest way to take this data file and output it to an html table ?

My plan is to have the script run in a cron job and all I'll need to do is view the html page.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Search for word in huge logfile and need to continue to print few lines from that line til find date

Guys i need an idea for one logic..in shell scripting am struggling with a logic...So the thing is... i need to search for a word in a huge log file and i need to continue to print few more lines from that line and the consecutive line has to end when it finds the line with date..because i know... (1 Reply)
Discussion started by: Prathi
1 Replies

2. UNIX for Dummies Questions & Answers

Extracting the two lines where the first line is matched

Hi, If I have a file of something like @hg19_gold_AL122127.6-131160 GCTTCATCATGCATGGATAGGCTGGCGCCTTTCCTGAGGCCATATGCCGATGGATATG @hg19_gold_AL122127.6-131159 CTTTAATATTTCCGCCACCATCCTGAGTGAATCCCAGCAAGGACAGTCTTTGGGGATT @hg19_gold_AL122127.6-131158... (4 Replies)
Discussion started by: jyu429
4 Replies

3. Shell Programming and Scripting

Extracting lines after nth LINE from an output

Hi all, Here is my problem for which i am breaking my head for past three days.. I have parted command output as follows.. Model: ATA WDC WD5000AAKS-0 (scsi) Disk /dev/sdb: 500GB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type ... (3 Replies)
Discussion started by: selvarajvs
3 Replies

4. Shell Programming and Scripting

extracting data froma logfile

You are given a 1 year logfile with each line starting with a date in the form “YYYY-MM-DD”. How would you extract logs from the 4th day of each month and put them into a new file (3 Replies)
Discussion started by: DOkuwa
3 Replies

5. Shell Programming and Scripting

Extracting line matching a phrase and then the next lines after it

Hi all, I was wondering if someone could tell me a way to extract from a file lines where you search for a phrase and then also extract the next X lines after it (i.e. take a block of text from the file)? Example { id=123 time=10:00:00 date=12/12/09 { ........ ... (6 Replies)
Discussion started by: muay_tb
6 Replies

6. Shell Programming and Scripting

sed / awk to concatenate lines until blank line

Sample input (line feed indicated by ) --------------- The red fox jumped over the brown fence of the red hous He then went into the orchard --------------- Desired Output --------------- The red fox jumped over the brown fence of the red house He then went into the orchard (11 Replies)
Discussion started by: dunstonrocks
11 Replies

7. UNIX for Advanced & Expert Users

need to concatenate two lines if the line doesnt end with quotes

Hi I am getting a source file where the columns are seperated by comma and double Quotes Eg1 : "AAA","BBB","CCCC" in the same source file i am also getting few lines where my last columns double quotes are ending in its next line or the next next line Eg2: "AAA","BBB","CCC CC"... (9 Replies)
Discussion started by: laxmi131
9 Replies

8. Shell Programming and Scripting

concatenate and display 2 lines as 1 with a condition for 2 line ?

I have 2 pattern of lines (SQL query and Time taken)in a log i need to capture all SQL queries with time taken >20 sec and need to display as one line. 2 lines from log: 2007-10-23 11:39:17,061 DEBUG - SQL Query : SELECT A.GROUP_CD , C.FN_CD FROM UP_GROUP A , PRD_GROUP_TO_FN B , PRD_FN... (7 Replies)
Discussion started by: vithala
7 Replies

9. UNIX for Advanced & Expert Users

Need solution concatenate and display 2 lines as 1 with a condition for 2 line ?

I have 2 pattern of lines (SQL query and Time taken)in a log i need to capture all SQL queries with time taken >20 sec and need to display as one line. 2 lines from log: 2007-10-23 11:39:17,061 DEBUG - SQL Query : SELECT A.GROUP_CD , C.FN_CD FROM UP_GROUP A , PRD_GROUP_TO_FN B , PRD_FN... (1 Reply)
Discussion started by: vithala
1 Replies

10. Shell Programming and Scripting

Extracting Logfile Entries

Hello All, I am trying to write a KSH script that would extract lines from a logfile, within a certain timeframe, in reverse order. Ex. -Sample logfile- INFO 2006-07-12 00:23:00 - ModeDecision Time in WF:27 INFO 2006-07-12 00:24:00 - NRT_MQ_RES:108 INFO 2006-07-12 00:25:00 - getRecords()... (2 Replies)
Discussion started by: harpdl
2 Replies
Login or Register to Ask a Question