concatenate log file lines up to timestamp


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
concatenate log file lines up to timestamp

Hi,

Using sed awk or perl I am trying to do something similar to

https://www.unix.com/shell-programmin...nk-line-2.html

but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines start with the timestamp line and any lines that do no start with a timestamp are appended to the last line with a timestamp. Optionally I would like to do this up to the first semicolon.

A simplified input would be somthing like this

2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

and ideally the output would be

2009-05-27 02:37:27.283 The quick brown fox;
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

although this is also acceptable

2009-05-27 02:37:27.283 The quick brown fox; The quick brown fox
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

The log files can be up to 10MB in size and there can be a hundred lines or more between timestamps. The purpose of this is to format the file so that it can be loaded into a database.

Any suggestions/solutions would be greatly appreciated.

Thanks,
Alan
# 2  
something to start with - adjust the date pattern as needed.

nawk -f alan.awk myFile

alan.awk:
Code:
BEGIN {
   PATdate="^[12][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]"
}
$0 ~ PATdate {printf("%c%s%c", (p)?ORS:"",$0, (/;$/)?ORS:"") ;p=(/;$/)?0:1;next}
p && /;$/ { p=0; print}
p {printf(" %s", $0)}


Last edited by vgersh99; 05-28-2009 at 05:38 PM..
# 3  
if your system have Python,
Code:
#!/usr/bin/env python
fh=open("file")
s=""
f=0
for items in fh:
    items=items.strip()
    if f and  items.startswith("2009"):
        if ";" in s:
            ind=s.index(";")
            print s[:ind] #print from start till where ; is
        else:
            print s 
        s=""  
        f=0        
    if items.startswith("2009"): 
        f=1 #set flag        
        print items,
        continue
    if f and not items.startswith("2009"):
        # join up those lines that doesn't start with 2009
        s=s+items
fh.close() #close the file

output
Code:
# more file
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;
adsf
sldkfdf
2009-05-28 10:10:28.000 Mary test
tester fmsd
2009-05-28 10:10:28.000

# ./test.py
2009-05-27 02:37:27.283 The quick brown fox
2009-05-28 10:10:28.000 Mary had alittle lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill; adsfsldkfdf
2009-05-28 10:10:28.000 Mary test tester fmsd
2009-05-28 10:10:28.000

# 4  
sed:
Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:
Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?<=.)(?=[0-9]{4}-[0-9]{2}-[0-9]{2})/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

-----Post Update-----

sed:
Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:
Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?:(?<=.))(?:(?=[0-9]{4}-[0-9]{2}-[0-9]{2}))/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

# 5  
Thank you all very much. All your responses were excellent. It seems like the awk or python examples will work best for me.

Thanks,
-Alan
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Check/Parse log file's lines using time difference/timestamp
cele_82
I was looking at this script which outputs the two lines which differs less than one sec. #!/usr/bin/perl -w use strict; use warnings; use Time::Local; use constant SEC_MILIC => 1000; my $file='infile'; ## Open for reading argument file. open my $fh, "<", $file or die "Cannot...... Shell Programming and Scripting
1
Shell Programming and Scripting
Concatenate lines in file shell script
systemoper
Hi colleagues, I have a file in this format. "/cccc/pppp/dddd/ggg/prueba.txt". ERROR" THE error bbbbbbbbbb finish rows. "/kkkk/mmmm/hhhh/jjj/ejemplo.txt". ERROR This is other error rows.I need my file in this format. "/cccc/pppp/dddd/ggg/prueba.txt". ERROR" THE error bbbbbbbbbb finish rows....... UNIX for Advanced & Expert Users
3
UNIX for Advanced & Expert Users
Concatenate two lines in a fIle
uday.sena.m
Hi All, Can any one help me in finding the solution for concatenating two or more lines in a file and writing them to a temporary file. for Example: He is a wise student. So he got first rank. This is in a file i want the output as He is a wise student so he got first rank. into a file...... Programming
3
Programming
copy lines from log files based on timestamp and sysdate
percvs88
I am sorry to repost this question. it was not clear, and I had the meeting and didn't response the question on time. I do really need help and appreciate your help very much. I'm looking for a simple shell script that will read lots of audit log file (*.aud) in a log fold every 10 minutes,...... Shell Programming and Scripting
1
Shell Programming and Scripting
Copy lines from a log file based on timestamp
ranjiadmin
how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:...... UNIX for Advanced & Expert Users
1
UNIX for Advanced & Expert Users