concatenate log file lines up to timestamp

Tags
shell scripts

 
Thread Tools Search this Thread
# 1  
Old 05-28-2009
concatenate log file lines up to timestamp

Hi,

Using sed awk or perl I am trying to do something similar to

https://www.unix.com/shell-programmin...nk-line-2.html

but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines start with the timestamp line and any lines that do no start with a timestamp are appended to the last line with a timestamp. Optionally I would like to do this up to the first semicolon.

A simplified input would be somthing like this

2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

and ideally the output would be

2009-05-27 02:37:27.283 The quick brown fox;
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

although this is also acceptable

2009-05-27 02:37:27.283 The quick brown fox; The quick brown fox
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

The log files can be up to 10MB in size and there can be a hundred lines or more between timestamps. The purpose of this is to format the file so that it can be loaded into a database.

Any suggestions/solutions would be greatly appreciated.

Thanks,
Alan
# 2  
Old 05-28-2009
something to start with - adjust the date pattern as needed.

nawk -f alan.awk myFile

alan.awk:
Code:
BEGIN {
   PATdate="^[12][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]"
}
$0 ~ PATdate {printf("%c%s%c", (p)?ORS:"",$0, (/;$/)?ORS:"") ;p=(/;$/)?0:1;next}
p && /;$/ { p=0; print}
p {printf(" %s", $0)}


Last edited by vgersh99; 05-28-2009 at 06:38 PM..
# 3  
Old 05-28-2009
if your system have Python,
Code:
#!/usr/bin/env python
fh=open("file")
s=""
f=0
for items in fh:
    items=items.strip()
    if f and  items.startswith("2009"):
        if ";" in s:
            ind=s.index(";")
            print s[:ind] #print from start till where ; is
        else:
            print s 
        s=""  
        f=0        
    if items.startswith("2009"): 
        f=1 #set flag        
        print items,
        continue
    if f and not items.startswith("2009"):
        # join up those lines that doesn't start with 2009
        s=s+items
fh.close() #close the file

output
Code:
# more file
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;
adsf
sldkfdf
2009-05-28 10:10:28.000 Mary test
tester fmsd
2009-05-28 10:10:28.000

# ./test.py
2009-05-27 02:37:27.283 The quick brown fox
2009-05-28 10:10:28.000 Mary had alittle lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill; adsfsldkfdf
2009-05-28 10:10:28.000 Mary test tester fmsd
2009-05-28 10:10:28.000

# 4  
Old 05-29-2009
sed:
Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:
Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?<=.)(?=[0-9]{4}-[0-9]{2}-[0-9]{2})/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

-----Post Update-----

sed:
Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:
Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?:(?<=.))(?:(?=[0-9]{4}-[0-9]{2}-[0-9]{2}))/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

# 5  
Old 06-29-2009
Thank you all very much. All your responses were excellent. It seems like the awk or python examples will work best for me.

Thanks,
-Alan

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Grep lines between last hour timestamp and current timestamp nms Shell Programming and Scripting 1 09-15-2017 05:18 AM
Bash Script Closest Timestamp (LOG FILE) roberto999 Shell Programming and Scripting 3 09-13-2015 02:36 PM
To check timestamp in logfile and display lines upto 3 hours before current timestamp srkmish Shell Programming and Scripting 12 09-19-2014 02:18 PM
Concatenate three lines into one FredAtArrow UNIX for Dummies Questions & Answers 7 08-14-2014 05:03 PM
Check/Parse log file's lines using time difference/timestamp cele_82 Shell Programming and Scripting 1 10-01-2013 12:14 PM
Help using telnet to capture log file with autogenerate timestamp eddlinux Shell Programming and Scripting 4 08-20-2013 07:02 AM
Concatenate lines in file shell script systemoper UNIX for Advanced & Expert Users 3 01-14-2012 06:40 PM
Concatenate two lines in a fIle uday.sena.m Programming 3 09-22-2011 12:50 AM
Concatenate lines between lines starting with a specific pattern s052866 Shell Programming and Scripting 9 10-24-2010 01:09 PM
Delete log file entries based on the Date/Timestamp within log file vikram3.r Shell Programming and Scripting 3 08-05-2010 05:26 AM
Concatenate two lines in one Orbix Shell Programming and Scripting 5 04-29-2010 05:40 PM
copy lines from log files based on timestamp and sysdate percvs88 Shell Programming and Scripting 4 06-25-2009 10:35 AM
copy lines from log files based on timestamp and sysdate percvs88 Shell Programming and Scripting 1 06-25-2009 10:33 AM
Copy lines from a log file based on timestamp ranjiadmin UNIX for Advanced & Expert Users 1 05-22-2008 04:16 AM
Spooling a log file with timestamp ukadmin UNIX for Dummies Questions & Answers 2 03-23-2004 09:06 AM