concatenate log file lines up to timestamp

Tags
shell scripts

Login to Reply

 
Thread Tools Search this Thread
# 1  
Old 05-28-2009
concatenate log file lines up to timestamp

Hi,

Using sed awk or perl I am trying to do something similar to

https://www.unix.com/shell-programmin...nk-line-2.html

but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines start with the timestamp line and any lines that do no start with a timestamp are appended to the last line with a timestamp. Optionally I would like to do this up to the first semicolon.

A simplified input would be somthing like this

2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

and ideally the output would be

2009-05-27 02:37:27.283 The quick brown fox;
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

although this is also acceptable

2009-05-27 02:37:27.283 The quick brown fox; The quick brown fox
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

The log files can be up to 10MB in size and there can be a hundred lines or more between timestamps. The purpose of this is to format the file so that it can be loaded into a database.

Any suggestions/solutions would be greatly appreciated.

Thanks,
Alan
# 2  
Old 05-28-2009
something to start with - adjust the date pattern as needed.

nawk -f alan.awk myFile

alan.awk:
Code:
BEGIN {
   PATdate="^[12][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]"
}
$0 ~ PATdate {printf("%c%s%c", (p)?ORS:"",$0, (/;$/)?ORS:"") ;p=(/;$/)?0:1;next}
p && /;$/ { p=0; print}
p {printf(" %s", $0)}


Last edited by vgersh99; 05-28-2009 at 06:38 PM..
# 3  
Old 05-28-2009
if your system have Python,
Code:
#!/usr/bin/env python
fh=open("file")
s=""
f=0
for items in fh:
    items=items.strip()
    if f and  items.startswith("2009"):
        if ";" in s:
            ind=s.index(";")
            print s[:ind] #print from start till where ; is
        else:
            print s 
        s=""  
        f=0        
    if items.startswith("2009"): 
        f=1 #set flag        
        print items,
        continue
    if f and not items.startswith("2009"):
        # join up those lines that doesn't start with 2009
        s=s+items
fh.close() #close the file

output
Code:
# more file
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;
adsf
sldkfdf
2009-05-28 10:10:28.000 Mary test
tester fmsd
2009-05-28 10:10:28.000

# ./test.py
2009-05-27 02:37:27.283 The quick brown fox
2009-05-28 10:10:28.000 Mary had alittle lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill; adsfsldkfdf
2009-05-28 10:10:28.000 Mary test tester fmsd
2009-05-28 10:10:28.000

# 4  
Old 05-29-2009
sed:
Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:
Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?<=.)(?=[0-9]{4}-[0-9]{2}-[0-9]{2})/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

-----Post Update-----

sed:
Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:
Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?:(?<=.))(?:(?=[0-9]{4}-[0-9]{2}-[0-9]{2}))/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

# 5  
Old 06-29-2009
Thank you all very much. All your responses were excellent. It seems like the awk or python examples will work best for me.

Thanks,
-Alan
Login to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Similar Threads More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Grep lines between last hour timestamp and current timestamp nms Shell Programming and Scripting 1 09-15-2017 05:18 AM
Bash Script Closest Timestamp (LOG FILE) roberto999 Shell Programming and Scripting 3 09-13-2015 02:36 PM
To check timestamp in logfile and display lines upto 3 hours before current timestamp srkmish Shell Programming and Scripting 12 09-19-2014 02:18 PM
Concatenate three lines into one FredAtArrow UNIX for Dummies Questions & Answers 7 08-14-2014 05:03 PM
Check/Parse log file's lines using time difference/timestamp cele_82 Shell Programming and Scripting 1 10-01-2013 12:14 PM
Concatenate lines in file shell script systemoper UNIX for Advanced & Expert Users 3 01-14-2012 06:40 PM
Concatenate two lines in a fIle uday.sena.m Programming 3 09-22-2011 12:50 AM
Concatenate lines between lines starting with a specific pattern s052866 Shell Programming and Scripting 9 10-24-2010 01:09 PM
Delete log file entries based on the Date/Timestamp within log file vikram3.r Shell Programming and Scripting 3 08-05-2010 05:26 AM
Concatenate two lines in one Orbix Shell Programming and Scripting 5 04-29-2010 05:40 PM
How to search backwards in a log file by timestamp of entries? kevinmccallum Shell Programming and Scripting 2 09-27-2009 05:07 AM
copy lines from log files based on timestamp and sysdate percvs88 Shell Programming and Scripting 4 06-25-2009 10:35 AM
copy lines from log files based on timestamp and sysdate percvs88 Shell Programming and Scripting 1 06-25-2009 10:33 AM
Copy lines from a log file based on timestamp ranjiadmin UNIX for Advanced & Expert Users 1 05-22-2008 04:16 AM
Spooling a log file with timestamp ukadmin UNIX for Dummies Questions & Answers 2 03-23-2004 09:06 AM
All times are GMT -4. The time now is 01:12 AM.

Unix & Linux Forums Content Copyright 1993-2018. All Rights Reserved.
UNIX.COM Login
Username:
Password:  
Show Password





Not a Forum Member?
Forgot Password?