concatenate log file lines up to timestamp


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting concatenate log file lines up to timestamp
# 1  
Old 05-28-2009
concatenate log file lines up to timestamp

Hi,

Using sed awk or perl I am trying to do something similar to

https://www.unix.com/shell-programmin...nk-line-2.html

but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines start with the timestamp line and any lines that do no start with a timestamp are appended to the last line with a timestamp. Optionally I would like to do this up to the first semicolon.

A simplified input would be somthing like this

2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

and ideally the output would be

2009-05-27 02:37:27.283 The quick brown fox;
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

although this is also acceptable

2009-05-27 02:37:27.283 The quick brown fox; The quick brown fox
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

The log files can be up to 10MB in size and there can be a hundred lines or more between timestamps. The purpose of this is to format the file so that it can be loaded into a database.

Any suggestions/solutions would be greatly appreciated.

Thanks,
Alan
# 2  
Old 05-28-2009
something to start with - adjust the date pattern as needed.

nawk -f alan.awk myFile

alan.awk:
Code:
BEGIN {
   PATdate="^[12][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]"
}
$0 ~ PATdate {printf("%c%s%c", (p)?ORS:"",$0, (/;$/)?ORS:"") ;p=(/;$/)?0:1;next}
p && /;$/ { p=0; print}
p {printf(" %s", $0)}


Last edited by vgersh99; 05-28-2009 at 06:38 PM..
# 3  
Old 05-28-2009
if your system have Python,
Code:
#!/usr/bin/env python
fh=open("file")
s=""
f=0
for items in fh:
    items=items.strip()
    if f and  items.startswith("2009"):
        if ";" in s:
            ind=s.index(";")
            print s[:ind] #print from start till where ; is
        else:
            print s 
        s=""  
        f=0        
    if items.startswith("2009"): 
        f=1 #set flag        
        print items,
        continue
    if f and not items.startswith("2009"):
        # join up those lines that doesn't start with 2009
        s=s+items
fh.close() #close the file

output
Code:
# more file
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;
adsf
sldkfdf
2009-05-28 10:10:28.000 Mary test
tester fmsd
2009-05-28 10:10:28.000

# ./test.py
2009-05-27 02:37:27.283 The quick brown fox
2009-05-28 10:10:28.000 Mary had alittle lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill; adsfsldkfdf
2009-05-28 10:10:28.000 Mary test tester fmsd
2009-05-28 10:10:28.000

# 4  
Old 05-29-2009
sed:
Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:
Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?<=.)(?=[0-9]{4}-[0-9]{2}-[0-9]{2})/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

-----Post Update-----

sed:
Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:
Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?:(?<=.))(?:(?=[0-9]{4}-[0-9]{2}-[0-9]{2}))/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

# 5  
Old 06-29-2009
Thank you all very much. All your responses were excellent. It seems like the awk or python examples will work best for me.

Thanks,
-Alan
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep lines between last hour timestamp and current timestamp

So basically I have a log file and each line in this log file starts with a timestamp: MON DD HH:MM:SS SEP 15 07:30:01 I need to grep all the lines between last hour timestamp and current timestamp. Then these lines will be moved to a tmp file from which I will grep for particular strings. ... (1 Reply)
Discussion started by: nms
1 Replies

2. Shell Programming and Scripting

To check timestamp in logfile and display lines upto 3 hours before current timestamp

Hi Friends, I have the following logfile. Currently time in india is 07/31/2014 12:33:34 and i have the following content in logfile. I want to display only those entries which contain string 'Exception' within last 3 hours. In this case, it would be the last line only I can get the... (12 Replies)
Discussion started by: srkmish
12 Replies

3. Shell Programming and Scripting

Check/Parse log file's lines using time difference/timestamp

I was looking at this script which outputs the two lines which differs less than one sec. #!/usr/bin/perl -w use strict; use warnings; use Time::Local; use constant SEC_MILIC => 1000; my $file='infile'; ## Open for reading argument file. open my $fh, "<", $file or die "Cannot... (1 Reply)
Discussion started by: cele_82
1 Replies

4. UNIX for Advanced & Expert Users

Concatenate lines in file shell script

Hi colleagues, I have a file in this format. "/cccc/pppp/dddd/ggg/prueba.txt". ERROR" THE error bbbbbbbbbb finish rows. "/kkkk/mmmm/hhhh/jjj/ejemplo.txt". ERROR This is other error rows.I need my file in this format. "/cccc/pppp/dddd/ggg/prueba.txt". ERROR" THE error bbbbbbbbbb finish rows.... (3 Replies)
Discussion started by: systemoper
3 Replies

5. Programming

Concatenate two lines in a fIle

Hi All, Can any one help me in finding the solution for concatenating two or more lines in a file and writing them to a temporary file. for Example: He is a wise student. So he got first rank. This is in a file i want the output as He is a wise student so he got first rank. into a file... (3 Replies)
Discussion started by: uday.sena.m
3 Replies

6. Shell Programming and Scripting

Concatenate lines between lines starting with a specific pattern

Hi, I have a file such as: --- >contig00001 length=35524 numreads=2944 gACGCCGCGCGCCGCGGCCAGGGCTGGCCCA CAGGCCGCGCGGCGTCGGCTGGCTGAG >contig00002 length=4242 numreads=43423 ATGCCGAAGGTCCGCCTGGGGCTGG CGCCGGGAGCATGTAGCG --- I would like to concatenate the lines not starting with ">"... (9 Replies)
Discussion started by: s052866
9 Replies

7. Shell Programming and Scripting

Delete log file entries based on the Date/Timestamp within log file

If a log file is in the following format 28-Jul-10 ::: Log message 28-Jul-10 ::: Log message 29-Jul-10 ::: Log message 30-Jul-10 ::: Log message 31-Jul-10 ::: Log message 31-Jul-10 ::: Log message 1-Aug-10 ::: Log message 1-Aug-10 ::: Log message 2-Aug-10 ::: Log message 2-Aug-10 :::... (3 Replies)
Discussion started by: vikram3.r
3 Replies

8. Shell Programming and Scripting

copy lines from log files based on timestamp and sysdate

I'm looking for a command or simple script that will read lots of audit log file (*.aud) in log fold every 10 minutes, and will output to one file based on sysdate - 10 minutes. assume the script is run at 11:12:20, and it should grep the line from Wed Jun 17 11:02:43 2009 to end of file. after... (4 Replies)
Discussion started by: percvs88
4 Replies

9. Shell Programming and Scripting

copy lines from log files based on timestamp and sysdate

I am sorry to repost this question. it was not clear, and I had the meeting and didn't response the question on time. I do really need help and appreciate your help very much. I'm looking for a simple shell script that will read lots of audit log file (*.aud) in a log fold every 10 minutes,... (1 Reply)
Discussion started by: percvs88
1 Replies

10. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:... (1 Reply)
Discussion started by: ranjiadmin
1 Replies
Login or Register to Ask a Question