Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

concatenate log file lines up to timestamp

Shell Programming and Scripting


Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 05-28-2009   -   Original Discussion by AlanC
AlanC's Unix or Linux Image
AlanC AlanC is offline
Registered User
 
Join Date: May 2009
Last Activity: 29 June 2009, 11:39 AM EDT
Posts: 2
Thanks: 0
Thanked 0 Times in 0 Posts
concatenate log file lines up to timestamp

Hi,

Using sed awk or perl I am trying to do something similar to

https://www.unix.com/shell-programmin...nk-line-2.html

but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines start with the timestamp line and any lines that do no start with a timestamp are appended to the last line with a timestamp. Optionally I would like to do this up to the first semicolon.

A simplified input would be somthing like this

2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

and ideally the output would be

2009-05-27 02:37:27.283 The quick brown fox;
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

although this is also acceptable

2009-05-27 02:37:27.283 The quick brown fox; The quick brown fox
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

The log files can be up to 10MB in size and there can be a hundred lines or more between timestamps. The purpose of this is to format the file so that it can be loaded into a database.

Any suggestions/solutions would be greatly appreciated.

Thanks,
Alan
Sponsored Links
    #2  
Old Unix and Linux 05-28-2009   -   Original Discussion by AlanC
vgersh99's Unix or Linux Image
vgersh99 vgersh99 is online now Forum Advisor  
Forum Advisor
 
Join Date: Feb 2005
Last Activity: 20 April 2018, 12:20 PM EDT
Location: Foxborough, MA
Posts: 8,253
Thanks: 240
Thanked 811 Times in 757 Posts
something to start with - adjust the date pattern as needed.

nawk -f alan.awk myFile

alan.awk:


Code:
BEGIN {
   PATdate="^[12][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]"
}
$0 ~ PATdate {printf("%c%s%c", (p)?ORS:"",$0, (/;$/)?ORS:"") ;p=(/;$/)?0:1;next}
p && /;$/ { p=0; print}
p {printf(" %s", $0)}


Last edited by vgersh99; 05-28-2009 at 05:38 PM..
Sponsored Links
    #3  
Old Unix and Linux 05-28-2009   -   Original Discussion by AlanC
ghostdog74's Unix or Linux Image
ghostdog74 ghostdog74 is offline
Registered User
 
Join Date: Sep 2006
Last Activity: 28 January 2015, 8:30 AM EST
Posts: 2,669
Thanks: 0
Thanked 18 Times in 18 Posts
if your system have Python,


Code:
#!/usr/bin/env python
fh=open("file")
s=""
f=0
for items in fh:
    items=items.strip()
    if f and  items.startswith("2009"):
        if ";" in s:
            ind=s.index(";")
            print s[:ind] #print from start till where ; is
        else:
            print s 
        s=""  
        f=0        
    if items.startswith("2009"): 
        f=1 #set flag        
        print items,
        continue
    if f and not items.startswith("2009"):
        # join up those lines that doesn't start with 2009
        s=s+items
fh.close() #close the file

output


Code:
# more file
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;
adsf
sldkfdf
2009-05-28 10:10:28.000 Mary test
tester fmsd
2009-05-28 10:10:28.000

# ./test.py
2009-05-27 02:37:27.283 The quick brown fox
2009-05-28 10:10:28.000 Mary had alittle lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill; adsfsldkfdf
2009-05-28 10:10:28.000 Mary test tester fmsd
2009-05-28 10:10:28.000

    #4  
Old Unix and Linux 05-29-2009   -   Original Discussion by AlanC
summer_cherry's Unix or Linux Image
summer_cherry summer_cherry is offline Forum Advisor  
Registered User
 
Join Date: Jun 2007
Last Activity: 11 November 2016, 3:44 AM EST
Location: Beijing China
Posts: 1,305
Thanks: 0
Thanked 26 Times in 26 Posts
sed:


Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:


Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?<=.)(?=[0-9]{4}-[0-9]{2}-[0-9]{2})/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

-----Post Update-----

sed:


Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:


Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?:(?<=.))(?:(?=[0-9]{4}-[0-9]{2}-[0-9]{2}))/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

Sponsored Links
    #5  
Old Unix and Linux 06-29-2009   -   Original Discussion by AlanC
AlanC's Unix or Linux Image
AlanC AlanC is offline
Registered User
 
Join Date: May 2009
Last Activity: 29 June 2009, 11:39 AM EDT
Posts: 2
Thanks: 0
Thanked 0 Times in 0 Posts
Thank you all very much. All your responses were excellent. It seems like the awk or python examples will work best for me.

Thanks,
-Alan
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Concatenate lines in a single paragraph cgkmal Shell Programming and Scripting 4 05-17-2009 01:02 AM
sed / awk to concatenate lines until blank line dunstonrocks Shell Programming and Scripting 11 03-30-2009 11:11 PM
How to concatenate consecutive lines shivi707 UNIX Desktop Questions & Answers 1 01-12-2009 08:08 AM
concatenate and display 2 lines as 1 with a condition for 2 line ? vithala Shell Programming and Scripting 7 07-11-2008 01:01 AM
Copy lines from a log file based on timestamp ranjiadmin UNIX for Advanced & Expert Users 1 05-22-2008 03:16 AM



All times are GMT -4. The time now is 12:23 PM.