Shell Programming and Scripting

BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

concatenate log file lines up to timestamp

👤 Login to reply

    #1  
Old 05-28-2009
AlanC AlanC is offline
Registered User
 
concatenate log file lines up to timestamp

Hi,

Using sed awk or perl I am trying to do something similar to

https://www.unix.com/shell-programmin...nk-line-2.html

but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines start with the timestamp line and any lines that do no start with a timestamp are appended to the last line with a timestamp. Optionally I would like to do this up to the first semicolon.

A simplified input would be somthing like this

2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

and ideally the output would be

2009-05-27 02:37:27.283 The quick brown fox;
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

although this is also acceptable

2009-05-27 02:37:27.283 The quick brown fox; The quick brown fox
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

The log files can be up to 10MB in size and there can be a hundred lines or more between timestamps. The purpose of this is to format the file so that it can be loaded into a database.

Any suggestions/solutions would be greatly appreciated.

Thanks,
Alan
Sponsored Links
    #2  
Old 05-28-2009
vgersh99's Unix or Linux Image
vgersh99 vgersh99 is offline Forum Advisor  
Forum Advisor
 
something to start with - adjust the date pattern as needed.

nawk -f alan.awk myFile

alan.awk:
Code:
BEGIN {
   PATdate="^[12][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]"
}
$0 ~ PATdate {printf("%c%s%c", (p)?ORS:"",$0, (/;$/)?ORS:"") ;p=(/;$/)?0:1;next}
p && /;$/ { p=0; print}
p {printf(" %s", $0)}


Last edited by vgersh99; 05-28-2009 at 05:38 PM..
Sponsored Links
    #3  
Old 05-28-2009
ghostdog74 ghostdog74 is offline
Registered User
 
if your system have Python,
Code:
#!/usr/bin/env python
fh=open("file")
s=""
f=0
for items in fh:
    items=items.strip()
    if f and  items.startswith("2009"):
        if ";" in s:
            ind=s.index(";")
            print s[:ind] #print from start till where ; is
        else:
            print s 
        s=""  
        f=0        
    if items.startswith("2009"): 
        f=1 #set flag        
        print items,
        continue
    if f and not items.startswith("2009"):
        # join up those lines that doesn't start with 2009
        s=s+items
fh.close() #close the file

output
Code:
# more file
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;
adsf
sldkfdf
2009-05-28 10:10:28.000 Mary test
tester fmsd
2009-05-28 10:10:28.000

# ./test.py
2009-05-27 02:37:27.283 The quick brown fox
2009-05-28 10:10:28.000 Mary had alittle lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill; adsfsldkfdf
2009-05-28 10:10:28.000 Mary test tester fmsd
2009-05-28 10:10:28.000

    #4  
Old 05-29-2009
summer_cherry summer_cherry is offline Forum Advisor  
Registered User
 
sed:
Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:
Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?<=.)(?=[0-9]{4}-[0-9]{2}-[0-9]{2})/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

-----Post Update-----

sed:
Code:
sed -n '/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ {
1 {
	h
}
1 !{
	x
	s/\n/ /g
	p
	$ {
		x
		p
	}
	$ !{
	d
	}
}
}
/[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\}/ !{
 H
}' a.txt

perl:
Code:
undef $/;
my $str=<DATA>;
$str=~s/\n/ /g;
$str=~s/(?:(?<=.))(?:(?=[0-9]{4}-[0-9]{2}-[0-9]{2}))/\n/g;
print $str;
__DATA__
2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

Sponsored Links
    #5  
Old 06-29-2009
AlanC AlanC is offline
Registered User
 
Thank you all very much. All your responses were excellent. It seems like the awk or python examples will work best for me.

Thanks,
-Alan
Sponsored Links
👤 Login to reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Concatenate lines in a single paragraph cgkmal Shell Programming and Scripting 4 05-17-2009 01:02 AM
sed / awk to concatenate lines until blank line dunstonrocks Shell Programming and Scripting 11 03-30-2009 11:11 PM
How to concatenate consecutive lines shivi707 UNIX Desktop Questions & Answers 1 01-12-2009 08:08 AM
concatenate and display 2 lines as 1 with a condition for 2 line ? vithala Shell Programming and Scripting 7 07-11-2008 01:01 AM
Copy lines from a log file based on timestamp ranjiadmin UNIX for Advanced & Expert Users 1 05-22-2008 03:16 AM



All times are GMT -4. The time now is 06:53 AM.

Unix & Linux Forums Content Copyright©1993-2018. All Rights Reserved.
×
UNIX.COM Login
Username:
Password:  
Show Password





Not a Forum Member?
Forgot Password?