Sponsored Content
Top Forums Shell Programming and Scripting concatenate log file lines up to timestamp Post 302320707 by AlanC on Thursday 28th of May 2009 04:30:10 PM
Old 05-28-2009
concatenate log file lines up to timestamp

Hi,

Using sed awk or perl I am trying to do something similar to

https://www.unix.com/shell-programmin...nk-line-2.html

but my requirement is slightly different. What I am trying to accomplish is to reformat a logfile such that all lines start with the timestamp line and any lines that do no start with a timestamp are appended to the last line with a timestamp. Optionally I would like to do this up to the first semicolon.

A simplified input would be somthing like this

2009-05-27 02:37:27.283 The quick
brown fox;
The quick
brown fox
2009-05-28 10:10:28.000 Mary
had a
little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

and ideally the output would be

2009-05-27 02:37:27.283 The quick brown fox;
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

although this is also acceptable

2009-05-27 02:37:27.283 The quick brown fox; The quick brown fox
2009-05-28 10:10:28.000 Mary had a little lamb.
2009-06-01 19:37:29.000 Jack and Jill ran up the hill;

The log files can be up to 10MB in size and there can be a hundred lines or more between timestamps. The purpose of this is to format the file so that it can be loaded into a database.

Any suggestions/solutions would be greatly appreciated.

Thanks,
Alan
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Copy lines from a log file based on timestamp

how to copy lines from a log file based on timestamp. INFO (RbrProcessFlifoEventSessionEJB.java:processFlight:274) - E_20080521_110754_967: rbrAciInfoObjects listing complete! INFO (RbrPnrProcessEventSessionEJB.java:processFlight:197) - Event Seq: 1647575217; Carrier: UA; Flt#: 0106; Origin:... (1 Reply)
Discussion started by: ranjiadmin
1 Replies

2. Shell Programming and Scripting

copy lines from log files based on timestamp and sysdate

I am sorry to repost this question. it was not clear, and I had the meeting and didn't response the question on time. I do really need help and appreciate your help very much. I'm looking for a simple shell script that will read lots of audit log file (*.aud) in a log fold every 10 minutes,... (1 Reply)
Discussion started by: percvs88
1 Replies

3. Shell Programming and Scripting

copy lines from log files based on timestamp and sysdate

I'm looking for a command or simple script that will read lots of audit log file (*.aud) in log fold every 10 minutes, and will output to one file based on sysdate - 10 minutes. assume the script is run at 11:12:20, and it should grep the line from Wed Jun 17 11:02:43 2009 to end of file. after... (4 Replies)
Discussion started by: percvs88
4 Replies

4. Shell Programming and Scripting

Delete log file entries based on the Date/Timestamp within log file

If a log file is in the following format 28-Jul-10 ::: Log message 28-Jul-10 ::: Log message 29-Jul-10 ::: Log message 30-Jul-10 ::: Log message 31-Jul-10 ::: Log message 31-Jul-10 ::: Log message 1-Aug-10 ::: Log message 1-Aug-10 ::: Log message 2-Aug-10 ::: Log message 2-Aug-10 :::... (3 Replies)
Discussion started by: vikram3.r
3 Replies

5. Shell Programming and Scripting

Concatenate lines between lines starting with a specific pattern

Hi, I have a file such as: --- >contig00001 length=35524 numreads=2944 gACGCCGCGCGCCGCGGCCAGGGCTGGCCCA CAGGCCGCGCGGCGTCGGCTGGCTGAG >contig00002 length=4242 numreads=43423 ATGCCGAAGGTCCGCCTGGGGCTGG CGCCGGGAGCATGTAGCG --- I would like to concatenate the lines not starting with ">"... (9 Replies)
Discussion started by: s052866
9 Replies

6. Programming

Concatenate two lines in a fIle

Hi All, Can any one help me in finding the solution for concatenating two or more lines in a file and writing them to a temporary file. for Example: He is a wise student. So he got first rank. This is in a file i want the output as He is a wise student so he got first rank. into a file... (3 Replies)
Discussion started by: uday.sena.m
3 Replies

7. UNIX for Advanced & Expert Users

Concatenate lines in file shell script

Hi colleagues, I have a file in this format. "/cccc/pppp/dddd/ggg/prueba.txt". ERROR" THE error bbbbbbbbbb finish rows. "/kkkk/mmmm/hhhh/jjj/ejemplo.txt". ERROR This is other error rows.I need my file in this format. "/cccc/pppp/dddd/ggg/prueba.txt". ERROR" THE error bbbbbbbbbb finish rows.... (3 Replies)
Discussion started by: systemoper
3 Replies

8. Shell Programming and Scripting

Check/Parse log file's lines using time difference/timestamp

I was looking at this script which outputs the two lines which differs less than one sec. #!/usr/bin/perl -w use strict; use warnings; use Time::Local; use constant SEC_MILIC => 1000; my $file='infile'; ## Open for reading argument file. open my $fh, "<", $file or die "Cannot... (1 Reply)
Discussion started by: cele_82
1 Replies

9. Shell Programming and Scripting

To check timestamp in logfile and display lines upto 3 hours before current timestamp

Hi Friends, I have the following logfile. Currently time in india is 07/31/2014 12:33:34 and i have the following content in logfile. I want to display only those entries which contain string 'Exception' within last 3 hours. In this case, it would be the last line only I can get the... (12 Replies)
Discussion started by: srkmish
12 Replies

10. Shell Programming and Scripting

Grep lines between last hour timestamp and current timestamp

So basically I have a log file and each line in this log file starts with a timestamp: MON DD HH:MM:SS SEP 15 07:30:01 I need to grep all the lines between last hour timestamp and current timestamp. Then these lines will be moved to a tmp file from which I will grep for particular strings. ... (1 Reply)
Discussion started by: nms
1 Replies
IO::Capture::Overview(3pm)				User Contributed Perl Documentation				IO::Capture::Overview(3pm)

NAME
IO::Capture -- Overview of "IO::Capture" Module, and classes derived from it. DESCRIPTION
The modules in this distribution are designed to allow you to capture and process output sent to STDOUT and/or STDERR. I initial created the modules to use in building module tests. I wanted to be able to intentionally cause errors, and insure the module responded correctly. E.g., Call a class method without a required argument. Using IO::Capture keeps the user from seeing these inten- tional errors when running 'make test'. I have also found this useful on occasion in Perl Tk apps, where I wanted to capture output from a Perl module I was using. I could then capture, then put the text into a log or message window. Note: None of the modules currently distributed will capture from the 'system' Perl function, or the like. It could be done, but gener- ally, if you would like to capture from a system command, you don't need this module, just use the backticks operators. my $output = '/usr/bin/ls'; They are small, lightweight modules. Instead of designing in a lot of features, we designed it to be easily reusable and adaptable. A module can be quickly built, that incorporates custom methods, but reuses all existing features of one of the derived classes. See the sec- tion on "ADDING FEATURES" Or, if you need to change the actual capture mechanism, "WRITING YOUR OWN DERIVED CLASS". (Don't worry, it's a piece of cake) DERIVED CLASSES
There are several classes derived from "IO::Capture". IO::Capture::Stdout Module to capture "STDOUT" from program. See IO::Capture::Stdout. IO::Capture::Stderr Module to capture "STDERR" from program. See IO::Capture::Stderr. IO::Capture::ErrorMessages This method has been depreciated. The only difference between this one and Stderr.pm was the trap for WARN. I found it was fixed in 5.8 so just check in Stderr now. I.e., Just use Stderr now. It (Stderr) will detect what version of perl you are using, and act accordingly. The two ("IO::Capture::ErrorMessages" and "IO::Capture::Stderr") are currently identical, and "IO::Capture::ErrorMessages" will be removed in a future release. If you would like to add features to any of these, or build your own module using "IO::Capture" as a base, read on. ADDING FEATURES
If one of these modules takes care of your problem, install it and have fun! But let's say you would like to add a feature to one of the derived classes, say IO::Capture::Stdout. No need to re-write the whole mod- ule, just use it as the base, and write your one feature. Here is a somewhat simplified example. # # Example module to add a grep_it method # # Give your package a name package MyPackage; #use IO:Capture:Stdout as the base use base 'IO::Capture::Stdout'; #define your method sub grep_it { my $self = shift; my $string = shift; my @found_lines; # Making a ref to the array makes it easier to read :-) my $arrayref = @{$self->{'IO::Capture::messages'}}; for my $line (@$arrayref) { push @found_lines, $line if $line =~ /$string/; } return wantarray ? @found_lines : scalar(@found_lines); } 1; Using it in this script #!/usr/sbin/perl use strict; use warnings; use MyPackage; my $capture = MyPackage->new(); $capture->start; print "The quick brown fox jumped over ..."; print "garden wall"; print "The quick red fox jumped over ..."; print "garden wall"; $capture->stop; for my $line ($capture->grep_it("fox")) { print "$line "; } Results in $ grep_it The quick brown fox jumped over ... The quick red fox jumped over ... WRITING YOUR OWN DERIVED CLASS
Before starting your own sub-class, be sure to read through IO::Capture. Pay special attention to the internal methods that are only defined as abstract methods in "IO::Capture". For examples, look at the sub-classes included with this distribution. ("IO::Capture::Std- out", "IO:Capture::Stderr". You can start by copying one of these and using it as a template. They have the required private methods defined already, and you may very well be able to use them as is. Change any methods, and add any new ones, as needed. For example, here is a commented copy of "IO::Capture::Stderr". # # Example module using abstract class IO::Capture # # Change this to give your class it's own name package IO::Capture::Stderr; # Make IO::Capture the base class use base qw/IO::Capture/; # If using included utility module in '_start()' use IO::Capture::Tie_STDx; # Override the three abstract methods needed to make a valid # module. See IO::Capture manpage # 1) _start - Starts the data capture. Is run from public method # start(); # # 2) _retrieve_captured_text() - Move the captured text into the # object hash key, "IO::Capture::messages". Called by public method # # 3) _stop - Stop the data capture. Called by public method 'stop()' # after private method '_retrieve_captured_text()' returns. # sub _start { tie *STDERR, "IO::Capture::Tie_STDx"; } sub _retrieve_captured_text { my $self = shift; # making a reference to it makes it more readable ;-) my $messages = @{$self->{'IO::Capture::messages'}}; @$messages = <STDERR>; } sub _stop { untie *STDERR; return 1; } 1; Lets say you don't want to capture all the text. You just want to grab the lines that have the word "Error" in them. The only thing you need to change is _retrieve_captured_text. (Besides the package name) Something like: sub _retrieve_captured_text { my $self = shift; # making a reference to it makes it more readable ;-) my $messages = @{$self->{'IO::Capture::messages'}}; while (<STDERR>) { push @$messages, $_ if /error/i; } } Yes. You could do this easier by just using "IO::Capture::Stderr" as the base and overriding "_retrieve_captured_text" like in "ADDING FEA- TURES", but hey, we needed an easy example. :-) If you want your class to have arguments that users can pass in, just use the default "new()" method and have the arguments passed in as an anonymous array. See the "IO::Capture::Stderr" module for an example. BUGS
Please report bugs on http://rt.cpan.org/ CREDITS
Special thanks to James E Keenan for many bug fixes and tests he provided. AUTHOR
Mark Reynolds reynolds<at>sgi.com Note: "Change <at" to 'at' sign.> COPYRIGHT
Copyright (c) 2003-2005, Mark Reynolds. All Rights Reserved. This module is free software. It may be used, redistributed and/or modified under the same terms as Perl itself. perl v5.8.8 2007-07-30 IO::Capture::Overview(3pm)
All times are GMT -4. The time now is 08:31 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy