Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Combine logs span across multiple lines

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Combine logs span across multiple lines
# 1  
Old 12-20-2016
Combine logs span across multiple lines
Hi All,
I am having a log file

Code:
ERROR 2016-12-08 10:22:23.542 some data
**** some data******
**** some data******
**** some data******
DEBUG 2016-12-08 10:23:23.542 some data
**** some data******
**** some data******
**** some data******

when i grep the log file with
Code:
ERROR

am getting only the first line, but actual log line with respect to that timestamp spans across multiple lines.
Would like to know how we can combine the log lines in one line to get the entire line as a part of grep comand
# 2  
Old 12-20-2016
IF you have "markers" and you know them all - DEBUG ERROR (you fill in the rest)

Code:
awk '  BEGIN {ok=0}
         { 
            if( index("|DEBUG|ERROR|", $1) ) 
            {
                 ok=( $1 == "ERROR")
            }
            if(ok) {print $0}
       } '  logfile

Note: this could be transformed into a one-liner awk statement. But since you seem not to know awk, I feel this is more helpful for understanding.


Change "|DEBUG|ERROR|" to include all of the "marker" words, the words you use to delimit your search.
This User Gave Thanks to jim mcnamara For This Post:
rbatte1
# 3  
Old 12-20-2016
Hi.

An alternate solution using a (non-standard) member of the grep family:
Code:
cgrep +w <delimiter-string-list> <search-string> <filename>

will search <filename> for <search-string>. When found, it will also copy lines up to a line containing a match for <delimiter-string-list>.

Here are examples using an extended dataset:
Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate extraction of blocks of delimited text, cgrep.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C cgrep

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

markers='ERROR|DEBUG|CLARK|ZEPHYR'
search=CLARK
pl " Results, looking for $search:"
cgrep -D -E +I2 +w "$markers" $search $FILE

search=ERROR
pl " Results, looking for $search:"
cgrep -D -E +I2 +w "$markers" $search $FILE

search=DEBUG
pl " Results, looking for $search:"
cgrep -D -E +I2 +w "$markers" $search $FILE

exit 0

producing:
Code:
$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.6 (jessie) 
bash GNU bash 4.3.30
cgrep ATT cgrep 8.15

-----
 Input data file data1:
ERROR 2016-12-08 10:22:23.542 some data
**** some data****** 1
**** some data****** 2
**** some data****** 3
DEBUG 2016-12-08 10:23:23.542 some data
**** some data****** 4
**** some data****** 5
**** some data****** 6
CLARK 2016-11-08 10:23:23.542 some data
**** some data****** 7
**** some data****** 8
**** some data****** 9
**** some data****** 9a
ZEPHYR 2015-11-08 10:23:23.542 some data
**** some data****** 10
**** some data****** 11
**** some data****** 12
DEBUG 2014-12-08 10:23:23.542 some data
**** some data****** 13
**** some data****** 14
**** some data****** 15

-----
 Results, looking for CLARK:
CLARK 2016-11-08 10:23:23.542 some data
**** some data****** 7
**** some data****** 8
**** some data****** 9
**** some data****** 9a

-----
 Results, looking for ERROR:
ERROR 2016-12-08 10:22:23.542 some data
**** some data****** 1
**** some data****** 2
**** some data****** 3

-----
 Results, looking for DEBUG:
DEBUG 2016-12-08 10:23:23.542 some data
**** some data****** 4
**** some data****** 5
**** some data****** 6
DEBUG 2014-12-08 10:23:23.542 some data
**** some data****** 13
**** some data****** 14
**** some data****** 15

The cgrep code needs to be acquired, compiled, and made available. I have done this several times over the years. An additional advantage is that cgrep is very fast.

Here are some details:
Code:
cgrep   shows context of matching patterns found in files (man)
Path    : ~/executable/cgrep
Version : 8.15
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYSV ...)
Home    : http://sourceforge.net/projects/cgrep/

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
rbatte1
# 4  
Old 12-20-2016
Code:
awk '/^[[:upper:]]{2}/{f=0} /^ERROR/{f=1}f' file

Last edited by Scrutinizer; 12-20-2016 at 03:51 PM..
This User Gave Thanks to Scrutinizer For This Post:
rbatte1
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

awk to combine lines if fields match in lines

In the awk below, what I am attempting to do is check each line in the tab-delimeted input, which has ~20 lines in it, for a keyword SVTYPE=Fusion. If the keyword is found I am splitting $3 using the . (dot) and reading the portion before and after the dot in an array a. If it does have that... (12 Replies)
Discussion started by: cmccabe
12 Replies

3. Shell Programming and Scripting

Awk: Combine multiple lines based on number of fields

If a file has following kind of data, comma delimited 1,2,3,4 1 1 1,2,3,4 1,2 2 2,3,4 My required output must have only 4 columns with comma delimited 1,2,3,4 111,2,3,4 1,222,3,4 I have tried many awk command using ORS="" but couldnt progress (10 Replies)
Discussion started by: mdkm
10 Replies

4. Shell Programming and Scripting

Combine multiple lines into single line

Hi All , I have a file with below data # User@Host: xyz @ # Query_time: t1 Lock_time: t2 Rows_sent: n1 Rows_examined: n2 SET timestamp=1396852200; select count(1) from table; # Time: 140406 23:30:01 # User@Host: abc @ # Query_time: t1 Lock_time: t2 Rows_sent: n1 Rows_examined:... (6 Replies)
Discussion started by: rakesh_411
6 Replies

5. Shell Programming and Scripting

Shell Script for viewing multiple logs from multiple server

I am new to Shell scripting and below is my requirement. I need to search some specific word e.g. "exception" or "transaction" from log file. We have multiple env e.g. Level1 , Level2 etc and each env have Multiple boxes e.g. For Level 1 env we have "test11.test.com" , "test12.test.com". Each... (1 Reply)
Discussion started by: peeyush
1 Replies

6. Shell Programming and Scripting

Combine multiple unique lines from event log text file into one line, use PERL or AWK?

I can't decide if I should use AWK or PERL after pouring over these forums for hours today I decided I'd post something and see if I couldn't get some advice. I've got a text file full of hundreds of events in this format: Record Number : 1 Records in Seq : ... (3 Replies)
Discussion started by: Mayday22
3 Replies

7. Shell Programming and Scripting

Combine multiple lines in file based on specific field

Hi, I have an issue to combine multiple lines of a file. I have records as below. Fields are delimited by TAB. Each lines are ending with a new line char (\n) Input -------- ABC 123456 abcde 987 890456 7890 xyz ght gtuv ABC 5tyin 1234 789 ghty kuio ABC ghty jind 1234 678 ght ... (8 Replies)
Discussion started by: ratheesh2011
8 Replies

8. Shell Programming and Scripting

select the lines in between some time span

Hi Everyone ! i want to take all the lines from a file that falls in between some date... and every line in a file has a time stamp.. ---some text---- 01/Jan/2010 ---- some other text ---- ---some text---- 10/Jan/2010 ---- some other text ---- ---some text---- 20/Dec/2010 ---- some... (3 Replies)
Discussion started by: me_newbie
3 Replies

9. Shell Programming and Scripting

Combine multiple lines in single line

This is related to one of my previous post but now with a slight difference: I need the "Updated:" to be in one line as well as the "Information:" on one line as well. These are in multiple lines right now as seen below. These can have 2 or more lines that needs to be in one line. System name:... (8 Replies)
Discussion started by: The One
8 Replies

10. Shell Programming and Scripting

Combine multiple columns from multiple files

Hi there, I was wondering if someone can help me with this. I am trying the combine multiple columns from multiple files into one file. Example file 1: c0t0d0 c0t2d0 # hostname vgname c0t0d1 c0t2d1 # hostname vgname c0t0d2 c0t2d2 # hostname vgname c0t1d0 c0t3d0 # hostname vgname1... (5 Replies)
Discussion started by: martva
5 Replies
Login or Register to Ask a Question