Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Combine logs span across multiple lines
Top Forums Shell Programming and Scripting Combine logs span across multiple lines Post 302988100 by drl on Tuesday 20th of December 2016 11:19:58 AM
Old 12-20-2016
Hi.

An alternate solution using a (non-standard) member of the grep family:
Code:
cgrep +w <delimiter-string-list> <search-string> <filename>

will search <filename> for <search-string>. When found, it will also copy lines up to a line containing a match for <delimiter-string-list>.

Here are examples using an extended dataset:
Code:
#!/usr/bin/env bash

# @(#) s1       Demonstrate extraction of blocks of delimited text, cgrep.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C cgrep

FILE=${1-data1}

pl " Input data file $FILE:"
cat $FILE

markers='ERROR|DEBUG|CLARK|ZEPHYR'
search=CLARK
pl " Results, looking for $search:"
cgrep -D -E +I2 +w "$markers" $search $FILE

search=ERROR
pl " Results, looking for $search:"
cgrep -D -E +I2 +w "$markers" $search $FILE

search=DEBUG
pl " Results, looking for $search:"
cgrep -D -E +I2 +w "$markers" $search $FILE

exit 0

producing:
Code:
$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.6 (jessie) 
bash GNU bash 4.3.30
cgrep ATT cgrep 8.15

-----
 Input data file data1:
ERROR 2016-12-08 10:22:23.542 some data
**** some data****** 1
**** some data****** 2
**** some data****** 3
DEBUG 2016-12-08 10:23:23.542 some data
**** some data****** 4
**** some data****** 5
**** some data****** 6
CLARK 2016-11-08 10:23:23.542 some data
**** some data****** 7
**** some data****** 8
**** some data****** 9
**** some data****** 9a
ZEPHYR 2015-11-08 10:23:23.542 some data
**** some data****** 10
**** some data****** 11
**** some data****** 12
DEBUG 2014-12-08 10:23:23.542 some data
**** some data****** 13
**** some data****** 14
**** some data****** 15

-----
 Results, looking for CLARK:
CLARK 2016-11-08 10:23:23.542 some data
**** some data****** 7
**** some data****** 8
**** some data****** 9
**** some data****** 9a

-----
 Results, looking for ERROR:
ERROR 2016-12-08 10:22:23.542 some data
**** some data****** 1
**** some data****** 2
**** some data****** 3

-----
 Results, looking for DEBUG:
DEBUG 2016-12-08 10:23:23.542 some data
**** some data****** 4
**** some data****** 5
**** some data****** 6
DEBUG 2014-12-08 10:23:23.542 some data
**** some data****** 13
**** some data****** 14
**** some data****** 15

The cgrep code needs to be acquired, compiled, and made available. I have done this several times over the years. An additional advantage is that cgrep is very fast.

Here are some details:
Code:
cgrep   shows context of matching patterns found in files (man)
Path    : ~/executable/cgrep
Version : 8.15
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYSV ...)
Home    : http://sourceforge.net/projects/cgrep/

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
rbatte1
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Combine multiple columns from multiple files

Hi there, I was wondering if someone can help me with this. I am trying the combine multiple columns from multiple files into one file. Example file 1: c0t0d0 c0t2d0 # hostname vgname c0t0d1 c0t2d1 # hostname vgname c0t0d2 c0t2d2 # hostname vgname c0t1d0 c0t3d0 # hostname vgname1... (5 Replies)
Discussion started by: martva
5 Replies

2. Shell Programming and Scripting

Combine multiple lines in single line

This is related to one of my previous post but now with a slight difference: I need the "Updated:" to be in one line as well as the "Information:" on one line as well. These are in multiple lines right now as seen below. These can have 2 or more lines that needs to be in one line. System name:... (8 Replies)
Discussion started by: The One
8 Replies

3. Shell Programming and Scripting

select the lines in between some time span

Hi Everyone ! i want to take all the lines from a file that falls in between some date... and every line in a file has a time stamp.. ---some text---- 01/Jan/2010 ---- some other text ---- ---some text---- 10/Jan/2010 ---- some other text ---- ---some text---- 20/Dec/2010 ---- some... (3 Replies)
Discussion started by: me_newbie
3 Replies

4. Shell Programming and Scripting

Combine multiple lines in file based on specific field

Hi, I have an issue to combine multiple lines of a file. I have records as below. Fields are delimited by TAB. Each lines are ending with a new line char (\n) Input -------- ABC 123456 abcde 987 890456 7890 xyz ght gtuv ABC 5tyin 1234 789 ghty kuio ABC ghty jind 1234 678 ght ... (8 Replies)
Discussion started by: ratheesh2011
8 Replies

5. Shell Programming and Scripting

Combine multiple unique lines from event log text file into one line, use PERL or AWK?

I can't decide if I should use AWK or PERL after pouring over these forums for hours today I decided I'd post something and see if I couldn't get some advice. I've got a text file full of hundreds of events in this format: Record Number : 1 Records in Seq : ... (3 Replies)
Discussion started by: Mayday22
3 Replies

6. Shell Programming and Scripting

Shell Script for viewing multiple logs from multiple server

I am new to Shell scripting and below is my requirement. I need to search some specific word e.g. "exception" or "transaction" from log file. We have multiple env e.g. Level1 , Level2 etc and each env have Multiple boxes e.g. For Level 1 env we have "test11.test.com" , "test12.test.com". Each... (1 Reply)
Discussion started by: peeyush
1 Replies

7. Shell Programming and Scripting

Combine multiple lines into single line

Hi All , I have a file with below data # User@Host: xyz @ # Query_time: t1 Lock_time: t2 Rows_sent: n1 Rows_examined: n2 SET timestamp=1396852200; select count(1) from table; # Time: 140406 23:30:01 # User@Host: abc @ # Query_time: t1 Lock_time: t2 Rows_sent: n1 Rows_examined:... (6 Replies)
Discussion started by: rakesh_411
6 Replies

8. Shell Programming and Scripting

Awk: Combine multiple lines based on number of fields

If a file has following kind of data, comma delimited 1,2,3,4 1 1 1,2,3,4 1,2 2 2,3,4 My required output must have only 4 columns with comma delimited 1,2,3,4 111,2,3,4 1,222,3,4 I have tried many awk command using ORS="" but couldnt progress (10 Replies)
Discussion started by: mdkm
10 Replies

9. Shell Programming and Scripting

awk to combine lines if fields match in lines

In the awk below, what I am attempting to do is check each line in the tab-delimeted input, which has ~20 lines in it, for a keyword SVTYPE=Fusion. If the keyword is found I am splitting $3 using the . (dot) and reading the portion before and after the dot in an array a. If it does have that... (12 Replies)
Discussion started by: cmccabe
12 Replies

10. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies
All times are GMT -4. The time now is 12:27 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy