Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

String search between patterns using sed

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting String search between patterns using sed
# 1  
Old 03-28-2013
String search between patterns using sed
Hi,
I am trying to find a way to get sed/awk/grep to help me find a string in a log file that exists between two datestamps and then print the preceding datestamp up to the next datestamp.

Here is an example of my logfile:
Code:
+++ 2013/03/28 17:01:37.085 SIGNALING HIGH ACTIVE

Failure Response Measurements:
400: 1

+++ 2013/03/28 17:01:37.085 SIGNALING HIGH ACTIVE

Failure Response Measurements:
408: 2
487: 4

+++ 2013/03/28 17:02:37.085 SIGNALING HIGH ACTIVE

Failure Response Measurements:
428: 2

+++ 2013/03/28 17:09:49.885 SIGNALING HIGH ACTIVE 

Failure Response Measurements:
487: 3

+++ 2013/03/28 18:44:48.926 SIGNALING HIGH ACTIVE

Failure Response Log:
Function Trace:
1188ce09 107f9f0f 11845800 11818639 11816b3c 118069cf
11807613 1075d735 11849f10 11848b58 11afb295 f7f4384e
f7d970aa

Response Code: 487
Request: 
<several lines of data here>

+++ 2013/03/28 19:41:28.369 SIGNALING HIGH ACTIVE

Failure Response Log:
Function Trace:
1188ce09 107f9f0f 10799a15 10797f42 107f6607 1073b770
11afb295 f7f7f84e f7dd30aa

Response Code: 408
Request:
<several lines of data here>

So for instance I have been trying to use:
Code:
sed -n '/\+\+\+/,/\+\+\+/ p' <logfile.log> |grep 487

This of course doesn't work. My desired result would be to search for "487" and get that line and all the lines that exist between the preceding datestamp and the following datestamp so that my results would look something like this:

Code:
Example desired output:

+++ 2013/03/28 17:01:37.085 SIGNALING HIGH ACTIVE

Failure Response Measurements:
408: 2
487: 4

+++ 2013/03/28 17:09:49.885 SIGNALING HIGH ACTIVE 

Failure Response Measurements:
487: 3

+++ 2013/03/28 18:44:48.926 SIGNALING HIGH ACTIVE

Failure Response Log:
Function Trace:
1188ce09 107f9f0f 11845800 11818639 11816b3c 118069cf
11807613 1075d735 11849f10 11848b58 11afb295 f7f4384e
f7d970aa

Response Code: 487
Request: 
<several lines of data here>

Any insight would be appreciated.
Last edited by Scrutinizer; 03-28-2013 at 10:54 PM.. Reason: extra code tags
# 2  
Old 03-28-2013
Code:
awk '
{
        R = ( R == "" ? $0 : R RS $0 )
        if ( $0 ~ /487/ )
                ++v
        if ( $0 ~ /^\+/ )
        {
                if ( v == 1 )
                        print R
                v = 0
                R = RS $0
        }

} ' logfile

# 3  
Old 03-28-2013
Quote:
Originally Posted by raytx
Hi,
I am trying to find a way to get sed/awk/grep to help me find a string in a log file that exists between two datestamps and then print the preceding datestamp up to the next datestamp.
It is relatively easy: read the log file in parts, where "part" is from one timestamp to the next. Accumulate these parts in the hold space. When you find a time stamp you finish the last part read in: exchange hold space and pattern space, check if the last part contains the string you searched for and print it if this is the case.


The following script which does that assumes a POSIX-compatible "sed", if you use a GNU-sed you might have to escape the "+":

Code:
sed -n '/^+++/ {
                  x
                  /<searchpattern here>/p
                  d
              }
        H' /path/to/input

I hope this helps.

bakunin
# 4  
Old 03-28-2013
Try with GNU awk (or mawk):
Code:
gawk '/487/{print "+++" $0}' RS='\+\+\+' ORS= file

With regular awk you could try
Code:
awk '/487/{print "+++" $0}' RS=+ ORS= file

But that would break if there is so much as a single +-sign in the record.
You could try replacing the +++ with a single character first if you know for sure that it will not occur in the text.
Code:
sed 's/^+++/|/' file | awk '/487/{print "+++" $0}' RS=\| ORS=

Anyway, probably best to use something like this:
Code:
awk '/487/{f=1} /^\+\+\+/{ if(f)print s ; f=0; s=$0; next } {s=s RS $0} END{ if(f)print s }' file

--
@Bakunin: very nice, but it would not work for the last record in the file.
@Yoda: also would not work for the last record and it prints the next "+++ header"..
Last edited by Scrutinizer; 03-29-2013 at 12:01 AM..
This User Gave Thanks to Scrutinizer For This Post:
raytx
# 5  
Old 03-29-2013
Are you stuck with the log file format?

I think it would be a lot simpler to process, and the resulting scripts easier to maintain, if you had a marker (### or whatever you might choose) for the end of each time stamp record.
Code:
+++ 2013/03/28 17:01:37.085 SIGNALING HIGH ACTIVE

Failure Response Measurements:
400: 1

###

+++ 2013/03/28 17:01:37.085 SIGNALING HIGH ACTIVE

# 6  
Old 03-29-2013
Quote:
Originally Posted by Scrutinizer
Try with GNU awk (or mawk):
Code:
gawk '/487/{print "+++" $0}' RS='\+\+\+' ORS= file

With regular awk you could try
Code:
awk '/487/{print "+++" $0}' RS=+ ORS= file

But that would break if there is so much as a single +-sign in the record.
You could try replacing the +++ with a single character first if you know for sure that it will not occur in the text.
Code:
sed 's/^+++/|/' file | awk '/487/{print "+++" $0}' RS=\| ORS=

Anyway, probably best to use something like this:
Code:
awk '/487/{f=1} /^\+\+\+/{ if(f)print s ; f=0; s=$0; next } {s=s RS $0} END{ if(f)print s }' file

--
@Bakunin: very nice, but it would not work for the last record in the file.
@Yoda: also would not work for the last record and it prints the next "+++ header"..
Thanks for the help and nice catch on the others not grabbing that last line. This seems to be working for me, so I can build around it.
Appreciate the help!

Ray
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Awk/sed command to extract the string between 2 patterns but having some particular value

Hi - i have one file with content as below. ***** BEGIN 123 ***** BASH is awesome ***** END ***** ***** BEGIN 365 ***** KSH is awesome ***** END ***** ***** BEGIN 157 ***** KSH is awesome ***** END ***** ***** BEGIN 7123 ***** C is awesome ***** END ***** I am trying to find all... (4 Replies)
Discussion started by: reldb
4 Replies

2. Shell Programming and Scripting

sed - search and replace whole string which contains dot

Hello. I would like to search exactly "string1.string2.string3" and replace it by "new_string1.new_string2.new_string3" And I would like to search exactly "string2.string3" and replace it by "new_string2.new_string3" And I would not found in the result : "string1.new_string2.new_string3"... (3 Replies)
Discussion started by: jcdole
3 Replies

3. Shell Programming and Scripting

Search for 2 string in 2 lines with sed ?

Hi ! I want to search a string in all lines with sed. If that string is there, i want to look for an other string in the next line.If that string is there i want to put an other line under it. eg: aaa bbb ccc ddd cat bla.txt | sed -e '/aaa/a\' -e ' \!!!' in the upper case, i would... (6 Replies)
Discussion started by: fugitivus
6 Replies

4. Shell Programming and Scripting

Recursive replacement of search string using sed

Dear Unix Forum Group Members, Please do let me know how I can replace the double pipe with single pipe recursively on single record. Sample Input Data: DN set|Call prefix||| Called number address nature 0||| *789|||||||ALL number types 0||| 00||||||||ALL number types 10||... (5 Replies)
Discussion started by: srinu.kadem
5 Replies

5. Shell Programming and Scripting

Script using Sed :Search all patterns & after the last Patter, insert a newLine with Comma Sep Value

I am trying to search the pattern "ARS (11)" and after the LAST pattern, i am trying to open new line and enter text using sed. My Existing Text file is Users.txtpaul, Paul Smith, Stevn Smiley, REQ000001, ARS (11) sam, Sam Martin, Stevn Smiley, REQ000001, ARS (11) mike, Mike Conway, Stevn... (8 Replies)
Discussion started by: evrurs
8 Replies

6. Shell Programming and Scripting

How to use SED or AWK to search and replace an exact string

I have a file DS1 DDS DS I want to replace only "DS" to "DSmail.blah.com" in a lot of files. I tried sed 's/DS/DSmail.blah.com' but it changes all the lines . thanks in advance (2 Replies)
Discussion started by: gubbu
2 Replies

7. Shell Programming and Scripting

awk/sed string search and replace

Need help with either sed or awk to acheive the following file1 ----- In the amazon forest The bats eat all the time... mon tue wed they would eat berries In the tropical forest The bats eat all the time... on wed bats eat nuts In the rain forest The bats eat all the time... on... (2 Replies)
Discussion started by: jville
2 Replies

8. Shell Programming and Scripting

String: Grep / SED for multy line search

Hi, At first I want to please you to provide the solution with grep/sed if possible. :cool: File looks like: wished result: so I want in a new file BLUE@@RED string from first line like: grep "/folder_start" cs_src > tmp1 string from second line: grep "/main" cs_src... (14 Replies)
Discussion started by: unknown7
14 Replies

9. Shell Programming and Scripting

sed search alternately for patterns

hi all, I am trying to do search on a gzip file. The file has <pattern1> Data.. <pattern2> data <pattern1> data <patter2> I want to print each pattern 1 and the corrresponding pattern2. If pattern 2 fails to appear and pattern 1 appears, I do not want to print pattern1 and... (3 Replies)
Discussion started by: baskar123
3 Replies

10. Shell Programming and Scripting

Need to replace all occurences of a search string using sed

All, Here is what I am searching for using sed. 1 00640000106798 I want to replace that with the following. 8 0064B0000106798 I can do this easy enough from the command line using sed but I need to put the search string in a file and then execute the sed command within a... (2 Replies)
Discussion started by: mjs3221
2 Replies
Login or Register to Ask a Question