Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Replacing lines matching a multi-line pattern (sed/perl/awk)

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Replacing lines matching a multi-line pattern (sed/perl/awk)
# 8  
Old 02-25-2014
Code:
$
$
$ cat input1
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL memmove @PA1 0x0012f740 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL BlaCall @PA1 0x0012f741 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL memmove @PA1 0x0012f740 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
$
$ perl -lne 'BEGIN {$in = 0; $count = 0}
             if (/\@CAL RtlInitAnsiString/) {
                 if ($x[0] =~ /\@CAL RtlInitAnsiString/) {print foreach (@x); @x = (); $count=0}
                 push @x, $_; $in = 1; $count++
             }
             elsif (/\@CAL memmove/ and $x[0] =~ /\@CAL RtlInitAnsiString/ and $in) {
                 print "\@MATCH"; @x = (); $in = 0; $count = 0
             }
             elsif ($in) {
                 $count++;
                 if ($count > 3) {print foreach (@x); @x=(); $count=0; $in=0}
                 else { push @x, $_; $count++ }
             }
             else {print}
            ' input1
@MATCH
@MATCH
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
$
$
$ cat input2
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL BlaCall @PA1 0x0012f741 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL memmove @PA1 0x0012f740 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
$
$ perl -lne 'BEGIN {$in = 0; $count = 0}
             if (/\@CAL RtlInitAnsiString/) {
                 if ($x[0] =~ /\@CAL RtlInitAnsiString/) {print foreach (@x); @x = (); $count=0}
                 push @x, $_; $in = 1; $count++
             }
             elsif (/\@CAL memmove/ and $x[0] =~ /\@CAL RtlInitAnsiString/ and $in) {
                 print "\@MATCH"; @x = (); $in = 0; $count = 0
             }
             elsif ($in) {
                 $count++;
                 if ($count > 3) {print foreach (@x); @x=(); $count=0; $in=0}
                 else { push @x, $_; $count++ }
             }
             else {print}
            ' input2
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@MATCH
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
$
$
$ cat input3
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL BlaCall @PA1 0x0012f741 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL BlaCall @PA1 0x0012f741 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL memmove @PA1 0x0012f740 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
$
$ perl -lne 'BEGIN {$in = 0; $count = 0}
             if (/\@CAL RtlInitAnsiString/) {
                 if ($x[0] =~ /\@CAL RtlInitAnsiString/) {print foreach (@x); @x = (); $count=0}
                 push @x, $_; $in = 1; $count++
             }
             elsif (/\@CAL memmove/ and $x[0] =~ /\@CAL RtlInitAnsiString/ and $in) {
                 print "\@MATCH"; @x = (); $in = 0; $count = 0
             }
             elsif ($in) {
                 $count++;
                 if ($count > 3) {print foreach (@x); @x=(); $count=0; $in=0}
                 else { push @x, $_; $count++ }
             }
             else {print}
            ' input3
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL BlaCall @PA1 0x0012f741 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@MATCH
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
$
$
$ cat input4
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL memmove @PA1 0x0012f740 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL BlaCall @PA1 0x0012f741 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL memmove @PA1 0x0012f740 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL memmove @PA1 0x0012f740 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
$
$
$ perl -lne 'BEGIN {$in = 0; $count = 0}
             if (/\@CAL RtlInitAnsiString/) {
                 if ($x[0] =~ /\@CAL RtlInitAnsiString/) {print foreach (@x); @x = (); $count=0}
                 push @x, $_; $in = 1; $count++
             }
             elsif (/\@CAL memmove/ and $x[0] =~ /\@CAL RtlInitAnsiString/ and $in) {
                 print "\@MATCH"; @x = (); $in = 0; $count = 0
             }
             elsif ($in) {
                 $count++;
                 if ($count > 3) {print foreach (@x); @x=(); $count=0; $in=0}
                 else { push @x, $_; $count++ }
             }
             else {print}
            ' input4
@MATCH
@MATCH
@LIB ADVAPI32.dll @CAL RtlInitAnsiString @PA1 0x0012f740 @PA2 "CriticalSectionTimeout" @RET0
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL RtlAnsiStringToUnicodeString @PA1 0x7ffdfbf8 @PA2 0x0012f740 @PA3 FALSE @RET STATUS_SUCCESS
@LIB ADVAPI32.dll @CAL memmove @PA1 0x0012f740 @PA2 0x0012f68c @PA3 4 @RET 0x0012f8bc
$
$
$

This User Gave Thanks to durden_tyler For This Post:
thefang
# 9  
Old 02-25-2014
You should pick one tool and keep stick to it.
A starting point to give python a try:
Code:
#!/usr/bin/env python
import re
import sys

pattern = re.compile(r'''
    ^[^\n]+@CAL\sRtlInitAnsiString\s@PA1\s0x0012f740[^\n]+\n
    (?:(?!^[^\n]+RtlInitAnsiString)[^\n]+\n){0,3}
    ^[^\n]+@CAL\smemmove\s@PA1\s0x0012f740[^\n]+
    ''', re.X|re.M|re.S)

with open(sys.argv[1], 'r') as f:
    text = f.read()

tfilter = re.sub(pattern, '@MATCH', text)

print tfilter 

sys.exit(0)

This User Gave Thanks to Klashxx For This Post:
thefang
# 10  
Old 02-25-2014
Also a starting point to give awk a try:

Code:
awk \
  -v S="@CAL RtlInitAnsiString @PA1 0x0012f740" \
  -v E="@CAL memmove @PA1 0x0012f740" \
  -v M="@MATCH" \
  -v L=3 '
$0~S{if(R)print V;V=$0;R=FNR+L;next}
FNR==R{print V; R=V=x}
R&&$0~E {$0=M; R=V=x}
R{V=V"\n"$0;next}
1
END{if(V)print V}' $1

This User Gave Thanks to Chubler_XL For This Post:
thefang
# 11  
Old 02-26-2014
Wow, three very nice approaches using three powerful tools (awk integrates best into the existing analysis scripts but the python and perl solutions look very promising too) - I will now test them on different and (much) larger sets of input files/patterns and then probably stick to the fastest one.

Thanks again, your input was extremely helpful and sure motivates me to get deeper into this kind of scripting.

Cheers!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk with sed to combine lines and remove specific odd # pattern from line

In the awk piped to sed below I am trying to format file by removing the odd xxxx_digits and whitespace after, then move the even xxxx_digit to the line above it and add a space between them. There may be multiple lines in file but they are in the same format. The Filename_ID line is the last line... (4 Replies)
Discussion started by: cmccabe
4 Replies

2. Shell Programming and Scripting

Sed: printing lines AFTER pattern matching EXCLUDING the line containing the pattern

'Hi I'm using the following code to extract the lines(and redirect them to a txt file) after the pattern match. But the output is inclusive of the line with pattern match. Which option is to be used to exclude the line containing the pattern? sed -n '/Conn.*User/,$p' > consumers.txt (11 Replies)
Discussion started by: essem
11 Replies

3. Shell Programming and Scripting

sed multiple multi line blocks of text containing pattern

Hi, I have a log file which has sessionids in it, each block in the log starts with a date entry, a block may be a single line or multiple lines. I need to sed (or awk) out the lines/blocks with that start with a date and include the session id. The files are large at several Gb. My... (3 Replies)
Discussion started by: andyatit
3 Replies

4. Shell Programming and Scripting

Sed/awk/perl command to replace pattern in multiple lines

Hi I know sed and awk has options to give range of line numbers, but I need to replace pattern in specific lines Something like sed -e '1s,14s,26s/pattern/new pattern/' file name Can somebody help me in this.... I am fine with see/awk/perl Thank you in advance (9 Replies)
Discussion started by: dani777
9 Replies

5. Shell Programming and Scripting

sed to replace a line with multi lines from a var

I am trying to find a line in a file ("Replace_Flag") and replace it with a variable which hold a multi lined file. myVar=`cat myfile` sed -e 's/Replace_Flag/'$myVar'/' /pathto/test.file myfile: cat dog boy girl mouse house test.file: football hockey Replace_Flag baseball ... (4 Replies)
Discussion started by: bblondin
4 Replies

6. Shell Programming and Scripting

Summing over specific lines and replacing the lines with the sum using sed, awk

Hi friends, This is sed & awk type question. I have a text file which has numbers spread all over the file. I want to sum the series of numbers whenever i find it and produce an output file with the sum. For example ###start of input text file #### abc def ghi 1 2 3 4 kjld random... (3 Replies)
Discussion started by: kaaliakahn
3 Replies

7. Shell Programming and Scripting

sed or awk delete character in the lines before and after the matching line

Sample file: This is line one, this is another line, this is the PRIMARY INDEX line l ; This is another line The command should find the line with “PRIMARY INDEX” and remove the last character from the line preceding it (in this case , comma) and remove the first character from the line... (5 Replies)
Discussion started by: KC_Rules
5 Replies

8. Shell Programming and Scripting

replacing multi lines with 1 line

I have an xml file that is stripped down to output that looks bacically like; <!-- TABLEA header --> <tablea> some fields </tablea> <!-- TABLEB header --> <!-- TABLEC header --> <tablec> some fields </tablec> I want to remove the header... (3 Replies)
Discussion started by: Griffs_Revenge
3 Replies

9. Shell Programming and Scripting

How to use sed to modify a line above or below matching pattern?

I couldn't figure out how to use sed or any other shell to do the following. Can anyone help? Thanks. If seeing a string (e.g., TODAY) in the line, replace a string in the line above (e.g, replace "Raining" with "Sunny") and replace a string in the line below (e.g., replace "Reading" with... (7 Replies)
Discussion started by: sprinner
7 Replies

10. Shell Programming and Scripting

AWK - Pattern Matching & Replacing - Performance

Experts, I am a beginner to Unix Shell Scripting We have source as a flat file which contains CTRL+F character as the delimiter. We need to count the number of records in the file (CTRL+F) to perform file validation Following command being used: awk '{cnt+=gsub(//,"&")}END {print cnt}'... (4 Replies)
Discussion started by: srivijay81
4 Replies
Login or Register to Ask a Question