Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract line between two patterns

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract line between two patterns
# 1  
Old 12-02-2012
Extract line between two patterns
Hi All,
I need a script to extract a lines between two patterns.I have done this using grep,cut,tail and head.But its very slow, because my input file contain more than a lakh.

Code:
COMMAND:XXXXXXXXXXXXXXXXXXXX
yyyyy
zzzzzz
REQUESTSTRING:aaaaaaaaaaaaaaa;11111
222222
333333
SEARCHPATTERN
44444444444444
55555555
66666

COMMAND:XXXXXXXXXXXXXXXXXXXX
yyyyy
zzzzzz
REQUESTSTRING:bbbbb SEARCHPATTERN bbbbbbbb;11111
222222
333333
44444444444444
55555555
66666

COMMAND:XXXXXXXXXXXXXXXXXXXX
yyyyy
zzzzzz
REQUESTSTRING:cccccccccccc;11111
222222
333333
SEARCHPATTERN
44444444444444
55555555
66666

COMMAND:XXXXXXXXXXXXXXXXXXXX
yyyyy
zzzzzz
REQUESTSTRING:dddddddddddd;111111
222222
333333
44444444444444
55555555
66666

From the above sample,

Extract single COMMAND and search for SEARCHPATTERN, if SEARCHPATTERN available print the REQUESTSTRING line before ";" .

Note:
The search pattern may present in separate line or in REQUESTSTRING line itself.

Expected output is:

Code:
REQUESTSTRING:aaaaaaaaaaaaaaa
REQUESTSTRING:bbbbb SEARCHPATTERN bbbbbbbb
REQUESTSTRING:ccccccccccccccc

My sample code using grep..

Code:
grep -n -e "COMMAND" input.txt > tmp.txt
startline=0
while read line
do
	if [ "$startline" = "0" ]
	then
		read line
	fi
	endline=`echo $line |cut -d ":" -f 1`
	endline=`expr $endline - 1`
#	echo "Startline:$startline,Endline:$endline"
	noofline=`expr $endline - $startline`
	head -$endline input.txt > t1.txt
	tail -$noofline t1.txt > t2.txt 
	grep "SEARCHPATTERN " t2.txt > /dev/dull
	if [ "$?" = "0" ] 
	then
		grep "REQUESTSTRING " t2.txt | cut -d ";" -f 1 
	fi
	startline=$endline
done < tmp.txt

# 2  
Old 12-02-2012
With assumptions:
Code:
awk '/SEARCHPATTERN/ && NF==2{split($2,a,/;/);print FS a[1]}' FS='REQUESTSTRING:' RS= file

# 3  
Old 12-02-2012
I believe this does what you want:
Code:
awk -F';' '/^COMMAND:/{
        found=1
        if(saved) print saved
}
/^REQUESTSTRING:/{
        if(found) saved = $1
}' input

----------------------------
Oops. I looked too closely at the expected output and missed the part about needing SEARCHPATTERN. The above script happens to work with the sample input, but is not correct in the general case.

What I should have said is something more like:
Code:
awk -F';' '/^COMMAND:/{
        if(fc && fs && saved) print saved
        fc = 1
        saved = ""
        fs = 0
}
/^REQUESTSTRING:/{
        if(fc) saved = $1
}
/SEARCHPATTERN/{
        fs = 1
}
END {   if(fc && fs && saved) print saved}' input

but elixir_sinari's proposal is shorter and should work fine as long as the input file doesn't violate the assumptions made. And, given the sample input, the assumptions look perfectly reasonable to me.
Last edited by Don Cragun; 12-02-2012 at 07:59 AM..
This User Gave Thanks to Don Cragun For This Post:
rajamohan
# 4  
Old 12-02-2012
Through Sed..
Code:
 sed -n 'H;/^$/{x;/COMMAND.*SEARCHPATTERN/s/.*\(REQUESTSTRING[^;]*\);.*/\1/p}' inputfile

This User Gave Thanks to michaelrozar17 For This Post:
rajamohan
# 5  
Old 12-03-2012
Quote:
Originally Posted by Don Cragun
I believe this does what you want:
Code:
awk -F';' '/^COMMAND:/{
        found=1
        if(saved) print saved
}
/^REQUESTSTRING:/{
        if(found) saved = $1
}' input

----------------------------
Oops. I looked too closely at the expected output and missed the part about needing SEARCHPATTERN. The above script happens to work with the sample input, but is not correct in the general case.

What I should have said is something more like:
Code:
awk -F';' '/^COMMAND:/{
        if(fc && fs && saved) print saved
        fc = 1
        saved = ""
        fs = 0
}
/^REQUESTSTRING:/{
        if(fc) saved = $1
}
/SEARCHPATTERN/{
        fs = 1
}
END {   if(fc && fs && saved) print saved}' input

but elixir_sinari's proposal is shorter and should work fine as long as the input file doesn't violate the assumptions made. And, given the sample input, the assumptions look perfectly reasonable to me.
Thanks a lot.. Don Cragun.. its working good..

---------- Post updated at 01:55 AM ---------- Previous update was at 01:53 AM ----------

Quote:
Originally Posted by michaelrozar17
Through Sed..
Code:
 sed -n 'H;/^$/{x;/COMMAND.*SEARCHPATTERN/s/.*\(REQUESTSTRING[^;]*\);.*/\1/p}' inputfile

Thanks a lot michaelrozar17.
Its very simple and working good..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract lines between patterns

I have a list in the format below, how do I read through the list and extract the lines between the ##START## and ##END##, so i can check for specific values between each ##START## & ##END## pattern ##START## RANDOMTEXT DFGSD SDFSDF ##END## ##START## morestuff sdfggfg sdfsdf... (10 Replies)
Discussion started by: squrcles
10 Replies

2. Shell Programming and Scripting

Extract all the sentences that matched two patterns

Hi I have two lists of patterns named A and B consisting of around 200 entries in each and I want to extract all the sentences from a big text file which match atleast one pattern from both A and B. For example, pattern list A consists of : ama ani ahum mari ... ... and pattern... (1 Reply)
Discussion started by: my_Perl
1 Replies

3. Debian

Extract Lines Between 2 patterns if exist...

Hello. I am not having luck with sed or awk today. $ echo "$BrackListFinal" DSCF3649-DSCF3651_Brkt DSCF3649.JPG 2014-07-21 13:34:44 On 1 DSCF3649.RAF 2014-07-21 13:34:44 On 1 DSCF3650.JPG 2014-07-21 13:34:45 On 2 DSCF3650.RAF 2014-07-21 13:34:45 On 2... (3 Replies)
Discussion started by: DSommers
3 Replies

4. Shell Programming and Scripting

Find matched patterns and print them with other patterns not the whole line

Hi, I am trying to extract some patterns from a line. The input file is space delimited and i could not use column to get value after "IN" or "OUT" patterns as there could be multiple white spaces before the next digits that i need to print in the output file . I need to print 3 patterns in a... (3 Replies)
Discussion started by: redse171
3 Replies

5. Shell Programming and Scripting

Extract all the lines in between of 2 patterns and merge them

Hi, I have a file with many lines and need to extract lines between 2 patterns (AAA and BBB) and merge all the in-between lines into single line separated by space. $ cat file1 blah blah blah blah AAA 1 2 3 blah BBB blah blah blah blah blah blah blah blah blah AAA 5 6 blah blah... (4 Replies)
Discussion started by: prvnrk
4 Replies

6. Shell Programming and Scripting

Need to extract text repetitively between two patterns

Hi All, I want to extract the text between some pattern which occurs repeatedly in a file. For example my input is like, /home/..... ..........java:25: cannot find symbol ............ /home/...... /home/....... I want to display... (2 Replies)
Discussion started by: Vignesh58
2 Replies

7. Shell Programming and Scripting

PERL: extract lines between two patterns

Hello Perl-experts, I am new to perl and need help to solve a problem. I have a table in below format. <Text A> <Pattern1> A Value B Value C Value D Value <Pattern2> <Text B> This table is in file1. I want to extract lines between Pattern1 and Pattern2 and write it into file2.... (11 Replies)
Discussion started by: mnithink
11 Replies

8. Shell Programming and Scripting

Extract patterns and copy them in different files

Hi All, I have a file which looks like this: Name1;A01 Name2;A01.047 Name3;A01.047.025 Newname1;B01 NewName2;B01.056.32 NewName3;B04.09.43 NewNewName1;C01.03 NewNewName2;C01.034.44As you can see, in the file there is some name and followed by the name is some identifier. These... (5 Replies)
Discussion started by: shoaibjameel123
5 Replies

9. Shell Programming and Scripting

Extract certain patterns from file.

Hi All, I tried extracting this pattern using grep but it did not work. What I have is a file which has contents like this: file:///channel/add-adhd.html file:///channel/allergies.html file:///channel/arthritis.html http://mail.yahoo.com/ http://messenger.yahoo.com/... (2 Replies)
Discussion started by: shoaibjameel123
2 Replies

10. Shell Programming and Scripting

To extract the string between two patterns

Sample input: Loading File System Networking in nature Closing the System now i need to extract the patterns between the words File and Closing: i.e. sample output: System Networking in Nature Thanks in advance !!!!!!!!!!!!!!!!! (6 Replies)
Discussion started by: aajan
6 Replies
Login or Register to Ask a Question