Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract lines between patterns

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract lines between patterns
# 1  
Old 01-15-2016
Extract lines between patterns
I have a list in the format below, how do I read through the list and extract the lines between the ##START## and ##END##, so i can check for specific values between each ##START## & ##END## pattern


Code:
##START##
RANDOMTEXT
DFGSD
SDFSDF
##END##
##START##
morestuff
sdfggfg
sdfsdf
##END##


EG.

Code:
X="morestuff sdfggfg sdfsdf"

Moderator's Comments:
Mod Comment Please use CODE tags (not ICODE tags) for multi-line samples of input, output, and code.
Last edited by Don Cragun; 01-15-2016 at 03:04 PM.. Reason: Change ICODE tags to CODE tags and add ICODE tags.
# 2  
Old 01-15-2016
What tool/command (shell? awk? sed?)?

---------- Post updated at 18:26 ---------- Previous update was at 18:04 ----------

shell (bash):
Code:
while read T
    do  case $T in
        ("##START##")   continue;;
        ("##END##")     echo $X
                        unset X   
                        continue;;
        (*)             X+=$T" ";;
        esac   
   done < file
RANDOMTEXT DFGSD SDFSDF
morestuff sdfggfg sdfsdf

Last edited by RudiC; 01-15-2016 at 01:43 PM.. Reason: typo
# 3  
Old 01-15-2016
Quote:
Originally Posted by squrcles
I have a list in the format below, how do I read through the list and extract the lines between the ##START## and ##END## , so i can check for specific values between each ##START## & ##END## pattern



##START##
RANDOMTEXT
DFGSD
SDFSDF
##END##
##START##
morestuff
sdfggfg
sdfsdf
##END##


EG.
X="morestuff sdfggfg sdfsdf"
Hello squrcles,

Could you please try following and let me know if this helps.
Code:
awk '/##START##/{A=1;if(Q){print Q;Q=""};next} /##END##/{A=0} A{Q=Q?Q OFS $0:$0} END{print Q}'  Input_file

Output will be as follows.
Code:
RANDOMTEXT  DFGSD  SDFSDF 
morestuff  sdfggfg  sdfsdf

Thanks,
R. Singh
# 4  
Old 01-15-2016
And expanding on RavinderSingh13's solution to put it into your variable 'x':-
Code:
#!/bin/bash
# test.sh
echo '##START##
RANDOMTEXT
DFGSD
SDFSDF
##END##
##START##
morestuff
sdfggfg
sdfsdf
##END##' > /tmp/text
x=$(awk '/##START##/{A=1;if(Q){print Q;Q=""};next} /##END##/{A=0} A{Q=Q?Q OFS $0:$0} END{print Q}' < /tmp/text)
n=1
while read line
do
	echo "$n = $line"
	n=$(($n+1))
done <<< "$x"

Results on OSX 10.7.5, default bash terminal...
Code:
Last login: Fri Jan 15 19:35:11 on ttys000
AMIGA:barrywalker~> cd Desktop/Code/Shell
AMIGA:barrywalker~/Desktop/Code/Shell> chmod 755 test.sh
AMIGA:barrywalker~/Desktop/Code/Shell> ./test.sh
1 = RANDOMTEXT DFGSD SDFSDF
2 = morestuff sdfggfg sdfsdf
AMIGA:barrywalker~/Desktop/Code/Shell> _

Last edited by wisecracker; 01-15-2016 at 04:05 PM.. Reason: Correct copy and paste error in the results window.
# 5  
Old 01-15-2016
I was having some fun with doing this in sed to produce a script to pipe to shell. The idea is that each START / END segment is numbered and the text is piped to a shell function with the segment number (first one being 1).

Might not be worth anything...
Place the following into test.sed

Code:
1i\
seg=0
/##START##/b a
b
:a
s/.*$/seg=`expr $seg + 1`/p
n
s/^/echo '/p
:b
n
/##END##/b e
p
b b
:e
s/.*$/' | call_func $seg/p

Then you can get an idea of where this could go by doing:

Code:
sed -f test.sed my-input-file.txt

# 6  
Old 01-15-2016
Slight variation with a trailing space:
Code:
awk '/##END##/{p=0; print s; s=x} p{s=s $0 FS}; /##START##/{p=1}' file


Another sed, testing if the hold buffer is non-empty ...
Code:
sed -n '/##START##/{n;h;d;}; /##END##/{g;s/\n/ /g;p;s/.*//;h;d;}; x;/./{x;H;}'


Shorter, but less efficient, using more memory:
Code:
sed -n '/##START##/{n;h;d;}; /##END##/{g;s/\n/ /g;p;d;}; H' file

Last edited by Scrutinizer; 01-16-2016 at 06:41 AM..
# 7  
Old 01-16-2016
Thanks for all you suggestions,
i'm my haste I did not make it clear what my end required result was.

whilst reading through the list I need to capture the text between each individual ##START## and ##END## pair to run a check against the contents.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to print lines from a files with specific start and end patterns and pick only the last lines?

Hi, I need to print lines which are matching with start pattern "SELECT" and END PATTERN ";" and only select the last "select" statement including the ";" . I have attached sample input file and the desired input should be as: INPUT FORMAT: SELECT ABCD, DEFGH, DFGHJ, JKLMN, AXCVB,... (5 Replies)
Discussion started by: nani2019
5 Replies

2. UNIX for Beginners Questions & Answers

Delete multiple lines between blank lines containing two patterns

Hi all, I'm looking for a way (sed or awk) to delete multiple lines between blank lines containing two patterns ex: user: alpha parameter_1 = 15 parameter_2 = 1 parameter_3 = 0 user: alpha parameter_1 = 15 parameter_2 = 1 parameter_3 = 0 user: alpha parameter_1 = 16... (3 Replies)
Discussion started by: ce9888
3 Replies

3. Shell Programming and Scripting

Extract all the sentences that matched two patterns

Hi I have two lists of patterns named A and B consisting of around 200 entries in each and I want to extract all the sentences from a big text file which match atleast one pattern from both A and B. For example, pattern list A consists of : ama ani ahum mari ... ... and pattern... (1 Reply)
Discussion started by: my_Perl
1 Replies

4. Debian

Extract Lines Between 2 patterns if exist...

Hello. I am not having luck with sed or awk today. $ echo "$BrackListFinal" DSCF3649-DSCF3651_Brkt DSCF3649.JPG 2014-07-21 13:34:44 On 1 DSCF3649.RAF 2014-07-21 13:34:44 On 1 DSCF3650.JPG 2014-07-21 13:34:45 On 2 DSCF3650.RAF 2014-07-21 13:34:45 On 2... (3 Replies)
Discussion started by: DSommers
3 Replies

5. Shell Programming and Scripting

Extract all the lines in between of 2 patterns and merge them

Hi, I have a file with many lines and need to extract lines between 2 patterns (AAA and BBB) and merge all the in-between lines into single line separated by space. $ cat file1 blah blah blah blah AAA 1 2 3 blah BBB blah blah blah blah blah blah blah blah blah AAA 5 6 blah blah... (4 Replies)
Discussion started by: prvnrk
4 Replies

6. Shell Programming and Scripting

Extract line between two patterns

Hi All, I need a script to extract a lines between two patterns.I have done this using grep,cut,tail and head.But its very slow, because my input file contain more than a lakh. COMMAND:XXXXXXXXXXXXXXXXXXXX yyyyy zzzzzz REQUESTSTRING:aaaaaaaaaaaaaaa;11111 222222 333333... (4 Replies)
Discussion started by: rajamohan
4 Replies

7. Shell Programming and Scripting

PERL: extract lines between two patterns

Hello Perl-experts, I am new to perl and need help to solve a problem. I have a table in below format. <Text A> <Pattern1> A Value B Value C Value D Value <Pattern2> <Text B> This table is in file1. I want to extract lines between Pattern1 and Pattern2 and write it into file2.... (11 Replies)
Discussion started by: mnithink
11 Replies

8. Shell Programming and Scripting

Extract certain patterns from file.

Hi All, I tried extracting this pattern using grep but it did not work. What I have is a file which has contents like this: file:///channel/add-adhd.html file:///channel/allergies.html file:///channel/arthritis.html http://mail.yahoo.com/ http://messenger.yahoo.com/... (2 Replies)
Discussion started by: shoaibjameel123
2 Replies

9. Shell Programming and Scripting

Searching patterns in 1 file and deleting all lines with those patterns in 2nd file

Hi Gurus, I have a file say for ex. file1 which has 3500 lines in it which are different account numbers and another file (file2) which has 230000 lines in it. I want to read all the lines in file1 and delete all those lines from file2 which has that same pattern as in file1. I am not quite... (4 Replies)
Discussion started by: toms
4 Replies

10. Shell Programming and Scripting

To extract the string between two patterns

Sample input: Loading File System Networking in nature Closing the System now i need to extract the patterns between the words File and Closing: i.e. sample output: System Networking in Nature Thanks in advance !!!!!!!!!!!!!!!!! (6 Replies)
Discussion started by: aajan
6 Replies
Login or Register to Ask a Question