Deleting lines from a stream after matching a pattern


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Deleting lines from a stream after matching a pattern
# 1  
Old 03-02-2012
Deleting lines from a stream after matching a pattern

Hi,

I have a requirement to to an ldapsearch and remove the shadow attributes in the output file.
What I do is ldapsearch() | operation to remove shadow > FILE
The ldapsearch gives output like this(with same line formation):

Code:
objectClass: FSConfig
objectClass: extensibleObject
fsCAIP: 1.1.1.1
fsCAPort-shadow: '<fsCAPort-shadow status="previous" currentValu
 e="ODA4MA==">ODA4MA==</fsCAPort-shadow>'
fsCAIP-shadow: '<fsCAIP-shadow status="previous" currentValue="M
 TAuOC4yMDcuNTQ=">MTAuOC4yMzIuMTcx</fsCAIP-shadow>'
fsCAPort: 8080
fsName: C=IN
fsName-shadow: '<fsName-shadow s
 tatus="previous" currentValue="Qz1JTg==">Qz1JTg==</fsN
 ame-shadow>'
...

I need to remove content starting from *-shadow to -shadow' for all instances.
i.e., My expected output is
Code:
objectClass: FSConfig
objectClass: extensibleObject
fsCAIP: 1.1.1.1
fsName: C=IN

How do I achieve this with sed/awk/grep/ anything else. Your help in this regard is much appreciated!
# 2  
Old 03-02-2012
Code:
awk '/shadow:/{x=1}/shadow>./{x=0;next}!x' yourfile

---------- Post updated at 03:44 PM ---------- Previous update was at 03:27 PM ----------

Code:
awk '{x=1}/shadow:/,/shadow>./{x=0}x' yourfile

---------- Post updated at 03:46 PM ---------- Previous update was at 03:44 PM ----------

Code:
awk '/shadow:/,/shadow>./{next}1' yourfile

This User Gave Thanks to ctsgnb For This Post:
# 3  
Old 03-02-2012
Thanks ctsgnb, works like a charm!
# 4  
Old 03-05-2012
Hi,

There is one problem with the code you had provided.
Code:
dn: fsDomain=default
 fsDomain: default
 objectClass: FSDomainCredentials
 objectClass: extensibleObject
 fsAttribA: xyz
 fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValu
  e="MTk2ODU=">MTk2ODU=</fsAttribA-shadow>'
 fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValue="WmZ
  MdC11U3BZLURGVnYtZEpMeQ==">WmZMdC11U3BZLURGVnYtZEpMeQ==</fsAttribA-shadow>
  '
 fsAttribB: 19685

When I executed the command on the above LDAP content, fsAttribB gets lost.

Code:
dn: fsDomain=default
 fsDomain: default
 objectClass: FSDomainCredentials
 objectClass: extensibleObject
 fsAttribA: xyz

How can I overcome this?
# 5  
Old 03-05-2012
This is because in this case, the single quote is not concatenated with shadow>

so the /shadow>./ cannot match /shadow>'/

Maybe you can give a try to this :

Code:
awk '/shadow:/,/shadow>/{next}length($0)<4{next}1' yourfile

( note that i removed the dot after the > signe)
here are some brief explanation about the code :

awk 'start of awk code
/shadow:/,/shadow>/every lines between (and including) a line containing pattern "shadow:" and pattern "shadow>"
{next}do not process further command, skip the current line, and process next line from the beginning of awk code
length($0)<4if the length of the current line is less than 4 characters
{next}do not process further command, skip the current line, and process next line from the beginning of awk code
1 "1" (means "true" ) the default behaviours is therefore to display the current line
' yourfile end of current awk statement and passing "yourfile" as argument
You can also give a try to :
Code:
awk '/shadow:/,/shadow>/{next}/^   *./{next}1' yourfile

(note: there are 3 spaces between the ^ and the *

You can also try to this last one , the matching pattern has been enhanced a little.

Code:
awk '/shadow:/,/shadow>/{next}/^ *.$/{next}1' yourfile


Last edited by ctsgnb; 03-05-2012 at 12:30 PM..
# 6  
Old 03-05-2012
Still some issues

Thanks,
But still some issue Smilie
Now I get duplicate entries in my output.
Eg:
For input
Code:
fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValu
 e="MTk2ODU=">MTk2ODU=</fsAttribA-shadow>'
fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValue="WmZ
 MdC11U3BZLURGVnYtZEpMeQ==">WmZMdC11U3BZLURGVnYtZEpMeQ==</fsAttribA-shadow>
 '
fsAttribA: 19685

I get
Code:
fsAttribA: 19685
...
fsAttribA: 19685

# 7  
Old 03-05-2012
1. Please upload your original input file.

2. Your initial requirement was to filter out things relating to "-shadow" stuff, not to look for duplicate entries.

If you don't care about the order, then you can just

Code:
awk '.... ' yourfile | sort | uniq

Code:
awk '.... ' yourfile | sort -u

Otherwise, please give all your requirements at once

---------- Post updated at 05:43 PM ---------- Previous update was at 05:38 PM ----------

I don't understand why you got duplicate entries : for me it worked fine :

Code:
$ cat f1
fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValu
 e="MTk2ODU=">MTk2ODU=</fsAttribA-shadow>'
fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValue="WmZ
 MdC11U3BZLURGVnYtZEpMeQ==">WmZMdC11U3BZLURGVnYtZEpMeQ==</fsAttribA-shadow>
 '
fsAttribA: 19685

Code:
$ awk '/shadow:/,/shadow>/{next}/^ *.$/{next}1' f1
fsAttribA: 19685

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Grep -v lines starting with pattern 1 and not matching pattern 2

Hi all! Thanks for taking the time to view this! I want to grep out all lines of a file that starts with pattern 1 but also does not match with the second pattern. Example: Drink a soda Eat a banana Eat multiple bananas Drink an apple juice Eat an apple Eat multiple apples I... (8 Replies)
Discussion started by: demmel
8 Replies

2. Shell Programming and Scripting

Deleting lines on matching certain pattern

hi I have a large xml file from which i have taken few lines . In this file I have to find for the string </invoices> and check if the 3 rd line after this string does not begin with <portCode> ,then i have to delete the string </invoices> and the next line having the string </shippingBill>... (13 Replies)
Discussion started by: sunnyboy
13 Replies

3. Shell Programming and Scripting

Sed: printing lines AFTER pattern matching EXCLUDING the line containing the pattern

'Hi I'm using the following code to extract the lines(and redirect them to a txt file) after the pattern match. But the output is inclusive of the line with pattern match. Which option is to be used to exclude the line containing the pattern? sed -n '/Conn.*User/,$p' > consumers.txt (11 Replies)
Discussion started by: essem
11 Replies

4. Shell Programming and Scripting

Matching a pattern 250 characters up and down stream

hii all i have a file a which contains some thing like this strand smthg position + yyx 3020 - yyw 10,000 now i have another file (file2) which contains the data starting from 1 to n positions i want to refer first file if + ... (4 Replies)
Discussion started by: anurupa777
4 Replies

5. Shell Programming and Scripting

Deleting a matching string(line) which is also in other lines

Hi, i need help with my shell script I have a file input.txt containing the following contents /. /usr /usr/share /usr/share/doc /usr/share/doc/wine /usr/share/doc/wine/copyright /usr/share/doc/wine/changelog.Debian.gz I need output as /usr/share/doc/wine /usr/share/doc/wine/copyright... (3 Replies)
Discussion started by: Amit0991
3 Replies

6. Shell Programming and Scripting

Removing all lines prior to the last pattern in a file/stream

Hi all, I didn't find anything that specifically answers this after searching for a bit, so please forgive me if this has been covered before. I'm looking to delete all lines prior to the last occurrence of a string in a file or stream from within a shell script (bash.) A bit of... (4 Replies)
Discussion started by: LivinFree
4 Replies

7. Shell Programming and Scripting

Finding lines matching the Pattern and their previous lines in a file

Hi, I am trying to locate the occurences of certain pattern like 'Possible network disconnect' in a text file. I can get the actual lines matching the pttern using: grep -w 'Possible network disconnect' file_name. But I am more interested in getting the timing of these events which are... (7 Replies)
Discussion started by: sagarparadkar
7 Replies

8. Shell Programming and Scripting

pattern matching lines using the date, and then joining the lines

Hi Guys, Was trying to attempt the below using awk and sed, have no luck so far, so any help would be appreciated. Current Text File: The first line has got an "\n", and the second line has got spaces/tabs then the word and "\n" TIME SERVER/CLIENT TEXT... (6 Replies)
Discussion started by: eo29
6 Replies

9. Shell Programming and Scripting

pattern matching over multiple lines and deleting the first

I've got a longish log file with content such as Uplink traffic: Downlink traffic: I want to parse the log file and remove any line that contains the string "Uplink traffic:" at the beginning of the line, but only if the line following it beginnings with the string "Downlink traffic:" (in... (7 Replies)
Discussion started by: Yorkie99
7 Replies

10. Shell Programming and Scripting

counting the lines matching a pattern, in between two pattern, and generate a tab

Hi all, I'm looking for some help. I have a file (very long) that is organized like below: >Cluster 0 0 283nt, >01_FRYJ6ZM12HMXZS... at +/99% 1 279nt, >01_FRYJ6ZM12HN12A... at +/99% 2 281nt, >01_FRYJ6ZM12HM4TS... at +/99% 3 283nt, >01_FRYJ6ZM12HM946... at +/99% 4 279nt,... (4 Replies)
Discussion started by: d.chauliac
4 Replies
Login or Register to Ask a Question

Featured Tech Videos