delete two patterns and remove one pattern

ppat7046 · #1 (**permalink**) 04-07-2009

Friends,

I have .txt file with following format.
START
ABC|Prashant1|Patel1
ABC|Prashant2|Patel2
ABC|Prashant1|Patel1
ABC|Prashant2|Patel2
END

I would like to do:
1) Delete line with START
2) Delete line with END
3) Remove ABC|
4) Delete duplicate records

The following command works fine which deletes line with START and END
sed -e /^START/d -e /^END/d Filename.txt

How do I incorporate task 3 and 4?
NOTE: The file will have more than 500,000 thousand rows.

Thanks in advance for suggestion,
Prashant

pinnacle · #2 (**permalink**) 04-07-2009

Quote:

Originally Posted by ppat7046

Friends,

I have .txt file with following format.
START
ABC|Prashant1|Patel1
ABC|Prashant2|Patel2
ABC|Prashant1|Patel1
ABC|Prashant2|Patel2
END

I would like to do:
1) Delete line with START
2) Delete line with END
3) Remove ABC|
4) Delete duplicate records

The following command works fine which deletes line with START and END
sed -e /^START/d -e /^END/d Filename.txt

How do I incorporate task 3 and 4?
NOTE: The file will have more than 500,000 thousand rows.

Thanks in advance for suggestion,
Prashant

Code:

nawk -F'|'  'NF==3{print $2,$3}' patel

tostay2003 · #3 (**permalink**) 04-07-2009

Quote:

Originally Posted by zenith

Code:

nawk -F'|'  'NF==3{print $2,$3}' patel

This code is not doing uniq + removes pipe delimiter

use this

Code:

grep "|" test | sed 's/^ABC|//g' | sort -u

pinnacle · #4 (**permalink**) 04-07-2009

Quote:

Originally Posted by tostay2003

This code is not doing uniq + removes pipe delimiter

use this

Code:

grep "|" test | sed 's/^ABC|//g' | sort -u

Tostay2003:
What if the data doesnt start with ABC your logic fails

use this
nawk -F'|' 'NF==3{print $2,$3}' patel | sort -u

tostay2003 · #5 (**permalink**) 04-07-2009

Quote:

Originally Posted by zenith

Tostay2003:
What if the data doesnt start with ABC your logic fails

use this
nawk -F'|' 'NF==3{print $2,$3}' patel | sort -u

I assumed from the authors description that the first field remains common in the file

Quote:

Use this if you didnt mean that the first field woudl be same.

Code:

grep "|" test | cut -d'|' -f2,3 | sort -u

or with slight amendment to code written by zenith i.e. by adding OFS

vgersh99 · #6 (**permalink**) 04-07-2009

Code:

nawk -F'|' 'NF==3 && !a[$2,$3]++ {print $2,$3}' patel

ppat7046 · #7 (**permalink**) 04-08-2009

Thank you all for your reply.

Code:

nawk -F'|' 'NF==3 && !a[$2,$3]++ {print $2,$3}' patel

I had made small change in print statment because it was not printing | symbol.

Code:

nawk -F'|' 'NF==3 && !a[$2,$3]++ {print $2,"|",$3}' patel

However, it prints the space after and before | symbol.

Thanks,
Prashant

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
SED: how to remove newline after pattern?	nico.ben	Shell Programming and Scripting	1	01-02-2009 10:59 PM
sed/awk help to match list of patterns and remove from org file	rajan_san	Shell Programming and Scripting	1	12-07-2008 02:56 AM
Delete lines between two patterns without deleting the second pattern	Ilja	Shell Programming and Scripting	1	11-14-2008 09:53 AM
comment/delete a particular pattern starting from second line of the matching pattern	imas	Shell Programming and Scripting	4	10-13-2008 02:37 AM
sed remove everything up to the pattern	katrvu	Shell Programming and Scripting	4	04-08-2008 09:35 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread: