Delete multiple lines starting with a specific pattern


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Delete multiple lines starting with a specific pattern
# 1  
Old 01-26-2012
[SOLVED] Delete multiple lines starting with a specific pattern

Hi, just tried some script, awk, sed for the last 2 hours and now need help.

Let's say I have a huge file of 800,000 lines like this :

Quote:
UPDATE abc.INVENTORY
SET ORD_30DAYS=33,
LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621'
WHERE INVENTORY_ID=4274

UPDATE abc.INVENTORY
SET LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621002'
WHERE INVENTORY_ID=5564
It's a tedious job to look through it, I'd like to remove those useless lines in it as there's a few thousands :

Quote:
UPDATE abc.INVENTORY
SET LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621002'
WHERE INVENTORY_ID=5564
Or to be even more precise :
if line1 = UPDATE*
and line2 = SET LAST_UPD_TIMESTAMP=*
and line3 = WHERE*
then delete those lines

As I said, it's a huge file, the faster, the better. Thanks for any help.

Last edited by Zurd; 01-26-2012 at 05:16 AM.. Reason: Now solved
# 2  
Old 01-26-2012
Code:
sed "/UPDATE\|SET LAST_UPD_TIMESTAMP\|WHERE/d" infile

Use -i option to make the changes inline.

--ahamed
# 3  
Old 01-26-2012
Thanks for answering that fast, however, it's not working as I really want it to be.

Quote:
> cat temp2
UPDATE test.INVENTORY
SET ORD_30DAYS=33,
LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621'
WHERE INVENTORY_ID=4274

UPDATE test.INVENTORY
SET LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621002'
WHERE INVENTORY_ID=5564

z ~/logs
> sed "/UPDATE\|SET LAST_UPD_TIMESTAMP\|WHERE/d" temp2
SET ORD_30DAYS=33,
LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621'

z ~/logs
> sed -i "/UPDATE\|SET LAST_UPD_TIMESTAMP\|WHERE/d" temp2
(nothing displayed after this command)
I don't want it to remove all the lines having UPDATE, SET LAST_UPD_TIMESTAMP and WHERE, only if they are close to each other on 3 lines. Any clue?
# 4  
Old 01-26-2012
Code:
sed "/^UPDATE\|^SET LAST_UPD_TIMESTAMP\|^WHERE/d" infile

--ahamed
# 5  
Old 01-26-2012
Unfortunately it produces the same exact results.

Quote:
> cat temp2
UPDATE test.INVENTORY
SET ORD_30DAYS=33,
LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621'
WHERE INVENTORY_ID=4274

UPDATE test.INVENTORY

SET LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621002'
WHERE INVENTORY_ID=5564

z ~/logs

> sed "/^UPDATE\|^SET LAST_UPD_TIMESTAMP\|^WHERE/d" temp2
SET ORD_30DAYS=33,
LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621'

It should be like this :
Quote:
> cat temp2
UPDATE test.INVENTORY
SET ORD_30DAYS=33,
LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621'
WHERE INVENTORY_ID=4274

UPDATE test.INVENTORY

SET LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621002'
WHERE INVENTORY_ID=5564

z ~/logs

> sed `do something with sed/awk or whatever here`
UPDATE test.INVENTORY
SET ORD_30DAYS=33,
LAST_UPD_TIMESTAMP='2012/jan/18 06:11:11.621'
WHERE INVENTORY_ID=4274
# 6  
Old 01-26-2012
Sorry, I missed that part of the requirement! Try this...

Code:
sed '/UPDATE/{N;/SET LAST_UPD_TIMESTAMP/{N;/WHERE/d}}' infile

--ahamed
This User Gave Thanks to ahamed101 For This Post:
# 7  
Old 01-26-2012
Thanks! sed is awesome, it's executing it so fast also, I really like it.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Grep -v lines starting with pattern 1 and not matching pattern 2

Hi all! Thanks for taking the time to view this! I want to grep out all lines of a file that starts with pattern 1 but also does not match with the second pattern. Example: Drink a soda Eat a banana Eat multiple bananas Drink an apple juice Eat an apple Eat multiple apples I... (8 Replies)
Discussion started by: demmel
8 Replies

2. Shell Programming and Scripting

Extract specific line in an html file starting and ending with specific pattern to a text file

Hi This is my first post and I'm just a beginner. So please be nice to me. I have a couple of html files where a pattern beginning with "http://www.site.com" and ending with "/resource.dat" is present on every 241st line. How do I extract this to a new text file? I have tried sed -n 241,241p... (13 Replies)
Discussion started by: dejavo
13 Replies

3. Shell Programming and Scripting

Delete lines that contain a pattern from specific line to the end.

Gents, I am trying to delete all lines which start with "H" character, but keeping the fist header. Example In the input file I will delete all lines starting from line 8 which contents character "H" to the end of the file. I try sed '8,10000{/^H/d;}' file But as don't know the end... (1 Reply)
Discussion started by: jiam912
1 Replies

4. Shell Programming and Scripting

How to delete lines starting with specific string?

Dear all, I would like to delete even lines starting with "N" together with their respective titles which are actually odd lines. Below is the example of input file. I would like to remove line 8 and 12 together with its title line, i.e., line 7 and 11, respectively.... (2 Replies)
Discussion started by: huiyee1
2 Replies

5. Shell Programming and Scripting

delete lines starting with a pattern

i have a file sample.txt containing i want to delete lines starting with 123 neglecting spaces and tabs. but not lines containing 123. i.e. i want files sample.txt as help me thanxx (4 Replies)
Discussion started by: yashwantkumar
4 Replies

6. Shell Programming and Scripting

how to find entries, NOT starting with specific pattern

Hey,I have a file in following format >1 ABC........ >2 XYZ..... >3 ABC........ >4 MNO....... >5 ABC....... now I would like to find only those entries that doesn't start with ABC (specific pattern)e.g preferred output: >2 XYZ.... >4 MNO....... it will be nice if anybody how... (2 Replies)
Discussion started by: ankitachaurasia
2 Replies

7. Shell Programming and Scripting

how to delete lines from a file which starts with a specific pattern

I need to delete those lines from a file, which starts with 45. How to do it? (3 Replies)
Discussion started by: mady135
3 Replies

8. Shell Programming and Scripting

Concatenate lines between lines starting with a specific pattern

Hi, I have a file such as: --- >contig00001 length=35524 numreads=2944 gACGCCGCGCGCCGCGGCCAGGGCTGGCCCA CAGGCCGCGCGGCGTCGGCTGGCTGAG >contig00002 length=4242 numreads=43423 ATGCCGAAGGTCCGCCTGGGGCTGG CGCCGGGAGCATGTAGCG --- I would like to concatenate the lines not starting with ">"... (9 Replies)
Discussion started by: s052866
9 Replies

9. Shell Programming and Scripting

comment/delete a particular pattern starting from second line of the matching pattern

Hi, I have file 1.txt with following entries as shown: 0152364|134444|10.20.30.40|015236433 0233654|122555|10.20.30.50|023365433 ** ** ** In file 2.txt I have the following entries as shown: 0152364|134444|10.20.30.40|015236433 0233654|122555|10.20.30.50|023365433... (4 Replies)
Discussion started by: imas
4 Replies

10. Shell Programming and Scripting

Delete lines starting with XX or YY or ZZ or ....

Hi There! My final task for today is to delete lines starting with certain numbers for e.g., my text block is and i want to delete all lines starting with 11 or 17 or 21 I know i can use multiple sed commands like sed '/^11,/d' <filename> sed '/^17,/d' <filename> sed '/^21,/d'... (2 Replies)
Discussion started by: orno
2 Replies
Login or Register to Ask a Question