Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Delete multiple lines containting a variable string using SED.

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Delete multiple lines containting a variable string using SED.
# 1  
Old 04-27-2007
Question Delete multiple lines containting a variable string using SED.
Good morning,
Novice scripter in Unix here, and I've run into and sed task I can't quite wrap my head around. I'm pulling my hair out fast enough as it is and thought I would go to the knowledge bank.

I have a sorted file that I'm trying to trim down by deleting any line whose first few characters are repeats of a previous line.
i.e.

1 ABCD
1 CDEF
1 EFGH
2 ACDE
2 GLKGI
2 KLIGH
.
.
.
10 ABSD
10 OIHIHN
10 OHOIN
.
.
.
XX LIHIN
XX OIHNM
XX OHINK

I need to delete any line for which the line header (first three charachters are a repeat.) So for the previous lines it would keep the first line that begins with "1 " the first line that begins with "2 " etc. So the end result of the would be

1 ABCD
2 ACDE
.
10 ABSD
.
XX LIHIN

The first three characters are always whole numbers, no more than 2 digits, and followed by a space, but the maximum number changes (it could anywhere between 11 and 40)

I suppose the other option would be to print out the first line that contains "1 ', the first line that contains "2 " etc. and drop them into a new file.
I'm more familiar with SED, but using AWK or something else would be fine too.

Thanks in advance!
# 2  
Old 04-27-2007
Code:
awk ' !arr[$1]++' oldfile > newfile

This finds the first occurrence of the first field and prints the line. Assuming I got what you want.
# 3  
Old 04-27-2007
Worked like a dream!
It's much appreciated!!
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Delete multiple lines between blank lines containing two patterns

Hi all, I'm looking for a way (sed or awk) to delete multiple lines between blank lines containing two patterns ex: user: alpha parameter_1 = 15 parameter_2 = 1 parameter_3 = 0 user: alpha parameter_1 = 15 parameter_2 = 1 parameter_3 = 0 user: alpha parameter_1 = 16... (3 Replies)
Discussion started by: ce9888
3 Replies

2. Shell Programming and Scripting

Sed find exact string and delete line with variable

All, I am trying to read in a variable and search a file then delete based on that string, but i want to match exact word. This works but it matches all, i don't want to match anthing that contains the string, just the exact string. sed -i "/$feedname/d" file I tried sed... (1 Reply)
Discussion started by: markdjones82
1 Replies

3. Shell Programming and Scripting

Sed/awk to delete single lines that aren't touching other lines

Hello, I'm trying to figure out how to use sed or awk to delete single lines in a file. By single, I mean lines that are not touching any other lines (just one line with white space above and below). Example: one two three four five six seven eight I want it to look like: (6 Replies)
Discussion started by: slimjbe
6 Replies

4. Shell Programming and Scripting

search and replace, when found, delete multiple lines, add new set of lines?

hey guys, I tried searching but most 'search and replace' questions are related to one liners. Say I have a file to be replaced that has the following: $ cat testing.txt TESTING AAA BBB CCC DDD EEE FFF GGG HHH ENDTESTING This is the input file: (3 Replies)
Discussion started by: DeuceLee
3 Replies

5. Shell Programming and Scripting

Grep a string from input file and delete next three lines including the line contains string in xml

Hi, 1_strings file contains $ cat 1_strings /home/$USER/Src /home/Valid /home/Review$ cat myxml <projected value="some string" path="/home/$USER/Src"> <input 1/> <estimate value/> <somestring/> </projected> <few more lines > <projected value="some string" path="/home/$USER/check">... (4 Replies)
Discussion started by: greet_sed
4 Replies

6. Shell Programming and Scripting

replace (sed?) a string in file with multiple lines (string) from variable

Can someone tell me how I can do this? e.g: a=$(echo -e wert trewt ertert ertert ertert erttert erterte rterter tertertert ert) How do i replace the STRING with $a? I try this: sed -i 's/STRING/'"$a"'/g' filename.ext but this don' t work (2 Replies)
Discussion started by: jforce
2 Replies

7. Shell Programming and Scripting

sed to conditionally delete multiple lines

Hello I'd like to remove any line in an output file that is preceded by one or more warning messages (each warning is on a separate line). Example : WARNING: Estimation of lower confidence limit of \rho failed; setting it to 0. 822 28447 ... (4 Replies)
Discussion started by: jossojjos
4 Replies

8. Shell Programming and Scripting

replace (sed?) a single line/string in file with multiple lines (string) from another file??

Can someone tell me how I can do this? e.g: Say file1.txt contains: today is monday the 22 of NOVEMBER 2010 and file2.txt contains: the 11th month of How do i replace the word NOVEMBER with (5 Replies)
Discussion started by: tuathan
5 Replies

9. Shell Programming and Scripting

using sed command to delete a string spanning multiple lines

file1 contains the following data sssssssssss firstline secondline pppppppppp ssssssssss Using sed comamnd i am trying to delete firtsline secondline. so, output should be sssssssssss pppppppppp ssssssssss I tried in the following the way, but it is not working. sed ... (9 Replies)
Discussion started by: radha.kalivar
9 Replies

10. Shell Programming and Scripting

Delete multiple lines w/ sed

Hi all, I am trying to figure out the syntx to delete multiple lines w/ sed. I know the following syntax will delete lines 1 THROUGH 5 from filex: sed 1,5d filex But I wan to delete lines 1 AND 5 (keeping lines 2,3, and 4). Does anyone know how to do this in a single sed statement? ... (2 Replies)
Discussion started by: bookoo
2 Replies
Login or Register to Ask a Question