removing line and duplicate line

Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers removing line and duplicate line
# 1  
Old 01-29-2007
removing line and duplicate line


I have 3 lines in a text file that is similar to this (as a result of a diff between 2 files):

< DATA[0].EVENT[0].EVENT_ID.s = "3661208"
< DATA[0].EVENT[0].EVENT_ID.s = "3661208"

I am trying to get it down to just this:

DATA[0].EVENT[0].EVENT_ID.s = "3661208"

How can I do this? Please note the number "35,36d34" and "3661208" is always different because I get this file from a background process that always changes.

thanks guys
# 2  
Old 01-29-2007
Do you care which, if any process number is preserved?
# 3  
Old 01-29-2007
I will always need the line AND the number in the quotes, i.e.

DATA[0].EVENT[0].EVENT_ID.s = "3661208"

the number here is always different. But i need it along with the rest of the line
# 4  
Old 01-29-2007
Could it be as simple as
sort -u
# 5  
Old 01-30-2007
i tried sort -u but it only removed one of the lines. I ended up with this:

< DATA[0].EVENT[0].EVENT_ID.s = "3661208"

but i am trying to just get this:

DATA[0].EVENT[0].EVENT_ID.s = "3661208"

# 6  
Old 01-30-2007
diff file1 file2 | sort -u | sed -n "/^[<>]/s/^[<>] *//p"

# 7  
Old 01-30-2007
thanks anbu23 you saved my life! Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Get an output of lines in pattern 1st line then 10th line then 11th line then 20th line and so on.

Input file: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 (6 Replies)
Discussion started by: Sagar Singh
6 Replies

2. Shell Programming and Scripting

Honey, I broke awk! (duplicate line removal in 30M line 3.7GB csv file)

I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code: awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies

3. Shell Programming and Scripting

Duplicate third column to every line

Dear All, I have file input like this: INP901 5173 4114 INP902 5227 INP903 5284 INP904 5346 INP905 5400 INP906 5456 INP907 5511 INP908 5572 INP909 5622 INP910 5678 INP911 5739 INP912 5796 INP913 5845 INP914 5910 INP915 5965 (2 Replies)
Discussion started by: attila
2 Replies

4. UNIX for Dummies Questions & Answers

Duplicate a line n times

I want a command equivalent to sed 'p;p' file My input file contains 1 2 3 If i want to input file like this 1 1 1 2 2 2 3 3 3 I use (2 Replies)
Discussion started by: nsuresh316
2 Replies

5. Shell Programming and Scripting

how to read the contents of two files line by line and compare the line by line?

Hi All, I'm trying to figure out which are the trusted-ips and which are not using a script file.. I have a file named 'ip-list.txt' which contains some ip addresses and another file named 'trusted-ip-list.txt' which also contains some ip addresses. I want to read a line from... (4 Replies)
Discussion started by: mjavalkar
4 Replies

6. Shell Programming and Scripting

Removing a line IF the next line contains string

So, I've been working on a project which takes layer 7 metadata from pcap dumps and archives it. However, there is a lot of dataless information that I don't want in my output. I know of ways to produce the output I want from the input file below, but I want a method of doing this, regardless of... (2 Replies)
Discussion started by: eh3civic
2 Replies

7. UNIX for Dummies Questions & Answers

Delete duplicate second line

Hi ALL I need a help I need to retain only the first line of 035 if I have two line before =040 , if only one then need to take that Eg: Input =035 (ABC)12324141241 =035 (XYZPQR)704124 =040 AB$QS$WEWR =035 (ABC)08080880809 =035 (XYZPQR)9809314 =040 ... (4 Replies)
Discussion started by: umapearl
4 Replies

8. Shell Programming and Scripting

Duplicate Line Report per Section

I've been working on a script (/bin/sh) in which I have requested and received help here (in which I am very grateful for!). The client has modified their requirements (a tad), so without messing up the script to much, I come once again for assistance. Here are the file.dat contents: ABC1... (4 Replies)
Discussion started by: petersf
4 Replies

9. Shell Programming and Scripting

remove duplicate words in a line

Hi, Please help! I have a file having duplicate words in some line and I want to remove the duplicate words. The order of the words in the output file doesn't matter. INPUT_FILE pink_kite red_pen ball pink_kite ball yellow_flower white no white no cloud nine_pen pink cloud pink nine_pen... (6 Replies)
Discussion started by: sam_2921
6 Replies

10. UNIX for Dummies Questions & Answers

Remove Duplicate line

Hi, I have a scenario here where I have created a flatfile with the below mentioned information. File as you can see is dispalyed in three columns 1st column is FileNameString 2nd column is Report_Name (this has spaces) 3rd column is Flag Result file needed is, removal of duplicate... (1 Reply)
Discussion started by: Student37
1 Replies
Login or Register to Ask a Question

Featured Tech Videos