"the cat is fat"
"I like eggs"
"fish live in water"
"the moon is made of cheese"
"fish have houses in the water"
"the cake is a lie"
"I like my eggs scrambled"
What I would like to do is remove the duplicate lines from file1.txt, the file with the numbers, but at the same time remove the equivalent line from file2.txt.
In other words: If line 5 from file1.txt is removed because it is a duplicate, then line 5 from file2.txt should also be removed.
I know that
Code:
awk '!x[$0]++' file1.txt > file1.txt.new
will work for removing duplicates, but it does not provide coherence with file2.txt.
thanks for the help in advance
Last edited by Scott; 08-09-2011 at 04:48 PM..
Reason: Added code tags to file[12].txt
It's not clear, exactly, what output should be given, so here goes!
Code:
$ paste -d\| file1.txt file2.txt | sort -uk1,1 | awk -F\| '{print $2}'
"the cat is fat"
"I like eggs"
"fish live in water"
"the moon is made of cheese"
"the cake is a lie"
It's not clear, exactly, what output should be given, so here goes!
Code:
$ paste -d\| file1.txt file2.txt | sort -uk1,1 | awk -F\| '{print $2}'
"the cat is fat"
"I like eggs"
"fish live in water"
"the moon is made of cheese"
"the cake is a lie"
Nice solution, but you missed an option "-t\|" in your sort.
I agree that it's a nice solution. However, if the first file's content is as advertised (comma-delimited numbers), there's really no need to specify an alternate delimiter; the default \t suffices.
Hi,
I have tried to remove dublicate lines based on first column with pipe delimiter . but i ma not able to get some uniqu lines
Command : sort -t'|' -nuk1 file.txt
Input :
38376KZ|09/25/15|1.057
38376KZ|09/25/15|1.057
02006YB|09/25/15|0.859
12593PS|09/25/15|2.803... (2 Replies)
Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted!
1. The problem statement, all variables and given/known data:
You will write a script that will remove all HTML tags from an HTML document and remove any consecutive... (3 Replies)
Hello,
I am in need of removing duplicate lines from within a file per section.
File:
ABC1 012345 header
ABC2 7890-000
ABC3 012345 Header Table
ABC4
ABC5 593.0000 587.4800
ABC5 593.5000 587.6580 <= dup need to remove
ABC5 593.5000 ... (5 Replies)
Hi,
I need to concatenate three files in to one destination file.In this if some duplicate data occurs it should be deleted.
eg:
file1:
-----
data1 value1
data2 value2
data3 value3
file2:
-----
data1 value1
data4 value4
data5 value5
file3:
-----
data1 value1
data4 value4 (3 Replies)
Hi,
I am trying to remove duplicate lines from a file. For example the contents of example.txt is:
this is a test
2342
this is a test
34343
this is a test
43434
and i want to remove the "this is a test" lines only and end up with the numbers in the file, that is, end up with:
2342... (4 Replies)
08-09-2011
