Help removing lines with duplicated columns


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help removing lines with duplicated columns
# 1  
Old 05-16-2008
Help removing lines with duplicated columns

Hi Guys...

Please Could you help me with the following ?

aaaa bbbb cccc sdsd
aaaa bbbb cccc qwer

as you can see, the 2 lines are matched in three fields...
how can I delete this pupicate ? I mean to delete the second one if 3 fields were duplicated ?

Thanks
# 2  
Old 05-16-2008
i) Use useful topics!
ii)
Code:
awk -F" " '! something[$3]++' inputfile

# 3  
Old 05-16-2008
Hi again

Could you explain this command for me .... Smilie
# 4  
Old 05-16-2008
Hi again

and what do u mean by " Use useful topics "...

Regards
# 5  
Old 05-16-2008
You've named this thread - your question - simply "help". It would be much better to use something like "removing duplicate lines".

And well, I see you've finally used the board search https://www.unix.com/shell-programmin...#post302196106 .

Let's put it together:
i) you've used a meaningless topic
ii) you've posted your problem three times
iii) you've already found a thread in which you problem is answered and explained

Do you really believe this will motivate me to explain this to you? I don't.
# 6  
Old 05-16-2008
Hi again

Dear fabtagon.....

I posted this topic in 3 different places to make sure that my problem can be handled by someone who can deal well enough with Unix.. and as I expected, someone came, like you, and gave a solution that makes no sense at all... thats why I had to post it in different places.. GOT IT !!

Please if you have a stright answer... be my guest, otherwise... go and practice som unix commands.....
Cappich ???

Warm regards Bozo
# 7  
Old 05-16-2008
Your help again.

I went back and checked the problem stated here https://www.unix.com/shell-programmin...#post302196106 .
, believe me it's different....

I have 3 matched fields out of 4, NOT one out of 4.
Here is another example.

SSSSS DDDDDDD 10:10:00 15:22:22
XXXXX AAAAAAA 00:00:11 00:02:11
XXXXX AAAAAAA 00:00:11 06:02:10
EEEEEE VVVVVVV 04:12:00 01:10:02
EEEEEE VVVVVVV 04:12:00 05:12:00
SSSSS DDDDDDD 10:10:00 13:23:21
EEEEEE FFFFFFFF 20:20:20 24:00:00

I want the output to be like

XXXXX AAAAAAA 00:00:11 06:02:10
EEEEEE VVVVVVV 04:12:00 05:12:00
SSSSS DDDDDDD 10:10:00 13:23:21
EEEEEE FFFFFFFF 20:20:20 24:00:00

If 2 lines are matched by 3 fields, I want to delete the first.

Thanks...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing carriage returns from multiple lines in multiple files of different number of columns

Hello Gurus, I have a multiple pipe separated files which have records going over multiple Lines. End of line separator is \n and records going over multiple lines have <CR> as separator. below is example from one file. 1|ABC DEF|100|10 2|PQ RS T|200|20 3| UVWXYZ|300|30 4| GHIJKL|400|40... (7 Replies)
Discussion started by: dJHa
7 Replies

2. Shell Programming and Scripting

Removing duplicated first field rows

Hello, I am trying to eliminate rows where the first field is duplicated, leaving the row where the last field is "NET". Data file: 345234|22.34|LST 546543|55.33|LST 793929|98.23|LST 793929|64.69|NET 149593|49.22|LST Desired output: 345234|22.34|LST 546543|55.33|LST... (2 Replies)
Discussion started by: palex
2 Replies

3. Shell Programming and Scripting

How to remove duplicated lines?

Hi, if i have a file like this: Query=1 a a b c c c d Query=2 b b b c c e . . . (7 Replies)
Discussion started by: the_simpsons
7 Replies

4. UNIX for Dummies Questions & Answers

Find duplicated values in two columns out of three

hi! could u help in the following? I have the data (long list!) that looks like (three coumns white space separated): rs3094315 0.0665173 742429 rs12562034 0.0738998 758311 rs3934834 0.396449 995669 rs9442372 0.402693 1008567 rs3737728 0.406271 1011278 rs6687776 0.435429 1020428 rs9651273... (4 Replies)
Discussion started by: kush
4 Replies

5. UNIX for Dummies Questions & Answers

Removing duplicated lines??

Hi Guys.. I have a problem for some reason my database has copied everything 4 times. My Database looks like this: >BAC233456 rhjieaheiohjteo tjtjrj6jkk6k6 j54ju54jh54jh >ANI124365 afrhtjykulilil htrjykuk rtkjryky ukrykyrk >BAC233456 rhjieaheiohjteo tjtjrj6jkk6k6 j54ju54jh54jh... (6 Replies)
Discussion started by: Iifa
6 Replies

6. Shell Programming and Scripting

finding duplicates in columns and removing lines

I am trying to figure out how to scan a file like so: 1 ralphs office","555-555-5555","ralph@mail.com","www.ralph.com 2 margies office","555-555-5555","ralph@mail.com","www.ralph.com 3 kims office","555-555-5555","kims@mail.com","www.ralph.com 4 tims... (17 Replies)
Discussion started by: totus
17 Replies

7. Shell Programming and Scripting

using sed to get rid of duplicated columns...

I can not figure out this one, so I turn to unix.com for help, I have a file, in which there are some lines containing continuously duplicate columns, like the following adb abc abc asd adfj 123 123 123 345 234 444 444 444 444 444 23 and the output I want is adb abc asd adfj 123 345... (5 Replies)
Discussion started by: fedora
5 Replies

8. Shell Programming and Scripting

remove duplicated columns

hi all, i have a file contain multicolumns, this file is sorted by col2 and col3. i want to remove the duplicated columns if the col2 and col3 are the same in another line. example fileA AA BB CC DD CC XX CC DD BB CC ZZ FF DD FF HH HH the output is AA BB CC DD BB CC ZZ FF... (6 Replies)
Discussion started by: kamel.seg
6 Replies

9. UNIX for Dummies Questions & Answers

Removing lines that are (same in content) based on columns

I have a file which looks like AA BB CC DD EE FF GG HH KK AA BB GG HH KK FF CC DD EE AA BB CC DD EE UU VV XX ZZ AA BB VV XX ZZ UU CC DD EE .... I want the script to give me only one line based on duplicate contents: AA BB CC DD EE FF GG HH KK AA BB CC DD EE UU VV XX ZZ (7 Replies)
Discussion started by: adsforall
7 Replies

10. Shell Programming and Scripting

remove duplicated lines without sort

Hi Just wondering whether or not I can remove duplicated lines without sort For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal. Normally, I would do who | cut -d" " -f1 |... (6 Replies)
Discussion started by: lalelle
6 Replies
Login or Register to Ask a Question