Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

Remove duplicate in colomn

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Remove duplicate in colomn
# 1  
Old 05-02-2018
Remove duplicate in colomn
i have a file which have two columns, 2nd column have duplicate values and i want to delete and keep only distinct on basis on 2nd column.

Code:
-bash-4.1$ cat file_re 
7440    713543695
7441    713543695
4603    714457614
4602    714457614
40301   717937765
40281   717937765
33741   721208982
33645   721208982
33646   721208982


i tried below but it is not working:
nawk '!A[$13]++' file_re
# 2  
Old 05-02-2018
Hello mirwasim,

You are using correct logic but wrong field number, try following and let me know if this helps you.
Code:
awk '!a[$2]++'   Input_file

Where $2 denotes 2nd field of the current line in Input_file.

Thanks,
R. Singh
# 3  
Old 05-02-2018
For duplicates, which of the first field values should be retained?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance (10 Replies)
Discussion started by: tinku981
10 Replies

2. Shell Programming and Scripting

Remove duplicate

Hi , I have a pipe seperated file repo.psv where i need to remove duplicates based on the 1st column only. Can anyone help with a Unix script ? Input: 15277105||Common Stick|ESHR||Common Stock|CYRO AB 15277105||Common Stick|ESHR||Common Stock|CYRO AB 16111278||Common Stick|ESHR||Common... (12 Replies)
Discussion started by: samrat dutta
12 Replies

3. Shell Programming and Scripting

How to remove duplicate ID's?

HI I have file contains 1000'f of duplicate id's with (upper and lower first character) as below i/p: a411532A411532a508661A508661c411532C411532 Requirement: But i need to ignore lowercase id's and need only below id's o/p: A411532 A508661 C411532 (9 Replies)
Discussion started by: buzzme
9 Replies

4. Shell Programming and Scripting

How to exclude certain colomn from the file?

Hi I have a file in the following format: 4135 f4135acc: 39798 rmtdb: 0 /t1/data/f4135acc.dta 4135 f4135pdb: 39795 rmtdb: 0 /bb/data/f4135pdb.dta 4135 p4135eng: 0 rmtdb: 0 /bb/bin/p4135eng 4135 r4135eng: 14142 rmtdb: 0 ... (6 Replies)
Discussion started by: aoussenko
6 Replies

5. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC (6 Replies)
Discussion started by: ccp
6 Replies

6. Shell Programming and Scripting

switch line to colomn

Hi It's possible to switch the line to colon from a file using Perl or AWK? for example my file have somthing like this: 10 11 12 13 14 20 21 22 23 24 30 31 32 33 34 I want to have a file with the line switched like : 10 20 30 11 21 31 12 22 32 13 23 33 14 24 34 Thanks a lot (4 Replies)
Discussion started by: rauchy
4 Replies

7. Shell Programming and Scripting

changing colomn to the line

I have a file which has only one colomn of numbers,ex: 122 173 292 400 979 2152 2339 2376 2387 2446 2450 What ksh / unix command should I use to create a file in which those numbers will be in one line,like this 122 173 292 400 979 .... etc Thanks a lot for help (9 Replies)
Discussion started by: aoussenko
9 Replies

8. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006... (2 Replies)
Discussion started by: sabercats
2 Replies

9. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006... (3 Replies)
Discussion started by: sabercats
3 Replies

10. Shell Programming and Scripting

remove duplicate

i have a text its contain many record, but its written in one line, i want to remove from that line the duplicate record, not record have fixed width ex: width = 4 inputfile test.txt =abc cdf abc abc cdf fgh fgh abc abc i want the outputfile =abc cdf fgh only those records can any one help... (4 Replies)
Discussion started by: kazanoova2
4 Replies
Login or Register to Ask a Question