Removing all the duplicates

 
Thread Tools Search this Thread
Homework and Emergencies Emergency UNIX and Linux Support Removing all the duplicates
# 8  
Old 08-16-2011
Can you post sample of your real data?
# 9  
Old 08-16-2011
the data is like below:
Code:
12116|  |12116     |C                  |M                 |                         |8913   |189  |111189  |12119249  |8000       |E|029|W Clock| ger                 |0|E 12th Street                      |                                        |  |FL |60       |U |111189      | 

12116|  |12116     |k               |Dsd                   |Y                    |10   |124  |224  |19621192 |850       |E|D007| |SMr                 |0|. J- 12      |                                        |Wrs            |FL |3331       |US |111224      | 

i need to find the duplictaes based on 10th field.

---------- Post updated at 03:06 PM ---------- Previous update was at 02:40 PM ----------

Anything i need to chnage in the below code for that?
Code:
 
awk -F"|" '{a[$10]++;b[$10]=b[$10]?b[$10]"\n"$0:$0}END{for(i in a){if(a[i]==1){print b[i]>"file1"}else{print b[i]>"file2"}}}' input

# 10  
Old 08-16-2011
I've checked that code for following data:
Code:
12116|  |12116     |C                  |M                 |                         |8913   |189  |111189  |12119249  |8000       |E|029|W Clock| ger                 |0|E 12th Street                      |                                        |  |FL |60       |U |111189      | 
22116|  |12116     |C                  |M                 |                         |8913   |189  |111189  |12119249  |8000       |E|029|W Clock| ger                 |0|E 12th Street                      |                                        |  |FL |60       |U |111189      | 
12116|  |12116     |k               |Dsd                   |Y                    |10   |124  |224  |19621192 |850       |E|D007| |SMr                 |0|. J- 12      |                                        |Wrs            |FL |3331       |US |111224      |

And got following result:
Code:
solaris% cat file1
12116|  |12116     |k               |Dsd                   |Y                    |10   |124  |224  |19621192 |850       |E|D007| |SMr                 |0|. J- 12      |                                        |Wrs            |FL |3331       |US |111224      | 
solaris% cat file2
12116|  |12116     |C                  |M                 |                         |8913   |189  |111189  |12119249  |8000       |E|029|W Clock| ger                 |0|E 12th Street                      |                                        |  |FL |60       |U |111189      | 
22116|  |12116     |C                  |M                 |                         |8913   |189  |111189  |12119249  |8000       |E|029|W Clock| ger                 |0|E 12th Street                      |                                        |  |FL |60       |U |111189      |

So it is working as expected for this sample... Can you post sample data that gives incorrect results?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies

2. Shell Programming and Scripting

Removing duplicates except the last occurrence

Hi All, i have a file like below, @DB_FCTS\src\Data\Scripts\Delete_CU_OM_BIL_PRT_STMT_TYP.sql @DB_FCTS\src\Data\Scripts\Delete_CDP_BILL_LBL_MSG.sql @DB_FCTS\src\Data\Scripts\Delete_OM_BIDDR.sql @DB_FCTS\src\Data\Scripts\Insert_CU_OM_LBL_MSG.sql... (11 Replies)
Discussion started by: mechvijays
11 Replies

3. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

4. Shell Programming and Scripting

Help in removing duplicates

I have an input file abc.txt with info like: abcd rateuse inklite robet rateuse abcd I need to remove duplicates from the file (eg: abcd,rateuse) from the file and need to place the contents in same file abc.txt if needed can be placed in another file. can anyone help me in this :( (4 Replies)
Discussion started by: rkrish
4 Replies

5. Shell Programming and Scripting

Removing duplicates

I have a test file with the following 2 columns: Col 1 | Col 2 T1 | 1 <= remove T5 | 1 T4 | 2 T1 | 3 T3 | 3 T4 | 1 <= remove T1 | 2 <= remove T3 ... (7 Replies)
Discussion started by: gctex
7 Replies

6. UNIX for Advanced & Expert Users

removing duplicates.

Hi All In unix ,we have a file ,there we have to remove the duplicates by using one specific column. Can any body tell me the command. ex: file1 id,name 1,ww 2,qwq 2,asas 3,asa 4,asas 4,asas o/p: 1,ww 2,qwq 3,asa (7 Replies)
Discussion started by: raju4u
7 Replies

7. Shell Programming and Scripting

Removing duplicates

Hi, I have a file in the below format., test test (10) to to (25) see see (45) and i need the output in the format of test 10 to 25 see 45 Some one help me? (6 Replies)
Discussion started by: imdadulla
6 Replies

8. Shell Programming and Scripting

removing duplicates

Hi I have a file that are a list of people & their credentials i recieve frequently The issue is that whne I catnet this list that duplicat entries exists & are NOT CONSECUTIVE (i.e. uniq -1 may not weork here ) I'm trying to write a scrip that will remove duplicate entries the script can... (5 Replies)
Discussion started by: stevie_velvet
5 Replies

9. UNIX for Dummies Questions & Answers

removing duplicates and sort -k

Hello experts, I am trying to remove all lines in a csv file where the 2nd columns is a duplicate. I am try to use sort with the key parameter sort -u -k 2,2 File.csv > Output.csv File.csv File Name|Document Name|Document Title|Organization Word Doc 1.doc|Word Document|Sample... (3 Replies)
Discussion started by: orahi001
3 Replies

10. Shell Programming and Scripting

Removing duplicates

Hi, I've been trying to removed duplicates lines with similar columns in a fixed width file and it's not working. I've search the forum but nothing comes close. I have a sample file: 27147140631203RA CCD * 27147140631203RA PPN * 37147140631207RD AAA 47147140631203RD JNA... (12 Replies)
Discussion started by: giannicello
12 Replies
Login or Register to Ask a Question