How to find dupicated records?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to find dupicated records?
# 1  
Old 09-04-2015
How to find dupicated records?

Hi gurus,

I have a file which contains duplicated records. the records is comma delimited with more than 30 columns. I need to find the records which have exact value for each column.

thanks in advance.
# 2  
Old 09-04-2015
man uniq:
Quote:
-d, --repeated
only print duplicate lines, one for each group
# 3  
Old 09-04-2015
man uniq
Quote:
Note: 'uniq' does not detect repeated lines unless they are adjacent. You may want to sort the input first
Try,
Code:
sort ken6503.file | uniq -d

or
Code:
perl -ne '$seen{$_}++ and print' ken6503.file

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Find highest records in table

Dear All, I have table files with different measurements for the same sensors. The first column indicate the sensor (7 different sensors and 16 measurements in the example below). I would like to find the best measurement for each sensor. The best measurement is determined by the higher value of... (10 Replies)
Discussion started by: GDC
10 Replies

2. Shell Programming and Scripting

Find records using epoch time.

How do i find the record which has been edit the last 10 minutes? from a.txt which has last field is epoch time updated. 10/17/2012 1:47 PM||||||In Use|chicken||1350005487 10/17/2012 2:53 PM||||||Available|chicken||13500000 10/17/2012 3:20 PM||||||In Use|cat||1351000000 10/17/2012 3:22... (2 Replies)
Discussion started by: sabercats
2 Replies

3. Shell Programming and Scripting

how to find these records.

Hi, Unix Gurus, I need find records in a file which first 5 character is empty. eg. 12344567 233 467 345 435 456 2334455555 4t54 eee 233445555 444 aaa eerer eree ereei want to find 345 435 456 eerer eree eree:wall: Thanks in advance (2 Replies)
Discussion started by: ken002
2 Replies

4. UNIX for Dummies Questions & Answers

Find records with matching patterns

Hi, I need to find records with a search string from a file. Search strings are provided in a file. For eg. search_String.txt file is like below chicago mexico newark sanhose and the file from where the records need to be fetched is given below src_file:... (1 Reply)
Discussion started by: sbhuvana20
1 Replies

5. Shell Programming and Scripting

Find records between two files which are not exists in one another in one

Hello all, Would like to know how to find records between two files which are not exists in one another in one. For example: I've two files "fileA" and "fileB" and want to find record from "fileB" which does not exists in "fileA". fileA -------- ABCD DEFG GHIJ KLMN NOPQ RSTU VUWX... (5 Replies)
Discussion started by: nvkuriseti
5 Replies

6. UNIX for Advanced & Expert Users

Find out records in between 50 to 60

my file contain 1 to 100 numbers and i want to display 50 to 60 numbers in a seperate file. how many ways this is possible in unix:confused: (3 Replies)
Discussion started by: mailkrissh
3 Replies

7. UNIX for Dummies Questions & Answers

How to find particular records in a file?

Hi all, i have one file contains 20 records 1 2 - - - 20 i want to find 5th records and 8th records and 14th plz tell me which command use? Thanks to advance (4 Replies)
Discussion started by: venkatreddy
4 Replies

8. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies

9. Shell Programming and Scripting

find duplicate records... again

Hi all: Let's suppose I have a file like this (but with many more records). XX ME 342 8688 2006 7 6 3c 60.029 -38.568 2901 0001 74 4 7603 8 969.8 958.4 3.6320 34.8630 985.5 973.9 3.6130 34.8600 998.7 986.9 3.6070 34.8610 1003.6 991.7 ... (4 Replies)
Discussion started by: rleal
4 Replies

10. UNIX for Advanced & Expert Users

find and group records in a file

Hi, I have this file which has 3 columns, District , stores and unit. What I want is all rows belonging to one district to be created separately under each district, the districts may vary every day , the source file may have 3 districts today and may have 160 tomorrow, so what I need is a... (20 Replies)
Discussion started by: thumsup9
20 Replies
Login or Register to Ask a Question