Sponsored Content
Top Forums Shell Programming and Scripting compare fields in a file with duplicate records Post 302281357 by rleal on Wednesday 28th of January 2009 05:50:57 PM
Old 01-28-2009
compare fields in a file with duplicate records

Hi:

I've been searching the net but didnt find a clue. I have a file in which, for some records, some fields coincide. I want to compare one (or more) of the dissimilar fields and retain the one record that fulfills a certain condition. For example, on this file:

Code:
99  TR   1991  5  06   60.000  -19.995  277
64  TR   1991  5  06   60.000  -19.995  275
60  TA   1990  7  17   60.000  -19.998  300
89  TA   1990  7  17   60.000  -19.998  277
46  CU   1992  8  29   61.020  -16.880  009
35  CU   1992  8  29   61.020  -16.880  003

retain the record with greater value on the last field for each duplicate:

Code:
99  TR   1991  5  06   60.000  -19.995  277
60  TA   1990  7  17   60.000  -19.998  300
46  CU   1992  8  29   61.020  -16.880  009

while sending the unwanted data to another file:

Code:
64  TR   1991  5  06   60.000  -19.995  275
89  TA   1990  7  17   60.000  -19.998  277
35  CU   1992  8  29   61.020  -16.880  003

Thanks, in advance.

r.-
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete Duplicate records from a tilde delimited file

Hi All, I want to delete duplicate records from a tilde delimited file. Criteria is considering the first 2 fields, the combination of which has to be unique, below is a sample of records in the input file 1620000010338~2446694087~0~20061130220000~A00BCC1CT... (5 Replies)
Discussion started by: irshadm
5 Replies

2. Shell Programming and Scripting

Remove all instances of duplicate records from the file

Hi experts, I am new to scripting. I have a requirement as below. File1: A|123|NAME1 A|123|NAME2 B|123|NAME3 File2: C|123|NAME4 C|123|NAME5 D|123|NAME6 1) I have 2 merge both the files. 2) need to do a sort ( key fields are first and second field) 3) remove all the instances... (3 Replies)
Discussion started by: vukkusila
3 Replies

3. UNIX for Advanced & Expert Users

Duplicate records from oracle to text file.

Hi, I want to fetch duplicate records from an external table to a text file. Pls suggest me. Thanks (1 Reply)
Discussion started by: shilendrajadon
1 Replies

4. Shell Programming and Scripting

How to find Duplicate Records in a text file

Hi all pls help me by providing soln for my problem I'm having a text file which contains duplicate records . Example: abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 tas 3420 3562 ... (1 Reply)
Discussion started by: G.Aavudai
1 Replies

5. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies

6. Shell Programming and Scripting

Find Duplicate records in first Column in File

Hi, Need to find a duplicate records on the first column, ANU4501710430989 0000000W20389390 ANU4501710430989 0000000W67065483 ANU4501130050520 0000000W80838713 ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies

7. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

8. Shell Programming and Scripting

Remove somewhat Duplicate records from a flat file

I have a flat file that contains records similar to the following two lines; 1984/11/08 7 700000 123456789 2 1984/11/08 1941/05/19 7 700000 123456789 2 The 123456789 2 represents an account number, this is how I identify the duplicate record. The ### signs represent... (4 Replies)
Discussion started by: jolney
4 Replies

9. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies

10. UNIX for Beginners Questions & Answers

Discarding records with duplicate fields

Hi, My input looks like this (tab-delimited): grp1 name2 firstname M 55 item1 item1.0 grp1 name2 firstname F 55 item1 item1.0 grp2 name1 firstname M 55 item1 item1.0 grp2 name2 firstname M 55 item1 item1.0 Using awk, I am trying to discard the records with common fields 2, 4, 5, 6, 7... (4 Replies)
Discussion started by: beca123456
4 Replies
ldns-compare-zones(1)					      General Commands Manual					     ldns-compare-zones(1)

NAME
ldns-compare-zones - read and compare two zonefiles and print differences SYNOPSIS
ldns-compare-zones [-c] [-i] [-d] [-z] [-s] ZONEFILE1 ZONEFILE2 DESCRIPTION
ldns-compare-zones reads two DNS zone files and prints number of differences. Output is formated to: +NUM_INS -NUM_DEL ~NUM_CHG The major comparison is based on the owner name. If an owner name is present in zonefile 1, but not in zonefile 2, the resource records with this owner name are considered deleted, and counted as NUM_DEL. If an owner name is present in zonefile 2, but not in zonefile 1, the resource records with this owner name are considered inserted, and counted as NUM_INS. If an owner name is present in both, but there is a difference in the amount or content of the records, these are considered changed, and counted as NUM_CHG. OPTIONS
-c Print resource records whose owner names are in both zone files, but with different resource records. (a.k.a. changed) -i Print resource records whose owner names are present only in ZONEFILE2 (a.k.a. inserted) -d Print resource records whose owner names are present only in ZONEFILE1 (a.k.a. deleted) -a Print all changes. Specifying this option is the same as specifying -c -i amd -d. -z Suppress zone sorting; this option is not recommended; it can cause records to be incorrectly marked as changed, depending of the nature of the changes. -s Do not exclude the SOA record from the comparison. The SOA record may then show up as changed due to a new serial number. Off by default since you may be interested to know if (other zone apex elements) have changed. -h Show usage and exit -v Show the version and exit AUTHOR
Written by Ondej Sury <ondrej@sury.org> for CZ.NIC, z.s.p.o. (czech domain registry) REPORTING BUGS
Report bugs to <ondrej@sury.org>. COPYRIGHT
Copyright (C) 2005 CZ.NIC, z.s.p.o.. This is free software. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 17 Oct 2007 ldns-compare-zones(1)
All times are GMT -4. The time now is 08:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy