Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Discarding records with duplicate fields Post 303043655 by beca123456 on Monday 3rd of February 2020 06:30:47 AM
Old 02-03-2020
Your code works. Thanks !
But I would like to understand why mine does not. It seems fine to me...
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Records Duplicate

Hi Everyone, I have a flat file of 1000 unique records like following : For eg Andy,Flower,201-987-0000,12/23/01 Andrew,Smith,101-387-3400,11/12/01 Ani,Ross,401-757-8640,10/4/01 Rich,Finny,245-308-0000,2/27/06 Craig,Ford,842-094-8740,1/3/04 . . . . . . Now I want to duplicate... (9 Replies)
Discussion started by: ganesh123
9 Replies

2. Shell Programming and Scripting

compare fields in a file with duplicate records

Hi: I've been searching the net but didnt find a clue. I have a file in which, for some records, some fields coincide. I want to compare one (or more) of the dissimilar fields and retain the one record that fulfills a certain condition. For example, on this file: 99 TR 1991 5 06 ... (1 Reply)
Discussion started by: rleal
1 Replies

3. Shell Programming and Scripting

combine duplicate records

I have a .DAT file like below 23666483030000653-B94030001OLFXXX000000120081227 23797049900000654-E71060001OLFXXX000000220081227 23699281320000655 E71060002OLFXXX000000320081227 22885068900000652 B86860003OLFXXX592123320081227 22885068900000652 B86860003ODL-SP592123420081227... (8 Replies)
Discussion started by: kshuser
8 Replies

4. UNIX for Dummies Questions & Answers

Getting non-duplicate records

Hi, I have a file with these records abc xyz xyz pqr uvw cde cde In my o/p file , I want all the non duplicate rows to be shown. o/p abc pqr uvw Any suggestions how to do this? Thanks for the help. rs (2 Replies)
Discussion started by: rs123
2 Replies

5. UNIX for Dummies Questions & Answers

Need to keep duplicate records

Consider my input is 10 10 20 then, uniq -u will give 20 and uniq -dwill return 10. But i need the output as , 10 10 How we can achieve this? Thanks (4 Replies)
Discussion started by: pandeesh
4 Replies

6. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

7. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies

8. Shell Programming and Scripting

Remove duplicate records

Hi, i am working on a script that would remove records or lines in a flat file. The only difference in the file is the "NOT NULL" word. Please see below example of the input file. INPUT FILE:> CREATE a ( TRIAL_CLIENT NOT NULL VARCHAR2(60), TRIAL_FUND NOT NULL... (3 Replies)
Discussion started by: reignangel2003
3 Replies

9. Shell Programming and Scripting

Duplicate records

Gents, I have a file which contends duplicate records in column 1, but the values in column 2 are different. 3099753489 3 3099753489 5 3101954341 12 3101954341 14 3102153285 3 3102153285 5 3102153297 3 3102153297 5 I will like to get something like this: output desired... (16 Replies)
Discussion started by: jiam912
16 Replies

10. Shell Programming and Scripting

Duplicate records

Gents, Please give a help file --BAD STATUS NOT RESHOOTED-- *** VP 41255/51341 in sw 2973 *** VP 41679/51521 in sw 2973 *** VP 41687/51653 in sw 2973 *** VP 41719/51629 in sw 2976 --BAD COG NOT RESHOOTED-- *** VP 41689/51497 in sw 2974 *** VP 41699/51677 in sw 2974 *** VP... (18 Replies)
Discussion started by: jiam912
18 Replies
TABLIFY(1p)						User Contributed Perl Documentation					       TABLIFY(1p)

NAME
tablify - turn a delimited text file into a text table SYNOPSIS
tablify [options] file Options: -h|--help Show help --no-headers Assume first line is data, not headers --no-pager Do not use $ENV{'PAGER'} even if defined --strip-quotes Strip " or ' around fields -l|--list List the fields in the file (for use with -f) -f|--fields=f1[,f2] Show only fields in comma-separated list; when used in conjunction with "no-headers" the list should be field numbers (starting at 1); otherwise, should be field names -w|where=f<cmp>v Apply the "cmp" Perl operator to restrict output where field "f" matches the value "v"; acceptable operators include ==, eq, >, >=, <=, and =~ -v|--vertical Show records vertically --limit=n Limit to first "n" records --fs=x Use "x" as the field separator (default is tab " ") --rs=x Use "x" as the record separator (default is newline " ") --as-html Create an HTML table instead of plain text DESCRIPTION
This script is essentially a quick way to parse a delimited text file and view it as a nice ASCII table. By selecting only certain fields, employing a where clause to only select records where a field matches some value, and using the limit to only see some of the output, you almost have a mini-database front-end for a simple text file. EXAMPLES
Given a data file like this: name,rank,serial_no,is_living,age George,General,190293,0,64 Dwight,General,908348,0,75 Attila,Hun,,0,56 Tojo,Emporor,,0,87 Tommy,General,998110,1,54 To find the fields you can reference, use the list option: $ tablify --fs ',' -l people.dat +-----------+-----------+ | Field No. | Field | +-----------+-----------+ | 1 | name | | 2 | rank | | 3 | serial_no | | 4 | is_living | | 5 | age | +-----------+-----------+ To extract just the name and serial numbers, use the fields option: $ tablify --fs ',' -f name,serial_no people.dat +--------+-----------+ | name | serial_no | +--------+-----------+ | George | 190293 | | Dwight | 908348 | | Attila | | | Tojo | | | Tommy | 998110 | +--------+-----------+ 5 records returned To extract the first through third fields and the fifth field (where field numbers start at "1" -- tip: use the list option to quickly determine field numbers), use this syntax for fields: $ tablify --fs ',' -f 1-3,5 people.dat +--------+---------+-----------+------+ | name | rank | serial_no | age | +--------+---------+-----------+------+ | George | General | 190293 | 64 | | Dwight | General | 908348 | 75 | | Attila | Hun | | 56 | | Tojo | Emporor | | 87 | | Tommy | General | 998110 | 54 | +--------+---------+-----------+------+ 5 records returned To select only the ones with six serial numbers, use a where clause: $ tablify --fs ',' -w 'serial_no=~/^d{6}$/' people.dat +--------+---------+-----------+-----------+------+ | name | rank | serial_no | is_living | age | +--------+---------+-----------+-----------+------+ | George | General | 190293 | 0 | 64 | | Dwight | General | 908348 | 0 | 75 | | Tommy | General | 998110 | 1 | 54 | +--------+---------+-----------+-----------+------+ 3 records returned To find Dwight's record, you would do this: $ tablify --fs ',' -w 'name eq "Dwight"' people.dat +--------+---------+-----------+-----------+------+ | name | rank | serial_no | is_living | age | +--------+---------+-----------+-----------+------+ | Dwight | General | 908348 | 0 | 75 | +--------+---------+-----------+-----------+------+ 1 record returned To find the name of all the people with a serial number who are living: $ tablify --fs ',' -f name -w 'is_living==1' -w 'serial_no>0' people.dat +-------+ | name | +-------+ | Tommy | +-------+ 1 record returned To filter outside of program and simply format the results, use "-" as the last argument to force reading of STDIN (and probably assume no headers): $ grep General people.dat | tablify --fs ',' -f 1-3 --no-headers - +---------+--------+--------+ | Field1 | Field2 | Field3 | +---------+--------+--------+ | General | 190293 | 0 | | General | 908348 | 0 | | General | 998110 | 1 | +---------+--------+--------+ 3 records returned When dealing with data lacking field names, you can specify "no-headers" and then refer to fields by number (starting at one), e.g.: $ tail -5 people.dat | tablify --fs ',' --no-headers -w '3 eq "General"' - +--------+---------+--------+--------+--------+ | Field1 | Field2 | Field3 | Field4 | Field5 | +--------+---------+--------+--------+--------+ | George | General | 190293 | 0 | 64 | | Dwight | General | 908348 | 0 | 75 | | Tommy | General | 998110 | 1 | 54 | +--------+---------+--------+--------+--------+ 3 records returned If your file has many fields which are hard to see across the screen, consider using the vertical display with "-v" or "--vertical", e.g.: $ tablify --fs ',' -v --limit 1 people.dat ************ Record 1 ************ name: George rank: General serial_no: 190293 is_living: 0 age : 64 1 record returned SEE ALSO
o Text::RecordParser o Text::TabularDisplay o DBD::CSV Although I don't DBD::CSV this module, the idea was much the inspiration for this. I just didn't want to have to install DBI and DBD::CSV to get this kind of functionality. I think my interface is simpler. AUTHOR
Ken Youens-Clark <kclark@cpan.org>. LICENSE AND COPYRIGHT
Copyright (C) 2006-10 Ken Youens-Clark. All rights reserved. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; version 2. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. perl v5.10.1 2010-07-26 TABLIFY(1p)
All times are GMT -4. The time now is 02:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy