12-05-2007
Delete Duplicate records from a tilde delimited file
Hi All,
I want to delete duplicate records from a tilde delimited file. Criteria is considering the first 2 fields, the combination of which has to be unique, below is a sample of records in the input file
1620000010338~2446694087~0~20061130220000~A00BCC1CT
1620000126196~2446694087~0~20061130220000~A00BCC1CT
1620000126196~2446694087~1~20061430220000~A00BCC1CT
1620000127475~2446694087~0~20061130220000~A00BCC1CT
1620000134743~2446694087~0~20061130220000~A00BCC1CT
1620000134743~2446694087~0~20060930220000~A00BCC1CT
here we notice that record 3 and 6 are duplicate records. let me how to do this in shell script.
Thanks in Advance
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
i have a file in unix in which datas are like this
07 01 abc data entry Z3 data entry ASSISTANT Z3 39
08 01 POD peadiatrist Z4 POD PeDIATRY Z4 67
01 operator specialist 00 operator UNSPECIFIED A0 00
... (12 Replies)
Discussion started by: trichyselva
12 Replies
2. UNIX for Advanced & Expert Users
Hi,
I want to fetch duplicate records from an external table to a text file.
Pls suggest me.
Thanks (1 Reply)
Discussion started by: shilendrajadon
1 Replies
3. Shell Programming and Scripting
Hi all
pls help me by providing soln for my problem
I'm having a text file which contains duplicate records .
Example:
abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452
tas 3420 3562 ... (1 Reply)
Discussion started by: G.Aavudai
1 Replies
4. Shell Programming and Scripting
For example suppose I have a file which contains data as:
$cat data
800,2
100,9
700,3
100,9
200,8
100,3
Now I want the output as
200,8
700,3
800,2
Key is first three characters, I don't want any reords which are having duplicate keys.
Like sort +0.0 -0.3 data can we use... (9 Replies)
Discussion started by: sumitc
9 Replies
5. Shell Programming and Scripting
Hi:
I've been searching the net but didnt find a clue. I have a file in which, for some records, some fields coincide. I want to compare one (or more) of the dissimilar fields and retain the one record that fulfills a certain condition. For example, on this file:
99 TR 1991 5 06 ... (1 Reply)
Discussion started by: rleal
1 Replies
6. Shell Programming and Scripting
Dear All,
I have one file which looks like :
account1:passwd1
account2:passwd2
account3:passwd3
account1:passwd4
account5:passwd5
account6:passwd6
you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies
7. UNIX for Dummies Questions & Answers
the data in my file is has no delimiters. it looks like this:
H52082320024740010PH333200612290000930 0.0020080131
D5208232002474000120070306200703060580T1502 TT 1.00
H52082320029180003PH333200702150001 30 100.0020080205
D5208232002918000120070726200707260580T1502 ... (3 Replies)
Discussion started by: jclanc8
3 Replies
8. Shell Programming and Scripting
I have a flat file that contains records similar to the following two lines;
1984/11/08 7 700000 123456789 2
1984/11/08 1941/05/19 7 700000 123456789 2
The 123456789 2 represents an account number, this is how I identify the duplicate record.
The ### signs represent... (4 Replies)
Discussion started by: jolney
4 Replies
9. Shell Programming and Scripting
I have 2 files
"File 1" is delimited by ";" and "File 2" is delimited by "|".
File 1 below (3 record shown):
Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones
Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull
Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies
10. UNIX for Advanced & Expert Users
actually i post about this issue before but many folkz miss-understood with my quesion,
We are checking for the delimited file records validation
Delimited file will have data like this:
Aaaa|sdfhxfgh|sdgjhxfgjh|sdgjsdg|sgdjsg|
Aaaa|sdfhxfgh|sdgjhxfgjh|sdgjsdg|sgdjsg|... (3 Replies)
Discussion started by: Seshendranath
3 Replies
LEARN ABOUT DEBIAN
recset
RECSET(1) User Commands RECSET(1)
NAME
recset - manipulate fields in a recfile
SYNOPSIS
recset [OPTION]... [FILE]...
DESCRIPTION
Alter or delete fields in records.
--no-external
don't use external descriptors.
--force
alter the records even if violating record restrictions.
--help print a help message and exit.
--version
show version and exit.
Record selection options:
-i, --case-insensitive
make strings case-insensitive in selection expressions.
-t, --type=TYPE
operate on records of the specified type only.
-e, --expression=EXPR
selection expression.
-q, --quick=STR
select records with fields containing a string.
-n, --number=NUM,...
select specific records by position, with ranges.
-R, --random=NUM
select a given number of random records.
Field selection options:
-f, --fields=FIELDS
comma-separated list of field names with optional subscripts.
Actions:
-s, --set=VALUE
change the value of the selected fields.
-a, --add=VALUE
add the selected fields with the given value.
-S, --set-add=VALUE
change the value of the selected fields. If they don't exist then add a new field with that value.
-r, --rename=NAME
rename the selected fields to a given name. If an entire record set is selected then the field is renamed in the record descriptor
as well.
-d, --delete
delete the selected fields.
-c, --comment
comment out the selected fields.
AUTHOR
Written by Jose E. Marchesi.
REPORTING BUGS
Report bugs to: bug-recutils@gnu.org
GNU recutils home page: <http://www.gnu.org/software/recutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
COPYRIGHT
Copyright (C) 2010, 2011, 2012 Jose E. Marchesi. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO
The full documentation for recset is maintained as a Texinfo manual. If the info and recset programs are properly installed at your site,
the command
info recset
should give you access to the complete manual.
recset 1.4.93 January 2012 RECSET(1)