Sponsored Content
Top Forums Shell Programming and Scripting Delete Duplicate records from a tilde delimited file Post 302149249 by irshadm on Wednesday 5th of December 2007 11:27:56 AM
Old 12-05-2007
Delete Duplicate records from a tilde delimited file

Hi All,

I want to delete duplicate records from a tilde delimited file. Criteria is considering the first 2 fields, the combination of which has to be unique, below is a sample of records in the input file

1620000010338~2446694087~0~20061130220000~A00BCC1CT
1620000126196~2446694087~0~20061130220000~A00BCC1CT
1620000126196~2446694087~1~20061430220000~A00BCC1CT
1620000127475~2446694087~0~20061130220000~A00BCC1CT
1620000134743~2446694087~0~20061130220000~A00BCC1CT
1620000134743~2446694087~0~20060930220000~A00BCC1CT

here we notice that record 3 and 6 are duplicate records. let me how to do this in shell script.

Thanks in Advance
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to extract a tilde delimited file in unix

i have a file in unix in which datas are like this 07 01 abc data entry Z3 data entry ASSISTANT Z3 39 08 01 POD peadiatrist Z4 POD PeDIATRY Z4 67 01 operator specialist 00 operator UNSPECIFIED A0 00 ... (12 Replies)
Discussion started by: trichyselva
12 Replies

2. UNIX for Advanced & Expert Users

Duplicate records from oracle to text file.

Hi, I want to fetch duplicate records from an external table to a text file. Pls suggest me. Thanks (1 Reply)
Discussion started by: shilendrajadon
1 Replies

3. Shell Programming and Scripting

How to find Duplicate Records in a text file

Hi all pls help me by providing soln for my problem I'm having a text file which contains duplicate records . Example: abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 tas 3420 3562 ... (1 Reply)
Discussion started by: G.Aavudai
1 Replies

4. Shell Programming and Scripting

How to delete duplicate records based on key

For example suppose I have a file which contains data as: $cat data 800,2 100,9 700,3 100,9 200,8 100,3 Now I want the output as 200,8 700,3 800,2 Key is first three characters, I don't want any reords which are having duplicate keys. Like sort +0.0 -0.3 data can we use... (9 Replies)
Discussion started by: sumitc
9 Replies

5. Shell Programming and Scripting

compare fields in a file with duplicate records

Hi: I've been searching the net but didnt find a clue. I have a file in which, for some records, some fields coincide. I want to compare one (or more) of the dissimilar fields and retain the one record that fulfills a certain condition. For example, on this file: 99 TR 1991 5 06 ... (1 Reply)
Discussion started by: rleal
1 Replies

6. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies

7. UNIX for Dummies Questions & Answers

Extract records by column value - file non-delimited

the data in my file is has no delimiters. it looks like this: H52082320024740010PH333200612290000930 0.0020080131 D5208232002474000120070306200703060580T1502 TT 1.00 H52082320029180003PH333200702150001 30 100.0020080205 D5208232002918000120070726200707260580T1502 ... (3 Replies)
Discussion started by: jclanc8
3 Replies

8. Shell Programming and Scripting

Remove somewhat Duplicate records from a flat file

I have a flat file that contains records similar to the following two lines; 1984/11/08 7 700000 123456789 2 1984/11/08 1941/05/19 7 700000 123456789 2 The 123456789 2 represents an account number, this is how I identify the duplicate record. The ### signs represent... (4 Replies)
Discussion started by: jolney
4 Replies

9. Shell Programming and Scripting

Deleting duplicate records from file 1 if records from file 2 match

I have 2 files "File 1" is delimited by ";" and "File 2" is delimited by "|". File 1 below (3 record shown): Doc1;03/01/2012;New York;6 Main Street;Mr. Smith 1;Mr. Jones Doc2;03/01/2012;Syracuse;876 Broadway;John Davis;Barbara Lull Doc3;03/01/2012;Buffalo;779 Old Windy Road;Charles... (2 Replies)
Discussion started by: vestport
2 Replies

10. UNIX for Advanced & Expert Users

Wanted best way to validate delimited file records

actually i post about this issue before but many folkz miss-understood with my quesion, We are checking for the delimited file records validation Delimited file will have data like this: Aaaa|sdfhxfgh|sdgjhxfgjh|sdgjsdg|sgdjsg| Aaaa|sdfhxfgh|sdgjhxfgjh|sdgjsdg|sgdjsg|... (3 Replies)
Discussion started by: Seshendranath
3 Replies
PMUNPACKEVENTRECORDS(3) 				     Library Functions Manual					   PMUNPACKEVENTRECORDS(3)

NAME
pmUnpackEventRecords - unpack event records C SYNOPSIS
#include <pcp/pmapi.h> int pmUnpackEventRecords(pmValueSet *vsp, int idx, pmResult ***rap); cc ... -lpcp DESCRIPTION
Event records are encoded as a packed array of records within a pmResult using a container metric with a value of type PM_TYPE_EVENT. pmUnpackEventRecords may be used to unpack event records from a metric value identified by vsp and idx. If the metric has a singular value, idx should be 0, else the ordinal instance value identified by idx will be unpacked, i.e. vsp->vlist[idx]. The unpacked records are turned into pmResult structures, one per event record and one metric per event parameter, and rap is returned as a pointer to an array (NULL pointer terminated) of pointers to the pmResult structures. Some control information from the packed event records is unpacked into additional ``anonymous'' metrics as follows: 1. If the event record has a non-zero flags value, then the corresponding pmResult will have the flags value encoded with the additional metric event.flags that is inserted ahead of all other event parameters. 2. If the event record flag is set to PM_EVENT_FLAG_MISSED, then the corresponding pmResult will have one metric event.missed with a value that equals the number of event records ``missed'' because either the PMDA could not keep up, or the PMAPI client did not collect the event records fast enough. pmUnpackEventRecords returns the number of pmResult structures as the return value, which is >= 0 for success. rset and the associated pmResult structures may be freed using the convenience function pmFreeEventResult(3). RETURN VALUE
The following errors are possible: PM_ERR_CONV The values associated with vsp are not encoded using the format PM_VAL_DPTR or PM_VAL_SPTR, or the flags at the head of the event record has an unexpected value. PM_ERR_INST The value associated with vsp is not singular as expected. PM_ERR_TYPE vsp is not a value of type PM_TYPE_EVENT. PM_ERR_TOOSMALL The value identified by vbp is not legal because the value length is less than the minimum size, or the number of event records encoded in the (value header) pmEventArray structure is negative, or the number of missed event records in the pmEventArray array is negative. PM_ERR_TOOBIG Either vsp indicates more than one value is present (all the event records are expected to be packed in a single metric value), or when unpacking the event records, the processing continues past the end of the enclosing value. Indicates corruption of the packed event record. PM_ERR_TYPE Event parameters must have one of the arithmetic types, else PM_TYPE_AGGREGATE, PM_TYPE_STRING or PM_TYPE_AGGREGATE_STATIC. other values < 0 refer to pmErrStr(3). SEE ALSO
PMAPI(3) and pmFreeEventResult(3). Performance Co-Pilot PCP PMUNPACKEVENTRECORDS(3)
All times are GMT -4. The time now is 02:10 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy