04-03-2008
Show input
Can you post what the unsorted input data looks like.
10 More Discussions You Might Find Interesting
1. Solaris
Can any one give me command How to delete duplicate records with out sort.
Suppose if the records like below:
345,bcd,789
123,abc,456
234,abc,456
712,bcd,789
out tput should be
345,bcd,789
123,abc,456
Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (2 Replies)
Discussion started by: svenkatareddy
2 Replies
2. Shell Programming and Scripting
Can any one give me command How to delete duplicate records with out sort.
Suppose if the records like below:
345,bcd,789
123,abc,456
234,abc,456
712,bcd,789
out tput should be
345,bcd,789
123,abc,456
Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (19 Replies)
Discussion started by: svenkatareddy
19 Replies
3. Shell Programming and Scripting
I have a file which consists of 1000 entries. Out of 1000 entries i have 500 Duplicate Entires. I want to remove the first Duplicate Entry (i,e entire Line) in the File.
The example of the File is shown below:
8244100010143276|MARISOL CARO||MORALES|HSD768|CARR 430 KM 1.7 ... (1 Reply)
Discussion started by: ravi_rn
1 Replies
4. Shell Programming and Scripting
I have a pipe delimited file. Key is field 2, date is field 5 (as example, my real file is more complicated of course, but the KEY and DATE are accurate)
There can be duplicate rows for a key with different dates.
I need to keep only rows with latest date in this case.
Example data: ... (4 Replies)
Discussion started by: LisaS
4 Replies
5. Shell Programming and Scripting
How do we sort and remove duplicate on column 1,2 retaining the record with maximum date (in feild 3) for the file with following format.
aaa|1234|2010-12-31
aaa|1234|2010-11-10
bbb|345|2011-01-01
ccc|346|2011-02-01
bbb|345|2011-03-10
aaa|1234|2010-01-01
Required Output
... (5 Replies)
Discussion started by: mabarif16
5 Replies
6. Shell Programming and Scripting
I'm looking to remove duplicate rows from a CSV file with a twist.
The first row is a header.
There are 31 columns. I want to remove duplicates when the first 29 rows are identical ignoring row 30 and 31 BUT the duplicate that is kept should have the shortest total character length in rows 30... (6 Replies)
Discussion started by: Michael Stora
6 Replies
7. UNIX for Advanced & Expert Users
I have an input file of 5GB which contains duplicate records and have to remove duplicate records by retaing first instance of that record .
Based on 5 fields the duplicates has to be removed .
Kindly request to help me in writing a Unix Script.
Thanks
Asim (11 Replies)
Discussion started by: duplicate
11 Replies
8. UNIX for Dummies Questions & Answers
I want to delete partical duplicate file
>gma-miR156d Gm01,PACID=26323927 150.00 -18.28 2 18 17 35 16 75.00% 81.25%
>>gma-miR156d Gm01,PACID=26323927 150.00 -18.28 150.00 -18.28 1 21 119 17
I want to order by the second column and delete the... (1 Reply)
Discussion started by: grace_shen
1 Replies
9. Shell Programming and Scripting
I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code:
awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies
10. UNIX for Beginners Questions & Answers
I am using DB2 v9 and trying to get country values in comma seperated format using below query
SELECT distinct LISTAGG(COUNTRIES, ',') WITHIN GROUP(ORDER BY EMPLOYEE)
FROM LOCATION ;
Output Achieved
MEXICO,UNITED STATES,INDIA,JAPAN,UNITED KINGDOM,MEXICO,UNITED STATES
The table... (4 Replies)
Discussion started by: Perlbaby
4 Replies
aclsort(3C) aclsort(3C)
NAME
aclsort() - sort an Access Control List (JFS File Systems only)
SYNOPSIS
DESCRIPTION
The routine sorts JFS Access Control List (ACL) entries into the correct order to be accepted by the acl(2) system call.
points to a buffer containing ACL entries; if non-zero, indicates that the permissions should be recalculated; and specifies the number of
ACL entries in the buffer.
sorts the contents of the ACL buffer as follows:
Entries will be in order and
Entries of type and will be sorted in increasing order by numeric ID.
The call will succeed if all of the following are true:
There is exactly one entry each of type and
There is at most one entry each of type and
Entries of type or may not contain duplicate entries. A duplicate entry is one of the same type containing the same numeric id,
irrespective of permission bits.
If the argument is zero and there are no entries of type and no entries of type the permissions of the and entries must be the same.
If there are no entries of type and no entries of type and the entry is specified, then the entry must also be specified, and the
permissions of the and entries must be the same.
RETURN VALUE
Upon successful completion, the return value is 0. If there are duplicate entries, the return value is the position of the first duplicate
entry. If there is more than one entry of type or they are treated as duplicate entries, and the return value is the position of the first
duplicate entry. For all other errors, the return value is -1.
NOTICES
The buffer is sorted by type and ID before checking for any failures. Therefore the buffer is always sorted, even if there is a failure.
The position of a duplicate entry returned on failure is not the byte offset of the duplicate entry from its base; rather it refers to the
entry number of the duplicate entry within the sorted buffer.
Checks will be performed in order of entry type. If there are multiple failures, the failure returned will be the first encountered, for
example, if the ACL buffer contains a duplicate entry and does not contain an entry, the return value will be the first duplicate entry.
ACLs do not have to be sorted with prior to passing them to acl(2).
DEPENDENCIES
is supported only on JFS file systems on the standard HP-UX operating system.
AUTHOR
was developed by AT&T.
SEE ALSO
acl(2), aclv(5).
aclsort(3C)