Sponsored Content
Top Forums Shell Programming and Scripting Sort and Remove Duplicate on file Post 302521923 by yinyuemi on Thursday 12th of May 2011 06:29:27 PM
Old 05-12-2011
how about this?
Code:
echo 'aaa|1234|2010-12-31
aaa|1234|2010-11-10
bbb|345|2011-01-01
ccc|346|2011-02-01
bbb|345|2011-03-10
aaa|1234|2010-01-01' |sort -t '|' -k1,1 -k3,3r |awk -F"|" '++a[$1"|"$2]==1'
aaa|1234|2010-12-31
bbb|345|2011-03-10
ccc|346|2011-02-01

 

9 More Discussions You Might Find Interesting

1. Solaris

How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort. Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (2 Replies)
Discussion started by: svenkatareddy
2 Replies

2. Shell Programming and Scripting

How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort. Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (19 Replies)
Discussion started by: svenkatareddy
19 Replies

3. Shell Programming and Scripting

Shellscript to sort duplicate files listed in a text file

I have many pdf's scattered across 4 machines. There is 1 location where I have other Pdf's maintained. But the issues it the 4 machines may have duplicate pdf's among themselves, but I want just 1 copy of each so that they can be transfered to that 1 location. What I have thought is: 1) I have... (11 Replies)
Discussion started by: deaddevil
11 Replies

4. Shell Programming and Scripting

remove of duplicate line from a file

I have a file a.txt having content like deepak ram sham deepram sita kumar I Want to delete the first line containing "deep" ... I tried using... grep -i 'deep' a.txt It gives me 2 rows...I want to delete the first one.. + need to know the command to delete the line from... (5 Replies)
Discussion started by: saluja.deepak
5 Replies

5. Shell Programming and Scripting

Remove duplicate lines based on field and sort

I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
Discussion started by: cokedude
8 Replies

6. Shell Programming and Scripting

Remove duplicate chars and sort string [SED]

Hi, INPUT: DCBADD OUTPUT: ABCD The SED script should alphabetically sort the chars in the string and remove the duplicate chars. (5 Replies)
Discussion started by: jds93
5 Replies

7. Shell Programming and Scripting

How to Remove duplicate value from file?

if different branch code is available for same BIC code and one of the branch code is XXX.only one row will be stored and with branch code as XXX .rest of the rows for the BIC code will not be stored. for example if $7 is BIC code and $8 is branch code INPUT file are following... (9 Replies)
Discussion started by: mohan sharma
9 Replies

8. UNIX for Dummies Questions & Answers

Sort and delete partical duplicate file

I want to delete partical duplicate file >gma-miR156d Gm01,PACID=26323927 150.00 -18.28 2 18 17 35 16 75.00% 81.25% >>gma-miR156d Gm01,PACID=26323927 150.00 -18.28 150.00 -18.28 1 21 119 17 I want to order by the second column and delete the... (1 Reply)
Discussion started by: grace_shen
1 Replies

9. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Discussion started by: refrain
8 Replies
XML::Filter::Sort::BufferMgr(3pm)			User Contributed Perl Documentation			 XML::Filter::Sort::BufferMgr(3pm)

NAME
XML::Filter::Sort::BufferMgr - Implementation class used by XML::Filter::Sort DESCRIPTION
The documentation is targetted at developers wishing to extend or replace this class. For user documentation, see XML::Filter::Sort. Two classes are used to implement buffering records and spooling them back out in sorted order as SAX events. One instance of the XML::Filter::Sort::Buffer class is used to buffer each record and one or more instances of the XML::Filter::Sort::BufferMgr class are used to manage the buffers. API METHODS
The API of this module as used by XML::Filter::Sort::Buffer consists of the following sequence of method calls: 1. When the first 'record' in a sequence is encountered, XML::Filter::Sort creates a XML::Filter::Sort::BufferMgr object using the "new()" method. 2. XML::Filter::Sort calls the buffer manager's "new_buffer()" method to get a XML::Filter::Sort::Buffer object and all SAX events are directed to this object until the end of the record is encountered. The following events are supported by the current buffer implementation: start_element() characters() comment() processing_instruction() end_element() 3. When the end of the record is detected, XML::Filter::Sort calls the buffer manager's "close_buffer()" method, which in turn calls the buffer's "close()" method. The "close()" method returns a list of values for the sort keys and the buffer manager uses these to store the buffer for later recall. Subsequent records are handled as per step 2. 4. When the last record has been buffered, XML::Filter::Sort calls the buffer manager's "to_sax()" method. The buffer manager retrieves each of the buffers in sorted order and calls the buffer's "to_sax()" method. Each buffer attempts to match the sort key paths as SAX events are received. Once a value has been found for a given key, that same path match is not attempted against subsequent events. For efficiency, the code to match each key is compiled into a closure. For even more efficiency, this compilation is done once when the XML::Filter::Sort object is created. The "compile_matches()" method in the buffer manager class calls the "compile_matches()" method in the buffer class to achieve this. DATA STRUCTURES
In the current implementation, the XML::Filter::Sort::BufferMgr class simply uses a hash to store the buffer objects. If only one sort key was defined, only a single hash is required. The values in the hash are arrayrefs containing the list of buffers for records with identical keys. If two or more sort keys are defined, the hash values will be XML::Filter::Sort::BufferMgr objects which in turn will contain the buffers. The following illustration may clarify the relationship (BM=buffer manager, B=buffer): BM +----------------+---------------+ | | BM BM +-----+--------+ +-----+----------+ | | | | BM BM BM BM +-----+----+ +----+------+ +----+----+ +------+------+ | | | | | | | | | | | | [B,B,B] [B] [B,B] [B] [B,B] [B,B,B] [B] [B,B] [B] [B,B] [B,B,B] [B,B] This layered storage structure is transparent to the XML::Filter::Sort object which instantiates and interacts with only one buffer manager (the one at the top of the tree). COPYRIGHT
Copyright 2002 Grant McLean <grantm@cpan.org> This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.12.4 2002-06-14 XML::Filter::Sort::BufferMgr(3pm)
All times are GMT -4. The time now is 09:48 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy