Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate lines based on field and sort Post 302608371 by balajesuri on Saturday 17th of March 2012 10:57:40 AM
Old 03-17-2012
Code:
sort -t, -nuk1 sorting.csv

 

10 More Discussions You Might Find Interesting

1. Solaris

How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort. Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (2 Replies)
Discussion started by: svenkatareddy
2 Replies

2. Shell Programming and Scripting

Remove lines, Sorted with Time based columns using AWK & SORT

Hi having a file as follows MediaErr.log 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:12:16 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:22:47 84 Server1 Policy1 Schedule1 master1 05/08/2008 03:41:26 84 Server1 Policy1 ... (1 Reply)
Discussion started by: karthikn7974
1 Replies

3. Shell Programming and Scripting

How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort. Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (19 Replies)
Discussion started by: svenkatareddy
19 Replies

4. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

5. Shell Programming and Scripting

Sort and Remove Duplicate on file

How do we sort and remove duplicate on column 1,2 retaining the record with maximum date (in feild 3) for the file with following format. aaa|1234|2010-12-31 aaa|1234|2010-11-10 bbb|345|2011-01-01 ccc|346|2011-02-01 bbb|345|2011-03-10 aaa|1234|2010-01-01 Required Output ... (5 Replies)
Discussion started by: mabarif16
5 Replies

6. UNIX for Dummies Questions & Answers

remove duplicate lines based on two columns and judging from a third one

hello all, I have an input file with four columns like this with a lot of lines and for example, line 1 and line 5 match because the first 4 characters match and the fourth column matches too. I want to keep the line that has the lowest number in the third column. So I discard line 5.... (5 Replies)
Discussion started by: TheTransporter
5 Replies

7. Shell Programming and Scripting

Remove lines with duplicate first field

Trying to cut down the size of some log files. Now that I write this out it looks more dificult than i thought it would be. Need a bash script or command that goes sequentially through all lines of a file, and does this: if field1 (space separated) is the number 2012 print the entire line. Do... (7 Replies)
Discussion started by: ajp7701
7 Replies

8. Shell Programming and Scripting

Remove duplicate value based on two field $4 and $5

Hi All, i have input file like below... CA009156;20091003;M;AWBKCA72;123;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;321;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;231;;CANADIAN... (2 Replies)
Discussion started by: mohan sharma
2 Replies

9. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

10. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Discussion started by: refrain
8 Replies
Tree::Simple::Visitor::LoadDirectoryTree(3pm)		User Contributed Perl Documentation	     Tree::Simple::Visitor::LoadDirectoryTree(3pm)

NAME
Tree::Simple::Visitor::LoadDirectoryTree - A Visitor for loading the contents of a directory into a Tree::Simple object SYNOPSIS
use Tree::Simple::Visitor::LoadDirectoryTree; # create a Tree::Simple object whose # node is path to a directory my $tree = Tree::Simple->new("./"); # create an instance of our visitor my $visitor = Tree::Simple::Visitor::LoadDirectoryTree->new(); # set the directory sorting style $visitor->setSortStyle($visitor->SORT_FILES_FIRST); # create node filter to filter # out certain files and directories $visitor->setNodeFilter(sub { my ($item) = @_; return 0 if $item =~ /CVS/; return 1; }); # pass the visitor to a Tree::Simple object $tree->accept($visitor); # the tree now mirrors the structure of the directory DESCRIPTION
This visitor can be used to load a directory tree into a Tree::Simple hierarchy. METHODS
new There are no arguments to the constructor the object will be in its default state. You can use the "setNodeFilter" and "setSortStyle" methods to customize its behavior. setNodeFilter ($filter_function) This method accepts a CODE reference as its $filter_function argument and throws an exception if it is not a code reference. This code reference is used to filter the tree nodes as they are created. The function is given the current directory or file being added to the tree, and it is expected to return either true(1) of false(0) to determine if that directory should be traversed or file added to the tree. setSortStyle ($sort_function) This method accepts a CODE reference as its $sort_function argument and throws an exception if it is not a code reference. This function is used to sort the individual levels of the directory tree right before it is added to the tree being built. The function is passed the the current path, followed by the two items being sorted. The reason for passing the path in is so that sorting operations can be performed on the entire path if desired. Two pre-built functions are supplied and described below. SORT_FILES_FIRST This sorting function will sort files before directories, so that files are sorted alphabetically first in the list followed by directories sorted alphabetically. Here is example of how that would look: Tree/ Simple.pm Simple/ Visitor.pm VisitorFactory.pm Visitor/ PathToRoot.pm SORT_DIRS_FIRST This sorting function will sort directories before files, so that directories are sorted alphabetically first in the list followed by files sorted alphabetically. Here is example of how that would look: Tree/ Simple/ Visitor/ PathToRoot.pm Visitor.pm VisitorFactory.pm Simple.pm visit ($tree) This is the method that is used by Tree::Simple's "accept" method. It can also be used on its own, it requires the $tree argument to be a Tree::Simple object (or derived from a Tree::Simple object), and will throw and exception otherwise. The node value of the $tree argument (gotten by calling "getNodeValue") is considered the root directory from which we begin our traversal. We use File::Spec to keep our paths cross-platform, but it is expected that you will feed in a valid path for your OS. If the path either does not exist, or is not a directory, then an exception is thrown. The $tree argument which is passed to "visit" must be a leaf node. This is because this Visitor will create all the sub-nodes for this tree. If the tree is not a leaf, an exception is thrown. We do not require the tree to be a root though, and this Visitor will not affect any nodes above the $tree argument. BUGS
None that I am aware of. Of course, if you find a bug, let me know, and I will be sure to fix it. CODE COVERAGE
See the CODE COVERAGE section in Tree::Simple::VisitorFactory for more inforamtion. SEE ALSO
These Visitor classes are all subclasses of Tree::Simple::Visitor, which can be found in the Tree::Simple module, you should refer to that module for more information. AUTHOR
stevan little, <stevan@iinteractive.com> COPYRIGHT AND LICENSE
Copyright 2004, 2005 by Infinity Interactive, Inc. <http://www.iinteractive.com> This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.10.1 2005-07-14 Tree::Simple::Visitor::LoadDirectoryTree(3pm)
All times are GMT -4. The time now is 12:47 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy