Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remove duplicate lines based on field and sort
Top Forums Shell Programming and Scripting Remove duplicate lines based on field and sort Post 302608361 by cokedude on Saturday 17th of March 2012 10:21:08 AM
Old 03-17-2012
Remove duplicate lines based on field and sort
I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method you can think of, but awk would be my preferred tool if possible.

Code:
cut -f 1 -d , sorting.csv | sort | uniq

Code:
55,I,like,cookies,2,8,9
44,I,like,cookies,2,8,9
88,I,like,cookies,5,7,8
88,I,like,cookies,2,8,9
99,I,like,cookies,5,7,8
99,I,like,cookies,2,8,9
77,I,like,cookies,5,7,8
77,I,like,cookies,2,8,9
66,I,like,cookies,5,7,8
66,I,like,cookies,2,8,9
55,I,like,cookies,5,7,8
44,I,like,cookies,5,7,8

 

10 More Discussions You Might Find Interesting

1. Solaris

How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort. Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (2 Replies)
Discussion started by: svenkatareddy
2 Replies

2. Shell Programming and Scripting

Remove lines, Sorted with Time based columns using AWK & SORT

Hi having a file as follows MediaErr.log 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:12:16 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:22:47 84 Server1 Policy1 Schedule1 master1 05/08/2008 03:41:26 84 Server1 Policy1 ... (1 Reply)
Discussion started by: karthikn7974
1 Replies

3. Shell Programming and Scripting

How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort. Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (19 Replies)
Discussion started by: svenkatareddy
19 Replies

4. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

5. Shell Programming and Scripting

Sort and Remove Duplicate on file

How do we sort and remove duplicate on column 1,2 retaining the record with maximum date (in feild 3) for the file with following format. aaa|1234|2010-12-31 aaa|1234|2010-11-10 bbb|345|2011-01-01 ccc|346|2011-02-01 bbb|345|2011-03-10 aaa|1234|2010-01-01 Required Output ... (5 Replies)
Discussion started by: mabarif16
5 Replies

6. UNIX for Dummies Questions & Answers

remove duplicate lines based on two columns and judging from a third one

hello all, I have an input file with four columns like this with a lot of lines and for example, line 1 and line 5 match because the first 4 characters match and the fourth column matches too. I want to keep the line that has the lowest number in the third column. So I discard line 5.... (5 Replies)
Discussion started by: TheTransporter
5 Replies

7. Shell Programming and Scripting

Remove lines with duplicate first field

Trying to cut down the size of some log files. Now that I write this out it looks more dificult than i thought it would be. Need a bash script or command that goes sequentially through all lines of a file, and does this: if field1 (space separated) is the number 2012 print the entire line. Do... (7 Replies)
Discussion started by: ajp7701
7 Replies

8. Shell Programming and Scripting

Remove duplicate value based on two field $4 and $5

Hi All, i have input file like below... CA009156;20091003;M;AWBKCA72;123;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;321;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;231;;CANADIAN... (2 Replies)
Discussion started by: mohan sharma
2 Replies

9. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

10. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Discussion started by: refrain
8 Replies

LEARN ABOUT DEBIAN

dancer::cookies

Dancer::Cookies(3pm)					User Contributed Perl Documentation				      Dancer::Cookies(3pm)

NAME

       Dancer::Cookies - a singleton storage for all cookies

SYNOPSIS

	   use Dancer::Cookies;

	   my $cookies = Dancer::Cookies->cookies;

	   foreach my $name ( keys %{$cookies} ) {
	       my $cookie = $cookies->{$name};
	       my $value  = $cookie->value;
	       print "$name => $value
";
	   }

	   cookie lang => "fr-FR"; #set a cookie and return its value
	   cookie lang => "fr-FR", expires => "2 hours";
	   cookie "lang"	   #return a cookie value

DESCRIPTION

       Dancer::Cookies keeps all the cookies defined by the application and makes them accessible and provides a few helper functions for cookie
       handling with regards to the stored cookies.

METHODS

   init
       This method is called when "->new()" is called. It creates a storage of cookies parsed from the environment using "parse_cookies_from_env"
       described below.

   cookies
       Returns a hash reference of all cookies, all objects of Dancer::Cookie type.

       The key is the cookie name, the value is the Dancer::Cookie object.

   cookie
       "cookie" method is useful to query or set cookies easily.

	   cookie lang => "fr-FR";		# set a cookie and return its value
	   cookie lang => "fr-FR", expires => "2 hours";   # extra cookie info
	   cookie "lang"			# return a cookie value

   parse_cookie_from_env
       Fetches all the cookies from the environment, parses them and creates a hashref of all cookies.

       It also returns all the hashref it created.

AUTHOR

       Alexis Sukrieh

LICENSE AND COPYRIGHT

       Copyright 2009-2010 Alexis Sukrieh.

       This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as
       published by the Free Software Foundation; or the Artistic License.

       See http://dev.perl.org/licenses/ for more information.

perl v5.14.2							    2012-01-28						      Dancer::Cookies(3pm)
All times are GMT -4. The time now is 11:55 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy