Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remove duplicate lines based on field and sort
Top Forums Shell Programming and Scripting Remove duplicate lines based on field and sort Post 302608361 by cokedude on Saturday 17th of March 2012 10:21:08 AM
Old 03-17-2012
Remove duplicate lines based on field and sort
I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method you can think of, but awk would be my preferred tool if possible.

Code:
cut -f 1 -d , sorting.csv | sort | uniq

Code:
55,I,like,cookies,2,8,9
44,I,like,cookies,2,8,9
88,I,like,cookies,5,7,8
88,I,like,cookies,2,8,9
99,I,like,cookies,5,7,8
99,I,like,cookies,2,8,9
77,I,like,cookies,5,7,8
77,I,like,cookies,2,8,9
66,I,like,cookies,5,7,8
66,I,like,cookies,2,8,9
55,I,like,cookies,5,7,8
44,I,like,cookies,5,7,8

 

10 More Discussions You Might Find Interesting

1. Solaris

How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort. Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (2 Replies)
Discussion started by: svenkatareddy
2 Replies

2. Shell Programming and Scripting

Remove lines, Sorted with Time based columns using AWK & SORT

Hi having a file as follows MediaErr.log 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:12:16 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:22:47 84 Server1 Policy1 Schedule1 master1 05/08/2008 03:41:26 84 Server1 Policy1 ... (1 Reply)
Discussion started by: karthikn7974
1 Replies

3. Shell Programming and Scripting

How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort. Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). (19 Replies)
Discussion started by: svenkatareddy
19 Replies

4. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

5. Shell Programming and Scripting

Sort and Remove Duplicate on file

How do we sort and remove duplicate on column 1,2 retaining the record with maximum date (in feild 3) for the file with following format. aaa|1234|2010-12-31 aaa|1234|2010-11-10 bbb|345|2011-01-01 ccc|346|2011-02-01 bbb|345|2011-03-10 aaa|1234|2010-01-01 Required Output ... (5 Replies)
Discussion started by: mabarif16
5 Replies

6. UNIX for Dummies Questions & Answers

remove duplicate lines based on two columns and judging from a third one

hello all, I have an input file with four columns like this with a lot of lines and for example, line 1 and line 5 match because the first 4 characters match and the fourth column matches too. I want to keep the line that has the lowest number in the third column. So I discard line 5.... (5 Replies)
Discussion started by: TheTransporter
5 Replies

7. Shell Programming and Scripting

Remove lines with duplicate first field

Trying to cut down the size of some log files. Now that I write this out it looks more dificult than i thought it would be. Need a bash script or command that goes sequentially through all lines of a file, and does this: if field1 (space separated) is the number 2012 print the entire line. Do... (7 Replies)
Discussion started by: ajp7701
7 Replies

8. Shell Programming and Scripting

Remove duplicate value based on two field $4 and $5

Hi All, i have input file like below... CA009156;20091003;M;AWBKCA72;123;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;321;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;231;;CANADIAN... (2 Replies)
Discussion started by: mohan sharma
2 Replies

9. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

10. Shell Programming and Scripting

Remove duplicate lines, sort it and save it as file itself

Hi, all I have a csv file that I would like to remove duplicate lines based on 1st field and sort them by the 1st field. If there are more than 1 line which is same on the 1st field, I want to keep the first line of them and remove the rest. I think I have to use uniq or something, but I still... (8 Replies)
Discussion started by: refrain
8 Replies

LEARN ABOUT CENTOS

kcookiejar4

KCOOKIEJAR4(8)							 KDE User's Manual						    KCOOKIEJAR4(8)

NAME

       kcookiejar4 - KDE HTTP cookie daemon

SYNOPSIS

       kcookiejar4 [--help] [Generic-options] [Qt-options] [KDE-options] [--shutdown] [--remove domain] [--remove-all] [--reload-config]

DESCRIPTION

       kcookiejar4 handles the HTTP cookies providing a D-BUS service to store/retrieve/clean cookies.

GENERIC OPTIONS

       --author
	   Show author information.

       --help
	   Show help about options.

       --help-all
	   Show all options.

       --help-kde
	   Show KDE specific options.

       --help-qt
	   Show Qt specific options.

       --license
	   Show license information.

       -v--version
	   Show version information

APPLICATION OPTIONS

       --shutdown
	   Shut down cookie jar and the D-BUS service.

       --remove domain
	   Removes cookies for domain from the cookie jar.

       --remove-all
	   Removes all the cookies from the cookie jar.

       --reaload-config
	   Reloads the configuration file.

USAGE

       KDE web browser konqueror uses kcookiejar4 for storing and managing cookies using the D-Bus service kcookiejar4 provides.

       When kcookiejar4 is started without parameters, it provides a D-BUS service to handle HTTP cookies.

       When kcookiejar4 is started with some parameters, it does additional tasks to the cookies jar it provides, like removing the cookies from
       one domain.

SEE ALSO

       kdeoptions(7), qtoptions(7)

BUGS

       There are probably tons of bugs. Use bugs.kde.org[1] to report them.

AUTHORS

       Waldo Bastian <bastian@kde.org>
	   Author.

       Dawit Alemayehu <adawit@kde.org>
	   Author.

NOTES

	1. bugs.kde.org
	   http://bugs.kde.org

0.01.01 							    2008-10-14							    KCOOKIEJAR4(8)
All times are GMT -4. The time now is 03:32 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy