How can i delete the duplicates based on one column of a line Post: 302340627

Sponsored Content

Top Forums Shell Programming and Scripting How can i delete the duplicates based on one column of a line Post 302340627 by rdhanek on Tuesday 4th of August 2009 05:20:34 AM

08-04-2009

Registered User

How can i delete the duplicates based on one column of a line

I have my data something like this

Code:

(08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb
(08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa
(08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts
(08/03/2009 22:57:42.425)(:) Ravi vvvvvvvvvvvvvvvvvvsssssssss bsbbbbs
(08/03/2009 22:57:42.426)(:) John bgbhhhhhhhhhhhhhhhhh dddddddddddddd
(08/03/2009 22:57:42.427)(:) king hhhhhhhhhhhhhssssss rr

Here i need to take the 3rd column as the key foir finding the duplicate rows. I need the output to have the rows with only one king,one john and so on...

Output expected :

Code:

(08/03/2009 22:57:42.414)(:) king aaaaaaaaaaaaaaaa bbbbbbbbbbbbbbbbbbbbbb
(08/03/2009 22:57:42.416)(:) John cccccccccccc cccccvssssssssss baaaaa
(08/03/2009 22:57:42.417)(:) Michael ddddddd tststststtststts
(08/03/2009 22:57:42.425)(:) Ravi vvvvvvvvvvvvvvvvvvsssssssss bsbbbbs

can some expert help me with this? this will be very helpful for my script.

rdhanek

View Public Profile for rdhanek

Find all posts by rdhanek

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

delete a line based on first character of the line

Hi, I need to delete all lines in a file which starts with "|" character. Can some one assist me? Thanks

2. Shell Programming and Scripting

how to delete duplicate rows based on last column

hii i have a huge amt of data stored in a file.Here in this file i need to remove duplicates rows in such a way that the last column has different data & i must check for greatest among last colmn data & print the largest data along with other entries but just one of other duplicate entries is...

3. UNIX for Dummies Questions & Answers

Remove duplicates based on a column in fixed width file

Hi, How to output the duplicate record to another file. We say the record is duplicate based on a column whose position is from 2 and its length is 11 characters. The file is a fixed width file. ex of Record: DTYU12333567opert tjhi kkklTRG9012 The data in bold is the key on which...

4. Shell Programming and Scripting

Delete Duplicates on the basis of two column values.

Hi All, i need ti delete two duplicate processss which are running on the same device type (column 1) and port ID (column 2). here is the sample data p1sc1m1 15517 11325 0 01:00:24 ? 0:00 scagntclsx25octtcp 2967 in3v mvmp01 0 8000 N S 969 750@751@752@ p1sc1m1 15519 11325 0 01:00:24 ? ...

5. Shell Programming and Scripting

Delete lines based on line number

I have a file with ~200K lines, I need to delete 4K lines in it. There is no range. I do have the line numbers of the lines which I want to be deleted. I did tried using > cat del.lines sed '510d;12d;219d;......;3999d' file > source del.lines Word too long. I even tried...

6. Shell Programming and Scripting

need to remove duplicates based on key in first column and pattern in last column

Given a file such as this I need to remove the duplicates. 00060011 PAUL BOWSTEIN ad_waq3_921_20100826_010517.txt 00060011 PAUL BOWSTEIN ad_waq3_921_20100827_010528.txt 0624-01 RUT CORPORATION ad_sade3_10_20100827_010528.txt 0624-01 RUT CORPORATION ...

7. Shell Programming and Scripting

remove duplicates based on single column

Hello, I am new to shell scripting. I have a huge file with multiple columns for example: I have 5 columns below. HWUSI-EAS000_29:1:105 + chr5 76654650 AATTGGAA HHHHG HWUSI-EAS000_29:1:106 + chr5 76654650 AATTGGAA B@HYL HWUSI-EAS000_29:1:108 + ...

8. Shell Programming and Scripting

delete from line and remove duplicates

My Input.....file1 ABCDE4435 Connected to 107.71.136.122 (SubNetwork=ONRM_RootMo_R SubNetwork=XYVLTN29CRBR99 MeContext=ABCDE4435 ManagedElement=1) ABCDE4478 Connected to 166.208.30.57 (SubNetwork=ONRM_RootMo_R SubNetwork=KLFMTN29CR0R04 MeContext=ABCDE4478 ManagedElement=1) ABCDE4478...

9. Shell Programming and Scripting

Search for duplicates and delete but remain the first one based on a specific pattern

Hi all, I have been trying to delete duplicates based on a certain pattern but failed to make it works. There are more than 1 pattern which are duplicated but i just want to remove 1 pattern only and remain the rest. I cannot use awk '!x++' inputfile.txt or sed '/pattern/d' or use uniq and sort...

LEARN ABOUT DEBIAN

bitten-slave

BITTEN-SLAVE(1) 						   User Commands						   BITTEN-SLAVE(1)

NAME

       bitten-slave - Performs a build and sends result to the Bitten master.

SYNOPSIS

       bitten-slave [--version] [-h] [--help] [--name=name] [-f file] [--config=file] [-u username] [--user=username] [-p password]
		    [--password=password] [-d dir] [--work-dir=dir] [--build-dir=build_dir] [-k] [--keep-files] [-s] [--single] [-n] [--dry-run]
		    [-i seconds] [--interval=seconds] [-l filename] [--log=filename] [-v] [--verbose] [-q] [--quiet] [--dump-reports] [URL] [URLs]

DESCRIPTION

       bitten-slave performs a continuous integration build and reports the result back to the bitten-master. Run bitten-slave --help for details.

COPYRIGHT

       Copyright 2009 W. Martin Borgert. Redistribution and use in source and binary forms, with or without modification, are permitted under the
       terms of the BSD License.

AUTHOR

       W. Martin Borgert
	   Author.

COPYRIGHT

       Copyright (C) 2009 W. Martin Borgert

bitten-slave							    2009-10-16							   BITTEN-SLAVE(1)