The UNIX and Linux Forums  


Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
.
google unix.com



Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How to remove duplicate records with out sort svenkatareddy SUN Solaris 2 02-28-2008 08:38 AM
Duplicate records from oracle to text file. shilendrajadon UNIX for Advanced & Expert Users 1 01-10-2008 11:21 AM
Remove all instances of duplicate records from the file vukkusila Shell Programming and Scripting 3 12-12-2007 07:50 AM
Records Duplicate ganesh123 Shell Programming and Scripting 9 02-22-2007 08:47 AM
How to extract duplicate records with associated header record run_eim UNIX for Dummies Questions & Answers 17 01-16-2007 11:46 PM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Bulgarian Greek Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 02-28-2008
svenkatareddy svenkatareddy is offline
Registered User
  
 

Join Date: Nov 2007
Posts: 22
How to remove duplicate records with out sort

Can any one give me command How to delete duplicate records with out sort.

Suppose if the records like below:

345,bcd,789
123,abc,456
234,abc,456
712,bcd,789

out tput should be
345,bcd,789
123,abc,456

Key for the records is 2nd and 3rd fields.fields are seperated by colon(,).
  #2 (permalink)  
Old 02-28-2008
Klashxx's Avatar
Klashxx Klashxx is offline Forum Advisor  
HP-UX/Linux/Oracle
  
 

Join Date: Feb 2006
Location: Almerķa, Spain
Posts: 393
Try:


Code:
awk '!($2$3 in a){a[$2$3]++;print}' FS=',' file

  #3 (permalink)  
Old 02-28-2008
krishmaths krishmaths is offline
Registered User
  
 

Join Date: Sep 2006
Location: Mysore, India
Posts: 191
Post How about this code?

for i in `cat filename | cut -d "," -f2 | uniq`
do
sed -n '/'${i}'/{p;q;}' filename
done
  #4 (permalink)  
Old 02-28-2008
radoulov's Avatar
radoulov radoulov is offline Forum Staff  
addict
  
 

Join Date: Jan 2007
Location: Варна, България / Milano, Italia
Posts: 2,915
Or:


Code:
awk '!x[$2,$3]++' FS="," file

Use nawk or /usr/xpg4/bin/awk on Solaris.
  #5 (permalink)  
Old 02-29-2008
girish.batra girish.batra is offline
Registered User
  
 

Join Date: Jan 2008
Posts: 74
you can use 'uniq' command also to remove the duplicate records which is ver easy to use. Check out the man pages of this command
  #6 (permalink)  
Old 02-29-2008
radoulov's Avatar
radoulov radoulov is offline Forum Staff  
addict
  
 

Join Date: Jan 2007
Location: Варна, България / Milano, Italia
Posts: 2,915
Quote:
Originally Posted by girish.batra View Post
you can use 'uniq' command also to remove the duplicate records which is ver easy to use. Check out the man pages of this command
You need a sorted input for uniq ....
  #7 (permalink)  
Old 06-05-2008
kenneth.mcbride kenneth.mcbride is offline
Registered User
  
 

Join Date: Jun 2008
Location: Redmond, WA
Posts: 14
awk '!x[$2,$3]++' FS="," file

This has been a true grit work horse one liner.

I have use it extensively.

Does anyone know if it can be instructed to work within a range of values?
I suspect it wouldn't be a one liner.

For example:

My file contents are all numbers (a mixture of intergers and floating point), where field one is a unique point number, field two is an X or Easting coordinate, field 3 is a Y or Northing coordinate, and field four is an elevation:

1,2.1,3.1,1.1
2,2.2,3.2,2.2
3,2.3,3.3,3.3
4,3.4,3.4,4.4
5,3.5,3.5,5.5
6,3.6,3.6,6.6
7,4.7,4.7,7.1
8,4.8,4.8,8.8
9,4.9,4.9,9.9

I would like to process the file via fields two and three and have the result be:

1,2.1,3.1,1.1
3,2.3,3.3,3.3
4,3.4,3.4,4.4
6,3.6,3.6,6.6
7,4.7,4.7,7.1
9,4.9,4.9,9.9

So I think what I am asking is, that field 2 and 3 be considered duplicates if they are in the range "$2-0.1 to $2+0.1" and "$3-0.1 to $3+0.1".
That's the best way I have of decribing it.

Thanks in advance,
Kenny.
Closed Thread

Bookmarks

Tags
solaris, unix commands

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On




All times are GMT -4. The time now is 02:27 PM.


Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0