![]() |
|
|
google unix.com
|
|||||||
| Forums | Register | Forum Rules | Links | Albums | FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
| Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here. |
More UNIX and Linux Forum Topics You Might Find Helpful
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| How to remove duplicate records with out sort | svenkatareddy | SUN Solaris | 2 | 02-28-2008 08:38 AM |
| Duplicate records from oracle to text file. | shilendrajadon | UNIX for Advanced & Expert Users | 1 | 01-10-2008 11:21 AM |
| Remove all instances of duplicate records from the file | vukkusila | Shell Programming and Scripting | 3 | 12-12-2007 07:50 AM |
| Records Duplicate | ganesh123 | Shell Programming and Scripting | 9 | 02-22-2007 08:47 AM |
| How to extract duplicate records with associated header record | run_eim | UNIX for Dummies Questions & Answers | 17 | 01-16-2007 11:46 PM |
![]() |
|
|
LinkBack | Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
|
|
||||
|
How to remove duplicate records with out sort
Can any one give me command How to delete duplicate records with out sort.
Suppose if the records like below: 345,bcd,789 123,abc,456 234,abc,456 712,bcd,789 out tput should be 345,bcd,789 123,abc,456 Key for the records is 2nd and 3rd fields.fields are seperated by colon(,). |
|
||||
|
awk '!x[$2,$3]++' FS="," file
This has been a true grit work horse one liner. I have use it extensively. Does anyone know if it can be instructed to work within a range of values? I suspect it wouldn't be a one liner. ![]() For example: My file contents are all numbers (a mixture of intergers and floating point), where field one is a unique point number, field two is an X or Easting coordinate, field 3 is a Y or Northing coordinate, and field four is an elevation: 1,2.1,3.1,1.1 2,2.2,3.2,2.2 3,2.3,3.3,3.3 4,3.4,3.4,4.4 5,3.5,3.5,5.5 6,3.6,3.6,6.6 7,4.7,4.7,7.1 8,4.8,4.8,8.8 9,4.9,4.9,9.9 I would like to process the file via fields two and three and have the result be: 1,2.1,3.1,1.1 3,2.3,3.3,3.3 4,3.4,3.4,4.4 6,3.6,3.6,6.6 7,4.7,4.7,7.1 9,4.9,4.9,9.9 So I think what I am asking is, that field 2 and 3 be considered duplicates if they are in the range "$2-0.1 to $2+0.1" and "$3-0.1 to $3+0.1". That's the best way I have of decribing it. Thanks in advance, Kenny. ![]() |
![]() |
| Bookmarks |
| Tags |
| solaris, unix commands |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|