Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: remove duplicates based on a field and criteria
Top Forums UNIX for Dummies Questions & Answers remove duplicates based on a field and criteria Post 302611011 by Scrutinizer on Thursday 22nd of March 2012 05:38:56 AM
Old 03-22-2012
Hi, try:
Code:
sort -t \; -k3.2,3.3rn infile | awk -F\; '!A[$4]++'

Make sure there are not spaces in the input file.
This User Gave Thanks to Scrutinizer For This Post:
wanderingmind16
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove lines based on score criteria

Hi guys, Please guide for Solution. PART-I INPUT FILE (has 2 columns ID and score) TC5584_1 93.9 DV161411_2 79.5 BP132435_5 46.8 EB682112_1 34.7 BP132435_4 29.5 TC13860_2 10.1 OUTPUT FILE (It shudn't contain the line ' BP132435_4 29.5 ' as BP132435 is repeated... (2 Replies)
Discussion started by: smriti_shridhar
2 Replies

2. Shell Programming and Scripting

split large file based on field criteria

I have a file containing date/time sorted data of the form ... 2009/06/10,20:59:59.950,XAG/USD,Q,1,1115, 14.3025,100,1,1 2009/06/10,20:59:59.950,XAG/USD,Q,1,1116, 14.3026,125,1,1 2009/06/10,20:59:59.950,XAG/USD,R,0,0, , 0,0,0 2009/06/10,20:59:59.950,XAG/USD,R,1,0, 14.1910,100,1,1... (6 Replies)
Discussion started by: asriva
6 Replies

3. Shell Programming and Scripting

need Shell script for Sort BASED ON FIRST FIELD and PRINT THE WHOLE FILE WITHOUT DUPLICATES

Can some one provide me a shell script. I have file with many columns and many rows. need to sort the first column and then remove the duplicates records if exists.. finally print the full data with first coulm as unique. Sort BASED ON FIRST FIELD and remove the duplicates if exists... (2 Replies)
Discussion started by: tuffEnuff
2 Replies

4. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

5. Shell Programming and Scripting

Remove duplicates based on the two key columns

Hi All, I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file. Input : Input.txt 1234,0,x 1234,1,y 5678,10,z 9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies

6. Shell Programming and Scripting

remove duplicates based on single column

Hello, I am new to shell scripting. I have a huge file with multiple columns for example: I have 5 columns below. HWUSI-EAS000_29:1:105 + chr5 76654650 AATTGGAA HHHHG HWUSI-EAS000_29:1:106 + chr5 76654650 AATTGGAA B@HYL HWUSI-EAS000_29:1:108 + ... (4 Replies)
Discussion started by: Diya123
4 Replies

7. Shell Programming and Scripting

Remove the partial duplicates by checking the length of a field

Hi Folks - I'm quite new to awk and didn't come across such issues before. The problem statement is that, I've a file with duplicate records in 3rd and 4th fields. The sample is as below: aaaaaa|a12|45|56 abbbbaaa|a12|45|56 bbaabb|b1|51|45 bbbbbabbb|b2|51|45 aaabbbaaaa|a11|45|56 ... (3 Replies)
Discussion started by: asyed
3 Replies

8. Shell Programming and Scripting

CSV with commas in field values, remove duplicates, cut columns

Hi Description of input file I have: ------------------------- 1) CSV with double quotes for string fields. 2) Some string fields have Comma as part of field value. 3) Have Duplicate lines 4) Have 200 columns/fields 5) File size is more than 10GB Description of output file I need:... (4 Replies)
Discussion started by: krishnix
4 Replies

9. Shell Programming and Scripting

Remove duplicates based on a field's value

Hi All, I have a text file with three columns. I would like a simple script that removes lines in which column 1 has duplicate entries, but use the largest value in column 3 to decide which one to keep. For example: Input file: 12345a rerere.rerere len=23 11111c fsdfdf.dfsdfdsf len=33 ... (3 Replies)
Discussion started by: anniecarv
3 Replies

10. Shell Programming and Scripting

Trying to remove duplicates based on field and row

I am trying to see if I can use awk to remove duplicates from a file. This is the file: -==> Listvol <== deleting /vol/eng_rmd_0941 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_1006 deleting /vol/eng_rmd_1012 rearrange /vol/eng_rmd_0943 ... (6 Replies)
Discussion started by: newbie2010
6 Replies

LEARN ABOUT OPENSOLARIS

iconv_mac_cyr

iconv_mac_cyr(5)                                        Standards, Environments, and Macros                                       iconv_mac_cyr(5)

NAME

       iconv_mac_cyr - code set conversion tables for Macintosh Cyrillic

DESCRIPTION

       The following code set conversions are supported:

       +---------------------------------------------------------------------+
       |                   Code Set Conversions Supported                    |
       +--------------+--------+--------------+--------+---------------------+
       |    Code      |Symbol  |Target Code   |Symbol  |       Target        |
       +--------------+--------+--------------+--------+---------------------+
       |Output        |        |              |        |                     |
       +--------------+--------+--------------+--------+---------------------+
       |Mac Cyrillic  |mac     |ISO 8859-5    |iso5    | ISO 8859-5 Cyrillic |
       +--------------+--------+--------------+--------+---------------------+
       |Mac Cyrillic  |mac     |KOI8-R        |koi8    | KOI8-R              |
       +--------------+--------+--------------+--------+---------------------+
       |Mac Cyrillic  |mac     |PC Cyrillic   |alt     | Alternative PC      |
       +--------------+--------+--------------+--------+---------------------+
       |Cyrillic      |        |              |        |                     |
       +--------------+--------+--------------+--------+---------------------+
       |Mac Cyrillic  |mac     |MS 1251       |win5    | Windows Cyrillic    |
       +--------------+--------+--------------+--------+---------------------+

CONVERSIONS

       The conversions are performed according to the following tables. All values in the tables are given in octal.

   Mac Cyrillic to ISO 8859-5
       For the conversion of Mac Cyrillic to ISO 8859-5, all characters not in the following table are mapped unchanged.

       +-----------------------------------------------------------------+
       |                     Conversions Performed                       |
       | Mac Cyrillic     ISO 8859-5      Mac Cyrillic      ISO 8859-5   |
       |24               4               276              252            |
       |200              260             277              372            |
       |201              261             300              370            |
       |202              262             301              245            |
       |203              263             302-311          40             |
       |204              264             312              240            |
       |205              265             313              242            |
       |206              266             314              362            |
       |207              267             315              254            |
       |210              270             316              374            |
       |211              271             317              365            |
       |212              272             320-327          40             |
       |213              273             330              256            |
       |214              274             331              376            |
       |215              275             332              257            |
       |216              276             333              377            |
       |217              277             334              360            |
       |220              300             335              241            |
       |221              301             336              361            |
       |222              302             337              357            |
       |223              303             340              320            |
       |224              304             341              321            |
       |225              305             342              322            |
       |226              306             343              323            |
       |227              307             344              324            |
       |230              310             345              325            |
       |231              311             346              326            |
       |232              312             347              327            |
       |233              313             350              330            |
       |234              314             351              331            |
       |235              315             352              332            |
       |236              316             353              333            |
       |237              317             354              334            |
       |240-246          40              355              335            |
       |247              246             356              336            |
       |250-252          40              357              337            |
       |253              242             360              340            |
       |254              362             361              341            |
       |255              40              362              342            |
       |256              243             363              343            |
       |257              363             364              344            |
       |260-263          40              365              345            |
       |264              366             366              346            |
       |265-266          40              367              347            |
       |267              250             370              350            |
       |270              244             371              351            |
       |271              364             372              352            |
       |272              247             373              353            |
       |273              367             374              354            |
       |274              251             375              355            |
       |275              371             376              356            |
       |375              370                                             |
       +-----------------------------------------------------------------+

   Mac Cyrillic to KOI8-R
       For the conversion of Mac Cyrillic to KOI8-R, all characters not in the following table are mapped unchanged.

       +-----------------------------------------------------------------+
       |               |     Conversions|Performed       |               |
       | Mac Cyrillic  |    KOI8-R      | Mac Cyrillic   |    KOI8-R     |
       |24             | 4              |276             |272            |
       |200            | 341            |277             |252            |
       |201            | 342            |300             |250            |
       |202            | 367            |301             |265            |
       |203            | 347            |302-311         |40             |
       |204            | 344            |312             |240            |
       |205            | 345            |313             |261            |
       |206            | 366            |314             |241            |
       |207            | 372            |315             |274            |
       |210            | 351            |316             |254            |
       |211            | 352            |317             |245            |
       |212            | 353            |320-327         |40             |
       |213            | 354            |330             |276            |
       |214            | 355            |331             |256            |
       |215            | 356            |332             |277            |
       |216            | 357            |333             |257            |
       |217            | 360            |334             |260            |
       |220            | 362            |335             |263            |
       |221            | 363            |336             |243            |
       |222            | 364            |337             |321            |
       |223            | 365            |340             |301            |
       |224            | 346            |341             |302            |
       |225            | 350            |342             |327            |
       |226            | 343            |343             |307            |
       |227            | 376            |344             |304            |
       |230            | 373            |345             |305            |
       |231            | 375            |346             |326            |
       |232            | 377            |347             |332            |
       |233            | 371            |350             |311            |
       |234            | 370            |351             |312            |
       |235            | 374            |352             |313            |
       |236            | 340            |353             |314            |
       |237            | 361            |354             |315            |
       |240-246        | 40             |355             |316            |
       |247            | 266            |356             |317            |
       |250-252        | 40             |357             |320            |
       |253            | 261            |360             |322            |
       |254            | 241            |361             |323            |
       |255            | 40             |362             |324            |
       |256            | 262            |363             |325            |
       |257            | 242            |364             |306            |
       |260-263        | 40             |365             |310            |
       |264            | 246            |366             |303            |
       |265-266        | 40             |367             |336            |
       |267            | 270            |370             |333            |
       |270            | 264            |371             |335            |
       |271            | 244            |372             |337            |
       |272            | 267            |373             |331            |
       |273            | 247            |374             |330            |
       |274            | 271            |375             |334            |
       |275            | 251            |376             |300            |
       |375            | 370            |                |               |
       +---------------+----------------+----------------+---------------+

   Mac Cyrillic to PC Cyrillic
       For the conversion of Mac Cyrillic to PC Cyrillic, all characters not in the following table are mapped unchanged.

       +-----------------------------------------------------------------+
       |               |     Conversions|Performed       |               |
       | Mac Cyrillic  |  PC Cyrillic   | Mac Cyrillic   | PC Cyrillic   |
       |24             | 4              |355             |255            |
       |240-334        | 40             |356             |256            |
       |335            | 360            |357             |257            |
       |336            | 361            |360             |340            |
       |337            | 357            |361             |341            |
       |340            | 240            |362             |342            |
       |341            | 241            |363             |343            |
       |342            | 242            |364             |344            |
       |343            | 243            |365             |345            |
       |344            | 244            |366             |346            |
       |345            | 245            |367             |347            |
       |346            | 246            |370             |350            |
       |347            | 247            |371             |351            |
       |350            | 250            |372             |352            |
       |351            | 251            |373             |353            |
       |352            | 252            |374             |354            |
       |353            | 253            |375             |355            |
       |354            | 254            |376             |356            |
       |303            | 366            |                |               |
       +---------------+----------------+----------------+---------------+

   Mac Cyrillic to MS 1251
       For the conversion of Mac Cyrillic to MS 1251, all characters not in the following table are mapped unchanged.

       +-----------------------------------------------------------------+
       |               |     Conversions|Performed       |               |
       | Mac Cyrillic  |    MS 1251     | Mac Cyrillic   |   MS 1251     |
       |24             | 4              |255             |40             |
       |200            | 300            |256             |201            |
       |201            | 301            |257             |203            |
       |202            | 302            |260-263         |40             |
       |203            | 303            |264             |263            |
       |204            | 304            |266             |264            |
       |205            | 305            |267             |243            |
       |206            | 306            |270             |252            |
       |207            | 307            |271             |272            |
       |210            | 310            |272             |257            |
       |211            | 311            |273             |277            |
       |212            | 312            |274             |212            |
       |213            | 313            |275             |232            |
       |214            | 314            |276             |214            |
       |215            | 315            |277             |234            |
       |216            | 316            |300             |274            |
       |217            | 317            |301             |275            |
       |220            | 320            |302             |254            |
       |221            | 321            |303-306         |40             |
       |222            | 322            |307             |253            |
       |223            | 323            |310             |273            |
       |224            | 324            |311             |205            |
       |225            | 325            |312             |240            |
       |226            | 326            |313             |200            |
       |227            | 327            |314             |220            |
       |230            | 330            |315             |215            |
       |231            | 331            |316             |235            |
       |232            | 332            |317             |276            |
       |233            | 333            |320             |226            |
       |234            | 334            |321             |227            |
       |235            | 335            |322             |223            |
       |236            | 336            |323             |224            |
       |237            | 337            |324             |221            |
       |240            | 206            |325             |222            |
       |241            | 260            |326             |40             |
       |242            | 245            |327             |204            |
       |243            | 40             |330             |241            |
       |244            | 247            |331             |242            |
       |245            | 267            |332             |217            |
       |246            | 266            |333             |237            |
       |247            | 262            |334             |271            |
       |250            | 256            |335             |250            |
       |252            | 231            |336             |270            |
       |253            | 200            |337             |377            |
       |254            | 220            |362             |324            |
       +---------------+----------------+----------------+---------------+

FILES

       /usr/lib/iconv/*.so             conversion modules

       /usr/lib/iconv/*.t              conversion tables

       /usr/lib/iconv/iconv_data       list of conversions supported by conversion tables

SEE ALSO

       iconv(1), iconv(3C), iconv(5)

SunOS 5.10                                                          18 Apr 1997                                                   iconv_mac_cyr(5)
All times are GMT -4. The time now is 11:37 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy