Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: remove duplicates based on a field and criteria
Top Forums UNIX for Dummies Questions & Answers remove duplicates based on a field and criteria Post 302611011 by Scrutinizer on Thursday 22nd of March 2012 05:38:56 AM
Old 03-22-2012
Hi, try:
Code:
sort -t \; -k3.2,3.3rn infile | awk -F\; '!A[$4]++'

Make sure there are not spaces in the input file.
This User Gave Thanks to Scrutinizer For This Post:
wanderingmind16
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove lines based on score criteria

Hi guys, Please guide for Solution. PART-I INPUT FILE (has 2 columns ID and score) TC5584_1 93.9 DV161411_2 79.5 BP132435_5 46.8 EB682112_1 34.7 BP132435_4 29.5 TC13860_2 10.1 OUTPUT FILE (It shudn't contain the line ' BP132435_4 29.5 ' as BP132435 is repeated... (2 Replies)
Discussion started by: smriti_shridhar
2 Replies

2. Shell Programming and Scripting

split large file based on field criteria

I have a file containing date/time sorted data of the form ... 2009/06/10,20:59:59.950,XAG/USD,Q,1,1115, 14.3025,100,1,1 2009/06/10,20:59:59.950,XAG/USD,Q,1,1116, 14.3026,125,1,1 2009/06/10,20:59:59.950,XAG/USD,R,0,0, , 0,0,0 2009/06/10,20:59:59.950,XAG/USD,R,1,0, 14.1910,100,1,1... (6 Replies)
Discussion started by: asriva
6 Replies

3. Shell Programming and Scripting

need Shell script for Sort BASED ON FIRST FIELD and PRINT THE WHOLE FILE WITHOUT DUPLICATES

Can some one provide me a shell script. I have file with many columns and many rows. need to sort the first column and then remove the duplicates records if exists.. finally print the full data with first coulm as unique. Sort BASED ON FIRST FIELD and remove the duplicates if exists... (2 Replies)
Discussion started by: tuffEnuff
2 Replies

4. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

5. Shell Programming and Scripting

Remove duplicates based on the two key columns

Hi All, I needs to fetch unique records based on a keycolumn(ie., first column1) and also I needs to get the records which are having max value on column2 in sorted manner... and duplicates have to store in another output file. Input : Input.txt 1234,0,x 1234,1,y 5678,10,z 9999,10,k... (7 Replies)
Discussion started by: kmsekhar
7 Replies

6. Shell Programming and Scripting

remove duplicates based on single column

Hello, I am new to shell scripting. I have a huge file with multiple columns for example: I have 5 columns below. HWUSI-EAS000_29:1:105 + chr5 76654650 AATTGGAA HHHHG HWUSI-EAS000_29:1:106 + chr5 76654650 AATTGGAA B@HYL HWUSI-EAS000_29:1:108 + ... (4 Replies)
Discussion started by: Diya123
4 Replies

7. Shell Programming and Scripting

Remove the partial duplicates by checking the length of a field

Hi Folks - I'm quite new to awk and didn't come across such issues before. The problem statement is that, I've a file with duplicate records in 3rd and 4th fields. The sample is as below: aaaaaa|a12|45|56 abbbbaaa|a12|45|56 bbaabb|b1|51|45 bbbbbabbb|b2|51|45 aaabbbaaaa|a11|45|56 ... (3 Replies)
Discussion started by: asyed
3 Replies

8. Shell Programming and Scripting

CSV with commas in field values, remove duplicates, cut columns

Hi Description of input file I have: ------------------------- 1) CSV with double quotes for string fields. 2) Some string fields have Comma as part of field value. 3) Have Duplicate lines 4) Have 200 columns/fields 5) File size is more than 10GB Description of output file I need:... (4 Replies)
Discussion started by: krishnix
4 Replies

9. Shell Programming and Scripting

Remove duplicates based on a field's value

Hi All, I have a text file with three columns. I would like a simple script that removes lines in which column 1 has duplicate entries, but use the largest value in column 3 to decide which one to keep. For example: Input file: 12345a rerere.rerere len=23 11111c fsdfdf.dfsdfdsf len=33 ... (3 Replies)
Discussion started by: anniecarv
3 Replies

10. Shell Programming and Scripting

Trying to remove duplicates based on field and row

I am trying to see if I can use awk to remove duplicates from a file. This is the file: -==> Listvol <== deleting /vol/eng_rmd_0941 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_0943 deleting /vol/eng_rmd_1006 deleting /vol/eng_rmd_1012 rearrange /vol/eng_rmd_0943 ... (6 Replies)
Discussion started by: newbie2010
6 Replies

LEARN ABOUT SUNOS

iconv_8859-1

iconv_8859-1(5) 					Standards, Environments, and Macros					   iconv_8859-1(5)

NAME

       iconv_8859-1 - code set conversion tables for ISO 8859-1 (Latin 1)

DESCRIPTION

       The following code set conversions are supported:

       +--------------------------------------------------------------------+
       |		  Code Set Conversions Supported		    |
       +--------------+--------+--------------+--------+--------------------+
       |    Code      |Symbol  |Target Code   |Symbol  |   Target Output    |
       +--------------+--------+--------------+--------+--------------------+
       |ISO 8859-1    |8859    |ISO 646       |646     | 7 bit ASCII	    |
       +--------------+--------+--------------+--------+--------------------+
       |ISO 8859-1    |8859    |ISO 646de     |646de   | German 	    |
       +--------------+--------+--------------+--------+--------------------+
       |ISO 8859-1    |8859    |ISO 646da     |646da   | Danish 	    |
       +--------------+--------+--------------+--------+--------------------+
       |ISO 8859-1    |8859    |ISO 646en     |646en   | English ASCII	    |
       +--------------+--------+--------------+--------+--------------------+
       |ISO 8859-1    |8859    |ISO 646es     |646es   | Spanish	    |
       +--------------+--------+--------------+--------+--------------------+
       |ISO 8859-1    |8859    |ISO 646fr     |646fr   | French 	    |
       +--------------+--------+--------------+--------+--------------------+
       |ISO 8859-1    |8859    |ISO 646it     |646it   | Italian	    |
       +--------------+--------+--------------+--------+--------------------+
       |ISO 8859-1    |8859    |ISO 646sv     |646sv   | Swedish	    |
       +--------------+--------+--------------+--------+--------------------+

CONVERSIONS

       The conversions are performed according to the following tables. All values in the tables are given in octal.

   ISO 8859-1 to ISO 646 (7-bit ASCII)
       For the conversion of ISO 8859-1 to ISO 646, all characters not in the following table are mapped unchanged.

       Converted to Underscore '_' (137)
       ---------------------------------
       200 201 202 203 204 205 206 207
       210 211 212 213 214 215 216 217
       220 221 222 223 224 225 226 227
       230 231 232 233 234 235 236 237
       240 241 242 243 244 245 246 247
       250 251 252 253 254 255 256 257
       260 261 262 263 264 265 266 267
       270 271 272 273 274 275 276 277
       300 301 302 303 304 305 306 307
       310 311 312 313 314 315 316 317
       320 321 322 323 324 325 326 327
       330 331 332 333 334 335 336 337
       340 341 342 343 344 345 346 347
       350 351 352 353 354 355 356 357
       360 361 362 363 364 365 366 367
       370 371 372 373 374 375 376 377

   ISO 8859-1 to ISO 646de (GERMAN)
       For the conversion of ISO 8859-1 to ISO 646de, all characters not in the following tables are mapped unchanged.

       +-----------------------------------------------------------------+
       |	       |     Conversions|Performed	 |		 |
       |  ISO 8859-1   |   ISO 646de	|  ISO 8859-1	 |  ISO 646de	 |
       |247	       | 100		|337		 |176		 |
       |304	       | 133		|344		 |173		 |
       |326	       | 134		|366		 |174		 |
       |334	       | 135		|374		 |175		 |
       +---------------+----------------+----------------+---------------+

       Converted to Underscore '_' (137)
       ---------------------------------
       100 133 134 135 173 174 175 176
       200 201 202 203 204 205 206 207
       210 211 212 213 214 215 216 217
       220 221 222 223 224 225 226 227
       230 231 232 233 234 235 236 237
       240 241 242 243 244 245 246
       250 251 252 253 254 255 256 257
       260 261 262 263 264 265 266 267
       270 271 272 273 274 275 276 277
       300 301 302 303	   305 306 307
       310 311 312 313 314 315 316 317
       320 321 322 323 324 325	   327
       330 331 332 333	   335 336 337
       340 341 342 343	   345 346 347
       350 351 352 353 354 355 356 357
       360 361 362 363 364 365	   367
       370 371 372 373	   375 376 377

   ISO 8859-1 to ISO 646da (DANISH)
       For the conversion of ISO 8859-1 to ISO 646da, all characters not in the following tables are mapped unchanged.

       +-----------------------------------------------------------------+
       |		     Conversions Performed			 |
       +---------------+----------------+----------------+---------------+
       |  ISO 8859-1   |   ISO 646da	|  ISO 8859-1	 |     ISO	 |
       +---------------+----------------+----------------+---------------+
       |646da	       |		|		 |		 |
       +---------------+----------------+----------------+---------------+
       |305	       | 135		|345		 |175		 |
       +---------------+----------------+----------------+---------------+
       |306	       | 133		|346		 |173		 |
       +---------------+----------------+----------------+---------------+
       |330	       | 134		|370		 |174		 |
       +---------------+----------------+----------------+---------------+

       Converted to Underscore '_' (137)
       ---------------------------------
       133 134 135 173 174 175
       200 201 202 203 204 205 206 207
       210 211 212 213 214 215 216 217
       220 221 222 223 224 225 226 227
       230 231 232 233 234 235 236 237
       240 241 242 243 244 245 246 247
       250 251 252 253 254 255 256 257
       260 261 262 263 264 265 266 267
       270 271 272 273 274 275 276 277
       300 301 302 303 304	   307
       310 311 312 313 314 315 316 317
       320 321 322 323 324 325 326 327
	   331 332 333 334 335 336 337
       340 341 342 343 344	   347
       350 351 352 353 354 355 356 357
       360 361 362 363 364 365 366 367
       371 372 373 374	       376 377

   ISO 8859-1 to ISO 646en (ENGLISH ASCII)
       For the conversion of ISO 8859-1 to ISO 646en, all characters not in the following tables are mapped unchanged.

       +-----------------------------------------------------------+
       |		  Conversions Performed 		   |
       +-----------------------------+-----------------------------+
       |	ISO 8859-1	     |	       ISO 646en	   |
       +-----------------------------+-----------------------------+
       |243			     |043			   |
       +-----------------------------+-----------------------------+

       Converted to Underscore '_' (137)
       ---------------------------------
       043
       200 201 202 203 204 205 206 207
       210 211 212 213 214 215 216 217
       220 221 222 223 224 225 226 227
       230 231 232 233 234 235 236 237
       240 241 242     244 245 246 247
       250 251 252 253 254 255 256 257
       260 261 262 263 264 265 266 267
       270 271 272 273 274 275 276 277
       300 301 302 303 304 305 306 307
       310 311 312 313 314 315 316 317
       320 321 322 323 324 325 326 327
       330 331 332 333 334 335 336 337
       340 341 342 343 344 345 346 347
       350 351 352 353 354 355 356 357
       360 361 362 363 364 365 366 367
       370 371 372 373 374 375 376 377

   ISO 8859-1 to ISO 646fr (FRENCH)
       For the conversion of ISO 8859-1 to ISO 646fr, all characters not in the following tables are mapped unchanged.

       +-----------------------------------------------------------------+
       |	       |     Conversions|Performed	 |		 |
       |  ISO 8859-1   |   ISO 646fr	|  ISO 8859-1	 |  ISO 646fr	 |
       |243	       | 043		|347		 |134		 |
       |247	       | 135		|350		 |175		 |
       |250	       | 176		|351		 |173		 |
       |260	       | 133		|371		 |174		 |
       |340	       | 100		|		 |		 |
       +---------------+----------------+----------------+---------------+

       Converted to Underscore '_' (137)
       ---------------------------------
       043
       100 133 134 135 173 174 175 176
       200 201 202 203 204 205 206 207
       210 211 212 213 214 215 216 217
       220 221 222 223 224 225 226 227
       230 231 232 233 234 235 236 237
       240 241 242     244 245 246
	   251 252 253 254 255 256 257
	   261 262 263 264 265 266 267
       270 271 272 273 274 275 276 277
       300 301 302 303 304 305 306 307
       310 311 312 313 314 315 316 317
       320 321 322 323 324 325 326 327
       330 331 332 333 334 335 336 337
	   341 342 343 344 345 346
	       352 353 354 355 356 357
       360 361 362 363 364 365 366 367
       370     372 373 374 375 376 377

   ISO 8859-1 to ISO 646it (ITALIAN)
       For the conversion of ISO 8859-1 to ISO 646it, all characters not in the following tables are mapped unchanged.

       +-----------------------------------------------------------------+
       |	       |     Conversions|Performed	 |		 |
       |  ISO 8859-1   |   ISO 646it	|  ISO 8859-1	 |  ISO 646it	 |
       |243	       | 043		|350		 |175		 |
       |247	       | 100		|351		 |135		 |
       |260	       | 133		|354		 |176		 |
       |340	       | 173		|362		 |174		 |
       |347	       | 134		|371		 |140		 |
       +---------------+----------------+----------------+---------------+

       Converted to Underscore '_' (137)
       ---------------------------------
       043
       100 133 134 135 173 174 175 176
       200 201 202 203 204 205 206 207
       210 211 212 213 214 215 216 217
       220 221 222 223 224 225 226 227
       230 231 232 233 234 235 236 237
       240 241 242     244 245 246
       250 251 252 253 254 255 256 257
	   261 262 263 264 265 266 267
       270 271 272 273 274 275 276 277
       300 301 302 303 304 305 306 307
       310 311 312 313 314 315 316 317
       320 321 322 323 324 325 326 327
       330 331 332 333 334 335 336 337
	   341 342 343 344 345 346
	       352 353 354 355 356 357
       360 361 362 363 364 365 366 367
       370     372 373 374 375 376 377

   ISO 8859-1 to ISO 646es (SPANISH)
       For the conversion of ISO 8859-1 to ISO 646es, all characters not in the following tables are mapped unchanged.

       +-----------------------------------------------------------------+
       |	       |     Conversions|Performed	 |		 |
       |  ISO 8859-1   |   ISO 646es	|  ISO 8859-1	 |  ISO 646es	 |
       |241	       | 133		|321		 |134		 |
       |247	       | 100		|347		 |175		 |
       |260	       | 173		|361		 |174		 |
       |277	       | 135		|		 |		 |
       +---------------+----------------+----------------+---------------+

       Converted to Underscore '_' (137)
       ---------------------------------
       100 133 134 135 173 174 175
       200 201 202 203 204 205 206 207
       210 211 212 213 214 215 216 217
       220 221 222 223 224 225 226 227
       230 231 232 233 234 235 236 237
       240     242 243 244 245 246
       250 251 252 253 254 255 256 257
	   261 262 263 264 265 266 267
       270 271 272 273 274 275 276
       300 301 302 303 304 305 306 307
       310 311 312 313 314 315 316 317
       320     322 323 324 325 326 327
       330 331 332 333 334 335 336 337
       340 341 342 343 344 345 346
       350 351 352 353 354 355 356 357
       360     362 363 364 365 366 367
       370 371 372 373 374 375 376 377

   ISO 8859-1 to ISO 646sv (SWEDISH)
       For the conversion of ISO 8859-1 to ISO 646sv, all characters not in the following tables are mapped unchanged.

       +-----------------------------------------------------------------+
       |	       |     Conversions|Performed	 |		 |
       |  ISO 8859-1   |   ISO 646sv	|  ISO 8859-1	 |  ISO 646sv	 |
       |304	       | 133		|344		 |173		 |
       |305	       | 135		|345		 |175		 |
       |311	       | 100		|351		 |140		 |
       |326	       | 134		|366		 |174		 |
       |334	       | 136		|374		 |176		 |
       +---------------+----------------+----------------+---------------+

       Converted to Underscore '_' (137)
       ---------------------------------
       100 133 134 135 136 140
       173 174 175 176
       200 201 202 203 204 205 206 207
       210 211 212 213 214 215 216 217
       220 221 222 223 224 225 226 227
       230 231 232 233 234 235 236 237
       240 241 242 243 244 245 246 247
       250 251 252 253 254 255 256 257
       260 261 262 263 264 265 266 267
       270 271 272 273 274 275 276 277
       300 301 302 303	       306 307
       310     312 313 314 315 316 317
       320 321 322 323 324 325	   327
       330 331 332 333	   335 336 337
       340 341 342 343	       346 347
       350     352 353 354 355 356 357
       360 361 362 363 364 365	   367
       370 371 372 373	   375 376 377

FILES

       /usr/lib/iconv/*.so	       conversion modules

       /usr/lib/iconv/*.t	       conversion tables

       /usr/lib/iconv/iconv_data       list of conversions supported by conversion tables

SEE ALSO

       iconv(1), iconv(3C), iconv(5)

SunOS 5.10							    28 Apr 1997 						   iconv_8859-1(5)
All times are GMT -4. The time now is 02:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy