Sponsored Content
Top Forums UNIX for Dummies Questions & Answers remove duplicate lines based on two columns and judging from a third one Post 302555275 by bartus11 on Wednesday 14th of September 2011 10:06:06 AM
Old 09-14-2011
Try this beautiful code:
Code:
perl -ane '/^.{4}/;if ($m{$&}{$F[3]}>$F[2]||$m{$&}{$F[3]}==undef){$m{$&}{$F[3]}=$F[2];$a{$&}{$F[3]}=$_}END{for $i (keys %a) {for $j (keys %{$a{$i}}){print $a{$i}{$j}}}}' file

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove lines, Sorted with Time based columns using AWK & SORT

Hi having a file as follows MediaErr.log 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:12:16 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:22:47 84 Server1 Policy1 Schedule1 master1 05/08/2008 03:41:26 84 Server1 Policy1 ... (1 Reply)
Discussion started by: karthikn7974
1 Replies

2. UNIX for Dummies Questions & Answers

Duplicate columns and lines

Hi all, I have a tab-delimited file and want to remove identical lines, i.e. all of line 1,2,4 because the columns are the same as the columns in other lines. Any input is appreciated. abc gi4597 9997 cgcgtgcg $%^&*()()* abc gi4597 9997 cgcgtgcg $%^&*()()* ttt ... (1 Reply)
Discussion started by: dr_sabz
1 Replies

3. Shell Programming and Scripting

Remove duplicate columns in input file

hello, I have an input file which looks like this: 2 C:G 17 -0.14 8.75 33.35 3 G:C 16 -2.28 0.98 28.22 4 C:G 15 0.39 11.06 29.31 5 G:C 14 2.64 5.17 36.07 6 G:C 13 -0.65 2.05 21.94 7 C:G 11 138.96 21.64 14.40 9 C:G 27 -2.40 6.95 27.98 10 C:G 26 2.89 15.60 34.33 11 G:C... (7 Replies)
Discussion started by: linux_usr
7 Replies

4. Shell Programming and Scripting

Remove duplicate lines based on field and sort

I have a csv file that I would like to remove duplicate lines based on field 1 and sort. I don't care about any of the other fields but I still wanna keep there data intact. I was thinking I could do something like this but I have no idea how to print the full line with this. Please show any method... (8 Replies)
Discussion started by: cokedude
8 Replies

5. Shell Programming and Scripting

Remove duplicate based on Group

Hi, How can I remove duplicates from a file based on group on other column? for example: Test1|Test2|Test3|Test4|Test5 Test1|Test6|Test7|Test8|Test5 Test1|Test9|Test10|Test11|Test12 Test1|Test13|Test14|Test15|Test16 Test17|Test18|Test19|Test20|Test21 Test17|Test22|Test23|Test24|Test5 ... (2 Replies)
Discussion started by: yale_work
2 Replies

6. Shell Programming and Scripting

Remove Duplicate by considering multiple columns

hi friends, my input chr1 exon 35204 35266 gene_id "GOLGB1"; transcript_id "GOLGB1"; chr1 exon 42357 42473 gene_id "GOLGB1"; transcript_id "GOLGB1"; chr1 exon 45261 45404 gene_id "GOLGB1"; transcript_id "GOLGB1"; chr1 exon 50701 50778 gene_id "GOLGB1"; transcript_id "GOLGB1";... (2 Replies)
Discussion started by: jacobs.smith
2 Replies

7. Shell Programming and Scripting

Remove duplicate value based on two field $4 and $5

Hi All, i have input file like below... CA009156;20091003;M;AWBKCA72;123;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;321;;CANADIAN WESTERN BANK;EDMONTON;;2300, 10303, JASPER AVENUE;;T5J 3X6;; CA009156;20091003;M;AWBKCA72;231;;CANADIAN... (2 Replies)
Discussion started by: mohan sharma
2 Replies

8. Shell Programming and Scripting

How To Remove Duplicate Based on the Value?

Hi , Some time i got duplicated value in my files , bundle_identifier= B Sometext=ABC bundle_identifier= A bundle_unit=500 Sometext123=ABCD bundle_unit=400 i need to check if there is a duplicated values or not if yes , i need to check if the value is A or B when Bundle_Identified ,... (2 Replies)
Discussion started by: OTNA
2 Replies

9. Shell Programming and Scripting

Remove columns with duplicate entries

I have a 13gb file. It has the following columns: The 3rd column is basically correlation values. I want to delete those rows which are repeated between the columns: A B 0.04 B C 0.56 B B 1 A A 1 C D 1 C C 1 Desired Output: (preferably in a .csv format A,B,0.04 B,C,0.56 C,D,1... (3 Replies)
Discussion started by: Sanchari
3 Replies

10. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies
Locale::Codes::LangVar(3pm)				 Perl Programmers Reference Guide			       Locale::Codes::LangVar(3pm)

NAME
Locale::Codes::LangVar - standard codes for language variation identification SYNOPSIS
use Locale::Codes::LangVar; $lvar = code2langvar('acm'); # $lvar gets 'Mesopotamian Arabic' $code = langvar2code('Mesopotamian Arabic'); # $code gets 'acm' @codes = all_langvar_codes(); @names = all_langvar_names(); DESCRIPTION
The "Locale::Codes::LangVar" module provides access to standard codes used for identifying language variations, such as those as defined in the IANA language registry. Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default IANA language registry codes will be used. SUPPORTED CODE SETS
There are several different code sets you can use for identifying language variations. A code set may be specified using either a name, or a constant that is automatically exported by this module. For example, the two are equivalent: $lvar = code2langvar('en','alpha-2'); $lvar = code2langvar('en',LOCALE_CODE_ALPHA_2); The codesets currently supported are: alpha This is the set of alphanumeric codes from the IANA language registry, such as 'arevela' for Eastern Armenian. This code set is identified with the symbol "LOCALE_LANGVAR_ALPHA". This is the default code set. ROUTINES
code2langvar ( CODE [,CODESET] ) langvar2code ( NAME [,CODESET] ) langvar_code2code ( CODE ,CODESET ,CODESET2 ) all_langvar_codes ( [CODESET] ) all_langvar_names ( [CODESET] ) Locale::Codes::LangVar::rename_langvar ( CODE ,NEW_NAME [,CODESET] ) Locale::Codes::LangVar::add_langvar ( CODE ,NAME [,CODESET] ) Locale::Codes::LangVar::delete_langvar ( CODE [,CODESET] ) Locale::Codes::LangVar::add_langvar_alias ( NAME ,NEW_NAME ) Locale::Codes::LangVar::delete_langvar_alias ( NAME ) Locale::Codes::LangVar::rename_langvar_code ( CODE ,NEW_CODE [,CODESET] ) Locale::Codes::LangVar::add_langvar_code_alias ( CODE ,NEW_CODE [,CODESET] ) Locale::Codes::LangVar::delete_langvar_code_alias ( CODE [,CODESET] ) These routines are all documented in the Locale::Codes::API man page. SEE ALSO
Locale::Codes The Locale-Codes distribution. Locale::Codes::API The list of functions supported by this module. http://www.iana.org/assignments/language-subtag-registry The IANA language subtag registry. AUTHOR
See Locale::Codes for full author history. Currently maintained by Sullivan Beck (sbeck@cpan.org). COPYRIGHT
Copyright (c) 2011-2013 Sullivan Beck This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.18.2 2014-01-06 Locale::Codes::LangVar(3pm)
All times are GMT -4. The time now is 12:34 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy