Sponsored Content
Top Forums Shell Programming and Scripting Filter file to remove duplicate values in first column Post 302977897 by RudiC on Saturday 23rd of July 2016 03:15:12 AM
Old 07-23-2016
It is always worthwhile to comb through these fora for similar problems and their solutions; five examples are usually offered at the bottom of this page (of which at least three solve your problem), and more may be available, helping you to help yourself.

Anyway, try
Code:
awk '!T[$1]++' file
num     Name            PCA_A1     PCA_A2       PCA_A3
0       compound_00     -3.5054     -1.1207     -2.4372
1       compound_01     -2.2641     0.4287      -1.6120
3       compound_03     -1.3053     1.8495      -1.0224
4       compound_04     -1.1845     -0.3377     -2.9453
7       compound_07     -0.2988     1.3539      -1.6114
8       compound_08     2.6872     -1.3726      -5.9732
9       compound_09     -1.4546     -0.8284     -3.5016

 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Filter/remove duplicate .dat file with certain criteria

I am a beginner in Unix. Though have been asked to write a script to filter(remove duplicates) data from a .dat file. File is very huge containig billions of records. contents of file looks like 30002157,40342424,OTC,mart_rec,100, ,0 30002157,40343369,OTC,mart_rec,95, ,0... (6 Replies)
Discussion started by: mukeshguliao
6 Replies

2. UNIX for Dummies Questions & Answers

[SOLVED] remove lines that have duplicate values in column two

Hi, I've got a file that I'd like to uniquely sort based on column 2 (values in column 2 begin with "comp"). I tried sort -t -nuk2,3 file.txtBut got: sort: multi-character tab `-nuk2,3' "man sort" did not help me out Any pointers? Input: Output: (5 Replies)
Discussion started by: pathunkathunk
5 Replies

3. Shell Programming and Scripting

Check to identify duplicate values at first column in csv file

Hello experts, I have a requirement where I have to implement two checks on a csv file: 1. Check to see if the value in first column is duplicate, if any value is duplicate script should exit. 2. Check to verify if the value at second column is between "yes" or "no", if it is anything else... (4 Replies)
Discussion started by: avikaljain
4 Replies

4. Linux

Filter a .CSV file based on the 5th column values

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , comma","string with or, without commas","string 1","USD","12","70%","08/01/2013",""... (2 Replies)
Discussion started by: dhruuv369
2 Replies

5. Shell Programming and Scripting

Identify duplicate values at first column in csv file

Input 1,ABCD,no 2,system,yes 3,ABCD,yes 4,XYZ,no 5,XYZ,yes 6,pc,noCode used to find duplicate with regard to 2nd column awk 'NR == 1 {p=$2; next} p == $2 { print "Line" NR "$2 is duplicated"} {p=$2}' FS="," ./input.csv Now is there a wise way to de-duplicate the entire line (remove... (4 Replies)
Discussion started by: deadyetagain
4 Replies

6. Shell Programming and Scripting

Remove duplicate values in a column(not in the file)

Hi Gurus, I have a file(weblog) as below abc|xyz|123|agentcode=sample code abcdeeess,agentcode=sample code abcdeeess,agentcode=sample code abcdeeess|agentadd=abcd stereet 23343,agentadd=abcd stereet 23343 sss|wwq|999|agentcode=sample1 code wqwdeeess,gentcode=sample1 code... (4 Replies)
Discussion started by: ratheeshjulk
4 Replies

7. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

8. Shell Programming and Scripting

Filter duplicate records from csv file with condition on one column

I have csv file with 30, 40 columns Pasting just three column for problem description I want to filter record if column 1 matches CN or DN then, check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345 and if column 2 contains 6789, 6789... (5 Replies)
Discussion started by: as7951
5 Replies

9. Shell Programming and Scripting

CSV File:Filter duplicate records from column1 & another column having unique record

Hi Experts, I have csv file with 30, 40 columns Pasting just 2 column for problem description. Need to print error if below combination is not present in file check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same. For all such rows, the field... (7 Replies)
Discussion started by: as7951
7 Replies
Locale::Currency(3pm)					 Perl Programmers Reference Guide				     Locale::Currency(3pm)

NAME
Locale::Currency - standard codes for currency identification SYNOPSIS
use Locale::Currency; $curr = code2currency('usd'); # $curr gets 'US Dollar' $code = currency2code('Euro'); # $code gets 'eur' @codes = all_currency_codes(); @names = all_currency_names(); DESCRIPTION
The "Locale::Currency" module provides access to standard codes used for identifying currencies and funds, such as those defined in ISO 4217. Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default ISO 4217 three-letter codes will be used. SUPPORTED CODE SETS
There are several different code sets you can use for identifying currencies. A code set may be specified using either a name, or a constant that is automatically exported by this module. For example, the two are equivalent: $curr = code2currency('usd','alpha'); $curr = code2currency('usd',LOCALE_CURR_ALPHA); The codesets currently supported are: alpha, LOCALE_CURR_ALPHA This is a set of three-letter (uppercase) codes from ISO 4217 such as EUR for Euro. Two of the codes specified by the standard (XTS which is reserved for testing purposes and XXX which is for transactions where no currency is involved) are omitted. This is the default code set. num, LOCALE_CURR_NUMERIC This is the set of three-digit numeric codes from ISO 4217. ROUTINES
code2currency ( CODE [,CODESET] ) currency2code ( NAME [,CODESET] ) currency_code2code ( CODE ,CODESET ,CODESET2 ) all_currency_codes ( [CODESET] ) all_currency_names ( [CODESET] ) Locale::Currency::rename_currency ( CODE ,NEW_NAME [,CODESET] ) Locale::Currency::add_currency ( CODE ,NAME [,CODESET] ) Locale::Currency::delete_currency ( CODE [,CODESET] ) Locale::Currency::add_currency_alias ( NAME ,NEW_NAME ) Locale::Currency::delete_currency_alias ( NAME ) Locale::Currency::rename_currency_code ( CODE ,NEW_CODE [,CODESET] ) Locale::Currency::add_currency_code_alias ( CODE ,NEW_CODE [,CODESET] ) Locale::Currency::delete_currency_code_alias ( CODE [,CODESET] ) These routines are all documented in the Locale::Codes::API man page. SEE ALSO
Locale::Codes The Locale-Codes distribution. Locale::Codes::API The list of functions supported by this module. http://www.iso.org/iso/support/currency_codes_list-1.htm The ISO 4217 data. AUTHOR
See Locale::Codes for full author history. Currently maintained by Sullivan Beck (sbeck@cpan.org). COPYRIGHT
Copyright (c) 1997-2001 Canon Research Centre Europe (CRE). Copyright (c) 2001 Michael Hennecke Copyright (c) 2001-2010 Neil Bowers Copyright (c) 2010-2012 Sullivan Beck This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. perl v5.16.2 2012-10-11 Locale::Currency(3pm)
All times are GMT -4. The time now is 07:55 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy