Sponsored Content
Top Forums Shell Programming and Scripting Filter file to remove duplicate values in first column Post 302977895 by LMHmedchem on Saturday 23rd of July 2016 02:15:21 AM
Old 07-23-2016
Filter file to remove duplicate values in first column

Hello,

I have a script that is generating a tab delimited output file.

Code:
num     Name            PCA_A1     PCA_A2       PCA_A3
0       compound_00     -3.5054     -1.1207     -2.4372
1       compound_01     -2.2641     0.4287      -1.6120
3       compound_03     -1.3053     1.8495      -1.0224
0       compound_00     -3.5054     -1.1207     -2.4372
4       compound_04     -1.1845     -0.3377     -2.9453
7       compound_07     -0.2988     1.3539      -1.6114
8       compound_08     2.6872     -1.3726      -5.9732
9       compound_09     -1.4546     -0.8284     -3.5016
4       compound_04     -1.1845     -0.3377     -2.9453
7       compound_07     -0.2988     1.3539      -1.6114
8       compound_08     2.6872     -1.3726      -5.9732

I need to trim this down so that there a no duplicates in the first column. Actually, the entire row would be a duplicate, but I don't see any reason to look at anything other than the index value. There is no particular rational to the order and there could be any number of duplicates of a given row.

The final results should look like this,
Code:
num     Name            PCA_A1     PCA_A2       PCA_A3
0       compound_00     -3.5054     -1.1207     -2.4372
1       compound_01     -2.2641     0.4287      -1.6120
3       compound_03     -1.3053     1.8495      -1.0224
4       compound_04     -1.1845     -0.3377     -2.9453
7       compound_07     -0.2988     1.3539      -1.6114
8       compound_08     2.6872     -1.3726      -5.9732
9       compound_09     -1.4546     -0.8284     -3.5016

I need one, and only one, instance of each index value ("num" column value) in the file, not just the lines with num values that appear only once. There always seems to be some confusion about that with discussions of "unique" lines.

The only thing I could think of was to sort the rows on the num column value and then loop through checking if the num value was equal to the previous line. If it is not equal, copy it to a new array, etc.

Any suggestions? There always seems to be some simple one line solution that I don't know about.

LMHmedchem
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Filter/remove duplicate .dat file with certain criteria

I am a beginner in Unix. Though have been asked to write a script to filter(remove duplicates) data from a .dat file. File is very huge containig billions of records. contents of file looks like 30002157,40342424,OTC,mart_rec,100, ,0 30002157,40343369,OTC,mart_rec,95, ,0... (6 Replies)
Discussion started by: mukeshguliao
6 Replies

2. UNIX for Dummies Questions & Answers

[SOLVED] remove lines that have duplicate values in column two

Hi, I've got a file that I'd like to uniquely sort based on column 2 (values in column 2 begin with "comp"). I tried sort -t -nuk2,3 file.txtBut got: sort: multi-character tab `-nuk2,3' "man sort" did not help me out Any pointers? Input: Output: (5 Replies)
Discussion started by: pathunkathunk
5 Replies

3. Shell Programming and Scripting

Check to identify duplicate values at first column in csv file

Hello experts, I have a requirement where I have to implement two checks on a csv file: 1. Check to see if the value in first column is duplicate, if any value is duplicate script should exit. 2. Check to verify if the value at second column is between "yes" or "no", if it is anything else... (4 Replies)
Discussion started by: avikaljain
4 Replies

4. Linux

Filter a .CSV file based on the 5th column values

I have a .CSV file with the below format: "column 1","column 2","column 3","column 4","column 5","column 6","column 7","column 8","column 9","column 10 "12310","42324564756","a simple string with a , comma","string with or, without commas","string 1","USD","12","70%","08/01/2013",""... (2 Replies)
Discussion started by: dhruuv369
2 Replies

5. Shell Programming and Scripting

Identify duplicate values at first column in csv file

Input 1,ABCD,no 2,system,yes 3,ABCD,yes 4,XYZ,no 5,XYZ,yes 6,pc,noCode used to find duplicate with regard to 2nd column awk 'NR == 1 {p=$2; next} p == $2 { print "Line" NR "$2 is duplicated"} {p=$2}' FS="," ./input.csv Now is there a wise way to de-duplicate the entire line (remove... (4 Replies)
Discussion started by: deadyetagain
4 Replies

6. Shell Programming and Scripting

Remove duplicate values in a column(not in the file)

Hi Gurus, I have a file(weblog) as below abc|xyz|123|agentcode=sample code abcdeeess,agentcode=sample code abcdeeess,agentcode=sample code abcdeeess|agentadd=abcd stereet 23343,agentadd=abcd stereet 23343 sss|wwq|999|agentcode=sample1 code wqwdeeess,gentcode=sample1 code... (4 Replies)
Discussion started by: ratheeshjulk
4 Replies

7. Shell Programming and Scripting

Find duplicate values in specific column and delete all the duplicate values

Dear folks I have a map file of around 54K lines and some of the values in the second column have the same value and I want to find them and delete all of the same values. I looked over duplicate commands but my case is not to keep one of the duplicate values. I want to remove all of the same... (4 Replies)
Discussion started by: sajmar
4 Replies

8. Shell Programming and Scripting

Filter duplicate records from csv file with condition on one column

I have csv file with 30, 40 columns Pasting just three column for problem description I want to filter record if column 1 matches CN or DN then, check for values in column 2 if column contain 1235, 1235 then in column 3 values must be sequence of 2345, 2345 and if column 2 contains 6789, 6789... (5 Replies)
Discussion started by: as7951
5 Replies

9. Shell Programming and Scripting

CSV File:Filter duplicate records from column1 & another column having unique record

Hi Experts, I have csv file with 30, 40 columns Pasting just 2 column for problem description. Need to print error if below combination is not present in file check for column-1 (DocumentNumber) and filter columns where value in DocumentNumber field is same. For all such rows, the field... (7 Replies)
Discussion started by: as7951
7 Replies
Num(3o)                                                            OCaml library                                                           Num(3o)

NAME
Num - Operation on arbitrary-precision numbers. Module Module Num Documentation Module Num : sig end Operation on arbitrary-precision numbers. Numbers (type num ) are arbitrary-precision rational numbers, plus the special elements 1/0 (infinity) and 0/0 (undefined). type num = | Int of int | Big_int of Big_int.big_int | Ratio of Ratio.ratio The type of numbers. === Arithmetic operations === val (+/) : num -> num -> num Same as Num.add_num . val add_num : num -> num -> num Addition val minus_num : num -> num Unary negation. val (-/) : num -> num -> num Same as Num.sub_num . val sub_num : num -> num -> num Subtraction val ( */ ) : num -> num -> num Same as Num.mult_num . val mult_num : num -> num -> num Multiplication val square_num : num -> num Squaring val (//) : num -> num -> num Same as Num.div_num . val div_num : num -> num -> num Division val quo_num : num -> num -> num Euclidean division: quotient. val mod_num : num -> num -> num Euclidean division: remainder. val ( **/ ) : num -> num -> num Same as Num.power_num . val power_num : num -> num -> num Exponentiation val abs_num : num -> num Absolute value. val succ_num : num -> num succ n is n+1 val pred_num : num -> num pred n is n-1 val incr_num : num Pervasives.ref -> unit incr r is r:=!r+1 , where r is a reference to a number. val decr_num : num Pervasives.ref -> unit decr r is r:=!r-1 , where r is a reference to a number. val is_integer_num : num -> bool Test if a number is an integer === The four following functions approximate a number by an integer : === val integer_num : num -> num integer_num n returns the integer closest to n . In case of ties, rounds towards zero. val floor_num : num -> num floor_num n returns the largest integer smaller or equal to n . val round_num : num -> num round_num n returns the integer closest to n . In case of ties, rounds off zero. val ceiling_num : num -> num ceiling_num n returns the smallest integer bigger or equal to n . val sign_num : num -> int Return -1 , 0 or 1 according to the sign of the argument. === Comparisons between numbers === val (=/) : num -> num -> bool val (</) : num -> num -> bool val (>/) : num -> num -> bool val (<=/) : num -> num -> bool val (>=/) : num -> num -> bool val (<>/) : num -> num -> bool val eq_num : num -> num -> bool val lt_num : num -> num -> bool val le_num : num -> num -> bool val gt_num : num -> num -> bool val ge_num : num -> num -> bool val compare_num : num -> num -> int Return -1 , 0 or 1 if the first argument is less than, equal to, or greater than the second argument. val max_num : num -> num -> num Return the greater of the two arguments. val min_num : num -> num -> num Return the smaller of the two arguments. === Coercions with strings === val string_of_num : num -> string Convert a number to a string, using fractional notation. val approx_num_fix : int -> num -> string See Num.approx_num_exp . val approx_num_exp : int -> num -> string Approximate a number by a decimal. The first argument is the required precision. The second argument is the number to approximate. Num.approx_num_fix uses decimal notation; the first argument is the number of digits after the decimal point. approx_num_exp uses scien- tific (exponential) notation; the first argument is the number of digits in the mantissa. val num_of_string : string -> num Convert a string to a number. === Coercions between numerical types === val int_of_num : num -> int val num_of_int : int -> num val nat_of_num : num -> Nat.nat val num_of_nat : Nat.nat -> num val num_of_big_int : Big_int.big_int -> num val big_int_of_num : num -> Big_int.big_int val ratio_of_num : num -> Ratio.ratio val num_of_ratio : Ratio.ratio -> num val float_of_num : num -> float OCamldoc 2012-06-26 Num(3o)
All times are GMT -4. The time now is 01:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy