Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Select unique names while removing the duplicates from a column Post 303044290 by MadeInGermany on Tuesday 18th of February 2020 04:30:29 AM
Old 02-18-2020
Code:
awk -F, 'a[$1]++==0' filename

is quick and dirty because it stores an unnecessary integer value.
The full and efficient code is
Code:
awk -F, '!($1 in a) { a[$1]; print }' filename

That you can condense again to an implicit print
Code:
awk -F, '!(($1 in a) || a[$1])' filename

or
Code:
awk -F, '!($1 in a) && !a[$1]' filename

This User Gave Thanks to MadeInGermany For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

select unique values from duplicates in linux

I have values in the variable as so the for loop just fetches one by one params=$'$HEW_SRC_DATABASE_LIB\nprmAttunityUser\nprmAttunityPwd\nprmODBCDataSource\nprmLoadInd\nprmSrc_Lib_ATM\nprmODBCDataSource_ATM' and i have a grep command like this ret=`grep \$y $pf` ... (0 Replies)
Discussion started by: vee_789
0 Replies

2. Shell Programming and Scripting

Change unique file names into new unique filenames

I have 84 files with the following names splitseqs.1, spliseqs.2 etc. and I want to change the .number to a unique filename. E.g. change splitseqs.1 into splitseqs.7114_1#24 and change spliseqs.2 into splitseqs.7067_2#4 So all the current file names are unique, so are the new file names.... (1 Reply)
Discussion started by: avonm
1 Replies

3. Shell Programming and Scripting

Select Unique Value

HOW CAN I SELECT AN UNIQUE STRING FROM A FIELD? ACTUALLY I WANT TO PRINT RECORDS THAT 2ND FIELD OF THAT HAVE ONE CHARACTER AND IT MUST BE "P" AWK '$2~"" {PRINT $0}' IN > OUTBUT THIS CODE PRINT ALL RECORDS WHICH 2ND FIELDS OF THEM START WITH "P" AND MAY CONTAINS ANOTHER CHARACTER! (1 Reply)
Discussion started by: saeed.soltani
1 Replies

4. Shell Programming and Scripting

Print unique names in each row of a specific column using awk

Is it possible to remove redundant names in the 4th column? input cqWE 100 200 singapore;singapore AZO 300 400 brazil;america;germany;ireland;germany .... .... output cqWE 100 200 singapore AZO 300 400 brazil;america;germany;ireland (4 Replies)
Discussion started by: quincyjones
4 Replies

5. Shell Programming and Scripting

Print unique names in a specific column using awk

Is it possible to modify file like this. 1. Remove all the duplicate names in a define column i.e 4th col 2. Count the no.of unique names separated by ";" and print as a 5th col thanx in advance!! Q input c1 30 3 Eh2 c10 96 3 Frp c41 396 3 Ua5;Lop;Kol;Kol c62 2 30 Fmp;Fmp;Fmp ... (5 Replies)
Discussion started by: quincyjones
5 Replies

6. Programming

Query to SELECT only Column Names that Contain a Specific String?

Hey Guys, I'm using SQuirreL SQL v3.5 GUI to fetch some data that I need for something I'm working on. I'm also using the IBM Informix Driver (*Version 3.5) to connect to the Database. What I want to do, if it's even possible, is to show all COLUMNS if they contain the word "Email". So in... (2 Replies)
Discussion started by: mrm5102
2 Replies

7. UNIX for Dummies Questions & Answers

Removing rows that contain non-unique column entry

Background: I have a file of thousands of potential SSR primers from Batch Primer 3. I can't use primers that will contain the same sequence ID or sequence as another primer. I have some basic shell scripting skills, but not enough to handle this. What you need to know: I need to remove the... (1 Reply)
Discussion started by: msatseqs
1 Replies

8. Shell Programming and Scripting

Removing duplicates on a single "column" (delimited file)

Hello ! I'm quite new to linux but haven't found a script to do this task, unfortunately my knowledge is quite limited on shellscripts... Could you guys help me removing the duplicate lines of a file, based only on a single "column"? For example: M202034357;01/2008;J30RJ021;Ciclo 01... (4 Replies)
Discussion started by: Rufinofr
4 Replies

9. Shell Programming and Scripting

Count occurrence of column one unique value having unique second column value

Hello Team, I need your help on the following: My input file a.txt is as below: 3330690|373846|108471 3330690|373846|108471 0640829|459725|100001 0640829|459725|100001 3330690|373847|108471 Here row 1 and row 2 of column 1 are identical but corresponding column 2 value are... (4 Replies)
Discussion started by: angshuman
4 Replies

10. Shell Programming and Scripting

Duplicates and unique segregation

Hi All, I have multiple files and i need to segregate unique and duplicates into files. Eg: /source/ -- path abc_12092016.csv abc_11092016.csv abc_12092016.csv ID,NAME,NUMBER 1,XYZ,1234 2,SDF,3456 1,XYZ,1234 abc_11092016.csv 4,RTY,7890 6,WER,5678 8,YUI,0987 6,WER,5678 in the... (1 Reply)
Discussion started by: ajayr1982
1 Replies
COMM(1) 							   User Commands							   COMM(1)

NAME
comm - compare two sorted files line by line SYNOPSIS
comm [OPTION]... FILE1 FILE2 DESCRIPTION
Compare sorted files FILE1 and FILE2 line by line. When FILE1 or FILE2 (not both) is -, read standard input. With no options, produce three-column output. Column one contains lines unique to FILE1, column two contains lines unique to FILE2, and column three contains lines common to both files. -1 suppress column 1 (lines unique to FILE1) -2 suppress column 2 (lines unique to FILE2) -3 suppress column 3 (lines that appear in both files) --check-order check that the input is correctly sorted, even if all input lines are pairable --nocheck-order do not check that the input is correctly sorted --output-delimiter=STR separate columns with STR --total output a summary -z, --zero-terminated line delimiter is NUL, not newline --help display this help and exit --version output version information and exit Note, comparisons honor the rules specified by 'LC_COLLATE'. EXAMPLES
comm -12 file1 file2 Print only lines present in both file1 and file2. comm -3 file1 file2 Print lines in file1 not in file2, and vice versa. AUTHOR
Written by Richard M. Stallman and David MacKenzie. REPORTING BUGS
GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Report comm translation bugs to <http://translationproject.org/team/> COPYRIGHT
Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. SEE ALSO
join(1), uniq(1) Full documentation at: <http://www.gnu.org/software/coreutils/comm> or available locally via: info '(coreutils) comm invocation' GNU coreutils 8.28 January 2018 COMM(1)
All times are GMT -4. The time now is 04:20 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy