Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Select unique names while removing the duplicates from a column Post 303044284 by jim mcnamara on Monday 17th of February 2020 11:38:50 PM
Old 02-18-2020
I think you need to specify a field separator as a comma.

Code:
Owner@Owner-PC ~
$ awk -F, '!a[$1]++' filename
ENSG00000003137,ENST00000001146
ENSG00000003402,ENST00000309955


Owner@Owner-PC ~
$ awk  '!a[$1]++' filename
ENSG00000003137,ENST00000001146
ENSG00000003137,ENST00000412253
ENSG00000003402,ENST00000309955
ENSG00000003402,ENST00000443227
ENSG00000003402,ENST00000341222

I used the sample data
These 2 Users Gave Thanks to jim mcnamara For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

select unique values from duplicates in linux

I have values in the variable as so the for loop just fetches one by one params=$'$HEW_SRC_DATABASE_LIB\nprmAttunityUser\nprmAttunityPwd\nprmODBCDataSource\nprmLoadInd\nprmSrc_Lib_ATM\nprmODBCDataSource_ATM' and i have a grep command like this ret=`grep \$y $pf` ... (0 Replies)
Discussion started by: vee_789
0 Replies

2. Shell Programming and Scripting

Change unique file names into new unique filenames

I have 84 files with the following names splitseqs.1, spliseqs.2 etc. and I want to change the .number to a unique filename. E.g. change splitseqs.1 into splitseqs.7114_1#24 and change spliseqs.2 into splitseqs.7067_2#4 So all the current file names are unique, so are the new file names.... (1 Reply)
Discussion started by: avonm
1 Replies

3. Shell Programming and Scripting

Select Unique Value

HOW CAN I SELECT AN UNIQUE STRING FROM A FIELD? ACTUALLY I WANT TO PRINT RECORDS THAT 2ND FIELD OF THAT HAVE ONE CHARACTER AND IT MUST BE "P" AWK '$2~"" {PRINT $0}' IN > OUTBUT THIS CODE PRINT ALL RECORDS WHICH 2ND FIELDS OF THEM START WITH "P" AND MAY CONTAINS ANOTHER CHARACTER! (1 Reply)
Discussion started by: saeed.soltani
1 Replies

4. Shell Programming and Scripting

Print unique names in each row of a specific column using awk

Is it possible to remove redundant names in the 4th column? input cqWE 100 200 singapore;singapore AZO 300 400 brazil;america;germany;ireland;germany .... .... output cqWE 100 200 singapore AZO 300 400 brazil;america;germany;ireland (4 Replies)
Discussion started by: quincyjones
4 Replies

5. Shell Programming and Scripting

Print unique names in a specific column using awk

Is it possible to modify file like this. 1. Remove all the duplicate names in a define column i.e 4th col 2. Count the no.of unique names separated by ";" and print as a 5th col thanx in advance!! Q input c1 30 3 Eh2 c10 96 3 Frp c41 396 3 Ua5;Lop;Kol;Kol c62 2 30 Fmp;Fmp;Fmp ... (5 Replies)
Discussion started by: quincyjones
5 Replies

6. Programming

Query to SELECT only Column Names that Contain a Specific String?

Hey Guys, I'm using SQuirreL SQL v3.5 GUI to fetch some data that I need for something I'm working on. I'm also using the IBM Informix Driver (*Version 3.5) to connect to the Database. What I want to do, if it's even possible, is to show all COLUMNS if they contain the word "Email". So in... (2 Replies)
Discussion started by: mrm5102
2 Replies

7. UNIX for Dummies Questions & Answers

Removing rows that contain non-unique column entry

Background: I have a file of thousands of potential SSR primers from Batch Primer 3. I can't use primers that will contain the same sequence ID or sequence as another primer. I have some basic shell scripting skills, but not enough to handle this. What you need to know: I need to remove the... (1 Reply)
Discussion started by: msatseqs
1 Replies

8. Shell Programming and Scripting

Removing duplicates on a single "column" (delimited file)

Hello ! I'm quite new to linux but haven't found a script to do this task, unfortunately my knowledge is quite limited on shellscripts... Could you guys help me removing the duplicate lines of a file, based only on a single "column"? For example: M202034357;01/2008;J30RJ021;Ciclo 01... (4 Replies)
Discussion started by: Rufinofr
4 Replies

9. Shell Programming and Scripting

Count occurrence of column one unique value having unique second column value

Hello Team, I need your help on the following: My input file a.txt is as below: 3330690|373846|108471 3330690|373846|108471 0640829|459725|100001 0640829|459725|100001 3330690|373847|108471 Here row 1 and row 2 of column 1 are identical but corresponding column 2 value are... (4 Replies)
Discussion started by: angshuman
4 Replies

10. Shell Programming and Scripting

Duplicates and unique segregation

Hi All, I have multiple files and i need to segregate unique and duplicates into files. Eg: /source/ -- path abc_12092016.csv abc_11092016.csv abc_12092016.csv ID,NAME,NUMBER 1,XYZ,1234 2,SDF,3456 1,XYZ,1234 abc_11092016.csv 4,RTY,7890 6,WER,5678 8,YUI,0987 6,WER,5678 in the... (1 Reply)
Discussion started by: ajayr1982
1 Replies
SHMCTL(2)						      BSD System Calls Manual							 SHMCTL(2)

NAME
shmctl -- shared memory control operations SYNOPSIS
#include <sys/shm.h> int shmctl(int shmid, int cmd, struct shmid_ds *buf); DESCRIPTION
The shmctl() system call performs some control operations on the shared memory area specified by shmid. Each shared memory segment has a data structure associated with it, parts of which may be altered by shmctl() and parts of which determine the actions of shmctl(). This structure is defined as follows in <sys/shm.h>: struct shmid_ds { struct ipc_perm shm_perm; /* operation permissions */ int shm_segsz; /* size of segment in bytes */ pid_t shm_lpid; /* pid of last shm op */ pid_t shm_cpid; /* pid of creator */ short shm_nattch; /* # of current attaches */ time_t shm_atime; /* last shmat() time*/ time_t shm_dtime; /* last shmdt() time */ time_t shm_ctime; /* last change by shmctl() */ void *shm_internal; /* sysv stupidity */ }; The ipc_perm structure used inside the shmid_ds structure is defined in <sys/ipc.h> and looks like this: struct ipc_perm { uid_t uid; /* Owner's user ID */ gid_t gid; /* Owner's group ID */ uid_t cuid; /* Creator's user ID */ gid_t cgid; /* Creator's group ID */ mode_t mode; /* r/w permission (see chmod(2)) */ unsigned short _seq; /* Reserved for internal use */ key_t _key; /* Reserved for internal use */ }; The operation to be performed by shmctl() is specified in cmd and is one of: IPC_STAT Gather information about the shared memory segment and place it in the structure pointed to by buf. IPC_SET Set the value of the shm_perm.uid, shm_perm.gid and shm_perm.mode fields in the structure associated with shmid. The values are taken from the corresponding fields in the structure pointed to by buf. This operation can only be executed by the super-user, or a process that has an effective user id equal to either shm_perm.cuid or shm_perm.uid in the data structure associated with the shared memory segment. IPC_RMID Remove the shared memory segment specified by shmid and destroy the data associated with it. Only the super-user or a process with an effective uid equal to the shm_perm.cuid or shm_perm.uid values in the data structure associated with the queue can do this. The read and write permissions on a shared memory identifier are determined by the shm_perm.mode field in the same way as is done with files (see chmod(2) ), but the effective uid can match either the shm_perm.cuid field or the shm_perm.uid field, and the effective gid can match either shm_perm.cgid or shm_perm.gid. RETURN VALUES
Upon successful completion, a value of 0 is returned. Otherwise, -1 is returned and the global variable errno is set to indicate the error. ERRORS
shmctl() will fail if: [EACCES] The command is IPC_STAT and the caller has no read permission for this shared memory segment. [EFAULT] buf specifies an invalid address. [EINVAL] shmid is not a valid shared memory segment identifier. cmd is not a valid command. [EPERM] cmd is equal to IPC_SET or IPC_RMID and the caller is not the super-user,nor does the effective uid match either the shm_perm.uid or shm_perm.cuid fields of the data structure associated with the shared memory segment. An attempt is made to increase the value of shm_qbytes through IPC_SET but the caller is not the super-user. LEGACY SYNOPSIS
#include <sys/types.h> #include <sys/ipc.h> #include <sys/shm.h> All of these include files are necessary. LEGACY DESCRIPTION
The ipc_perm structure used inside the shmid_ds structure, as defined in <sys/ipc.h>, looks like this: struct ipc_perm { __uint16_t cuid; /* Creator's user id */ __uint16_t cgid; /* Creator's group id */ __uint16_t uid; /* Owner's user id */ __uint16_t gid; /* Owner's group id */ mode_t mode; /* r/w permission (see chmod(2)) */ __uint16_t seq; /* Reserved for internal use */ key_t key; /* Reserved for internal use */ }; This structure is maintained for binary backward compatibility with previous versions of the interface. New code should not use this inter- face, because ID values may be truncated. Specifically, LEGACY mode limits the allowable uid/gid ranges to 0-32767. If the user has a UID that is out of this range (e.g., "nobody"), software using the LEGACY API will not behave as expected. SEE ALSO
shmat(2), shmdt(2), shmget(2), compat(5) BSD
August 17, 1995 BSD
All times are GMT -4. The time now is 04:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy