Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Select unique names while removing the duplicates from a column Post 303044284 by jim mcnamara on Monday 17th of February 2020 11:38:50 PM
Old 02-18-2020
I think you need to specify a field separator as a comma.

Code:
Owner@Owner-PC ~
$ awk -F, '!a[$1]++' filename
ENSG00000003137,ENST00000001146
ENSG00000003402,ENST00000309955


Owner@Owner-PC ~
$ awk  '!a[$1]++' filename
ENSG00000003137,ENST00000001146
ENSG00000003137,ENST00000412253
ENSG00000003402,ENST00000309955
ENSG00000003402,ENST00000443227
ENSG00000003402,ENST00000341222

I used the sample data
These 2 Users Gave Thanks to jim mcnamara For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

select unique values from duplicates in linux

I have values in the variable as so the for loop just fetches one by one params=$'$HEW_SRC_DATABASE_LIB\nprmAttunityUser\nprmAttunityPwd\nprmODBCDataSource\nprmLoadInd\nprmSrc_Lib_ATM\nprmODBCDataSource_ATM' and i have a grep command like this ret=`grep \$y $pf` ... (0 Replies)
Discussion started by: vee_789
0 Replies

2. Shell Programming and Scripting

Change unique file names into new unique filenames

I have 84 files with the following names splitseqs.1, spliseqs.2 etc. and I want to change the .number to a unique filename. E.g. change splitseqs.1 into splitseqs.7114_1#24 and change spliseqs.2 into splitseqs.7067_2#4 So all the current file names are unique, so are the new file names.... (1 Reply)
Discussion started by: avonm
1 Replies

3. Shell Programming and Scripting

Select Unique Value

HOW CAN I SELECT AN UNIQUE STRING FROM A FIELD? ACTUALLY I WANT TO PRINT RECORDS THAT 2ND FIELD OF THAT HAVE ONE CHARACTER AND IT MUST BE "P" AWK '$2~"" {PRINT $0}' IN > OUTBUT THIS CODE PRINT ALL RECORDS WHICH 2ND FIELDS OF THEM START WITH "P" AND MAY CONTAINS ANOTHER CHARACTER! (1 Reply)
Discussion started by: saeed.soltani
1 Replies

4. Shell Programming and Scripting

Print unique names in each row of a specific column using awk

Is it possible to remove redundant names in the 4th column? input cqWE 100 200 singapore;singapore AZO 300 400 brazil;america;germany;ireland;germany .... .... output cqWE 100 200 singapore AZO 300 400 brazil;america;germany;ireland (4 Replies)
Discussion started by: quincyjones
4 Replies

5. Shell Programming and Scripting

Print unique names in a specific column using awk

Is it possible to modify file like this. 1. Remove all the duplicate names in a define column i.e 4th col 2. Count the no.of unique names separated by ";" and print as a 5th col thanx in advance!! Q input c1 30 3 Eh2 c10 96 3 Frp c41 396 3 Ua5;Lop;Kol;Kol c62 2 30 Fmp;Fmp;Fmp ... (5 Replies)
Discussion started by: quincyjones
5 Replies

6. Programming

Query to SELECT only Column Names that Contain a Specific String?

Hey Guys, I'm using SQuirreL SQL v3.5 GUI to fetch some data that I need for something I'm working on. I'm also using the IBM Informix Driver (*Version 3.5) to connect to the Database. What I want to do, if it's even possible, is to show all COLUMNS if they contain the word "Email". So in... (2 Replies)
Discussion started by: mrm5102
2 Replies

7. UNIX for Dummies Questions & Answers

Removing rows that contain non-unique column entry

Background: I have a file of thousands of potential SSR primers from Batch Primer 3. I can't use primers that will contain the same sequence ID or sequence as another primer. I have some basic shell scripting skills, but not enough to handle this. What you need to know: I need to remove the... (1 Reply)
Discussion started by: msatseqs
1 Replies

8. Shell Programming and Scripting

Removing duplicates on a single "column" (delimited file)

Hello ! I'm quite new to linux but haven't found a script to do this task, unfortunately my knowledge is quite limited on shellscripts... Could you guys help me removing the duplicate lines of a file, based only on a single "column"? For example: M202034357;01/2008;J30RJ021;Ciclo 01... (4 Replies)
Discussion started by: Rufinofr
4 Replies

9. Shell Programming and Scripting

Count occurrence of column one unique value having unique second column value

Hello Team, I need your help on the following: My input file a.txt is as below: 3330690|373846|108471 3330690|373846|108471 0640829|459725|100001 0640829|459725|100001 3330690|373847|108471 Here row 1 and row 2 of column 1 are identical but corresponding column 2 value are... (4 Replies)
Discussion started by: angshuman
4 Replies

10. Shell Programming and Scripting

Duplicates and unique segregation

Hi All, I have multiple files and i need to segregate unique and duplicates into files. Eg: /source/ -- path abc_12092016.csv abc_11092016.csv abc_12092016.csv ID,NAME,NUMBER 1,XYZ,1234 2,SDF,3456 1,XYZ,1234 abc_11092016.csv 4,RTY,7890 6,WER,5678 8,YUI,0987 6,WER,5678 in the... (1 Reply)
Discussion started by: ajayr1982
1 Replies
S3MKBUCKET(1p)						User Contributed Perl Documentation					    S3MKBUCKET(1p)

NAME
s3mkbucket - Create Amazon AWS S3 buckets SYNOPSIS
s3mkbucket [options] [bucket ...] Options: --access-key AWS Access Key ID --secret-key AWS Secret Access Key --acl-short private|public-read|public-read-write|authenticated-read Environment: AWS_ACCESS_KEY_ID AWS_ACCESS_KEY_SECRET OPTIONS
--help Print a brief help message and exits. --man Prints the manual page and exits. --verbose Print a message for each created bucket. --access-key and --secret-key Specify the "AWS Access Key Identifiers" for the AWS account. --access-key is the "Access Key ID", and --secret-key is the "Secret Access Key". These are effectively the "username" and "password" to the AWS account, and should be kept confidential. The access keys MUST be specified, either via these command line parameters, or via the AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY_SECRET environment variables. Specifying them on the command line overrides the environment variables. --secure Uses SSL/TLS HTTPS to communicate with the AWS service, instead of HTTP. --acl-short Apply a "canned ACL" to the bucket when it is created. To set a more complex ACL, use the "s3acl" tool after the bucket is created. The following canned ACLs are currently defined by S3: private Owner gets "FULL_CONTROL". No one else has any access rights. This is the default. public-read Owner gets "FULL_CONTROL". The anonymous principal is granted "READ" access. public-read-write Owner gets "FULL_CONTROL". The anonymous principal is granted "READ" and "WRITE" access. This is a useful policy to apply to a bucket, if you intend for any anonymous user to PUT objects into the bucket. authenticated-read Owner gets "FULL_CONTROL" . Any principal authenticated as a registered Amazon S3 user is granted "READ" access. bucket One or more bucket names. As many as possible will be created. A user may have no more than 100 buckets. Bucket names must be between 3 and 255 characters long, and can only contain alphanumeric characters, underscore, period, and dash. Bucket names are case sensitive. Buckets with names containing uppercase characters or underscores are not accessible using the virtual hosting method. Buckets are unique in a global namespace. That means if someone has created a bucket with a given name, someone else cannot create another bucket with the same name. If a bucket name begins with one or more dashes, it might be mistaken for a command line option. If this is the case, separate the command line options from the bucket names with two dashes, like so: s3mkbucket --verbose -- --bucketname ENVIRONMENT VARIABLES
AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY_SECRET Specify the "AWS Access Key Identifiers" for the AWS account. AWS_ACCESS_KEY_ID contains the "Access Key ID", and AWS_ACCESS_KEY_SECRET contains the "Secret Access Key". These are effectively the "username" and "password" to the AWS service, and should be kept confidential. The access keys MUST be specified, either via these environment variables, or via the --access-key and --secret-key command line parameters. If the command line parameters are set, they override these environment variables. CONFIGURATION FILE
The configuration options will be read from the file "~/.s3-tools" if it exists. The format is the same as the command line options with one option per line. For example, the file could contain: --access-key <AWS access key> --secret-key <AWS secret key> --secure This example configuration file would specify the AWS access keys and that a secure connection using HTTPS should be used for all communications. DESCRIPTION
Create buckets in the Amazon Simple Storage Service (S3). BUGS
Report bugs to Mark Atwood mark@fallenpegasus.com. Making a bucket that already exists and is owned by the user does not fail. It is unclear whether this is a bug or not. Occasionally the S3 service will randomly fail for no externally apparent reason. When that happens, this tool should retry, with a delay and a backoff. Access to the S3 service can be authenticated with a X.509 certificate, instead of via the "AWS Access Key Identifiers". This tool should support that. It might be useful to be able to specify the "AWS Access Key Identifiers" in the user's "~/.netrc" file. This tool should support that. Errors and warnings are very "Perl-ish", and can be confusing. AUTHOR
Written by Mark Atwood mark@fallenpegasus.com. Many thanks to Wotan LLC <http://wotanllc.com>, for supporting the development of these S3 tools. Many thanks to the Amazon AWS engineers for developing S3. SEE ALSO
These tools use the Net::Amazon:S3 Perl module. The Amazon Simple Storage Service (S3) is documented at <http://aws.amazon.com/s3>. perl v5.10.0 2009-03-08 S3MKBUCKET(1p)
All times are GMT -4. The time now is 05:37 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy