Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk to print unique text in field

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk to print unique text in field
# 1  
Old 10-08-2015
awk to print unique text in field
I am trying to use awk to print the unique entries in $2

So in the example below there are 3 lines but 2 of the lines match in $2 so only one is used in the output.


File.txt
Code:
chr17:29667512-29667673 NF1:exon.1;NF1:exon.2;NF1:exon.38;NF1:exon.4;NF1:exon.46;NF1:exon.47 703.807
chr16:89877104-89877220 FANCA:exon.4;FANCA:exon.5 159.284
chr16:89838075-89838232 FANCA:exon.23;FANCA:exon.4 583.497

Desired output
Code:
NF1
FANCA

Code:
awk '!seen[$2]++ {lines[i++]=$2}      END {for (i in lines) if (seen[lines[i]]==1) print lines[i]}' file.txt > output.txt

is close but uses the entire line not the unique text

Current output
Code:
NF1:exon.14;NF1:exon.16;NF1:exon.8
NF1:exon.13;NF1:exon.22;NF1:exon.28;NF1:exon.30;NF1:exon.4
CTC1:exon.1;CTC1:exon.16;CTC1:exon.2;CTC1:exon.3;CTC1:exon.5
CTC1:exon.1;CTC1:exon.10;CTC1:exon.2;CTC1:exon.3

Desired output
Code:
NF1
CTC1

Thank you Smilie
# 2  
Old 10-08-2015
You could try something like this:
Code:
awk '{n=split($2,F,/[:;]/); for(i=1; i<=n; i+=2) A[F[i]]} END{for(i in A) print i}' file

The idea is that you only use the array for the fields that you want to find. The fields are a subset $2 and here split is used to break it down further.

-- Edit --
It could be shortened so that the end section can be left out:
Code:
awk '{n=split($2,F,/[:;]/); for(i=1; i<=n; i+=2) if(!A[F[i]]++) print F[i]}' file

--
Without the split() function, using the field separator to split the fields:
Code:
awk -F'[:; ]' '{for(i=3; i<=NF-2; i+=2) if(!A[$i]++) print $i}' file

Last edited by Scrutinizer; 10-08-2015 at 03:17 PM..
This User Gave Thanks to Scrutinizer For This Post:
cmccabe
# 3  
Old 10-08-2015
Hi.

Also with the usual commands:
Code:
cut -f2 -d" " file |
tee f1 |
cut -f1 -d":" |
tee f2 |
sort -u

producing:
Code:
FANCA
NF1

on a system like:
Code:
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian 5.0.8 (lenny, workstation) 
cut (GNU coreutils) 6.10
sort (GNU coreutils) 6.10

See files f1,f2 for intermediate output.

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
cmccabe
# 4  
Old 10-08-2015
@drl, that works if there is always only one kind of label per line...
This User Gave Thanks to Scrutinizer For This Post:
cmccabe
# 5  
Old 10-08-2015
Hi, Scrutinizer.
Quote:
Originally Posted by Scrutinizer
@drl, that works if there is always only one kind of label per line...
Yes, I see what you mean, thanks; however, more kinds of labels were not presented in the sample data. If I have time, I'll think about a solution for that ... cheers, drl

---------- Post updated at 15:20 ---------- Previous update was at 15:10 ----------

Hi.

To correct possible flaw noted by Scrutinizer:
Code:
#!/usr/bin/env bash

# @(#) s2	Demonstrate extract unique string from specific field.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C cut tr sort

FILE=${1-data2}

pl " Input data file $FILE:"
cat $FILE

pl " Results:"
cut -f2 -d" " $FILE |
tee f1 |
tr ';' '\n' |
tee f2 |
cut -f1 -d":" |
tee f3 |
sort -u

exit 0

produciing:
Code:
$ ./s2

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian 5.0.8 (lenny, workstation) 
bash GNU bash 3.2.39
cut (GNU coreutils) 6.10
tr (GNU coreutils) 6.10
sort (GNU coreutils) 6.10

-----
 Input data file data2:
chr17:29667512-29667673 NF1:exon.1;NF1:exon.2;NF1:exon.38;NF1:exon.4;NF1:exon.46;NF1:exon.47 703.807
chr16:89877104-89877220 FANCA:exon.4;FANCA:exon.5 159.284
chr16:89838075-89838232 FANCA:exon.23;FANCA:exon.4 583.497
chr18:89838075-89838232 DARK:exon.23;FANCA:exon.4 583.497
chr19:89838075-89838232 PARK:exon.23;DARK:exon.4 583.497

-----
 Results:
DARK
FANCA
NF1
PARK

Best wishes ... cheers, drl
This User Gave Thanks to drl For This Post:
cmccabe
# 6  
Old 10-09-2015
Thank you both Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Awk: count unique elements in a field and sum their occurence across the entire file

Hi, Sure it's an easy one, but it drives me insane. input ("|" separated): 1|A,B,C,A 2|A,D,D 3|A,B,B I would like to count the occurence of each capital letters in $2 across the entire file, knowing that duplicates in each record count as 1. I am trying to get this output... (5 Replies)
Discussion started by: beca123456
5 Replies

2. Shell Programming and Scripting

awk to print text in field if match and range is met

In the awk below I am trying to match the value in $4 of file1 with the split value from $4 in file2. I store the value of $4 in file1 in A and the split value (using the _ for the split) in array. I then strore the value in $2 as min, the value in $3 as max, and the value in $1 as chr. If A is... (6 Replies)
Discussion started by: cmccabe
6 Replies

3. UNIX for Beginners Questions & Answers

Print lines based upon unique values in Nth field

For some reason I am having difficulty performing what should be a fairly easy task. I would like to print lines of a file that have a unique value in the first field. For example, I have a large data-set with the following excerpt: PS003,001 MZMWR/ L-DWD// * PS003,001... (4 Replies)
Discussion started by: jvoot
4 Replies

4. Shell Programming and Scripting

awk to print lines based on text in field and value in two additional fields

In the awk below I am trying to print the entire line, along with the header row, if $2 is SNV or MNV or INDEL. If that condition is met or is true, and $3 is less than or equal to 0.05, then in $7 the sub pattern :GMAF= is found and the value after the = sign is checked. If that value is less than... (0 Replies)
Discussion started by: cmccabe
0 Replies

5. Shell Programming and Scripting

awk to print unique text in field before hyphen

Trying to print the unique values in $2 before the -, currently the count is displayed. Hopefully, the below is close. Thank you :). file chr2:46603668-46603902 EPAS1-902|gc=54.3 253.1 chr2:211471445-211471675 CPS1-1205|gc=48.3 264.7 chr19:15291762-15291983 NOTCH3-1003|gc=68.8 195.8... (3 Replies)
Discussion started by: cmccabe
3 Replies

6. Shell Programming and Scripting

awk to parse field and include the text of 1 pipe in field 4

I am trying to parse the input in awk to include the |gc= in $4 but am not able to. The below is close: awk so far: awk '{sub(/\|]+]++/, ""); print }' input.txt Input chr1 955543 955763 AGRN-6|pr=2|gc=75 0 + chr1 957571 957852 AGRN-7|pr=3|gc=61.2 0 + chr1 970621 ... (7 Replies)
Discussion started by: cmccabe
7 Replies

7. Shell Programming and Scripting

Print unique names in a specific column using awk

Is it possible to modify file like this. 1. Remove all the duplicate names in a define column i.e 4th col 2. Count the no.of unique names separated by ";" and print as a 5th col thanx in advance!! Q input c1 30 3 Eh2 c10 96 3 Frp c41 396 3 Ua5;Lop;Kol;Kol c62 2 30 Fmp;Fmp;Fmp ... (5 Replies)
Discussion started by: quincyjones
5 Replies

8. Shell Programming and Scripting

Print unique records in 2 columns using awk

Is it possible to print the records that has only 1 value in 2nd column. Ex: input awex1 1 awex1 2 awex1 3 assww 1 ader34 1 ader34 2 output assww 1 (5 Replies)
Discussion started by: quincyjones
5 Replies

9. Shell Programming and Scripting

Compare Tab Separated Field with AWK to all and print lines of unique fields.

Hi. I have a tab separated file that has a couple nearly identical lines. When doing: sort file | uniq > file.new It passes through the nearly identical lines because, well, they still are unique. a) I want to look only at field x for uniqueness and if the content in field x is the... (1 Reply)
Discussion started by: rocket_dog
1 Replies

10. Shell Programming and Scripting

awk, comma as field separator and text inside double quotes as a field.

Hi, all I need to get fields in a line that are separated by commas, some of the fields are enclosed with double quotes, and they are supposed to be treated as a single field even if there are commas inside the quotes. sample input: for this line, 5 fields are supposed to be extracted, they... (8 Replies)
Discussion started by: kevintse
8 Replies
Login or Register to Ask a Question