Visit Our UNIX and Linux User Community


awk to add text to each line of matching id


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk to add text to each line of matching id
# 1  
Old 01-18-2019
awk to add text to each line of matching id

The awk below executes as expected if the id in $4 (like in f) is unique. However most of my data is like f1 where the same id can appear multiple times. I think that is the reason why the awk is not working as expected. I added a comment on the line that I can not change without causing the script to abort. Each line in f2 is searched and must contain the id, in this case COL1A2 but that id may not be a single entry. That is the id may appear 5 times, but each line with that is in f2 is searched. Using the $4 in f1 as the id and reading each $1, $2, and $3 value into a variable min and max.

The $4 is then split on the _ in f2 and read into array. The same id from f1 may appear in multiple lines of f2 however each will have unique $2 and $3 values. Each value in the split will match a $4 id in f1. The min and max must match the $1 of f2 and be between the $2 and $3 values in f2. An exact match is not needed rather just that the min or max variables falls within $2 and $3. If that is true then exon is printed in $5 of f2 if it is not true then intron is printed in $5. Most of this works as expected I just did not account for the possibity for multiple enteries and am nut sure how to adjust for it. Thank you Smilie

For example using the contents of the f1 where COL1A2 appears 3 times, each entry or line is searched in f2. Currently, I believe since COL1A2 is not unique not match is found in f2 as the min and max are not set per entry or line. Thank you Smilie.

awk w/ desired output
Code:
awk '
 BEGIN{
  SUBSEP=","
}
FNR==NR{
  max[$1,$NF]=$3
  min[$1,$NF]=$2
  next
}
{
 split($4,array,"_")   # How do I change/modify this so it only looks a each line with this id `COL1A2` in it?
}
(($1,array[1]) in max){
if(($2>min[array[5],array[1]] && $2<max[array[5],array[1]]) || ($3>max[array[5],array[1]] && $3<max[array[5],array[1]])){
  print array[5],array[1],min[array[5],array[1]],max[array[5],array[1]],"exon"
  next
}
}
{
  print $0,"intron"}' f f2 

chr7    94024333    94024423    COL1A2_cds_0_0_chr7_94024344_f  0   + intron
chr7    94027049    94027080    COL1A2_cds_1_0_chr7_94027060_f  0   + intron
chr7 COL1A2 94027591 94027701 exon

awk w/ current output
Code:
  .... }' f1 f2

chr7    94024333    94024423    COL1A2_cds_0_0_chr7_94024344_f  0   + intron
chr7    94027049    94027080    COL1A2_cds_1_0_chr7_94027060_f  0   + intron
chr7    94027683    94027718    COL1A2_cds_2_0_chr7_94027694_f  0   + intron

Code:
contents of f single COL1A2 entry

chr7    94027591    94027701    COL1A2

Code:
contents of f1  multiple COL1A2 entry, this is most of the actual data, very few are single entries though there are some

chr7    94027591    94027701    COL1A2  
chr7    94027799    94027811    COL1A2  
chr7    94030799    94030847    COL1A2

Code:
contents of f2 always the same format

chr7    94024333    94024423    COL1A2_cds_0_0_chr7_94024344_f  0   +
chr7    94027049    94027080    COL1A2_cds_1_0_chr7_94027060_f  0   +
chr7    94027683    94027718    COL1A2_cds_2_0_chr7_94027694_f  0   +

# 2  
Old 01-18-2019
Hi cmccabe,
If we rewrite the code that reads your first input file to be:
Code:
FNR==NR{
	max[$1,$NF,++count[$1,$NF]]=$3
	min[$1,$NF,count[$1,$NF]]=$2
	next
}

does that give you enough of a hint for what you then need to do in the loop you need to add in the code that reads your second input file?
This User Gave Thanks to Don Cragun For This Post:
# 3  
Old 01-19-2019
Thank you for the hint, I made two adjustments to the script and commented them. The output is the same but maybe I have the idea just not implementing it correctly? Thank you Smilie.

Code:
awk '
 BEGIN{
  SUBSEP=","
}
FNR==NR{
  max[$1,$NF,++count[$1,$NF]]=$3  # read with count each line of f1 max
  min[$1,$NF,count[$1,$NF]]=$2
  next
}
{ for (i in count)   # start a loop with setting each line in id to i
 split($4,array,"_")   
}
(($1,array[1],i++) in max){    # search each matching id line in f2
if(($2>min[array[5],array[1]] && $2<max[array[5],array[1]]) || ($3>max[array[5],array[1]] && $3<max[array[5],array[1]])){
  print array[5],array[1],min[array[5],array[1]],max[array[5],array[1]],"exon"
  next
  }
 }
 {
  print $0,"intron"}' f1 f2

chr7    94024333    94024423    COL1A2_cds_0_0_chr7_94024344_f  0   + intron
chr7    94027049    94027080    COL1A2_cds_1_0_chr7_94027060_f  0   + intron
chr7    94027683    94027718    COL1A2_cds_2_0_chr7_94027694_f  0   + intron

# 4  
Old 01-19-2019
Hi cmccabe,
If you create an array with three subscripts, you have to use three subscripts when you try to access an element of that array.

Are the ranges given in your first input file always in increasing numerical order for each $1,$4 set of values (as in your sample file f1)? If they are we can use that information to make your code run faster.

Is the fifth subfield of $4 in your second input file always identical to the $1 value on the same input line (as in your sample files)? If they are, we can use that information to make your code run faster.

You note that your input files fields are separated by tabs. Do you want the output file to be tab delimited too; or do you want the output to be delimited by spaces as shown in your sample output?

Note that in your original code (and in your updated code) you have the line:
Code:
if(($2>min[array[5],array[1]] && $2<max[array[5],array[1]]) || ($3>max[array[5],array[1]] && $3<max[array[5],array[1]])){

and a $3 value can never be less than a max[] value and greater than the same value. Can we assume that you intended to write:
Code:
if(($2>min[array[5],array[1]] && $2<max[array[5],array[1]]) || ($3>min[array[5],array[1]] && $3<max[array[5],array[1]])){

or more likely that you meant:
Code:
if(($2>=min[array[5],array[1]] && $2<=max[array[5],array[1]]) || ($3>=min[array[5],array[1]] && $3<=max[array[5],array[1]])){

(i.e., are the min-max ranges inclusive of the endpoints or exclusive of the endpoints)?
This User Gave Thanks to Don Cragun For This Post:
# 5  
Old 01-19-2019
Sorry... I missed one question concerning the line of code:
Code:
if(($2>min[array[5],array[1]] && $2<max[array[5],array[1]]) || ($3>max[array[5],array[1]] && $3<max[array[5],array[1]])){

Is it your intent to print the line containing exon if either endpoint is in an entry in the first input file for that $1,$4 pair, or should it only print the exon line if both endpoints are in range?
This User Gave Thanks to Don Cragun For This Post:
# 6  
Old 01-19-2019
Quote:
Are the ranges given in your first input file always in increasing numerical order for each $1,$4 set of values (as in your sample file f1)? If they are we can use that information to make your code run faster.
Yes, these should always be sorted like in f1

Quote:
Is the fifth subfield of $4 in your second input file always identical to the $1 value on the same input line (as in your sample files)? If they are, we can use that information to make your code run faster.
Yes, this will always be the case if $4 is found as in f1

Quote:
You note that your input files fields are separated by tabs. Do you want the output file to be tab delimited too; or do you want the output to be delimited by spaces as shown in your sample output?
f1 will always be tab-delimited except for a whitespace after $3 and $4, but the output would be tab-delimited I did and OFS="\t" but I think the whitespaces are making that not work

You are correct in that I meant to be looking for inclusive endpoints so the >=/<= is what I should have used.

Quote:
Is it your intent to print the line containing exon if either endpoint is in an entry in the first input file for that $1,$4 pair, or should it only print the exon line if both endpoints are in range?
I used the || statement to make sure the script works as expected but it could be && as both coordinates should lie within the endpoints (trying to think of a situation where its not the case and not coming up with anything).

Thank you very much Smilie.
# 7  
Old 01-19-2019
Quote:
Originally Posted by cmccabe
Yes, these should always be sorted like in f1



Yes, this will always be the case if $4 is found as in f1



f1 will always be tab-delimited except for a whitespace after $3 and $4, but the output would be tab-delimited I did and OFS="\t" but I think the whitespaces are making that not work

You are correct in that I meant to be looking for inclusive endpoints so the >=/<= is what I should have used.



I used the || statement to make sure the script works as expected but it could be && as both coordinates should lie within the endpoints (trying to think of a situation where its not the case and not coming up with anything).

Thank you very much Smilie.
Thanks for the responses.

Unfortunately, upon looking closer at your example input files, there are no entries in f1 where both endpoints of any line in f2 fall within the ranges specified in f1. In the last line of f2 $2 falls inside the range specified in the first line in f1 but $3 does not.

And, despite what you said about the input files being tab delimited, the samples you provided don't contain any <tab> characters. And, since you said that the real files you're using do contain some <space>s between field 3 and 4 and after field4, the following code assumes that strings of one or more <space> and/or <tab> characters separate field and that any <space> and <tab> characters after field 4 are to be ignored. (As written, the code shown below will not work if a line in either input file contains any leading <space> or <tab> characters.)

So, given the above and assuming that you just want there to be some overlap between the ranges specified in a line in f1 and in a line in f2, maybe the following will do what you want:
Code:
#!/bin/ksh
awk '
BEGIN{	FS = "[[:blank:]]+|_"
	OFS = "\t"
}
FNR == NR {
	# Note that this code assumes that the min and max ranges in the first
	# input file are presented with the minimum values arranged in
	# increasing order for each $1, $NF set of values.
	max[$1, $4, ++count[$1, $4]] = $3	# Save max value with count of
						# lines with the same $1 and $4
						# value from the first input
						# file.
	min[$1, $4, count[$1, $4]] = $2		# Save corresponding min value.
	next					# Skip to next input line.
}
{	for(i = 1; i <= count[$1, $4]; i++) {
		# If the minimum value on this line is greater than the saved
		# maximum value we are looking at now or the maximum value on
		# this line is less than thee save minimum value we are looking
		# at now, there cannot be a matching entry for this $1,$4 value
		# pair.
		if($2 > max[$1, $4, i] || $3 < min[$1, $4, i])
			break
		# If the minimum or maximum on this line is within range, we
		# have found what we are looking for.
		if(($2 >= min[$1, $4, i]) || $3 <= max[$1, $4, i]) {
			print $1, $4, min[$1, $4, i], max[$1, $4, i], "exon"
			next
		}
	}

	# No entry was found for this $1,$4 value pair.  Report "intron" found.
	print $0, "intron"
}' "${1:-f1}" f2

This uses <tab> as the output field separator, but on output lines that end in "intron", <space>s in the input will not be converted to <tab>s in the output.

If you run the above script with no operands (or with the operand f1 or with the operand f) from your sample input files, it will produce the output:
Code:
chr7    94024333    94024423    COL1A2_cds_0_0_chr7_94024344_f  0   +	intron
chr7    94027049    94027080    COL1A2_cds_1_0_chr7_94027060_f  0   +	intron
chr7	COL1A2	94027591	94027701	exon

Note that the above code does not set SUBSEP since it was not used in your script and isn't needed in the code shown above. Note also that the field separator I'm using the code above uses any sequence of one or more <space>s and/or <tab>s to treated as a field separator and uses every <underscore> as a field separator. That means that the subfields you were splitting into the array named array in your code will all be treated as separate fields in the code above. (That means I don't have to call split() to break that string into subfields.)

The sample files you provided to test any of the "corner" cases where I might have missed something. I think it will work OK, but I haven't performed enough extensive testing to give you any kind of guarantee.

Hope this helps,
Don
This User Gave Thanks to Don Cragun For This Post:

Previous Thread | Next Thread
Test Your Knowledge in Computers #576
Difficulty: Medium
All programming languages have automatic garbage collection that monitors the dynamically allocated pieces of memory and determine if any variable in the program still references it. If the memory is no longer referenced, it is 'garbage' and becomes eligible to be 'collected'.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk to update file with partial matching line in another file and append text

In the awk below I am trying to cp and paste each matching line in f2 to $3 in f1 if $2 of f1 is in the line in f2 somewhere. There will always be a match (usually more then 1) and my actual data is much larger (several hundreds of lines) in both f1 and f2. When the line in f2 is pasted to $3 in... (4 Replies)
Discussion started by: cmccabe
4 Replies

2. Shell Programming and Scripting

awk to add text to matching pattern in field

In the awk I am trying to add :p.=? to the end of each $9 that matches the pattern NM_. The below executes andis close but I can not seem to figure out why the :p.=? repeats in the split as in the green in the current output. I have added comments as well. Thank you :). file ... (4 Replies)
Discussion started by: cmccabe
4 Replies

3. Shell Programming and Scripting

awk to lookup stored variable in file and print matching line

The bash bash below extracts the oldest folder from a directory and stores it in filename That result will match a line in bold in input. In the matching line there is an_xxx digit in italics that (once the leading zero is removed) will match a line in link. That is the lint to print in output.... (2 Replies)
Discussion started by: cmccabe
2 Replies

4. Shell Programming and Scripting

awk to copy previous line matching a particular columns

Hello Help, 2356798 7689867 999 000 123678 20385907 9797 666 17978975 87468976 968978 98798 I am trying to have out put which actually look for the third column value of 9797 and then it insert line there after with first, second column value exactly as the previous line and replace the third... (3 Replies)
Discussion started by: Indra2011
3 Replies

5. Shell Programming and Scripting

Matching and printing line with awk

Hi there, I'm trying to use awk to print out the entire line that contains a match to a certain regex and then append some text,plus the match to the end of the line. So far I have: awk -F: '{print "RG:Z:" $2}' file Which prints out the match I want plus the additional text, but I'm stuck... (3 Replies)
Discussion started by: jim_lad
3 Replies

6. Shell Programming and Scripting

Replace and add line in file with line in another file based on matching string

Hi, I want to achieve something similar to what described in another post: The difference is I want to add the line if the pattern is not found. File 1: A123, valueA, valueB B234, valueA, valueB C345, valueA, valueB D456, valueA, valueB E567, valueA, valueB F678, valueA, valueB ... (11 Replies)
Discussion started by: jyu3
11 Replies

7. Shell Programming and Scripting

sed or awk delete character in the lines before and after the matching line

Sample file: This is line one, this is another line, this is the PRIMARY INDEX line l ; This is another line The command should find the line with “PRIMARY INDEX” and remove the last character from the line preceding it (in this case , comma) and remove the first character from the line... (5 Replies)
Discussion started by: KC_Rules
5 Replies

8. Shell Programming and Scripting

AWK : Add Fields of lines with matching field

Dear All, I would like to add values of a field, if the lines match in a certain field. Then I would like to divide the sum though the number of lines that have a matched field. This is the Input: Input: Test1 5 Test1 10 Test2 2 Test2 5 Test2 13 Test3 4 Output: Test1 7.5 Test1 7.5... (6 Replies)
Discussion started by: DerSeb
6 Replies

9. Shell Programming and Scripting

using command line arguments as columns for pattern matching using awk

Hi, I wish to use a column, as inputted by a user from command line, for pattern matching. awk file: { if($1 ~ /^8/) { print $0> "temp2.csv" } } something like this, but i want '$1' to be any column as selected by the user from command line. ... (1 Reply)
Discussion started by: invinclible0009
1 Replies

10. UNIX for Dummies Questions & Answers

how to delete line with matching text and line immediately after

hello experts, I have a file: File1 Sample Test1 This is a Test Sample Test2 Another Test Final Test3 A final Test I can use sed to delete the line with specific text ie: sed '/Test2/d' File1.txt > File2.txt How can I delete the line with the matching text and the line immediately... (6 Replies)
Discussion started by: orahi001
6 Replies

Featured Tech Videos