The awk below executes as expected if the id in $4 (like in f) is unique. However most of my data is like f1 where the same id can appear multiple times. I think that is the reason why the awk is not working as expected. I added a comment on the line that I can not change without causing the script to abort. Each line in f2 is searched and must contain the id, in this case COL1A2 but that id may not be a single entry. That is the id may appear 5 times, but each line with that is in f2 is searched. Using the $4 in f1 as the id and reading each $1, $2, and $3 value into a variable min and max.
The $4 is then split on the _ in f2 and read into array. The same id from f1 may appear in multiple lines of f2 however each will have unique $2 and $3 values. Each value in the split will match a $4 id in f1. The min and max must match the $1 of f2 and be between the $2 and $3 values in f2. An exact match is not needed rather just that the min or max variables falls within $2 and $3. If that is true then exon is printed in $5 of f2 if it is not true then intron is printed in $5. Most of this works as expected I just did not account for the possibity for multiple enteries and am nut sure how to adjust for it. Thank you
For example using the contents of the f1 where COL1A2 appears 3 times, each entry or line is searched in f2. Currently, I believe since COL1A2 is not unique not match is found in f2 as the min and max are not set per entry or line. Thank you .
Hi cmccabe,
If we rewrite the code that reads your first input file to be:
does that give you enough of a hint for what you then need to do in the loop you need to add in the code that reads your second input file?
This User Gave Thanks to Don Cragun For This Post:
Thank you for the hint, I made two adjustments to the script and commented them. The output is the same but maybe I have the idea just not implementing it correctly? Thank you .
Hi cmccabe,
If you create an array with three subscripts, you have to use three subscripts when you try to access an element of that array.
Are the ranges given in your first input file always in increasing numerical order for each $1,$4 set of values (as in your sample file f1)? If they are we can use that information to make your code run faster.
Is the fifth subfield of $4 in your second input file always identical to the $1 value on the same input line (as in your sample files)? If they are, we can use that information to make your code run faster.
You note that your input files fields are separated by tabs. Do you want the output file to be tab delimited too; or do you want the output to be delimited by spaces as shown in your sample output?
Note that in your original code (and in your updated code) you have the line:
and a $3 value can never be less than a max[] value and greater than the same value. Can we assume that you intended to write:
or more likely that you meant:
(i.e., are the min-max ranges inclusive of the endpoints or exclusive of the endpoints)?
This User Gave Thanks to Don Cragun For This Post:
Sorry... I missed one question concerning the line of code:
Is it your intent to print the line containing exon if either endpoint is in an entry in the first input file for that $1,$4 pair, or should it only print the exon line if both endpoints are in range?
This User Gave Thanks to Don Cragun For This Post:
Are the ranges given in your first input file always in increasing numerical order for each $1,$4 set of values (as in your sample file f1)? If they are we can use that information to make your code run faster.
Yes, these should always be sorted like in f1
Quote:
Is the fifth subfield of $4 in your second input file always identical to the $1 value on the same input line (as in your sample files)? If they are, we can use that information to make your code run faster.
Yes, this will always be the case if $4 is found as in f1
Quote:
You note that your input files fields are separated by tabs. Do you want the output file to be tab delimited too; or do you want the output to be delimited by spaces as shown in your sample output?
f1 will always be tab-delimited except for a whitespace after $3 and $4, but the output would be tab-delimited I did and OFS="\t" but I think the whitespaces are making that not work
You are correct in that I meant to be looking for inclusive endpoints so the >=/<= is what I should have used.
Quote:
Is it your intent to print the line containing exon if either endpoint is in an entry in the first input file for that $1,$4 pair, or should it only print the exon line if both endpoints are in range?
I used the || statement to make sure the script works as expected but it could be && as both coordinates should lie within the endpoints (trying to think of a situation where its not the case and not coming up with anything).
Yes, this will always be the case if $4 is found as in f1
f1 will always be tab-delimited except for a whitespace after $3 and $4, but the output would be tab-delimited I did and OFS="\t" but I think the whitespaces are making that not work
You are correct in that I meant to be looking for inclusive endpoints so the >=/<= is what I should have used.
I used the || statement to make sure the script works as expected but it could be && as both coordinates should lie within the endpoints (trying to think of a situation where its not the case and not coming up with anything).
Thank you very much .
Thanks for the responses.
Unfortunately, upon looking closer at your example input files, there are no entries in f1 where both endpoints of any line in f2 fall within the ranges specified in f1. In the last line of f2$2 falls inside the range specified in the first line in f1 but $3 does not.
And, despite what you said about the input files being tab delimited, the samples you provided don't contain any <tab> characters. And, since you said that the real files you're using do contain some <space>s between field 3 and 4 and after field4, the following code assumes that strings of one or more <space> and/or <tab> characters separate field and that any <space> and <tab> characters after field 4 are to be ignored. (As written, the code shown below will not work if a line in either input file contains any leading <space> or <tab> characters.)
So, given the above and assuming that you just want there to be some overlap between the ranges specified in a line in f1 and in a line in f2, maybe the following will do what you want:
This uses <tab> as the output field separator, but on output lines that end in "intron", <space>s in the input will not be converted to <tab>s in the output.
If you run the above script with no operands (or with the operand f1 or with the operand f) from your sample input files, it will produce the output:
Note that the above code does not set SUBSEP since it was not used in your script and isn't needed in the code shown above. Note also that the field separator I'm using the code above uses any sequence of one or more <space>s and/or <tab>s to treated as a field separator and uses every <underscore> as a field separator. That means that the subfields you were splitting into the array named array in your code will all be treated as separate fields in the code above. (That means I don't have to call split() to break that string into subfields.)
The sample files you provided to test any of the "corner" cases where I might have missed something. I think it will work OK, but I haven't performed enough extensive testing to give you any kind of guarantee.
Hope this helps,
Don
This User Gave Thanks to Don Cragun For This Post:
In the awk below I am trying to cp and paste each matching line in f2 to $3 in f1 if $2 of f1 is in the line in f2 somewhere. There will always be a match (usually more then 1) and my actual data is much larger (several hundreds of lines) in both f1 and f2. When the line in f2 is pasted to $3 in... (4 Replies)
In the awk I am trying to add :p.=? to the end of each $9 that matches the pattern NM_. The below executes andis close but I can not seem to figure out why the :p.=? repeats in the split as in the green in the current output. I have added comments as well. Thank you :).
file
... (4 Replies)
The bash bash below extracts the oldest folder from a directory and stores it in filename
That result will match a line in bold in input. In the matching line there is an_xxx digit in italics that
(once the leading zero is removed) will match a line in link. That is the lint to print in output.... (2 Replies)
Hello Help,
2356798 7689867 999 000
123678 20385907 9797 666
17978975 87468976 968978 98798
I am trying to have out put which actually look for the third column value of 9797 and then it insert line there after with first, second column value exactly as the previous line and replace the third... (3 Replies)
Hi there,
I'm trying to use awk to print out the entire line that contains a match to a certain regex and then append some text,plus the match to the end of the line.
So far I have:
awk -F: '{print "RG:Z:" $2}' file
Which prints out the match I want plus the additional text, but I'm stuck... (3 Replies)
Hi,
I want to achieve something similar to what described in another post:
The difference is I want to add the line if the pattern is not found.
File 1:
A123, valueA, valueB
B234, valueA, valueB
C345, valueA, valueB
D456, valueA, valueB
E567, valueA, valueB
F678, valueA, valueB
... (11 Replies)
Sample file:
This is line one,
this is another line,
this is the PRIMARY INDEX line
l ;
This is another line
The command should find the line with “PRIMARY INDEX” and remove the last character from the line preceding it (in this case , comma) and remove the first character from the line... (5 Replies)
Dear All,
I would like to add values of a field, if the lines match in a certain field. Then I would like to divide the sum though the number of lines that have a matched field. This is the Input:
Input:
Test1 5
Test1 10
Test2 2
Test2 5
Test2 13
Test3 4
Output:
Test1 7.5
Test1 7.5... (6 Replies)
Hi,
I wish to use a column, as inputted by a user from command line, for pattern matching.
awk file:
{
if($1 ~ /^8/)
{
print $0> "temp2.csv"
}
}
something like this, but i want '$1' to be any column as selected by the user from command line.
... (1 Reply)
hello experts,
I have a file: File1 Sample Test1
This is a Test
Sample Test2
Another Test
Final Test3
A final Test
I can use sed to delete the line with specific text
ie: sed '/Test2/d' File1.txt > File2.txt
How can I delete the line with the matching text and the line immediately... (6 Replies)