Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

awk to combine lines if fields match in lines

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk to combine lines if fields match in lines
# 8  
Old 05-14-2017
Hello cmccabe,

When I run my script I got following output. Where 4 lines in which string Fusion is there is coming to output.
Code:
chr12:12006495-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr15:88483984-chr12:12006495 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr12:12022903-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
chr15:88483984-chr12:12022903 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
chr17    7577108    COSM10749;COSM43737    C    A,T    149.594    PASS    AF=0.0830415,0.0;AO=372,2;DP=4420;FAO=166,0;FDP=1999;FR=.,.,REALIGNEDx0.0865;FRO=1833;FSAF=82,0;FSAR=84,0;FSRF=952;FSRR=881;FWDB=0.0072184,-0.0207142;FXX=4.99998E-4;HRUN=1,1;LEN=1,1;MLLD=293.795,80.5366;OALT=A,T;OID=COSM10749,COSM43737;OMAPALT=A,T;OPOS=7577108,7577108;OREF=C,C;PB=.,.;PBP=.,.;QD=0.299338;RBI=0.00721997,0.02565;REFB=1.40155E-4,-7.81395E-4;REVB=1.50579E-4,0.0151276;RO=4043;SAF=187,1;SAR=185,1;SRF=2118;SRR=1925;SSEN=0,0;SSEP=0,0;SSSB=-0.0251826,-5.12306E-4;STB=0.52327,0.5;STBP=0.541,1.0;TYPE=snp,snp;VARB=-0.00153404,0.0;HS;FUNC=[{'origPos':'7577108','origRef':'C','normalizedRef':'C','gene':'TP53','normalizedPos':'7577108','normalizedAlt':'A','polyphen':'1.0','gt':'pos','codon':'TTT','coding':'c.830G>T','sift':'0.0','grantham':'205.0','transcript':'NM_000546.5','function':'missense','protein':'p.Cys277Phe','location':'exonic','origAlt':'A','exon':'8','oncomineGeneClass':'Loss-of-Function','oncomineVariantClass':'Hotspot'}]    GT:GQ:DP:FDP:RO:FRO:AO:FAO:AF:SAR:SAF:SRF:SRR:FSAR:FSAF:FSRF:FSRR:QT    0/1:149:4420:1999:4043:1833:372,2:166,0:0.0830415,0.0:185,1:187,1:2118:1925:84,0:82,0:952:881:1
chr10    89624278    .    G    T    62.8836    PASS    AF=0.0785393;AO=297;DP=4155;FAO=157;FDP=1999;FR=.;FRO=1842;FSAF=77;FSAR=80;FSRF=908;FSRR=934;FWDB=0.0113997;FXX=4.99998E-4;HRUN=1;LEN=1;MLLD=117.237;OALT=T;OID=.;OMAPALT=T;OPOS=89624278;OREF=G;PB=.;PBP=.;QD=0.12583;RBI=0.040843;REFB=5.39678E-4;REVB=-0.0392199;RO=3844;SAF=150;SAR=147;SRF=1936;SRR=1908;SSEN=0;SSEP=0;SSSB=0.00159791;STB=0.502301;STBP=0.96;TYPE=snp;VARB=-0.00676678;FUNC=[{'origPos':'89624278','origRef':'G','normalizedRef':'G','gene':'PTEN','normalizedPos':'89624278','normalizedAlt':'T','gt':'pos','codon':'TAG','coding':'c.52G>T','transcript':'NM_000314.4','function':'nonsense','protein':'p.Glu18Ter','location':'exonic','origAlt':'T','exon':'1'}]    GT:GQ:DP:FDP:RO:FRO:AO:FAO:AF:SAR:SAF:SRF:SRR:FSAR:FSAF:FSRF:FSRR:QT    0/1:62:4155:1999:3844:1842:297:157:0.0785393:147:150:1936:1908:80:77:908:934:1

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
cmccabe
# 9  
Old 05-14-2017
Yes, I see now... it is the awk -F'\t' '!seen[$4]++' that is removing the 4th line.

Code:
chr12:12006495-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr15:88483984-chr12:12006495 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr12:12022903-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
chr15:88483984-chr12:12022903 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833

Since the lines in bold have the same $4 value only one of the lines needs to be printed.
The lines in italics have the same. This will only apply to those lines where SVTYPE=Fusion.

would adding the code in purple only remove the duplicates in the lines where SVTYPE=Fusion? Thank you very much Smilie.
Code:
   };
                        match($0,/oncomineGeneClass.*,/);
                        print "Locus\tType\tFunction\tGene\tReads"
                        print $1":"$2 "-" VAL OFS svtype OFS substr($0,RSTART+20,RLENGTH-22) OFS $3 OFS read_count;
                                   '{ if (a[$4]++ == 0) print $0; }' "$@"
            next
                    }
    1

desired output
Code:
chr12:12006495-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E4N15 1868
chr12:12022903-chr15:88483984 Fusion Gain-of-Function ETV6NTRK3-E5N15 414833
chr17 entire line.... (this line does not have the keyword in it so is printed in the output)
chr10 entire line...   (this line does not have the keyword in it so is printed in the output

# 10  
Old 05-15-2017
Hello cmccabe,

Based on your shown sample output, could you please try following and let me know if this helps you.
Code:
awk '/SVTYPE=Fusion/{
                        match($5,/].*]/);
                        sub(/.COS.*/,"",$3);
                        sub(/-/,"",$3);
                        sub(/\./,"-",$3);
                        VAL=substr($5,RSTART+1,RLENGTH-2);
                        num=split($8, array,";");
                        for(i=1;i<=num;i++){
                                                if(array[i] ~  /SVTYPE/){
                                                sub(/.*=/,"",array[i]);
                                                svtype=array[i]
                                                                        };
                                                if(array[i] ~ /READ_COUNT/){
                                                sub(/.*=/,"",array[i]);
                                                read_count=array[i]
                                                                           }
                                           };
                        match($0,/oncomineGeneClass.*,/);
                        if(++A[$3]==1)
                                    {
                                        print $1":"$2 "-" VAL OFS svtype OFS substr($0,RSTART+20,RLENGTH-22) OFS $3 OFS read_count;
                                    }
                    next
                    }
        1
    '    Input_file

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
cmccabe
# 11  
Old 05-15-2017
Thank you as always for the great help and explanations Smilie.

---------- Post updated at 07:30 AM ---------- Previous update was at 07:00 AM ----------

I am still a bit confused on this concept:

Quote:
#### Putting the values of subtring whose value starts from RSTART+1 to RLENGTH-2, here point to be noted RSTART and RLENGTH are the OOTB awk's keywords which will be SET when a match is found, we have done this match for 5th field above
I guess my confusion is what defines RSTART+1 and RLENGTH-2 in this line but RSTART+20 and RLENGH-22 in the print?

In the first RSTART $5 was used as the substring, so chr15:88483984, so is RSTART+1 the chr15 and RLENGTH-2 = to :88483984? Thank you Smilie.
# 12  
Old 05-15-2017
Quote:
Originally Posted by cmccabe
Thank you as always for the great help and explanations Smilie.
---------- Post updated at 07:30 AM ---------- Previous update was at 07:00 AM ----------
I am still a bit confused on this concept:
I guess my confusion is what defines RSTART+1 and RLENGTH-2 in this line but RSTART+20 and RLENGH-22 in the print?
In the first RSTART $5 was used as the substring, so chr15:88483984, so is RSTART+1 the chr15 and RLENGTH-2 = to :88483984? Thank you Smilie.
Hello cmccabe,

No need to be confuse Smilie

I will provide a guidance here and you could play with it. Let us assume you are printing simply RSTART and RLENGTH(RSTART will keep the index of starting point of the REGEX and RLENGTH will have the last character etc's index in it) variable's values then you will see some additional characters/digits are coming(which are not in your requirement) so this + and - to them is the adjustment of moving the cursor to print the exact required output. I hope I was clear in it, if you have doubt then kindly do let me know on same, will try to explain more on same.

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
cmccabe
# 13  
Old 05-17-2017
Thank you very much that helps a lot Smilie.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to remove lines that do not start with digit and combine line or lines

I have been searching and trying to come up with an awk that will perform the following on a converted text file (original is a pdf). 1. Since the first two lines are (begin with) text they are removed 2. if $1 is a number then all text is merged (combined) into one line until the next... (3 Replies)
Discussion started by: cmccabe
3 Replies

2. Shell Programming and Scripting

awk to combine matching lines in file

I am trying to combine all matching lines in the tab-delimited using awk. The below runs but no output results. Thank you :). input chrX 110925349 110925532 ALG13 chrX 110925349 110925532 ALG13 chrX 110925349 110925532 ALG13 chrX 47433390 47433999 SYN1... (3 Replies)
Discussion started by: cmccabe
3 Replies

3. UNIX for Beginners Questions & Answers

How to count lines of CSV file where 2 fields match variables?

I'm trying to use awk to count the occurrences of two matching fields of a CSV file. For instance, for data that looks like this... Joe,Blue,Yes,No,High Mike,Blue,Yes,Yes,Low Joe,Red,No,No,Low Joe,Red,Yes,Yes,Low I've been trying to use code like this... countvar=`awk ' $2~/$color/... (4 Replies)
Discussion started by: nmoore2843
4 Replies

4. Shell Programming and Scripting

awk to combine lines from line with pattern match to a line that ends in a pattern

I am trying to combine lines with these conditions: 1. First line starts with text of "libname VALUE db2 datasrc" where VALUE can be any text. 2. If condition1 is met then continue to combine lines through a line that ends with a semicolon. 3. Ignore case when matching patterns and remove any... (5 Replies)
Discussion started by: Wes Kem
5 Replies

5. Shell Programming and Scripting

Awk: Combine multiple lines based on number of fields

If a file has following kind of data, comma delimited 1,2,3,4 1 1 1,2,3,4 1,2 2 2,3,4 My required output must have only 4 columns with comma delimited 1,2,3,4 111,2,3,4 1,222,3,4 I have tried many awk command using ORS="" but couldnt progress (10 Replies)
Discussion started by: mdkm
10 Replies

6. UNIX for Dummies Questions & Answers

awk - (URGENT!) Print lines sort and move lines if match found

URGENT HELP IS NEEDED!! I am looking to move matching lines (01 - 07) from File1 and 77 tab the matching string from File2, to File3.txt. I am almost done but - Currently, script is not printing lines to File3.txt in order. - Also the matching lines are not moving out of File1.txt ... (1 Reply)
Discussion started by: High-T
1 Replies

7. Shell Programming and Scripting

Print only lines where fields concatenated match strings

Hello everyone, Maybe somebody could help me with an awk script. I have this input (field separator is comma ","): 547894982,M|N|J,U|Q|P,98,101,0,1,1 234900027,M|N|J,U|Q|P,98,101,0,1,1 234900023,M|N|J,U|Q|P,98,54,3,1,1 234900028,M|H|J,S|Q|P,98,101,0,1,1 234900030,M|N|J,U|F|P,98,101,0,1,1... (2 Replies)
Discussion started by: Ophiuchus
2 Replies

8. UNIX for Dummies Questions & Answers

awk display the match and 2 lines after the match is found.

Hello, can someone help me how to find a word and 2 lines after it and then send the output to another file. For example, here is myfile1.txt. I want to search for "Error" and 2 lines below it and send it to myfile2.txt I tried with grep -A but it's not supported on my system. I tried with awk,... (4 Replies)
Discussion started by: eurouno
4 Replies

9. Shell Programming and Scripting

search and combine lines in awk

Hi All, I have 1 "keyword" file like this: 00-1F-FB-00-04-18 00-19-CB-8E-66-DF 00-1F-FB-00-48-9C 00-1F-FB-00-AA-4F .... and the 2nd "details" file like this: Wed Feb 11 00:00:02 2009 NAS-IP-Address = xxxxxxxxxxxxxxxxxx Class = "P1-SHT-AAA01;1233704662;4886720" ... (6 Replies)
Discussion started by: xajax7
6 Replies

10. UNIX for Dummies Questions & Answers

SImple HELP! how to combine two lines together using sed or awk..

hi..im new to UNIX... ok i have this information in the normal shell... there are 2 lines display like this: h@hotmail.com k@hotmail.com i want it to display like this with a space betweem them h@hotmail.com k@hotmail.com the information is stored in a text file.... anyone... (10 Replies)
Discussion started by: forevercalz
10 Replies
Login or Register to Ask a Question