Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Deleting sequences based on character frequency

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Deleting sequences based on character frequency
# 8  
Old 06-17-2010
Hi, I think I understand what you mean, for the first question:
Code:
awk '{ for (i=1;i<=length($5);i++) F[(substr($5,i,1)),i]+=$4; A[NR]=$0; S[NR]=$5}
     END {for (j=1;j<=NR;j++){{p=1; for(i=1;i<=length(S[j]);i++) if(F[(substr(S[j],i,1)),i]<=5) p=0} if (p){print A[j]}}}' RS=">" ORS=">" infile

If your file is really big and you run out of memory then you could try this
Code:
awk 'NR==FNR { for (i=1;i<=length($5);i++) F[(substr($5,i,1)),i]+=$4; next }
     {p=1; for(i=1;i<=length($5);i++) if(F[(substr($5,i,1)),i]<=5) p=0} p' RS=">" ORS=">" infile infile

Output first Sample:
Code:
> ID 2 Prev 31
A-TGCTAGCTACGTCGTACGT
> ID 3 Prev 30
A-TGCTAGCTACGTCGTTCGT
> ID 4 Prev 30
A-TGCTAGCTACGTCGTACGA

Output second sample:
Code:
> Seq 1 Freq 1
ATC
> Seq 2 Freq 33
ACG
> Seq 3 Freq 33
TTG
> Seq 4 Freq 33
TCC

# 9  
Old 06-17-2010
Thank you very, very much!
It is working great!

---------- Post updated at 08:33 PM ---------- Previous update was at 06:19 PM ----------

For a reason I can not understand, when I work with my actual data the freq is added to a second line instead to the firts one:

Original file
Quote:
>GF2FOAC04I8T0F
ACGAGTGCGTTTGATGTGCCAGCTGCCGTTGGTGTTAATGAGCTGAATGTTCTGCTGAGGGCCATGGCTGAACACGCCGGCAATCACGTTGGTGGAACGT GCAACAGCGCCTCCAACGGTGGTGGTGCCCGCGTCCACCCCAGCGGCCAGCAGAAGGATGACAATGACCTTCGCCCAC
>GF2FOAC04J305H
ACGAGTGCGTTTGATGTGCCAGCTGCCGTTGGTGTTAATGAGCTGAATGTTCTGCTGAGGGCCATGGCTGAACACGCCGGCAATCACGTTGGTGGAACGT GCAACAGCGCCTCCAACGGTGGTGGTGCCCGCGTCCACCCCAGCGGCCAGCAGAAGGATGACAATGACCTCCGCCCAC
>GF2FOAC04J3QXL
ACGAGTGCGTTTGATGTGCCAGCTGCCGTTGGTGTTAATGAGCTGAATGTTCTGCTGAGGGCCATGGCTGAACACGCCGGCAATCACGTTGGTGGAACGT GCAACAGCGCCTCCAAggGGTGGTGCCCCGCGTCCACCCCAGCGGCCAGCAGAAGGATGACAATGACCTTCGCCCACG
after using your code:
Code:
awk '/^>/{id=$0;next}length>=5&&length<=18{F[$1]++;if (!I[$1]) I[$1]=id}END{for(i in I)printf "%s Freq %s\n%s\n",I[i],F[i],i}' inputfile > Outputfile

I get the following:
Quote:

>GF2FOAC04IFFXE
Freq 1
ACGAGTGCGTTTGATGTGCCAGCTGCCTTGGTGTTAATGAGCTGAATGTTCTGCTGAGGGCCATGGCTGAACACGCCGGCAATCACGTTGGTGGAACGTG CAACAGCGCCTCCAACGGTGGTGGTGCCCGCGTCCACCCCAGCGGCCAGCAGAAGGATGACAATGACCCTCGCCCACG
>GF2FOAC04I1YPB
Freq 1
ACGAGTGCGTTGATGTGCCAGCTGCCGTTGGTGTTAATGAGCTGAATGTTCTGCTGAGGCCATGGCTGAACACGCCGGCAGTCACGTTGGTGGAACGTGC AACAGCGCCTCCAACGGTGGTGGTGCCCGCGTCCACCCCAGCGGCCAGCAGAAGGATGACAATGACCTTCGCCCACGC
>GF2FOAC04JODR7
Freq 1
ACGAGTGCGTTGATGTGCCAGCTGCCGTTGGTGTTAATGAGCTGAATGTTCTGCTGAGGGCCGATGGCTGAACACGCCGGCAATCACGTTGGTGGAACGT GCAACAGCGCCTCCAACGGTGGTGGTGCCCGCGTCCACCCCACGCGGCCAGCAGAAGGATGACAATGACCTTCGCCCA
>GF2FOAC04JI8HO
Freq 1
ACGAGTGCGTTTGATGTGCCAGCTGCCGTTGGTGTTAATGAGCTGATGTTCTGCTGgGCCGATGGCTGAACACGCCGGCAATCACGTTGGTGGagTGCAA CAgCGCCTCCAACGGTCGGTGGTGCCCGCGTCCACCCCAGCGGCCAGCAGagATGACAATGACCTTCGCCCACGCTCC
The problem is that the FASTA format is distorted and I cannot process the data afterwards. Any ideas how to avoid that problem and instead generate the ID followed by the Freq:
Quote:
>GF2FOAC04JI8HO Freq 1
ACGAGTGCGTTTGATGTGCCAGCTGCCGTTGGTGTTAATGAGCTGATGTTCTGCTGgGCCGATGGCTGAACACGCCGGCAATCACGTTGGTGGagTGCAA CAgCGCCTCCAACGGTCGGTGGTGCCCGCGTCCACCCCAGCGGCCAGCAGagATGACAATGACCTTCGCCCACGCTCC
Thanks!
# 10  
Old 06-24-2010
File
Scrutinizer,

Here you have a couple of examples of the input files I am trying to process using your code
Code:
awk '{ for (i=1;i<=length($5);i++) F[(substr($5,i,1)),i]+=$4; A[NR]=$0; S[NR]=$5}
     END {for (j=1;j<=NR;j++){{p=1; for(i=1;i<=length(S[j]);i++) if(F[(substr(S[j],i,1)),i]<=5) p=0} if (p){print A[j]}}}' RS=">" ORS=">" inputfile > outputfile

Thanks once again!
Examples.zip (5.3 KB) 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Outputting sequences based on length with sed

I have this file: >ID1 AA >ID2 TTTTTT >ID-3 AAAAAAAAA >ID4 TTTTTTGGAGATCAGTAGCAGATGACAG-GGGGG-TGCACCCC Add I am trying to use this script to output sequences longer than 15 characters: sed -r '/^>/N;{/^.{,15}$/d}' The desire output would be this: >ID4... (8 Replies)
Discussion started by: Xterra
8 Replies

2. UNIX for Dummies Questions & Answers

Filling positions based on frequency

I have files with hundreds of sequences with frequency values reported as "Freq X" and missing characters represented by a dash ("-"), something like this >39sample Freq 4 TAGATGTGCCCGTGGGTTTCCCGTCAACACCGGATAGTAGCAGCACTA >22sample Freq 15 T-GATGTCGTGGGTTTCCCGTCAACACCGGCAAATAGTAGCAGCACTA... (12 Replies)
Discussion started by: Xterra
12 Replies

3. Shell Programming and Scripting

Print sequences from file2 based on match to, AND in same order as, file1

I have a list of IDs in file1 and a list of sequences in file2. I can print sequences from file2, but I'm asking for help in printing the sequences in the same order as the IDs appear in file1. file1: EN_comp12952_c0_seq3:367-1668 ES_comp17168_c1_seq6:1-864 EN_comp13395_c3_seq14:231-1088... (5 Replies)
Discussion started by: pathunkathunk
5 Replies

4. Shell Programming and Scripting

Eliminating sequences based on Distances

I have to remove sequences from a file based on the distance value. I am attaching the file containing the distances (Distance.xls) The second file looks something like this: Sequences.txt >Sample1 Freq 59 ggatatgatgatgaactggt >Sample1 Freq 54 ggatatgatgttgaactggt >Sample1 Freq 44... (2 Replies)
Discussion started by: Xterra
2 Replies

5. Shell Programming and Scripting

Selecting sequences based on scores

I have two files with thousands of sequences of different lengths. infile1 contains the actual sequences and infile2 the scores for each A, T, G and C in infile1. Something like this: infile1: >HZVJKYI01ECH5R TTGATGTGCCAGCTGCCGTTGGTGTGCCAA >HZVJKYI01AQWJ8 GGATATGATGATGAACTGGTTTGGCACACC... (4 Replies)
Discussion started by: Xterra
4 Replies

6. Shell Programming and Scripting

Extract sequences based on the list

Hi, I have a file with more than 28000 records and it looks like below.. >mm10_refflat_ABCD range=chr1:1234567-2345678 tgtgcacactacacatgactagtacatgactagac....so on >mm10_refflat_BCD range=chr1:3234567-4545678... tgtgcacactacacatgactagtatgtgcacactacacatgactagta . . . . . so on ... (2 Replies)
Discussion started by: Diya123
2 Replies

7. Shell Programming and Scripting

Trimming sequences based on Reference

My file looks something like this Wnat I need is to look for the Reference sequence (">Reference1") and based on the length of that sequence trim all the entries in that file. So, the rersulting file will contain all sequences with the same length, like this Thus, all sequences will keep... (5 Replies)
Discussion started by: Xterra
5 Replies

8. Shell Programming and Scripting

Removing low frequency sequences

If I have a file with the following information And I would like to remove all the sequences with Freq less than 3, so I end up having the following file: I am currently using awk to accomplish this task but I am not getting the results I actually want. Any help will be greatly appreciated. (3 Replies)
Discussion started by: Xterra
3 Replies

9. Shell Programming and Scripting

Trimming sequences based on specific pattern

My files look like this And I need to cut the sequences at the last "A" found in the following 'pattern' -highlighted for easier identification, the pattern is the actual file is not highlighted. The expected result should look like this Thus, all the sequences would end with AGCCCTA... (2 Replies)
Discussion started by: Xterra
2 Replies

10. Shell Programming and Scripting

Deleting all characters from 350th character to 450th character from the log file

Hi All, I have a big log file i want to delete all characters (between 350th to 450th characters) starting at 350th character position to 450th character position. please advice or sample code. (6 Replies)
Discussion started by: rajeshorpu
6 Replies
Login or Register to Ask a Question