Shell Programming and Scripting

In that case the example that I gave above should put you on track. Split the AGRN or whatever part of $5 that could possibly been *exactly* matched by the search pattern. This will avoid searching a pattern in a longer string which is rather greedy.

Instead of reading 750 MB, you are reading 3GB to operate on. With the four input files in arrays and an extended algorithm, the performance might be way faster.
If we had some meaningful samples, we could work out a small test script...

one of the 4 files used is below (all four are 1 field and a list of names)

PAH.bed

Code:

AGRN
CCDC39 
CCDC40 
CFTR
DNAAF1
DNAAF2 
DNAAF3 
DNAH11 
DNAH5 
DNAI1 
DNAI2 
DNAL1 
DYX1C1
HEATR2 
HYDIN 
LRRC6 
NME8 
OFD1
RPGR
RSPH4A 
RSPH9

The file that is searched in is 11,137,660 lines in the format below:

Code:

chr1    955543    955763    chr1:955543    AGRN-6|gc=75    1    0
chr1    955543    955763    chr1:955543    AGRN-6|gc=75    2    2
chr1    955543    955763    chr1:955543    AGRN-6|gc=75    3    2

Thank you .

Part of what is confusing is that you have input files with the filename extension .bed that you show as having a single field such as:

Code:

AGRN
CCDC39 
CCDC40 
CFTR
DNAAF1
...

and you have four output files (three of which have the same filename extension in a completely different format):

Code:

chr1:955543    AGRN-6|gc=75     3

and one other output file has the filename extension .. Why aren't the names of your output files consistent? Why aren't all files with the filename extension .bed in the same format?

And we have an unknown number of files matching the pattern /home/cmccabe/Desktop/HiQ/*base_counts.txt and no indication of what is actually matched by the asterisk. Please give us some actual sample pathnames that this pattern might match.

You have said your input files have more than 11 million lines each and have shown us the 3 line sample:

Code:

chr1    955543    955763    chr1:955543    AGRN-6|gc=75    1    0
chr1    955543    955763    chr1:955543    AGRN-6|gc=75    2    2
chr1    955543    955763    chr1:955543    AGRN-6|gc=75    3    2

and your code accumulates totals based on the string AGRN-6 and prints results assuming that AGRN-6 and AGRN-6|gc=75 select the same set of lines from your huge input files. Please give us a few more lines (some with strings that will be selected for output from the .bed input file and some that won't. And show us the exact output you hope to get in your four output files for that sample input. (Note that that means we need to see four sample .bed input files and four corresponding output files in your sample.)

From your description I am assuming that there could be multiple AGRN-x values in the input but for a given AGRN-x the string following the | will be a constant. I.e., for $5 in your code having the value AGRN-6 the only value for$6 will be gc=75, but there could be an AGRN-otherstring and all AGRN-otherstring entries would have a string something like xyz=somenumber but xyz and somenumber would always be the same for any given AGRN-otherstring. Is this assumption correct?

Adding | as a field separator character seems to be creating unneeded work for you. It would seem that using - as a field separator instead of using | as a field separator would help. Will there ever be more than one - in an input line?

Well, I obviously missed that: there are n *base_counts.txt files, and for each of them you scan four times through the huge file, so 4 * n * 11,137,660 lines are read.
As has been offered before, with some meaningful samples we perhaps could give some decent help.
Please post at least two ???_unix_corrected.bed (partly) , two *base_counts.txt (partly) files, and a representative part of the huge file so we can build a meaningful test scenario.