UNIX for Dummies Questions & Answers

Cool would do that thanks!

Here is the top of the pattern file:

Chr1 711
Chr1 892
Chr1 956
Chr1 10904
Chr1 32210
Chr1 37388
Chr1 49438
Chr1 71326
Chr1 71348
Chr1 88300
Chr1 90571
Chr1 90606
Chr1 90809
Chr1 90864
Chr1 96770
Chr1 97473

Here is the top of the Master file I'm trying to grep out of

Chr1 3658 A 1 ^M. C
Chr1 3659 A 1 . C
Chr1 3660 A 1 . C
Chr1 3661 C 2 .^M. F@
Chr1 3662 A 2 .. F@
Chr1 3663 A 2 .. F@
Chr1 3664 A 2 .. F@
Chr1 3665 T 2 .. FD
Chr1 3666 A 2 .. HD
Chr1 3667 C 2 .. HD
Chr1 3668 A 4 ..^K,^K. H:.$
Chr1 3669 T 4 ..,. GA1*
Chr1 3670 A 4 ..,. HCD9
Chr1 3671 A 4 ..,. GDDJ
Chr1 3672 T 4 ..,. JFBI
Chr1 3673 C 4 ..,. JBDJ
Chr1 3674 G 6 ..,.^:,^:. IBDJ?=
Chr1 3675 G 6 ..,.,. J8DHDJ



So I have a 12GB file and I want to pull out the lines that correspond to about 350,000 specific positions on the chromosomes. Each is a separate line.

I'm using Ubuntu. The Master file is a .bcf made with samtools and the coordinates (pattern file) was made from a table in a paper and is tab delimited. I added a third column with just a tab so as not to pull out multiple entries beginning with the search pattern eg only want Chr1 3675 pulling not other entries like Chr1 367567 for example.

This starts to ouput the lines with the correct patterns to the terminal within a few seconds:

Code:

cat C2L4_all.bcf | grep -f lertest.txt

But this just greps indefinately, creates the output file but it stays empty. After about 10 min I kill the command.

Code:

cat C2L4_all.bcf | grep -f lertest.txt > plop.txt

I tried the script command to copy the terminal to a file but it has all kinds if formatting info in it as well. Must be an easier way.

---------- Post updated at 01:29 AM ---------- Previous update was at 01:21 AM ----------

Should I let it run overnight? Its just the non-write to file version only takes a few seconds!!

I'm wondering if it scans the whole 12 GB file before writing to file whereas if it finds something it may instantly display it in the terminal?

Seems odd if this is the case though!

1) I can't reproduce your problem with the sample data you gave above as there's no match between patterns and data.
2) Are you sure that there's all those strange char sequences near end of line in the data file?
3) why don't you do some testing with way smaller but still matching files?
4) use code tags, not icode tags.

Thanks RudiC,
I just took the top 100 lines of my 12GB file and did a mock run with 10 patterns I know are entries. It worked fine both to the terminal and written to file. So it looks like the problem is the size of the file that is being searched for the patterns.

Can I tell grep to only look for one match? I guess that may make it faster.

FGPonce

Reduce the pattern file to just one line, or give the pattern on the command line. You may want to switch off regex matching (which is compute intensive) with the -F (fixed string) option to grep.

Note 1: If your Master file contains any null bytes, the standards don't guarantee that you'll get the results you want. The behavior of grep is only specified when all of the input files are text files. By definition, text files have no lines longer than LINE_MAX bytes (including the terminating <newline> character) and do not contain any null bytes. Since you're seeing some output quickly, I doubt that this is your problem.

Note 2: I changed the ICODE tags to CODE tags in the quote from your posting above.

Note 3: The tabs in your input files have been converted to spaces (presumably by the tools used to copy and paste text from your files into this posting). You should do something like:

Code:

head C2L4_all.bcf lertest.txt | od -c

to verify that both of these files do have tabs rather than spaces separating fields. Note also that the last line you show above for your pattern file does NOT include a trailing tab (or spaces) on the last line. (However, this could also just be a copy and paste artifact.) Again, since you're seeing some output quickly, this probably isn't problem. But if some lines have tabs and others have spaces in either file (or both files), you could miss a lot of data you're trying to extract.

As RudiC mentioned, searching for fixed strings rather than basic regular expressions will be faster. And, as I mentioned before, getting rid of the unneeded cat will also be more efficient. Together, this would be:

Code:

grep -Ff lertest.txt C2L4_all.bcf > plop.txt

Note 4: You said that the command:

Code:

cat C2L4_all.bcf | grep -f lertest.txt

or more efficiently:

Code:

grep -Ff lertest.txt C2L4_all.bcf

starts writing output to the terminal within seconds. How much data did your command write to the terminal in ten minutes?

There is a good chance that the ouput produced by grep is being buffered when output is not directed to a terminal. If the amount of output you're seeing from the above command in ten minutes fits in less than BUFSIZ bytes on your system, you shouldn't expect to see anything in plop.txt in ten minutes. You should be able to find the value of BUFSIZ on your system in /usr/include/stdio.h or in a file that it #includes (assuming that your system includes the tools to build C or C++ applications and the headers are installed in the conventional location). From the data we've seen so far, I'm guessing this is your problem. If it is, letting the command run overnight may indeed solve your problem.

Hi,
I ran it overnight but found this error this morning:

Code:

grep: memory exhausted

However I did go for gold and point a 350,000 pattern list at the 12 GB file.

I did a bit of playing and got 300 patterns looked for and written to file in 1 min 45s, 3000 patterns in 2 min 39, 30,000 stopped after 60 mins 0 bytes output. Best ratio was 5,000 patterns in 2 min 47 s.

Thought I'd have to dribble 5000 patterns in at a time somehow.

Then tried Dons version:

Code:

grep -Ff lertest.txt C2L4_all.bcf

Did all 350,000 patterns in about 5 minutes............nice one Don! What does the -F do?

FGPonce