Here is the top of the Master file I'm trying to grep out of
So I have a 12GB file and I want to pull out the lines that correspond to about 350,000 specific positions on the chromosomes. Each is a separate line.
I'm using Ubuntu. The Master file is a .bcf made with samtools and the coordinates (pattern file) was made from a table in a paper and is tab delimited. I added a third column with just a tab so as not to pull out multiple entries beginning with the search pattern eg only want Chr1 3675 pulling not other entries like Chr1 367567 for example.
This starts to ouput the lines with the correct patterns to the terminal within a few seconds:
But this just greps indefinately, creates the output file but it stays empty. After about 10 min I kill the command.
I tried the script command to copy the terminal to a file but it has all kinds if formatting info in it as well. Must be an easier way.
---------- Post updated at 01:29 AM ---------- Previous update was at 01:21 AM ----------
Should I let it run overnight? Its just the non-write to file version only takes a few seconds!!
I'm wondering if it scans the whole 12 GB file before writing to file whereas if it finds something it may instantly display it in the terminal?
Seems odd if this is the case though!
Note 1: If your Master file contains any null bytes, the standards don't guarantee that you'll get the results you want. The behavior of grep is only specified when all of the input files are text files. By definition, text files have no lines longer than LINE_MAX bytes (including the terminating <newline> character) and do not contain any null bytes. Since you're seeing some output quickly, I doubt that this is your problem.
Note 2: I changed the ICODE tags to CODE tags in the quote from your posting above.
Note 3: The tabs in your input files have been converted to spaces (presumably by the tools used to copy and paste text from your files into this posting). You should do something like:
to verify that both of these files do have tabs rather than spaces separating fields. Note also that the last line you show above for your pattern file does NOT include a trailing tab (or spaces) on the last line. (However, this could also just be a copy and paste artifact.) Again, since you're seeing some output quickly, this probably isn't problem. But if some lines have tabs and others have spaces in either file (or both files), you could miss a lot of data you're trying to extract.
As RudiC mentioned, searching for fixed strings rather than basic regular expressions will be faster. And, as I mentioned before, getting rid of the unneeded cat will also be more efficient. Together, this would be:
Note 4: You said that the command:
or more efficiently:
starts writing output to the terminal within seconds. How much data did your command write to the terminal in ten minutes?
There is a good chance that the ouput produced by grep is being buffered when output is not directed to a terminal. If the amount of output you're seeing from the above command in ten minutes fits in less than BUFSIZ bytes on your system, you shouldn't expect to see anything in plop.txt in ten minutes. You should be able to find the value of BUFSIZ on your system in /usr/include/stdio.h or in a file that it #includes (assuming that your system includes the tools to build C or C++ applications and the headers are installed in the conventional location). From the data we've seen so far, I'm guessing this is your problem. If it is, letting the command run overnight may indeed solve your problem.
This User Gave Thanks to Don Cragun For This Post:
I 'm trying to grep 2 fieldds on 2 differnt lines. Like this:
psit > file
egrep -e '(NS|ES)' $file. Not working. If this succeeds then run next cmd else exit. Pls Help
Gundu (13 Replies)
Hey guys:
I've been meaning to post this question for awhile...it is regarding grep. Let's say for example that the following entry is in logxx:
Wed Feb 2 07:44:11 <vsm> 91030 Line 5 Severity 1 Vps 6
Call Answered - DN:8753101 CLID:5164665761 PI:83
If I do a grep 91030... (27 Replies)
Hi.
I have this format on a textfile:
VG Name /dev/vg00
PV Name /dev/dsk/c16t0d0
PV Name /dev/dsk/c18t0d0
PV Name /dev/dsk/c16t4d0
VG Name /dev/vg01
PV Name ... (6 Replies)
Guys I am having a problem with being able to find missing monitors in a configuration check script I am trying to create for accountability purposes for managing a large number of systems. What I am trying to do is run a script that will look at the raw config data in a file and pull all the pool... (7 Replies)
I want to search files (basically .cc files) in /xx folder and subfolders.
Those files (*.cc files) must contain #include "header.h" AND x() function.
I am writing it another way to make it clear,
I wanna list of *.cc files that have 'header.h' & 'x()'. They must have two strings, header.h... (2 Replies)
hi
i have kind of below text in a file.
I want to get a complete paragraph starting with START and ending with before another START) which has a particular string say XYZ or ABC
START XYZ hshjghkjh 45 ljkfd
fldjlj d jldf
START 3493u ABC 454
4545454
4545454 45454
4545454
START ...... (3 Replies)
How do you grep 'select * from table_name' string from a script if the select * and from table_name are on 2 different lines ? like
select *
from table_name
Any help would be greatly appreciated !!!
Thanks
RDR (4 Replies)
I have 1300 files (SearchFiles0001.txt, SearchFiles0002.txt, etc.) , each with 650,000 lines, tab-delimited data.
I have a pattern file, with about 1000 lines with a single word. Each single word is found in the 1300 files once.
If I grep -f PatternFile.txt SearchFiles*.txt >OutputFile.txt... (2 Replies)
I want to grep multiple lines from a text file. I want to grep all lines containing X,Y and NA in a single command. How do I go about doing that?
This is what my text files look like:
rs1983866 0.0983 10 100016313
rs1983865 0.5994 X 100016339
rs1983864 0.3272 11 100017453
rs7077266... (2 Replies)
Hi Everyone,
Im currently using the below code to pull data from a large CSV file and put it into smaller files with just the data associated with the number that I "grep".
grep 'M053' test.csv > test053.csv
Is there a way that I can use grep to run through my file like the example below... (6 Replies)