UNIX for Dummies Questions & Answers

I have a very large file (over 700 million lines) that has some lines that I need to delete. An example of 5 lines of the file:

HS4_80:8:2303:19153:193032 153 k80:138891 [...]
HS4_80:8:2105:5544:43174 89 k88:81949 [...]
165 k88:81949 323 0 * = 323 0 [...]
HS4_80:8:2105:5544:19502 73 k64:351700 [...]
HS4_80:8:2303:19154:108202 137 k72:245019 [...]

Two questions:
1. How can I confirm that the first character in line 3 above (beginning with 165) is a tab, so that I might use that as a regular expression, i.e. delete all lines that begin with tab. (Not all the lines I want to keep begin with HS4, nor do all the lines I want to delete begin with 165).

1. Assuming I find the right regular expression, how do I create a new file that is exactly the same, minus the lines I want to delete (like line 3 above)?

I'm a biologist who is new to programming. I know a lot more than I did a few weeks ago but it's slow going...so thanks for any help!

Hi pathunkathunk,

This sed command will delete all lines that begin with a tab character. I think this is what you want, if I understood your question. And it should be very fast processing that big file.

Code:

$ sed '/^\t/ d' infile

EDIT to add the command that redirects output:

Code:

$ sed '/^\t/ d' infile >outfile

Thanks but this doesn't seem to work for me.

I tried it on my large file and did not notice a difference. Then I created a smaller test file of only 6 lines, ensuring that I used the tab key to indent every other line. Still, the output file was identical to the input.

command: sed '/^\t/ d' test2 >test2b

#test2
abc1
abc2
ab3
abc4
abc5

#test2b
abc1
abc2
ab3
abc4
abc5

Then I don't know what you want to achieve.

The regular expression of my previous post deletes all lines that begin with a tab character. But in your files I don't see them. They begin with a letter, no spaces.

With older seds you cannot use "\t" and you need to do this:

Code:

sed 's/^    //d' infile

You cannot copy-paste this. The white space is one TAB-character entered as CTRL-V TAB

With perl. Original file is retained as inputfile.bak.

Code:

perl -i.bak -ne '!/^\t/ && print' inputfile

Code:

awk '!/^\t/' infile

Code:

grep -v '^\t' infile