Shell Programming and Scripting

All,

I have a problem with grep/fgrep/egrep. Basically I am building a 200 times 200 correlation matrix. The entries of this matrix need to be retrieved from another very large matrix (~100G). I tried to use the grep/fgrep/egrep to locate each entry and put them into one file. It looks very slow with grep/egrep + reg exp. I am thinking that fgrep may will be faster, but not sure.

Does anyone have experience on this topic? or you may suggest some other commands/ways to solve this problem.

Regards,
Gary

simply reading 100GB is going to take several minutes, so you should get everything you need on one pass thru the file. If you read through the entire file 200 times you are doomed to wait for looong time.

Actually, you can "tune" a regex. You can also use awk with a regex. awk can output simultaneously to OPEN_MAX differentt files.

Code:

 getconf OPEN_MAX

show us what your input is like and what you want as output.