Shell Programming and Scripting

All,

Thanks for your inputs.
The suggested solution works for me

Code:

awk -F\| -v X="an" 'match ($NF, ".*"X) {print $0, RLENGTH-length(X)+1}' OFS=\| file

The project involves searching a list of keywords in a master input file.
1. master_file.txt containing strings. Record Count : 20000000 String Records.
2. keywords.txt containing keywords. Record Count : 200,000 Unique Keywords.

I run a shell script which reads a keyword and runs the above mentioned command to search in master_file.txt to append desired output.

The end result is being achieved as per our expectations.
However, i am concerned about the performance and response time of this utility.
I tried with 16000 keywords and 20000 master records and the process took around 25 minutes.

I am looking to reduce this number and considered the following:
1. Split up file into n part and run searches in parallel and then collate results?
2. Possible tweaking for commands ?
3. Is text mining in shell correct from a design and feasibility perspective ?

Please provide your inputs.

You shouldn't run one command (= one process) per keyword, that's definitely too ineffective. On the other hand, the numbers you indicated might be too high for grep or awk. There's programs/applications/databases out there designed for text mining - I'm sure they'd be more appropriate for your task.

Thanks Rudi.

Running in parallel is certainly one option and i am exploring that.
Certainly agree there are specific programs designed for it; but i wanted to invest some time to find out something at the ground level.

I find shell script plus awk/sed/grep a great way to prototype a concept, but it's worth knowing the limitations of the tools you are using. I'd suggest sticking with a smaller subset and finalizing your prototype and in the background start researching text mining tools/relational databases etc.