Shell Programming and Scripting

Thanks vgersh99!
Hope I did not offend you or hurt your feeling when I said your script took ~45hours to process my data. Originally I was thinking the scenario for one column comparison of substrings, then the two-columns situation (to handle array in awk which I did not catch well), which should be a new post. But I knew there are many experts in this forum, and I just posted it when the question popped up my mind. Sometime I felt guilty that you guys helped me so much but I did nothing. And my learning seems very slow.
Thank you!

no-no, no problem on this end.
I know how to make the code take any number of columns into account - that's pretty easy.
I also know where the scalability issue is - it's doing the full table scan of the already cached rows.
The problem is I cannot think of an easy and non-convoluted way of fixing this....
Sorry....

What awk were you using?
I think it's gawk, definitely I will give it a try with mawk.

Is the order of output important by the way?
No, the order does not matter. Column 1 has unique keys for each row, so that the order can be controlled afterwards. Thank you!

any chance of you providing either a representative 'section' of your file OR the full file in the gzip-ed format attached to this thread?
I'd like to have a 'big enough' sample so that could gauge the performance gain...

Attached two files should be unzipped and concatenated to have a single one. I am aware col1 and col3 are identical but I need them as they are to match all the previous posts. Let me know if you have any question.
Thank you!