Good point. Nuts, I thought that was a portable option. Throw that whole script out, then. I did note it expected one pattern per line. Argh. My script doesn't work, then.

Looking less and less like there's really going to be an efficient solution if you have to parse that thing the hard way line by individual line on systems and shells with no features.

---------- Post updated at 01:31 PM ---------- Previous update was at 12:49 PM ----------

OK, here's a brute-force version in awk:

Code:

BEGIN   {       FS="|"  ;       OFS="|"
                # Read in list of ID's
                while(getline < "file-1")       n[$2]=$1;
        }

{
        for(i in n)     if(index(i, $0) > 0)
                o[FILENAME "|" n[i]]++;
}

END     {       for(k in o)     print k, o[k];  }

Don' think it'll be quite as efficient as grep but ought to do. It ran in two seconds on 5,000 files cached, maybe 5-10 seconds uncached. I don't believe I used any GNU specific features either.

Put that in extract.awk and run with

Code:

xargs awk -f extract.awk < file-2

You can change o[FILENAME "|" n[i]]++; to o[FILENAME "|" i]++; if I somehow got name vs ID backwards again.

---------- Post updated at 01:54 PM ---------- Previous update was at 01:31 PM ----------

Estimate that to work in 60 minutes on data similar to yours on a 'fast' machine. use 'xargs -n 10' to prevent it from eating all your memory. Maybe not so good. If there was some sort of pattern to the OIDs/names, that'd help a lot for finding them without having to brute-force check all 50,000 individually... Or if you knew for a fact you had or could get GNU grep, and didn't have more than one object per line, grep's hardwired performance is going to be way better than awk's scripted performance...