Shell Programming and Scripting

hmmm... have problems visualizing this. Could you provide a sample, pls.
It works for the OP's sample... Just trying to see the boarder conditions......

After re-reading the OP's previous two posts, I'm not certain if I correctly understood the task. If I am mistaken, apologies for the noise.

What I have in mind is the following scenario:

Code:

1 aaaa 10 aaaa
2 bbbb 20 bbbb
3 aaa 30 bbb

At the time that I wrote my previous post, I was under the impression that line 3 should be excluded because $2 is a substring of line1 and $4 of line 2.

However, your interpretation may well be correct. Perhaps the substring matches must be constrained to the same line. I am no longer confident in my assumption. Both of the fields of the lines excluded from the sample data in post #10 match the same preceding line, but nothing in that data set precludes independent column matching.

Perhaps I read too much into it.

Regards,
Alister

There should not be cross column comparison. Line 3 3 aaa 30 bbbis unique as its field2 and field4 are not substring of their corresponding field of any previous lines at the same time.

Code:

1 aaaa 10 aaaa
3 aaa  30 bbb

Only the script is slow for my 76,000 lines, still running after ~1hr on Linux 2.6.32-431.5.1.el6.x86_64. Thank you very much, both of you!

see if this makes it faster - getting away from the associate array and split-ing....:

Code:

awk '{
      for(i=1;i<=c;i++) {
        if (index(a[i],$2) && index(b[i],$4))
           next
        if (index($2, a[i]) && index($4,b[iA])) {
           delete a[i]
           delete b[i]
        }
      }
      a[++c]=$2
      b[c]=$4
      all[c]=$0
   }
END {
   for (i=1; i in all;i++) print all[i]
}' myFile

probably there's a better way to handle delete-d array elements that doesn't create 'holes' to be iterated over and over again, but... First let's see if this change makes any difference

If you add an array of keys, for the added memory, it's possible to completely bypass a scan of a and b:

Code:

awk '{
    if (($2 SUBSEP $4) in k) next
    for(i=1;i<=c;i++) {
...

Regards,
Alister

actually, it should be as simple as this for the sample file provided:

Code:

awk  '{
      for(i=1;i<=c;i++)
        if (index(a[i],$2) && index(b[i],$4))
           next
      a[++c]=$2
      b[c]=$4
      all[c]=$0
   }
END {
   for (i=1; i in all;i++) print all[i]
}' myFile

If you had a better, more representative data sample, maybe we could tweak the above - it works for the provided sample.

That looks wrong to me. Every line is added to all, but none of its members are ever removed (even when members of a or b are deleted).

Regards,
Alister