Shell Programming and Scripting

Thank you everyone! Lot's of awesome solutions for this problem. Very much appreciated!

It is working as designed
Unix.unix should be printed NOT

---------- Post updated at 11:19 AM ---------- Previous update was at 07:38 AM ----------

Please, refer to the perldoc to know what \g1 does.

Ok, one more experts posting.
The \g1 was introduced in Perl 5.10 and behaves like \1 (I tested with Perl 5.8 only, my bad).
The perl solution treats Unix.unix as two words while the awk solution treats it as one word.
Regarding my \b comment, only my version prints both

Code:

No duplicat sentence with the word duplicate
No duplicate sentence with the word duplicat

(Now I have tested with perl 5.8 and 5.18)

Could that be a bug or oversight in the AWK sugestion? Maybe is enough for the OP intention, however, a word normally is not only defined by characters separated by spaces.

Seeing all these elaborate awk solutions i wonder if sed wouldn't be easier:

Code:

sed '/\([^ ]*\) \1/d' file

It is little known that back references ("\1") can be used not only in the replacement string but also in the search regexp.

Btw.: "word" here is something surrounded by whitespace, not a certain number of characters. It is easy to put such a further restriction in if it is indeed needed.

I hope this helps.

bakunin

@Bakunin, that would only work with adjacent words and would also match partial patterns:

Code:

$ echo foo foobar | sed '/\([^ ]*\) \1/d'
$

And because of the zero or more match:

Code:

$ echo abc def ghi | sed '/\([^ ]*\) \1/d'
$

Yes, the boundary \b metacharacter is an anchor and I did not stop to think that it will not be saved as part of the group match.
From perl -ne 'print unless /(\b\w+\b).*\g1/i' to perl -ne 'print unless /\b(\w+)\b.*\b\g1\b/i' would had been a more appropriated suggestion. If your Perl version does not support the \g{} then, there's other bugs to consider.