Shell Programming and Scripting

As said before, removing duplicate non-empty lines is easier in awk:

Code:

awk '!X[$0]++' file

will do the job for you.

Perfect! Thank you all!

The previous awk deletes duplicate lines thruout the file. Therefore it needs to learn (store in memory) the whole file.
Consecutive identical lines can be deleted with storing only two lines

Code:

sed '$!N; /^\(.*\)\n\1$/!P;D'

Code:

awk 'prev!=$0 {print} {prev=$0}'

Code:

uniq

Danke schon!

Let's open it up for ease of maintenance. I never used -r for extended regex, as usually my sed is not GNU! The third substitute does not need it, as [] is implicitly and or.

Code:

sed -rn '
  /DIP-10219/{
    s/^.*[^,](cn=.*com).*$/\1/
    s/hbo/&-ns/g
    s/[(]|[)]/\\&/g
    p
   }
 ' in_file >out_file

Removing duplicates can be done by sed only if the file is sorted, but it can be done by 'sort -u' or 'uniq', if you are talking about whole line duplicates or well defined key field duplicates. In some situations, people need to remove later or earlier lines with the same key fields that are not entirely duplicate lines. One behaviour of 'sort -u' that is very useful is that all later lines for the key are deleted; the first survives. Sometimes we sort the file in reverse order if the last for a key is desired. To detect duplicates on the fly, you need a tool that can store past lines or keys and look up each line, like awk, ksh or bash.