UNIX for Beginners Questions & Answers

FM:Chicago:Development
FM:Chicago:Development:Score
SR:Cary:Testing:Testcases
PM:Newyork:Scripting
PM:Newyork:Scripting:Audit

FM:Chicago:Development:Score
SR:Cary:Testing:Testcases
PM:Newyork:Scripting:Audit

awk '!seen[$0]++' file

awk '
        {T[$0]
        }
END     {for (t1 in T)
           for (t2 in T)
             if (t1 ~ t2 && length(t1) != length(t2)) delete T[t2]
         for (t in T) print t
        }
' file
PM:Newyork:Scripting:Audit
FM:Chicago:Development:Score
SR:Cary:Testing:Testcases

awk: {T[$0]}END{for (t1 in T)for(t2 in T)if(t1 ~ t2 && length(t1)!=length(t2))delete T[t2] for (t in T) print }
awk:                                                                                       ^ syntax error

#!/usr/bin/env bash

# @(#) s1       Demonstrate elimination of duplicate lines, local uniq.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C sed my-uniq dixf

FILE=${1-data1}
E=expected-output.txt

pl " Input data file $FILE:"
cat $FILE

pl " Input data file transform separator:"
sed 's/:/_/' $FILE |
tee t1

pl " Expected output:"
cat $E

pl " Results, re-transform separator:"
my-uniq --separator=":" --last --field=1 t1 |
sed 's/_/:/' |
tee f1

pl " Verify results if possible:"
C=$HOME/bin/pass-fail
[ -f $C ] && $C || ( pe; pe " Results cannot be verified." ) >&2

pl " Help in my-uniq:"
my-uniq -h

dixf my-uniq

exit 0

$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-4-amd64, x86_64
Distribution        : Debian 8.7 (jessie) 
bash GNU bash 4.3.30
sed (GNU sed) 4.2.2
my-uniq (local) 1.11
dixf (local) 1.42

-----
 Input data file data1:
FM:Chicago:Development
FM:Chicago:Development:Score
SR:Cary:Testing:Testcases
PM:Newyork:Scripting
PM:Newyork:Scripting:Audit

-----
 Input data file transform separator:
FM_Chicago:Development
FM_Chicago:Development:Score
SR_Cary:Testing:Testcases
PM_Newyork:Scripting
PM_Newyork:Scripting:Audit

-----
 Expected output:
FM:Chicago:Development:Score
SR:Cary:Testing:Testcases
PM:Newyork:Scripting:Audit

-----
 Results, re-transform separator:
FM:Chicago:Development:Score
SR:Cary:Testing:Testcases
PM:Newyork:Scripting:Audit

-----
 Verify results if possible:

-----
 Comparison of 3 created lines with 3 lines of desired results:
 Succeeded -- files (computed) f1 and (standard) expected-output.txt have same content.

-----
 Help in my-uniq:

my-uniq - print or omit unique lines in non-sorted file

Synopsis

This code exists because one common requirement of a task is to
find (or omit) unique (or replicated) lines in a file, but also
to preserve the original order of the lines.  Standard versions
of "uniq" have usually required a sorted input file.

An additional common requirement is to consider only the content
of one field in each line rather than the entire line.  my-uniq
satisfies these requirements.

Usage: my-uniq options files

options:

--count
place count on each processed line, default is off.

--duplicate
print items that have more than one occurrence, default off.

--unique
print items that have only one occurrence, default is off.

--field=n
select a specific field, delimited by the separator, to be
used for the comparison, the default is the entire line.

--separator=string
choose an alternate separator, such as "|", or ",", the
default separator is "whitespace".

--last
allows over-writing, effectively keeping the most-recently
seen instance. Some versions of uniq on other *nix systems use
the most recent (Solaris), the default is compatibility with
GNU/Linux uniq, which keeps the first occurrence.

--quick
omit the operation that prints the lines in the order that
they were read. This prints according to a hash order,
therefore somewhat random -- a quick way to re-order a
file. This also requires less storage, a consideration for
large-volume files.

--help
print this and quit.

--version
print version number and quit.

my-uniq Like GNU/Linux uniq, but files need not be sorted. (what)
Path    : ~/bin/my-uniq
Version : 1.11
Length  : 282 lines
Type    : Perl script, ASCII text executable
Shebang : #!/usr/bin/perl
Help    : probably available with --help
Modules : (for perl codes)
 warnings       1.23
 strict 1.08
 Carp   1.3301
 Data::Dumper   2.151_01
 Getopt::Long   2.42