Shell Programming and Scripting

I have a file contails as below

I/P:

Code:

123456
123456
234567
987654
678905
678905

Like above i have 1000's of entries
I need output as below
O/P:

Code:

123456
678905

I'm using uniq -d filename it is showing results but it is missing few duplicate entries and i dont know why.Please help me.

Code:

awk 'A[$0]++==1' file

Better to use $1 instead of $0 to avoid skipping some duplicate numbers due to leading/trailing whitespace.

Hi Buzzme,

Code:

awk '!d[$0]++' file

Correction: This ll have only unique entries

That will not work.
Revisit the OP's requirement.

elixir_sinari,
you are correct, that ll not work!. I did not understand the problem at first shot. Thanks.. for correcting me.

Buzzme,
> I'm using uniq -d filename it is showing results but it is missing few duplicate entries and i dont know why
- You may need to use

Code:

sort

before

Code:

uniq -d

, to have it work correctly. Wondering if you have tried it.

Please check it out with sorting numerical order:

Code:

sort -n file|uniq -d

Here is onother version with uniq that ll give output inclduing a numerical sorted output:

Code:

sort -n file|uniq -c|awk '{if ($1>1) print $2}'

Enjoy Have fun!.

You cannot meaningfully use a numeric sort, since uniq expects its data to be sorted lexicographically.

uniq will not consider "01" to be equal to "1", nor 1.0 to 1.00, and nor " 1" to "1". If leading/trailing zeroes/whitespace are a concern, then either the file needs to be preprocessed to normalize the entries, or a more capable tool should be used, e.g. perl or AWK.

Demonstration:

Code:

$ printf '%s\n' 1 01 001 '  1'
1
01
001
  1
$ printf '%s\n' 1 01 001 '  1' | sort -un
1
$ printf '%s\n' 1 01 001 '  1' | sort -n | uniq
  1
001
01
1

Notice how sort -un knows that it's doing a numeric comparison and considers all 4 terms to be equal. However, uniq considers each value to be distinct.

Regards,
Alister