Amit,

I tried sed '$!N; /^\(.*\)\n\1$/!P; D'
to remove duplicates. It didn't work:

ex:
file test1.txt has the following rows:
123
123
145
123
123

I used the following command to remove duplicates:
sort.sh test1.txt > test2.txt

sort.sh script has your sed command:
#!/bin/ksh
file1=$1
sed '$!N; /^\(.*\)\n\1$/!P; D' < $file1

Do you know for sure this sed command works? Or is there some thing that I am doing wrong. Because, result file test2.txt has the following rows: It didn't remove all the duplicates?

123
123
145
123

Appreciate any pointers.
Radhika.

Why don't try to sort first & then take unique records.....
Is it not a good idea????