It's equivalent to uniq, so it won't help you.
If your data is in fact already sorted then just use `uniq` instead of `sort -u`

No, my data is not sorted.

The best possible approach will be push all the data in oracle using sqlloader.
Create index on the fly for the key u want unique.
And fire query to get the unique records.

Any better alternatives?

I am not sure if I want to reload all that data again into another table and .....

As I am pulling data from a table using select * from table name into a text file and then doing sort -u file1 > file2.

Although, I could try doing a select distinct columns from the table.... and see if it will take more time than it took my original approach. Is it worth trying? I don't know.

I just don't have the luxury of trying different options at my will as it is a production database unless I know it's worth trying.

It's already in a database!
Just do add a sort by in the select clause and
index the appropriate fields.

Definetly its worth a try.

Precautions u can take are:

1. Make sure all distinct columns are indexed.
2. If it is one table, then u need not worry about joins...else make sure the joins are in such a way that you get maximum throughput instead of least response time
3. Run the query at such a time when no other big activity is going on in same table, bcos if query will be long...it can give rollback segmetn too old error.

All the best.

Sorry for reply back ....

>> Hi Amit,



>> sed '$!N; /^\(.*\)\n\1$/!P; D'

>> Could you explain the command - bit by bit if you don't mind.

>> Thanks!

I think u can refer the man page of sed and look for sed Addresses

I think the topic is self explainatory...

BTW ...

I tested this command with more than 1GB file.

it took about 13 min to sort that file. Much Much Faster than sort command.