Shell Programming and Scripting

Hi, I have a large file with number of records in there. I need some help to find only first row based on a key and ignore other rows with the same key. I tried few things but file is huge(30 million rows). So need some solution that is very efficient.

e.g

Code:

Junk|Apple|7|Random|data|here...
Junk|Apple|1|Random|data|here...
Junk|Apple|5|Random|data|here...
Junk|Orange|1|Random|data|here...
Junk|Orange|9|Random|data|here...

Here second field is the key. So I want only first record with 'Apple' and then first record with next key (in this case Orange). So output shall be

Code:

Junk|Apple|7|Random|data|here...
Junk|Orange|1|Random|data|here.

Since the file is large, I need help with some solution that do not run out memory.

Thank you...

Code:

awk -F\| '!a[$2]++' file

If records with the same key are always contiguous (as in your example), an even more efficient solution is possible.

Code:

awk -F\| '$2 != o; {o=$2}' file

For the corner case of the first record, that implementation assumes that the key field is not empty.

Regards,
Alister

Code:

perl -F'\|' -alne  '{if(!$hash{$F[1]}){$hash{$F[1]}++;print $_;}}' input_file

Solution cut down :

Code:

perl -F'\|' -alne  '{if(!$hash{$F[1]}++){print}}' input_file