Shell Programming and Scripting

I am looking for a way to delete duplicate entries in a VERY large file (approx 2gb)

However I need to compare several fields before determining if this is a duplicate. I setup a hash in perl but it seems to not function correctly.

Any help appreciated.

of the 19 comma separated fields I need to check the second, the 5th and 10th to determine if they are truly duplicated. I cannot use the whole line as some of the fields are variable.


I have tried several iterations of hashes in perl but when I use a

my $USER=(split /,/)[1];
my $DAY=(split /,/)[4];
my $NUM=(split /,/)[9];

if (exists $seen{$USER} && exists $seen{$DAY} && exists $seen{$NUM})
.....


This seems to not work at all...I'm a newbie at perl so any help appreciated.

Thanks
G

guess should be sth like below, if not understand you wrongly

Code:

my $hash;
while(<FH>){
	my @tmp=split(",",$_);
	my $key=sprintf("%s%s%s",$tmp[2],$tmp[5],$tmp[11]);
	if (not exists $hash{$key}){
		print;
		$hash{$key} = 1;
	}
}

Ok that worked! I am suprised honestly. I guess I need to understand more about hashes. also I am curious about the use of sprintf.


优秀 !
感谢.

G