So if the tenth field is identical to one we have seen before, remove the whole line?
Code:
perl -ane 'print unless $seen{$F[9]}++' my_log.txt
(
Perl arrays are numbered from zero, so
$F[9] is the tenth field. The
-a option causes
Perl to split the input line into the array
@F, somewhat similarly to how
awk works.)
The input line is printed unless the hash
%seen already has an entry for the tenth field. We also add one to its value, which causes it to be set (to one) if it didn't exist before. Thus, the
%seen value for the current seed will be set the next time it is encountered.
For the sample file you posted, this reduces 1,274 lines to just 72 lines.
