UNIX for Dummies Questions & Answers

Code:

awk '!_[$0]++'

Most importantly, I want to know what the underscore does "!_"

But ideally, please breakdown the whole thing. It is supposed to remove duplicate lines when found in a file.

The _ is an array name. You can change it to whatever valid identifier name you like - !my_awk_array[$0]++ would be the same.

! is a logical negation: NOT my_awk_array[$1]++

The expression !my_awk_array[$1]++ uses the current value of the array my_awk_array with a key $1, it goes like this:

Code:

zsh-4.3.11[t]% printf '%s\n'  a b a b a | awk '{ print "$1 is:", $1, "my_awk_array[$1]++ is:", my_awk_array[$1]++ }'
$1 is: a my_awk_array[$1]++ is: 0
$1 is: b my_awk_array[$1]++ is: 0
$1 is: a my_awk_array[$1]++ is: 1
$1 is: b my_awk_array[$1]++ is: 1
$1 is: a my_awk_array[$1]++ is: 2

So a post-incremented array element has the value 0 only the first time we see a value.

So how we get only unique values with the code above?

1. We have 0 only the first time we see a value.
2. 0 gets evaluated to false in Boolean expressions.
3. NOT false returns true, so we get only those records.

Makes sense?