UNIX for Dummies Questions & Answers

That was helpful. If I've understood correctly, bipinajith's pattern is telling awk to treat the data as a single column array, assign keys to each value, (hash table?), then if the same key comes up again, it's negated.
It's still hard for me to parse the pattern, though. It seems awk is doing an awful lot with very little. But I guess these are all just operators, and all the functions are built into awk.
I still have questions, of course. But I feel like I need to do some reading, first. I appreciate all of your help, everyone!

$0 means 'the entire unmodified line', pure and simple. You can change what 'line' means to awk, but by default, it uses newlines like everything else.

You can think of it as a hash table if you like. It could actually be built from a tree or list, but that doesn't really matter -- the point is, you can do a["qwerty"]=5; print a["qwerty"] and get 5 out. awk is like grep or sed, in that it has a built-in loop which runs code on every line individually. But it's like perl or shell in that it has no hardcoded function.

awk statements are like conditional { code block }. Whenever the conditional is true, it runs the { code block }. If you leave off { code block }, it assumes { print }, which will print the entire unmodified line.

So, awk '1' acts like cat, because 1 is always true. awk '/regex/' acts like grep "regex", because /regex/ is true whenever the current line matches the regular expression.

Now imagine what happens for every line for awk 'A[$0]++

The first time a line is seen, the value of A for that line will be "", a blank string. awk will consider this false, and not print the line. Next time, it will be a nonzero number, which awk considers true, causing it to print the line.

Code:

awk '!arr[$0]++' wordlist_file

Sorry, had a bit on my plate over the last week, so I had to come back to this.

So, I tried bipinajith's one-liner:

Code:

awk '!arr[$0]++' ~/path/to/file.txt

While it did remove doubles, the output only went to stdout, and the file was untouched. I thought simple redirection would do the job, but no soap. I get a "no such file" complaint.

Code:

awk '!arr[$0]++' ~/path/to/file.txt >~/path/to/newfile.txt

After a while, I figured out that the print statement would get me there:

Code:

awk '!arr[$0]++' ~/path/to/file.txt print$0 >~/path/to/newfile.txt

But I'm still wondering how you get it to "work in place" like sed's -i flag? Isn't that the behavior bipinajith was expecting from his original line?
Also, I got this error, although as far I can tell, everything worked as expected:

Code:

awk: can't open file print-bash
 input record number 9427508, file print-bash
 source line number 1

What does that mean?

This should definitely work!!

Code:

awk '!arr[$0]++' ~/path/to/file.txt > ~/path/to/newfile.txt

Please double check and verify if you are missing something.

Huh, you're right, I must of overlooked something. Of course it should work, and it does. But didn't you expect your original code to work in place? In other words, to over-write the original file? Or did I misinterpret?

The contents of a field, as seen by awk, can be changed within an awk program; this changes what awk perceives as the current input record. The actual input is untouched; awk never modifies the input file.

So you should redirect the output to another file and rename it back to original file if required.