UNIX for Dummies Questions & Answers

Hi!
I recently downloaded a wordlist file called 2of12.txt, which is a wordlist of common words, part of the 12dicts package. I've been getting unexpected results from grepping it, such as getting no matches when clearly there ought to be, or returns that are simply wrong.
Par exemple:

Code:

egrep ^...a.....n.$ /usr/share/dict/2of12.txt |head -5

apparition
bipartisan
cavalryman
defamation
dilatation

Clearly I'm asking for an eleven-letter word, and getting ten-letter words, (but at least the letters I'm asking for are in the right places). If I grep any other wordlist, I get the expected results.

Code:

egrep ^...a.....n.$ /usr/share/dict/sowpods.txt |head -5

advancement
advantaging
alkalescent
alkalifying
antalkaline

But if I add an extra dot at the end, I get the correct results. Well, not the correct results, but you know what I mean:

Code:

egrep ^...a.....n..$ /usr/share/dict/2of12.txt |head -5

advancement
arraignment
arrangement
derangement
devastating

I opened 2of12.txt in TextWrangler, showing invisibles, to see if there were some kind of extra white space characters in there, but I could see nothing wrong. It looks like they're all single words, followed by a newline.
Something must be wrong with this file, but I have no idea what it might be. I had read here 12dicts - Helpful that the file contained annotations after certain words, but I can find none of these. Does anyone have any idea what might cause this behaviour in a text file? If so, how can I find and fix this problem?
Thanks!

Maybe this can help:

Code:

egrep ^...a.....n. /usr/share/dict/2of12.txt | head -5
advancement
apparition
arraignment
arrangement
audaciousness

I think that a period matches any character, including the record separator, in a regular expression.

---------- Post updated at 10:44 PM ---------- Previous update was at 10:28 PM ----------

Maybe THIS can help better:

Code:

egrep -w ...a.....n. /usr/share/dict/2of12.txt | head -5
advancement
apparition
arraignment
arrangement
bipartisan

The -w option stands for --word-regexp in egrep's man. Still, a period may match a record separator and thus you get ten character words OR eleven character words...

Now THIS may be quite what you wanted in the first place:

Code:

egrep -w '...a.....n[^^M]' /usr/share/dict/2of12.txt | head -5
advancement
arraignment
arrangement
derangement
devastating

The regular expression ends with a character range that means 'any character EXCEPT a record separator'; you may get it as a caret followed by a newline, both enclosed in brackets (open bracket, caret, control-V, newline, close bracket). Just a single tiny side-effect: I get 'self-advancement' somewhere down the list of results...

The dot will match any sort of whitespace - space, tab, whatever. What other kind of record separator is there, besides newlines? The way you've tried it, it returns words that are eleven or more letters when we want only eleven-letter words.

Code:

^...........$

should return exactly eleven characters. It certainly does in every other wordlist, (or text file), that I have. You need the 'end of line' stop, or it'll look at the first eleven characters, then keep on going. Yet in 2of12, it returns ten-letter words. What can it be matching that I can't see when I have my text editor showing invisibles? Why does this 2of12 file behave differently? There must be something there I can't see.

This is not true. The dot does not match a newline. If it did, egrep "." file would match an empty line (^$) which it doesn't.

The problem is that there are probably ctl-M characters in the file just before the newline (Dos style newline which is 0x0d0x0a combinations rather than just a single 0x0a). Run the file through dos2unix, or someother conversion tool, which will delete the ctl-M characters from the file.

---------- Post updated at 23:36 ---------- Previous update was at 23:33 ----------

If you want an easy to verify that CTL-M characters are present, use cat:

Code:

cat -v 2of12.txt |head

This will likely generate the output:

Code:

a^M
aardvark^M
abaci^M
aback^M
abacus^M
abaft^M
abalone^M
abandon^M
abandoned^M
abandonment^M

Now THIS may be quite what you wanted in the first place:

Code:

egrep -w '...a.....n[^^M]' /usr/share/dict/2of12.txt | head -5
advancement
arraignment
arrangement
derangement
devastating

The regular expression ends with a character range that means 'any character EXCEPT a record separator'; you may get it as a caret followed by a newline, both enclosed in brackets (open bracket, caret, control-V, newline, close bracket). Just a single tiny side-effect: I get 'self-advancement' somewhere down the list of results...[/QUOTE]

OK, that looks like it works. I need to figure out how to use that to try and strip that junk out of the file. I think we get 'self-advancement' because we didn't start with a caret. Again, why don't we need the -w flag in any other file? If you don't use it, even with the starting caret, you get longer words.

---------- Post updated at 11:43 PM ---------- Previous update was at 11:40 PM ----------

That was it, all right!

Code:

whom^M
whomever^M
whomsoever^M
whoop^M
whoopee^M
whooper^M
whoops^M
whoosh^M
whopper^M
whopping^M
whore^M

---------- Post updated at 11:56 PM ---------- Previous update was at 11:43 PM ----------

dos2unix did the trick.