UNIX for Advanced & Expert Users

I have a file like this:

Code:

[token1]=value1
[token2]=value2
.
.
.
[token n]=valuen

The issue is that if we get to have i.e. the [token17] line duplicated it may incurr into errors in our application.
I tried to find those repeated lines with something like

Code:

uniq -cd prueba1.txt

But it only found the repeated lines that are inmediately after the other i.e.

Code:

$ cat prueba1.txt
uno
dos
tres
tres
cuatro
cinco
seis
cuatro
siete
ocho
$ uniq -cd prueba1.txt
      2 tres

Only finding "tres" when it should also find "cuatro"
Any idea on how to fix that?

To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums

---------- Post updated at 05:09 PM ---------- Previous update was at 05:09 PM ----------

Sort your file first - 'man sort'

Why don't you sort before running uniq?

Also, just a starting point, look at arrays usage in awk:

Code:

 
awk '
{
        key = $0; # use your key field here
        if(key in regarr) {
                duparr[key] = key
        }
        else {
                regarr[key] = key
        }
}
END {
        for(idx in duparr) {
                print idx;
        }
}
'

Thanks guys
I appreciate your flash responses, this worked better than I expected
My final version is this command:

Code:

tr ' ] ' ' ' < prueba.txt | sort | awk '{print $1} ' | uniq -d | wc -l

That way, it should return 0, if it returns any higher, we have problems!
Let me know what you think about it, perhaps there is a way to make it shorter, I think it's really long...

A little bit smaller:

Code:

 cut -d']' -f1 prueba2.txt|sort|uniq -d|wc -l

Nice one, at least is smaller that mine, hehe