Hi Gurus,

I had a question regarding avoiding duplicates.i have a file abc.txt
abc.txt
-------
READER_1_1_1> HIER_28056 XML Reader: Error [UnterminatedXMLDecl] occurred while parsing:[Error at (file /home/abc.xml, line 6, char 1 ): Invalid document structure.]; line number [6]; column number [1]
READER_1_3_1> Sun Mar 23 23:52:48 2008
READER_1_3_1> HIER_28056 XML Reader: Error [UnterminatedXMLDecl] occurred while parsing:[Error at (file /home/abc.xml, line 6, char 1 ): Invalid document structure.]; line number [6]; column number [1]
READER_1_2_1> Sun Mar 23 23:52:48 2008
READER_1_2_1> HIER_28056 XML Reader: Error [UnterminatedXMLDecl] occurred while parsing:[Error at (file /home/abc.xml, line 6, char 1 ): Invalid document structure.]; line number [6]; column number [1]
READER_1_1_1> Sun Mar 23 23:52:48 2008
READER_1_1_1> HIER_28058 XML Reader Error
READER_1_2_1> Sun Mar 23 23:52:48 2008
READER_1_2_1> HIER_28058 XML Reader Error
READER_1_3_1> Sun Mar 23 23:52:48 2008
READER_1_3_1> HIER_28058 XML Reader Error.


I will get this file when i egrep the session log file for error specific messages.

here i want to remove the repeated lines in this file?
i am try to use uniq command, but here READER_1_3_1> will be varying in the above abc.txt file
i.e., in every line we have 1,2,3 in place of READER_1_**_1>.

So i am not getting the desired output.

Can any one let me know how to get the requirement done.

My output should be
READER_1_1_1> HIER_28056 XML Reader: Error [UnterminatedXMLDecl] occurred while parsing:[Error at (file /home/abc.xml, line 6, char 1 ): Invalid document structure.]; line number [6]; column number [1]
READER_1_1_1> HIER_28058 XML Reader Error



Thanks & Regards,
San

use :
"uniq -q" prints only the unique lines in a file.

My suggestion would be to write a Perl script incorporating Regular Expression to solve your issue. It may be easier to pull out your unique values that way.

My question is: Does every message have 2 duplicates in input data?

You gave abc.txt example, where every messages is duplicated with 'READER_1_2_1>' and 'READER_1_3_1>' headers added. If all logs are presented in such way you can:

I could misunderstand you. It can be too obvious answer for your question. I don't know, if "header" (before '>' sign) changes.

Thnaks for ur reply

No since it reads from 3 sources it will be giving out 3 errors.But the source is same so i get 3 similar messages with only reader variation.
Here in reader place i may get any other word .

All you need to trim a file down to unique lines is:

However, your lines aren't straight duplicates, so you'll need to pre-process the file. If your lines will all be the same except for the initial "reader" part, then just strip out any occurrence of READER_X_X_X.

ShawnMilo

Thanks for ur reply
But here all the lines are unique since reader_1 will be changed Reader_2.only that variation will be there,

so if i use uniq command it not getting the desired output.