Shell Programming and Scripting

Hello Folks,

Can some one help me with the removal of duplicate lines from a log file and send it to another log file. It's bit complicated as two lines are same but only difference is the timestamp, but some lines are uniq. Line has been seperated by colon's.

Log file:
=================================================

FORTE:APP:ERROR:nachversorgung.sh logarchivloeschung.sql FP:CRON:for3 :W:0001:Fehler: ##### Fehler in logarchivloeschung, Anzahl Redologs zu gross, 22 Logs nach 10 Laeufen!!! #####:TM 2011.12.20-12:40:43

FORTE:APP:ERROR:nachversorgung.sh logarchivloeschung.sql FP:CRON:for3 :W:0001:Fehler: ##### Fehler in logarchivloeschung, Anzahl Redologs zu klein, 7 Logs #####:TM 2011.12.19-16:35:36

FORTE:APP:ERROR:nachversorgung.sh logarchivloeschung.sql FP:CRON:for3 :W:0001:Fehler: ##### Fehler in logarchivloeschung, Anzahl Redologs zu gross, 22 Logs nach 10 Laeufen!!! #####:TM 2011.12.16-12:42:55

FORTE:APP:ERROR:nachversorgung.sh logarchivloeschung.sql FP:CRON:for3 :W:0001:Fehler: ##### Fehler in logarchivloeschung, Anzahl Redologs zu klein, 7 Logs #####:TM 2011.12.15-16:32:46

FORTE:APP:ERROR:nachversorgung.sh logarchivloeschung.sql FP:CRON:for3 :W:0001:Fehler: ##### Fehler in logarchivloeschung, Anzahl Redologs zu klein, 7 Logs #####:TM 2011.12.15-12:33:36
==================================================================

It should remove the duplicate lines and display atleast like this & send this data to another file.
==================================================================
FORTE:APP:ERROR:nachversorgung.sh logarchivloeschung.sql FP:CRON:for3 :W:0001:Fehler: ##### Fehler in logarchivloeschung, Anzahl Redologs zu gross, 22 Logs nach 10 Laeufen!!! #####:TM 2011.12.20-12:40:43

FORTE:APP:ERROR:nachversorgung.sh logarchivloeschung.sql FP:CRON:for3 :W:0001:Fehler: ##### Fehler in logarchivloeschung, Anzahl Redologs zu klein, 7 Logs #####:TM 2011.12.19-16:35:36

==================================================================

Only difference b/w lines is time stamp. Hope i m clear..

If you're okay with the new file not including timestamps, it would be really simple:

Code:

sed 's/#####:TM.*//' file | sort -u > newfile

If you need timestamps, I can offer this BASH snippet, but it can only get rid of duplicate ADJACENT lines:
[EDIT: Removed. Compared to the elegance of CarloM's awk, this was ridiculous]

I'm sure there's a better way. I think some awk-master is gonna need to post for a perfect solution.

Code:

awk -F"TM" '{lines[$1]=$2} END {for (i in lines) {print i lines[i]}}' inputfile

or

Code:

awk -F"TM" '!(lines[$1]) {lines[$1]=$2} END {for (i in lines) {print i lines[i]}}' inputfile

You could sort the file on input.

The first occurrence is of a line along with its time stamp is retained.

Code:

perl -ane '$t=pop @F;$r=join(" ",@F);if($.==1){$x{$r}=$t}for(keys %x){unless($k eq $r){$x{$r}=$t}}END{for(keys %x){print "$_ $x{$_}\n"}}' inputfile

Man that's awesome.

To the original poster: Carlo could give all the details, but the big difference I see between the two solutions he gave is that one grabs the first occurrence of a line, and the other grabs the last (meaning, you'll either get the first timestamp, or--probably more helpful--the timestamp of the last occurrence, depending on which you pick). The impressive perl solution also grabs the last occurrence.

Oh yeah, but your file is backwards from traditional logfiles--I forgot. Anywhoo.. I'm sure you'll work it out. Glad to read these solutions.

Thanks Everyone. All the solutions are working