Remove Similar Lines from a File

Nysif Steve · #1 (**permalink**) 08-31-2007

I have a log file "logreport" that contains several lines as seen below:

04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping

I would like to edit the report to remove entries that report duplicate events (event being the portion highlighted in red). I have no knowledge of what the events will be or how long they are. I am trying to produce an output close to what is seen below:

04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
This Error was reproduced 2 times

aigles · #2 (**permalink**) 09-02-2007

Try and adapt the following awk program.
steve.awk :

Code:

{
   if (match($0, /\[/) == 0) {
      Lines[++LinesCount] = $0;
      LineIds[LinesCount] = ""
      next;
   }

   id  = substr($0, RSTART);

   if (++Ids[id] == 1) {
      Lines[++LinesCount] = $0;
      LineIds[LinesCount] = id;
   }
}

END {
   for (i=1; i<=LinesCount; i++) {
      print Lines[i];
      if (id = LineIds[i]) {
         if (Ids[id] > 1) {
            print "This Error was reproduced", Ids[id], "times";
         }
      }
   }
}

Execute the awk program :

Code:

awk -f steve.awk logreport

Jean-Pierre.

namishtiwari · #3 (**permalink**) 09-03-2007

You can use --

cat logfile | sort | uniq -c > newlogfile

Thanks
Namish

Nysif Steve · #4 (**permalink**) 09-04-2007

I ended up using the code:

Code:

cat logfile | sort | uniq -c -n6 >> logreport

For the uniq command the -c flag will print the number of occurences before each line while the -n6 flag will ignore the first 6 fields for comparison. The end result is exactly what I needed. Thank you for your help everyone.

uniesh · #5 (**permalink**) 03-13-2009

cat file1|sort|uniq -c|cut -f2 >file2

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Remove Similar entries in a File	Nysif Steve	UNIX for Advanced & Expert Users	2	03-13-2009 03:50 AM
vi to remove lines in file	kjbaumann	UNIX for Dummies Questions & Answers	2	03-13-2009 03:40 AM
remove lines from file	bluemoon1	Shell Programming and Scripting	9	11-09-2007 01:40 AM
Remove Duplicate lines from File	Nysif Steve	UNIX for Dummies Questions & Answers	18	09-09-2007 08:57 AM
Remove Duplicate Lines in File	Teh Tiack Ein	Shell Programming and Scripting	5	01-12-2006 08:30 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode	Rate This Thread:


	Powered by