![]() |
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.
|
|
google unix.com
|
|||||||
| Forums | Register | Forum Rules | Links | Albums | FAQ | Members List | Calendar | Search | Today's Posts | Mark Forums Read |
| UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !! |
More UNIX and Linux Forum Topics You Might Find Helpful
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Remove duplicate lines in log files | karthikn7974 | Shell Programming and Scripting | 4 | 03-21-2009 06:41 PM |
| Remove all instances of duplicate records from the file | vukkusila | Shell Programming and Scripting | 3 | 12-12-2007 07:50 AM |
| how to remove duplicate lines | fredao | Shell Programming and Scripting | 3 | 12-13-2006 12:51 PM |
| Duplicate lines in the file | guptan | UNIX for Advanced & Expert Users | 3 | 05-18-2006 05:28 AM |
| Remove Duplicate Lines in File | Teh Tiack Ein | Shell Programming and Scripting | 5 | 01-12-2006 08:30 AM |
![]() |
|
|
LinkBack | Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
||||
|
Remove Duplicate lines from File
I have a log file "logreport" that contains several lines as seen below:
04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping I would like to edit the report to remove entries that report duplicate events. I am trying to produce an output close to what is seen below: 04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping This Error was reproduced 2 times |
|
||||
|
Quote:
My code works only for uniq words ![]() -ilan PS:not getting any idea on top of my head to serve your question!! |
|
||||
|
What exactly is the output from this? I can't get it to run correctly.
Code:
awk '{x[substr($0,9)]++;y[substr($0,9)]=$1}END{
for(i in x)
printf "%s\nThis Error was reproduced %d times\n",y[i]i,x[i]
}' logfile
|
|
|||||
|
I tried radoulov's script and it works for me. I had a sample log file on my desktop called "logfile.log" with these entries:
Code:
04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING 07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping 09:20:00 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING 10:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping 11:11:05 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING Code:
awk '{x[substr($0,9)]++;y[substr($0,9)]=$1}END{\
for(i in x) \
printf "%s\nThis Error was reproduced %d times\n",y[i]i,x[i]\
}' logfile.log
Code:
10:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping This Error was reproduced 3 times 11:11:05 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING This Error was reproduced 3 times |
| Sponsored Links | ||
|
|
![]() |
| Bookmarks |
| Tags |
| linux, sendmail |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|