The UNIX and Linux Forums  

Go Back   The UNIX and Linux Forums > Top Forums > UNIX for Dummies Questions & Answers
Remove Duplicate lines from File Remove Duplicate lines from File
Google UNIX.COM
Forums Portal Register Forum Rules FAQContribute Members List Arcade Search Today's Posts Mark Forums Read


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove duplicate lines in log files karthikn7974 Shell Programming and Scripting 2 04-24-2008 09:14 PM
Remove all instances of duplicate records from the file vukkusila Shell Programming and Scripting 3 12-12-2007 03:50 AM
how to remove duplicate lines fredao Shell Programming and Scripting 3 12-13-2006 08:51 AM
Duplicate lines in the file guptan UNIX for Advanced & Expert Users 3 05-18-2006 02:28 AM
Remove Duplicate Lines in File Teh Tiack Ein Shell Programming and Scripting 5 01-12-2006 04:30 AM

Reply
Page 1 of 3 1 23 >
 
Submit Tools LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 08-30-2007
Registered User
  

Join Date: Aug 2007
Location: Albany, NY
Posts: 26
Remove Duplicate lines from File
I have a log file "logreport" that contains several lines as seen below:

04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping


I would like to edit the report to remove entries that report duplicate events. I am trying to produce an output close to what is seen below:

04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
This Error was reproduced 2 times
Reply With Quote
Forum Sponsor
  #2 (permalink)  
Old 08-30-2007
Registered User
  

Join Date: Jul 2007
Posts: 74
you can use uniq.

can be like this:
for i in `more logreport | sort |uniq `; do echo "no.of time &i:`grep -c $i logreport`"; done

-ilan
Last edited by ilan; 08-30-2007 at 11:49 AM.
Reply With Quote
  #3 (permalink)  
Old 08-30-2007
radoulov's Avatar
addict
  

Join Date: Jan 2007
Location: Milan, Italy/Varna, Bulgaria
Posts: 1,551
Code:
awk '{x[substr($0,9)]++;y[substr($0,9)]=$1}END{
	for(i in x)
printf "%s\nThis Error was reproduced %d times\n",y[i]i,x[i]
}' logfile
As always, use nawk or /usr/xpg4/bin/awk on Solaris.
Reply With Quote
  #4 (permalink)  
Old 08-30-2007
Registered User
  

Join Date: Jul 2007
Posts: 74
Quote:
Originally Posted by ilan View Post
you can use uniq.

can be like this:
for i in `more logreport | sort |uniq `; do echo "no.of time &i:`grep -c $i logreport`"; done

-ilan
stand corrected!!
My code works only for uniq words

-ilan

PS:not getting any idea on top of my head to serve your question!!
Reply With Quote
  #5 (permalink)  
Old 08-31-2007
Registered User
  

Join Date: Aug 2007
Location: Albany, NY
Posts: 26
What exactly is the output from this? I can't get it to run correctly.

Code:
awk '{x[substr($0,9)]++;y[substr($0,9)]=$1}END{
	for(i in x)
printf "%s\nThis Error was reproduced %d times\n",y[i]i,x[i]
}' logfile
Reply With Quote
  #6 (permalink)  
Old 08-31-2007
radoulov's Avatar
addict
  

Join Date: Jan 2007
Location: Milan, Italy/Varna, Bulgaria
Posts: 1,551
Quote:
Originally Posted by Nysif Steve View Post
What exactly is the output from this? I can't get it to run correctly.
What is the output you get?

Code:
zsh 4.3.2% cat file
04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to peng
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to pong
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to pong
zsh 4.3.2% awk '{x[substr($0,9)]++;y[substr($0,9)]=$1}END{
for(i in x)
printf "%s\nThis Error was reproduced %d times\n",y[i]i,x[i]
}' file
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to pong
This Error was reproduced 2 times
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to peng
This Error was reproduced 1 times
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
This Error was reproduced 3 times
Reply With Quote
  #7 (permalink)  
Old 08-31-2007
cassj's Avatar
Supporter
  

Join Date: Jul 2003
Location: Interweb
Posts: 107
I tried radoulov's script and it works for me. I had a sample log file on my desktop called "logfile.log" with these entries:

Code:
04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
09:20:00 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING
10:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
11:11:05 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING
I cd'd to my desktop, and then I used radoulov's script with the slight modification of adding "\" to the end of each line so I could copy and paste it to the bash command line.

Code:
awk '{x[substr($0,9)]++;y[substr($0,9)]=$1}END{\
for(i in x) \
printf "%s\nThis Error was reproduced %d times\n",y[i]i,x[i]\
}' logfile.log
Press return and voila!:

Code:
10:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
This Error was reproduced 3 times
11:11:05 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING
This Error was reproduced 3 times
Nice job radoulov!
Reply With Quote
Google The UNIX and Linux Forums
Reply
Page 1 of 3 1 23 >

Tags
linux, sendmail

« removing a line from a file and then placing into another file | Combining find, grep and gzip »
Thread Tools
Display Modes
Linear Mode Linear Mode




All times are GMT -7. The time now is 06:42 AM.

The UNIX and Linux Forums RSS Feeds - Contact Us - The UNIX and Linux Forums - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited.
The UNIX and Linux Forums Content Copyright ©1993-2008. All Rights Reserved.Ad Management by RedTyger Visit The Global Fact Book

Content Relevant URLs by vBSEO 3.2.0