The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > UNIX for Dummies Questions & Answers
Remove Duplicate lines from File Remove Duplicate lines from File
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


UNIX for Dummies Questions & Answers If you're not sure where to post a UNIX or Linux question, post it here. All UNIX and Linux newbies welcome !!

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove duplicate lines in log files karthikn7974 Shell Programming and Scripting 4 03-21-2009 06:41 PM
Remove all instances of duplicate records from the file vukkusila Shell Programming and Scripting 3 12-12-2007 07:50 AM
how to remove duplicate lines fredao Shell Programming and Scripting 3 12-13-2006 12:51 PM
Duplicate lines in the file guptan UNIX for Advanced & Expert Users 3 05-18-2006 05:28 AM
Remove Duplicate Lines in File Teh Tiack Ein Shell Programming and Scripting 5 01-12-2006 08:30 AM

Closed Thread
Page 1 of 3 1 23
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 08-30-2007
Nysif Steve Nysif Steve is offline
Registered User
  
 

Join Date: Aug 2007
Location: Albany, NY
Posts: 26
Remove Duplicate lines from File
I have a log file "logreport" that contains several lines as seen below:

04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping


I would like to edit the report to remove entries that report duplicate events. I am trying to produce an output close to what is seen below:

04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
This Error was reproduced 2 times
  #2 (permalink)  
Old 08-30-2007
ilan ilan is offline
Registered User
  
 

Join Date: Jul 2007
Posts: 101
you can use uniq.

can be like this:
for i in `more logreport | sort |uniq `; do echo "no.of time &i:`grep -c $i logreport`"; done

-ilan
Last edited by ilan; 08-30-2007 at 02:49 PM..
  #3 (permalink)  
Old 08-30-2007
radoulov's Avatar
radoulov radoulov is offline Forum Staff  
addict
  
 

Join Date: Jan 2007
Location: Варна, България / Milano, Italia
Posts: 2,847
Code:
awk '{x[substr($0,9)]++;y[substr($0,9)]=$1}END{
	for(i in x)
printf "%s\nThis Error was reproduced %d times\n",y[i]i,x[i]
}' logfile
As always, use nawk or /usr/xpg4/bin/awk on Solaris.
  #4 (permalink)  
Old 08-30-2007
ilan ilan is offline
Registered User
  
 

Join Date: Jul 2007
Posts: 101
Quote:
Originally Posted by ilan View Post
you can use uniq.

can be like this:
for i in `more logreport | sort |uniq `; do echo "no.of time &i:`grep -c $i logreport`"; done

-ilan
stand corrected!!
My code works only for uniq words

-ilan

PS:not getting any idea on top of my head to serve your question!!
  #5 (permalink)  
Old 08-31-2007
Nysif Steve Nysif Steve is offline
Registered User
  
 

Join Date: Aug 2007
Location: Albany, NY
Posts: 26
What exactly is the output from this? I can't get it to run correctly.

Code:
awk '{x[substr($0,9)]++;y[substr($0,9)]=$1}END{
	for(i in x)
printf "%s\nThis Error was reproduced %d times\n",y[i]i,x[i]
}' logfile
  #6 (permalink)  
Old 08-31-2007
radoulov's Avatar
radoulov radoulov is offline Forum Staff  
addict
  
 

Join Date: Jan 2007
Location: Варна, България / Milano, Italia
Posts: 2,847
Quote:
Originally Posted by Nysif Steve View Post
What exactly is the output from this? I can't get it to run correctly.
What is the output you get?

Code:
zsh 4.3.2% cat file
04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to peng
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to pong
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to pong
zsh 4.3.2% awk '{x[substr($0,9)]++;y[substr($0,9)]=$1}END{
for(i in x)
printf "%s\nThis Error was reproduced %d times\n",y[i]i,x[i]
}' file
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to pong
This Error was reproduced 2 times
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to peng
This Error was reproduced 1 times
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
This Error was reproduced 3 times
  #7 (permalink)  
Old 08-31-2007
cassj's Avatar
cassj cassj is offline VIP Member  
Supporter
  
 

Join Date: Jul 2003
Location: Interweb
Posts: 109
I tried radoulov's script and it works for me. I had a sample log file on my desktop called "logfile.log" with these entries:

Code:
04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
09:20:00 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING
10:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
11:11:05 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING
I cd'd to my desktop, and then I used radoulov's script with the slight modification of adding "\" to the end of each line so I could copy and paste it to the bash command line.

Code:
awk '{x[substr($0,9)]++;y[substr($0,9)]=$1}END{\
for(i in x) \
printf "%s\nThis Error was reproduced %d times\n",y[i]i,x[i]\
}' logfile.log
Press return and voila!:

Code:
10:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
This Error was reproduced 3 times
11:11:05 /usr/lib/snmp/snmpdx: [ID 999999 daemon.error] Agent snmpd appeared dead but responded to DING
This Error was reproduced 3 times
Nice job radoulov!
Sponsored Links
Closed Thread
Page 1 of 3 1 23

Bookmarks

Tags
linux, sendmail

« removing a line from a file and then placing into another file | Combining find, grep and gzip »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Linear Mode Linear Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 03:01 PM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0