Remove Similar entries in a File

 
Thread Tools Search this Thread
Special Forums UNIX and Linux Applications Infrastructure Monitoring Remove Similar entries in a File
# 1  
Old 08-30-2007
Remove Similar entries in a File

--------------------------------------------------------------------------------

I have a log file "logreport" that contains several lines as seen below:


04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
06:38:08 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
07:11:05 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping


I would like to edit the report to remove entries that report duplicate events. I am trying to produce an output close to what is seen below:


04:20:00 /usr/lib/snmp/snmpdx: [ID 702911 daemon.error] Agent snmpd appeared dead but responded to ping
This Error was reproduced 2 times
# 2  
Old 08-30-2007
Forum rules - no duplicate posts !!!
kamitsin
# 3  
Old 03-13-2009
cat file1|sort|uniq -c | cut -f2
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

REMOVE DUPLICATE IN a ROW AFTER CHECKING THE FIRST SIMILAR NAME

Hi all I have a big file like this in rows and columns from 2 column onwards the next column is desciption of previous column means 3rd columns is description of 2 columns and 5 column is description of 4 column. All cloumns are separated by comma ... (1 Reply)
Discussion started by: manigrover
1 Replies

2. Shell Programming and Scripting

Kindly check:remove duplicates with similar data in front of it

Hi all, I have 2 files containing data like this: so if there is same entry repeated in the column like1,2,3,4 I have to check if there is different entries column like 2,4 but similar entries for duplicatein column 2 like1,3 the output shuld be like this for first file ... (5 Replies)
Discussion started by: manigrover
5 Replies

3. Shell Programming and Scripting

Merge two files with similar column entries

Hi , I have few files which contains user name and data transfer rate in MBs and this data is collected for year and for each month report is saved in 12 different files I have to merge all the files to prepare the final report Files are as below Similarly I have 10 more files ... (5 Replies)
Discussion started by: pratapsingh
5 Replies

4. Shell Programming and Scripting

extracting lines from a file with similar first name

consider i have two files cat onlyviews1.sql CREATE VIEW V11 AS SELECT id, name, FROM etc etc WHERE etc etc; CREATE VIEW V22 AS SELECT id, name, FROM etc etc WHERE etc etc; CREATE VIEW V33 AS (10 Replies)
Discussion started by: vivek d r
10 Replies

5. Shell Programming and Scripting

Filter a .kml file (xml) to remove unwanted entries

Ok, i have a .kml file that that i want to trim down and get rid of the rubbish from. its formatted like so: <Placemark> <name><!]></name> <description><!</b><br/>Frequency: <b>2437</b><br/>Timestamp: <b>1304892397000</b><br/>Date: <b>2011-05-08... (11 Replies)
Discussion started by: Phear46
11 Replies

6. Shell Programming and Scripting

remove one of each similar lines in a file

Hello folks I have a question for you gurus of sed or grep (maybe awk, but I would prefer the first two) I have a file (f1) that says: (actually, these are not numbers but md5sum, but for simplicity, let's assume these numbers.) 1 2 3 4 5And I have a file (f2) that says 1|a 1|b 1|c 2|d... (3 Replies)
Discussion started by: tukuyomi
3 Replies

7. Shell Programming and Scripting

how to remove certain entries within a file in AIX

I have a rather large log file on our AIX server that contains syslog entries, with each line beginning with the date/time the message was received. The date goes back to Aug 17th of last year. I need to remove all lines within this log file that have its messages earlier than Nov 5th, and keep... (1 Reply)
Discussion started by: iaminit
1 Replies

8. Infrastructure Monitoring

Remove Similar Lines from a File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead... (4 Replies)
Discussion started by: Nysif Steve
4 Replies

9. Shell Programming and Scripting

Need Help: Collecting similar messages in a file

Hi All, Since i am very new to shell scripting, i need help from you guys. Suppose there is a file containing: Log message: Ashish "asasasa" asasa asasa asasas.info1 Log message: Kapil "asasasa" asasa asasa asasas..info1 (2 Replies)
Discussion started by: ashish.kapil
2 Replies

10. Shell Programming and Scripting

replace a similar field in a file

Hello, I am having a file where I have to replace the port values with the variable I defined. The file is an extract of an xml file: <NameValuePair> <name>Service1</name> <value>tcp:32406</value> </NameValuePair> <NameValuePair> ... (2 Replies)
Discussion started by: chiru_h
2 Replies
Login or Register to Ask a Question