The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
Remove duplicate lines in log files Remove duplicate lines in log files
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Remove Duplicate lines from File Nysif Steve UNIX for Dummies Questions & Answers 18 09-09-2007 08:57 AM
how to remove duplicate lines fredao Shell Programming and Scripting 3 12-13-2006 12:51 PM
Remove duplicate sabercats Shell Programming and Scripting 2 03-31-2006 12:35 PM
remove duplicate files in a directory asinha63 Shell Programming and Scripting 1 03-13-2006 03:46 PM
Remove Duplicate Lines in File Teh Tiack Ein Shell Programming and Scripting 5 01-12-2006 08:30 AM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 04-24-2008
karthikn7974's Avatar
karthikn7974 karthikn7974 is offline VIP Member  
Supporter
  
 

Join Date: Jul 2007
Location: Singapore
Posts: 58
Lightbulb Remove duplicate lines in log files
Hi
Having a file as follows
Code:
==================================================
Date           Time         error code      server 
======================================================
12-29-08    10:10        121221          server A
12-29-08    10:12        121221          server A
12-29-08    10:10        121221          server B
12-29-08    10:10        121221          server B
need to remove the duplicate lines with the following conditions

if date=12-29-08 and server=server A .
Must delete the line with old time and keep the latest.

Found similiar post in this forum but dont know how to use with my condition.

anyone can suggest

Thanks in advance
  #2 (permalink)  
Old 04-25-2008
adhit adhit is offline
Registered User
  
 

Join Date: Apr 2008
Location: Jakarta
Posts: 11
uniq -f 2 file
hope that's help
  #3 (permalink)  
Old 04-25-2008
karthikn7974's Avatar
karthikn7974 karthikn7974 is offline VIP Member  
Supporter
  
 

Join Date: Jul 2007
Location: Singapore
Posts: 58
can u give more details
  #4 (permalink)  
Old 12-03-2008
rohit22hamirpur rohit22hamirpur is offline
Registered User
  
 

Join Date: Nov 2008
Location: delhi
Posts: 8
even u can use

sort -u filename(this will sort as well as remove duplicate rows)
  #5 (permalink)  
Old 03-21-2009
quynhdang quynhdang is offline
Registered User
  
 

Join Date: Mar 2009
Posts: 1
Additional Question of "sort -u filename"
Hi there,

I use the same command you suggested but I found out it also sort the order. What else command I can remove any duplicated line but without change the order of the file.

Thank in advanced!
QuynhDang
Sponsored Links
Closed Thread

Bookmarks

Tags
shell script, shell scripting, unix scripting, unix scripting basics

« cron job - shell code correction | How to suspend a user account? »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Linear Mode Linear Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 05:44 PM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0