Removing duplicates from log file? - The UNIX and Linux Forums

		The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
Removing duplicates from log file?

google unix.com

Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
removing duplicates of a pattern from a file	ashisharora	UNIX for Dummies Questions & Answers	3	09-04-2008 07:25 AM
removing duplicates	stevie_velvet	Shell Programming and Scripting	5	07-07-2008 08:42 AM
Removing duplicates in a sorted file by field.	kinksville	Shell Programming and Scripting	1	04-29-2008 03:03 PM
removing duplicates from a file	trichyselva	UNIX for Dummies Questions & Answers	2	03-25-2008 10:49 AM
Removing duplicates	giannicello	Shell Programming and Scripting	12	09-14-2005 07:12 PM


	Powered by

LinkBack

Thread Tools

Search this Thread

Rate Thread

Display Modes

#1 (permalink)

01-21-2009

Ilja

Registered User

Join Date: Nov 2008

Posts: 22

Removing duplicates from log file?

I have a log file with posts looking like this:
[arrival time]-[message]-[id number]

Messages can be delivered by different systems at different times. The id number is used to sort out duplicate messages. What I need is to strip the arrival time from each post, sort posts by id number, and reattach arrival time to respective post.

Raw data:

Code:

200901211501-hello1-010
200901211504-hello1-010
200901211507-hello2-052
200901211512-hello2-052
200901211522-hello3-713
200901211544-hello4-220
200901211559-hello5-117
200901211612-hello5-117
200901211630-hello5-117

This is what I want to achieve (dupes removed):

Code:

200901211501-hello1-010
200901211507-hello2-052
200901211522-hello3-713
200901211544-hello4-220
200901211559-hello5-117

I've used different combinations of sed and uniq, but I seem to fail.

Grateful for any help.

Bookmarks

Tags

remove duplicates

« How to reduce font size in a file | Help on IF loop »

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Switch to Linear Mode Switch to Hybrid Mode Threaded Mode	Rate This Thread:

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Trackbacks are On Pingbacks are On Refbacks are On Forum Rules

All times are GMT -4. The time now is 05:46 AM.