Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Remove duplicate text
Top Forums Shell Programming and Scripting Remove duplicate text Post 302203292 by dejavu88 on Saturday 7th of June 2008 05:58:27 PM
Old 06-07-2008
The original file format looks like this:

Code:
UserID: 7
Start Time: Sat May 24 23:24:13 CEST 2008
End Time: Sat May 24 23:26:57 CEST 2008
Total time spent: 2.73072 minutes / 163.843 seconds
Date: Sat Jun 7 16:09:03 CEST 2008

UserID: 8
Start Time: Sun May 25 00:14:30 CEST 2008
End Time: Sun May 25 00:14:32 CEST 2008
Total time spent: 0.0304667 minutes / 1.828 seconds
Date: Sat Jun 7 16:10:02 CEST 2008

UserID: 9
Start Time: Sun May 25 00:14:30 CEST 2008
End Time: Sun May 25 00:14:32 CEST 2008
Total time spent: 0.0304667 minutes / 1.828 seconds
Date: Sat Jun 7 16:11:01 CEST 2008

I forgot to mention that there's an extra line at the beginning of the file followed by an empty line. Which says something about my log file. Thought it would be possible to get awk to read from the end of the file instead of from the top.

[edit2]

My fault, it's also in html format, not plain text in a log file, so when awk reads the file, it reads multiple lines in the .html file instead of what you see in the output of the log in a browser. There are a lot of lines in the file, should I just paste them here or can I send you the file?
Last edited by dejavu88; 06-07-2008 at 07:24 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate ???

Hi all, I have a out.log file CARR|02/26/2006 10:58:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:11:30.107|CDxAcct=1405157051 CARR|02/26/2006 11:18:30.107|CDxAcct=7659579782 CARR|02/26/2006 11:28:30.107|CDxAcct=9534922327 CARR|02/26/2006 11:38:30.107|CDxAcct=9534922327 CARR|02/26/2006... (3 Replies)
Discussion started by: sabercats
3 Replies

2. Shell Programming and Scripting

Remove duplicate

Hi all, I have a text file fileA.txt DXRV|02/28/2006 11:36:49.049|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:37:06.404|SAC||||CDxAcct=6070970034 DXRV|02/28/2006 11:37:25.740|SAC||||CDxAcct=2420991350 DXRV|02/28/2006 11:38:32.633|SAC||||CDxAcct=6070970034 DXRV|02/28/2006... (2 Replies)
Discussion started by: sabercats
2 Replies

3. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

4. Shell Programming and Scripting

remove duplicate

Hi, I am tryung to use shell or perl to remove duplicate characters for example , if I have " I love google" it will become I love ggle" or even "I loveggle" if removing duplicate white space Thanks CC (6 Replies)
Discussion started by: ccp
6 Replies

5. Shell Programming and Scripting

Filter or remove duplicate block of text without distinguishing marks or fields

Hello, Although I have found similar questions, I could not find advice that could help with our problem. The issue: We have several hundreds text files containing repeated blocks of text (I guess back at the time they were prepared like that to optmize printing). The block of texts... (13 Replies)
Discussion started by: samask
13 Replies

6. Shell Programming and Scripting

Remove duplicate

Hi , I have a pipe seperated file repo.psv where i need to remove duplicates based on the 1st column only. Can anyone help with a Unix script ? Input: 15277105||Common Stick|ESHR||Common Stock|CYRO AB 15277105||Common Stick|ESHR||Common Stock|CYRO AB 16111278||Common Stick|ESHR||Common... (12 Replies)
Discussion started by: samrat dutta
12 Replies

7. UNIX for Dummies Questions & Answers

Remove duplicate

Hi, How can I replace || with space and then remove duplicate from following text? T111||T222||T444||T222||T555 Thanks in advance (10 Replies)
Discussion started by: tinku981
10 Replies

8. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

9. Shell Programming and Scripting

How to remove duplicate text blocks from a file?

Hi All I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file. All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this. 10.14.22.22... (3 Replies)
Discussion started by: mahasona
3 Replies

10. Shell Programming and Scripting

Remove duplicate occurrences of text pattern

Hi folks! I have a file which contains a 1000 lines. On each line i have multiple occurrences ( 26 to be exact ) of pattern folder#/folder#. # is depicting the line number in the file some text here folder1/folder1 some text here folder1/folder1 some text here folder1/folder1 some text... (7 Replies)
Discussion started by: martinsmith
7 Replies

LEARN ABOUT MOJAVE

datetime::locale::en

DateTime::Locale::en(3) 				User Contributed Perl Documentation				   DateTime::Locale::en(3)

NAME

       DateTime::Locale::en

SYNOPSIS

	 use DateTime;

	 my $dt = DateTime->now( locale => 'en' );
	 print $dt->month_name();

DESCRIPTION

       This is the DateTime locale package for English.

DATA

       This locale inherits from the DateTime::Locale::root locale.

       It contains the following data.

   Days
       Wide (format)

	 Monday
	 Tuesday
	 Wednesday
	 Thursday
	 Friday
	 Saturday
	 Sunday

       Abbreviated (format)

	 Mon
	 Tue
	 Wed
	 Thu
	 Fri
	 Sat
	 Sun

       Narrow (format)

	 M
	 T
	 W
	 T
	 F
	 S
	 S

       Wide (stand-alone)

	 Monday
	 Tuesday
	 Wednesday
	 Thursday
	 Friday
	 Saturday
	 Sunday

       Abbreviated (stand-alone)

	 Mon
	 Tue
	 Wed
	 Thu
	 Fri
	 Sat
	 Sun

       Narrow (stand-alone)

	 M
	 T
	 W
	 T
	 F
	 S
	 S

   Months
       Wide (format)

	 January
	 February
	 March
	 April
	 May
	 June
	 July
	 August
	 September
	 October
	 November
	 December

       Abbreviated (format)

	 Jan
	 Feb
	 Mar
	 Apr
	 May
	 Jun
	 Jul
	 Aug
	 Sep
	 Oct
	 Nov
	 Dec

       Narrow (format)

	 J
	 F
	 M
	 A
	 M
	 J
	 J
	 A
	 S
	 O
	 N
	 D

       Wide (stand-alone)

	 January
	 February
	 March
	 April
	 May
	 June
	 July
	 August
	 September
	 October
	 November
	 December

       Abbreviated (stand-alone)

	 Jan
	 Feb
	 Mar
	 Apr
	 May
	 Jun
	 Jul
	 Aug
	 Sep
	 Oct
	 Nov
	 Dec

       Narrow (stand-alone)

	 J
	 F
	 M
	 A
	 M
	 J
	 J
	 A
	 S
	 O
	 N
	 D

   Quarters
       Wide (format)

	 1st quarter
	 2nd quarter
	 3rd quarter
	 4th quarter

       Abbreviated (format)

	 Q1
	 Q2
	 Q3
	 Q4

       Narrow (format)

	 1
	 2
	 3
	 4

       Wide (stand-alone)

	 1st quarter
	 2nd quarter
	 3rd quarter
	 4th quarter

       Abbreviated (stand-alone)

	 Q1
	 Q2
	 Q3
	 Q4

       Narrow (stand-alone)

	 1
	 2
	 3
	 4

   Eras
       Wide

	 Before Christ
	 Anno Domini

       Abbreviated

	 BC
	 AD

       Narrow

	 B
	 A

   Date Formats
       Full

	  2008-02-05T18:30:30 = Tuesday, February 5, 2008
	  1995-12-22T09:05:02 = Friday, December 22, 1995
	 -0010-09-15T04:44:23 = Saturday, September 15, -10

       Long

	  2008-02-05T18:30:30 = February 5, 2008
	  1995-12-22T09:05:02 = December 22, 1995
	 -0010-09-15T04:44:23 = September 15, -10

       Medium

	  2008-02-05T18:30:30 = Feb 5, 2008
	  1995-12-22T09:05:02 = Dec 22, 1995
	 -0010-09-15T04:44:23 = Sep 15, -10

       Short

	  2008-02-05T18:30:30 = 2/5/08
	  1995-12-22T09:05:02 = 12/22/95
	 -0010-09-15T04:44:23 = 9/15/-10

       Default

	  2008-02-05T18:30:30 = Feb 5, 2008
	  1995-12-22T09:05:02 = Dec 22, 1995
	 -0010-09-15T04:44:23 = Sep 15, -10

   Time Formats
       Full

	  2008-02-05T18:30:30 = 6:30:30 PM UTC
	  1995-12-22T09:05:02 = 9:05:02 AM UTC
	 -0010-09-15T04:44:23 = 4:44:23 AM UTC

       Long

	  2008-02-05T18:30:30 = 6:30:30 PM UTC
	  1995-12-22T09:05:02 = 9:05:02 AM UTC
	 -0010-09-15T04:44:23 = 4:44:23 AM UTC

       Medium

	  2008-02-05T18:30:30 = 6:30:30 PM
	  1995-12-22T09:05:02 = 9:05:02 AM
	 -0010-09-15T04:44:23 = 4:44:23 AM

       Short

	  2008-02-05T18:30:30 = 6:30 PM
	  1995-12-22T09:05:02 = 9:05 AM
	 -0010-09-15T04:44:23 = 4:44 AM

       Default

	  2008-02-05T18:30:30 = 6:30:30 PM
	  1995-12-22T09:05:02 = 9:05:02 AM
	 -0010-09-15T04:44:23 = 4:44:23 AM

   Datetime Formats
       Full

	  2008-02-05T18:30:30 = Tuesday, February 5, 2008 6:30:30 PM UTC
	  1995-12-22T09:05:02 = Friday, December 22, 1995 9:05:02 AM UTC
	 -0010-09-15T04:44:23 = Saturday, September 15, -10 4:44:23 AM UTC

       Long

	  2008-02-05T18:30:30 = February 5, 2008 6:30:30 PM UTC
	  1995-12-22T09:05:02 = December 22, 1995 9:05:02 AM UTC
	 -0010-09-15T04:44:23 = September 15, -10 4:44:23 AM UTC

       Medium

	  2008-02-05T18:30:30 = Feb 5, 2008 6:30:30 PM
	  1995-12-22T09:05:02 = Dec 22, 1995 9:05:02 AM
	 -0010-09-15T04:44:23 = Sep 15, -10 4:44:23 AM

       Short

	  2008-02-05T18:30:30 = 2/5/08 6:30 PM
	  1995-12-22T09:05:02 = 12/22/95 9:05 AM
	 -0010-09-15T04:44:23 = 9/15/-10 4:44 AM

       Default

	  2008-02-05T18:30:30 = Feb 5, 2008 6:30:30 PM
	  1995-12-22T09:05:02 = Dec 22, 1995 9:05:02 AM
	 -0010-09-15T04:44:23 = Sep 15, -10 4:44:23 AM

   Available Formats
       d (d)

	  2008-02-05T18:30:30 = 5
	  1995-12-22T09:05:02 = 22
	 -0010-09-15T04:44:23 = 15

       EEEd (d EEE)

	  2008-02-05T18:30:30 = 5 Tue
	  1995-12-22T09:05:02 = 22 Fri
	 -0010-09-15T04:44:23 = 15 Sat

       Hm (H:mm)

	  2008-02-05T18:30:30 = 18:30
	  1995-12-22T09:05:02 = 9:05
	 -0010-09-15T04:44:23 = 4:44

       hm (h:mm a)

	  2008-02-05T18:30:30 = 6:30 PM
	  1995-12-22T09:05:02 = 9:05 AM
	 -0010-09-15T04:44:23 = 4:44 AM

       Hms (H:mm:ss)

	  2008-02-05T18:30:30 = 18:30:30
	  1995-12-22T09:05:02 = 9:05:02
	 -0010-09-15T04:44:23 = 4:44:23

       hms (h:mm:ss a)

	  2008-02-05T18:30:30 = 6:30:30 PM
	  1995-12-22T09:05:02 = 9:05:02 AM
	 -0010-09-15T04:44:23 = 4:44:23 AM

       M (L)

	  2008-02-05T18:30:30 = 2
	  1995-12-22T09:05:02 = 12
	 -0010-09-15T04:44:23 = 9

       Md (M/d)

	  2008-02-05T18:30:30 = 2/5
	  1995-12-22T09:05:02 = 12/22
	 -0010-09-15T04:44:23 = 9/15

       MEd (E, M/d)

	  2008-02-05T18:30:30 = Tue, 2/5
	  1995-12-22T09:05:02 = Fri, 12/22
	 -0010-09-15T04:44:23 = Sat, 9/15

       MMM (LLL)

	  2008-02-05T18:30:30 = Feb
	  1995-12-22T09:05:02 = Dec
	 -0010-09-15T04:44:23 = Sep

       MMMd (MMM d)

	  2008-02-05T18:30:30 = Feb 5
	  1995-12-22T09:05:02 = Dec 22
	 -0010-09-15T04:44:23 = Sep 15

       MMMEd (E, MMM d)

	  2008-02-05T18:30:30 = Tue, Feb 5
	  1995-12-22T09:05:02 = Fri, Dec 22
	 -0010-09-15T04:44:23 = Sat, Sep 15

       MMMMd (MMMM d)

	  2008-02-05T18:30:30 = February 5
	  1995-12-22T09:05:02 = December 22
	 -0010-09-15T04:44:23 = September 15

       MMMMEd (E, MMMM d)

	  2008-02-05T18:30:30 = Tue, February 5
	  1995-12-22T09:05:02 = Fri, December 22
	 -0010-09-15T04:44:23 = Sat, September 15

       ms (mm:ss)

	  2008-02-05T18:30:30 = 30:30
	  1995-12-22T09:05:02 = 05:02
	 -0010-09-15T04:44:23 = 44:23

       y (y)

	  2008-02-05T18:30:30 = 2008
	  1995-12-22T09:05:02 = 1995
	 -0010-09-15T04:44:23 = -10

       yM (M/yyyy)

	  2008-02-05T18:30:30 = 2/2008
	  1995-12-22T09:05:02 = 12/1995
	 -0010-09-15T04:44:23 = 9/-010

       yMEd (EEE, M/d/yyyy)

	  2008-02-05T18:30:30 = Tue, 2/5/2008
	  1995-12-22T09:05:02 = Fri, 12/22/1995
	 -0010-09-15T04:44:23 = Sat, 9/15/-010

       yMMM (MMM y)

	  2008-02-05T18:30:30 = Feb 2008
	  1995-12-22T09:05:02 = Dec 1995
	 -0010-09-15T04:44:23 = Sep -10

       yMMMEd (EEE, MMM d, y)

	  2008-02-05T18:30:30 = Tue, Feb 5, 2008
	  1995-12-22T09:05:02 = Fri, Dec 22, 1995
	 -0010-09-15T04:44:23 = Sat, Sep 15, -10

       yMMMM (MMMM y)

	  2008-02-05T18:30:30 = February 2008
	  1995-12-22T09:05:02 = December 1995
	 -0010-09-15T04:44:23 = September -10

       yQ (Q yyyy)

	  2008-02-05T18:30:30 = 1 2008
	  1995-12-22T09:05:02 = 4 1995
	 -0010-09-15T04:44:23 = 3 -010

       yQQQ (QQQ y)

	  2008-02-05T18:30:30 = Q1 2008
	  1995-12-22T09:05:02 = Q4 1995
	 -0010-09-15T04:44:23 = Q3 -10

   Miscellaneous
       Prefers 24 hour time?

       No

       Local first day of the week

       Monday

SUPPORT

       See DateTime::Locale.

AUTHOR

       Dave Rolsky <autarch@urth.org>

COPYRIGHT

       Copyright (c) 2008 David Rolsky. All rights reserved. This program is free software; you can redistribute it and/or modify it under the
       same terms as Perl itself.

       This module was generated from data provided by the CLDR project, see the LICENSE.cldr in this distribution for details on the CLDR data's
       license.

perl v5.18.2							    2017-10-06						   DateTime::Locale::en(3)
All times are GMT -4. The time now is 02:44 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy