Sponsored Content
Top Forums Shell Programming and Scripting How to find duplicate line in log file? Post 302424720 by wax_light on Wednesday 26th of May 2010 06:52:44 AM
Old 05-26-2010
wow that is really fast, thank you,

I will try it, I use Bourne Shell

I hope it will work
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to find Duplicate Records in a text file

Hi all pls help me by providing soln for my problem I'm having a text file which contains duplicate records . Example: abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 abc 1000 3452 2463 2343 2176 7654 3452 8765 5643 3452 tas 3420 3562 ... (1 Reply)
Discussion started by: G.Aavudai
1 Replies

2. Shell Programming and Scripting

How to : Find duplicate number from file? with bash

Thanks AVKlinux (6 Replies)
Discussion started by: avklinux
6 Replies

3. Shell Programming and Scripting

find out duplicate records in file?

Dear All, I have one file which looks like : account1:passwd1 account2:passwd2 account3:passwd3 account1:passwd4 account5:passwd5 account6:passwd6 you can see there're two records for account1. and is there any shell command which can find out : account1 is the duplicate record in... (3 Replies)
Discussion started by: tiger2000
3 Replies

4. Shell Programming and Scripting

Find c28 to next comma in line of log file

I have somelogfile.log where I'm trying to find only the numbers which would start at c28 on the third line from the bottom and end at the next comma, so for somelogfile.log: bunch of text - 0, some other text - 123 or bunch of text - 123, some other text - 192831923 where I want either the... (5 Replies)
Discussion started by: unclecameron
5 Replies

5. Shell Programming and Scripting

Find Duplicate records in first Column in File

Hi, Need to find a duplicate records on the first column, ANU4501710430989 0000000W20389390 ANU4501710430989 0000000W67065483 ANU4501130050520 0000000W80838713 ANU4501210170685 0000000W69246611... (3 Replies)
Discussion started by: Murugesh
3 Replies

6. UNIX for Dummies Questions & Answers

CSV file:Find duplicates, save original and duplicate records in a new file

Hi Unix gurus, Maybe it is too much to ask for but please take a moment and help me out. A very humble request to you gurus. I'm new to Unix and I have started learning Unix. I have this project which is way to advanced for me. File format: CSV file File has four columns with no header... (8 Replies)
Discussion started by: arvindosu
8 Replies

7. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

8. Shell Programming and Scripting

How to find duplicate line in Linux?

Hi, Gurus, I need find the duplicate record in unix file. what command I should use for this. Thanks in advance (4 Replies)
Discussion started by: ken6503
4 Replies

9. Shell Programming and Scripting

Honey, I broke awk! (duplicate line removal in 30M line 3.7GB csv file)

I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code: awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies

10. UNIX for Dummies Questions & Answers

Log file - Delete duplicate line & keep last date

Hello All ! I need your help on this case, I have a csv file with this: ITEM105;ARI FSR;2016-02-01 08:02;243 ITEM101;ARI FSR;2016-02-01 06:02;240 ITEM032;RNO TLE;2016-02-01 11:03;320 ITEM032;RNO TLE;2016-02-02 05:43;320 ITEM032;RNO TLE;2016-02-01 02:03;320 ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies
Pod::Simple::PullParser(3pm)				 Perl Programmers Reference Guide			      Pod::Simple::PullParser(3pm)

NAME
Pod::Simple::PullParser -- a pull-parser interface to parsing Pod SYNOPSIS
my $parser = SomePodProcessor->new; $parser->set_source( "whatever.pod" ); $parser->run; Or: my $parser = SomePodProcessor->new; $parser->set_source( $some_filehandle_object ); $parser->run; Or: my $parser = SomePodProcessor->new; $parser->set_source( $document_source ); $parser->run; Or: my $parser = SomePodProcessor->new; $parser->set_source( @document_lines ); $parser->run; And elsewhere: require 5; package SomePodProcessor; use strict; use base qw(Pod::Simple::PullParser); sub run { my $self = shift; Token: while(my $token = $self->get_token) { ...process each token... } } DESCRIPTION
This class is for using Pod::Simple to build a Pod processor -- but one that uses an interface based on a stream of token objects, instead of based on events. This is a subclass of Pod::Simple and inherits all its methods. A subclass of Pod::Simple::PullParser should define a "run" method that calls "$token = $parser->get_token" to pull tokens. See the source for Pod::Simple::RTF for an example of a formatter that uses Pod::Simple::PullParser. METHODS
my $token = $parser->get_token This returns the next token object (which will be of a subclass of Pod::Simple::PullParserToken), or undef if the parser-stream has hit the end of the document. $parser->unget_token( $token ) $parser->unget_token( $token1, $token2, ... ) This restores the token object(s) to the front of the parser stream. The source has to be set before you can parse anything. The lowest-level way is to call "set_source": $parser->set_source( $filename ) $parser->set_source( $filehandle_object ) $parser->set_source( $document_source ) $parser->set_source( @document_lines ) Or you can call these methods, which Pod::Simple::PullParser has defined to work just like Pod::Simple's same-named methods: $parser->parse_file(...) $parser->parse_string_document(...) $parser->filter(...) $parser->parse_from_file(...) For those to work, the Pod-processing subclass of Pod::Simple::PullParser has to have defined a $parser->run method -- so it is advised that all Pod::Simple::PullParser subclasses do so. See the Synopsis above, or the source for Pod::Simple::RTF. Authors of formatter subclasses might find these methods useful to call on a parser object that you haven't started pulling tokens from yet: my $title_string = $parser->get_title This tries to get the title string out of $parser, by getting some tokens, and scanning them for the title, and then ungetting them so that you can process the token-stream from the beginning. For example, suppose you have a document that starts out: =head1 NAME Hoo::Boy::Wowza -- Stuff B<wow> yeah! $parser->get_title on that document will return "Hoo::Boy::Wowza -- Stuff wow yeah!". If the document starts with: =head1 Name Hoo::Boy::W00t -- Stuff B<w00t> yeah! Then you'll need to pass the "nocase" option in order to recognize "Name": $parser->get_title(nocase => 1); In cases where get_title can't find the title, it will return empty-string (""). my $title_string = $parser->get_short_title This is just like get_title, except that it returns just the modulename, if the title seems to be of the form "SomeModuleName -- description". For example, suppose you have a document that starts out: =head1 NAME Hoo::Boy::Wowza -- Stuff B<wow> yeah! then $parser->get_short_title on that document will return "Hoo::Boy::Wowza". But if the document starts out: =head1 NAME Hooboy, stuff B<wow> yeah! then $parser->get_short_title on that document will return "Hooboy, stuff wow yeah!". If the document starts with: =head1 Name Hoo::Boy::W00t -- Stuff B<w00t> yeah! Then you'll need to pass the "nocase" option in order to recognize "Name": $parser->get_short_title(nocase => 1); If the title can't be found, then get_short_title returns empty-string (""). $author_name = $parser->get_author This works like get_title except that it returns the contents of the "=head1 AUTHOR Paragraph... " section, assuming that that section isn't terribly long. To recognize a "=head1 Author Paragraph " section, pass the "nocase" otpion: $parser->get_author(nocase => 1); (This method tolerates "AUTHORS" instead of "AUTHOR" too.) $description_name = $parser->get_description This works like get_title except that it returns the contents of the "=head1 DESCRIPTION Paragraph... " section, assuming that that section isn't terribly long. To recognize a "=head1 Description Paragraph " section, pass the "nocase" otpion: $parser->get_description(nocase => 1); $version_block = $parser->get_version This works like get_title except that it returns the contents of the "=head1 VERSION [BIG BLOCK] " block. Note that this does NOT return the module's $VERSION!! To recognize a "=head1 Version [BIG BLOCK] " section, pass the "nocase" otpion: $parser->get_version(nocase => 1); NOTE
You don't actually have to define a "run" method. If you're writing a Pod-formatter class, you should define a "run" just so that users can call "parse_file" etc, but you don't have to. And if you're not writing a formatter class, but are instead just writing a program that does something simple with a Pod::PullParser object (and not an object of a subclass), then there's no reason to bother subclassing to add a "run" method. SEE ALSO
Pod::Simple Pod::Simple::PullParserToken -- and its subclasses Pod::Simple::PullParserStartToken, Pod::Simple::PullParserTextToken, and Pod::Simple::PullParserEndToken. HTML::TokeParser, which inspired this. SUPPORT
Questions or discussion about POD and Pod::Simple should be sent to the pod-people@perl.org mail list. Send an empty email to pod-people-subscribe@perl.org to subscribe. This module is managed in an open GitHub repository, <https://github.com/theory/pod-simple/>. Feel free to fork and contribute, or to clone <git://github.com/theory/pod-simple.git> and send patches! Patches against Pod::Simple are welcome. Please send bug reports to <bug-pod-simple@rt.cpan.org>. COPYRIGHT AND DISCLAIMERS
Copyright (c) 2002 Sean M. Burke. This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. AUTHOR
Pod::Simple was created by Sean M. Burke <sburke@cpan.org>. But don't bother him, he's retired. Pod::Simple is maintained by: o Allison Randal "allison@perl.org" o Hans Dieter Pearcey "hdp@cpan.org" o David E. Wheeler "dwheeler@cpan.org" perl v5.18.2 2013-11-14 Pod::Simple::PullParser(3pm)
All times are GMT -4. The time now is 04:42 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy