Sponsored Content
Top Forums Shell Programming and Scripting Filter or remove duplicate block of text without distinguishing marks or fields Post 302563525 by samask on Tuesday 11th of October 2011 12:33:32 PM
Old 10-11-2011
Excellent! It works flawlessly now.

I felt bad that I bothered you to tweak the code, but now I am happy. In that way, looking at how you have improved it, I can learn even more Smilie

Thank you so much for your valuable advice.

PS: If it is all right, I added a rating to this thread, but it should really be a rating to your nice code, more than the thread itself.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate text

Hello, I have a log file which is generated by a script which looks like this: userid: 7 starttime: Sat May 24 23:24:13 CEST 2008 endtime: Sat May 24 23:26:57 CEST 2008 total time spent: 2.73072 minutes / 163.843 seconds date: Sat Jun 7 16:09:03 CEST 2008 userid: 8 starttime: Sun May... (7 Replies)
Discussion started by: dejavu88
7 Replies

2. Shell Programming and Scripting

Filter duplicate block of text using SED

Hi, I would like to print a block of text between 2 regular expression using Sed, This can be achieved by using the command as shown below, however my problem is the same block of text is repeated twice. I would like to eliminate the duplicate block of text. For Example If my file... (5 Replies)
Discussion started by: dkumar91
5 Replies

3. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

4. Shell Programming and Scripting

Filter/remove duplicate .dat file with certain criteria

I am a beginner in Unix. Though have been asked to write a script to filter(remove duplicates) data from a .dat file. File is very huge containig billions of records. contents of file looks like 30002157,40342424,OTC,mart_rec,100, ,0 30002157,40343369,OTC,mart_rec,95, ,0... (6 Replies)
Discussion started by: mukeshguliao
6 Replies

5. Shell Programming and Scripting

Find duplicate based on 'n' fields and mark the duplicate as 'D'

Hi, In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'. In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies

6. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

7. Shell Programming and Scripting

Remove duplicate lines from file based on fields

Dear community, I have to remove duplicate lines from a file contains a very big ammount of rows (milions?) based on 1st and 3rd columns The data are like this: Region 23/11/2014 09:11:36 41752 Medio 23/11/2014 03:11:38 4132 Info 23/11/2014 05:11:09 4323... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

8. Shell Programming and Scripting

How to remove duplicate text blocks from a file?

Hi All I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file. All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this. 10.14.22.22... (3 Replies)
Discussion started by: mahasona
3 Replies

9. Shell Programming and Scripting

Remove duplicate occurrences of text pattern

Hi folks! I have a file which contains a 1000 lines. On each line i have multiple occurrences ( 26 to be exact ) of pattern folder#/folder#. # is depicting the line number in the file some text here folder1/folder1 some text here folder1/folder1 some text here folder1/folder1 some text... (7 Replies)
Discussion started by: martinsmith
7 Replies

10. Shell Programming and Scripting

Filter file to remove duplicate values in first column

Hello, I have a script that is generating a tab delimited output file. num Name PCA_A1 PCA_A2 PCA_A3 0 compound_00 -3.5054 -1.1207 -2.4372 1 compound_01 -2.2641 0.4287 -1.6120 3 compound_03 -1.3053 1.8495 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies
Template::Plugin::Gravatar(3pm) 			User Contributed Perl Documentation			   Template::Plugin::Gravatar(3pm)

NAME
Template::Plugin::Gravatar - configurable TT2-based generation of Gravatar URLs from email addresses. VERSION
0.05 SYNOPSIS
[% USE Gravatar %] [% FOR user IN user_list %] <img src="[% Gravatar( email => user.email ) | html %]" alt="[% user.name | html %]" /> [% END %] # OR a mini CGI example use strict; use CGI qw( header start_html end_html ); use Template; my %config = ( # ... your other config stuff GRAVATAR => { default => "http://myhost.moo/local/image.png", size => 80, rating => "R" }, ); # note the "default" must be an absolute URI to work correctly my $tt2 = Template->new(\%config); my $user = { email => 'whatever@wherever.whichever', rating => "PG", name => "Manamana", size => 75 }; print header(), start_html(); $tt2->process(*DATA, { user => $user }) or warn $Template::ERROR; print end_html(); __DATA__ [% USE Gravatar %] [% FILTER html %] <img src="[% Gravatar( user ) | html %]" alt="[% user.name | html %]" /> [% END %] DESCRIPTION
Please see <http://site.gravatar.com/site/implement> for more on the service interface and <http://site.gravatar.com/> for information about Gravatars (globally recognized avatars) in general. All of the options supported in Gravatars--default, rating, size, and border--can be used here. The gravatar_id is generated from a given email. INTERFACE
/SETTINGS new Not called directly. Called when you "USE" the plugin. Takes defaults from the template config hash and mixes them with any per template defaults. E.g., [% USE Gravatar %] Use config arguments if any. [% USE Gravatar(default => 'http://mysite.moo/local/default-image.gif') %] Mix config arguments, if any, with new instance arguments. Arguments email (required) The key to using Gravatars is a hex hash of the user's email. This is generated automatically and sent to gravatar.com as the "gravatar_id". default (optional) The local (any valid absolute image URI) image to use if there is no Gravatar corresponding to the given email. size (optional) Gravatars are square. Size is 1 through 80 (pixels) and sets the width and the height. rating (optional) G|PG|R|X. The maximum rating of Gravatar you wish returned. If you have a family friendly forum, for example, you might set it to "G." border (optional) A hex color, e.g. FF00FF or F0F. base (developer override) This is provided as a courtesy for the one or two developers who might need it. More below. gravatar_id (not allowed) This is not an option but a generated variable. It is an MD5 hex hash of the email address. The reason is it not supported as an optional variable is it would allow avatar hijacking. The only argument that must be given when you call the "Gravatar" plugin is the email. Everything else -- rating, default image, border, and size -- can be set in three different places: the config, the "USE" call, or the "Gravatar" call. All three of the following produce the same Gravatar URL. Settings via config Used if the entire "site" should rely on one set of defaults. use Template; my %config = ( GRAVATAR => { default => "http://mysite.moo/img/avatar.png", rating => "PG", size => 80, } ); my $template = <<; [% USE Gravatar %] [% Gravatar(email => 'me@myself.ego') | html %] my $tt2 = Template->new(\%config); $tt2->process($template); Settings via instance Used if a particular template needs its own defaults. use Template; my $template = <<; [% USE Gravatar( rating => "PG", size => 80 ) %] [% Gravatar(email => 'me@myself.ego') | html %] my $tt2 = Template->new(); $tt2->process($template); Any other calls with different emails will share the defaults in this template. Settings in the Gravatar call Used for per URL control. use Template; my $template = <<; [% USE Gravatar %] [% Gravatar(email => 'me@myself.ego', default => "http://mysite.moo/img/avatar.png", rating => "PG", size => 80 ) | html %] my $tt2 = Template->new(); $tt2->process($template); Base URL (for developers only) You may also override the base URL for retrieving the Gravatars. It's set to use the service from www.gravatar.com. It can be overridden in the config or the "USE". DIAGNOSTICS
Email is the only required argument. Croaks without it. Size, border, and rating are also validated on each call. Croaks if an invalid size (like 0 or 100) or rating (like MA or NC-17) or border (like ff0 or FF) is given. CONFIGURATION AND ENVIRONMENT
No configuration is necessary. You may use the configuration hash of your new template to pass default information like the default image location for those without Gravatars. You can also set it in the "USE" call per template if needed. DEPENDENCIES (SEE ALSO) Template, Template::Plugin, Carp, Digest::MD5, and URI::Escape. http://www.gravatar.com/ BUGS AND LIMITATIONS
None known. I certainly appreciate bug reports and feedback via "bug-template-plugin-gravatar@rt.cpan.org", or through the web interface at <http://rt.cpan.org/>. AUTHOR
Ashley Pond V "<ashley@cpan.org>". LICENSE
Copyright 2007, Ashley Pond V. This program is free software; you can redistribute it and modify it under the same terms as Perl itself. See http://www.perl.com/perl/misc/Artistic.html. SEE ALSO
Gravatar::URL - standalone Gravatar URL generation. <http://www.gravatar.com> - The Gravatar web site. <http://site.gravatar.com/site/implement> - The Gravatar URL implementation guide. perl v5.12.4 2007-09-09 Template::Plugin::Gravatar(3pm)
All times are GMT -4. The time now is 04:36 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy