09-02-2011
Removing Duplicates from file
Hi Experts,
Please check the following new requirement. I got data like the following in a file.
HTML Code:
FILE_HEADER
01cbbfde7898410| 3477945| home| 1
01cbc275d2c122| 3478234| WORK| 1
01cbbe4362743da| 3496386| Rich Spare| 1
01cbc275d2c122| 3478234| WORK| 1
This is pipe separated file with column 2,3 as key columns. The file should be formatted to the following output files
1) All records other than the duplicates
HTML Code:
FILE_HEADER
01cbbfde7898410| 3477945| home| 1
01cbbe4362743da| 3496386| Rich Spare| 1
2) The dupicate key file
Any thoughts on this.
Note:- The 'FILE_HEADER' should be there in the first file.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
i have a file with some 1000 entries it will contain entries like
1000,ram
2000,pankaj
1001,rahim
1000,ram
2532,govind
2000,pankaj
3000,venkat
2532,govind
what i want is i want to extract only the distinct rows from this file
so my output should contain only
1000,ram... (2 Replies)
Discussion started by: trichyselva
2 Replies
2. Shell Programming and Scripting
I have data like this:
It's sorted by the 2nd field (TID).
envoy,90000000000000634600010001,04/11/2008,23:19:27,RB00266,0015,DETAIL,ERROR,
envoy,90000000000000634600010001,04/12/2008,04:23:45,RB00266,0015,DETAIL,ERROR,... (1 Reply)
Discussion started by: kinksville
1 Replies
3. UNIX for Dummies Questions & Answers
hey all,
I need some help.
I have a text file with names in it.
My target is that if a particular pattern exists in that file more than once..then i want to rename all the occurences of that pattern by alternate patterns..
for e.g if i have PATTERN occuring 5 times then i want to... (3 Replies)
Discussion started by: ashisharora
3 Replies
4. Shell Programming and Scripting
I have a log file with posts looking like this:
--
Messages can be delivered by different systems at different times. The id number is used to sort out duplicate messages. What I need is to strip the arrival time from each post, sort posts by id number, and reattach arrival time to respective... (2 Replies)
Discussion started by: Ilja
2 Replies
5. Shell Programming and Scripting
Hi,
I have a file that I want to change the format of. It is a large file in rows but I want it to be comma separated (comma then a space).
The current file looks like this:
HI, Joe, Bob, Jack, Jack
After I would want to remove any duplicates so it would look like this:
HI, Joe,... (2 Replies)
Discussion started by: kylle345
2 Replies
6. Shell Programming and Scripting
Hi all,
I am working with a huge amount of files in a Linux environment and I was trying to filter my data. Here's what my data looks like
Name............................Size
OLUSDN.gf.gif-1.JPEG.......5 kb
LKJFDA01.gf.gif-1.JPEG.....3 kb
LKJFDA01.gf.gif-2.JPEG.....1 kb... (7 Replies)
Discussion started by: Error404
7 Replies
7. UNIX for Dummies Questions & Answers
Hi All,
I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file
I,01,000131,764,2,4.00
I,01,000131,765,2,4.00
I,01,000131,772,2,4.00
I,01,000131,773,2,4.00
I,01,000168,762,2,2.00
I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies
8. UNIX for Dummies Questions & Answers
I have been using grep to output whole lines using a pattern file with identifiers (fileA):
fig|562.2322.peg.1
fig|562.2322.peg.3
fig|562.2322.peg.3
fig|562.2322.peg.3
fig|562.2322.peg.7
From fileB with corresponding identifiers in the second column:
NODE_0 fig|562.2322.peg.1 peg ... (2 Replies)
Discussion started by: Mauve
2 Replies
9. Shell Programming and Scripting
i hav two files like
i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies
10. Shell Programming and Scripting
i hav two files like
i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3.I have tried previous post also,but in that complete line must be similar.In this case i have to verify first column only regardless what is the content in succeeding columns. (3 Replies)
Discussion started by: sagar_1986
3 Replies
LEARN ABOUT MOJAVE
html::filter5.18
HTML::Filter(3) User Contributed Perl Documentation HTML::Filter(3)
NAME
HTML::Filter - Filter HTML text through the parser
NOTE
This module is deprecated. The "HTML::Parser" now provides the functionally of "HTML::Filter" much more efficiently with the the "default"
handler.
SYNOPSIS
require HTML::Filter;
$p = HTML::Filter->new->parse_file("index.html");
DESCRIPTION
"HTML::Filter" is an HTML parser that by default prints the original text of each HTML element (a slow version of cat(1) basically). The
callback methods may be overridden to modify the filtering for some HTML elements and you can override output() method which is called to
print the HTML text.
"HTML::Filter" is a subclass of "HTML::Parser". This means that the document should be given to the parser by calling the $p->parse() or
$p->parse_file() methods.
EXAMPLES
The first example is a filter that will remove all comments from an HTML file. This is achieved by simply overriding the comment method to
do nothing.
package CommentStripper;
require HTML::Filter;
@ISA=qw(HTML::Filter);
sub comment { } # ignore comments
The second example shows a filter that will remove any <TABLE>s found in the HTML file. We specialize the start() and end() methods to
count table tags and then make output not happen when inside a table.
package TableStripper;
require HTML::Filter;
@ISA=qw(HTML::Filter);
sub start
{
my $self = shift;
$self->{table_seen}++ if $_[0] eq "table";
$self->SUPER::start(@_);
}
sub end
{
my $self = shift;
$self->SUPER::end(@_);
$self->{table_seen}-- if $_[0] eq "table";
}
sub output
{
my $self = shift;
unless ($self->{table_seen}) {
$self->SUPER::output(@_);
}
}
If you want to collect the parsed text internally you might want to do something like this:
package FilterIntoString;
require HTML::Filter;
@ISA=qw(HTML::Filter);
sub output { push(@{$_[0]->{fhtml}}, $_[1]) }
sub filtered_html { join("", @{$_[0]->{fhtml}}) }
SEE ALSO
HTML::Parser
COPYRIGHT
Copyright 1997-1999 Gisle Aas.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
perl v5.18.2 2013-03-25 HTML::Filter(3)