Sponsored Content
Top Forums Shell Programming and Scripting PERL "filtering the log file removing the duplicates Post 302837537 by Subbeh on Friday 26th of July 2013 04:02:28 AM
Old 07-26-2013
scriptscript, did you try to do it yourself? With a little bit of research on how to open files and how to use variables in perl you could do it yourself:

Code:
#!/usr/bin/perl

my (%h, $k, $v);

open(my $fh, '<', '/path/to/file') or die "Unable to open file, $!";
while (<$fh>) {
        $_ =~ s/(type|state|TYPE)[0-9]/<$1>/;
        $h{$_}++;
}
close($fh);

while (($k, $v) = each %h) {
        print "$h{$k}\t$k"
}

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

removing duplicates from a file

i have a file with some 1000 entries it will contain entries like 1000,ram 2000,pankaj 1001,rahim 1000,ram 2532,govind 2000,pankaj 3000,venkat 2532,govind what i want is i want to extract only the distinct rows from this file so my output should contain only 1000,ram... (2 Replies)
Discussion started by: trichyselva
2 Replies

2. Shell Programming and Scripting

Removing duplicates in a sorted file by field.

I have data like this: It's sorted by the 2nd field (TID). envoy,90000000000000634600010001,04/11/2008,23:19:27,RB00266,0015,DETAIL,ERROR, envoy,90000000000000634600010001,04/12/2008,04:23:45,RB00266,0015,DETAIL,ERROR,... (1 Reply)
Discussion started by: kinksville
1 Replies

3. UNIX for Dummies Questions & Answers

removing duplicates of a pattern from a file

hey all, I need some help. I have a text file with names in it. My target is that if a particular pattern exists in that file more than once..then i want to rename all the occurences of that pattern by alternate patterns.. for e.g if i have PATTERN occuring 5 times then i want to... (3 Replies)
Discussion started by: ashisharora
3 Replies

4. Shell Programming and Scripting

Removing duplicates from log file?

I have a log file with posts looking like this: -- Messages can be delivered by different systems at different times. The id number is used to sort out duplicate messages. What I need is to strip the arrival time from each post, sort posts by id number, and reattach arrival time to respective... (2 Replies)
Discussion started by: Ilja
2 Replies

5. Shell Programming and Scripting

Removing Duplicates from file

Hi Experts, Please check the following new requirement. I got data like the following in a file. FILE_HEADER 01cbbfde7898410| 3477945| home| 1 01cbc275d2c122| 3478234| WORK| 1 01cbbe4362743da| 3496386| Rich Spare| 1 01cbc275d2c122| 3478234| WORK| 1 This is pipe separated file with... (3 Replies)
Discussion started by: tinufarid
3 Replies

6. Shell Programming and Scripting

formatting a file and removing duplicates

Hi, I have a file that I want to change the format of. It is a large file in rows but I want it to be comma separated (comma then a space). The current file looks like this: HI, Joe, Bob, Jack, Jack After I would want to remove any duplicates so it would look like this: HI, Joe,... (2 Replies)
Discussion started by: kylle345
2 Replies

7. UNIX for Dummies Questions & Answers

Removing duplicates from a file

Hi All, I am merging files coming from 2 different systems ,while doing that I am getting duplicates entries in the merged file I,01,000131,764,2,4.00 I,01,000131,765,2,4.00 I,01,000131,772,2,4.00 I,01,000131,773,2,4.00 I,01,000168,762,2,2.00 I,01,000168,763,2,2.00... (5 Replies)
Discussion started by: Sri3001
5 Replies

8. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3 (2 Replies)
Discussion started by: sagar_1986
2 Replies

9. Shell Programming and Scripting

Removing duplicates from new file

i hav two files like i want to remove/delete all the duplicate lines in file2 which are viz unix,unix2,unix3.I have tried previous post also,but in that complete line must be similar.In this case i have to verify first column only regardless what is the content in succeeding columns. (3 Replies)
Discussion started by: sagar_1986
3 Replies

10. Shell Programming and Scripting

Removing duplicates on a single "column" (delimited file)

Hello ! I'm quite new to linux but haven't found a script to do this task, unfortunately my knowledge is quite limited on shellscripts... Could you guys help me removing the duplicate lines of a file, based only on a single "column"? For example: M202034357;01/2008;J30RJ021;Ciclo 01... (4 Replies)
Discussion started by: Rufinofr
4 Replies
open(3pm)						 Perl Programmers Reference Guide						 open(3pm)

NAME
open - perl pragma to set default PerlIO layers for input and output SYNOPSIS
use open IN => ":crlf", OUT => ":bytes"; use open OUT => ':utf8'; use open IO => ":encoding(iso-8859-7)"; use open IO => ':locale'; use open ':utf8'; use open ':locale'; use open ':encoding(iso-8859-7)'; use open ':std'; DESCRIPTION
Full-fledged support for I/O layers is now implemented provided Perl is configured to use PerlIO as its IO system (which is now the default). The "open" pragma serves as one of the interfaces to declare default "layers" (also known as "disciplines") for all I/O. Any open(), read- pipe() (aka qx//) and similar operators found within the lexical scope of this pragma will use the declared defaults. With the "IN" subpragma you can declare the default layers of input streams, and with the "OUT" subpragma you can declare the default lay- ers of output streams. With the "IO" subpragma you can control both input and output streams simultaneously. If you have a legacy encoding, you can use the ":encoding(...)" tag. if you want to set your encoding layers based on your locale environment variables, you can use the ":locale" tag. For example: $ENV{LANG} = 'ru_RU.KOI8-R'; # the :locale will probe the locale environment variables like LANG use open OUT => ':locale'; open(O, ">koi8"); print O chr(0x430); # Unicode CYRILLIC SMALL LETTER A = KOI8-R 0xc1 close O; open(I, "<koi8"); printf "%#x ", ord(<I>), " "; # this should print 0xc1 close I; These are equivalent use open ':utf8'; use open IO => ':utf8'; as are these use open ':locale'; use open IO => ':locale'; and these use open ':encoding(iso-8859-7)'; use open IO => ':encoding(iso-8859-7)'; The matching of encoding names is loose: case does not matter, and many encodings have several aliases. See Encode::Supported for details and the list of supported locales. Note that ":utf8" PerlIO layer must always be specified exactly like that, it is not subject to the loose matching of encoding names. When open() is given an explicit list of layers they are appended to the list declared using this pragma. The ":std" subpragma on its own has no effect, but if combined with the ":utf8" or ":encoding" subpragmas, it converts the standard file- handles (STDIN, STDOUT, STDERR) to comply with encoding selected for input/output handles. For example, if both input and out are chosen to be ":utf8", a ":std" will mean that STDIN, STDOUT, and STDERR are also in ":utf8". On the other hand, if only output is chosen to be in ":encoding(koi8r)", a ":std" will cause only the STDOUT and STDERR to be in "koi8r". The ":locale" subpragma implicitly turns on ":std". The logic of ":locale" is as follows: 1. If the platform supports the langinfo(CODESET) interface, the codeset returned is used as the default encoding for the open pragma. 2. If 1. didn't work but we are under the locale pragma, the environment variables LC_ALL and LANG (in that order) are matched for encod- ings (the part after ".", if any), and if any found, that is used as the default encoding for the open pragma. 3. If 1. and 2. didn't work, the environment variables LC_ALL and LANG (in that order) are matched for anything looking like UTF-8, and if any found, ":utf8" is used as the default encoding for the open pragma. If your locale environment variables (LC_ALL, LC_CTYPE, LANG) contain the strings 'UTF-8' or 'UTF8' (case-insensitive matching), the default encoding of your STDIN, STDOUT, and STDERR, and of any subsequent file open, is UTF-8. Directory handles may also support PerlIO layers in the future. NONPERLIO FUNCTIONALITY
If Perl is not built to use PerlIO as its IO system then only the two pseudo-layers ":bytes" and ":crlf" are available. The ":bytes" layer corresponds to "binary mode" and the ":crlf" layer corresponds to "text mode" on platforms that distinguish between the two modes when opening files (which is many DOS-like platforms, including Windows). These two layers are no-ops on platforms where bin- mode() is a no-op, but perform their functions everywhere if PerlIO is enabled. IMPLEMENTATION DETAILS
There is a class method in "PerlIO::Layer" "find" which is implemented as XS code. It is called by "import" to validate the layers: PerlIO::Layer::->find("perlio") The return value (if defined) is a Perl object, of class "PerlIO::Layer" which is created by the C code in perlio.c. As yet there is noth- ing useful you can do with the object at the perl level. SEE ALSO
"binmode" in perlfunc, "open" in perlfunc, perlunicode, PerlIO, encoding perl v5.8.0 2002-06-01 open(3pm)
All times are GMT -4. The time now is 05:11 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy