Sponsored Content
Top Forums Shell Programming and Scripting Perl- Finding average "frequency" of occurrence of duplicate lines Post 302545287 by acsg on Tuesday 9th of August 2011 12:49:37 AM
Old 08-09-2011
Perl- Finding average "frequency" of occurrence of duplicate lines

Hello,

I am working with a perl script that tries to find the average "frequency" in which lines are duplicated. So far I've only managed to find the way to count how many times the lines are repeated, the code is as follows:

Code:
perl -ae'
my $filename= $ENV{'i'};
open (FILE, "$filename") or die  $!;

my %seen= ();


while(my $line = <FILE>){
  my @fields = split(/\s+/, $line);
  my @fields2= @fields[3..16];
  my $niin= join("\t", @fields2);
  $seen{$niin}++;
  }

foreach my $keys (sort {$seen{$b} <=> $seen{$a}} keys %seen){
    print "$keys = $seen{$keys}\n";
}

close (FILE);


'

Which produces this type of output:

Code:
225    1    225    2    225    3    225    4    225    5    225    6    225    7 = 31789
225    10    225    11    225    12    225    13    225    14    225    15    225    0 = 31772
225    8    225    9    225    10    225    11    225    12    225    13    225    14 = 31714
225    3    225    4    225    5    225    6    225    7    225    8    225    9 = 31686

Now, what I want to do is find a way to find out the number of (in average) "every how many lines a certain line is repeated". So I was wondering if it's possible to have some sort of record and then in the end just calculate the average?

I actually have another way to calculate this frequency. In the original file being read, the first field is a unix timestamp (which i "cut out" for the counting of the duplicate lines). So I thought it would be possible as well to try to keep a record of the "time between repetitions" and then make an average in the end. Of course this would imply keeping a record for each duplicate line, which seems like a rather intricate operation. An example of the lines is :

Code:
1301892853.870    1316    efc0696e        225    1    225    2    225    3    225    4    225    5    225    6    225    7

The first field being the unix timestamp. The first, second and third field are ignored for the comparison of duplicate lines.

Any help is deeply appreciated.

---------- Post updated 08-09-11 at 07:49 AM ---------- Previous update was 08-08-11 at 08:40 AM ----------

Is this really not accomplishable the way I asked for in perl? Is there any other way to do it? Any ideas please? Smilie

Thanks again...

Last edited by pludi; 08-08-2011 at 03:24 AM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies

2. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

3. Solaris

"Load Average" vs "virtual processor"

Hi, I have one question regarding the understanding of “load average” in a platform with virtual processors. Suppose in this situation: Total number of physical processors: 1 Number of virtual processors: 32 Total number of cores: 4 Number of cores per physical... (1 Reply)
Discussion started by: MDING
1 Replies

4. Shell Programming and Scripting

finding the strings beween 2 characters "/" & "/" in .txt file

Hi all. I have a .txt file that I need to sort it My file is like: 1- 88 chain0 MASTER (FF-TE) FFFF 1962510 /TCK T FD2TQHVTT1 /jtagc/jtag_instreg/updateinstr_reg_1 dff1 (TI,SO) 2- ... (10 Replies)
Discussion started by: Behrouzx77
10 Replies

5. Shell Programming and Scripting

Cant get awk 1liner to remove duplicate lines from Delimited file, get "event not found" error..help

Hi, I am on a Solaris8 machine If someone can help me with adjusting this awk 1 liner (turning it into a real awkscript) to get by this "event not found error" ...or Present Perl solution code that works for Perl5.8 in the csh shell ...that would be great. ****************** ... (3 Replies)
Discussion started by: andy b
3 Replies

6. Shell Programming and Scripting

Find lines with "A" then change "E" to "X" same line

I have a bunch of random character lines like ABCEDFG. I want to find all lines with "A" and then change any "E" to "X" in the same line. ALL lines with "A" will have an "X" somewhere in it. I have tried sed awk and vi editor. I get close, not quite there. I know someone has already solved this... (10 Replies)
Discussion started by: nightwatchrenba
10 Replies

7. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

8. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

9. AIX

Apache 2.4 directory cannot display "Last modified" "Size" "Description"

Hi 2 all, i have had AIX 7.2 :/# /usr/IBMAHS/bin/apachectl -v Server version: Apache/2.4.12 (Unix) Server built: May 25 2015 04:58:27 :/#:/# /usr/IBMAHS/bin/apachectl -M Loaded Modules: core_module (static) so_module (static) http_module (static) mpm_worker_module (static) ... (3 Replies)
Discussion started by: penchev
3 Replies
SNMPTABLE(1)							     Net-SNMP							      SNMPTABLE(1)

NAME
snmptable - retrieve an SNMP table and display it in tabular form SYNOPSIS
snmptable [COMMON OPTIONS] [-Cb] [-CB] [-Ch] [-CH] [-Ci] [-Cf STRING] [-Cw WIDTH] AGENT TABLE-OID DESCRIPTION
snmptable is an SNMP application that repeatedly uses the SNMP GETNEXT or GETBULK requests to query for information on a network entity. The parameter TABLE-OID must specify an SNMP table. snmptable is an SNMP application that repeatedly uses the SNMP GETNEXT or GETBULK requests to query for information on a network entity. The parameter TABLE-OID must specify an SNMP table. AGENT identifies a target SNMP agent, which is instrumented to monitor the gievn objects. At its simplest, the AGENT specification will consist of a hostname or an IPv4 address. In this situation, the command will attempt communication with the agent, using UDP/IPv4 to port 161 of the given target host. See snmpcmd(1) for a full list of the possible formats for AGENT. OPTIONS
COMMON OPTIONS Please see snmpcmd(1) for a list of possible values for COMMON OPTIONS as well as their descriptions. -Cb Display only a brief heading. Any common prefix of the table field names will be deleted. -CB Do not use GETBULK requests to retrieve data, only GETNEXT. -Cc CHARS Print table in columns of CHARS characters width. -Cf STRING The string STRING is used to separate table columns. With this option, each table entry will be printed in compact form, just with the string given to separate the columns (useful if you want to import it into a database). Otherwise it is printed in nicely aligned columns. -Ch Display only the column headings. -CH Do not display the column headings. -Ci This option prepends the index of the entry to all printed lines. -Cl Left justify the data in each column. -Cr REPEATERS For GETBULK requests, REPEATERS specifies the max-repeaters value to use. For GETNEXT requests, REPEATERS specifies the number of entries to retrieve at a time. -Cw WIDTH Specifies the width of the lines when the table is printed. If the lines will be longer, the table will be printed in sections of at most WIDTH characters. If WIDTH is less than the length of the contents of a single column, then that single column will still be printed. EXAMPLES
$ snmptable -v 2c -c public localhost at.atTable SNMP table: at.atTable RFC1213-MIB::atTable atIfIndex atPhysAddress atNetAddress 1 8:0:20:20:0:ab 130.225.243.33 $ snmptable -v 2c -c public -Cf + localhost at.atTable SNMP table: at.atTable atIfIndex+atPhysAddress+atNetAddress 1+8:0:20:20:0:ab+130.225.243.33 $ snmptable localhost -Cl -CB -Ci -OX -Cb -Cc 16 -Cw 64 ifTable SNMP table: ifTable Index Descr Type Mtu Speed PhysAddress AdminStatus OperStatus LastChange InOctets InUcastPkts InNUcastPkts InDiscards InErrors InUnknownProtos OutOctets OutUcastPkts OutNUcastPkts OutDiscards OutErrors OutQLen Specific index: [1] 1 lo softwareLoopbac 16436 10000000 up up ? 2837283786 3052466 ? 0 0 ? 2837283786 3052466 ? 0 0 0 zeroDotZero index: [2] 2 eth0 ethernetCsmacd 1500 10000000 0:5:5d:d1:f7:cf up up ? 2052604234 44252973 ? 0 0 ? 149778187 65897282 ? 0 0 0 zeroDotZero BUGS
The test for TABLE-OID actually specifying a table is rather heuristic. Note also that the test requires the defining MIB file to be loaded. SEE ALSO
snmpcmd(1), variables(5). 4th Berkeley Distribution 25 Jul 2003 SNMPTABLE(1)
All times are GMT -4. The time now is 11:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy