Sponsored Content
Top Forums Shell Programming and Scripting Perl- Finding average "frequency" of occurrence of duplicate lines Post 302545287 by acsg on Tuesday 9th of August 2011 12:49:37 AM
Old 08-09-2011
Perl- Finding average "frequency" of occurrence of duplicate lines

Hello,

I am working with a perl script that tries to find the average "frequency" in which lines are duplicated. So far I've only managed to find the way to count how many times the lines are repeated, the code is as follows:

Code:
perl -ae'
my $filename= $ENV{'i'};
open (FILE, "$filename") or die  $!;

my %seen= ();


while(my $line = <FILE>){
  my @fields = split(/\s+/, $line);
  my @fields2= @fields[3..16];
  my $niin= join("\t", @fields2);
  $seen{$niin}++;
  }

foreach my $keys (sort {$seen{$b} <=> $seen{$a}} keys %seen){
    print "$keys = $seen{$keys}\n";
}

close (FILE);


'

Which produces this type of output:

Code:
225    1    225    2    225    3    225    4    225    5    225    6    225    7 = 31789
225    10    225    11    225    12    225    13    225    14    225    15    225    0 = 31772
225    8    225    9    225    10    225    11    225    12    225    13    225    14 = 31714
225    3    225    4    225    5    225    6    225    7    225    8    225    9 = 31686

Now, what I want to do is find a way to find out the number of (in average) "every how many lines a certain line is repeated". So I was wondering if it's possible to have some sort of record and then in the end just calculate the average?

I actually have another way to calculate this frequency. In the original file being read, the first field is a unix timestamp (which i "cut out" for the counting of the duplicate lines). So I thought it would be possible as well to try to keep a record of the "time between repetitions" and then make an average in the end. Of course this would imply keeping a record for each duplicate line, which seems like a rather intricate operation. An example of the lines is :

Code:
1301892853.870    1316    efc0696e        225    1    225    2    225    3    225    4    225    5    225    6    225    7

The first field being the unix timestamp. The first, second and third field are ignored for the comparison of duplicate lines.

Any help is deeply appreciated.

---------- Post updated 08-09-11 at 07:49 AM ---------- Previous update was 08-08-11 at 08:40 AM ----------

Is this really not accomplishable the way I asked for in perl? Is there any other way to do it? Any ideas please? Smilie

Thanks again...

Last edited by pludi; 08-08-2011 at 03:24 AM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies

2. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

3. Solaris

"Load Average" vs "virtual processor"

Hi, I have one question regarding the understanding of “load average” in a platform with virtual processors. Suppose in this situation: Total number of physical processors: 1 Number of virtual processors: 32 Total number of cores: 4 Number of cores per physical... (1 Reply)
Discussion started by: MDING
1 Replies

4. Shell Programming and Scripting

finding the strings beween 2 characters "/" & "/" in .txt file

Hi all. I have a .txt file that I need to sort it My file is like: 1- 88 chain0 MASTER (FF-TE) FFFF 1962510 /TCK T FD2TQHVTT1 /jtagc/jtag_instreg/updateinstr_reg_1 dff1 (TI,SO) 2- ... (10 Replies)
Discussion started by: Behrouzx77
10 Replies

5. Shell Programming and Scripting

Cant get awk 1liner to remove duplicate lines from Delimited file, get "event not found" error..help

Hi, I am on a Solaris8 machine If someone can help me with adjusting this awk 1 liner (turning it into a real awkscript) to get by this "event not found error" ...or Present Perl solution code that works for Perl5.8 in the csh shell ...that would be great. ****************** ... (3 Replies)
Discussion started by: andy b
3 Replies

6. Shell Programming and Scripting

Find lines with "A" then change "E" to "X" same line

I have a bunch of random character lines like ABCEDFG. I want to find all lines with "A" and then change any "E" to "X" in the same line. ALL lines with "A" will have an "X" somewhere in it. I have tried sed awk and vi editor. I get close, not quite there. I know someone has already solved this... (10 Replies)
Discussion started by: nightwatchrenba
10 Replies

7. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

8. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

9. AIX

Apache 2.4 directory cannot display "Last modified" "Size" "Description"

Hi 2 all, i have had AIX 7.2 :/# /usr/IBMAHS/bin/apachectl -v Server version: Apache/2.4.12 (Unix) Server built: May 25 2015 04:58:27 :/#:/# /usr/IBMAHS/bin/apachectl -M Loaded Modules: core_module (static) so_module (static) http_module (static) mpm_worker_module (static) ... (3 Replies)
Discussion started by: penchev
3 Replies
RCORDER(8)						    BSD System Manager's Manual 						RCORDER(8)

NAME
rcorder -- print a dependency ordering of interdependent files SYNOPSIS
rcorder [-k keep] [-s skip] file ... DESCRIPTION
The rcorder utility is designed to print out a dependency ordering of a set of interdependent files. Typically it is used to find an execu- tion sequence for a set of shell scripts in which certain files must be executed before others. Each file passed to rcorder must be annotated with special lines (which look like comments to the shell) which indicate the dependencies the files have upon certain points in the sequence, known as ``conditions'', and which indicate, for each file, which ``conditions'' may be expected to be filled by that file. Within each file, a block containing a series of ``REQUIRE'', ``PROVIDE'', ``BEFORE'' and ``KEYWORD'' lines must appear. The format of the lines is rigid. Each line must begin with a single '#', followed by a single space, followed by ``PROVIDE:'', ``REQUIRE:'', ``BEFORE:'', or ``KEYWORD:''. No deviation is permitted. Each dependency line is then followed by a series of conditions, separated by whitespace. Multi- ple ``PROVIDE'', ``REQUIRE'', ``BEFORE'' and ``KEYWORD'' lines may appear, but all such lines must appear in a sequence without any interven- ing lines, as once a line that does not follow the format is reached, parsing stops. The options are as follows: -k Add the specified keyword to the ``keep list''. If any -k option is given, only those files containing the matching keyword are listed. -s Add the specified keyword to the ``skip list''. If any -s option is given, files containing the matching keyword are not listed. An example block follows: # REQUIRE: networking syslog # REQUIRE: usr # PROVIDE: dns nscd This block states that the file in which it appears depends upon the ``networking'', ``syslog'', and ``usr'' conditions, and provides the ``dns'' and ``nscd'' conditions. A file may contain zero ``PROVIDE'' lines, in which case it provides no conditions, and may contain zero ``REQUIRE'' lines, in which case it has no dependencies. There must be at least one file with no dependencies in the set of arguments passed to rcorder in order for it to find a starting place in the dependency ordering. DIAGNOSTICS
The rcorder utility may print one of the following error messages and exit with a non-zero status if it encounters an error while processing the file list. Requirement %s has no providers, aborting. No file has a ``PROVIDE'' line corresponding to a condition present in a ``REQUIRE'' line in another file. Circular dependency on provision %s, aborting. A set of files has a circular dependency which was detected while processing the stated con- dition. Circular dependency on file %s, aborting. A set of files has a circular dependency which was detected while processing the stated file. SEE ALSO
rc(8) HISTORY
The rcorder utility first appeared in NetBSD 1.5. AUTHORS
Written by Perry E. Metzger <perry@piermont.com> and Matthew R. Green <mrg@eterna.com.au>. BUGS
The ``REQUIRE'' keyword is misleading: It doesn't describe which daemons have to be running before a script will be started. It describes which scripts must be placed before it in the dependency ordering. For example, if your script has a ``REQUIRE'' on ``named'', it means the script must be placed after the ``named'' script in the dependency ordering, not necessarily that it requires named(8) to be started or enabled. BSD
August 5, 2011 BSD
All times are GMT -4. The time now is 11:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy