How to calculate the entropy of a single directory that contains many files Post: 302358619

Sponsored Content

Top Forums Shell Programming and Scripting How to calculate the entropy of a single directory that contains many files Post 302358619 by draxmas on Saturday 3rd of October 2009 01:27:14 PM

10-03-2009

Registered User

How to calculate the entropy of a single directory that contains many files

Hello,
I'm new member of shell scripting and i face some difficulties. To begin, i try to write an algorithm that calculate from one directory containing nfdump files (288) the entropy of one day 24hours. Each of the file is 5 min interval
(nfdump -r nfcapd.200908250000 -s srcip) 1st
(nfdump -r nfcapd.200908250005 -s srcip) 2nd
(nfdump -r nfcapd.200908250010 -s srcip) 3rd
...
(nfdump -r nfcapd.200908252350 -s srcip) last

taking the flows (first 10) to calculate the entropy of the first 5 min interval i did it (i hope so in the right way) and i took the results of the 1st 5 min interval entropy. the code that i have written is:

Code:

#!/bin/sh
#x1 to x10 are variables which include values. Top 10 srcip Addr ordered by flows
x1=7576
x2=5691
x3=5130
x4=4239
x5=4136
x6=3387
x7=3070
x8=2780
x9=2125
x10=2011
#sum is the total flows
sum=551892
#calculate the values for each srcip Addr (10)
calx1=`echo "scale=6;$x1 / $sum" | bc`
calx2=`echo "scale=6;$x2 / $sum" | bc`
calx3=`echo "scale=6;$x3 / $sum" | bc`
calx4=`echo "scale=6;$x4 / $sum" | bc`
calx5=`echo "scale=6;$x5 / $sum" | bc`
calx6=`echo "scale=6;$x6 / $sum" | bc`
calx7=`echo "scale=6;$x7 / $sum" | bc`
calx8=`echo "scale=6;$x8 / $sum" | bc`
calx9=`echo "scale=6;$x9 / $sum" | bc`
calx10=`echo "scale=6;$x10 / $sum" | bc`
#calculate the logarithmic values from x1 to x10
l1=`echo "scale=6;l (${calx1})" | bc -l`
l2=`echo "scale=6;l (${calx2})" | bc -l`
l3=`echo "scale=6;l (${calx3})" | bc -l`
l4=`echo "scale=6;l (${calx4})" | bc -l`
l5=`echo "scale=6;l (${calx5})" | bc -l`
l6=`echo "scale=6;l (${calx6})" | bc -l`
l7=`echo "scale=6;l (${calx7})" | bc -l`
l8=`echo "scale=6;l (${calx8})" | bc -l`
l9=`echo "scale=6;l (${calx9})" | bc -l`
l10=`echo "scale=6;l (${calx10})" | bc -l`
#calculate the total entropy for 5 min interval
RESULT=`echo "scale=6;${calx1} * ${l1} + ${calx2} * ${l2} + ${calx3} * ${l3} + ${calx4} * ${l4} + ${calx5} * ${l5} + ${calx6} * ${l6} + ${calx7} * ${l7} + ${calx8} * ${l8} + ${calx9} * ${l9} + ${calx10} * ${l10}" | bc -l`
echo "Entropy for 5 min interval is ${RESULT}"

I used the shannon entropy formula for this.

My main problem now is how to make a matrix -array of all 288 files of 5 min interval and how is possible to combine the above algorithm with the matrix? Please if someone knows how to do this just to give me some help - advise
Thanks

draxmas

View Public Profile for draxmas

Find all posts by draxmas

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to calculate file's size in directory and subdirectory

Hi, I have written one script to calculate total space of all file in one directory, ignoring subdirectory, it works fine. Now, I've been trying to calculate all files which includes files in any subdirectories. I use recursive function to do this, but it can work only if there is only one...

2. Emergency UNIX and Linux Support

calculate internal fragmentation in directory

hi sorry for grammar mistakes but i am writting these fro tablet and am not realy used to ot so well yet.... i am in the middle of doing work here and hope some1 ca answear my question asap please :-) How to calculate the amount of internal fragmentation using script. cd directory ......

3. Shell Programming and Scripting

Find all the files under a specific directory and zip them into a single file.

Hi, Is there a way to find all the files from a specific location and then zip them into a single file, even if they are in multiple directories?

4. Shell Programming and Scripting

how to find first files in a directory and combine them as a single file?

i have below list of files in a directory. /root/admin/files/file1.txt /root/admin/files/file2.txt /root/admin/files/file3.txt /root/admin/files/pattern.txt /root/admin/files/server.txt i need combine the above text files in the below sequence, file1.txt, pattern.txt,server.txt =>...

5. Shell Programming and Scripting

Change to directory and search some file in that directory in single command

I am trying to do the following task : export ENV=aaa export ENV_PATH=$(cd /apps | ls | grep $ENV) However, it's not working. What's the way to change to directory and search some file in that directory in single command Please help.

6. Shell Programming and Scripting

Paste 2 single column files to a single file

Hi, I have 2 csv/txt files with single columns. I am trying to merge them using paste, but its not working.. output3.csv: flowerbomb everlon-jewelry sofft steve-madden dolce-gabbana-watchoutput2.csv: http://www1.abc.com/cms/slp/2/Flowerbomb http://www1.abc.com/cms/slp/2/Everlon-Jewelry...

7. Shell Programming and Scripting

Bash to calculate average of all files in directory and output by part of filename

I am trying to use awk to calculate the average of all lines in $2 for every file in a directory. The below bash seems to do that, but I cannot figure out how to capture the string before the _ as the output file name and have it be tab-delimeted. Thank you :). Filenames in...

8. Shell Programming and Scripting

Copy files recursively to one single directory

I need to copy a complete directory structure into a new location. But I want to have all files copied into one directory and leave out the directory structure. So all files must be placed in one directory.

9. Shell Programming and Scripting

Move multiple files to different directory using a single command

I have multiple files test1, test2, test3 etc. I want to move to a different directory with ABC_ prefixed to every file and and current dat time as postfix using a single command. (I will be using this is sftp with ! (command for local server). I have tried the following but it gives error ...

LEARN ABOUT DEBIAN

dlassq.f

dlassq.f(3)							      LAPACK							       dlassq.f(3)

NAME

       dlassq.f -

SYNOPSIS

   Functions/Subroutines
       subroutine dlassq (N, X, INCX, SCALE, SUMSQ)
	   DLASSQ

Function/Subroutine Documentation
   subroutine dlassq (integerN, double precision, dimension( * )X, integerINCX, double precisionSCALE, double precisionSUMSQ)
       DLASSQ

       Purpose:

	    DLASSQ  returns the values	scl  and  smsq	such that

	       ( scl**2 )*smsq = x( 1 )**2 +...+ x( n )**2 + ( scale**2 )*sumsq,

	    where  x( i ) = X( 1 + ( i - 1 )*INCX ). The value of  sumsq  is
	    assumed to be non-negative and  scl  returns the value

	       scl = max( scale, abs( x( i ) ) ).

	    scale and sumsq must be supplied in SCALE and SUMSQ and
	    scl and smsq are overwritten on SCALE and SUMSQ respectively.

	    The routine makes only one pass through the vector x.

       Parameters:
	   N

		     N is INTEGER
		     The number of elements to be used from the vector X.

	   X

		     X is DOUBLE PRECISION array, dimension (N)
		     The vector for which a scaled sum of squares is computed.
			x( i )	= X( 1 + ( i - 1 )*INCX ), 1 <= i <= n.

	   INCX

		     INCX is INTEGER
		     The increment between successive values of the vector X.
		     INCX > 0.

	   SCALE

		     SCALE is DOUBLE PRECISION
		     On entry, the value  scale  in the equation above.
		     On exit, SCALE is overwritten with  scl , the scaling factor
		     for the sum of squares.

	   SUMSQ

		     SUMSQ is DOUBLE PRECISION
		     On entry, the value  sumsq  in the equation above.
		     On exit, SUMSQ is overwritten with  smsq , the basic sum of
		     squares from which  scl  has been factored out.

       Author:
	   Univ. of Tennessee

	   Univ. of California Berkeley

	   Univ. of Colorado Denver

	   NAG Ltd.

       Date:
	   November 2011

       Definition at line 104 of file dlassq.f.

Author
       Generated automatically by Doxygen for LAPACK from the source code.

Version 3.4.1							  Sun May 26 2013						       dlassq.f(3)