Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How big is my awk array?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How big is my awk array?
# 1  
Old 09-10-2007
How big is my awk array?
Hi All,

I'm creating a script that goes through some csv files (output from sar) trying to get some statistics on system performance. This is the code I have so far:

Code:
 
awk -F"\",\"" 'NR != 1 {
                           per[$10]++
                           sum += $10
                   }
                   END {
                       print ARGV[ARGIND]
                       for (i in per)
                           print i" : "per[i]
                       print "Total : "sum
                       print "Lines : "(NR -1)
                   }' $FILES

The input files have a header line which accounts for the "NR !=1" and "(NR - 1)".

What I need to do is normalise the data working out the 90th and 95th percentile (I haven't quite got to the math part yet - I'm going to try and calculate the cumulative frequency first... ). So what I believe I need to do next is sort my array - but I don't know how many elements are stored in it. Can anyone tell me how to do this?

Some example output here (from just one input file):

Code:
snsax-psmpw-5-2007-01-07_MemorySwapUtilisation.csv
9.63 : 1
9.64 : 20
9.6 : 1
9.65 : 29
9.66 : 17
9.87 : 11
9.78 : 1
9.89 : 1
9.61 : 38
9.62 : 24
Total : 1380.4
Lines : 143

Thanks in advance Smilie
# 2  
Old 09-10-2007
Use a counter:

Code:
awk -F"\",\"" 'NR != 1 {
                           per[$10]++
                           sum += $10
                           if ( per[$10] == 1 )
                                c++
                   }
                   END {
                       print ARGV[ARGIND]
                       print "Array obj:",c
                       for (i in per)
                           print i" : "per[i]
                       print "Total : "sum
                       print "Lines : "(NR -1)
                   }' $FILES

# 3  
Old 09-10-2007
Hi Klashxx,

It works - thanks!

Can you explain it to me because I'm a little confused... I can't understand why it works at any other time apart from the first pass - when the variable per[$10] is equal to one...

Many thanks.

p.
# 4  
Old 09-10-2007
or altrnatively.. change
Code:
                           per[$10]++
                           sum += $10
                           if ( per[$10] == 1 )
                                c++

TO
Code:
                           if ( !($10 in per) ) c++;
                           per[$10]++
                           sum += $10

# 5  
Old 09-10-2007
You're completely right vgersh99 ,but i thought the other piece was more clear for this particular case.

pondlife , you 're using an associative array ,where the index is the 10th element of your input and its value is a counter ,the number of times that the element is repeated.

Each element of the index is unique , so the first time you add a new entry to the array (new index) its counter value wil be 1
Code:
per[$10] = 1

On the other hand if the index exits the only variation will be the increase of the value of the counter so:
Code:
per[$10] >1

finally we can say that:

Code:
   if ( per[$10] == 1 )
      c++

Or:
Code:
if ( !($10 in per) ) c++

indicates that a new element was added to the array.

Hope this helps.

Regards
Last edited by Klashxx; 09-10-2007 at 02:45 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split a big file into multiple files using awk

this thread is a continuation from previous thread https://www.unix.com/shell-programming-and-scripting/223901-split-big-file-into-multiple-files-based-first-four-characters.html ..I am using awk to split file and I have a syntax error while executing the below code I am using AIX 7.2... (4 Replies)
Discussion started by: etldev
4 Replies

2. Shell Programming and Scripting

How to Assign an shell array to awk array?

Hello All, Can you please help me with the below. #!/bin/bash ARR="No Differences In Stage Between HASH_TOTALS & HASH_TOTALS_COMP For UNINUM:0722075 PROVIDER:5 EXTRACT_DT:30-SEP-12 VER_NUM:1" ARR="No Differences In Stage Between HASH_TOTALS & HASH_TOTALS_COMP For UNINUM:0722075 PROVIDER:5... (14 Replies)
Discussion started by: Ariean
14 Replies

3. Shell Programming and Scripting

AWK - Parse a big file

INPUT SAMPLE Symmetrix ID : 000192601507 Masking View Name : TS00P22_13E_1 Last updated at : 05:10:18 AM on Tue Mar 22,2011 Initiator Group Name : 10000000c960b9cd Host Initiators { WWN : 10000000c960b9cd } Port Group Name :... (8 Replies)
Discussion started by: greycells
8 Replies

4. Shell Programming and Scripting

awk column comparison big file

Hi all, I would like to compare a column in one file to a column in another file and when there is a match it prints the first column and the corresponding second column. Example File1 ABA ABC ABE ABF File 2 ABA 123 ABB 124 ABD 125 ABC 126 So what I would like printed to a... (6 Replies)
Discussion started by: pcg
6 Replies

5. Shell Programming and Scripting

awk with really big files

Hi, I have a text file that is around 7Gb which is basically a matrix of numbers (FS is a space and RS is \n). I need the most efficient way of plucking out a number from a specified row and column in the file. For example, for the value at row 15983, col 26332, I'm currently I'm using: ... (1 Reply)
Discussion started by: Jonny2Vests
1 Replies

6. Shell Programming and Scripting

Perl or awk/egrep from big files??

Hi experts. In one thread i have asked you how to grep the string from the below sample file- Unfortunately the script did not gave proper output (it missed many strings). It happened may be i did gave you the proper contents of the file That was the script- "$ perl -00nle'print join... (13 Replies)
Discussion started by: thepurple
13 Replies

7. Shell Programming and Scripting

Big data file - sed/grep/awk?

Morning guys. Another day another question. :rolleyes: I am knocking up a script to pull some data from a file. The problem is the file is very big (up to 1 gig in size), so this solution: for results in `grep "^\ ... works, but takes ages (we're talking minutes) to run. The data is held... (8 Replies)
Discussion started by: dlam
8 Replies

8. Shell Programming and Scripting

awk not working as expected with BIG files ...

I am facing some strange problem. I know, there is only one record in a file 'test.txt' which starts with 'X' I ensure that with following command, awk /^X/ test.txt | wc -l This gives me output = '1'. Now I take out this record out of the file, as follows : awk /^X/ test.txt >... (1 Reply)
Discussion started by: videsh77
1 Replies
Login or Register to Ask a Question