Want to get average value for each hour


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Want to get average value for each hour
# 1  
Old 04-02-2015
Want to get average value for each hour

I want to get CPU average value only (not required user CPU & memory) with each hours on individual date. The sample output is below

Code:
                |           |        | User  |Memory| User |
   Date       |  Time  |CPU %|CPU % |  %   |Mem % |
03/02/2015|00:00:00| 24.56| 20.66| 89.75| 63.48|
03/02/2015|00:05:00| 24.40| 20.72| 89.88| 63.47|
03/02/2015|00:10:00| 23.23| 19.98| 90.12| 63.48|
03/02/2015|00:15:00| 26.88| 23.25| 90.30| 63.56|
03/02/2015|00:20:00| 27.57| 23.39| 89.94| 63.49|
03/02/2015|00:25:00| 24.87| 20.46| 89.55| 63.41|
03/02/2015|00:30:00| 33.23| 29.12| 89.48| 63.35|
03/02/2015|00:35:00| 30.05| 26.04| 89.49| 63.36|
03/02/2015|00:40:00| 35.62| 31.81| 89.38| 63.25|
03/02/2015|00:45:00| 20.50| 17.03| 89.83| 63.71|
03/02/2015|00:50:00| 20.49| 17.81| 89.61| 63.50|
03/02/2015|00:55:00| 23.48| 19.37| 89.59| 63.56|
03/02/2015|01:00:00| 56.81| 49.93| 89.97| 63.88|
03/02/2015|01:05:00| 44.84| 39.46| 90.05| 63.98|
03/02/2015|01:10:00| 22.45| 19.12| 89.64| 63.57|
03/02/2015|01:15:00| 17.34| 14.53| 89.66| 63.59|
03/02/2015|01:20:00| 17.29| 14.75| 89.65| 63.58|
03/02/2015|01:25:00| 14.08| 11.54| 89.49| 63.44|
03/02/2015|01:30:00| 19.08| 16.77| 89.33| 63.28|
03/02/2015|01:35:00| 18.99| 16.80| 89.31| 63.26|

Total count of the page is = 8919

Each days contains 288 entries and totally 31 days output contains.

Could you please help me out to simplify in shell programming.

Last edited by Don Cragun; 04-02-2015 at 07:32 PM.. Reason: Add CODE tags.
# 2  
Old 04-02-2015
Try this awk solution if I understand what you are looking for (untested as more sample data is needed).

Code:
awk -F'|' 'NR>2 {hour=" " substr($2,1,2); array[$1 hour] += $3 count[$1 hour]++ } END { for (a in array) {print a, array[a]/count[a] } } ' file | sort
03/02/2015 00 26.2439
03/02/2015 01 26.3635
03/03/2015 01 18.99

# 3  
Old 04-03-2015
Quote:
Originally Posted by mjf
Try this awk solution if I understand what you are looking for (untested as more sample data is needed).

Code:
awk -F'|' 'NR>2 {hour=" " substr($2,1,2); array[$1 hour] += $3 count[$1 hour]++ } END { for (a in array) {print a, array[a]/count[a] } } ' file | sort
03/02/2015 00 26.2439
03/02/2015 01 26.3635
03/03/2015 01 18.99

I have only tested this with the given data, but I'm surprised that you're showing sample output that includes the data marked in red above when there was no sample input for that date. When I tried running the above code with the given sample input, I didn't get the last line of the output shown above.

It also looks like there is a missing semicolon. Did you perhaps intend the following?:
Code:
awk -F'|' 'NR>2 {hour=" " substr($2,1,2); array[$1 hour] += $3; count[$1 hour]++ } END { for (a in array) {print a, array[a]/count[a] } } ' file | sort

which produces the output:
Code:
03/02/2015 00 26.24
03/02/2015 01 26.36

from the given sample input. And, this seems to match the given sample input.
These 2 Users Gave Thanks to Don Cragun For This Post:
# 4  
Old 04-03-2015
Don,
Yes, I added another test record with 3/3/2015 date to test the break logic and forgot to remove it from posted output (as OP did not include it in his file). I missed the semi-colon although I'm not clear at the moment why it impacted the output (the extra decimal places). Thanks for pointing this out.
This User Gave Thanks to mjf For This Post:
# 5  
Old 04-03-2015
Thanks for providing such a wonderful script and also help me to get the following items.

Daily Average
Hourly Average.

Output sample updated in the code.

Code:
Daily Average

03/11/15	35.52%	21.62%
03/12/15	40.20%	22.23%
03/13/15	34.92%	20.53%
03/14/15	51.13%	22.42%
03/15/15	35.94%	14.47%
03/16/15	33.22%	18.70%
03/17/15	35.30%	19.89%

Hourly Average for the month

12AM	40.53%	29.66%
1AM	36.41%	23.77%
2AM	34.87%	23.10%
3AM	40.30%	23.49%
4AM	39.05%	19.84%
5AM	41.52%	18.91%
6AM	42.05%	19.29%

Original Output File 
   Date   |  Time  |CPU % |CPU % |  %   |Mem % |
03/02/2015|00:00:00| 24.56| 20.66| 89.75| 63.48|
03/02/2015|00:05:00| 24.40| 20.72| 89.88| 63.47|
03/02/2015|00:10:00| 23.23| 19.98| 90.12| 63.48|
03/02/2015|00:15:00| 26.88| 23.25| 90.30| 63.56|
03/02/2015|00:20:00| 27.57| 23.39| 89.94| 63.49|
03/02/2015|00:25:00| 24.87| 20.46| 89.55| 63.41|
03/02/2015|00:30:00| 33.23| 29.12| 89.48| 63.35|
03/02/2015|00:35:00| 30.05| 26.04| 89.49| 63.36|
03/02/2015|00:40:00| 35.62| 31.81| 89.38| 63.25|
03/02/2015|00:45:00| 20.50| 17.03| 89.83| 63.71|
03/02/2015|00:50:00| 20.49| 17.81| 89.61| 63.50|
03/02/2015|00:55:00| 23.48| 19.37| 89.59| 63.56|
03/02/2015|01:00:00| 56.81| 49.93| 89.97| 63.88|
03/02/2015|01:05:00| 44.84| 39.46| 90.05| 63.98|
03/02/2015|01:10:00| 22.45| 19.12| 89.64| 63.57|
03/02/2015|01:15:00| 17.34| 14.53| 89.66| 63.59|
03/02/2015|01:20:00| 17.29| 14.75| 89.65| 63.58|
03/02/2015|01:25:00| 14.08| 11.54| 89.49| 63.44|
03/02/2015|01:30:00| 19.08| 16.77| 89.33| 63.28|
03/02/2015|01:35:00| 18.99| 16.80| 89.31| 63.26|
03/02/2015|01:40:00| 15.59| 13.32| 89.28| 63.23|
03/02/2015|01:45:00| 20.91| 18.27| 89.28| 63.23|
03/02/2015|01:50:00| 27.69| 24.68| 89.32| 63.27|
03/02/2015|01:55:00| 29.00| 25.71| 89.32| 63.27|
03/02/2015|02:00:00| 30.21| 25.44| 89.17| 63.21|
03/02/2015|02:05:00| 27.62| 22.90| 89.16| 63.22|
03/02/2015|02:10:00| 23.68| 19.24| 88.84| 63.19|
03/02/2015|02:15:00| 24.77| 20.00| 88.86| 63.21|
03/02/2015|02:20:00| 29.70| 25.90| 88.92| 63.27|
03/02/2015|02:25:00| 28.41| 24.24| 88.97| 63.32|
03/02/2015|02:30:00| 25.70| 22.29| 88.97| 63.29|
03/02/2015|02:35:00| 17.59| 14.77| 88.92| 63.25|
03/02/2015|02:40:00| 21.38| 18.68| 88.91| 63.23|
03/02/2015|02:45:00| 18.21| 15.20| 88.98| 63.33|
03/02/2015|02:50:00| 26.29| 23.21| 88.91| 63.26|
03/02/2015|02:55:00| 23.37| 20.37| 88.91| 63.26|
03/02/2015|03:00:00| 30.18| 25.81| 88.72| 63.05|
03/02/2015|03:05:00| 27.22| 23.42| 88.78| 63.15|
03/02/2015|03:10:00| 30.25| 26.39| 88.93| 63.30|
03/02/2015|03:15:00| 43.21| 38.60| 88.99| 63.38|
03/02/2015|03:20:00| 28.94| 25.25| 88.94| 63.33|
03/02/2015|03:25:00| 24.68| 21.22| 88.96| 63.23|
03/02/2015|03:30:00| 30.62| 25.71| 88.73| 63.13|
03/02/2015|03:35:00| 29.18| 26.12| 88.58| 62.99|
03/02/2015|03:40:00| 23.66| 21.63| 88.49| 62.90|
03/02/2015|03:45:00| 24.87| 22.66| 88.49| 62.91|
03/02/2015|03:50:00| 14.66| 12.61| 88.55| 62.97|
03/02/2015|03:55:00| 12.54| 10.53| 88.51| 62.96|
03/02/2015|04:00:00| 16.39| 13.42| 88.66| 63.09|
03/02/2015|04:05:00| 10.22|  8.06| 88.74| 63.17|


Last edited by Saravanan_0074; 04-03-2015 at 04:14 PM..
# 6  
Old 04-03-2015
Quote:
Originally Posted by mjf
Don,
Yes, I added another test record with 3/3/2015 date to test the break logic and forgot to remove it from posted output (as OP did not include it in his file). I missed the semi-colon although I'm not clear at the moment why it impacted the output (the extra decimal places). Thanks for pointing this out.
Unlike many languages where two adjacent strings representing numbers separated by a space would be a syntax error, awk is happy to concatenate them and treat them as a single numeric string.

So if we look at the first few lines of the input file:
Code:
                |           |        | User  |Memory| User |
   Date       |  Time  |CPU %|CPU % |  %   |Mem % |
03/02/2015|00:00:00| 24.56| 20.66| 89.75| 63.48|
03/02/2015|00:05:00| 24.40| 20.72| 89.88| 63.47|
03/02/2015|00:10:00| 23.23| 19.98| 90.12| 63.48|

and the first part of your script:
Code:
awk -F'|' 'NR>2 {hour=" " substr($2,1,2); array[$1 hour] += $3 count[$1 hour]++ }

On lines 3 through 5, hour will be set to 00 and (since $1 is also a constant on these three lines) the subscript in array[] and count[] on these three lines will be 03/02/2015 00. And for lines 3 through 5 we end up with:
Code:
line	array["03/02/2015 00"]		count["03/02/2015 00"]
  3	24.560					1
  4	24.560 + 24.401 (48.961)		2
  5	48.961 + 23.232 (72.193)		3

So, the higher than expected results come from the extra digits being added to the values that are being summed.
This User Gave Thanks to Don Cragun For This Post:
# 7  
Old 04-03-2015
Facebook

I understand the concatenation but what I'm not getting is why the extra digits being added are off by one for each line. For example, for line 3, I would expect the extra digit in red to be 1 instead of 0 as count is 1. So why it is that count[$1 hour]++ is 0 when it's concatenated to the array for line 3?

Code:
 
line    array["03/02/2015 00"]        count["03/02/2015 00"]  
  3     24.560                                 1    
  4     24.560 + 24.401 (48.961)               2 
  5     48.961 + 23.232 (72.193)               3

This User Gave Thanks to mjf For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Average of multiple time-stamped data every half hour

Hi All, Thank you for reading through my post and helping me figure out how I would be able to perform this task. For example: I have a list of continuous output collected into a file in the format as seen below: Date...........Time........C....A......... B ==========================... (5 Replies)
Discussion started by: terrychen
5 Replies

2. Shell Programming and Scripting

How to get next hour?

Hi guys I want any script to get me next hour For example Nexthour.sh 2013022823 It get me result 2013030100 Thanks a lot , I'm using Solaris 10 (5 Replies)
Discussion started by: teefa
5 Replies

3. Shell Programming and Scripting

How to convert 24 hour time to 12 hour timing?

Hi friends, I want to convert 24 hour timing to 12 hour please help me... my data file looks like this.. 13-Nov-2011 13:27:36 15.32044 72.68502 13-Nov-2011 12:08:31 15.31291 72.69807 16-Nov-2011 01:16:54 15.30844 72.74028 15-Nov-2011 20:09:25 15.35096 ... (13 Replies)
Discussion started by: nex_asp
13 Replies

4. UNIX for Dummies Questions & Answers

Load average spikes once an hour

Hi, I am getting a high load average, around 7, once an hour. It last for about 4 minutes and makes things fairly unusable for this time. How do I find out what is using this. Looking at top the only thing running at the time is md5sum. I have looked at the crontab and there is nothing... (10 Replies)
Discussion started by: sm9ai
10 Replies

5. Shell Programming and Scripting

Getting the last hour from a log.

I have a log like this: Jan 26 13:59:41 server2 ntpdate: step time server 91.189.94.4 offset 0.065456 sec Jan 26 13:59:41 server2 ntpd: ntpd 4.2.4p8@1.1612-o Fri Aug 6 22:49:54 UTC 2010 (1) Jan 26 13:59:41 server2 ntpd: precision = 1.000 usec Jan 26 13:59:41 server2 ntpd: ntp_io: estimated max... (2 Replies)
Discussion started by: Jotne
2 Replies

6. What is on Your Mind?

Power Hour?

I had some free time at work today so I decided to get a little practice with my shell scripts (I'm pretty new to the whole UNIX thing). I'm sure I'm not the only college student here so maybe this code will come in handy for future weekends. #!/bin/sh if then echo "No playlist... (0 Replies)
Discussion started by: thedoobieman5
0 Replies

7. Shell Programming and Scripting

Counting average data per hour

Hi i have log like this : Actually i will process the data become Anybody can help me ? (6 Replies)
Discussion started by: justbow
6 Replies

8. Shell Programming and Scripting

help in hour grep

i have this script that checks for yesterday date and also specific hour in that ----------------------------------------------------------------- TZ=`date +%Z`+24 ;a=`date +%Y-%m-%d %k` cd logs count=0 for i in DBMaint.log do cat $i | grep $a >> file12.txt done... (0 Replies)
Discussion started by: ali560045
0 Replies

9. UNIX for Dummies Questions & Answers

an hour less in 24 hour system

My program: __________________________________ #!/bin/ksh DAY=`date +%y%m%d` H=`date +%H` M=`date +%M` day=`date +%m/%d/%y` let h=$H-1 echo DAY $DAY echo H $H echo M $M echo day $day echo h $h _____________________________________ My result: (3 Replies)
Discussion started by: bobo
3 Replies

10. UNIX for Dummies Questions & Answers

1 Hour less

set DAY=`date +%y%m%d` set H=`date +%H` set M=`date +%M` mailx -s "$H-Mydata" myemail@mail.com<mydata I am looking to set the current hour to have 1 hour less in the subject header: For example: let's say the system time is 8 I want to have "7-Mydata" not "8-Mydata" Can some1... (6 Replies)
Discussion started by: bobo
6 Replies
Login or Register to Ask a Question