Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Averaging data every 30 mins using AWK

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Averaging data every 30 mins using AWK
# 1  
Old 07-25-2011
Averaging data every 30 mins using AWK
A happy Monday to you all,

I have a .csv file which contains data taken every 5 seconds. I want to average these 5 second data points into 30 minute averages!

Code:
date                          co2

25/06/2011 08:04        8.31
25/06/2011 08:04        8.32
25/06/2011 08:04        8.33
25/06/2011 08:04        8.32
25/06/2011 08:04        8.31
25/06/2011 08:04        8.30
25/06/2011 08:04        8.29
25/06/2011 08:04        8.34
25/06/2011 08:04        8.43
25/06/2011 08:04        8.33
25/06/2011 08:04        8.32
25/06/2011 08:04        8.30
25/06/2011 08:05        8.31
25/06/2011 08:05        8.32
25/06/2011 08:05        8.30
25/06/2011 08:05        8.29
25/06/2011 08:05        8.30
25/06/2011 08:05        8.31
25/06/2011 08:05        8.33
25/06/2011 08:05        8.32
25/06/2011 08:05        8.32
25/06/2011 08:05        8.35
25/06/2011 08:05        8.34
25/06/2011 08:05        8.36
25/06/2011 08:06        8.37
25/06/2011 08:06        8.34
25/06/2011 08:06        8.35
25/06/2011 08:06        8.35
25/06/2011 08:06        8.34
25/06/2011 08:06        8.38
25/06/2011 08:06        8.35
25/06/2011 08:06        8.36
25/06/2011 08:06        8.32
25/06/2011 08:06        8.31
25/06/2011 08:06        8.30
25/06/2011 08:06        8.32

Is there any way of doing this in AWK?

Cheers!
# 2  
Old 07-25-2011
Check if it is giving you desired results:
Code:
awk '{a[int((NR-1)/720)]+=$3};END{for (i=0;i<NR/720;i++){print a[i]/720}}' file

# 3  
Old 07-25-2011
Ah good to hear from you again Bartus!

I should have specified sorry, there are two columns I want to average, a 'CO2' column and a 'CH4' column, both in columns 12 and 14 respectively.

Also when I put in that code, the Shell just replied with multiple '0's.
Last edited by gd9629; 07-25-2011 at 07:23 AM..
# 4  
Old 07-25-2011
Try:
Code:
awk 'NR>2' file | awk '{a[int((NR-1)/720)]+=$12;b[int((NR-1)/720)]+=$14};END{print "CO2 CH4";for (i=0;i<NR/720;i++){print a[i]/720,b[i]/720}}'

Last edited by bartus11; 07-25-2011 at 07:33 AM.. Reason: Updated to omit first two lines
# 5  
Old 07-25-2011
hmm now the Shell outputs two columns of 0's, and the .csv file is unaltered.
# 6  
Old 07-25-2011
Can you upload the file that you are trying to analyze?
# 7  
Old 07-25-2011
I've attached just a snippet of the file. Hope it helps.
june2temp.xls (745.0 KB) 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Script to search log file for last 15 mins data

Hi All, I have an issue which I'm trying to understand a way of doing, I have several nodes which contain syslog events which I want to force trigger an email initially (eventually leading to another method of alerting but to start with an email). Basically the syslog file will have hours worth... (6 Replies)
Discussion started by: mutley2202
6 Replies

2. Shell Programming and Scripting

Script (ksh) to get data in every 30 mins interval for the given date

Hello, Since I m new to shell, I had a hard time to sought out this problem. I have a log file of a utility which tells that batch files are successful with timestamp. Given below is a part of the log file. 2013/03/07 00:13:50 Apache/1.3.29 (Unix) configured -- resuming normal operations... (12 Replies)
Discussion started by: rpm120
12 Replies

3. Shell Programming and Scripting

How to perform averaging of values for particular timestamp using awk or anythoing else??

I have a file of the form. 16:00:26,83.33 16:05:26,83.33 16:10:26,83.33 16:15:26,83.33 16:20:26,90.26 16:25:26,83.33 16:30:26,83.33 17:00:26,83.33 17:05:26,83.33 17:10:26,83.33 17:15:26,83.33 17:20:26,90.26 17:25:26,83.33 17:30:26,83.33 For the timestamp 16:00:00 to 16:55:00, I need to... (5 Replies)
Discussion started by: Saidul
5 Replies

4. Shell Programming and Scripting

Averaging help in awk

Hi all, I have a data file like below, where Time is in the second column DATE TIME FRAC_DAYS_SINCE_JAN1 2011-06-25 08:03:20.000 175.33564815 2011-06-25 08:03:25.000 175.33570602... (10 Replies)
Discussion started by: gd9629
10 Replies

5. Shell Programming and Scripting

Hourly averaging using Awk

Hey all, I have a set of 5-second data as shown below. I need to find an hourly average of this data. date co2 25/06/2011 08:04:00 8.30 25/06/2011 08:04:05 8.31 25/06/2011 08:04:10 8.32 25/06/2011 08:04:15 8.33 25/06/2011 08:04:20 ... (5 Replies)
Discussion started by: gd9629
5 Replies

6. Shell Programming and Scripting

Averaging Data From Multiple Columns, Using Header if Possible

Hi, I have a file with multiple tab delimited columns and I would like to have the average of each column: Iteration Tree No Lh HMean 1000 1 -78.834717 -78.834717 1100 1 -77.991031 -78.624046 1200 1 -79.416055 -78.761861 1300 1 -79.280494 -78.968099 1400 1 -82.846275 -80.808696 ... (4 Replies)
Discussion started by: mikey11415
4 Replies

7. Shell Programming and Scripting

Averaging in increments using awk & head/tail

Hi, I only have a very limited understanding and experience with writing code and I was hoping I could get some help. I have a dataset of two columns (txt format, numbers in each row separated by a tab) Eg. 1 5 2 5 3 6 4 7 5 6 6 6 7 ... (5 Replies)
Discussion started by: Emred_Skye
5 Replies

8. UNIX for Dummies Questions & Answers

Averaging the rows using 'awk'

Dear all, I have the data in the following format. I want to do average of each NR= 5 (rows) for all the 3 ($1,$2, $3) columns and want to print average result in another file in the same format. I dont know how to write code for this in 'awk', can some one help me to write a code for this in... (1 Reply)
Discussion started by: arvindr
1 Replies

9. Shell Programming and Scripting

averaging column values with awk

Hello. Im just starting to learn awk so hang in there with me...I have a large text file formatted as such everything is in a single column ID001 value 1 value 2 value....n ID002 value 1 value 2 value... n I want to be able to calculate the average for values for each ID from the... (18 Replies)
Discussion started by: johnmillsbro
18 Replies

10. Shell Programming and Scripting

AWK - averaging $3 by info in $1

Hello, I have three columns of data of the format below: <name> <volume> <size> a 2 1.2 a 2 1.1 b 3 1.7 c 0.7 1.9 c 0.7 1.9 c 0.7 1.8 What I... (3 Replies)
Discussion started by: itisthus
3 Replies
Login or Register to Ask a Question