Average of multiple time-stamped data every half hour


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Average of multiple time-stamped data every half hour
# 1  
Old 10-12-2014
Average of multiple time-stamped data every half hour

Hi All,

Thank you for reading through my post and helping me figure out how I would be able to perform this task.

For example: I have a list of continuous output collected into a file in the format as seen below:
Code:
 Date...........Time........C....A......... B
==========================
"2014-10-13 15:45:00",0,-1.92925,-0.529
"2014-10-13 15:45:00.1",1,-2.0245,-0.65
"2014-10-13 15:45:00.2",2,-2.0245,-0.65

The first two values are date and time (in hh:mm:ss:millisecond , every 10ms =1 second), the third value (C) doesnt really matter, and the last two values are the numbers lets say A and B. Could you please suggest me, how I would be able to calculate the average every half an hour. If possible I would like to store the values to another file once the averages are calculated.

For example the average of values A from 15:45:00 to 16:14:59:9

Thank you for your suggestions.

Regards,
Terry

Last edited by vbe; 10-13-2014 at 08:58 AM.. Reason: Clarity
# 2  
Old 10-12-2014
Even though you edited your post for clarity, your request is both confusing and ambiguous.

Is this a homework assignment?

What have you tried to solve your problem?

A millisecond is .001 second; 10ms is not 1 second; 10ms is .01 second.

What average are you trying to calculate (the average of the A values and the average of the B values, the average of the sums of the A and B values, the average of the A and B values)?
This User Gave Thanks to Don Cragun For This Post:
# 3  
Old 10-13-2014
Hi Don,

Thanks a lot for your reply.

Regarding your question, if this is a homework assignment, it's not. I collect a stream of data and use excel to take an average since I am not a programmer or someone with Unix background. Maybe it's about time I can learn some perl etc. and make my life a little easier than having to use excel to take an average.

Like you said, I was wrong in saying the last item on the value for time is millisecond. It actually goes from 0 till 9 for the next value and then the value for second increases by one.

Also, I am trying to calculate average of A values and average of B values for the duration of 30 minutes.

Regards,
Terry
# 4  
Old 10-13-2014
If you have GNU awk try this:

Code:
awk -F, '
{
 split($0,T,"\"");
 gsub("[-:]", " ", T[2])
 split(T[2],V,".")
 D=mktime(V[1])
 if(D>B) {
    if(C) print "\"" strftime("%Y-%m-%d %H:%M:%S",B) "\"", sumA/C, sumB/C
    B=D+30*60
    sumA=sumB=C=0
 }
 C++
 sumA=sumA+$3
 sumB=sumB+$4
}
END {if(C) print "\"" strftime("%Y-%m-%d %H:%M:%S",B) "\"", sumA/C, sumB/C}' OFS=, infile


Last edited by Chubler_XL; 10-13-2014 at 07:06 PM.. Reason: changed from 15 to 30mins
This User Gave Thanks to Chubler_XL For This Post:
# 5  
Old 10-14-2014
Many thanks Chubler_XL.

Now, I am trying to figure out how I can have access to GNU awk and I will keep you updated regarding how I come across it.

Regards,
Terry
# 6  
Old 10-14-2014
Quote:
Originally Posted by terrychen
Many thanks Chubler_XL.

Now, I am trying to figure out how I can have access to GNU awk and I will keep you updated regarding how I come across it.

Regards,
Terry
If you have perl already then it should be pretty simple to solve this problem...
This User Gave Thanks to shamrock For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Want to get average value for each hour

I want to get CPU average value only (not required user CPU & memory) with each hours on individual date. The sample output is below | | | User |Memory| User | Date | Time |CPU %|CPU % | % |Mem % | 03/02/2015|00:00:00| 24.56| 20.66| 89.75| 63.48|... (13 Replies)
Discussion started by: Saravanan_0074
13 Replies

2. Shell Programming and Scripting

How to convert 24 hour time to 12 hour timing?

Hi friends, I want to convert 24 hour timing to 12 hour please help me... my data file looks like this.. 13-Nov-2011 13:27:36 15.32044 72.68502 13-Nov-2011 12:08:31 15.31291 72.69807 16-Nov-2011 01:16:54 15.30844 72.74028 15-Nov-2011 20:09:25 15.35096 ... (13 Replies)
Discussion started by: nex_asp
13 Replies

3. UNIX for Dummies Questions & Answers

Load average spikes once an hour

Hi, I am getting a high load average, around 7, once an hour. It last for about 4 minutes and makes things fairly unusable for this time. How do I find out what is using this. Looking at top the only thing running at the time is md5sum. I have looked at the crontab and there is nothing... (10 Replies)
Discussion started by: sm9ai
10 Replies

4. Shell Programming and Scripting

Counting average data per hour

Hi i have log like this : Actually i will process the data become Anybody can help me ? (6 Replies)
Discussion started by: justbow
6 Replies

5. AIX

Time getting reduced by 1 hour

I am setting TZ=EST5EDT,M3.2.0/02:00:00,M11.1.0/02:00:00 Then Setting the date to Mar 14 01:40 EST date 0314014010 Sun Mar 14 01:40:36 EDT 2010 Note that it show it EST. According to my TZ variable 01:40 Should be in EST only. On executing date command once again it shows date Sun Mar... (4 Replies)
Discussion started by: januuj23
4 Replies

6. Shell Programming and Scripting

calculate the average of time series data using AWK

Hi, I have two time series data (below) merged into a file. t1 and t2 are in unit of second I want to calculate the average of V1 every second and count how many times "1" in V2 is occur within a second Input File: t1 V1 t2 V2 10.000000... (5 Replies)
Discussion started by: nica
5 Replies

7. Shell Programming and Scripting

Help in extracting multiple files and taking average at same time

Hi, I have 20 files which have respective 50 lines with different values. I would like to process each line of the 50 lines in these 20 files one at a time and do an average of 3rd field ($3) of these 20 files. This will be output to an output file. Instead of using join to generate whole... (8 Replies)
Discussion started by: ahjiefreak
8 Replies

8. HP-UX

calculate average of multiple line data

I have a question as below and i need to write a shell or perl script for this query:My Input file looks like below RNo Marks 12 50 15 70 18 80 12 40 13 55 18 88 13 75... (4 Replies)
Discussion started by: smacherla
4 Replies

9. Programming

problem deleting date-time stamped file in a directory

I have a number of files of the format filename.xfr_mmddyy_%H%M%S which i get in a specified directory daily. Now i want to search in the specified directory & delete the files which are more than 2 days old .So I use a command find $DIR/backup/* -ctime +2 -exec rm -f {} \; But after executing... (1 Reply)
Discussion started by: dharmesht
1 Replies
Login or Register to Ask a Question