Sponsored Content
Top Forums Shell Programming and Scripting calculate the average of time series data using AWK Post 302270893 by nica on Tuesday 23rd of December 2008 07:18:10 AM
Old 12-23-2008
calculate the average of time series data using AWK

Hi,

I have two time series data (below) merged into a file.
t1 and t2 are in unit of second

I want to calculate the average of V1 every second and count how many times "1" in V2 is occur within a second


Input File:

t1 V1 t2 V2
10.000000 4.387413 10.139355302 1
10.100000 4.397372 10.252770182 1
10.200000 4.406951 10.398060182 1
10.300000 3.940732 10.515105302 1
10.400000 4.044359 10.645365302 1
10.500000 4.139778 10.768800182 1
10.600000 4.222087 10.929725222 1
10.700000 4.299174 11.106285302 1
10.800000 2.941378 11.216505302 1
10.900000 3.081282 11.324910182 1
11.000000 3.219284 11.626115222 1
11.100000 3.354575 11.822715302 1
11.200000 3.486347 11.968005302 1
11.300000 3.613792 12.107075222 1
11.400000 3.730119 12.233535302 1
11.500000 3.846800 12.377615222 1
11.600000 3.956768 12.494055302 1
11.700000 4.059215 12.642540182 1
11.800000 4.153333 12.742740182 1
11.900000 4.234293 12.853565222 1
12.000000 4.309844 13.093440182 1
12.100000 2.107283 13.209275222 1
12.200000 2.234828 13.343940182 1
12.300000 2.371988 13.471005302 1
12.400000 2.511328 13.635125222 1
12.500000 2.652041 13.824900182 1
12.600000 2.793317 13.955160182 1
12.700000 2.934348 14.082225302 1
12.800000 3.067364 14.185620182 1
12.900000 3.205592 14.302665302 1
13.000000 4.130738 14.421090182 1
13.100000 3.929949 14.707265222 1
13.200000 2.160613 14.828715302 1
13.300000 2.296229 14.938935302 1
13.400000 2.434470 15.114285302 1
13.500000 2.574528 15.242730182 1
13.600000 3.865811 15.485025302 1
13.700000 4.273357 15.660375302 1
13.800000 4.357861 15.895845302 1
13.900000 4.371735 16.034310182 1
14.000000 4.377158 16.150145222 1
..............
..............



Desired Output:

t1 V1 V2
10.000000 3.986053 7
11.000000 3.765453 6
12.000000 2.818793 7
13.000000 3.439529 7
...............
...............


Please, can anyone tell me AWK code for calculating this..??

Thanks
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Use awk to calculate average of column 3

Suppose I have 500 files in a directory and I need to Use awk to calculate average of column 3 for each of the file, how would I do that? (6 Replies)
Discussion started by: grossgermany
6 Replies

2. HP-UX

calculate average of multiple line data

I have a question as below and i need to write a shell or perl script for this query:My Input file looks like below RNo Marks 12 50 15 70 18 80 12 40 13 55 18 88 13 75... (4 Replies)
Discussion started by: smacherla
4 Replies

3. UNIX for Dummies Questions & Answers

Iterate a min/max awk script over time-series temperature data

I'm trying to iterate a UNIX awk script that returns min/max temperature data for each day from a monthly weather data file (01_weath.dat). The temperature data is held in $5. The temps are reported each minute so each day contains 1440 temperature enteries. The below code has gotten me as far as... (5 Replies)
Discussion started by: jgourley
5 Replies

4. Shell Programming and Scripting

Calculate average time using a script

Hello, I'm hoping to get some help on calculating an average time from a list of times (hour:minute:second). Here's what my list looks like right now, it will grow (I can get the full date or change the formatting of this as well): 07:55:31 09:42:00 08:09:02 09:15:23 09:27:45 09:49:26... (4 Replies)
Discussion started by: jaredhanks
4 Replies

5. Shell Programming and Scripting

AWK novice - calculate the average

Hi, I have the following data in a file for example: P1 XXXXXXX.1 YYYYYYY.1 ZZZ.1 P1 XXXXXXX.2 YYYYYYY.2 ZZZ.2 P1 XXXXXXX.3 YYYYYYY.3 ZZZ.3 P1 XXXXXXX.4 YYYYYYY.4 ZZZ.4 P1 XXXXXXX.5 YYYYYYY.5 ZZZ.5 P1 XXXXXXX.6 YYYYYYY.6 ZZZ.6 P1 XXXXXXX.7 YYYYYYY.7 ZZZ.7 P1 XXXXXXX.8 YYYYYYY.8 ZZZ.8 P2... (6 Replies)
Discussion started by: alex2005
6 Replies

6. Shell Programming and Scripting

Calculate Average AWK

I want to calculate the average line by line of some files with several lines on them, the files are identical, just want to average the 3rd columns of those files.:wall: Example file: File 1 001 0.046 0.667267 001 0.047 0.672028 001 0.048 0.656025 001 0.049 ... (2 Replies)
Discussion started by: AriasFco
2 Replies

7. Shell Programming and Scripting

Calculate average for repeated ID within a data

I have an awk script that gives the following output: Average end-to-end transmission delay 2.7 to 5.7 is 0.635392 seconds Average end-to-end transmission delay 2.1 to 5.1 is 0.66272 seconds Average end-to-end transmission delay 2.1 to 5.1 is 0.691712 seconds Average end-to-end transmission... (4 Replies)
Discussion started by: ENG_MOHD
4 Replies

8. Programming

Resample time series data with replacement any way to do this in awk or just bash script

I have some time series data that I need to resample or downsample at some specific time intervals. The firs column is time in decimal hours. I am tryiong to resample this data every 3 minutse. So I need a data value ever 0.05. Here is the example data and as you can see, there time slot for 0.1500... (3 Replies)
Discussion started by: malandisa
3 Replies

9. Shell Programming and Scripting

Average of multiple time-stamped data every half hour

Hi All, Thank you for reading through my post and helping me figure out how I would be able to perform this task. For example: I have a list of continuous output collected into a file in the format as seen below: Date...........Time........C....A......... B ==========================... (5 Replies)
Discussion started by: terrychen
5 Replies

10. Shell Programming and Scripting

Calculate Average time of one column

Hello dears, I have a log file with records like below and want to get a average of one column based on the search of one specific keyword. 2015-02-07 08:15:28 10.102.51.100 10.112.55.101 "kevin.c" POST ... (2 Replies)
Discussion started by: Newman
2 Replies
ATSA(1) 						  The Canonical Csound Reference						   ATSA(1)

NAME
atsa - Performs ATS analysis on a soundfile. . DESCRIPTION
ATS analysis for use with the Csound ATS Resynthesis opcodes. SYNTAX
csound -U atsa [flags] infilename outfilename INITIALIZATION
The following flags can be set for atsa (The default values are stated in parenthesis): -b start (0.000000 seconds) -e duration (0.000000 seconds or end) -l lowest frequency (20.000000 Hertz) -H highest frequency (20000.000000 Hertz) -d frequency deviation (0.100000 of partial freq.) -c window cycles (4 cycles) -w window type (type: 1) (Options: 0=BLACKMAN, 1=BLACKMAN_H, 2=HAMMING, 3=VONHANN) -h hop size (0.250000 of window size) -m lowest magnitude (-60.000000) -t track length (3 frames) -s min. segment length (3 frames) -g min. gap length (3 frames) -T SMR threshold (30.000000 dB SPL) -S min. segment SMR (60.000000 dB SPL) -P last peak contribution (0.000000 of last peak's parameters) -M SMR contribution (0.500000) -F File Type (type: 4) (Options: 1=amp.and freq. only, 2=amp.,freq. and phase, 3=amp.,freq. and residual, 4=amp.,freq.,phase, and residual) PARAMETERS
ATS analysis was devised by Juan Pampin. For complete information on ATS visit: http://www-ccrma.stanford.edu/~juan/ATS.html. Analysis parameters must be carefully tuned for the Analysis Algorithm (ATSA) to properly capture the nature of the signal to be analyzed. As there are a significant number of them, ATSH offers the possibility of Saving/Loading them in a Binary File carrying the extension "*.apf". The extension is not mandatory, but recommended. A brief explanation of each Analysis Parameters follows: 1. Start (secs.): the starting time of the analysis in seconds. 2. Duration (secs.): the duration time of the analysis in seconds. A zero means the whole duration of the input sound file. 3. Lowest Frequency (Hz.): this parameter will partially determine the size of the Analysis Window to be used. To compute the size of the Analysis Window, the period of the Lowest Frequency in samples (SR / LF) is multiplied by the number of cycles of it the user wants to fit in the Analysis Window (see parameter 6). This value is rounded to the next power of two to determine the size of the FFT for the analysis. The remaining samples are zero-padded. If the signal is a single, harmonic sound, then the value of the Lowest Frequency should be its fundamental frequency or a sub-harmonic of it. If it is not harmonic, then its lowest significant frequency component may be a good starting value. 4. Highest Frequency (Hz.): highest frequency to be taken into account for Peak Detection. Once it is determined that no relevant information is found beyond a certain frequency, the analysis may be faster and more accurate setting the Highest Frequency parameter to that value. 5. Frequency Deviation (Ratio): frequency deviation allowed for each peak in the Peak Continuation Algorithm, as a ratio of the frequency involved. For instance, considering a peak at 440 Hz and a Deviation of .1 will produce that the Peak Continuation Algorithm will only try to find candidates for its continuation between 396 and 484 Hz (10% above and below the frequency of the peak). A small value is likely to produce more trajectories whilst a large value will reduce them, but at the cost of rendering information difficult to be further processed. 6. Number of Cycles of Lowest Frequency to fit in Analysis Window: this will also partially determine the size of the Fourier Analysis Window to be used. See Parameter 3. For single harmonic signals, it is supposed to be more than one (typically 4). 7. Hop Size (Ratio): size of the gap between one Analysis Window and the next expressed as a ratio of the Window Size. For instance, a Hop Size value of .25 will "jump" by 512 samples (Windows will overlap for a 75% of their size). This parameter will also determine the size of the analysis frames obtained. Signals that change their spectra very fast (such as Speech sounds) may need a high frame rate in order to properly track their changes. 8. Amplitude Threshold (dB): the highest amplitude value to be taken into account for Peak Detection. 9. Window Type: the shape of the smoothing function to be used for the Fourier Analysis. There are four choices available at present: Blackman, Blackman-Harris, Von Hann, and Hanning. Precise specifications about them are easily found on D.S.P. bibliography. 10. Track Length (Frames): The Peak Continuation Algorithm will "look-back" by Length frames in order to do its job better, preventing frequency trajectories from curving too much and loosing stability. However, a large value for this parameter will slow down the analysis significantly. 11. Minimal Segment Length (Frames): once the analysis is done, the spectral data can be further "cleaned" up during post-processing. Trajectories shorter than this value are suppressed if their average SMR is below Minimal Segment SMR (see parameters 16 and 14). This might help to avoid non-relevant sudden changes while keeping a high frame rate, reducing also the number of intermittent sinusoids during synthesis. 12. Minimal Gap Length (Frames): as parameter 11, this one is also used to clean up the data during post-processing. In this case, gaps (zero amplitude values, i.e. theoretical "silence") longer than Length frames are filled up with amplitude/frequency values obtained by linear interpolation of the adjacent active frames. This parameter prevents sudden interruptions of stable trajectories while keeping a high frame rate. 13. SMR Threshold (dB SPL): also a post-processing parameter, the SMR Threshold is used to eliminate partials with low averages. 14. Minimal Segment SMR (dB SPL): this parameter is used in combination with parameter 11. Short segments with SMR average below this value will be removed during post-processing. 15. Last Peak Contribution (0 to 1): as explained in Parameter 10, the Peak Continuation Algorithm "looks-back" several number of frames to do its job better. This parameter will help to weight the contribution of the first precedent peak over the others. A zero value means that all precedent peaks (to the size of Parameter 10) are equally taken in account. 16. SMR Contribution (0 to 1): In addition to the proximity in frequency of the peaks, the ATS Peak Continuation Algorithm may use psycho-acoustic information (the Signal-to-Mask-Ratio, or SMR) to improve the perceptual results. This parameter indicates how much the SMR information is used during tracking. For instance, a value of .5 makes the Peak Continuation Algorithm to use a 50% of SMR information and a 50% of Frequency Proximity information to decide which is the best candidate to continue a sinusoidal track. EXAMPLES
The following command: atsa -b0.1 -e1 -l100 -H10000 -w2 audiofile.wav audiofile.ats Generates the ATS analysis file 'audiofile.ats' from the original 'audiofile.wav' file. It begins analysis from second 0.1 of the file and the analysis is performed for 1 second thereafter. The lowest frequency stored is 100 Hz and the highest is 10kHz. A Hamming window is used for each analysis frame. AUTHORS
Barry Vercoe MIT Media Lab Author. Dan Ellis MIT Media Lab, Cambridge Massachussetts Author. COPYRIGHT
5.07 06/23/2009 ATSA(1)
All times are GMT -4. The time now is 10:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy