Sponsored Content
Top Forums Programming Find gaps in time data and replace missing time value and column 2 value by interpolation in awk Post 302931906 by malandisa on Friday 16th of January 2015 10:11:21 AM
Old 01-16-2015
Dear RudiC and everyone,

Thank you. this solution works and does exactly what I really needed. The interpolation is just alright. As you can see this is time series data for some quantity that depends on the sun, which exhibits higher values during the day that at night. this is morning and the variation of the quantity more complex due to too many factors. So this interpolation does just fine. I have just increased the precision of the D0/D1 to include more significant figures and that solves the problem you indicated with the
Code:
D1 > D0

.

Again may I say thank you and am sure this will help others as well.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

calculate the average of time series data using AWK

Hi, I have two time series data (below) merged into a file. t1 and t2 are in unit of second I want to calculate the average of V1 every second and count how many times "1" in V2 is occur within a second Input File: t1 V1 t2 V2 10.000000... (5 Replies)
Discussion started by: nica
5 Replies

2. Shell Programming and Scripting

How to get data between the start time and end time?

Hi, Can anyone help me how can I get the line that between the start time and end time. file1.txt 15/03/2009 20:45:03 Request: - Data of this line 15/03/2009 20:45:12 Response: - Data of this line 15/03/2009 22:10:40 Request: - Data of this line 15/03/2009 22:10:42 Response: - Data of... (1 Reply)
Discussion started by: tanit
1 Replies

3. Shell Programming and Scripting

Searching for Gaps in Time

I am very new to shell scripting. We use C-Shell here and I know the issues that surround it. I hope a solution can be created using awk, sed, etc... instead of having to write a program. I have an input file that is sorted by date and time in ascending order ... (2 Replies)
Discussion started by: jclanc8
2 Replies

4. Shell Programming and Scripting

Find and Convert UTC Time to PST Time

Hello All - I have a script that grabs data from the net and outputs the following data 46029 46.144 -124.510 2010 07 26 22 50 320 4.0 6.0 2.2 9 6.8 311 1012.1 -0.9 13.3 13.5 13.3 - - 46041 47.353 -124.731 2010 07 26 22 50 250 2.0 3.0 1.6 8 6.4 - 1011.6 - ... (0 Replies)
Discussion started by: drexnefex
0 Replies

5. Shell Programming and Scripting

awk/sed to search & replace data in first column

Hi All, I need help in manipulating the data in first column in a file. The sample data looks like below, Mon Jul 18 00:32:52 EDT 2011,NULL,UAT Jul 19 2011,NULL,UAT 1] All field in the file are separated by "," 2] File is having weekly data extracted from database 3] For eg.... (8 Replies)
Discussion started by: gr8_usk
8 Replies

6. UNIX for Dummies Questions & Answers

Converting string date time to unix time in AWK

I'd like to convert a date string in the form of sun aug 19 09:03:10 EDT 2012, to unixtime timestamp using awk. I tried This is how each line of the file looks like, different date and time in this format Sun Aug 19 08:33:45 EDT 2012, user1(108.6.217.236) all: test on the 17th ... (2 Replies)
Discussion started by: bkkid
2 Replies

7. Shell Programming and Scripting

awk : collecting all data between two time frame

Hi Experts , I need your help to collect the complete data between two time frame from the log files, when I try awk it's collecting the data only which is printed with time stamp for example, awk works well from "16:00 to 17:30" but its not collecting <line*> "from 17:30 to 18:00" ... (8 Replies)
Discussion started by: zenkarthi
8 Replies

8. Programming

Resample time series data with replacement any way to do this in awk or just bash script

I have some time series data that I need to resample or downsample at some specific time intervals. The firs column is time in decimal hours. I am tryiong to resample this data every 3 minutse. So I need a data value ever 0.05. Here is the example data and as you can see, there time slot for 0.1500... (3 Replies)
Discussion started by: malandisa
3 Replies

9. Shell Programming and Scripting

XY interpolation by time in awk

Hi I am a newbie in awk scripting. I'm working with a file with xy coordinates that were acquired with a time stamp. All the time stamps were recorded but not the XY coordinates. Let see an example: FFID X Y UNIX TIME TIMEGAP... (8 Replies)
Discussion started by: Joćo Noiva
8 Replies

10. Shell Programming and Scripting

Awk: time intervals based on epoch time

I have a list of epoch times delimited by "-" as follows: 1335078000 - 1335176700 1335340800 - 1335527400 1335771300 - 1335945600 1336201200 - 1336218000 The corresponding dates are: 20120422 1000 - 20120423 1325 20120425 1100 - 20120427 1450 20120430 1035 - 20120502 1100 ... (3 Replies)
Discussion started by: alex2005
3 Replies
SPLINE(1)						      GNU Plotting Utilities							 SPLINE(1)

NAME
spline - interpolate datasets using splines under tension SYNOPSIS
spline [ options ] [ files ] DESCRIPTION
spline reads datasets from standard input or from one or more files, and fits a smooth curve (a "spline") through each dataset. An inter- polated version of each dataset, consisting of points from the smooth curve, is written to standard output. Unless the -a or -A options are used (see below), each dataset should be a sequence of values for a vector-valued function of a single scalar variable. That is, each dataset should be a sequence of data points, given as alternating t and y values. t is a scalar indepen- dent variable, and y is a vector-valued dependent variable. The dimensionality of y is specified with the -d option (the default dimen- sionality is 1). Between each data point and the next, t should increase. An input file may contain more than a single dataset. If an input file is in ASCII format (the default), its datasets should be separated by blank lines. The t and y values of the data points in each dataset may be arranged arbitrarily, so long as they are separated by white space. Besides datasets, an input file may contain any number of comment lines, which should begin with the comment character `#'. Com- ment lines are ignored. They are not treated as blank, i.e., they do not interrupt a dataset in progress. Options and file names may be interspersed on the command line, but the options are processed before the file names are read. If -- is seen, it is interpreted as the end of the options. If no file names are specified, or the file name - is encountered, the standard input is read. The type of interpolation, and the format of the input and output files, may be selected by command-line options. OPTIONS
Interpolation-Related Options -f --filter Use a local interpolation algorithm (the cubic Bessel algorithm), so that spline can be used as a real-time filter. The slope of the interpolating curve at each point in a dataset will be chosen by fitting a quadratic function through that point and the two adjacent points in the dataset. If -f is specified then the -t option, otherwise optional, must be used as well. Also, if -f is specified then the -k, -p, and -T options may not be used. If -f is not specified, then the default (global) interpolation algorithm will be used. -k k --boundary-condition k Set the boundary condition parameter for each constructed spline to be k. (The default value is 1.0.) In each of its components, the spline will satisfy the two boundary conditions y"[0]=ky"[1] and y"[n]=ky"[n-1]. Here y[0] and y[1] signify the values of a specified component of the vector-valued dependent variable y at the first two points of a dataset, and y[n-1] and y[n] the values at the last two points. Setting k to zero will yield a "natural" spline, i.e., one that has zero curvature at the two ends of the dataset. The -k option may not be used if -f or -p is specified. -n n --no-of-intervals n Subdivide the interval over which interpolation occurs into n subintervals. The number of data points computed, and written to the output, will be n+1. The default value for n is 100. -p --periodic Construct a periodic spline. If this option is specified, the y values for the first and last points in each dataset must be equal. The -f and -k options may not be used if -p is specified. -T tension --tension tension Each interpolating curve will be a spline under tension. This option sets the tension value (the default is 0.0). If tension equals zero, the curve will be a piecewise cubic spline. Increasing the tension above zero makes the curve "tighter", and reduces the likelihood of spurious inflection points. That is because between each pair of successive points in a dataset, the curve will satisfy the fourth-order differential equation y""=sgn(tension)*(tension^2)y" in each of its components. As tension increases to positive infinity, it will converge to a polygonal line. The -T option may not be used if -f is specified. -t tmin tmax [tspacing] --t-spacing tmin tmax [tspacing] For each dataset, set the interval over which interpolation occurs to be the interval between tmin and tmax. If tspacing is not specified, the interval will be divided into the number of subintervals specified by the -n option. If the -t option is not used, the interval over which interpolation occurs will be the entire range of the independent variable in the dataset. The -t option must always be used if the -f option is used to request filter-like behavior (see above). Format-Related Options -d dimension --y-dimension dimension Set the dimensionality of the dependent variable y in the input and output files to be dimension. The default dimension is 1. -I data-format --input-format data-format Set the data format for the input file(s) to be data-format, which may be one of the following. a ASCII format (the default). Each file is a sequence of floating point numbers, interpreted as the t and y coordinates of the successive data points in a dataset. If y is d-dimensional, there will be d+1 numbers for each point. The t and y coordi- nates of a point need not appear on the same line, and points need not appear on different lines. But if a blank line occurs (i.e., two newlines in succession are seen), it is interpreted as the end of a dataset, and the beginning of the next. f Single precision binary format. Each file is a sequence of floating point numbers, interpreted as the t and y coordinates of the successive data points in a dataset. If y is d-dimensional, there will be d+1 numbers for each point. Successive datasets are separated by a single occurrence of the quantity FLT_MAX, which is the largest possible single precision float- ing point number. On most machines this is approximately 3.4x10^38. d Double precision binary format. Each file is a sequence of double precision floating point numbers, interpreted as the t and y coordinates of the successive data points in a dataset. If y is d-dimensional, there will be d+1 numbers for each point. Successive datasets are separated by a single occurrence of the quantity DBL_MAX, which is the largest possible double preci- sion floating point number. On most machines this is approximately 1.8x10^308. i Integer binary format. Each file is a sequence of integers, interpreted as the t and y coordinates of the successive data points in a dataset. If y is d-dimensional, there will be d+1 numbers for each point. Successive datasets are separated by a single occurrence of the quantity INT_MAX, which is the largest possible integer. On most machines this is 2^31-1. -a [step_size [lower_limit]] --auto-abscissa [step_size [lower_limit]] Automatically generate values for t, the independent variable (the default values of step_size and lower_limit are 1.0 and 0.0, respectively). Irrespective of data format (`a', `f', `d', or `i'), this option specifies that the values of t are missing from the input file: the dataset(s) to be read contain only values of y, the dependent variable. So if y is d-dimensional, there will be only d numbers for each point. The increment from each t value to the next will be step_size, and the first t value will be lower_limit. This option is useful, e.g., when interpolating curves rather than functions. -A --auto-dist-abscissa Automatically generate values for t, the independent variable. This is a variant form of the -a option. The increment from each t value to the next will be the distance in d-dimensional space between the corresponding y values, and the first t value will be 0.0. That is, t will be "polygonal arclength". This option is useful when interpolating curves rather than functions. -O data-format --output-format data-format Set the data format for the output file to be data-format. The interpretation of data-format is the same as for the -I option. The default is `a', i.e., ASCII format. -P significant-digits --precision significant-digits Set the numerical precision for the t and y values in the output file to be significant-digits. This takes effect only if the out- put file is written in `a' format, i.e., in ASCII. significant-digits must be a positive integer (the default is 6). -s --suppress-abscissa Omit the independent variable t from the output file; for each point, supply only the dependent variable y. If y is d-dimensional, there will be only d numbers for each point, not d+1. This option is useful when interpolating curves rather than functions. Informational Options --help Print a list of command-line options, and exit. --version Print the version number of spline and the plotting utilities package, and exit. EXAMPLES
Typing echo 0 0 1 1 2 0 | spline will produce on standard output an interpolated dataset consisting of 101 data points. If graphed, this interpolated dataset will yield a parabola. It is sometimes useful to interpolate between a sequence of arbitrarily placed points in d-dimensional space, i.e., to "spline a curve" rather than a function. The -a and -s options are used for this. For example, echo 0 0 1 0 1 1 0 1 | spline -d 2 -a -s will produce on standard output a 101-point dataset that interpolates between the four points (0,0), (1,0), (1,1), and (0,1). The -d 2 option specifies that the dependent variable y is two-dimensional. The -a option specifies that the t values are missing from the input and should be automatically generated. The -s option specifies that the t values should be stripped from the output. AUTHORS
spline was written by Robert S. Maier (rsm@math.arizona.edu), starting with an earlier version by Rich Murphey (rich@freebsd.org). The algorithms for constructing splines under tension are similar to those used in the FITPACK subroutine library, and are ultimately due to Alan K. Cline (cline@cs.utexas.edu). SEE ALSO
"The GNU Plotting Utilities Manual". BUGS
Email bug reports to bug-gnu-utils@gnu.org. FSF
Dec 1998 SPLINE(1)
All times are GMT -4. The time now is 05:24 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy