Sponsored Content
Top Forums Programming Find gaps in time data and replace missing time value and column 2 value by interpolation in awk Post 302931838 by Don Cragun on Thursday 15th of January 2015 09:59:35 PM
Old 01-15-2015
This seems very similar to several earlier requests in these forums. A good one to look at for ideas would be XY interpolation by time in awk. If that doesn't give you what you need, the last post in that thread provides pointers to several similar threads.

Please look at those first, and if you can't find anything in them that helps, explain how this case is different from the problem solved in the above thread and we'll try to help you get a working solution.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

calculate the average of time series data using AWK

Hi, I have two time series data (below) merged into a file. t1 and t2 are in unit of second I want to calculate the average of V1 every second and count how many times "1" in V2 is occur within a second Input File: t1 V1 t2 V2 10.000000... (5 Replies)
Discussion started by: nica
5 Replies

2. Shell Programming and Scripting

How to get data between the start time and end time?

Hi, Can anyone help me how can I get the line that between the start time and end time. file1.txt 15/03/2009 20:45:03 Request: - Data of this line 15/03/2009 20:45:12 Response: - Data of this line 15/03/2009 22:10:40 Request: - Data of this line 15/03/2009 22:10:42 Response: - Data of... (1 Reply)
Discussion started by: tanit
1 Replies

3. Shell Programming and Scripting

Searching for Gaps in Time

I am very new to shell scripting. We use C-Shell here and I know the issues that surround it. I hope a solution can be created using awk, sed, etc... instead of having to write a program. I have an input file that is sorted by date and time in ascending order ... (2 Replies)
Discussion started by: jclanc8
2 Replies

4. Shell Programming and Scripting

Find and Convert UTC Time to PST Time

Hello All - I have a script that grabs data from the net and outputs the following data 46029 46.144 -124.510 2010 07 26 22 50 320 4.0 6.0 2.2 9 6.8 311 1012.1 -0.9 13.3 13.5 13.3 - - 46041 47.353 -124.731 2010 07 26 22 50 250 2.0 3.0 1.6 8 6.4 - 1011.6 - ... (0 Replies)
Discussion started by: drexnefex
0 Replies

5. Shell Programming and Scripting

awk/sed to search & replace data in first column

Hi All, I need help in manipulating the data in first column in a file. The sample data looks like below, Mon Jul 18 00:32:52 EDT 2011,NULL,UAT Jul 19 2011,NULL,UAT 1] All field in the file are separated by "," 2] File is having weekly data extracted from database 3] For eg.... (8 Replies)
Discussion started by: gr8_usk
8 Replies

6. UNIX for Dummies Questions & Answers

Converting string date time to unix time in AWK

I'd like to convert a date string in the form of sun aug 19 09:03:10 EDT 2012, to unixtime timestamp using awk. I tried This is how each line of the file looks like, different date and time in this format Sun Aug 19 08:33:45 EDT 2012, user1(108.6.217.236) all: test on the 17th ... (2 Replies)
Discussion started by: bkkid
2 Replies

7. Shell Programming and Scripting

awk : collecting all data between two time frame

Hi Experts , I need your help to collect the complete data between two time frame from the log files, when I try awk it's collecting the data only which is printed with time stamp for example, awk works well from "16:00 to 17:30" but its not collecting <line*> "from 17:30 to 18:00" ... (8 Replies)
Discussion started by: zenkarthi
8 Replies

8. Programming

Resample time series data with replacement any way to do this in awk or just bash script

I have some time series data that I need to resample or downsample at some specific time intervals. The firs column is time in decimal hours. I am tryiong to resample this data every 3 minutse. So I need a data value ever 0.05. Here is the example data and as you can see, there time slot for 0.1500... (3 Replies)
Discussion started by: malandisa
3 Replies

9. Shell Programming and Scripting

XY interpolation by time in awk

Hi I am a newbie in awk scripting. I'm working with a file with xy coordinates that were acquired with a time stamp. All the time stamps were recorded but not the XY coordinates. Let see an example: FFID X Y UNIX TIME TIMEGAP... (8 Replies)
Discussion started by: Joćo Noiva
8 Replies

10. Shell Programming and Scripting

Awk: time intervals based on epoch time

I have a list of epoch times delimited by "-" as follows: 1335078000 - 1335176700 1335340800 - 1335527400 1335771300 - 1335945600 1336201200 - 1336218000 The corresponding dates are: 20120422 1000 - 20120423 1325 20120425 1100 - 20120427 1450 20120430 1035 - 20120502 1100 ... (3 Replies)
Discussion started by: alex2005
3 Replies
FILTER1D(l)															       FILTER1D(l)

NAME
filter1d - Time domain filtering of 1-D time series SYNOPSIS
filter1d [ infile ] -F<type><width> [ -Dincrement ] [ -E ] [ -H[nrec] ] [ -Iignore_val ] [ -Llack_width ] [ -Nn_cols/t_col ] [ -Qq_factor ] [ -Ssymmetry_factor ] [ -Tstart/stop/int ] [ -V ] [ -bi[s][n] ] [ -bo[s][n] ] DESCRIPTION
filter1d is a general time domain filter for multiple column time series data. The user specifies the number of columns of input and which column is the time. (See -N option below). The fastest operation occurs when the input time series are equally spaced and have no gaps or outliers and the special options are not needed. filter1d has options L, Q, and S for unevenly sampled data with gaps. infile Multi-column ASCII (or binary, see -b) file holding data values to be filtered. -F sets Filtertype, type is one of b(oxcar), c(osine arch), g(aussian), m(edian), or p(maximum likelihood Probability estimator -- a mode estimator), and specify full filter width in same units as time column, OR, use -Ffname to give the name of a one-column file of your own coefficients. Upper case type B, C, G, M, P, F will use robust filter versions: i.e., replace outliers (2.5 L1 scale off median) with median during filtering. OPTIONS
-D increment is used when series is NOT equidistantly sampled. Then increment will be the abscissae resolution, i.e., all abscissae will be rounded off to a multiple of increment. Alternatively, resample data with sample1d. -E Include Ends of time series in output. Default loses half the filter-width of data at each end. -H Input file(s) has Header record(s). Number of header records can be changed by editing your .gmtdefaults file. If used, GMT default is 1 header record. -I To ignore values; If an input value equals ignore_val it will be set to NaN. -L Checks for Lack of data condition. If input data has a gap exceeding width then no output will be given at that point [Default does not check Lack]. -N Sets number of columns in input and which column contains the independent variable (time). The left-most column is # 0, the right- most is # (n_cols - 1). [Default is n_cols = 2, t_col = 0; i.e., file has t, f(t) pairs]. -Q assess Quality of output value by checking mean weight in convolution. Enter q_factor between 0 and 1. If mean weight < q_factor, output is suppressed at this point [Default does not check Quality]. -S Checks symmetry of data about window center. Enter a factor between 0 and 1. If ( (abs(n_left - n_right)) / (n_left + n_right) ) > factor, then no output will be given at this point [Default does not check Symmetry]. -T Make evenly spaced timesteps from start to stop by int [Default uses input times]. -V Selects verbose mode, which will send progress reports to stderr [Default runs "silently"]. -bi Selects binary input. Append s for single precision [Default is double]. Append n for the number of columns in the binary file(s). -bo Selects binary output. Append s for single precision [Default is double]. EXAMPLES
To filter the data set in the file cruise.gmtd containing evenly spaced gravity, magnetics, topography, and distance (in m) with a 10 km Gaussian filter, removing outliers, and output a filtered value every 2 km between 0 and 100 km, try filter1d cruise.gmtd -T0/1.0e5/2000 -FG10000 -N4/3 -V > filtered_cruise.gmtd Data along track often have uneven sampling and gaps which we do not want to interpolate using sample1d. To find the median depth in a 50 km window every 25 km along the track of cruise v3312, stored in v3312.dt, checking for gaps of 10km and asymmetry of 0.3, try filter1d v3312.dt -FM50 -T0/100000/25 -L10 -S0.3 > v3312_filt.dt SEE ALSO
gmt(1gmt), sample1d(1gmt) 1 Jan 2004 FILTER1D(l)
All times are GMT -4. The time now is 05:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy