XY interpolation by time in awk


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting XY interpolation by time in awk
# 1  
Old 12-15-2014
XY interpolation by time in awk

Hi I am a newbie in awk scripting.

I'm working with a file with xy coordinates that were acquired with a time stamp. All the time stamps were recorded but not the XY coordinates. Let see an example:

Code:
FFID          	X	    Y    	  UNIX TIME     TIMEGAP
10001	469557,9	4266045,60	1386557563,76	
10002	469558,32	4266046,88	1386557564,65	
10003	469558,65	4266048,13	1386557565,55	
10004		                	1386557566,45	
10005	                		1386557567,36	
10006	                		1386557568,25	
10007	469559,82	4266051,82	1386557569,16	3,61
10008	469560,48	4266053,00	1386557570,05	
10009	469561,10	4266054,12	1386557570,95	
10010	469561,64	4266055,28	1386557571,85	
10011	469562,00	4266056,50	1386557572,76	
10012	469562,25	4266057,80	1386557573,66	
10013	469562,52	4266059,11	1386557574,55	
10014		                	1386557575,45	
10015		                	1386557576,35	
10016	                		1386557577,25	
10017		                 	1386557578,16	
10018	469564,31	4266063,98	1386557579,06	4,51
.
.
.
.
.

What I need to implement in awk is a interpolation by time e.g:

For the gap 10004-10006 fill the X gaps with this formula

x(i) = x1 + t(i)-t(1)/timegap e.g

10004 = x(10003) + [[t(10004)-t(10003)]/3.61]
10005 = x(10003) + [[t(10005)-t(10003)/3.61]
10006 = x(10003) + [[t(10006)-t(10003)]/3.61]

where t is UNIX TIME and so one for the next gaps.

Can anyone please help me.

Thanks in advance.

Last edited by Joćo Noiva; 12-15-2014 at 01:40 PM..
# 2  
Old 12-15-2014
Please reconsider the formulae you presented. If used as given, x(10004) = 1002939159,51. If parenthesizing the difference, as I assume it should be, x(10007) = 469559,65, so the interpolation doesn't fit. It's even worse for the second gap.
# 3  
Old 12-15-2014
I RudiC,

Thanks for your reply.

In fact I forgot parenthesizing. Is now correct. I need only to fill the gaps with the interpolation by time. The low value after the last gap in this example x (10007) = 469559,65 don't need to be interpolated because was measured. What I need is to interpolate only the gaps (that are variable)

For example for the formula that I want the result for the gaps is:

Code:
10004	469558,8993
10005	469559,1514
10006	469559,3979

My need is to automatize this process.

Last edited by Joćo Noiva; 12-15-2014 at 01:42 PM.. Reason: to complete the idea
# 4  
Old 12-15-2014
Printing the values you gave and the deltas, you can see that this is not a correct interpolation as the last delta is too large
Code:
469558.65
                0.2493 
469558.8993
                0.2521
469559.1514
                0.2465
469559.3979
                0.4221
469559.82

# 5  
Old 12-15-2014
Quote:
Originally Posted by RudiC
Printing the values you gave and the deltas, you can see that this is not a correct interpolation as the last delta is too large
Code:
469558.65
                0.2493 
469558.8993
                0.2521
469559.1514
                0.2465
469559.3979
                0.4221
469559.82

This data was acquired by a moving vessel. The timetag
was always recorded but the not the coordinates. So as the vessel speed is not constant I will have always will have deltas in the interpolation.
# 6  
Old 12-15-2014
Your interpolation formula is adding a percentage of the number one to the coordinates; not a percentage of the range between two coordinate values. Instead of the formula:
Code:
x(i) = x1 + (t(i)-t(1))/timegap

you need something more like:
Code:
x(i) = x(1) + (x(N) - x(1)) * ((t(i) - t(1)) / (t(N) - t(1)))

where N is the index of the next known coordinate point and timestamp.

Using this formula, the awk script:
Code:
awk '
NR == 1 {
	print
	next
}
NF == 2{t[++ic] = $2
	id[ic] = $1
	next
}
{	if(ic > 1) interpolate()
	print
	t[ic = 1] = $4
	x = $2
	y = $3
}
function interpolate(	X, Y, i, j) {
#	id[++ic] = $1;t[ic] = $4	# Test Only
	for(i = 2; i <= ic; i++) {
		X = x + ($2 - x) * ((t[i] - t[1]) / ($4 - t[1]))
		Y = y + ($3 - y) * ((t[i] - t[1]) / ($4 - t[1]))
		printf("%s\t%.4f\t%.4f\t%s\n", id[i], X, Y, t[i])
	}
}' file

(with your sample input with commas converted to periods (since I'm in the US instead of Portugal)) produces the output:
Code:
FFID          	X	    Y    	  UNIX TIME     TIMEGAP
10001	469557.9	4266045.60	1386557563.76	
10002	469558.32	4266046.88	1386557564.65	
10003	469558.65	4266048.13	1386557565.55	
10004	469558.9417	4266049.0499	1386557566.45
10005	469559.2366	4266049.9801	1386557567.36
10006	469559.5251	4266050.8898	1386557568.25
10007	469559.82	4266051.82	1386557569.16	3.61
10008	469560.48	4266053.00	1386557570.05	
10009	469561.10	4266054.12	1386557570.95	
10010	469561.64	4266055.28	1386557571.85	
10011	469562.00	4266056.50	1386557572.76	
10012	469562.25	4266057.80	1386557573.66	
10013	469562.52	4266059.11	1386557574.55	
10014	469562.8772	4266060.0818	1386557575.45
10015	469563.2344	4266061.0537	1386557576.35
10016	469563.5916	4266062.0255	1386557577.25
10017	469563.9528	4266063.0082	1386557578.16
10018	469564.31	4266063.98	1386557579.06	4.51

As a sanity check, you can uncomment line before the for loop. In that mode it produces the output:
Code:
FFID          	X	    Y    	  UNIX TIME     TIMEGAP
10001	469557.9	4266045.60	1386557563.76	
10002	469558.32	4266046.88	1386557564.65	
10003	469558.65	4266048.13	1386557565.55	
10004	469558.9417	4266049.0499	1386557566.45
10005	469559.2366	4266049.9801	1386557567.36
10006	469559.5251	4266050.8898	1386557568.25
10007	469559.8200	4266051.8200	1386557569.16
10007	469559.82	4266051.82	1386557569.16	3.61
10008	469560.48	4266053.00	1386557570.05	
10009	469561.10	4266054.12	1386557570.95	
10010	469561.64	4266055.28	1386557571.85	
10011	469562.00	4266056.50	1386557572.76	
10012	469562.25	4266057.80	1386557573.66	
10013	469562.52	4266059.11	1386557574.55	
10014	469562.8772	4266060.0818	1386557575.45
10015	469563.2344	4266061.0537	1386557576.35
10016	469563.5916	4266062.0255	1386557577.25
10017	469563.9528	4266063.0082	1386557578.16
10018	469564.3100	4266063.9800	1386557579.06
10018	469564.31	4266063.98	1386557579.06	4.51

The lines shown in red confirm that the interpolated values for the end of the range match the actual end of range values.

The provided values are all shown with the number of decimal places provided in the input; the interpolated values are shown with four decimal places. If you'd like the interpolated values to be printed with two decimal places, change both occurrences of %.4f in the printf format string to %.2f. (I used a different number of decimal places to make it obvious which values were interpolated.)

Note that if you have other code that is generating the TIMEGAP field in your sample input, it isn't needed for this code to work.

You haven't said what OS you're using. If you want to run this script on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk, /usr/xpg6/bin/awk or nawk.
This User Gave Thanks to Don Cragun For This Post:
# 7  
Old 12-17-2014
Dear Don,

First I want to thank you for you suggestion and correction of the equation that was obviously wrong as RudiC also said.

Secondly I also thank you for your clean, simple and functional code!

Thirdly I ask you if you don't my mind a little more help.

I have a second pack of data that I need to interpolate based on the previous interpolated file.

I try to be short and clear:

Based on the RESULT.txt file (with all the data interpolated) I need to interpolate the values from a second file AIS_557.txt that have (due to issues during data acquisition) very few data points and with time stamps without decimals, based on first file. See example:


Code:
RESULT.TXT
010001 0469557.90 4266045.60 1386557563.76
010002 0469558.32 4266046.88 1386557564.65
010003 0469558.65 4266048.13 1386557565.55
010004 0469558.94 4266049.05 1386557566.45
010005 0469559.24 4266049.98 1386557567.36
010006 0469559.53 4266050.89 1386557568.25
010007 0469559.82 4266051.82 1386557569.16
010008 0469560.48 4266053.00 1386557570.05
010009 0469561.10 4266054.12 1386557570.95
010010 0469561.64 4266055.28 1386557571.85
010011 0469562.00 4266056.50 1386557572.76
010012 0469562.25 4266057.80 1386557573.66
010013 0469562.52 4266059.11 1386557574.55
010014 0469562.88 4266060.08 1386557575.45
010015 0469563.23 4266061.05 1386557576.35
010016 0469563.59 4266062.03 1386557577.25
010017 0469563.95 4266063.01 1386557578.16
010018 0469564.31 4266063.98 1386557579.06
010019 0469564.85 4266065.22 1386557579.95
010020 0469565.34 4266066.47 1386557580.85
010021 0469565.79 4266067.67 1386557581.76
...........


Code:
 AIS_557.txt
468953.73 4264882.13 1386533790
468944.97 4264871.07 1386533820
468936.22 4264860.01 1386533850
468927.46 4264848.94 1386533880
468918.70 4264837.88 1386533910
468918.65 4264826.79 1386533940
468909.89 4264815.72 1386533970
468892.38 4264793.60 1386534030
468883.62 4264782.54 1386534060
468874.90 4264782.57 1386534090
468866.14 4264771.51 1386534150
468857.43 4264771.54 1386534180
468839.95 4264760.51 1386534210
468822.43 4264738.39 1386534240
468813.67 4264727.33 1386534270
..........

So I need to find the best fit for each time stamp of AIS file on the RESULT file. Interpolate X Y (for each time stamp of RESULT file) for the second file (AIS file).

Me and a colleague we are trying to adapt your function to do this but it is not working yet. The code is this one:

Code:
awk '
function interpola(i, P1, P2,  Xi, Yi) {
	Xi= Xais[P1] +((Xais[P2]-Xais[P1])*(Tsis[i]-Tais[P1]))/(Tais[P2]-Tais[P1])
	Yi= Yais[P1] +((Yais[P2]-Yais[P1])*(Tsis[i]-Tais[P1]))/(Tais[P2]-Tais[P1])
	printf("%06i %010.2f %010.2f %013.2f\n", FFID[i], Xi, Yi, T[i])
#	print FFID[i], T[i], P1, P2 #test fitting points for interpolation (keeps line number where are P1 or P2)
}
{
NR==FNR{Tsis[FNR]=$4; Sis[FNR]=$0; next} # Reads the first file with principal serie 
{Xais[FNR]=$1; Yais[FNR]=$2; Tais[FNR]=$3 } # Reads the second file with AIS values to interpolate

}
END {
for (i in Tsis[FNR]) {
	for (j=1; j<FNR; j++) {	# Runs down until find P1
		if (Tsis[i]>Tais[FNR]) {
			P1=j
			break
		}
	}
	for (k=j; k<=FNR; k++) { # Runs down until find P2
		if ((Tsis[i]<Tais[FNR]) ) {
			P2=k
			break
		}
	}	
	interpola(i, P1, P2) 	#interpolation and print interpolated values
}

}' '  RESULT.txt AIS_557.txt  > AIS_INTERPOLATION.txt

Can you give us, please, a suggestion!

Thanks in advance
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

An interpolation between two files

Dear all, I always appreciate your help. I am an electrical engineer. I am using a tool for timing analysis of a circuit. I would like to interpolate results from two timing reports at different voltages (0.945V and 0.78V). If voltage is decreased, data arrival time is increased. For... (4 Replies)
Discussion started by: jypark22
4 Replies

2. Shell Programming and Scripting

Interpolation of two values in two different files

Dear All, I have two files which contain numerical data and strings. I want to create a new file that only revise numerical data from two files using interpolation. I guess AWK works, but I am new on AWK. FileA.txt . . index_2("0.1, 1, 2, 4, 8, 16, 32"); values("0.0330208, 0.0345557,... (17 Replies)
Discussion started by: jypark22
17 Replies

3. Shell Programming and Scripting

Awk: time intervals based on epoch time

I have a list of epoch times delimited by "-" as follows: 1335078000 - 1335176700 1335340800 - 1335527400 1335771300 - 1335945600 1336201200 - 1336218000 The corresponding dates are: 20120422 1000 - 20120423 1325 20120425 1100 - 20120427 1450 20120430 1035 - 20120502 1100 ... (3 Replies)
Discussion started by: alex2005
3 Replies

4. Programming

Find gaps in time data and replace missing time value and column 2 value by interpolation in awk

Dear all, I am kindly seeking assistance on the following issue. I am working with data that is sampled every 0.05 hours (that is 3 minutes intervals) here is a sample data from the file 5.00000 15.5030 5.05000 15.6680 5.10000 16.0100 5.15000 16.3450 5.20000 16.7120 5.25000... (4 Replies)
Discussion started by: malandisa
4 Replies

5. Shell Programming and Scripting

Hi ! whether it is possible to do interpolation in scripting...

Hi ! Experts... I just wanted to know whether it is possible in scripting...to do interpolation.... if so....have a look on my data file I need temperature and salinity value with a bin size of 0.5 m output looks somewhat like this dep temp sal 0.5 25 0.077 1 25 ... (12 Replies)
Discussion started by: nex_asp
12 Replies

6. UNIX for Dummies Questions & Answers

Converting string date time to unix time in AWK

I'd like to convert a date string in the form of sun aug 19 09:03:10 EDT 2012, to unixtime timestamp using awk. I tried This is how each line of the file looks like, different date and time in this format Sun Aug 19 08:33:45 EDT 2012, user1(108.6.217.236) all: test on the 17th ... (2 Replies)
Discussion started by: bkkid
2 Replies

7. Shell Programming and Scripting

Interpolation using awk

Hi all, Consider I have a text file containing: 1003 60 1005 80 1100 110 Based on that file I need to create another file which is containing value from 1001 till 1100 which is a linear interpolation between two point (for 1004; 1006;1007 until 1109) and extrapolation based on 2... (7 Replies)
Discussion started by: ardy_yana
7 Replies

8. Shell Programming and Scripting

Variable interpolation in "print" of awk command

Hi, I have a snippet like below. Based on variable i, i wish to print 1,2,3,4,5th columns to Sample files. For each loop, one column from contetn and results will be pused to sample files. But i have a problem here i=1 while ; do `awk -F"\t" '{print $($i)}' $content > Sample_${i}_original`;... (4 Replies)
Discussion started by: forums123456
4 Replies

9. Web Development

JavaScript variable interpolation

Hi everybody, Firstly, this would be the first time I'm using JavaScript. My background is mainly Perl. Nevertheless, here's my problem. So I've created a function function linkout(url){ setTimeout("window.open(url)",5000) //in milliseconds } However because the variable "url" is... (0 Replies)
Discussion started by: z1dane
0 Replies

10. UNIX for Dummies Questions & Answers

variable interpolation

I've become obsessed with trying to get this to work. As of yet, I am unable to figure it out. Unfortunately, I don't have Linux or UNIX available when I get home. Anyone have tips for me on how I can pass param1 to ID via use of COUNTER and loop? thx. LIMIT=6 param1="999999999" export... (0 Replies)
Discussion started by: egkumpe
0 Replies
Login or Register to Ask a Question