Shell or awk script to compute average of all the points within a circle


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell or awk script to compute average of all the points within a circle
# 1  
Old 11-21-2015
Shell or awk script to compute average of all the points within a circle

HI Help,

I have a file which looks like below ---

Input file --->

Code:
1970113.00000 3460.00000 1.09516 
1970116.00000 3791.00000 1.06350 
1970120.00000 4120.00000 1.07588 
1970115.00000 4450.00000 1.09591 
1970116.00000 4780.00000 1.09965 
1970120.00000 5109.00000 1.06733 
1970122.00000 5440.00000 1.03760 
1970124.00000 5770.00000 1.02025 
1970123.00000 6100.00000 1.00998 
1970120.00000 6430.00000 0.96426

What I want to do ? ----
Code:
For each line (#NR= 1 to 10) I would like to search X($1),Y($2) within the I/P file  which lies within a radius of 50 and then average out all the $3 within that circle of radius 50.

Problem ??

My script below does the job but it takes ages to go through the input file which is TOO large.

Code:
 
set first = 1    # NR=1 ; first line fo the input file
set last = 10   # NR = 10; Last line of the input file
 
set num = ${first}
while (${num} <= ${last})
 
set X0 = `cat S2 | awk -v line=${num} 'NR==line' | awk '{print $1}'` # $1,X, for each Record
set Y0 = `cat S2 | awk -v line=${num} 'NR==line' | awk '{print $2}'` # $2,Y, for each Record
set XS = `cat S2 | awk -v line=${num} 'NR==line' | awk '{print $3}'` # $3,Value that will be averaged out, for each Record
set R0 = `echo "50"` # Search radius
 
set AVG = `cat S2 | awk -v X=${X0} -v Y=${Y0} -v R=${R0} '{print $1,$2,$3,sqrt(($1-X)*($1-X) + ($2-Y)*($2-Y)) <=R}'  | awk '{if($4==1){print $0}}' | awk '{ sum+=$3} END {print sum/NR}'`# Search all the points within the input file which lies within R #Average all the $3,values#
 
echo "${X0} ${Y0} ${XS} ${AVG}" >> TMP #For each record print a new line with existing $1,$2,$3 and $AVG
 
@ num++ # end of loop and it goes to next NR
end

This could be very easy for you experts.

Thanks,

Last edited by Indra2011; 11-22-2015 at 11:29 AM..
# 2  
Old 11-22-2015
Without understanding what exactly you are doing to that file, I can see that you are creating 15 processes to run 15 commands on every single line, which will have a serious effect if done for many lines / a large file. On the other hand, for 10 lines only that should not be noticed at all.

I'd say all of the above could be done in one single (awk?) script, accelerating the entire processing considerably.

Aside: the XS value will always be 0 as $4 does not exist in the input file (assuming "S2" IS the input file).
This User Gave Thanks to RudiC For This Post:
# 3  
Old 11-22-2015
Thanks a lot RudiC.

The catch was fantastic. You are right considering the input file that should be
Code:
$3

and not
Code:
$4

.

Let me explain what I am trying to do......

Code:
For NR==1,
 
I am trying to look through or search through the entire input file and find X ($1, NR >1 to NR = last record) and Y ($2, NR>1 to NR=last record) which lies within a radius of 50meter from the X0 ($1 for NR=1) and Y0($1 for NR=1). If 'ANY' found , print "1" and then add $3 for all those points and divide by the numbers of the point found inside the circle of radius R=50.
 
For NR==2 ------
Repeating the same process above.
 
For NR== 3------etc etc till last line of the record.

So, basically searching the points which lies within a radius of 50 from each points within that file and then averaging out the
Code:
$3

with number of the points found inside that circle.

Thanks,

Last edited by Indra2011; 11-22-2015 at 12:27 PM..
# 4  
Old 11-22-2015
So - is S2 the input file? How many lines? Will the calculations be done for every line in the file (lets call that ALLINES) or just for the first num lines? Yielding ALLINES * ALLINES result lines as opposed to num * ALLINES result lines? Will all the results go to one single output file?
This User Gave Thanks to RudiC For This Post:
# 5  
Old 11-22-2015
Yes Rudi.
Code:
S2

is the input file.
The file has almost 80000 lines.
Calculations will be done for every line (ALLLINES), yielding ALLLINES * ALLLINES result.

Thanks,
# 6  
Old 11-22-2015
I'm not sure I interpreted your requirements correctly. Your sample file doesn't have any line's coordinates within radius 50 from any other, so any test is impossible for that set (it did some avaraging for R = 1500).
Try
Code:
awk -vR0=50 '
        {X[NR]=$1
         Y[NR]=$2
         V[NR]=$3
        }
END     {for (n=1; n<=NR; n++)  {X0 = X[n]
                                 Y0 = Y[n]
                                 SUM = CNT = 0
                                 for (i=1; i<=NR; i++)  {R = sqrt((X[i]-X0)*(X[i]-X0) + (Y[i]-Y0)*(Y[i]-Y0))
                                                         if (R<=R0)     {SUM += V[i]
                                                                         CNT++
                                                                        }
                                                        }
                                 print X0, Y0, V[n], SUM/CNT
                                }
        }
' file
1970113.00000 3460.00000 1.09516 1.09516
1970116.00000 3791.00000 1.06350 1.0635
1970120.00000 4120.00000 1.07588 1.07588
1970115.00000 4450.00000 1.09591 1.09591
1970116.00000 4780.00000 1.09965 1.09965
1970120.00000 5109.00000 1.06733 1.06733
1970122.00000 5440.00000 1.03760 1.0376
1970124.00000 5770.00000 1.02025 1.02025
1970123.00000 6100.00000 1.00998 1.00998
1970120.00000 6430.00000 0.96426 0.96426

Not sure how this approach would handle 80000 lines, though...

---------- Post updated at 20:05 ---------- Previous update was at 20:00 ----------

I have to correct myself - that won't be ALLINES * ALLINES result lines but ALLINES result lines, still ALLINES * ALLINES computations...
These 2 Users Gave Thanks to RudiC For This Post:
# 7  
Old 11-22-2015
Many many thanks RudiC. Besides solution, it is always great learning from the script you create. I highly appreciate your skills.

Best Regards
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk or sed script to count number of occurrences and creating an average

Hi Friends , I am having one problem as stated file . Having an input CSV file as shown in the code U_TOP_LOGIC/U_HPB2/U_HBRIDGE2/i_core/i_paddr_reg_2_/Q,1,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,0,0... (4 Replies)
Discussion started by: kshitij
4 Replies

2. Shell Programming and Scripting

Compute average based on field values

Im looking for a way to average the values in field 14 (when field 2 is equal to 2016) and fields 3 and 4 (when field 2 is equal to 2017). Any help is appreciated. 001001 2016 33.22 38.19 48.07 51.75 59.77 67.68 70.86 72.21 66.92 53.67 42.31 40.15 001001 2017 ... (10 Replies)
Discussion started by: ncwxpanther
10 Replies

3. Shell Programming and Scripting

Linux Mount Points usage check with shell script

Hi Everyone, Just in need of your help again, I have managed to get a script to check the linux disk usage stuff. But wanted to tweak it little more to get the desired output. Requirement: Now its querying only one mount point, As its not saving in array instead calling it as variables. So... (1 Reply)
Discussion started by: thiyagoo
1 Replies

4. Shell Programming and Scripting

Compute average ignoring outliers of different segments within a dat file using awk

I have data files that look like this, say data.txt 0.00833 6.34 0.00833 6.95 0.00833 7.08 0.00833 8.07 0.00833 8.12 0.00833 8.26 0.00833 8.70 0.00833 9.36 0.01667 20.53 0.01667 6.35 0.01667 6.94 0.01667 7.07 0.01667 8.06 0.01667 8.10 0.01667 8.25 0.01667 8.71 0.01667 9.31... (7 Replies)
Discussion started by: malandisa
7 Replies

5. Shell Programming and Scripting

AWK script to split data and find average

Input: 2.58359023380340e+02 1.43758864405595e+02 -7.65700666212508e+00 1.06460208083228e+02 1.26185441783936e+02 -3.41389169427027e+01 -1.40393299309592e+02 -3.07758776849508e+01 1.45067703495838e+02 1.79405834959073e+02 5.06666234594205e+01 OUT 2.0105894389e+02 (average of... (8 Replies)
Discussion started by: chrisjorg
8 Replies

6. Shell Programming and Scripting

awk based script to find the average of all the columns in a data file

Hi All, I need the modification for the below mentioned code (found in one more post https://www.unix.com/shell-programming-scripting/27161-script-generate-average-values.html) to find the average values for all the columns(but for a specific rows) and print the averages side by side. I have... (4 Replies)
Discussion started by: ks_reddy
4 Replies

7. Homework & Coursework Questions

Shell Script average runtime

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: Make a bash script that calculates average runtime for the first two scripts you made. The average should be... (17 Replies)
Discussion started by: navlelo
17 Replies

8. Shell Programming and Scripting

shell script for finding average runtime of other script

so I've made a shell script that downloads 6 files in succession from a given url, then deletes them. Now I want to time the script, and the average time it uses by running it ~100 times. My problem is tho, how do I store the time it takes for each run through of the script? I know time writes to... (3 Replies)
Discussion started by: navlelo
3 Replies

9. Shell Programming and Scripting

Compute the median of a set of numbers with AWK?

Is there a way in awk to compute the median of a set of numbers in a file in the following format. 34 67 78 100 23 45 67 (3 Replies)
Discussion started by: Lucky Ali
3 Replies

10. Shell Programming and Scripting

Shell Script to see the mount points.

Hi all, First of all I dont even know the ABC of scripting .. But now I want a Script to see the mount points of the file systems Can any body help plsssssssss :o (1 Reply)
Discussion started by: priky
1 Replies
Login or Register to Ask a Question