Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Awk: Iterate over all records, stop when value < threshold

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awk: Iterate over all records, stop when value < threshold
# 1  
Old 03-27-2014
Awk: Iterate over all records, stop when value < threshold
Hello I have data arranged in two columns

Code:
0 7.7272086460791485
1 7.560656885151559
2 7.019966124180658
3 7.2403229620576015
4 7.200882072174212
5 7.3231008505909445
6 7.284184600697482
7 6.933936488650265
8 7.311824413328238
9 7.651279125577341
10 7.536873675349084
11 6.9220655676488025
12 7.168193668875762
13 6.856087051713454
14 7.40589866332712
15 6.723136760289026
16 7.303837907836767
17 7.25148060447339
18 7.25969711845886
19 7.303844998557891
20 7.337370442953407
21 7.091735597353506
22 6.789705841533072
23 6.826330143275894
24 7.095998255319059
...
...
127 6.667292681807827
128 6.930002598091201
129 7.422422147885169
130 7.259891211725509
131 7.281619006386211
132 6.4913982829224075
133 6.041526310918811
134 5.387829354522078
135 3.991014879884833
136 4.052976930635175
137 4.184443617333475
138 3.763408797879499
139 4.4784425565732
140 5.107326919922296
141 3.632494498884792
142 4.138827640820262
143 3.9833286712642084
144 4.027717561305877
145 3.9407127615889617


I wish to iterate down until NR. I want the script to stop when arr[$2]<cutoff value I choose.


I tried with


Code:
{for(i=1;i<=NR;i++) {n[i]++; arr[i]=$1; arr2[i]=$2}}
END { for(i=1;i<=NR;i++) {if(arr2[i]<4.5) {print arr[i]; arr2[i]; i; exit}}


But it doesn't work yet. Is there anyone who could help me attempt this?
I would also like to work out a RMSD relative to the first value, here 7.560656885151559. It's to prove that it plateaus.
Last edited by Scrutinizer; 03-28-2014 at 03:19 PM.. Reason: quote tags to code tags
# 2  
Old 03-27-2014
Not clear. Please explain in depth what you need, and lay out in English what algorithms you want to be used.
# 3  
Old 03-27-2014
  • I wish to use AWK to parse my data,
  • AWK must read $1 into arr[i] and $2 into arr2[i]
  • I wish the parsing to stop when the value in arr2 is less than a constant cutoff value I pick.
  • At this point, I want to print index i, the value of $2 i.e. arr2[i], and I want AWK to break from the for loop.
# 4  
Old 03-27-2014
I'm not sure what you need the for loop for.
To do what you specify in above post, try
Code:
awk '{arr[NR]=$1; arr2[NR]=$2} $2<cutoff {exit} END{print NR, $2}' cutoff="4.5" file

# 5  
Old 03-27-2014
Thanks, that works.


I also want to work out the difference of every value value in the array relative to the first entry. I would like to print this into a file.

arr2[1]-arr2[i]



---------- Post updated at 04:17 PM ---------- Previous update was at 04:02 PM ----------

EDIT: This works now.
Also, how could I introduce the double conditional that

$2<cutoff && the difference between subsequent elements in array < threshold chosen?


Code:
  awk '{arr[NR]=$1; arr2[NR]=$2} {if($2<cutoff && arr2[NR]-arr2[NR-1]<threshold) {exit}} END{print NR, arr2[NR]-arr2[NR-1], arr2[NR], arr2[NR-1], $2}' cutoff="4.5" threshold="2"

# 6  
Old 03-27-2014
Like so:
Code:
awk     '                                       {arr[NR]=$1; arr2[NR]=$2}
         $2<cutoff && ($2-arr2[NR-1])<threshold {exit}
         END                                    {print NR, arr2[NR]-arr2[NR-1], arr2[NR], arr2[NR-1], $2}
        ' cutoff="4.5" threshold="1" file

?
# 7  
Old 03-27-2014
Yes, that works.

What would be super, is if I could extend the

Code:
&& ($2-arr2[NR-1])<threshold

condition so that say, 10 iterations have to hold true before it exits? I'm trying to probe my data has reached a plateau.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using awk to assign binary values to data above/below a certain threshold?

Hello, I have files containing a large amount of values in columns, and I want to simplify the data by making the values binary, i.e. assigning 1/0 for each value above or below a certain threshold. I am curious to know if it's possible to use awk to do this (or another way in bash). I've... (2 Replies)
Discussion started by: ksennin
2 Replies

2. Shell Programming and Scripting

Skip first and last n records with awk

Hi, I have an awk code that reads an input file, checks the 4th column and tells if its fine. #!/bin/ksh { if ($4 == 0) print "fine" else print "some problem" }' FILENAME My problem is that, I dont want to check the first 3 and last 3 lines. This can be hard coded by using BEGIN and END... (9 Replies)
Discussion started by: gotam
9 Replies

3. Shell Programming and Scripting

iterate through list of numbers and print specific lines with awk

Could someone please point me in the right direction with the following? I have a program that generates logs that contains sections like this: IMAGE INPUT 81 0 0.995 2449470 0 1726 368 1 0.0635 0.3291 82 0 1.001 2448013 0 1666 365 1 0.0649 ... (4 Replies)
Discussion started by: euval
4 Replies

4. Shell Programming and Scripting

Substituting variable value in AWK /start/,/stop/

Hi all u brilient people on the forum... I am trying to call the variable value in awk command for search pattern /start/,/stop/ but i am nt able to do this .... wat i did is ..i have created two variable YESTERDAY and TODAY and passed the y'day n 2'days dates in it...like this ... (14 Replies)
Discussion started by: whomi
14 Replies

5. Shell Programming and Scripting

Counting records with AWK

I've been working with an awk script and I'm wondeing id it's possible to count records in a file which DO NOT contain, in this instance fields 12 and 13. With the one script I am wanting to display the count for the records WITH fields 12 and 13 and a seperate count of records WITHOUT fields... (2 Replies)
Discussion started by: Glyn_Mo
2 Replies

6. UNIX for Dummies Questions & Answers

Iterate a min/max awk script over time-series temperature data

I'm trying to iterate a UNIX awk script that returns min/max temperature data for each day from a monthly weather data file (01_weath.dat). The temperature data is held in $5. The temps are reported each minute so each day contains 1440 temperature enteries. The below code has gotten me as far as... (5 Replies)
Discussion started by: jgourley
5 Replies

7. UNIX for Advanced & Expert Users

AWK aggregate records

Hy all, I have a problem...can some one help me... I have a file of records sort: 30|239|ORD|447702936929 |blackberry.net |20080728|141304|00000900|2|0000000000000536|28181|0000000006|0000000001|10|1 30|239|ORD|447702936929 |blackberry.net ... (4 Replies)
Discussion started by: anaconga
4 Replies

8. Shell Programming and Scripting

awk - Number of records

Hi, Is it possible to find the total number of records processed by awk at begining. NR gives the value at the end. Is there any variable available to find the value at the begining? Thanks ---------- Suman (1 Reply)
Discussion started by: suman_jakkula
1 Replies

9. Shell Programming and Scripting

Stop awk adding a new line

In a loop, I want to append some text to a file without generating a new line (and then force a new line before re-iterating the loop). In the code below the first 'echo' command is OK as it uses '--n' for no new line. For the 'awk' line I *thought* I could solve it by using printf rather than... (1 Reply)
Discussion started by: TobyR
1 Replies

10. UNIX for Dummies Questions & Answers

awk | stop after specified number of results

I am searching some rather large text files using grep and or awk. What I would like to know is if there is a way (either with grep, awk, or realy any other unix tool) to stop the search when a predifined number of results are returned. I would like to do this for speed purpuses. When i get... (6 Replies)
Discussion started by: evan108
6 Replies
Login or Register to Ask a Question