Awk: Iterate over all records, stop when value < threshold


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awk: Iterate over all records, stop when value < threshold
# 15  
Old 03-28-2014
Hello,


yes it's the same data,
the data has an exponential-type decay, to a plateau value.

I wish to start analysing the data once it crosses a cutoff value ($2<cutoff).

But I've had trouble putting only the subset of data into an array.
# 16  
Old 03-31-2014
Code:
awk     '(FILENUM != ARGIND) { 
          FILENUM=ARGIND; arr[FNR]=$1; arr2[FNR]=$2}
         (ARGIND == 1) {
         if($2<4.5) {
                complete="YES"
                exit}
         else   {
                complete="NO"
                exit
         }
         }
         END{printf "%s    %s%6d  %s   %s   %4.3f   %s  %s ", file, "time", FNR, "ps",  "iteration ended at COM distance", $2, "finished pulling?", complete}

         (ARGIND == 2 && $2<4.5) {
         arr3[FNR]+=$2

         }

         END{printf "%4.3f  %6d %4.3f", arr3[FNR], FNR, arr3[FNR]/FNR}' file="$old" $old $old > $old.result

I know this looks painful, but how could I amend the script to make it

First parsing
I) parse until $2<4.5


Second parsing
II) parse all lines $2<4.5
# 17  
Old 04-01-2014
If anyone could help I would be very grateful.
# 18  
Old 04-01-2014
Moderator's Comments:
Mod Comment Bumping up posts or double posting is not permitted in these forums.

Please read the rules, which you agreed to when you registered, if you have not already done so.

You may receive an infraction for this. If so, don't worry, just try to follow the rules more carefully. The infraction will expire in the near future

Thank You.

The UNIX and Linux Forums.


I don't understand what those 'exits' are for.

I also don't understand why you have multiple END sections. END always does the same thing.

I also don't understand why you're suddenly wedging in FNR everywhere, including the END statements. 99% of the data you store in your arrays never gets used for anything!

Every time I post in this thread I get more and more confused to your requirements. If you could post a good example of the output you want and how it's calculated from the input you gave it'd be better than a million posts saying "program doesn't work how do I fix this". I can't tell what you want from a program that doesn't do what you want...

Last edited by Corona688; 04-01-2014 at 04:28 PM..
# 19  
Old 04-01-2014
Quote:
For the data

0 7.722388226523387
1 7.7660013123828096
2 8.290526837606086
3 6.77884110037098
4 7.029582689281424
5 6.715810839331927
6 7.517426758402166
7 6.183903363655094
8 7.048724452467266
9 5.8582750452037935
10 6.481938261085246
11 6.769077500888982
12 6.413736963496258
13 6.961597909625096
14 6.6894614128619985
15 6.535369421593066
16 6.540483079147737
17 6.641503210089296
18 6.5401793348950195
19 7.013346631720046
20 6.962681568569234
21 6.101765805639436
22 6.513251521133769
23 6.338325888236119
24 6.10717577973319
25 6.20675401281057
26 6.50695728982179
27 6.357749919413864
28 7.409494628637002
29 6.840173288471167
30 6.7239904997599185
31 6.74352427303535
32 6.64770184328439
33 6.754765081034419
34 6.059294935944365
35 6.850334994202908
36 6.612677990890236
37 6.71504474361227
38 6.569996401046845
39 6.6011511309164455
40 7.013453227125966
41 7.228367551888497
42 7.331141091573926
43 6.527488465738153
44 6.851614386790745
45 6.781393097665132
46 6.747845089814984
47 7.324883604117072
48 6.830582078298863
49 6.910474901865819
50 6.861282607112627
51 7.048093038158803
52 7.11671226863833
53 6.784646894031629
54 6.728933862592124
55 6.839299531337964
56 6.9248351147716285
57 6.707443598582235
58 6.567005640364306
59 7.135919424886934
60 6.8901293151100695
61 7.192896318331181
62 6.424458951708234
63 6.740565438718713
64 6.463817391585307
65 7.073175282345635
66 7.280386984993104
67 6.848030199739276
68 6.800067935371437
69 6.720153678071956
70 6.6733076601989065
71 7.052336379040527
72 7.2855001092764
73 6.7828749786768
74 6.588902858372063
75 7.183111022760112
76 7.025074001561643
77 6.298917194125965
78 5.060227277866115
79 3.7067556568399405
80 3.7385422657156244
81 3.651782457306803
82 4.129905327856894
83 3.6246016188137897
84 3.7543591923187885
85 3.6423328039916885
86 3.937790731265336
87 3.951315156802678
88 3.607711381535056
89 3.6769293836203003
90 3.675821941808219
91 4.035941680407786
92 3.933839462461634
93 3.9893465006947655
94 4.001331166527052
95 3.7301856192807996
96 3.786365060300995
97 3.367445018143988
98 3.9833897220631926
99 3.464373610471933
100 4.1261964574527115
101 3.961761165286271
102 3.631884666760846
103 3.808670191420319
104 3.481851851199455
105 3.9129751688983516
106 3.864645351207803
107 3.9388607209143904
108 3.9917072232946462
109 4.173913989059009
110 3.8644608113488714
111 3.4909078095798494
112 3.857727526173673
113 3.9955722761950208
114 3.841472483862828
115 4.327350762790359
116 4.222059771755263
117 3.6938834462886407
118 4.082307570304044
119 3.869427172854604
120 3.8297893596578865
121 4.048114028065467
122 3.7362481012905313
123 3.68296301638126
124 3.807018865189885
125 3.7602551272676807
126 3.791213404646831
127 3.793813085414115
128 3.9073068483890436
129 3.8311558374044252
130 3.766559261531107
131 3.9191333626800557
132 3.9302111763343803
133 3.9784828086329047
134 3.8369050113764396
135 4.038057743574033
136 3.9182467428937615
137 4.007878767130461
138 4.108746406819583
139 3.796683992663568
140 3.7923734572273684
141 4.144883005724992
142 3.803391914935476
143 4.067326518137691
144 3.344418506428391
145 3.7629409571082935
146 3.4253301410095784
147 4.0086181974652435
148 4.288692167965181
149 4.044528641807874
150 3.8274927201409454
151 4.055321679187342
152 4.156569414459632
153 3.843598172836961
154 3.9078377234413724
155 4.131821303727747
156 3.723268729747613
157 3.9046451202882317
158 3.3076098756852423
159 3.705148379807119
160 3.9408940535877828
161 3.9302243193675155
162 3.872291403254681
163 4.142381359249118
164 3.7924616760678598
165 3.7605461524752384
166 3.731087886745309
167 4.156014272602539
168 4.256958171035382
169 3.8075483595119137
170 3.7224969883961117
171 3.8538025773174382
172 4.468919387005501
173 3.9708056475507028
174 3.6764405676629064
175 3.865485812998111
176 3.2381932840236733
177 4.3109393367004625
178 3.84070854535715
179 3.8962591015573995
180 3.895709035880224
181 3.915425264517956
182 3.9089665481223
183 4.08992911568025
184 4.310764539083257
185 4.185574941400778
186 3.549381773516515
187 3.702343034626684
188 3.9676793710887224
189 3.6871518686786975
190 3.8861650798429674
191 3.6516982023067146
192 3.96665364882544
193 3.8707987942742483
194 4.025032293232348
195 3.64333297623619
196 4.005121087856924
197 3.677139051757457
198 3.541561929883151
199 4.121166712469015



I want on the first parsing, to parse until $2<cutoff,

At this point, I want to exit. I want to print NR, and $2.

Not all data files I input will meet the $2<cutoff, hence I set the variable complete=YES if it is true, otherwise NO.




I'm using FNR because otherwise NR is for both times I parse the file.


Code:
if($2<4.5) {
                complete="YES"
                exit}
         else   {
                complete="NO"
                exit
         }
         }
         END{printf "%s    %s%6d  %s   %s   %4.3f   %s  %s ", file, "time", FNR, "ps",  "iteration ended at COM distance", $2, "finished pulling?", complete}






Second parsing:

I want to start parsing after the first time $2<cutoff.

I want to compute an array arr3+=$2, which should sum the parse values. At the end of the file, I want to print arr3 and arr3/FNR.


The output for the EXAMPLE DATA BELOW

Quote:

79 3.7067556568399405
80 3.7385422657156244
81 3.651782457306803
82 4.129905327856894
83 3.6246016188137897
84 3.7543591923187885
85 3.6423328039916885
86 3.937790731265336
87 3.951315156802678
88 3.607711381535056
89 3.6769293836203003
90 3.675821941808219
91 4.035941680407786
92 3.933839462461634
93 3.9893465006947655
94 4.001331166527052
95 3.7301856192807996
96 3.786365060300995
97 3.367445018143988
98 3.9833897220631926
99 3.464373610471933
100 4.1261964574527115
101 3.961761165286271
102 3.631884666760846
103 3.808670191420319
104 3.481851851199455
105 3.9129751688983516
106 3.864645351207803
107 3.9388607209143904
108 3.9917072232946462
109 4.173913989059009
110 3.8644608113488714
111 3.4909078095798494
112 3.857727526173673
113 3.9955722761950208
114 3.841472483862828
115 4.327350762790359
116 4.222059771755263
117 3.6938834462886407
118 4.082307570304044
119 3.869427172854604
120 3.8297893596578865
121 4.048114028065467
122 3.7362481012905313
123 3.68296301638126
124 3.807018865189885
125 3.7602551272676807
126 3.791213404646831
127 3.793813085414115
128 3.9073068483890436
129 3.8311558374044252
130 3.766559261531107
131 3.9191333626800557
132 3.9302111763343803
133 3.9784828086329047
134 3.8369050113764396
135 4.038057743574033
136 3.9182467428937615
137 4.007878767130461
138 4.108746406819583
139 3.796683992663568
140 3.7923734572273684
141 4.144883005724992
142 3.803391914935476
143 4.067326518137691
144 3.344418506428391
145 3.7629409571082935
146 3.4253301410095784
147 4.0086181974652435
148 4.288692167965181
149 4.044528641807874
150 3.8274927201409454
151 4.055321679187342
152 4.156569414459632
153 3.843598172836961
154 3.9078377234413724
155 4.131821303727747
156 3.723268729747613
157 3.9046451202882317
158 3.3076098756852423
159 3.705148379807119
160 3.9408940535877828
161 3.9302243193675155
162 3.872291403254681
163 4.142381359249118
164 3.7924616760678598
165 3.7605461524752384
166 3.731087886745309
167 4.156014272602539
168 4.256958171035382
169 3.8075483595119137
170 3.7224969883961117
171 3.8538025773174382
172 4.468919387005501
173 3.9708056475507028
174 3.6764405676629064
175 3.865485812998111
176 3.2381932840236733
177 4.3109393367004625
178 3.84070854535715
179 3.8962591015573995
180 3.895709035880224
181 3.915425264517956
182 3.9089665481223
183 4.08992911568025
184 4.310764539083257
185 4.185574941400778
186 3.549381773516515
187 3.702343034626684
188 3.9676793710887224
189 3.6871518686786975
190 3.8861650798429674
191 3.6516982023067146
192 3.96665364882544
193 3.8707987942742483
194 4.025032293232348
195 3.64333297623619
196 4.005121087856924
197 3.677139051757457
198 3.541561929883151
199 4.121166712469015

should be

Quote:
print {arr3, FNR, arr3/FNR}
468.098 121 3.86858



Is this clearer?

Last edited by chrisjorg; 04-01-2014 at 07:07 PM..
# 20  
Old 04-01-2014
From your first sample in that reply:
Code:
$ awk '{ T+=$2 ; TN++ }; $2<CUTOFF {C=1}; !C { Q+=$2; QN++ } END { print T, TN, T/TN; print Q, QN, Q/QN }' CUTOFF=4.5 data2
1003.86 200 5.01932
535.766 79 6.78185
$

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using awk to assign binary values to data above/below a certain threshold?

Hello, I have files containing a large amount of values in columns, and I want to simplify the data by making the values binary, i.e. assigning 1/0 for each value above or below a certain threshold. I am curious to know if it's possible to use awk to do this (or another way in bash). I've... (2 Replies)
Discussion started by: ksennin
2 Replies

2. Shell Programming and Scripting

Skip first and last n records with awk

Hi, I have an awk code that reads an input file, checks the 4th column and tells if its fine. #!/bin/ksh { if ($4 == 0) print "fine" else print "some problem" }' FILENAME My problem is that, I dont want to check the first 3 and last 3 lines. This can be hard coded by using BEGIN and END... (9 Replies)
Discussion started by: gotam
9 Replies

3. Shell Programming and Scripting

iterate through list of numbers and print specific lines with awk

Could someone please point me in the right direction with the following? I have a program that generates logs that contains sections like this: IMAGE INPUT 81 0 0.995 2449470 0 1726 368 1 0.0635 0.3291 82 0 1.001 2448013 0 1666 365 1 0.0649 ... (4 Replies)
Discussion started by: euval
4 Replies

4. Shell Programming and Scripting

Substituting variable value in AWK /start/,/stop/

Hi all u brilient people on the forum... I am trying to call the variable value in awk command for search pattern /start/,/stop/ but i am nt able to do this .... wat i did is ..i have created two variable YESTERDAY and TODAY and passed the y'day n 2'days dates in it...like this ... (14 Replies)
Discussion started by: whomi
14 Replies

5. Shell Programming and Scripting

Counting records with AWK

I've been working with an awk script and I'm wondeing id it's possible to count records in a file which DO NOT contain, in this instance fields 12 and 13. With the one script I am wanting to display the count for the records WITH fields 12 and 13 and a seperate count of records WITHOUT fields... (2 Replies)
Discussion started by: Glyn_Mo
2 Replies

6. UNIX for Dummies Questions & Answers

Iterate a min/max awk script over time-series temperature data

I'm trying to iterate a UNIX awk script that returns min/max temperature data for each day from a monthly weather data file (01_weath.dat). The temperature data is held in $5. The temps are reported each minute so each day contains 1440 temperature enteries. The below code has gotten me as far as... (5 Replies)
Discussion started by: jgourley
5 Replies

7. UNIX for Advanced & Expert Users

AWK aggregate records

Hy all, I have a problem...can some one help me... I have a file of records sort: 30|239|ORD|447702936929 |blackberry.net |20080728|141304|00000900|2|0000000000000536|28181|0000000006|0000000001|10|1 30|239|ORD|447702936929 |blackberry.net ... (4 Replies)
Discussion started by: anaconga
4 Replies

8. Shell Programming and Scripting

awk - Number of records

Hi, Is it possible to find the total number of records processed by awk at begining. NR gives the value at the end. Is there any variable available to find the value at the begining? Thanks ---------- Suman (1 Reply)
Discussion started by: suman_jakkula
1 Replies

9. Shell Programming and Scripting

Stop awk adding a new line

In a loop, I want to append some text to a file without generating a new line (and then force a new line before re-iterating the loop). In the code below the first 'echo' command is OK as it uses '--n' for no new line. For the 'awk' line I *thought* I could solve it by using printf rather than... (1 Reply)
Discussion started by: TobyR
1 Replies

10. UNIX for Dummies Questions & Answers

awk | stop after specified number of results

I am searching some rather large text files using grep and or awk. What I would like to know is if there is a way (either with grep, awk, or realy any other unix tool) to stop the search when a predifined number of results are returned. I would like to do this for speed purpuses. When i get... (6 Replies)
Discussion started by: evan108
6 Replies
Login or Register to Ask a Question