Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Replace outliers in bash, with one-line command?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Replace outliers in bash, with one-line command?
# 1  
Old 01-07-2019
Replace outliers in bash, with one-line command?
I need to replace outliers from time series, an example below (outlier at x-position 386). The problem is that I do not want to smooth this line, I would like to replace the outlier with, for instance, an average value from the two consecutive values (i.e., average value from positions 385 and 387). Could someone suggest a one-line command using linux tools (awk) for this please? Any hints are appreciated.

Code:
Y X
0.204416 200
0.20432 201
0.204368 202
0.204416 203
0.204416 204
0.204368 205
0.204368 206
0.204464 207
0.204464 208
0.204512 209
0.204416 210
0.204416 211
0.204416 212
0.204464 213
0.20456 214
0.204512 215
0.204512 216
0.204608 217
0.204608 218
0.204608 219
0.204512 220
0.204464 221
0.204368 222
0.20432 223
0.204176 224
0.204224 225
0.204224 226
0.204272 227
0.20432 228
0.204272 229
0.204272 230
0.204464 231
0.204512 232
0.204416 233
0.204272 234
0.204224 235
0.204224 236
0.204272 237
0.204224 238
0.204224 239
0.204272 240
0.20432 241
0.20432 242
0.204368 243
0.204368 244
0.204368 245
0.20432 246
0.204368 247
0.204368 248
0.204272 249
0.204272 250
0.204272 251
0.204272 252
0.204272 253
0.20432 254
0.204272 255
0.204224 256
0.204224 257
0.204272 258
0.204272 259
0.204272 260
0.204224 261
0.204224 262
0.204272 263
0.204176 264
0.204128 265
0.20408 266
0.20408 267
0.20408 268
0.204128 269
0.204032 270
0.20408 271
0.204128 272
0.204176 273
0.204272 274
0.204176 275
0.20408 276
0.204128 277
0.204176 278
0.204128 279
0.204128 280
0.204176 281
0.204176 282
0.20408 283
0.204032 284
0.203936 285
0.204032 286
0.203984 287
0.204032 288
0.203984 289
0.203936 290
0.203936 291
0.203888 292
0.203936 293
0.20384 294
0.20384 295
0.203888 296
0.203936 297
0.203984 298
0.203936 299
0.203984 300
0.204032 301
0.20408 302
0.203984 303
0.203888 304
0.20384 305
0.20384 306
0.20384 307
0.203744 308
0.203744 309
0.203792 310
0.20384 311
0.203888 312
0.20384 313
0.20384 314
0.20384 315
0.203888 316
0.203936 317
0.203984 318
0.203936 319
0.203936 320
0.203984 321
0.203936 322
0.203936 323
0.203984 324
0.203888 325
0.20384 326
0.203888 327
0.20384 328
0.20384 329
0.20384 330
0.203792 331
0.203696 332
0.203696 333
0.203648 334
0.203696 335
0.203792 336
0.20384 337
0.203792 338
0.204128 339
0.20408 340
0.204032 341
0.203552 342
0.2036 343
0.2036 344
0.2036 345
0.203648 346
0.203648 347
0.203696 348
0.203648 349
0.2036 350
0.203552 351
0.203552 352
0.2036 353
0.203648 354
0.203696 355
0.203744 356
0.203744 357
0.203744 358
0.203744 359
0.203696 360
0.203696 361
0.203744 362
0.203792 363
0.203744 364
0.203792 365
0.203792 366
0.203792 367
0.203792 368
0.203744 369
0.203648 370
0.203504 371
0.203552 372
0.203552 373
0.203504 374
0.203552 375
0.203504 376
0.203504 377
0.203552 378
0.203504 379
0.203504 380
0.203504 381
0.203552 382
0.203936 383
0.203888 384
0.203888 385
0.10208 386
0.203456 387
0.203456 388
0.203504 389
0.203504 390
0.203552 391
0.203648 392
0.203648 393
0.203648 394
0.203648 395
0.203552 396
0.203552 397
0.203504 398
0.203504 399
0.2036 400
0.203696 401
0.203696 402
0.203552 403
0.203504 404
0.203552 405
0.203504 406
0.203456 407
0.203408 408
0.203408 409
0.203456 410
0.203504 411
0.203552 412
0.203552 413
0.203552 414
0.203552 415
0.2036 416
0.203552 417
0.203456 418
0.203456 419
0.203456 420
0.203456 421
0.203504 422
0.203504 423
0.203456 424
0.203456 425
0.203504 426
0.203456 427
0.203408 428
0.203408 429
0.203264 430
0.203216 431
0.203216 432
0.203312 433
0.203408 434
0.203408 435
0.203408 436
0.20336 437
0.203408 438
0.20336 439
0.20336 440
0.203312 441
0.203264 442
0.203216 443
0.203264 444
0.203312 445
0.203264 446
0.203168 447
0.203168 448
0.203168 449
0.203216 450
0.203264 451
0.203264 452
0.203312 453
0.203264 454
0.203264 455
0.203216 456
0.20312 457
0.203072 458
0.20312 459
0.203264 460
0.203312 461
0.203264 462
0.203216 463
0.203168 464
0.20312 465
0.203024 466
0.202976 467
0.202976 468
0.203024 469
0.203072 470
0.203072 471
0.203072 472
0.203168 473
0.203216 474
0.203216 475
0.203216 476
0.203216 477
0.203168 478
0.203168 479
0.20312 480
0.203072 481
0.203072 482
0.203024 483
0.202976 484
0.203024 485
0.203072 486
0.203216 487
0.20312 488
0.203072 489
0.202976 490
0.202976 491
0.203024 492
0.203024 493
0.203024 494
0.203024 495
0.203072 496
0.203024 497
0.203024 498
0.202976 499

Replace outliers in bash, with one-line command?-ppng
fb.txt (3.8 KB) 
# 2  
Old 01-07-2019
Quote:
Originally Posted by Gery
I need to replace outliers from time series, an example below (outlier at x-position 386). The problem is that I do not want to smooth this line, I would like to replace the outlier with, for instance, an average value from the two consecutive values (i.e., average value from positions 385 and 387). Could someone suggest a one-line command using linux tools (awk) for this please? Any hints are appreciated.
OK, i can more or less see where your problem is but you need to define your goal a bit more precisely:

Define "outlier" - i can see that line 386 is far from the average but i suppose this is just a sample from a larger dataset. The values, as far as i can see, are neither falling nor climbing monotonely so an algorithm like "if X is not between X-1 and X+1 then correct" is not possible. How far from the average (or median? or weighted average? or fixed value? or something else entirely?) is an "outlier"?

I hope this helps.

bakunin
This User Gave Thanks to bakunin For This Post:
wisecracker
# 3  
Old 01-08-2019
Thanks Bakunin for the reply, I think the best way to define outlier here is something higher than the average, you are right to point out that this is only a sample from a larger data set, actually not all time series are similar. In this case this a "flat" time series, so outliers is anything that goes beyond the average. The issue would be when the trend would have an exponential shape, for instance.

--- Post updated 01-08-19 at 04:19 PM ---

This works:

Code:
awk 'NR==1; 
       NR>3 {print pp,ppi} 
            {if((pp-p)^2>0.01 && (p-$1)^2>0.01) p=(pp+$1)/2; 
             pp=p; p=$1; ppi=pi; pi=$2} 
       END  {print pp,ppi; print p,pi}' file

Last edited by Gery; 01-08-2019 at 05:26 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to run several bash commands put in bash command line?

How to run several bash commands put in bash command line without needing and requiring a script file. Because I'm actually a windows guy and new here so for illustration is sort of : $ bash "echo ${PATH} & echo have a nice day!" will do output, for example:... (4 Replies)
Discussion started by: abdulbadii
4 Replies

2. UNIX for Advanced & Expert Users

BASH Internal : Replace pattern with string without external command

Morning, I'm trying step up my scripting game .. :rolleyes::confused::D Is there a way to do the replacement with an or without using an external command ? I did try but no joy. var=${var//\(|\)/} #!/bin/bash var="lulus.UbiRwidgets.com (10.1.1.1)" var=${var//\(/}... (5 Replies)
Discussion started by: popeye
5 Replies

3. Shell Programming and Scripting

sed command to replace a line in a file using line number from the output of a pipe.

Sed command to replace a line in a file using line number from the output of a pipe. Is it possible to replace a whole line piped from someother command into a file at paritcular line... here is some basic execution flow.. the line number is 412 lineNo=412 Now i have a line... (1 Reply)
Discussion started by: vivek d r
1 Replies

4. Shell Programming and Scripting

sed command to replace a line at a specific line number with some other line

my requirement is, consider a file output cat output blah sdjfhjkd jsdfhjksdh sdfs 23423 sdfsdf sdf"sdfsdf"sdfsdf"""""dsf hellow there this doesnt look good et cetc etc etcetera i want to replace a line of line number 4 ("this doesnt look good") with some other line ... (3 Replies)
Discussion started by: vivek d r
3 Replies

5. Shell Programming and Scripting

how to replace html line into a command line?

hi! i'm trying to use sed for this, but i'm struggling a lot. i want to convert <div class="middle" id="middle">Friday, 20 April 2012<br /> <span class="hex">728CB5</span> <br /></div> into fbsetroot -solid '#728CB5' considering all information between 'id="middle">' and '<br />... (2 Replies)
Discussion started by: nitrofurano
2 Replies

6. UNIX for Dummies Questions & Answers

Bash: using SED, trying to replace some characters except first or last line

Hi, I require to replace 2 items: 1. replace start of all lines in a file with ' except the first line 2. replace end of all lines in a file with '||chr( except last line I am able to do the entire file using sed -e s/^/\'/g -e s/$/\'\|\|chr\(/g "$file" > newfile.txt but am not yet... (3 Replies)
Discussion started by: Chella15
3 Replies

7. Shell Programming and Scripting

Bash: using SED, trying to replace some characters except first or last line

Hi, I require to replace 2 items: 1. replace start of all lines in a file with ' except the first line 2. replace end of all lines in a file with '||chr( except last line I am able to do the entire file using sed -e s/^/\'/g -e s/$/\'\|\|chr\(/g "$file" > newfile.txt but am not yet able... (0 Replies)
Discussion started by: Chella15
0 Replies

8. Shell Programming and Scripting

Bash - Loading a command's output line by line into an array

I have been trying this a lot of different ways and haven't found too much online. Here's what I've got so far: j=0 declare -a first zero=(`cat $tmpfile`) for i in "${zero}" do command $i >> "${first}" ... (4 Replies)
Discussion started by: Azrael
4 Replies

9. UNIX for Dummies Questions & Answers

Running set options from the command line and bash command

I'm reading about debugging aids in bash and have come across the set command. It says in my little book that an addition to typing set you can also use them "on the command line when running a script..." and it lists this in a small table: set -o option Command Line... (5 Replies)
Discussion started by: Straitsfan
5 Replies
Login or Register to Ask a Question