Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Selecting highest value within a range Post 302740523 by markymarkg123 on Thursday 6th of December 2012 09:50:32 AM
Old 12-06-2012
Selecting highest value within a range

Within millions of lines of data, there are perhaps 20 "spikes" that are very narrow. I want only the highest value from each spike within a range of 10 rows. Possible?

My data looks like this, 8 columns of integers, millions of rows. There are clear spikes when you graph columns.

Code:
2.14883e+05  1.22992e+05  7.96926e+03  -1.37694e+03  3.95054e+03  -1.62924e+04  8.21638e+03  1.01061e+04  
2.14357e+05  1.22730e+05  8.20939e+03  -1.54033e+03  4.28164e+03  -1.61322e+04  7.97054e+03  1.01922e+04  
2.13361e+05  1.22889e+05  8.05019e+03  -1.18045e+03  4.02582e+03  -1.61925e+04  7.99161e+03  1.02380e+04  
2.68777e+05  1.17178e+05  5.12913e+04  -1.40305e+04  2.95355e+04  -2.65120e+04  2.14739e+04  9.34042e+03  
4.09316e+05  1.80414e+05  5.32998e+04  -1.06297e+04  3.04299e+04  -2.75763e+04  2.09896e+04  8.12131e+03  
4.94380e+05  2.46756e+05  1.36658e+04  6.56373e+03  6.79386e+03  -1.70254e+04  7.70163e+03  9.14013e+03  
4.92551e+05  2.48154e+05  1.39390e+04  6.94251e+03  6.73128e+03  -1.65537e+04  7.33397e+03  9.21148e+03  
4.91403e+05  2.48659e+05  1.49110e+04  6.85990e+03  7.53969e+03  -1.67406e+04  7.14302e+03  9.71328e+03  
4.89192e+05  2.52866e+05  1.39238e+04  8.07381e+03  7.38389e+03  -1.64431e+04  6.44513e+03  1.00028e+04  
4.90260e+05  2.47635e+05  1.45364e+04  6.72570e+03  7.24393e+03  -1.69678e+04  6.90001e+03  1.01961e+04

Currently, my code will give me all the consecutive-row increases in column 1 over a specified value of 5000:

Code:
awk '(NR>1) && (d=$1-x)>=5000 {print "increase of" " " d, "at Line" " " 'NR' " "}{x=$1}' test | sort -n

The output looks like this:
increase of 140539 at Line 5
increase of 55416 at Line 4
increase of 85064 at Line 6

However, I want the output to look like this, filtering out the smaller increases within that data range:
increase of 140539 at Line 5
increase of 5662 at Line 48924
increase of 10334 at Line 589332
...
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

sort with highest wc

Hi :) I'm a unix beginner and i've recently got an assignment to write up a script to print the most common IP address that made requests from a webserver. I'm really lost in this one...and if someone could pls tell me where to start i'll be really greatful ! thanx (1 Reply)
Discussion started by: ymf
1 Replies

2. Shell Programming and Scripting

Selecting files between a user inputed date range

Hi all! I'm working on a KSH script to select files between a user inputed date range (stored in a variable) and then move them and unzip them. I'm stuck at how to get the files between the user inputed date range selected. Any help would be greatly appreciated! The files are as such: ... (6 Replies)
Discussion started by: kelldan
6 Replies

3. UNIX for Dummies Questions & Answers

Printing highest value from one column

Hi, I have a file that looks like this: s6 98 s6 91 s6 56 s5 32 s5 10 s5 4 So what I want to do is print only the highest value for each value in the column: So the file will look like this: s6 98 s5 32 Thanks (4 Replies)
Discussion started by: phil_heath
4 Replies

4. Shell Programming and Scripting

print range between two patterns if it contains a pattern within the range

I want to print between the range two patterns if a particular pattern is present in between the two patterns. I am new to Unix. Any help would be greatly appreciated. e.g. Pattern1 Bombay Calcutta Delhi Pattern2 Pattern1 Patna Madras Gwalior Delhi Pattern2 Pattern1... (2 Replies)
Discussion started by: joyan321
2 Replies

5. Shell Programming and Scripting

Extract the highest number out

Hi Gurus, I've using HPUX B.11.23 U ia64 with shell = sh. I've been having some problem get the highest number of this script. Actually I wanted to get the highest number from this listing (TEST123 data and based on this highest number, there will be email being sent out. For example,... (6 Replies)
Discussion started by: superHonda123
6 Replies

6. Shell Programming and Scripting

Selecting a range of Lines

Hi All, Is there a way to get a range of lines from a file??? I want to search through a set of scripts and need to select the group of lines which do the FTP. Say, Line1 Line2 ftp SERVER user UNAME PASS send FILE_TO_BE_SENT close Line3 Line4 Line5 ftp SERVER1 user USER1 PASS1... (6 Replies)
Discussion started by: beinthemiddle
6 Replies

7. Shell Programming and Scripting

Finding the highest value(in negative)

Hi all, I have a simple problem. I have given an example of the problem below. There are 4 space-delimited columns. 2655 96 IA -0.8179 2655 96 IA -0.9144 2655 96 CPU -0.4275 2655 96 RMA -0.3407 2655 96 IA -0.9373 2655 96 ... (2 Replies)
Discussion started by: jaysean
2 Replies

8. Shell Programming and Scripting

Printing highest value from a list

Hi all, I'm trying to get the item with the maximum value, and was wondering if someone can help me with it. Heres my input file: apples 15 books 15 books 17 pens 12 pens 15 umbrella 13Here's what my output file should look like: apples 15 books 17 pens 15 umbrella 13 Can... (2 Replies)
Discussion started by: r4v3n
2 Replies

9. Shell Programming and Scripting

Selecting lowest and highest values in columns 1 and 2, based on subsets in column 3

Hi, I have a file with the following columns: 361459 447394 CHL1 290282 290282 CHL1 361459 447394 CHL1 361459 447394 CHL1 178352861 178363529 AGA 178352861 178363529 AGA 178363657 178363657 AGA Essentially, using CHL1 as an example. For any line that has CHL1 in... (2 Replies)
Discussion started by: hubleo
2 Replies

10. Shell Programming and Scripting

Highest value matrix parsing

Hi All I do have a matrix in the following format a_2 a_3 s_4 t_6 b 0 0.9 0.004 0 c 0 0 1 0 d 0 0.98 0 0 e 0.0023 0.96 0 0.0034 I have thousands of rows I would like to parse the maximum value in each of the row and out put that highest value along the column header of... (2 Replies)
Discussion started by: Kanja
2 Replies
MHPATH(1)                                                            [nmh-1.5]                                                           MHPATH(1)

NAME
mhpath - print full pathnames of nmh messages and folders SYNOPSIS
mhpath [+folder] [msgs] [-version] [-help] DESCRIPTION
mhpath expands and sorts the message list `msgs' and writes the full pathnames of the messages to the standard output separated by new- lines. If no `msgs' are specified, mhpath outputs the current mail folder's pathname instead. If the only argument is `+', your nmh "Path" is output; this can be useful in shell scripts. Contrasted with other nmh commands, a message argument to mhpath may often be intended for writing. Because of this: 1) the name "new" has been added to mhpath's list of reserved message names (the others are "first", "last", "prev", "next", "cur", and "all"). The new message is equivalent to the message after the last message in a folder (and equivalent to 1 in a folder without mes- sages). The "new" message may not be used as part of a message range. 2) Within a message list, the following designations may refer to messages that do not exist: a single numeric message name, the single message name "cur", and (obviously) the single message name "new". All other message designations must refer to at least one existing message, if the folder contains messages. 3) An empty folder is not in itself an error. A message number less than that of the smallest existing message in a folder is treated as if the message already exists. A message number greater than that of the highest existing message in a folder causes an "out of range" error message to be displayed. As part of a range designation that contains messages that do exist, message numbers less than the smallest, or greater than the highest, existing message in a folder are ignored. Examples: The current folder foo contains messages 3 5 6. Cur is 4. % mhpath /r/phyl/Mail/foo % mhpath all /r/phyl/Mail/foo/3 /r/phyl/Mail/foo/5 /r/phyl/Mail/foo/6 % mhpath 2001 mhpath: message 2001 out of range 1-6 % mhpath 1-2001 /r/phyl/Mail/foo/3 /r/phyl/Mail/foo/5 /r/phyl/Mail/foo/6 % mhpath new /r/phyl/Mail/foo/7 % mhpath last new /r/phyl/Mail/foo/6 /r/phyl/Mail/foo/7 % mhpath last-new mhpath: bad message list last-new % mhpath cur /r/phyl/Mail/foo/4 % mhpath 1-2 mhpath: no messages in range 1-2 % mhpath first:2 /r/phyl/Mail/foo/3 /r/phyl/Mail/foo/5 % mhpath 1 2 /r/phyl/Mail/foo/1 /r/phyl/Mail/foo/2 mhpath is also useful in back-quoted operations: % cd `mhpath +inbox` % echo `mhpath +` /r/phyl/Mail FILES
$HOME/.mh_profile The user profile PROFILE COMPONENTS
Path: To determine the user's nmh directory Current-Folder: To find the default current folder SEE ALSO
folder(1) DEFAULTS
`+folder' defaults to the current folder `msgs' defaults to none CONTEXT
None BUGS
Like all nmh commands, mhpath expands and sorts [msgs]. So don't expect mv `mhpath 501 500` to move 501 to 500. Quite the reverse. But mv `mhpath 501` `mhpath 500` will do the trick. Out of range message 0 is treated far more severely than large out of range message numbers. MH.6.8 11 June 2012 MHPATH(1)
All times are GMT -4. The time now is 02:34 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy