Reoccuring peak values in large data file and print the line..


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Reoccuring peak values in large data file and print the line..
# 1  
Old 10-16-2015
Reoccuring peak values in large data file and print the line..

Hi i have some large data files that contain several fields and rows the data in a field have a numeric value that is in a sine wave pattern what i would like todo is locate each peak and pick the highest value and print that complete line. the data looks something like this it is field nr4 which contains the sine wave pattern it varies from -10... to -600. ..
Code:
139,080 8,398 2,608 -593,820
139,100 8,398 2,608 -594,540
139,120 8,398 2,608 -595,140
139,140 8,398 2,608 -595,740
139,160 8,398 2,608 -596,280
139,180 8,398 2,608 -596,760
139,200 8,398 2,608 -597,240
139,220 8,398 2,608 -597,660
139,240 8,398 2,608 -598,020
139,260 8,398 2,608 -598,320
139,280 8,398 2,608 -598,620
139,300 8,398 2,608 -598,800
139,320 8,398 2,608 -599,040
139,340 8,398 2,608 -599,160
139,360 8,398 2,608 -599,220
139,380 8,398 2,608 -599,280
139,400 8,398 2,608 -599,280
139,420 8,398 2,608 -599,280
139,440 8,398 2,608 -599,160
139,460 8,398 2,608 -599,040
139,480 8,398 2,608 -598,920
139,500 8,398 2,608 -598,680
139,520 8,398 2,608 -598,440
139,540 8,398 2,608 -598,140
139,560 8,398 2,608 -597,780
139,580 8,398 2,608 -597,300
139,600 8,398 2,608 -596,820
139,620 8,398 2,608 -596,280
139,640 8,398 2,608 -595,620
139,660 8,398 2,608 -594,960
139,680 8,398 2,608 -594,300
.........
149,160 8,398 2,608 -597,000
149,180 8,398 2,608 -597,540
149,200 8,398 2,608 -597,960
149,220 8,398 2,608 -598,380
149,240 8,398 2,608 -598,740
149,260 8,398 2,608 -599,100
149,280 8,398 2,608 -599,340
149,300 8,398 2,608 -599,580
149,320 8,398 2,608 -599,760
149,340 8,398 2,608 -599,940
149,360 8,398 2,608 -600,000
149,380 8,398 2,608 -600,060
149,400 8,398 2,608 -600,060
149,420 8,398 2,608 -600,000
149,440 8,398 2,608 -599,940
149,460 8,398 2,608 -599,820
149,480 8,398 2,608 -599,640
149,500 8,398 2,608 -599,460
149,520 8,398 2,608 -599,220
149,540 8,398 2,608 -598,860
149,560 8,398 2,608 -598,500
149,580 8,398 2,608 -598,080
149,600 8,398 2,608 -597,540
149,620 8,398 2,608 -597,000
149,640 8,398 2,608 -596,400
149,660 8,398 2,608 -595,740

it would really help if someone can me give some hints..

Last edited by ninjaunx; 10-16-2015 at 05:55 AM..
# 2  
Old 10-16-2015
Any attempt from your side?

---------- Post updated at 11:19 ---------- Previous update was at 11:17 ----------

And, are you looking for
- the highest or lowest value?
- a relative or absolute extremum?
- noise involved?
# 3  
Old 10-16-2015
not any good ones Smilie
the noise is not any problem as i intend to use it. what i try todo is file which for each row contains the peak value and the other fields if the peak value is exactly true is not important the only thing is that it can have two peaks from the same "real" peak.
# 4  
Old 10-16-2015
Please post the not-so-good-ones.

---------- Post updated at 12:34 ---------- Previous update was at 11:45 ----------

Howsoever, try
Code:
awk '
function sign(X)        {return X>0?+1:X==0?0:-1
                        }
                {TMP=$4
                 sub(/,/, ".", TMP)
                 DELTA = sign (TMP-LastVal)
                }
DELTA != TREND  {print LastLine
                }
                {LastLine = $0
                 LastVal = TMP
                 TREND = DELTA
                }
' file
139,380 8,398 2,608 -599,280
139,420 8,398 2,608 -599,280
149,380 8,398 2,608 -600,060
149,400 8,398 2,608 -600,060

# 5  
Old 10-16-2015
i did some sort and grep arguments which feels like it almost works...
in your case i get some extra lines i get two peaks that are to close to each other.. the peaks must be at least 1s(the first column is timestamp) couple of seconds apart from each other... but if i do sort uniq on the first column and ignoring the decimal part i get almost to the real thing...also i need to add threshold

Quote:
Originally Posted by RudiC
Please post the not-so-good-ones.

---------- Post updated at 12:34 ---------- Previous update was at 11:45 ----------

Howsoever, try
Code:
awk '
function sign(X)        {return X>0?+1:X==0?0:-1
                        }
                {TMP=$4
                 sub(/,/, ".", TMP)
                 DELTA = sign (TMP-LastVal)
                }
DELTA != TREND  {print LastLine
                }
                {LastLine = $0
                 LastVal = TMP
                 TREND = DELTA
                }
' file
139,380 8,398 2,608 -599,280
139,420 8,398 2,608 -599,280
149,380 8,398 2,608 -600,060
149,400 8,398 2,608 -600,060

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Perl to extract values and print at end of each line

In the below perl I am trying to extract and print the values AF1=, the GT value, and F or QUAL diveded by 33 (rounded to the nearest whole #). The GT value is at the end after the GT:PL so all the possibilities are read into a hash h, then depending on the value that is in the line the... (1 Reply)
Discussion started by: cmccabe
1 Replies

2. Shell Programming and Scripting

Print line if values in fields matches number and text

datafile: 2017-03-24 10:26:22.098566|5|'No Route for Sndr:RETEK RMS 00040 /ZZ Appl:PF Func:PD Txn:832 Group Cntr:None ISA CntlNr:None Ver:003050 '|'2'|'PFI'|'-'|'EAI_ED_DeleteAll'|'EAI_ED'|NULL|NULL|NULL|139050594|ActivityLog| 2017-03-27 02:50:02.028706|5|'No Route for... (7 Replies)
Discussion started by: SkySmart
7 Replies

3. Shell Programming and Scripting

awk to print line is values between two fields in separate file

I am trying to use awk to find all the $3 values in file2 that are between $2 and $3 in file1. If a value in $3 of file2 is between the file1 fields then it is printed along with the $6 value in file1. Both file1 and file2 are tab-delimited as well as the desired output. If there is nothing to... (4 Replies)
Discussion started by: cmccabe
4 Replies

4. Shell Programming and Scripting

Reading off values from a large file

Hi, I have a large output file (star.log), with many lines of the following type *** T vavg unburnt: 723.187 / burnt: 2662.000 What I would like to do is pick the values 723.187 and 2662.000 and What I've got so far is awk '/unburnt:.*burnt:/{Tu=$6;Tb=$NF}END{print Tu, Tb}'... (6 Replies)
Discussion started by: lost.identity
6 Replies

5. UNIX for Dummies Questions & Answers

Large file data handling issue

I have a single record large file, semicolon ';' and pipe '|' separated. I am doing a vi on the file. It is throwing an error "File to long" I need to actually remove the last | symbol from this file. sed -e 's/\|*$//' filename is working fine for small files. But not working on this big... (13 Replies)
Discussion started by: Gurkamal83
13 Replies

6. Shell Programming and Scripting

Compare values in two files. For matching rows print corresponding values from File 1 in File2.

- I have two files (File 1 and File 2) and the contents of the files are mentioned below. - I am trying to compare the values of Column1 of File1 with Column1 of File2. If a match is found, print the corresponding value from Column2 of File1 in Column5 of File2. - I tried to modify and use... (10 Replies)
Discussion started by: Santoshbn
10 Replies

7. UNIX for Dummies Questions & Answers

[Solved] Print a line using a max and a min values of different columns

Hi guys, I already search on the forum but i can't solve this on my own. I have a lot of files like this: And i need to print the line with the maximum value in last column but if the value is the same (2 in this exemple for the 3 last lines) i need get the line with the minimum value in... (4 Replies)
Discussion started by: MetaBolic0
4 Replies

8. Shell Programming and Scripting

How to remove a subset of data from a large dataset based on values on one line

Hello. I was wondering if anyone could help. I have a file containing a large table in the format: marker1 marker2 marker3 marker4 position1 position2 position3 position4 genotype1 genotype2 genotype3 genotype4 with marker being a name, position a numeric... (2 Replies)
Discussion started by: davegen
2 Replies

9. Shell Programming and Scripting

Find line number of bad data in large file

Hi Forum. I was trying to search the following scenario on the forum but was not able to. Let's say that I have a very large file that has some bad data in it (for ex: 0.0015 in the 12th column) and I would like to find the line number and remove that particular line. What's the easiest... (3 Replies)
Discussion started by: pchang
3 Replies

10. Shell Programming and Scripting

To Delete the repeated occurances and print in same line by appending values

Hi All, I am trying to work on below text a b c 1 a b c 2 a b c 3 x y z 6 x y z 44 a b c 89 Need to delete the occurances and get in single line like below: a b c 1 2 3 89 x y z 6 44 89 Please help me i am new into unix scripting ..... ---------- Post updated at 03:00... (8 Replies)
Discussion started by: shaliniyadav
8 Replies
Login or Register to Ask a Question