Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to insert gaussian noise to a data with an order with awk?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to insert gaussian noise to a data with an order with awk?
# 15  
Old 07-23-2012
Right where the arrow is, I noticed i seems to be missing. Did you copy-paste the example?
# 16  
Old 07-23-2012
Quote:
Originally Posted by Scrutinizer
Something like this?
Code:
awk '
  {
    for(i=1;i<=NF;i++)                             
      if($i~/[A-Z]{5}[0-9]$/){                  
        getline p<f                                   
        split(p,N)                                     
        sub($(i+1)+0,sprintf("%.3f",$(i+1)+N[2]),$(i+1))  value
      }
  }
  1                                                   
' f=noise data

noise and data are arbitrary file names



--
if you are using gawk, use gawk --posix
This wil not work with mawk
On Solaris use /usr/xpg4/bin/awk
I do not believe that this solution is correct. This approach depends on whitespace between fields. In the sample data, there are instances where there is no whitespace. This suggests that the data format consists of fixed-width records with fixed width fields. Further, there are some records that do not contain any station names at all.

The format looks like 6 characters for the station immediately followed by 6 characters for the time. If the time is less than 6 characters, it is right-justified and space padded. 6 pairs of station-time per record for a total of 72 characters per record.

Perhaps the OP could elaborate on the data format. Along with my assumptions above, responses to the following questions would be helpful: Do all station fields contain at least one letter? If so, should station fields with a numeric value, along with the paired time field, be skipped?

By the way, Scrutinizer, why the qualifications for gawk and mawk? Something about the sub() regular expression?

Regards,
Alister
Last edited by alister; 07-23-2012 at 11:39 AM..
This User Gave Thanks to alister For This Post:
Scrutinizer
# 17  
Old 07-23-2012
You are right alister, there is no white space everywhere, so the suggestion will indeed not work for cases without whitespace..

mawk and gawk (without --posix or --re-interval) do not support interval expressions..

Perhaps run it through sed 's/....../& /g' first..
Last edited by Scrutinizer; 07-23-2012 at 11:57 AM..
# 18  
Old 07-23-2012
Quote:
Originally Posted by Scrutinizer
mawk and gawk (without --posix or --re-interval) do not support interval expressions..

Perhaps run it through sed 's/....../& /g' first..
My first thought was to just use fold, pipe into awk to do the math, then paste. But it seems that some lines aren't 72 characters long. Specifically, when a station field doesn't contain a letter, it signals a series of numeric values that may not necessarily end on the 72nd character of a line.

Regarding gawk and interval expressions, I was bitten hard by that in the late 90s. Nearly 15 years later, I still vividly recall what the code was supposed to accomplish, the details of the hardware, the hum of the machines in the room, what was hanging on the walls, the scent of the air freshener, and that after what seemed a frustrating eternity, I stumbled across the --re-interval option. Fortunately, it's no longer the case.

From Regexp Operators - The GNU Awk User's Guide

Quote:
Interval expressions were not traditionally available in awk. They were added as part of the POSIX standard to make awk and egrep consistent with each other.

Initially, because old programs may use ‘{’ and ‘}’ in regexp constants, gawk did not match interval expressions in regexps.

However, beginning with version 4.0, gawk does match interval expressions by default. This is because compatibility with POSIX has become more important to most gawk users than compatibility with old programs.
From Gawk 4.0.0 Now Available
Quote:
From: Aharon Robbins
Subject: Gawk 4.0.0 Now Available
Date: Thu, 30 Jun 2011 11:26:43 +0300
Although it's probably worth a heads up since it's only been about a year.

I assume mawk still doesn't support them. Shame.

Regards,
Alister
# 19  
Old 07-24-2012
Hello everybody

Yes there header lines between eveeeery record. Which are starting with date:

Code:
0910 1 2048 32.38 40 20.88  28 35.53  9.62  0.00

This line contains 48 characters.

And the station information is 6 characters also the time information is 6 characters again. This data is an output of f code so the spaces are pre-defined in the code for the station name and the time.

There also lines with

Code:
MOLLP010.640KLYTP311.900TRNMP016.570     0 0.000     0 0.000     0 0.000
     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000

In the first line zeros help the line to be finish till the 72nd character. So the number of the zeros after the last time information is not fixed it depends on the number of the stations recorded for an event.

The second line is a mark between the events. a full line with zeros till the 72 nd character.

Except the full zero lines and header lines all the lines contain station names. A station name can be 4 or 3 character after that information there is another information like P0, S0..... So if the station name is 3 character is becomes :
Code:
ISK P0

Thanks everyone helping me...
# 20  
Old 07-24-2012
Try this (tested in hp-ux)

Code:
$awk 'BEGIN{f1=0;i=1;k=1;while (getline < "file2"){a[i++]=$2}}
{if($1 ~ /^[A-Z]/){l=length($0);for(j=1;j<=l;j=j+6){val=substr($0,j,6);if(val ~ /^[A-Z]/){f1=1;printf val}else{if(f1==1){s=substr($0,j,6);gsub(" ","",s);f1=0;sm=s+a[k];k++;printf "%6.3f",sm}else{printf val}}};printf "\n"}
else{print}}' file1

and output is
Code:
0910 1 2048 32.38 40 20.88  28 35.53  9.62  0.00
MSDMP0 1.804MSDMS1 2.869KCTXP2 4.384KRCMP0 4.506BOZMP0 5.609BOZMS2 9.378
ARMTP2 6.611ARMTS211.582FRR6P0 8.297FRR6S014.046CINRP0 8.842FRR7P0 8.560
FRR7S015.376TRNMP013.088CEKMP016.103ATIMP216.153KRBGP319.275     0 0.000
     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000
0910 1 2328 55.48 40 40.72  29 13.24  8.31  0.00
KLCMP0 3.432KLCMS1 5.981SAKIP0 3.837SAKIS1 6.378ARCEP0 4.149ARCES1 6.837
GBZMP0 4.496GBZMS2 7.617IBBTP0 4.643IBBTS2 7.878BUYMP0 4.610BUYMS2 7.680
GOZTP0 4.860GOZTS2 8.140CINRP0 4.000KRMLP0 5.110KRMLS0 8.020SABAP0 5.290
SABAS2 8.950ESKMP0 5.220ESKMS2 9.040GEMTP2 5.220MOLLP0 6.140MOLLS0 9.680
KURNP0 6.140KURNS210.270FRONP0 5.770ARMTP2 6.320ARMTS210.820CEKMP0 7.320
CEKMS012.720FRR7P0 7.160BOZMP0 7.680BOZMS313.180ISK P2 7.320ISK S212.720
IGDMP0 8.570IGDMS214.410MDNYP2 8.360ADVTP2 8.840FRR6P010.680FRR6S018.000
SILTP210.020KLYTP211.180MSDMP011.360MSDMS219.970BGKTP210.790ELBAP014.820
CTKSP313.910KCTXP315.860KRCMP115.720SNLMP118.520SNLMS232.140SLVTP317.020
TRNMP121.250GONEP324.790KRBGP326.340RKY P327.110LAP P332.180     0 0.000
     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000
0910 2 20 9 50.48 40 34.52  28 54.61 14.09  0.00
ARMTP2 2.140ARMTS2 4.220ESKMP0 2.680ESKMS1 4.400CINRP0 3.180BOZMP0 3.500
BOZMS2 5.570MDNYP2 5.180FRR7P0 4.790FRR7S0 8.640GEMTP2 5.790FRONP0 6.000
FRR6P0 6.610BUYMP0 7.070BUYMS211.790MSDMP0 7.250MSDMS212.140IBBTP0 8.500
IBBTS214.860KLCMP0 7.860IGDMP0 8.220IGDMS213.680SABAP0 9.640SABAS216.520
KURNP010.000KURNS216.820ISK P3 8.820KCTXP310.250KRCMP010.790KRCMS219.360
MOLLP010.640KLYTP311.900TRNMP016.570     0 0.000     0 0.000     0 0.000
     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000

This User Gave Thanks to raj_saini20 For This Post:
miriammiriam
# 21  
Old 07-24-2012
Quote:
Originally Posted by raj_saini20
Try this (tested in hp-ux)

Code:
$awk 'BEGIN{f1=0;i=1;k=1;while (getline < "file2"){a[i++]=$2}}
{if($1 ~ /^[A-Z]/){l=length($0);for(j=1;j<=l;j=j+6){val=substr($0,j,6);if(val ~ /^[A-Z]/){f1=1;printf val}else{if(f1==1){s=substr($0,j,6);gsub(" ","",s);f1=0;sm=s+a[k];k++;printf "%6.3f",sm}else{printf val}}};printf "\n"}
else{print}}' file1

and output is
Code:
0910 1 2048 32.38 40 20.88  28 35.53  9.62  0.00
MSDMP0 1.804MSDMS1 2.869KCTXP2 4.384KRCMP0 4.506BOZMP0 5.609BOZMS2 9.378
ARMTP2 6.611ARMTS211.582FRR6P0 8.297FRR6S014.046CINRP0 8.842FRR7P0 8.560
FRR7S015.376TRNMP013.088CEKMP016.103ATIMP216.153KRBGP319.275     0 0.000
     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000
0910 1 2328 55.48 40 40.72  29 13.24  8.31  0.00
KLCMP0 3.432KLCMS1 5.981SAKIP0 3.837SAKIS1 6.378ARCEP0 4.149ARCES1 6.837
GBZMP0 4.496GBZMS2 7.617IBBTP0 4.643IBBTS2 7.878BUYMP0 4.610BUYMS2 7.680
GOZTP0 4.860GOZTS2 8.140CINRP0 4.000KRMLP0 5.110KRMLS0 8.020SABAP0 5.290
SABAS2 8.950ESKMP0 5.220ESKMS2 9.040GEMTP2 5.220MOLLP0 6.140MOLLS0 9.680
KURNP0 6.140KURNS210.270FRONP0 5.770ARMTP2 6.320ARMTS210.820CEKMP0 7.320
CEKMS012.720FRR7P0 7.160BOZMP0 7.680BOZMS313.180ISK P2 7.320ISK S212.720
IGDMP0 8.570IGDMS214.410MDNYP2 8.360ADVTP2 8.840FRR6P010.680FRR6S018.000
SILTP210.020KLYTP211.180MSDMP011.360MSDMS219.970BGKTP210.790ELBAP014.820
CTKSP313.910KCTXP315.860KRCMP115.720SNLMP118.520SNLMS232.140SLVTP317.020
TRNMP121.250GONEP324.790KRBGP326.340RKY P327.110LAP P332.180     0 0.000
     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000
0910 2 20 9 50.48 40 34.52  28 54.61 14.09  0.00
ARMTP2 2.140ARMTS2 4.220ESKMP0 2.680ESKMS1 4.400CINRP0 3.180BOZMP0 3.500
BOZMS2 5.570MDNYP2 5.180FRR7P0 4.790FRR7S0 8.640GEMTP2 5.790FRONP0 6.000
FRR6P0 6.610BUYMP0 7.070BUYMS211.790MSDMP0 7.250MSDMS212.140IBBTP0 8.500
IBBTS214.860KLCMP0 7.860IGDMP0 8.220IGDMS213.680SABAP0 9.640SABAS216.520
KURNP010.000KURNS216.820ISK P3 8.820KCTXP310.250KRCMP010.790KRCMS219.360
MOLLP010.640KLYTP311.900TRNMP016.570     0 0.000     0 0.000     0 0.000
     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000

Heey it works perfectly and could you please explain the code a little bit it is really compilicated for me to understand Smilie

And Again thank you everyone who tryed to help Smilie

---------- Post updated at 09:10 AM ---------- Previous update was at 04:09 AM ----------

Quote:
Originally Posted by raj_saini20
Try this (tested in hp-ux)

Code:
$awk 'BEGIN{f1=0;i=1;k=1;while (getline < "file2"){a[i++]=$2}}
{if($1 ~ /^[A-Z]/){l=length($0);for(j=1;j<=l;j=j+6){val=substr($0,j,6);if(val ~ /^[A-Z]/){f1=1;printf val}else{if(f1==1){s=substr($0,j,6);gsub(" ","",s);f1=0;sm=s+a[k];k++;printf "%6.3f",sm}else{printf val}}};printf "\n"}
else{print}}' file1

and output is
Code:
0910 1 2048 32.38 40 20.88  28 35.53  9.62  0.00
MSDMP0 1.804MSDMS1 2.869KCTXP2 4.384KRCMP0 4.506BOZMP0 5.609BOZMS2 9.378
ARMTP2 6.611ARMTS211.582FRR6P0 8.297FRR6S014.046CINRP0 8.842FRR7P0 8.560
FRR7S015.376TRNMP013.088CEKMP016.103ATIMP216.153KRBGP319.275     0 0.000
     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000
0910 1 2328 55.48 40 40.72  29 13.24  8.31  0.00
KLCMP0 3.432KLCMS1 5.981SAKIP0 3.837SAKIS1 6.378ARCEP0 4.149ARCES1 6.837
GBZMP0 4.496GBZMS2 7.617IBBTP0 4.643IBBTS2 7.878BUYMP0 4.610BUYMS2 7.680
GOZTP0 4.860GOZTS2 8.140CINRP0 4.000KRMLP0 5.110KRMLS0 8.020SABAP0 5.290
SABAS2 8.950ESKMP0 5.220ESKMS2 9.040GEMTP2 5.220MOLLP0 6.140MOLLS0 9.680
KURNP0 6.140KURNS210.270FRONP0 5.770ARMTP2 6.320ARMTS210.820CEKMP0 7.320
CEKMS012.720FRR7P0 7.160BOZMP0 7.680BOZMS313.180ISK P2 7.320ISK S212.720
IGDMP0 8.570IGDMS214.410MDNYP2 8.360ADVTP2 8.840FRR6P010.680FRR6S018.000
SILTP210.020KLYTP211.180MSDMP011.360MSDMS219.970BGKTP210.790ELBAP014.820
CTKSP313.910KCTXP315.860KRCMP115.720SNLMP118.520SNLMS232.140SLVTP317.020
TRNMP121.250GONEP324.790KRBGP326.340RKY P327.110LAP P332.180     0 0.000
     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000
0910 2 20 9 50.48 40 34.52  28 54.61 14.09  0.00
ARMTP2 2.140ARMTS2 4.220ESKMP0 2.680ESKMS1 4.400CINRP0 3.180BOZMP0 3.500
BOZMS2 5.570MDNYP2 5.180FRR7P0 4.790FRR7S0 8.640GEMTP2 5.790FRONP0 6.000
FRR6P0 6.610BUYMP0 7.070BUYMS211.790MSDMP0 7.250MSDMS212.140IBBTP0 8.500
IBBTS214.860KLCMP0 7.860IGDMP0 8.220IGDMS213.680SABAP0 9.640SABAS216.520
KURNP010.000KURNS216.820ISK P3 8.820KCTXP310.250KRCMP010.790KRCMS219.360
MOLLP010.640KLYTP311.900TRNMP016.570     0 0.000     0 0.000     0 0.000
     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000     0 0.000

I am so sory but we needed to change the format of the input file a little bit I tried to modify your code but could not write full station names

here is the input
Code:
0910 1 2048 32.38 40N20.01  28W36.24  10.85   0.00                   32.00     computed  24-Jul-12 ^@^@^@^@^@^@^@^@
MSDMP0    3.49KCTXP2    6.60KRCMP0    6.63BOZMP0    6.94ARMTP2    8.03FRR6P0    9.30
CINRP0   10.33FRR7P0   10.20TRNMP0   14.25CEKMP0   17.68ATIMP2   18.06KRBGP3   21.01
0
0910 1 2328 55.48 40N41.07  29W13.91  12.06   0.00                   55.35
KLCMP0    4.92SAKIP0    5.03ARCEP0    5.59GBZMP0    5.88IBBTP0    5.30BUYMP0    5.36
GOZTP0    5.94CINRP0    5.50KRMLP0    6.55SABAP0    6.86ESKMP0    6.42GEMTP2    6.44
MOLLP0    7.38KURNP0    7.68FRONP0    6.61ARMTP2    7.71CEKMP0    8.13FRR7P0    7.62
BOZMP0    8.91ISK P2    9.23IGDMP0    9.41MDNYP2    9.64ADVTP2   10.78FRR6P0   10.92
SILTP2   12.62KLYTP2   12.65MSDMP0   13.14BGKTP2   13.06ELBAP0   16.22CTKSP3   16.26
KCTXP3   17.35KRCMP1   17.37SNLMP1   19.64SLVTP3   19.66TRNMP1   22.69GONEP3   26.98
KRBGP3   29.44RKY P3   29.98LAP P3   35.99
0

Only there are,this time, the spaces between station information and the time information increased and there are only 2 digits for the time information the rest is the same



I will be very happy if you tell me how can I use this input for the same process...Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Order of data in Spool File

Hello, I have a shell script through which I am executing .sql file and spooling the result of Query from .sql . I want to spool the result in ascending order. Is there any parameter to be set to print result in ascending or descending order. Thanks in advance. (4 Replies)
Discussion started by: Aparna.N
4 Replies

2. Shell Programming and Scripting

How order a data matrix using awk?

is it possible to order the following row clusters from ascending to descending. thanx in advance input 1 2 4 0 1 2 4 0 3 3 3 3 1 5 1 0 1 5 1 0 6 0 0 0 5 1 1 1... (4 Replies)
Discussion started by: quincyjones
4 Replies

3. Shell Programming and Scripting

Parsing XML (and insert data) then output data (bash / Solaris)

Hi folks I have a script I wrote that basically parses a bunch of config and xml files works out were to add in the new content then spits out the data into a new file. It all works - apart from the xml and config file format in the new file with XML files the original XML (that ends up in... (2 Replies)
Discussion started by: dfinch
2 Replies

4. Shell Programming and Scripting

how to extract data from numbered files using linux in the numerical order-

Hi experts, I have a list of files containing forces as the only number as follows. Force1.txt Force2.txt Force3.txt Force4.txt Force5.txt . . . . . . . . . Force100.txt I want to put all the data(only a number ) in these forces files in the file with the same order like 1,2,3 ..100 .... (2 Replies)
Discussion started by: hamnsan
2 Replies

5. Shell Programming and Scripting

Help with sort data based on descending order problem

Input file 9.99331e-13 8.98451e-65 9.98418e-34 7.98319e-08 365592 111669 74942.9 0 Desired output 365592 111669 74942.9 7.98319e-08 1.99331e-13 6.98418e-34 (2 Replies)
Discussion started by: perl_beginner
2 Replies

6. Shell Programming and Scripting

Generating Gaussian Distributed Random Numbers

I want to generate an awk function that generated a Gaussian distributed set of random numbers. I need to implement the thing below in awk. Rnd is just a uniform random number between 0 and 1 function rgaussian(r1, r2) { Do v1 = 2 * Rnd - 1 v2 = 2 * Rnd - 1 ... (0 Replies)
Discussion started by: kristinu
0 Replies

7. Shell Programming and Scripting

Awk: Data Insert

Hi guys, I'm having some difficulties in insert some details to the following contents. I need to insert "TEST" under MIR & a value "25" to the next line. So far, I am able to insert "TEST" by using awk to capture MIR as the identifier. However, I am having some difficulties in inserting "25"... (3 Replies)
Discussion started by: nantheless
3 Replies

8. Shell Programming and Scripting

How to insert data befor some field in a row of data depending up on values in row

Hi I need to do some thing like "find and insert before that " in a file which contains many records. This will be clear with the following example. The original data record should be some thing like this 60119827 RTMS_LOCATION_CDR INSTANT_POSITION_QUERY 1236574686123083rtmssrv7 ... (8 Replies)
Discussion started by: aemunathan
8 Replies

9. Solaris

Not able to insert data

Hi All, I enhanced a perl script that creates a table with xyz_yyyymm and insert data from xyz table before truncate xyz.I have tested it successfully in dev but when i ran it on production new table xyz_yyyymm created but did not insert any records from xyz.(No errors were thrown)The perl... (1 Reply)
Discussion started by: megh
1 Replies

10. Shell Programming and Scripting

AWK - printing certain fields when field order changes in data file

I'm hoping someone can help me on this. I have a data file that greatly simplified might look like this: sec;src;dst;proto 421;10.10.10.1;10.10.10.2;tcp 426;10.10.10.3;10.10.10.4;udp 442;10.10.10.5;10.10.10.6;tcp sec;src;fac;dst;proto 521;10.10.10.1;ab;10.10.10.2;tcp... (3 Replies)
Discussion started by: eric4
3 Replies
Login or Register to Ask a Question