Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Selection of records y time and offset.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Selection of records y time and offset.
# 8  
Old 02-02-2017
allowedsplittime() matches with the slope from the graphic and with the values in your array. Notice how the slope of the allowed time changes from 3 sec per km to 1 sec per km at the 6 km point.

Below I run some of the distance values from analysis_25.csv thru this function and you can see the result matches with the contents of the Allowed_SlipTime column:

Code:
   Distance          Result
 11698.3675          0.3016
  9531.0848          2.4689
  6661.5539          5.3384
  5277.8263          8.1665
  3600.4959         13.1985

This User Gave Thanks to Chubler_XL For This Post:
jiam912
# 9  
Old 02-02-2017
Dear Xl

I have test the code and works perfect.

There is only one think to change.

the
Code:
 stime

should be in sec , then i was trying to change it like this.

# Skip lookahead when diff is greater than 25sec

Code:
	    stime1 = (dtime[2] - dtime[1])
	    	        
	    if((((dtime[2] - dtime[1])-100)+60) > 25) next

         	       	    
	    if(stime1>=40)stime = (((dtime[2] - dtime[1])-100)+60)
	    else stime = (dtime[2] - dtime[1])

kindly, can you check it. because i try it but does not work.

Everything else is working perfect.

Thanks for your help
# 10  
Old 02-02-2017
We need to add a new function to convert jdtime to seconds; changes highlighted in red below:

Code:
awk -F, 'FNR>1 {
    gsub(/"/, "")
    line=substr($10,1,5)
    point=substr($11,1,5)
    ffid=$2
    sw=$5
    tape=$6;
    x=$49;
    y=$50;
    tb=$46
    printf("%4d %5d %5d %4d %4d %10.1f %10.1f %s\n",
      tape,line,point,ffid,sw,x,y,tb)}' 25.csv |
sort -k8n | awk '{v=substr($8,16,1); gsub(/2/,"0",v);print substr($0,1,64) v}' > toto
awk '
   BEGIN {
       split("analysis.csv bad_records.csv", OUT)
       for(j=1;j<3;j++) {
          printf "Tape1,LineName1,PointNumber1,FFID1,X1,Y1,JD1,HHMMSS1," > OUT[j]
          printf "Tape2,LineName2,PointNumber2,FFID2,X2,Y2,JD2,HHMMSS2," > OUT[j]
          printf "Offset,SlipTime,Allowed_SlipTime\n" > OUT[j]
       }
    }
    function gpstojdtime(gpstime) {
        gpss = (gpstime/1000000)+10800;
        jtmp = (gpss-((2017-1980)*365+10-6)*86400-18)/86400;
        jday = int(jtmp);
        htmp = 24*(jtmp - jday);
        hour = int(htmp);
        mtmp = 60*((htmp - hour));
        minu = int(mtmp);
        stmp = 60*(mtmp - minu);
        secd = (stmp);
        return hour*10000+minu*100+secd;
    }

    function jdtimetosec(value) {
       hour = int(value/10000)
       minu = int((value%10000)/100)
       secd = value%100

       return 3600*hour + 60*minu + secd
    }

    function allowedsliptime(distance) {
        if(distance < 2000) return 18
        if(distance > 12000) return 0
        if(distance < 6000) return 24 - distance / 333.333
        return 12 - distance / 1000
    }

    function calcoffset(x,y) {
        return sqrt((x[1]-x[2])^2+(y[1]-y[2])^2)
    }

    FNR==NR { L[FNR]=$0 ; next}
    {
        tape[1]=$1
        splin[1]=$2
        spnum[1]=$3
        sprec[1]=$4
        sw[1]=$5
        x[1]=$6
        y[1]=$7
        spgps[1]=$8
        dtime[1]=gpstojdtime(spgps[1])
        day[1]=jday

        # Look up to 10 records ahead
        for(i=1; i< 10 && (i+FNR in L); i++) {
            split(L[i+FNR],V)
            tape[2]=V[1]
            splin[2]=V[2]
            spnum[2]=V[3]
            sprec[2]=V[4]
            sw[2]=V[5]
            x[2]=V[6]
            y[2]=V[7]
            spgps[2]=V[8]
            dtime[2]=gpstojdtime(spgps[2])
            day[2]=jday

            offset = calcoffset(x,y)
            stime = jdtimetosec(dtime[2]) - jdtimetosec(dtime[1])
            allow = allowedsliptime(offset)

            # Skip lookahead when diff is greater than 25sec
            if(stime > 25) next

            # decide on output file based on splittime vrs allowedsplit
            if(stime < allow) of=OUT[2]
            else of=OUT[1]

            for(j=1;j<3;j++)
                printf"%d,%d,%d,%d,%.1f,%.1f,%d,%.2f,",
                    tape[j],splin[j],spnum[j],sprec[j],
                    x[j],y[j],day[j], dtime[j]  > of

            printf "%.4f,%.3f,%5.4f\n", offset, stime, allow > of
        }
    }' toto toto

This User Gave Thanks to Chubler_XL For This Post:
jiam912
# 11  
Old 02-03-2017
Dear Xl

Great, really wonderful job. Appreciate your help.. It works perfect.

Please the last thing here..

it is possible to add other output here:
Code:
split("analysis.csv bad_records.csv all.csv", OUT)

Code:
of=OUT[3]

in order to have complete data..
I can concatenate both outputs and get complete one, but will be great to add this in the code.

Thanks Again...
# 12  
Old 02-03-2017
Try this:

Code:
awk -F, 'FNR>1 {
    gsub(/"/, "")
    line=substr($10,1,5)
    point=substr($11,1,5)
    ffid=$2
    sw=$5
    tape=$6;
    x=$49;
    y=$50;
    tb=$46
    printf("%4d %5d %5d %4d %4d %10.1f %10.1f %s\n",
      tape,line,point,ffid,sw,x,y,tb)}' 25.csv |
sort -k8n | awk '{v=substr($8,16,1); gsub(/2/,"0",v);print substr($0,1,64) v}' > toto
awk '
   BEGIN {
       split("analysis.csv bad_records.csv all.csv", OUT)
       for(j=1;j<=3;j++) {
          printf "Tape1,LineName1,PointNumber1,FFID1,X1,Y1,JD1,HHMMSS1," > OUT[j]
          printf "Tape2,LineName2,PointNumber2,FFID2,X2,Y2,JD2,HHMMSS2," > OUT[j]
          printf "Offset,SlipTime,Allowed_SlipTime\n" > OUT[j]
       }
    }
    function gpstojdtime(gpstime) {
        gpss = (gpstime/1000000)+10800;
        jtmp = (gpss-((2017-1980)*365+10-6)*86400-18)/86400;
        jday = int(jtmp);
        htmp = 24*(jtmp - jday);
        hour = int(htmp);
        mtmp = 60*((htmp - hour));
        minu = int(mtmp);
        stmp = 60*(mtmp - minu);
        secd = (stmp);
        return hour*10000+minu*100+secd;
    }

    function jdtimetosec(value) {
       hour = int(value/10000)
       minu = int((value%10000)/100)
       secd = value%100

       return 3600*hour + 60*minu + secd
    }

    function allowedsliptime(distance) {
        if(distance < 2000) return 18
        if(distance > 12000) return 0
        if(distance < 6000) return 24 - distance / 333.333
        return 12 - distance / 1000
    }

    function calcoffset(x,y) {
        return sqrt((x[1]-x[2])^2+(y[1]-y[2])^2)
    }

    FNR==NR { L[FNR]=$0 ; next}
    {
        tape[1]=$1
        splin[1]=$2
        spnum[1]=$3
        sprec[1]=$4
        sw[1]=$5
        x[1]=$6
        y[1]=$7
        spgps[1]=$8
        dtime[1]=gpstojdtime(spgps[1])
        day[1]=jday

        # Look up to 10 records ahead
        for(i=1; i< 10 && (i+FNR in L); i++) {
            split(L[i+FNR],V)
            tape[2]=V[1]
            splin[2]=V[2]
            spnum[2]=V[3]
            sprec[2]=V[4]
            sw[2]=V[5]
            x[2]=V[6]
            y[2]=V[7]
            spgps[2]=V[8]
            dtime[2]=gpstojdtime(spgps[2])
            day[2]=jday

            offset = calcoffset(x,y)
            stime = jdtimetosec(dtime[2]) - jdtimetosec(dtime[1])
            allow = allowedsliptime(offset)

            # Skip lookahead when diff is greater than 25sec
            if(stime > 25) next

            # decide on output file based on splittime vrs allowedsplit
            if(stime < allow) of=OUT[2]
            else of=OUT[1]

            for(j=1;j<3;j++) {
                printf"%d,%d,%d,%d,%.1f,%.1f,%d,%.2f,",
                    tape[j],splin[j],spnum[j],sprec[j],
                    x[j],y[j],day[j], dtime[j]  > of

                printf"%d,%d,%d,%d,%.1f,%.1f,%d,%.2f,",
                    tape[j],splin[j],spnum[j],sprec[j],
                    x[j],y[j],day[j], dtime[j]  > OUT[3]
            }
            printf "%.4f,%.3f,%5.4f\n", offset, stime, allow > of
            printf "%.4f,%.3f,%5.4f\n", offset, stime, allow > OUT[3]
        }
    }' toto toto

This User Gave Thanks to Chubler_XL For This Post:
jiam912
# 13  
Old 02-03-2017
Dear Xl

Many, thanks for the code.. As I say previously your job is wonderful.

Thanks a lot for your time, support and help.

Regards..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with split a list of records into each line with 200 coordinate at a time

Input File: E 3359799 3360148 350 X D 3471287 3471607 321 X E 3359799 3360740 942 X E 4359790 4360039 250 X . . . Desired Output File: E 3359799 3359998 200 X E 3359999 3360148 150 X D 3471287 3471486 200 X D 3471487 3471607 121 X E 3359799 3359998 200 X E 3359999 3360198 200 X... (1 Reply)
Discussion started by: perl_beginner
1 Replies

2. Shell Programming and Scripting

Grep 'time' in save records

Hi Team, Is there a way to grep time taken to save records. Its like there is one webpage where when I click save button taking so much time to save result. Therefore, I want to grep that time taken to save that record from file.log Thanks in advance. (1 Reply)
Discussion started by: TCS
1 Replies

3. UNIX for Dummies Questions & Answers

File name offset

Dear all, I want to offset the file numbers. can you please make some awk code or linux code for the same. Example: input file names ANI_WFMASS_PIST00001.gif ANI_WFMASS_PIST00002.gif . . . ANI_WFMASS_PIST0000n.gif offset --> 30 ANI_WFMASS_PIST00031.gif ANI_WFMASS_PIST00032.gif... (14 Replies)
Discussion started by: kri321shna
14 Replies

4. UNIX for Advanced & Expert Users

Grep --byte-offset not returning the offset (Grep version 2.5.1)

Hi, I am trying to get the position of a repeated string in a line using grep -b -o "pattern" In my server I am using GNU grep version 2.14 and the code is working fine. However when I am deploying the same code in a different server which is using GNU grep version 2.5.1 the code is not... (3 Replies)
Discussion started by: Subhamoy
3 Replies

5. Shell Programming and Scripting

Find records using epoch time.

How do i find the record which has been edit the last 10 minutes? from a.txt which has last field is epoch time updated. 10/17/2012 1:47 PM||||||In Use|chicken||1350005487 10/17/2012 2:53 PM||||||Available|chicken||13500000 10/17/2012 3:20 PM||||||In Use|cat||1351000000 10/17/2012 3:22... (2 Replies)
Discussion started by: sabercats
2 Replies

6. Shell Programming and Scripting

Scroll records from database, one at a time

Hi, I need to come up with a site that will display all the records in the database, but one at a time. Not sure how to go about it. Please pour in your suggestions. Thanks ---------- Post updated at 04:38 AM ---------- Previous update was at 12:52 AM ---------- Can... (3 Replies)
Discussion started by: sh_kk
3 Replies

7. Shell Programming and Scripting

awk scripting - matching records and summing up time

Hello. I just found out about awk, and it appears that this could handle the problem I'm having right now. I first stumbled on the thread How to extract first and last line of different record from a file, and that problem is almost similar to mine. In my case, an ASCII file will contain the... (0 Replies)
Discussion started by: Gonik
0 Replies

8. Programming

Negative Offset

Function: int fcntl(int fd, int cmd, struct flock * lock) Data Type: struct flock This structure is used with the fcntl function to describe a file lock. It has these members: off_t l_start This specifies the offset of the start of the region to which the lock applies, and... (1 Reply)
Discussion started by: DNAx86
1 Replies

9. Shell Programming and Scripting

deleting multiple records from a huge file at one time

I have a very big file of 5gb size and there are about 50 million records in there. I have to delete the records based on recrord number that I know fromoutside with out opening the file. The record numbers are very random like 5000678, 7890005 etc. Can somebody let me know how i can... (5 Replies)
Discussion started by: dsravan
5 Replies

10. Shell Programming and Scripting

Perform selection and deletion at the same time

I am performing some operation like: SQL Statement below: UPDATE table2 SET column1_table2 = ( SELECT column1_table1 FROM table1 WHERE column2_table1 = column2_table2 LIMIT 1) LIMIT 2; So what I'm trying to do is to find a corresponding value and then I'm adding it into table2. Now how... (2 Replies)
Discussion started by: Legend986
2 Replies
Login or Register to Ask a Question