Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Dont have a clue which program to use to process flying bat data

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Dont have a clue which program to use to process flying bat data
# 8  
Old 02-24-2011
If these are just input lines for optical sensors, and not gray code or something, I don't think your original events make much sense:
Code:
# Event A
# No sensors blocked
1,1
# Bat blocks sensor A.  OK.
0,1
# Bat teleports past sensor A to block sensor B.
1,0
# Bat does the moonwalk and blocks both.
0,0

I only find fourteen of these, and all of them backwards... Shouldn't the patterns be more like this?
Code:
#Event A
# No sensors blocked.
1,1
# Bat blocks sensor A
0,1
# Bat blocks both sensors
0,0
# Bat blocks only sensor B
1,0
# Elvis has left the building
1,1

#Event B
#No sensors blocked
1,1
# Sensor B blocked
1,0
# Both blocked
0,0
# Only A blocked
0,1
# Flying rat is cleared for landing.
1,1

This works a little better: 20 events.

But why does it have to start with no sensors blocked? The shorter the pattern you look for, the less you'll be messed up by flapping and noise. I tried checking for this:
Code:
#Event A
# Something's blocked both beams.
0,0
# ...but it's leaving.
1,0
# Hooray!
1,1

...and vice-versa for event B. This clocks 112 events, 58 A's and 54 B's. There's even a pattern to when they happen, though I'm less sure that makes sense(bats at high noon? what time zone was your logger? Smilie)

Code:
$ cat batmon.sh
#!/bin/bash

# Loop over all commandline parameters 2011-*
while [ "$#" -gt 0 ]
do
        BASE=$(basename "$1")

        # The datafiles contain \r, which must be stripped somehow
        while IFS=$'\r\n' read LINE
        do
                # Convert HH:MM:SS into HH,MM,SS to make awk easier
                TIME="${LINE:4:8}" ; TIME="${TIME//:/,}"
                # We also convert YYYY-MM-DD into YYYY,MM,DD
                echo "${BASE//-/,},${LINE:0:2},${TIME},${LINE:13}"
        done < "$1/LogFile.txt"
        shift
done | awk -F "," '
BEGIN { OT=0 # Time of previous measurement
                MAX=10  # Max num of seconds between valid events
                CA=0            ;       CB=0
                # Length of the patterns
                L=3
                # Patterns to check against
                A[0]="0,0";     A[1]="0,1";     A[2]="1,1"
                B[0]="0,0";     B[1]="1,0";     B[2]="1,1"
}

{
        if($8 == "pv")
        {       # Calculate number of seconds from date
                # Needs GAWK.
                T=mktime($1 " " $2 " " $3 " " $5 " " $6 " " $7);

                # If too much time has passed since the last event, start over.
                if((T-OT) > MAX)        # Blank the array
                        for(N=0; N<(L-1); N++)  C[N]="";
                else    # Shift elements toward the front
                        for(N=0; N<(L-1); N++)  C[N]=C[N+1];

                OT=T    # Set prev time to this one.

                C[L-1]=$9 "," $10;      # Set the latest event in the array

                FOUNDA=1;       FOUNDB=1;
                for(N=0; N<L; N++)
                {
                        if(A[N] != C[N]) FOUNDA=0;
                        if(B[N] != C[N]) FOUNDB=0;
                }

                # Count the events and mark the hour they occurred in
                if(FOUNDA)      {       CA++;   AH[$5]++;       }
                if(FOUNDB)      {       CB++;   BH[$5]++;       }
        }
}
END {   # Print the event counts
        printf("A %2d\nB %2d\nT %2d\n", CA, CB, CA+CB);

        # Print a list of hours from 1-23
        printf("H");
        for(N=1; N<=23; N++)    printf(" %2d", N);

        # Print hourly counts for event A
        printf("\nA");
        for(N=1; N<=23; N++)    printf(" %2d", AH[sprintf("%02d", N)]);

        # Hourly counts for event B
        printf("\nB");
        for(N=1; N<=23; N++)
                printf(" %2d", BH[sprintf("%02d",N)]);

        printf("\n");
}'

$ ./batmon.sh 2011-*
A 58
B 54
T 112
H  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
A  1  3  0  1  0  0  0  1  7  7 10  6  3  2  0  0  6  2  5  3  1  0  0
B  3  4  0  0  0  5  5  0  6  4  6  8  0  1  3  0  4  0  4  0  1  0  0
$

I know you didn't want shell, but even getting the list of directories into awk would've been a chore, let alone sorting it. Processing all the dates would've been very inefficient in BASH but is a snap in GAWK.

Doing a final bit of changes to add more stuff you asked for.

---------- Post updated at 12:11 PM ---------- Previous update was at 11:57 AM ----------

This version creates one big datafile, output.txt. It also prints individual events and their times to standard output, and final statistics to standard error. Modify the match sequences however you want but make sure you tell it the new length in the L variable.

Code:
$ cat batmon.sh
#!/bin/bash

# Loop over all commandline parameters 2011-*
while [ "$#" -gt 0 ]
do
        BASE=$(basename "$1")

        # The datafiles contain \r, which must be stripped somehow
        while IFS=$'\r\n' read LINE
        do
                # Convert HH:MM:SS into HH,MM,SS to make awk easier
                TIME="${LINE:4:8}" ; TIME="${TIME//:/,}"
                # We also convert YYYY-MM-DD into YYYY,MM,DD
                echo "${BASE//-/,},${LINE:0:2},${TIME},${LINE:13}"
        done < "$1/LogFile.txt"
        shift
done | tee output.txt | awk -F "," '
BEGIN { OT=0 # Time of previous measurement
                MAX=10  # Max num of seconds between valid events
                CA=0            ;       CB=0
                # Length of the patterns
                L=3
                # Patterns to check against
                A[0]="0,0";     A[1]="0,1";     A[2]="1,1"
                B[0]="0,0";     B[1]="1,0";     B[2]="1,1"
}

{
        if($8 == "pv")
        {       # Calculate number of seconds from date
                # Needs GAWK.
                T=mktime($1 " " $2 " " $3 " " $5 " " $6 " " $7);

                # If too much time has passed since the last event, start over.
                if((T-OT) > MAX)        # Blank the array
                        for(N=0; N<(L-1); N++)  C[N]="";
                else    # Shift elements toward the front
                        for(N=0; N<(L-1); N++)  C[N]=C[N+1];

                OT=T    # Set prev time to this one.

                C[L-1]=$9 "," $10;      # Set the latest event in the array

                FOUNDA=1;       FOUNDB=1;
                for(N=0; N<L; N++)
                {
                        if(A[N] != C[N]) FOUNDA=0;
                        if(B[N] != C[N]) FOUNDB=0;
                }

                # Count the events and mark the hour they occurred in
                if(FOUNDA)
                {
                        printf("A@%s-%s-%s %s:%s:%s\n",$1,$2,$3,$5,$6,$7);
                        CA++;   AH[$5]++;
                }

                if(FOUNDB)
                {
                        printf("B@%s-%s-%s %s:%s:%s\n",$1,$2,$3,$5,$6,$7);
                        CB++;   BH[$5]++;
                }
        }
}
END {   # The final statistics will be printed to stderr, to easily
        # seperate them from the event times printed to stdout.

        # Print the event counts
        printf("A %2d\nB %2d\nT %2d\n", CA, CB, CA+CB) > "/dev/stderr";

        # Print a list of hours from 1-23
        STR="H";
        for(N=1; N<=23; N++)    STR=STR sprintf(" %2d", N);;
        print STR > "/dev/stderr";

        # Print hourly counts for event A
        STR="A";
        for(N=1; N<=23; N++)
                STR=STR sprintf(" %2d", AH[sprintf("%02d", N)]);
        print STR > "/dev/stderr";

        # Hourly counts for event B
        STR="B";
        for(N=1; N<=23; N++)
                STR=STR sprintf(" %2d", BH[sprintf("%02d",N)]);
        print STR > "/dev/stderr";
}'

# 9  
Old 02-24-2011
method to the madness!
Hi, yes sorry dinn't want to confuse things with the times if not necessary but since you noticed:
its not actualy a time zone thing, its me twealing the time to make it work -sort of...
the logger automatically starts a new directory ever midnight, I have no control of this so what I did is make the PC clock 16 hours early thus 4pm (when bats start to flyin in winter) = midnight in pc/logger time and a new directory is started.Smilie

as regards the data I know its a bit confusing:
bit zero is the first port which is triggered by the beams. bit 1 is the secont port which is triggerd by the beams. -however... to make it "fun", the state of the ports is also recorded as a zero or a one - there is only one state per line of text

sometimes bats will break one beam by turning in the lightbox this obv screws things up and makes it more difficult. thus we need to search for definite patterns and ignore all others.

I am now going to try to make sure I have explained things the right way
for each line of text (they are installed in remote locations hours away from here)
here is a sample from the manufacturer showing more than two ports in operation,
this shows that the port is logged first then the state.
Code:
<-16:24:14,pv,4,1
Rx<-16:24:14,pv,2,0
Rx<-16:24:14,pv,4,0
Rx<-16:24:14,pv,2,1
Rx<-16:24:14,pv,4,1
Rx<-16:24:14,pv,2,0
Rx<-16:24:14,pv,4,0
Rx<-16:24:14,pv,0,1
Rx<-16:24:14,pv,2,1
Rx<-16:24:14,pv,4,1
Rx<-16:24:14,pv,0,0
Rx<-16:24:14,pv,2,0
Rx<-16:24:14,pv,4,0
Rx<-16:24:15,pv,1,1
Rx<-16:24:15,pv,3,1
Rx<-16:24:15,pv,5,1
Rx<-16:24:15,pv,1,0
Rx<-16:24:15,pv,3,0
Rx<-16:24:15,pv,5,0

Last edited by Scott; 02-24-2011 at 03:59 PM.. Reason: Code tags
# 10  
Old 02-24-2011
Hey, Corona688:

I think you misunderstand the nature of the data representation. I know I did and was puzzled by the bat teleportation phenomenon. I don't think the final fields each represent a beam's state. Instead, the penultimate field identifies the beam and the ultimate its state. At least, that's the only way it makes sense to me.

Regards,
Alister

---------- Post updated at 01:52 PM ---------- Previous update was at 01:32 PM ----------

Hello, cmp260:


Quote:
Originally Posted by cmp260
Code:
(date,time,beam,signal)
20/01/2011,10:10,1,1
20/01/2011,10:10,0,1
20/01/2011,10:10,1,0
20/01/2011,10:10,0,0

the above is a single transit in directiion "A" the opposite would be a transit in direction "B" as follows:
Code:
20/01/2011,10:15,0,1
20/01/2011,10:15,1,1
20/01/2011,10:15,0,0
20/01/2011,10:15,1,0

the abvoe two examples are they only valid transits.
Is the order in which the beams report high important? If the first two lines in each of those patterns were reversed, would it still constitute a valid transit (since both beams were high prior to both going low)? If yes, then the following would also be a valid transit.
Code:
Rx<-07:04:55,pv,0,1
Rx<-07:04:55,pv,1,1
Rx<-07:04:55,pv,1,0
Rx<-07:04:55,pv,0,0

Also, your newer sample data has increased in resolution from minutes to seconds. Should an otherwise valid transit be invalidated if it crosses a second boundary? Or is the following a valid transit?
Code:
Rx<-07:04:55,pv,1,1
Rx<-07:04:55,pv,0,1
Rx<-07:04:56,pv,1,0
Rx<-07:04:56,pv,0,0

Regards,
Alister
# 11  
Old 02-24-2011
Blarg. Oh well, you can plug whatever patterns you want into my script. Glad to know I wasn't catching dozens of false-positives from daylight bat gremlins. I'll work on it.

I still think you can make your patterns a whole lot shorter to catch a lot more valid events on the assumption that any bug big enough to simultaneously block two sensors is probably a bat. In fact, since it just records changes, the lines also tell us what the state used to be, so the pattern can be really really short:

Code:
# Getting these two events in a row implies both beams were blocked.
0,1     # Sensor 0 becomes unobstructed.
1,1     # Sensor 1 becomes unobstructed.

# Or, the other way:
1,1     # Sensor 1 becomes unobstructed.
0,1     # Sensor 0 becomes unobstructed.

These give the following event counts:
Code:
$ 
$ ./batmon.sh 2011* > /dev/null
A 147
B 148
T 295
H  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
A  6  6  0  2  1  0  1  6 24 14 23 18  6  5  2  0 11  4 11  4  3  0  0
B  4  7  0  1  1  7  7  3 15 15 18 18  9  2  8  5  9  6  8  3  2  0  0

...and that's with the accuracy increased to within a second(MAX=1).

Or to make them a bit more sure:

Code:
# one direction
0,0 # Beam 0 obstructed.
1,1 # Beam 1 unobstructed.
0,1 # Beam 0 unobstructed.  Two of these in a row means both sensors were blocked after 0,0

# another direction
1,0 # beam 1 obstructed.
0,1 # beam 0 unobstructed.
1,1 # beam 1 unobstructed.  Two of these in a row means both sensors were blocked after 1,0

That nets you
Code:
$ ./batmon.sh 2011* > /dev/null
A 18
B 15
T 33
H  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
A  1  1  0  0  0  0  1  1  3  1  2  3  1  0  0  2  0  1  0  1  0  0  0
B  2  1  0  0  0  0  0  0  4  1  4  1  1  0  0  0  0  0  1  0  0  0  0
$

If you want to deal with multiple sets of sensors in one data file I think you'll need a perl script, since awk doesn't have multidimensional arrays...
Last edited by Corona688; 02-24-2011 at 03:29 PM..
# 12  
Old 02-24-2011
blarg
hi, you have mixed it up a bit: state zero means unblocked beam so your statement below is wrong:
Getting these two events in a row implies both beams were blocksed.
Code:
0,1     # Sensor 0 becomes unobstructed.
1,1     # Sensor 1 becomes unobstructed.

this should be:
Code:
0,0     # first Sensor becomes unobstructed.
1,0     # second sensor becomes unobstructed.
2,0     # third becomes unobstructed.
3,0     # fourth becomes unobstructed.
etc. etc.

however, there are only two sensors in actual fact -above is just to illustrate the point
Last edited by Scott; 02-24-2011 at 04:00 PM..
# 13  
Old 02-24-2011
Ah. Correcting the patterns gets you:

Code:
# one direction
0,1 # Beam 0 obstructed.
1,0 # Beam 1 unobstructed.
0,0 # Beam 0 unobstructed.  Two of these in a row means both sensors were blocked after 0,1

# another direction
1,1 # beam 1 obstructed.
0,0 # beam 0 unobstructed.
1,0 # beam 1 unobstructed.  Two of these in a row means both sensors were blocked after 1,1

Which works much better, even at maximum accuracy(MAX=0):
Code:
$ ./batmon.sh 2011* > /dev/null
A 115
B 109
T 224
H  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
A  4  6  0  1  1  1  1  2 12 15 15 14  7  2  5  4  8  6  6  3  2  0  0
B  4  2  0  2  0  0  1  6 14 12 17 14  6  5  4  0  8  2  8  3  1  0  0

# 14  
Old 02-24-2011
Alastair,
as regards your Q: if this is a valid transit?
Code:
Rx<-07:04:55,pv,0,1
Rx<-07:04:55,pv,1,1
Rx<-07:04:55,pv,1,0
Rx<-07:04:55,pv,0,0

the answer is sort of - but it would not help to count it as one since it would indicate that the bat turned around and flew back from where it had come whilst inside the beams.

hope this helpsSmilie
Last edited by Scott; 02-24-2011 at 04:00 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

I dont want to exit the program by ctrl + c

Hey , guys I am new to shell programing ,, so need a help from you guys ... I have to write a shell script that accepts file name or directory name from the user if it is a directory then throw an error if it is a file then give the user two options . 1.overwrite the content 2.append the... (2 Replies)
Discussion started by: coolashu
2 Replies

2. Shell Programming and Scripting

How can I get parent process program name?

Hi: I have 2 script on AIX server. The child script is called by parent script. For example: The parent script full name is /home/op/def/parent.spt, which calls /home/op/abc/child.spt I want to get the parent program name with full path name (i.e. /home/op/def/parent.spt), in... (3 Replies)
Discussion started by: cstsang
3 Replies

3. Shell Programming and Scripting

dont have a clue again- flying bat data new logger

hi, this is related to this thread https://www.unix.com/shell-programming-scripting/154822-dont-have-clue-program-use-process-flying-bat-data.html using another type of logger (ipcas ipether232) which produces csv data but not seperated into differnt folders. I need to work out how to get... (2 Replies)
Discussion started by: cmp260
2 Replies

4. UNIX for Dummies Questions & Answers

defunct process occuring in a particular program

All, I am getting a wired scenario, Not all times but some times i am getting the defunct process for a program. If i rerun the same program it is working fine.This defunct scenario is not occuring often. And this is happening only for the one program in my system that to sometimes Can you... (1 Reply)
Discussion started by: arunkumar_mca
1 Replies

5. Programming

Process creation in Unix with C Program

Hi, can somebody please help me regarding this? How can i print parent's PID from child's and vice versa. I have tried with getpid() & ngetppid() function but it is not giving me the correct data. The logic i used is: if (pid == 0) { getpid() : Chil'd PID getppid():... (1 Reply)
Discussion started by: dsudipta
1 Replies

6. Programming

How to start a process in linux using C program??

I have a set of attributes such as group id,group name,etc related to a linux process. Iwant to know how to start a process in linux using C program.Plz do help me. (3 Replies)
Discussion started by: vigneshinbox
3 Replies

7. Programming

Getting process info in a program

Hi, I am wondering if there is a way to find out in a C software program if a particular process is running without having to write a file. In the past, I have been using the system command to execute a pgrep and output the info to a file. Then the C program reads the file to make the... (1 Reply)
Discussion started by: herbmiller
1 Replies

8. HP-UX

get program name give a process id

Hi , I have query regarding to get a program name given a pid in HP-Ux . give procees id ( PID) i would like to retrieve the process/program through a C program ? my input to c program will be will be pid and i would like to know what is process name /program name . Many Thanks ... (1 Reply)
Discussion started by: naren_chella
1 Replies

9. HP-UX

IO by Process/program

HI there, I'm trying to find a way of showing the IO's performed by individual programs. Number of reads/writes IO time that kind of thing. Anybody know of any way to show this information. Free little downloads that kind of thing? Cheers Phil (2 Replies)
Discussion started by: cpiuk
2 Replies
Login or Register to Ask a Question