Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Dont have a clue which program to use to process flying bat data

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Dont have a clue which program to use to process flying bat data
# 15  
Old 02-24-2011
Or an exact pattern including all steps of the process from all unblocked to all blocked gets this:
Code:
# One direction
1,1 #Block 1
0,1 #Block 0
1,0 #Unblock 1
0,0 #Unblock 0

# Other dir
0,1 #Block 0
1,1 #Block 1
0,0 #Unblock 0
0,1 #Unblock 1

Code:
A 65
B 74
T 139
H  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
A  2  4  0  0  1  1  1  0  4 10  4  9  6  2  4  2  5  3  3  3  1  0  0
B  3  2  0  1  0  0  0  4 12  8 12  9  4  2  2  0  5  1  7  2  0  0  0

What are you researching by the way?
This User Gave Thanks to Corona688 For This Post:
cmp260
# 16  
Old 02-24-2011
totals explained
it might help at this stage to confuse the issue more and explain how it is planned to get daily total counts:
these bat species do a weird thing called "light sampling" where they go in and out multiple times until the conditons are right and finally leave to forage for the night. Obv, this causes a total screw up if you just simply count the total number exiting - you end up with an order of magnitude more bats than actually exist in the roost. To eliminate this as best as possible we need to total the ins and outs until a peak number is arrived at as follows:
If we set A or "out" = plus one and B or "in" = minus one
thus:
Code:
dinrection total
out +1   1
out +1   2
out +1   3
out +1   4
in -1    3
in -1    2
out +1   3

etc. this way at some point in each evening there is a maximum in the total column which will Smiliemore or less equate to the total number in the roost

---------- Post updated at 07:54 PM ---------- Previous update was at 07:47 PM ----------

Hi Corona, not research, monitoring the poplulations since they are protected species.Smilie
Greater and Lesser horseshoe bats.
Last edited by Scott; 02-24-2011 at 04:01 PM.. Reason: Please use code tags
# 17  
Old 02-24-2011
Well that sure changes the picture.

The 'guess' is the absolute value of the daily count because, for eight bats to leave and not return means there WERE at least eight bats in there right?

[edit] Bug fixed. There's still that funny day where eight left but reviewing the events I see there are eight more B's than A's that day.

Code:
$ cat batmon.sh
#!/bin/bash

# Loop over all commandline parameters 2011-*
while [ "$#" -gt 0 ]
do
        BASE=$(basename "$1")

        # The datafiles contain \r, which must be stripped somehow
        while IFS=$'\r\n' read LINE
        do
                # Convert HH:MM:SS into HH,MM,SS to make awk easier
                TIME="${LINE:4:8}" ; TIME="${TIME//:/,}"
                # We also convert YYYY-MM-DD into YYYY,MM,DD
                echo "${BASE//-/,},${LINE:0:2},${TIME},${LINE:13}"
        done < "$1/LogFile.txt"
        shift
done | tee output.txt | awk -F "," '
BEGIN { OT=0 # Time of previous measurement
                MAX=0   # Max num of seconds between valid events
                DAY=""; # Current day
                CA=0            ;       CB=0
                # Running total of bats leaving and entering
                TOTALBATS=0;
                # The highest TOTALBATS has ever been
                MAXBATS=0;
                # Length of the patterns
                L=4
                # Patterns to check against
                # Block 1       Block 0         Unblock 1       Unblock 0
                A[0]="1,1";     A[1]="0,1";     A[2]="1,0";     A[3]="0,0";
                # Block 0       Block 1         Unblock 0       Unblock 1
                B[0]="0,1";     B[1]="1,1";     B[2]="0,0";     B[3]="1,0";

#               A[0]="0,0";     A[1]="0,1";     A[2]="1,1"
#               B[0]="0,0";     B[1]="1,0";     B[2]="1,1"
}

{
        if((DAY != $1 $2 $3) && (DAY != ""))
        {
                I=TOTALBATS;    if(I<0) I=-I;
                STR=sprintf("COUNT@%s COUNT %+d RET %d LEFT %d GUESS %d", DAY,
                        TOTALBATS, MAXBATS, -MINBATS, I);
                print STR >"/dev/stderr"
                TOTALBATS=0;
                MAXBATS=0;
                MINBATS=0;
        }

        DAY=$1 $2 $3;

        if($8 == "pv")
        {       # Calculate number of seconds from date
                # Needs GAWK.
                T=mktime($1 " " $2 " " $3 " " $5 " " $6 " " $7);

                # If too much time has passed since the last event, start over.
                if((T-OT) > MAX)        # Blank the array
                        for(N=0; N<(L-1); N++)  C[N]="";
                else    # Shift elements toward the front
                        for(N=0; N<(L-1); N++)  C[N]=C[N+1];

                OT=T    # Set prev time to this one.

                C[L-1]=$9 "," $10;      # Set the latest event in the array

                FOUNDA=1;       FOUNDB=1;
                for(N=0; N<L; N++)
                {
                        if(A[N] != C[N]) FOUNDA=0;
                        if(B[N] != C[N]) FOUNDB=0;
                }

#               DAY=$1 $2 $3;


                # Count the events and mark the hour they occurred in
                if(FOUNDA)
                {
                        printf("A@%s-%s-%s %s:%s:%s\n",$1,$2,$3,$5,$6,$7);
                        CA++;   AH[$5]++;
                        TOTALBATS++;
                }

                if(FOUNDB)
                {
                        printf("B@%s-%s-%s %s:%s:%s\n",$1,$2,$3,$5,$6,$7);
                        CB++;   BH[$5]++;
                        TOTALBATS--;
                }

                if(MAXBATS < TOTALBATS) MAXBATS=TOTALBATS;
                if(MINBATS > TOTALBATS) MINBATS=TOTALBATS;
        }
}
END {   # The final statistics will be printed to stderr, to easily
        # seperate them from the event times printed to stdout.

        # The last daily count
        I=TOTALBATS;    if(I<0) I=-I;
        STR=sprintf("COUNT@%s COUNT %+d RET %d LEFT %d GUESS %d", DAY,
                TOTALBATS, MAXBATS, -MINBATS, I);
        print STR >"/dev/stderr"

        # Print the event counts
        printf("A %2d\nB %2d\nT %2d\nM %2d\n", CA, CB, CA+CB, MAXBATS) > "/dev/stderr";

        # Print a list of hours from 1-23
        STR="H";
        for(N=1; N<=23; N++)    STR=STR sprintf(" %2d", N);;
        print STR > "/dev/stderr";

        # Print hourly counts for event A
        STR="A";
        for(N=1; N<=23; N++)
                STR=STR sprintf(" %2d", AH[sprintf("%02d", N)]);
        print STR > "/dev/stderr";

        # Hourly counts for event B
        STR="B";
        for(N=1; N<=23; N++)
                STR=STR sprintf(" %2d", BH[sprintf("%02d",N)]);
        print STR > "/dev/stderr";
}'

$ ./batmon.sh 2011* > /dev/null
COUNT@20110123 COUNT -2 RET 0 LEFT 4 GUESS 2
COUNT@20110124 COUNT +3 RET 3 LEFT 2 GUESS 3
COUNT@20110125 COUNT +2 RET 2 LEFT 1 GUESS 2
COUNT@20110126 COUNT +0 RET 0 LEFT 0 GUESS 0
COUNT@20110127 COUNT +0 RET 0 LEFT 0 GUESS 0
COUNT@20110128 COUNT +3 RET 3 LEFT 2 GUESS 3
COUNT@20110129 COUNT -1 RET 0 LEFT 3 GUESS 1
COUNT@20110130 COUNT -6 RET 0 LEFT 8 GUESS 6
COUNT@20110131 COUNT -8 RET 2 LEFT 10 GUESS 8
COUNT@20110201 COUNT +0 RET 0 LEFT 0 GUESS 0
COUNT@20110202 COUNT +0 RET 0 LEFT 0 GUESS 0
A 65
B 74
T 139
M  0
H  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
A  2  4  0  0  1  1  1  0  4 10  4  9  6  2  4  2  5  3  3  3  1  0  0
B  3  2  0  1  0  0  0  4 12  8 12  9  4  2  2  0  5  1  7  2  0  0  0
$

Last edited by Corona688; 02-24-2011 at 04:58 PM..
# 18  
Old 02-25-2011
the counts will hardly ever match completely since there usually a few bats that roost elswhere but return a day or two later

---------- Post updated at 11:31 AM ---------- Previous update was at 12:19 AM ----------

thanks for all your work on this Corona.
a few more questions (remember, I dont have a clueSmilie as yet)
What is the program called? a bash script?
and how would I go about running it? How do I read the output?
are there any online tutorials on it you know of so I can teach myself?
Is there any way to run the program from the unix equivalent of a dos batch file?

If you had the time, it would be nice to be able to see what time of night the maximum daily count was achieved. I know I can manually put the data in excel and adjust the time by -16hours using formulas then paste it into the directories again prior to running your script, but is there anyway to do this programatically?
many thnksSmilie
# 19  
Old 02-25-2011
Quote:
Originally Posted by cmp260
thanks for all your work on this Corona.
a few more questions (remember, I dont have a clueSmilie as yet)
What is the program called? a bash script?
A shell script, yes. It needs BASH or a relatively new version of KSH, as well as gawk. Your linux system shouldn't have any trouble with it.
Quote:
and how would I go about running it?
I've been including the instructions for running it with almost every post, but, to give a little more detail:
Code:
# go to your data files.  Where you put them is up to you.
$ cd ~/code/sh/batmon/
# ls is like 'dir'.
# We are in a directory full of your data folders.
$ ls -l
total 144
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-01-23
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-01-24
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-01-25
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-01-26
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-01-27
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-01-28
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-01-29
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-01-30
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-01-31
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-02-01
drwxr-xr-x 2 tyler users  4096 Feb 24 09:21 2011-02-02
-rw-rw---- 1 tyler users 11997 Feb 24 09:18 File.zip
-rwxr-xr-x 1 tyler users  3074 Feb 25 09:27 batmon.sh
-rw-r--r-- 1 tyler users 80012 Feb 25 09:27 output.txt
# batmon.sh is a text file containing the contents I've been showing you.
# You run it like this in the shell.  The shell knows how to turn 2011* into
# 2011-02, etc.
$ ./batmon.sh 2011*
...

Quote:
How do I read the output?
Code:
# This saves the individual events into events.txt and 
# outputs the rest of the data to the screen.
$ ./batmon.sh 2011* > events.txt
# These are the daily counts.  total A - total B = -2 on 20110123, etc.
COUNT@20110123 COUNT -2 RET 0 LEFT 4 GUESS 2
COUNT@20110124 COUNT +3 RET 3 LEFT 2 GUESS 3
COUNT@20110125 COUNT +2 RET 2 LEFT 1 GUESS 2
COUNT@20110126 COUNT +0 RET 0 LEFT 0 GUESS 0
COUNT@20110127 COUNT +0 RET 0 LEFT 0 GUESS 0
COUNT@20110128 COUNT +3 RET 3 LEFT 2 GUESS 3
COUNT@20110129 COUNT -1 RET 0 LEFT 3 GUESS 1
COUNT@20110130 COUNT -6 RET 0 LEFT 8 GUESS 6
COUNT@20110131 COUNT -8 RET 2 LEFT 10 GUESS 8
COUNT@20110201 COUNT +0 RET 0 LEFT 0 GUESS 0
COUNT@20110202 COUNT +0 RET 0 LEFT 0 GUESS 0
# This is the complete total number of A events
A 65
# This is the complete total number of B events.
B 74
# This is the complete total number of events, A and B.
T 139
# These are hours from 1am onwards.
H  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
# These are the total number of A events in every hour.
A  2  4  0  0  1  1  1  0  4 10  4  9  6  2  4  2  5  3  3  3  1  0  0
# These are the total number of B events in every hour.
B  3  2  0  1  0  0  0  4 12  8 12  9  4  2  2  0  5  1  7  2  0  0  0

Quote:
are there any online tutorials on it you know of so I can teach myself?
Tell you what, I'll see if I can find a decent version of bash and gawk for windows and zip something up for you...
Quote:
Is there any way to run the program from the unix equivalent of a dos batch file?
It is the UNIX equivalent of a batch file.
Quote:
If you had the time, it would be nice to be able to see what time of night the maximum daily count was achieved.
I'll take a look.
Quote:
I know I can manually put the data in excel and adjust the time by -16hours using formulas then paste it into the directories again prior to running your script, but is there anyway to do this programatically?
I'll take a look..
# 20  
Old 02-25-2011
hi, thanks for the help again! I do know a bit about linux commands and dir structure, ls, cd etc, probably just enough to be a danger to myself, but havent messed with in it a few years.

Ive just installed cygwin on XP which includes GAWK, would that work?
Im happy to use linux but Win would be useful for other non techhie peopel to bel be able to use it too. Smilie
# 21  
Old 02-25-2011
I doubt it... It's really not a UNIX environment, same awk or not.

But I managed to hack something that'll work by splitting the awk part into its own file. That code works in both. Now there's two little stubs, a batchfile in Windows and a shell script in Linux, that paste all the data together, add dates, and feed it into awk, dumping the output into three files:
  • output.txt -- all the data
  • stats.txt -- mostly what I've been showing you. The daily count and some totals.
  • events.txt -- the time and date and direction of every event.

Some missing utilities were gotten from UnxUtils | Download UnxUtils software for free at SourceForge.net

I fixed the dates and added the time of maximum. (Many days turned out to have no maximum.) If you need any changes now hopefully you can just update the awk bit and it'll work in both.
batwin.zip (325.4 KB) 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

I dont want to exit the program by ctrl + c

Hey , guys I am new to shell programing ,, so need a help from you guys ... I have to write a shell script that accepts file name or directory name from the user if it is a directory then throw an error if it is a file then give the user two options . 1.overwrite the content 2.append the... (2 Replies)
Discussion started by: coolashu
2 Replies

2. Shell Programming and Scripting

How can I get parent process program name?

Hi: I have 2 script on AIX server. The child script is called by parent script. For example: The parent script full name is /home/op/def/parent.spt, which calls /home/op/abc/child.spt I want to get the parent program name with full path name (i.e. /home/op/def/parent.spt), in... (3 Replies)
Discussion started by: cstsang
3 Replies

3. Shell Programming and Scripting

dont have a clue again- flying bat data new logger

hi, this is related to this thread https://www.unix.com/shell-programming-scripting/154822-dont-have-clue-program-use-process-flying-bat-data.html using another type of logger (ipcas ipether232) which produces csv data but not seperated into differnt folders. I need to work out how to get... (2 Replies)
Discussion started by: cmp260
2 Replies

4. UNIX for Dummies Questions & Answers

defunct process occuring in a particular program

All, I am getting a wired scenario, Not all times but some times i am getting the defunct process for a program. If i rerun the same program it is working fine.This defunct scenario is not occuring often. And this is happening only for the one program in my system that to sometimes Can you... (1 Reply)
Discussion started by: arunkumar_mca
1 Replies

5. Programming

Process creation in Unix with C Program

Hi, can somebody please help me regarding this? How can i print parent's PID from child's and vice versa. I have tried with getpid() & ngetppid() function but it is not giving me the correct data. The logic i used is: if (pid == 0) { getpid() : Chil'd PID getppid():... (1 Reply)
Discussion started by: dsudipta
1 Replies

6. Programming

How to start a process in linux using C program??

I have a set of attributes such as group id,group name,etc related to a linux process. Iwant to know how to start a process in linux using C program.Plz do help me. (3 Replies)
Discussion started by: vigneshinbox
3 Replies

7. Programming

Getting process info in a program

Hi, I am wondering if there is a way to find out in a C software program if a particular process is running without having to write a file. In the past, I have been using the system command to execute a pgrep and output the info to a file. Then the C program reads the file to make the... (1 Reply)
Discussion started by: herbmiller
1 Replies

8. HP-UX

get program name give a process id

Hi , I have query regarding to get a program name given a pid in HP-Ux . give procees id ( PID) i would like to retrieve the process/program through a C program ? my input to c program will be will be pid and i would like to know what is process name /program name . Many Thanks ... (1 Reply)
Discussion started by: naren_chella
1 Replies

9. HP-UX

IO by Process/program

HI there, I'm trying to find a way of showing the IO's performed by individual programs. Number of reads/writes IO time that kind of thing. Anybody know of any way to show this information. Free little downloads that kind of thing? Cheers Phil (2 Replies)
Discussion started by: cpiuk
2 Replies
Login or Register to Ask a Question