I see what you mean by the piping and being a bash script. I actually played with that a bit but didn't get any better results.

Unfortunately I can't get you a better representation of the file just because I'm at home right now and it's something I'm doing for work. I can do that tomorrow however.

In the meantime, you can assume the word "rewrite" on the lines that I want and that will be fine. Even better, the grep part I am not really concerned about, I know grep well and regex very well so we can just assume the grep is doing it's job or will be after I get awk under control. It's the awk I'm very new and raw on.

I must have copied the wrong version of the code for you to look at, or, your highlighter got it wrong (it happens quite a bit with shell scripts and quotes within quotes within...). In any case the part that goes

Code:

echo "$4 $5" |
        sed -e 's/\//-/g' -e 's/:/ /1' |
        tr -d '[]' |

is necessary.
Basically it turns this [25/Jun/2014:18:57:07 --700] into this 25-Jun-2014 18:57:07 --700 which is then digestible by date so we can return a UNIX timestamp.

Obviously this means that both RefDate & LogDate are UNIX timestamps where RefDate is fixed for the whole script at some time set to "less than now", probably -1 hour so it will be: now-1 hour not now-2 seconds as I showed it.

The one liner was exactly as you said, too long, that's why I was putting it into a script and breaking it out into lines.

The first date variable only needs to be set once since it's compared against as a constant for all the lines processed. The second variable needs to be called for every line since the date is actually picked out of the line (fields 4 and 5 hence the echo $4 $5|....

I really want to get the awk stuff in it's own script file, that's the exercise, making that work.

Thinking of it like a bash script of piping then I would break it up into two files.

Perhaps like this? :

File: monitor-rewrite.sh

Code:

#!bin/sh
cat '/var/log/httpd/rewrite.log' | \
awk -f monitor-rewrite.awk | \
grep -B 1 -A 0 -inP '(rewrite\s+.?assets/.+.?\s+->\s+.?sites/default/files/.+.?|rewrite .?sites/default/files/.+.?\s+->\s+.?assets/.+.?)'

And then:
File: monitor-rewrite.awk
(You'll see I've played around with the quotes etc. on the variable assignments but to no avail so far.)

Code:

BEGIN {
  RefDate=$(date -d "now-3 hours" "+%s"); 
  LogDate=$(echo $4 $5|sed -e "s#/#-#g" -e "s/:/ /1"|tr -d "[]"|date "+%s")
}
{
  if (LogDate > RefDate) {
    print RefDate, LogDate, RefDate-LogDate, $0; 
  } else {
    print RefDate, LogDate, RefDate-LogDate; 
  }
}

And this is really what I want to achieve, breaking the awk code out into it's own file for the sake of cleaner code but mostly so I learn how since it seems the syntax and rules change a bit once moved from the command line to a program file.

BTW, the print statements in the file will simply be:

Code:

  if (LogDate > RefDate) 
    print $0;

What you see above is just so you can see the variables to know if they are getting set correctly.

Things I have noticed when moving the awk code into an awk program file:
It's the variables I'm really stuck on. Perhaps on the one liner, even though they were after the awk they were still being interpreted at the shell script level. Or perhaps I should not have put them in the BEGIN {} area and they actually go somewhere else -- the second variable I suspect that's the case since BEGIN isn't the loop area and it needs to change for every line (the only reason I haven't moved it yet is that I was trying to get rid of the syntax errors first).

I agree there are probably more efficient ways to read the file into awk but it's fine for now. Best efficiency is not nearly as important as figuring this out and just making it work.

Hopefully this explains what I am attempting a little better.