It has one bug which is that the LogDate variable doesn't seem to update on a per line basis. I'm assuming that's because it's in the setup section and so I'm not too fussed about that as I figured it was just a matter of moving it to the the loop area.
However, before tackling that I wanted to get it all into a script file. This however I have found not so easy. Mostly parse errors and also the call to the date function I can't get to work even when I fix the parsing errors.
Also, you see the last line is the piping to the awk program. This doesn't work inside this file how I have it here, I have to pipe that to the file from outside it but I would like all the code self-contained in one file if possible. I did try using a bash shell script instead of a .awk program file but I came across similar problems in parsing etc. so at that point I decided to ask the bright people here for some help
Any help greatly appreciated. Here's what I came up with for an awk file:
---------- Post updated at 09:18 PM ---------- Previous update was at 09:16 PM ----------
One more thing, I know there's many other tools out there but I would really like to make this work with awk. It's as much an exercise with awk for me as it is solving a real world problem.
Shell command substitution ($(date -d'now-3 hours' '+%s')) is not an awk command.
Similarly, echo, grep, sed, and tr and the pipelines connecting them are not awk commands. In addition to that you're using options to some of these utilities that are not available on all systems (such as the date -d option and the forms of BREs you're giving to grep), so I can't run your script to figure out what it is doing.
Please show us the output you're trying to produce and explain in English exactly what you want this script to do.
Last edited by Don Cragun; 06-26-2014 at 12:54 AM..
Reason: Fix typo.
Basically what I am doing is looking through an apache rewrite.log file and filtering out lines I don't want. I use awk to pre-filtering the lines by date so that the "...now-2 seconds..." part is just setting up a date/time so that I only look at lines up to 2 seconds old. That 2 seconds was just left over from testing. In reality it would 1 hour or something like that.
After awk filters the lines by date I run them through grep to get just the ones I am interested in by way of a regex.
To answer your more specific questions:
Code:
awk -v RefDate="$(date -d'now-2 second' '+%s')"
(Note: above is from the one liner, the -v is not on the multi-line version but is instead encased in braces with a begin, as in: BEGIN {...}.)
Yes, I see that there is a system call and that system call is the date command. However, this part of the line to me looks like it says "awk awk, I want you to make a variable -v called RefDate RefDate= and do so by getting the output of the system date function "$(date -d'now-2 second' '+%s')". Actually the '2 seconds' part was just testing, it reality it would be hours or a day.
In my mind this is an awk feature of using variables that just happens to in this case call a system command. So to me that part is an awk script with a bash sub-script, not a bash script with an awk sub-script.
However, the second part, where I am piping filtered lines from awk to grep, certainly that part is not an awk script. I guess that means all of that part must be outside the .awk file. I was hoping to just tag it onto the last line of the .awk file just as it's the last part of the one liner ...kind of hoping the same philosophy would following from the one liner to the awk program file but I guess that's simply not the case. Or is there a way?
Basically what I am doing is looking through an apache rewrite.log file and filtering out lines I don't want. I use awk to pre-filtering the lines by date so that the "...now-2 seconds..." part is just setting up a date/time so that I only look at lines up to 2 seconds old. That 2 seconds was just left over from testing. In reality it would 1 hour or something like that.
After awk filters the lines by date I run them through grep to get just the ones I am interested in by way of a regex.
To answer your more specific questions:
Code:
awk -v RefDate="$(date -d'now-2 second' '+%s')"
(Note: above is from the one liner, the -v is not on the multi-line version but is instead encased in braces with a begin, as in: BEGIN {...}.)
Yes, I see that there is a system call and that system call is the date command. However, this part of the line to me looks like it says "awk awk, I want you to make a variable -v called RefDate RefDate= and do so by getting the output of the system date function "$(date -d'now-2 second' '+%s')". Actually the '2 seconds' part was just testing, it reality it would be hours or a day.
In my mind this is an awk feature of using variables that just happens to in this case call a system command. So to me that part is an awk script with a bash sub-script, not a bash script with an awk sub-script.
However, the second part, where I am piping filtered lines from awk to grep, certainly that part is not an awk script. I guess that means all of that part must be outside the .awk file. I was hoping to just tag it onto the last line of the .awk file just as it's the last part of the one liner ...kind of hoping the same philosophy would following from the one liner to the awk program file but I guess that's simply not the case. Or is there a way?
Almost your entire problem is thinking that you have an awk script instead of a bash or ksh shell script. What you have is a shell pipeline where the first command in the pipeline happens to be awk. If your 1-liner works when typed into your shell from your terminal window, adding the full pathname of your shell after #! as the 1st line of your script and your 1-liner as the 2nd line of the script would work.
Note that the date utility is not a system call. I haven't seen any OS that has a date() system call. You may be thinking about the awk system() function which can be used to invoke a shell to run a command, but you don't want to invoke a shell from awk to process each line you're reading from a file. If you need to do that, you're probably wasting a lot of time by using awk instead of just reading the file directly in the shell.
Note that a 1-liner that long is impossible to read and update (assuming you may want to make changes later). So you should probably reformat your 1-liner into more readable chunks. You haven't said what shell you're using, so I'll use the shell I normally use (ksh) as an example. For this script ksh and bash are interchangeable:
And, looking at this we can more easily see that all of the code marked in red above from your 1-liner is a no-op (the date utility does not read anything from its standard input; so the echo, sed, and tr commands don't do anything useful. And, the echo "$4 $5" is referring to the 4th and 5th arguments passed to your shell (not the 4th and 5th fields from a line in the file awk is reading, so that pipeline is just passing a <space> character followed by a <newline> character to date anyway).
and still produce the same output.
And, as long as RefDate is always set to now minus something and LogDate is always set to now, the script can be further simplified to:
and still produce the same output. Note also that since the input line you showed us does not contain the string "rewrite", grep is not going to match any lines and you won't have any output. I, therefore, have to assume that there are other lines in your log file that are in a different format.
If you will show us a more representative sample of your input file, and be more clear about what you're really trying to do, I think I can help you reformat the way you set RefDate so we can compare that to the date and time in the 4th and 5th fields from lines in the format you showed us. But, unless the lines you're using to match with the grep have the same data in the 4th and 5th fields, you aren't going to get the output you want.
PLEASE show us some representative sample input AND show us the output you want to get from your script!
If you have the dates and times shown in human readable terms (instead of a seconds since the Epoch), does your output really need the number of seconds between the two? If you need that difference to be shown in seconds, your script will run MUCH slower.
Last edited by Don Cragun; 06-26-2014 at 04:12 AM..
Reason: Fix typo (you -> your).
I see what you mean by the piping and being a bash script. I actually played with that a bit but didn't get any better results.
Unfortunately I can't get you a better representation of the file just because I'm at home right now and it's something I'm doing for work. I can do that tomorrow however.
In the meantime, you can assume the word "rewrite" on the lines that I want and that will be fine. Even better, the grep part I am not really concerned about, I know grep well and regex very well so we can just assume the grep is doing it's job or will be after I get awk under control. It's the awk I'm very new and raw on.
I must have copied the wrong version of the code for you to look at, or, your highlighter got it wrong (it happens quite a bit with shell scripts and quotes within quotes within...). In any case the part that goes
is necessary.
Basically it turns this [25/Jun/2014:18:57:07 --700] into this 25-Jun-2014 18:57:07 --700 which is then digestible by date so we can return a UNIX timestamp.
Obviously this means that both RefDate & LogDate are UNIX timestamps where RefDate is fixed for the whole script at some time set to "less than now", probably -1 hour so it will be: now-1 hour not now-2 seconds as I showed it.
The one liner was exactly as you said, too long, that's why I was putting it into a script and breaking it out into lines.
The first date variable only needs to be set once since it's compared against as a constant for all the lines processed. The second variable needs to be called for every line since the date is actually picked out of the line (fields 4 and 5 hence the echo $4 $5|....
I really want to get the awk stuff in it's own script file, that's the exercise, making that work.
Thinking of it like a bash script of piping then I would break it up into two files.
And this is really what I want to achieve, breaking the awk code out into it's own file for the sake of cleaner code but mostly so I learn how since it seems the syntax and rules change a bit once moved from the command line to a program file.
BTW, the print statements in the file will simply be:
Code:
if (LogDate > RefDate)
print $0;
What you see above is just so you can see the variables to know if they are getting set correctly.
Things I have noticed when moving the awk code into an awk program file:
no -v in front of variables
as you pointed out, the quotes make a line like RefDate="$(date -d'now-3 hours' '+%s')" stop working and I noticed that it doesn't like single quotes at all in the file.
no single quote around the whole thing
It's the variables I'm really stuck on. Perhaps on the one liner, even though they were after the awk they were still being interpreted at the shell script level. Or perhaps I should not have put them in the BEGIN {} area and they actually go somewhere else -- the second variable I suspect that's the case since BEGIN isn't the loop area and it needs to change for every line (the only reason I haven't moved it yet is that I was trying to get rid of the syntax errors first).
I agree there are probably more efficient ways to read the file into awk but it's fine for now. Best efficiency is not nearly as important as figuring this out and just making it work.
Hopefully this explains what I am attempting a little better.
Last edited by nybbles2bytes; 06-26-2014 at 05:28 AM..
Reason: adding more info.
I must have copied the wrong version of the code for you to look at, or, your highlighter got it wrong (it happens quite a bit with shell scripts and quotes within quotes within...). In any case the part that goes
is necessary.
Basically it turns this [25/Jun/2014:18:57:07 --700] into this 25-Jun-2014 18:57:07 --700 which is then digestible by date so we can return a UNIX timestamp.
... ... ...
NO! It does not. The command:
Code:
LogDate="$(echo "$4 $5" |
sed -e 's/\//-/g' -e 's/:/ /1' |
tr -d '[]' |
date '+%s')"
and the command:
Code:
LogDate="$(date '+%s')"
will both set LogDate to the current time as a integer value specifying the number of seconds since the Epoch. I repeat: The date utility does not read from standard input! The echo, sed, and tr utilities in that pipeline have absolutely no effect on the value returned other than that the time needed to start those three additional processes might delay response by a second or two depending on your hardware and system load.
I repeat: The code in red above is shell commands; not awk commands. Sticking shell commands inside the action section of a BEGIN clause in an awk script does not make them awk commands. THIS WILL NOT WORK!
I understand that you want to prepend a reference date and a reformatted version of the date from a line at the start of some lines in your log file. Processing would be hundreds to thousands of times faster per line if we can put the timestamps at the start of the line as YYYYMMDDHHmmSS instead of as seconds since the Epoch, but it isn't easy to get the seconds between the two from this format. I repeat: Do you NEED the number of seconds between these two timestamps on each line?
should produce exactly the same output as your 1-liner did. If you changed your 1-liner to use now-3 hours instead of now-2 second, you can also make that change. It will affect the timestamp printed in the awk output, but will not affect anything else. I can't verify that they are equivalent because date on my system does not support the -d option you're using, the input you have given me does not contain any lines that will be matched by your grep command, and the RE you're using with grep is not supported by the grep I'm using; but I'm sure they will produce the same output if given the same input at the same time.
It's well past my bedtime, so I'm going to bed. Please post a more complete sample of your log file along with an answer to the above question about the need for the difference between the two timestamps when you get to work later this morning and I'll take another look at it this afternoon.
Fair enough. I think I have learned enough to get exactly what I want now. This seems to set the variables exactly as they should be. Let me know if you see anything wrong.
Test output included (print statement is purely test data output) which also includes the input lines. I have also removed anything but the awk code for the sake of simplicity.
---------- Post updated at 07:19 PM ---------- Previous update was at 03:56 PM ----------
I saw what you meant by not piping to date so I've modified this to work the way date wants to work. So here's the code, as you can see I've pulled some functionality into th awk and got rid of the need for grep for the regex.
What I don't know how to do is get rid of using "date" through the loop as I am sure it's making it a very slow script by having to call it that way. One way I suppose would be to split the datetimes up and reorder them by YYYYMMDDHHMMSS and then just do a string compare but I'm not really sure how to do that in awk.
And of course any clues you have on whatever else might make it slow that we could change since I don't really know the ins and outs of awk.
Called with: sudo cat /var/log/httpd/rewrite.log|./monitor-rewrites.awk -v LookBack="10 hours"
Code:
#!/bin/awk -f
BEGIN{
#
# LookBack should be something like "1 hour" or "1 day" and is set on the command line with
# -v LookBack='1 hour'. Best to make it the same and the crontab set frequency for calling
# this script.
#
"date -d \"-"LookBack"\" \"+%s\""|getline RefDate;
close("date -d \"-"LookBack"\" \"+%s\"");
print "BEGIN";
}
{
"date -d \"$(echo "$4"|tr -d \"[]\"|sed -e \"s#/#-#g\" -e \"s/:/ /1\")\" \"+%s\""|getline LogDate;
close("date -d \"$(echo "$4"|tr -d \"[]\"|sed -e \"s#/#-#g\" -e \"s/:/ /1\")\" \"+%s\"");
#
# Uncomment the following for testing
# print "R:",RefDate, " L:",LogDate, " Diff:",(LogDate - RefDate), " Line: "$0;
#
if (RefDate < LogDate) {
if (/rewrite +.?(assets\/.+.? +-> +.?sites\/default\/files\/|sites\/default\/files\/.+.? +-> +.?assets\/)./) {
print $4, $5, $10, $11, $12, $13;
}
}
}
END {
print "END";
}
I'm trying to create a shell script that takes a awk script that I wrote and a filename as an argument. I was able to get that done but I'm having trouble figuring out how to keep the header of the output at the top but sort the rest of the rows alphabetically. This is what I have now but it is... (1 Reply)
Hi
I have a text file (Input.txt) with two column entries separated by tab as given below:
aaa str1
bbb str2
cccccc str3
dddd str4
eee str3
ssss str2
sdf str3
hhh str1
fff str2
ccc str3
.....
.....
..... (1 Reply)
I have a shell script (.sh) and I want to pass a parameter value to the awk command but I am getting exception, please assist.
diff=$1$2.diff
id=$2 new=new_$diff
echo "My id is $1"
echo "I want to sync for user account $id"
##awk command I am using is as below
cat $diff |... (1 Reply)
Discussion started by: Sarita Behera
1 Replies
4. Post Here to Contact Site Administrators and Moderators
Variable I have in my shell script
diff=$1$2.diff
id=$2
new=new_$diff
echo "My id is $1"
echo "I want to sync for user account $id"
##awk command I am using is as below
cat $diff | awk -F'~' ''$2 == "$id"' {print $0}' > $new
I could see value of $id is not passing to the awk... (0 Replies)
I am getting the following error while passing parameter to a shell script called within awk script. Any idea what's causing this issue and how to ix it ? Thanks
sh: -c: line 0: syntax error near unexpected token `newline'
sh: -c: line 0: `./billdatecalc.sh ... (10 Replies)
This is the final first release of the dynamic menu generator for pekwm (WM).
#!/bin/bash
function param_val {
awk "/^${1}=/{gsub(/^${1}="'/,""); print; exit}' $2
}
echo "Dynamic {"
for CF in `ls -c1 /usr/share/applications/*.desktop`
do
name=$(param_val Name $CF)
... (3 Replies)
hi everyone
i am trying to do this
bash> cat abc.sh
deepak()
{
echo Deepak
}
deepak
bash>./abc.sh
Deepak
so it is giving me write simply i created a func and it worked
now i modified it like this way
bash> cat abc.sh (2 Replies)
Hello,
I have this awk script that I want to execute by passing parameters through a shell script.
I'm a little confused. This awk script removes duplicates from an input file.
Ok, so I have a .sh file called rem_dups.sh
#!/usr/bin/sh... (4 Replies)
Hi guys
I have a shell script that executes sql statemets and sends the output to a file.the script takes in parameters executes sql and sends the result to an output file.
#!/bin/sh
echo " $2 $3 $4 $5 $6 $7
isql -w400 -U$2 -S$5 -P$3 << xxx
use $4
go
print"**Changes to the table... (0 Replies)
