Sponsored Content
Full Discussion: Help with awk script
Top Forums UNIX for Dummies Questions & Answers Help with awk script Post 302907122 by Don Cragun on Thursday 26th of June 2014 03:06:53 AM
Old 06-26-2014
Quote:
Originally Posted by nybbles2bytes
Hi Don,

Thanks for the reply.

Basically what I am doing is looking through an apache rewrite.log file and filtering out lines I don't want. I use awk to pre-filtering the lines by date so that the "...now-2 seconds..." part is just setting up a date/time so that I only look at lines up to 2 seconds old. That 2 seconds was just left over from testing. In reality it would 1 hour or something like that.

After awk filters the lines by date I run them through grep to get just the ones I am interested in by way of a regex.

To answer your more specific questions:
Code:
awk -v RefDate="$(date -d'now-2 second' '+%s')"

(Note: above is from the one liner, the -v is not on the multi-line version but is instead encased in braces with a begin, as in: BEGIN {...}.)

Yes, I see that there is a system call and that system call is the date command. However, this part of the line to me looks like it says "awk awk, I want you to make a variable -v called RefDate RefDate= and do so by getting the output of the system date function "$(date -d'now-2 second' '+%s')". Actually the '2 seconds' part was just testing, it reality it would be hours or a day.

In my mind this is an awk feature of using variables that just happens to in this case call a system command. So to me that part is an awk script with a bash sub-script, not a bash script with an awk sub-script.

However, the second part, where I am piping filtered lines from awk to grep, certainly that part is not an awk script. I guess that means all of that part must be outside the .awk file. I was hoping to just tag it onto the last line of the .awk file just as it's the last part of the one liner ...kind of hoping the same philosophy would following from the one liner to the awk program file but I guess that's simply not the case. Or is there a way?
Almost your entire problem is thinking that you have an awk script instead of a bash or ksh shell script. What you have is a shell pipeline where the first command in the pipeline happens to be awk. If your 1-liner works when typed into your shell from your terminal window, adding the full pathname of your shell after #! as the 1st line of your script and your 1-liner as the 2nd line of the script would work.

Note that the date utility is not a system call. I haven't seen any OS that has a date() system call. You may be thinking about the awk system() function which can be used to invoke a shell to run a command, but you don't want to invoke a shell from awk to process each line you're reading from a file. If you need to do that, you're probably wasting a lot of time by using awk instead of just reading the file directly in the shell.

Note that a 1-liner that long is impossible to read and update (assuming you may want to make changes later). So you should probably reformat your 1-liner into more readable chunks. You haven't said what shell you're using, so I'll use the shell I normally use (ksh) as an example. For this script ksh and bash are interchangeable:
Code:
#!/bin/ksh
awk -v RefDate="$(date -d'now-2 second' '+%s')" \
	-v LogDate="$(echo "$4 $5" |
		sed -e 's/\//-/g' -e 's/:/ /1' |
		tr -d '[]' |
		date '+%s')" '
{	if(LogDate > RefDate)
		print RefDate, LogDate, RefDate-LogDate, $0
}' "/var/log/httpd/rewrite.log" |
grep -B 1 -A 0 -inP '(rewrite\s+.?assets/.+.?\s+->\s+.?sites/default/files/.+.?|rewrite .?sites/default/files/.+.?\s+->\s+.?assets/.+.?)'

And, looking at this we can more easily see that all of the code marked in red above from your 1-liner is a no-op (the date utility does not read anything from its standard input; so the echo, sed, and tr commands don't do anything useful. And, the echo "$4 $5" is referring to the 4th and 5th arguments passed to your shell (not the 4th and 5th fields from a line in the file awk is reading, so that pipeline is just passing a <space> character followed by a <newline> character to date anyway).

So, this script can be further simplified to:
Code:
#!/bin/ksh
awk -v RefDate="$(date -d'now-2 second' '+%s')" \
	-v LogDate="$(date '+%s')" '
{	if(LogDate > RefDate)
		print RefDate, LogDate, RefDate-LogDate, $0
}' "/var/log/httpd/rewrite.log" |
grep -B 1 -A 0 -inP '(rewrite\s+.?assets/.+.?\s+->\s+.?sites/default/files/.+.?|rewrite .?sites/default/files/.+.?\s+->\s+.?assets/.+.?)'

and still produce the same output.
And, as long as RefDate is always set to now minus something and LogDate is always set to now, the script can be further simplified to:
Code:
#!/bin/ksh
awk -v RefDate="$(date -d'now-2 second' '+%s')" \
	-v LogDate="$(date '+%s')" '
{	print RefDate, LogDate, RefDate-LogDate, $0
}' "/var/log/httpd/rewrite.log" |
grep -B 1 -A 0 -inP '(rewrite\s+.?assets/.+.?\s+->\s+.?sites/default/files/.+.?|rewrite .?sites/default/files/.+.?\s+->\s+.?assets/.+.?)'

and still produce the same output. Note also that since the input line you showed us does not contain the string "rewrite", grep is not going to match any lines and you won't have any output. I, therefore, have to assume that there are other lines in your log file that are in a different format.

If you will show us a more representative sample of your input file, and be more clear about what you're really trying to do, I think I can help you reformat the way you set RefDate so we can compare that to the date and time in the 4th and 5th fields from lines in the format you showed us. But, unless the lines you're using to match with the grep have the same data in the 4th and 5th fields, you aren't going to get the output you want.

PLEASE show us some representative sample input AND show us the output you want to get from your script!

If you have the dates and times shown in human readable terms (instead of a seconds since the Epoch), does your output really need the number of seconds between the two? If you need that difference to be shown in seconds, your script will run MUCH slower.

Last edited by Don Cragun; 06-26-2014 at 04:12 AM.. Reason: Fix typo (you -> your).
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

create a shell script that calls another script and and an awk script

Hi guys I have a shell script that executes sql statemets and sends the output to a file.the script takes in parameters executes sql and sends the result to an output file. #!/bin/sh echo " $2 $3 $4 $5 $6 $7 isql -w400 -U$2 -S$5 -P$3 << xxx use $4 go print"**Changes to the table... (0 Replies)
Discussion started by: magikminox
0 Replies

2. Shell Programming and Scripting

want to pass parameters to awk script from shell script

Hello, I have this awk script that I want to execute by passing parameters through a shell script. I'm a little confused. This awk script removes duplicates from an input file. Ok, so I have a .sh file called rem_dups.sh #!/usr/bin/sh... (4 Replies)
Discussion started by: script_op2a
4 Replies

3. Shell Programming and Scripting

Call shell script function from awk script

hi everyone i am trying to do this bash> cat abc.sh deepak() { echo Deepak } deepak bash>./abc.sh Deepak so it is giving me write simply i created a func and it worked now i modified it like this way bash> cat abc.sh (2 Replies)
Discussion started by: aishsimplesweet
2 Replies

4. Shell Programming and Scripting

Help: How to convert this bash+awk script in awk script only?

This is the final first release of the dynamic menu generator for pekwm (WM). #!/bin/bash function param_val { awk "/^${1}=/{gsub(/^${1}="'/,""); print; exit}' $2 } echo "Dynamic {" for CF in `ls -c1 /usr/share/applications/*.desktop` do name=$(param_val Name $CF) ... (3 Replies)
Discussion started by: alexscript
3 Replies

5. Shell Programming and Scripting

Passing awk variable argument to a script which is being called inside awk

consider the script below sh /opt/hqe/hqapi1-client-5.0.0/bin/hqapi.sh alert list --host=localhost --port=7443 --user=hqadmin --password=hqadmin --secure=true >/tmp/alerts.xml awk -F'' '{for(i=1;i<=NF;i++){ if($i=="Alert id") { if(id!="") if(dt!=""){ cmd="sh someScript.sh... (2 Replies)
Discussion started by: vivek d r
2 Replies

6. Shell Programming and Scripting

Calling shell script within awk script throws error

I am getting the following error while passing parameter to a shell script called within awk script. Any idea what's causing this issue and how to ix it ? Thanks sh: -c: line 0: syntax error near unexpected token `newline' sh: -c: line 0: `./billdatecalc.sh ... (10 Replies)
Discussion started by: Sudhakar333
10 Replies

7. Post Here to Contact Site Administrators and Moderators

Unable to pass shell script parameter value to awk command in side the same script

Variable I have in my shell script diff=$1$2.diff id=$2 new=new_$diff echo "My id is $1" echo "I want to sync for user account $id" ##awk command I am using is as below cat $diff | awk -F'~' ''$2 == "$id"' {print $0}' > $new I could see value of $id is not passing to the awk... (0 Replies)
Discussion started by: Ashunayak
0 Replies

8. UNIX for Dummies Questions & Answers

Passing shell script parameter value to awk command in side the script

I have a shell script (.sh) and I want to pass a parameter value to the awk command but I am getting exception, please assist. diff=$1$2.diff id=$2 new=new_$diff echo "My id is $1" echo "I want to sync for user account $id" ##awk command I am using is as below cat $diff |... (1 Reply)
Discussion started by: Sarita Behera
1 Replies

9. Shell Programming and Scripting

awk script to call another script based on second column entry

Hi I have a text file (Input.txt) with two column entries separated by tab as given below: aaa str1 bbb str2 cccccc str3 dddd str4 eee str3 ssss str2 sdf str3 hhh str1 fff str2 ccc str3 ..... ..... ..... (1 Reply)
Discussion started by: my_Perl
1 Replies

10. UNIX for Beginners Questions & Answers

Shell script to call and sort awk script and output

I'm trying to create a shell script that takes a awk script that I wrote and a filename as an argument. I was able to get that done but I'm having trouble figuring out how to keep the header of the output at the top but sort the rest of the rows alphabetically. This is what I have now but it is... (1 Reply)
Discussion started by: Eric7giants
1 Replies
All times are GMT -4. The time now is 07:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy