Sponsored Content
Full Discussion: Help with awk script
Top Forums UNIX for Dummies Questions & Answers Help with awk script Post 302907132 by Don Cragun on Thursday 26th of June 2014 05:12:41 AM
Old 06-26-2014
Quote:
Originally Posted by nybbles2bytes
... ... ...

I must have copied the wrong version of the code for you to look at, or, your highlighter got it wrong (it happens quite a bit with shell scripts and quotes within quotes within...). In any case the part that goes
Code:
echo "$4 $5" |
        sed -e 's/\//-/g' -e 's/:/ /1' |
        tr -d '[]' |

is necessary.
Basically it turns this [25/Jun/2014:18:57:07 --700] into this 25-Jun-2014 18:57:07 --700 which is then digestible by date so we can return a UNIX timestamp.

... ... ...
NO! It does not. The command:
Code:
LogDate="$(echo "$4 $5" |
		sed -e 's/\//-/g' -e 's/:/ /1' |
		tr -d '[]' |
		date '+%s')"

and the command:
Code:
LogDate="$(date '+%s')"

will both set LogDate to the current time as a integer value specifying the number of seconds since the Epoch. I repeat: The date utility does not read from standard input! The echo, sed, and tr utilities in that pipeline have absolutely no effect on the value returned other than that the time needed to start those three additional processes might delay response by a second or two depending on your hardware and system load.
Quote:
Originally Posted by nybbles2bytes
Code:
BEGIN {
  RefDate="$(date -d'now-3 hours' '+%s')" 
  LogDate="$(echo "$4 $5"|sed -e 's/\//-/g' -e 's/:/ /1'|tr -d '[]'|date '+%s')" 
}
{
  if (LogDate > RefDate) 
    print RefDate, LogDate, RefDate-LogDate, $0; 
}

I repeat: The code in red above is shell commands; not awk commands. Sticking shell commands inside the action section of a BEGIN clause in an awk script does not make them awk commands. THIS WILL NOT WORK!

I understand that you want to prepend a reference date and a reformatted version of the date from a line at the start of some lines in your log file. Processing would be hundreds to thousands of times faster per line if we can put the timestamps at the start of the line as YYYYMMDDHHmmSS instead of as seconds since the Epoch, but it isn't easy to get the seconds between the two from this format. I repeat: Do you NEED the number of seconds between these two timestamps on each line?

The script that I posted:
Code:
#!/bin/ksh
awk -v RefDate="$(date -d'now-2 second' '+%s')" \
	-v LogDate="$(date '+%s')" '
{	print RefDate, LogDate, RefDate-LogDate, $0
}' "/var/log/httpd/rewrite.log" |
grep -B 1 -A 0 -inP '(rewrite\s+.?assets/.+.?\s+->\s+.?sites/default/files/.+.?|rewrite .?sites/default/files/.+.?\s+->\s+.?assets/.+.?)'

should produce exactly the same output as your 1-liner did. If you changed your 1-liner to use now-3 hours instead of now-2 second, you can also make that change. It will affect the timestamp printed in the awk output, but will not affect anything else. I can't verify that they are equivalent because date on my system does not support the -d option you're using, the input you have given me does not contain any lines that will be matched by your grep command, and the RE you're using with grep is not supported by the grep I'm using; but I'm sure they will produce the same output if given the same input at the same time.

It's well past my bedtime, so I'm going to bed. Please post a more complete sample of your log file along with an answer to the above question about the need for the difference between the two timestamps when you get to work later this morning and I'll take another look at it this afternoon.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

create a shell script that calls another script and and an awk script

Hi guys I have a shell script that executes sql statemets and sends the output to a file.the script takes in parameters executes sql and sends the result to an output file. #!/bin/sh echo " $2 $3 $4 $5 $6 $7 isql -w400 -U$2 -S$5 -P$3 << xxx use $4 go print"**Changes to the table... (0 Replies)
Discussion started by: magikminox
0 Replies

2. Shell Programming and Scripting

want to pass parameters to awk script from shell script

Hello, I have this awk script that I want to execute by passing parameters through a shell script. I'm a little confused. This awk script removes duplicates from an input file. Ok, so I have a .sh file called rem_dups.sh #!/usr/bin/sh... (4 Replies)
Discussion started by: script_op2a
4 Replies

3. Shell Programming and Scripting

Call shell script function from awk script

hi everyone i am trying to do this bash> cat abc.sh deepak() { echo Deepak } deepak bash>./abc.sh Deepak so it is giving me write simply i created a func and it worked now i modified it like this way bash> cat abc.sh (2 Replies)
Discussion started by: aishsimplesweet
2 Replies

4. Shell Programming and Scripting

Help: How to convert this bash+awk script in awk script only?

This is the final first release of the dynamic menu generator for pekwm (WM). #!/bin/bash function param_val { awk "/^${1}=/{gsub(/^${1}="'/,""); print; exit}' $2 } echo "Dynamic {" for CF in `ls -c1 /usr/share/applications/*.desktop` do name=$(param_val Name $CF) ... (3 Replies)
Discussion started by: alexscript
3 Replies

5. Shell Programming and Scripting

Passing awk variable argument to a script which is being called inside awk

consider the script below sh /opt/hqe/hqapi1-client-5.0.0/bin/hqapi.sh alert list --host=localhost --port=7443 --user=hqadmin --password=hqadmin --secure=true >/tmp/alerts.xml awk -F'' '{for(i=1;i<=NF;i++){ if($i=="Alert id") { if(id!="") if(dt!=""){ cmd="sh someScript.sh... (2 Replies)
Discussion started by: vivek d r
2 Replies

6. Shell Programming and Scripting

Calling shell script within awk script throws error

I am getting the following error while passing parameter to a shell script called within awk script. Any idea what's causing this issue and how to ix it ? Thanks sh: -c: line 0: syntax error near unexpected token `newline' sh: -c: line 0: `./billdatecalc.sh ... (10 Replies)
Discussion started by: Sudhakar333
10 Replies

7. Post Here to Contact Site Administrators and Moderators

Unable to pass shell script parameter value to awk command in side the same script

Variable I have in my shell script diff=$1$2.diff id=$2 new=new_$diff echo "My id is $1" echo "I want to sync for user account $id" ##awk command I am using is as below cat $diff | awk -F'~' ''$2 == "$id"' {print $0}' > $new I could see value of $id is not passing to the awk... (0 Replies)
Discussion started by: Ashunayak
0 Replies

8. UNIX for Dummies Questions & Answers

Passing shell script parameter value to awk command in side the script

I have a shell script (.sh) and I want to pass a parameter value to the awk command but I am getting exception, please assist. diff=$1$2.diff id=$2 new=new_$diff echo "My id is $1" echo "I want to sync for user account $id" ##awk command I am using is as below cat $diff |... (1 Reply)
Discussion started by: Sarita Behera
1 Replies

9. Shell Programming and Scripting

awk script to call another script based on second column entry

Hi I have a text file (Input.txt) with two column entries separated by tab as given below: aaa str1 bbb str2 cccccc str3 dddd str4 eee str3 ssss str2 sdf str3 hhh str1 fff str2 ccc str3 ..... ..... ..... (1 Reply)
Discussion started by: my_Perl
1 Replies

10. UNIX for Beginners Questions & Answers

Shell script to call and sort awk script and output

I'm trying to create a shell script that takes a awk script that I wrote and a filename as an argument. I was able to get that done but I'm having trouble figuring out how to keep the header of the output at the top but sort the rest of the rows alphabetically. This is what I have now but it is... (1 Reply)
Discussion started by: Eric7giants
1 Replies
All times are GMT -4. The time now is 02:00 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy