Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using awk with the date command and escape characters

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using awk with the date command and escape characters
# 1  
Old 06-19-2010
Using awk with the date command and escape characters
I have a file that is a log file for web traffic. I would like to convert the timestamp in it to unix time or epoch time.

I am using the date command in conjunction with awk to try to do this. Just

myfile:
Code:
28/Aug/1995:00:00:38 1 /pub/atomicbk/catalog/home.gif 813
28/Aug/1995:00:00:38 1 /pub/atomicbk/catalog/catalog.gif 693
28/Aug/1995:00:00:39 1 /pub/atomicbk/catalog/laser.gif 129
28/Aug/1995:00:00:40 1 /pub/atomicbk/catalog/logo2.gif 12871

I can run the date command like this:
Code:
date -j -f "%d/%b/%Y:%T" 28/Aug/1995:00:00:38[/FONT][FONT=Courier New] +%s

And I get the intended result:
Code:
809593238

(809593230 seconds since Jan 1, 1970)

So myfile above, the expected output should be:
Code:
809593238 1 /pub/atomicbk/catalog/home.gif 813
809593238 1 /pub/atomicbk/catalog/catalog.gif 693
809593239 1 /pub/atomicbk/catalog/laser.gif 129
809593240 1 /pub/atomicbk/catalog/logo2.gif 12871

I'm close but I'm getting an error.

I run the following command:
Code:
cat myfile | awk '{ date -j -f "%d/%b/%Y:%T" print $1 echo +%s }'

The error I get is:
Code:
awk: syntax error at source line 1
 context is
    { date -j -f "%d/%b/%Y:%T" >>>  print <<<  $1 echo " +%s" }
awk: illegal statement at source line 1

I've tried various combinations of this but with no success. I think it could be because 1) commands like date may not work inside an awk statement and 2) I need to properly escape the '+' and the '%' inside an awk statement so it can be passed on to the date command correctly. Any help is much appreciated.

Thanks!
Last edited by Scott; 06-19-2010 at 11:10 PM.. Reason: Please use code tags
# 2  
Old 06-20-2010
You need a bit more code to make it work with awk Smilie.

Anyway, I would use Perl:

Code:
perl -MTime::Local -lane'BEGIN {
  $month{$_} = ++ $c 
    for "JANFEBMARAPRMAYJUNJULAUGSEPOCTNOVDEC" =~ /.../g
  }
  @dt = split /[\/:]/, $F[0];
  $F[0] = timelocal $dt[5], $dt[4], $dt[3], 
    $dt[0], $month{uc $dt[1]} - 1 , $dt[2] - 1900;
  print join " ", @F;
  ' infile

This a quick and dirty solution, there are non-standard modules on CPAN that will parse the date for you and your code will be more robust.
This User Gave Thanks to radoulov For This Post:
jontjioe
# 3  
Old 06-20-2010
Or shell:
Code:
while read fdate rest; do 
  echo $(date -j -f "%d/%b/%Y:%T" $fdate +%s) "$rest"
done < myfile

This User Gave Thanks to Scrutinizer For This Post:
jontjioe
# 4  
Old 06-20-2010
Thank you so much!

Both the perl and the while read solution worked great! I'll let you know if one runs faster than the other as I will be running this on approximately 1 million lines.

---------- Post updated at 01:02 PM ---------- Previous update was at 08:49 AM ----------

Ok, so I ran both the Perl and the "while read" code blocks on a 309MB log file that had 6,397,194 lines. While both code blocks performed the same end result, the Perl block of code performed exponentially faster, so I had to use that instead of the while read. I waited about 10 minutes on the "while read" block and it was still going. The Perl block finished under 3 minutes. So I ended up using the Perl block in my script.

Thank you both for your help!
This User Gave Thanks to jontjioe For This Post:
Scrutinizer
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Escape characters in a variable

Debian 9 64x - LXDE How can i disable escape sequences in a variable? #!/bin/bash #mainscript . "./links.bash" echo "$red_start This text should be red $color_end"#!/bin/bash #links.bash #colors red_start="\eOutput that i get: \e Output expected: This text should be... (5 Replies)
Discussion started by: int3g3r
5 Replies

2. Shell Programming and Scripting

How to escape all special characters?

I have an application which I am integrating with that accepts the password via a CLI. I am running in to issues with passwords that contain special characters. I tried to escape them all, but I ran in to an issue where I cannot escape the characters ' ] My attempt is as follows: $... (2 Replies)
Discussion started by: AMG1978
2 Replies

3. Shell Programming and Scripting

Escape characters

i am executing script from A server which will execute the script in B server , as below. ssh A 'ssh B echo 'select * from testing where name ='test'' i am getting the below output. select * from testing where name=test but i need the output where clause with quotes , tried with... (3 Replies)
Discussion started by: expert
3 Replies

4. Shell Programming and Scripting

Replace special characters with Escape characters?

i need to replace the any special characters with escape characters like below. test!=123-> test\!\=123 !@#$%^&*()-= to be replaced by \!\@\#\$\%\^\&\*\(\)\-\= (8 Replies)
Discussion started by: laknar
8 Replies

5. Shell Programming and Scripting

awk print $1 escape all special characters

I'm using awk '{print $1}' and it works most of the time to print the contents of a mysql query loop, but occationally I get a field with some special character in it, is there a way to tell awk to ignore all special characters between my FS? I have >186K records, so building a list of ALL special... (6 Replies)
Discussion started by: unclecameron
6 Replies

6. UNIX for Dummies Questions & Answers

Escape Characters on various shells

Hi, I want to know if escape charaters work on all the popular UNIX shells. More specifically I want to know if echo "\c" will work on most of the UNIX shells and are there any specific shells on which \c won't work. Please help. Thanks, Vineet (2 Replies)
Discussion started by: vineetd
2 Replies

7. Shell Programming and Scripting

Searching for escape characters

Hi all I have been trying to write a script to look for a set of specific escape characters in a file. On viewing the file via vi it shows this : ^ I understand this means no end of line. I have tried a vary of grep parameters such as grep ^\^. filename grep --binary-file=binary without... (8 Replies)
Discussion started by: timcs
8 Replies

8. Shell Programming and Scripting

escape characters..

hey i want to know the unix commands to replace all the character escape sequences with their "C" values in a string... thanks in advance..! Regards, Sharanya (9 Replies)
Discussion started by: sharsin2001
9 Replies

9. Shell Programming and Scripting

number of escape characters?

Hi, I am trying to execute the following command from a batch script, but no matter how many escape characters I put in it doesn't execute properly. It works fine from the command line with quotes around the -exec part. #!/bin/sh /usr/local/bin/sudo /usr/atria/bin/cleartool setview -exec... (0 Replies)
Discussion started by: Sebarry
0 Replies

10. UNIX for Advanced & Expert Users

lp FormFeed Escape characters

I'm trying to modify the /usr/lib/lp/model/netstandard file to generate a header for all the print jobs that are sent, but there is no formfeed defined so the the job prints right after the header with no page break. What is the sequence I need in order to generate a formfeed? Or, do you have... (4 Replies)
Discussion started by: jgordon
4 Replies
Login or Register to Ask a Question