Shell Programming and Scripting

I am trying to delete lines in archived Apache httpd logs

Each line has the pattern:

<ip-address> - - <date-time> <document-request-URL> <http-response> <size-of-req'd-doc> <referring-document-URL>

This pattern is shown in the example of 6 lines from the log in the code box below. These 6 lines are in a row, and all start with same IP address. These are actually the first 6 lines of about 20-25 lines, in which a document was served with multiple gif images.

My purpose is to delete lines under the following conditions:

First, determine whether a line in the log becomes a "reference line"
This happens if the line being tested ("subsequent line")
Note that a "subsequent line" is one that does not qualify to become a "reference line."

I have only one sed command line now, basically the regular expression to correspond to the pattern which identifies the line and parenthesized expressions/fields in the line. To be safe, I am using old-style regular expression syntax and not any "extended" kind, such as using `\d` metacharacters to indicate digits

Code:

/(([0-9]{1,3}\.){3}[0-9]{1,3}).*([0-9]{2}\/[a-zA-Z]{3}\/[0-9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2}) \"GET (.*) HTTP\/1\.1\" [0-9]{3} [0-9]{1,} \"http:\/\/my\.website\.org\//

I have wrapped three fields: the IP address, Date, requested document (between GET and HTTP/1.1), in parentheses.

This becomes the line to be tested: the tests are to do as above. 1) check for IP address difference, (2) check for time difference, (3) examine the requested docfield for file types ( gif | ico | css | js | png | jpg | jpeg | etc ) basically if they are not html, they get deleted (4) make sure the hostname/server name in the referring document is 'http://my.website.org/'

I am thinking that I need to use the Hold and eXchange pattern system, but am not sure how to go about that. More importantly, I must do comparisons on the text, converting date/time expressions into integers to be compared, and more doing string comparisons. The sed utility, as far as I know, has no built-in features for this, so I may have to pass these as parameters to a shell (?) to do the comparisons and return a result that sed can work with.

I have even more of a challenge too: see the NB below.

What I need is a good pointer or reference to what I should be telling sed to do, aside from just being given the answer. Thanks.

Code:

[line lengths broken up to avoid an annoying presentation]

172.16.77.182 - - [18/Sep/2012:20:48:16 +0300] "GET /reference/imagesHistoHTML/ethyl-eosin.gif HTTP/1.1" 200 3300 "http://www.google.com/imgres?hl=en&sa=X&rlz=1C1CHKZ_enUS433US433&biw=1280&
   bih=670&tbm=isch&prmd=imvns&tbnid=fEILNrTkTl2MzM:&imgrefurl=http://my.website.org/reference/histo.html&docid=r1tvojxQaLyVFM&imgurl=http://my.website.org/reference/imagesHistoHTML/ethyl-eosin.gif
   &w=366&h=265&ei=ybNYUPD5CNSO0QHoqYG4BQ&zoom=1&iact=hc&vpx=635&vpy=80&dur=2959&hovh=191&hovw=264&tx=168&ty=103&sig=114532110125230912831&page=1&tbnh=120&tbnw=166&start=0&
   ndsp=18&ved=1t:429,r:15,s:0,i:123" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
172.16.77.182 - - [18/Sep/2012:20:48:17 +0300] "GET /reference/style/std.css HTTP/1.1" 200 5429 "http://my.website.org/reference/histo.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 
   (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
172.16.77.182 - - [18/Sep/2012:20:48:17 +0300] "GET /style/std.css HTTP/1.1" 200 5429 "http://my.website.org/reference/histo.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) 
    Chrome/21.0.1180.89 Safari/537.1"
172.16.77.182 - - [18/Sep/2012:20:48:16 +0300] "GET /reference/histo.html HTTP/1.1" 200 118818 "http://www.google.com/imgres?hl=en&sa=X&rlz=1C1CHKZ_enUS433US433&biw=1280&bih=670&tbm=isch&
     prmd=imvns&tbnid=fEILNrTkTl2MzM:&imgrefurl=http://my.website.org/reference/histo.html&docid=r1tvojxQaLyVFM&imgurl=http://my.website.org/reference/imagesHistoHTML/ethyl-eosin.gif
    &w=366&h=265&ei=ybNYUPD5CNSO0QHoqYG4BQ&zoom=1&iact=hc&vpx=635&vpy=80&dur=2959&hovh=191&hovw=264&tx=168&ty=103&sig=114532110125230912831&page=1&tbnh=120
    &tbnw=166&start=0&ndsp=18&ved=1t:429,r:15,s:0,i:123" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
172.16.77.182 - - [18/Sep/2012:20:48:18 +0300] "GET /reference/imagesHistoHTML/dichlorotriazinyl.gif HTTP/1.1" 200 1406 "http://my.website.org/reference/histo.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) 
    AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"
172.16.77.182 - - [18/Sep/2012:20:48:19 +0300] "GET /reference/imagesHistoHTML/nitroso%20dye%20structure.gif HTTP/1.1" 200 2259 "http://my.website.org/reference/histo.html" "Mozilla/5.0
    (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1"

(to respect privacy of those accessing the server, I changed IP address to a recognized private LAN address [I hope])


NB: I am running this sed script as sed.exe (GNU sed version 4.2.1 (c) 2009) under Microsoft Windows 7, thus solutions requiring use of a shell should be a shell command processor installed (MS cmd version 6..7601 or or installable within Windos 7. I am aware that I can process the text of the logs within a VM running a Linux distro (I have, for instance, Ubuntu and TinyCore Linux installed as VMs), but (1) I have not kicked the MS Windows environment as an every-day use system and (2) my facility in bash scripting was more than a decade ago.

Sed scripts that handle multiple lines usually have a different flavor -- I like to call them loopers. My sed to remove extra blank lines in a row:

Code:

sed '
  :loop
  $b
  N
  s/^\n$//
  t loop
  P
  s/.*\n//
  t loop
 '