Apache log file pharsing, need help!!!!


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Apache log file pharsing, need help!!!!
# 1  
Old 10-22-2007
Apache log file pharsing, need help!!!!

I`m new to shell scripting and I need some help here

I`m trying to pharse Apache log and I encountered a problem so I need some help...

How to break line into fields having different field separators?

let`s say I want to break line into 9 fields

and the lines format is:

text1 text2 text3 [text4] "text5" text6 text7 "text8" " text9"

so teh fields would be:
field1 = text1
field2 = text2
field3 = text3
field4 = text4
field5 = text5
field6 = text6
field7 = text7
field8 = text8
field9 = text9

example:

88.118.118.214 - - [22/Oct/2007:20:15:13 +0300] "GET /~tomasa/kg/data/uzd.html HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

field1 = 88.118.118.214
field2 = -
field3 = -
field4 = 22/Oct/2007:20:15:13 +0300
field5 = GET /~tomasa/kg/data/uzd.html HTTP/1.1
field6 = 304
field7 = -
field8 = -
field9 = Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)

Any thoughts how can I do that (I know I should use awk for this task but how to change field separator individual field?)???

Any help i wellcome. Thank you in advance.
# 2  
Old 10-24-2007
awk

Hi,

A little complex, but it should be ok. I have tested it on solaris.

input:
Code:
88.118.118.214 - - [22/Oct/2007:20:15:13 +0300] "GET /~tomasa/kg/data/uzd.html HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
text1 text2 text3 [text4] "text5" text6 text7 "text8" "text9"

output:
Code:
field1= 88.118.118.214
field2= -
field3= -
field4= 22/Oct/2007:20:15:13 +0300
field5= GET /~tomasa/kg/data/uzd.html HTTP/1.1
field6= 304
field7= -
field8= -
field9= Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
field1= text1
field2= text2
field3= text3
field4= text4
field5= text5
field6= text6
field7= text7
field8= text8
field9= text9

code:
Code:
nawk '
function getVal(str,pos,sta,end)
{
	for(i=1;i<=pos;i++)
	{
		str=substr(str,index(str,sta)+1)
	}
	a=substr(str,1,index(str,end)-1)
	return a
}
{
print "field1= "$1
print "field2= "$2
print "field3= "$3
print "field4= "getVal($0,1,"[","]")
print "field5= "getVal($0,1,"\"","\"")
print "field6= "getVal(getVal($0,2,"\"","\""),1," "," ")
print "field7= "getVal(getVal($0,2,"\"","\""),2," "," ")
print "field8= "getVal($0,3,"\"","\"")
print "field9= "getVal($0,5,"\"","\"")
}' a

# 3  
Old 03-04-2009
Code:
#!/usr/bin/perl
my $str='88.118.118.214 - - [22/Oct/2007:20:15:13 +0300] "GET /~tomasa/kg/data/uzd.html HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"';
$str=~s/\[([^\]]*)\]/"$1"/g;
$str=~s/ (?=[^"]*$|([^"]*"[^"]*"[^"]*)*$)/\n/g;
my @arr=split("\n",$str);
for(my $i=0;$i<=$#arr;$i++){
	$arr[$i]=~s/"//g;
	print "field",$i+1," = ",$arr[$i],"\n";
}

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Parse apache log file with three different time formats

Hi, I want to parse below file and Write a function to extract the logs between two given timestamp. Apache (Unix) Log Samples - MonitorWare The challenge here is there are three date and time format. First :- 07/Mar/2004:16:05:49 Second :- Sun Mar 7 16:02:00 2004 Third :- 29-Mar... (6 Replies)
Discussion started by: sahil_shine
6 Replies

2. Red Hat

Apache log rotate configuration

HI i was trying to configure logrotate for my apache server and it's not working properly. Os: Red Hat 6 here is my lodrotate configuration /var/log/httpd/*log { daily missingok notifempty sharedscripts compress delaycompress postrotate ... (3 Replies)
Discussion started by: bentech4u
3 Replies

3. UNIX for Advanced & Expert Users

Apache log rotate configuration

HI i was trying to configure logrotate for my apache server and it's not working properly. here is my lodrotate configuration /var/log/httpd/*log { daily missingok notifempty sharedscripts compress delaycompress postrotate /sbin/service httpd... (1 Reply)
Discussion started by: bentech4u
1 Replies

4. Shell Programming and Scripting

Script that watches an apache log file

i'm trying to write a basic script that "watches" the apache access.log file and prints out lines that correspond to slow requests, Im checking for how microseconds it's taken. For the sake of testing, i'm using any number field. so far I have:- watch --interval=1 "tail access.log | cut -d '... (3 Replies)
Discussion started by: ssaini2014
3 Replies

5. Web Development

Apache/2.2.15 custom error log

Hello, I've updated my apache access log to include the x-forward-for IP instead of my client(loadbalancer) ip. However, i can't seem to find a way to do the same for the error logs. Can someone please assist. Thank you. -K (0 Replies)
Discussion started by: kmaq7621
0 Replies

6. Linux

Generating apache log reports

Hello all, I'm trying to find some tool on generating reports based on apache access_log files (of Common format). I found some of them (awstats, lire/logreport, weblog expert, apache logs viewer, etc..) but they generate some global and general report about the log file. Also some perl... (0 Replies)
Discussion started by: enux
0 Replies

7. Web Development

how to wait after apache log rotation

My solaris server utilize the freeware savelog program to rotate apache logs. One server has become very busy and even after doing a graceful restart it continues to log to the saved gzip log file. Has anyone been able to come up with a way or script to issue a "wait" type command so that the... (2 Replies)
Discussion started by: csross
2 Replies

8. Web Development

Apache log with long strings of Xs in GET request

Hi everybody, I was looking at my apache2 log and I found GET requests as such: Some ip - - "GET... (4 Replies)
Discussion started by: z1dane
4 Replies

9. Solaris

Apache localhost-access.log

The localhost-access.log has a size 3gb. What can apache2 break log on the parts 300mb, or the other issue, make log every week and index it with prifix current date(localhost-access_date.log)? Please help. (3 Replies)
Discussion started by: sotich82
3 Replies
Login or Register to Ask a Question