Sponsored Content
Full Discussion: Web log analysis
Special Forums UNIX and Linux Applications Web log analysis Post 302497953 by treesloth on Friday 18th of February 2011 01:13:47 PM
Old 02-18-2011
Web log analysis

I'm looking for good web log analysis software for 95% Apache / 5% IIS logs. There's a lot out there, but our needs are unusual. We're a web hosting company with a large number of VPS servers. For a variety of reasons, we like to develop models of virtual server activity. Among other things, this means obtaining Apache logs from a large number of virtual servers and analyzing them for all sorts of characteristics. Each virtual server maintains its own set of Apache logs. These logs are pulled to a central analysis server and analyzed.

We're not really interested in analyzing in a domain-specific way. We're much more interested in creating composite reports on each of many virtual servers. We can then create a "typical virtual server" model from such reports.

The following would be very useful:

1) The ability to calculate mean and standard deviation of page size. This is far from an exact science, of course, but statistically-reasonable estimates would be nice.
2) Same as 1), but for number of elements per page.
3) Transaction rates, preferably per minute or second.
4) All the other "stuff" that's common to most log analyzers.

Are there any packages that can do what I've described here?
 

6 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Problems with Perl/KSH Web Log Script

Hi, I am writing a series of scripts for work to analyse intranet access logs. All of the scripts do as they should when run individually from the shell, but only when run from certain directories. This sounds like it may be a PATH issue but I am not sure. When I run a certain script, say... (3 Replies)
Discussion started by: mmanders
3 Replies

2. UNIX for Advanced & Expert Users

WEB Server Log File Analysis using awk/sed/grep

I'm trying to find a way to show large page sizes (page size in K) from multiple web server log files. Essentially I want to show only rows from a file where a specific column is larger than some value. Has anyone ever done this type of log analysis? If so, a snippet of code would be very... (2 Replies)
Discussion started by: mike_cataldo@ad
2 Replies

3. Shell Programming and Scripting

Log File Analysis

Hi All, Given a list of date and time, how can i split them in intervals of 1 hour. The input file is listed as below f.e.: 16-OCT-2009 09:11:47 10.65.4.24. . . (1 Reply)
Discussion started by: x-plicit78
1 Replies

4. Homework & Coursework Questions

Parse a Web Server Access Log

1. The problem statement, all variables and given/known data: Write a parser for a web server access log that will provide the statistics outlined below. Remember to format your output in a neat form. You may complete this assignment with one Awk script or a shell script using a combination of... (6 Replies)
Discussion started by: codyhazelwood
6 Replies

5. Shell Programming and Scripting

Log Analysis with AWK with Time difference

I would like to write a shell script that calculated the time difference bettween the log entries. If the time difference is higher as 200 sec. print the complette lines out. My Problem is, i am unable to jump in the next line and calculate the time difference. Thank you for your Help. ... (5 Replies)
Discussion started by: fabian3010
5 Replies

6. OS X (Apple)

Web Request log

Hey all, I would like to track what applications are requesting access at the unix level. is there a log or a way to capture all network access requests? Not with applications. (1 Reply)
Discussion started by: Phorn
1 Replies
clfsplit(1)							     logtools							       clfsplit(1)

NAME
clfsplit - split Common-Log Format web logs based on IP address SYNOPSIS
clfsplit [--help] [-i input] -d defaultfile -f file -s spec [-f file -s spec] DESCRIPTION
The clfsplit will split up large CLF format web logs based on IP address. This is for creating separate log analysis passes for internal and external users of web pages. OVERVIEW
The defaultfile parameter specifies where data goes if it doesn't match any of the IP ranges. This could be /dev/null depending on your aims. The -i input parameter gives the file to take input from (default standard input). The -f file parameter must be given before the list of IP addresses. The spec parameter is the IP addresses that go to the file in question. It is of the form start[-end][:start[-end]] where start and end specify the start and ends of ranges of IPs. Also the CIDR notation can be used or a single IP address. If there is a large number of IP ranges then a file name can be given which contains a set of IP ranges, one range per line. EXIT STATUS
0 No errors 1 Bad parameters 2 Can't open input 3 Can't open/write to output file 4 Can't open and read from spec file AUTHOR
This program, its manual page, and the Debian package were written by Russell Coker <russell@coker.com.au>. SEE ALSO
clfmerge(1),clfdomainsplit(1) Russell Coker <;russell@coker.com.au> 0.06 clfsplit(1)
All times are GMT -4. The time now is 11:04 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy