Sponsored Content
Full Discussion: Web log analysis
Special Forums UNIX and Linux Applications Web log analysis Post 302497953 by treesloth on Friday 18th of February 2011 01:13:47 PM
Old 02-18-2011
Web log analysis

I'm looking for good web log analysis software for 95% Apache / 5% IIS logs. There's a lot out there, but our needs are unusual. We're a web hosting company with a large number of VPS servers. For a variety of reasons, we like to develop models of virtual server activity. Among other things, this means obtaining Apache logs from a large number of virtual servers and analyzing them for all sorts of characteristics. Each virtual server maintains its own set of Apache logs. These logs are pulled to a central analysis server and analyzed.

We're not really interested in analyzing in a domain-specific way. We're much more interested in creating composite reports on each of many virtual servers. We can then create a "typical virtual server" model from such reports.

The following would be very useful:

1) The ability to calculate mean and standard deviation of page size. This is far from an exact science, of course, but statistically-reasonable estimates would be nice.
2) Same as 1), but for number of elements per page.
3) Transaction rates, preferably per minute or second.
4) All the other "stuff" that's common to most log analyzers.

Are there any packages that can do what I've described here?
 

6 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Problems with Perl/KSH Web Log Script

Hi, I am writing a series of scripts for work to analyse intranet access logs. All of the scripts do as they should when run individually from the shell, but only when run from certain directories. This sounds like it may be a PATH issue but I am not sure. When I run a certain script, say... (3 Replies)
Discussion started by: mmanders
3 Replies

2. UNIX for Advanced & Expert Users

WEB Server Log File Analysis using awk/sed/grep

I'm trying to find a way to show large page sizes (page size in K) from multiple web server log files. Essentially I want to show only rows from a file where a specific column is larger than some value. Has anyone ever done this type of log analysis? If so, a snippet of code would be very... (2 Replies)
Discussion started by: mike_cataldo@ad
2 Replies

3. Shell Programming and Scripting

Log File Analysis

Hi All, Given a list of date and time, how can i split them in intervals of 1 hour. The input file is listed as below f.e.: 16-OCT-2009 09:11:47 10.65.4.24. . . (1 Reply)
Discussion started by: x-plicit78
1 Replies

4. Homework & Coursework Questions

Parse a Web Server Access Log

1. The problem statement, all variables and given/known data: Write a parser for a web server access log that will provide the statistics outlined below. Remember to format your output in a neat form. You may complete this assignment with one Awk script or a shell script using a combination of... (6 Replies)
Discussion started by: codyhazelwood
6 Replies

5. Shell Programming and Scripting

Log Analysis with AWK with Time difference

I would like to write a shell script that calculated the time difference bettween the log entries. If the time difference is higher as 200 sec. print the complette lines out. My Problem is, i am unable to jump in the next line and calculate the time difference. Thank you for your Help. ... (5 Replies)
Discussion started by: fabian3010
5 Replies

6. OS X (Apple)

Web Request log

Hey all, I would like to track what applications are requesting access at the unix level. is there a log or a way to capture all network access requests? Not with applications. (1 Reply)
Discussion started by: Phorn
1 Replies
Catalyst::Manual::Deployment::FastCGI(3pm)		User Contributed Perl Documentation		Catalyst::Manual::Deployment::FastCGI(3pm)

NAME
Catalyst::Manual::Deployment::FastCGI - Deploying Catalyst with FastCGI FastCGI Deployment FastCGI is a high-performance extension to CGI. It is suitable for production environments, and is the standard method for deploying Catalyst in shared hosting environments. Pros Speed FastCGI performs equally as well as mod_perl. Don't let the 'CGI' fool you; your app runs as multiple persistent processes ready to receive connections from the web server. App Server When using external FastCGI servers, your application runs as a standalone application server. It may be restarted independently from the web server. This allows for a more robust environment and faster reload times when pushing new app changes. The frontend server can even be configured to display a friendly "down for maintenance" page while the application is restarting. Load-balancing You can launch your application on multiple backend servers and allow the frontend web server to perform load-balancing among all of them. And of course, if one goes down, your app continues to run. Multiple versions of the same app Each FastCGI application is a separate process, so you can run different versions of the same app on a single server. Can run with threaded Apache Since your app is not running inside of Apache, the faster mpm_worker module can be used without worrying about the thread safety of your application. Widely supported. FastCGI is compatible with many server implementations, not just Apache. Cons You may have to disable mod_deflate. If you experience page hangs with mod_fastcgi then remove deflate.load and deflate.conf from mods-enabled/ More complex environment With FastCGI, there are more things to monitor and more processes running than when using mod_perl. Standalone FastCGI Server In server mode the application runs as a standalone server and accepts connections from a web server. The application can be on the same machine as the web server, on a remote machine, or even on multiple remote machines. Advantages of this method include running the Catalyst application as a different user than the web server, and the ability to set up a scalable server farm. To start your application in server mode, install the FCGI::ProcManager module and then use the included fastcgi.pl script. $ script/myapp_fastcgi.pl -l /tmp/myapp.socket -n 5 Command line options for fastcgi.pl include: -d -daemon Daemonize the server. -p -pidfile Write a pidfile with the pid of the process manager. -l -listen Listen on a socket path, hostname:port, or :port. -n -nproc The number of processes started to handle requests. See below for the specific web server configurations for using the external server. WEB SERVERS
Any web server which supports FastCGI should work with Catalyst. Configuration recipies for well-known web servers are linked below, and we would welcome contributions from people deploying Catalyst on other web servers. Apache Catalyst::Manual::Deployment::Apache::FastCGI nginx Catalyst::Manual::Deployment::nginx::FastCGI lighttpd Catalyst::Manual::Deployment::lighttpd::FastCGI Microsoft IIS Catalyst::Manual::Deployment::IIS::FastCGI AUTHORS
Catalyst Contributors, see Catalyst.pm COPYRIGHT
This library is free software. You can redistribute it and/or modify it under the same terms as Perl itself. perl v5.14.2 2012-01-20 Catalyst::Manual::Deployment::FastCGI(3pm)
All times are GMT -4. The time now is 04:56 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy