Unix/Linux Go Back    


CentOS 7.0 - man page for pmdaweblog (centos section 1)

Linux & Unix Commands - Search Man Pages
Man Page or Keyword Search:   man
Select Man Page Set:       apropos Keyword Search (sections above)


PMDAWEBLOG(1)									    PMDAWEBLOG(1)

NAME
       pmdaweblog - performance metrics domain agent (PMDA) for Web server logs

SYNOPSIS
       $PCP_PMDAS_DIR/weblog/pmdaweblog  [-Cp]	[-d  domain] [-h helpfile] [-i port] [-l logfile]
       [-n idlesec] [-S num] [-t delay] [-u socket] [-U username] configfile

DESCRIPTION
       pmdaweblog is a Performance Metrics Domain Agent (PMDA(3)) that scans Web server  logs  to
       extract	metrics  characterizing  Web server activity.  These performance metrics are then
       made available through the infrastructure of the Performance Co-Pilot (PCP).

       The configfile specifies which Web servers are to be monitored,	their  associated  access
       logs  and error logs, and a regular-expression based scheme for extracting detailed infor-
       mation about each Web access.  This file is maintained as part of  the  PMDA  installation
       and/or	de-installation   by   the   scripts   Install	 and   Remove  in  the	directory
       $PCP_PMDAS_DIR/weblog.  For more details, refer to the section  below  covering	installa-
       tion.

       Once  started,  pmdaweblog  monitors  a	set of log files and in response to a request for
       information, will process any new information that has been appended  to  the  log  files,
       similar	to  a tail(1).	There is also periodic "catch up" to process new information from
       all log files, and a scheme to detect the rotation of log files.

       Like all other PMDAs, pmdaweblog is launched by pmcd(1) using command line options  speci-
       fied in $PCP_PMCDCONF_PATH - the Install script will prompt for appropriate values for the
       command line options, and update $PCP_PMCDCONF_PATH.

       A brief description of the pmdaweblog command line options follows:

       -C     Check the configuration and exit.

       -d domain
	      Specify the domain number.  It is absolutely crucial that the  performance  metrics
	      domain  number  specified here is unique and consistent.	That is, domain should be
	      different for every PMDA on the one host, and the same domain number should be used
	      for the pmdaweblog PMDA on all hosts.

	      For   most   installations,   the  default  domain  as  encapsulated  in	the  file
	      $PCP_PMDAS_DIR/weblog/domain.h  will  suffice.	For   alternate   values,   check
	      $PCP_PMCDCONF_PATH  for the domain values already in use on this host, and the file
	      $PCP_VAR_DIR/pmns/stdpmid contains a repository of ``well  known''  domain  assign-
	      ments that probably should be avoided.

       -h helpfile
	      Get the help text from the supplied helpfile rather than from the default location.

       -i port
	      Communicate with pmcd(1) on the specified Internet port (which may be a number or a
	      name).

       -l logfile
	      Location of the log file.  By default, a log file named weblog.log  is  written  in
	      the   current   directory   of   pmcd(1)	 when	pmdaweblog   is   started,   i.e.
	      $PCP_LOG_DIR/pmcd.  If the log file cannot be created or is not writable, output is
	      written to the standard error instead.

       -n idlesec
	      If  a  Web server log file has not been modified for idlesec seconds, then the file
	      will be closed and re-opened.  This is the only way pmdaweblog can detect any asyn-
	      chronous	rotation  of  the logs by Web server administrative scripts.  The default
	      period is 20 seconds.  This value may be changed dynamically  using  pmstore(1)  to
	      modify the value of the performance metric web.config.check.

       -p     Communicate with pmcd(1) via a pipe.

       -S num Specify  the  maximum number of Web servers per sproc.  It may be desirable (from a
	      latency and load balancing perspective) or necessary (due to file  descriptor  lim-
	      its)  to	delegate  responsibility for scanning the Web server log files to several
	      sprocs.  pmdaweblog will ensure that each sproc handles the log files for  at  most
	      num Web servers.	The default value is 80 Web servers per sproc.

       -t delay
	      To avoid the need to scan a lot of information from the Web server logs in response
	      to a single request for performance metrics, all log files will be checked at least
	      once  every  delay  seconds.  The default is 15 seconds.	This value may by changed
	      dynamically using pmstore(1) to modify the value of the performance metric web.con-
	      fig.catchup.

       -u socket
	      Communicate with pmcd(1) via the given Unix domain socket.

       -U     User  account  under which to run the agent.  The default is the unprivileged "pcp"
	      account in current versions of PCP, but in older	versions  the  superuser  account
	      ("root") was used by default.

INSTALLATION
       The  PCP  framework allows metrics to be collected on one host and monitored from another.
       These hosts are referred to as collector and monitor hosts, respectively.  A host  may  be
       both a collector and a monitor.

       Collector  hosts  require the installation of the agent, while monitoring hosts require no
       agent installation at all.

       For collector hosts do the following as root:

	 # cd $PCP_PMDAS_DIR/weblog
	 # ./Install

       The installation procedure prompts for a default or non-default installation.   A  default
       installation  will  search for known server configurations and automatically configure the
       PMDA for any server log files that  are	found.	 A  non-default  installation  will  step
       through	each  server, prompting the user for other server configurations and arguments to
       pmdaweblog.  The end result of a collector installation is to build a  configuration  file
       that is passed to pmdaweblog via the configfile argument.

       If you want to undo the installation, do the following as root:

	 # cd $PCP_PMDAS_DIR/weblog
	 # ./Remove

       pmdaweblog  is launched by pmcd(1) and should never be executed directly.  The Install and
       Remove scripts notify pmcd(1) when the agent is installed or removed.

CONFIGURATION
       The configuration file for the weblog PMDA is an ASCII file that can be	easily	modified.
       Empty  lines  and  lines beginning with '#' are ignored.  All other lines must be either a
       regular expression or server specification.

       Regular expressions, which are used on both the access and error log files, must be of the
       form:

	 regex regexName regexp
       or

	 regex_posix regexName ordering regexp_posix

       The  regexName  is  a  word which uniquely identifies the regular expression.  This is the
       reference used in the server specification.  The regexp for access logs is in  the  format
       described  for regcmp(3).  The regexp_posix for access logs is in the format described for
       regcomp(3).  The argument ordering is explained below. The Posix form should be	available
       on all platforms.

       The  regular expression requires the specification of up to four arguments to be extracted
       from each line of a Web server access log, depending on the type of server.  In	the  most
       common case there are two arguments representing the method and the size.

       For  the  non-  Posix  version, argument $0 should contain the method: GET, HEAD , POST or
       PUT.  The method PUT is treated as a synonym for POST, and anything else is categorized as
       OTHER.

       The second argument, $1, should contain the size of the request.  A size of ``-'' or `` ''
       is treated as unknown.

       Argument $3 should contain the status code returned to the client browser and argument  $4
       should  contain	the  status code returned to the server from a remote host.  These latter
       two arguments are used for caching servers and must be specified as a pair (or $3 will  be
       ignored).   For	 further   information	 on   status   codes,	refer  to  the	web  site
       http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

       Some legal non- Posix regex expression specifications for monitoring an access log are:

	 # pattern for CERN, NCSA, Netscape etc Access Logs
	 regex CERN ] "([A-Za-z][-A-Za-z]+)$0 .*" [-0-9]+ ([-0-9]+)$1

	 # pattern for FTP Server access logs (normally in SYSLOG)
	 regex SYSLOG_FTP ftpd[.*]: ([gp][-A-Za-z]+)$0( )$1

       There is 1 special types of access logs with the RegexName SQUID.  This formats extract	4
       parameters  but	since the Squid log file uses text-based status codes, it is handled as a
       special case.

       In the examples below, NS_PROXY parses the Netscape/W3C Common  Extended  Log  Format  and
       SQUID parses the default Squid Object Cache format log file.

	 # pattern for Netscape Proxy Server Extended Logs
	 regex NS_PROXY ] "([A-Za-z][-A-Za-z]+)$0 .*" ([-0-9]+)$2 \
	      ([-0-9]+)$1 ([-0-9]+)$3

	 # pattern for Squid Cache logs
	 regex SQUID [0-9]+.[0-9]+[ ]+[0-9]+ [a-zA-Z0-9.]+ \
	      ([_A-Z]+)$3([0-9]+)$2 ([0-9]+)$1 ([A-Z]+)$0

       The  regexp  for  the error logs does not require any arguments, only a match.  Some legal
       expressions are:

	 # pattern for CERN, NCSA, Netscape etc Error Logs
	 regex CERN_err .

	 # pattern for FTP Server error logs (normally in SYSLOG)
	 regex SYSLOG_FTP_err FTP LOGIN FAILED

       If POSIX compliant regular expressions are used, additional information is required  since
       the order of parameters cannot be specified in the regular expression.  For backwards com-
       patibility, the common case of two parameters the order may be specified as method,size or
       size,method  In	the general case, the ordering is specified by one of the following meth-
       ods:

       n1,n2,n3,n4
	    where nX is a digit between 1 and 4. Each comma-seperated field represents (in order)
	    the argument number for method,size,client_status,server_status

       -    Used for cases like the error logs where the content is ignored.

       As  for	the  non- Posix format, the SQUID RegexName is treated as a special case to match
       the non-numerical status codes.

       Some legal Posix regex expression specifications for monitoring an access log are:

	 # pattern for CERN, NCSA, Netscape, Apache etc Access Logs
	 regex_posix CERN method,size ][ \]+"([A-Za-z][-A-Za-z]+) \
	      [^"]*" [-0-9]+ ([-0-9]+)

	 # pattern for CERN, NCSA, Netscape, Apache etc Access Logs
	 regex_posix CERN 1,2 ][ \]+"([A-Za-z][-A-Za-z]+) \
	      [^"]*" [-0-9]+ ([-0-9]+)

	 # pattern for FTP Server access logs (normally in SYSLOG)
	 regex_posix SYSLOG_FTP method,size ftpd[.*]: \
	      ([gp][-A-Za-z]+)( )

	 # pattern for Netscape Proxy Server Extended Logs
	 regex_posix NS_PROXY 1,3,2,4 ][ ]+"([A-Za-z][-A-Za-z]+) \
	      [^"]*" ([-0-9]+) ([-0-9]+) ([-0-9]+)

	 # pattern for Squid Cache logs
	 regex_posix SQUID 4,3,2,1 [0-9]+.[0-9]+[ ]+[0-9]+ \
	      [a-zA-Z0-9.]+ ([_A-Z]+)([0-9]+) ([0-9]+) ([A-Z]+)

	 # pattern for CERN, NCSA, Netscape etc Error Logs
	 regex_posix CERN_err - .

	 # pattern for FTP Server error logs (normally in SYSLOG)
	 regex_posix SYSLOG_FTP_err - FTP LOGIN FAILED

       A Web server can be specified using this syntax:

	 server serverName on|off accessRegex accessFile errorRegex errorFile

       The serverName must be unique for each server, and is the name given to the  instance  for
       the  associated	performance  metrics.	See  PMAPI(3)  for  a  discussion of PCP instance
       domains.  The on or off flag indicates whether the server is to be monitored when the PMDA
       is   installed.	  This	 can   altered	 dynamically  using  pmstore(1)  for  the  metric
       web.perserver.watched, which has one instance for each Web server named in configfile.

       Two files are monitored for each Web server, the access and  the  error	log.   Each  file
       requires  the  name of a previously declared regular expression, and a file name.  The log
       files specified for each server do not have to exist when the weblog  PMDA  is  installed.
       The  PMDA  will	continue to check for non-existent log files and open them when possible.
       Some legal server specifications are:

	 # Netscape Server on Port 80 at IP address 127.55.555.555
	 server 127.55.555.555:80 on CERN /logs/access CERN_err /logs/errors

	 # FTP Server.
	 server ftpd on SYSLOG_FTP /var/log/messages SYSLOG_FTP_err /var/log/messages

CAVEATS
       Specifying regular expressions with an incorrect number of arguments, anything other  than
       2  for  access logs, and none for error logs, may cause the PMDA to behave incorrectly and
       even crash. This is due to limitations in the interface of regex(3).

FILES
       $PCP_PMDAS_DIR/weblog
		 installation directory for the weblog PMDA

       $PCP_PMDAS_DIR/weblog/Install
		 installation script for the weblog PMDA

       $PCP_PMDAS_DIR/weblog/Remove
		 de-installation script for the weblog PMDA

       $PCP_LOG_DIR/pmcd/weblog.log
		 default log file for error reporting

       $PCP_PMCDCONF_PATH
		 pmcd configuration file that specifies the command line options to be used  when
		 pmdaweblog is launched

       $PCP_LOG_DIR/NOTICES
		 log of PMDA installations and removals

       $PCP_VAR_DIR/config/web/weblog.conf
		 likely location of the weblog PMDA configuration file

       $PCP_DOC_DIR/pcpweb/index.html
		 the online HTML documentation for PCPWEB

PCP ENVIRONMENT
       Environment variables with the prefix PCP_ are used to parameterize the file and directory
       names used by PCP.  On each installation, the file /etc/pcp.conf contains the local values
       for these variables.  The $PCP_CONF variable may be used to specify an alternative config-
       uration file, as described in pcp.conf(5).

SEE ALSO
       pmcd(1),  pmchart(1),   pmdawebping(1),	 pminfo(1),   pmstore(1),   pmview(1),	 tail(1),
       weblogvis(1), webvis(1), PMAPI(3), PMDA(3) and regcmp(3).

Performance Co-Pilot			       PCP				    PMDAWEBLOG(1)
Unix & Linux Commands & Man Pages : ©2000 - 2018 Unix and Linux Forums


All times are GMT -4. The time now is 01:32 PM.