Apache2 logs analysis Post: 302980013

Sponsored Content

Top Forums Shell Programming and Scripting Apache2 logs analysis Post 302980013 by Chubler_XL on Monday 22nd of August 2016 09:17:38 PM

08-22-2016

Moderator

I have had a quick try at simplifying this script for you.

I managed to identify 3 different tests you are doing and created a check() function that
will cover these cases. It checks for a match and returns zero of no match. Otherwise it logs when required and returns 1. The return value is added to each of your counters.

I'm sure there could be much more simplification if you specified you expressions and counter names in another config file. But you would still need to edit the config file to change the tests so I doubt much more would be gained going that way.

Below, I use check() function to increment counters for your 3 different test cases - your job is to extend this for the full testing set. Note there is no need to initialise the counters as they will be set to zero automatically once the first line is processed.

Code:

#!/usr/bin/awk -f
function check(Fld, mtch, ex) {
   # ex will always be null (false) if it is not passed in,
   # otherwise it must equate to true to continue
   if(!ex && (Fld !~ mtch)) return 0

   x[IP]++
   if (SHOWLOG) printf("%s\t\033[1;32m%s\033[0m\t\t\033[1;32m%s\033[0m\n", IP, $2, mtch)
   return 1
}

BEGIN { FS="\""; SHOWLOG=1; IGNORECASE=1 }

{
  split($1,a," ")
  IP = a[2]

  # Case 1 - match to $2
  WebManage += check($2, "webmanage")
  WebSQL    += check($2, "websql")
  Digit200  += check($2, "[0-9]{200,}")

  # Case 2 - match to $(NF - 1)
  PrintNF   += check($(NF -1), "print")
  BinShNF   += check($(NF -1), "bin/sh")

  # Case 3 - complex expression
  Hexa      += check("", "[a-z0-9]", ( $(NF-1) !~ /Mozilla/ && $(NF-1) ~ /\\x[a-fA-Z0-9]+/ ))
  ConnectNF += check("", "connect", ( $(NF-1) !~ /Mozilla/ &&  $(NF-1) !~ /Outlook/ && $(NF-1) !~ /internal dummy connection/ && $3 !~ /200/ && $(NF-1) ~ /connect/))
}

END {
  printf("%-20s\t%d\n","webManage", WebManage);
  printf("%-20s\t%d\n","WebSQL", WebSQL);
  printf("%-20s\t%d\n","Digit200", Digit200);
  printf("%-20s\t%d\n","PrintNF", PrintNF);
  printf("%-20s\t%d\n","BinShNF", BinShNF);
  printf("%-20s\t%d\n","Hexa", Hexa);
  printf("%-20s\t%d\n","ConnectNF", ConnectNF);

  for ( j in x )  {
      print j
  }
}

Last edited by Chubler_XL; 08-23-2016 at 03:48 PM.. Reason: Better variable names - remove initialise of vars

Chubler_XL

View Public Profile for Chubler_XL

Find all posts by Chubler_XL

8 More Discussions You Might Find Interesting

1. Solaris

Logs Analysis Software ?

Hi, What is the best log analysis software for Solaris ?? Regards

2. Programming

Regarding stack analysis

I would like to know how I could do the following : void func(){ int a = 100; b=0; int c = a/b; } void sig_handler (int sig,siginfo_t *info,void *context){ //signal handling function //here I want to access the variables of func() } int main(){ struct sigaction *act =...

3. Shell Programming and Scripting

Grep yesterday logs from weblogic logs

Hi, I am trying to write a script which would go search and get the info from the logs based on yesterday timestamp and write yesterday logs in new file. The log file format is as follows: """"""""""""""""""""""""""...

4. Shell Programming and Scripting

Metacharacters analysis

:confused:Hi , Can someone please advise what is the meaning of metacharacters in below code? a_PROCESS=${0##*/} a_DPFX=${a_PROCESS%.*} a_LPFX="a_DPFX : $$ : " a_UPFX="Usage: $a_PROCESS" Regards, gehlnar

5. Shell Programming and Scripting

Analysis of a script

what does this line in a script mean?? I have tried to give it at the command prompt and here is what it returns ksh: /db2home/db2dap1/sqllib/db2profile: not found. . /db2home/db2dap1/sqllib/db2profile i have tried the same thing for my home directory too and the result is the same ....

6. UNIX for Dummies Questions & Answers

Text analysis

Hey Guys, Does anyone know how to count the separate amount of words in a text file? e.g the 5 and 20 Furthermore does anyone know how to convert whole numbers in decimals? Thanks

7. Infrastructure Monitoring

Nmon Analysis

Dear All, I am an performance tester. Now i am working in project where we are using linux 2.6.32. Now I got an oppurtunity to learn the monitoring the server. As part of this task i need to do analysis of the Nmon report. I was completely blank in this. So please suggest me how to start...

8. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention...

LEARN ABOUT DEBIAN

apache2::mpm

libapache2-mod-perl2-2.0.7::docs::api::Apache2::MPM(3pm)User Contributed Perl Documentatiolibapache2-mod-perl2-2.0.7::docs::api::Apache2::MPM(3pm)

NAME

       Apache2::MPM - Perl API for accessing Apache MPM information

Synopsis
	 use Apache2::MPM ();

	 # check whether Apache MPM is threaded
	 if (Apache2::MPM->is_threaded) { do_something() }

	 # which mpm is used
	 my $mpm = lc Apache2::MPM->show;

	 # query mpm properties
	 use Apache2::Const -compile => qw(:mpmq);
	 if (Apache2::MPM->query(Apache2::Const::MPMQ_STATIC)) { ... }

Description
       "Apache2::MPM" provides the Perl API for accessing Apache MPM information.

API

       "Apache2::MPM" provides the following functions and/or methods:

   "query"
       Query various attributes of the MPM

	 my $query = Apache2::MPM->query($const);

       obj: $class ( "Apache2::MPM class" )
	   the class name

       arg1: $const ( "Apache2::Const :mpmq group constant" )
	   The MPM attribute to query.

       ret: $query ( boolean )
	   the result of the query

       since: 2.0.00

       For example to test whether the mpm is static:

	 use Apache2::Const -compile => qw(MPMQ_STATIC);
	 if (Apache2::MPM->query(Apache2::Const::MPMQ_STATIC)) { ... }

   "is_threaded"
       Check whether the running Apache MPM is threaded.

	 my $is_threaded = Apache2::MPM->is_threaded;

       obj: $class ( "Apache2::MPM class" )
	   the class name

       ret: $is_threaded ( boolean )
	   threaded or not

       since: 2.0.00

       Note that this functionality is just a shortcut for:

	 use Apache2::Const -compile => qw(MPMQ_IS_THREADED);
	 my $is_threaded = Apache2::MPM->query(Apache2::Const::MPMQ_IS_THREADED);

   "show"
       What mpm is used

	 my $mpm = Apache2::MPM->show();

       obj: $class ( "Apache2::MPM class" )
	   the class name

       ret: $mpm ( string )
	   the name of the MPM. e.g., "Prefork".

       since: 2.0.00

See Also
       mod_perl 2.0 documentation.

Copyright
       mod_perl 2.0 and its core modules are copyrighted under The Apache Software License, Version 2.0.

Authors
       The mod_perl development team and numerous contributors.

perl v5.14.2							    2011-02-08		  libapache2-mod-perl2-2.0.7::docs::api::Apache2::MPM(3pm)