Apache2 logs analysis Post: 302980013

Sponsored Content

Top Forums Shell Programming and Scripting Apache2 logs analysis Post 302980013 by Chubler_XL on Monday 22nd of August 2016 09:17:38 PM

08-22-2016

Moderator

I have had a quick try at simplifying this script for you.

I managed to identify 3 different tests you are doing and created a check() function that
will cover these cases. It checks for a match and returns zero of no match. Otherwise it logs when required and returns 1. The return value is added to each of your counters.

I'm sure there could be much more simplification if you specified you expressions and counter names in another config file. But you would still need to edit the config file to change the tests so I doubt much more would be gained going that way.

Below, I use check() function to increment counters for your 3 different test cases - your job is to extend this for the full testing set. Note there is no need to initialise the counters as they will be set to zero automatically once the first line is processed.

Code:

#!/usr/bin/awk -f
function check(Fld, mtch, ex) {
   # ex will always be null (false) if it is not passed in,
   # otherwise it must equate to true to continue
   if(!ex && (Fld !~ mtch)) return 0

   x[IP]++
   if (SHOWLOG) printf("%s\t\033[1;32m%s\033[0m\t\t\033[1;32m%s\033[0m\n", IP, $2, mtch)
   return 1
}

BEGIN { FS="\""; SHOWLOG=1; IGNORECASE=1 }

{
  split($1,a," ")
  IP = a[2]

  # Case 1 - match to $2
  WebManage += check($2, "webmanage")
  WebSQL    += check($2, "websql")
  Digit200  += check($2, "[0-9]{200,}")

  # Case 2 - match to $(NF - 1)
  PrintNF   += check($(NF -1), "print")
  BinShNF   += check($(NF -1), "bin/sh")

  # Case 3 - complex expression
  Hexa      += check("", "[a-z0-9]", ( $(NF-1) !~ /Mozilla/ && $(NF-1) ~ /\\x[a-fA-Z0-9]+/ ))
  ConnectNF += check("", "connect", ( $(NF-1) !~ /Mozilla/ &&  $(NF-1) !~ /Outlook/ && $(NF-1) !~ /internal dummy connection/ && $3 !~ /200/ && $(NF-1) ~ /connect/))
}

END {
  printf("%-20s\t%d\n","webManage", WebManage);
  printf("%-20s\t%d\n","WebSQL", WebSQL);
  printf("%-20s\t%d\n","Digit200", Digit200);
  printf("%-20s\t%d\n","PrintNF", PrintNF);
  printf("%-20s\t%d\n","BinShNF", BinShNF);
  printf("%-20s\t%d\n","Hexa", Hexa);
  printf("%-20s\t%d\n","ConnectNF", ConnectNF);

  for ( j in x )  {
      print j
  }
}

Last edited by Chubler_XL; 08-23-2016 at 03:48 PM.. Reason: Better variable names - remove initialise of vars

Chubler_XL

View Public Profile for Chubler_XL

Find all posts by Chubler_XL

8 More Discussions You Might Find Interesting

1. Solaris

Logs Analysis Software ?

Hi, What is the best log analysis software for Solaris ?? Regards

2. Programming

Regarding stack analysis

I would like to know how I could do the following : void func(){ int a = 100; b=0; int c = a/b; } void sig_handler (int sig,siginfo_t *info,void *context){ //signal handling function //here I want to access the variables of func() } int main(){ struct sigaction *act =...

3. Shell Programming and Scripting

Grep yesterday logs from weblogic logs

Hi, I am trying to write a script which would go search and get the info from the logs based on yesterday timestamp and write yesterday logs in new file. The log file format is as follows: """"""""""""""""""""""""""...

4. Shell Programming and Scripting

Metacharacters analysis

:confused:Hi , Can someone please advise what is the meaning of metacharacters in below code? a_PROCESS=${0##*/} a_DPFX=${a_PROCESS%.*} a_LPFX="a_DPFX : $$ : " a_UPFX="Usage: $a_PROCESS" Regards, gehlnar

5. Shell Programming and Scripting

Analysis of a script

what does this line in a script mean?? I have tried to give it at the command prompt and here is what it returns ksh: /db2home/db2dap1/sqllib/db2profile: not found. . /db2home/db2dap1/sqllib/db2profile i have tried the same thing for my home directory too and the result is the same ....

6. UNIX for Dummies Questions & Answers

Text analysis

Hey Guys, Does anyone know how to count the separate amount of words in a text file? e.g the 5 and 20 Furthermore does anyone know how to convert whole numbers in decimals? Thanks

7. Infrastructure Monitoring

Nmon Analysis

Dear All, I am an performance tester. Now i am working in project where we are using linux 2.6.32. Now I got an oppurtunity to learn the monitoring the server. As part of this task i need to do analysis of the Nmon report. I was completely blank in this. So please suggest me how to start...

8. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention...

LEARN ABOUT PHP

maxdb_get_host_info

MAXDB_GET_HOST_INFO(3)							 1						    MAXDB_GET_HOST_INFO(3)

maxdb_get_host_info - Returns a string representing the type of connection used

       Procedural style

SYNOPSIS

       string maxdb_get_host_info (resource  $link)

DESCRIPTION

       Object oriented style
       string$maxdb->host_info ()

	The maxdb_get_host_info(3) function returns a string describing the connection represented by the $link parameter is using.

RETURN VALUES

	A character string representing the server hostname and the connection type.

EXAMPLES

       Example #1

	      Object oriented style

	      <?php
	      $maxdb = new maxdb("localhost", "MONA", "RED", "DEMODB");

	      /* check connection */
	      if (maxdb_connect_errno()) {
		 printf("Connect failed: %s
", maxdb_connect_error());
		 exit();
	      }

	      /* print host information */
	      printf("Host info: %s
", $maxdb->host_info);

	      /* close connection */
	      $maxdb->close();
	      ?>

       Example #2

	      Procedural style

	      <?php
	      $link = maxdb_connect("localhost", "MONA", "RED", "DEMODB");

	      /* check connection */
	      if (maxdb_connect_errno()) {
		 printf("Connect failed: %s
", maxdb_connect_error());
		 exit();
	      }

	      /* print host information */
	      printf("Host info: %s
", maxdb_get_host_info($link));

	      /* close connection */
	      maxdb_close($link);
	      ?>

       The above example will output something similar to:

       Host info: localhost

SEE ALSO

       maxdb_get_proto_info(3).

PHP Documentation Group 												    MAXDB_GET_HOST_INFO(3)