Apache2 logs analysis Post: 302980013

Sponsored Content

Top Forums Shell Programming and Scripting Apache2 logs analysis Post 302980013 by Chubler_XL on Monday 22nd of August 2016 09:17:38 PM

08-22-2016

Moderator

I have had a quick try at simplifying this script for you.

I managed to identify 3 different tests you are doing and created a check() function that
will cover these cases. It checks for a match and returns zero of no match. Otherwise it logs when required and returns 1. The return value is added to each of your counters.

I'm sure there could be much more simplification if you specified you expressions and counter names in another config file. But you would still need to edit the config file to change the tests so I doubt much more would be gained going that way.

Below, I use check() function to increment counters for your 3 different test cases - your job is to extend this for the full testing set. Note there is no need to initialise the counters as they will be set to zero automatically once the first line is processed.

Code:

#!/usr/bin/awk -f
function check(Fld, mtch, ex) {
   # ex will always be null (false) if it is not passed in,
   # otherwise it must equate to true to continue
   if(!ex && (Fld !~ mtch)) return 0

   x[IP]++
   if (SHOWLOG) printf("%s\t\033[1;32m%s\033[0m\t\t\033[1;32m%s\033[0m\n", IP, $2, mtch)
   return 1
}

BEGIN { FS="\""; SHOWLOG=1; IGNORECASE=1 }

{
  split($1,a," ")
  IP = a[2]

  # Case 1 - match to $2
  WebManage += check($2, "webmanage")
  WebSQL    += check($2, "websql")
  Digit200  += check($2, "[0-9]{200,}")

  # Case 2 - match to $(NF - 1)
  PrintNF   += check($(NF -1), "print")
  BinShNF   += check($(NF -1), "bin/sh")

  # Case 3 - complex expression
  Hexa      += check("", "[a-z0-9]", ( $(NF-1) !~ /Mozilla/ && $(NF-1) ~ /\\x[a-fA-Z0-9]+/ ))
  ConnectNF += check("", "connect", ( $(NF-1) !~ /Mozilla/ &&  $(NF-1) !~ /Outlook/ && $(NF-1) !~ /internal dummy connection/ && $3 !~ /200/ && $(NF-1) ~ /connect/))
}

END {
  printf("%-20s\t%d\n","webManage", WebManage);
  printf("%-20s\t%d\n","WebSQL", WebSQL);
  printf("%-20s\t%d\n","Digit200", Digit200);
  printf("%-20s\t%d\n","PrintNF", PrintNF);
  printf("%-20s\t%d\n","BinShNF", BinShNF);
  printf("%-20s\t%d\n","Hexa", Hexa);
  printf("%-20s\t%d\n","ConnectNF", ConnectNF);

  for ( j in x )  {
      print j
  }
}

Last edited by Chubler_XL; 08-23-2016 at 03:48 PM.. Reason: Better variable names - remove initialise of vars

Chubler_XL

View Public Profile for Chubler_XL

Find all posts by Chubler_XL

8 More Discussions You Might Find Interesting

1. Solaris

Logs Analysis Software ?

Hi, What is the best log analysis software for Solaris ?? Regards

2. Programming

Regarding stack analysis

I would like to know how I could do the following : void func(){ int a = 100; b=0; int c = a/b; } void sig_handler (int sig,siginfo_t *info,void *context){ //signal handling function //here I want to access the variables of func() } int main(){ struct sigaction *act =...

3. Shell Programming and Scripting

Grep yesterday logs from weblogic logs

Hi, I am trying to write a script which would go search and get the info from the logs based on yesterday timestamp and write yesterday logs in new file. The log file format is as follows: """"""""""""""""""""""""""...

4. Shell Programming and Scripting

Metacharacters analysis

:confused:Hi , Can someone please advise what is the meaning of metacharacters in below code? a_PROCESS=${0##*/} a_DPFX=${a_PROCESS%.*} a_LPFX="a_DPFX : $$ : " a_UPFX="Usage: $a_PROCESS" Regards, gehlnar

5. Shell Programming and Scripting

Analysis of a script

what does this line in a script mean?? I have tried to give it at the command prompt and here is what it returns ksh: /db2home/db2dap1/sqllib/db2profile: not found. . /db2home/db2dap1/sqllib/db2profile i have tried the same thing for my home directory too and the result is the same ....

6. UNIX for Dummies Questions & Answers

Text analysis

Hey Guys, Does anyone know how to count the separate amount of words in a text file? e.g the 5 and 20 Furthermore does anyone know how to convert whole numbers in decimals? Thanks

7. Infrastructure Monitoring

Nmon Analysis

Dear All, I am an performance tester. Now i am working in project where we are using linux 2.6.32. Now I got an oppurtunity to learn the monitoring the server. As part of this task i need to do analysis of the Nmon report. I was completely blank in this. So please suggest me how to start...

8. Shell Programming and Scripting

If I ran perl script again,old logs should move with today date and new logs should generate.

Appreciate help for the below issue. Im using below code.....I dont want to attach the logs when I ran the perl twice...I just want to take backup with today date and generate new logs...What I need to do for the below scirpt.............. 1)if logs exist it should move the logs with extention...

LEARN ABOUT SUSE

sdl_wasinit

SDL_WasInit(3)							 SDL API Reference						    SDL_WasInit(3)

NAME

       SDL_WasInit - Check which subsystems are initialized

SYNOPSIS

       #include "SDL.h"

       Uint32 SDL_WasInit(Uint32 flags);

DESCRIPTION

       SDL_WasInit  allows you to see which SDL subsytems have been initialized. flags is a bitwise OR'd combination of the subsystems you wish to
       check (see SDL_Init for a list of subsystem flags).

RETURN VALUE

       SDL_WasInit returns a bitwised OR'd combination of the initialized subsystems.

EXAMPLES

       /* Here are several ways you can use SDL_WasInit() */

       /* Get init data on all the subsystems */
       Uint32 subsystem_init;

       subsystem_init=SDL_WasInit(SDL_INIT_EVERYTHING);

       if(subsystem_init&SDL_INIT_VIDEO)
	 printf("Video is initialized.
       ");
       else
	 printf("Video is not initialized.
       ");

       /* Just check for one specfic subsystem */

       if(SDL_WasInit(SDL_INIT_VIDEO)!=0)
	 printf("Video is initialized.
       ");
       else
	 printf("Video is not initialized.
       ");

       /* Check for two subsystems */

       Uint32 subsystem_mask=SDL_INIT_VIDEO|SDL_INIT_AUDIO;

       if(SDL_WasInit(subsystem_mask)==subsystem_mask)
	 printf("Video and Audio initialized.
       ");
       else
	 printf("Video and Audio not initialized.
       ");

SEE ALSO

       SDL_Init, SDL_Subsystem

SDL
							      Tue 11 Sep 2001, 23:00						    SDL_WasInit(3)