The Bot Hunter: An Event Processing Challenge Post: 302225305

Sponsored Content

Special Forums UNIX and Linux Applications Virtualization and Cloud Computing The Bot Hunter: An Event Processing Challenge Post 302225305 by Linux Bot on Friday 15th of August 2008 05:40:08 AM

08-15-2008

Registered User

The Bot Hunter: An Event Processing Challenge

Tim Bass
08-15-2008 02:35 AM
Recently we penned The Attack of the Spiders from the Clouds where we mentioned how cloud computing infrastructures can be used to stage malicous or accidential network attacks.

Today I challenge our CEP/ESP/EP vendors (or SIs) to create the following solution to detect and block rogue bots on Apache web sites.** I will install and test each submitted solution on The UNIX Forums and post the results here.

Here are some basic requirements:

Your solution must run on Linux and be installable and configurable remotely with SSH or HTTP.* There will be no physical access to the server. No exceptions.
Preferrably, the configuration can be done with a Web-Based Interface (WBI) - a browser.
Your solution will listen to continuous updates to the Apache2 access log, exact location configurable in your solution, and identify robots ( bots), also known as spiders, from the log.
Your solution will provide a confidence metric, key indicator (KI), for each bot detected, from 0 to 10, where 10 indicates “absolutely a bot,” 0 is “absolutely not a bot.”
Your solution will update the IP address of each bot and KI you identify in a file/table called, for example, ./bot_scorecard.txt where each line is an IP address of a bot, followed by a semicolon (or other delimiter of your choice) and the confidence factor, for example,* 10.0.0.1;10 means that 10.0.0.1 is a bot, 100% sure.
Your solution must compare bots detected to a file/table called, for example, ./bots_allowed.txt and ./bots_denied.txt that are in the format IP address/mask, for example 10.0.0.1/24, or 10.0.0.1/32.
If the KI “confidence factor” of the IP address of your detected bot is higher than the tunable “is a bot” KI, then your solution should update the tables/files and then call iptables and block the bot.
It should send an email to one or more email addresses with a message, for example:* “New Bot Detected - Confidence 8″ with IP address, etc. in the message.* Another example would be an email, “Bot Blocked” - with details, etc.
You cannot automatically block any traffic that is not a bot.* Blocking one “non-bot” results in failure, no exceptions.

These are some basic requirements; I don’t want to restrict your thinking or solution, so be creative!* Feel free to ask any questions in the comment section of this thread.

Remember, sometimes you may have to manage the state of IP addresses for days, or hours, before you can accurately deterimine if it is a bot based on behavior alone.** So, you will need to work with both long and short time windows.* Latency is not important. Detection accurate is importance.

Anyone care to submit a solution for testing?

Source...

Linux Bot

View Public Profile for Linux Bot

Find all posts by Linux Bot

LEARN ABOUT DEBIAN

event::generic

Event::generic(3pm)					User Contributed Perl Documentation				       Event::generic(3pm)

NAME

       Event::generic - generic event handling

SYNOPSIS

	use Event::generic;

	$source = Event::generic::Source->new;

	$w = Event->generic(source => $source, ...);
	$w = $source->watch(...);

	$source = $w->source;
	$w->source($source);

	$source->event;
	$source->event($data);

	$data = $event->data;

DESCRIPTION

       This module provides a watcher type within the "Event" framework.  You must understand the architecture of the "Event" system in order to
       understand this document.

       This module provides a system of reified event sources and watchers watching those sources.  Events are generated solely by a method on the
       event source object.  The events may carry arbitrary data to the event handler callbacks.  This module is intended for situations where the
       events of interest are best determined by Perl code.

CLASSES

       Event::generic::Source
	   A reified event source.

       Event::generic
	   A watcher that can watch "Event::generic::Source" event sources.

       Event::Event::Dataful
	   A (target) event that can carry arbitrary data.

EVENT SOURCE CLASS

   Constructor
       Event::generic::Source->new
	   Creates and returns a new event source.

   Methods
       $source->event
       $source->event(DATA)
	   The invocation of this method is a source event for watchers of the event source.  When this method is called, each active watcher of
	   the event source generates a target event.  The DATA, if supplied, is copied into the target event objects, from which it can be
	   retrieved using the "data()" method.

       $source->watch(ATTR => VALUE, ...)
	   Generates and returns a new watcher, configured to watch this event source.	The standard watcher attributes may be specified as
	   arguments.  The watcher returned is an ordinary "Event::generic", and may be reconfigured to watch a different event source.

WATCHER CLASS

   Type-specific attributes
       source => $source
	   The event source to watch.  This must be either an "Event::generic::Source" object or "undef".  When set to "undef", no source is being
	   watched, and the watcher cannot be started.

EVENT CLASS

   Type-specific methods
       $event->data
	   Returns the data associated with the event, which may be any scalar.  This is read-only, and is set by the event source.

SEE ALSO

       Event

perl v5.14.2							    2007-05-22						       Event::generic(3pm)