The Bot Hunter: An Event Processing Challenge Post: 302225305

Sponsored Content

Special Forums UNIX and Linux Applications Virtualization and Cloud Computing The Bot Hunter: An Event Processing Challenge Post 302225305 by Linux Bot on Friday 15th of August 2008 05:40:08 AM

08-15-2008

Registered User

The Bot Hunter: An Event Processing Challenge

Tim Bass
08-15-2008 02:35 AM
Recently we penned The Attack of the Spiders from the Clouds where we mentioned how cloud computing infrastructures can be used to stage malicous or accidential network attacks.

Today I challenge our CEP/ESP/EP vendors (or SIs) to create the following solution to detect and block rogue bots on Apache web sites.** I will install and test each submitted solution on The UNIX Forums and post the results here.

Here are some basic requirements:

Your solution must run on Linux and be installable and configurable remotely with SSH or HTTP.* There will be no physical access to the server. No exceptions.
Preferrably, the configuration can be done with a Web-Based Interface (WBI) - a browser.
Your solution will listen to continuous updates to the Apache2 access log, exact location configurable in your solution, and identify robots ( bots), also known as spiders, from the log.
Your solution will provide a confidence metric, key indicator (KI), for each bot detected, from 0 to 10, where 10 indicates “absolutely a bot,” 0 is “absolutely not a bot.”
Your solution will update the IP address of each bot and KI you identify in a file/table called, for example, ./bot_scorecard.txt where each line is an IP address of a bot, followed by a semicolon (or other delimiter of your choice) and the confidence factor, for example,* 10.0.0.1;10 means that 10.0.0.1 is a bot, 100% sure.
Your solution must compare bots detected to a file/table called, for example, ./bots_allowed.txt and ./bots_denied.txt that are in the format IP address/mask, for example 10.0.0.1/24, or 10.0.0.1/32.
If the KI “confidence factor” of the IP address of your detected bot is higher than the tunable “is a bot” KI, then your solution should update the tables/files and then call iptables and block the bot.
It should send an email to one or more email addresses with a message, for example:* “New Bot Detected - Confidence 8″ with IP address, etc. in the message.* Another example would be an email, “Bot Blocked” - with details, etc.
You cannot automatically block any traffic that is not a bot.* Blocking one “non-bot” results in failure, no exceptions.

These are some basic requirements; I don’t want to restrict your thinking or solution, so be creative!* Feel free to ask any questions in the comment section of this thread.

Remember, sometimes you may have to manage the state of IP addresses for days, or hours, before you can accurately deterimine if it is a bot based on behavior alone.** So, you will need to work with both long and short time windows.* Latency is not important. Detection accurate is importance.

Anyone care to submit a solution for testing?

Source...

Linux Bot

View Public Profile for Linux Bot

Find all posts by Linux Bot

LEARN ABOUT DEBIAN

ibid-plugin

IBID-PLUGIN(1)						     Ibid - Multi-protocol Bot						    IBID-PLUGIN(1)

NAME

       ibid-plugin - Plugin testing developer environment for Ibid

SYNOPSIS

       ibid-plugin [options...]  [plugin[-]|plugin.Processor[-]...]

DESCRIPTION

       This utility is for testing Ibid plugins without the full bot environment.  This means testing can be performed offline and without loading
       all the available plugins.

       This should be run in a configured Ibid bot directory.

       All the listed plugins and Processors will be loaded on start-up.  Naming a plugin loads the complete plugin.  Suffixing a - to	the  name,
       ignores that plugin or Processor instead of loading it.

OPTIONS

       -c, --configured
	      Load all configured plugins, instead of only the core and requested plugins.

       -o, --only
	      Don't  load  the	Ibid core plugins, only the plugins requested.	Note that without the core plugin to pre- and post-process events,
	      most other plugins won't function correctly.

       -p, --public
	      By default, ibid-plugin emulates a private conversation with the bot.  With this option, the conversation is considered to be public
	      and the bot will have to be addressed to provoke a response.

       -v, --verbose
	      Increase verbosity.  The final form of each Event object will be displayed before any responses.

       -h, --help
	      Show a help message and exit.

FILES

       ibid.ini
	      Locates the database to act upon by looking for the [databases].ibid value in the bot configuration file in the current directory.

BUGS

       ibid-plugin doesn't emulate a complete Ibid environment, and will ignore all of the following:

       *      Delayed and periodically executed functions.

       *      Messages to alternate sources.

       *      Messages directly dispatched, rather than added to responses.

       *      Permissions.  All permissions are granted to the user.

SEE ALSO

       ibid(1), ibid.ini(5), ibid-setup(1), http://ibid.omnia.za.net/

Ibid 0.1							    March 2010							    IBID-PLUGIN(1)