unix and linux commands - unix shell scripting

Detection Theory: Signature Versus Anomaly Detection


 
Thread Tools Search this Thread
# 1  
Old 11-29-2008
Detection Theory: Signature Versus Anomaly Detection

Detection experts understand that the optimal detection design and architecture is generally a combination of both signature and anomaly detection engines.*** In event processing, signature detection involves the real-time pattern matching analysis of events.*** A core advantage of signature detection is that basic pattern matching models are easy to understand and develop when you know exactly what pattern you are looking for.** Designers then use a pattern, often called a signature, that searches for exact strings within an event object to detect, track, or observe an object of interest.** Pattern matching can be executed very quickly, efficiently and inexpensively on most all computing platforms today.

Howver, signature detection engines have weaknesses.* Generally speaking, signature detection engines can only detect known patterns based on a posteriori models.** An a posteriori signature must be created for every pattern under observation.* Therefore, unknown or modified patterns and situations will generally go undetected; hence, in practice, signature detection engines can suffer from both false positives and false negatives.

Pattern matching works well when detecting signatures in a known, deterministic model.* However, pattern matching does not work well against changing, self-modifying or adaptive behavior.*** In addition, signature detection is made more difficult by advanced techniques that attempt to conceal the "real" pattern by generating easy to detect "decoy" patterns.*** An example of this deception is chaff, small metal objects used to deceive signature-based radar detection.** The same type of deception is easily created in cyberspace.* Furthermore, the overall capability of a signature-based engine to scale upwards against adaptive and deceptive behavior is constrained by the fact that a new signature must be created for each variation, and as the rule set grows, the detection engine performance decreases.

In threat detection, signature-based detection often reduces, in practice, to a race condition between the malicious user (the threat) and the signature developers where the advantage goes to the threat because malicious users can develop new threats faster than new detection signatures can be written, tested and deployed to the detection engine.

On the other side of the detection coin, anomaly detection focuses on the concept of baselining normal behavior and detecting variations from the baseline. The baseline is learned and/or specified by system designers.** Detected situations in an anomaly detection engine are created by any situation that falls outside the predefined boundary of the anomaly detection model.

A key component of baselining in anomaly detection is the capability of the detection engine to detect deviations from situational models at many different layers.* This means that anomaly detection engines are initially computationally expensive. However, one trade-off is that anomaly detection engines tends to scale better than a signature detection engines as the event data-set grows.* As a result,* designers often see fewer false positives in anomaly detection.

A known disadvantage of anomaly-detection engines, similar to signature detection, is the difficultly of predefining rules.** Even more challenging, the detailed knowledge of normal baseline situations must be constructed and transferred into the engine memory for accurate detection.* However, once a robust baseline has been established and normal behavior or situational pattern defined, anomaly detection engines tends to scale more quickly and easily than signature-based engines because a new signature does not have to be designed, tested and uploaded for every new variant that comes along.

Detection experts know that the optimal detection design is generally a combination of both signature and anomaly detection engines.** Because anomaly detection engines tends to be adaptive, learning systems, the current trend is for anomaly detection engines based on statistical learning algorithms such as artificial neural networks or dynamic Bayesian networks.

In other words, it is well established in detection systems design that the optimal approach for most non-trivial detection-oriented problems is a combination of both signature and anomaly detection engines.** If you are buying a detection engine and the engine is only capable of signature or pattern detection, you will be using a suboptimal detection architecture.** This is a well established systems engineering principal in detection-oriented systems design.

Note:* This post is adapted from my original post, Quintessential Event Processing: Signature Versus Anomaly Detection

Image
Image

More...
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Curl detection of redirect

This URL: Internet Archive Wayback Machine ..will redirect to this URL when using a browser: https://web.archive.org/web/20140110200053/http://rapor.tuik.gov.tr/reports/rwservlet?adnksdb2&ENVID=adnksdb2Env&report=wa_idari_yapi_10sonrasi.RDF&p_il1=17&p_yil=2012&p_dil=2&desformat=html But... (0 Replies)
Discussion started by: Mid Ocean
0 Replies

2. Shell Programming and Scripting

SQL Injection Detection

I want to grep/awk /var/log/httpd/mysite-access_log.log and check if 2 words from the following appear in a single line: benchmark union information_schema drop truncate group_concat into file case hex lpad group order having insert union select from (12 Replies)
Discussion started by: koutroul
12 Replies

3. Shell Programming and Scripting

File delta detection

Hello, I need to compare two flat files (ASCII format), say file OLD and file NEW. Both have similar structure. These files are | delimitted files and have around few million of records (lines) each. Each file has same set columns and same set of key columns (i.e. the 3rd and 5th column of the... (7 Replies)
Discussion started by: manubatham20
7 Replies

4. Programming

Parallel Processing Detection and Program Return Value Detection

Hey, for the purpose of a research project I need to know if a specific type of parallel processing is being utilized by any user-run programs. Is there a way to detect whether a program either returns a value to another program at the end of execution, or just utilizes any form of parallel... (4 Replies)
Discussion started by: azar.zorn
4 Replies

5. Shell Programming and Scripting

NAT detection

hellou, can anybody help me with nat detection in real time ? i prefer some detection script because i try some nat detection program's for example p0f or i'm using tcpdump, but i would get contain of specific packet. Some ideas? (1 Reply)
Discussion started by: TheTechnic
1 Replies

6. Shell Programming and Scripting

begin end detection

Hi all, i am new to scripting. i need to write a code to detect begin and end of word that either begins or ends with t,th,d,dh,s,sh i have a set of words in a file containg one word per line. let the filename be aaa.txt. i have an another file bbb.txt which has two lines, just specifying the... (7 Replies)
Discussion started by: blkanth
7 Replies

7. IP Networking

modem detection

How to get information that where is my modem configured in /dev. I have two modems configured in my device .. one is USB and other is PCI modem.. USB is detected as /dev/USB0. but how to see about PCI modem? (0 Replies)
Discussion started by: s123.radha
0 Replies

8. UNIX for Dummies Questions & Answers

virus detection

IS there virus software for unix? I worked in a Solaris environment and dont remember having anything. I also ask because the current enviroment i am working on is Microsoft and they argue that they do not use unix because of virus detection. Any input would be greatly appreciated. (1 Reply)
Discussion started by: pbonilla
1 Replies
Login or Register to Ask a Question