Complex Event Processing RSS News

Louis Lovas
Tue, 13 Nov 2007 22:23:54 -0500
Bending the Nail<spanstyle='font-size:10.0pt;font-family:Verdana'>I am both humbled and appreciativeof all the accolades, constructive comments (hey, fix that misspelled word) and yes, criticism on my latest <ahref="http://apama.typepad.com/my_weblog/2007/11/bending-the-nai.html"><spanclass=SpellE>blog</span> about using SQL for Complex Event Processing.<spanstyle='mso-spacerun:yes'> </span>I was expecting some measure of response, asshown by this <ahref="http://hansgilde.wordpress.com/2007/11/06/fair-and-unfair-criticism-of-an-sql-ep-approach/">rebuttalgiven the somewhat polarizing nature of using the SQL language for CEPapplications. For every viewpoint thereis always an opposing, yet arguably valid outlook. I welcome any and allcommentary. Again thanks all reading and commenting.</span>
<spanstyle='font-size:10.0pt;font-family:Verdana'> </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'> </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'>There were two main themes of thecriticism that I received. One was on the viability of the SQL language, theother on my commentary on the use of Java and C++ for CEP applications. I wouldlike to clarify and reinforce a few points that I made. </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'> </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'>I chose the aggregation use-caseas an example to highlight limitations of SQL because its one that I haverecent experience with. I have been both directly and indirectly involved insix aggregation projects for our financial services customers over the pastyear. In those endeavors I've both learned much and leveraged much.<spanstyle='mso-spacerun:yes'> </span>As I tried to describe in a condensednarrative, aggregation is a challenging problem. One that is best solved by useof complex nested data structures and associated program logic.<spanstyle='mso-spacerun:yes'> </span>Trying to represent this in SQL is a tallorder given the flat 2-dimensional nature of SQL tables and the limited abilityfor semantic expression. To try to further explain this, I have taken a snippetof a MonitorScript-based implementation, presentingit as a bit of pseudo code that describes the nested structure I am referringto. I've intentionally avoided any specific language syntax and I've condensedthe structures to just the most relevant elements. But suffice to say, definingthese structures is clearly possible in Java and C++ and Apama'sMonitorScript. I would also like to give credit where it's due and acknowledge many ofcolleagues for the (abridged) definition I'm using below from one of ourcustomer implementations.</span>

<spanstyle='font-size:9.0pt;font-family:"Courier New"'> </span>
structure Provider {
stringsymbol; // symbol(unique to this provider)
stringmarketId; //the market identifier of the price point
integerquantity; // quantity theprovider is offering
floattimestamp; // time of thepoint
floatcost; // transactioncost for this provider
<span style='font-size:9.0pt;font-family:"Courier New";color:navy'> <spanclass=GramE>hashmap</span> extraParams;<spanstyle='mso-tab-count:1'> </span>// used for storing extra information onpoint
}

structure PricePoint
{
floatprice;// the price (either a bid or ask)
<span style='font-size:9.0pt;font-family:"Courier New";color:navy'> arrayproviders; // array of providers atthis price
integertotalQty;<spanstyle='mso-spacerun:yes'> </span>// total quantity across allproviders at this price
}

structure AggregatedOrderBook
{
integer<spanstyle='font-size:9.0pt;font-family:"Courier New";color:navy'> <spanclass=SpellE>sequenceId</span>; // counter incremented each time the book is updated<spanstyle='mso-spacerun:yes'> </span></span>
integertotalBidQuantity;<spanstyle='mso-spacerun:yes'> </span>// total volume available on the bidside
integertotalAskQuantity; // total volume available onthe ask side
integertotalProviders;<spanstyle='mso-spacerun:yes'> </span>// total number of providers
<span style='font-size:9.0pt;font-family:"Courier New";color:navy'> array bids;<spanstyle='mso-spacerun:yes'> </span>// list of bids, sorted by price
<span style='font-size:9.0pt;font-family:"Courier New";color:navy'> array asks;<spanstyle='mso-spacerun:yes'> </span>// list of asks, sorted by price
}
<spanstyle='font-size:9.0pt;font-family:"Courier New";color:navy'> </span>
<spanstyle='font-size:9.0pt;font-family:"Courier New"'> </span>

<spanstyle='font-size:10.0pt;font-family:Verdana'>An aggregation engine would createan instance of </span><spanstyle='font-size:10.0pt;font-family:Verdana;mso-bidi-font-family:"Courier New";color:navy'>AggregatedOrderBook</span> for each symbol, tracking prices per market data <istyle='mso-bidi-font-style:normal'>Provider</i><spanstyle='font-size:10.0pt;font-family:Verdana'>. As market data quotes arrivethey are decomposed and inserted (sort/merged) into the appropriate </span><spanclass=SpellE>PricePoint</span><spanstyle='font-size:10.0pt;font-family:Verdana'> and total values are calculated.This is an oversimplification of what transpires per incoming quote, but theaim here is to provide a simplified yet representative example of thecomplexities in representing an Aggregated Order Book. </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'> </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'>Furthermore, after each quote isprocessed and the aggregate order book is updated it's imperative that it bemade available to trading strategies expeditiously. Minimizing the <istyle='mso-bidi-font-style:normal'>signal-to-trade</i> latency is a key measureof success of algorithmic trading. Aggregation is a heavyweight, compute intensive operation. It takes alot of processing power to aggregate 1,000 symbols across 5 Exchanges. As such,it is one (of many) opposing forces to the goal of minimizing latency.<spanstyle='mso-spacerun:yes'> </span>So this presents yet another critical aspectof aggregation, how best to design it so that is can deliver its content toeagerly awaiting strategies. One meansof minimizing that latency is to have the aggregation component and tradingstrategies co-resident within the CEP runtime engine. Passing (or otherwiseproviding) the aggregated order book to the strategies becomes a simple '<istyle='mso-bidi-font-style:normal'>tap-on-the-shoulder</i>' codingconstruct. But it does imply the CEPlanguage has the semantic expressiveness to design and implement bothaggregation and trading strategies and then the ability to load and run themside-by-side within the CEP engine. Anyother model implies not only multiple languages (i.e. java and <spanclass=SpellE>streamSQL</span>) but likely some sort of distributed, networkedmodel. Separating aggregation from its consumers, the trading strategies willlikely incur enough overhead that it impacts that all important <istyle='mso-bidi-font-style:normal'>signal-to-trade</i> latency measure.<spanstyle='mso-spacerun:yes'> </span>I do realize that the CEP vendors using astreaming SQL variant have begun to add imperative syntax <ahref="http://www.dbta.com/e-newsletters/fmbdi=5_Minute_Briefing_Data_Integration/10-30-07.html">tosupport complex prodedural logic and "loop"constructs something I'm quite glad to see happening. It only validatesthe claim I've been making all along. The SQL language at its core isunsuitable for full-fledged CEP-style applications. The unfortunate side effectof these vendor-specific additions is that it will fracture attempts atstandardization. </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'> </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'> </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'>In my previous <ahref="http://apama.typepad.com/my_weblog/2007/11/bending-the-nai.html"><spanclass=SpellE>blog</span>, I wanted to point out the challenges of the SQLlanguage to both implement logic and manage application state. To that end, Iprovided a small snippet of a streamSQL variant.<spanstyle='mso-spacerun:yes'> </span>A criticism leveled against it states thatit's an</span> unnecessarilyinefficient bit of code. I won't argue that point, andI won't take credit for writing it either. I simply borrowed it from a sampleapplication provided with another SQL-based CEP product.<spanstyle='mso-spacerun:yes'> </span>The sample code a vendor includes with theirproduct is all too often taken as gospel. A customer's expectation is that it represents best practice usage. Vendorsshould take great care in providing samples, portions of which inevitably endup in production code.
<spanstyle='font-size:10.0pt;font-family:Verdana'> </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'>The second criticism I receivedwas on a few unintentionally scathing comments I made against Java andC++. I stated that using C++ and/or Java"</span>means you startan application's implementation at the bottom rung of the ladder<spanstyle='font-size:10.0pt;font-family:Verdana;mso-bidi-font-family:Verdana;color:#333333'>". </span>Myintent was to draw an analogy to CEP with its language and surroundinginfrastructure. All CEP engines provide much more than just <istyle='mso-bidi-font-style:normal'>language</i>. They provide a runtime engineor virtual machine, connectivity components, visualization tools andmanagement/deployment tools. CEP vendors like all infrastructure vendors liveand die by the features, performance and quality of their product. All toooften I've witnessed customers take a "notinvented here" attitude. They may survey the (infrastructure)landscape and decide "we can dobetter". For a business' IT group chartered with servicing the business to think they can implement infrastructurethemselves is a naïve viewpoint. Granted, on occasion requirements might be sounique that the only choice is to start slinging C++ code, but weighing themerits of commercial (and open source) infrastructure should not be overlooked.
<spanstyle='font-size:10.0pt;font-family:Verdana'> </span>
<spanstyle='font-size:10.0pt;font-family:Verdana'>My goal in this and past <spanclass=SpellE>blogs</span> is to provide concrete use-cases and opinions on CEPdrawn from my own experiences with designing, building and deploying <spanclass=SpellE>Apama</span> CEP applications. In doing so I was quite aware thatI drawing a big red bulls-eye on my back making me an easy target fordetractors to take aim. Surprisingly, I have received much more positivecommentary than I ever expected and fully professional criticisms. I thank allthat have taken the time to read my editorials, I am quite flattered.</span>
<spanstyle='font-size:10.0pt;font-family:Verdana'> </span>



Source...