Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Microsoft StreamInsight Allan Mitchell Konesans Limited Who am I? • Allan Mitchell • Joint author on “First to Market” SSIS Book by Wrox • Websites – www.SQLIS.com – www.SQLDTS.com – www.konesans.com • StreamInsight and SQL Azure Advisory Boards • Microsoft SQL Server MVP • Twitter: allanSQLIS • eMail: [email protected] Agenda • What is Stream Processing • Understanding Streaming Data – Differences to RDBMS • Scenarios • StreamInsight Architecture • Concepts • Typical Patterns What is Event Stream Processing Query • Database loads data from disk • You query database • Data is stale What is Stream Processing Query StreamInsight • • • • Continuous processing of event streams from multiple sources based on a declarative query language with near-zero latency Understanding Streaming Data Using an RDBMS: Q. Given a time interval tell me the amount of Electricity units used per household per time interval from now until I say stop. Not Easy Requires requerying/polling Need to store the end of the last interval Understanding Streaming Data Answering with StreamInsight • Create a window over the event stream • Partition/Group By Household • SUM() readings within window This is the streaming data paradigm in a nutshell – ask questions about data in flight. What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Query Paradigm Database Applications Event-driven Applications Ad-hoc queries or requests Continuous standing queries What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database Applications Event-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries Latency Seconds, hours, days Milliseconds or less What is CEP? Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. Database Applications Event-driven Applications Query Paradigm Ad-hoc queries or requests Continuous standing queries Latency Seconds, hours, days Milliseconds or less Data Query Rate Hundreds of events/sec > Tens of thousands of events/sec Latency Scenarios for Event-Driven Applications Months StreamInsight Target Scenarios Days Relational Database Applications Operational Analytics Applications, e.g., Logistics, etc. Data Warehousing Applications Web Analytics Applications hours Minutes Seconds 100 ms Manufacturing Applications Monitoring Applications < 1ms 0 10 100 1000 Financial trading Applications 10000 100000 ~1million Aggregate Data Rate (Events/sec.) 12 Who might need CEP • • • • Fraud Detection Real-Time Trade Risking Algorithmic Trading/Betting Meter throughputs – Oil, Gas, Water, Electricity – Use to drive alarms, alerts etc • Telemetry Data Windowing • Hopping – Tumbling • Snapshot • Count Time Windows Hopping Window Time Windows Tumbling Window Snapshot Windows Snapshot Windows StreamInsight Key Features • Runtime environment for continuous and eventdriven processing • In-memory processing, no persistence • Declarative query language: LINQ • .NET APIs for Input/Output Adapters • Flexible deployment models • Extensibility: Aggregates, Operators • Rich diagnostic & debugging interface StreamInsight Platform StreamInsight Application Development Event sources Devices, Sensors StreamInsight Application at Runtime Input Adapters StreamInsight Engine Output Adapters Event targets Pagers & Monitoring devices Standing Queries Web servers Event stores & Databases Stock ticker, news feeds ` Query Logic KPI Dashboards, SharePoint UI Query Logic Trading stations Query Logic Event stores & Databases Microsoft StreamInsight CEP Application Development Event Development experience with .NET, C#, LINQ and Visual Studio 2008/10 CEP platform from Microsoft to build sources event-driven applications CEP applications Engine Event-driven are Standing Queries fundamentally different from Event Event traditional database Event Event applications: queries are Event continuous, consume and Event produce streams, and compute Event Event results incrementally Event C_ID C_NAME C_ZIP Static reference data ` Output Adapters Input Adapters Flexible adapter SDK with high performance to connect to different event sources and sinks Event target The CEP platform does the heavy lifting for you to deal with temporal characteristics of event stream data 21 Central Concepts • Events – – – User Payload + temporal properties Type based on .NET structure or class Point / Interval / Edge – – Sequence of events with specific type Flow into standing queries – – – Composed of query operators Apply desired semantics on events In-memory, event-triggered, incremental computations – Produce / Consume event streams (Connection Manager) • Streams • Queries • Adapters Typical Query & Analytics Patterns • • • • • • • • • • • • Introduce additional event fields Filter out events Grouping by one or more event properties Aggregation for each event group over a set (a window) Rank events within windows Correlate and synchronize event streams Check for absence of events Create result only when input changes Remove duplicates Reduce event rate, resample stream Enrich events with reference data Collection of assets may change over time LINQ Query Examples LINQ Example – JOIN, PROJECT, FILTER: from e1 in MyStream1 join e2 in MyStream2 on e1.ID equals e2.ID where e1.f2 == “foo” select new { e1.f1, e2.f4 }; Join Filter Project LINQ Example – GROUP&APPLY, WINDOW: from e3 in MyStream3 group e3 by e3.i into SubStream from win in SubStream.HoppingWindow( FiveMinutes,ThreeSeconds) select new { i = SubStream.Key, a = win.Avg(e => e.f) }; Grouping Window Project & Aggregate Extensibility • Need for domain-specific extensions – Integrate with functionality available in existing libraries • User-defined operators, functions, aggregates • Code written in .NET, deployed as .NET assembly • Window-based semantics for UDOs, UDAs – Receive a set of events – Produce a value / set of events Demo • • • • • Adapter model Windows Grouping Altering event lifetime Extending For More Information • Download: http://go.microsoft.com/fwlink/?LinkId=160598 • StreamInsight MSDN documentation: http://msdn.microsoft.com/en-us/library/ee362541(SQL.105).aspx • Blogs: http://blogs.msdn.com/streaminsight/ http://www.SQLIS.com/ • Forum: http://social.msdn.microsoft.com/Forums/en-US/streaminsight • StreamInsight MSDN portal: http://msdn.microsoft.com/en-us/ee476990.aspx • Samples: http://streaminsight.codeplex.com • Twitter: http://twitter.com/streaminsight