Download Extending SSIS with Custom Tasks

Microsoft StreamInsight
Allan Mitchell
Konesans Limited
Who am I?
• Allan Mitchell
• Joint author on “First to Market” SSIS Book by Wrox
• Websites
– www.SQLIS.com
– www.SQLDTS.com
– www.konesans.com
• StreamInsight and SQL Azure Advisory Boards
• Microsoft SQL Server MVP
• Twitter: allanSQLIS
• eMail: [email protected]
Agenda
• What is Stream Processing
• Understanding Streaming Data
– Differences to RDBMS
• Scenarios
• StreamInsight Architecture
• Concepts
• Typical Patterns
What is Event Stream Processing
Query
• Database loads data from disk
• You query database
• Data is stale
What is Stream Processing
Query
StreamInsight
•
•
•
•
Continuous processing of event streams
from multiple sources
based on a declarative query language
with near-zero latency
Understanding Streaming Data
Using an RDBMS:
Q. Given a time interval tell me the amount of Electricity
units used per household per time interval from now until
I say stop.
Not Easy
Requires requerying/polling
Need to store the end of the last interval
Understanding Streaming Data
Answering with StreamInsight
• Create a window over the event stream
• Partition/Group By Household
• SUM() readings within window
This is the streaming data paradigm in a nutshell –
ask questions about data in flight.
What is CEP?
Complex Event Processing (CEP) is the continuous
and incremental processing of event streams from
multiple sources based on declarative query and
pattern specifications with near-zero latency.
What is CEP?
Complex Event Processing (CEP) is the continuous
and incremental processing of event streams from
multiple sources based on declarative query and
pattern specifications with near-zero latency.
Query
Paradigm
Database
Applications
Event-driven
Applications
Ad-hoc queries or
requests
Continuous standing
queries
What is CEP?
Complex Event Processing (CEP) is the continuous
and incremental processing of event streams from
multiple sources based on declarative query and
pattern specifications with near-zero latency.
Database
Applications
Event-driven
Applications
Query
Paradigm
Ad-hoc queries or
requests
Continuous standing
queries
Latency
Seconds, hours,
days
Milliseconds or less
What is CEP?
Complex Event Processing (CEP) is the continuous
and incremental processing of event streams from
multiple sources based on declarative query and
pattern specifications with near-zero latency.
Database
Applications
Event-driven
Applications
Query
Paradigm
Ad-hoc queries or
requests
Continuous standing
queries
Latency
Seconds, hours,
days
Milliseconds or less
Data Query
Rate
Hundreds of
events/sec
> Tens of thousands
of events/sec
Latency
Scenarios for Event-Driven
Applications
Months
StreamInsight
Target Scenarios
Days
Relational Database Applications
Operational Analytics
Applications, e.g., Logistics,
etc.
Data Warehousing
Applications
Web Analytics Applications
hours
Minutes
Seconds
100 ms
Manufacturing
Applications
Monitoring
Applications
< 1ms
0
10
100
1000
Financial trading
Applications
10000
100000
~1million
Aggregate Data Rate (Events/sec.)
12
Who might need CEP
•
•
•
•
Fraud Detection
Real-Time Trade Risking
Algorithmic Trading/Betting
Meter throughputs
– Oil, Gas, Water, Electricity
– Use to drive alarms, alerts etc
• Telemetry Data
Windowing
• Hopping
– Tumbling
• Snapshot
• Count
Time Windows
Hopping Window
Time Windows
Tumbling Window
Snapshot Windows
Snapshot Windows
StreamInsight Key Features
• Runtime environment for continuous and eventdriven processing
• In-memory processing, no persistence
• Declarative query language: LINQ
• .NET APIs for Input/Output Adapters
• Flexible deployment models
• Extensibility: Aggregates, Operators
• Rich diagnostic & debugging interface
StreamInsight Platform
StreamInsight
Application Development
Event sources
Devices, Sensors
StreamInsight Application at Runtime
Input
Adapters
StreamInsight Engine
Output
Adapters
Event targets
Pagers &
Monitoring devices
Standing Queries
Web servers
Event stores & Databases
Stock ticker, news feeds
`
Query
Logic
KPI Dashboards,
SharePoint UI
Query
Logic
Trading stations
Query
Logic
Event stores & Databases
Microsoft StreamInsight
CEP Application
Development
Event
Development experience with .NET,
C#, LINQ and Visual Studio 2008/10
CEP platform from Microsoft to build
sources
event-driven applications
CEP applications
Engine
Event-driven
are
Standing Queries
fundamentally different from
Event
Event
traditional database
Event
Event
applications:
queries
are
Event
continuous, consume and Event
produce streams,
and compute
Event
Event
results incrementally
Event
C_ID
C_NAME
C_ZIP
Static reference data
`
Output Adapters
Input Adapters
Flexible adapter SDK with
high performance to
connect to different event
sources and sinks
Event target
The CEP platform does the
heavy lifting for you to deal
with temporal characteristics
of event stream data
21
Central Concepts
• Events
–
–
–
User Payload + temporal properties
Type based on .NET structure or class
Point / Interval / Edge
–
–
Sequence of events with specific type
Flow into standing queries
–
–
–
Composed of query operators
Apply desired semantics on events
In-memory, event-triggered, incremental computations
–
Produce / Consume event streams (Connection Manager)
• Streams
• Queries
• Adapters
Typical Query & Analytics Patterns
•
•
•
•
•
•
•
•
•
•
•
•
Introduce additional event fields
Filter out events
Grouping by one or more event properties
Aggregation for each event group over a set (a window)
Rank events within windows
Correlate and synchronize event streams
Check for absence of events
Create result only when input changes
Remove duplicates
Reduce event rate, resample stream
Enrich events with reference data
Collection of assets may change over time
LINQ Query Examples
LINQ Example – JOIN, PROJECT, FILTER:
from e1 in MyStream1
join e2 in MyStream2
on e1.ID equals e2.ID
where e1.f2 == “foo”
select new { e1.f1, e2.f4 };
Join
Filter
Project
LINQ Example – GROUP&APPLY, WINDOW:
from e3 in MyStream3
group e3 by e3.i into SubStream
from win in SubStream.HoppingWindow(
FiveMinutes,ThreeSeconds)
select new { i = SubStream.Key,
a = win.Avg(e => e.f) };
Grouping
Window
Project &
Aggregate
Extensibility
• Need for domain-specific extensions
– Integrate with functionality available in existing libraries
• User-defined operators, functions, aggregates
• Code written in .NET, deployed as .NET
assembly
• Window-based semantics for UDOs,
UDAs
– Receive a set of events
– Produce a value / set of events
Demo
•
•
•
•
•
Adapter model
Windows
Grouping
Altering event lifetime
Extending
For More Information
• Download:
http://go.microsoft.com/fwlink/?LinkId=160598
• StreamInsight MSDN documentation:
http://msdn.microsoft.com/en-us/library/ee362541(SQL.105).aspx
• Blogs:
http://blogs.msdn.com/streaminsight/
http://www.SQLIS.com/
• Forum:
http://social.msdn.microsoft.com/Forums/en-US/streaminsight
• StreamInsight MSDN portal:
http://msdn.microsoft.com/en-us/ee476990.aspx
• Samples:
http://streaminsight.codeplex.com
• Twitter:
http://twitter.com/streaminsight