Download Comparing M2M Performance

Designing Complex Tabular Models
Bob Duffy
Database Architect
Say thank you to our Sponsors :
Speaker - Bob Duffy







20 years in database sector, 250+ projects
Senior Consultant with Microsoft 2005-2008
One of about 25 MCA for SQL Server globally (aka SQL Ranger)
SQL MCM on SQL 2005 and 2008
SQL Server MVP 2009+
SSAS Maestro
Database Architect at Prodata SQL Centre of Excellence
 http://blogs.prodata.ie/author/bob.aspx
 [email protected]
What we will cover
 What is difficult in tabular models ?
 Introducing the SCOM Data Warehouse
 Modelling Components




Managed Entities (Parent Child)
Groups (Many to Many Versioned Parent Child)
Discovery (Versioned XML EAV Pattern)
Real Time Processing
What is Complex for Tabular Models





Clean Star Schema
Simple Joins
Basic Calculations (SUM/AVG)
Full Processing
Basic Analytics








Highly relational Data Source
Many to Many
Parent Child
Versioned Fact Tables
Semi Additive Measures
Pivoted Data or XML
Analytical Dimensons
Real Time Processing
The SCOM Data Warehouse Model
Managed Entities
Rule
Performance
Alert
State
Discovery
Source: https://technet.microsoft.com/en-us/library/gg508713.aspx
Managed Entities and Hierarchies
 How Many Computers do I have ?
 How Many SQL Databases do I have ?
Computer
Computer
CPU 1-n
DISK 1-n
SQL Instance
SQL DB
Engine
SQL
Database
File Group
SQL Log File
SQL Agent
SQL Job
SQL Data File
What did we learn about P/C
 Perfectly “do-able” in tabular
 Needs to be flattened
 TSQL Naturalization best for recalc performance
 Use of Path functions best for agility
 Ragged Hierarchies possible
 Dax approaches gets too complex
 HideMemberIIF not supported and can crash
http://www.daxpatterns.com/parent-child-hierarchies/
Many to Many (P/C Versioned)
 How many SQL 2005 Databases are there?
 How many Servers does Bob manage ?
Managed Entity
PK
Group
ManagedEntityRowID
PK
Display Name
GroupID
Group Name
attribute name
Group Membership
ManagedEntityRowID
GroupID
FromDateTime
ToDateTime
Lessons on M2M





Lots of work on optimising them from SQLBI
We used helper “measures” to simplify
Roll on SQL 2016 with native support
Versioning and Parent-Child do-able with DAX
Performance better than expected
See https://www.sqlbi.com/articles/optimize-many-to-many-calculation-in-dax-with-summarize-and-cross-table-filtering/
Performance of M2M
Comparing M2M Performance
140
900
800
120
700
100
600
80
500
400
60
300
40
200
20
100
0
0
Object Count
M2M Basic
M2M Optimised
Queries
M2M Versioned
Time
M2M Versioned Optimised
Measure M2M
Discovery (EAV XML Design Pattern)
 Classes in SCOM have custom “tables” in OLTP
 100s of tables created dynamically on demand
 The data warehouse converts all these into XML
 Think hybrid of Entity Attribute View and XML
Lessons Learned on Discovery Attributes




Dimensional design is better for “fixed” schema
Performance on lots of attributes not great
Improvements with separating numeric values
Attributes “can” be in facts instead of dimensions
 We need to promote into columns to slice on attribute
 Leaving as measures is more flexible version wise
 LENE Needs Date AND Time in one column
Performance Counter Expressions
Semi Additive Measures with LENE
Real Time Processing
 How to Process large model every 15 mins?
 Need ProcessAdd & Change Tracking
 No Updates or deletes allowed to tables
ProcessAdd Mechanism
 TSQL Query locates last watermark in DWH
 DAX Query locates last watermark in cube
 Stored Procure retrieves only new data
Almost Real Time Processing
Thank You
 Special Thanks to:





Carmel for rubber ducking
Esteban for Processing Framework
Michael for early prototyping
Alberto for helping review the model
SQLBI for their
 Advanced DAX course
 DAX Formatter / Studio input
 DAX Patterns
Stick around for RAFFLE and the AFTER EVENT!
 All our volunteers and organisers do not get paid for
organizing this event – If you see them, please:





Give them a hug
Shake their hand
Say thank you
Spread the word
Get involved yourself
 Don’t forget to thank the sponsors for their support
 Thank the speakers for donating their time, energy and
expenses