Download SQL_Saturday_511_Redmond_April_2_2016_

ELASTIC DATABASE CAPABILITIES
WITH AZURE SQL DB
Silvia Doomra
Azure SQL DB Program Management
ELASTIC DATABASE TOOLS & SERVICES - GOALS
Simplify the creation and operation of SaaS solutions in Azure
that grow to use large numbers of databases
 Develop OLTP applications with scaled out data tiers in Azure SQL DB
 Scale (grow or shrink) Azure SQL DB resources as needed
 Manage operations over many Azure DB databases
SHARDING AND TENANCY MODELS
Single tenant per database

Each tenant’s data is stored in a different database

Better isolation of tenants as compared to multi-tenant model
Multiple tenants per database

Multiple tenants share the same database

Less isolation of tenants as compared to single tenant model
Hybrid model

Some tenants share databases, others get their own database

E.g., premium or paying customers get their own databases, while free tier customers share databases
ELASTIC SCALE FOR YOUR DATA TIER
[shardmaps_global]
 Shard map management:
Manage membership in a scaled out data tier
Range: contiguous values

List: explicit values
name
1
RangeShardMap
[shards_global]
 Two types of shard maps

smid
sid
smid
Datasourc
e
Databasena
me
1
1
serverName
DB2
2
1
serverName
DB3
[shard_mappings_global]
 Types of sharding keys

INT, BIGINT, GUID, VARBINARY

All datetime types
Shard Map
Manager
DB1
DB2
DB3
DB4
DB5
DB6
[0-100)
[100-200)
[200-300)
[300-400)
[400-500)
[500-600)
mid
smid
min
ma
x
Sid
1
1
0
100
1
2
1
100
200
2
...
DBn
[n-n+100)
DATA DEPENDENT ROUTING
Scenario: query a shard with a specific shardlet key/tenant-id
Shard Map
Manager
Client App
Application
Developer
DDR APIs ( )
Admin/
DevOps
SELECT *
FROM customers
WHERE customer ID = 104
DB1
DB2
DB3
DB4
DB5
DB5
[0-100)
[100-200)
[200-300)
[300-400)
[400-500)
[500-600)
...
DBn
[n-n+100)
Data Dependent Routing
// Get a routed connection for a given shardingKey
using (SqlConnection conn = ShardMap.OpenConnectionForKey(
shardingKey,
connectionString /* Credentials Only */ ,
ConnectionOptions.Validate
/* Validate */ ));
{
using (SqlCommand cmd = new SqlCommand()
{
cmd.Connection = conn;
cmd.CommandText = "SELECT dbNameField, TestIntField, TestBigIntField
FROM ShardedTable";
SqlDataReader sdr = cmd.ExecuteReader();
// Now consume results from the data reader…
}
}
DATA DEPENDENT ROUTING
Caching: improve performance of shard operations

Global Shard Map (GSM) – state of all shards in the Shard Map

Local Shard Map (LSM) – state of all shards on a particular shard

Client Cache (eager/lazy) – state of all shards in the Shard Map/known shards
Application
Developer
Client App
DDR APIs ( )
Shard Map
Manager
GSM
Cache
DB1
[0-100)
LSM
ELASTIC SCALE CONNECTION OPENING FLOW
ShardMap.OpenConnectionForKey(
104 /* Tenant ID */ ,
“…” /* Credentials Only */ ,
ConnectionOptions.Validate /* Validate */ ));
 OpenConnectionForKey call with validation on
 Check for shardlet key in cache
 Cache miss: Fetch mapping info from GSM
Client
Application
Cache
Shard Map
Manager
 Connect to shard
GSM
[100, 200): DB2
 Validate on shard
DB2
[100, 300)
LSM
spValidate
 Validation fails: Go back to cache miss
MULTI-SHARD QUERY
Scenario: execute a query across a set of shards (returns a UNION ALL result set)
Shard Map
Manager
Client App
Application
Developer
Admin/
DevOps
MSQ APIs ( )
SELECT count(*)
FROM customers
UNION ALL result set
DB1
DB2
DB3
DB4
DB5
DB5
[0-100)
[100-200)
[200-300)
[300-400)
[400-500)
[500-600)
...
DBn
[n-n+100)
MULTI-SHARD QUERY
using (MultiShardConnection conn = new MultiShardConnection(m_shardMap.GetAllShards(null),
MultiShardTestUtils.GetTestSqlCredential()))
{
using (MultiShardCommand cmd = conn.CreateCommand())
{
cmd.CommandText = "SELECT dbNameField, TestIntField, TestBigIntField FROM ShardedTable";
cmd.CommandType = CommandType.Text;
cmd.Policy = MultiShardPolicy.PartialResults;
using (MultiShardDataReader sdr = cmd.ExecuteReader(includeShardNameColumn: true))
{
while (sdr.Read())
{
var dbNameField = sdr.GetString(0);
var testIntField = sdr.GetFieldValue<int>(1);
var testBigIntField = sdr.GetFieldValue<Int64>(2);
string shardIdPseudoColumn = sdr.GetFieldValue<string>(3);
}
}
}
}
GETTING STARTED IS EASY!
http://azure.microsoft.com/enus/documentation/articles/sql-database-elastic-scaledocumentation-map/
5 minute experience to a running app in Visual studio!
DEMO
ELASTIC DATABASE TOOLS
DATA MOVEMENT: SPLIT/MERGE
Scenario: perform a split or merge action

Split: create two distinct shards from one

Merge: create one shard from two distinct shards
Customer Hosted
Services
(SM)
Application
Developer
Split
Merge
DB1
DB2
DB3
DB4
DB5
DB5
[0-100)
[100-200)
[200-300)
[300-400)
[400-500)
[500-600)
DB2.1
DB5.1
DB5.2
[100-300)
[400-450)
[450-500)
...
DBn
[n-n+100)
Admin/
DevOps
ELASTIC DB QUERY
PowerBI
OLTP Cloud Application
SQL TDS, ODBC, JDBC, ADO
SQL DB Elastic Query
Elastic Tools Libraries
DB
DB
DB
DB
DB
Azure SQL Database
DB
DB
DB
DB
DB
DB
DB
DB
DB
DB
DB
DB
Azure SQL
DB v12
DB
DB
DB
DEMO
ELASTIC DATABASE QUERY
ELASTIC DATABASE JOBS
 Scenario: Perform management operations across many DBs (e.g. index
maintenance, DDL and DML, even queries delivering merged results)
Shard Map
Manager DB
Results DB
Control DB
Admin/
DevOps
Database
Jobs
(cloud service)
DB1
DB2
DB3
DB4
DB5
DB5
[0-100)
[100-200)
[200-300)
[300-400)
[400-500)
[500-600)
...
DBn
[n-n+100)
ELASTIC DATABASE JOBS: COMMON USE CASES
• Make changes to many databases
• Deploy schema changes across many databases, using T-SQL
• Update data common to many databases, e.g., reference data
• Deploy new versions of stored procedures
• Maintain indexes across a set of databases
• Manage permissions and logins across a set of databases
• Collect results from many databases
• Collect database telemetry from DMVs to monitor data tier performance
• Gather application-specific metrics and KPIs
DATABASE JOBS – CAPABILITIES
 Summary of key capabilities
 Execute T-SQL scripts – on-demand, scheduled
 Asynchronous, parallel (configurable) execution across databases
 Monitor execution progress and review status
 Automatic retry in case of failures
 Operations across collections of Azure SQL Databases

Elastic Pools

Shard Sets

All DBs on a Server

Ad-hoc collections of DBs
 Use stored secured credentials
ELASTIC DATABASE TRANSACTION
DEMO
ELASTIC DATABASE TRANSACTIONS
ELASTIC DATABASE TRANSACTION
using (var scope = new TransactionScope())
{
using (var conn1 = new SqlConnection(connStrDb1))
{
conn1.Open();
SqlCommand cmd1 = conn1.CreateCommand();
cmd1.CommandText = string.Format("insert into T1 values(1)");
cmd1.ExecuteNonQuery();
}
using (var conn2 = new SqlConnection(connStrDb2))
{
conn2.Open();
var cmd2 = conn2.CreateCommand();
cmd2.CommandText = string.Format("insert into T2 values(2)");
cmd2.ExecuteNonQuery();
}
scope.Complete();
}
DYNAMIC MANAGEMENT VIEWS (DMV)
All DMVs related to transactions are relevant for distributed transactions in Azure SQL DB.
sys.dm_tran_active_snapshot_database_transactions
sys.dm_tran_active_transactions
sys.dm_tran_current_snapshot
sys.dm_tran_current_transaction
sys.dm_tran_database_transactions
sys.dm_tran_locks
sys.dm_tran_session_transactions
sys.dm_tran_top_version_generators
sys.dm_tran_transactions_snapshot
sys.dm_tran_version_store
ELASTIC DB TOOLS: SUMMARY OF CAPABILITIES
Client .NET APIs
Management Services
Shard map management (SMM)
 Define groups of shards for your application
 Manage mapping of routing keys to shards
Data dependent routing (DDR)

Route incoming requests to the correct shard, e.g.,
given a customer ID

Ensure correct routing as tenants move

Cache routing information for efficiency
Multi-shard query (MSQ)

Interactive processing across several shards

Same statement executed on all shards with UNION
all semantics
Elastic DB Transactions

Support for distributed transactions across Azure SQL
DBs
Database Jobs (DJ)
• Asynchronous processing across several databases
Split/Merge (SM)
• Grow or shrink capacity by adding or removing
scale units
• Easily move data between scale units
Querying Capabilities
Elastic Queries (EQ)
• Interactive reporting across several shards
• Rich query semantics beyond MSQ
• Familiar SQL DB connection experience and protocol
support
ROADMAP & NEXT STEPS
Elastic Scale APIs and customer-deployed services

Client Library and Split/Merge: Generally Available

Open Sourced at: https://github.com/Azure/elastic-db-tools

Elastic Database Jobs: currently in Preview

Elastic Database Query: currently in Preview

Elastic Database Transactions: Generally Available
THANKS!
Silvia Doomra- [email protected]