Download BNCOD2006

An Overview of a Scalable
Distributed Database System:
SD-SQL Server
Witold LITWIN, Soror SAHRI & Thomas SCHWARZ
[email protected]
[email protected]
Ceria Laboratory
Paris-Dauphine University
BNCOD 2006
[email protected]
Comp. Eng. Dep.
Santa Clara U.
Overview
1. Introduction
2. Architecture
3. Command Interface
4. Processing
5. Performance
6. Conclusion & Future Work
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Partitioned Tables

Most DBSs have distributed/parallel versions with
partitioned tables

SQL Server, Oracle, DB2, MySQL, Postgres…
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
BENEFITS OF PARTITIONING
BENEFITS OF PARTITIONING
Partitioning can provide tremendous benefits to a wide variety of applications by
improving manageability, performance, and availability. It is not unusual for
partitioning to improve the performance of certain queries or maintenance
operations by an order of magnitude. Moreover, partitioning can greatly simplify
common administration tasks.
Partitioning also enables database designers and administrators to tackle some of
the toughest problems posed by cutting-edge applications. Partitioning is a key
tool for building multi-terabyte systems or systems with extremely high
availability requirements.
Partitioning in
Oracle Database 10g Release 2
An Oracle White Paper
May 2005
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Hassle of Partitioning

DBSs require manual partitioning


And manual repartitioning when tables scale-up
DBSs do not provide dynamically scalable tables
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Facts
 http://ceria.dauphine.fr/CERIA-publications.html
 Research Report, December 2005
 [Oracle Database 10g]
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
SD-SQL Server Goal
Scalable Distributed Partitioning of
Relational Tables
Scalable Distributed Database System
SD-DBS
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Role Model: an SDDS


A scalable distributed data structure
Specifically designed for possibly very large
data on multi-computers or networks of WSs
 P2P & Grids in modern vocabulary
 Why SDDS Role Model?



Several SDDS schemes are well-known by now:
 LH*, RP*, k-RP*, LH*RS…Chord, VBI & most of P2P schemes
The domain has over 20.000 references on Google
An SD-DBS reuses SDDS design principles
 With DB management specificity
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
SD-SQL Server
The first and yet the only SD-DBS
 Implements the SD-DBS architecture

• Litwin, Schwartz & Risch (2002)

Runs on Microsoft SQL Server 2000
Shared Nothing Architecture
 Up to 250 nodes at present

BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Gross Architecture
User/Application
User/Application
sd_insert
sd_create_table
SD-SQL
Server
Managers
Linked
SQL
Servers
SD-SQL
client
SD-SQL
server
SD-SQL
server
S
S
C
D1_T
P
T
_D1_T
NDBs
SD-SQL
peer
D1
BNCOD 2006 – July 18th, 2006
D2
_D1_T
Di
Split
_D1_T
Di+1
An Overview of a Scalable Distributed Database System: SD-SQL Server
Nodes, SDBs & NDBs
MDB
Node1
Node2
DB1
DB2
DB1 SDB
DB2 SDB
BNCOD 2006 – July 18th, 2006
Node3
DB1
DB2
……
Nodei
DB1
An Overview of a Scalable Distributed Database System: SD-SQL Server
NDB Types

Client NDB
• Interfaces applications & users
• Carries only images
• No actual tables with application data
 Server NDB
• Carries actual tables
• segments

Primary NDB
 First for an SDB
Peer NDB
• Both functions
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable (Distributed) Table



For the application: a table of an SDB
Internally: a collection of segments behind client images
A segment is an SQL table
• One per NDB of the SDB
• Sharing the scalable table scheme
• Except its check constraint
• Min and Max value of the partition key
• With size limit
• Splitting when overflows occur

The check constraints partition the partition key space
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable (Distributed) Table

The primary segment
First allocated for a new table
 At some server or peer NDB of SDB

• The peer creating the table
• The primary server of the client creating the table
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Gross Architecture
User/Application
User/Application
sd_insert
sd_create_table
SD-SQL
Server
Managers
Linked
SQL
Servers
SD-SQL
client
SD-SQL
server
SD-SQL
server
S
S
C
D1_T
P
T
_D1_T
NDBs
SD-SQL
peer
D1
BNCOD 2006 – July 18th, 2006
D2
_D1_T
Di
Split
_D1_T
Di+1
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable (Distributed) Table

For the application, it the client image is the table


The image name is the table name
Primary image

Created during the scalable table creation
• at the client or peer NDB creating the table

Secondary images




Created later on
On other NDBs of the SDB
For local applications
By a dedicated command
• sd_Create Image….
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable (Distributed) Table

Internally, every image is a specific SQL
Server view of the segments:
 Distributed
partitioned union view
CREATE VIEW T AS SELECT * FROM N2.DB1.SD._N1_T
UNION ALL SELECT * FROM N3.DB1.SD._N1_T
UNION ALL SELECT * FROM N4.DB1.SD._N1_T
 Updatable
• Through the check constraints
 With or without Lazy Schema Validation
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Gross Architecture
User/Application
User/Application
sd_insert
sd_create_table
SD-SQL
Server
Managers
Linked
SQL
Servers
SD-SQL
client
SD-SQL
server
SD-SQL
server
S
S
C
D1_T
P
T
_D1_T
NDBs
SD-SQL
peer
D1
BNCOD 2006 – July 18th, 2006
D2
_D1_T
Di
Split
_D1_T
Di+1
An Overview of a Scalable Distributed Database System: SD-SQL Server
SD SQL Server Meta-Tables


Store various SD-SQL Server meta-data
In particular about each scalable table

At each server or peer NDB
• SD.Size meta-table
• Segment capacity
• The number of stored tuples triggering a split
• Same for every segment at present
• SD.RP meta-table
• the actual partitioning of the scalable table
• The location of each segment
• SD.Primary table
• The location of the SD.RP table for each segment in the NDB
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
SD SQL Server Meta-Tables

At every client or peer NDB
 In SD.Image table
• All the local images
• The name of the image
• The type
• Primary or secondary
• The number of segments
• As seen by an image
• Not necessarily the actual one
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
SD SQL Server Meta-Tables

At every NDB




SD.SDBNode points towards the primary NDB
SD.MDBNode points towards the MDB
At MDB

SD.Nodes indicates all the available SD-SQL Server nodes
• Over linked SQL Server nodes

SD.SDB describes all the SDBs
At every primary NDB

SD.NDB points to every NDB of the SDB
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable Tables: Meta-data
DB1 SDB
…….
N1.DB1
N2.DB1
N3.DB1
Ni.DB1
Primary
T Scalable Table
Size
SDBNode
Nodes
N1.DB1
1000
Ni.DB1
N1.DB1
RP
N2.DB1
N3.DB1
Meta-Tables
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable Table Expansion

The number of segments in a scalable table
may grow

An overflowing segment splits
• Creating one or more new segments
A split occurs when an insert overflows the
segment capacity
 The trigger launches the split as an
asynchronous job called splitter


To avoid the application level timeout
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Gross Architecture
User/Application
User/Application
sd_insert
sd_create_table
SD-SQL
Server
Managers
Linked
SQL
Servers
SD-SQL
client
SD-SQL
server
SD-SQL
server
S
S
C
D1_T
P
T
_D1_T
NDBs
SD-SQL
peer
D1
BNCOD 2006 – July 18th, 2006
D2
_D1_T
Di
Split
_D1_T
Di+1
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable Table Expansion

Every new segment

Is basically created at an existing NDB that does
not yet have any segments of the expanding
table
• provided there is any

Otherwise a new NDB is first appended to SDB
• Provided there is an available SD SQL Server node
Inherits the “father”’s schema
 Gets its new check constraint
 Gets indexes as defined at the “father”

BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Single Segment Split
Single Tuple Insert
Check Constraint?
p=INT(b/2)
C( S)=  { c: c < h = c (b+1-p)}
C( S1)={c: c > = c (b+1-p)}
b+1
b
b+1-p
S
p
S
S1
SELECT
TOP Pi
* INTO
FROM
S ORDER
BYBY
C ASC
SELECT TOP
Pi * WITH
TIES
INTONi.Si
Ni.S1
FROM
S ORDER
C ASC
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Single Segment Split
Bulk Insert
(a)
(b)
b+t
b
(c)
P1
b
b
b
b
Pn
b+t-np
p
b+t-np
S
S
S
p
P1
Pn
S1
SN
p = INT(b/2)
C(S) = {c: l < c < h }
 { c: l ≤ c < h’ = c (b+t–Np)}
C(S1) = {c: c (b+t-p) < c < h}
…
C(SN) = {c: c (b+t-Np) ≤ c < c (b+t-(N-1)p)}
Single segment split
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Multi-Segment Split
Bulk Insert
b
b
b
b
b
b
p
p
S
Sk
S1
S1, n1
Sk
Sk, nk
Multi-segment split
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Split with SDB Expansion
sd_create_node_database
sd_create_node
N1
N2
N3
N4
Ni
NDB
NDB
NDB
NDB
NDB
DB1
DB1
DB1
DB1
DB1
SDB DB1
sd_insert
sd_insert
sd_insert
…….
Scalable Table T
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Image Adjustment
The splits do not modify synchronously the images
 Any split makes every image outdated
 The client or peer verifies every image dynamically
when a query to the image comes in

Image checking
 Image adjustment if necessary

BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Image Adjustment




Get the number of segments presented in the image,
N1
Get the number of segments of the scalable table,
N2
Compare N1 and N2:
If N1<N2 then Image Adjustment

Alter the partitioned view definition
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Image: Example
DB1 SDB
N1.DB1
Primary
Image
N2.DB1
N3.DB1
N4.DB1
T Image
T Scalable Table
CREATE VIEW T AS SELECT * FROM N2.DB1.SD._N1_T
UNION ALL SELECT * FROM N3.DB1.SD._N1_T
UNION ALL SELECT * FROM N4.DB1.SD._N1_T
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Application Interface
 The application interface manipulates scalable
tables through SD-SQL Server commands
 The SD-SQL Server commands start with ‘sd_’ to
distinguish from SQL Server commands for static
tables
INSERT
sd_insert
CREATE TABLE
BNCOD 2006 – July 18th, 2006
sd_create_table
An Overview of a Scalable Distributed Database System: SD-SQL Server
Nodes Management

Node Creation
sd_create_node ‘Dell1’ /* Server by default */
 sd_create_node ‘Ceria’, ‘client’


Node Alteration

sd_alter_node ‘Ceria’, ‘ADD server’ /* Becomes
peer*/

Node Removal

sd_drop_node ‘Ceria’
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
SDB & NDB Management

SDB Creation

sd_create_scalable_database
‘SkyServer’, ‘Dell1’, ‘Server’, 2
/* Creates the primary SkyServer NDB as well at Dell1*/

SDB Alteration


sd_create_node_database
‘SkyServer’, ‘Ceria’, ‘Client’
SDB Removal

sd_drop_scalable_database ‘SkyServer’
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable Tables Management

Scalable Table Creation

sd_create_table ‘PhotoObj (objid BIGINT
PRIMARY KEY…)’, 10000
• No foreign keys yet

Scalable Table Alteration
sd_alter_table ‘PhotoObj ADD t INT’, 1000
 sd_create_index ‘run_index ON Photoobj (run)’
 sd_drop_index ‘PhotoObj.run_index’


Scalable Table Removal

sd_drop_table ‘PhotoObj’
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Image Adjustment

Secondary Image Creation
sd_create_image ‘Ceria’, ‘PhotoObj’
 sd_create_image ‘Dell2’, ‘PhotoObj’


Secondary Image Removal

sd_drop_image 'PhotoObj’
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable View

A view of an image
Involving perhaps static tables
 And perhaps static views
…


Declared under SD-SQL Server by the SQL
Server CREATE VIEW command
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable Queries Management
USE SkyServer

Scalable Update Queries


/* SQL Server command */
sd_insert ‘INTO PhotoObj SELECT * FROM
Ceria5.Skyserver-S.PhotoObj’
Scalable Search Queries
sd_select ‘* FROM PhotoObj’
 sd_select ‘TOP 5000 * INTO PhotoObj1
FROM PhotoObj’, 500

BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Command Processing

Let Q a scalable query using the PhotoObj image:

sd_select ‘COUNT (*) FROM PhotoObj’
Find Images in Q
Image
Binding
Check PhotoObj Image for Correctness
Adjust PhotoObj Image if needed
Send Q’ to SQL Server for Execution
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Concurrency
 SD-SQL Server processes every command as SQL
distributed transaction at Repeatable Read
isolation level
 Tuple level locks
 Shared locks
 Exclusive 2PL locks
 Much less blocking than the Serializable Level
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Concurrency
 Splits use exclusive locks on segments and
tuples in RP meta-table.
 Shared locks on other meta-tables: Primary, NDB
meta-tables
 Scalable queries use basically shared locks
on meta-tables and any other table involved
 All the conccurent executions can be shown
serializable
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Concurrency: Example
Splitter
X
Exclusive Lock
Dell1
RP
sd_alter_table
Dell2
Waiting
Shared
Lock
Dell3
X
Exclusive Lock
Exclusive Lock
PhotoObj
Dell1.SkyServer
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Experimental Environment
 6 Machines Pentium IV 1.7 GHz

RAM: 780 Mb & 1 Gb
 Operating System: Windows 2K Server
 Ethernet Network: max bandwidth of 1 Gb/s
 Use of SQL Analyzer for editing queries
 Use of SQL Profiler to take measurements
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
The SkyServer Benchmark

We use SkyServer database as benchmark


Provided and installed at Ceria by Dr. Gray
SkyServer brings the entire database of the Sloan
Digital Sky Survey, SDSS
We use of the PhotoObj table as an example
scalable table
 In our experiments, PhotoObj has almost 159 K
tuples (about 260 MB)

• Originally, it has 14 M tuples
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Split Time
Splitting PhotoObj with 160 k tuples into 2…5 segments,
according to segment capacity
10000 tuples
20000 tuples
80000 tuples
160000 tuples
40000 tuples
Split Time (sec)
200
165.11
146.57
150
100
50
0
66.53
46.17
22.42
10.54
7.11
2
77.86
56.79
38.87
18.93
12.15
3
104.01
104.15
59.59
60.73
32.08
21.40
37.12
26.42
4
5
Number of Segments
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Split Time
Splitting PhotoObj with 160 k tuples and indexes into 2… 5
segments according to segment capacity
0 index
1 index
2 index
3 index
Execution Time (s)
250
200
150
100
50
0
2
3
4
5
Number of Segments
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Split Time Analysis

Longer split time may timeout a query put on wait

Future solution: Incremental Splitting

The splitter moves tuples by an
increment at a time
• Let us say 1000 tuples

Then ends up by calling upon itself
• The query may proceed as the splitter
releases the exclusive lock on the RP
tuple

The process continues for next
increment etc as long as there are
tuples to move
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Image Adjustment
Temps
d'exécution de
(sec)
(Q) sd_select ‘COUNT (*) FROM PhotoObj’
Adjustment on a Peer
Checking on a Peer
SQL Server Peer
Adjustment on a Client
Checking on a Clientj
SQL Server client
2
1,5
1
0,5
0
39500
79000
158000
Capacité de PhotoObj
Query (Q1) execution time
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable View Processing
(Q) sd_select ‘COUNT (*) FROM Ti’
Execution Time (sec)
With PhotoObj Image Adjustment
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
With only PhotoObj Image Checking
0.811
0.826
0.846
0.853
0.11
0.126
0.14
0.153
Niveau 0
Niveau 1
Niveau 2
Niveau 3
Scalable View Level
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
SD-SQL Server / SQL Server
Temps d'exécution
(sec)
 (Q): sd_select ‘COUNT (*) FROM PhotoObj’
SQL Server-Distr
SD-SQL Server
SQL Server-Centr
SD-SQL Server LSV
500
436
400
300
200
100
0
93
106
93
16
1
203
164
156
76
2
283
226
356
256
220
343
326
250
203
220
4
5
123
3
Nombre de segments
Execution time of (Q) on SQL Server and SD-SQL Server
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Conclusion

Scalable tables are now a reality
with SD-SQL Server
 No more manual repartitioning

• Unlike in any other DBS we know about

Performance analysis proves
Efficiency of our design
 Immediate utility of SD-SQL Server

BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Future Works



SQL Server 2005 portage
Incremental splits
Virtual repository of eGov documents





Foreign keys for scalable tables
More performance measurements
 Skyserver & other benchmarks
Error processing
High availability


SQL Server XML View
Parity segments
Application to other DBMSs

Oracle, DB2, etc.
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server
Thank you for your attention
Work performed between 2003 -2006
Partly founded by
MsResearch
EEC Icons Project
EEC E-Gov Project
BNCOD 2006 – July 18th, 2006
An Overview of a Scalable Distributed Database System: SD-SQL Server