Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
An Overview of a Scalable Distributed Database System: SD-SQL Server Witold LITWIN, Soror SAHRI & Thomas SCHWARZ [email protected] [email protected] Ceria Laboratory Paris-Dauphine University BNCOD 2006 [email protected] Comp. Eng. Dep. Santa Clara U. Overview 1. Introduction 2. Architecture 3. Command Interface 4. Processing 5. Performance 6. Conclusion & Future Work BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Partitioned Tables Most DBSs have distributed/parallel versions with partitioned tables SQL Server, Oracle, DB2, MySQL, Postgres… BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server BENEFITS OF PARTITIONING BENEFITS OF PARTITIONING Partitioning can provide tremendous benefits to a wide variety of applications by improving manageability, performance, and availability. It is not unusual for partitioning to improve the performance of certain queries or maintenance operations by an order of magnitude. Moreover, partitioning can greatly simplify common administration tasks. Partitioning also enables database designers and administrators to tackle some of the toughest problems posed by cutting-edge applications. Partitioning is a key tool for building multi-terabyte systems or systems with extremely high availability requirements. Partitioning in Oracle Database 10g Release 2 An Oracle White Paper May 2005 BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Hassle of Partitioning DBSs require manual partitioning And manual repartitioning when tables scale-up DBSs do not provide dynamically scalable tables BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Facts http://ceria.dauphine.fr/CERIA-publications.html Research Report, December 2005 [Oracle Database 10g] BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server SD-SQL Server Goal Scalable Distributed Partitioning of Relational Tables Scalable Distributed Database System SD-DBS BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Role Model: an SDDS A scalable distributed data structure Specifically designed for possibly very large data on multi-computers or networks of WSs P2P & Grids in modern vocabulary Why SDDS Role Model? Several SDDS schemes are well-known by now: LH*, RP*, k-RP*, LH*RS…Chord, VBI & most of P2P schemes The domain has over 20.000 references on Google An SD-DBS reuses SDDS design principles With DB management specificity BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server SD-SQL Server The first and yet the only SD-DBS Implements the SD-DBS architecture • Litwin, Schwartz & Risch (2002) Runs on Microsoft SQL Server 2000 Shared Nothing Architecture Up to 250 nodes at present BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Gross Architecture User/Application User/Application sd_insert sd_create_table SD-SQL Server Managers Linked SQL Servers SD-SQL client SD-SQL server SD-SQL server S S C D1_T P T _D1_T NDBs SD-SQL peer D1 BNCOD 2006 – July 18th, 2006 D2 _D1_T Di Split _D1_T Di+1 An Overview of a Scalable Distributed Database System: SD-SQL Server Nodes, SDBs & NDBs MDB Node1 Node2 DB1 DB2 DB1 SDB DB2 SDB BNCOD 2006 – July 18th, 2006 Node3 DB1 DB2 …… Nodei DB1 An Overview of a Scalable Distributed Database System: SD-SQL Server NDB Types Client NDB • Interfaces applications & users • Carries only images • No actual tables with application data Server NDB • Carries actual tables • segments Primary NDB First for an SDB Peer NDB • Both functions BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable (Distributed) Table For the application: a table of an SDB Internally: a collection of segments behind client images A segment is an SQL table • One per NDB of the SDB • Sharing the scalable table scheme • Except its check constraint • Min and Max value of the partition key • With size limit • Splitting when overflows occur The check constraints partition the partition key space BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable (Distributed) Table The primary segment First allocated for a new table At some server or peer NDB of SDB • The peer creating the table • The primary server of the client creating the table BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Gross Architecture User/Application User/Application sd_insert sd_create_table SD-SQL Server Managers Linked SQL Servers SD-SQL client SD-SQL server SD-SQL server S S C D1_T P T _D1_T NDBs SD-SQL peer D1 BNCOD 2006 – July 18th, 2006 D2 _D1_T Di Split _D1_T Di+1 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable (Distributed) Table For the application, it the client image is the table The image name is the table name Primary image Created during the scalable table creation • at the client or peer NDB creating the table Secondary images Created later on On other NDBs of the SDB For local applications By a dedicated command • sd_Create Image…. BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable (Distributed) Table Internally, every image is a specific SQL Server view of the segments: Distributed partitioned union view CREATE VIEW T AS SELECT * FROM N2.DB1.SD._N1_T UNION ALL SELECT * FROM N3.DB1.SD._N1_T UNION ALL SELECT * FROM N4.DB1.SD._N1_T Updatable • Through the check constraints With or without Lazy Schema Validation BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Gross Architecture User/Application User/Application sd_insert sd_create_table SD-SQL Server Managers Linked SQL Servers SD-SQL client SD-SQL server SD-SQL server S S C D1_T P T _D1_T NDBs SD-SQL peer D1 BNCOD 2006 – July 18th, 2006 D2 _D1_T Di Split _D1_T Di+1 An Overview of a Scalable Distributed Database System: SD-SQL Server SD SQL Server Meta-Tables Store various SD-SQL Server meta-data In particular about each scalable table At each server or peer NDB • SD.Size meta-table • Segment capacity • The number of stored tuples triggering a split • Same for every segment at present • SD.RP meta-table • the actual partitioning of the scalable table • The location of each segment • SD.Primary table • The location of the SD.RP table for each segment in the NDB BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server SD SQL Server Meta-Tables At every client or peer NDB In SD.Image table • All the local images • The name of the image • The type • Primary or secondary • The number of segments • As seen by an image • Not necessarily the actual one BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server SD SQL Server Meta-Tables At every NDB SD.SDBNode points towards the primary NDB SD.MDBNode points towards the MDB At MDB SD.Nodes indicates all the available SD-SQL Server nodes • Over linked SQL Server nodes SD.SDB describes all the SDBs At every primary NDB SD.NDB points to every NDB of the SDB BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable Tables: Meta-data DB1 SDB ……. N1.DB1 N2.DB1 N3.DB1 Ni.DB1 Primary T Scalable Table Size SDBNode Nodes N1.DB1 1000 Ni.DB1 N1.DB1 RP N2.DB1 N3.DB1 Meta-Tables BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable Table Expansion The number of segments in a scalable table may grow An overflowing segment splits • Creating one or more new segments A split occurs when an insert overflows the segment capacity The trigger launches the split as an asynchronous job called splitter To avoid the application level timeout BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Gross Architecture User/Application User/Application sd_insert sd_create_table SD-SQL Server Managers Linked SQL Servers SD-SQL client SD-SQL server SD-SQL server S S C D1_T P T _D1_T NDBs SD-SQL peer D1 BNCOD 2006 – July 18th, 2006 D2 _D1_T Di Split _D1_T Di+1 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable Table Expansion Every new segment Is basically created at an existing NDB that does not yet have any segments of the expanding table • provided there is any Otherwise a new NDB is first appended to SDB • Provided there is an available SD SQL Server node Inherits the “father”’s schema Gets its new check constraint Gets indexes as defined at the “father” BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Single Segment Split Single Tuple Insert Check Constraint? p=INT(b/2) C( S)= { c: c < h = c (b+1-p)} C( S1)={c: c > = c (b+1-p)} b+1 b b+1-p S p S S1 SELECT TOP Pi * INTO FROM S ORDER BYBY C ASC SELECT TOP Pi * WITH TIES INTONi.Si Ni.S1 FROM S ORDER C ASC BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Single Segment Split Bulk Insert (a) (b) b+t b (c) P1 b b b b Pn b+t-np p b+t-np S S S p P1 Pn S1 SN p = INT(b/2) C(S) = {c: l < c < h } { c: l ≤ c < h’ = c (b+t–Np)} C(S1) = {c: c (b+t-p) < c < h} … C(SN) = {c: c (b+t-Np) ≤ c < c (b+t-(N-1)p)} Single segment split BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Multi-Segment Split Bulk Insert b b b b b b p p S Sk S1 S1, n1 Sk Sk, nk Multi-segment split BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Split with SDB Expansion sd_create_node_database sd_create_node N1 N2 N3 N4 Ni NDB NDB NDB NDB NDB DB1 DB1 DB1 DB1 DB1 SDB DB1 sd_insert sd_insert sd_insert ……. Scalable Table T BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Image Adjustment The splits do not modify synchronously the images Any split makes every image outdated The client or peer verifies every image dynamically when a query to the image comes in Image checking Image adjustment if necessary BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Image Adjustment Get the number of segments presented in the image, N1 Get the number of segments of the scalable table, N2 Compare N1 and N2: If N1<N2 then Image Adjustment Alter the partitioned view definition BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Image: Example DB1 SDB N1.DB1 Primary Image N2.DB1 N3.DB1 N4.DB1 T Image T Scalable Table CREATE VIEW T AS SELECT * FROM N2.DB1.SD._N1_T UNION ALL SELECT * FROM N3.DB1.SD._N1_T UNION ALL SELECT * FROM N4.DB1.SD._N1_T BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Application Interface The application interface manipulates scalable tables through SD-SQL Server commands The SD-SQL Server commands start with ‘sd_’ to distinguish from SQL Server commands for static tables INSERT sd_insert CREATE TABLE BNCOD 2006 – July 18th, 2006 sd_create_table An Overview of a Scalable Distributed Database System: SD-SQL Server Nodes Management Node Creation sd_create_node ‘Dell1’ /* Server by default */ sd_create_node ‘Ceria’, ‘client’ Node Alteration sd_alter_node ‘Ceria’, ‘ADD server’ /* Becomes peer*/ Node Removal sd_drop_node ‘Ceria’ BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server SDB & NDB Management SDB Creation sd_create_scalable_database ‘SkyServer’, ‘Dell1’, ‘Server’, 2 /* Creates the primary SkyServer NDB as well at Dell1*/ SDB Alteration sd_create_node_database ‘SkyServer’, ‘Ceria’, ‘Client’ SDB Removal sd_drop_scalable_database ‘SkyServer’ BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable Tables Management Scalable Table Creation sd_create_table ‘PhotoObj (objid BIGINT PRIMARY KEY…)’, 10000 • No foreign keys yet Scalable Table Alteration sd_alter_table ‘PhotoObj ADD t INT’, 1000 sd_create_index ‘run_index ON Photoobj (run)’ sd_drop_index ‘PhotoObj.run_index’ Scalable Table Removal sd_drop_table ‘PhotoObj’ BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Image Adjustment Secondary Image Creation sd_create_image ‘Ceria’, ‘PhotoObj’ sd_create_image ‘Dell2’, ‘PhotoObj’ Secondary Image Removal sd_drop_image 'PhotoObj’ BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable View A view of an image Involving perhaps static tables And perhaps static views … Declared under SD-SQL Server by the SQL Server CREATE VIEW command BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable Queries Management USE SkyServer Scalable Update Queries /* SQL Server command */ sd_insert ‘INTO PhotoObj SELECT * FROM Ceria5.Skyserver-S.PhotoObj’ Scalable Search Queries sd_select ‘* FROM PhotoObj’ sd_select ‘TOP 5000 * INTO PhotoObj1 FROM PhotoObj’, 500 BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Command Processing Let Q a scalable query using the PhotoObj image: sd_select ‘COUNT (*) FROM PhotoObj’ Find Images in Q Image Binding Check PhotoObj Image for Correctness Adjust PhotoObj Image if needed Send Q’ to SQL Server for Execution BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Concurrency SD-SQL Server processes every command as SQL distributed transaction at Repeatable Read isolation level Tuple level locks Shared locks Exclusive 2PL locks Much less blocking than the Serializable Level BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Concurrency Splits use exclusive locks on segments and tuples in RP meta-table. Shared locks on other meta-tables: Primary, NDB meta-tables Scalable queries use basically shared locks on meta-tables and any other table involved All the conccurent executions can be shown serializable BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Concurrency: Example Splitter X Exclusive Lock Dell1 RP sd_alter_table Dell2 Waiting Shared Lock Dell3 X Exclusive Lock Exclusive Lock PhotoObj Dell1.SkyServer BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Experimental Environment 6 Machines Pentium IV 1.7 GHz RAM: 780 Mb & 1 Gb Operating System: Windows 2K Server Ethernet Network: max bandwidth of 1 Gb/s Use of SQL Analyzer for editing queries Use of SQL Profiler to take measurements BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server The SkyServer Benchmark We use SkyServer database as benchmark Provided and installed at Ceria by Dr. Gray SkyServer brings the entire database of the Sloan Digital Sky Survey, SDSS We use of the PhotoObj table as an example scalable table In our experiments, PhotoObj has almost 159 K tuples (about 260 MB) • Originally, it has 14 M tuples BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Split Time Splitting PhotoObj with 160 k tuples into 2…5 segments, according to segment capacity 10000 tuples 20000 tuples 80000 tuples 160000 tuples 40000 tuples Split Time (sec) 200 165.11 146.57 150 100 50 0 66.53 46.17 22.42 10.54 7.11 2 77.86 56.79 38.87 18.93 12.15 3 104.01 104.15 59.59 60.73 32.08 21.40 37.12 26.42 4 5 Number of Segments BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Split Time Splitting PhotoObj with 160 k tuples and indexes into 2… 5 segments according to segment capacity 0 index 1 index 2 index 3 index Execution Time (s) 250 200 150 100 50 0 2 3 4 5 Number of Segments BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Split Time Analysis Longer split time may timeout a query put on wait Future solution: Incremental Splitting The splitter moves tuples by an increment at a time • Let us say 1000 tuples Then ends up by calling upon itself • The query may proceed as the splitter releases the exclusive lock on the RP tuple The process continues for next increment etc as long as there are tuples to move BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Image Adjustment Temps d'exécution de (sec) (Q) sd_select ‘COUNT (*) FROM PhotoObj’ Adjustment on a Peer Checking on a Peer SQL Server Peer Adjustment on a Client Checking on a Clientj SQL Server client 2 1,5 1 0,5 0 39500 79000 158000 Capacité de PhotoObj Query (Q1) execution time BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Scalable View Processing (Q) sd_select ‘COUNT (*) FROM Ti’ Execution Time (sec) With PhotoObj Image Adjustment 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 With only PhotoObj Image Checking 0.811 0.826 0.846 0.853 0.11 0.126 0.14 0.153 Niveau 0 Niveau 1 Niveau 2 Niveau 3 Scalable View Level BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server SD-SQL Server / SQL Server Temps d'exécution (sec) (Q): sd_select ‘COUNT (*) FROM PhotoObj’ SQL Server-Distr SD-SQL Server SQL Server-Centr SD-SQL Server LSV 500 436 400 300 200 100 0 93 106 93 16 1 203 164 156 76 2 283 226 356 256 220 343 326 250 203 220 4 5 123 3 Nombre de segments Execution time of (Q) on SQL Server and SD-SQL Server BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Conclusion Scalable tables are now a reality with SD-SQL Server No more manual repartitioning • Unlike in any other DBS we know about Performance analysis proves Efficiency of our design Immediate utility of SD-SQL Server BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Future Works SQL Server 2005 portage Incremental splits Virtual repository of eGov documents Foreign keys for scalable tables More performance measurements Skyserver & other benchmarks Error processing High availability SQL Server XML View Parity segments Application to other DBMSs Oracle, DB2, etc. BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server Thank you for your attention Work performed between 2003 -2006 Partly founded by MsResearch EEC Icons Project EEC E-Gov Project BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server