Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Entity–attribute–value model wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Functional Database Model wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Relational model wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
SQL Clone Redgate's New Tool for SQL Server Database Provisioning Who am I? Kevin G. Boles SQL Server Consultant “The Wolfe” for the Relational Engine Indicium Resources, Inc. @TheSQLGuru [email protected] MCITP, MCT, MVP 2007-2012, yada-yada What do I do? • World-class Better than average relational engine expert • Over 45,000 man-hours invested in SQL Server • Absolutely LOVE doing targeted performance analysis and tuning work • VERY good at making others better at interacting with SQL Server • Couldn’t SSAS my way out of a paper bag … with both ends open! Agenda • Why database provisioning? • Who does SQL Clone benefit? • How does SQL Clone work? • Demos • Usage Scenarios SQL Server Provisioning Study - 2017 Database Sizes NOTE: This is just dev/test. There are undoubtedly many large databases not being dropped down due to time/space constraints Why Database Provisioning? • Quality (at speed). Realistic testing with repeatable processes. Move problems ‘left’. • Replace current (painful) processes. • It’s a job that’s ‘delegated up’. Problems: Quality at Speed • Matching production volumes • Mirroring data distribution of real data • Repeatable processes • Time taken Problems: Drive Space • Multiple teams and multiple branches • Realistic data • Isolated environments • N times the amount of production data Problems: Pressure on Environments • Multiple development teams • Multiple developers per team • Multiple branches of code • Main branch • Hot fixes • New Development • Automated testing and QA Problems: Regulatory Compliance • Business regulations are increasing • Health Care • Banking • Insurance • International compliance (GDPR) • Cost of data breaches! Challenges of Using Realistic Data • Shouldn’t be on production (it is a pet peeve of mine to see test databases on production box!) • Moving/restoring large amounts of data takes time • Storing large amounts of data costs money • Data may be sensitive Primary Target Audience • Who is responsible for data provisioning? • DBA • IT Manager • Head of Data Services • Chief Data Officer Who Benefits from Production-realistic Data? • Developers • Testers/QA • BI/Reporting teams • Performance Troubleshooting staff What Is SQL Clone? “SQL Clone is a SQL Server database cloning tool that allows you to create editable copies of databases within seconds while using very little additional disk space.” Technical details and architecture Volume Shadow Copy (VSS) • 64-bit Windows technology • Back up/Copy a file/disk at a point in time while in use • Go to that point in time quickly • Other uses: • SQL Server snapshots • VMs Virtual Disk Service? • 64-bit Windows technology • Allocate logical storage without taking disk space • Mount remote storage as local • Mount read-only with differencing disks • Virtual hard drive ‘chaining’ Image Creation In Action • Create an initial, small VHD, at the data image location (shown as 10MB in size in Figure 8) • Create a mount point to this VHD on the local operating system of the machine hosting the source database • Use the VSC service to “remote copy” the MDF and LDF files to the data image location • Un-mount the VHD DEMOS Solutions: Realistic Data • Clean the data as necessary • Create a SQL Clone database copy • Share that copy, including realistic data, with all teams • Troubleshoot not on production Solutions: Drive Space • Clones are small (save up to 99% of disk space) • Differential storage only • Refresh of an environment resets space used Solutions: Shared Environments Isolated Environments! • Database is a mutable object. Multiple people can cause conflicts, breakages, etc • Multiple clones per data image • Multiple environments • Clones are very small • Creating clones takes seconds Solutions: Regulatory Compliance • Clean data in a clone or a copy of database • Create new data image • Distribute the cleaned data image as clones • Works with database internal processes such as Dynamic Data Masking (if your security is the same) Solutions: Data Obfuscation Clean data prior to clone process OR 1. Create a clone from a data image 2. Clean the clone 3. Create a data image of the cleaned clone 4. Clone the new data image (Pursuing clone chaining) Solutions: A/B Testing Clone A Series A Mods Comparative Analytics or Testing \\FileShare\Images\Financial Clone B Series B Mods Solutions: Do More • Faster than normal restores • Smaller than normal restores • Fully automatable • Self-service Features & limitations V1 features • Central Management Server • Image from live DB • Image from backup file • Image from Clone – full sized • Automation – PowerShell cmdlets • Permissions – Basic V1 limitations • 2TB image limit • Image from backup with passwords • 64 bit Windows 2008R2/7 upwards only • SQL Server 2008 upwards only* • Beware default image location in C drive * Clones do not circumvent Express 10GB database size limit since SQL Server still perceives the clone as its original size. Data Masking Workflows • Image > Clone > Mask > Image • In V1 and Image of Clone is full size • Data Generator • Custom script • Active Research for Masking and Chaining Roadmap • Advanced access and permissions • Create new image from a Clone using chaining • Potential: • Incrementing images from changes • Extending the 2TB limit • Data masking SQL Clone Goals 1. Encourage you to give your developers, analysts, troubleshooters, etc. the necessary data for more accurate and timely work 2. Provide a mechanism for automating and managing provisioning of development/test/reporting databases 3. Enable you to better support a DevOps approach within your environment Bonus Round • SQL Clone Competition (25 Amazon Echo Dots, 5 Clone Licenses!!). Open until April 30! • https://www.red-gate.com/blog/sql-clone-competition What About TDE?? • Extracting image from live database works fine • Creating a clone database requires a Database Master Key (can be different from source server) and restores of the original Certificate and Database Encryption Key • Extracting from backup also requires DMK and Cert Packaging and Pricing • The product is clearly team focused, so it is sold in user packs • Pricing scales as number of seats goes up • UNLIMITED usage regarding Images and Clones • Primary other entry in this market is Delphix. It is upscale and does already have data masking, but it also runs six-figures SQL Clone Licensing [email protected] Questions? Links http://www.red-gate.com/products/dba/sql-clone/database-provisioning-report https://www.red-gate.com/blog/database-lifecycle-management/sql-clone-anddatabases-protected-with-tde https://www.red-gate.com/blog/database-lifecycle-management/sql-clone-and-backupsprotected-with-tde Tell them I sent you!!! I may get a small spiff - and I may split it with you! [email protected] @TheSQLGuru