Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Base Content Slide Larry Ellison CEO, Oracle "By having all of the pieces in the stack—from the silicon all the way up to the application—we'll be able to deliver systems that run faster, are faulttolerant, are highly secure—much more secure, much more performance, much more cost-effective, much easier to use than we ever could have delivered by simply delivering components." <Insert Picture Here> Extreme Performance Data Warehousing Çetin Özbütün Vice President, Data Warehousing Technologies The Rise of the Intelligent Economy “From recession comes an opportunity to reset a number of industry structures…there is an opportunity to infuse industries with technologies that position them to operate more effectively in the next 50 years.” Lessons Learned in Building the Intelligent Economy, May 2010 All Businesses Want Better Insight Industry Retail Typical Questions What stores should be closed or sold? Which customers will respond to new promotion? Telecommunications What are the issues effecting churn by region? What is the average revenue per user (ARPU)? Healthcare What are most common patient service requests? What is average level of clinical supplies on-hand? Financial Services How will new online services impact deposits? How does average loan compare to last year? Utilities Who do we target for energy efficiency program? What resources are needed to restore an outage? Public Sector What is the trend on budget and expenditures? What is most cost-effective way to manage waste? Challenge: Much More Data to Analyze Data Warehouse Size and Growth 34% More than 10 TB 17% 25% 3 - 10 TB 19% 18% 21% 1 - 3 TB 12% 500 GB - 1 TB Less than 500 GB 20% 5% 21% In 3 Years Source: TDWI Next Generation Data Warehouse Platforms Report, 2009 Today Challenge: No Single Source of Truth Expensive Data Warehouse Architecture Data Marts OLAP Data Mining ETL Data Marts ETL OLAP Data Mining Challenge: User Requirements Not Met High Churn in Data Warehouse Platforms Poor query response 45% Can't support advanced analytics 40% Inadequate data load speed 39% Can't scale to large data volumes 37% Cost of scaling up is too expensive 33% Poorly suited to real-time or on demand workloads 29% Current platform is a legacy we must phase out 23% Can't support data modeling we need 23% We need platform that supports mixed workloads Source: TDWI Next Generation Data Warehouse Platforms Report, 2009 21% DW Strategy • Single source of truth • Extreme performance • Lower cost of ownership • Deeper Insight DW Strategy • Single source of truth • Extreme performance • Lower cost of ownership • Deeper Insight A Single Source of Truth? Movie location see footnote Consolidate Onto a Single Platform Faster Performance, Single Source of Truth Data Marts Online Analytics ETL Data Mining Oracle Database 11g Oracle Exadata Database Machine Oracle Exadata Database Machine For OLTP, Data Warehousing & Consolidated Workloads • Improve query performance by 10x – Better insight into customer requirements – Expand revenue opportunities • Consolidate OLTP and analytic workloads – Lower admin and maintenance costs – Reduce points of failure • Integrate analytics and data mining – Complex and predictive analytics • Lower risk – Streamline deployment – One support contact Oracle Exadata Database Machine Family Oracle Exadata Database Machine X2-2 Oracle Database Server Grid • 8 2-processor Database Servers – 96 CPU Cores – 768 GB Memory Exadata Storage Server Grid • 14 Storage Servers – 5 TB Smart Flash Cache – 336 TB Disk Storage Unified Server/Storage Network • 40 Gb/sec Infiniband Links Available in full, half, quarter racks Oracle Exadata Database Machine Family Oracle Exadata Database Machine X2-8 Oracle Database Server Grid • 2 8-processor Database Servers – 128 CPU Cores – 2 TB Memory – Oracle Linux or Solaris 11 Express Exadata Storage Server Grid • 14 Storage Servers – 5 TB Smart Flash Cache – 336 TB Disk Storage Unified Server/Storage Network • 40 Gb/sec Infiniband Links Traditional Query Problem What Were Yesterday’s Sales? Select sum(sales) where salesdate= ‘22-Jan-2010’… Return entire Sales table Discard most of sales table Sum • Data is pushed to database server for processing • I/O rates are limited by speed and number of disk drives • Network bandwidth is strained, limiting performance and concurrency Exadata Smart Scan Improve Query Performance by 10x or More What Were Yesterday’s Sales? Select sum(sales) where salesdate= ‘22-Jan-2010’… Return Sales for Jan 22 2010 Sum • Off-load data intensive processing to Exadata Storage Server • Exadata Storage Server only returns relevant rows and columns • Wide Infiniband connections eliminate network bottlenecks Exadata Storage Index Transparent I/O Elimination with No Overhead A B C D Index 1 3 Min B = 1 Max B =5 5 5 8 Select * from Table where B<2 Only first set of rows can match Min B = 3 Max B =8 3 • Maintain summary information about table data in memory • Eliminate disk I/Os if MIN / MAX never match “where” clause • Completely automatic and transparent Exadata Hybrid Columnar Compression Reduce Disk Space Requirements 100 90 Data – Terabytes 80 1.4x 70 60 50 40 2.5 x 3x 30 20 10 10x 15x DW Data Archive Data 0 Uncompressed Data Warehouse Data Appliances OLTP Data Oracle Built-in Analytics Secure, Scalable Platform for Advanced Analytics Oracle OLAP Analyze and summarize Oracle Data Mining Uncover and predict • Complex and predictive analytics embedded into Oracle Database 11g • Reduce cost of additional hardware, management resources • Improve performance by eliminating data movement and duplication Exadata Smart Flash Cache Extreme Performance for OLTP Applications Frequently Used Data Infrequently Used Data • Automatically caches frequently-accessed ‘hot’ data in flash storage • Assigns the rest to less expensive disk drives • Know when to avoid trying to cache data that will never be reused • Process data at 50GB/sec and up to 1million I/Os per second Benefits Multiply Converting Terabytes to Gigabytes 10 TB of User Data 1 TB of User Data 100 GB of User Data 10 TB of User Data With 10x Compression With Partition Pruning 20 GB of User Data 5 GB of User Data Sub second “10 TB” Scan With Smart Scan No Indexes 10 TB of User Data With Storage Indexes ETL with Oracle Staging Raw Files BCP Unload FTP Parallel Loads Non-Oracle Source Data Pump Unload SCP Oracle Source • Fast data loading using DBFS and External Tables • Fast transforms in Oracle Database 11g via Parallel DML operations • Best-in-class performance for large batch oriented data loads Turkcell Runs 10x Faster on Exadata Compresses Data Warehouse by 10x • Replaced high-end SMP Server and 10 Storage Cabinets • Reduced Data Warehouse from 250TB to 27TB – Using OLTP & Hybrid Columnar Compression – Ready for future growth where data doubles every year • Experiencing 10x faster query performance – Delivering over 50,000 reports per month – Average report runs reduced from 27 to 2.5 mins – Up to 400x performance gain on some reports Softbank Runs 2x–8x Faster on Exadata 36 Teradata Racks Replaced by 3 Exadata Racks Teradata 36 Racks Exadata 3 Racks Workload Management for DW Setting Up a Workload Management System Workload Management Define Workload Plans Define Workloads Filter Exceptions Manage Resources Execute Workloads Monitor Workloads Adjust Plans RAC IORM Adjust Workload Plans OEM DBRM Monitor Workloads Workload Management Request Each request: • Executes on a RAC Service • Which limits the physical resources • Allows scalability across racks Assign Each consumer group has: Each request assigned to a consumer group: •OS or DB Username •Application or Module •Action within Module •Administrative function •Resource Allocation (example: 10% of CPU/IO resources) •Directives (example: 20 active sessions) •Thresholds (example: no jobs longer than 2 min) Ad-hoc Workload Downgrade Queue Reject Execute Workload Management Real-Time ETL Batch ETL Request Analytic Reports Assign Execute OLTP Requests Ad-hoc Workload Downgrade Queue Reject Workload Management Real-Time ETL Queue R-T 10% Batch ETL Queue Analytic Reports Queue Analytic Reports 50% OLTP Requests Queue OLTP 5% Batch 10% Request Assign Ad-hoc 25% Ad-hoc Workload Downgrade Queue Reject Movie location see Oracle Exadata for Data Warehousing footnote Oracle Exadata Momentum Rapid adoption in all geographies and industries Oracle Database 11g The Best Database for Data Warehousing Real Application Clusters Advanced Compression Partitioning OLAP Data Mining • World record performance for fast access to information • Manage growing volumes of information cost-effectively • Reduce costs through server and data consolidation The Concept of Partitioning Maintain Consistent Performance as Database Grows SALES SALES SALES Europe USA Jan Feb Jan Feb Large Table Partition Composite Partition • Difficult to Manage • Divide and Conquer • Higher Performance • Easier to Manage • Match to business needs • Improve Performance Partition for Performance Partition Pruning Sales Table 5/19 What was the total sales amount for May 20 and May 21 2010? Select sum(sales_amount) From SALES 5/20 Where sales_date between to_date(‘05/20/2010’,’MM/DD/YYYY’) And to_date(‘05/22/2010’,’MM/DD/YYYY’); 5/21 5/22 • Performs operations only on relevant partitions • Dramatically reduces amount of data retrieved from disk • Improves query performance and optimizes resource utilization Partition to Manage Data Growth Compress Data and Lower Storage Costs Archive Data Read Only Data Active Data 15-50x Archive Compression 10-15x DW Compression 3x OLTP Compression • Distribute partitions across multiple compression tiers • Free up storage space and execute queries faster • No changes to existing applications In-Memory Parallel Execution Efficient use of memory on clustered servers In-Memory Parallel Query in Database Tier • Compress more data into available memory on cluster • Intelligent algorithm – Places table fragments in memory on different nodes • Reduces disk IO and speeds query execution © 2010 Oracle Corporation Automated Degree of Parallelism Queue statements if not enough parallel servers available 64 32 16 When required number of servers are available, execute first statement Automatically determine DOP 8 Enough parallel servers available Execute immediately • Optimizer derives the best Degree of Parallelism • Based on resource requirements of all concurrent operations • Less DBA management, better resource utilization Summary Management Improve Response Time with Materialized Views Region SQL Query Date Query Rewrite Products Relational Star Schema Sales by Region Sales by Date Sales by Product Sales by Channel Channel Materialized Views • Pre-summarized information stored within Oracle Database 11g • Separate database object, transparent to queries • Supports sophisticated transparent query rewrite • Fast incremental refresh of changed data Cube Organized Materialized Views Region SQL Query Summaries Date Query Rewrite Automatic Refresh Products Channel • Exposes Oracle OLAP cubes as relational materialized views • Provides SQL access to data stored in an OLAP cubes • Any BI tool or SQL application can leverage OLAP cubes DW Strategy • Single source of truth • Extreme performance • Lower cost of ownership • Deeper Insight In-database Analytics Bring Algorithms to the Data, Not Data to the Algorithms • Analytic computations done in the database – Dimensional analysis – Statistical analysis – Data Mining OLAP Statistics Data Mining • • • • Scalability Security Backup & Recovery Simplicity Oracle OLAP Built-in Access to Analytic Calculations • How do sales in the Western region this quarter compare with sales a year ago? • What will sales next quarter be? • What factors can we alter to improve the sales forecast? • Multidimensional analytic engine that analyzes summary data • Offers improved query performance and fast, incremental updates • Embedded in Oracle Database instance and storage Oracle OLAP and OBIEE Calculations Computed Faster in OLAP Engine Oracle Data Mining Find Hidden Patterns, Make Predictions Retail Financial Services • Customer Segmentation • Response Modeling • Credit Scoring • Possibility of default Communications Utilities • Customer churn • Network intrusion • Product bundling • Predict power line failure Healthcare Public Sector • Patient outcome prediction • Fraud detection • Tax fraud • Crime analysis • Collection of data mining algorithms that solve business problems • Simplifies development of predictive BI applications • Embedded in Oracle Database instance and storage Oracle Data Mining and OBIEE Prediction and Probability Results Integrated in Reports Oracle Spatial and OBIEE • Enrich BI with map visualization of Oracle Spatial data • Enable location analysis in reporting, alerts and notifications • Use maps to guide data navigation, filtering and drill-down • Increase ROI from geospatial and non-spatial data Oracle Exadata Intelligent Warehouse For Industries Data Models Business Intelligence Exadata • Combine deep industry knowledge with data warehousing expertise • Help jump-start design and implementation of data warehouses • Available for Retail and Communications industries Oracle Industry Data Models Reference Data Model Aggregate Data Model Derived Data Model Relational (STAR) for BI OLAP for Analytical Data Mining/Complex Reports/Query Base Data Model (3NF) Atomic Level of Transaction Data • Combine deep industry knowledge with data warehousing expertise • Help jump-start design and implementation of data warehouses • Optimized for Oracle Database 11g and Oracle Exadata Oracle Data Warehousing What Customers Think… Movie location see footnote Henry Lovoy Data Manager HealthSouth Corporation “Oracle Database 11g, along with Oracle Real Application Clusters, Advanced Compression and Partitioning, all lend themselves to delivering highly available, high performance data warehousing.” Extreme Performance Data Warehousing Integrated Technology Stack BI Applications • Single source of truth BI Tools ELT Tools Data Models • Easy to deploy and manage • Extreme performance • Meets all end user requirements • Lower cost of ownership Database Smart Storage Data Warehouse Reference Architecture Data Warehouse Reference Architecture Base data warehouse schema Atomic-level data, 3nf design Supports general end-user queries Data feeds to all dependent systems Application-specific performance structures Summary data / materialized views Dimensional view of data Supports specific end-users, tools, and applications Oracle #1 for Data Warehousing Source: IDC, July 2009 – “Worldwide Data Warehouse Management Tools 2008 Vendor Shares”