Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Serializability wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Functional Database Model wikipedia , lookup
Relational model wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Concurrency control wikipedia , lookup
Database model wikipedia , lookup
White Paper HIGH PERFORMANCE SOLUTION FOR ORACLE 11g RAC WITH XTREMCACHE 2.0.1 ON VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition Optimum IOPS for OLTP workloads Sustained Average Response Time Less Than One Millisecond EMC Solutions Abstract This solution demonstrates the benefits of deploying EMC® XtremCache™ and VMAX® to increase IOPS and decrease latency for OLTP databases. It provides scalability, high performance, and ease of use for mission-critical business demands. December 2013 Copyright © 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H11828.2 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 2 Table of contents Executive summary............................................................................................................................... 5 The business challenge.................................................................................................................... 5 The technology solution: open, best-in-class components ............................................................... 5 Open standards benefits ............................................................................................................. 6 Operational advantages .............................................................................................................. 6 Solution overview ............................................................................................................................ 6 Key results and recommendations ................................................................................................... 6 Introduction.......................................................................................................................................... 8 Purpose ........................................................................................................................................... 8 Scope .............................................................................................................................................. 8 Audience ......................................................................................................................................... 8 Terminology ..................................................................................................................................... 9 Technology overview .......................................................................................................................... 10 Architecture layers ......................................................................................................................... 10 Architecture diagram ...................................................................................................................... 10 EMC Oracle Performance Solution: Server layer ................................................................................. 11 Server hardware ............................................................................................................................. 12 Server software .............................................................................................................................. 13 Red Hat Enterprise Linux ................................................................................................................ 13 Oracle Grid Infrastructure and Database 11g Release 2 .................................................................. 14 XtremSF and XtremCache deployment ............................................................................................ 14 XtremSF and XtremCache overview ................................................................................................ 14 EMC XtremCache configuration.................................................................................................. 15 EMC Oracle Performance Solution: Network layer .............................................................................. 16 EMC Oracle Performance Solution: Storage layer ............................................................................... 18 Introduction ................................................................................................................................... 18 Storage components ...................................................................................................................... 18 Storage hardware........................................................................................................................... 18 EMC Symmetrix VMAX 40K eight-engine configuration .................................................................. 19 Storage software ............................................................................................................................ 20 EMC Oracle Performance Solution: Oracle Database layer .................................................................. 21 Introduction ................................................................................................................................... 21 Storage virtual provisioning design ................................................................................................ 21 Drive type....................................................................................................................................... 21 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 3 ASM disk group configuration for OLTP database ........................................................................... 21 OLTP database and workload profile .............................................................................................. 22 EMC Oracle Performance Solution: Performance testing .................................................................... 23 Overview ........................................................................................................................................ 23 Test objective................................................................................................................................. 24 Query only test scenarios and methodology ................................................................................... 24 Read test results ............................................................................................................................ 24 Update only test scenarios and methodology ................................................................................. 26 Peak write test results .................................................................................................................... 27 Conclusion ......................................................................................................................................... 29 Summary ....................................................................................................................................... 29 Findings ......................................................................................................................................... 29 References.......................................................................................................................................... 30 EMC documentation ....................................................................................................................... 30 Oracle documentation.................................................................................................................... 30 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 4 Executive summary The business challenge The technology solution: open, best-in-class components Customers require an open, scalable, tiered, highly available, and high performance infrastructure to run their critical Oracle systems. IT organizations must strive for better performance and increased efficiency in their Oracle infrastructure and Oracle database and storage administration operations, including the following requirements: Reduce capital expenditures and operational expenditures by deploying an open, non lock-in technology Consolidate many Oracle databases (Oracle database versions 10gR1 to 11gR2 and Oracle 12c) and database workloads, including OLTP and Data Warehouse, to maximize the efficiency of the data center infrastructure Deliver maximum performance while effectively utilizing the existing arrays and Oracle infrastructure Maintain the highest performance levels and provide predictable performance to deliver the quality of service required in these Oracle mixed workload environments The EMC Proven High Performance Solution for Oracle Real Application Clusters (RAC) on EMC Symmetrix® VMAX® is an open architecture that incorporates open, best-inclass Intel servers with EMC server-side flash storage (XtremSF™) and EMC VMAX storage arrays. The solution uses optimal servers to balance performance, scalability, and Oracle license costs. The use of Xtrem™ technologies XtremSF and XtremCache™ software in the servers provides distinct performance and operational advantages over equivalent systems that do not contain server-side flash technologies. XtremCache does the following to accelerate an Oracle RAC environment: Features an ultra-performance tier: Accelerates any application that benefits from low-latency, high bandwidth physical read I/O Hottest data resides on database server flash Data is as close to the Oracle Database server CPU as any storage model will allow Cooperates with Oracle Clusterware: Oracle Clusterware is the final authority on all node membership information in an Oracle RAC deployment Has no awareness of database instances: Has no concern for content of blocks of cached LUNs Only XtremCache nodes can access LUNs cached by XtremCache Does not impose a performance penalty on active transactions for cache insertions or cache coherency High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 5 Offers optimized performance: VMAX arrays are relieved of read IOPS, leaving more bandwidth for handling writes. VMAX FAST enables automatic data placement as data goes from hot to cool usage. Delivers the highest performance levels in the industry: This solution delivers the highest performance for mixed workload Oracle environments. EMC Proven Solutions for Oracle have demonstrated sustained metrics over 3.7 million IOPS with latency of less than half a millisecond for OLTP workload, and Data Warehouse workloads with sustained throughput of 32 GBs with a data load rate of 21 TB/hour. Key results and recommendations on page 7 provides details. Open standards benefits This solution is based on open standards. Advantages resulting from the open standards commitment include the following: Flexible adaptation to existing and future customer needs and open industry standards Lower capital investment and operational expense without vendor lock-in Operational advantages EMC open architecture not only supports different releases of Oracle Database software (10g, 11g, and 12c), but it also provides the capability of running the databases concurrently; that is, it supports database consolidation. Open architecture and flexible adaptation means that application modification is not required for database deployment for this solution, lessening the potential for unforeseen impact to business operations and systemic data flow throughout the enterprise. Solution overview The purpose of the solution is to build an EMC High Performance Solution for Oracle on VMAX infrastructure, based on an open architecture, and demonstrate the following capabilities of the infrastructure: High performance and flexibility Low operational costs Reduced risk This white paper validates the performance of the solution and provides guidelines to build similar solutions. This document is not a comprehensive guide to every aspect of this solution. Key results and recommendations The core advantages of the EMC Proven High Performance Solution for Oracle RAC on VMAX are as follows: Delivers the highest performance for Oracle RAC workload environments. EMC Proven High Performance Solution for Oracle RAC on VMAX has demonstrated sustained metrics over 3.7 million IOPS with latency of less than ½ millisecond for OLTP workload. This impressive performance is achieved by utilizing open High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 6 best-in-class components at the computer, network and storage layers. Details are listed in Table 1: Table 1. IOPS test results with workload when XtremCache is enabled Workload type Performance statistics One node Two nodes Four nodes Eight nodes Read only workload IOPS 457,136 962,155 1,914,963 3,765,176 Response time (ms) 0.74 0.69 0.68 0.75 UPDATE transaction workload Aggregate IOPS 53,492 99,649 190,809 303,330 Write IOPS 26,950 50,110 95,932 153,276 Read IOPS 26,542 49,539 94,877 150,054 Redo throughput (MB/s) 20 37 71 115 Note: Because EMC XtremCache is write through cache, for the UPDATE workload, the data blocks that have been read into buffer cache are accelerated by EMC XtremCache. Meanwhile, the dirty blocks that have been flushed by DBWR are passing through EMC XtremCache and are directly written to the back-end VMAX array. Uses the following EMC technology enablers in the reference architecture: EMC VMAX 40K XtremSF PCIe flash card XtremCache caching software This solution provides a foundation that can be scaled in a flexible, predictable, and nearly linear way using additional server resources, including CPUs and memory, HBA ports, and front-end ports, to provide higher IOPS and throughput based on the configuration in this solution. High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 7 Introduction Purpose The purpose of this white paper is to describe an EMC High Performance Solution for Oracle on VMAX infrastructure based on an open architecture, and to demonstrate the following capabilities of the infrastructure: High performance and flexibility Low operational costs Reduced risk This solution validates the performance of the solution and provides guidelines to build similar solutions. This document is not a comprehensive guide to every aspect of this solution. Scope Audience This white paper serves the following purposes: Introduces the key solution technologies Describes the solution architecture and design Describes the solution test scenarios and present the results of performance testing Identifies the key business benefits of the solution This white paper is intended for chief information officers (CIOs), data center directors, Oracle DBAs, storage administrators, system administrators, technical managers, and any others involved in evaluating, acquiring, managing, operating, or designing Oracle database environments. High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 8 Terminology Table 2 lists the terminology used in this solution paper. Table 2. Terminology Term Definition AWR Automatic Workload Repository ASM Automatic Storage Management DML Data Manipulation Language IOPS I/Os Per Second LUN Logical Unit Number OLTP Online Transaction Processing PCIe Peripheral Component Interconnect Express PGA Program Global Area RAID Redundant Array of Independent Disks RAC Real Application Clusters SATA Serial Advanced Technology Attachment SLOB Silly Little Oracle Benchmark SGA System Global Area High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 9 Technology overview This section describes the architecture of the solution. Architecture layers The solution provides an optimal cost-to-performance ratio for Oracle mission-critical application environments. The architecture is composed of the following layers: Architecture diagram Server–Eight Cisco UCS C240 M3 servers each with: Two 700 GB EMC XtremSF SLC flash card (1400 GB total) EMC XtremSF driver and firmware Network–Two Cisco Director - MDS 9506 switches Storage–EMC Symmetrix VMAX 40K storage system Database–One eight-node Oracle RAC database is installed on the eight servers Figure 1 depicts the EMC Proven High Performance Solution for Oracle RAC on VMAX. Figure 1. Solution architecture High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 10 EMC Oracle Performance Solution: Server layer Comprising the server layer of the solution, eight Cisco UCS C240 M3 servers utilize a total of 128 cores with 2.90 GHz E5-2690 processors, 2.56 TB RAM, and 11 TB XtremSF flash PCIe cards. The Cisco UCS C240 M3 is an enterprise-class rack server designed for performance and expandability. As part of the EMC Proven High Performance Solution for Oracle RAC on VMAX, the server layer enables a highperforming, consolidated approach to an Oracle infrastructure, resulting in deployment flexibility without the need for application modification. Features and benefits include the following: Intel Xeon processor E5-2600 product family for hardened protection for virtual and cloud environments Fully integrated quad-port gigabit Ethernet Figure 2 shows one of the eight Cisco UCS C240 M3 rack servers used in the EMC Proven High Performance Solution for Oracle RAC on VMAX solution. Figure 2. Cisco UCS C240 M3 rack server (1 of 8) High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 11 Server hardware Table 3 describes the various hardware components of the EMC Proven High Performance Solution for Oracle on the VMAX server layer. Table 3. Server hardware Server hardware Cisco UCS C240 M3 Quantity Configuration Description 8 2 x 8-Core Sandy-Bridge E52690 Processors – 512 GB RAM - 4x 200 GB SSD server Servers Each server consists of the following components: PCIEHHS7XXM 2 700 GB SLC PCIe card EMC XtremSF UCSC-C240M3S 1 UCS C240 M3 SFF w/o CPU mem HD PCIe with rail kit expdr Server housing UCS-CPU-E52690 16 2.90 GHz E5-2690/130W 4C/10 MB Cache/DDR3 1600 MHz CPUs UCS-ML1X324RY-A 16 32 GB DDR3-1600 MHz LR DIMM/PC3-12800/quad rank/x4/1.35v DRAM UCSSD200G0KA 2-E 4 200 GB std height 15 mm SATA SSD hot plug/drive sled mounted Internal hard drives UCSC-SD16G-C240 1 16 GB SD card module for C240 servers SD card UCSC-RAIL2U 1 2U rail kit for UCS C-Series servers Rail kit N20-BBLKD 20 UCS 2.5-inch HDD blanking panel HDD panels UCSC-HSC240M3 2 Heat sink for UCS C240 M3 rack server Heat sinks UCSC-PCIF01F 4 Full-height PCIe filler for CSeries PCI slot fillers UCSC-PCIF01H 1 Half-height PCIe filler for UCS PCI slot filler UCSC-RAID11-C240 1 LSI 2008 SAS RAID Mezzanine Card for UCS C240 server RAID card CAB-C13C14-AC 2 Power cord C13 to C14 (recessed receptacle) 10A Power cables UCSC-PSU650W 2 650 W power supply for CSeries rack servers Power supplies High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 12 Server hardware Server software Quantity Configuration Description LPE12004M8 2 Emulex Quad Channel 8Gb FC PCIe HBA Fibre Channel cards E10G42BTDA 1 Intel X520-DA2- Network adapter - PCI Express 2.0 x8 low profile - 10 gigabit Ethernet - 2 ports 10 GbE network card E10GSFPSR 2 Intel Ethernet SFP+ SR Optics - SFP+ transceiver module 1000Base-SX, 10GBase-SR 850 nm Optical ports for FC Table 4 describes the various software components of the solution server layer. Table 4. Red Hat Enterprise Linux Server software Server software Configuration Description Red Hat Enterprise Linux 6.3 Operating system for database servers Oracle Grid Infrastructure 11g Release 2 Enterprise Edition 11.2.0.3 Software provides Clusterware and ASM storage volume management Oracle Database 11g Release 2 Enterprise Edition 11.2.0.3 Oracle Database software EMC XtremCache software 2.0.1 EMC XtremCache software for server-side flash cache Red Hat Enterprise Linux includes enhancements and new capabilities, including developer tools, virtualization features, security, scalability, file systems, and storage. Red Hat Enterprise Linux is a versatile platform that can be deployed on physical systems, as a guest on the major hypervisors, or in the cloud. It supports all leading hardware architectures with compatibility across releases. High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 13 Oracle Grid Infrastructure and Database 11g Release 2 Oracle Database 11gR2 is available in a variety of editions tailored to meet the business and IT needs of an organization. This solution utilizes Oracle Database 11gR2 Enterprise Edition (EE). Oracle Database 11g R2 EE delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. The database supports advanced features such as Virtual Private Database, and data warehousing options such as partitioning and Advanced Analytics, that are either included or available as extra-cost options. XtremSF and XtremCache deployment XtremSF and XtremCache overview In this solution, two 700 GB EMC XtremSF flash cards are used in each RAC node, and one XtremCache cache device is created from one XtremSF card. EMC XtremSF is an advanced flash storage technology deployed in a server and designed to deliver unprecedented performance acceleration by reducing latency and increasing I/O throughput. EMC XtremSF allows applications to access data in the most efficient manner possible. Residing on the server PCIe interconnect bus, XtremSF reduces application response time from milliseconds to microseconds by caching the frequently accessed data at the server side, which reduces the number of I/O requests sent to backend disk array. EMC XtremCache is server flash-caching software that reduces latency and accelerates throughput to dramatically improve application performance when coupled with EMC XtremSF PCIe flash technology. XtremCache accelerates reads and protects data by using a write-through cache policy to the networked storage to deliver persistent high availability, integrity, and disaster recovery. XtremCache coupled with array-based EMC FAST™ software provides the most efficient and intelligent I/O path from the application to the underlying disk array. The result is a networked infrastructure that is dynamically optimized for performance, intelligence, and protection for both physical and virtual environments. Benefits of XtremCache include the following: Provides performance acceleration for read-intensive workloads with the protection of a back-end, networked storage array Provides an intelligent path for the I/O and ensures that the right data is in the XtremCache of the servers at the right time By offloading flash and wear-level management onto the XtremSF PCIe flash card, uses minimal CPU and memory resources from the server Works in both physical and virtual environments Provides better data protection: since XtremCache is a write-through cache, it does not comprise data consistency in the storage array, even if the cards fail in the middle of I/O processing No need to warm up the XtremCache across database instances reboot Works for any kind of I/O; for example, any applications and any database platforms Is supported on various operating systems and server platforms High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 14 Allows customers flexibility in choice of cache capacity on the cards Supports the Oracle RAC database, and even these RAC databases are “stretched” with EMC VPLEX As XtremCache is installed in a greater number of servers in the environment, more I/O processing is offloaded from the storage array to XtremCache configured on the servers. This provides a highly scalable performance model in the storage environment. For more detail, refer to: Introduction to EMC XtremCache for Oracle Real Application Clusters listed in References Introduction to EMC XtremCache for Oracle Real Application Clusters video listed in References EMC XtremCache configuration XtremCache supports Oracle RAC using a distributed cache coherency algorithm. XtremCache automatically recognizes the presence of Oracle RAC, and switches operation to clustering mode. All working Oracle RAC nodes must have XtremCache installed for the distributed Cache feature to come online. EMC recommends using XtremCache with Oracle RAC to cache LUNs that holding data files and TEMP files. Do not use XtremCache to cache redo logs, archives, or clusterware files. Steps for configuring XtremCache devices are shown in the XtremCache manual. High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 15 EMC Oracle Performance Solution: Network layer The switch component level is made up of two Cisco MDS 9506 director-class SAN switches (like the one shown in Figure 3). The Cisco MDS 9506 is designed for deployment in storage networks supporting virtualized data centers and enterprise clouds. The switch combines high performance and low total cost of ownership, a core architectural requirement at all levels of the VMAX Performance Block. Figure 3. Cisco MDS 9506 Multilayer Director The Cisco MDS 9506 also offers these benefits: Highly available scalability through a combination of nondisruptive software upgrades, stateful process failover, and full redundancy of all core components Superior platform for accelerated, intelligent storage applications such as EMC replication and backup, data migration, and storage media encryption Virtual machine transparency and end-to-end visibility all the way from the virtual machine down to the EMC storage, enabling scalable, mobile virtual machines Nexus 5.2(8) software is used in the EMC High Performance Solution for Oracle on VMAX. Table 5 lists the hardware components of the network layer of this solution. High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 16 Table 5. Hardware components of the network layer Network hardware Quantity Configuration Description MDS-PBF-ADV32 8 Cisco 32-port 8-Gbps FC Port Module Line cards MDS-9506-V2 2 Chassis SUP2 no ports Director Director chassis FC10M-50MLC 128 FCHNL 10M 50/125 LC-LC LC adapters MDS-PW19-TWST 2 Cisco 9506 Twist Lock Power Cord US Power cord FC1M-50MLC 96 FCHNL 1M 50/125 LC-LC MDS-8G-SW 192 MDS 2/4/8-Gbps FC shortwave SFP LC MDS-ENT-9500 2 Enterprise license key 9500 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 17 EMC Oracle Performance Solution: Storage layer Introduction EMC is not only the world’s largest storage provider, but also the leader in storage infrastructure in Oracle Database environments. Storage User Demand Study, 2012 – Fall Edition: Many Uses of Internal Storage provides more details. Storage components The storage components comprising the EMC Proven High Performance Solution for Oracle on VMAX include the following: Storage hardware EMC Symmetrix VMAX 40K with eight engines (the specification of the engine is shown in Figure 4) EMC PowerPath® Dynamic cache partitioning (VMAX) Figure 4. EMC Symmetrix VMAX 40K Table 6. EMC Symmetrix VMAX storage hardware Storage hardware Quantity Configuration SD-DBV-DIR-1P 4 VMAX 40K DBV DIR 1 Phase SD-DBV-SPS 16 VMAX 40K SPSV SD-1P 1 VMAX 40K 1P INFRAST SVDBSOLDOR1P 4 VMAX 40K Drive Bay Solid Door 1P SYMV2-MIGRBAS 1 Symmetrix 40K Migration Bundle PP-SE-SYM 1 PPATH SE SYM Description High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 18 EMC Symmetrix VMAX 40K eight-engine configuration Storage hardware Quantity Configuration SD-VCONFIG32 1 VMAX 40K VCONFIG 32 SD-FE80000E 16 VMAX 40K 8 MM 8 G Fibre SD-INTBKVKIT 8 VMAX 40K Internal Cable Bracket Titan SD-PW40U-US 10 30A 1Phase Namer Japan l6-30P Power VL4103001B 376 VMAX 40K 4 G 10K 300 GB SAS Drive Disk drives SD-ADD192C 7 VMAX 40K Add Engine-192GB-C VMAX engine SD-192-BASEC 1 VMAX 40K Base-192GB-C VMAX cache SD-DE25-DIR 64 VMAX 40K 25SLT DR ENCL Description Fibre Ports The EMC Symmetrix VMAX 40K is designed for high efficiency, scalability, and secure data persistence. Built on the strategy of powerful, trusted, and smart storage, and founded in EMC Virtual Matrix Architecture that allows for seamless, cost-effective growth, the EMC Symmetrix VMAX 40K offers the following: Zero downtime migration technology and lower cost and greater efficiency through automated tiering More scalability for less management complexity and operational expense The configuration supporting this solution is described in Table 7: Table 7. EMC Symmetrix VMAX 40K configurations Component Quantity Configuration Engines 8 192 GB cache each, total of 1536 GB raw cache Directors 16 Eight ports on each director with 8 GB Fibre Channel Bays 5 1 system, 4 disk 10K SAS drives 376 100 TB Raw, 45 TB Usable (RAID1 Configured) High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 19 Storage software Table 8 lists the software used in the solution storage layer. Table 8. Storage software Storage software Configuration Description VMAX Enginuity™ code 5876 VMAX micro code EMC Solutions Enabler 7.6 Host CLI storage management software EMC PowerPath 5.7 SP1 Multipathing and load balancing software High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 20 EMC Oracle Performance Solution: Oracle Database layer Introduction In Oracle 11g R2, Oracle ASM and Oracle Clusterware have been integrated into the Oracle Grid Infrastructure. In the solution, we use ASM to store the database files. Storage virtual provisioning design EMC Virtual Provisioning™ automatically stripes data across all data devices in a virtual pool and balances the workload across storage devices. To ensure even striping of data, all data devices in a virtual pool should be the same size. Table 9 shows the RAID selections and number of spindles for each virtual pool. In this solution, Oracle data files and redo log files are located on thin devices using RAID 1 protection and all physical spindles for the best performance and capacity. Table 9. ASM disk group configuration for OLTP database Virtual pool design on VMAX 40K Virtual pool RAID protection Drive type Physical spindles size FC_R1_1 RAID 1 (2-way, mirror) FC 10K 300 GB Number of active spindles 376+ 5 (spare disk) Item CRS, DATA, REDO Table 10 details the RAC database Oracle ASM (automatic storage management) disk group design. On each database, we used three ASM disk groups to store the relevant database files, including data files, control files, online redo log files, and temporary files. Default settings are used for ASM disk groups. Table 10. ASM disk group design for RAC database Item LUN size (GB) Number of LUNs ASM disk group name CRS 10 2 +CRS DATA 1024 18 +DATA REDO 60 4 +REDO High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 21 OLTP database and Table 11 details each OLTP database workload profile for the solution. We used the SLOB toolkit to generate an OLTP database and deliver the OLTP-like workloads, workload profile including the query-only and UPDATE-only workloads required for the solution. Table 11. Database workload profile for each OLTP database Profile characteristic Details Database type OLTP Database size 16 TB Oracle Database 11gR2 eight-node RAC database on ASM Instance configuration for read only workload SGA size: 16 GB Workload profile OLTP-like workload simulated by SLOB Network connectivity 8 Gb Fibre Channel for SAN; 10 GbE for IP Note: Because more SGA causes less I/O activity, we reduced SGA to this small number to generate a stable and high I/O workload High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 22 EMC Oracle Performance Solution: Performance testing Overview The solution characterizes the Oracle OLTP system performance on VMAX array with EMC XtremSF cards installed on database servers. We created an eight-node Oracle RAC database on eight physical servers. Silly Little Oracle Benchmark (SLOB) was used to generate workload because it is the premiere SQL workload for driving maximum physical random I/O from a database platform. SLOB is a SQL-driven Oracle database I/O generator, instead of a synthetic I/O generator. SLOB uniquely drives massive physical I/O using minimal host CPU resources, and it specifically targets the Oracle I/O subsystem. SLOB performs all of its physical I/O buffered in the Oracle SGA. No physical I/O buffered in the Oracle PGA is performed. SLOB possesses the following characteristics: Supports testing Oracle logical read (SGA buffer gets) scaling Supports testing physical, random single-block reads (db file sequential read/db file parallel read) Supports testing random single block writes (db file parallel write) Supports testing extreme REDO logging I/O Consists of simple PL/SQL Is entirely free of all application contention We used SLOB to generate an OLTP-like workload on the eight-node Oracle RAC database to demonstrate sustained flash and storage array IOPS. The database performance metrics including IOPS and latency were gathered primarily from the AWR report. In addition, we gathered metrics for I/O throughput at the server/database and storage levels. Notes: Benchmark results are highly dependent on workload, specific application requirements, and system design and implementation. Relative system performance will vary because of these and other factors. Do not use solution test workloads as a substitute for a specific customer application benchmark when making critical capacity planning or product evaluation decisions All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute. High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 23 Test objective This test measures the physical I/O scalability along with the scaling of the number of concurrent SLOB zero-think-time sessions (simulated concurrent users) and the number of RAC nodes. Multiple concurrent sessions (reader sessions) executing similar query SQL statements were run to validate read only workload, and multiple concurrent sessions (writer sessions) executing similar UPDATE SQL statements were run to validate the physical read/write workload. Query only test scenarios and methodology We gradually increased the number of instances of the RAC database and the number of concurrent users with each user running similar OLTP queries simultaneously. When we added a RAC node, additional resources were added, including CPU power and XtremSF cards. With the addition of new RAC nodes, we tested the system again by running the similar SLOB workload. For this test, workloads were running simultaneously from all the RAC nodes added. We then increased the number of concurrent users and measured the performance scalability. The test process included the following steps: Read test results 1. Run the OLTP query only workload with 64 concurrent simulated users (zerothink-time sessions) on the first node of an eight-node RAC database using SLOB. 2. Add the second node into the system, then run the workload with 64 concurrent users on each node; that is, with 128 concurrent users in total running simultaneously on the two-node RAC database. 3. Add two additional nodes into the system, then run the workload with 64 concurrent users on each node separately; that is, with a total of 256 concurrent users running simultaneously on the four-node RAC database. 4. Add four additional nodes into the system, then run the workload with 64 concurrent users on each node separately; that is, with a total of 512 concurrent users running simultaneously on the eight node RAC database. Performance statistics were captured using Oracle Automatic Workload Repository (AWR) RAC reports. We observed the “physical reads” value in the AWR report to assess read IOPS statistics. Query average response time was calculated from the “db file parallel read” and “db file sequential read” record in the “Top Timed Events” section of the AWR report, as shown in Figure 5. Figure 5. AWR RAC report snippet for read IO response time calculation High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 24 We used the following formula to calculate the I/O latency: For “db file sequential read” The total wait time is T1 which is 19,485.48 seconds as shown in Figure 5. The total number of waits is N1 which is 30,614,070 as shown in Figure 5. For “db file parallel read” The total wait time is T2 which is 9,803.47 seconds as shown in Figure 5. The total number of waits is N2 which is 8,610,615 as shown in Figure 5. The average read response time is: (T1+T2) / (N1+N2), which is (19,485.48 + 9,803.47) * 1,000/ (30,614,070 + 8,610,615) as shown in Figure 5, and 0.75ms is the average response time. Table 12 depicts how IOPS increase when RAC nodes scale up. Table 12. Scaling of nodes and resulting increases in IOPS Metrics 1 node 2 nodes 4 nodes 8 nodes IOPS 457,136 962,155 1,914,963 3,765,176 Average response time (ms) 0.74 0.69 0.68 0.75 Figure 6 shows that IOPS increase when the number of RAC nodes scales, while average read I/O response time remains under one millisecond. Figure 6. Query only IOPS scaling and average physical read I/O response time As Figure 6 shows, we achieved a total of 3,765,176 read IOPS and an average latency of 0.75 milliseconds with eight-node RAC database when running 64 concurrent sessions executing similar queries SQL statements on each of the RAC nodes. High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 25 The IOPS increased nearly linearly with each additional node that was added into the Oracle RAC database. For example, the total IOPS of four RAC nodes reached 1,914,963. After another four RAC nodes are added for a total of eight database nodes, the IOPS are almost doubled to 3,765,176. The read hit ratio for XtremCache is about 98 percent for each cache device during the test. Two percent of the I/Os are served from the storage array. The statistics can be monitored by running the following command line: vfcmt display -cache_dev <device> Update only test scenarios and methodology We gradually increased the number of RAC nodes and ran a total of 56 concurrent sessions on the database to generate stable and high read and write IOPS, with each session running similar SLOB UPDATE SQL statements We decreased buffer cache for each database instance to push consistent write I/O workload to the back-end storage. The write workload was driven by the UPDATE SQL statement. Generally, it incurs the following operations sequentially: 1. Read the data blocks that need to be updated into the buffer cache. 2. Update the rows in the data blocks. 3. Commit the updated rows and trigger LGWR flushing redo entries to online log files. During the SQL UPDATE only workload running, the background DBWR process flushes the dirty blocks out of the buffer cache into data files. Because we used a very small buffer cache, the data blocks were read into the buffer cache and written out of the buffer cache soon after the rows were updated. Thus the execution of each UPDATE operation caused physical reads, which were accelerated by the EMC XtremCache (when cache hit) or the back-end VMAX array (cache miss). When the updated data blocks were written out of the buffer cache by the DBWR process, as being a write through cache for EMC XtremCache , the data blocks were written to the back-end VMAX array. Once the application got the acknowledgement from the backend array, the application I/O request was complete, and the updated data was asynchronously written to XtremCache. The test process included the following steps: 1. Run the Update only workload with 56 concurrent users on one RAC Database with only one node using the scripts in SLOB. 2. Add one additional node into the RAC database, then run the workload with 56 concurrent users; that is, with 28 concurrent users running simultaneously on each node of this two-node RAC. 3. Add two additional nodes into the RAC database, then run the workload with 56 concurrent users; that is, with 14 concurrent users running simultaneously on each node of this four-node RAC. 4. Add four additional nodes into the RAC database, then run the workload with 56 concurrent users; that is, with seven concurrent users running simultaneously on each node of this eight-node RAC High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 26 Peak write test results Performance statistics are captured using AWR reports. We read the “physical writes” row in the AWR report for the peak write IOPS statistics. Because the write workload is generated by UPDATE statements as described previously, we also collected “physical reads” from the AWR report for the peak read IOPS statistics that were part of the write transaction. We calculated updated write average response time by dividing the “Total Wait Time(s)” by the “Waits” of the “db file parallel write” record in the “Top Timed Events” section of the AWR report, as shown in Figure 7. We also calculated LGWR latency by dividing the “Total Wait Time(s)” by the “Waits” of the “log file parallel write” record in the “Top Timed Events” section of the AWR report, as shown in Figure 7. For example, in following AWR snippet, the total wait time of the “db file parallel write” wait event is 287.00 seconds, which is 287,000 ms, and the number of waits is 319,941; thus, the write average response time can be calculated as 287,000 / 319,941= 0.90 ms. The LGWR latency is: 1,087.81*1,000/1,323,505 = 0.82. Figure 7. Update only write average response time measurement from the RAC AWR report Table 13 and Figure 8 depict the peak read/write IOPS increases as the RAC node scaling. Table 13. Scaling of users and resulting increases in peak disk array IOPS IOPS 1 node 2 nodes 4 nodes 8 nodes Write 26,950 50,110 95,932 153,276 Read 26,542 49,539 94,877 150,054 Aggregate 53,492 99,649 190,809 303,330 Write response time 0.30 0.40 0.30 0.90 Read response time(ms) 0.90 0.80 0.20 0.30 Redo size (MB/s) 20 37 71 115 LGWR latency (ms) 0.55 0.54 0.66 0.82 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 27 Figure 8 shows the peak IOPS, and update-only IOPS for RAC node scaling, while average response time remains under one millisecond. Figure 8. Peak writes – update-only IOPS scaling along with RAC nodes scaling During an UPDATE transaction, the backend VMX only needs to handle the writing I/O activities, as the reading I/O activities have been cached and accelerated by XtremCache. Because of this, the solution can scale to accommodate a very heavy transaction workload, as confirmed in testing. As shown in Table 13 and Figure 8, when running 56 concurrent sessions executing similar UPDATE SQL statements on eight-node RAC database, we achieved 303,330 aggregated IOPS including 153,276 write IOPS and 150,054 read IOPS, which were part of the write transaction. The average latency of write was 0.9 milliseconds, because we used very small buffer cache (only 128 MB, almost no data cached in the server) to generate a high physical write I/O workload. The IOPS increased nearly linearly when additional RAC nodes were added into the workload. For example, the aggregate IOPS were 53,492 when running write workload on one RAC node, and this increased to 99,649 when running workload on two RAC nodes. Redo size is also a key metric used to measure the transaction capability. As demonstrated through testing, the workload on one node generated 20 MB/second redo entries, and almost doubled to 37 MB/second with the workload running on two nodes. When we ran workload on four nodes the redo throughput is almost doubled again to 71 MB/second. The transaction capability is scaling along with the node scaling. High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 28 Conclusion Summary Implementing the EMC Proven High Performance Solution for Oracle RAC on VMAX with innovative and proven products like EMC’s VMAX gives customers choices within an open infrastructure, enabling them to easily integrate into existing data center operations and take advantage of new technologies such as XtremCache, which now also fully supports Oracle RAC. Customers can efficiently use people, processes, and technologies through virtualization, database, and applications consolidation. Customers can also independently scale capacity and processing capability without the limitations imposed by a single-purpose appliance As the infrastructure undergoes changes to applications, databases, and nondatabase software, this stack can align with the shifting technical demands imposed by the business needs. This white paper explains how to balance OLTP workloads while maintaining the protection and resiliency of the data. That adaptability to change and the ability to apply the technology where it is needed protects the capital investment, and can be fluid as the requirements change without sacrificing any of the other datacenter operations. Findings The core advantages of this solution are as follows: With its open architecture, supports Oracle OLTP workload built on superior functional integration throughout the solution Leverages the following EMC technology enablers in the solution: EMC VMAX 40K XtremSF flash cards XtremCache Provides full support for EMC Performance Boost, HA, and replication technologies Increases IOPS for OLTP workloads when servers are added The aggregate read IOPS increased from 457,136 to 1,914,963 when RAC nodes were scaled out from one to four, and it increased to 3,765,176 when eight RAC nodes ran the workload together. Average response time less than one millisecond (0.75ms) The aggregate IOPS for UPDATE transaction workload was 53,492 on one RAC node and caused 20 MB/s redo throughput, and it increased to 190,809 and caused 71 MB/s redo throughput when four RAC nodes were added. The IOPS increased to 303,330 and generated 115 MB/s redo throughput when running on eight RAC nodes. Average response time for DBWR background process less than one millisecond (0.90 ms) This solution provides a foundation that can be scaled in a flexible, predictable, and near-linear way, by adding additional server resources including CPUs and memory, HBA ports, and front-end ports, to provide higher IOPS and throughput based on the described configuration. High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 29 References EMC documentation The following documents provide additional and relevant information. Access to these documents depends on your login credentials. If you do not have access to a document, contact your EMC representative. The following reference materials are available on EMC Powerlink: EMC Infrastructure for High Performance Microsoft and Oracle Database Systems Introduction to EMC XtremCache EMC XtremCache Data Sheet In addition, XtremCache documentation is available at EMC Online Support: https://support.emc.com/products/25208_XtremCache-Cache/Documentation/ A video entitled Introduction to EMC XtremCache for Oracle Real Application Clusters is visible via this link: https://community.emc.com/videos/6740 Oracle documentation For additional information, see the following documents: Oracle Grid Infrastructure Installation Guide 11g Release 2(11.2) for Linux Oracle Database Installation Guide 11g Release 2 (11.2) for Linux. High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition 30