Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download EMC INFRASTRUCTURE FOR HIGH PERFORMANCE MICROSOFT AND ORACLE DATABASE SYSTEMS
Document related concepts
Entity–attribute–value model wikipedia , lookup
Concurrency control wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Tandem Computers wikipedia , lookup
Microsoft Access wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Team Foundation Server wikipedia , lookup
Relational model wikipedia , lookup
Database model wikipedia , lookup
Clusterpoint wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Transcript
White Paper EMC INFRASTRUCTURE FOR HIGH PERFORMANCE MICROSOFT AND ORACLE DATABASE SYSTEMS EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 Simplified storage management with FAST VP Accelerated performance with XtremCache EMC Solutions Abstract This white paper describes an automated storage tiering solution for multiple mission-critical applications virtualized with VMware vSphere on the EMC® Symmetrix® VMAX® 40K storage platform. With EMC XtremCache™ enabled on the host, read I/O is cached and offloaded from the VMAX storage virtual pool. December 2013 Copyright © 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H11035.3 EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 2 Table of contents Executive summary............................................................................................................................... 6 Business case .................................................................................................................................. 6 Solution overview ............................................................................................................................ 6 Key results ....................................................................................................................................... 6 Introduction.......................................................................................................................................... 7 Purpose ........................................................................................................................................... 7 Scope .............................................................................................................................................. 7 Audience ......................................................................................................................................... 7 Terminology ..................................................................................................................................... 7 Key technology components ................................................................................................................ 9 Overview .......................................................................................................................................... 9 EMC XtremSF and XtremCache ......................................................................................................... 9 Server-side flash caching for maximum speed ............................................................................. 9 Write-through caching to the array for total protection ................................................................. 9 Application agnostic.................................................................................................................. 10 Integration with vSphere ........................................................................................................... 10 Minimum impact on system resources ...................................................................................... 10 XtremCache active/passive clustering support .......................................................................... 10 EMC Symmetrix VMAX 40K ............................................................................................................. 10 EMC Virtual Provisioning ................................................................................................................ 10 EMC FAST VP .................................................................................................................................. 11 NEC Express5800/A1080a-E .......................................................................................................... 11 VMware vSphere 5 components ..................................................................................................... 11 VMware vSphere 5 .................................................................................................................... 11 VMware vCenter Server.............................................................................................................. 11 EMC PowerPath/VE.................................................................................................................... 11 Oracle Database 11g R2 ................................................................................................................. 11 Oracle Automatic Storage Management ..................................................................................... 11 Oracle Grid Infrastructure .......................................................................................................... 12 Microsoft SQL Server 2012 ............................................................................................................. 12 SQL Server Failover Clustering ................................................................................................... 12 Solution architecture and design........................................................................................................ 13 Overview ........................................................................................................................................ 13 Physical architecture ...................................................................................................................... 13 Hardware resources ....................................................................................................................... 14 Software resources ........................................................................................................................ 15 Storage connectivity ...................................................................................................................... 15 EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 3 Storage Virtual Provisioning design ................................................................................................ 16 FAST VP configuration .................................................................................................................... 17 Storage design considerations .................................................................................................. 17 Oracle database and workload profile ............................................................................................ 18 Oracle database schema................................................................................................................ 19 Oracle database services ............................................................................................................... 19 Oracle LUN configuration ............................................................................................................... 20 Microsoft SQL Server workload type ............................................................................................... 20 SQL Server 2012 DSS workload and profile .................................................................................... 20 SQL Server 2012 DSS LUN configuration ........................................................................................ 21 SQL Server 2012 OLTP workload and profile................................................................................... 21 SQL Server 2012 OLTP LUN configuration ....................................................................................... 22 SQL Server 2012 and Windows 2008 R2 settings for DSS and OLTP workloads .............................. 22 Operating system and SQL Server instance settings .................................................................. 22 Database settings ..................................................................................................................... 22 VMware vSphere configuration ...................................................................................................... 23 VMware virtual machine configuration ........................................................................................... 23 SQL Server 2012 clustering on VMware .......................................................................................... 24 EMC Virtual Storage Integrator ....................................................................................................... 25 XtremCache configuration with VMware ......................................................................................... 26 Performance testing processes .......................................................................................................... 29 Overview ........................................................................................................................................ 29 Validation ...................................................................................................................................... 29 Application workloads ................................................................................................................... 29 Test procedure ............................................................................................................................... 30 Test scenarios ................................................................................................................................ 30 Three-tier FAST VP without and with XtremCache ............................................................................... 31 Objective ....................................................................................................................................... 31 Test scenarios ................................................................................................................................ 31 Test result summary ....................................................................................................................... 31 Three-tier FAST VP performance results .......................................................................................... 33 Two-tier FAST VP without and with XtremCache ................................................................................. 34 Overview ........................................................................................................................................ 34 Test scenarios ................................................................................................................................ 34 Two-tier FAST VP OLTP performance results .................................................................................... 34 Two-tier XtremCache and FAST VP OLTP performance results .......................................................... 35 Two-tier FAST VP performance results breakdown .......................................................................... 37 XtremCache impact on FAST VP .......................................................................................................... 38 EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 4 Overview ........................................................................................................................................ 38 Three-tier FAST VP behavior with XtremCache................................................................................. 38 XtremSF and XtremCache with DSS workload ..................................................................................... 40 Overview ........................................................................................................................................ 40 Caching.......................................................................................................................................... 40 Local storage ................................................................................................................................. 41 XtremCache with SQL Server failover cluster instance ........................................................................ 42 Overview ........................................................................................................................................ 42 Microsoft failover clustering (active/passive) support .................................................................... 42 Validation ...................................................................................................................................... 43 Conclusion ......................................................................................................................................... 44 Summary ....................................................................................................................................... 44 Findings ......................................................................................................................................... 44 References.......................................................................................................................................... 45 White papers ................................................................................................................................. 45 Product documentation.................................................................................................................. 45 Other documentation ..................................................................................................................... 45 EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 5 Executive summary Business case Solution overview As enterprises move their databases and applications to the private cloud, their IT organizations must strive for more efficiency and improved quality of service, including: Extending the high performing flash technology from storage to host to support mixed workloads. Moving the workload from the storage array to host-based flash so that the array can serve more I/Os for other applications. Reducing capital expenditures and ongoing costs. Maintaining high performance levels and providing predictable performance to deliver the quality of service required in these environments. It is essential that infrastructure and tools simplify storage management processes and improve performance with a minimum of manual tasks. EMC® Symmetrix® VMAX® 40K, and associated management tools, have been developed to be the foundation of this infrastructure and to meet real business needs: Performance optimization—Optimizing and prioritizing business applications, allowing customers to dynamically allocate resources within a single array. Ease of management—Elimination of manually tiering applications when performance objectives change over time. Host-side storage acceleration—Accelerating application performance to extreme levels and placing hot data closest to server memory through consolidation with EMC XtremSF™ and EMC XtremCache™. NEC Express5800/A1080a-E is the base server platform of this solution. Representing the fifth generation of enterprise server architecture from NEC, this line of server maintains the NEC legacy for developing scalable enterprise servers that offer exceptional configuration flexibility, capacity, reliability and availability features. Pairing the NEC Express5800/A1080a-E with VMware vSphere 5, the platform creates an outstanding solution for enterprise virtualization needs. Key results Our testing shows that this solution, based on Symmetrix VMAX with Enginuity™ 5876, EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP™), XtremSF, and XtremCache, provides the following performance results: XtremSF and XtremCache improve OLTP performance by offloading much of the read I/O traffic from the storage array for other applications. XtremSF and XtremCache can solidly support an OLTP workload with SAN-based central storage, and can improve the application performance on two-tier FAST VP. Active/passive-hypervisor and physical-cluster support means that XtremSF and XtremCache can ensure data integrity while accelerating application performance in a highly available environment. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 6 Introduction Purpose This white paper describes the design, testing, and validation of an enterprise VMware infrastructure using the Symmetrix VMAX 40K storage platform with Enginuity 5876, XtremSF, and XtremCache as its foundation. This solution demonstrates how XtremSF and XtremCache complement FAST VP in providing performance, scalability, and application-specific functionality to the solution using representative application environments, including Microsoft SQL Server and Oracle. Specifically, this solution: Validates that XtremSF and XtremCache can be shared to and serve multiple applications in a VMware virtualized environment. Validates that XtremCache can be consolidated with FAST VP enabled on Symmetrix VMAX, and also that recently accessed data within workloads can be effectively offloaded from SAN-based central storage to XtremSF. Validates that XtremSF and XtremCache can improve performance to provide excellent response times for the read-intensive workloads. Demonstrates that data integrity can be guaranteed by adding XtremCache to a clustered SQL Server instance. Scope This white paper discusses multiple EMC products as well as those from other vendors. Some general configuration and operational procedures are outlined. However, for detailed product installation information, refer to the user documentation provided with those products. Audience This white paper is intended for EMC employees, partners, and customers, including IT planners, virtualization architects and administrators, and any other IT professionals involved in evaluating, acquiring, managing, operating, or designing infrastructure that leverages EMC technologies. Throughout this white paper, we assume that you have some familiarity with the concepts and operations related to enterprise storage and virtualization technologies and their use in information infrastructures. Terminology Table 1 lists several terms used in this paper. Table 1. Terminology Term Definition ASM Oracle Automatic Storage Management DSS Decision Support System (that is, data warehouse) FAST VP Fully Automated Storage Tiering for Virtual Pools FC Fibre Channel FCI Failover Cluster Instance EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 7 Term Definition HBA Host bus adapters HS Hot swap IOPS I/Os per second LUN Logical unit number NIC Network interface controller OLTP Online transaction processing pRDM physical Raw Device Mapping RAID Redundant array of independent disks SAN Storage area network SAS Serial Attached SCSI SATA Serial Advanced Technology Attachment SCSI Small Computer System Interface SLC Single-Level Cell TDev Thin device TPS Transactions per second VSI Virtual Storage Integrator EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 8 Key technology components Overview This solution used the following key hardware and software components: EMC XtremSF EMC XtremCache Note: Any mention of VFCache or XtremSW Cache in this white paper refers to XtremCache. EMC XtremSF and EMC XtremCache EMC Symmetrix VMAX 40K EMC Virtual Provisioning™ EMC FAST VP NEC Express5800/A1080a-E VMware vSphere Oracle Database 11g R2 Enterprise Edition Microsoft SQL Server 2012 Enterprise Edition EMC XtremSF is PCIe flash hardware deployed in the server to dramatically improve application performance by reducing latency and accelerating throughput. XtremSF can be used as a local storage device to accelerate read and write performance. It can also be used in conjunction with server flash caching software EMC XtremCache for accelerated read performance with data protection. XtremCache is intelligent caching software that leverages server-based flash technology to reduce latency and accelerate throughput for dramatic application performance improvement. A number of XtremCache features are highlighted below. For more information refer to the XtremCache Installation and Administration Guide. Server-side flash caching for maximum speed XtremCache software caches the most frequently referenced data on the flash in the server (XtremSF or other), which puts the data closer to the application. XtremCache automatically adapts to changing workloads by determining which data is most frequently referenced and promoting it to the server flash. This means that the “hottest” (that is, the most active) data automatically resides on the PCIe card in the server for faster access. XtremCache offloads the read traffic from the storage array, which allows it to allocate greater processing power to other applications. While one application is accelerated with XtremCache, the array’s performance for other applications is maintained or even slightly enhanced. Write-through caching to the array for total protection XtremCache accelerates reads and protects data by using a write-through cache to the storage to deliver persistent high availability, integrity, and disaster recovery. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 9 Application agnostic XtremCache is transparent to applications, therefore no rewriting, retesting, or recertification is required to deploy XtremCache in the environment. Integration with vSphere Integration of the VSI plug-in with VMware vSphere vCenter simplifies the management and monitoring of XtremCache. Minimum impact on system resources XtremCache does not require a significant amount of memory or CPU cycles, as a majority of flash management is done on XtremSF. Unlike other server flash solutions, there is no significant overhead from using XtremSF and XtremCache on server resources. XtremCache active/passive clustering support XtremCache clustering support ensures data integrity of an active/passive clustered application. The XtremCache-enabled cluster also accelerates application performance. EMC Symmetrix VMAX 40K EMC Symmetrix VMAX 40K with Enginuity 5876 provided the tiered storage configuration used in the test environment. Built on the strategy of powerful, trusted, smart storage, this solution incorporated a highly scalable Virtual Matrix Architecture™ that enables Symmetrix VMAX arrays to grow seamlessly and cost-effectively. Symmetrix VMAX supports flash drives, Fibre Channel (FC) drives, and SATA drives within a single array, as well as an extensive range of RAID types. The EMC Enginuity operating environment controls all components in the Symmetrix VMAX array. Enginuity 5876 for Symmetrix VMAX offers: EMC Virtual Provisioning More efficiency: New zero downtime technology for migrations (technology refreshes) and lower costs with automated tiering. More scalability: Up to two times more performance, with the ability to manage up to 10 times more capacity per storage administrator. More security: Built-in encryption, RSA-integrated key management, increased value for virtual server and mainframe environments, replication enhancements, and a new electronic licensing model. EMC Virtual Provisioning is the EMC implementation of thin provisioning. It is designed to simplify storage management, improve capacity utilization, and enhance performance. Virtual Provisioning provides for the separation of physical storage devices from the storage devices as perceived by host systems. This enables nondisruptive provisioning and more efficient storage use. This solution uses virtually provisioned storage for all deployed applications. For detailed information on Virtual Provisioning, refer to the EMC Solutions Enabler Symmetrix Array Controls CLI v7.4 Product Guide. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 10 EMC FAST VP EMC FAST VP is a feature of Enginuity 5875 and higher, that provides automatic storage tiering at the sub-LUN level. Virtual pools are Virtual Provisioning thin pools. FAST VP provides support for sub-LUN data movement in thin-provisioned environments. It combines the advantages of Virtual Provisioning with automatic storage tiering at the sub-LUN level to optimize performance and cost, while simplifying storage management and increasing storage efficiency. FAST VP data movement between tiers is based on performance measurement and user-defined policies, and is executed automatically and non-disruptively. NEC Express5800/A10 80a-E The NEC Express5800/A1080a has many key design features that are ideal for mixedworkload large-scale virtualization. With a maximum memory configuration of 2 TB, eight CPU sockets (160 threads) and 14 PCI Express 2.0 slots, consolidating entire database, application, and Web infrastructures onto a single NEC Express5800/A1080a is the preferred solution to growing IT needs. VMware vSphere For this solution, the Microsoft SQL Server and Oracle application servers are fully virtualized using VMware vSphere 5. This section describes the virtualization infrastructure, which uses the following components and options: VMware vSphere 5.0.1 VMware vCenter Server EMC PowerPath®/VE for VMware vSphere Version 5.7 VMware vSphere 5 VMware vSphere 5 is a complete, scalable, and powerful virtualization platform, with infrastructure services that transform IT hardware into a high-performance shared computing platform, and application services that help IT organizations deliver the highest levels of availability, security, and scalability. VMware vCenter Server VMware vCenter is the centralized management platform for vSphere environments, enabling control and visibility at every level of the virtual infrastructure. EMC PowerPath/VE EMC PowerPath/VE for VMware vSphere delivers PowerPath multipathing features to optimize VMware vSphere virtual environments. PowerPath/VE installs as a kernel module on the VMware ESXi host and works as a multipathing plug-in (MPP) that provides enhanced path management capabilities to ESXi hosts. Oracle Database 11g R2 Oracle Database 11g Release 2 Enterprise Edition delivers performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. It provides comprehensive features for transaction processing, business intelligence, and content management applications. Oracle Automatic Storage Management Oracle Automatic Storage Management (ASM) is a volume manager and a file system for Oracle database. ASM is Oracle's recommended storage management solution EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 11 that provides an alternative to conventional volume managers, file systems, and raw devices. ASM uses disk groups to store data files. An ASM disk group is a collection of disks that ASM manages as a unit. Within a disk group, ASM exposes a file system interface for Oracle database files. The content of files stored in a disk group is evenly distributed, or striped, to eliminate hot spots and to provide uniform performance across the disks. The performance is comparable to the performance of raw devices. Oracle Grid Infrastructure For this solution, Oracle Grid Infrastructure was installed with the Standalone Server option. The Oracle Grid Infrastructure for a standalone server provides system support for an Oracle database including volume management, file system, and automatic restart capabilities. If you plan to use Oracle Restart or Oracle ASM, then you must install Oracle Grid Infrastructure before you install and create the database. Oracle Grid Infrastructure for a standalone server combines Oracle Restart and Oracle ASM into a single set of binaries that is installed in the Oracle Grid Infrastructure home. Microsoft SQL Server 2012 Microsoft SQL Server 2012 is Microsoft’s database management and analysis system for e-commerce, line-of-business, and data warehousing solutions. By enabling a modern data platform with SQL Server 2012, users can get built-in, mission-critical capabilities and enable breakthrough insights across the organization with familiar analytics tools and enterprise-ready Big Data solutions. SQL Server Failover Clustering In SQL Server failover clustering, the operating system and SQL Server work together to provide availability in case of an application failure, hardware failure, or operating system error. Failover clustering provides hardware redundancy through a configuration in which vital, shared resources are automatically transferred from a failing computer to an equally configured server. SQL Server failover clustering in the active/passive mode is for one instance of a set of databases. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 12 Solution architecture and design Overview EMC solutions are validated architectures that are designed to reflect real-world deployments. This section describes the key components, resources, and overall architecture that make up this solution and its environment. Physical architecture Figure 1 depicts the physical architecture for this solution. Figure 1. Physical architecture diagram This solution is built on an EMC Symmetrix VMAX 40K array running Enginuity 5876. The array provides a mix of flash, FC, and SATA drives. FAST VP continually monitors and tunes performance by relocating data across storage tiers, based on access patterns and predefined FAST policies. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 13 We provisioned Microsoft SQL Server 2012 (two OLTP workloads and one Decision Support System (DSS)) and Oracle 11g R2 (OLTP). We also built Microsoft failover clustering on the virtualized environment to validate XtremCache and Microsoft Cluster Service (MSCS) consolidation. These applications ran on virtual machines in a VMware vSphere 5 cluster environment on EMC VMAX 40K storage. Load generation tools drove these applications simultaneously to validate the infrastructure and acceleration function from XtremCache. Failover was performed for SQL Server failover clustering to verify XtremCache and Windows Server Failover Clustering (WSFC) integration. The effects of applying the FAST policy are documented in Performance testing processes. Hardware resources Table 2 lists the hardware resources used in this solution environment. Table 2. Hardware resources Equipment Quantity Configuration EMC Symmetrix VMAX 40K Enginuity 5876 1 Three-engine, 128 GB cache per engine 33 flash 200 GB (including 1 HS) 132 × 600 GB 10k FC drives (including 6 HS) 70 × 2 TB 7.2k SATA drives (including 3 HS) 64 x 450 GB 15k FC drives (including 2 HS) NEC Express5800/A1080a-E 2 8-socket (10 C/2.40 GHZ/30 MB cache) 1 TB RAM 4 GbE IP ports 4 × 146 GB 2.5-in. 15k SAS disks 1 internal RAID controller 12 × 8 PCIe slots/2 × 16 PCIe slots 2 dual-port 8 Gb/s HBAs (4 FC) 1 quad-port GbE NIC SAN 1 8 Gb enterprise-class FC switch EMC XtremSF 2 700 GB SLC PCIe cards EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 14 Software resources Table 3 lists the software resources used in this solution environment. Table 3. Storage connectivity Software resources Software Version EMC Symmetrix VMAX Enginuity code 5876 EMC XtremCache 1.5 EMC PowerPath/VE for VMware 5.7 EMC Unisphere® for VMAX 1 EMC Solutions Enabler 7.4 VMware vSphere 5 (Enterprise Plus) 5.0.1 Oracle ASMlib 2.0.5 Oracle Database 11g R2 11.2.0.3 Microsoft Windows Server 2008 R2 SP1 Microsoft SQL Server 2012 RTM Microsoft TPC-E toolkit 1.12.0 Quest Benchmark Factory 5.8.1 Red Hat Enterprise Linux Server 5.7 Swingbench 2.3 The application workloads were logically separated using masking views within the VMAX 40K and HBAs. Figure 2 shows the front-end port use for each application. Oracle and SQL Server OLTP workloads were on separate ESXi hosts but shared the same eight front-end ports on the VMAX. The SQL Server OLTP and DSS workloads were on the same ESXi host but the front-end ports on the VMAX were separate. The purpose was to separate the application by the I/O size. The OLTP application is typically 8k–64k I/O in size and is measured in IOPS. For DSS with large I/O sizes ranging from 8k to 256k, disk performance is typically measured by throughput (in megabytes per second). EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 15 Figure 2. Storage Virtual Provisioning design Logical grouping of ports to applications EMC Virtual Provisioning greatly simplifies storage design. We created four thin pools on the array, based on the drive types available. Table 4 shows the thin pool definitions. Table 4. Thin pool configuration Thin pool name Drive size/ technology/ RPM RAID protection No. of drives Data device size No. of data devices Pool capacity FAST VP usage FLASH_3RAID5 200 GB flash RAID 5 3+1 32 68.8 GB 64 4.2 TB Oracle/SQL Server OLTP FC10K_RAID1 600 GB FC 10k RAID 1 126 66 GB 504 32 TB Oracle/SQL Server OLTP FC15K_RAID1 450 GB FC 15k RAID 1 64 49.2 GB 256 12.2 TB DSS SATA_6RAID6 2 TB SATA 7.2k RAID 6 6+2 72 240 GB 256 60 TB Oracle/SQL Server OLTP/DSS For this solution, the Oracle and Microsoft OLTP applications were bound to the FC10K_RAID1 pool. The Microsoft DSS application was bound to the FC 15K_RAID1 pool, which was backed by a smaller number of drives. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 16 FAST VP configuration VMAX administrators can set high-performance policies that use more flash drive capacity for critical applications, and cost-optimized policies that use more SATA drive capacity for less critical applications. The ideal FAST VP policy is to specify a 100-percent allocation for each of the tiers included. Such a policy provides the greatest amount of flexibility to an associated storage group, as it allows 100 percent of the storage group’s capacity to be promoted or demoted to any tier within the policy. However, data warehouse applications tend to issue scan-intensive operations that access large portions of the data at a time and also commonly perform bulk loading operations. These operations result in larger I/O sizes than OLTP workloads, and they require a storage subsystem that can provide the required throughput. This makes the throughput or megabytes per second (MB/s) the critical metric. Although flash disk storage can provide more than a 100 MB/s throughput, generally it is best suited to serving a small portion of the database’s hot data. Therefore, in this solution, we used a two-tier policy consisting of FC and SATA storage to provide a cost-efficient mix of storage to satisfy the needs of DSS workloads. Table 5 shows the FAST VP policies used for the application workloads in this solution for Oracle, SQL Server OLTP, and SQL Server DSS. Table 5. FAST VP policy for Oracle, SQL Server OLTP, and SQL Server DSS Storage group FAST policy name Flash FC SATA MSSQL_OLTP MSSQL_OLTP 100 percent 100 percent 100 percent MSSQL_DSS MSSQL_DSS 0 percent 100 percent 100 percent Oracle Oracle 100 percent 100 percent 100 percent Storage design considerations The design incorporates the following recommended practices for mission-critical database applications with FAST VP: Use separate storage volumes for data files and log files. Use separate file groups for large databases. For ASM, EMC recommends separate ASM disk groups for DATA, REDO, FRA, and TEMP. Bind all thin devices to the FC tier. Pin log devices and temp files to the FC tier. Figure 3 shows an overview of how each critical application is configured for FAST VP. In this implementation, only data LUNs are managed by FAST VP. LUNs for OS, temp, and LOG are pinned to the FC tier, excluding them from FAST VP decisions and movement. Note: The DSS SQL Server instance tempdb is moved out after moving the DSS tempdb to XtremCache. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 17 Figure 3. Oracle database and workload profile General view of FAST VP configuration for mission-critical applications The Swingbench Order Entry—PL/SQL Server (SOE) schema was used to deliver the OLTP workloads required by this solution. Swingbench consists of a load generator, a coordinator, and a cluster overview. The software enables a load to be generated and the transactions and response times to be charted. Table 6 details the Oracle database and workload profile for this solution. Table 6. Oracle database and workload profile Profile characteristic Details Database size 2 TB Database version Oracle Database 11g R2 single instance Storage type Oracle ASM Oracle system global area (SGA) 24 GB Workload type OLTP Workload profile Swingbench Order Entry (TPC-C-like) workload Database metric Transactions per second (TPS) Workload read/write ratio 80/20 Swingbench sessions 1,800 EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 18 Oracle database schema Two identical schemas were used to deliver the OLTP workloads required by this solution: Schemas SOE1 and SOE2. A Swingbench Order Entry workload was generated and run against schema SOE1. Since the I/O distribution across the database was completely even and random, this reduced sub-LUN skewing (since the entire database was highly active), therefore the second schema SOE2 remained idle to simulate a more normal environment where some objects are not highly accessed. Table 7 lists the tables and indexes for the SOE schema used in this solution (SOE1). Table 7. Oracle database services SOE schema Table name Index CUSTOMERS CUSTOMERS_PK (UNIQUE), CUST_ACCOUNT_MANAGER_IX, CUST_EMAIL_IX, CUST_LNAME_IX, CUST_UPPER_NAME_IX INVENTORIES INVENTORY_PK (UNIQUE), INV_PRODUCT_IX, INV_WAREHOUSE_IX ORDERS ORDER_PK (UNIQUE), ORD_CUSTOMER_IX, ORD_ORDER_DATE_IX, ORD_SALES_REP_IX, ORD_STATUS_IX ORDER_ITEMS ORDER_ITEMS_PK (UNIQUE), ITEM_ORDER_IX, ITEM_PRODUCT_IX PRODUCT_DESCRIPTIONS PRD_DESC_PK (UNIQUE), PROD_NAME_IX PRODUCT_INFORMATION PRODUCT_INFORMATION_PK (UNIQUE), PROD_SUPPLIER_IX WAREHOUSES WAREHOUSES_PK (UNIQUE) LOGON – Database services are entry points to an Oracle database that enable the management of workloads across the cluster. For this solution, each of the test schemas had a corresponding database service mapped to it, as shown in Table 8. This enabled monitoring of the Oracle I/O to be mapped to each individual service and hence schema. Table 8. Oracle database services Schema Service Instance Swingbench sessions SOE1 SOE1.oracledb.ie orafast 1800 SOE2 SOE2.oracledb.ie orafast 0 EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 19 Oracle LUN configuration Microsoft SQL Server workload type SQL Server 2012 DSS workload and profile Table 9 lists the LUN configuration for the Oracle application. Table 9. Oracle LUN configuration ASM disk group TDev hyper size No. of TDevs Capacity (GB) +DATA 64 GB 2 128 +SOE1 64 GB 15 960 +SOE2 64 GB 15 960 +FRA 64 GB 2 128 +TEMP 64 GB 1 64 +REDO 64 GB 1 64 Total (GB) 2304 In the test environment, the following two applications generated the different workload patterns running on the Microsoft SQL Server 2012 enterprise class platform: A TPC-E-like application, acting as a typical OLTP application A TPC-H-like application, acting as a typical DSS application The test workload for the SQL Server 2012 DSS application was based on a TPC-H-like workload. The TPC-H-like application models the analysis part of the business environment where trends are computed and refined data is produced to support the making of sound business decisions. In a TPC-H-like application, periodic refresh functions are performed against a DSS database whose content is queried on behalf of, or by, various decision makers. Table 10 details the SQL Server DSS database and workload profile for this solution. Table 10. SQL Server DSS database and application profile Profile characteristic Details Total SQL Server database capacity 2 TB Number of SQL Server instances 1 Concurrent users 2 Read/write ratio (user databases) 100:0 (typical) EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 20 SQL Server 2012 DSS LUN configuration Table 11 shows the LUN utilization for Microsoft SQL Server DSS LUNs, and how they were used. Table 11. Table LUN use for Microsoft SQL Server DSS Purpose LUN size No. of TDevs Capacity (GB) SQL Server DSS virtual machine data store 128 1 128 File group1–8 480 8 3840 Log 64 1 64 tempdb log 32 1 32 tempdb 64 6 384 Total (GB) 4,448 The test workload for the SQL Server 2012 OLTP application instances was based on SQL Server 2012 OLTP workload and a TPC-E-like workload. It was composed of a set of transactional operations that simulate an online stock trading floor, which is latency-sensitive and combines profile multiple query types. These include insert and updates per application transaction. The OLTP applications are primarily heavily indexed to support low latency retrieval of small numbers of rows from data sets that often have little historical data volume. These types of database operations induce significant disk head movement and generate classic random I/O scan patterns. Table 12 details the SQL Server OLTP database and workload profile for this solution. Table 12. SQL Server OLTP database and application profile Profile characteristic Details Total SQL Server database capacity 1 TB Number of SQL Server instances 2 Number of user databases for each virtual machine 1 (400 GB, 600 GB) Concurrent users Mixed workloads to simulate both a hot and a warm application environment. Read/write ratio 90:10 EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 21 SQL Server 2012 OLTP LUN configuration Table 13 and Table 14 show the LUN use for Microsoft SQL Server LUNs. Table 13. Microsoft SQL Server LUN use—SQL Server OLTP instance #1 Purpose LUN size No. of TDevs Capacity (GB) SQL Server OLTP virtual machine data store 128 1 128 SQL1 tpce root 64 1 64 SQL1 file group 1–8 128 8 1024 SQL1_Log 256 1 256 SQL1 tempdb log 64 1 64 SQL1 tempdb 64 4 256 Total (TB) 1,792 Table 14. Microsoft SQL Server LUN use—SQL Server OLTP instance #2 Purpose LUN size No. of TDevs Capacity (GB) SQL Server OLTP virtual machine data store 128 1 128 SQL2 tpce root 32 1 32 SQL2 file group 1–8 64 8 512 SQL2_Log 128 1 128 SQL2 tempdb log 64 1 64 SQL2 tempdb 64 4 256 Total (TB) 1,120 SQL Server 2012 Operating system and SQL Server instance settings and Windows 2008 For the SQL Server 2012 tests we used Windows 2008 R2 as the operating system. R2 settings for Our settings for DSS and OLTP workloads were: DSS and OLTP Large-page memory support was enabled for the SQL Server instance by workloads enabling the 834 startup parameter. The Lock pages in memory option was used for the SQL Server instances. All data and log LUNs were formatted using a 64 KB allocation unit size. Database settings For user databases, we used these settings: For DSS user database multiple data files—16 data files on 16 thin devices (TDevs). For OLTP user database multiple data files—eight data files on eight TDevs. Disabled the autogrow option for data files and manually grew all data files. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 22 For tempdb, we used these settings: Pre-allocated space and added a single data file per LUN. We ensured all files were the same size. Assigned temp log files to one of the LUNs dedicated to log files. Enabled autogrow—In general, the use of a large growth increment is appropriate for data warehouse workloads. A value equivalent to 10 percent of the initial file size is a reasonable starting point. We followed standard SQL Server best practices for database and tempdb sizing considerations. For more information, see Capacity Planning for tempdb in SQL Server Books Online. For the transaction log we used this configuration: VMware vSphere configuration Created a single transaction log file per database on one of the LUNs assigned to the transaction log space. Spread log files for different databases across available LUNs or use multiple log files for log growth, as required. Enabled the autogrow option for log files. VMware vCenter Server provided a scalable and extensible platform to centrally manage vSphere environments, providing control and visibility at every level of the virtual infrastructure. We connected two ESXi5 hosts to the VMAX 40K array. Host A ran the virtual machine for Oracle, and also ran the FCI standby SQL Server virtual machine. Host B ran the virtual machines for the SQL Server OLTP, the DSS, and ran the FCI active SQL Server virtual machine. VMware virtual machine configuration The clustered SQL Server data LUNs used physical Raw Device Mapping (pRDM) disks, besides these shared disks, all virtual machines in this configuration used virtual machine disks (VMDK) from VMware Virtual Machine File System (VMFS) data store volumes, including the OS and boot LUNs. Each VMFS data store hosts a single VMDK disk, ensuring high performance and zero contention. This practice also ensures you have the ability to restore at an application level with EMC TimeFinder Clone and Snap on the VMAX 40K array. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 23 Table 15 shows the virtual machines CPU and memory allocation for each application virtual machine. Table 15. SQL Server 2012 clustering on VMware Virtual machine CPU and memory allocation Application Virtual machine name CPU count Memory size (MB) Oracle ORACLEDB 32 54,272 Microsoft SQL Server DSS SQLTPCH01 32 131,072 Microsoft SQL Server OLTP SQLTPCE01 32 16,000 SQLTPCE02 32 16,000 Domain controller 4 4,096 In this solution, a clustered SQL Server instance was built across two ESXi hosts for the XtremCache function test. The cluster requires specific hardware and software. The ESXi hosts had the following configuration: One physical network adapter dedicated to the VMkernel. Shared storage must be on an FC SAN. In this solution, the two shared disks were from VMAX 40K. RDM in physical compatibility (pass-through) mode. Table 16. Microsoft SQL Server Failover Cluster Instance (FCI) LUN use Purpose Quantity of LUNs Capacity (GB) SQL Server FCI boot LUN data store 2 160 Microsoft Distributed Transaction Coordinator (MSDTC) data store 1 200 Microsoft SQL Server user database store 2 200 Total (TB) 0.92 EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 24 EMC Virtual Storage Integrator EMC Virtual Storage Integrator (VSI) provides enhanced visibility into Symmetrix VMAX 40K directly from the vCenter GUI. Figure 4 shows the data store and storage pool information, which provides information about virtual pool usage for the Oracle_SOE1_1 data store. Figure 4. Data store and storage pool information viewed from VSI VMAX 40K volumes hosted the VMFS data stores and pRDM disks for this solution. Figure 4 shows the ESXi server and the Symmetrix VMAX 40K storage mapping with details about VMFS data stores and the LUNs. The VSI Storage Viewer feature identifies details about VMFS data stores such as the VMAX storage volumes hosting the data store, the paths to the physical storage, pool usage information, and data store performance statistics. Figure 5 shows the LUN view from VSI. From here, administrators can identify the Symmetrix device ID for LUNs and data stores, if user-defined labels are set on VMAX LUNs. Administrators can export these listings to CSV files for manipulation with VMware PowerCLI scripts for the rapid provisioning of data stores to the ESXi hosts. Figure 5. EMC Virtual Storage Integrator LUN view EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 25 XtremCache configuration with VMware In a VMware environment, the XtremCache card resides on the ESXi server, while XtremCache software is installed on each of the virtual machines that are accelerated by XtremCache. The XtremCache VSI plug-in, which resides on the vCenter client, is used to manage XtremCache. XtremCache can accelerate performance for either RDM or VMFS LUNs in a VMware environment. The XtremCache installation is distributed over various vSphere system components. Figure 6 illustrates the location of these installed components. XtremCache software is installed on the guest machines and the XtremCache VSI plug-in is installed on a vSphere client. Figure 6. XtremCache in VMware environment EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 26 Multiple virtual machines on the same ESXi server can share the performance advantages of a single XtremSF card. As shown in Figure 7, the flash device (VMFS) is carved into virtual disks and presented to the virtual machines. Refer to XtremCache Installation Guide for VMware 1.5 for the detailed XtremCache VMware configuration. Figure 7. XtremCache in VMware environment XtremCache is integrated with VSI plug-ins to simplify XtremCache management and monitoring. Figure 8 shows how VSI is used to manage XtremCache in the VMware environment. Figure 8. VSI XtremCache management EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 27 We monitored and observed how many I/Os were offloaded by the XtremSF card, as shown in Figure 9. Figure 9. VSI XtremSF monitor EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 28 Performance testing processes Overview This section describes how we tested the applications in the solution environment. Each test is described in more detail in later sections. Notes Benchmark results are highly dependent on workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning or product evaluation decisions are contemplated. The environment was rigorously controlled; results obtained in other operating environments may vary significantly. EMC Corporation does not warrant that a user can or will achieve performances similar to these. Validation To validate the environment, we deployed all applications and populated them with test data. Each of the applications (Oracle, SQL Server OLTP, and SQL Server DSS) was deployed at the production location, and workloads were driven against each application running simultaneously on the VMAX 40K storage array. We used Unisphere’s Performance Analyzer module on VMAX to monitor and gather storage performance data in addition to application performance monitoring tools. Application workloads For each application, we used load generation tools to simulate realworld user interactions. The details are as follows: We used a Microsoft TPC-E toolkit on the client virtual machines to generate TPC-E-like loads simultaneously for SQL Server OLTP databases. This emulated warm and hot workloads. The SQL Server OLTP application I/O pattern is typically 8 KB read/write, with a read/write ratio of 90:10 percent, respectively. We used Quest Benchmark Factory to generate a TPC-H-like load for the SQL Server DSS database. The DSS application I/O pattern is typically 64 KB, with 100 percent read ratio on the data LUNs. We generated a Swingbench TPC-C-like order entry workload with 1,800 users and ran it against the Oracle database. The Oracle I/O pattern is 8 KB read and 8 KB write, with a read/write ratio of 80/20 percent, respectively. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 29 Test procedure The test procedure was as follows: 1. Baseline test The baseline performance metrics were measured when the application workloads (Oracle, SQL Server OLTP, and SQL Server DSS) were run together and stabilized within three hours. We measured each application’s performance to ensure it was within predefined KPIs and that all workloads co-existed without a negative impact on each other. 2. Enable XtremCache After the application workloads stabilized, we enabled XtremCache on the OLTP workload and measured the performance acceleration and workload offloading from three-tiered FAST VP storage to XtremSF. We enabled a card on each of the SQL Server OLTP and Oracle virtual machines, and configured as much space as possible for all three workloads, according to demand. Enabling XtremCache on a DSS workload was verified as cache and as a local disk store for tempdb. The minimum space requirement for the XtremCache is 25 GB. The two 700 GB XtremSF cards (651 GB usable space) were divided by capacity and allocated to the four virtual machines on the two ESXi servers. The detailed allocation is listed in Table 17. Table 17. Test scenarios XtremSF allocation on virtual machines XtremSF allocation per application/virtual machine ESXi 01 (allocation unit: GB) ESXi 02 (allocation unit: GB) Oracle OLTP 600 N/A SQL Server DSS N/A 200 SQL Server OLTP 01 N/A 200 SQL Server OLTP 02 N/A 200 Total 625 625 The performance tests contained the following scenarios: Three-tier FAST VP without and with XtremCache Two-tier FAST VP without and with XtremCache XtremCache impact on FAST VP XtremCache with DSS workload XtremCache with SQL Server failover cluster instance EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 30 Three-tier FAST VP without and with XtremCache Objective The objective of this test was to validate the solution build under normal operating conditions for a normal work day, with FAST VP storage tiering enabled. Tests were run without and with XtremCache enabled to evaluate the heavy workloads offloading from three-tiered FAST VP storage to XtremSF. We evaluated all aspects of this solution, including the VMware vSphere server and virtual machine performance, Oracle, SQL Server OLTP, and SQL Server DSS server and client experiences. Test scenarios Test result summary The XtremCache offloading test had two scenarios: Before enabling XtremCache: All OLTP workloads were on VMAX with three-tier FAST VP enabled. The storage can support the workload with an excellent application response time; however the workload was heavy on the array. After enabling XtremCache on the virtual machines: The read I/O can be offloaded to XtremCache. The array still has three-tier FAST VP enabled and can handle other I/O requests. The test result summary is as follows: Without XtremCache enabled the array received more than 40,000 IOPS from the host side. With XtremCache enabled this number fell to only 12,000 IOPS (approximately) from the host side. XtremCache significantly reduced the IOPS and back-end adapter utilization of the storage array in the three-tier FAST VP configuration, while there was no change in the OLTP TPS performance. The array was then freed up for other I/O requests. The SQL Server OLTP, Oracle Database, and LUN response times decreased after XtremCache was enabled. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 31 As shown in Figure 10, the IOPS received by the VMAX 40K array front-end adapter fell from approximately 40,000 to 12,000, while the back-end adapter busy percentage decreased from 67 percent to 33 percent. This means that approximately 28,000 IOPS were offloaded by XtremCache. Figure 10. Array workload before and after XtremCache was enabled The test procedure was carried out with the three application (Oracle, SQL Server OLTP, and SQL Server DSS) workloads running together. Figure 11 shows XtremCache reduced storage device IOPS for Oracle and SQL Server OLTP data LUNs respectively. Figure 11. Storage device IOPS before and after XtremCache is enabled EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 32 Three-tier FAST VP performance results Table 18 shows detailed results of running workloads on the source array for Oracle and SQL Server OLTP applications before and after enabling XtremCache on the threetier FAST VP. With flash, FC, and SATA tiers in the FAST VP pool, after enabling XtremCache: The storage IOPS reduced significantly from more than 40,000 to approximately 12,000, because the XtremCache offloaded the IOPS and the SAN-based storage could serve I/Os from other applications. As the result of the decrease in back-end storage utilization, the virtual machine and ESXi CPU utilization increased. There was no obvious increase in OLTP TPS because the three-tier FAST VP could serve the application with excellent performance. Response times decreased, because the XtremCache solution cached the most frequently referenced data on the server-based PCIe card, thereby putting the data closer to the application. Table 18. Detailed results before and after enabling XtremCache Components Performance Three tiers configured without XtremCache VMAX VMAX OLTP IOPS (total) 40,605 12,275 ESXi 01 Average CPU utilization 65 percent 77 percent ESXi 02 Average CPU utilization 42 percent 43 percent Swingbench TPS 8,535 8,537 Average Oracle database response time (ms) 4 3 vCPU utilization 84.4 percent 91.4 percent SQL01 latency (ms) (read/write/transfer) 7/8/7 3/5/3 SQL02 latency (ms) (read/write/transfer) 3/4/4 2/4/2 SQL01 vCPU utilization 30 percent 73 percent SQL02 vCPU utilization 66 percent 81 percent Transaction/sec 5,725 5,846 Oracle OLTP SQL Server Three tiers configured with XtremCache EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 33 Two-tier FAST VP without and with XtremCache Overview With only FC and SATA tiers in the FAST VP pool, central storage may be a performance bottleneck because the limited spindle numbers may not provide excellent response times for heavy workloads. However, with XtremCache enabled on the virtual machines, the performance bottleneck can be overcome and the application continues to experience excellent levels of storage latency. Test scenarios The XtremCache offloading test had two scenarios: Two-tier FAST VP OLTP performance results Before enabling XtremCache: All OLTP workloads were on VMAX with two-tier FAST VP enabled. The two-tier storage has limited spindles to provide excellent response times for read I/O intensive applications. After enabling XtremCache on the virtual machines: XtremCache can significantly improve application performance. After disabling the flash tier in FAST VP, most of the workload was served by the FC tier. The FC tier disk utilization was very high and the SAN-based central storage (FC and SATA) could not support enough IOPS or provide acceptable response times for the I/O-intensive OLTP applications. The test results were as follows: The average FC disk IOPS was 142, which in theory is the maximum value for 10K FC disks. The maximum FC disk utilization was almost at 100 percent. The total OLTP IOPS was approximately 10,000 on average; the storage could not serve more IOPS. For SQL Server OLTP the average disk response time was more than 20 ms, and for Oracle, the application response time was more than 35 ms. These times exceed vendor-recommended limits. CPU utilization for the ESXi and the virtual machine was low: ESXi CPU utilization was less than 20 percent. Virtual machine CPU utilization was less than 5 percent. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 34 Figure 12 shows the FC-tier disk heat map. The red color means the disk workload was very high (hot). Utilization reached 100 percent of the total IOPS capacity. Figure 12. Two-tier XtremCache and FAST VP OLTP performance results FC-tier disk heat map without XtremCache Performance improved after enabling the XtremCache on the Oracle and SQL Server OLTP virtual machines. The test results were as follows: The average FC disk IOPS was 43, which was approximately 30 percent of the maximum capacity for 10K FC disk IOPS. The FC disk maximum utilization was approximately 65 percent. For SQL Server OLTP, the average disk response time was no more than 3 ms, and for Oracle, the application response time was no more than 3 ms. ESXi and virtual machine CPU utilization increased greatly when compared with the results before XtremCache was enabled: On ESXi-1 CPU utilization increased from 16 percent to 21 percent, and on ESXi-2 increased from 2 percent to 40 percent. The SQL Server OLTP database CPU utilization increased from less than 2 percent to 60-70 percent; for the Oracle database it increased from 48 percent to 89 percent. XtremCache can provide excellent response times for a read I/O intensive OLTP workload, and reduce the storage workload from the system. As a result, the system was able to handle more OLTP TPS. ESXi and virtual machine CPU usage increased because of the increased SQL Server and Oracle utilization with transactional processing. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 35 Figure 13 shows the FC-tier disk heat map. The yellow color means the disk workload was normal compared with those in Figure 12. The previous heavy workload on this tier was removed by XtremCache. Figure 13. FC tiers disk heat map with XtremCache Figure 14 shows that high Oracle latency and high SQL Server data LUN latency on the two-tiered SAN-based storage can be effectively eliminated by adding XtremCache to the virtual machine hosting the applications. The average response times fell by approximately six to ten times for SQL Server OLTP and Oracle, respectively. Figure 14. Oracle and SQL Server latency before and after XtremCache EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 36 Two-tier FAST VP performance results breakdown Table 19 shows the detailed performance metrics for storage, ESXi, and virtual machines of the running workloads on the source array for Oracle and SQL Server OLTP applications before and after enabling XtremCache on the two-tier FAST VP. With FC and SATA tiers in the FAST VP pool, after enabling XtremCache: OLTP application TPS and response times significantly improved because XtremCache can offload read I/O processing from the storage array, while reducing disk latencies, thus enabling higher transactional throughput. It can address “hot-spots” in the data center and alleviate high utilization of a twotier FAST VP storage environment. CPU usage increased because of the increased SQL Server and Oracle utilization with transactional processing. With XtremCache enabled the system was able to handle more SQL Server and Oracle TPS. Table 19. Performance metrics for storage, ESXi, and virtual machines Components Performance Two tiers configured without XtremCache VMAX VMAX IOPS 23,514 13,798 ESXi 01 Average CPU utilization 16 percent 21 percent ESXi 02 Average CPU utilization 2 percent 40 percent Swingbench TPS 6,653 8,590 Average Oracle response (ms) 35 3 vCPU utilization 47.6 percent 89 percent SQL01 latency (ms) (read/write/transfer) 22/4/21 3/2/3 SQL02 latency (ms) (read/write/transfer) 21/4/21 2/3/2 SQL01 vCPU utilization 1.20 percent 69.84 percent SQL02 vCPU utilization 1.46 percent 63.04 percent Transaction/sec 2,073 6,054 Oracle OLTP SQL Server Two tiers configured with XtremCache EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 37 XtremCache impact on FAST VP Overview In this solution, we evaluated the impact of XtremCache on FAST VP to measure if XtremCache offloading caused an impact on the existing three-tiered FAST VP. FAST VP moves data between tiers. The ingress and egress track records the data moving in or out from each tier. If there are many ingress tracks and egress tracks per second, the back-end performance is impacted when the application is running on the corresponding tiers. Generally, FAST VP has a quality of service (QoS) setting to control the reallocation rate of the data movement. The minimum value of the setting is 10; this means 1 GB/sec is the maximum moving rate. If the ingress/egress of each tier is far less than this value, the impact to the back-end is minimal. In the solution, the ingress/egress track of each tier (FLASH, FC, and SATA) was monitored when we enabled the XtremCache on the virtual machines. The purpose was to evaluate if the XtremCache causes lots of data movement between the FAST VP controlled storage pools, and if the FLASH tier can be released automatically when we enable the XtremCache. Three-tier FAST VP behavior with XtremCache In the three-tier FAST VP implementation, after enabling XtremCache, the flash tier ingress/egress tracks per second were less than 200 tracks per second, as shown in Figure 15, Figure 16, and Table 20. These charts show that XtremCache started taking the load off the flash tier of the three-tier FAST VP to the FC and SATA tier, freeing up these resources for other I/O intensive applications. This movement was controlled by FAST VP. Figure 15. Track ingress of three tiers after enabling XtremCache on three-tiered FAST VP EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 38 Figure 16. Track egress of three tiers after enabling XtremCache on three-tiered FAST VP Table 20. Average track ingress/egress per second FC tier Flash tier SATA tier FAST ingress track per second 78.03 1.68 55.68 FAST egress track per second 49.55 78.30 18.95 The maximum ingress or egress data between tiers was approximately 200 x 64 KB = 13 MB/sec. The moving rate has a minimum impact to the back-end disk workload. Since FAST VP has a long demotion period (demoting the cold data from the flash or FC tier to the lower tier), the running workload with XtremCache does not actively demote the data. This means that although the previously frequently accessed data can be served by XtremCache and the workload is freed from the flash tier, the flash capacity can still be occupied. The flash tier in the FAST VP can be proactively freed to handle more IOPS. While XtremCache can accelerate the performance for the SQL Server or Oracle OLTP workloads, FAST VP with flash tier is still complementary to XtremCache. The central storage can be a cost-effective solution with XtremCache. When the read-intensive I/Os are served by XtremCache, FAST VP with flash tier enabled can handle other I/O requests. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 39 XtremSF and XtremCache with DSS workload Overview XtremCache has a split-card functionality in which a part of the XtremSF card can be used as a cache and the other part can be used as local storage. We tested XtremCache in split-card mode with a DSS workload as the cache and the SQL Server instance tempdb as the direct-attached storage (DAS). Caching We carved a 200 GB LUN from the 700 GB available capacity and tested this as a cache to accelerate the TPC-H-like database data LUNs, as shown in Figure 17. The Max IO parameter is the maximum cached I/O size for XtremCache, which means that users can adjust the maximum cached I/O size for different application I/O patterns. In this solution, it was set to 128 KB for DSS workload. This means that any I/O size less than 128 KB is cached; whereas all other I/O greater than 128 KB is bypassed by XtremCache. Figure 17. LUN creation Figure 18 shows the space allocation for DSS from the 700 GB available capacity. Figure 18. XtremCache allocation for DSS EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 40 We performed the baseline SQL Server DSS performance test with other application (Oracle and SQL Server OLTP) workloads running together with FAST VP enabled. We defined a performance baseline to represent the DSS environment after applying the FAST VP policies, which stabilized the workload as follows: The average bandwidth was 650 MB/s The peak bandwidth was more than 1.2 GB/s After enabling XtremCache, no actual query bandwidth increase was observed. The reason is that XtremCache is not large enough to cache the entire data/index of the 2 TB DSS database. Local storage We carved 200 GB from the 700 GB capacity for use as the tempdb database data and log store. One TPC-H-like query (Q2 in the 22 TPC-H-like queries) was picked up as the DSS query to measure the performance for tempdb, with five concurrent executions. SQL Server tempdb was heavily used for sorting while the DSS query was running. Table 21 shows the performance results before and after. Table 21. Performance comparison before and after using XtremSF Without local storage With local storage Bandwidth (MB/sec) 270 396 Average LUN latency(ms) 13 1 Average read latency 13 1 Average write latency 3 1 Maximal LUN latency 84 20 vCPU utilization 74.7 percent 89.4 percent As the tempdb store for DSS workloads, XtremSF can: Increase the actual query bandwidth from 270 MB/sec to 396 MB/sec Reduce the average tempdb data LUN latency from 13 ms to 1 ms Reduce the peak tempdb data LUN latency from 84 ms to 20 ms Because of the faster I/O for tempdb to store the TPC-H-like query intermediate results, the application was able to execute more queries and consequently the virtual machine CPU utilization increased accordingly. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 41 XtremCache with SQL Server failover cluster instance Overview This section introduces XtremCache support for a SQL Server FCI on an active and passive Windows clustered environment for OLTP applications. Microsoft failover clustering (active/passive) support SQL Server within a SQL Server failover cluster (with multiple server cluster nodes) can also be accelerated using XtremCache, as the data is first written to the shared storage device, in this case VMAX, and then synchronously to the server flash device, in this case XtremSF. In the case of a failover, the SQL Server virtual instance can be started on the standby node and stopped on the active node if accessible, and continues to write to the shared LUN, as normal operation. If the new node has XtremCache enabled, SQL Server I/O caching now begins on that new cluster node. The previous node’s (failover from) device no longer receives I/O, as the application has moved. When the SQL Server fails back to the original node, the application retrieves data from the cache device, but now this device can contain stale data. Configuring the supplied XtremCache clustering script ensures that stale data is never retrieved. The scripts use Cluster Management events that relate to an application service start/stop transition to trigger a mechanism that purges the application cache. Cluster support is currently provided for clusters configured to operate in active/standby mode, where XtremCache is installed and operating on the single, active node and on any combination of the standby nodes. Only one application in a cluster can use XtremCache. To use XtremCache for several applications, you must configure these applications as separate resources in a single application, so that they fail over between hosts as a single unit. The following steps and recommendations outline how to configure the XtremCache for Microsoft Cluster Service: 1. Create the XtremCache resource in the SQL Server instance that uses XtremCache using Add a resource > Generic Script. 2. Unzip the XtremCache_State_Control.vbs under the EMC XtremCache installation folder and save it into the same folder on the active and passive node of the clustered SQL Server. For example, put it under C:\Program Files\EMC\VFC\XtremCache_Cluster_Support_1.5\XtremCache_Cluster_Supp ort\Microsoft_Cluster_Service. 3. Bring the resource (Generic Script) online. 4. Set the dependence: For the XtremCache resource, set it depending on the accelerated shared source LUNs. For the SQL Server service, set it depending on the XtremCache resource. 5. On the passive node, run the script to release all source devices and enable their acquisition by an active node: vfcmt set -clustermode passive For more information, refer to EMC XtremCache Installation and Administration Guide. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 42 Validation The following steps are used to validate the A/P Microsoft Cluster Service clustering supported and accelerated by XtremCache. 1. On the primary SQL Server, create a test table and insert 40,000 rows. One column type is the varchar. 2. Set the cache and source LUN to accelerate the source LUN stored in the database and the test table. 3. Query all data in the SQL Server instance to get the ‘to-be-stale’ data into the XtremCache. 4. Fail over the SQL Server instance to the passive node and update each row to set the varchar column to a new string. 5. Fail back the SQL Server instance, query the table, and validate it with and without the XtremCache clustering script enabled. Table 22 shows results without and with the XtremCache clustering script enabled. Table 22. Results with and without XtremCache clustering script No XtremCache clustering script Enable XtremCache clustering script Dirty read Yes No User database status Suspect and need to manually restore Healthy Shared LUN status The file system structure on the disk is corrupt and unusable Healthy SQL Server service Offline because of the disk error Online EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 43 Conclusion Summary Findings This EMC solution has shown the implementation of multiple, business-critical applications in a VMware private cloud environment hosted by VMAX 40K storage and XtremCache installed on the ESXi server. Each application had different workload characteristics and placed varying demands on the underlying storage. XtremCache provides better performance for the applications that involve heavy read I/O: With three-tier FAST VP configuration, XtremCache significantly offloads array IOPS. Arrays were then freed up for other I/O requests. With two-tier FAST VP configuration, XtremCache can improve application performance with excellent response times. The key findings of the tests show that: XtremCache improves OLTP performance by offloading much of the read I/O traffic from the storage array. In this solution, 70 percent of IOPS is offloaded to XtremCache from the storage array. XtremCache solidly supports OLTP workloads. When the SAN-based central storage has limited spindles to support the read I/O intensive workload or removes the flash tier from FAST VP for other applications, the impact to FAST VP is minimal. In this solution, the average response time for the Oracle database decreased to 3 ms from 35 ms, which is almost a 12 times improvement. The SQL Server database TPS increased to 162 from 56, which is almost a three times improvement. XtremCache works well with a failover clustered SQL Server instance and ensures source LUN acceleration while guaranteeing data integrity. EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 44 References White papers For additional information, see the white papers listed below. XtremCache Installation and Administration Guide v1.5 XtremCache Installation Guide for VMware 1.5 XtremCache Troubleshooting Guide 1.5 XtremCache Troubleshooting Guide for VMware v1.5 XtremCache VMware VSI Plug-in Guide 1.5 Implementing Virtual Provisioning on EMC Symmetrix VMAX with Oracle Database 10g and 11g—Applied Technology EMC Mission Critical Infrastructure for Microsoft SQL Server 2012 Provisioning EMC Symmetrix VMAXe Storage for VMware vSphere Environments Maximize Operational Efficiency for Oracle RAC with EMC Symmetrix FAST VP (Automated Tiering) and VMware vSphere—An Architectural Overview EMC Symmetrix Virtual Provisioning—Applied Technology FAST VP Theory and Practices for Planning and Performance—Technical Notes Best Practices for Fast, Simple Capacity Allocation with EMC Symmetrix Virtual Provisioning—Technical Notes Implementing Fully Automated Storage Tiering for Virtual Pools (FAST VP) for EMC Symmetrix VMAX Series Arrays EMC Storage Optimization and High Availability for Microsoft SQL Server 2008 R2 Product documentation For additional information, see EMC Solutions Enabler Symmetrix Array Controls CLI Version 7.4 Product Guide. Other documentation For additional information, see the documents listed below. SQL Server Best Practices Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Linux Oracle Real Application Clusters Installation Guide 11g Release 2 (11.2) for Linux Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Oracle Database Storage Administrator's Guide 11g Release 2 (11.2) EMC Infrastructure for High Performance Microsoft and Oracle Database Systems EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, and VMware vSphere 5 45
