* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download g EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 11 OLTP
Survey
Document related concepts
Entity–attribute–value model wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Functional Database Model wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Ingres (database) wikipedia , lookup
Relational model wikipedia , lookup
Concurrency control wikipedia , lookup
Database model wikipedia , lookup
Transcript
IMPLEMENTATION GUIDE EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 11g OLTP EMC VSPEX Abstract This Implementation Guide describes the high-level steps and best practices required to implement the VSPEX Proven Infrastructure for Virtualized Oracle Databases on a VSPEX Private Cloud with VMware vSphere enabled by VNX and EMC backup and recovery systems. April 2013 Copyright © 2013 EMC Corporation. All rights reserved. Published in the USA. EMC believes the information in this publication is accurate of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. For the most up-to-date regulatory document for your product line, go to the technical documentation and advisories section on the EMC online support website. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Part Number H11445 2 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Contents Chapter 1 Introduction ........................................................................ 11 Purpose of this guide .................................................................................... 12 Business value.............................................................................................. 12 Scope............................................................................................................ 13 Audience....................................................................................................... 13 Terminology .................................................................................................. 14 Chapter 2 Before You Start .................................................................. 15 Overview ....................................................................................................... 16 Pre-deployment tasks .......................................................................................... 16 Documentation workflow .............................................................................. 17 Deployment prerequisites ............................................................................. 17 Planning and sizing Oracle Database 11gR2 ................................................. 19 Support resources......................................................................................... 19 Design guide ........................................................................................................ 19 Solution overviews............................................................................................... 19 VSPEX Proven Infrastructure ................................................................................. 19 VMware documentation ....................................................................................... 19 Chapter 3 Solution Overview ............................................................... 21 Overview ....................................................................................................... 22 Solution architecture............................................................................................ 22 Key components ........................................................................................... 24 Introduction ......................................................................................................... 24 EMC VSPEX .......................................................................................................... 24 Oracle Database 11g............................................................................................ 26 VMware vSphere 5.1 ............................................................................................ 27 VMware vSphere HA ............................................................................................. 27 VMware vSphere Distributed Resource Scheduler ................................................ 27 VMware vSphere PowerCLI ................................................................................... 28 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 3 Contents Red Hat Enterprise Linux 6.3 ................................................................................ 28 EMC VNX series .................................................................................................... 28 EMC Virtual Storage Integrator ............................................................................. 29 EMC Unisphere .................................................................................................... 29 EMC Avamar......................................................................................................... 29 EMC Data Domain ................................................................................................ 30 Chapter 4 Solution Implementation..................................................... 31 Overview ....................................................................................................... 32 Setting up the physical components ............................................................. 32 Implementing the network ............................................................................ 33 Implementing the storage array .................................................................... 35 Overview .............................................................................................................. 35 Setting up initial VNX configuration ...................................................................... 35 Provisioning storage for the NFS datastore .......................................................... 36 Configuring Unified Storage Management ............................................................ 36 Configuring FAST Cache (optional) ....................................................................... 39 Example storage layout ........................................................................................ 41 Implementing ESXi and vCenter .................................................................... 42 Preparing the Oracle Database 11gR2 virtual machine template................... 43 Overview .............................................................................................................. 43 Creating the Oracle Database 11gR2 virtual machine ........................................... 43 Completing the Oracle install pre-requisites......................................................... 44 Installing the Oracle Database 11gR2 binaries and patches ................................. 46 Preparing the Oracle Database 11gR2 virtual machine template .......................... 48 Customizing the VMware template ....................................................................... 48 Deploying the Oracle Database 11gR2 virtual machine ................................. 49 Overview .............................................................................................................. 49 Deploying new virtual machines........................................................................... 49 Cloning the Oracle binaries ona new virtual machine ........................................... 50 Determining the database prerequisites .............................................................. 51 Configuring the database file layout for NFS ......................................................... 52 Configuring the initialization parameters ............................................................. 53 Enabling the HugePages setting ........................................................................... 54 Creating the database using DBCA ....................................................................... 55 Performing the database post creation activities .................................................. 55 Implementing backup and recovery .............................................................. 58 Overview .............................................................................................................. 58 Minimum backup software and hardware requirements ....................................... 59 Deploying the backup software ............................................................................ 59 4 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Contents Installing the backup software ............................................................................. 60 Configuring the backup software .......................................................................... 61 Using the Data Domain system ............................................................................ 66 Chapter 5 Solution Verification ........................................................... 69 Verifying the baseline infrastructure ............................................................. 70 Overview .............................................................................................................. 70 Verifying the ESXi functionality............................................................................. 70 Verifying the solution component redundancy ..................................................... 70 Verifying the Oracle Database configuration......................................................... 71 Using the health monitoring tools ........................................................................ 71 Verifying the Oracle Database ....................................................................... 72 Overview .............................................................................................................. 72 Understanding the key metrics............................................................................. 72 Reviewing the Oracle dataset ............................................................................... 74 Backup and recovery verification methodology ............................................. 74 Overview .............................................................................................................. 74 Backing up an Oracle database ............................................................................ 75 Changing the database contents ......................................................................... 75 Restoring the Oracle database ............................................................................. 75 Chapter 6 Reference Documentation ................................................... 77 White papers................................................................................................. 78 Oracle .................................................................................................................. 78 EMC ..................................................................................................................... 78 VMware................................................................................................................ 78 Product documentation ................................................................................ 79 EMC ..................................................................................................................... 79 Oracle .................................................................................................................. 79 VMWare ............................................................................................................... 79 Other documentation .................................................................................... 80 Links ............................................................................................................. 80 Appendix A Configuration Worksheet ..................................................... 81 Configuration worksheet for Oracle Database 11gR2 ................................... 82 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 5 Contents 6 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Figures Figure 1. Figure 2. Figure 3. Figure 4. Figure 5. Figure 6. Figure 7. Figure 8. Figure 9. Figure 10. Figure 11. Figure 12. Figure 13. Figure 14. Figure 15. Figure 16. Figure 17. Figure 18. Figure 19. Figure 20. Figure 21. Figure 22. Figure 23. Figure 24. Figure 25. Figure 26. Figure 27. Figure 28. Figure 29. Figure 30. Figure 31. Figure 32. Figure 33. Figure 34. Architecture of the validated infrastructure ......................................... 23 VSPEX Proven Infrastructure ............................................................... 25 Storage network configuration............................................................ 34 Oracle Database 11gR2 storage elements .......................................... 35 Configuring EMC Unified Storage Management in vSphere Client........ 36 Configuring NFS Storage on VNX for Oracle VMs ................................. 37 Configuring VNX storage pools ........................................................... 37 Unisphere Create File System Wizard.................................................. 38 EMC Unisphere Create NFS Export Wizard ........................................... 39 Enabling FAST Cache in the Storage Pool Properties dialog ................. 40 Example storage layout for VNX .......................................................... 41 Oracle Universal Installer – Install choose database software only ..... 46 Enable Oracle Direct NFS client ........................................................... 47 Script root_clone.sh to be created and owned by root ........................ 47 Script cloneHome.sh to be created and owned by Oracle ................... 47 Details for template, vspx-Orac11203EE-2vcpu-8GB-tmplt ................. 48 Details custom specification, Oracle_2vCPU_8GB_11203_EE_spec ... 48 Example PowerCLI script to deploy a virtual machine from template ... 50 Sample PowerCLI script to clone and reattach the ORACLE_HOME ...... 51 Oracle Net Configuration Assistant utility ........................................... 51 Sample /etc/mtab NFS mount opitons ............................................... 52 Oranfstab example ............................................................................. 53 Oracle Database Configuration Assistant setup screen ....................... 55 Oracle startup/shutdown script /etc/init.d/dbora .............................. 57 Avamar server and Avamar Plug-in for Oracle...................................... 58 Link to 'Documents and Downloads' below for Avamar documentation and server/client agents and plug-ins. ............................................... 60 Avamar Administrator authentication and server (Grid) selection ....... 61 Avamar dataset for Oracle RMAN (LINUX) ............................................ 62 Specifying the Avamar Oracle dataset information ............................. 63 Avamar Schedule named 'BRS' in force for RMAN dataset named ‘Oracle’ in this case. ........................................................................... 64 Avamar Retention policy named 'BRSORACLE' in force for this dataset and earlier set schedule. .................................................................... 65 Avamar Policy tab highlighting Group Name of 'BRS' for the 'Oracle' dataset with prior Schedules and Retention Periods in force per the created policy. .................................................................................... 66 Avamar's Server panel ........................................................................ 67 Avamar’s 'Add Data Domain System' panel ........................................ 68 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 7 Figures Figure 35. Figure 36. Figure 37. Figure 38. Figure 39. 8 Swingbench interface load panel screenshot ..................................... 73 Swingbench order entry screen for workload simulation ..................... 73 Status of Completed Avamar backup job. ........................................... 75 Avamar Administrator Backup and Restore tab ................................... 76 Avamar's Activity panel showing a Completed job .............................. 76 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Tables Table 1. Table 2. Table 3. Table 4. Table 5. Table 6. Table 7. Table 8. Table 9. Table 10. Table 11. Table 12. Table 13. Table 14. Table 15. Table 16. Table 17. Table 18. Table 19. Table 20. Table 21. Table 22. Table 23. Table 24. Table 25. Table 26. Table 27. Table 28. Table 29. Table 30. Terminology........................................................................................ 14 Tasks for pre-deployment ................................................................... 16 Solution deployment process ............................................................. 17 Deployment prerequisites checklist .................................................... 18 Reference virtual machine characteristics ........................................... 25 Mapping the Oracle sizing model to the VSPEX reference virtual machine ........................................................ 26 Tasks for physical setup ..................................................................... 32 Tasks for switch and network configuration ........................................ 33 Tasks for storage configuration........................................................... 35 VNX storage layouts for Oracle Database ............................................ 36 Steps for installing and configuring ESXi and vCenter ......................... 42 Virtual machine template specification............................................... 43 Oracle install prerequisites ................................................................. 44 Red Hat Enterprise Linux 6 RPM packages for Oracle Database 11gR2 45 Installing the Oracle Database binaries and patches .......................... 46 Oracle Database 11gR2 reference virtual machine model ................... 49 Database file layout for NFS ................................................................ 52 Hardware requirements for Avamar Grid ............................................. 59 Install Avamar Admin and configure Oracle Server .............................. 59 Tasks for verifying the VSPEX Proven Infrastructure installation ......... 70 Tools for monitoring the solution ........................................................ 71 General operations ............................................................................. 72 Schema tables and indexes ................................................................ 74 Avamar backup terminology ............................................................... 74 Common server information ............................................................... 82 ESXi server information ...................................................................... 82 Array information................................................................................ 83 Network infrastructure information ..................................................... 83 VLAN information ............................................................................... 84 Service accounts ................................................................................ 84 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 9 Tables 10 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Chapter 1 Introduction This chapter presents the following topics: Purpose of this guide ................................................................................. 12 Business value .......................................................................................... 12 Scope ........................................................................................................ 13 Audience ................................................................................................... 13 Terminology............................................................................................... 14 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 11 Introduction Purpose of this guide EMC® VSPEX™ Proven Infrastructures are optimized for virtualizing business-critical applications. VSPEX provides modular solutions built with technologies that enable faster deployment, more simplicity, greater choice, higher efficiency, and lower risk. VSPEX provides partners with the ability to design and implement the virtual assets required to support a fully integrated virtualized solution for Oracle relational database management systems on a VSPEX private cloud infrastructure. The VSPEX for virtualized Oracle infrastructure provides customers with a modern system, capable of hosting a virtualized database solution that is scalable and delivers a constant performance level. This solution uses VMware vSphere™ to provide virtualization, backed by EMC VNX® providing storage, and EMC Avamar® and EMC Data Domain® for backup. The compute and network components, while vendordefinable, are designed to provide redundancy and sufficient power to handle the processing and data needs of the virtual machine environment. This Implementation Guide describes how to implement, using best practices, the VSPEX Proven Infrastructure for virtualized Oracle Databases on a VSPEX Private Cloud with VMware vSphere. Business value Database management systems software continues to be the dominant software used to manage data in nearly all commercial segments. This growth is expected to continue despite the increasing market share of other data management tools. This growth is expected to accelerate as customers continue to diversify their infrastructures and supporting technologies and drive towards more hardware and software appliances and configurations. This VSPEX Proven Infrastructure is focused on helping EMC partners understand the value that the VNX series, EMC backup and recovery systems, and Oracle bring to customers who often have growing, isolated IT environments running server-centric applications and who face increasing Oracle backup and recovery issues. This VSPEX solution is designed to meet the customer’s Oracle database challenges while enabling customers to grow in performance, scalability, reliability, and automation. By consolidating their database applications on EMC VNX, they can consolidate onto a single centralized storage platform that enables them to more effectively manage the exploding growth in data that is challenging businesses today. This solution has been sized and proven with EMC backup and recovery systems to: 12 • Reduce the customer’s backup storage requirements and costs • Meet backup windows • Enable fast disk-based recovery EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Introduction Scope This Implementation Guide describes the high-level steps required to deploy the VSPEX virtualization solution for Oracle RDBMS and Online Transaction Processing (OLTP) systems. This VSPEX application solution is based on the current VSPEX Proven Infrastructure: VMware Private Cloud and has been modified to support Oracle Database requirements. This solution uses the VSPEX standard or “reference virtual machine” building blocks of vCPU, memory, and storage/ I/O configurations, and describes the implementation based on three (small, medium, and large) different database workload thresholds. This Implementation Guide applies to and assumes that a VSPEX Proven Infrastructure with VMware vSphere with VNX and EMC backup and recovery systems already exists in the customer’s environment. Audience This guide is intended for internal EMC personnel and qualified EMC VSPEX partners. This guide assumes the VSPEX partners who intend to deploy this solution are: • Qualified by EMC to sell, install, and configure the EMC VNX family of storage systems • Qualified to sell, install, and configure the network and server products required for VSPEX Proven Infrastructures • Certified for selling VSPEX Proven Infrastructure Partners implementing this solution should also have the necessary technical training and background to install and configure: • VMware vSphere • Redhat Enterprises Linux 6.3 • Oracle Database 11gR2 or above • EMC next-generation backup, which includes EMC Avamar and EMC Data Domain This document provides external references where applicable. EMC recommends that readers are familiar with these documents. For details, refer to the section entitled Support resources. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 13 Introduction Terminology Table 1 lists the terminology used in the guide. Table 1. 14 Terminology Term Definition AWR Automatic Workload Repository DNFS Oracle Direct NFS client DNS Domain name system FAST VP Fully Automated Storage Tiering for Virtual Pools FQDN Fully Qualified Domain Name IOPS Input/output operations per second FRA Fast Recovery Area (Oracle) NFS Network File System NL-SAS Near-line serial-attached SCSI ODM Oracle Disk Manager OLTP Online transaction processing Oracle EE Oracle Enterprise Edition Oracle SE Oracle Standard Edition PowerCLI A Windows PowerShell interface to the VMware vSphere and vCloud APIs Reference virtual machine Represents a unit of measure for a single virtual machine to qualify the compute resources in a VSPEX Proven Infrastructure SGA System global area Statspack Oracle database monitoring and reporting utilities TPS Transactions per second VMDK VMware virtual machine disk VMFS VMware virtual machine file system EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Chapter 2 Before You Start This chapter presents the following topics: Overview ................................................................................................... 16 Documentation workflow ........................................................................... 17 Deployment prerequisites .......................................................................... 17 Planning and sizing Oracle Database 11gR2 ............................................... 19 Support resources ..................................................................................... 19 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 15 Before You Start Overview Before you layer the Oracle OLTP on a VSPEX private cloud, EMC recommends you complete the pre-deployment tasks listed in Table 2. Pre-deployment tasks Pre-deployment tasks include procedures that do not directly relate to environment installation and configuration, but whose results are needed at the time of installation. Examples of pre-deployment tasks include the collection of hostnames, IP addresses, VLAN IDs, license keys, installation media, and so on. Perform these tasks before visiting customer to decrease the time required on site. Table 2. 16 Tasks for pre-deployment Task Description Reference Gather documents Gather the related documents listed in Support resources. These are used throughout the text of this document to provide details on setup procedures and deployment best practices for the various components of the solution. Support resources Gather tools Gather the required and optional tools for the deployment. Use Table 4 to confirm that all equipment, software, and appropriate licenses are available before the deployment process. Deployment prerequisites Gather data Collect the customer-specific configuration data for networking, naming, and required accounts. Enter this information into the Customer Oracle Database 11gR2 configuration worksheet for reference during the deployment process. Appendix A EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Before You Start Documentation workflow EMC recommends that you follow the process flow in Table 3 to design and implement your VSPEX Proven Infrastructure for virtualized Oracle Database 11gR2 solution. Table 3. Solution deployment process Step Action 1 Use the VSPEX qualification worksheet to collect user requirements. The qualification worksheet is in Appendix A of the companion Design Guide. 2 Use the VSPEX Sizing Tool to determine the recommended VSPEX Proven Infrastructure for a virtualized Oracle Database 11gR2 solution. For more information about the VSPEX Sizing Tool, refer to the VSPEX Sizing Tool on the EMC Business Value Portal. Note You need to register the first time you access the tool. If the VSPEX Sizing Tool is not available, you can manually size the application using the sizing guidelines in Appendix B of the companion Design Guide. 3 To determine the final design for the VSPEX Proven Infrastructure for virtualized Oracle Database 11g, refer to the companion Design Guide. Note Ensure that all application requirements are considered, not just this application. 4 To select and order the right solution, refer to the appropriate VSPEX Proven Infrastructure document in Support resources. 5 To deploy and test your VSPEX solution, refer to this guide. Deployment prerequisites Table 4 itemizes the hardware and software requirements to configure the solution. For additional information, refer to the hardware and software tables in the relevant document in the Support resources section. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 17 Before You Start Table 4. Requirement Deployment prerequisites checklist Description Version Physical servers: sufficient physical server capacity to host the required number of virtual machines as recommended by the companion Design Guide and VSPEX Sizing Tool. Hardware Reference notes Reference Architecture: EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 500 Virtual Machines VMware vSphere 5.1 servers to the host virtual infrastructure servers. This requirement may be covered in the existing infrastructure. EMC Avamar 6.1 hardware and licenses Networking: switch port capacity and capabilities as required by the virtual server infrastructure. EMC Data Domain hardware and licenses EMC VNX: multiprotocol storage array with the required disk layout. EMC Backup: Avamar 6.1, Data Domain Software VNX OE for file Release 7.1.47-5 VNX OE for block Release 05.32.000.5 .008 EMC VSI for VMware vSphere: Unified Storage Management 5.4 EMC VSI for VMware vSphere: Storage Viewer 5.4 VMware ESXi™ installation media 5.1 VMware vCenter Server installation media 5.1 EMC Avamar 6.1 in server and client versions Red Hat Enterprise Linux 6.3 Red Hat Enterprise Linux 6.3 media www.redhat.com VMware vSphere license keys. This may be covered in the VSPEX Proven Infrastructure. 5.1 www.vmware.com Red Hat Enterprise Linux 6.3 www.redhat.com Oracle Database 11gR2 11.2.0.x www.oracle.com EMC online support Licenses 18 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Before You Start Planning and sizing Oracle Database 11gR2 To plan and size your Oracle Database 11gR2 deployment on the VSPEX infrastructure, you should follow the recommendations in the companion Design Guide. Size the storage accordingly to information obtained from the VSPEX for Virtualized Oracle 11g Qualification worksheet as detailed in Appendix A & B of the companion Design Guide. Support resources Before implementing the solution described in this document, EMC recommends that you read the following documents, available from the VSPEX space in the EMC Community Network or from EMC.com or the VSPEX Proven Infrastructure partner portal. . Design guide • EMC VSPEX for Virtualized Oracle Database 11g OLTP Design Guide Solution overviews • EMC VSPEX Server Virtualization for Midmarket Businesses • EMC VSPEX Server Virtualization for Small and Medium Businesses • Reference Architecture: EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 500 Virtual Machines • White Paper: EMC Avamar Backup for Oracle Environments • White Paper: EMC Avamar Backup with Data Domain VSPEX Proven Infrastructure VMware documentation VMware VSphere Product Documentation EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 19 Before You Start 20 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Chapter 3 Solution Overview This chapter presents the following topics: Overview ................................................................................................... 22 Key components ........................................................................................ 24 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 21 Solution Overview Overview This chapter provides an overview of the VSPEX Proven Infrastructure for Oracle Database 11gR2 and the key technologies used in this solution. This Implementation Guide documents a virtualized Oracle solution built upon on a VSPEX private cloud that supports up to 500 virtual machines. This VSPEX Proven Infrastructure for virtualized Oracle Database solution was validated using VNX and VMware virtualized Linux platforms to consolidate storage and server hardware. The solution described in this guide includes servers, storage, network, backup and recovery, and Oracle Database components that are focused on midrange environments. The solution enables customers to quickly and consistently deploy a medium virtualized Oracle Database in a VSPEX Proven Infrastructure. The VNX storage arrays are multiprotocol platforms that can support the iSCSI, NFS, and CIFS protocols depending on the customer’s specific needs. Solution architecture Figure 1 shows the architecture that characterizes the infrastructure validated for an Oracle Database 11gR2 overlay on a VSPEX infrastructure. To validate this solution, we1: • Deployed all Oracle Database 11gR2 servers as virtual machines on VMware vSphere 5.1 • Used the VSPEX sizing tool for Oracle Database 11gR2 to determine the number of, and the detailed compute resources for, each Oracle Database 11gR2 server role. Figure 1 displays the three sizing options (small, medium, and large) for this solution. Use the sizing tools provided with this solution to size your customer’s environment and choose the option that best suits your customer. • Determined the recommended storage layout for Oracle Database 11gR2 and the virtual infrastructure pool in the VNX series storage arrays (using the VSPEX sizing tool). 1 In this Implementation Guide, "we" refers to the EMC Solutions engineering team that validated the solution. 22 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Overview Figure 1. Architecture of the validated infrastructure EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 23 Solution Overview Key components Introduction EMC VSPEX This section provides an overview of the key technologies used in this solution, including: • EMC VSPEX • Oracle Database 11g • VMware vSphere 5.1 • VMware vSphere HA • vSphere Distributed Resource Scheduler • VMware vSphere PowerCLI • Red Hat Enterprise Linux 6.3 • Reference virtual machine • Oracle Small/Medium/Large Model • EMC VNX® series • Unisphere • EMC Virtual Storage Integrator (VSI) • EMC Avamar 6.1 • EMC DataDomain Deduplication Storage EMC has joined forces with the industry’s leading providers of IT infrastructure to create a complete virtualization solution that accelerates the deployment of private cloud technologies. Built with best-of-breed technologies, VSPEX enables faster deployment, more simplicity, greater choice, higher efficiency, and lower risk. VSPEX provides a virtual infrastructure for customers looking to gain simplicity that is characteristic of truly converged infrastructures while at the same time gaining more choice in individual components of the stack. VSPEX solutions are proven by EMC and packaged and sold exclusively by EMC channel partners. VSPEX provides channel partners more opportunity, faster sales cycle, and end-to-end enablement. By working even more closely together, EMC and its channel partners can now deliver an infrastructure that accelerates the journey to the cloud for even more customers. As shown in Figure 2, a VSPEX Proven Infrastructure is a modular, virtualized system validated by EMC and delivered by EMC partners. VSPEX includes a virtualization layer, server, network, and storage, designed by EMC to deliver reliable and predictable performance. VSPEX provides the flexibility to choose the best network, server, and virtualization technologies that fit a customer’s environment to create a complete virtualization solution. 24 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Overview Figure 2. VSPEX Proven Infrastructure Reference virtual machine To simplify the virtual infrastructure discussion, the VSPEX solution has defined a representative customer reference workload. For VSPEX solutions, the reference virtual machine is defined as a measure unit of a single virtual machine to qualify the compute resources in the VSPEX virtual infrastructure. Table 5 lists the characteristics of this virtual machine. Table 5. Reference virtual machine characteristics Characteristic Value Virtual processors per virtual machine 1 RAM per virtual machine 2 GB Available storage capacity per virtual machine 100 GB I/O operations per second (IOPS) per virtual machine 25 I/O pattern Random I/O read/write ratio 2:1 VSPEX for virtualized Oracle 11g sizing model Scale-up testing formed part of the validation process. Using a standard computesizing model for Oracle simplified and standardized the validation testing. During the testing, we defined three general computer model thresholds based on general concurrent user loads. This enabled us to identify the configuration required to run a TPC-C like OLTP workload with a 60:40 read/write ratio yielding acceptable response times for those thresholds. Table 6 shows how we mapped the Oracle sizing model to the VSPEX reference virtual machine. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 25 Solution Overview Table 6. Mapping the Oracle sizing model to the VSPEX reference virtual machine Oracle model Resources Equivalent reference virtual machine Small—virtual machine for up to 150 users Compute requirement 4 • 2 vCPU • 8 GB memory Storage Requirement (OS & Oracle binaries) • 100 GB • Medium—virtual machine for up to 250 users 25 IOPS Compute requirement • 4 vCPU • 8 16 GB of Memory Storage requirement (OS & Oracle binaries) Large—virtual machine for more than 250 Users • 100 GB • 25 IOPS Compute requirement • 8 vCPU • 16 32 GB of memory Storage requirement (OS & Oracle binaries) • 100 GB • 25 IOPS We calculated the database storage I/O thresholds and capacity separately from those required by the VSPEX reference virtual machine. Oracle Database 11g Oracle Database 11gR2 is available in a variety of editions tailored to meet the business and IT needs of an organization. In this solution we will be considering: • Oracle Database 11g Release 2 Standard Edition (SE) • Oracle Database 11g Release 2 Enterprise Edition (EE) Oracle SE is an affordable, full-featured data management solution that is ideal for all companies. It is available on single or clustered servers, has a maximum capacity of four processor sockets, and is licensed by the socket regardless of core count. The SE license includes Oracle Real Application Clusters (RAC) as a standard feature with no additional cost. 26 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Overview Oracle Database 11g R2 EE delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. It supports advanced features, either included or as extra-cost options, that are not available with Oracle Database 11gR2 SE. These include security features such as Virtual Private Database, and data warehousing options such as Partitioning and Advanced Analytics. Oracle Database 11g Release 2 EE extends the processor-licensing model for multi-core processors and is priced using the following formula: (number of processors) x (number of cores) x (Oracle processor core factor) For example, two 10-core Intel® Xeon® Processors E7-2870 (Oracle processor core factor of 0.5) are licensed as follows • Oracle Database 11g Release 2 SE: Two processor socket SE licenses • Oracle Database 11g Release 2 EE: 2 x 10 x 0.5 = 10 EE licenses The Oracle Database 11g R2 edition can affect the licensing cost and the size and number of VMware ESXi clusters you can configure. This affects how you place and manage the virtual machines. Refer to the document entitled Design Guide for EMC VSPEX for Virtualized Oracle Database 11g OLTP, in the DRS Host Affinity and Oracle processor licensing section, for more information on virtualization and Oracle processor licensing. VMware vSphere 5.1 VMware vSphere 5.1 abstracts applications and information from the complexity of underlying infrastructure through comprehensive virtualization of server, storage, and networking hardware. This transformation creates fully functional virtual machines that run isolated and encapsulated operating systems and applications just like physical computers. This virtualization of hardware resources enables efficiencies through consolidation of multiple applications on fewer physical servers. VMware vSphere HA VMware vSphere High Availability (HA) provides easy-to-use, cost-effective high availability for applications running in virtual machines. In the event of physical server failure, affected virtual machines are automatically restarted on other production servers with spare capacity. HA enables you to create a cluster from multiple ESXi servers, allowing you to protect virtual machines so that if of one of the hosts in the cluster fails, the impacted virtual machines are automatically restarted on other ESXi hosts within that same VMware vSphere cluster. VMware vSphere Distributed Resource Scheduler VMware vSphere Distributed Resource Scheduler (DRS) is an infrastructure service run by VMware vCenter Server (vCenter). DRS aggregates ESXi host resources into clusters and automatically distributes these resources to virtual machines by monitoring utilization and continuously optimizing virtual machine distribution across ESXi hosts. DRS can also use vMotion and storage vMotion (SvMotion) to ensure that virtual machines have access to rebalance resource capacity and make room for larger virtual machines. VMware recommends enabling DRS to achieve higher consolidation ratios. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 27 Solution Overview VMware vSphere PowerCLI VMware vSphere PowerCLI provides a Windows PowerShell interface for the users of vSphere 5.x and above and VMware Infrastructure 4.x and above. VMware vSphere PowerCLI is a powerful command-line tool that lets you automate all aspects of vSphere management, including network, storage, virtual machine, guest OS, and more. PowerCLI is distributed as a Windows PowerShell snapin, and includes 330 PowerShell cmdlets for managing and automating vSphere and vCloud, along with documentation and samples. Red Hat Enterprise Linux 6.3 Red Hat Enterprise Linux 6.3 includes enhancements and new capabilities that provide rich functionality, especially the developer tools, virtualization features, security, scalability, file systems, and storage. Red Hat Enterprise Linux is a versatile platform for x86 and x86-64 that can be deployed on physical systems, as a guest on the major hypervisors, or in the cloud. It supports all leading hardware architectures with compatibility across releases. EMC VNX series The EMC VNX family is optimized for virtual applications delivering industry-leading innovation and enterprise capabilities for file, block, and object storage in a scalable, easy-to-use solution. This next-generation storage platform combines powerful and flexible hardware with advanced efficiency, management, and protection software to meet the demanding needs of today’s enterprises. The VNX Series is powered by the Intel® Xeon® processor for intelligent storage that automatically and efficiently scales in performance, while ensuring data integrity and security. VNX customer benefits • Next-generation unified storage, optimized for virtualized applications • Capacity optimization features including compression, deduplication, thin provisioning, and application-centric copies • High availability, designed to deliver five 9s availability • Automated tiering with FAST™ VP (Fully Automated Storage Tiering for Virtual Pools) and FAST Cache that can be optimized for the highest system performance and lowest storage cost simultaneously • Simplified management with EMC Unisphere® for a single management interface for all NAS, SAN, and replication needs • Up to three times improvement in performance with the latest Intel Xeon multicore processor technology, optimized for Flash VNX Software suites available 28 • FAST Suite: Automatically optimizes for the highest system performance and the lowest storage cost simultaneously. • Local Protection Suite: Practices safe data protection and repurposing. • Remote Protection Suite: Protects data against localized failures, outages, and disasters. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Overview • Application Protection Suite: Automates application copies and proves compliance. • Security and Compliance Suite: Keeps data safe from changes, deletions, and malicious activity. Software packs available EMC Virtual Storage Integrator • Total Efficiency Pack: Includes all five software suites. • Total Protection Pack: Includes local, remote, and application protection suites. The EMC VSI for VMware vSphere Unified Storage Management feature can provision Network File System (NFS) datastores on NAS storage and Virtual Machine File System (VMFS) datastores and Raw Device Mapping (RDM) volumes on block storage, and perform array-based compression and array-based cloning of virtual machines in NFS datastores. The cloning functions include Full Clones (copies) and Fast Clones (snaps) of Virtual Machine Disk (VMDK) files. VMware administrators can use the feature to manage NAS and block storage in VMware environments by using the existing vSphere Client user interface. VMware administrators can use the tool to: • Provision new NFS, VMFS, and RDM storage • Extend existing NFS and VMFS storage • Compress virtual machines in NFS datastores • Clone virtual machines in NFS datastores • Provide Fast Clones support – Limited to the same file system EMC Unisphere EMC Unisphere is the central management platform for the VNX series, providing a single, combined view of file and block systems, with all features and functions available through a common interface. Unisphere is optimized for virtual applications and provides industry-leading VMware integration, automatically discovering virtual machines and ESX servers and providing end-to-end, virtual-to-physical mapping. Unisphere also simplifies configuration of FAST Cache and FAST VP on VNX platforms. EMC Avamar If you decide to implement a backup solution, EMC recommends Avamar. Avamar deduplication backup software and system performs variable-length deduplication at the client, so that backup data is reduced before it moves across networks (LAN or WAN). Avamar identifies duplicate data segments and sends only unique segments across the network to the backup appliance. This means shorter backup windows, less backup storage consumed, and maximum leverage of available bandwidth. Avamar provides: • Flexible deployment options. Avamar offers flexibility in solution deployments, depending on the specific use case and recovery requirements. Avamar is a turnkey backup and recovery solution that integrates with EMCcertified hardware for streamlined deployment. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 29 Solution Overview EMC Data Domain 30 • Scalability, high availability, and reliability. Avamar uses a scalable grid architecture, enabling linear performance and storage increases by simply adding storage nodes. • Manageability and support. You can securely access Avamar systems through existing network links and integrate them with management frameworks to use SNMP for remote access. EMC Data Domain deduplication storage system deduplicates data inline so that the data lands on disk already deduplicated, which requires less disk space than the original dataset. With Data Domain, you can retain backup and archive data on site longer to quickly and reliably restore data from disk. The Data Domain software suite includes the following options: • EMC Data Domain Replication • Virtual Tape Library (VTL) • Data Domain Boost • Retention Lock • Encryption • Extended Retention EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Chapter 4 Solution Implementation This chapter presents the following topics: Overview ................................................................................................... 32 Setting up the physical components........................................................... 32 Implementing the network ......................................................................... 33 Implementing the storage array .................................................................. 35 Implementing ESXi and vCenter.................................................................. 42 Preparing the Oracle Database 11gR2 virtual machine template .................. 43 Deploying the Oracle Database 11gR2 virtual machine ............................... 49 Implementing backup and recovery ............................................................ 58 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 31 Solution Implementation Overview This chapter discusses the steps required to install, configure, and implement the VSPEX Proven Infrastructure environment, which involves: • Preparing the components and their physical setup. • Implementing the network, including the configuration steps for physical and virtual switches in line with vendor guidelines and The vSphere Networking Guide. • Configuring the VNX storage array and optional FASTCache feature. • Building the VMware vSphere environment with ESXi server and vCenter implementation. • Preparing a virtual machine template for rapid deployment of Oracle Database 11gR2 pre-requisites. • Quickly and simply deploying an Oracle Database 11gR2 virtual machine using PowerCLI. • Implementing backup and recovery. Setting up the physical components This section includes information about the how to prepare the solution’s physical components. After you complete the steps listed in Table 7, the new hardware components will be racked, cabled, powered, and ready for network connection. Note If you already have a VSPEX Proven Infrastructure environment, you can skip this section. Table 7. Tasks for physical setup Task Description Reference Prepare network switches Install switches in the rack and connect them to power. Your vendor’s installation guide Prepare servers Install the servers in the rack and connect them to power. Your vendor’s installation guide Prepare VNX Install the VNX in the rack and connect it to power. EMC VNX System Installation Guide For details of the physical setup, refer to Reference Architecture: EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 500 Virtual Machines 32 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Implementing the network This section documents requirements for the network infrastructure needed to support this architecture. Table 8 provides a summary of the tasks for switch and network configuration and references for further information. Note If you already have a VSPEX Proven Infrastructure environment, you can skip this section. Table 8. Tasks for switch and network configuration Task Description Reference Network Configuration Configure storage network and host infrastructure networking as specified in the solution reference architecture. Refer to the appropriate document in Support resources. Complete network cabling Connect: N/A Switch interconnect ports VNX ports ESXi NiC/CNA ports Configure VLAN Configure private and public VLANs as required. Vendor’s switch configuration guide Configure VMare vSphere networking Create virtual switches for separation of application, management, and storage networks. vSphere Networking Guide For details of network implementation, refer to the appropriate document in Support resources. Figure 3 shows the configured vSphere network for this implementation. As highlighted, redundant 10GbE network adapters have been dedicated for storage network traffic. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 33 Solution Implementation Figure 3. 34 Storage network configuration EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Implementing the storage array Overview This section describes how to configure the VNX storage array. In this solution, the VNX provides Network File System (NFS) or Virtual Machine File System (VMFS) data storage for the VMware host. Table 9. Tasks for storage configuration Task Description Set up initial VNX configuration Configure the IP address information and other key parameters on the VNX. Provision storage for NFS mounts and NFS datastores Create NFS file systems that will be mounted as NFS datastores Reference • VNX System Installation Guide • VNX Series Configuration Worksheet Figure 4 depicts the high-level architecture of this solution. Figure 4. Setting up initial VNX configuration Oracle Database 11gR2 storage elements Ensure that network interfaces, IP address information, and other key parameters such as DNS and NTP are configured on the VNX before provisioning the storage. For more information on how to configure the VNX platform, refer to the appropriate document in Support resources. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 35 Solution Implementation Before you provision the storage for the NFS datastore, follow the recommendations Provisioning storage for the NFS and VSPEX Sizing Tool proposals introduced in the companion Design Guide. datastore Before you start, refer to the Reference Architecture titled EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 500 Virtual Machines to configure NFS file system and provision storage for virtual machine operating system on VNX. Table 10 shows an example of how to lay out the storage for the Oracle Database 11gR2 data in VNX. For more information about the storage layout recommendations and design, refer to the Design Guide. Table 10. VNX storage layouts for Oracle Database Storage pool name RAID type Disk type No. of disks Oracle Data pool RAID 5 (4+1) 15,000 rpm SAS disks 30 Oracle FRA pool RAID 6 (6+2) 7,200 rpm SAS disks 8 Oracle Redo pool RAID 5 (4+1) 15,000 rpm SAS disks 5 Present the NFS filesystem for the virtual machine datastore to all ESXi hosts, using either EMC Unisphere or the VSI plugin for VMware vSphere client. Figure 6 displays how to configure the NFS datastores that are used to store virtual servers on the VNX array using VSI. Configuring Unified Storage Management The EMC Unified Storage Management plugin for VMware vSphere provides an interface for creating and managing VMFS datastores for VMware ESXi host directly from the vSphere Client interface. Figure 5 shows the configuration screen accessed through Home >Solutions and applications >EMC from the vSphere Client, which is accessible after running the installer on your management host. Figure 5. Configuring EMC Unified Storage Management in vSphere Client Unified storage management enables the vSphere administrator to provision NFS datastores and mount them directly to the ESXi cluster nodes from vSphere Client, as shown in Figure 6. 36 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Figure 6. Configuring NFS Storage on VNX for Oracle VMs To configure NFS network settings, storage pools for file, file systems, and NFS exports on the VNX array in EMC Unisphere, perform the following steps: 1. In Unisphere, select the VNX array for this solution. 2. Select Settings Network Settings for File 3. Configure the IP address for network ports used for NFS. For detailed steps refer to EMC Procedure Generator for VNX. 4. Select Storage Storage Configuration Figure 7. 5. Configuring VNX storage pools Select the Storage Pools for File tab and create the additional storage pools in the VNX for Oracle Database files, as shown in Figure 7. Refer to Table 10 for detailed information. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 37 Solution Implementation 6. Figure 8. 7. 38 Create the required file systems. Figure 8 shows the Create File System Wizard from Unisphere. Unisphere Create File System Wizard Create the NFS exports for the database components that the Oracle nfs client will mount on the database virtual machines. Figure 9 shows the Unisphere Create NFS Export Wizard. Root and Access permissions are granted to the NIC. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Figure 9. EMC Unisphere Create NFS Export Wizard Complete the following steps in EMC Unisphere to configure the NFS file systems that are used to store virtual servers on the VNX array: Configuring FAST Cache (optional) 1. Create the storage pools in VNX. Refer to Table 10. 2. If you are using VNX, provision the LUNs in these pools. For detailed best practices on the number of the LUNs to be created in the storage pool for the NFS datastore, refer to the EMC VSPEX for Virtualized Oracle Database 11g OLTP Design Guide. 3. Create the file system from the storage pool to present to the ESXi servers as the NFS datastore in the VNX array. 4. Export the file systems using NFS and give root access to the ESXi servers. Enabling FAST Cache is a transparent operation to Oracle Database 11gR2 and no reconfiguration or downtime is necessary. To make the best use of either of the FAST technologies, EMC recommends that you first enable FAST Cache on the Oracle Database 11gR2 storage pool. For more details, refer to the EMC VSPEX for Virtualized Oracle Database 11g OLTP Design Guide. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 39 Solution Implementation To create and configure FAST Cache, use the following steps: 1. Refer to the Reference Architecture paper entitled EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 500 Virtual Machines for detailed steps about how to create FAST Cache. 2. In Unisphere, after creating the FAST Cache, click the Storage tab and select Storage Pool. Select Data Pool, and click Properties. 3. Select the Advanced tab in Storage Pool Properties and click Enabled to enable FAST Cache, as shown in Figure 10. 4. Click OK to complete the configuration. Note The FAST Cache feature on the VNX series array does not cause an instant performance improvement. The system must collect data about access patterns and promote frequently used information into the cache. This process can take a few hours during which the performance of the array steadily improves. Figure 10. 40 Enabling FAST Cache in the Storage Pool Properties dialog EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Example storage layout Figure 11 shows an example storage layout for VNX with FAST Cache enabled on the Oracle Database 11gR2 pool. Figure 11. Example storage layout for VNX EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 41 Solution Implementation Implementing ESXi and vCenter This section documents the requirements for the installation and configuration of the ESXi hosts, vCenter configuration, and infrastructure servers required to support the architecture. Table 11 describes the tasks that must be completed. Table 11. Steps for installing and configuring ESXi and vCenter Task Description Reference documents Install ESXi Install the ESXi 5.1 hypervisor on the physical servers that are being deployed for the solution. vSphere Installation and Setup Configure and deploy vCenter Server Install VMware vCenter Server 5.1 vCenter Server and Host Management Guide Installing vCenter Server 5.1 best practices Configure ESXi networking Configure ESXi networking including NIC trunking, VMkernel port, virtual machine port groups, and Jumbo Frames. vSphere Networking Connect VMware datastores Connect the VMware datastore to the ESXi hosts deployed for the solution. vSphere Storage Create a virtual data center Create a virtual data center. vCenter Server and Host Management Apply vSphere license keys Type the vSphere license keys in the vCenter licensing menu. vSphere Installation and Setup Add ESXi hosts Connect vCenter to ESXi hosts. vCenter Server and Host Management Configure vSphere clustering Create a vSphere cluster vSphere Resource Management Perform array ESXi host discovery Perform ESXi host discovery within the Unisphere console. Using EMC VNX Storage with VMware vSphere–TechBook Enable VMware High Availability (HA), DRS, and vMotion functionality Once you enable DRS, you may need to consider the use of VMware vSphere DRS Affinity and Anti-Affinity must rules for specific groups of virtual machines, for example, Oracle VMs may be restricted to a particular set of VMs due to licensing Automating High Availability (HA) Services with VMware HA –VMware Technical note Install EMC VNX USM Install the EMC VNX USM on the administration console. EMC VSI for VMware vSphere: Unified Storage Management— Product Guide Install the EMC VSI plug-in Install the EMC VSI plug-in on the administration console. EMC VSI for VMware vSphere: Unified Storage Management— Product Guide For details of ESXi and vCenter, refer to the Reference Architecture titled EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 500 Virtual Machines. 42 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Preparing the Oracle Database 11gR2 virtual machine template Overview VMware provides the ability to create a template that you can use as a master copy to quickly create and provision virtual machines. By using a template, you can install a guest OS and apply it to a virtual machine with application users and software configured and ready for use with minimal administrative intervention. This minimizes deployment time and avoids repetitive installation and configuration tasks for each virtual machine that is required. Customization specifications, maintained in vCenter, further simplify the rollout of virtual machines. A deployment wizard, automation tool, or script can use these templates to automatically pre-create or amend server settings (such as server name, time zone, and network configuration) prior to building the new virtual machine. This section documents the preparation of a VMware virtual machine template to enable rapid deployment of new Oracle Database 11gR2 environments. To configure the template with the requirements and prerequisites for the Oracle software install, we will complete the following steps: 1. Creating the Oracle Database 11gR2 virtual machine Create a virtual machine with the following resources: 2 vCPUs 8 GB of vRAM 100 GB virtual disk 2. Install the operating system and RPM packages 3. Configure the system and kernel parameters 4. Create the OS groups and users 5. Install and detach the Oracle binaries ready for clone 6. Convert the virtual machine to a VMware template Create a virtual machine on the ESXi server with the customer’s guest OS configuration. Table 12 lists an example specification used when building the virtual machine template. Table 12. Virtual machine template specification Part Description CPU 2 vCPUs Memory 8 GB Virtual Disk 100GB • 8GB swap partition • 92GB bootable root partition Operating system Redhat Enterprise Linux Server release 6.3 Kernel 2.6.32 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 43 Solution Implementation Part Description Network interfaces Eth0: public/management IP network Eth1: Storage Network Completing the Oracle install prerequisites OS users Username: oracle OS groups Group: oinstall, dba Oracle 11gR2 Binaries 11.2.0.4 (Oracle 11.2.0.3 installed and patched) RPM packages installed Refer to the Oracle Installation Guide for prerequisites System configuration See the Oracle Installation Guide for prerequisites. This section describes how to install and configure the operating system and Oracle prerequisites in the virtual machine prior to converting it to a template. Table 13. Oracle install prerequisites Task Description Reference Install the guest OS Install the Red Hat Enterprise Linux 6.3 64-bit operating system and configure the network interfaces. Red Hat Enterprise Linux 6 Installation Guide Install the required RPM packages Install or update required RPM packages as listed in Table 14. Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Set the kernel parameters Configure the kernel with Oracle-specific settings for semaphores, virtual and shared memory, network, and asynchronous I/O. Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Set the resource limits for the Oracle user Check and adjust the resource limits for the Oracle software installation users Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Create the required software directories Create the required directories for the Oracle software Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Configure Huge Pages HugePages reduce the number of memory pages and reduces virtual memory management overhead. Oracle Database 11g Release 2 on Red Hat Enterprise Linux 6 Deployment Recommendations Table 14 lists the Red Hat Enterprise Linux 6 RPM packages required to install Oracle database 11gR2. 44 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Table 14. Red Hat Enterprise Linux 6 RPM packages for Oracle Database 11gR2 binutils-2.20.51.0.2-5.11.el6 (x86_64) libstdc++-4.4.4-13.el6.i686 compat-libcap1-1.10-1 (x86_64) libstdc++-devel-4.4.4-13.el6 (x86_64) compat-libstdc++-33-3.2.3-69.el6 (x86_64) libstdc++-devel-4.4.4-13.el6.i686 compat-libstdc++-33-3.2.3-69.el6.i686 libaio-0.3.107-10.el6 (x86_64) gcc-4.4.4-13.el6 (x86_64) libaio-0.3.107-10.el6.i686 gcc-c++-4.4.4-13.el6 (x86_64) libaio-devel-0.3.107-10.el6 (x86_64) glibc-2.12-1.7.el6 (i686) libaio-devel-0.3.107-10.el6.i686 glibc-2.12-1.7.el6 (x86_64) make-3.81-19.el6 glibc-devel-2.12-1.7.el6 (x86_64)10 sysstat-9.0.4-11.el6 (x86_64) glibc-devel-2.12-1.7.el6.i686 On Oracle Linux 6, and Red Hat Enterprise Linux 6: ksh unixODBC-2.2.14-11.el6 (x86_64) or later libgcc-4.4.4-13.el6 (i686) unixODBC-2.2.14-11.el6.i686 or later libgcc-4.4.4-13.el6 (x86_64) unixODBC-devel-2.2.14-11.el6 (x86_64) or later libstdc++-4.4.4-13.el6 (x86_64) unixODBC-devel-2.2.14-11.el6.i686 or later EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 45 Solution Implementation Installing the Oracle Database 11gR2 binaries and patches As advised in the Design Guide, decide whether to install Oracle SE or Oracle EE Database 11gR2 software. Both editions use the same installation files. Table 15 includes the steps required to install the Oracle Database binaries and patches. Table 15. Installing the Oracle Database binaries and patches Task Description Reference Install the Oracle binaries Run the Oracle Universal installer and choose to “Install choose database software only” as shown in Figure 12. Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Install the latest PSU and CPU Install the latest database Patch Set Update and instructions available from the My Oracle Support website. A login ID and password as well as a support contract are required. See Critical Patch Updates, Security Alerts and Third Party Bulletin Enable DNFS Oracle Database is not shipped with Direct NFS enabled by default. To enable Direct NFS, refer to Error! Reference source not found.. Oracle Database Installation Guide 11g Release 2 (11.2) for Linux Detach the Oracle Home directory? To remove the Oracle Database 11gR2 software from the Oracle Universal Installer inventory run the script $ORACLE_HOME/oui/bin/detachhome.sh Oracle Universal Installer and OPatch User's Guide 11g Release 2 (11.2) for Windows and UNIX Chapter 2 Create the ORACLE_HOME cloning scripts To automate the cloning Oracle software process, create the scripts /root/root_clone.sh shown in Figure 24 And $OHOME/oui/bin/cloneHome.sh shown in 0 Oracle Universal Installer and OPatch User's Guide 11g Release 2 (11.2) for Windows and UNIX Chapter 6 As well as the official installation guide, Oracle Database Installation Guide 11g Release 2 (11.2) for Linux, Red Hat has produced a helpful whitepaper titled Oracle Database 11g Release 2 on Red Hat Enterprise Linux 6 Deployment Recommendations which offers guidance specific to the Red Hat operating system. Figure 12. 46 Oracle Universal Installer – Install choose database software only EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Oracle Database is not shipped with Direct NFS enabled by default. To enable Direct NFS, complete the following steps: Change the directory to $ORACLE_HOME/rdbms/lib. cp libodm11.so libodm11.so_stub ln -s libnfsodm11.so libodm11.so Enter the following command: make -f ins_rdbms.mk dnfs_on Figure 13. Enable Oracle Direct NFS client [root@vspex-ora02 ~]# cat /root/root_clone.sh #!/bin/bash OHOME=/u01/app/oracle/11.2.0.3 echo "Cloning Oracle Binaries as Oracle at $(date +%Y%m%d%H%M)" su - oracle -c "$OHOME/oui/bin/cloneHome.sh" echo "Clone Finished at $(date +%Y%m%d%H%M)" echo "Running root.sh at $(date +%Y%m%d%H%M)" $OHOME/root.sh echo "Install Finished at $(date +%Y%m%d%H%M)" Figure 14. Script root_clone.sh to be created and owned by root [root@vspex-ora02 ~]# cat $OHOME/oui/bin/cloneHome.sh #!/bin/sh OHOME=/u01/app/oracle/11.2.0.3 OHOMENAME=OraDb11g_home1 OBASE=/u01/app/oracle CUR_DIR=`pwd` cd $OHOME/oui/bin ${OHOME}/clone/bin/clone.pl ORACLE_HOME="${OHOME}" ORACLE_HOME_NAME="${OHOMENAME}" ORACLE_BASE="${OBASE}" -ignoreSysPrereqs cd $CUR_DIR Figure 15. Script cloneHome.sh to be created and owned by Oracle EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 47 Solution Implementation Preparing the Oracle Database 11gR2 virtual machine template Convert the virtual machine to a template called vspx-Orac11203EE-2vcpu-8GB-tmplt as shown in Figure 16. Figure 16. Customizing the VMware template Create a custom specification, Oracle_2vCPU_8GB_11203_EE_spec, using the vCenter client. This requests the new virtual machine name and sets place holder IP addresses for the network interfaces as shown in Figure 17. Figure 17. 48 Details for template, vspx-Orac11203EE-2vcpu-8GB-tmplt Details custom specification, Oracle_2vCPU_8GB_11203_EE_spec EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Deploying the Oracle Database 11gR2 virtual machine Overview Table 16 shows the mapping of the compute requirements for the Oracle Database 11gR2 virtual machines. Table 16. Oracle Database 11gR2 reference virtual machine model Virtual Machine Compute Requirement for Oracle Database 11gr2 Deploying new virtual machines Memory in GB Total reference virtual machines Small 8 4 Medium 16 8 Large 32 16 There are several methods available to deploy a virtual machine from a template using custom specifications: • Manually in vCenter by choosing Deploy Virtual Machine from this Template • Choosing Customize using an existing customization specification • By creating a workflow with an orchestration tool such as vCenter Orchestrator • By creating a custom PowerCLI script that can be edited or parameterized The example PowerCLI script in Figure 18: • Assigns variables for: template name, custom specification name, virtual machine name, ESXi host name, NFS or VMFS datastore name, virtual machine public IP address, virtual machine storage IP address, and the number of vCPUs, and vRAM size in MB. • Assigns the virtual machine public and storage IP addresses to the two vNICs included in the custom specification in Figure 17. • Creates the new virtual machine using the template and custom specification. • Changes the CPU and VRAM assignment for the virtual machine. • Starts the virtual machine. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 49 Solution Implementation # Connect to Vcenter Connect-VIServer -server 192.168.28.33 -user Administrator -password <Password> { #Assign Variables for template, specification, VM, ESXi Host , Datastore # IP addresses for the Public and Storage network interfaces # and the vCPU and vRAM assigned to the Virtual Machine $Template="vspx-Orac11203EE-2vcpu-8GB-tmplt" $Customization="Oracle_2vCPU_8GB_11203_EE_spec" $VMName="<Virtual Machine Name>" $VMHost=" <ESXi Host Name>" $Datastore="<Datastore Name>" $IPAdressPublic=”< Public IP Address>” $IPAdressStorage=”<Storage IP Address>” $NumVCPU=”4 ” $vRAMMB=”16384” # Assign IP addresses to the two vNICs included in the custom specification Get-OSCustomizationSpec $Customization | Get-OSCustomizationNicMapping | where { $_.Position -eq '1'} | Set-OSCustomizationNicMapping -IpMode UseStaticIP -IpAddress $IPAdressPublic -Subnetmask 255.255.255.0 -DefaultGateway 192.168.28.1 Get-OSCustomizationSpec $Customization | Get-OSCustomizationNicMapping | where { $_.Position -eq '2'} | Set-OSCustomizationNicMapping -IpMode UseStaticIP -IpAddress $IPAdressStorage -Subnetmask 255.255.255.0 -DefaultGateway 192.168.40.1 # Create the new VM using the Template and custom spec New-VM -Name $VMName -OSCustomizationSpec $Customization -Template $Template VMHost $VMHost -Datastore $Datastore # Change the CPU and memory assignment for the VM Set-VM $VMName -NumCPU $NumVCPU -MemoryMB $vRAMMB -Confirm:$false # Start up the VM Start-VM -VM $VMName -Confirm:$false } Figure 18. Example PowerCLI script to deploy a virtual machine from template Edit and run the example PowerCLI script, or edit it to read a parameter file and deploy multiple virtual machines. Cloning the Oracle binaries ona new virtual machine 50 Run the clone process and reattach the Oracle home directory to edit and run a second PowerCLI to complete the virtual machine deployment. This script, shown in Figure 19, calls the root_clone.sh from Figure 14 which does this. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation { $script="/root/root_clone.sh" $huser="root" $hpwd="<ROOT Password>" $VMName="<Virtual Machine Name>" Invoke-VMScript -ScriptText $script -VM $VMName -GuestUser $huser -GuestPassword $hpwd -ScriptType bash } Figure 19. Determining the database prerequisites Sample PowerCLI script to clone and reattach the ORACLE_HOME Use the VSPEX Oracle Qualification Worksheet and the Design Guide to size the virtualization infrastructure and the overall NFS filesystems in which to store the Oracle database. These documents also provide guidance on the Oracle initialization parameters required by the database. Table 16 shows the sizing of the virtual machine. Determine the size of the individual datafiles based on the schema design and tablespace placement of tables and indexes. Datafile placement is covered in the section Configuring the database file layout for NFS. For guidance on configuring the initialization parameters, see the Design Guide and the Configuring the initialization parameters section of this document. Before creating the database, you must create a listener process to receive the incoming client connection requests using the Oracle Net Configuration Assistant (NETCA) utility, as shown in Figure 20. Figure 20. Oracle Net Configuration Assistant utility EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 51 Solution Implementation Consider these items before creating an Oracle 11gR2 database: Configuring the database file layout for NFS • Select the required database character set to meet your localization and globalization needs. • Consider the time zones your database must support. • Ensure the SYSTEM and SYSAUX tablespace are created as Locally Managed Tablespaces, appropriately sized, and have AUTOEXTEND enabled. • Plan to use an undo tablespace to manage your undo data. The components of the Oracle Database: datafiles, online redo log files, temp files, control files, and FRA files reside on NFS file systems. These file systems are designed (in terms of the RAID level and number of disks used) to be appropriate for each type of file. Table 17 lists each file type and indicates where they reside. Table 17. Database file layout for NFS Content Location Datafiles /u02/app/oracle/oradata/${ORACLE_SID} Temp files /u04/app/oracle/oradata/${ORACLE_SID} Online redo logs, control files /u03/app/oracle/oradata/${ORACLE_SID} FRA files /u05/app/oracle/oradata/${ORACLE_SID} This solution uses the Oracle Direct NFS client, which offers improved performance over the standard NFS client supplied with the operating system. By default, Direct NFS serves the mount entries found in the file /etc/mtab, which lists all currently mounted filesystems along with their initialization options. This should match the system configuration file /etc/fstab where the mount options are defined. Figure 21 provides a sample /etc/mtab with NFS mount options: NFSServer1:/vspex_ora2_data /u02/app/oracle/oradata/VSPEX1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,t imeo=600,addr=192.168.40.11 0 0 NFSServer2:/vspex_ora2_redo /u03/app/oracle/oradata/VSPEX1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,t imeo=600,addr=192.168.40.10 0 0 NFSServer1:/vspex_ora2_temp /u04/app/oracle/oradata/VSPEX1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,t imeo=600,addr=192.168.40.11 0 0 NFSServer1:/vspex_ora2_fra /u05/app/oracle/oradata/VSPEX1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,t imeo=600,addr=192.168.40.11 0 0 Figure 21. Sample /etc/mtab NFS mount options Oracle recommends that you set the NFS buffer size parameters rsize and wsize to 32768 in the system configuration file /etc/fstab. However, you can use the file oranfstab to specify additional Oracle Database specific options for Direct NFS. 52 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation The oranfstab is located in one of the two following locations: • /etc/oranfstab where the options are global to all Oracle databases on the server • $ORACLE_HOME/dbs/oranfstab where the options are specific to a single database The Oracle Database requires that mount points be mounted by the kernel NFS system even when served through Direct NFS. To enable Direct NFS, you can optionally create an oranfstab file with the following attributes for each NFS server to be accessed using Direct NFS: • Server: The NFS server name. • Path: Up to four network paths to the NFS server, specified either by IP address, or by name, as displayed using the ifconfig command on the filer. • Local: Up to four local paths on the database host, specified by IP address or by name, as displayed using the ifconfig command run on the database host. • Export: The exported path from the NFS server. • Mount: The corresponding local mount point for the exported volume. • Dontroute: Specifies that outgoing messages should not be routed by the operating system, but instead sent using the IP address they are bound to. Please note that this attribute does not work on Linux with multiple paths in the same subnet. • mnt_timeout: Specifies (in seconds) how long Direct NFS client should wait for a successful mount before timing out. This parameter is optional and the default timeout is 10 minutes. Figure 22 provides an example of an oranfstab file with two NFS server entries: server: NFSServer1 local: vspex_ora2 path: NFSServer1 export: /vspex_ora2_data mount: /u02/app/oracle/oradata/vspex1 export: /vspex_ora2_temp mount: /u04/app/oracle/oradata/vspex1 export: /vspex_ora2_fra mount: /u05/app/oracle/oradata/vspex1 server: NFSServer2 local: vspex_ora2 path: NFSServer2 export: /vspex_ora2_redo mount: /u03/app/oracle/oradata/vspex1 Figure 22. Configuring the initialization parameters Oranfstab example To configure the initialization parameters, you can use one of the following methods. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 53 Solution Implementation Automatic Shared Memory Management To automate the management of shared memory structures, implement Automatic Shared Memory Management (ASMM). The following shared memory structures will be automatically managed: • DB_CACHE_SIZE • SHARED_POOL_SIZE • LARGE_POOL_SIZE • JAVA_POOL_SIZE • STREAMS_POOL_SIZE When implemented, ASMM enables the Oracle database to distribute memory between these components based on workload requirements. Hence, these components are considered autotuned parameters. The background process, Memory Manager (MMAN), coordinates the sizing of these components and moves memory to where it is needed most. To implement this feature, the set the following initialization parameters: • SGA_TARGET set to a nonzero value • STATISTICS_LEVEL=TYPICAL (or ALL) Automatic Memory Management Do not use Oracle Automatic Memory Management (AMM) since AMM is incompatible with HugePages. If you want to use HugePages make sure that both MEMORY_TARGET / MEMORY_MAX_TARGET initialization parameters are not set. For more information, see My Oracle Support Note ID 749851.1. I/O operations for file system files To enable direct I/O and async I/O for file system files, set FILESYSTEMIO_OPTIONS=SETALL With async I/O, processing continues while the I/O request is submitted and processed. This allows async I/O to bypass some of the performance bottlenecks associated with I/O operations. Direct I/O is a feature available in the filesystem that delivers data directly to the application without caching in the file system buffer cache. Direct NFS does not depend on the value of FILESYSTEMIO_OPTIONS. Direct NFS always issues async and direct I/O as it does not depend on OS support. However, so that there is a backup to the OS NFS client in case of mis-configuration, set filesystemio_options to SETALL if the OS supports it. HugePages was enabled in the Linux Kernel as part of the Template creation process Enabling the HugePages setting described in Table 13. The database initialization parameter, USE_LARGE_PAGES, was from 11.2.0.2 to manage HugePages for use by the database. From version 11.2.0.3, setting 54 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation USE_LARGE_PAGES to AUTO causes the oradism process to try to reconfigure the Linux kernel to increase the number of HugePages to match the database requirements. See My Oracle Support article USE_LARGE_PAGES To Enable HugePages In 11.2 [ID 1392497.1] for more details. Creating the database using DBCA The Database Configuration Assistant (DBCA) is the preferred way to create a database because it is a more automated approach and the database is ready to use when DBCA completes. Launch DBCA as a standalone tool to create an Oracle Database. To start DBCA on UNIX, Linux, or at a Windows command-line prompt, enter the following command: dbca Note: The dbca utility is typically located in the ORACLE_HOME/bin directory. Figure 23. Oracle Database Configuration Assistant setup screen Oracle DBCA runs in interactive mode. Using the screens in Oracle DBCA, you can either modify one of the preconfigured database types or customize the database. General Purpose/Transaction Processing is a preconfigured database template that you can create and customize during the installation. This involves amending the Performing the database post creation activities • Initialization parameters • Size and placement of the database files to use the NFS mount points After you create a database as described in Creating the database using DBCA, the instance is open and available for normal database use. However, the Oracle database will not start up or shut down automatically with the virtual machine. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 55 Solution Implementation With Oracle Database 11gR2, there are two methods to automate the Oracle Database startup and shutdown process: • Use an initialization script and the /etc/oratab file as defined by My Oracle Support Document How to Automate Startup/Shutdown of Oracle Database on Linux [ID 222813.1] • Install the additional 11.2 Grid Infrastructure software using the single instance option. This will use a modified Cluster Ready Services (CRS) component to automate the startup of the database, listener, and Oracle Automatic Storage Management (a grid Infrastructure component) if required. This solution uses the initialization script shown in Figure 24. Follow these steps to set up automated startup and shutdown of the database and listener. 1. Update the /etc/oratab file with an entry in the following format: <ORACLE_SID>:<ORACLE_HOME>:Y Where: ORACLE_SID is the name of the database ORACLE_HOME is the directory where the Oracle 11gR2 database binaries are installed Y states that the database can be started up and shutdown automatically For example: VPLEXM1: /u01/app/oracle/11.2.0.3:Y 2. 56 As root, create the service script /etc/init.d/dbora as shown in Figure 24. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation #!/bin/bash # # chkconfig run levels and script numbering # chkconfig: 35 99 10 # description: Starts and stops Oracle processes # # Set ORA_HOME to be equivalent to the $ORACLE_HOME # from which you wish to execute dbstart and dbshut; # Set ORA_OWNER to the user id of the owner of the # Oracle database in ORA_HOME. # ORA_HOME=/u01/app/oracle/11.2.0.3 ORA_OWNER=oracle case "$1" in 'start') # Start the TNS Listener su - $ORA_OWNER -c "$ORA_HOME/bin/lsnrctl start" # Start the Oracle databases: su - $ORA_OWNER -c "$ORA_HOME/bin/dbstart"& touch /var/lock/subsys/dbora ;; 'stop') # Stop the TNS Listener su - $ORA_OWNER -c "$ORA_HOME/bin/lsnrctl stop" # Stop the Oracle databases: su - $ORA_OWNER -c "$ORA_HOME/bin/dbshut" rm -f /var/lock/subsys/dbora ;; esac # End of script dbora Figure 24. 3. Oracle startup/shutdown script /etc/init.d/dbora Execute the script: chmod 755 /etc/init.d/dbora 4. Register the service to the Linux service mechanism for automatic startup at run level 3 and 5 by issuing the following command. /sbin/chkconfig --add dbora With the infrastructure build tasks complete, the database is now ready for use. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 57 Solution Implementation Implementing backup and recovery Overview The Avamar Plug-in for Oracle works with Oracle and Oracle Recovery Manager (RMAN) to backup an Oracle database, tablespace, or datafiles to an Avamar server or a Data Domain system. The Avamar Plug-in for Oracle serves as a backup module and the Avamar server and/or Data Domain system as a storage device. RMAN initiates the backup and recovery operations. The Avamar plug-in for Oracle interprets RMAN backup and recovery commands, and then routes the commands to the Avamar server. The Avamar server sends commands to the Avamar Plug-in for Oracle to perform the backup and restore activity. Figure 25 illustrates how the Avamar server and the Avamar Plug-in for Oracle interface with an Oracle database and RMAN. Figure 25. Note 58 Avamar server and Avamar Plug-in for Oracle You can also use EMC Data Domain as the backup target for Avamar as an option. The Avamar client and plug-ins are installed in the same way as when using Avamar as the backup target. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Minimum backup software and hardware requirements Table 18 below lists the hardware requirements for the Avamar administrator server on Windows or LINUX. Table 18. Hardware requirements for Avamar Grid Description Requirement vCPU 2 RAM 2 GB Storage 20 GB Network 1 Gbps OS platform Windows or Linux server Required account privileges You must have domain-level administrator access for account privileges. Typically, the Administrator account you used when installing and configuring the Oracle server is also a member of the Administrators group on each server. Deploying the backup software Table 19 below lists the steps to install the Avamar Administrator management console and to configure the Oracle server. For more detailed information, refer to Avamar 6.1 for Oracle User Guide. Table 19. Install Avamar Admin and configure Oracle Server Task Description Reference Install the Avamar Admin console Launch a web browser eg: http://avamarserver Download the Avamar administrator software Avamar 6.1 for Oracle User Guide Install the Avamar administrator on a Windows server Avamar 6.1 for Oracle User Guide Download the Avamar agent from the Avamar server Avamar 6.1 for Oracle User Guide Download the Avamar RMAN plug-in from the Avamar server Avamar 6.1 for Oracle User Guide Install the Avamar client on the Oracle server Avamar 6.1 for Oracle User Guide Install the Avamar RMAN plug-in on the Oracle server Avamar 6.1 for Oracle User Guide Install the Avamar agent on the Oracle server EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 59 Solution Implementation Installing the backup software Task Description Reference Register the Oracle server agent to the Avamar server Use the scripts provided by Avamar. Avamar 6.1 for Oracle User Guide Create a backup user on the Oracle server Use Oracle SQL commands to create a backup user with sysdba privileges Avamar 6.1 for Oracle User Guide To install the backup software, follow these steps: 1. Install MMC on a Windows server. Launch a web browser and point to the Avamar server name as shown in Figure 26. Figure 26. 2. Link to 'Documents and Downloads' below for Avamar documentation and server/client agents and plug-ins. Install the Avamar agent client package on the Oracle server using the following command: rpm –ivh AvamarClient-linux-rhel5-x86_64.VERSION.rpm 3. Register the Avamar client (Oracle server) to the Avamar server using the following command: /usr/local/avamar/bin/avregister 60 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation 4. Install the Avamar RMAN plug-in packages on the Oracle server using the following command: rpm –ivh AvamarRMAN-linux-rhel5-x86_64.VERSION.rpm 5. Create the backup user with sysdba privileges using the following commands: SQL> create user backupuser identified by manager; SQL> grant sysdba to backupuser; Configuring the backup software To configure the backup software, follow these steps: 1. Launch Avamar Administrator (shown in Figure 27) and configure Avamar to back up the entire Oracle Database 11gR2. Figure 27. Avamar Administrator authentication and server (Grid) selection EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 61 Solution Implementation 2. From Avamar Administrator, select Tools -> Manage Create datasets -> New. The New Dataset dialog box displays as shown in Figure 28. Figure 28. 62 Avamar dataset for Oracle RMAN (LINUX) EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation 3. Click the Options tab to display the Edit Dataset options as shown in Figure 29. Enter Oracle instance name, backup user name and password, the number of channels, and the database backup options. Figure 29. Specifying the Avamar Oracle dataset information EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 63 Solution Implementation 4. Create a schedule from Avamar Administrator by selecting Tools -> Manage schedules -> New. The Manage All Schedules screen displays as shown in Figure 30. Figure 30. 5. 64 Avamar Schedule named 'BRS' in force for RMAN dataset named ‘Oracle’ in this case. Create retention policy. From Avamar Administrator, select Tools -> Manage retention policies -> New. The Manage all Retention Policies dialog box displays as shown in Figure 31. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Figure 31. Avamar Retention policy named 'BRSORACLE' in force for this dataset and earlier set schedule. 6. Create backup group. From Avamar Administrator, select Tools -> action -> group -> new group. 7. On the New Group screen, pecify a group name, select the datasets, a schedule, the retention policy, and the client (which is Oracle server here). EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 65 Solution Implementation Figure 32. 8. Using the Data Domain system Avamar Policy tab highlighting Group Name of 'BRS' for the 'Oracle' dataset with prior Schedules and Retention Periods in force per the created policy. With the backup configuration complete, you can to perform backup jobs. Configuring the Data Domain system To integrate a Data Domain system to the Avamar configuration, prepare the Data Domain system by enabling DD Boost and the creating a DD Boost user account so that Avamar can access the Data Domain system for backups, restores, and replication, if applicable. Perform the following steps: 1. Enable DD Boost on the Data Domain system by issuing the following command: ddboost enable 2. Create a DD Boost account with admin privileges by issuing the following command: user add USER priv admin 3. Set the new account as the DD Boost user by issuing the following command: ddboost set user-name USER 4. Disable and enable DD Boost to allow the changes by issuing the following commands” ddboost disable ddboost enable 66 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Solution Implementation Adding a Data Domain system This section describes how to add the Data Domain system. To add Data Domain to the backup configuration, perform the following steps: 1. In Avamar Administrator, select Server to launch the Avamar Server window as shown in Figure 33. Figure 33. Avamar's Server panel 2. Select Actions -> Add Data Domain System. The Add Data Domain System screen displays as shown in Figure 34. 3. Specify the Data Domain system name, DDBoost user name, and password, maximum number of streams, indicate if you wish to use the system default replication storage, and then click Get Info. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 67 Solution Implementation Figure 34. 4. Avamar’s 'Add Data Domain System' panel Click the SNMP tab and verify that the SNMP configuration is correct and click OK. This completes the process to add a Data Domain system to an Avamar server. 68 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Chapter 5 Solution Verification This chapter presents the following topics: Verifying the baseline infrastructure ........................................................... 70 Verifying the Oracle Database .................................................................... 72 Backup and recovery verification methodology ........................................... 74 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 69 Solution Verification Verifying the baseline infrastructure Overview This section provides a list of items that you need to review after configuring the solution. The goal of this section is to verify the configuration and functionality of the solution, and to ensure that the configuration supports the core availability requirements. Table 20 describes the tasks that must be completed. Table 20. Tasks for verifying the VSPEX Proven Infrastructure installation Task Description Reference Verify ESXi functionality Verify the basic ESXi functionality of the solution with a post-installation checklist. Using EMC VNX storage with VMware vSphere http://www.vmware.com Verify redundancy of the solution components Verify the redundancy of the solution of the components: Vendor documentation Storage ESXi host Network switch Verify the Oracle Database configuration in the VSPEX Proven Infrastructure Verify the Oracle Database configuration. http://support.oracle.com Monitor the health of the VSPEX Proven Infrastructure for virtualized Oracle Database Use the tool to monitor the VSPEX Proven Infrastructure’s health View alert logs (Oracle Database 11gR2) View storage status from Unisphere management console View VM logs from vSphere 5.1 client Verifying the ESXi functionality EMC recommends that you verify the ESXi configurations before deploying the system to production on each ESXi server. For more detailed information, refer to appropriate documents in VSPEX Proven Infrastructure. Verifying the solution component redundancy To ensure that the various components of the solution maintain availability requirements, it is important to test specific scenarios related to maintenance or hardware failure. EMC recommends that you verify redundancy of the solution components including storage, ESXi hosts, and network switches. For detailed steps, refer to the appropriate documents in VSPEX Proven Infrastructure. 70 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Verifying the Oracle Database configuration Using the health monitoring tools To ensure that Oracle Database 11gR2 is running smoothly in the VSPEX infrastructure, use these steps to verify and monitor the Oracle Database configuration: 1. Check the installed Oracle Database contents and directory location. Make sure Oracle Database has been installed successfully. 2. Verify that the Oracle Database that you plan to use is accessible. Use the TNSPING utility to test connectivity from the client with the following command: tnsping net_service_name count. Health monitoring is a simplified measurement that reflects the reliability, stability, and performance of the entire VSPEX Proven Infrastructure for virtualized Oracle Database. Table 21 lists some tools for you to monitor and troubleshoot your VSPEX solution. Table 21. Tools for monitoring the solution Tools Description esxtop The esxtop tool provides a real-time view (updated every five seconds, by default) of ESXi Hypervisor performance metrics, such as CPU. It is important that you understand the performance for hypervisor in order measure or troubleshoot the Oracle Database performance-related issues. Unisphere Analyzer Unisphere Analyzer software is preloaded on all VNX storage systems with the FLARE/block bundles (or later). It is an easy to monitor, collect, and analyze the all aspects of storage related data from it. Oracle Enterprise Manager Database Control Oracle Enterprise Manager Database Control (Database Control) is the primary tool for managing your Oracle database. It is installed with the database. Using Database Control, you can perform administrative tasks such as creating schema objects (tables, views, indexes, and so on), managing user security, managing database memory and storage, backing up and recovering your database, and importing and exporting data. You can also view performance and status information about your database. Automatic Workload Repository (AWR) The Automatic Workload Repository (AWR) collects, processes, and maintains performance statistics for problem detection and selftuning purposes. This data is both in memory and stored in the database. The gathered data can be displayed in both reports and views. EMC recommends that you use the above tools to monitor the performance of the entire Oracle Database environment. For details about performance verification and test methodology, refer to the companion Design Guide. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 71 Solution Verification Verifying the Oracle Database Overview This section provides a verification example of an Oracle OLTP Database in the VSPEX Proven Infrastructure for virtualized Oracle Database solution. The purpose is to measure the performance of the Oracle Database to ensure that the VSPEX Proven Infrastructure meets the customer’s business requirements. Before you verify your own VSPEX Proven Infrastructure, EMC recommends that you read the Application validation methodology section in the companion Design Guide to understand the test methodology. Understanding the key metrics After you identify the business requirements, you need to know the key metrics for the Oracle Database testing when running the Oracle Database validation tests. The key metrics are identified in the Design Guide in the Application validation methodology section. My Oracle Support Note 1275596.1 provides a list of typical acceptable I/O response times. Loading the test configuration The Swingbench Order Entry – PL/SQL (SOE) schema delivers the OLTP workloads required by the solution. The Order Entry schema introduces heavy contention on a small number of tables and is designed to stress CPU and memory. To establish an optimal configuration, we tested different users accessing the order entry schema. Before running an Oracle Database 11gR2 test, it is important to understand the test scenarios according to the business requirements. You can easily create your own test scenarios using Swingbench. By editing the transaction list, you can define the different operations that end users perform in the Oracle Database. Table 22 lists some of the general operations we considered in the Oracle Database 11gR2 VSPEX solution. Table 22. General operations Workload Operation Weight Swingbench Order Entry Customer Registration 30 Browse Products 60 Order Products 50 Process Orders 30 Browse Orders 10 Swingbench allows you to select a variable “think time” (delay) between transactions. Figure 35 shows the load panel on the Swingbench interface, with the appropriate think time parameters set. 72 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Figure 35. Swingbench interface load panel screenshot We simulated the workload using Swingbench against the database as shown in Figure 36. Figure 36. Swingbench order entry screen for workload simulation Once the test is complete, you can use the Automatic Workload Repository (AWR) report to check the results and verify that you have achieved the key metrics in your test environment. Since the metrics can vary with different kinds of Oracle Database configuration and VSPEX offerings for VMware, refer to the VSPEX Sizing Tool for the detailed metrics. After enabling FAST Cache on the existing database, we observed a reduction in I/O access latency (in milliseconds). If the test results are not ideal, it is easy to identify the bottleneck from the AWR report. For detailed information about the AWR report, refer to the My Oracle Support Note ID 1359094.1. After identifying potential bottlenecks, refer to the Implementing the storage section to make sure that you have the correct configuration of VNX, switches, hypervisor, or load balancer for the VSPEX for virtualized Oracle solution. System bottlenecks represent a point of contention where there are insufficient resources to service user transaction requests. Consider adding more resources into the solution to meet the requirement. Often, the reason for the bottleneck is inefficient custom code or thirdparty solutions due to customization. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 73 Solution Verification Reviewing the Oracle dataset In this VSPEX validation test, we used the Swingbench Order Entry - PL/SQL (SOE) schema to deliver the OLTP workloads. This schema models a traditional OLTP database. Tables and indexes reside in separate tablespaces. The SOE schema used in this solution contains the tables and indexes shown in Table 23. Table 23. Schema tables and indexes Table name Index CUSTOMERS CUSTOMERS_PK (UNIQUE), CUST_ACCOUNT_MANAGER_IX, CUST_EMAIL_IX, CUST_LNAME_IX, CUST_UPPER_NAME_IX INVENTORIES INVENTORY_PK (UNIQUE), INV_PRODUCT_IX, INV_WAREHOUSE_IX ORDERS ORDER_PK (UNIQUE), ORD_CUSTOMER_IX, ORD_ORDER_DATE_IX, ORD_SALES_REP_IX, ORD_STATUS_IX ORDER_ITEMS ORDER_ITEMS_PK (UNIQUE), ITEM_ORDER_IX, ITEM_PRODUCT_IX PRODUCT_DESCRIPTIONS PRD_DESC_PK (UNIQUE), PROD_NAME_IX PRODUCT_INFORMATION PRODUCT_INFORMATION_PK (UNIQUE), PROD_SUPPLIER_IX WAREHOUSES WAREHOUSES_PK (UNIQUE) LOGON – Backup and recovery verification methodology Overview This section provides a verification example of an Oracle OLTP Database in the VSPEX Proven Infrastructure for virtualized Oracle Database solution. The purpose is to perform backup and recovery on an Oracle Database using Avamar backup software to ensure the database is fully protected and meets your business requirements. Before testing backup and recovery of your Oracle 11gR2 database, you need to understand Avamar backup terminology: Table 24. 74 Avamar backup terminology Term Description Datasets Avamar datasets are lists of directories and files to back up from a client. Assigning a dataset to a client or group enables you to save backup selections. Each dataset defines a source data list, an exclusion list, an inclusion list, and plug-in options. Schedules Schedules are reusable objects that control system backup activity at one of the supported intervals. The intervals include daily, weekly, monthly, and on-demand. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Term Description Retention policies Backup retention policies enable you to specify how long to retain a backup in the system. The typical retention settings include a retention period, a specific end date, or no end date. Backup groups Avamar uses groups to implement various policies that automate backups and enforce consistent rules and system behavior across an entire grid or most likely, a group. Group members are client machines that have been added to a particular group to perform the same policy-based scheduled backups. Backing up an Oracle database To run a backup job, from the Avamar Administrator interface, click backup to start the backup process. While the backup job is running, use the Activity Monitor screen to monitor the backup process. After a backup job completes, the job status indicates Completed. On the Activity Monitor screen, select a completed backup job, review the number of datasets that were backed up, and then check the deduplication ratio of the backup dataset. Figure 37 shows the status of the backup job as Completed on the Avamar Administrator Activity Monitor screen. Figure 37. Status of Completed Avamar backup job. Changing the database contents To change the database contents, rerun the Swingbench tool against the Oracle database for a few minutes and then use SQL statements or Oracle Enterprise Manager (OEM) to ensure that the size of the database has been changed. Restoring the Oracle database After the Oracle database is backed up successfully, you can change the database to suit the test plan. To simulate recovering an entire database to its last full database backup, use Avamar Administrator’s Select for Restore option in the Backup and Restore panel shown in Figure 38. Select the Oracle client and then select the date to be recovered followed by how much of ‘Contents of Backup’ is to be recovered. Choose the entire 11g database (but if you want to back up to a more granular level, you can click on the ‘+’ sign to the left to reveal tablespaces and datafiles). From the Action menu, select Restore now to restore the database. EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 75 Solution Verification Figure 38. Avamar Administrator Backup and Restore tab Once the restore job is complete, the job status changes to Completed as shown on the Avamar administrator activity screen displayed in Figure 39. Figure 39. Avamar's Activity panel showing a Completed job Use Oracle Enterprise Manager GUI or SQL statements to verify that the datasets are successfully recovered. 76 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Chapter 6 Reference Documentation This chapter presents the following topics: White papers ............................................................................................. 78 Product documentation ............................................................................. 79 Other documentation ................................................................................. 80 Links ......................................................................................................... 80 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 77 Configuration Worksheet White papers Oracle • Oracle Edition Comparisons http://www.oracle.com/us/products/database/enterpriseedition/comparisons/index.html • Oracle Software Investment Guide http://www.oracle.com/us/corporate/pricing/sig-070616.pdf • Oracle Database Licensing http://www.oracle.com/us/corporate/pricing/databaselicensing-070584.pdf • Oracle Processor Core Factor Table http://www.oracle.com/us/corporate/contracts/processor-core-factor-table070634.pdf • Installing and Using Standby Statspack in 11gR2 [ID 454848.1] • How to Tell if the IO of the Database is Slow • HugePages on Linux: What It Is... and What It Is Not... [ID 361323.1] [Article ID 1275596.1] https://support.oracle.com (Requires Login) EMC • Deploying Oracle Database on EMC VNX Unified Storage http://www.emc.com/collateral/hardware/white-papers/h8242-deployingoracle-vnx-wp.pdf VMware • Understanding Oracle Certification Support and Licensing in Vmware Environments http://www.vmware.com/files/pdf/techpaper/vmw-understanding-oraclecertification-supportlicensing-environments.pdf • Oracle Databases on VMware Best Practices Guide http://www.vmware.com/files/pdf/partners/oracle/Oracle_Databases_on_VM ware_-_Best_Practices_Guide.pdf • Best Practices for running VMware vSphere on NFS. http://www.vmware.com/files/pdf/VMware_NFS_BestPractices_WP_EN.pdf • Performance Best Practices for VMware vSphere™ 5.1 http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.1.pdf 78 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Product documentation All of the links included in this section were working correctly at the time of publication. EMC • EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 500 Virtual Machines – Reference Architecture • VNX System Installation Guide • VNX Series Configuration Worksheet • Using EMC VNX Storage with VMware vSphere–TechBook • EMC VSI for VMware vSphere: Unified Storage Management—Product Guide • EMC VSI for VMware vSphere: Unified Storage Management—Product Guide • VNX family http://www.emc.com/storage/vnx/vnx-family.htm • VNX series documentation on EMC online support site https://support.emc.com/products/12781_VNX-Series/Documentation/ Oracle • Oracle Database 11gR2 Documentation Library 11g Release 2 (11.2) http://www.oracle.com/pls/db112/homepage http://support.oracle.com • • VMWare Oracle Database 11g Release 2 documentation: http://docs.oracle.com/cd/E11882_01/install.112/e24321.pdf http://www.oracle.com/technetwork/indexes/documentation/index.html Oracle Database Release Notes for Linux • My Oracle Support (https://support.oracle.com/ latest OPatch and latest PSU • My Oracle Support Note ID 749851.1 • My Oracle Support Note ID 1392497.1 • My Oracle Support Note ID 361468.1 • My Oracle Support Note ID 1359094.1 • Sphere Installation and Setup • vCenter Server and Host Management Guide • Installing vCenter Server 5.1 best practices • vSphere Networking • vSphere Storage • vCenter Server and Host Management EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 79 Configuration Worksheet • vSphere Installation and Setup vCenter Server and Host Management • vSphere Resource Management • Automating High Availability (HA) Services with VMware HA –VMware Technical note • General documentation: http://www.vmware.com • VMware vSphere Documentation http://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-pubs.html • vSphere PowerCLI Documentation http://www.vmware.com/support/developer/PowerCLI/ Other documentation • Oracle Database 11g Release 2 on Red Hat Enterprise Linux 6 Deployment Recommendations http://www.redhat.com/resourcelibrary/reference-architectures/deployingoracle-11gr2-on-rhel-6 Links • Everything Oracle at EMC https://community.emc.com/community/connect/everything_oracle • My Oracle Support: Oracle Support Website https://support.oracle.com (user account required) 80 FAQ: How to Use AWR reports to Diagnose Database Performance Issues [ID 1359094.1] Requirements for Installing Oracle 11gR2 RDBMS on RHEL6 or OL6 64-bit (x86-64) [ID 1441282.1] How to Tell if the IO of the Database is Slow [Article ID 1275596.1] HugePages on Linux: What It Is... and What It Is Not... [ID 361323.1] USE_LARGE_PAGES To Enable HugePages In 11.2 [ID 1392497.1] HugePages on Oracle Linux 64-bit [ID 361468.1] Installing and Using Standby Statspack in 11g [ID 454848.1] Installing and Configuring StatsPack Package[Article ID 149113.1] EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Appendix A Configuration Worksheet This appendix presents the following topic: Configuration worksheet for Oracle Database 11gR2................................... 82 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 81 Configuration Worksheet Configuration worksheet for Oracle Database 11gR2 Before configuring Oracle Database 11gR2 for this solution, you need to gather some customer-specific configuration information such as IP addresses, hostnames, and so on. The following tables provide a worksheet that you can use to record the information. You can also print and use the worksheet as a customer “leave behind” document for future reference. To confirm the customer information, cross-reference with the relevant array configuration worksheet: EMC VNX File and Unified Worksheets. Table 25. Common server information Server Name Purpose Primary IP Domain Controller DNS Primary DNS Secondary DHCP NTP SMTP SNMP vCenter Server Swingbench server Database server 01 Database server 02 Database server 03 Table 26. Server Name ESXi server information Purpose Primary IP Private Net (storage) addresses ESXi Host 1 ESXi Host 2 … 82 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide Configuration Worksheet Table 27. Array information Array name Root password NAS admin password CS0 Primary hostname CS1 Secondary hostname (if applicable) CS0 Primary IP address CS1 Secondary IP address (if applicable) SPA IP address SPB IP address Virtual machine infrastructure pool name Virtual machine infrastructure file system name Datafile storage pool name Datafile file system name Redo log storage pool name Redo log file system name Fra storage pool name Fra file system name Table 28. Name Network infrastructure information Purpose IP Subnet Mask Default Gateway Ethernet Switch 1 Ethernet Switch 2 … EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide 83 Configuration Worksheet Table 29. Name VLAN information Network Purpose VLAN ID Allowed Subnets Virtual Machine Networking Management NFS networking vMotion Table 30. Account Service accounts Purpose Password (optional, secure appropriately) vCenter Server administrator Array administrator Oracle Database administrator 84 EMC VSPEX for Virtualized Oracle Database 11g OLTP Implementation Guide