Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Extensible Storage Engine wikipedia , lookup
Relational model wikipedia , lookup
Database model wikipedia , lookup
Microsoft Access wikipedia , lookup
Clusterpoint wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Object-relational impedance mismatch wikipedia , lookup
White Paper EMC AUTOMATED PERFORMANCE OPTIMIZATION for MICROSOFT APPLICATIONS EMC VMAX, FAST VP, and Microsoft Hyper-V • Automated performance optimization • Cloud-ready infrastructure • Simplified, automated management EMC Solutions Group Abstract This white paper presents a solution that explores the scalability and performance for shared application workloads using EMC® Fully Automated Storage Tiering for Virtual Pools (FAST VP) and highlights the ease of management with Microsoft System Center Virtual Machine Manager (SCVMM 2008 R2). March 2012 Copyright © 2012 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H8763.1 EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 2 Table of contents Executive summary............................................................................................................................... 7 Business case .................................................................................................................................. 7 Solution overview ............................................................................................................................ 7 Introduction.......................................................................................................................................... 9 Purpose ........................................................................................................................................... 9 Scope .............................................................................................................................................. 9 Audience ......................................................................................................................................... 9 Terminology ................................................................................................................................... 10 Technology overview .......................................................................................................................... 11 EMC Symmetrix VMAX .................................................................................................................... 12 EMC Symmetrix Management Console ........................................................................................... 12 Microsoft Hyper-V .......................................................................................................................... 12 Microsoft SCVMM........................................................................................................................... 13 Benefits..................................................................................................................................... 13 EMC Replication Manager .............................................................................................................. 13 EMC FAST VP .................................................................................................................................. 13 Consolidated applications storage design on EMC VMAX ................................................................... 15 Overview ........................................................................................................................................ 15 Application environment highlights ............................................................................................... 15 General design guidelines.............................................................................................................. 16 FAST VP .......................................................................................................................................... 17 FAST VP theory .......................................................................................................................... 18 FAST VP storage design for applications ............................................................................................ 19 FAST VP storage sizing for Microsoft applications ........................................................................... 19 FAST VP building block for Exchange 2010 ..................................................................................... 19 Phase 1 – Collect user requirements ......................................................................................... 20 Phase 2 - Design the storage architecture based on user requirements ..................................... 22 IOPS calculation example .......................................................................................................... 22 Disk space calculations ............................................................................................................. 22 Capacity calculation example .................................................................................................... 23 Phase 3 – Validate design ......................................................................................................... 24 Microsoft SQL Server ...................................................................................................................... 24 Phase 1 – Collect user requirements ......................................................................................... 24 Phase 2 – Design the storage architecture based on user requirements .................................... 25 IOPS calculation ........................................................................................................................ 25 Capacity calculation .................................................................................................................. 26 EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 3 Microsoft SharePoint ................................................................................................................. 27 Phase 1 – Collect user requirements ......................................................................................... 27 Phase 2 – Design the storage architecture based on user requirements .................................... 28 IOPS calculation ........................................................................................................................ 28 Disk space calculation............................................................................................................... 28 Final best configuration ................................................................................................................. 29 FAST VP management tools ............................................................................................................ 29 FAST VP policy................................................................................................................................ 29 Applications building block design on Symmetrix VMAX .................................................................... 31 Microsoft Exchange ........................................................................................................................ 31 Exchange Server 2010 database and DAG design ...................................................................... 31 Hyper-V and virtual machine design .......................................................................................... 32 Microsoft SQL Server ...................................................................................................................... 33 SQL Server configuration overview ............................................................................................ 33 SQL Server test application ....................................................................................................... 34 SQL Server database design overview ....................................................................................... 34 Hyper-V and virtual machine design .......................................................................................... 34 Microsoft SharePoint ..................................................................................................................... 35 SharePoint 2010 farm design considerations ............................................................................ 35 SharePoint farm design overview .............................................................................................. 36 Server role overview .................................................................................................................. 36 SharePoint WFE server ............................................................................................................... 37 Database server ........................................................................................................................ 37 SharePoint crawl server ............................................................................................................. 37 SharePoint query server ............................................................................................................ 37 SharePoint database design...................................................................................................... 37 Hyper-V and virtual machine design .......................................................................................... 38 Microsoft SQL Server and SharePoint Hyper-V design ................................................................ 39 Virtual Provisioning and FAST VP design on the Symmetrix VMAX................................................... 39 Space reclaim utilities.................................................................................................................... 41 Microsoft SCVMM........................................................................................................................... 43 Backup and restore of Microsoft applications using Replication Manager.......................................... 45 Overview ........................................................................................................................................ 45 Replication Manager design ........................................................................................................... 45 Replication Manager theory and best practices ......................................................................... 45 Exchange................................................................................................................................... 46 SQL Server................................................................................................................................. 46 SharePoint ................................................................................................................................ 46 Replication Manager design layout ............................................................................................ 46 Microsoft Exchange ................................................................................................................... 47 EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 4 Microsoft SQL Server and SharePoint ........................................................................................ 47 Rapid restore using Replication Manager................................................................................... 49 Microsoft SQL Server ................................................................................................................. 49 Rapid restore using Replication Manager................................................................................... 50 Microsoft SharePoint ................................................................................................................. 52 Rapid Restore using Replication Manager .................................................................................. 55 Performance testing and validation results ........................................................................................ 57 Notes ............................................................................................................................................. 57 Methodology and tools .................................................................................................................. 57 LoadGen.................................................................................................................................... 58 TPC-E ......................................................................................................................................... 58 VSTS ......................................................................................................................................... 58 Data collection points .................................................................................................................... 60 Microsoft Exchange ................................................................................................................... 60 Microsoft SQL Server ................................................................................................................. 60 Microsoft SharePoint ................................................................................................................. 60 Microsoft Exchange test results with FAST VP ................................................................................. 60 Validation results with Jetstress ................................................................................................ 60 Environment validation with LoadGen ....................................................................................... 61 Test 1: End-to-end validation with FAST VP for Exchange under normal conditions .................... 62 Objectives ................................................................................................................................. 62 Configuration ............................................................................................................................ 62 Performance results and analysis .............................................................................................. 62 Test 2: End-to-end validation with FAST VP for Exchange in a failover condition ......................... 63 Objectives ................................................................................................................................. 63 Configuration ............................................................................................................................ 63 Performance results and analysis .............................................................................................. 63 Exchange client results.............................................................................................................. 64 Consolidated applications test results with FAST VP ...................................................................... 65 Environment validation for all Microsoft applications ................................................................ 65 Objectives ................................................................................................................................. 66 Configuration ............................................................................................................................ 66 Microsoft Exchange ........................................................................................................................ 66 Hyper-V and virtual machines performance results .................................................................... 66 Exchange client results.............................................................................................................. 67 Microsoft SQL Server ................................................................................................................. 68 Performance results and analysis .............................................................................................. 68 SharePoint ................................................................................................................................ 70 Test methodology design and data preparation ......................................................................... 71 SharePoint test results summary ............................................................................................... 71 Virtual machine performance results ......................................................................................... 72 EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 5 SharePoint test results .............................................................................................................. 73 Metalogix StoragePoint test summary ............................................................................................ 75 Metalogix StoragePoint BLOB externalization overview ............................................................. 75 BLOB externalization results ..................................................................................................... 75 RBS externalization best practices ............................................................................................ 78 Impact of TimeFinder/snapshot ................................................................................................. 78 Storage and performance ............................................................................................................... 79 Snapshot sizing ............................................................................................................................. 81 Hyper-V HA migration ..................................................................................................................... 82 Conclusion ......................................................................................................................................... 84 Summary ....................................................................................................................................... 84 Findings ......................................................................................................................................... 84 Appendix ............................................................................................................................................ 85 SQL Server restore using Replication Manager ............................................................................... 85 SharePoint Restore using Replication Manager .............................................................................. 86 References.......................................................................................................................................... 92 Product documentation.................................................................................................................. 92 EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 6 Executive summary This white paper outlines the design guidance and architecture of a solution for a mixed Microsoft application workload on EMC® Symmetrix VMAX™ utilizing Microsoft Hyper-V as the hypervisor. The demonstrated applications are Microsoft Exchange Server 2010, SharePoint 2010, SQL Server 2008 R2, and System Center Virtual Machine Manager (SCVMM 2008 R2), to provide for management of the Hyper-V virtual machines. Business case When deploying multiple applications on a storage array, today’s customers need to take performance, total cost of ownership, and ease of management into consideration, and architect a solution that best meets all their needs. To achieve this, many customers have completely separated physical infrastructures. This can lead to an increase in complexity and higher operational costs, management costs, and capital costs. For today’s customers, it is a constant critical business challenge to maintain or improve the performance of a company's mission-critical applications, while reducing costs and providing for simplified management to administer the environment. One way to reduce costs and eliminate bottlenecks is through manual application layouts on static storage tiers and virtual provisioning bottlenecks but the complexity of managing multiple applications does not justify it any longer. This solution provides a simplified architecture to host the different business applications of a company, ensuring that each business line’s information is separated from that of the other. It greatly simplifies the overall environment and reduces operational and management costs. By leveraging additional EMC software such as EMC Symmetrix Fully Automated Storage Tiering for Virtual Pools (FAST VP), Virtual Provisioning™, and Microsoft SCVMM, the solution provides for automated storage tiering and enables quick provisioning of virtual machines and rapid scaling of additional workloads. Solution overview This solution simulates a scenario demonstrating three distinct applications, each with separate requirements for user profile and size, where the physical infrastructure is shared. For example, this architecture could be used for a large enterprise customer whose IT department is looking to provide for ease of management, growth, and automated performance-tuning, using storage tiering for the different business units within the company. There is a growing need among customers to be able to run multiple workloads and applications on shared infrastructure and meet expected performance levels at lower costs, as dictated by business service level agreements (SLAs). This solution showcases a comprehensive design methodology to run a consolidated workload across the Symmetrix VMAX platform, and demonstrates a private cloud solution for customers who are looking for enterprise consolidation with VMAX. The environment includes Exchange Server 2010, SharePoint 2010, and SQL Server 2008 R2. The architecture includes management provided by SCVMM to manage the Hyper-V environment. In an enterprise environment, a customer must allow for requirements where multiple applications are hosted on the same hardware. This solution, on the EMC VMAX array, is ideal because it enables virtual machines to be automatically allocated in a EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 7 balanced way, using Hyper-V's performance and resource optimization (PRO) loadbalancing feature, while at the same time continuing to meet or exceed performance levels per the business SLA, by leveraging EMC’s FAST VP automated tiering technology. Microsoft SCVMM allows rapid provisioning of virtual machines on demand by each business unit for rapid scaling of additional workloads. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 8 Introduction Purpose This white paper presents a solution that explores the scalability and performance of shared application workloads using FAST VP and highlights the ease of management with Microsoft SCVMM. It utilizes the VMAX platform to host and protect the consolidated Microsoft workload, which appeals to the enterprise customer market. Scope The scope of this paper is to document: Audience • The deployment of consolidated Microsoft applications on the VMAX array • Performance and test results: automated storage performance using FAST VP storage tiering technology • Protection of the consolidated environment using EMC Replication Manager • Application protection using database availability groups (DAGs) for Microsoft Exchange • The impact of hardware failure on applications The intended audience for the white paper is: • Customers • EMC partners • Internal EMC personnel EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 9 Terminology This paper includes the following terminology: Table 1. Terminology Term Definition EMC VMAX The array offering by EMC to provide high-end storage for the virtual data center. Its innovative EMC Symmetrix Virtual Matrix Architecture™ seamlessly scales performance, capacity, and connectivity on demand to meet all application requirements. EMC Replication Manager The EMC Replication Manager product provides simplified management of storage replication and integration with critical business applications to take disk-based copies that serve as foundation for recovery operations. DAG DAG is the base component of the high availability (HA) and site resilience framework built into Microsoft Exchange Server 2010. A DAG is a group of up to 16 mailbox servers that hosts a set of databases and provides automatic database-level recovery from failures that affect individual servers or databases. SCVMM Microsoft System Center Virtual Machine Manager 2008 R2 (SCMM), Service Pack 1 (SP1) helps enable the centralized management of the physical and virtual IT infrastructure, increased server utilization, and dynamic resource optimization across multiple virtualization platforms. It includes end-to-end capabilities such as planning, deploying, managing, and optimizing the virtual infrastructure. FAST VP Fully Automated Storage Tiering for Virtual Pools supports automated storage tiering at the sub-LUN level. VP refers to virtual pools, which are virtual provisioning thin pools. VLUN The Virtual LUN technology enables migration of data between storage tiers within the same array, without any production disruption. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 10 Technology overview This solution provides a comprehensive design methodology aimed at helping customers to create a scalable building block for their business units, including Microsoft Exchange Server 2010, SharePoint Server 2010, and SQL Server 2008 R2. The testing simulates a consolidated multi-application scenario by demonstrating three separate environments—Messaging (Exchange), Online Transaction Processing (OLTP) Database (SQL), and enterprise-class collaboration (SharePoint)—each with separate requirements for user profile and size, where the physical infrastructure is shared across the VMAX platform and Hyper-V environment. The solution also includes local HA and backup and restore for all three applications. The following components are used in this solution: • EMC VMAX • EMC Symmetrix Management Console • Microsoft Hyper-V virtualization technology • Microsoft SCVMM • EMC Replication Manager • EMC FAST VP • Metalogix for SharePoint binary large object (BLOB) externalization Figure 1. Physical architecture design EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 11 EMC Symmetrix VMAX EMC Symmetrix VMAX Enginuity version 5875, with the strategy of simple, intelligent, modular storage, incorporates a new, highly scalable Virtual Matrix Architecture™ that enables VMAX arrays to grow seamlessly and cost-effectively from an entry-level configuration into the world’s largest storage system. It offers: • Unmatched application availability: For the highest level of information protection, the VMAX system is the only platform in the industry that can deliver comprehensive solutions for local, remote, and multi-site business continuity. • Redundancy: The VMAX platform provides for excellent redundancy, including multiple engines, with two integrated highly available directors per engine, and a mirrored cache across all the engines that prevent it from being a “single point of failure”. • More scalability: Up to twice the performance, with the ability to manage up to 10 times more capacity per storage administrator. • More security: Built-in encryption, RSA-integrated key management, increased value for virtual server and mainframe environments, replication enhancements, and a new eLicensing model. The tiered storage configuration used in the test environment is based on the following VMAX features: EMC Symmetrix Management Console Microsoft Hyper-V • Sub-LUN FAST VP • Virtual LUN VP mobility The Symmetrix VMAX storage system provides a built-in web browser interface, Symmetrix Management Console (SMC). SMC provides a centralized management to the entire VMAX storage infrastructure. In the context of FAST, SMC integrates easy-to-use wizards to: • Create thin pools • Create storage tiers • Define and associate storage groups • Set up FAST policies Microsoft Windows Server 2008 R2, with Hyper-V, builds on the architecture and functions of Windows Server 2008 Hyper-V by adding multiple new features that enhance product flexibility. Hyper-V provides server virtualization and has increased flexibility in the deployment and life cycle management of applications. Hyper-V virtualization helps to consolidate workloads and reduce server sprawl. Additionally, it allows deploying virtualization with clustering technologies to provide a robust IT infrastructure with high availability and quick disaster recovery. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 12 Microsoft SCVMM Microsoft Virtual Machine Manager 2008 R2, with Service Pack 1 (SP1), provides centralized management of the physical and virtual IT infrastructure, increased server utilization, and dynamic resource optimization across multiple virtualization platforms. It includes end-to-end capabilities such as planning, deploying, managing, and optimizing the virtual infrastructure. Benefits • Centrally creates and manages virtual machines across the entire datacenter. • Easily consolidates multiple physical servers onto virtual hosts. • Rapidly provisions and optimizes new and existing virtual machines. • PRO enables the dynamic management of virtual resources through management packs that are PRO-enabled. As an open and extensible platform, PRO encourages partners to design custom management packs that promote the compatibility of their products and solutions with PRO's powerful management capabilities. EMC Replication Manager EMC Replication Manager manages EMC’s point-in-time replication technologies through a centralized management console. Replication Manager coordinates the entire data replication process—from discovery and configuration to the management of multiple application consistent disk-based replicas. Auto-discover your replication environment and enable streamlined management by scheduling, recording, and cataloging replica information, including auto-expiration. With Replication Manager, you can put the right data in the right place at the right time—on-demand or based on schedules and policies that you define. This application-centric product allows you to simplify replication management with application consistency. EMC FAST VP FAST VP is EMC’s new automated storage tiering technology introduced in the latest Symmetrix Enginuity microcode version (5875). FAST VP combines a "sub-volume" auto-tiering technology from EMC and leverages Virtual Provisioning technology. It enables storage administrators to implement automated, policy-driven plans that perform dynamic, nondisruptive changes to the storage layouts of different applications. This is done by ensuring that the hot spots of a volume or LUN are served by high-performance drives and the inactive data is handled by cost-effective drives. With FAST VP, customers can achieve: • Most efficient utilization of Flash drives for high-performance workloads • Lower cost of storage with the majority of the less-accessed data on Serial Advanced Technology Attachment (SATA) drives • Better performance at a lower cost, requiring fewer drives, less power and cooling, and a smaller footprint • Radically simplified automated management in a tiered environment EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 13 FAST VP can move data among (up to) three tiers (Flash drive, fibre channel (FC), and SATA drives) to meet the performance and capacity demands of a broad range of applications. Frequently accessed data is moved to, or kept at, proper storage tiers, based on the access patterns of sub-volumes and the defined FAST policy. Based on the changing performance requirements of applications, FAST VP only promotes those active hot spots of a volume or LUN to high-performance drives such as Flash drives, but not the entire volume or LUN. At the same time, FAST VP also moves less accessed portions of a volume or LUN to low-cost drives such as SATA drives. Customers thus get the best of both worlds; high performance and low cost. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 14 Consolidated applications storage design on EMC VMAX Overview Storage design is a critical element to successfully deploy a mixed Microsoft application workload on the VMAX, hosted on the Hyper-V virtualization server platform. The process is essentially the same design for a physical environment as a virtual building block, from a disk perspective, except that in the virtual environment the virtual machine’s OS volume also needs to be accounted for. The virtualized Microsoft consolidated application storage design comprises a number of different pieces including: • Storage design requirements—Disk requirements and layout. • Virtual Provisioning—Pool design. • FAST VP automated storage tiering design. • Hyper-V virtual machine building block—Hyper-V root and virtual machine design and resource requirements. Each of these will be discussed in detail in this section. • Backup and restore design and capabilities using EMC TimeFinder® snapshots and Replication Manager. The high level objectives of this design are to: • Validate and show how FAST VP in a virtualized consolidated Microsoft environment can support multiple Exchange, SQL and SharePoint workloads across the same disks • Showcase how easy it is to provide each application the performance level and flexibility by using the right mix of drive types • Provide sizing recommendations for all of the applications in a consolidated FAST VP environment • Show how the VMAX and replication manager can be used to centrally manage protect multiple applications. • Validate Microsoft SCVMM: Application environment highlights Monitor the health and performance of the environment The solution was designed to showcase a mixed Microsoft application workload on the VMAX, hosted on the Hyper-V virtualization server platform. VMAX FAST VP automated storage tiering technology was leveraged to detect the I/O patterns of the different applications and handle unanticipated spikes, and provide optimal performance for multiple applications. Microsoft SCVMM was implemented to manage and monitor the virtualized environment. This solution simulates a multi-tenant scenario by demonstrating three distinct business units, each with unique requirements for user profile and size where the physical infrastructures are shared. This architecture could be leveraged by companies whose IT organizations service different departments. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 15 This solution architecture helps provide direction on how to run consolidated applications of different departments, simplify management of these workloads, and serve as guidance towards a private cloud deployment. The following applications are part of the solution: • • • General design guidelines Microsoft Exchange 2010 Total of 42 databases, with 500 users per database, to house 21,000 users 14 mailbox virtual machines and 14 HUB/CAS servers in a two-copy DAG configuration Four Hyper-V servers hosting 28 virtual machines, to support 21,000 Exchange 2010 DAG environment Replication Manager application sets and jobs are based on the grouping of three databases taking snapshots off the passive copy Microsoft SQL Server 2008 R2 Three SQL servers running hot, warm, and cold TPC-E-like OLTP workloads Total environment size is 1.2 TB A shared Hyper-V cluster with four nodes to support the SQL and SharePoint environment Replication Manager application sets and jobs per server, taking pointer based snapshots for each database Microsoft SharePoint 2010 3 TB SharePoint farm A shared Hyper-V cluster with four nodes to support the SQL and SharePoint environment Replication Manager application sets and jobs to take pointer-based snapshots of the SharePoint farm The following list provides general design guidance for running a mixed Microsoft application workload on a VMAX: • I/O requirements include user I/O (send/receive), any other overhead plus an additional 20 percent. (For Exchange, the 20 percent accounts for some overhead as well as log and BDM I/O) • Database and log I/O should be evenly distributed among the SAN and storage back-end • Use larger hyper volumes when creating LUNs to achieve better performance • A minimum of two HBAs are required per server to provide for redundancy • When using a hypervisor to virtualize the Microsoft Exchange servers a minimum of three IP connections are recommended for each Hyper-V server • Always calculate I/O spindle requirements first, then capacity requirements EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 16 • Use the following read/write ratio for the different Microsoft applications when unsure: Microsoft Exchange—3:2 in a mailbox resiliency configuration and 1:1 in a standalone configuration Microsoft SQL Server—85:15 Microsoft SharePoint Server—Content Databases - 90:10 R:W– Tempdb–Search 50:50 • Install EMC PowerPath® for optimal path management and maximum I/O performance. For more information on installing and configuring the PowerPath application, visit: http://www.emc.com/products/detail/software/powerpathmultipathing.htm • Follow these recommendations to ensure the best possible server performance: Format new NTFS volumes on Windows Server 2008 R2 with the allocation unit size (ALU) set to 64 KB. This can be done from the drop-down list in Disk Manager or through the command prompt using diskpart. Note Partition alignment is no longer required when running Microsoft Windows Server 2008 as partitions are automatically aligned to a 1 MB offset. • Visit the following Microsoft links for guidance on determining server memory and CPU requirements for the Microsoft Exchange 2010 Mailbox Server role: http://technet.microsoft.com/en-us/library/ee832793.aspx (Memory) http://technet.microsoft.com/en-us/library/ee712771.aspx (CPU) • Visit the following Microsoft links for guidance on determining server memory and CPU requirements for Microsoft SQL Server http://msdn.microsoft.com/en-us/library/ms143506.aspx • Visit the following Microsoft links for guidance on determining server memory and CPU requirements for Microsoft SharePoint http://technet.microsoft.com/en-us/library/cc261795(office.12).aspx FAST VP FAST VP is a policy-based system that automatically moves sub-LUN data across storage tiers to achieve performance service levels and cost targets. It enables the storage administrator to set high-performance policies that utilize more Flash drive capacity for critical applications, and cost-optimized policies that utilize more SATA drive capacity for less-critical applications. This section introduces FAST VP functionality and describes how the technology was leveraged in this solution to provide optimal performance for all applications. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 17 FAST VP provides automation to the tiered storage for Virtually Provisioned (VP) devices. Sub-LUN data movement for VP devices provides dramatically improved capacity utilization and a reduction in time and complexity to manage storage. This enables FAST VP to: • Be more responsive to changes in the production workload activity • Improve performance • Utilize capacity more efficiently by: Requiring fewer Flash drives in the system Placing more data on FC and SATA drives FAST VP theory FAST VP has three main components: • Storage tiers: A storage tier is a combination of drive technology and RAID protection in the VMAX array. The storage tiers can be either virtual pools or disk groups. • Storage group: Storage groups are collections of LUNs that are associated with a FAST VP policy. In this solution there are three storage groups (one per application) associated with FAST VP policies. • FAST policy: FAST policy is a setting that initiates data movement between storage tiers based on compliance and performance policy thresholds. A FAST VP policy combines storage groups with storage tiers, and defines the configured capacities, as a percentage that a storage group is allowed to consume on that tier. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 18 FAST VP storage design for applications It is necessary to perform disk sizing for all applications individually before designing the best configuration for FAST VP. Sizing will depend on multiple factors, such as disk type, protection type, and cache. The following is a general guideline approach for sizing a consolidated workload with FAST VP: • Perform sizing as per FAST VP policy requirements. A recommended starting point would be 80/20 skew assumption. Note Skew can vary and will depend on the actual application profile. • Choose RAID 5 or RAID 6 protection types for faster tiers; FC and EFD to yield the best total cost of ownership (TCO) • Mirrored protection for SATA normally yields best performance results • For Exchange only: Separate DAG copies on different spindles Size the tiers to account for a full failover scenario where all the database copies are running on a single server Perform sizing exercise for only FC and SATA tiers and allow a small amount of EFD to handle unanticipated spikes Adopt a building block approach–size for one server and scale This section addresses the sizing for each of the Microsoft applications that are part of this solution. FAST VP storage sizing for Microsoft applications Sizing and configuring storage for use with Microsoft applications can be a complicated process, driven by many variables and factors, which vary from organization to organization. Properly configured storage, combined with properly sized server and network infrastructure, can guarantee smooth operation and the best user experience. The sizing for this solution was performed for consolidated workloads of multiple Microsoft applications. The following sizing exercise provides general guidance on FAST VP sizing for consolidated workloads. The sizing was performed for this solution to best leverage a three-tier FAST VP solution to support a mixed Microsoft application workload. The storage requirements for Microsoft Exchange, often a capacity-driven application, FAST VP building block for Exchange can in most cases be satisfied by SATA drives. This solution, nevertheless, showcases how Exchange works with FAST VP, particularly in a consolidated application 2010 environment leveraging multiple storage tiers. One of the methods that can be used to simplify the sizing and configuration of large Microsoft Exchange environments is to define a unit of measure—a building block. A building block can be defined as the amount of disk and server resources required to support a specific number of Microsoft Exchange Server 2010 users. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 19 The amount of required resources is based on: • Specific user profile type • Mailbox size • Disk requirements The building block approach simplifies the implementation of Microsoft Exchange Server 2010 Mailbox server. Once the initial building block is designed, it can be easily reproduced to support the required number of total users in an organization. This approach serves as a baseline for Microsoft Exchange administrators to create their own building blocks, based on their company’s specific Microsoft Exchange environment requirements. This approach is very helpful when future growth is expected, as it makes Microsoft Exchange environment expansion much easier, and straightforward. EMC’s best practices involving the building block approach for Microsoft Exchange Server design has proved to be very successful throughout many customer implementations. Phase 1 – Collect user requirements The user requirements used to validate both the building block storage design methodology and VMAX performance are detailed in Table 2. Table 2. User requirements Item User equipment Total number of users 21,000 User I/O profile 150 messages sent\received per day = .15 IOPS per user per day in a DAG setup User mailbox size Start with 500 MB, grow to 1 GB Deleted item retention 14 days Concurrency 100 percent Recovery point objective (RPO) Remote < 5 minutes, local = 6 hours Recovery time objective (RTO) 60 minutes Mailbox resiliency solution (DAG) Yes Backup/restore required Yes (hardware volume shadow service (VSS)) The requirements include starting with a user mailbox size of 500 MB with the ability to seamlessly grow to 1 GB. This document shows how this can be easily accomplished, using the VMAX Virtual Provisioning feature (Virtual Provisioning and FAST VP design on the Symmetrix VMAX). EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 20 Based on the user requirements, a virtual Microsoft Exchange Building block of 3,000 users per server was created. The decision for the number of users per server was based on a number of factors, including: • Total number of users—use this number to find a figure that can be evenly divided by a per-server number. • User profile—a larger user I/O profile or large mailbox usually dictates fewer users per building block. • Recovery Time Objective—with a smaller RTO and depending on the backup and restore technology used, fewer users may be supported within a single building block. • Array features—the ability of an intelligent array, such as the VMAX, to backup and restore larger amounts of Microsoft Exchange data in seconds, makes it much easier to achieve good consolidation. • Simplicity and ease of design—fewer larger databases will help and using a SAN and intelligent storage will make this easier to achieve. • Hyper-V server configuration, memory and number of virtual CPUs available— the building block must fit the hardware resources so that the Hyper-V resources are factored in Table 3. Table 3. Required information for building block design Building block characteristic Value Maximum number of Microsoft Exchange Server 2010 users per server 3,000 Number of databases per server 6 Number of users per database 500 Disk type Mixed disk type supported by FAST VP storage tiering technology Number of Hyper-V servers for Microsoft Exchange Mailbox virtual machines 4 Total number of Microsoft Exchange Mailbox virtual machines 14 Database read/write ratio 3:2 Logs truncation failure buffer 6 EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 21 Phase 2 - Design the storage architecture based on user requirements It is recommended to first calculate the spindle requirements per building block from an I/O perspective, and then the space requirements. When performing spindle calculations to satisfy I/O requirements, follow the guidelines outlined below: 1. Calculate total Exchange I/O as using the following formula: Total I/O = (No. of users * I/O profile) + 20 percent 2. Apply a FAST policy split to size the tiers. It is recommended to start with an 80/20 skew–80 percent of I/Os to be serviced by faster tier, 20 percent to be serviced by slower tiers 3. Total I/O for faster tier = (0.8 or the larger percentage) * (Total Exchange I/O) 4. Total I/O for slower tier = (0.2 or the smaller percentage) * (Total Exchange I/O) 5. Array IOPs = (Total IOPs *.60) + RAID penalty (Total IOPs *.40) 6. Disks required = Array IOPs / IOPs per spindle Notes • Tasks 5 and 6 need to be performed for each tier involved. • When using thick or thin devices with Microsoft Exchange 2010, the initial spindle requirements for the pools must meet the I/O requirements. Once that has been accomplished, the sizing calculations should follow. IOPS calculation example • Total I/O for 3,000 users= 3000 * .15 = 450 + 20 percent = 540 IOPS • Total I/O for FC = (80 percent of 540) = 432 • Total I/O for SATA = (20 percent of 540) = 108 • SATA disks required to service 108 I/Os in a RAID 1 configuration • FC disks required to service 432 I/Os in a RAID 5 configuration • (108*.60)*W 2(108*.40)= 152 / 55 ~ 4 (432*.60)*W 4(432*.40)= 948 / 130 ~ 8 From an I/O sizing perspective, using 80 percent FC, 20 percent SATA split, the following disks are required per server: 4 7.2k 2 TB SATA 8 10k 600 GB FC Disk space calculations EMC recommends that the Microsoft calculator be used when performing disk space calculations. Current Microsoft calculations require more than 60 percent capacity over the user mailbox target size. EMC’s Virtual Provisioning is a key to reducing upfront storage purchase and addressing any unforeseen growth. The LUN calculations were performed to account for the final mailbox size (1 GB) and the spindle calculations for the initial size (0.5 GB). EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 22 When performing capacity calculations, follow the outlined steps below: • Calculate total capacity based on mailbox requirements • Apply a FAST policy split to size the tiers. EMC recommends to start with an 80/20 skew–80 percent of capacity to reside on slower tier • Total capacity for slower tier = (0.8 or the larger percentage) * (Total Exchange Capacity) • Total capacity for faster tier = (0.2 or the smaller percentage) * (Total Exchange Capacity) • Spindle requirement per server = <Total Capacity>/ <Usable Capacity> Note Sub-task 6 needs to be performed for each tier involved. Capacity calculation example • Capacity requirements = Total database capacity per server + Total log capacity per server • Database size = <Number of mailboxes> x <Mailbox size on disk> x <Database overhead growth factor> = 500 users x 1.23 GB + 20 percent = 738 GB Based on this, the database LUN capacity can be calculated as follows: • Database LUN size = <Database size> + <Content Index Catalog / (1 - Free space percentage requirement) = (738 GB + 73.8 GB)/0.8 = 1024 GB • Log LUN size = Logs per day per user x number of users x truncation failure tolerance x 20 percent = 30 logs @ 1 MB x 500 users x 6 days x .20 = 115 GB • Total database capacity per server (for initial 0.5 GB mailbox size) = <Database LUN size> * <No. of databases> = 1024*6 = 3072 GB • Total log capacity per server = <Log LUN size> * <No. of databases> = 115*6 = 345 GB • Total capacity = < Total database capacity per server> + < Total log capacity per server> = 3072 + 345= 3417 GB • Usable capacity available per 2 TB SATA drive = 1754 GB • Usable capacity available per 600 GB 10K FC drive = 536 GB • Spindle requirement per server = <Total capacity> / <Usable capacity> • Disk capacity skew recommendations are 80 percent SATA, 20 percent FC • Capacity on SATA is 3417*0.8 = 2733 GB • Capacity on FC is 3417*0.2 = 684 GB • Required spindles (SATA mirrored) – 2733 / 1754 ~ 4 (mirrored disks) • Required spindles (FC RAID 5) – 684 / 536 ~ 4 EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 23 • From a capacity sizing perspective, using an 80 percent SATA and 20 percent FC split, the following disks would be required per server: 4 7.2k 2 TB SATA 4 10k 600 GB FC The best configuration based on both I/O and capacity requirements is as follows: 1 GB mailbox state No. of spindles required to satisfy both I/O and capacity 4 7.2K 2 TB SATA drives Thin LUN size (Database) 1024 GB Thin LUN size (Log) 115 GB 8 10K 600 GB FC For Exchange, the sizing does not account for Flash drives, which are used in small quantities to handle any unanticipated spikes. In many cases, as Exchange 2010 is capacity bound, requirements can be satisfied by SATA drives as opposed to a tiered solution. Phase 3 – Validate design Microsoft Exchange Jetstress and Microsoft Exchange LoadGen tools were used to validate the storage design. For a complete summary of Jetstress and LoadGen findings, see the Performance testing and validation results section of this white paper. Microsoft SQL Server To maintain flexibility, performance, and granularity of recovery, ensure that the storage sizing and back-end configuration for SQL Server is optimal. Outlined in this section is the approach adopted when sizing SQL Server in a FAST VP configuration. Phase 1 – Collect user requirements The SQL configuration for this environment is outlined in Table 4. Table 4. SQL configuration user requirements Item User equipment Total number of users 100,000 Database users per server 20,000, 30,000, 50,000 respectively Total IOPS 6,000 Number of databases 3 Database profile Hot / Warm / Cold RPO Remote < 5 minutes, local = 6 hours EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 24 Item User equipment RTO 60 minutes Read/write ratio 85:15 Backup/restore required Yes (hardware VSS) Phase 2 – Design the storage architecture based on user requirements It is recommended to first calculate the spindles for the SQL Server to satisfy I/O requirements, and then the space requirements. Shown below is the sizing calculation for this solution. IOPS calculation • Total I/O for 225,00 users = 6,000 + 20 percent = 6,000 + 1,200 = 7,200 IOPS • Calculate the back-end I/O as per FAST VP policy requirements. In this solution, the FAST VP sizing was calculated based on the following skew for I/O 75 percent SATA, 15 percent FC and 10 percent Flash. • • Calculate the backend I/O for each tier: Total backend I/O for RAID 1/0 SATA = (10 percent of 7,200) = (720*0.85) + 2 (720*0.15) = 828 Total I/O for RAID 5 FC = (15 percent of 7,200) = (1,080*0.85) + 4(1,080*0.15) = 1566 Total I/O for RAID 5 Flash = (75 percent of 7,200) = (5,040*0.85) + 4 (5,040*0.15) = 7308 Total Back-end I/O = 10,224 SATA disks required to service 808 I/Os in a RAID 1/0 configuration • FC disks required to service 2,088 I/Os in a RAID 5 configuration • 1,566 / 130= ~12 FLASH disks required to service 7,308 I/Os in a RAID 5 configuration Note • 828/50=~17 round up to 18 for R1/0 7,308/1,800=~4 When calculating for performance, the fastest tier needs to service the maximum number of I/Os. From an I/O sizing perspective, using the above policy settings, the following disks are for the environment: 18 7.2k 2 TB SATA drives 12 10k 600 GB FC drives 4 200 GB Flash drives EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 25 Capacity calculation • • User database size Hot – 200 GB Warm – 300 GB Cold – 600 GB Calculate the database LUN size based on the user database sizes: Database LUN size = <Database size> + Free space percentage requirement (20 percent) • Hot = 300 + 20 percent = 360 GB Warm = 400 + 20 percent = 480 GB Cold = 700 + 20 percent = 840 GB Calculate the tempdb and log LUN sizes for each of the databases. The log and tempdb sizes are calculated as 20 percent the size of the database: Log and tempdb size Hot database – 20 percent of 300 = 60 GB Warm database – 20 percent of 400 = 80 GB Cold database – 20 percent of 700 = 140 GB The user database log and the tempdb are laid out on a separate LUN for each database. Based on this, the log LUNs were sized at 120 GB for the hot and warm databases and 140 GB for the cold database: • Total database size = Sum of the sizes of all the databases = 2,448 GB • Usable capacity available per 2 TB SATA drive = 1,754 GB • Usable capacity available per 600 GB 10K FC drive = 536 GB • FAST policy skew used was 75 percent SATA, 15 percent FC and 10 percent Flash • Capacity on each tier is: SATA = 2,448*0.75 = 1,836 GB FC = 2,448*0.15 = 368 GB Flash = 2,448*0.1 = 245 GB • Spindle requirement = <Total capacity> / <Usable capacity> • Spindles required for each tier is: Note SATA (mirrored) = 4 FC (RAID 5 3+1) = 4 Flash (RAID 5 3+1) = 4 When calculating for capacity, the slowest tier needs to host the majority of the data. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 26 • From a capacity sizing perspective, using these policy settings, the following disks are required for the environment: 4 7.2k 2 TB SATA drives 4 10k 600 GB FC drives 4 200 GB Flash drives The best configuration based on both I/O and capacity requirements are as follows: 1 TB ( 200GB, 300GB, 500GB) SQL Server database No. of spindles required to satisfy both I/O and capacity 18 7.2K 2 TB SATA drives 12 10K 600 GB FC drives 4 200 GB Flash drives Hot—360 GB Thin LUN sizes (Database) Warm—480 GB Cold—840 GB Hot—120 GB Thin LUN size (Log) Warm—120 GB Cold—140 GB Microsoft SharePoint The SharePoint Server farm has storage requirements that typically are mid to low IOPS on its content database search components. With FAST VP on VMAX, the SharePoint farm storage configuration can be simplified and the requirements of the disk I/O request in the farm can be dynamically satisfied with the auto-tiering technology. This section outlines the approach adopted when sizing SharePoint Servers in a FAST VP configuration. Phase 1 – Collect user requirements The user requirements used to validate both the building block storage design methodology and VMAX performance are detailed in Table 5. Table 5. Building block storage design methodology and VMAX performance user requirements SharePoint farm profile Quantity/size/type Total data 3 TB Document size range 200 KB-5 MB Total site count 40,000 Size per site 20 GB Total site collections count 16 Content database size 200 GB EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 27 SharePoint farm profile Quantity/size/type Total IOPS 2,000 Total site collection 1 Total user count 30,000-40,000 heavy users (Microsoft defined) Usage profile(s) (% browse / % search / % modify) 80% / 10% / 10% User concurrency 10% Phase 2 – Design the storage architecture based on user requirements It is recommended to first calculate the spindles for SQL Server and other SharePoint Servers that require storage to satisfy I/O requirements, and then for space requirements. In a SharePoint farm, the requirement for IOPS is much less than applications such as SQL Server. A two-tier (SATA and FC) configuration can be used to satisfy its disk needs. Shown below is the sizing calculation for this solution. IOPS calculation • Total I/O for the farm = 2,000*(1 + 20 percent)= 2,400 IOPS • Apply a FAST policy split to size the tiers. EMC recommends to start with a 90/10 skew – 90 percent of I/Os to be serviced by faster tier, 10 percent to be serviced by slower tiers • Calculate back-end I/O for each tier: Total back-end I/O for RAID 1/0 SATA (10 percent of 2,400) = (240*0.85) + 2 (240*0.15) = 276 Total I/O for RAID 5 FC (90 percent of 2,400) = (2,160*0.85) + 4 (2,160*0.15) = 3,132 Total back-end I/O = 3,408 Note • When calculating for performance, the fastest tier needs to service the maximum number of I/Os. From an I/O sizing perspective, using the above policy settings, the following disks are required for the environment: 6 7.2k 2 TB SATA drives (276/50 ~ 6) 24 10k 600 GB FC drives (3132/130 ~ 24) Disk space calculation EMC’s Virtual Provisioning is key to reducing up-front storage purchase and handling any unforeseen growth. When performing capacity calculations, follow the steps below: • Calculate the total capacity based on SharePoint farm requirements • Apply a FAST policy split to size the tiers. In this solution, a 90/10 skew for the SharePoint farm, where 70 percent of the capacity resided on the slower tier, was used. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 28 • Total capacity for slower tier = (0.9 or the larger percentage) * 3 TB = 2.7 TB • Total capacity for faster tier = (0.1 or the smaller percentage) * 3 TB = 0.3 TB • Usable capacity available per 2 TB SATA drive = 1,754 GB • Usable capacity available per 600 GB 10K FC drive = 536 GB • Spindle requirement for SATA drives = 2.7 TB / 1,754 GB = 4 • Spindle requirement for FC drives = 0.3 TB/536 GB ~ 4 • From a capacity sizing perspective, using a 70 percent SATA, 30 percent FC split, the following disks would be required for the farm: 2 7.2k 2 TB SATA 4 10k 600 GB FC The ideal configuration based on both I/O and capacity requirements are as follows: SharePoint farm disk requirement Number of spindles required to satisfy both I/O and capacity Final best configuration 6 7.2K 2 TB SATA drives 24 10K 600 GB FC The final disk configuration for this environment needs to satisfy both I/O and capacity for all of the applications. The application that benefited the most with FAST VP in this environment was SQL Server since the TPC-E-like load was very heavy when compared to the other two applications. The final best disk configuration that satisfies both the I/O and capacity requirements in this mixed workload environment is as follows: Mixed Microsoft workload disk requirement Total number of spindles required for the mixed Microsoft application workload FAST VP management tools FAST VP policy 42 SATA 90 FC 8 Flash (4 introduced to handle unanticipated Exchange spikes) The following management options are available to configure FAST VP: • Solutions Enabler Command Line Interface (SYMCLI) • Symmetrix Management Console (SMC) FAST VP in a VMAX environment provides an easy way to employ the storage service specializations of an array configuration with a mixture of drive types. FAST VP offers a simple and cost-effective way to provide optimal performance of a given mixed configuration, by automatically tiering storage to the changing application needs. The FAST VP tiers for this solution were chosen per the application requirements and FAST EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 29 VP policies were set to allow data movement between the tiers to optimize performance. The following FAST VP tiers were used for this solution: Table 6. FAST VP tiers Tier name Drive technology MS_Flash 200 GB 15k Flash drives MS_FibreChannel 600 GB 10K FC drives MS_SATA 2 TB 7.2K SATA drives The FAST VP policy settings varied for each application since the workload pattern of each application differed and their needs were different. • For Exchange, FAST VP sizing was performed to leverage only FC and SATA tiers, and a small number of Flash drives were introduced to handle any unanticipated spikes and any hot-spots in the workload. This policy helped to automate performance and, at the same time, keep the cost in check. • For SQL, the testing tool emulated an OLTP TPC-E-type load, which performs wide-stripe reads, hence the FAST VP policy was designed to include more Flash to manage the high performance requirements. This provided the best blend of cost and performance. • SharePoint, being a fairly low I/O application, did not require the Flash tier, hence the policy was implemented to include only the FC and SATA tiers. Nevertheless, by sharing the same tiers, and allowing FAST VP to move data per the policy settings, optimal performance was achieved for each application. The skew for each application was obtained by using a modeling tool called Tier Advisor. Performance data is uploaded to the tool, which then models an optimal storage array configuration by enabling interactive experimentation with different storage tiers and storage policies until the desired cost and performance preferences are achieved. Tier Advisor helps define the amount of disk drives to use for each disk drive technology when configuring a tiered storage solution. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 30 Applications building block design on Symmetrix VMAX Once the disk calculations are computed for each application, the virtual machine and Hyper-V requirements can be calculated. The memory and CPU requirements are based on Microsoft best practices. This section details this for each application, separately. Microsoft Exchange Exchange Server 2010 database and DAG design A DAG is a group of up to 16 Mailbox servers that host a set of databases, and provide automatic Exchange database-level recovery from failures that affect individual servers or databases. A DAG is a boundary for mailbox database replication, database and server switchovers and failovers, and for an internal component called Active Manager. Active Manager is an Exchange Server 2010 component that manages switchovers and failovers, and runs on every server in a DAG. The Exchange design for this solution incorporates an active/passive DAG design and adheres to the previously described guidelines, deploying three active and three passive databases for each mailbox virtual machine. Each virtual machine is capable of handling all six databases or 3,000 users in a switchover/failover condition, but under normal conditions only 1,500 users would be active on each mailbox virtual machine. This grouping of three databases and 1,500 users was used and carried over to the backup configuration with Replication Manager, in an effort to make the management as easy as possible. The design also accounted for the reboot or loss of a Hyper-V root server. The active database on MBX1 residing on Hyper-V root server 1 has its passive copies on Hyper-V root server 2. If either Hyper-V root server needs to be rebooted, due to patching or maintenance, there is no loss of service to the user mailboxes. The design incorporated the following: • A total of 21 active and 21 passive databases, with 500 users per database to house the 21,000 users • Mailbox databases are grouped in collections of three (1,500 users) • Each mailbox virtual machine has three active and three passive mailbox databases • The active and passive database copies do not reside on the same virtual machine or on the same Hyper-V root server • Replication Manager application sets and jobs design based on groupings of three databases per Replication Manager backup job EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 31 The DAG configuration on the virtual machines and Hyper-V servers is described in Figure 2. Figure 2. DAG configuration on the virtual machines and Hyper-V servers Hyper-V and virtual machine design Once the user per building block and disk calculations are complete, the virtual machine and Hyper-V requirements can be calculated. Guidelines for memory configurations may be found at: http://technet.microsoft.com/en-us/library/dd346700.aspx CPU and memory requirement calculations start with the mailbox server role. Based on the requirements, the building block must be able to support 3,000 users per server. Provisioning sufficient megacycles so that mailbox server CPU utilization does not exceed 80 percent is also required. Table 7 lists the mailbox server megacycle and memory requirement calculations. Table 7. Mailbox CPU requirements Parameter Value Active megacycles 3,000 mailboxes x 3 megacycles per mailbox = 9,000 Passive megacycles 0 (sizing for all active) Replication megacycles 0 (sizing for all active) Maximum mailbox access concurrency 100% Total required megacycles during mailbox server failure 9,000 Using megacycle capacity to determine the number of mailbox users that a Microsoft Exchange Mailbox Server can support is not an exact science. A number of factors can lead to unexpected megacycle results in test and production environments. Therefore, megacycles should only be used to approximate the number of mailbox users that a Microsoft Exchange Mailbox Server can support. Also, per Microsoft recommendations, a 10 percent overhead needs to be factored in for hypervisor overhead. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 32 Remember that it is always better to be a bit conservative rather than overly aggressive during the capacity planning portion of the design process. Based on Microsoft guidelines and server vendor specifications, the CPU and memory requirements were determined for each virtual machine role. Table 8 provides the summary of the virtual machine CPU and memory configurations, which has taken into consideration a Hyper-V root server failure. Table 8. Virtual machine CPU and memory configurations summary vCPUs per virtual machine Memory per virtual machine Mailbox (to support 3000 users during failover) 4 20 GB HUB/CAS 4 20 GB Virtual machine role A 120 GB VHD volume for the mailbox server OS was provisioned. Pass-through disks were used for the database and log volumes, primarily to accommodate the requirement for a hardware Volume Shadow Service (VSS) snapshot. Table 9 describes those values. Table 9. Microsoft SQL Server Mailbox virtual machine resources Item Description Number of users supported 3,000 User profile supported 0.15 (150 messages / user / day) Database LUN 6 1 TB thin LUNs (pass-through) Log LUN 6 115 GB thin LUNs (pass-through) OS LUN 120 GB (VHD) Virtual machine CPU 4 vCPU Xeon X7560 2.27 GHz Virtual machine memory 20 GB SQL Server configuration overview The Windows and SQL Server 2008 R2 configuration of each virtual machine are as follows: • Grant "Lock pages in memory" to a SQL startup account • Use a 64 KB NTFS allocation unit size for the user data device • Four tempdb data files with an equal initialization size for every SQL instance • Grant the SQL Server Admin user “Perform volume maintenance tasks right”, which enables fast file initialization and supports thin provisioning EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 33 SQL Server test application The test environment for SQL Server is based on a TPC-E-like workload. It is composed of a set of transactional operations simulating the activity of a brokerage firm, such as managing customer accounts, executing customer trade orders, and other interactions with financial markets. SQL Server database design overview The SQL Server test configuration is based on the following profile: • Number of SQL users supported: 100,000 • Simulated user workload with one percent concurrency rate and zero think time, consistent with Microsoft testing methodologies • User data: 1.2 TB Table 10. SQL Server profile Profile Value Total SQL database capacity 1.2 TB Number of SQL instances 3 (1 per virtual machine) Number of user databases per instance 1 Number of virtual machines 3 Type of data store Pass-through SQL virtual machine configuration 4 virtual processors (vCPUs) with 16 GB memory (no over-commitment) Concurrent users Mixed workloads to simulate hot, warm, and cold databases Hyper-V and virtual machine design Once the SQL Server LUN design and disk calculations are complete the virtual machine and Hyper-V requirements can be calculated. Guidelines for memory configurations may be found at: http://msdn.microsoft.com/en-us/library/ms143506.aspx Based on the requirements, the SQL Server needs to support a CPU utilization of less than 80 percent. Based on Microsoft guidelines and server vendor specifications, the CPU and memory requirements were determined for each virtual machine. Table 11 provides the summary of the virtual machine CPU and memory, and storage configurations, which has taken into consideration a Hyper-V root server failure. A 120 GB VHD volume for the SQL server OS was provisioned. Pass-through disks were used for the database and log volumes, primarily to accommodate the requirement for a hardware VSS snapshot. Table 11 and Table 12 describe those values. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 34 Table 11. Microsoft SharePoint SQL Server 2008 R2 virtual machine resources Item SQL Server1 SQL Server 2 SQL Server3 Number of users supported 20,000 30,000 50,000 Database (pass-through thin LUN) 360 GB 480 GB 840 GB Log LUN (pass-through thin LUN) 60 GB 80 GB 120 GB Tempdb/log LUN ( pass-through thin LUN) 60 GB 80 GB 120 GB OS LUN (VHD) 120 GB 120 GB 120 GB Virtual machine CPU ( vCPU Xeon X7560 2.27 GHz) 4 4 4 Virtual machine memory 16 GB 16 GB 16 GB Table 12. Hyper-V server configuration (shared with SharePoint farm virtual machines) Item Description Number of Hyper-V Servers 3 Server type Xeon x7460, 4 sockets, 6 cores Total memory 192 GB Total vCPUs 32 Number of HBAs 2 dual-port 4 Gb QLogic 2462 SharePoint 2010 farm design considerations The SharePoint farm was designed for optimized performance, reduced bottlenecks, and ease of manageability. The SharePoint 2010 farm consists of: • A web application that was created by using the enterprise portal collaboration template • Sixteen enterprise document center site collections were created on 16 content databases on two SQL Server hosts • Two SharePoint crawl servers • Four SharePoint query servers • Four SharePoint web front-end servers • Metalogix StoragePoint installed on the web front-end (WFE) server EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 35 Table 13 describes the SharePoint environment profile. Table 13. Environment profile SharePoint farm profile Quantity/size/type SharePoint – Total Data 3 TB SharePoint – Document Size Range 200 KB-5 MB SharePoint – Total Site Count 40,000 SharePoint – Size Per Site 20 GB SharePoint – Total Site Collections Count 16 SharePoint – Content Database Size 200 GB SharePoint – Total User Count 30-40k SharePoint – Usage Profile(s) (% browse / % search / % modify) 80% / 10% / 10% SharePoint – User Concurrency 10% SharePoint farm design overview In SharePoint 2010, the crawl servers index the contents of the SharePoint farm, then populate the crawl and property stores on the SQL database server, and add content index (CI) files on the query server. In this test environment, 16 content databases on two SQL Server instances were populated with over two million documents, which added up to 3 TB of data. For each crawl server in the SharePoint farm, one 10 GB LUN was created to store the crawl index files. For each query server in the SharePoint farm, two LUNs of 240 GB each were created to store the CI files (the query index partition and its backup mirror) resulting from the crawl operations. The search server in SharePoint Server 2010 is architected to provide greater redundancy in a single farm, and to enable scalability in multiple directions. Each component that makes up the query architecture (the query servers, index partitions, and property database) and the crawling architecture (crawl servers, crawl database, and crawl property database) can be scaled-out separately, based on the needs and growth of an organization. Server role overview The SharePoint 2010 farm configuration details contain the following server roles: • SharePoint WFE server • SQL Server 2008 R2 database server • SharePoint crawl server EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 36 • SharePoint query server SharePoint WFE server • The WFE servers are deployed on a Windows Server 2008 R2 virtual machine • Metalogix StoragePoint 3.0 are installed on one of the WFE servers • Internet Information Service (IIS) is configured to provide web content to SharePoint clients • IIS web garden threads are set at the default value of one for optimal performance and ease of management • IIS logging is disabled to limit the unnecessary growth of log files and to optimize performance Database server • The resource-intensive database servers are deployed on a Windows Server 2008 R2 virtual machine • SQL server 2010 R2 Enterprise application servers are installed • Dedicated SQL Server I/O channels to the back-end disks are used • The database and log LUNs are attached through pass-through LUNs SharePoint crawl server • The crawl servers are installed on Windows Server 2008 R2 virtual machine • SharePoint index crawling roles are deployed to this host • Dedicated crawl index LUNs are attached as pass-through LUNs to store the CI files from the crawling operation SharePoint query server • The query servers are installed on Windows Server 2008 R2 virtual machine • SharePoint query roles are deployed to this host • Dedicated query index LUNs are attached as pass-through LUNs to each of the query server for its query index partitions and mirror • One of the query servers also serves as the Search Admin, and the query index LUN also hosts the Osearch Administration component SharePoint database design The sequence of building a SQL Server environment for SharePoint 2010 requires three different subsets of databases in the following order: • SQL system databases created during installation (master, model, msdb, and tempdb). • SharePoint databases created by the system from the deployment of SharePoint farms (SharePoint admin, configdb, crawl store, Windows SharePoint Services (WSS), and content database). EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 37 • SharePoint content databases, populated with user content documents in the SharePoint farm. • Content database design: SharePoint 2010 content databases are recommended to be 200 GB or less for general usage scenarios, and a maximum of 2,000 items per list. Larger database sizes are supported in certain situations. With over one million documents and the requirement of 3 TB of data, the SharePoint farm is designed to have 16 content databases for 16 sites, and 160 sub-sites. One 250 GB pass-through LUN is configured for each content database. Hyper-V and virtual machine design Once the SharePoint farm Server LUN design and disk calculations are complete, the virtual machine and Hyper-V requirements can be calculated. Guidelines for memory configurations for SharePoint may be found at: http://technet.microsoft.com/en-us/library/cc261795(office.12).aspx Based on the requirements, the SharePoint Server virtual machines need to support a CPU utilization of less than 80 percent. Based on Microsoft guidelines and server vendor specifications, the CPU and memory requirements were determined for each virtual machine role. Table 14 provides the summary of the virtual machine CPU and memory configurations, which has taken into consideration a Hyper-V root server failure. A 120 GB VHD volume for the SharePoint server OS was provisioned. Pass-through disks were used for the database and log volumes, primarily to accommodate the requirement for a hardware VSS snapshot. Table 14. Summary of the virtual machine CPU and memory configurations Virtual machine name Number of servers vCPU Memory (GB) WFE Server 4 4 4 Query Server 4 4 8 Crawl Server 2 4 8 Admin Server 1 2 4 SQL Server 2 4 16 50 100 Total EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 38 Microsoft SQL Server and SharePoint Hyper-V design Figure 3 shows the initial location of the virtual machines on the Hyper-V hosts for SQL Server and the SharePoint farm servers: Figure 3. Hyper-V virtual machine placement The goal is to ensure that the load is evenly distributed across all of the root servers to prevent any performance bottlenecks. The SQL servers and SharePoint machines were distributed evenly across the four Hyper-V root servers, as shown in Figure 3. Depending on the load, the less active database servers could be hosted by the same Hyper-V host. The design in Figure 3 is an example of where the least-active database server (SQL03) was placed with another, less active database server (SP SQL01) on the same Hyper-V machine. For the SharePoint farm, the key is to lay out the different servers with the same role to different Hyper-V hosts, in order to balance the server resource usage. Also, if there is an outage of one Hyper-V server, the requests will be easily serviced by other virtual machines in the farm, while the specific virtual machine fails over to another Hyper-V host and resumes their roles. Virtual Provisioning and FAST VP design on the Symmetrix VMAX This section details how VMAX Virtual Provisioning and FAST VP storage tiering technology was used to provide a well-performing, easy-to-use, and economical design for all of the applications. One of the main advantages of using VMAX Virtual Provisioning is the ability to easily increase the database volumes’ storage as the users’ mailboxes grow, without any server interruptions. This, combined with FAST VP, which allows for sub-LUN level movement between storage tiers, can deliver excellent performance, tremendous savings in disk cost, power, cooling, and a reduced footprint. It also provides for outstanding storage flexibility for a mixed workload Microsoft environment. VMAX Virtual Provisioning and FAST VP work well with Hyper-V and storage technologies such as snapshots, clones, and Symmetrix Remote Data Facility (SRDF). For more information on VMAX virtual provisioning, refer to the Symmetrix Virtual Provisioning Feature Specification paper. When leveraging FAST VP for Microsoft Exchange on a VMAX, the DAG copies were laid out on separate spindles per Microsoft’s best practice recommendations. Hence, two sets of Symmetrix VP tiers were designed to host each DAG copy. One set was scaled dynamically to include disks for SQL and SharePoint to support the consolidated workload. The Exchange FAST VP policy setting was identical for both of the DAG copies. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 39 As seen in Table 15, the policies differed for each application, since the workload pattern was different. SQL Server benefited the most with FAST VP, since the load generated while testing was TPC-E-like, random, heavy workload. The Flash tier was introduced for Exchange to account for any unanticipated spikes. Table 15. FAST VP design for Exchange Application copy DAG Copy 1 FAST VP policy settings Flash 1 percent Fibre Channel 14 percent SATA 85 percent Flash 1 percent Fibre Channel 14 percent SATA 85 percent Flash 10 percent Fibre Channel 15 percent SATA 75 percent Fibre Channel 10 percent SATA 90 percent Microsoft Exchange DAG Copy 2 Microsoft SQL Server Microsoft SharePoint General recommendations for the design of the disk layout with thin pools on VMAX are as follows: • Use large data devices (120 - 240 GB) for the thin pools • Use concatenated thin device meta volumes, since the data devices are already striped, from a disk group perspective The design used for this solution is as follows: • Three virtual pools governed by FAST VP policy • Each virtual pool tier was supported by the number of disks, as calculated during the storage sizing phase • 240 GB data devices were used for the SATA and FC tier, and 20 GB for Flash • EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 40 • • • • Database thin device details are as follows: Exchange: − A 1 TB database thin metaLUN and 115 GB thin log LUN was used to accommodate the 1 GB mailbox requirement − Six databases per server—500 users per database SQL Server: − Three thin metaLUNs (360 GB, 480 GB, and 840 GB, respectively) were carved out to meet the database requirements − Three thin log LUNs (60 GB, 80 GB, and 120 GB, respectively) were used to accommodate the log requirements − Three thin tempdb/log LUNs (60 GB, 80 GB, and 120 GB, respectively) were used for tempdb SharePoint: − Appropriate LUNs were carved out to satisfy the different SharePoint components size requirements. In order to prevent a situation where a thin pool runs out of space, it is recommended that the thin pool utilization threshold tool be used to monitor each pool’s disk space utilization. These settings can be found within the SMC. The tool can be set to send warnings and alerts when the pool’s space utilization reaches a certain level. Space reclaim utilities In many cases, old and unwanted data is deleted, but the actual space is still left allocated to the thin pool, which decreases space utilization efficiency, and increases business costs. When using virtual provisioning, there are two ways to reclaim space to reduce capacity requirements and TCO: • The zero space reclamation utility, which is part of the microcode, de-allocates the extents that contain all zeroes. This utility essentially reclaims the contiguous range of all zero data. • To reclaim non-zero data, host-level interaction is required. The StorReclaim utility is a Windows command line utility designed to free allocated, but unused, storage space as part of EMC’s Virtual Provisioning solution. As part of the solution, the StorReclaim utility was used to showcase how deleted/unused data, which is still allocated, can be reclaimed. The files on one of the mailbox servers were deleted. The data, from the host perspective, was appeared to be deleted but the data showed up as allocated in the thin pool. Figure 4 shows the capacity on the virtual provisioning tiers after the files were deleted: EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 41 Figure 4. Capacity on the virtual provisioning tiers The StorReclaim utility was then run to reclaim any of the unused space and allocate it back to the data devices. The tool was run online while the user applications were accessing the target volume. The command line to reclaim space is shown in Figure 5. Figure 5. Command line to reclaim space Once the command completed, the command to show the pool capacity was executed. As seen in Figure 6, approximately 200 GB of unused space was reclaimed, once the tool was run. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 42 Figure 6. Command line to reclaim space execution results The StorReclaim utility is an extremely useful tool, which can be run: Microsoft SCVMM • Regularly, to free any allocated but unused storage space, which would be released back to a thin pool • Ad-hoc manually, as required, if free space falls below a certain threshold • Online, as shown above, while user applications are accessing the target volume Microsoft SCVMM was used to enable rapid deployment of virtual machines, provide centralized control of the virtual datacenter, and delegate self-provisioning by authorized end-users. SCVMM also provided increased server utilization and dynamic resource optimization across virtualization platforms. Some important capabilities of SCVMM include planning, deploying, managing, and optimizing the virtual infrastructure. • In this solution, Virtual Machine Manager (VMM) was used to provide unified management for the two Hyper-V environments (Exchange and SQL Server/SharePoint infrastructure) from a single console. VMM helped consolidate physical servers in a virtual environment and provide for an aggregate monitoring of the clusters, hosts, and virtual machines in this environment. VMM provided the following benefits: Virtual machine placement – VMM helped ensure that the virtual machines were appropriately placed on the root servers such that the resources were balanced. The star-ranked feature provided by VMM helped with the placement of the virtual machines on the root servers. This star ranking was calculated based on performance data on the virtual hosts, and the resource consumption usage. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 43 Provisioning – With VMM, provisioning servers was easy and fast. The virtual machine templates were stored in the VMM library and helped speed the deployment of new machines, and ensure that standard hardware and software configurations were used. Business Continuity – VMM works with Windows Server 2008 failover clustering technology to provide enhanced availability for virtual machine hosts and guests. VMM is cluster-aware and all virtual machines were automatically configured with high availability settings. To offset the failure of a root server, VMM automatically moved the virtual machines to other root Hyper-V hosts, and also distributed them as evenly as possible between the remaining nodes. Live Migration and Quick Migration tests were performed to test this feature and are documented in the Performance testing and validation results section. Figure 7 shows the VMM environment for the SQL/SharePoint environment. The VMM GUI shows all of the Hyper-V root servers, the virtual machines that are on a particular physical host, and the status. The console allows managing an environment of large numbers of virtual machines effectively, and provides for all of the benefits previously outlined. Figure 7. VMM environment for the SQL/SharePoint environment EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 44 Backup and restore of Microsoft applications using Replication Manager Overview Replication Manager is a backup tool that relies upon disk-based replicas as the foundation of the recovery operation. Replicas can be used for granular backup and restore of databases and file-systems, or for simplifying and automating data refreshes in development testing, quality assurance, and reporting cycles. Specifically, Replication Manager provides the following: • Eliminates scripting and makes replication technologies easy to use through an intuitive point-and-click user interface and configuration wizards • Enables fast reaction or recovery of data in the event of corrupt or lost information • Saves operations time by automating the mounting, dismounting, scheduling, and expiration of replicas Utilizing the Replication Manager application-aware technology in the backup and restore design provides the following benefits: Replication Manager design • Improves RTO for application recovery. Recovery from a VSS integrated replica, via Replication Manager, is faster than a restore from tape or even a restore from a disk-based backup. • Improves RPO application recovery. Multiple VSS integrated replicas via Replication Manager. Replication Manager is a robust enterprise-level application that can be used to perform a number of functions and provide great benefits, in conjunction with TimeFinder local replication technology and Microsoft applications. Replication Manager in a FAST VP environment is no different from one where the storage-tiering technology is not used. The design principles are essentially the same in both cases. Replication Manager theory and best practices Replication Manager can be used with Microsoft applications and TimeFinder snapshots to: • Create database application sets • Create replicas of the databases using VSS • Provide on-demand mount and dismount • Allow the database instances to be mounted to an alternate server and bring them with different names for test and refresh purposes • For Exchange, create replicas of active and passive databases, protected as part of a Database Availability Group (DAG) • Run consistency checks on the replicated data • Perform point-in-time or roll-forward restores to the production databases in the event of a corruption EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 45 The following are some of the best practice guidelines for designing a Replication Manager backup environment for Microsoft applications: • It is recommended to separate the database and logs onto different LUNs to allow for restore of the different components, individually. • Zone the Replication Manager mount hosts’ HBAs to different array ports, then the production Exchange Server’s HBAs for performance reasons. • Use EMC PowerPath on the mount host for optimal multipathing and load balancing, particularly when running consistency checks. • Do not share spindles between the production volumes and snapshot copies. Always dedicate spindles for the save pool for better performance. • If possible, use a physical mount even in a virtual environment as this will reduce the time it takes to mount the volumes. Exchange • In a DAG environment, it is good practice to take snapshots of the passive copy to lessen any potential performance impact. • It is recommended to not exceed four databases (eight LUNs) per Replication Manager application set. Beyond that, the possibility of VSS timeouts and checksum timeouts increases, and this can result in job failures. • In Exchange Server 2010, it is no longer a requirement to run consistency checks on the mounted snapshot copy. It is, however, recommended to do this once a week. SQL Server • System database data and log files should not be on any of the user database/log LUNs. • When planning to utilize the replica for testing/reporting purposes, mount the copy to an alternate host. SharePoint • Do not place the system database, tempdb and log files on the same LUN as the OSearch components. Replication Manager design layout The following components comprise the environment that is part of the Replication Manager design for Microsoft Exchange 2010 in this solution: • Symmetrix VMAX array running 5875 microcode. • TimeFinder snapshots for local protection. • The Exchange configuration that is being replicated includes three passive copies (six LUNs in total being snapped) per mailbox virtual machines. • Single stand-alone server to run Replication Manager server software, which also acts as the mount host. A general rule of thumb is to have one mount host for every three Exchange servers. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 46 • All of the database data and log LUNs are pass-through LUNs. Microsoft Exchange Microsoft Exchange Server 2010 in a two-DAG configuration Four Hyper-V servers hosting fourteen virtual machine mailbox servers TimeFinder snapshots were taken off passive database copies Microsoft SQL Server and SharePoint • Four Hyper-V servers hosting three SQL Server virtual machines and thirteen SharePoint virtual machines • TimeFinder snapshots SharePoint Server database and Office SharePoint Server Search service (Osearch) components for local protection, as well as the StoragePoint database and the external binary large objects (BLOBs) Note This is a guideline and will vary based on customer requirements. Figure 8. Replication Manager design The following are the high-level steps Replication Manager performs when taking a snapshot of the Exchange Server 2010 databases: • The Replication Manager jobs are configured to first check the event logs for events that are reporting any database corruption issues. They verify if circular logging is disabled and map the Exchange databases to the back-end storage volumes. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 47 • The Replication Manager job then takes a TimeFinder snapshot copy of the databases defined in the application set. This copy is an application-consistent VSS copy of the source databases. The snapshots are taken from the passive DAG copy. • Once the snapshot is taken, Replication Manager mounts the replica (snapshot volumes) into a specified directory on the Replication Manager mount server. • Once successfully mounted, Replication Manager can initiate a checksum process against the mounted volumes to ensure consistency. If this process is successful, Replication Manager truncates the Exchange logs. If it is not successful, the Replication Manager job fails and current transaction logs are not truncated until the next scheduled backup. This step is not required to be run every time a snapshot is taken but it is recommended that consistency check be run at least once a week. • After the process has completed, the snapshot volumes remain mounted on the mount server until the next replica rotation. This allows adequate time for the data to be archived to tape. In this solution the jobs were scheduled to run four times a day, every day, to provide an RPO of six hours. Figure 9. Replication Manager breakdown–Microsoft Exchange Figure 9 highlights the Replication Manager design for this solution. A total of six jobs were created. They were scheduled to run 30 minutes apart. There were several considerations taken into account to design the Replication Manager backup environment: • Three databases were included in every application set. Since the databases all reside on separate RDM LUNs, this layout allows performing a point-in-time EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 48 restore of databases and logs, as well as roll-forward restores of just the databases, if needed. Placing three databases per application set also helps reduce the number of Replication Manager jobs, which results in simplified management. • Each Replication Manager job is directly associated with an application set and configured to take a snapshot of three databases. The job is configured to take snapshots from the passive DAG copy. This design approach was adopted to reduce any potential performance impact on the active database copy. • The local replication technology used for this solution is TimeFinder snapshots. Microsoft recommends hosting multiple, full copies of Exchange databases on separate servers to provide local recoverability capability. In most environments with large mailboxes, Exchange Server 2010 is capacity driven more so than performance. For this reason, using snapshots provides huge space savings. TimeFinder snapshots are pointer-based copies of the source LUN that store only the changed tracks; hence they use minimum space on the VMAX array. By taking snapshots of the passive DAG, any potential performance impact on the production Exchange databases is minimized. Finally, Replication Manager integrates well with the TimeFinder/snapshot technology and provides for instant restore capabilities, both point-in-time and roll-forward. The jobs were scheduled, using Replication Manager, to minimize administrative tasks and automate the solution. The snapshot jobs were scheduled to run four times a day. This lowered the RPO for recovery to three hours. This also worked well in both roll-forward and point-in-time restores. Since the databases and logs were located on separate LUNs, it also provided for a roll-forward restore capability and no data loss solution when the logs were intact. Rapid restore using Replication Manager The various scenarios and procedures for recovering Microsoft Exchange 2010 with Replication Manager are detailed in the EMC Virtual Infrastructure for Microsoft Exchange 2010 Enabled for Symmetrix VMAX and Replication Manager—Proven Solution Guide. Refer to this guide for step-by-step procedures. Microsoft SQL Server The following are the high-level steps performed by Replication Manager when taking a snapshot of the SQL Server 2008 databases: • One Replication Manager job is configured for each database. • The Replication Manager job takes a TimeFinder snapshot copy of the databases defined in the application set. This copy is an application-consistent VSS copy of the source databases. • The Replication Manager job mounts the databases on the mount host for tape backup (these snapshots are unmounted when the next replication starts). EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 49 In this solution, the jobs were scheduled to run four times a day, every day, to provide an RPO of six hours. Figure 10. Replication Manager breakdown–Microsoft SQL Server A total of three jobs were created and scheduled to run at the same time. The following represents several considerations for designing the Replication Manager backup environment: • One database is included in each application set. Since the databases all reside on separate pass-through LUNs, this layout enables a point-in-time restore of the databases and logs, as well as restores of only the databases, so that the database can be recovered to the latest point-in-time with the appropriate logs. • Each Replication Manager job is directly associated with an application set and configured to take a snapshot of that database. • The local replication technology used for this solution is TimeFinder snapshots. • The jobs were scheduled using Replication Manager to minimize administrative tasks and automate the solution. The snapshot jobs were scheduled to run four times a day to provide an RPO of six hours. Since the databases and logs were located on separate LUNs, it also provided for a roll-forward restore capability and no data loss solution when the logs were intact. Rapid restore using Replication Manager The same procedure for leveraging Replication Manager to restore the SQL Server database can be used in both the physical and virtual environment. One important advantage that Replication Manager provides is for test/refresh or an alternate form of recovery. Replication Manager enables the replica copy of the production database to be mounted on an alternate server, and brings up or restores the database with a different instance and/or database name. Customers can run any analysis or reporting application on the replica, thus offloading these activities from the production server. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 50 1. Select the replica to be mounted, as shown in Figure 11. Figure 11. 7. Replica to mount Specify the mount host, the mount path, and recovery options, as shown in Figure 12. Note To recover the database, the mount host must have SQL Server installed. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 51 Figure 12. Mount options When performing a restore activity, it is necessary to understand that the source copy will be overwritten. The tool allows for restoring the database to a point-in-time copy should the production database encounter any form of corruption. The restore process is done at a LUN level (since the tool leverages the underlying storage replication technology) and can be done at the filegroup level, as long as the specific file group is on a separate LUN with other filegroups in the replica. The process is simple: select the objects in the copy to restore and click OK. For the detailed procedure, refer to SQL Server restore using Replication Manager in the Appendix. Microsoft SharePoint The following are the high-level steps performed by Replication Manager when taking a snapshot of the SharePoint 2010 farm: • One Replication Manager application set is configured for the SharePoint farm. • One Replication Manager application set is configured for the Metalogix StoragePoint database. • One Replication Manager application set is configured for the external BLOB volumes. • The Replication Manager job then takes a TimeFinder snapshot copy of the SharePoint databases and its other components, as defined in each application set. This copy is an application-consistent VSS copy of the source databases. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 52 • The Replication Manager job will mount the databases on the Mount host for tape backup (these snapshots will be unmounted when the next replication starts). The jobs were scheduled to run four times a day, every day, to provide an RPO of six hours. Figure 13. RM Replication Breakdown – Microsoft SharePoint A total of three jobs were created and scheduled to run one after the other. The following considerations were taken when designing the Replication Manager backup environment: • The entire SharePoint farm is configured as one application set. Replication Manager can recognize a SharePoint component through any member SharePoint Server with VSS enabled. • Since all of the databases reside on separate pass-through LUNs, this layout allows performing a point-in-time restore of databases and logs or any SharePoint search component, as required. • Each Replication Manager job is directly associated with an application set and configured to take a snapshot of that database. • The local replication technology used for this solution is TimeFinder snapshots. • The three jobs for the three different application sets are configured to run one after another (linked jobs). The SharePoint farm job runs first, and when completed, triggers the StoragePoint database job. Finally, the external BLOB job is executed as a linked job. The sequence of the jobs is essential for eliminating the possibilities of creating orphaned BLOBs for restore. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 53 Figure 14. StoragePoint database job scheduled to run after SharePoint farm job finishes EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 54 Figure 15. External BLOB job scheduled to run after StoragePoint database job finishes Rapid Restore using Replication Manager Replication Manager allows customers to mount the replica on an alternate mount host and have access to the files that are backed up. If there is a need for one file, mount the replica or part of it, and restore that particular file. Any of the SharePoint components can be restored, as long as the replica taken is consistent in the farm. For example, if one of the content databases is corrupted but the transaction logs are still available, the content database could be restored separately. The corresponding logs can then be applied to the database, thus restoring the copy to its current state, and making the metadata available for the External BLOB. Execute the following steps when restoring the SharePoint content database: 1. Stop the SPTimer service on all SharePoint Servers. 8. Choose the content database to restore. Depending on your needs, there is an option to select to backup the transactions logs prior to restore, so that you can apply them to bring the database to the desired point in time. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 55 The procedure for recovering the SharePoint farm, when there is a need to restore the External BLOBs, is slightly different. During replication, the External BLOB is set to be replicated after the SharePoint content database and the storage point database are replicated. When restoring, the sequence needs to be reversed in this order: 1. Restore the External BLOB 9. Restore the StoragePoint database 10. Restore the content database Orphaned BLOBs do not affect SharePoint performance or make the content irretrievable. Orphaned BLOBs occur when a content database is restored and the corresponding BLOB store contains BLOBs that were added after the content database snapshot was taken. A BLOB garbage collector job is available from storage point to clean up the orphaned BLOBs. To perform the SharePoint restore operations, follow the, SharePoint Restore using Replication Manager procedure, in the Appendix. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 56 Performance testing and validation results The following section describes the approach and methodology used to validate this solution. To ensure that the design and guidance presented in this white paper deliver good performance, end-to-end validation testing was done on the entire infrastructure. The performance testing was conducted in four sections: Notes • End-to-end performance validation testing of the Exchange environment when leveraging FAST VP using the Microsoft LoadGen tool • End-to end performance validation testing of the entire environment when leveraging FAST VP, using the following tools: Microsoft Exchange–Microsoft LoadGen tool Microsoft SQL Server–TPC-E tool Microsoft SharePoint–Microsoft Visual Studio Team System (VSTS) tool • Replication Manager backup and restore performance results for all of the applications • TimeFinder/snapshot performance and impact for all applications Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated. All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute. Methodology and tools The solution involved a combination of a number of components. Each component needs to perform well for the solution to be deemed successful. The two main focuses for the testing validation of the consolidated solution were: • Overall testing of all the applications under normal conditions. • Overall testing of all the applications with TimeFinder snapshots active. The snapshots were application-consistent snapshots, taken with Replication Manager. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 57 Apart from the above tests, there were some application-specific tests performed to gauge the health and performance of each application: • Microsoft Exchange • Database failure–simulating a database failure scenario Microsoft SQL Server and Microsoft SharePoint Live Migration and Quick Migration of the SQL Server All tests were run multiple times to ensure that the results were consistent. LoadGen Exchange Load Generator (LoadGen) is used for a full end-to-end assessment of the Exchange Server 2010 environment. LoadGen can be used to perform pre-deployment validation and stress testing tasks that introduce various workload types into a test (non-production) Exchange messaging system. This test simulates the delivery of multiple MAPI, Outlook Web access, IMAP, POP, and SMTP client messaging requests to an Exchange server. TPC-E The test environment is based on a Microsoft Bench craft TPC-E testing tool. It generates TPC-E-like workload. The testing environment composed of a set of transactional operations simulating the activity of a brokerage firm, such as managing customer accounts, executing customer trade orders, and other interactions with financial markets. VSTS VSTS was used to simulate the load on the SharePoint. A client load emulation tool based on KnowledgeLake was used to ensure that the SharePoint farm was operating at the optimal performance level. The testing environment was built with VSTS team test rig that consisted of a single controller and four agents on four different hosts (with one agent on the same host of the controller). Throughput is the number of operations (browse, search, or modify) that a SharePoint farm can perform each second. Ideally, the number of operations requested per second is lower than the number targeted for a given level of performance. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 58 Table 16 describes throughput targets for four levels of user load: Table 16. User load throughput targets User load Request rate Supported user Light 20 requests per hour. An active user will generate a request every 180 seconds. Each response per second of throughput supports 180 simultaneous users and 1,800 total users. Typical 36 requests per hour. An active user will generate a request every 100 seconds. Each response per second of throughput supports 100 simultaneous users and 1,000 total users. Heavy 60 requests per hour. An active user will generate a request every 60 seconds. Each response per second of throughput supports 60 simultaneous users and 600 total users. Extreme 120 requests per hour. An active user will generate a request every 30 seconds. Each response per second of throughput supports 30 simultaneous users and 300 total users. In this solution, all users adhered to a Microsoft heavy user profile, which specifies 60 requests per hour. A think time of 0 percent was applied to all tests. “0 percent think time” is the elimination of typical user decision-making time when browsing, searching, or modifying, using Office SharePoint Server. Every user request is completed from start to finish without a pause, which generates a continuous workload on the system. Table 17 describes the required response time for different tests. Table 17. Required response time Test type Details Percentage in the overall test Required response time Browse User Browse 80 <3 seconds Search Unique value search 10 <3 seconds Modify Browse and metadata modify 10 <3 seconds Note LoadGen should only be used in a test lab configuration and in nonproduction Exchange environments. For more information on LoadGen visit: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=cf46 4be7-7e52-48cd-b852-ccfc915b29ef EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 59 Data collection points To validate the complete health of the solution and identify any bottlenecks, results were collected and analyzed from a number of places. The performance results are broken down into the following four areas: Microsoft Exchange • Hyper-V and virtual machine performance • Exchange Server and Exchange client performance • Storage and SAN performance • Replication Manager backup and restore performance Microsoft SQL Server • Hyper-V and virtual machine performance • SQL Server and client performance • Storage and SAN performance • Replication Manager backup and restore performance Microsoft SharePoint Microsoft Exchange test results with FAST VP • Hyper-V and virtual machine performance • SharePoint Server client performance • Storage and SAN performance • Replication Manager backup and restore performance Validation results with Jetstress In this solution, Jetstress version 14.01.225.017 was used to simulate an I/O profile of 0.15 IOPS per user. Jetstress was run for seven building blocks (21,000 users). The building blocks were validated using 2-hour performance tests. Jetstress was run only with SATA and FC tiers, as the Exchange sizing was done only for those two tiers. The latency was a little higher for a couple of databases than the rest, since FAST VP had moved only certain parts of the databases that exceeded the performance threshold to the faster tier. This was due to the compliance settings that put some of the databases on the faster FC tier and some on the slower SATA tier. Table 18 shows I/Os and the average latency across all servers, which reflects the aggregate performance of this solution. Table 18. Average performance per mailbox virtual machine 3,000 users Database I/O Value DB Disk Transfers/Sec 535 (target 450) DB Disk Read/Sec 310 DB Disk Write/Sec 225 Average DB Disk Read Latency (ms) 15.37 ms EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 60 Database I/O Value Average DB Disk Write Latency (ms) 5.16 ms Transaction Log I/O Log Disk Writes/sec 211 Average Log Disk Write Latency (ms) 1.66 BDM IOPS 179 Environment validation with LoadGen The LoadGen tool was used to simulate MAPI workload against the entire Exchange infrastructure. LoadGen testing is necessary to determine how each Exchange component performs under a real, close-to-production user load. LoadGen requires full deployment of the Exchange environment for validation testing. All LoadGen validation testing should be performed in an isolated lab environment where there is no connectivity to the production data. LoadGen generates users and the workloads against the entire Exchange environment, including Exchange Server virtual machine, network, and storage components. LoadGen simulates the entire mail flow, helping to determine any bottlenecks in the solution. It is the only tool that helps to determine the CPU and memory resources that are necessary to sustain the load for which the Exchange environment was designed. In this solution, Exchange Server Load Generator 2010 (LoadGen) was used to simulate Outlook 2007 online mode mailboxes with the following characteristics: • The Outlook 2007 profile with 150 messages produced 181 Outlook tasks per user, per day, and was used for all 21,000 users. • Each mailbox is 1,024 MB in size. • Fourteen LoadGen client machines were used in the testing. Each producing load for 1,500 very heavy users. Figure 16. Note LoadGen Simulation LoadGen was run to simulate a normal work day’s operation and change. The simulated workday duration is set to nine hours. During testing the following user load was produced: • The Outlook 2007 150 message profile generated a total of 9.5 million tasks per day across all eight servers. • The load produced an average of 11.8 GB of logs per database over 24 hours. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 61 • 30 logs a day were produced per user, which is a good measure to compare to customer environments. Test 1: End-to-end validation with FAST VP for Exchange under normal conditions Objectives The objective of this test was to validate the entire solution under normal operating conditions for a normal work day with FAST VP storage tiering enabled. All aspects of the solution were evaluated, including the Hyper-V root server and virtual machines performance, Exchange server and client experience, and how the VMAX array handled the load. Only a small number of Flash drives were introduced in the FAST VP policy to account for any unanticipated spikes and any hot spots in the workload. Configuration In this test, all Exchange databases and virtual machines were placed under normal operating conditions; three active and four passive copies per Exchange virtual machine. The LoadGen tool was configured to simulate a very heavy load with a 9hour work day. Performance results and analysis The Hyper-V and the virtual machine CPU utilization counters were analyzed. Figure 17 and Figure 18 represent the CPU utilization on the mailbox virtual machines during the daytime hours of the test. During this test, LoadGen ran at a very heavy load for the first nine hours. • The mailbox virtual machines averaged approximately 26 percent CPU utilization with spikes up to 46 percent. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 62 Figure 17. CPU utilization of Mailbox virtual machine Test 2: End-to-end validation with FAST VP for Exchange in a failover condition Objectives The objective of this test was to validate the entire solution in a failover operating condition when all of the databases are serviced by seven servers with FAST VP storage tiering enabled. As was the case in the previous test, all aspects of the solution were evaluated, including the Hyper-V root server and virtual machine performance, Exchange server and client experience, as well as how the VMAX array handled the load. Configuration In this test, all Exchange databases and virtual machines were active on seven mailbox servers. The LoadGen tool was configured to simulate very heavy load with a 9-hour work day. Performance results and analysis Hyper-V and virtual machine performance results The Hyper-V and the virtual machine CPU utilization counters were analyzed. Figure 18 represents the CPU utilization on the Hyper-V servers and the mailbox virtual machines during the daytime hours of the test. During this test, LoadGen ran at a very heavy load for the first nine hours. • The mailbox virtual machines averaged approximately 42 percent CPU utilization with spikes up to 85 percent. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 63 Figure 18. Virtual machine CPU utilization Overall, in both cases, the Hyper-V virtual environment performed well under the load and had some additional headroom to handle spikes. Exchange client results Mailbox server response times for client requests are recorded to determine the amount of time it takes the mailbox server to respond to a client request and gauge the overall client experience. The response time average per request should not exceed 20 milliseconds. The following performance monitor counter on the mailbox server was observed to monitor response time: • MSExchangeIS/RPC Averaged Latency The following performance monitor counters were tracked on the mailbox server to monitor the message sent and delivered rates: • MSExchangeIS Mailbox (_Total)/Messages Sent/sec • MSExchangeIS Mailbox (_Total)/Messages Delivered/sec From an Exchange client perspective, each test is validated by comparing the results of select performance counters to the allowed Microsoft-specified criteria. Performance counter data is collected at 10-second intervals for the duration of each test run. The results of the first and last hours are discarded, and the results are averaged over the remaining test duration. Table 19 shows the Exchange client results. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 64 Table 19. Exchange client results Counter Target Test 1 results Test 2 results Avg Processor(_Total)\% Processor Time <70% 29.21% 50.54% Avg Replay Queue Length <20 0.461 0 Avg Copy Queue Length <12 0.197 0 MSExchange Database\I/O Database Reads Average Latency <20 ms 9.14 ms 11.22 ms MSExchange Database\I/O Database Writes Average Latency <20 ms 2.522 ms 2.82 ms Avg MSExchangeIS\RPC Requests <70 1.035 2.619 Avg MSExchangeIS\RPC Averaged Latency <10 ms 2.063 ms 2.346 ms From an Exchange client perspective, all performance counters were well below the target and in both cases indicate a healthy client experience. For additional information about monitoring Exchange Server 2010 performance and other key performance counters, visit Performance and Scalability Counters and Thresholds on Microsoft’s TechNet site at: http://technet.microsoft.com/enus/library/dd335215.aspx One important question which often arises is the impact on active and passive databases when under full load. I/O comparisons were performed between the active and passive copies under full load. Table 20 shows that there is not much difference between the two. The comparisons shown below are based on I/O serviced by each database LUN. Table 20. Consolidated applications test results with FAST VP I/O comparisons under full load Counter Active copy Passive copy Total I/O/sec 56.35 46.70 Reads/sec 39.24 26.27 Write/sec 17.11 20.43 Environment validation for all Microsoft applications Once the Microsoft Exchange environment was tested, it was dynamically scaled to include two additional database environments–Microsoft SQL Server 2008 and Microsoft SharePoint. This mixed Microsoft workload was deployed in a shared disk infrastructure on the VMAX. The spindles were shared between the three applications. FAST VP automated storage tiering technology was leveraged to provide an optimal blend of cost and performance to the environment. The following testing EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 65 tools were used to run tests against each database and the performance was also measured: • Microsoft Exchange–Exchange LoadGen tool • Microsoft SQL Server–SQL TPC-E tool • Microsoft SharePoint–Microsoft Visual Studio Team System (VSTS) tool Objectives The objective of the testing was to validate the mixed Microsoft workload under normal operating conditions for a normal work day, with FAST VP storage tiering enabled, and to do the same with snapshots active. Performance for the different solution elements was measured and evaluated for all three applications. The applications, as well as storage and server performance, are highlighted below for each of the applications, separately. As explained in the EMC FAST VP section, the FAST policies were set differently for each application to cater to the workload patterns. Flash was only used to handle spikes for Exchange. The Flash tier was not part of the FAST policy for SharePoint since it is a low I/O application. The ease of management for the FAST VP environment for all of the mixed workload was highlighted during the testing process. Configuration Two types of tests were executed against the mixed Microsoft workload in this solution: Microsoft Exchange • Test 1 - (Normal operations without snapshots) running heavy load on all three applications – Exchange, SQL and SharePoint • Test 2 – (Normal operations with snapshots) running heavy load on all three applications – Exchange, SQL and SharePoint In both of these tests, all Exchange databases and virtual machines were placed under normal operating conditions; three active and three passive copies per Exchange virtual machine. The LoadGen tool was used to simulate a very heavy workload for a 9-hour normal day. Hyper-V and virtual machines performance results As with the previous test scenarios, the Hyper-V and the virtual machine CPU utilization counters were analyzed. Figure 19 represents the CPU utilization on the mailbox virtual machines, during the daytime hours, under normal load, with snapshots active. During this test, LoadGen ran at a very heavy load for the nine hours. • The mailbox virtual machine CPU averaged approximately 32 percent, peaking at 71 percent, thus indicating a well-performing server with enough room for growth. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 66 Figure 19. CPU utilization on the mailbox virtual machines Exchange client results From an Exchange client perspective, as with the Exchange-only testing scenario, each test is validated by comparing the results of select performance counters to the allowed Microsoft-specified criteria. Performance counter data was collected at 10-second intervals for the duration of each test run. The results of the first and last hours are discarded. The results are averaged over the remaining duration of the test. As can be seen from Table 21, the Exchange client performed very well and all performance counters were well below the target. There was not much difference in the performance with and without snapshots. This indicates a healthy client experience in both cases. Table 21. Exchange client results Counter Target Test 1 results Test 2 results Avg Processor(_Total)\% Processor Time <70% 26% 28.09% Avg Replay Queue Length <20 0.032 0.122 Avg Copy Queue Length <12 0.001 0.002 EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 67 Counter Target Test 1 results Test 2 results MSExchange Database\I/O Database Reads Average Latency <20 ms 9.50 ms 10.07 ms MSExchange Database\I/O Database Writes Average Latency <20 ms 2.77 ms 2.961 ms Avg MSExchangeIS\RPC Requests <70 0.987 1.298 Avg MSExchangeIS\RPC Averaged Latency <10 ms 1.94 2.267 Microsoft SQL Server The average SQL Server processed over 1,000 user requests per second over the duration of the 8-hour duration test. Microsoft’s acceptable user response times were comfortably met at all times. Note This 8-hour test was configured with realistic user values and user load concurrency counts. The aim of this test was not to run the SQL Server at maximum performance capacity, but to run the server at capacity with acceptable headroom for additional operations. The TPC-E-like user workload translated to the SQL Server CPU utilization of 70.9 percent on SQL01, 29 percent on SQL02, and two percent on SQL03, The activity level was very low, and memory and I/O subsystem suffered no pressure and had ample headroom. Average test response time stayed well within acceptable parameters. Throughout the duration of the 8-hour load tests, SQL Server performance remained stable. Due to checksum activity, some peaks and dips were recorded. Performance results and analysis Hyper-V and virtual machine performance results The Hyper-V and the virtual machine CPU utilization counters were analyzed. Figure 20 represents the CPU utilization on the SQL Server, and SharePoint server virtual machines during the daytime hours of the test. During this test, TPC-E and VSTS tests ran at a very heavy load for the eight hours. • The SQL Server virtual machines averaged approximately 27.5 percent CPU utilization with spikes up to 43.9 percent. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 68 Figure 20. CPU utilization on the SQL Server, and SharePoint server The SQL Server virtual machine CPU configuration is sufficient to handle the load and any additional spikes, if necessary. The Hyper-V servers were designed to host the virtual machines, with enough room to support any additional virtual machines if any failover occurs. SQL Server results SQL Server response times for client requests are recorded to determine the amount of time it takes the SQL Server to respond to a client request and gauge the overall client experience. The response time average per request should not exceed 20 milliseconds. The following performance counter on the SQL Server client was observed to monitor response time: • Transaction/sec The following performance monitor counters were tracked on the SQL Server to monitor the disk IO activities: • Avg Disk Reads/sec • Avg Disk Writes/sec • Disk Transfers/sec From a SQL Server client perspective, each test is validated by comparing the results of select performance counters to the allowed Microsoft-specified criteria. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 69 Performance counter data is collected at 10-second intervals for the duration of each test run. SQL Server client results • Test 1 - (Normal operations without snapshots) running heavy load on all three applications–Exchange, SQL and SharePoint • Test 2 – (Normal operations with snapshots) running heavy load on all three applications–Exchange, SQL and SharePoint Table 22 shows that all performance counters were within the targeted range and indicated a healthy client experience in both cases. From the I/O perspective, all of the database LUNS have very similar performance with/without the snapshot present. The performance impact of keeping a snapshot is most visible with the busiest database of the three, SQL01, which has the heaviest load of read and modification, and least visible for the database SQL03, which has the least heavy load. Table 22. Counter Target SQL Server client results Test 1 results (without TF/snapshot) Test 2 results (with TF/snapshot) SQL01/Ho t SQL02/War m SQL03/Col d SQL01/Ho t SQL02/War m SQL03/Col d Avg Disk Reads / sec <20 ms 6 ms 9 ms 15 ms 14 ms 8.76 ms 17 ms Avg Disk Writes / sec <20 ms 2 ms 4 ms 2 ms 2 ms 2 ms 2 ms 5671 2997 535 5222 2358 511 61.2 27.5 1.7 68.3 29.3 3 687 441 33 335 302 33 Disk Transfers / sec Processor Time Transaction /sec <70% SharePoint Test approach The test client used VSTS to test the FAST Search server for SharePoint 2010 performance. The test application mimics the operations of the SharePoint farm access. During validation, a Microsoft heavy user load profile was used to determine the maximum user count that the Microsoft SharePoint Server 2010 farm could sustain, while ensuring that average response times remained within acceptable limits. Microsoft standards state that a heavy user performs 60 requests per hour, that is, a request every 60 seconds. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 70 Microsoft recommends the response times from the WFE are categorized in the following way: http://technet.microsoft.com/en-us/library/cc261795(office.12).aspx: • Slow (3-5 seconds): User response times can slow to this rate without issue • Recommended (1-2 seconds): The average user response time target • Fast (<1 second): For organizations whose businesses demand speed User load profile A common mix of user profiles was used to emulate different types of business organizations. For example, some organizations are browse-intensive, while others are search- and/or modify-intensive. Table 23 shows the user load profile for this test. Table 23. User load profile User profile Percentage Browse/Search/Modify 80/10/10 All tests were run from a load controller host that spread the load evenly across each of the four load agent hosts. The load controller host also collected performance metrics for analysis from all of the load agents and hosts in the farm. Test methodology design and data preparation To prepare the SharePoint 2010 web farm content and file share content, the sample documents are downloaded from various sources. The documents were modified with PowerShell scripts and C# Win form application. Each sample generated over 400 copies in this fashion. The inserted words and timestamps were then used as search keywords for the SharePoint search performance test. This ensures the search result for the keyword is unique, even though most parts of the generated documents are the same as the original sample file. SharePoint test results summary SharePoint 2010 and Hyper-V performance data was logged for analysis during an 8hour test run lifecycle. This data presents an account of results from VSTS, which generates continuous workload (Browse/Search/Modify) to the WFEs of the SharePoint 2010 farm , while simultaneously consolidating the SQL Server 2008 R2 and Exchange 2010 DAG workload on the same Hyper-V data center. In the consolidation test, the average SharePoint farm processed a heavy user load over the duration of the 8-hour duration test. Microsoft’s acceptable user response times were comfortably met at all times. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 71 Note This 8-hour run test was configured with realistic user values and user load concurrency counts. The aim of this test was not to run the farm at maximum performance capacity, but to run the farm at capacity with acceptable headroom for additional operations. The SharePoint user workload translated to a back-end SQL Server CPU utilization of 22.3 percent on SQL01, and 11.4 percent on SQL02. Memory and I/O subsystem suffered no pressure and had ample headroom. Average test response time stayed well within acceptable parameters. Throughout the duration of the 8-hour load tests, SQL Server performance remained stable. Due to checksum activity, some peaks and dips were recorded. Virtual machine performance results The Hyper-V and the virtual machine CPU utilization counters were analyzed. Figure 21 and Figure 22 represent the CPU utilization on the Hyper-V servers, the SharePoint server, and the SharePoint Server virtual machines during the daytime hours of the test. During this test, TPC-E and VSTS tests ran at a very heavy load for the eight hours. • The SharePoint farm virtual machines serving different server roles have very different CPU usage patterns: Figure 21. The WFE server can experience a very heavy load on the CPUs. As shown in Figure 21, an average of 89.1 percent CPU utilization with spikes up to 97.6 percent was recorded. WFE CPU usage EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 72 Figure 22. The CPU utilization on the SharePoint SQL Server is not heavy. As shown in Figure 22, the average CPU utilization is approximately 22.3 percent with spikes up to 41.4 percent. SharePoint SQL Server CPU usage The virtual machines with different roles were distributed evenly across all of the Hyper-V root servers to ensure good performance. The SharePoint Server virtual machine CPU configuration is sufficient to handle the load and any additional spikes, if necessary. The Hyper-V CPU resource is also sufficient to support the virtual machines with enough room to support any additional virtual machines, if any failover occurs. SharePoint test results SharePoint server response times for client requests are recorded to determine the amount of time it takes the SharePoint farm to respond to a client request, and gauge the overall client experience. The response time average per request should not exceed 20 milliseconds. The following performance counter on the VSTS test client was observed to monitor response time: • Average test time (sec) • Average response time (sec) The following performance monitor counters were tracked on the SharePoint farm and the SQL Server to monitor the disk I/O activities: • Avg Disk Reads/sec • Avg Disk Writes/sec From a SharePoint client perspective, each test is validated by comparing the results of select performance counters to the Microsoft-specified criteria. Performance counter data is collected at 10-second intervals for the duration of each test run, and the results of the first and last hours are discarded. Results are averaged over the remaining duration of the test. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 73 The following tests were performed to measure the application performance: • Test 1 – (Normal operations without snapshots) running heavy load on all three applications – Exchange, SQL and SharePoint • Test 2 – (Normal operations with snapshots)running heavy load on all three applications – Exchange, SQL, and SharePoint Table 24 shows that all performance counters were within the targeted range and indicated a healthy client experience in both cases. From the I/O perspective, all of the database LUNS have very similar performance with or without the snapshot present. The performance impact of keeping a snapshot is most visible with modify tests because the snapshot uses a copy-on-first-write mechanism. Modify tests write to the source LUN and trigger the copy to snapshot, thus taking longer than the tests without a snapshot. Counter Avg Test time (sec) Table 24. SharePoint client results Target Test 1 (without TF/snapshot) <3 Avg response time (sec) Search Browser Modify Search Browser Modify 0.34 0.81 1.09 0.31 1.02 1.42 0.047 Avg Processor (_Total)\% Processor Time <70% SQL server Database\I /O Database Reads Average Latency <20 SQL server Database\I /O Database Writes Average Latency <20 Test 2 (with TF/snapshot) 0.11 SQL01 22.3% SQL02 11.4% SQL01 SQL02 12.9% 46.40 content db Crawl db content db 5 6 9 4 6 3 Crawl db 7 7 EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 74 Metalogix StoragePoint BLOB externalization overview Metalogix StoragePoint test summary Metalogix StoragePoint enables SharePoint users to store unstructured SharePoint data or BLOBs outside of SQL Server content databases on virtual storage, including SAN, NAS, and cloud platforms. Organizations can create a more scalable and betterperforming SharePoint environment and reduce overall storage costs by moving content to less-expensive storage. StoragePoint storage profiles determine when content is off loaded and are configured by the SharePoint Web application, content database, or site collection. For content within existing SharePoint environments, StoragePoint ships with an externalization timer job that can be scheduled to manage the off-loading process. BLOB externalization results Before externalization In the test environment, 16 content databases were populated with 1,252,244 documents aggregating to 3,094 GB of data. Each content database contained up to 200 GB of data. The content databases in SharePoint stored files, metadata (information about the files), context information (such as file location and security information), and other structured relational data in its tables. During externalization The externalization process is scheduled and run by a timer job. The observed result is that one server can only run one profile job, while multiple servers can run externalization timer jobs together against one content source. The estimated duration of one 200 GB content database is approximately 10 hours. A fine-tuned externalization plan can ensure that the web servers can be fully utilized to externalize the BLOBs as soon as possible. Externalization will not stop the web operations, such as Browse, Search, and Modify. After a storage profile is defined and activated, all new uploaded documents are stored in the BLOB store defined for that profile, while the BLOBs already in the content databases are migrated by the externalization process. Figure 23 shows the impact of the externalization process of a BLOB to the running process of the VSTS test code. The sudden dip of the request per seconds (RPS) occurred when the four profile jobs ran on the WFE servers. The jobs were rescheduled to the query servers and the profile jobs started the run after one hour of the VSTS test code. During the process, a performance degradation of 19 percent was observed in the four concurrent user profile tests. Thus, externalization should be scheduled to run during off hours to ensure customers are not affected by these operations. Note Externalization is not carried out during peaks hours, so users are not affected by these operations. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 75 Figure 23. Impact of externalizing BLOB Figure 24 shows the StoragePoint job status. Users may Abort or Suspend the current running jobs, and can set the thread count, dynamically. Figure 24. StoragePoint job status Table 25 shows the CPU utilization of the four query servers using default four thread counts on each externalization component, with one component running on one profile to perform the externalizing process concurrently. Table 25. CPU utilization of query servers Externalization component Average CPU UT (%) Max CPU UT (%) Query Server 1 80.11% 97.34% Query Server 2 80.02% 97.29% Query Server 3 80.39% 96.64% Query Server 4 81.73% 90.39% Table 26 shows the performance statistics of the externalization device. It indicates that the IOPS requirement for this device is 231, therefore the SATA device with high EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 76 capacity and fair performance can fully support the externalized device performance requirement. Table 26. Externalization LUN performance Performance statistics of externalization devices IOPS (read) IOPS (write) IOPS (total) IOPS (max) Read latency (ms) Write latency (ms) Averag e latency (ms) Averag e read size Averag e write size 12 219 231 723 19 7 8 52k 110k Table 27 shows the performance impact of the RPS for VSTS test code during the fully externalized process. Table 27. Performance impact of externalization process Profile Requests per second (RPS) Concurrency Maximum user capacity Average user response time in seconds (browse/search/modify) Before externalization 80/10/10 53* 10% 31,800 0.83/0.23/1.01 During externalization 80/10/10 43* 10% 25,800 1/0.26/2.95 Note The test is performed in a non-consolidation test scenario; therefore the test result RPS is higher than the result of the consolidation test, since the virtual machines of the SharePoint farm can fully leverage the host and storage resources. After setting up the storage profiles in the StoragePoint 3.0 provider, it took 35 hours to complete the process of externalizing BLOBs from 16 content databases with a total of 3,094 GB data. After externalization Figure 25 summarizes the results of the user profile test result through the externalization process: Figure 25. User profile test result of externalization process After externalization, using StoragePoint 3.0 provider, all of the BLOBs inside the SQL content databases were offloaded to the VMAX file share, except for the metadata, EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 77 context information, other structured data, and reference IDs of BLOBs in the content databases. Approximately 2921.69 GB (94.41 percent content database data) was offloaded to the BLOB store after completing the externalization process, as shown in Table 28. Table 28. Space saving by externalizing BLOBs Content size before externalization (MB) Content size after externalization (MB) Space saved percentage (%) 3,094,600 172,909 94.41% RBS externalization best practices • The remote BLOB storage (RBS) externalization process is CPU intensive. Two CPU machines can achieve a near-double externalization rate compared to one CPU machine with the same configuration. • One or two farm servers containing web application can externalize one content database. They can be scheduled on different start times, but finish at the same time. • After externalization is finished, each 200 GB content database has approximately 5-30 GB metadata left in the content databases. The available space can be released from the content database through the SQL Server database shrink operations. Externalization does not interrupt the normal operations on the SharePoint farm, such as Browse, Modify, and Search. It can impact the performance slightly, especially when running the job on the WFE servers. Note If the WFE servers have high workloads to serve the web requests, DO NOT run the externalization job on the WFE server. Impact of TimeFinder/snapshot TimeFinder/snapshot uses a process called copy-on-first-write (COFW) when handling writes to the production data, and when a snapshot is active on the production volume. Anytime the client attempts to write to the production volume, the original track is first copied to the save area, then the write is processed against the production volume. This process of pointers maintains the consistent, point-in-time copy of the data for the ongoing snapshot. Since SQL01 has the most active workload, the impact to the test result is the most visible. The read latency was more than doubled, and total transaction/sec was downgraded almost by half. To minimize the impact of storage replication, TimerFinder/Clone technology can be used. Once created, any activity on a clone will not impact the production LUN. While requiring a full size of the production storage, the clone is considered a more expensive replication technology. For any database with heavy transactions and frequent changes, keeping a snapshot for a long period of time will lose its advantage of using less storage, thus making clone replication a more suitable choice. Meanwhile, with the least active workload on SQL03, the read latency increased 10 percent and the impact to the transaction/sec is negligible. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 78 In the case of Exchange and SharePoint, the impact of snapshots was not as high. The impact to the average response time for the SharePoint 2010 farm operations was most visible on the modify operation during the test run–the average test time increased 30 percent. Since the snapshots were taken off the passive copy, the impact was negligible for Exchange. Storage and performance From a storage perspective, the array and thin pools performed well. During the testing, the disk pools reached 75 percent maximum utilization and averaged approximately 60 percent. It should be noted that when testing with Exchange Jetstress against the same configuration, with the same user profile, the disk utilization averaged approximately 80 percent. Figure 26, Figure 27, Figure 28, and Figure 29 show how the VMAX performed during normal load and with snapshots. It can be observed that the VMAX performance was well-rounded in both cases, and the utilization peaked slightly higher in the latter condition. The array disk heat map shows that while a number of the disks were reaching 70 percent utilization overall, the disk layout performed well. In Figure 26, Figure 27, Figure 28, and Figure 29, it can be observed how the different tiers performed. The Flash tier was designed to only handle the peaks for Exchange and SQL, and though it handled a large amount of I/O, these disks were approximately 40 percent utilized. This is because each Flash drive can service approximately 1,800 I/Os. As expected, the FC and SATA tiers were approximately 70 percent utilized. Figure 26. Normal condition (VMAX utilization) EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 79 Figure 27. Normal condition (VMAX disk heat map) Figure 28. With snapshots (VMAX utilization) EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 80 Figure 29. Snapshot sizing With snapshots (VMAX disk heat map) An important sizing question that often arises is how much snapshot space is required to protect the production data, particularly Microsoft Exchange. If there is too little snapshot space, you could end up losing the snapshot copy. If there is too much snapshot space, you could be wasting storage. To address the question, as part of testing, snapshot space requirements on the VMAX were measured over a period of 24 and 48 hours when LoadGen was running to help with sizing. Table 29 shows the snapshot space usage of a single snapshot over 6-hour increments when LoadGen was running. The total snapshot space for a single set of database and log LUNs was a little more than two percent a day, based on our testing results. The numbers shown in the snapshot sizing table can only act as a rough guideline to size the save pool for snapshots with Exchange. It is recommended to size the pool based on actual change rates. The results noted in Table 29 are based on: • Snapshots against 18 passive copies • Percent based on a fully provisioned database/log set of 1.2 TB • Total space being snapped is 50.4 TB EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 81 Table 29. Hyper-V HA migration VMAX snapshot usage with Exchange Server 2010 % of single DB and log size needed for snapshots Total DB and logs space (MB) Total snapshot space used (MB) Per single DB and logs (MB) Start 55,364,813 20,039 1,113 6 hours 55,364,813 230,538 12,087 0.96 12 hours 55,364,813 363,201 20,178 1.61 18 hours 55,364,813 444,556 24,698 1.96 24 hours 55,364,813 608,836 33,824 2.69 The solution design enables the SQL and SharePoint servers to move between the Hyper-V root servers. Essentially, this design makes a server available to customers, with the expectation that they will be able to access that server at all times, usually because some effort has been made to ensure continuous operation. In order to provide server-level resiliency, Hyper-V High Availability (HA) utilizes the Windows Server Failover Clustering framework. To configure high availability for the application, using the failover clustering feature offered by Hyper-V, follow the standard best practices that are applicable to any other application. • There must be at least two hosts in a virtual machine cluster • All hosts must be given a unique host name and configured with static IP addresses. When using DHCP, ensure that the IP addresses of the host(s) persist upon reboot. • All hosts must have access to the same management networks. There must be at least one management network in common among all hosts, and the best practice is to have at least two. • All of the application virtual machines must have access to the same virtual networks and clustered volumes so that they can run on any host in the cluster • All hosts in a HA cluster need to have DNS configured so that the short host name can be resolved to the appropriate IP address from any other host in the cluster As part of the HA testing, live migration was tested with and without a full TPC-E workload on the busiest SQL Server. Live migration from a SQL Server, running a full user load, took approximately 41 seconds longer than that without any load (that is, a 25 percent increase). The transactions running on a SQL virtual machine, during a live migration, were approximately 40-50 percent slower. The users continued to process the transactions during the migration. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 82 On the other hand, quick migration without load took four minutes and 39 seconds, with the disruption of services, making live migration a preferred choice for migration servers among the Hyper-V servers. Table 30 shows the results of the Hyper-V HA migration. Table 30. Hyper-V HA migration Server Duration Test scenario SQL Server 01 4:39 SQL quick migration SQL Server 01 2:41 SQL Live migration with light load SQL Server 01 3:22 SQ Live migration with heavy load EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 83 Conclusion Summary The virtualized Microsoft mixed-workload, building-block approach adopted in this solution provides a private cloud solution for customers who are looking for enterprise consolidation with the VMAX array. This solution offers automated performance management and enables scaling of the application environment in a cost-effective and flexible manner by combining EMC Virtual Provisioning technology and EMC FAST VP tiering technology for storage, EMC Replication Manager for backup and recovery, and Microsoft Hyper-V for server infrastructure and virtualization. Findings The solution presented in this white paper provides the following benefits: • With EMC FAST VP storage tiering technology, customers can automate the process of identifying data that can benefit from Flash drives or that can be kept on high–capacity, less-expensive SATA drives, without impacting performance. This automated performance management allows customers to optimize and prioritize business applications, allowing them to dynamically allocate resources per application needs within the array. • With EMC Virtual Provisioning, customers purchase only the storage needed for the initial requirement. As the databases or the user mailboxes grow, more storage can be seamlessly added with no effect on the applications, databases, or server performance. The only additional cost is the purchase of additional disk space. • Customers can achieve a 4:1 server consolidation ratio by incorporating Microsoft Hyper-V as the server virtualization platform. • With EMC Replication Manager, customers can gain a very small backup window, regardless of the database size, with little to no impact on the Microsoft application and database servers. A small percentage of the production space is required for the snapshot space compared to clones. During testing, with a heavy Exchange Server 2010 load, as little as two percent of production storage was required to protect the databases for a 24 hour period. • The automated and rapid recovery capabilities that Replication Manager and Symmetrix snapshots provide are unmatched. A typical database restore and recovery takes only minutes, regardless of the size of the database. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 84 Appendix This section details procedures for the SharePoint restore operations. SQL Server restore using Replication Manager During a restore operation, perform the following steps: 1. Select the replica restore option. 11. Select objects to be restored. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 85 12. Select the option to backup the transaction logs before restoring, to be able to recover to the latest point in time copy. SharePoint Restore using Replication Manager The following outlines the procedure to perform a SharePoint contentdb restore: 1. Stop SPTimer service on all SharePoint Servers. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 86 13. Choose content database. 14. Choose to backup the transaction logs before restoring and recover the database with No Recovery so that they can be applied to the database after recovery. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 87 15. Apply the transaction logs to bring the database to the point of time that is desired or to current state. This section focuses on the procedure to recover the SharePoint farm when there is a need to restore the External BLOBs. The external BLOB is set to be replicated after the SharePoint content database and the storage point database are replicated. When restoring, the sequence should be reversed. 1. Restore the external BLOB volume. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 88 16. Restore the StoragePoint database. 17. Select Recover the databases option and choose Recovery. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 89 18. Restore the SharePoint content databases. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 90 19. Select Recover the databases option and then select Recovery. Note To ensure consistency across the farm, the outlined restore sequence should be strictly followed. EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 91 References Product documentation For additional information, see the product documents listed below. • Symmetrix Virtual Provisioning Feature Sheet • EMC Virtual Infrastructure for Microsoft Exchange 2010 Enabled for Symmetrix VMAX and Replication Manager Proven Solution Guide EMC Automated Performance Optimization for Microsoft Applications EMC VMAX, FAST VP, and Microsoft Hyper-V 92