Download g HIGH PERFORMANCE SOLUTION FOR ORACLE 11 RAC WITH XTREMCACHE 2.0.1

White Paper
HIGH PERFORMANCE SOLUTION FOR
ORACLE 11g RAC WITH XTREMCACHE 2.0.1
ON VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat
Enterprise Linux, Oracle Database Enterprise Edition
 Optimum IOPS for OLTP workloads
 Sustained Average Response Time Less Than One Millisecond
EMC Solutions
Abstract
This solution demonstrates the benefits of deploying EMC® XtremCache™ and
VMAX® to increase IOPS and decrease latency for OLTP databases. It provides
scalability, high performance, and ease of use for mission-critical business
demands.
December 2013
Copyright © 2013 EMC Corporation. All Rights Reserved.
EMC believes the information in this publication is accurate as of its
publication date. The information is subject to change without notice.
The information in this publication is provided as is. EMC Corporation makes no
representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or
fitness for a particular purpose.
Use, copying, and distribution of any EMC software described in this
publication requires an applicable software license.
For the most up-to-date listing of EMC product names, see EMC Corporation
Trademarks on EMC.com.
All trademarks used herein are the property of their respective owners.
Part Number H11828.2
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
2
Table of contents
Executive summary............................................................................................................................... 5
The business challenge.................................................................................................................... 5
The technology solution: open, best-in-class components ............................................................... 5
Open standards benefits ............................................................................................................. 6
Operational advantages .............................................................................................................. 6
Solution overview ............................................................................................................................ 6
Key results and recommendations ................................................................................................... 6
Introduction.......................................................................................................................................... 8
Purpose ........................................................................................................................................... 8
Scope .............................................................................................................................................. 8
Audience ......................................................................................................................................... 8
Terminology ..................................................................................................................................... 9
Technology overview .......................................................................................................................... 10
Architecture layers ......................................................................................................................... 10
Architecture diagram ...................................................................................................................... 10
EMC Oracle Performance Solution: Server layer ................................................................................. 11
Server hardware ............................................................................................................................. 12
Server software .............................................................................................................................. 13
Red Hat Enterprise Linux ................................................................................................................ 13
Oracle Grid Infrastructure and Database 11g Release 2 .................................................................. 14
XtremSF and XtremCache deployment ............................................................................................ 14
XtremSF and XtremCache overview ................................................................................................ 14
EMC XtremCache configuration.................................................................................................. 15
EMC Oracle Performance Solution: Network layer .............................................................................. 16
EMC Oracle Performance Solution: Storage layer ............................................................................... 18
Introduction ................................................................................................................................... 18
Storage components ...................................................................................................................... 18
Storage hardware........................................................................................................................... 18
EMC Symmetrix VMAX 40K eight-engine configuration .................................................................. 19
Storage software ............................................................................................................................ 20
EMC Oracle Performance Solution: Oracle Database layer .................................................................. 21
Introduction ................................................................................................................................... 21
Storage virtual provisioning design ................................................................................................ 21
Drive type....................................................................................................................................... 21
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
3
ASM disk group configuration for OLTP database ........................................................................... 21
OLTP database and workload profile .............................................................................................. 22
EMC Oracle Performance Solution: Performance testing .................................................................... 23
Overview ........................................................................................................................................ 23
Test objective................................................................................................................................. 24
Query only test scenarios and methodology ................................................................................... 24
Read test results ............................................................................................................................ 24
Update only test scenarios and methodology ................................................................................. 26
Peak write test results .................................................................................................................... 27
Conclusion ......................................................................................................................................... 29
Summary ....................................................................................................................................... 29
Findings ......................................................................................................................................... 29
References.......................................................................................................................................... 30
EMC documentation ....................................................................................................................... 30
Oracle documentation.................................................................................................................... 30
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
4
Executive summary
The business
challenge
The technology
solution: open,
best-in-class
components
Customers require an open, scalable, tiered, highly available, and high performance
infrastructure to run their critical Oracle systems. IT organizations must strive for
better performance and increased efficiency in their Oracle infrastructure and Oracle
database and storage administration operations, including the following
requirements:

Reduce capital expenditures and operational expenditures by deploying an
open, non lock-in technology

Consolidate many Oracle databases (Oracle database versions 10gR1 to 11gR2
and Oracle 12c) and database workloads, including OLTP and Data Warehouse,
to maximize the efficiency of the data center infrastructure

Deliver maximum performance while effectively utilizing the existing arrays and
Oracle infrastructure

Maintain the highest performance levels and provide predictable performance
to deliver the quality of service required in these Oracle mixed workload
environments
The EMC Proven High Performance Solution for Oracle Real Application Clusters (RAC)
on EMC Symmetrix® VMAX® is an open architecture that incorporates open, best-inclass Intel servers with EMC server-side flash storage (XtremSF™) and EMC VMAX
storage arrays.
The solution uses optimal servers to balance performance, scalability, and Oracle
license costs. The use of Xtrem™ technologies XtremSF and XtremCache™ software in
the servers provides distinct performance and operational advantages over
equivalent systems that do not contain server-side flash technologies.
XtremCache does the following to accelerate an Oracle RAC environment:

Features an ultra-performance tier: Accelerates any application that benefits
from low-latency, high bandwidth physical read I/O

Hottest data resides on database server flash

Data is as close to the Oracle Database server CPU as any storage model
will allow

Cooperates with Oracle Clusterware: Oracle Clusterware is the final authority
on all node membership information in an Oracle RAC deployment

Has no awareness of database instances:

Has no concern for content of blocks of cached LUNs

Only XtremCache nodes can access LUNs cached by XtremCache

Does not impose a performance penalty on active transactions for cache
insertions or cache coherency
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
5

Offers optimized performance: VMAX arrays are relieved of read IOPS, leaving
more bandwidth for handling writes. VMAX FAST enables automatic data
placement as data goes from hot to cool usage.

Delivers the highest performance levels in the industry: This solution delivers
the highest performance for mixed workload Oracle environments. EMC Proven
Solutions for Oracle have demonstrated sustained metrics over 3.7 million IOPS
with latency of less than half a millisecond for OLTP workload, and Data
Warehouse workloads with sustained throughput of 32 GBs with a data load
rate of 21 TB/hour. Key results and recommendations on page 7 provides
details.
Open standards benefits
This solution is based on open standards. Advantages resulting from the open
standards commitment include the following:

Flexible adaptation to existing and future customer needs and open industry
standards

Lower capital investment and operational expense without vendor lock-in
Operational advantages
EMC open architecture not only supports different releases of Oracle Database
software (10g, 11g, and 12c), but it also provides the capability of running the
databases concurrently; that is, it supports database consolidation.
Open architecture and flexible adaptation means that application modification is not
required for database deployment for this solution, lessening the potential for
unforeseen impact to business operations and systemic data flow throughout the
enterprise.
Solution overview
The purpose of the solution is to build an EMC High Performance Solution for Oracle
on VMAX infrastructure, based on an open architecture, and demonstrate the
following capabilities of the infrastructure:

High performance and flexibility

Low operational costs

Reduced risk
This white paper validates the performance of the solution and provides guidelines to
build similar solutions. This document is not a comprehensive guide to every aspect
of this solution.
Key results and
recommendations
The core advantages of the EMC Proven High Performance Solution for Oracle RAC on
VMAX are as follows:

Delivers the highest performance for Oracle RAC workload environments. EMC
Proven High Performance Solution for Oracle RAC on VMAX has demonstrated
sustained metrics over 3.7 million IOPS with latency of less than ½ millisecond
for OLTP workload. This impressive performance is achieved by utilizing open
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
6
best-in-class components at the computer, network and storage layers. Details
are listed in Table 1:
Table 1.
IOPS test results with workload when XtremCache is enabled
Workload
type
Performance
statistics
One
node
Two
nodes
Four
nodes
Eight
nodes
Read only
workload
IOPS
457,136
962,155
1,914,963
3,765,176
Response time (ms)
0.74
0.69
0.68
0.75
UPDATE
transaction
workload
Aggregate IOPS
53,492
99,649
190,809
303,330
Write IOPS
26,950
50,110
95,932
153,276
Read IOPS
26,542
49,539
94,877
150,054
Redo throughput
(MB/s)
20
37
71
115
Note: Because EMC XtremCache is write through cache, for the UPDATE workload, the data
blocks that have been read into buffer cache are accelerated by EMC XtremCache.
Meanwhile, the dirty blocks that have been flushed by DBWR are passing through EMC
XtremCache and are directly written to the back-end VMAX array.

Uses the following EMC technology enablers in the reference architecture:

EMC VMAX 40K

XtremSF PCIe flash card

XtremCache caching software
This solution provides a foundation that can be scaled in a flexible, predictable, and
nearly linear way using additional server resources, including CPUs and memory, HBA
ports, and front-end ports, to provide higher IOPS and throughput based on the
configuration in this solution.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
7
Introduction
Purpose
The purpose of this white paper is to describe an EMC High Performance Solution for
Oracle on VMAX infrastructure based on an open architecture, and to demonstrate the
following capabilities of the infrastructure:

High performance and flexibility

Low operational costs

Reduced risk
This solution validates the performance of the solution and provides guidelines to
build similar solutions. This document is not a comprehensive guide to every aspect
of this solution.
Scope
Audience
This white paper serves the following purposes:

Introduces the key solution technologies

Describes the solution architecture and design

Describes the solution test scenarios and present the results of performance
testing

Identifies the key business benefits of the solution
This white paper is intended for chief information officers (CIOs), data center
directors, Oracle DBAs, storage administrators, system administrators, technical
managers, and any others involved in evaluating, acquiring, managing, operating, or
designing Oracle database environments.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
8
Terminology
Table 2 lists the terminology used in this solution paper.
Table 2.
Terminology
Term
Definition
AWR
Automatic Workload Repository
ASM
Automatic Storage Management
DML
Data Manipulation Language
IOPS
I/Os Per Second
LUN
Logical Unit Number
OLTP
Online Transaction Processing
PCIe
Peripheral Component Interconnect Express
PGA
Program Global Area
RAID
Redundant Array of Independent Disks
RAC
Real Application Clusters
SATA
Serial Advanced Technology Attachment
SLOB
Silly Little Oracle Benchmark
SGA
System Global Area
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
9
Technology overview
This section describes the architecture of the solution.
Architecture layers The solution provides an optimal cost-to-performance ratio for Oracle mission-critical
application environments. The architecture is composed of the following layers:

Architecture
diagram
Server–Eight Cisco UCS C240 M3 servers each with:

Two 700 GB EMC XtremSF SLC flash card (1400 GB total)

EMC XtremSF driver and firmware

Network–Two Cisco Director - MDS 9506 switches

Storage–EMC Symmetrix VMAX 40K storage system

Database–One eight-node Oracle RAC database is installed on the eight servers
Figure 1 depicts the EMC Proven High Performance Solution for Oracle RAC on VMAX.
Figure 1.
Solution architecture
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
10
EMC Oracle Performance Solution: Server layer
Comprising the server layer of the solution, eight Cisco UCS C240 M3 servers utilize a
total of 128 cores with 2.90 GHz E5-2690 processors, 2.56 TB RAM, and 11 TB
XtremSF flash PCIe cards. The Cisco UCS C240 M3 is an enterprise-class rack server
designed for performance and expandability. As part of the EMC Proven High
Performance Solution for Oracle RAC on VMAX, the server layer enables a highperforming, consolidated approach to an Oracle infrastructure, resulting in
deployment flexibility without the need for application modification. Features and
benefits include the following:

Intel Xeon processor E5-2600 product family for hardened protection for virtual
and cloud environments

Fully integrated quad-port gigabit Ethernet
Figure 2 shows one of the eight Cisco UCS C240 M3 rack servers used in the EMC
Proven High Performance Solution for Oracle RAC on VMAX solution.
Figure 2.
Cisco UCS C240 M3 rack server (1 of 8)
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
11
Server hardware
Table 3 describes the various hardware components of the EMC Proven High
Performance Solution for Oracle on the VMAX server layer.
Table 3.
Server hardware
Server
hardware
Cisco UCS
C240 M3
Quantity
Configuration
Description
8
2 x 8-Core Sandy-Bridge E52690 Processors – 512 GB
RAM - 4x 200 GB SSD server
Servers
Each server consists of the following components:
PCIEHHS7XXM
2
700 GB SLC PCIe card
EMC XtremSF
UCSC-C240M3S
1
UCS C240 M3 SFF w/o CPU
mem HD PCIe with rail kit
expdr
Server housing
UCS-CPU-E52690
16
2.90 GHz E5-2690/130W
4C/10 MB Cache/DDR3 1600
MHz
CPUs
UCS-ML1X324RY-A
16
32 GB DDR3-1600 MHz LR
DIMM/PC3-12800/quad
rank/x4/1.35v
DRAM
UCSSD200G0KA
2-E
4
200 GB std height 15 mm
SATA SSD hot plug/drive sled
mounted
Internal hard drives
UCSC-SD16G-C240
1
16 GB SD card module for
C240 servers
SD card
UCSC-RAIL2U
1
2U rail kit for UCS C-Series
servers
Rail kit
N20-BBLKD
20
UCS 2.5-inch HDD blanking
panel
HDD panels
UCSC-HSC240M3
2
Heat sink for UCS C240 M3
rack server
Heat sinks
UCSC-PCIF01F
4
Full-height PCIe filler for CSeries
PCI slot fillers
UCSC-PCIF01H
1
Half-height PCIe filler for UCS
PCI slot filler
UCSC-RAID11-C240
1
LSI 2008 SAS RAID
Mezzanine Card for UCS C240
server
RAID card
CAB-C13C14-AC
2
Power cord C13 to C14
(recessed receptacle) 10A
Power cables
UCSC-PSU650W
2
650 W power supply for CSeries rack servers
Power supplies
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
12
Server
hardware
Server software
Quantity
Configuration
Description
LPE12004M8
2
Emulex Quad Channel 8Gb FC
PCIe HBA
Fibre Channel cards
E10G42BTDA
1
Intel X520-DA2- Network
adapter - PCI Express 2.0 x8
low profile - 10 gigabit
Ethernet - 2 ports
10 GbE network card
E10GSFPSR
2
Intel Ethernet SFP+ SR Optics
- SFP+ transceiver module 1000Base-SX, 10GBase-SR 850 nm
Optical ports for FC
Table 4 describes the various software components of the solution server layer.
Table 4.
Red Hat Enterprise
Linux
Server software
Server software
Configuration
Description
Red Hat Enterprise Linux
6.3
Operating system for database servers
Oracle Grid Infrastructure
11g Release 2
Enterprise
Edition
11.2.0.3
Software provides Clusterware and ASM
storage volume management
Oracle Database 11g
Release 2
Enterprise
Edition
11.2.0.3
Oracle Database software
EMC XtremCache
software
2.0.1
EMC XtremCache software for server-side
flash cache
Red Hat Enterprise Linux includes enhancements and new capabilities, including
developer tools, virtualization features, security, scalability, file systems, and
storage. Red Hat Enterprise Linux is a versatile platform that can be deployed on
physical systems, as a guest on the major hypervisors, or in the cloud. It supports all
leading hardware architectures with compatibility across releases.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
13
Oracle Grid
Infrastructure and
Database 11g
Release 2
Oracle Database 11gR2 is available in a variety of editions tailored to meet the
business and IT needs of an organization. This solution utilizes Oracle Database
11gR2 Enterprise Edition (EE). Oracle Database 11g R2 EE delivers industry-leading
performance, scalability, security, and reliability on a choice of clustered or single
servers running Windows, Linux, or UNIX. The database supports advanced features
such as Virtual Private Database, and data warehousing options such as partitioning
and Advanced Analytics, that are either included or available as extra-cost options.
XtremSF and
XtremCache
deployment
XtremSF and XtremCache overview
In this solution, two 700 GB EMC XtremSF flash cards are used in each RAC node, and
one XtremCache cache device is created from one XtremSF card.
EMC XtremSF is an advanced flash storage technology deployed in a server and
designed to deliver unprecedented performance acceleration by reducing latency and
increasing I/O throughput. EMC XtremSF allows applications to access data in the
most efficient manner possible. Residing on the server PCIe interconnect bus,
XtremSF reduces application response time from milliseconds to microseconds by
caching the frequently accessed data at the server side, which reduces the number of
I/O requests sent to backend disk array.
EMC XtremCache is server flash-caching software that reduces latency and
accelerates throughput to dramatically improve application performance when
coupled with EMC XtremSF PCIe flash technology. XtremCache accelerates reads and
protects data by using a write-through cache policy to the networked storage to
deliver persistent high availability, integrity, and disaster recovery.
XtremCache coupled with array-based EMC FAST™ software provides the most
efficient and intelligent I/O path from the application to the underlying disk array. The
result is a networked infrastructure that is dynamically optimized for performance,
intelligence, and protection for both physical and virtual environments.
Benefits of XtremCache include the following:

Provides performance acceleration for read-intensive workloads with the
protection of a back-end, networked storage array

Provides an intelligent path for the I/O and ensures that the right data is in the
XtremCache of the servers at the right time

By offloading flash and wear-level management onto the XtremSF PCIe flash
card, uses minimal CPU and memory resources from the server

Works in both physical and virtual environments

Provides better data protection: since XtremCache is a write-through cache, it
does not comprise data consistency in the storage array, even if the cards fail in
the middle of I/O processing

No need to warm up the XtremCache across database instances reboot

Works for any kind of I/O; for example, any applications and any database
platforms

Is supported on various operating systems and server platforms
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
14

Allows customers flexibility in choice of cache capacity on the cards

Supports the Oracle RAC database, and even these RAC databases are
“stretched” with EMC VPLEX
As XtremCache is installed in a greater number of servers in the environment, more
I/O processing is offloaded from the storage array to XtremCache configured on the
servers. This provides a highly scalable performance model in the storage
environment. For more detail, refer to:

Introduction to EMC XtremCache for Oracle Real Application Clusters listed in
References

Introduction to EMC XtremCache for Oracle Real Application Clusters video
listed in References
EMC XtremCache configuration
XtremCache supports Oracle RAC using a distributed cache coherency algorithm.
XtremCache automatically recognizes the presence of Oracle RAC, and switches
operation to clustering mode.
All working Oracle RAC nodes must have XtremCache installed for the distributed
Cache feature to come online. EMC recommends using XtremCache with Oracle RAC
to cache LUNs that holding data files and TEMP files. Do not use XtremCache to cache
redo logs, archives, or clusterware files.
Steps for configuring XtremCache devices are shown in the XtremCache manual.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
15
EMC Oracle Performance Solution: Network layer
The switch component level is made up of two Cisco MDS 9506 director-class SAN
switches (like the one shown in Figure 3). The Cisco MDS 9506 is designed for
deployment in storage networks supporting virtualized data centers and enterprise
clouds. The switch combines high performance and low total cost of ownership, a
core architectural requirement at all levels of the VMAX Performance Block.
Figure 3.
Cisco MDS 9506 Multilayer Director
The Cisco MDS 9506 also offers these benefits:

Highly available scalability through a combination of nondisruptive software
upgrades, stateful process failover, and full redundancy of all core components

Superior platform for accelerated, intelligent storage applications such as EMC
replication and backup, data migration, and storage media encryption

Virtual machine transparency and end-to-end visibility all the way from the virtual
machine down to the EMC storage, enabling scalable, mobile virtual machines
Nexus 5.2(8) software is used in the EMC High Performance Solution for Oracle on
VMAX.
Table 5 lists the hardware components of the network layer of this solution.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
16
Table 5.
Hardware components of the network layer
Network
hardware
Quantity
Configuration
Description
MDS-PBF-ADV32
8
Cisco 32-port 8-Gbps FC Port
Module
Line cards
MDS-9506-V2
2
Chassis SUP2 no ports Director
Director chassis
FC10M-50MLC
128
FCHNL 10M 50/125 LC-LC
LC adapters
MDS-PW19-TWST
2
Cisco 9506 Twist Lock Power
Cord US
Power cord
FC1M-50MLC
96
FCHNL 1M 50/125 LC-LC
MDS-8G-SW
192
MDS 2/4/8-Gbps FC shortwave
SFP LC
MDS-ENT-9500
2
Enterprise license key 9500
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
17
EMC Oracle Performance Solution: Storage layer
Introduction
EMC is not only the world’s largest storage provider, but also the leader in storage
infrastructure in Oracle Database environments. Storage User Demand Study, 2012 –
Fall Edition: Many Uses of Internal Storage provides more details.
Storage
components
The storage components comprising the EMC Proven High Performance Solution for
Oracle on VMAX include the following:
Storage hardware

EMC Symmetrix VMAX 40K with eight engines (the specification of the engine is
shown in Figure 4)

EMC PowerPath®

Dynamic cache partitioning (VMAX)
Figure 4.
EMC Symmetrix VMAX 40K
Table 6.
EMC Symmetrix VMAX storage hardware
Storage
hardware
Quantity
Configuration
SD-DBV-DIR-1P
4
VMAX 40K DBV DIR 1 Phase
SD-DBV-SPS
16
VMAX 40K SPSV
SD-1P
1
VMAX 40K 1P INFRAST
SVDBSOLDOR1P
4
VMAX 40K Drive Bay Solid Door
1P
SYMV2-MIGRBAS
1
Symmetrix 40K Migration Bundle
PP-SE-SYM
1
PPATH SE SYM
Description
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
18
EMC Symmetrix
VMAX 40K
eight-engine
configuration
Storage
hardware
Quantity
Configuration
SD-VCONFIG32
1
VMAX 40K VCONFIG 32
SD-FE80000E
16
VMAX 40K 8 MM 8 G Fibre
SD-INTBKVKIT
8
VMAX 40K Internal Cable Bracket
Titan
SD-PW40U-US
10
30A 1Phase Namer Japan l6-30P
Power
VL4103001B
376
VMAX 40K 4 G 10K 300 GB SAS
Drive
Disk drives
SD-ADD192C
7
VMAX 40K Add Engine-192GB-C
VMAX engine
SD-192-BASEC
1
VMAX 40K Base-192GB-C
VMAX cache
SD-DE25-DIR
64
VMAX 40K 25SLT DR ENCL
Description
Fibre Ports
The EMC Symmetrix VMAX 40K is designed for high efficiency, scalability, and secure
data persistence. Built on the strategy of powerful, trusted, and smart storage, and
founded in EMC Virtual Matrix Architecture that allows for seamless, cost-effective
growth, the EMC Symmetrix VMAX 40K offers the following:

Zero downtime migration technology and lower cost and greater efficiency
through automated tiering

More scalability for less management complexity and operational expense
The configuration supporting this solution is described in Table 7:
Table 7.
EMC Symmetrix VMAX 40K configurations
Component
Quantity
Configuration
Engines
8
192 GB cache each, total of 1536 GB raw cache
Directors
16
Eight ports on each director with 8 GB Fibre
Channel
Bays
5
1 system, 4 disk
10K SAS drives
376
100 TB Raw, 45 TB Usable (RAID1 Configured)
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
19
Storage software
Table 8 lists the software used in the solution storage layer.
Table 8.
Storage software
Storage software
Configuration
Description
VMAX Enginuity™ code
5876
VMAX micro code
EMC Solutions Enabler
7.6
Host CLI storage management software
EMC PowerPath
5.7 SP1
Multipathing and load balancing software
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
20
EMC Oracle Performance Solution: Oracle Database layer
Introduction
In Oracle 11g R2, Oracle ASM and Oracle Clusterware have been integrated into the
Oracle Grid Infrastructure. In the solution, we use ASM to store the database files.
Storage virtual
provisioning
design
EMC Virtual Provisioning™ automatically stripes data across all data devices in a
virtual pool and balances the workload across storage devices. To ensure even
striping of data, all data devices in a virtual pool should be the same size.
Table 9 shows the RAID selections and number of spindles for each virtual pool. In
this solution, Oracle data files and redo log files are located on thin devices using
RAID 1 protection and all physical spindles for the best performance and capacity.
Table 9.
ASM disk group
configuration for
OLTP database
Virtual pool design on VMAX 40K
Virtual
pool
RAID
protection
Drive
type
Physical
spindles
size
FC_R1_1
RAID 1 (2-way,
mirror)
FC 10K
300 GB
Number of
active spindles
376+ 5 (spare
disk)
Item
CRS, DATA, REDO
Table 10 details the RAC database Oracle ASM (automatic storage management) disk
group design. On each database, we used three ASM disk groups to store the relevant
database files, including data files, control files, online redo log files, and temporary
files. Default settings are used for ASM disk groups.
Table 10.
ASM disk group design for RAC database
Item
LUN size (GB)
Number of LUNs
ASM disk group name
CRS
10
2
+CRS
DATA
1024
18
+DATA
REDO
60
4
+REDO
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
21
OLTP database and Table 11 details each OLTP database workload profile for the solution. We used the
SLOB toolkit to generate an OLTP database and deliver the OLTP-like workloads,
workload profile
including the query-only and UPDATE-only workloads required for the solution.
Table 11.
Database workload profile for each OLTP database
Profile characteristic
Details
Database type
OLTP
Database size
16 TB
Oracle Database
11gR2 eight-node RAC database on ASM
Instance configuration for read only
workload
SGA size: 16 GB
Workload profile
OLTP-like workload simulated by SLOB
Network connectivity
8 Gb Fibre Channel for SAN; 10 GbE for IP
Note: Because more SGA causes less I/O
activity, we reduced SGA to this small number
to generate a stable and high I/O workload
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
22
EMC Oracle Performance Solution: Performance testing
Overview
The solution characterizes the Oracle OLTP system performance on VMAX array with
EMC XtremSF cards installed on database servers. We created an eight-node Oracle
RAC database on eight physical servers. Silly Little Oracle Benchmark (SLOB) was
used to generate workload because it is the premiere SQL workload for driving
maximum physical random I/O from a database platform.
SLOB is a SQL-driven Oracle database I/O generator, instead of a synthetic I/O
generator. SLOB uniquely drives massive physical I/O using minimal host CPU
resources, and it specifically targets the Oracle I/O subsystem. SLOB performs all of
its physical I/O buffered in the Oracle SGA. No physical I/O buffered in the Oracle PGA
is performed. SLOB possesses the following characteristics:

Supports testing Oracle logical read (SGA buffer gets) scaling

Supports testing physical, random single-block reads (db file sequential
read/db file parallel read)

Supports testing random single block writes (db file parallel write)

Supports testing extreme REDO logging I/O

Consists of simple PL/SQL

Is entirely free of all application contention
We used SLOB to generate an OLTP-like workload on the eight-node Oracle RAC
database to demonstrate sustained flash and storage array IOPS. The database
performance metrics including IOPS and latency were gathered primarily from the
AWR report. In addition, we gathered metrics for I/O throughput at the
server/database and storage levels.
Notes:
Benchmark results are highly dependent on workload, specific application requirements,
and system design and implementation. Relative system performance will vary because of
these and other factors. Do not use solution test workloads as a substitute for a specific
customer application benchmark when making critical capacity planning or product
evaluation decisions
All performance data contained in this report was obtained in a rigorously controlled
environment. Results obtained in other operating environments may vary significantly.
EMC Corporation does not warrant or represent that a user can or will achieve similar
performance expressed in transactions per minute.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
23
Test objective
This test measures the physical I/O scalability along with the scaling of the number of
concurrent SLOB zero-think-time sessions (simulated concurrent users) and the
number of RAC nodes. Multiple concurrent sessions (reader sessions) executing
similar query SQL statements were run to validate read only workload, and multiple
concurrent sessions (writer sessions) executing similar UPDATE SQL statements were
run to validate the physical read/write workload.
Query only test
scenarios and
methodology
We gradually increased the number of instances of the RAC database and the number
of concurrent users with each user running similar OLTP queries simultaneously.
When we added a RAC node, additional resources were added, including CPU power
and XtremSF cards. With the addition of new RAC nodes, we tested the system again
by running the similar SLOB workload. For this test, workloads were running
simultaneously from all the RAC nodes added. We then increased the number of
concurrent users and measured the performance scalability.
The test process included the following steps:
Read test results
1.
Run the OLTP query only workload with 64 concurrent simulated users (zerothink-time sessions) on the first node of an eight-node RAC database using
SLOB.
2.
Add the second node into the system, then run the workload with 64
concurrent users on each node; that is, with 128 concurrent users in total
running simultaneously on the two-node RAC database.
3.
Add two additional nodes into the system, then run the workload with 64
concurrent users on each node separately; that is, with a total of 256
concurrent users running simultaneously on the four-node RAC database.
4.
Add four additional nodes into the system, then run the workload with 64
concurrent users on each node separately; that is, with a total of 512
concurrent users running simultaneously on the eight node RAC database.
Performance statistics were captured using Oracle Automatic Workload Repository
(AWR) RAC reports. We observed the “physical reads” value in the AWR report to
assess read IOPS statistics. Query average response time was calculated from the
“db file parallel read” and “db file sequential read” record in the “Top Timed Events”
section of the AWR report, as shown in Figure 5.
Figure 5.
AWR RAC report snippet for read IO response time calculation
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
24
We used the following formula to calculate the I/O latency:



For “db file sequential read”

The total wait time is T1 which is 19,485.48 seconds as shown in Figure 5.

The total number of waits is N1 which is 30,614,070 as shown in Figure 5.
For “db file parallel read”

The total wait time is T2 which is 9,803.47 seconds as shown in Figure 5.

The total number of waits is N2 which is 8,610,615 as shown in Figure 5.
The average read response time is: (T1+T2) / (N1+N2), which is (19,485.48 +
9,803.47) * 1,000/ (30,614,070 + 8,610,615) as shown in Figure 5, and
0.75ms is the average response time.
Table 12 depicts how IOPS increase when RAC nodes scale up.
Table 12.
Scaling of nodes and resulting increases in IOPS
Metrics
1 node
2 nodes
4 nodes
8 nodes
IOPS
457,136
962,155
1,914,963
3,765,176
Average response time (ms)
0.74
0.69
0.68
0.75
Figure 6 shows that IOPS increase when the number of RAC nodes scales, while
average read I/O response time remains under one millisecond.
Figure 6.
Query only IOPS scaling and average physical read I/O response time
As Figure 6 shows, we achieved a total of 3,765,176 read IOPS and an average
latency of 0.75 milliseconds with eight-node RAC database when running 64
concurrent sessions executing similar queries SQL statements on each of the RAC
nodes.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
25
The IOPS increased nearly linearly with each additional node that was added into the
Oracle RAC database. For example, the total IOPS of four RAC nodes reached
1,914,963. After another four RAC nodes are added for a total of eight database
nodes, the IOPS are almost doubled to 3,765,176.
The read hit ratio for XtremCache is about 98 percent for each cache device during the
test. Two percent of the I/Os are served from the storage array. The statistics can be
monitored by running the following command line:
vfcmt display -cache_dev <device>
Update only test
scenarios and
methodology
We gradually increased the number of RAC nodes and ran a total of 56 concurrent
sessions on the database to generate stable and high read and write IOPS, with each
session running similar SLOB UPDATE SQL statements
We decreased buffer cache for each database instance to push consistent write I/O
workload to the back-end storage. The write workload was driven by the UPDATE SQL
statement. Generally, it incurs the following operations sequentially:
1.
Read the data blocks that need to be updated into the buffer cache.
2.
Update the rows in the data blocks.
3.
Commit the updated rows and trigger LGWR flushing redo entries to online log
files.
During the SQL UPDATE only workload running, the background DBWR process
flushes the dirty blocks out of the buffer cache into data files. Because we used a very
small buffer cache, the data blocks were read into the buffer cache and written out of
the buffer cache soon after the rows were updated. Thus the execution of each
UPDATE operation caused physical reads, which were accelerated by the EMC
XtremCache (when cache hit) or the back-end VMAX array (cache miss). When the
updated data blocks were written out of the buffer cache by the DBWR process, as
being a write through cache for EMC XtremCache , the data blocks were written to the
back-end VMAX array. Once the application got the acknowledgement from the backend array, the application I/O request was complete, and the updated data was
asynchronously written to XtremCache.
The test process included the following steps:
1.
Run the Update only workload with 56 concurrent users on one RAC Database
with only one node using the scripts in SLOB.
2.
Add one additional node into the RAC database, then run the workload with
56 concurrent users; that is, with 28 concurrent users running simultaneously
on each node of this two-node RAC.
3.
Add two additional nodes into the RAC database, then run the workload with
56 concurrent users; that is, with 14 concurrent users running simultaneously
on each node of this four-node RAC.
4.
Add four additional nodes into the RAC database, then run the workload with
56 concurrent users; that is, with seven concurrent users running
simultaneously on each node of this eight-node RAC
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
26
Peak write test
results
Performance statistics are captured using AWR reports. We read the “physical writes”
row in the AWR report for the peak write IOPS statistics. Because the write workload is
generated by UPDATE statements as described previously, we also collected
“physical reads” from the AWR report for the peak read IOPS statistics that were part
of the write transaction.
We calculated updated write average response time by dividing the “Total Wait
Time(s)” by the “Waits” of the “db file parallel write” record in the “Top Timed
Events” section of the AWR report, as shown in Figure 7. We also calculated LGWR
latency by dividing the “Total Wait Time(s)” by the “Waits” of the “log file parallel
write” record in the “Top Timed Events” section of the AWR report, as shown in Figure
7.
For example, in following AWR snippet, the total wait time of the “db file parallel
write” wait event is 287.00 seconds, which is 287,000 ms, and the number of waits is
319,941; thus, the write average response time can be calculated as 287,000 /
319,941= 0.90 ms. The LGWR latency is: 1,087.81*1,000/1,323,505 = 0.82.
Figure 7.
Update only write average response time measurement from the RAC AWR
report
Table 13 and Figure 8 depict the peak read/write IOPS increases as the RAC node
scaling.
Table 13.
Scaling of users and resulting increases in peak disk array IOPS
IOPS
1 node
2 nodes
4 nodes
8 nodes
Write
26,950
50,110
95,932
153,276
Read
26,542
49,539
94,877
150,054
Aggregate
53,492
99,649
190,809
303,330
Write response
time
0.30
0.40
0.30
0.90
Read response
time(ms)
0.90
0.80
0.20
0.30
Redo size (MB/s)
20
37
71
115
LGWR latency
(ms)
0.55
0.54
0.66
0.82
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
27
Figure 8 shows the peak IOPS, and update-only IOPS for RAC node scaling, while
average response time remains under one millisecond.
Figure 8.
Peak writes – update-only IOPS scaling along with RAC nodes scaling
During an UPDATE transaction, the backend VMX only needs to handle the writing I/O
activities, as the reading I/O activities have been cached and accelerated by
XtremCache. Because of this, the solution can scale to accommodate a very heavy
transaction workload, as confirmed in testing.
As shown in Table 13 and Figure 8, when running 56 concurrent sessions executing
similar UPDATE SQL statements on eight-node RAC database, we achieved 303,330
aggregated IOPS including 153,276 write IOPS and 150,054 read IOPS, which were
part of the write transaction. The average latency of write was 0.9 milliseconds,
because we used very small buffer cache (only 128 MB, almost no data cached in the
server) to generate a high physical write I/O workload.
The IOPS increased nearly linearly when additional RAC nodes were added into the
workload. For example, the aggregate IOPS were 53,492 when running write workload
on one RAC node, and this increased to 99,649 when running workload on two RAC
nodes.
Redo size is also a key metric used to measure the transaction capability. As
demonstrated through testing, the workload on one node generated 20 MB/second
redo entries, and almost doubled to 37 MB/second with the workload running on two
nodes. When we ran workload on four nodes the redo throughput is almost doubled
again to 71 MB/second. The transaction capability is scaling along with the node
scaling.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
28
Conclusion
Summary
Implementing the EMC Proven High Performance Solution for Oracle RAC on VMAX
with innovative and proven products like EMC’s VMAX gives customers choices within
an open infrastructure, enabling them to easily integrate into existing data center
operations and take advantage of new technologies such as XtremCache, which now
also fully supports Oracle RAC. Customers can efficiently use people, processes, and
technologies through virtualization, database, and applications consolidation.
Customers can also independently scale capacity and processing capability without
the limitations imposed by a single-purpose appliance
As the infrastructure undergoes changes to applications, databases, and nondatabase software, this stack can align with the shifting technical demands imposed
by the business needs. This white paper explains how to balance OLTP workloads
while maintaining the protection and resiliency of the data. That adaptability to
change and the ability to apply the technology where it is needed protects the capital
investment, and can be fluid as the requirements change without sacrificing any of
the other datacenter operations.
Findings
The core advantages of this solution are as follows:

With its open architecture, supports Oracle OLTP workload built on superior
functional integration throughout the solution

Leverages the following EMC technology enablers in the solution:

EMC VMAX 40K

XtremSF flash cards

XtremCache

Provides full support for EMC Performance Boost, HA, and replication
technologies

Increases IOPS for OLTP workloads when servers are added

The aggregate read IOPS increased from 457,136 to 1,914,963 when RAC
nodes were scaled out from one to four, and it increased to 3,765,176
when eight RAC nodes ran the workload together.


Average response time less than one millisecond (0.75ms)
The aggregate IOPS for UPDATE transaction workload was 53,492 on one
RAC node and caused 20 MB/s redo throughput, and it increased to
190,809 and caused 71 MB/s redo throughput when four RAC nodes were
added. The IOPS increased to 303,330 and generated 115 MB/s redo
throughput when running on eight RAC nodes.

Average response time for DBWR background process less than
one millisecond (0.90 ms)
This solution provides a foundation that can be scaled in a flexible, predictable, and
near-linear way, by adding additional server resources including CPUs and memory,
HBA ports, and front-end ports, to provide higher IOPS and throughput based on the
described configuration.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
29
References
EMC
documentation
The following documents provide additional and relevant information. Access to
these documents depends on your login credentials. If you do not have access to a
document, contact your EMC representative.
The following reference materials are available on EMC Powerlink:

EMC Infrastructure for High Performance Microsoft and Oracle Database Systems

Introduction to EMC XtremCache

EMC XtremCache Data Sheet
In addition, XtremCache documentation is available at EMC Online Support:
https://support.emc.com/products/25208_XtremCache-Cache/Documentation/
A video entitled Introduction to EMC XtremCache for Oracle Real Application Clusters
is visible via this link: https://community.emc.com/videos/6740
Oracle
documentation
For additional information, see the following documents:

Oracle Grid Infrastructure Installation Guide 11g Release 2(11.2) for Linux

Oracle Database Installation Guide 11g Release 2 (11.2) for Linux.
High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX
Enabled by EMC VMAX 40K, XtremSF, XtremCache,
Red Hat Enterprise Linux, Oracle Database Enterprise Edition
30