Download Remote Site Recovery of Oracle Enterprise Data Warehouse

White Paper
REMOTE SITE RECOVERY OF ORACLE
ENTERPRISE DATA WAREHOUSE USING
EMC DATA DOMAIN
A Detailed Review
EMC SOLUTIONS GROUP
Abstract
This white paper describes how a 12 TB Oracle data warehouse was transported
from a data center in Massachusetts, U.S, to Cork, Ireland. The infrastructure
was re-created at the Cork site on EMC® Symmetrix® VMAX™ storage and the
database was restored and recovered. EMC Data Domain® Encryption was used
to secure the data during transport.
July 2011
Copyright © 2011 EMC Corporation. All Rights Reserved.
EMC believes the information in this publication is accurate as of its
publication date. The information is subject to change without notice.
The information in this publication is provided “as is.” EMC Corporation makes
no representations or warranties of any kind with respect to the information in
this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.
Use, copying, and distribution of any EMC software described in this
publication requires an applicable software license.
For the most up-to-date listing of EMC product names, see EMC Corporation
Trademarks on EMC.com.
All trademarks used herein are the property of their respective owners.
Part Number H8238.1
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
2
Table of contents
Executive summary ............................................................................................................... 5
Business case .................................................................................................................................. 5
Solution overview ............................................................................................................................ 5
Key results ....................................................................................................................................... 6
Introduction .......................................................................................................................... 7
Purpose ........................................................................................................................................... 7
Scope .............................................................................................................................................. 7
Audience.......................................................................................................................................... 7
Terminology ..................................................................................................................................... 7
Technology overview ............................................................................................................. 8
Overview .......................................................................................................................................... 8
EMC Data Domain DD660 ................................................................................................................. 8
EMC Data Domain Encryption ........................................................................................................... 9
EMC Data Domain file system encryption lock .................................................................................. 9
EMC Symmetrix VMAX ...................................................................................................................... 9
EMC Solutions Enabler ..................................................................................................................... 9
EMC Symmetrix TimeFinder .............................................................................................................. 9
EMC PowerPath .............................................................................................................................. 10
Oracle Database ............................................................................................................................ 10
Solution overview ............................................................................................................... 11
Use case requirements................................................................................................................... 11
The challenge................................................................................................................................. 11
Nondisruptive to production ...................................................................................................... 11
Distance and latency ................................................................................................................. 11
Database size............................................................................................................................ 12
The solution ................................................................................................................................... 12
Project timeline .............................................................................................................................. 14
Discovery – primary site ...................................................................................................... 15
Overview ........................................................................................................................................ 15
Security.......................................................................................................................................... 15
Business and technical .................................................................................................................. 16
Primary data center configuration information................................................................................ 16
Remote site infrastructure .............................................................................................................. 17
Architecture ............................................................................................................................... 17
Hardware resources................................................................................................................... 18
Software resources .................................................................................................................... 18
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
3
Backup appliance...................................................................................................................... 18
Oracle database backup – primary site ................................................................................ 19
Overview ........................................................................................................................................ 19
DD660 setup .................................................................................................................................. 19
Network settings ....................................................................................................................... 19
Encryption ................................................................................................................................. 20
RMAN backup ................................................................................................................................ 20
File system lock.............................................................................................................................. 20
Data compression results............................................................................................................... 21
Oracle server deployment and storage provisioning – remote site ........................................ 23
Overview ........................................................................................................................................ 23
Deploying the remote site server .................................................................................................... 23
Configuring the remote site storage ............................................................................................... 23
Primary data warehouse configuration information.................................................................... 23
Provisioning storage at the remote site ...................................................................................... 24
Setting up the Oracle database environment.................................................................................. 25
Oracle database recovery – remote site ............................................................................... 27
Overview ........................................................................................................................................ 27
DD660 setup .................................................................................................................................. 27
Network settings ....................................................................................................................... 27
Unlock file system ..................................................................................................................... 28
RMAN restore ................................................................................................................................. 29
Verifying the integrity of the database ............................................................................................ 32
Data Domain consolidation – remote site ............................................................................ 34
Overview ........................................................................................................................................ 34
Replication process........................................................................................................................ 34
Conclusion ......................................................................................................................... 37
Summary ....................................................................................................................................... 37
Findings ......................................................................................................................................... 37
References .......................................................................................................................... 38
White papers ................................................................................................................................. 38
Product documentation.................................................................................................................. 38
Other documentation ..................................................................................................................... 38
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
4
Executive summary
Business case
EMC offers an extensive series of solutions for quick, efficient, and nondisruptive
backup of large Oracle data warehouse environments. Increasingly, customers are
looking to repurpose or reuse these backups to facilitate operations such as:
•
Migration of Oracle data warehouses to new data center locations
•
Seeding of Oracle data warehouse environments at remote disaster recovery
locations
•
Deployment of remote database copies for test and development purposes
These scenarios present challenges in relation to determining the most appropriate
mechanism for transferring the data from the primary site to the remote site, and
developing validated procedures for re-creating the production environment and
recovering the database at the remote site.
The main considerations in relation to data transfer to the remote site include:
•
Security and corporate governance compliance
•
Distance and latency
•
Database size
•
Cost effectiveness
For one-off migration scenarios, where the data warehouse is many terabytes in size
and the distance between data centers is great, it is impractical to transfer the data
across the WAN. Consequently, customers are looking for an offline transport solution
that is economical and secure.
Solution overview
At its headquarters in Hopkinton, Massachusetts, EMC IT maintains a 12 TB Oracle
data warehouse, deployed on an Oracle Real Application Clusters (RAC) 11g
environment. To demonstrate the benefits and implementation of the solution
documented in this white paper, this real-world production database was cloned and
backed up, shipped by air from the primary data center in Hopkinton, Massachusetts
to the EMC Solutions Center in Cork, Ireland, and then restored and recovered for test
and development use.
The remote data center infrastructure consisted of a two-node Oracle RAC 11g cluster
that accessed the Oracle database on an EMC® Symmetrix® VMAX™ storage system.
This reflects the infrastructure at the primary data center.
The production database in Hopkinton was cloned using EMC TimeFinder® and then
backed up to an EMC Data Domain® DD660 appliance using Oracle Recovery Manager
(RMAN). Data Domain Encryption and file system encryption locking were used to
ensure security and corporate governance compliance during transport.
At the remote site, the DD660 was added to the environment and the file system was
unlocked. The database was then restored and recovered to a test system using
Oracle RMAN.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
5
After verifying that the database had been successfully restored, Data Domain
Replicator was used to move the database image from the DD660 to a shared Data
Domain DD880 appliance in the data center in Cork, so that the DD660 could be
repurposed for another project.
Key results
The key benefits demonstrated by the solution include:
•
Nondisruptive backup
Creating a TimeFinder clone has no impact on production. Mounting the clone
to a separate backup server provides a fully-usable, independent, point-in-time
copy that can be backed up to the Data Domain appliance with no disruption to
the production environment.
•
Reduced backup footprint and shipping costs
Data Domain deduplication reduces the footprint of the backup database
significantly. In this solution, a 70 percent reduction in capacity requirements
enabled use of the midrange DD660 appliance, which minimized shipping
costs.
•
Seamless Oracle RMAN integration
Data Domain systems integrate seamlessly into Oracle architectures,
supporting existing backup strategies, including Oracle RMAN.
•
Data security and corporate governance compliance
Data Domain Encryption ensures that all data is physically stored in an
encrypted manner and cannot be accessed without first decrypting it. Data
Domain file system encryption lock adds a further level of protection that
enables shipping on commercial air transport while satisfying corporate
governance compliance requirements.
•
Validated and documented end-to-end procedures
Database integrity testing at the end of the project demonstrated successful
recovery of the database at the remote data center.
This white paper documents the procedures for achieving these results, from
the initial discovery phase necessary for designing the infrastructure for the
remote site, to a complete restore and recovery of a working data warehouse.
These procedures are applicable to any solution involving recovery of an Oracle
database.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
6
Introduction
Purpose
This white paper documents a case study that involved cloning and backing up a realworld 12 TB database, shipping it by air from the primary data center in Hopkinton,
Massachusetts to the EMC Solutions Center in Cork, Ireland, and then restoring and
recovering the database for test and development use.
Scope
The white paper covers the following:
Audience
Terminology
•
An overview of the technologies involved in the solution
•
The discovery process carried out at the production site to develop a build plan
for the remote site
•
Re-creation of the data warehouse at the remote site
•
Backup and securing of the DD660 for transfer by commercial air transport
•
Restore and recovery of the database at the remote site
•
Validation of the recovered database
•
Replication of the database from the DD660 to a DD880 appliance
This white paper is intended for Oracle, storage, server, and backup administrators
who want to understand how to migrate an Oracle database to a remote site and the
build requirements at the remote site to enable recovery of the database. It is
assumed that the reader is familiar with the following EMC and Oracle products:
•
EMC Symmetrix VMAX
•
EMC Data Domain
•
Oracle Database 11g R1 Enterprise Edition with Oracle Clusterware
Table 1 defines terms used in this white paper.
Table 1.
Terminology
Term
Definition
AES
Advanced Encryption Standard
ASM
Automatic Storage Management
DD
Data Domain
DD OS
Data Domain operating system
RAC
Real Application Clusters
RMAN
Recovery Manager
SISL™
Stream-Informed Segment Layout architecture
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
7
Technology overview
Overview
EMC Data Domain
DD660
The case study uses the following hardware and software components:
•
EMC Data Domain DD660
•
EMC Data Domain Encryption
•
EMC Data Domain file system encryption lock
•
EMC Symmetrix VMAX
•
EMC Symmetrix TimeFinder
•
EMC PowerPath®
•
Oracle Database 11g R1 Enterprise Edition with Oracle Clusterware
•
Oracle RMAN
For the solution, an EMC Data Domain DD660 appliance was used to back up the
source database and transport it to the recovery site in Cork.
Data Domain deduplication storage systems dramatically reduce the amount of disk
storage needed to retain and protect enterprise data, requiring a storage footprint
that is 10 to 30 times smaller, on average, than that required by the original data set.
The DD660 is a midrange appliance that offers up to 2 TB/hour of aggregate inline
deduplication throughput, and up to 700 GB/hour for a single stream. The model
used in the case study provides 8 TB of storage capacity, with much higher data
storage capacity enabled by deduplication.
Like all Data Domain systems, the DD660 derives its performance advantages from
the Data Domain Stream-Informed Segment Layout (SISL) scaling architecture. SISL
identifies 99 percent of duplicate segments in RAM, inline, ensuring that only unique
data is stored to disk. The storage of unique data is invisible to backup software,
which sees the entire virtual file system. The integrity and recoverability of backup
data are secured by the Data Domain Operating System (DD OS) and Data
Invulnerability Architecture™, which ensures that data is stored and recoverable with
continuous write verification, fault-detection, and self-healing mechanisms.
When data enters the Data Domain system, using any of the supported protocols
(NFS, CIFS, VTL, OST, and Distributed Segmented Processing), the stream is
segmented, fingerprinted, deduplicated (Global Compression™), grouped into multisegment compression regions, and then locally compressed. When confidentiality is
required, deduplicated and compressed data can be encrypted inline and the file
system can then be locked.
Data Domain systems integrate seamlessly with Oracle RMAN, with no changes to
scripts, backup processes, or system architecture.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
8
EMC Data Domain
Encryption
Data Domain Encryption protects data if the Data Domain system is stolen or lost
during transit, and eliminates accidental exposure during replacement of failed
drives. DD Encryption is designed such that if an intruder circumvents other network
security controls and gains access to encrypted data, the data is unreadable and
unusable to that person without the proper cryptographic keys.
The inline encryption process integrates seamlessly with the inline deduplication
process, encrypting data before it is written to disk. Similar to the advantages of
inline deduplication, inline encryption requires minimal resources to provide fast,
reliable, and secure backup and recovery.
EMC Data Domain
file system
encryption lock
File system encryption lock is specifically designed to lock an encrypted Data Domain
appliance and its external storage prior to transporting the appliance between
locations.
File system encryption lock requires an encryption passphrase. This is a humanreadable key that is used to generate a machine-usable AES256 encryption key that
encrypts the Data Domain system encryption key. Locking the file system involves
changing the passphrase. This changes the encryption of the Data Domain system
key and removes the passphrase from the system. Locking the system allows a Data
Domain system to be transported with the encryption keys on the system. The keys
are not recoverable until the file system is unlocked using the passphrase (which is
not stored on the appliance).
EMC Symmetrix
VMAX
An EMC Symmetrix VMAX provides the storage platform at both the primary and
remote data centers.
Symmetrix VMAX storage arrays provide high performance and scalability for
demanding enterprise storage environments. Built on the strategy of simple,
intelligent, modular storage, the Symmetrix VMAX incorporates a highly scalable
Virtual Matrix Architecture™ that enables it to grow seamlessly and cost-effectively
from an entry-level configuration into the world’s largest storage system. The VMAX
supports Flash drives, Fibre Channel drives, and SATA drives within a single array, as
well as an extensive range of RAID types.
EMC Solutions
Enabler
EMC Solutions Enabler is the software that provides a host with the Symmetrix
Command Line Interface (SYMCLI), which is a comprehensive command set for
managing your Symmetrix storage environment. SYMCLI is used in single command
line entries and scripts to monitor device configuration and status, and perform
control operations on devices and data objects within a managed storage system.
EMC Symmetrix
TimeFinder
EMC Symmetrix TimeFinder software provides powerful replication capabilities for
creating local point-in-time copies that are independent of the hosts and operating
systems, applications, and databases. These copies are immediately available for
read and write access.
For the solution, a TimeFinder copy of the Oracle database was used to offload
backup overhead from the production environment to another server.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
9
EMC PowerPath
EMC PowerPath® is server-resident software that enhances performance and
application availability by supporting multiple I/O paths to logical devices and
intelligently distributing I/O requests across all available paths. PowerPath also
provides automatic failover in the event of a hardware failure by automatically
detecting the path failure and redirecting I/O to another path.
For the solution, PowerPath provides intelligent path management and automatic
path failover at the remote site.
Oracle Database
The solution involves an Oracle Database 11g R1 data warehouse and uses the
features of Oracle ASMLib, Oracle Database 11g, and Oracle Clusterware 11g such as
RAC, Automatic Storage Management (ASM), and RMAN.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
10
Solution overview
Use case
requirements
This solution is based on a real-world use case and is essentially a case study. The
requirements of the use case were to take a production database and transfer it to a
lab environment for test and development purposes.
The EMC IT Production DB resides in EMC IT corporate headquarters in Hopkinton,
Massachusetts, as shown in Figure 1. The test and development lab is located in the
EMC Solutions Center in Cork, Ireland. The database is approximately 12 TB.
Figure 1.
The challenge
An economical and secure mechanism was required to transport the data, and the
process also had to have zero impact to the EMC IT production system.
The challenge
Nondisruptive to production
As the project involved a real-world, production system, it was critical that the backup
procedure did not impact database operations. This factor dictated a nondisruptive
backup solution.
Distance and latency
The distance between the data centers in Hopkinton and Cork is approximately 4,740
kilometers (2,946 miles). The minimum latency for the round-trip is 47.4 ms, not
including other delays introduced by encryption or the network itself. In fact, the
average measured ping time between the data centers is 90 ms.
For example, a dedicated WAN link, with 47.4 ms round-trip latency, 1 Gb/s
bandwidth, and a TCP window size of 65,536 bytes, would restrict the maximum
throughput to 1.3 MB/s and take approximately 112 days to move 12 TB of data.
As this was a one-off project, it was not envisaged that sufficient dedicated
bandwidth would be made available on the corporate WAN for the data migration.
These factors dictated an offline method for transporting the data.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
11
Database size
The size of the database—approximately 12 TB— was a significant challenge and
another factor that made it impractical to transport the data across the WAN. Figure 2
indicates just how big 12 TB is in real terms.
Figure 2.
The solution
What 12 TB looks like
To facilitate transporting the data in an economical and secure fashion, a Data
Domain DD660 appliance was chosen. The reasons for this are:
•
Small form factor
The DD660 used has a 2U form factor and weighs 30 kg (66 lb)—perfectly
suited for low shipping costs.
•
Data compression
The DD660 has 8 TB of data storage space. However, Data Domain systems
store only unique data, using Global Compression to pool redundant data from
each backup image. This enables much higher data storage rates than the raw
capacity of the device.
Figure 3 shows the difference in size between the original database, the database
backed up by RMAN, and the deduplicated, encrypted database on the DD660.
Figure 3.
Original and deduplicated database sizes
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
12
•
Data integrity
The Data Domain Data Invulnerability Architecture protects against data loss
from hardware and software failures.
•
Encryption of data at rest
Data Domain Encryption protects data if the Data Domain system is stolen or
lost during transit.
•
Encrypted file system lock
Data Domain file system encryption lock protects an encryption-enabled Data
Domain system (and its external storage devices) while in transit. The encryption
keys are not recoverable without a passphrase, which is not stored with the
system.
To ensure that the production environment was not impacted by the project, it was
decided to create a TimeFinder clone of the database and mount this to a backup
server. This would provide a fully-usable, independent, point-in-time copy of the
production database that could be backed up to the DD660 nondisruptively.
Figure 4 illustrates the solution designed for the project and outlines the procedures
involved.
Figure 4.
The solution
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
13
Project timeline
The initial planning and feasibility phase of this project took place over several weeks
prior to the official kickoff of the project. After the official kickoff, the entire project
took just 22 working days to complete, from commencing discovery to verification of
the restored database at the remote site. Figure 5 shows the project schedule.
Predictably, discovery was the lengthiest phase (2 weeks) and was iterative in nature.
It progressed in parallel with, and was informed by, the build process at the remote
site. The comprehensive information gathering and planning during this phase was
key to ensuring successful completion of the project. The result was a complete and
successful restore and recovery of the database at the remote site in only 28 hours.
Figure 5.
Project timeline
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
14
Discovery – primary site
Overview
This section describes the discovery process carried out at the start of the project to
determine the feasibility of the project and to gather infrastructure and configuration
information from the production site. This information was used to determine the
transport mechanism between the two data centers and to create a build plan for the
remote site. This phase of the project involved the steps shown in Figure 6.
Figure 6.
Discovery process
The stakeholders involved in the project were:
•
•
Security
Primary site

Stakeholders: EMC IT; EMC IT Security and Compliance

Location: EMC IT headquarters, Hopkinton, Massachusetts, U.S.

Database: EMC IT Production DB
Remote site

Stakeholders: EMC Global Solutions (Oracle Engineering)

Location: EMC Solutions Center, Cork, Ireland (CSC)

Database: CSC test and development database
The security of the data to be transferred between the data centers was of critical
importance to the feasibility of the project. EMC IT and EMC IT Security considered
data security issues in terms of:
•
Corporate governance and compliance—that is, ensuring data security both in
transit and at the remote data center so as to meet legislative requirements and
EMC and industry standards.
•
Identifying any potential personal/sensitive data for masking/cleaning.
•
Choosing a backup/copy mechanism that would be nondisruptive to the
production database.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
15
This feasibility phase concluded that a copy of the EMC IT Production DB would be
supplied to EMC Global Solutions under the following conditions:
Business and
technical
•
The data would be stored in a secure EMC data center at the remote site.
•
Access to the data would be limited to specified EMC personnel.
•
No data would be made public or available to external parties.
•
A Data Domain DD660 would be used to transport the data, with Data Domain
Encryption and Data Domain file system encryption lock enabled.
To enable recovery and repurposing of the data warehouse, the infrastructure at the
primary site would be re-created at the remote site. In preparation for this, the
following business and technical issues were considered:
•
Mapping and configuration of the underlying infrastructure—servers, storage,
and IP and storage networks
•
Identifying hardware and software versions for the:
•

Server operating system

Oracle database

Symmetrix VMAX storage system

Data Domain system
Identifying the groups and individuals within EMC IT who could supply
information and carry out the necessary tasks in the production environment
A project plan was then agreed to that would minimize the impact on EMC IT staff and
would move the data to Cork with minimal delay.
Primary data center To facilitate the project, EMC IT made several important documents available to the
EMC Global Solutions team in Cork. These documents detailed the design and
configuration
information
architecture of the EMC IT data warehouse.
The Detailed Solutions Design document for the data warehouse was key to
understanding the production environment. This document contains the following
information:
•
High-level architecture
•
Hardware inventory and configuration information for the database and backup
servers and cluster, including CPU and memory, internal drives, and HBA and
network cards and ports
•
List of required Shared Infrastructure Services, including: SAN, storage array,
and IP networks—private (Cluster Interconnect), data center, and backup
•
Storage rationale and design, hardware and software, and configuration
•
Security—user and group access and authentication
•
Data center hardware provisioning—environment and cabling details
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
16
In addition, the following “live” information was supplied:
•
A copy of the current symapi_db.bin file from the production Symmetrix array
•
An extract of the Oracle data dictionary views including V$ASM_DISKGROUP
and V$ASM_DISK
•
A listing of the Oracle ASM disks
•
A mapping of the PowerPath devices and paths to the Symmetrix LUNs
•
Multiple sample Automatic Workload Repository (AWR) reports
•
A listing of the latest operating system and Oracle Clusterware and database
patch levels
Discovery was an iterative process, and continued in parallel with, and in response
to, the remote site build.
Remote site
infrastructure
Architecture
From the information gathered during the discovery phase, the infrastructure for the
remote site was designed as shown in Figure 7.
Figure 7.
Remote site architecture
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
17
Hardware resources
Table 2 details the hardware environment for the remote site.
Table 2.
Hardware resources
Purpose
Quantity
1
Storage array
Resources
Symmetrix VMAX with:
• 2 x VMAX 64 GB engines
• 8 Gb FC connectivity
• 450 GB 15k FC drives
Oracle RAC database servers
2
Linux server with:
• Quad-core CPU
• 96 GB RAM
FC switches
2
8 GB/s FC switches
Ethernet switches
2
1 Gigabit Ethernet switches
Software resources
Table 3 details the software resources for the remote site.
Table 3.
Software resources
Software
Purpose
Red Hat Enterprise Linux 5.5
Server OS for database and application servers
Oracle Database 11g R1 (11.1.0.7)
Database and cluster software
EMC Enginuity™ 5875
Symmetrix VMAX operating environment
Backup appliance
Table 4 details the configuration of the Data Domain appliance used for backing up
and transporting the database.
Table 4.
Data Domain appliance configuration
Appliance
Details
EMC Data Domain DD600
• 12 x 1 TB SATA drives
• 1 Gb IP connectivity
• DD OS 4.9
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
18
Oracle database backup – primary site
Overview
The process for configuring the DD660 and backing up the production database at
the primary data center involved the steps shown in Figure 8. When the process was
complete, the appliance was secure and ready to ship to Cork—that is, the data was
encrypted and the file system locked.
Local site
backup
Configure DD660
network settings &
add to network
Enable DD660 file
system encryption
Create clone copy
of production
database
RMAN backup to
DD660
Test database
restore
DD660 secure & ready to ship
Data encrypted
File system locked
Lock DD660 file
system
Figure 8.
DD660 setup
Backup
complete
Oracle database backup process
Network settings
The DD660 used for this solution had two 1 Gb network interfaces. To improve
network performance and resiliency, these were aggregated and used in parallel.
Figure 9 shows the link aggregation setup.
Figure 9.
DD660 link aggregation setup
To enable backup of the database, the appliance was then provided with an IP
address in the local network.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
19
Encryption
Data Domain Encryption was enabled prior to enabling the file system on the DD660.
Data was encrypted using the 128-bit Advanced Encryption Standard (AES) algorithm
(aes_128_cbc), as shown in Figure 10. This option strikes a reasonable balance
between security and performance for the use case.
Figure 10.
RMAN backup
DD660 encryption
On the production site, a fresh clone copy of the storage was created using
TimeFinder. An Oracle RMAN backup of this copy was then taken, with the database
mounted, but not opened, on a backup server. A recovery catalog was used to record
RMAN metadata.
With Data Domain Encryption enabled, the data being backed up to the DD660 was
deduplicated and then encrypted inline, before it was written to disk.
The following were backed up to the DD660:
•
A full RMAN backup of the Oracle database
•
The database control file
•
A schema export of the RMAN recovery catalog
The Oracle team on the production site then validated the backup by restoring the
control file and database to the backup server. Table 5 shows the duration of the
backup and restore operations.
Table 5.
File system lock
Backup and restore times
Process
Timings
Database backup to DD660
28 hours and 38 minutes
Database restore (for verification)
30 hours and 2 minutes
The filesys encryption lock command was used to lock the Data Domain file system.
This required a passphrase change, as shown in Figure 11, automatically re-encrypts
the system encryption key, and removes the new passphrase from the DD system.
The new passphrase was communicated separately to the remote site.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
20
Figure 11.
Data compression
results
Data Domain file system encryption lock
Data Domain deduplication eliminated redundant data segments during the backup
process. This allowed the Data Domain appliance to store and manage much more
data than would be possible with a traditional storage device.
Figure 12 shows a DD OS view that summarizes data storage and compression
statistics for the use case and illustrates the value of Data Domain deduplication
when transporting an Oracle database. Table 6 also summarizes these statistics.
Figure 12.
Use case compression summary
The total physical space available on the appliance is 7.5 TB, the total uncompressed
data written by RMAN to the appliance was 10.6 TB, and the space required to store
the compressed data was 3.2 TB. This demonstrates a 69.7 percent reduction in the
space required to store the database.
Table 6.
Use case compression summary
Item
Value
Total physical space available
7,528.6 GiB (7.5 TB)
Total data written to appliance (uncompressed)
10,613.6 GiB (10.6 TB)
Data stored on appliance (compressed)
3,217.0 GiB (3.2 TB)
Deduplication factor
3.3
Percent reduction—storage space saving
69.7%
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
21
Figure 13 shows the allocated, used, and free space information for the source
database. This closely matches the statistics gathered by the DD660 and shown in
Figure 12.
Figure 13.
SQL output showing database allocated and free space information
The database had 11.6 TB of storage assigned to its data files, not including temp
files, redo logs, control files and archive logs. Of this, 10.13 TB was allocated to
tables and indexes within the data files, with 1,504 GB of free space available.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
22
Oracle server deployment and storage provisioning – remote site
Overview
This section describes preparation of the hardware and software infrastructure at the
Solutions Center lab in Cork, as outlined in Figure 14.
Figure 14.
Deploying the
remote site server
Remote site preparation process
To simplify the restore process, the Oracle Solutions Engineering team decided to
restore the database as a single instance before migrating it to Oracle RAC.
The database server was configured with Red Hat Enterprise Linux 5.5 and prepared
for a standard Oracle ASM/Database install. Server configuration involved the
following steps:
1.
Install and configure EMC PowerPath.
2.
Install and configure ASMLIB.
Primary data warehouse configuration information
Configuring the
remote site storage During the discovery phase of the project, EMC IT provided the Oracle Solutions
Engineering team in Cork with the configuration database (symapi_db.bin) for the IT
data warehouse storage array.
At the remote site, the configuration information for the storage array was extracted
by querying symapi_db.bin in offline mode using standard Solutions Enabler display
commands. The procedure involved the following steps:
1.
Copy symapi_db.bin to a host with Solutions Enabler installed.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
23
2.
Use the following Solutions Enabler commands to set up Solutions Enabler to
query the database file:
export SYMCLI_OFFLINE=1
export SYMCLI_DB_FILE=”/DWproject/symapi_db.bin”
export SYMCLI_SID=XXX
3.
Use the following command to extract the configuration details for all devices
in the database storage group (GDWORAPRD), as shown in Figure 15:
symsg show SG_GDWORAPRD
Figure 15.
Extracting device configuration information from a storage group
With this information it was possible to understand how the production storage was
provisioned. In conjunction with the other information sources supplied by EMC IT
during the discovery phase, the Cork team was able to determine how much storage
was required on the target Symmetrix VMAX, the Oracle ASM disk groups required,
and the number of LUNs in each disk group.
Provisioning storage at the remote site
Provisioning storage at the remote site involved the following steps:
1.
Create the devices on the target Symmetrix VMAX array, as listed in Table 7.
Table 7.
Storage assigned to Symmetrix VMAX array at remote site
ASM disk group
2.
Number of devices
Device size (MB)
GDWP_TEMP
32
32,263
GDWP_ARCH
16
64,528
GDWP_REDO01
16
8,066
GDWP_REDO02
16
8,066
GDWP_T0_DATA01
16
129,056
GDWP_T1_DATA01
80
129,056
GDWP_T5_DATA01
40
129,056
Total Devices
216
Configure each device with a device identifier label that corresponds to the
name of the ASM disk group to which the device will belong. This label can be
read from the host using the Inquiry utility, which makes device identification
from the host much easier.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
24
Figure 16 shows these labels being created with Solutions Enabler.
Figure 16.
3.
Setting device identifiers with Solutions Enabler
Create a masking view for the devices so that they are visible to the RAC
cluster.
At this point, the storage is ready to be incorporated into the Oracle configuration at
the remote site.
Setting up the
Oracle database
environment
Setting up the Oracle database environment at the remote site involved the following
steps:
1.
Partition and assign the PowerPath devices for use with Oracle ASM.
Figure 17 shows the mapping of the Oracle ASM disks to the Symmetrix
assigned “device_name”.
Figure 17.
2.
Oracle ASM disk mapping to Symmetrix devices
Install Oracle Clusterware and the database binaries.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
25
3.
Configure the ASM instance and create the ASM disk groups listed in Table 8.
Table 8.
ASM disk groups and sizing
Allocation unit
Size
(MB)
(GB)
GDWP_ARCH
1
1,008
GDWP_REDO01
1
126
GDWP_REDO02
1
126
GDWP_T0_DATA01
16
2,016
GDWP_T1_DATA01
16
10,081
GDWP_T5_DATA01
16
5,041
GDWP_TEMP
1
1,008
Disk group
For example, Figure 18 lists the 16 Oracle ASM disks assigned to disk group
GDWP_ARCH.
Figure 18.
Oracle ASM disks assigned to disk group GDWP_ARCH
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
26
Oracle database recovery – remote site
Overview
This section describes the restore and recovery of the Oracle database at the remote
site, as outlined in Figure 19.
Figure 19.
DD660 setup
Database restore and recovery process
Network settings
When the appliance arrived at the remote data center, the IP address had to be
changed to enable connectivity to the remote network. Link aggregation was already
in place, so this was simply a case of changing the IP address of the virtual interface,
as shown in Figure 20.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
27
Figure 20.
Reset the IP address
Unlock file system
The Data Domain file system was locked during transit and could not be enabled
without the passphrase, as indicated in Figure 21. The passphrase was sent to Cork
separately from the DD660 appliance.
Figure 21.
Data Domain file system enable refused—passphrase required
The filesys encryption unlock command allows the user to enter the passphrase to
unlock the file system, as shown in Figure 22.
Figure 22.
Unlocking and enabling the Data Domain file system
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
28
RMAN restore
With the Data Domain file system unlocked, the database was then restored from the
DD660 using RMAN. This process involved the following steps:
1.
Mount the DD660 on the /backup directory, as shown in Figure 23.
Figure 23.
2.
Mounting the DD660 on the remote site server
Create the Oracle database and build the RMAN recovery catalog from the
export file held on the DD660. Figure 24 shows a summary listing of the
database and control file backups from the catalog.
Figure 24.
Summary of database and control file backups
3.
Use RMAN to connect to the target database and the recovery catalog.
4.
Start up the Oracle instance without mounting the database (nomount mode),
as shown in Figure 25.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
29
Figure 25.
5.
Restore the backup control file to the control_file location specified in the
SPFILE, as shown in Figure 26.
Figure 26.
6.
Starting up the Oracle instance in nomount mode
Restoring the backup database control file
Allocate the channels and restore the database, as shown in Figure 27.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
30
Figure 27.
7.
Database restore
Mount and open the database, as shown in Figure 28.
Figure 28.
Mounting and opening the database
In this use case, further recovery was not required as the database had been
backed up in a consistent state, a backup control file was used, and no
further archive logs needed to be applied.
8.
Re-create and size the Oracle temp files for the temporary tablespaces TEMP
and TEMP1. The SQL commands used to do this are shown in Figure 29.
Figure 29.
Re-creating the Oracle temp files
This completed the successful restore and recovery of the database and provided the
first step in validating the integrity of the database.
The total time taken for the restore and recovery operations was 27 hours, 8 minutes.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
31
Verifying the
integrity of the
database
Figure 30 shows a breakdown of the restored database objects and files and their
mappings to the Oracle ASM disk groups.
Figure 30.
Mapping of Oracle database objects to Oracle ASM disk groups
As part of the discovery process, EMC IT supplied the Cork team with Automatic
Workload Repository (AWR) reports from the production database. A number of key
SQL queries in the reports were identified to verify the integrity of the restored
database. At the remote site, each of these queries was extracted from the AWR
reports and run against the restored database.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
32
Figure 31 demonstrates this process for one of the queries. The first item is an extract
from the SQL ordered by Gets section of an AWR report. The second item is an extract
from the AWR report for SQL ID: 0v7knzwvnc9u7. The third item shows the Oracle
TKProf output of the trace file gathered for this SQL query when it was run against the
restored database.
SQL
query ID
SQL
statement
Figure 31.
Testing the integrity of the restored database
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
33
Data Domain consolidation – remote site
Overview
The Solutions Center in Cork uses a shared Data Domain DD880 for consolidated
backup on many projects within the lab. To ensure consistency, and to maintain a
copy of the EMC IT Production DB in its original state, Data Domain Replicator was
used to copy the database image to the DD880 at the end of the project, as shown in
Figure 32. The DD660 was then repurposed to another project.
Cork Solutions Center
EMC Data Domain
DD660
DD Replicator
EMC Data Domain DD880
Cork Solutions Center shared
backup target
Figure 32.
Database consolidation to shared DD880
Data Domain Replicator software provides automated, fast, and reliable replication of
data for disaster recovery (DR), remote office data protection, and multiple site tape
consolidation. Replicator duplicates the compressed, deduplicated data over a
network (including WANs), greatly reducing the demands on the network.
Once replication has been configured between a source and a destination, any new
data written to the source can be automatically replicated to the destination—for the
use case, a one-off copy was made to the DD880 but continued replication was not
required.
Replication
process
Setting up replication between the DD660 and DD880 involves simply creating a
replication pair by specifying the replica source and destination. This can be done
using the DD Enterprise Manager, as shown in Figure 33, or using the DD OS CLI.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
34
Figure 33.
Creating a replication pair
The Replication view in DD Enterprise Manager provides configuration, status,
performance, and topology information for the replication pair. Figure 34 shows this
information for the use case replication process at completion.
Figure 34.
Data Domain Enterprise Manager—Replication view
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
35
Figure 35 shows the timeline for the replication process, which took 23 hours to
complete.
Figure 35.
Data Domain replication timeline
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
36
Conclusion
Summary
This white paper captures and describes the process used to transport a 12 TB Oracle
data warehouse between two EMC data centers and to restore and recover the
database at the target site. The data centers were located on different continents and
over 4,000 kilometers apart.
Data security was a critical requirement. If the transport mechanism was not secure,
then it would not meet EMC corporate governance compliance and the move could
not happen. In addition, as the process involved a real-world data warehouse, the
solution had to be nondisruptive to EMC core production activities.
The large size of the database and the long distance involved meant that:
•
Transfer over a shared WAN link was neither practical nor efficient
•
Creating a dedicated WAN link was neither practical nor economical
In comparison, backing up the data to a Data Domain DD660 appliance, and
transporting this by commercial carrier to the target site, proved to be a secure,
practical, economical, and efficient solution.
At the primary site, EMC TimeFinder enabled offloading of the backup process to a
non-production server, and using an NFS mount allowed simple RMAN backup and
restore on the DD660.
Findings
The key findings of this solution include:
•
Nondisruptive backup
EMC TimeFinder and seamless Data Domain integration with Oracle RMAN
ensured zero impact to the production environment.
•
Security and corporate governance compliance
Data Domain data integrity, Data Domain Encryption, and Data Domain file
system encryption locking secured the system, enabling it to be shipped by
standard commercial transport.
•
Simple restore and recovery
Using an NFS mount allowed simple RMAN backup and restore from disk. Data
Domain technology ensured balanced backup and restore speeds.
The 12 TB data warehouse was available for use within 3 days of arrival in Cork.
•
Economic transport
Data Domain deduplication reduced the database storage and transport
requirements from 12 TB to 3.3 TB—a small form factor DD660 (2U, 60 kg)
minimized shipping costs.
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
37
References
White papers
Product
documentation
Other
documentation
For additional information, see the white papers listed below.
•
EMC IT's Migration to the Open, Expandable Oracle BI Grid — Applied
Technology
•
EMC Backup and Recovery for Oracle Database 11g Data Warehouse — A
Detailed Review
For additional information, see the product documents listed below.
•
EMC DD OS 4.9 Administration Guide
•
EMC Solutions Enabler Symmetrix CLI 7.2 Command Reference
For additional information, see the documents listed below.
•
Oracle Clusterware Administration and Deployment Guide 11g Release 1 (11.1)
•
Oracle Real Application Clusters Administration and Deployment Guide 11g
Release 1 (11.1)
•
Oracle Clusterware Installation Guide 11g Release 1 (11.1) for Linux
•
Oracle Database Installation Guide 11g Release 1 (11.1) for Linux
•
Oracle Real Application Clusters Installation Guide 11g Release 1 (11.1) for
Linux and UNIX
•
Oracle Database Backup and Recovery Reference 11g Release 1 (11.1)
•
Oracle Database Backup and Recovery User's Guide 11g Release 1 (11.1)
Remote Site Recovery of Oracle Enterprise Data Warehouse Using EMC Data Domain
A Detailed Review
38