Download Lessons learned from managing a large single instance database

Oracle Apps at EMC
© Copyright 2009 EMC Corporation. All rights reserved.
1
Oracle Applications Deployment
11.0.3
11i
Employees, Vendors
(Web, Forms, Citrix)
Employees
(Forms, Web)
Quoting
Pricing
Configurator
Incentive Compensation
Projects
Customer Service
Tele Service
InstallBase
Manufacturing
Finance
Order Entry
SSP
R12
Partners
Japan, China, Korea
Partners, Customers
(Web)
Internet
Internet
Quoting
Configurator
Firewall
Firewall
Load Balancer
Linux RHEL 4 Dell PE Nodes
Web/Forms Internal and External (67)
Concurrent Manager (4)
Solaris10, 144 GB, 36 Dual
Cores
2 Node RAC
Solaris 10, 224 GB, 224 Cores
Web External (9)
Concurrent Manager (3)
3 Node RAC Linux
CXLDBP1
EP01
0.5 TB
CRP01
2 TB
7 TB
Symm DMX 3
Symm DMX 3
Sun Fire 25K
velop and
Design
Product
Linux RHEL VMs
Celera (NAS)
Forms Servers (5)
Sun Enterprise E4500/E5500
Celera (NAS)
Solaris 10
Active – Passive Cluster
Oracle 8 Database
Solaris App Servers
Forms Interface
© Copyright 2009 EMC Corporation. All rights reserved.
Symm DMX 3
DELL R900
Sun Fire 25K
Active – Active 2 nodes RAC
Oracle 10g Database
Linux App Servers
Forms & Web Interfaces
Active – Active 3 nodes RAC
Oracle 11g Database
Linux VMs
Web Interface
2
Oracle Functional Footprint
EMC is leveraging virtually all of the Oracle Applications footprint.
Integrated
Sales
Cycle
Order-to
-Cash
Services
Channel
Management
Partner
Integration
Oracle Apps
(11.0.3, 11.5.10)
SW
Development
Process
Customer
Relationship
Mgmt.
© Copyright 2009 EMC Corporation. All rights reserved.
Commissions
Inventory
Management
Factory
Integration
3
Oracle EBS Infrastructure Design Objectives
The nature of EMC’s business required an extremely available
architecture for the new implementation - during certain periods, nothing
less than 100% availability is satisfactory.
Key design characteristics:

Highly Available (> 99% availability)
–

No Data Loss
–
–

Accomplished with a combination of hIGHLY Available RAC Cluster (DB Tier), Cisco ACE for
the Applications server farm (Application Tier), and Cisco fiber-channel switching (physical
layer)
An abrupt loss of the production site will not result in the loss of any committed transactions
Accomplished through a combination of EMC SRDF and Oracle standby database technology
< 4-hour Recovery Time
–
–
In the event of a site failure, IT must be able to fail over to the remote site and return Catalyst to
operational condition within 4 hours
Accomplished through rigorous attention to scripting and failover event scripting
© Copyright 2009 EMC Corporation. All rights reserved.
4
Oracle E-Business Suite at EMC

Very high usage penetration within the enterprise
–
–




70+ Application Tiers –VM /Linux
Two-node RAC Architecture - 224 cores
Oracle Database 10g R2, Sun Solaris 10
Extremely dense module usage
–
–

One of the largest
deployments of
Oracle eBusiness
Suite in the world.
50,000+ Named Users
4,000+ Concurrent Users at peak periods
Largest users of Service, Install Base and Contracts, Oracle Sales, Configurator
One of the largest users of iStore, Projects, TCA, Order Management, etc.
Application Continuity Tool – keeping 11i available during maintenance
System Statistics
Business Statistics
Quotes
2,263,231
Orders
674,458
DB Size
Number of Rows
Service Requests
9,210,593
IOPS
Service Contracts
1,391,325
Interconnect Traffic
Projects
59,937,589
Material Transactions
12,305,357
Parties
Relationships
Time Cards
9,429,011
10,356,648
7,336,539
© Copyright 2009 EMC Corporation. All rights reserved.
8.8 Billion
15K
3000-4000 Blocks/sec
Daily Volume
110,512
IB instances
7TB
Database Transactions
754/Sec
22 Million/day
Archive Log
1.2 TB
Conc. Jobs
70K
Workflow Events
Workflow Roles
250K
20,193,476
5
Oracle 11i Architecture
Modules
Employees, Vendors
(Web, Forms, Citrix)
Partners, Customers
Quoting
Pricing
Configurator
Incentive Compensation
Projects
Customer Service
Tele Service
InstallBase
Internet
Users
50,000 Users Worldwide
4000 Concurrent Users
Employees and Partners
Technology
Oracle 10g database
Oracle eBusiness Suite (11i)
EMC SAN and NAS Storage
Sun 25K Enterprise Servers (RAC)
Dell PowerEdge Linux App Servers
Cisco Load Balancer
Firewall
Cisco Load Balancer
crm.emc.com
Celera (NAS)
Linux RHEL 4
Linux RHEL 4
Linux RHEL 4
Powerlink
Web/Forms Internal
(53)
Dell PE 2850
Web/Forms External
(14)
Dell PE 2850
eServices
Concurrent Manager
(2)
Dell PE 6850
Catalyst
WebMethods
2 Node RAC
Solaris 10, 224 GB, 224 Cores
Outerbay
Sun Fire 25K
Intergation
Services
CRP01
7 TB
Dialhome
CRI
Configurator
Connectrix
3 Corporation. All rights reserved.
©Symm
CopyrightDMX
2009 EMC
6
Current Technology Stack
Storage
(EMC)
Unix
(Sun, Veritas, EMC)
Database
(Oracle)
Application
(Oracle, Redhat)
ACE
Users
Web/Forms Tier
(67 VM nodes)
HBA
RAC – 2 nodes
CRS
SQL*Net
File Systems
JDBC
JVM
Database
Forms
Instance 2
(CRP01b)
Apache
Instance 1
(CRP01a)
crm.emc.com
Powerpath
Reports
Listeners
Services
Instances
Database
CRS and Rdbms
Veritas odm
Interconnect
Volume Manager
Veritas Cluster
Solaris 10 OS
I/O Fencing
RAID 10 Protection
Concurrent
Concurrent Tier
(4 nodes)
Connectrix
Symm DMX 3 (4500)
Model - Sun 25 K
Oracle 10.2.0.3
EBusiness Suite
720 x 146 GB drives
OS - Solaris 10
Size 7 TB
Linux RHEL 4
95 TB Total RAW
CPUs – 224 Cores
Growth 150 GB / Month
Shared Appltop
RAID 10 Protection
Memory - 224 GB
SGA 30 GB each
PCP Enabled CM
Cache 256GB mirrored
Veritas RAC 5.0
6 Listeners
VMWARE Web
Powerpath 5.0.2
EDL Backup
Dell PE 6850 CM
© Copyright 2009 EMC Corporation. All rights reserved.
Aging component,
candidate for upgrade
7
Future 11i Technology Stack
Database
(Oracle)
CRS
SQL*Net
ASM
JDBC
Symmetrix V-Max
146 FC GB drives
SATA & EFD Drives
95 TB Total RAW
RAID 10 Protection
Cache 256GB mirrored
Model – Cisco UCS
(Nehalem)
OS – RHEL 5.3
CPUs – 2 (test)
Memory - 96GB (test)
Powerpath 5.3
© Copyright 2009 EMC Corporation. All rights reserved.
Listeners
Services
Instances
Database
CRS and Rdbms
10 Gb Interconnect
Oracle ASM
Oracle CRS
RHEL 5.3
Connectrix
Load
Balancer
(ACE)
Users
Forms
JVM
Instance 4
(CRP01d)
Powerpath
I/O Fencing
RAID 10 Protection
RAC – 4 nodes
Apache
Instance 3
(CRP01c)
Database
Web/Forms Tier
(67 nodes)
Instance 1 Instance 2
(CRP01a) (CRP01b)
Cisco UCS
HBA
Application
(Oracle, Redhat)
Concurrent Tier
(2 nodes)
crm.emc.com
Reports
Linux
(RedHat, Cisco, EMC)
Concurrent
Storage
(EMC)
Oracle 11G
EBusiness Suite
Size 7 TB
Linux RHEL 4
Growth 100 GB / Month
Shared Appltop
SGA 30+ GB each
PCP Enabled CM
6 Listeners load balancing
VMware Web
EDL Backup
VMware CM
8
Applications and Database Cloning
Challenge:
•
Copy production database and application code to test environment, apply post clone changes to make it test or
development environment.
Copy process has to be very fast to reduce the test and development downtime.
•
Business Benefits:
•
Solution:
Using Open Replicator to copy the data across SAN
Backup
•
•
2. Open Replicator Replication (4 hours)
Reporting
•
PROD
3. BCV Restore - (4 hours)
1. Backup – BCV (30 mins)
(Proxy)
•
TEST
Only 4 hours Downtime
for TEST or Dev for
Database Copy
Reduce Test / Dev
Database Size during
Clone (Subsetting)
No manual intervention
or scripting
Preserve Apps DBA
Resources
No network Impact
DEV
(7TB)
Production
Test / Dev
4. Post Clone, Migrations
© Copyright 2009 EMC Corporation. All rights reserved.
9
Optimal DR Solution
Westboro
Hopkinton
Data
X
Production
DR
Database Server
Redo
Arch
BCV
Dynamic SRDF
Dynamic SRDF
Dynamic SRDF
RDF Link Personality Switch
Data
Redo
Arch
Production
DR
Database Server
Recover Database
BCV
Roll
Roll
Switch
DR
Break
Fail
Forward
Forward
Database
Site
Database
Database
Data
- Initial
DR
DR
RDF
Failure
Database
to
Database
Build
Roles
DR
Link
Primary Database
Database
is Open Failure
– Read Only
Custom Script isRecognizes
Symetrix/V-Max
Running Failure
CustomArchive
Script is
BCV
Running
is Mounted
Archive Log
Dynamic
SRDF
FilesPersonality
are Registered
Switch
Background
ASM
is Restarted
Process
(Arch
Apply’s
BCV Logs
Hidden)
Alert Log
Redo
andisArch
Monitored
Luns Mount
for Complete/Errors
to Current DR
Recover Database
© Copyright 2009 EMC Corporation. All rights reserved.
10
Production Environments Layout
Disaster Recovery (Hopkinton)
Production (Westboro)
DR
Apply Logs Hourly to Keep in Sync
•
•
•
BCV
BCV
Daily 12AM
Daily 1 PM
•
•
DR
Prod
Reporting1
ACT
Reporting HA Reporting2 ACT DR
Monthly Rel
Initial Build using SRDF
Daily 5 AM
Backup
EMC Disk Library
Shareplex2 Near Real Time
•
Symmetrix DMX 3 (4500)
720 drives, 146 GB each
95 TB Total RAW
RAID 10 Protection
Symmetrix DMX 3 (4500)
720 drives, 146 GB each
95 TB Total RAW
RAID 10 Protection
30 days retention
•
•
Perf Test
Shareplex 1 Near Real Time
•
Zero data loss solution
Minimal overhead to
production
Automated recovery
No DB block corruption
DR cab be opened as
read only
DR Hardwar for
performance testing
Fastest backup method
No performance impact
to Prod during backup
Database validation
before backup
90 days retention
Tape Library
© Copyright 2009 EMC Corporation. All rights reserved.
11
Sky Rocketing Growth
Impact of Growth
• Slow Performance
• Infrastructure Costs
• Continuously adding storage
• Additional CPU and Memory
• Maintenance Costs
• Resource Costs
• DBA and development
resources
• Storage and Unix resources
© Copyright 2009 EMC Corporation. All rights reserved.
12
Storage Multiplier -2008
153 TB
Tape Backup of Prod
90 TB
63 TB
Backup of (Prod, Splx, Dev, Test, etc)
Onto EDL with RAID
28 TB
35 TB
3TB (Dev, Test, Training, Perf, etc) RAID
32 TB
12TB (Dev, Test, Training, Perf)
20 TB
5TB (Prod, Splx, SBY, ACT, Bkup) Mirror
15 TB
5TB (Prod, Splx, SBY, ACT, Bkup) DR
10 TB
5TB (Prod, Splx, SBY, ACT, Bkup) Mirror
5 TB
5TB (Prod, Splex, SBY, ACT, Bkup)
Oracle 11i Multiplier
1 TB of Data
© Copyright 2009 EMC Corporation. All rights reserved.
13
Application Data ILM - Strategy
Nearline Database
© Copyright 2009 EMC Corporation. All rights reserved.
14
Storage Multiplier Effect
153TB
Tape Backup of Prod
90 TB
63TB
Backup of (Prod, Splx, Dev, Test, etc)
onto EDL with RAID
28 TB
35TB
3 TB - Dev, Test, Training, Perf, etc RAID
32TB
12 TB - Dev, Test, Training, Perf
20TB
5 TB - Prod, Splx, SBY, ACT, Bkup Mirror
15TB
5 TB - Prod, Splx, SBY, ACT, Bkup DR
10TB
5 TB - Prod, Splx, SBY, ACT, Bkup Mirror
5TB
5 TB - Prod, Splx, SBY, ACT, Bkup
Oracle 11i Multiplier
1 TB of Data
© Copyright 2009 EMC Corporation. All rights reserved.
15
Storage Optimization – Archiving / Subsetting
Current
Year 2008
153 TB
Decomm of 3 Environments
Elimination of Tape Backups
Reduce Backup Retention
Rman Backup
Tape Backup (90)
EDL Backup (30)
Non Prod (12)
Disaster Recovery (5)
Production (5)
64 TB
40TB
Subsetting
EDL Backup (30)
Non Prod (9)
Disaster Recovery (5)
Production (5)
EDL Backup (30)
Non Prod (9)
Disaster Recovery (5)
Production (5)
Oracle 11i Multiplier Effect – 1 TB
© Copyright 2009 EMC Corporation. All rights reserved.
16