Download larger block size

VERITAS Confidential
Bart Drewes
Product Manager
VERITAS Software
VERITAS Confidential
Inside the Black Box
Tuning Oracle E-Business Suite
Applications
VERITAS Confidential
Agenda
• Introduction/ERP Overview
• Success Criteria for ERP Performance
Management
• Some Real World Experiences
• Starting Point to ERP Tuning
• Closing Summary
VERITAS Confidential
Why ERP Applications?
• Shorten the time and expense (theoretically!) required to
deliver high quality business solutions
• Buy (vs. build) functionality whenever and wherever
possible
• Avoid the expense and risk of building (and supporting)
applications from scratch
• Allow focus on core business functions
• Minimize efforts which dilute core competency
• Maintain competitiveness through employment of latest
technology
VERITAS Confidential
ERP Application Characteristics
• Wide-ranging core functionality that can be
customized to meet specific customer requirements
• Support for multiple platforms to accommodate
today’s heterogeneous computing environments
• Modular design to enable customers to “mix &
match” application components to fit their functional
business requirements
VERITAS Confidential
A Need for Performance
• Because ERP solutions are typically implemented to
automate mission-critical core business functions,
the requirement for high performance and
throughput must be met
• E-business modules further increase the exposure of
poor performance
• ERP-based applications inherently tend to be the
most transaction – and throughput – intensive of
your entire portfolio
VERITAS Confidential
Built-in Performance?
• Well…
Unfortunately, the requirement of high performance
cannot be explicitly designed into the application as
a core functional attribute
• But why not…?
The problem stems from the fact that each customer
environment is inherently different – different
customizations, usage patterns, data volumes, etc.
VERITAS Confidential
Performance Limitations from ERP
Design
• Wide-ranging, customizable core functionality
– Each customer/site will customize uniquely
• Support for multiple platforms
– Tuning is often platform- and release-specific
• Modular design in support of “mix & match”
– Modular application components
– Modular architectures
VERITAS Confidential
Complex, Multi-tier Architectures
Web Server
Application
Database
USER
URL
HTML
JPEG
GIF
JSP
“The More Tiers,
the More Tears!”
VERITAS Confidential
Storage
JSP
EJB
SERVLET
JDBC
SQL
SQL
INDEX
TABLE
DATAFILE
LVM
I/O
I/O
CHANNEL
LUN
DISK
Common Perception on ERP Tuning
•
•
•
•
•
•
It’s no use
Vendors do not allow
How can you tune black boxes?
Changes cause more problems
No way to measure performance gain
Would never catch up with dynamic business
changes
• Just follow the vendor’s best practice guidelines
VERITAS Confidential
Success Factors - Basic ERP
Performance Management
• Holistic view of application performance behavior
• Ability to measure end-to-end response time, and
segment into application tiers
• Continuous performance monitoring with little overhead
• Visibility to application structure
• Correlation of information across application tiers
• Ability to track true user activities across tiers
• Ability to track application module performance across
tiers
• In-context drill down for root cause analysis
• Quick identification of performance bottleneck
VERITAS Confidential
Success Factors - Advanced ERP
Performance Management
• Capture of historical performance data to enable:
–
–
–
–
–
Analysis of problems in the past
Baseline behavior for exception analysis
Trend analysis
Capacity planning
Building of Application knowledge base
• Ability to do proactive performance management
• Simulation for performance impact from changes
• Understanding of application usage patterns in
relation to business requirement changes
VERITAS Confidential
VERITAS i3
INFORM
INSIGHT
Where is the problem?
Alerting
Do I or will I have a
performance problem?
Performance
Warehouse
Client
Network
Web
Servers
Reporting
How is performance
tracking over time?
VERITAS Confidential
INDEPTH
How do I fix the problem?
App
Servers
DB
Servers
Storage
Case Study
The System
• Oracle Financials 11i running on a 4-tier architecture
• Two Form Servers balanced by the Forms Metric Server
• Eight worldwide offices are connected to the system
Web Clients
VERITAS Confidential
Internet
Web Server
Form Servers
Oracle Database
Server
StartPoint shows the system
architecture. It provides us with a
dashboard view of the system’s
health
Each sphere
represents one tier
of the architecture.
The atmosphere
represents the
alerts that relate to
the tier
We detect a
performance
alert in the
Oracle
Application
server
VERITAS Confidential
16
‘Top Activity’ alert is part of the
‘Performance’ alert metric set. It
indicates that an OA activity
response time has breached the
performance threshold
VERITAS Confidential
We were able to drill down
‘in context’ to find the
problematic Form that
breached the response
time.
We found the Transaction
form
This form exceeded the
response time that we set
for it by 15% of its normal
response time
VERITAS Confidential
We are looking at the Oracle Applications Savvy in
the Oracle Apps tier
We see most of the processing time
is in the Oracle Database tier
We have the database users that
called the Transaction form
VERITAS Confidential
We are in the Oracle Workspace
This displays the Oracle Activity associated with the
Transaction Form
SQL Statements associated
with Transaction Form.
One SQL statement has
high response time
VERITAS Confidential
We selected the most time consuming SQL statement and
looked at it using the Over Time view
It has increased dramatically.
Let us drill down in context on this statement
VERITAS Confidential
VERITAS Confidential
We have drilled down and are able to see the Form,
the Users, the longest running statements
We see this statement is the eight longest running
statement over the long term
VERITAS Confidential
The response time of this SQL statement has
increased only recently
VERITAS Confidential
Let us look at the most recent Execution Plan
VERITAS Confidential
We uncover that the table RA_CUSTOMER_TRX_ALL has
recently grown significantly. This table is accessed by the long
running query
VERITAS Confidential
The Over Time graph verifies this
VERITAS Confidential
VERITAS Confidential
VERITAS Confidential
These are the columns in the Index
VERITAS Confidential
The number of Distinct values have not
changed although the table size has
significantly increased
VERITAS Confidential
Find the SQL statement that
inserted values in the
RA_CUSTOMER_TRX_ALL table
VERITAS Confidential
VERITAS Confidential
Look at the behavior of the Insert
statement over time.
This statement consumed major I/O
resources on the 31st August
VERITAS Confidential
To find what activated the Insert statement we
associate it to the Forms. It show us that it is
the RAXTRX.exe (Autoinvoice Import Program)
VERITAS Confidential
To improve the situation we use the
SmarTune feature and receive
index recommendations
VERITAS Confidential
Simulate Changes check the affect of
adding the index, both positive or
negative for all statements
VERITAS Confidential
Verify the changes by looking at the
response time statistics of the
original problematic statement
VERITAS Confidential
The Over Time graph verifies that the
response time has returned to the normal
value after the index implementation
VERITAS Confidential
The Over Time graph verifies that the
response time has returned to the normal
value after the index implementation
VERITAS Confidential
Alerts are gone
VERITAS Confidential
Case Study
• Detect
–
Identify an upsurge in form response time
• Find
–
Problem is associated with slowed down performance of the
database
• Focus
–
An SQL statement has deteriorated in performance due to
increases in table size
• Improve
–
A suitable index was introduced
• Verify
–
VERITAS Confidential
The index has improved SQL time, and consequently the
performance of the problematic form
Starting Point To ERP Tuning
• Majority of performance challenges end up in
database tier
• Start measuring and collecting performance
indicators ASAP
• Set reasonable targets based on measurable
performance indicators
• Identify top 10 tuning targets (by user, by
business transaction, or other application entity)
VERITAS Confidential
Large Return Recognizable Through
Database Tuning - Oracle as Example
• Object Design and Maintenance
• Query Optimization
– Object statistics
– Indexing
• Environmental/Configuration Considerations
• Lifecycle Issues
VERITAS Confidential
Object Design and Maintenance — DOs
• Review base table and index growth - monitor extents
and fragmentation - frequently; preferably weekly
• If possible, anticipate (and plan for) occasional rebuilds
of your database (after lots of testing)
• Consider a larger database block size - 8K is a good
nominal value for large sites/implementations
• Make sure your busy tables and indexes are loadbalanced for optimal I/O (try to isolate large transactionintensive objects in their own tablespaces)
• Try to educate your users on the importance of workload
management - distribute log files and teach them how to
interpret them
VERITAS Confidential
Object Design and Maintenance —
DOs (II)
• Pay special attention to temporary and interface
tables - monitor their size very closely, and
truncate them as often as possible to reclaim
space and reset the high water mark in the
header block
– This can have a dramatic effect on full table scans!
• Monitor your index usage patterns (more on this
later)
VERITAS Confidential
Object Design and Maintenance —
DON’Ts
• Never accept the default storage parameters
– Diligent review should be standard policy
• Don’t expect “linear” growth
–
–
–
–
–
VERITAS Confidential
Data volume
Transaction volume
Server capacity
Users’ expertise
User demand on performance
Query Optimization
• Cost-based optimization is prevalent
– Tables must be ANALYZEd periodically to maintain
accurate data distribution statistics
– Check standard ANALYZE utility scripts packaged
with ERP administration function to ensure sufficient
sample size for ESTIMATE option
– Histogram for certain tables may be critical for
success
• Track explain plan change over time from object
statistics changes
VERITAS Confidential
Indexing – Most Power Tuning Key
• Heavy indexing is a common design feature –
–
–
–
–
–
Result from implicitly “generic” design approach
Monitor usage
Evaluate search efficiency
Informed decision to drop indexes
Informed decision to add appropriate indexes
• Can’t change SQL text from standard ERP
modules -- but can influence optimizer with
proper index design
VERITAS Confidential
Indexing (II)
• All “standard” tuning practices with regard to indexes
still apply in ERP environment:
– Try to enhance clustering whenever and wherever
possible (via reloads or CLUSTER creation)
– Re-organize indexes as frequently as is feasible
– Capitalize on larger block size (as previously
mentioned) to reduce b-tree depth
– Physically separate indexes from base table separate spindles, not just LVs
– Keep your statistics as current as possible to enhance
the effectiveness of CBO hints
VERITAS Confidential
Environmental/Configuration
Considerations
• Don’t “under-instance”, i.e., make sure you have enough
Apps. database instances to effectively manage your
entire lifecycle (more on this later)
• If possible, try to maintain pseudo-equivalence between
instances, i.e., make your test instances as close to
production as possible, ideally with on-demand or short
latency refresh capabilities
• Size your systems with absolutely as much physical
memory as you can afford - and then immediately budget
for more!
VERITAS Confidential
Environmental/Configuration
Considerations (II)
• Try to keep ad-hoc and OLAP reporting queries off your
production instance; consider a short-latency reporting
instance, preferably on a separate machine
• “Pin to Win!” - Design a package pinning strategy
• Adjust the PCTFREE parameter for static data, such as
financial data for closed accounting periods
• For tables which are subjected to high concurrent
INSERT activity, consider building them with more than
one free list
• Tables with high UPDATE activity may benefit from a
higher INITRANS value, although this does increase
block-level overhead
VERITAS Confidential
Environmental/Configuration
Considerations (III)
• Some influencing INIT.ORA parameters:
– CURSOR_SPACE_FOR_TIME - when set to TRUE, prevents
private SQL statements in the library cache from being deallocated until application cursor is closed; use to enhance online transaction performance, bear in mind that you will probably
want to increase the size of the shared pool library cache and
the number of open cursors (watch out for ORA-01000 errors)
– DB_FILE_MULTIBLOCK_READ_COUNT - set to a minimum of
32; this will greatly enhance the performance of full table scans
(remember that the maximum for this value is OS-dependent,
and calculated as OS maximum IO size/database block size)
VERITAS Confidential
Lifecycle Issues
• Formalize your change management policies and
practices before you go live; some bureaucracy is not
necessarily a bad thing...
• Make sure you have a very thorough code/design review
with your implementation consultants before they walk
out the door - document everything! - and don’t forget
database object parameters
• If you are going to do any post-production
customizations, make sure you have at least 3
(preferably full-sized) instances for migration development, test, and QA
– Need a separate instance specifically for validating patches
VERITAS Confidential
Lifecycle Issues (II)
• Remember - new/changed code will impact your
maintenance schedule as well (i.e., defrags,
rebuilds, extent monitoring, etc.)
• Tricks of the Trade - you might be able to justify
the additional capacity required by your full-scale
development/testing database(s) as part of a
disaster recovery strategy
VERITAS Confidential
QUESTIONS
VERITAS Confidential
ANSWERS
VERITAS Confidential