Download Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 1

1
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Performance Tuning: 10 New
Active Session History and
Automatic Workload
Repository Tips
Deba Chatterjee
Principal Product Manager
3 Types of
Performance
Management
Reactive
Performance
Management
3
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Proactive
Performance
Management
Preventive
Performance
Management
4
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Reactive Performance Management
Comparing Performance
Across Two Time Periods
Database Hang Analysis
SQL Performance Analysis –I
SQL Performance Analysis -II
5
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Comparing Performance Across
Two Periods
 Performance was fine yesterday, today my
application is really slow?
 Inconsistent Performance
– Over utilization of system resources
– High load ad hoc query consuming resources
– Change in execution plan of query
– Parallel execution downgrade
6
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Tip: Compare Period ADDM
SQL Commonality
AWR Snapshot
Period 1
Regressed SQL
I/O Bound
AWR Snapshot
Period 2
Compare
Period ADDM
Analysis Report
• Full ADDM analysis across two AWR snapshot periods
• Detects causes, measure effects, then correlates them
• Causes: workload changes, configuration changes
• Effects: regressed SQL, reach resource limits (CPU, I/O, memory, interconnect)
• Makes actionable recommendations along with quantified impact
7
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Undersized SGA
Compare Period ADDM: Method
Identify what changed
• Configuration changes, workload
changes
Quantify performance differences
• Uses DB Time as basis for
measuring performance
Identify root cause
• Correlate performance
differences with changes
8
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
•
30% smaller Buffer cache
•
10% new SQL
•
Top SQL increased 45%
•
Read I/O up 55%
•
Buffer cache reduction
caused read I/O increase
Reactive Performance Management
Comparing Performance Across
Two Time Periods
Database Hang Analysis
SQL Performance Analysis – I
SQL Performance Analysis – II
9
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Compare
Period ADDM
Database Hang Analysis
 My database is hung? I do not want to
bounce it.
 Database Hung state
– Blocking Sessions
– Memory allocation issues
– Library cache issues
– Unresponsive Storage (ASM)
– Interconnect problems
10
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Tip: Real-Time ADDM
EM Agent
Deadlocks
Diagnostic Connection
Unresponsive DB
Hangs
JDBC Connection
Latches
Real-Time Analysis
Database
•
•
•
•
ADDM
Analysis
Uses a pre-established diagnostic connection for unresponsive systems
Initiates a standard JDBC connection for real-time analysis
Diagnostic connection collects data without holding latches or running SQL
First intelligent advisor to diagnose problems in real-time as they occur, no matter
how sick the system is
11
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Real-Time ADDM
•
•
•
•
•
12
Real-time analysis of hung or slow database systems
Holistically identify global resource contentions and deadlocks
Quantified performance impact
Precise, actionable recommendations
Provide cluster-wide analysis for RAC
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Reactive Performance Management
Comparing Performance Across
Two Time Periods
Compare
Period ADDM
Database Hang Analysis
Real-Time
ADDM
SQL Performance Analysis – I
SQL Performance Analysis – II
13
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
SQL Performance Analysis
 I enabled parallel query, yet this query is taking
so long. Can you take a look?
 Parallel Downgrades
– Uncontrolled parallel execution
– Parallel Server availability
– Object level settings
– Session level settings
14
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Tip: Real-Time SQL Monitoring
Insert Executed with Parallel Hint
15
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Real-Time SQL Monitoring
Parallel Tab
• Parallel Coordinator busy for the entire duration!!
16
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Real-Time SQL Monitoring
Enabled Parallel DML
• Parallel Slaves busy for the entire duration!!!
17
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
SQL Performance Analysis
 My simple query takes so long. What’s wrong
with the database?
 SQL Performance Problems
– Statistics
– Resources
– Application issues
– Parallelism
– Initialization parameters
18
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Tip: Real-Time SQL Monitoring
SQL with Count and Group By
19
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Real-Time SQL Monitoring
SQL with Count and Group By
20
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Real-Time SQL Monitoring
PGA Size Increased
21
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Reactive Performance Management
Comparing Performance Across
Two Time Periods
Compare
Period ADDM
Database Hang Analysis
Real-Time
ADDM
SQL Performance Analysis – I
SQL Performance Analysis – II
22
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
SQL
Monitoring
23
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Proactive Performance Management
4.Proactively Monitoring Long
Running Programs
5.Analyzing Transient
Performance Problems
Understanding Workload Profile
6. Correlating ASH & AWR
7. Using ASH Analytics
24
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Reactive Tracing of long running
programs ?
 Can you trace my program ?
 What is wrong with tracing ?
– A very reactive way of looking at problems
– Overhead of writing data to trace files
– Programs we want to trace are usually the ones
with issues
– Impacts the performance of the production
system
25
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Monitoring Complex Database Operations
What’s Really Happening inside the Database
Challenge
• SQL & PL/SQL
Monitoring only
monitors a single
execution
• How does a DBA
monitor a composite
operation such as a
batch job?
26
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Solution
• Real-Time Database
Operations Monitoring
• Benefit:
Allows DBAs to
analyze and tune
complex composite DB
operations
Real-Time Database Operations Monitoring
Know What’s Happening and Resolve Issues Faster.
• Database monitoring of application jobs
• Grouping of SQLs, sessions for the application jobs
• Key scenarios: ETL operations, Quarter End Close jobs
• Real-time monitoring driven by application specified tagging
• Oracle Data Pump jobs automatically monitored
• Tagging ability in PL/SQL, OCI, JDBC
• Avoids the overhead of SQL*Trace
• Visibility of top SQL statements, system and session performance
metrics
27
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Naming a Database Operation
How to Setup a DBOP ?
– Naming or Bracketing
– Tagging
EXPLICIT
BEGIN_OPERATION
SQL
PL/SQL Blocks
…
SQL
SQL
END_OPERATION
28
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
IMPLICIT (For Java & OCI)
DBOP (Tag)
SQL
PL/SQL Blocks
…
SQL
SQL
Monitor Composite Database Operations
• Oracle Database 11g: Support for simple DB operations
• PL/SQL procedures/functions
• Oracle Database 12c: NEW support for composite operations
• Session(s) activity between 2 points of time defined by application code or DBA
• For example; SQL*Plus script, batch job, or ETL processing
• At most one DBOP per DB session
29
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Proactive Performance Management
4.Reactive Tracing of Long
Running Programs
5.Analyzing Transient
Performance Problems
Understanding Workload Profile
• 6. Correlating ASH & AWR
• 7. Using ASH Analytics
30
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Database
Operations
Analyzing Transient Performance
Problems
 What happened last night the batch job took
twice the time to finish ?
 No way to detect transient issues
– We look at AWR data
 Averaged out over the snapshot window
– On-disk ASH Data
 Sampled every 10 seconds
– Very difficult to detect such issues in the “past”
31
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Automatic Performance Diagnostics
The ADDM Family:
A Continuous Evolution in Database Performance Management
ADDM
Compare
Period ADDM
• Diagnose persistent • In-depth
performance issues
performance
comparison across
• Uses AWR
two periods
snapshots
• Automatically runs
every hour
32
Real-Time
ADDM
• Hung or extremely
slow databases
• Uses AWR data
• Uses a normal and
diagnostic mode
connection
• Manually triggered
• Manually triggered
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Enhanced
Real-Time ADDM
• Proactively detect &
diagnose
performance spikes
• Uses in-memory
data
• Automatically runs
every 3 seconds
Enhanced Real-Time ADDM
Database self-monitors for serious performance issues
• Proactive problem detection & analysis
• Very light weight (in memory, latchless) check runs every 3 seconds
• When detects bad performance trends, triggers further analysis
• Analyzes High CPU, I/O spikes, memory, interconnect, hangs, deadlocks
• Identifies a problem before it threatens application performance
• For current spikes, Real-Time ADDM can be manually triggered
• For short duration (5-min) performance spikes, i.e. transient, high impact problems
• Actionable advice for critical issues
• Richer data set available for analysis
• Reports (analysis and data) stored in AWR for historical analysis
33
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Triggers for Further Analysis
#
Rule
Condition
1
High Load
Average active sessions greater than 3 times the number of CPU
cores
2
I/O bound
Impact on active sessions based on single block read performance
3
CPU bound
Active sessions greater than 10% of total load and CPU utilization
great than 50%
4
Over-allocated
memory
Allocation over 95% of physical memory
5
Interconnect bound
Single block interconnect transfer time based
6
Session Limit
Session limit close to 100%
7
Process Limit
Process limit close to 100%
8
Hung Session
Significant number of hung sessions. If this number is greater than
10% of total sessions
9
Deadlock Detected
Any deadlock detected by hang analyzer
34
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Proactive Performance Management
4.Reactive Tracing of Long
Running Programs
Database
Operations
5.Analyzing Transient
Performance Problems
Real-Time
ADDM
Understanding Workload Profile
6. Correlating ASH & AWR
7. Using ASH Analytics
35
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Understanding Workload Profile
 The SQL Response Metric crossed the warning
threshold. What is wrong?
 Several factors can impact SQL Response time
– Increased or unusual load on system
– Hardware Issues
– Runaway queries consuming system resources
– Changes in execution plans
– Missing or stale object statistics
 Need a mechanism to quickly analyze in-memory
performance data
36
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Database Response Time Analysis—AWR
 AWR top 5 section
shows the Wait Class
which contributes most
to DB wait time
 Foreground Wait Class section in
AWR to see distribution of DB waits
over Waits classes

37
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Objects involved in TX row lock
contention can be identified in
Segment Statistics section of AWR
From AWR to ASH
 ASH report for the
period of increase
of Application waits
will show the same
waits as AWR
 Can I get the Application
Module which suffered
from this type of
contention ?
38
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Extracting More Data From ASH
 Identify SQL statements and sessions impacted by waits on
“Application” Wait Class
39
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Extracting More Data From ASH
 Get a list of blocking sessions and DB objects !
40
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Understanding Workload Profile
• Graphical ASH report for advanced analysis
• Different visualizations: Stacked chart or
Tree Map
• Provides visual filtering for recursive drill-downs
• Collaborate with others using Active
• Select any time period for analysis
Reports
• Analyze performance across many dimensions
41
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Proactive Performance Management
42
4.Reactive Tracing of Long
Running Programs
Database
Operations
5.Analyzing Transient
Performance Problems
Real-Time
ADDM
Understanding Workload Profile
6. Correlating ASH & AWR
7. Using ASH Analytics
ASH
Analytics
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
43
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Preventive Performance Management
Prevent Regression After
Upgrade
Ensure Optimal Resource
Allocation
Prevent Performance Issues
Due to Application Changes
44
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Case Study
DATABASE MANAGEMENT
187 Million Members
$252 Million in Revenue
800 Oracle Databases
1,400 Applications
FOCUS ON
Preventive Performance
Management
Challenge:
Using Oracle Enterprise Manager:
• LinkedIn’s ERP systems were being upgraded from
DB 10g to DB11g
• Presence of a large amount of custom code
• Limited time frame to complete the upgrade.
• Management concern about System performance
• Initial testing showed no major problems/concerns
• A week before go-live several potential
showstopper performance issues were noticed.
• Re-writing or tuning several pieces of code was not
feasible in a short window of time.
• Decision to use either SQL Profiles or Baselines to
regress to the DB10g plan in the interim
• Used EM to regress back to the old plan to run a
job that calls the slow performing SQL
• Used SQL Tuning Advisor to drilling down in to the
session and identify the SQL ID
• You can compare the explain plan and see the new
explain plan in the same window
• Click through guided wizard to implement the SQL
profile
• DONE!!!
45
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
SQL Tuning Advisor
Gather Missing or Stale
Statistics
Create a SQL Profile
SQL Profiling
Add Missing Access
Structures
Statistics Analysis
Modify SQL Constructs
Access Path Analysis
SQL Restructure Analysis
Alternative Plan Analysis
Parallel Query Analysis
Automatic Tuning Optimizer
•
•
•
•
SQL Tuning
Advisor
Adopt Alternative
Execution Plan (11.2)
Create Parallel SQL
Profile (11.2)
Administrator
Comprehensive SQL Tuning Recommendations
Analyzes statistics for accuracy
Recommends SQL Profiles for transparent application tuning
Suggest access structures and alternate SQL to speed up query execution
Identifies alternative execution plans using real-time and historical performance
data to recover from plan regression
• Recommends appropriate degree of parallelism for best performance
46
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Preventive Performance Management
Prevent Regression After
Upgrade
Ensure Optimal Resource
Allocation
Prevent Performance Issues
Due to Application Changes
47
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
SQL Tuning
Advisor
Ensure Optimal Resource Allocation
 We are going through a schema consolidation,
how can I ensure one schema / user will not
run away with all my system resources?
 Database resource manager directives prevent
a single user session to run away with all
resources
 Create a resource allocation strategy
 Allocate appropriate CPU and I/O (Exadata)
across consumer groups
48
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Allocating Resources in the Database
49
No Resource
Allocation
•
•
•
Gives maximum flexibility for each Consumer Group
Allows any Consumer Group to consume all available resource
Risky as one Consumer Group can run away with all resources.
Specify a minimum
allocation
•
•
•
•
Ensures all Consumer Group get a specific share of the resources
Allows any Consumer Group to consume any unused resources
Kicks in at 100% resource utilization.
Assumes that not all Consumer Group will use its allocated resources
Specify a minimum
and maximum
•
•
Ensures all Consumer Group get a specific share of the resources
Prevents a Consumer Group from taking more than the maximum value
assigned.
May result in unused capacity
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
•
Tip: Setting up Resource Manager in Oracle Enterprise
Manager
• Extremely simple to manage resource plans using Enterprise
Manager UI
50
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Preventive Performance Management
Prevent Regression After
Upgrade
SQL Tuning
Advisor
Ensure Optimal Resource
Allocation
DB Resource
Manager
Prevent Performance Issues
Due to Application Changes
51
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Prevent Performance Issues Due to
Application Changes
 The new BI system has very aggressive SLAs
defined. How can we ensure consistent
performance across the system?
 Code migration, new indices, objects can often
impact performance of the application
 How do we validate the performance of critical
queries before rolling out these changes ?
52
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Validate
Impact of Custom Code Migration
Central SPA System
Trial 1
Production
State 1
01001011001010100100100100100100100100100100100010010101001001001
00111001001001001001001000010010000010111001001010100100100101010
1001000100000101010010010101010011010100101010010010101001100101
Trial 2






53
Use SPA Guided Workflow (recommended) or PL/SQL APIs
Create a SQL tuning set of the top X (20 or 30) queries
Establish first trial remotely using current state – baseline
Make change – Create the indexes or migrate custom code
Establish second trial remotely using the same SQL Tuning Set
Review SPA report and rollout or rollback changes.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
State 2
Custom
Code Changes
Tip: Take the Guess Work Out!
 Run your trial
before and after
migrating the
change
 Make sure your
most important
queries are not
regressed
 Take the guess
work out
54
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Preventive Performance Management
Prevent Regression After
Upgrade
55
SQL Tuning
Advisor
Ensure Optimal Resource
Allocation
DB Resource
Manager
Prevent Performance Issues
Due to Application Changes
SQL
Performance
Analyzer
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Enterprise Manager at OpenWorld 2013
Session Title
CON1452
CON1590
Advanced Database Performance Analysis Techniques Using SQL Performance
Analyzer
DBA's guide to Data Masking and Data Subsetting
CON1628
Best Practices to Monitor and Manage WebLogic Server: Development to Production
CON1450
CON1589
CON1603
CON1451
Extreme Database Management with the Latest Generation of Database Technology
Advanced Management of Oracle Applications Unlimited and Fusion Applications
Application Performance Matters – Oracle Real User Experience Insight
Zero to Cloud in Hours with Oracle Enterprise Manager 12c
Oracle Performance Tuning Boot Camp: 10 New Problem-Solving Tips Using ASH &
AWR
Managing Heterogeneous Environments with Oracle Enterprise Manager
Oracle's Approach to Application and Infrastructure Testing
Oracle Enterprise Manager DBaaS: Database, Schema, or Terabytes of Data in
Minutes
Deep Dive into Exadata management with Oracle Enterprise Manager 12c
CON1587
CON1604
CON1588
CON1719
CON1453
GEN11423 Total Cloud Control: Oracle Enterprise Manager 12c Overview
CON1602 What's New in Oracle Exalogic Management?
CON1454 Database Lifecycle Management with Enterprise Manager 12c
56
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Date/Time
Tuesday , Jul 23, 2:15 PM , Room 426
Tuesday , Jul 23, 4:45 PM , Room 509
Wednesday, Jul 24, 12:00 PM , Room
420/421
Wednesday,
Wednesday,
Wednesday,
Wednesday,
Jul
Jul
Jul
Jul
24,
24,
24,
24,
12:00 PM , Room 431
12:00 PM , Room 509
2:00 PM , Room 509
2:00 PM , Room 617
Wednesday, Jul 24, 3:15 PM , Room 509
Wednesday, Jul 24, 5:15 PM , Room 509
Wednesday, Jul 24, 8:45 AM , Room 509
Wednesday, Jul 24, 8:45 AM , Room 61
Thursday , Jul 25, 12:00 PM , Room 42
Thursday , Jul 25, 2:00 PM , Blue Hall
Thursday , Jul 25, 9:00 AM , Room 509
Thursday , Jul 25, 9:00 AM , Room 617
57
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
58
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.