Download Document

1
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Automatic Data Optimization
and Oracle Intelligent
Storage Protocol
Kevin Jernigan
Senior Director Product Management
Oracle Product Development
Program Agenda
 Data Growth and Information Lifecycle Management
 Database Partitioning and Compression
 Heat Map and Automatic Data Optimization
 Automatic Data Optimization Use Cases
 Oracle Intelligent Storage Protocol and ZFS Storage
3
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Growth in Data Diversity and Usage
1,800 Exabytes of Data in 2011, 20x Growth by 2020
Today’s Drivers
Enterprise
45% per year growth
in database data
Cloud
80% of new
applications
and their data
Regulation
300 exabytes in
archives by 2015
4
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Emerging Growth Factors
Mobile
#1 Internet access device in 2013
Big Data
Large customers top 50PB
Social Business
$30B/year in commerce by 2015
Information Lifecycle Management
Managing Data Over its Lifetime
“The policies, processes,
practices, and tools used to align
the business value of information
with the most appropriate and
cost effective IT infrastructure
from the time information is
conceived through its final
disposition.”
Value at Risk
High
Value
Medium
Value
Low
Value
Storage Networking Industry Association
(SNIA) Data Management Forum
$$ $
$$$
Total Cost of Ownership (TCO)
5
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Information Lifecycle Management
• Improve DW and
batch performance
• Define
by 10X or more with
Data Classes
Partitioning and
•
*Structured
and
Compression
Unstructured
• Improve OLTP
performance with
Compression and
Flash Cache
• Reduce TCO for
database storage
by 10X or more with
Compression
• Automate tamper• Define and Enforce
proof history
Compliance
tracking with
Policies
Flashback Data
Archive
6
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Active
Mainframe Desktop
Servers
Data Optimization
Hardware Optimization
Software Optimization
Application Optimization
Less
Active
Historical
• Create Storage
Tiers for the Data
Classes
• Reduce Storage
Requirements with
Compression:
 2X – 4X with ACO
 10X – 50X with HCC
• Create Data Access and
Migration Policies
Use ZFS, SAM,
and StorageTek
tape solutions
Database Partitioning
7
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
The Concept of Partitioning
Simple Yet Powerful
ORDERS
ORDERS
ORDERS
USA
EUROPE
JAN
FEB
FEB
Large Table
Partition
Composite Partition
Difficult to Manage
Divide and Conquer
Better Performance
Easier to Manage
More flexibility to match
business needs
Improve Performance
Transparent to applications
8
JAN
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
What Can Be Partitioned?
 Tables
– Heap tables
Global Non-Partitioned Index
Global Partitioned Index
– Index-organized tables
 Indexes
– Global Indexes
Table
Partition
Table
Partition
Table
Partition
– Local Indexes
 Materialized Views
Local Partitioned Index
9
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Table
Partition
Partition for Performance
Partition Pruning
Q: What was the total sales for
the weekend of May 20 - 22
2008?
SELECT sum(sales_amount)
FROM sales
WHERE sales_date BETWEEN
to_date(‘05/19/2008’,’MM/DD/YYYY’)
AND
to_date(‘05/22/2008’,’MM/DD/YYYY’);
•
Only relevant partitions will be accessed
•
•
•
Minimizes I/O operations
•
10
Static pruning with known values in advance
Dynamic pruning at runtime
Provides massive performance gains
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Sales Table
May 18th 2008
May 19th 2008
May 20th 2008
May 21st 2008
May 22nd 2008
May 23rd 2008
May 24th 2008
Partition for Tiered Storage
ORDERS TABLE (10 years)
2003
95% Less Active
Low End Storage Tier
2-3x less per terabyte
11
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
2012
2013
5% Active
High End Storage Tier
Database Compression
12
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Compression Techniques
COMPRESSION TYPE:
SUITABLE FOR:
Basic Compression
“Mostly read” tables and partitions in Data Warehouse
environments or “inactive” data partitions in OLTP
environments
Active tables and partitions in OLTP and Data
Warehouse environments
Non-relational data in OLTP and Data Warehouse
environments
Advanced Row Compression
Advanced LOB Compression and
Deduplication
13
Advanced Network Compression and
Data Guard Redo Transport Compression
All environments
RMAN/Data Pump Backup Compression
All environments
Index Key Compression
Indexes on tables for OLTP and Data Warehouse
Hybrid Columnar Compression –
Warehouse Level
“Mostly read” tables and partitions in Data Warehouse
environments
Hybrid Columnar Compression – Archive
Level
“Inactive” data partitions in OLTP and Data Warehousing
environments
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Advanced Row Compression
• Partition/table/tablespace data compression
– Support for conventional DML Operations (INSERT, UPDATE)
– Customers indicate that 2x to 4x compression ratio’s typical
• Significantly eliminates/reduces write overhead of DML’s
– Batched compression minimizes impact on transaction performance
• “Database-Aware” compression
– Does not require data to be uncompressed – keeps data compressed
in memory
– Reads often see improved performance due to fewer I/Os and
enhanced memory efficiency
14
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Hybrid Columnar Compression
Compression Unit
• Hybrid Columnar Compressed Tables
– Compressed tables can be modified using conventional
DML operations
– Useful for data that is bulk loaded and queried
• How it Works
10x to 15x
Reduction
15
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
– Tables are organized into Compression Units (CUs)
• CUs are multiple database blocks
– Within Compression Unit, data is organized by column
instead of by row
• Column organization brings similar values close
together, enhancing compression
Data Compression
Reduce storage footprint, read compressed data faster
Hot Data
Warm Data
Archive Data
111010101010101
10101010111010100110101
001101010101011
11000010100010110111010
010001011011000
10100101001001000010001
110100101000001
01010110100101101001110
001110001010101
00010100100101000010010
101001011010010
00010001010101110011010
110001010010011
111001001000010
001010101101000
101010101110101
001101011100001
010001011011101
010100101001001
000010001010101
101001011010011
100001010010010
100001001000010
001010101101001
101010101110101
001101011100001
010001011011101
010100101001001
000010001010101
101001011010011
100001010010010
100001001000010
001010101101001
3X
10X
15X
Advanced Row Compression
16
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
10101010111010100110101110000101000101
10111010101001010010010000100010101011
01001011010011100001010010010100001001
00001000101010111001101110011000111010
10101010111010100110101110000101000101101110101
01001010010010000100010101011010010110100111000
01010010010100001001000010001010101110011011100
Columnar Query Compression Columnar Archive Compression
Compression Benefits
 Transparent: 100% Application Transparent
 Smaller: Reduces Footprint
– CapEx: Lowers server & storage costs for primary, standby, backup, test & dev databases …
– OpEx: Lowers heating, cooling, floor space costs …
 Additional ongoing savings over life of a database as database grows in size
 Faster: Transactional, Analytics, DW
– Greater speedup from in-memory: 3-10x more data fits in buffer cache & flash cache
– Faster queries
– Faster backup & restore speeds
 End-to-end Cost / Performance Benefits across CPU, DRAM,
Flash, Disk & Network
17
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Heat Map and Automatic
Data Optimization
18
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Understanding Data Usage Patterns
Database ‘heat map’
01 1 0 1 0 1 0 0
0101
11101
11 0 1 0 11
000
01010101
011
0101010101010
10
10
101000101010
1101
1
0 0 0 1 0 1 10 1 0 1 0 1 0 1 0 1
01010
1 1 0 1 0 1 0 1 01 1 0 1
011
0111010001110
10
10
101000101010
1101
1
0 0 1 1 0 1 10 1 0 0 0 1 0 1 0 1
01010
1 1 0 1 0 1 0 1 01 1 0 1
010 111000001110
11
10
101000101010
1101
10
101000101010
1101
01010
1 1 0 1 0 1 0 1 01 1 0 1
10
101000101010
1101
10
101100101010
1101
19
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
1
0 0 0 1 0 1 10 1 0 1 0 1 0 1 0 1
01010
1 1 0 1 0 1 0 1 01 1 0 1
011
0101010101010
10
0
0
1101000101011101
10
101000101010
1101
011
0 1 1 1 0 1 0 1 0 1 1 1 011
011
0111010001110
10
10
101000101010
1101
1
0 0 1 1 0 1 10 1 0 0 0 1 0 1 0 1
01010
1 1 0 1 0 1 0 1 01 1 0 1
010 111000001110
11
10
101000101010
1101
1
0 0 0 1 0 1 10 1 0 1 0 1 0 1 0 1
11010
1 1 0 1 0 1 0 1 01 1 0 1
011
0111010101110
11
10
101100101010
1101
1
0 0 0 1 0 11
0101010101
01010
1 1 0 1 0 1 0 10
1101
011
0101010101010
10
0
0
1101000101011101
1
0 0 0 1 0 11
0101010101
01010
1 1 0 1 0 1 0 10
1101
011
0111010001110
10
10
101000101010
1101
1
0 0 1 1 0 1 10 1 0 0 0 1 0 1 0 1
01010
1 1 0 1 0 1 0 1 01 1 0 1
10
101000101010
1101
10
101000101010
1101
011
0111010101110
11
11010
1 1 0 1 0 1 0 1 01 1 0 1
011
0111010101110
11
10
101100101010
1101
Understanding Data Usage Patterns
Database ‘heat map’
01 1 0 1 0 1 0 0
0101
11101
11 0 1 0 11
000
01010101
011
0101010101010
10
10
101000101010
1101
1
0 0 0 1 0 1 10 1 0 1 0 1 0 1 0 1
01010
1 1 0 1 0 1 0 1 01 1 0 1
011
0111010001110
10
10
101000101010
1101
1
0 0 1 1 0 1 10 1 0 0 0 1 0 1 0 1
01010
1 1 0 1 0 1 0 1 01 1 0 1
010 111000001110
11
10
101000101010
1101
10
101000101010
1101
01010
1 1 0 1 0 1 0 1 01 1 0 1
10
101000101010
1101
10
101100101010
1101
20
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
1
0 0 0 1 0 1 10 1 0 1 0 1 0 1 0 1
01010
1 1 0 1 0 1 0 1 01 1 0 1
011
0101010101010
10
0
0
1101000101011101
10
101000101010
1101
011
0 1 1 1 0 1 0 1 0 1 1 1 011
011
0111010001110
10
10
101000101010
1101
1
0 0 1 1 0 1 10 1 0 0 0 1 0 1 0 1
01010
1 1 0 1 0 1 0 1 01 1 0 1
010 111000001110
11
10
101000101010
1101
1
0 0 0 1 0 1 10 1 0 1 0 1 0 1 0 1
11010
1 1 0 1 0 1 0 1 01 1 0 1
011
0111010101110
11
10
101100101010
1101
1
0 0 0 1 0 11
0101010101
01010
1 1 0 1 0 1 0 10
1101
011
0101010101010
10
0
0
1101000101011101
1
0 0 0 1 0 11
0101010101
01010
1 1 0 1 0 1 0 10
1101
011
0111010001110
10
10
101000101010
1101
1
0 0 1 1 0 1 10 1 0 0 0 1 0 1 0 1
01010
1 1 0 1 0 1 0 1 01 1 0 1
10
101000101010
1101
10
101000101010
1101
011
0111010101110
11
11010
1 1 0 1 0 1 0 1 01 1 0 1
011
0111010101110
11
10
101100101010
1101
Heat Map
What it tracks
 “Heat Map” tracking
HOT
Active
– Access and modification times tracked for tables and
partitions
– Modification times tracked for database blocks
 Comprehensive
Frequent
Access
WARM
– Segment level shows both reads and writes
– Distinguishes index lookups from full table scans
– Automatically excludes stats gathering, DDLs, table
Occasional
Access
redefinitions, etc
 High Performance
Dormant
21
COLD
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
– Object level at no cost
– Block level < 5% cost
Heat Map
Enterprise Manager Visualization
22
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Automatic Data Optimization
Usage Based Data Compression
01110101010010
10000100010101
01011100001010
Hot Data
10101010111010100110101
11000010100010110111010
10100101001001000010001
011100001010001011011
01010110100101101001110
101010100101001001000
00010100100101000010010
010001010101101001011
00010001010101110011010
010101001010010010001
10100101001001000010001
Warm Data
Archive Data
1010101011101010011010111000010100
101010101110101001101011100001010001011011
0101101110101010010100100100001000
101010100101001001000010001010101101001011
1010101101001011010011100001010010
010011100001010010010100001001000010001010
0101101110101010010100100100001000
0101000010010000100010101011010010
101010101110101001101011100001010001011011
1010101101001011010011100001010010
1000010100100101001010110111000010
101010101110101001101011100001011101011001
1110010100100101001010110111011010
10X
3X
Advanced Row Compression
23
Copyright © 2012,
2013, Oracle and/or its affiliates. All rights reserved.
15X
Columnar Query Compression Columnar Archive Compression
Confidential – Oracle Restricted
Automatic Data Optimization
Simple Declarative SQL extension
ALTER TABLE sales ILM add
Active
Frequent
Access
Occasional
Access
Dormant
24
 Advanced Row Compression (2-4x)
 Affects ONLY candidate rows
 Cached in DRAM & FLASH
row store compress advanced row
after 2 days of no update
 Warehouse Compression(10x)
 High Performance Storage
column store compress for query low
after 1 week of no update
 Warehouse Compression(10x)
 Low Cost Storage
tier to SATA Tablespace
 Archive Compression(15-50X)
 Archival Storage
column store compress for archive high
after 6 months of no access
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Scheduled Policy Execution
Automatic Data Optimization
 Immediate and background policy execution
– Policies are executed in maintenance windows
– Can be manually executed as needed
 Policies can be extended to incorporate Business Rules
– Users can add custom conditions to control placement
(e.g. 3 months after the ship date of an order)
25
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Automatic Data
Optimization Use Cases
26
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Automatic Data Optimization
OLTP
Reporting
Compliance & Reporting
10x compressed
15x compressed
This Quarter
Row Store
for fast OLTP
27
This Year
Compressed
Column Store
for fast analytics
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Prior Years
Archive Compressed
Column Store
for max compression
As data cools
down, Automatic
Data Optimization
automatically
converts data to
columnar
compressed
Online
Up to 15x Smaller Footprint & Faster Queries
Automatic Data Optimization for OLTP
 Both Columnar & Archive Compression now complement Advanced
Row Compression
 Best Practice:
– Step 1: Use Advanced Row Compression for entire DB and then
– Step 2: ADO automatically converts into columnar compressed once
the updates cool down, and is used mainly for reporting
=> Query speed of Columnar & 10x smaller footprint
– Step 3: ADO automatically converts into archive compressed once
data cools down further and is no longer frequently queried
=> 15-50x smaller footprint
28
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Optimizes Data Based on Heat Map
Automatic Data Optimization for DW
 Data generally comes in via Bulk Loading
 Workload dominated by queries, even during loading
Step 1: Bulk Load directly into Columnar Compressed
– 10x smaller footprint, Query speed of Columnar
Step 2: ADO automatically converts to Archive Compressed
and moves to Lower Cost Storage once its queried infrequently
– Data remains online, with 15-50x smaller footprint, & lower
storage cost
29
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Fast, Flexible Loads & Queries on Columnar
Automatic Data Optimization – Mixed Use
 Fastest Load with uncompressed & Fastest Queries with columnar
– Mixed workloads often use Java app or 3rd party tools to insert and update data
that does not use Bulk Loads, so cannot use Columnar
 Step 1: Load into uncompressed, conventional inserts & updates
– Fast loading, & flexibility of using a regular OLTP app for loading
 Step 2: ADO moves to Row Compressed or Columnar
Compressed or Low Cost Storage once updates cool down
– Faster Queries, 3-10x smaller footprint
30
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Oracle Intelligent Storage
Protocol (OISP)
31
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Introduction to NFS
 NFS means Network File System
– Originally developed by Sun Microsystems in 1984, built on RPC
 NFS is a distributed file system protocol
– NFS is an open standard, anyone can implement it
– Secure data access anywhere in the enterprise
– Commonly implemented with TCP over IP, can use RPC/RDMA
 Thanks to Moore’s Law
– Block and file protocol is a matter of convenience, not performance
32
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Direct NFS Client
 New feature introduced in Oracle Database 11g Release 1
 NFS client embedded in Oracle database kernel
 Improves NAS storage performance for Oracle database
 Improves high availability of NAS storage
 Optimizes scalability of NAS storage
 Vastly reduces system CPU utilization
 Significant cost savings for database storage
 Simplifies client management and eliminates config errors
 Consistent configuration and performance across different platforms
– Even Microsoft Windows
33
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Direct NFS Architecture
Direct NFS client
NAS Storage
Database
…
…
LGWR I/O queue
DBWR I/O queue
DBWR TCP connection
…
PQ slave I/O queue
…
RMAN I/O queue
Parallel network
paths for scalability
and failover
34
LGWR TCP connection
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
PQ slave TCP connection
RMAN TCP connection
Direct NFS can issue 1000s
of concurrent operations due
to the parallel architecture
Every Oracle
process has its own
TCP connection
Oracle Database on ZFS
 Database files
– Control
– Data
– Online Redo
 Recovery files
– Archived redo
– Backup sets
– Image copies
35
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Oracle Intelligent Storage Protocol
Setting Up An Oracle Database Without OISP
Many Steps to Configure
21
7 Shares +
7 Record Size Tunes +
7 LogBias Tunes =
21 Steps
21
7 Shares +
7 Record Size Tunes +
7 LogBias Tunes =
21 Steps
12
4 Shares +
4 Record Size Tunes +
4 LogBias Tunes =
12 Steps
Test
21
7 Shares +
7 Record Size Tunes +
7 LogBias Tunes =
21 Steps
Dev
Production
Database
BU&R
36
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Oracle Intelligent Storage Protocol (OISP)
Only on Oracle: Simplify Setup and Tuning for Oracle Database 12c
Oracle Database 12c
Database
data
IO metadata
exchanged with
ZFS Storage Appliance
ZFS Storage Appliance
dynamically tunes critical
storage parameters
ZFS Storage Appliance
37
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Reduced
Administration
Time
OISP will dynamically tune
65% of the critical factors
for optimal database
performance
Setting Up An Oracle Database With OISP
Fewer Steps to Configure
2
2 Shares +
0 Record Size Tunes +
0 LogBias Tunes =
2 Steps
2
2 Shares +
0 Record Size Tunes +
0 LogBias Tunes =
2 Steps
2
2 Shares +
0 Record Size Tunes +
0 LogBias Tunes =
2 Steps
Test
2
2 Shares +
0 Record Size Tunes +
0 LogBias Tunes =
2 Steps
Dev
Production
Database
BU&R
38
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Auto Tuning of Record Size, LogBias
Without OISP
With OISP
OISP
Tunes
Logfile share
NFS
Server
NFS
Server
Multiple shares, each with
its own Record Size and
LogBias setting
mnt/dbname/redo (Admin sets RS, LB)
mnt/dbname/control (Admin sets RS, LB)
mnt/dbname/pfile (Admin sets RS, LB)
mnt/dbname/datafile (Admin sets RS, LB)
mnt/dbname/tempfile (Admin sets RS, LB)
mnt/dbname/chgtrack (Admin sets RS, LB)
mnt/dbname/backup (Admin sets RS, LB)
39
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Datafile share
Two shares:
logfile and datafile
mnt/dbname/logfile (OISP sets RS, LB)
redo
mnt/dbname/datafile (OISP sets RS, LB)
control
pfile
datafile
tempfile
chgtrack
backup
Architecture
ZFS File System
Oracle Instance
Hints
Hints
Hints on I/O requests
Oracle Direct NFS
NFS Server
RPC
RPC
TCP
TCP
IP
IP
NIC
40
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
LINK
NIC
Use Case Example: RMAN Backup
Non-OISP/Un-Tuned 420 MBPS
1.7x
Improvement
OISP-Enabled 734 MBPS
Equivalent to a Tuned setup*
* set share logbias = throughput
41
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Over Un-Tuned /
Non-OISP
Summary
Oracle Advanced Compression
Oracle Database 12c
Oracle Advanced
Compression
Advanced Row Compression
Advanced LOB Compression
Advanced LOB Deduplication
Heat Map
(Object and Row Level)
Automatic Data Optimization
+
Temporal Optimization
42
Hybrid Columnar Compression (Exadata, ZFSSA, Axiom)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
Summary
 Partitioning: An enabler to allow data classification
 Compression: Reduce data storage footprint and improve
performance
 Tiered Storage: Key ILM strategy to lower storage usage and costs
 Heat Map: Automatically tracks data access patterns
 Automatic Data Optimization
– Automates data movement for compression and storage tiering
– Enables ILM for Oracle Database data
 ZFS Storage Appliance: database-aware storage
 Oracle Intelligent Storage Protocol: ZFS-specific admin reduction
43
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
44
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
45
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.