Download datapumpdoag2009

Working with Oracle
Oracle since
1986
Data
Pump
Oracle DBA - OCP Oracle7, 8, 9, 10
Oracle DBA of the Year – 2002
Carl Dudley
Oracle ACE
Directorof Wolverhampton, UK
University
Regular Presenter at Oracle Conferences
Consultant and Trainer
Technical Editor for a number of Oracle texts
SIG Director
UK Oracle UserUKOUG
Group Director
Member of IOUC
Day job – University
of Wolverhampton, UK
[email protected]
Oracle Data Pump
Oracle10g Data Pump Environment
Data Pump Exports
The Master Table
Data Pump Import
Attaching to Data Pump Jobs
Performance Tests
Data Pump and External Tables
Summary
2
The Data Pump Utility
 Enhanced Export and Import utility used for a variety of purposes
— Produce logical dumps of database objects
— Reorganize database storage
— Transfer data across systems
— Upgrade (migrate) to different versions of Oracle
— Store data offline for future use
— Perform TableSpace Point-In-Time Recovery (TSPITR)
 Essential features
— Users may export/import their own objects (or row subsets)
• Data Pump can use direct path or external table method
— Easiest method to restore a single table
— Cannot be used to recover data
— Data Pump export file is a binary file in “internal” Oracle format
— Export does not drop exported objects
— Import can create objects as well as import rows
3
Data Pump Architecture
Client 1
Dump file set
Data, Metadata,
Master Table
Master
table
Shadow
process 1
Worker
Process 1
metadata
Status queue
Master
control
process
Worker
Process 2
Direct path
Shadow
process 2
Control
queue
Client 2
Worker
Process 3
External
table
Parallel
process 1
Parallel
process 2
Database
Log file
4
Data Pump Architecture (continued)
 Shadow Process
— Creates a job which includes master table, master process and queues
— Checks job status during the run
— If the client process detaches, the other processes remain active
• Another shadow process can be invoked to connect to the job
– Need to know job name – can be seen in user_datapump_jobs
– Allows a change of parameter e.g.PARALLEL
 Master Control Process
— Controls execution and sequencing
— Divides processing among worker processes
— Manages information in the master table and log file
 Worker Process
— Loads and unloads data and metadata
— When using external table API, number of worker processes can be set by
PARALLEL parameter (Enterprise Edition only)
— Maintains master table (type of object being handled etc.)
5
Directories for Data Pump
 Output is server based, so directory objects required to ensure security
 Directory objects must be created by the sys user
— Necessary because the privileged 'Oracle' account is used to write to the
files, thus presenting a security risk
CREATE DIRECTORY dpump_dir AS ‘c:\extfiles';
 Must grant READ and WRITE access to the Data Pump user on the directory
GRANT READ,WRITE ON dpump_dir TO fred;
— Oracle reads/writes files in the directory on the users behalf
 DATA_PUMP_DIR directory used by default when no DIRECTORY specified
— In windows and UNIX this is pre-created
— DATA_PUMP_DIR is pre_defined on install
— On Windows, if setting the environmental variable DATA_PUMP_DIR, the
directory name must be UPPERCASE
C:\> SET DATA_PUMP_DIR=DATA_PUMP_DIR
6
Finding Permissions on Directories
SELECT grantee
,privilege
,directory_name
FROM all_tab_privs t
,all_directories d
WHERE t.table_name = d.directory_name
ORDER BY d.directory_name
,t.privilege
GRANTEE
------------FRED
FRED
FRED
PUBLIC
PUBLIC
PRIVILEGE
--------READ
READ
WRITE
READ
WRITE
DIRECTORY_NAME
-------------FILE1_DIR
DPUMP1_DIR
DPUMP1_DIR
DPUMP2_DIR
DPUMP2_DIR
7
Data Pump Queues
 Two queues observed in dba_queues (names contain timestamps)
— KUPC$S_1_20060521193941
• Status queue
— KUPC$C_1_20060521193941
• Control queue
 Queue table used by both queues observed in dba_queue_tables
— KUPC$DATAPUMP_QUETAB
 In Release 2, Data Pump needs to have a Streams Pool configured
— Requires STREAMS_POOL_SIZE > 0
— Or use Automatic Shared Memory Management (ASMM)
• SGA_TARGET > 0
8
Methods of Exporting/Importing
 Can interactively stop and restart jobs by attaching from another session
— Multiple clients (expdp) can attach to the same export job
 Certain operations can be performed within OEM
 All imported rows are placed in new blocks beyond the table HWM (no
searching for free space)
 Data Pump uses a direct path mode whenever possible
— Structures such as clustered tables or tables with triggers and/or active
referential constraints prevent this
• The (slower) External Table API is used instead
 Do not use sys except at the request of Oracle Technical Support
9
Oracle Data Pump
Oracle10g Data Pump Environment
Data Pump Exports
The Master Table
Data Pump Import
Attaching to Data Pump Jobs
Performance Tests
Data Pump and External Tables
Summary
10
Data Pump Export Levels
 Table
— Specific tables can be exported (with or without the data)
— Specific partitions and subpartitions
— Row subsets using query specifications (forces external table method)
expdp amy/amypw DIRECTORY=dpump_dir DUMPFILE=amy_emp.dmp
QUERY=emp:"WHERE job='CLERK' AND sal<900"
 Schema (default level)
— Allows export of all objects owned by one user
— DBAs may use this to export a series of users
 Tablespace
— Transportable Tablespaces
— Tablespace level export
 Full
— DBAs may export all objects in database except those owned by sys
11
Exporting Tables from Different Schemas
 Original Export allowed DBAs to export tables owned by different users
exp system/manager tables = fred.emp, sh.sales, scott.dept
About to export specified tables by conventional path
Current user changed to FRED
..exporting table EMP 14 rows exported
Current user changed to SH
..exporting table SALES 100432 rows exported
:
:
 Cannot be done directly in Oracle10g Data Pump
— All specified tables must reside in same schema
— Must perform schema level exports to export objects across schemas
• Requires EXP_FULL_DATABASE privilege
• All objects (with their dependents) are exported
 Restriction removed in Oracle11g Release 2
12
Exporting Tables from Different Schemas - Workaround
 Can specify a list of tables from within the schemas
expdp system/manager DIRECTORY=dpump_dir
DUMPFILE=test.dmp SCHEMAS=fred,sh,scott
INCLUDE=TABLE:\"IN('EMP','SALES','DEPT')\"
— Note the need to escape the double quote characters (Windows)
• On UNIX all special characters ( ' " may need to be escaped
 Subqueries can also be used
expdp system/manager DIRECTORY=dpump_dir
DUMPFILE=test2.dmp INCLUDE=TABLE:\"IN (SELECT tname FROM
tab WHERE tname LIKE '%EMP%' AND tabtype = 'TABLE‘)\"
expdp scott/tiger DIRECTORY=dpump_dir
DUMPFILE=trg.dmp INCLUDE=TRIGGER:\"IN (SELECT trigger_name
FROM user_triggers WHERE table_name = 'EMP')\"
13
Exporting the Meta Data
 Data, metadata or both (default) can be exported
CONTENT=DATA_ONLY | METADATA_ONLY | ALL
— Metadata is written as XML (for portability) using dbms_metadata
• Seven times bigger than normal export
 Schema level dump of metadata
expdp fred/fred DIRECTORY=dpump_dir DUMPFILE=meta.dmp
CONTENT=METADATA_ONLY
14
Compressing and Encrypting
 In 10g R2, Compression of metadata can occur before an export
COMPRESSION = METADATA_ONLY | NONE
— Uncompressed during import
 Specific columns in tables may now be stored in encrypted form
— Such columns can be re-encrypted in dumpfile set if a password is supplied
• Otherwise encrypted column data is dumped in clear text
— ENCRYPTION_PASSWORD[ = PASSWORD]
— The same password is needed on import
— Requires Transparent Data Encryption
• Part of the Advanced Security Option
15
Filtering Objects to Export
16
Selecting Objects to Export
 Export can exclude OR include certain objects
— Can be used to exert fine control over what is exported
expdp amy/amypw DIRECTORY=dpump_dir DUMPFILE=amy.dmp
EXCLUDE=INDEX,TRIGGER:\"LIKE 'COPY%'\"
• Excludes all indexes, and triggers with names beginning with 'copy'
expdp amy/amypw DIRECTORY=dpump_dir DUMPFILE=amy.dmp
INCLUDE=TABLE/GRANT
• Exports only table level grants
expdp fred/fred DIRECTORY=dpump_dir DUMPFILE=amy3.dmp
INCLUDE=TABLE:\"LIKE 'DEP%'\"
If colon and escaped
characters are missed,
defaults to schema level!
Table name must
be uppercase
Escape character
17
Object Paths
 The set of object paths can be seen in datapump_paths
OBJECT_PATH
------------------------------:
INDEX
TABLE/INDEX
TABLE_EXPORT/TABLE/INDEX
INDEX
INDEX/INDEX
TABLE/INDEX/INDEX
TABLE_EXPORT/TABLE/INDEX/INDEX
CONSTRAINT
TABLE/CONSTRAINT
FULL_PATH
-----------------------------:
TABLE_EXPORT/TABLE/INDEX
TABLE_EXPORT/TABLE/INDEX
TABLE_EXPORT/TABLE/INDEX
TABLE_EXPORT/TABLE/INDEX/INDEX
TABLE_EXPORT/TABLE/INDEX/INDEX
TABLE_EXPORT/TABLE/INDEX/INDEX
TABLE_EXPORT/TABLE/INDEX/INDEX
TABLE_EXPORT/TABLE/CONSTRAINT
TABLE_EXPORT/TABLE/CONSTRAINT
— This view contains 1903 rows
— Types of exports can be seen in dba_export_objects (571 rows)
18
Data Pump Export Files
 Perform exports in parallel for increased performance (PARALLEL=integer)
— Dump file set will consist of one or more files to the value of PARALLEL
— Can use substitution variable in the filename for automatic naming
• For example : mydumpfileset%U.dmp
• %U can have values from 1 to 99
 The export can also be created in multiple files based on a file size limit
— FILESIZE=integer[B | K | M | G]
— Multiple file names or ‘%u’ is required if multiple files are needed
• Multiple directories can be used dpdir1:f1.dmp,dpdir2:f2.dmp
— Size of file is independent of direct or external table method
— Export will not overwrite existing files
 Master table cannot be stored across multiple files in an export
• 400,000 objects (10,000 tables) creates a master table of 189mb
• Make FILESIZE big enough to store the master table
19
Export and Block Corruption
 Data Pump Export does not detect corrupted blocks
SQL> select count(*) from scott.empb;
select count(*) from scott.empb
*
ERROR at line 1:
ORA-01578: ORACLE data block corrupted (file # 8, block # 17)
ORA-01110: data file 8: 'C:\T3_F1'
C:\>expdp scott/tiger dumpfile = dp:test.cor tables = empb
Export: Release 10.2.0.1.0 - Production on Sunday, 07 May, 2006
9:28:55
:
:
. . exported "SCOTT".“EMPB"
23.50 KB
928 rows
Master table "SCOTT"."SYS_EXPORT_TABLE_04" successfully
loaded/unloaded
********************************************************************
Dump file set for SCOTT.SYS_EXPORT_TABLE_04 is: C:\ATEST.COR
Job "SCOTT"."SYS_EXPORT_TABLE_04" successfully completed at 09:29:37
20
Data Pump Interfaces
Interactive command line
— Very limited functionality
Command line
Parameter file
Data Pump API
Database Control (OEM)
21
Interactive Method
 Schema level export is automatically invoked
— Can use a default directory
c:\>expdp directory = data_pump_dir
Export: Release 10.2.0.1.0 on Thursday, 22 September, 2005 20:46:45
Copyright (c) 2003, Oracle.
All rights reserved.
Username: scott
Password: xxxxx
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0
With the Partitioning, OLAP and Data Mining options
Starting "SCOTT"."SYS_EXPORT_SCHEMA_01": scott/******** directory=data_pump_dir
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 26.96 MB
Processing object type SCHEMA_EXPORT/PRE_SCHEMA/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/TABLE/TABLE
Processing object type SCHEMA_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
. . exported "SCOTT"."EMPBIG"
20.33 MB 458752 rows
Master table "SCOTT"."SYS_EXPORT_SCHEMA_01" successfully loaded/unloaded
******************************************************************************
Dump file set for SCOTT.SYS_EXPORT_SCHEMA_01 is:
D:\ORACLE\PRODUCT\10.2.0\ADMIN\ORA1\DPDUMP\EXPDAT.DMP
Job "SCOTT"."SYS_EXPORT_SCHEMA_01" successfully completed at 20:48:31
22
Command Line Method
C:\>expdp scott/tiger tables = empbig,emp directory = dp dumpfile = emp.dmp
job_name = q1
Export: Release 10.2.0.1.0 on Sunday, 25-July, 2005 18:20:05
Copyright (c) 2003, Oracle.
All rights reserved.
Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0
With the Partitioning, OLAP and Data Mining options
Starting "SCOTT"."Q1": scott/******** tables = empbig,emp directory = dp
dumpfile = emp.dmp job_name = q1
Estimate in progress using BLOCKS method...
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 25.06 MB
Processing object type TABLE_EXPORT/TABLE/TABLE
Processing object type TABLE_EXPORT/TABLE/INDEX/INDEX
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Processing object type TABLE_EXPORT/TABLE/INDEX/STATISTICS/INDEX_STATISTICS
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/REF_CONSTRAINT
Processing object type TABLE_EXPORT/TABLE/STATISTICS/TABLE_STATISTICS
. . exported "SCOTT"."EMPBIG"
20.33 MB 458752 rows
. . exported "SCOTT"."EMP"
7.820 KB
14 rows
Master table "SCOTT"."Q1" successfully loaded/unloaded
******************************************************************************
Dump file set for SCOTT.Q1 is:
C:\EMP.DMP
Job "SCOTT"."Q1" successfully completed at 18:20:50
23
Data Pump API
 PL/SQL interface to allow routines to be constructed and scheduled
— Can execute jobs and set parameters
BEGIN
handle1 :=
DBMS_DATAPUMP.OPEN('EXPORT','SCHEMA',NULL,‘EX_JOB_1','LATEST');
DBMS_DATAPUMP.SET_PARAMETER(handle1,'KEEP_MASTER',1);
DBMS_DATAPUMP.ADD_FILE(handle1,‘scott.dmp','DPDIR');
DBMS_DATAPUMP.METADATA_FILTER(handle1,'SCHEMA_EXPR', 'IN(''SCOTT'')');
DBMS_DATAPUMP.START_JOB(handle1);
DBMS_DATAPUMP.DETACH(handle1);
END;
/
24
Data Pump Export Features
 Flashback exports (time or SCN based) can be generated as of a specific
recent point in time (consistent exports)
FLASHBACK_SCN=234671
FLASHBACK_TIME='"2004-08-02 17:10:00"'
 Can be executed as a job and subsequently stopped and resumed
 Estimates size of export file(s) based on blocks or statistics
— Option to perform an 'estimates only' export (ESTIMATE_ONLY=Y)
— Does not account for metadata
 Export can be performed from a remote database across a network
25
Estimates of Export size
 Export estimates size of export file(s) based on blocks or statistics
ESTIMATE = BLOCKS|STATISTICS
— Option to perform an 'estimates only' export (ESTIMATE_ONLY=Y)
 Test conducted on a 220,000 employee table
Availability of statistics
Value of ESTIMATE
Estimated
Value
No statistics
ESTIMATE=BLOCKS
18M
No statistics
ESTIMATE=STATISTICS
8.058M
Statistics (computed)
ESTIMATE=BLOCKS
18M
Statistics (computed)
ESTIMATE=STATISTICS
14.83M
— Actual size of table export = 15.08M
26
Exporting from a Remote Database
 Based on database links and uses external tables
expdp scott/tiger DIRECTORY=dp NETWORK_LINK=source_database_link
DUMPFILE=net_exp.dmp LOGFILE=net_exp.log
— If export is run on a read only database, it must be remote
• Need to maintain the master table
— Creates the dumpfile set on the instance where the job is running
 Network bandwidth dictates performance
— Parallel operations could saturate
Remote Database
expdp using a database link
Export file on
local server
27
Oracle Data Pump
Oracle10g Data Pump Environment
Data Pump Exports
The Master Table
Data Pump Import
Attaching to Data Pump Jobs
Performance Tests
Data Pump and External Tables
Summary
28
Data Pump Master Table
 Created and maintained during the export operation
— Same name as the Data Pump job name, e.g. sys_export_schema_03
• Can be set with JOB_NAME parameter
— Avoid job names like emp-history
• Oracle will attempt to build a master table called emp
— Automatically dropped when export completes successfully
• Can be preserved using KEEP_MASTER=y
— Data Pump user will need quota for master table data
• Has over 70 columns
— Final object to be placed in dump file set
 Allows monitoring of export process and maintains context for stop/restart
29
Master Table Columns
PROCESS_ORDER
DUPLICATE
DUMP_FILEID
DUMP_POSITION
DUMP_LENGTH
DUMP_ALLOCATION
COMPLETED_ROWS
ERROR_COUNT
ELAPSED_TIME
OBJECT_TYPE_PATH
OBJECT_PATH_SEQNO
OBJECT_TYPE
IN_PROGRESS
OBJECT_NAME
OBJECT_LONG_
OBJECT_SCHEMA
ORIGINAL_
PARTITION_NAME
SUBPARTITION_NAME
FLAGS
PROPERTY
COMPLETION_TIME
OBJECT_TABLESPACE
SIZE_ESTIMATE
OBJECT_ROW
PROCESSING_STATE
PROCESSING_STATUS
BASE_PROCESS_ORDER
BASE_OBJECT_TYPE
BASE_OBJECT_NAME
BASE_OBJECT_SCHEMA
ANCESTOR_PROCESS_ORDER
DOMAIN_PROCESS_ORDER
UNLOAD_METHOD
PARALLELIZATION
GRANULES
SCN
GRANTOR
XML_CLOB
NAME
VALUE_T
VALUE_N
IS_DEFAULT
FILE_TYPE
USER_DIRECTORY
USER_FILE_NAME
FILE_NAME
EXTEND_SIZE
FILE_MAX_SIZE
PROCESS_NAME
LAST_UPDATE
WORK_ITEM
OBJECT_NUMBER
COMPLETED_BYTES
TOTAL_BYTES
METADATA_IO
DATA_IO
CUMULATIVE_TIME
PACKET_NUMBER
OLD_VALUE SEED
LAST_FILE
USER_NAME
OPERATION
JOB_MODE
CONTROL_QUEUE
STATUS_QUEUE
REMOTE_LINK
VERSION_DB_VERSION
TIMEZONE
STATE
PHASE
GUID
START_TIME
BLOCK_SIZE
METADATA_BUFFER_SIZE
DATA_BUFFER_SIZE
DEGREE
PLATFORM
ABORT_STEP
INSTANCE
30
Indexes on Master Table
 Indexes built in default tablespace
sys_mtable_00000d5f6_ind_1
object_schema
object_name
object_type
sys_mtable_00000d5f6_ind_2
base_process_order
sys_mtable_00000d5f6_ind_3
object_path_seqno
sys_c006183
Process_order
duplicate
31
Data Pump Master Table (continued)
 The reason why Data Pump cannot be used on a READ ONLY database
— Writes are performed on the master table
— But can perform on a READ ONLY database via a network connection
• Can export data out of standby read only databases
 Allows the restart of Data Pump jobs
— Records current state of every object imported or exported
• Holds locations in the dump file set, status of worker processes, current
job status and restart information
32
Identifying Contents of Master Table
 To find file names used by a Data Pump job
SELECT user_file_name
FROM <master_table_name>
WHERE process_order IN (-22,-21);
 To find the kinds of database objects in the export
SELECT object_type,COUNT(*)
FROM <master_table_name>
GROUP BY object_type;
 To find tables collected in the export
SELECT object_schema, object_name
FROM <master_table_name>
WHERE process_order > 0
AND object_type = 'TABLE';
33
Using Data Pump as Part of a Backup Strategy
 A full database-level export is a logical backup of the database
— Slower than OS physical backups but can be parallelized
— Useful for restoring single tables from a DROP command
 Database must be open to perform an export
— Export guarantees a read-consistent view as of the time of the export or at a
specified flashback SCN or time
— The database can be placed in RESTRICTed mode by DBAs to guarantee a
consistent full database export
STARTUP OPEN RESTRICT
• Only users with RESTRICTED SESSION system privilege can connect
34
Using Data Pump as Part of a Backup Strategy
(continued)
 Data Pump can be used to restore/reorganize a database
— Rebuild a database to effect a change in the block size
• Must prebuild all tablespaces first
— Move tables across users, reduce fragmentation, migration effects
— No rollforward recovery is possible
 Set VERSION for export so that it can be read by a previous Oracle release
— 9.2 is allowed!
35
Oracle Data Pump
Oracle10g Data Pump Environment
Data Pump Exports
The Master Table
Data Pump Import
Attaching to Data Pump Jobs
Performance Tests
Data Pump and External Tables
Summary
36
Data Pump Import
 The only utility that can read Data Pump export files
— Can selectively import individual database objects and types of objects
using EXCLUDE or INCLUDE
 Can import only the metadata to a special sqlfile
impdp fred/fred DIRECTORY=dpump_dir DUMPFILE=f1.dmp
SQLFILE=fred_ddl.sql
— File contains commands to recreate original objects
• Edit the create statements before submission to the target database
— Useful for moving from development to live database
• Contains all the code for the procedural objects
— If TRACE=2 is specified, the XML is also included
 If job stops on a corrupted object, the import can jump over it on restart
with START_JOB=SKIP_CURRENT submitted in an attached session
37
Data Pump Import DDL Transformations - Remapping
 Schemas can be remapped from one user to another with REMAP_SCHEMA
impdp system/manager DIRECTORY=dpump_dir DUMPFILE=users.dmp
SCHEMAS=fred,scott REMAP_SCHEMA=fred:amy
— Loads data from two schemas, fred’s data is loaded into amy’s schema
 Objects can be moved to different tablespaces using REMAP_TABLESPACE
— Much more convenient than original exp/imp
• The XML used for metadata allows easy transformation via XSL-T
 Files can be mapped to different file names using REMAP_DATAFILE
 MASTER_ONLY=Y (hidden parameter) will import only the master table
— OEM uses the master table for other purposes
 Consider disabling referential constraints and triggers during import
38
Data Pump Import DDL
Transformations (continued)
 The TRANSFORM parameter can prevent generation of
1. STORAGE and TABLESPACE clause
2. STORAGE clause only
• Applies to both tables and indexes unless TABLE or INDEX is specified
TRANSFORM=SEGMENT_ATTRIBUTES|STORAGE:{y|n}[:TABLE|INDEX]
 STORAGE
— Controls use of existing storage parameters (default is y)
TRANSFORM=STORAGE:n:TABLE
• Omits only the storage clauses for tables
— Storage parameters always ignored if SEGMENT_ATTRIBUTES is set to 'n'
 SEGMENT_ATTRIBUTES
— Controls the preservation of the tablespace (default is y)
39
Import of Data into Tables Already Present
 Import of rows based on value of TABLE_EXISTS_ACTION
Value
Action
SKIP
Leaves table unchanged
Default value when not in DATA_ONLY mode
APPEND
Adds new rows using external table method
Default value when in DATA_ONLY mode
TRUNCATE
Removes data before importing new rows using external
table method
REPLACE
Drops the table and recreates
40
Data Pump Import from a Remote Database
 Transfer data between development, production and standby databases
— Source database can be read only
• Schema owner(s) on the source must have access to a locally
managed temporary tablespace
 No creation of dumpfile
— Requires a database link – does not use network pipes
— Uses direct path and performs INSERT /*+APPEND*/ ...
CREATE DATABASE LINK test_public USING 'test';
impdp system/manager NETWORK_LINK=test DIRECTORY=dpump_dir
SCHEMAS=fred,scott
•
Log file is written to dpump_dir
41
Data Pump Import from a Remote Database
impdp scott/tiger TABLES=emp DIRECTORY=dpump_dir
NETWORK_LINK=source_database_link
Data Pump
Job
Server
Process
Remote
Database
impdp using a
database link
Target
Database
+
Log file
 Can specify flashback scn or time, only when importing from a remote
database
 Can also use ESTIMATE on import from a remote database
— Instructs the source system to estimate how much data will be generated
42
Oracle Data Pump
Oracle10g Data Pump Environment
Data Pump Exports
The Master Table
Data Pump Import
Attaching to Data Pump Jobs
Performance Tests
Data Pump and External Tables
Summary
43
Output from a Running Export Session
C:\>expdp sh/sh directory=ext_dir dumpfile=sh.dmp job_name=j1
Export: Release 10.1.0.2.0 - Production on Thursday, 07 October, 2004 12:10
Copyright (c) 2003, Oracle.
All rights reserved.
Connected to: Oracle Database 10g Enterprise Edition Release 10.1.0.2.0 –
Production With the Partitioning, OLAP and Data Mining options
FLASHBACK automatically enabled to preserve database integrity.
Starting "SH"."J1": sh/****** directory=ext_dir dumpfile=sh.dmp job_name=J1
Estimate in progress using BLOCKS method...
Processing object type SCHEMA_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 107.1 MB
Processing object type SCHEMA_EXPORT/SE_PRE_SCHEMA_PROCOBJACT/PROCACT_SCHEMA
Processing object type SCHEMA_EXPORT/TABLE/TABLE
Processing object type SCHEMA_EXPORT/TABLE/GRANT/OBJECT_GRANT
Processing object type SCHEMA_EXPORT/TABLE/INDEX/INDEX
.......
44
Setting up the Second (attach) Session
c:\>expdp sh/sh attach=j1
Job: J1
Owner: SH
Operation: EXPORT
Creator Privs: FALSE
GUID: A3FA0099CB2A49429F2B754C114F9E05
Start Time: Thursday, 07 October, 2004 12:14
Mode: SCHEMA
Instance: orac
Max Parallelism: 1
EXPORT Job Parameters:
Parameter Name
Parameter Value:
CLIENT_COMMAND
sh/******** directory = ext_dir dumpfile = sh.dmp job_name= j1
DATA_ACCESS_METHOD
AUTOMATIC
ESTIMATE
BLOCKS
INCLUDE_METADATA
1
LOG_FILE_DIRECTORY
EXT_DIR
LOG_FILE_NAME
export.log
TABLE_CONSISTENCY
0
State: EXECUTING
Bytes Processed: 0
Current Parallelism: 1
Job Error Count: 0
Dump File: C:\SH.DMP
bytes written: 4,096
Worker 1 Status:
State: EXECUTING
 Hitting 'CTRL-C' in the original session will automatically place the
session in the 'attach' mode
45
Controlling Data Pump Jobs
Export> STATUS
Job: J1
Operation: EXPORT
Mode: SCHEMA
State: EXECUTING
Bytes Processed: 0
Current Parallelism: 1
Job Error Count: 0
Dump File: C:\SH.DMP
bytes written: 4,096
Worker 1 Status:
State: EXECUTING
Export> STOP_JOB
Are you sure you wish to stop this job ([y]/n): y
 STOP_JOB preserves the master table – for future START_JOB commands
— KILL_JOB deletes the master table
— Could also issue ADD_FILE,TRACE and/or PARALLEL commands
— Gives the opportunity to fix space related problems and then restart export
— CONTINUE_CLIENT starts the job
• Logging info is sent to the client session
 STATUS 120 will display job status information every 2 minutes
46
Keywords in Interactive Mode
Command
Effect
ADD_FILE
Add additional dump files
CONTINUE_CLIENT
Exit interactive mode and enter logging mode
(Restarts job, if job is in the stopped state)
EXIT_CLIENT
Stop export client session, but leave job running
HELP
Display a summary of available commands
KILL_JOB
Detach attached client sessions and kill current job
PARALLEL
Adjust the number of active worker processes for the
current job (Enterprise Edition only)
START_JOB
Restart a stopped job to which you are attached
STATUS
Show status for current job and/or set status interval
STOP_JOB [= IMMEDIATE] Stop the current job for later restart
IMMEDIATE aborts worker processes
47
Monitoring and Removing Failed Export Jobs
 dba_datapump_jobs
 dba_datapump_sessions
 v$session_longops
 On stopping an export, the job remains visible in dba_datapump_jobs
— If the export file is unavailable or corrupted the job cannot be killed by
attaching to the export
• Need to physically drop the master table from the user schema
SQL> CONNECT scott/tiger
SQL> DROP TABLE SYS_EXPORT_SCHEMA_02;
•
Indirectly removes the job record from dba_datapump_jobs
48
dba_datapump_jobs
OWNER
NAME
----SCOTT
SCOTT
SCOTT
SCOTT
SCOTT
JOB_NAME
------------------X
SYS_IMPORT_TABLE_01
Y
Z
T
OPERATION
--------EXPORT
IMPORT
EXPORT
EXPORT
EXPORT
JOB_MODE
-------TABLE
TABLE
SCHEMA
TABLE
TABLE
STATE
DEGREE
------------ -----NOT RUNNING
0
NOT RUNNING
0
NOT RUNNING
0
NOT RUNNING
0
STOP PENDING
1
ATTACHED DATAPUMP
SESSIONS SESSIONS
-------- -------0
0
0
0
0
0
0
0
1
3
 Four exports have completed with KEEP_MASTER = Y
 Export T has been recently stopped by an attached session
 The table level export has a default job name
— Combination of owner name and job name uniquely identifies a job
 Y is a schema level export
 The datapump_sessions column is not documented
— Refers to the number of sessions attached to the job’s queues
49
Oracle Data Pump
Oracle10g Data Pump Environment
Data Pump Exports
The Master Table
Data Pump Import
Attaching to Data Pump Jobs
Performance Tests
Data Pump and External Tables
Summary
50
Performance – Example Scenario
 Test has 2.0 GB of data (16.2M rows) involving two fact tables
 Export (single stream)
— Original export 10min 40sec
— Data Pump export 3min 12sec
— Expect 1.5 – 2 times single stream speed
• Higher factors of improvement depending on degree of parallelism and
sufficient hardware
 Import
— Original import 2hr 26min 10sec
— Data Pump import 0hr 03min 05sec
51
Performance – Further Tests
 Timings taken for sample employee tables containing 1, 0.5m, 1m, 2m, 4m
and 8m rows
— Original export
— Data Pump export using direct path and external table
— Original import
— Data Pump export using direct path and external table
 Sizes of dump file sets compared for
— Original export
— Data Pump export using direct path and external table
EMPNO
----7369
7499
7521
7566
ENAME
----SMITH
ALLEN
WARD
JONES
JOB
-------CLERK
SALESMAN
SALESMAN
MANAGER
MGR
---7902
7698
7698
7839
HIREDATE
--------17-DEC-80
20-FEB-81
22-FEB-81
02-APR-81
SAL COMM DEPTNO SEQID
---- ---- ------ ----800
20
1
1600 300
30
2
1250 500
30
3
2975
20
4
52
Export Timings and Sizes
Timings (seconds)
Rows
1
0.5M
1M
2M
4M
8M
Data Pump
Direct path
28
38
58
69
90
114
Data Pump
External Tables
51
70
90
105
120
146
Original Export
2
21
32
52
100
192
Size of Export File (MB)
Rows
1
0.5M
1M
2M
4M
8M
Data Pump
Direct path
0.008
20
41
82
169
331
Data Pump
External Tables
0.008
20
41
82
169
331
Original Export
0.008
23
47
89
190
389
53
Response time
Export Performance
3 min
Export - conventional path
Data Pump - direct path
Data Pump - external table
2 min
1 min
0.5m 1m
2m
4m
8m
Rows (millions)
54
Import Timings
Timings (seconds)
Rows
1
0.5M
1M
2M
4M
8M
Data Pump
Direct path
7
12
16
25
46
86
Data Pump
External Tables
25
33
33
44
63
120
Original Import
2
15
33
73
157
306
55
Response time
Import Performance
5 min
4 min
Import
Data Pump - direct path
Data Pump - external table
3 min
2 min
1 min
0.5m 1m
2m
4m
Rows (millions)
8m
56
Access Methods
 Oracle uses external table mode for imports when restrictions due to
object types prevent the use of direct path
 External table mode is used when the table
— Has referential constraints
— Has active triggers
— Has a global index present during a single-partition load
— Has a domain index on a LOB column
— Is in a cluster
— Has fine-grained access control enabled in insert mode
— Contains BFILE columns or columns of opaque types
— Contains VARRAY columns with an embedded opaque type
57
Setting the Data Access Method
 Undocumented parameter ACCESS_METHOD
— Can have values of EXTERNAL_TABLE and DIRECT_PATH
 Restrictions may not necessarily cause the value to be overridden
— Actual method used can be observed in a trace of master process(es)
...
DATA_ACCESS_METHOD = ...
...
 Significant start up time spent in estimation
 Performance improvement highest with large tables
58
Data Pump Tracing
 Tracing can be enabled at a number of levels using the TRACE parameter
— Can also be achieved by setting an event in the server parameter file
EVENT='39089 TRACE NAME CONTEXT FOREVER, LEVEL 0nnn0300'
— The first four digits identify the level of tracing
• 1FF0300 (or 01FF0300) is the highest level
• 480300 is the setting for standard tracing and can identify most errors
Item to be traced
Setting
API
0001
MCP
0008
FILE
0010
QUEUE
0020
WORKER
0040
DATA
0080
METADATA API
0100
MCP = Master control Process
59
Data Pump Tracing
expdp amy/amypw DIRECTORY=dp DUMPFILE=amy.dmp TRACE=01FF0300
 Trace files are written to BACKGROUND_DUMP_DEST
— Format of trace filename for master process
<sid>_dm<integer>_<process_id>.trc
db1_dm00_2198.trc
— Format of trace filename(s) - one for each worker process
<sid>_dw<integer>_<process_id>.trc
db1_dw01_3167.trc
– There is also a 'component' trace file in USER_DUMP_DEST
 Data Pump user must be a DBA to perform traced exports or imports
60
Sampling the Data
 Can specify a percentage of data to be sampled and exported
 Allows a subset of data to be obtained for testing
SAMPLE = "SCOTT"."EMP":50
— Causes 50% of the rows to be exported
• Sample based on a random selection of blocks (.00001 up to 100%)
 Tablespace storage can be decreased on import
— Large datafiles in production mapped to smaller files in a test database
• PCTSPACE can be used to specify a percentage reduction
— Extent allocations are altered and sizes of datafiles are adjusted
TRANSFORM = PCTSPACE:50
— Needs to be used in conjunction with the SAMPLE parameter
• Size of sample and storage need to be compatible
61
Oracle Data Pump
Oracle10g Data Pump Environment
Data Pump Exports
The Master Table
Data Pump Import
Attaching to Data Pump Jobs
Performance Tests
Data Pump and External Tables
Summary
62
Handling External Tables
 External Table files can be read and written
Database
Original
Table(s)
Ext_tab1
External
Table File
Ext_tab2
63
Writing to External Tables
 Unload and transform data into a flat file
— In Oracle9i, external tables had read only access via the ORACLE_LOADER
access driver
— ORACLE_DATAPUMP access driver is now able to write to external tables
 Only CREATE TABLE AS SELECT statements are allowed
— Subsequent DML is not allowed
 The resultant flat file is of a platform independent Oracle proprietary format
— Allows transform operations on the data as it is moved
• Joins can be created on data as it is loaded or unloaded
— More flexible than simple data pump export/import
 Data is written out in granules to allow parallel processing
— Parallelism is possible even with a single output file
• Except for writes to tape devices
 Can use the external table from within different databases
64
Writing to External Tables - Example
 Create an external table by unloading data from the database
CREATE DIRECTORY ext_dir AS 'c:\oracle\db1\unload';
GRANT READ,WRITE ON DIRECTORY ext_dir TO fred;
-- necessary if FRED is going to use the directory
CREATE TABLE emp_dept_xt (ename,job,hiredate,dname,loc)
ORGANIZATION EXTERNAL
(TYPE ORACLE_DATAPUMP DEFAULT DIRECTORY ext_dir
LOCATION ('emp_dept_file.dmp'))
AS SELECT e.ename,e.job,e.hiredate,d.dname,d.loc
FROM emp e,dept d
WHERE e.deptno = d.deptno;
65
Writing to External Tables – Example (2)
 Create an external table from a previously created external table file
CREATE TABLE emp_dept_from_xt (
ename
VARCHAR2(10)
,job
VARCHAR2(9)
,hiredate
DATE
,dname
VARCHAR2(14)
,loc
VARCHAR2(12))
ORGANIZATION EXTERNAL
(TYPE ORACLE_DATAPUMP DEFAULT DIRECTORY ext_dir
LOCATION ('emp_dept_file.dmp'));
— Note the absence of the 'AS' clause
— Can create multiple tables containing subsets of columns from the file
66
Contents of the External Table
SELECT * FROM emp_dept_xt WHERE dname = 'SALES';
ENAME
---------ALLEN
WARD
MARTIN
BLAKE
TURNER
JAMES
JOB
--------SALESMAN
SALESMAN
SALESMAN
MANAGER
SALESMAN
CLERK
HIREDATE
----------20-FEB-1981
22-FEB-1981
28-SEP-1981
01-MAY-1981
08-SEP-1981
03-DEC-1981
DNAME
---------SALES
SALES
SALES
SALES
SALES
SALES
LOC
------CHICAGO
CHICAGO
CHICAGO
CHICAGO
CHICAGO
CHICAGO
 Test to see if there are any discrepancies between the data
SELECT e.ename,e.job,e.hiredate,d.dname,d.loc
FROM emp e,dept d
WHERE e.deptno = d.deptno
MINUS
SELECT * FROM emp_dept_xt;
no rows selected.
67
Oracle Data Pump
Oracle10g Data Pump Environment
Data Pump Exports
The Master Table
Data Pump Import
Attaching to Data Pump Jobs
Performance Tests
Data Pump and External Tables
Oracle11g New features
Summary
68
Compressed and Encrypted Export Parameters
 Compressed exports
COMPRESSION={ALL | DATA_ONLY | METADATA_ONLY | NONE}
— ALL and DATA_ONLY require Advanced Compression option
— METADATA_ONLY metadata is compressed (default)
— NONE disables compression
 Encrypted exports
ENCRYPTION = {ALL | DATA_ONLY | ENCRYPTED_COLUMNS_ONLY |
METADATA_ONLY | NONE}
• Default is ALL if ENCRYPTION_PASSWORD is set
• DATA_ONLY, METADATA_ONLY, ALL available only in 11g
ENCRYPTION_MODE = { DUAL | PASSWORD | TRANSPARENT }
• Governs whether it can be imported using a password, wallet, or either
ENCRYPTION_ALGORITHM = { AES128 | AES192 | AES256 }
• Available only in 11g
69
Oracle 11g Data Pump Additional Features
 Legacy mode
— Automatic translation of old exp commands into Datapump syntax
— Legacy scripts still work
 Overwrite existing dump files with REUSE_DUMPFILES
 Metadata differ
— Requires Change Management Licence
 Objects having deferred segment creation are exported even if not built
— Original export cannot do this
 REMAP_DATA
— Masks data values – useful for moving data from production to test
 Useful white papers with examples
— http://www.oracle.com/technology/products/database/utilities/index.html#
70
Compression and Encryption Performance
 Export 2 million row employee table with compression and encryption
Dumpfile size
Winzip size
from export
Time to Export
Normal
82.6M
17.3M
44 secs
Compressed
15.2M
11.7M
46 secs
Encrypted
82.6M
82.6M
44 secs
Encrypted and
compressed
15.2M
15.2M
44 secs
71
Oracle Data Pump
Oracle10g Data Pump Environment
Data Pump Exports
The Master Table
Data Pump Import
Attaching to Data Pump Jobs
Performance Tests
Data Pump and External Tables
Summary
72
Differences between Data Pump and EXP/IMP
 Startup time is longer
— Designed for big jobs
— Typical time could be 10 seconds (or longer)
— Stream format makes Data Pump exports 15% smaller
 Master table is placed at end of exported file using direct path mode
— Import needs to locate the master table and build its indexes
 Importing a small subset of the data from an export could take a long time
— The master table needs to be maintained
 Metadata construction performance is about the same
— Seven times bigger than metadata from original export
— Can use gzip to compress the metadata
73
Points to Consider
 Faster
— Watch out for lots of small tables
 More Flexible
— Can export a selection of procedures and functions
• The dumpfile can act as a backup of the source code
— Multiple files can easily be used in multiple directories
 Good for restructuring tables
— Set up a good PCTFREE before importing data
— Drop old tables, then import under new storage definitions
 Can import database objects without including rows
— Useful for moving from development to live database
 Can effect the transfer of tablespaces across database and networks
without the need for read only status
— Very useful for OLTP databases
74
Future of Original Export and Import Utilities
 Import will remain available in future releases for ever
— Will handle export files from all versions
 Export will be deprecated
— Oracle 9i export can be used with 10g for downgrade purposes
— New features in 10g and 11g are not supported in exp
 Schema containing 200 tables each with 14 rows of employee data
— Time for exp : 23 secs
— Time for expdp : 2mins 13 seconds
— Size of exp dumpfile : 246K
— Size of expdp dumpfile : 2764K
75
Oracle Data Pump
Carl Dudley
University of Wolverhampton, UK
UKOUG SIG Director
[email protected]