Download Downstream Capture/Apply: Oracle Streams

LowMart Corporation
Oracle 11g
Downstream
Capture/Warehousing
Using Streams
GROUP 4
Minh Vo, Susan Champigny, Ganapathi S. Santhana
12/8/2012
1 University Avenue
Lowell, MA 01854
Phone: 978 934-4000
LowMart Corporation
1
Plan of Action

PREPARATION
o



Disable Oracle Database Vault
CREATE DATA WAREHOUSE DATABASE: ORA11B
o
Adding a Warehouse Database with DBCA
o
Spotcheck Before Proceeding with Streams Setup
DOWNSTREAM CAPTURE/APPLY SETUP: ORACLE STREAMS
o
Setting System Parameters, Creating Streams Tablespace, and Creating STRMADMIN User
o
Configuring STREAMS_QUEUE, CAPTURE, PROPAGATION, and APPLY
o
Adding Another Apply Stream to Grab Selective Columns
o
Transformations!!!
APPENDIX
o
Created Complaints table in SYSDBA instead of HR
o
Ensure that strmadmin has all the Necessary Privileges
o
Run the Health Check Script
o
Things to look out for / Useful stuff:
o
Mike's Document
LowMart Corporation
2
Preparation
DISABLE ORACLE DATABASE VAULT
In your database, as sysdba, enter:
SQL> SELECT * FROM V$OPTION WHERE PARAMETER = 'Oracle Database Vault';
If TRUE, we have Data Vault installed. If FALSE, continue to the next section as it will not disrupt streams implementation.
If TRUE, we’ll need to disable it:
a. shutdown everything (asm, databases, listener)
b. perform (as oracle user):
$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk dv_off ioracle
c. restart ASM, databases, listener
Create Data Warehouse Database: Ora11b
ADDING A WAREHOUSE DATABASE WITH DBCA
On the VirtualBox machine where we installed the ASM implementation of Ora11, start dbca (Database Configuration
Assistant) and create the database ora11b.
LowMart Corporation
3
LowMart Corporation
4
LowMart Corporation
5
The warehouse database (ora11b) has been created. If we omitted the sample schemas option during setup, we’ll have to
create the HR user to proceed.
SPOTCHECK BEFORE PROCEEDING WITH STREAMS SETUP
Let’s doublecheck /etc/oratab so we can start setting . oraenv:
Agent has already set this up for us. Wonderful. We should also check our listener and tnsnames (done later).
LowMart Corporation
6
Downstream Capture/Apply: Oracle Streams
SETTING SYSTEM PARAMETERS, CREATING STREAMS TABLESPACE, AND CREATING STRMADMIN USER
On both databases (ora11 and ora11b) set the job_queue_processes = 1000, aq_tm_processes = 4 and add supplemental
log data. Be sure to create the complaints table in the HR schema as well.
On the production machine (ora11), create the streams_tbs tablespace and create the user strmadmin. Grant access to
strmadmin.
LowMart Corporation
7
For the Warehousing database, also create the streams_tbs tablespace, the strmadmin user, and grant privileges to
strmadmin. Make sure the DATAFILE location is specified correctly.
CONFIGURING STREAMS_QUEUE, CAPTURE, PROPAGATION , AND APPLY
Now as the strmadmin user on the source database, run the DBMS_STREAMS_ADM.REMOVE_QUEUE procedure to clear
out old queue if one exists.
Now create a new streams queue in ora11:
LowMart Corporation
8
Setup the source database to capture changes from the targeted table:
Drop the database link if one exists, then create the link to ora11b:
Drop existing propagation, just to be safe.
LowMart Corporation
9
And now, (re)create the propagation. MAKE SURE YOU’RE STILL THE STRMADMIN USER:
By now, the source database ora11 has been set up to capture changes in the hr.complaints table and send them over to
the streams_queue@ora11b_link. Next, we’ll work to start the Apply process on the target database.
On the TARGET database ora11b, connect as strmadmin/oracle and remove and recreate the queue.
LowMart Corporation
10
Create the apply process:
And link the apply process to the queue and table (hr.complaints).
Still on the Target database (ora11b) as strmadmin, drop and (re)create the database link to ora11 source database.
We tested the link with query below, but complained that the listener was not aware of the service requested.
SQL> select * from dual@ora11_link;
So, we’ll append to listener.ora under grid_home:
LowMart Corporation
11
TNSNAMES.ORA already set correctly.
Restart the listener service (lsnrctl stop/start) and retest the link:
Awesome! Links are working both to and from ora11/ora11b. While still on the Target machine, get the SCN number and
instantiate the table with the SCN number:
LowMart Corporation
12
Now instantiate to the newest SCN:
Start the Apply process with
SQL> EXEC DBMS_APPLY_ADM.START_APPLY('SYNC_APPLY');
From this point on, all changes in source table hr.complaints (ora11) are propagated to the target table hr.complaints
(ora11b).
To test, we’ll insert a record into source (ora11) hr.complaints and monitor the alert logs of both databases:
SQL> insert into hr.complaints values (9,'rogers','roy','phone','does not work','yes');
LowMart Corporation
13
Adding a record to the source db hr.complaints applies it to the target hr.complaints!
LowMart Corporation
14
ADDING ANOTHER APPLY STREAM TO GRAB SELECTIVE COLUMNS
Now let’s configure for selective columns. We’ll need to create the warehouse table DEFECT_TRENDS with the columns
PRODUCT and DEFECT on the target DB (ora11b).
Create new Apply process named “DEFECT_APPLY” so we can attach rules to it without breaking our replication to
hr.complaints on target.
BEGIN
dbms_apply_adm.create_apply(
queue_name => 'strmadmin.streams_queue',
apply_name => 'defect_apply',
apply_captured => false);
END;
/
Link the Apply process to the queue and table:
BEGIN
dbms_streams_adm.add_table_rules( table_name => 'hr.complaints',
streams_type => 'APPLY',
streams_name => 'DEFECT_APPLY',
queue_name => 'strmadmin.streams_queue',
include_dml => true,
include_ddl => false,
include_tagged_lcr => false,
source_database => 'ora11');
END;
/
Find the RULE_NAME to extend with declarative transformations:
SQL> select streams_type, streams_name, rule_owner, rule_name from dba_streams_rules;
LowMart Corporation
15
We see that we’ll need to extend STRMADMIN.COMPLAINTS25 since it belongs to the DEFECT_APPLY stream.
TRANSFORMATIONS !!!
We’ve done our sleuth-work. Now it’s time to add transformations on the rows being sent to us.
BEGIN
DBMS_STREAMS_ADM.RENAME_TABLE(
rule_name
=> ‘STRMADMIN.COMPLAINTS25’,
from_table_name => ‘hr.complaints’,
to_table_name => ‘hr.defect_trends’,
step_number => 0,
operation
=> 'ADD');
END;
/
BEGIN
DBMS_STREAMS_ADM.DELETE_COLUMN(
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.complaints',
column_name => 'CUSTNO',
value_type
=> '*',
step_number => 1,
operation
=> 'ADD');
DBMS_STREAMS_ADM.DELETE_COLUMN(
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.complaints',
column_name => 'CUSTLNAME',
value_type
=> '*',
step_number => 1,
operation
=> 'ADD');
DBMS_STREAMS_ADM.DELETE_COLUMN(
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.complaints',
column_name => 'CUSTFNAME',
value_type
=> '*',
step_number => 1,
operation
=> 'ADD');
DBMS_STREAMS_ADM.DELETE_COLUMN(
LowMart Corporation
16
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.complaints',
column_name => 'COMPLAINT',
value_type
=> '*',
step_number => 1,
operation
=> 'ADD');
END;
/
To display declarative transformations, run:
COLUMN RULE_OWNER HEADING
COLUMN RULE_NAME HEADING
COLUMN DECLARATIVE_TYPE HEADING
COLUMN PRECEDENCE HEADING
COLUMN STEP_NUMBER HEADING
SELECT
FROM
WHERE
'Rule Owner'
'Rule Name'
'Declarative|Type'
'Precedence'
'Step Number'
FORMAT A15
FORMAT A15
FORMAT A15
FORMAT 99999
FORMAT 99999
RULE_OWNER, RULE_NAME, DECLARATIVE_TYPE, PRECEDENCE, STEP_NUMBER
DBA_STREAMS_TRANSFORMATIONS
TRANSFORM_TYPE = 'DECLARATIVE TRANSFORMATION';
Match up the rows:
LowMart Corporation
17
Start DEFECT_APPLY with
BEGIN
DBMS_APPLY_ADM.START_APPLY(
apply_name => 'defect_apply');
END;
/
Doesn’t appear like it’s working yet, though… getting error:
Due to the step numbers assigned, turns out the ordering of my declarative rule-based transformations changed the
name of the table first, and then tried to delete columns off the original name, which no longer exists. Removing the
DELETE_COLUMN transformations and adding them back again with the proper table name (hr.defect_trends) worked
great!
BEGIN
DBMS_STREAMS_ADM.DELETE_COLUMN(
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.complaints ',
column_name => 'CUSTNO',
value_type
=> '*',
step_number => 1,
operation
=> 'REMOVE');
DBMS_STREAMS_ADM.DELETE_COLUMN(
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.complaints',
column_name => 'CUSTLNAME',
value_type
=> '*',
step_number => 1,
operation
=> 'REMOVE');
DBMS_STREAMS_ADM.DELETE_COLUMN(
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.complaints',
column_name => 'CUSTFNAME',
value_type
=> '*',
step_number => 1,
operation
=> 'REMOVE');
DBMS_STREAMS_ADM.DELETE_COLUMN(
LowMart Corporation
18
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.complaints',
column_name => 'COMPLAINT',
value_type
=> '*',
step_number => 1,
operation
=> 'REMOVE');
END;
/
Then finally:
BEGIN
DBMS_STREAMS_ADM.DELETE_COLUMN(
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.defect_trends',
column_name => 'CUSTNO',
value_type
=> '*',
step_number => 1,
operation
=> 'ADD');
DBMS_STREAMS_ADM.DELETE_COLUMN(
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.defect_trends',
column_name => 'CUSTLNAME',
value_type
=> '*',
step_number => 1,
operation
=> 'ADD');
DBMS_STREAMS_ADM.DELETE_COLUMN(
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.defect_trends',
column_name => 'CUSTFNAME',
value_type
=> '*',
step_number => 1,
operation
=> 'ADD');
DBMS_STREAMS_ADM.DELETE_COLUMN(
rule_name
=> 'STRMADMIN.COMPLAINTS25',
table_name
=> 'hr.defect_trends',
column_name => 'COMPLAINT',
value_type
=> '*',
step_number => 1,
operation
=> 'ADD');
END;
/
Inserting records to ora11 hr.complaints now successfully ships data to ora11b hr.complaints AND hr.defect_trends.
LowMart Corporation
19
For good measure, run the streams health check script after uploading it to /tmp:
$ sqlplus /nolog
SQL> spool /tmp/mysite.html
SQL> @/tmp/streams_hc_11_2_0_3.sql
SQL> exit
And open the generated mysite.html for review.
LowMart Corporation
20
Fin. Done. <High Five>
LowMart Corporation
21
Appendix
CREATED COMPLAINTS TABLE IN SYSDBA INSTEAD OF HR
Pretty self-explanatory… create the complaints table in HR schema instead.
ENSURE THAT STRMADMIN HAS ALL THE NECESSARY PRIVILEGES
SQL> connect / as sysdba
Connected.
SQL> GRANT SELECT ON CHANGE_SETS to strmadmin;
Grant succeeded.
SQL> GRANT EXECUTE on DBMS_CDC_PUBLISH to strmadmin;
Grant succeeded.
SQL> GRANT SELECT ANY TABLE to strmadmin;
Grant succeeded.
SQL> GRANT CREATE SESSION to strmadmin;
Grant succeeded.
SQL> GRANT CREATE TABLE to strmadmin;
Grant succeeded.
SQL> GRANT CREATE TABLESPACE to strmadmin;
Grant succeeded.
SQL> GRANT UNLIMITED TABLESPACE to strmadmin;
Grant succeeded.
SQL> GRANT SELECT_CATALOG_ROLE to strmadmin;
Grant succeeded.
SQL> GRANT EXECUTE_CATALOG_ROLE to strmadmin;
Grant succeeded.
SQL> GRANT CREATE SEQUENCE to strmadmin;
Grant succeeded.
SQL> GRANT DBA to strmadmin;
Grant succeeded.
SQL> GRANT EXECUTE on DBMS_CDC_PUBLISH to strmadmin;
Grant succeeded.
SQL> GRANT SELECT ANY TABLE to strmadmin;
Grant succeeded.
SQL> EXECUTE DBMS_STREAMS_AUTH.GRANT_ADMIN_PRIVILEGE(GRANTEE => 'strmadmin');
PL/SQL procedure successfully completed.
LowMart Corporation
22
RUN THE HEALTH CHECK SCRIPT
View the output:
At this point, it seems my capture setup is not correct. STRMADMIN_CAPTURE. Is this on Prod? Or Warehouse?
LowMart Corporation
23
THINGS TO LOOK OUT FOR/USEFUL STUFF :
DBA_ALERT_HISTORY
exec dbms_streams_adm.remove_streams_configuration;
select propagation_name, error_message from dba_propagation;
COLUMN OWNER HEADING 'Owner' FORMAT A10
COLUMN NAME HEADING 'Queue Name' FORMAT A28
COLUMN QUEUE_TABLE HEADING 'Queue Table' FORMAT A22
COLUMN USER_COMMENT HEADING 'Comment' FORMAT A15
SELECT q.OWNER, q.NAME, t.QUEUE_TABLE, q.USER_COMMENT
FROM DBA_QUEUES q, DBA_QUEUE_TABLES t
WHERE t.OBJECT_TYPE = 'SYS.ANYDATA' AND
q.QUEUE_TABLE = t.QUEUE_TABLE AND
q.OWNER
= t.OWNER;
Owner Queue Name
Queue Table
Comment
--------------------------------------------------------------------------------------------------------------------------STRMADMIN AQ$_STREAMS_QUEUE_TABLE_E STREAMS_QUEUE_TABLE exception queue
STRMADMIN STREAMS_QUEUE
STREAMS_QUEUE_TABLE
exec DBMS_STREAMS_ADM.REMOVE_QUEUE(‘STREAMS_QUEUE’,TRUE);
Alert log of ora11b:
LowMart Corporation
24
On target:
SQL> select apply_name, queue_name from dba_apply;
APPLY_NAME
QUEUE_NAME
-------------------------------------------------------------------------SYNC_APPLY
STREAMS_QUEUE
BEGIN DBMS_APPLY_ADM.ALTER_APPLY(
apply_name => 'SYNC_APPLY',
apply_user => 'HR'
);
END;
/
MIKE’S DOCUMENT
In your database, as sysdba, enter: SELECT * FROM V$OPTION WHERE PARAMETER = 'Oracle Database Vault';
If TRUE, you have DV installed. If FALSE, you are good and no concern with data vault in the process of implementing streams
If TRUE, you need to disable that before proceeding with implementing streams. Here is how to do that:
a. shutdown everything (asm, databases, listener)
b. perform (as oracle user):
$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk dv_off ioracle
c. restart ASM, databases, listener
You will also need to deal with log miner. the docs will tell you more on that but in general, streams plsql procedures will call upon
logminer procedures and if not there, streams procedures will fail.
create table complaints (
custno number(3) not null,
custlname varchar2(20),
custfname varchar2(20),
product varchar2(10),
complaint varchar2(100),
defect varchar2(30));
1. created a 2nd database called ora11b, used regular datafiles (non-asm database). Created an HR user in ora11b, granted connect,
resource to HR.
2. ensured that the listener (which is controlled by the grid infrastructure) could listen for both ora11 and ora11b
3. saved off the entire server to ensure i had a good fail back server to go to after steps 1 and 2 were done.
4. setup streams requirements for the source database (ora11): setup streams admin user, streams ts, queues, etc. (per oracle
documentation)
5. setup streams requirements for the target database (ora11b): same as above, per oracle documentation
6. much trial and error. Oracle documentation talks about using global names, logminer, etc. to set up the streams. That is true but
you can also avoid global names altogether (smart idea) and use db_links as the method of moving replicated streams data over from
source to target table. That is the route I suggest to consider.
Here is a sample insert record I used for the source table (hr.complaints):
insert into hr.complaints values (1,'johnston','dave','phone','does not work','yes');
The idea is to have this newly inserted record in the hr.complaints table replicate over to the ora11b database and into the
hr.complaints table there. That's the goal. Sounds easy enough right? Hmmm...
LowMart Corporation
25
for the new tablespace needed, as well as the streams administrator user, here is some handy code for that purpose:
for source db (ora11 on asm)
CREATE TABLESPACE streams_tbs DATAFILE '+DATA/ora11/datafile/streams' SIZE 25M AUTOEXTEND ON MAXSIZE 256M;
CREATE USER strmadmin IDENTIFIED BY oracle
DEFAULT TABLESPACE streams_tbs
QUOTA UNLIMITED ON streams_tbs;
grant connect, dba, imp_full_database, exp_full_database, aq_administrator_role to stradmin;
for target db (ora11b: non asm):
CREATE TABLESPACE streams_tbs DATAFILE '/u02/oradata/ora11b/streams.dbf' SIZE 25M AUTOEXTEND ON MAXSIZE 256M;
CREATE USER strmadmin IDENTIFIED BY oracle
DEFAULT TABLESPACE streams_tbs
QUOTA UNLIMITED ON streams_tbs;
grant connect, dba, imp_full_database, exp_full_database, aq_administrator_role to stradmin;
At this point, you must setup queues, capture rules, propagation rules on the source database then do the same on the target
database (per the documentation). Once done, and database links are built and working correctly, starting the apply process should
render decent results (one hopes). This is usually where the troubleshooting begins and the time required to get this working gets
more involved.
To setup and test your database links from each database, you might like using this code:
-- link in source database to connect to the target database
CREATE DATABASE LINK ora11b_link CONNECT TO strmadmin IDENTIFIED BY oracle USING 'ora11b';
-- test link
select * from dual@ora11b_link;
Well, since I spent around 8 hours straight today working on the solution for the project, I will assume many of you did not do the
same. So I will let you continue the effort given the above information as a decent road map on this project and see how you fair with
it.
http://blog.nominet.org.uk/tech/2011/06/03/how-to-corrupt-your-data-dictionary-with-oracle-streams/
I hit this very same issue yesterday as I was working on the implementation. This is a classic case of 'listening' to the documentation
and then getting burned by accident. Obviously in this case, I reloaded my server and started the process again but the 2nd time
around, I left those logminer objects right where they always live. In fact, since I mentioned to you about using database links to
move the data, the logminer facility is not even used to accomplish the replication of the table data. So avoid, at all costs, the part
where Oracle tells you to create a logminer tablespace and then move logminer objects into it. If you should ever drop that tablespace,
you will wipe your database out pretty nicely. Not a good thing, as the blogger suggested, if you were implementing a streams solution
on a production database.