* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Downstream Capture/Apply: Oracle Streams
Microsoft Access wikipedia , lookup
Entity–attribute–value model wikipedia , lookup
Functional Database Model wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Concurrency control wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Ingres (database) wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Oracle Database wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Clusterpoint wikipedia , lookup
ContactPoint wikipedia , lookup
LowMart Corporation Oracle 11g Downstream Capture/Warehousing Using Streams GROUP 4 Minh Vo, Susan Champigny, Ganapathi S. Santhana 12/8/2012 1 University Avenue Lowell, MA 01854 Phone: 978 934-4000 LowMart Corporation 1 Plan of Action PREPARATION o Disable Oracle Database Vault CREATE DATA WAREHOUSE DATABASE: ORA11B o Adding a Warehouse Database with DBCA o Spotcheck Before Proceeding with Streams Setup DOWNSTREAM CAPTURE/APPLY SETUP: ORACLE STREAMS o Setting System Parameters, Creating Streams Tablespace, and Creating STRMADMIN User o Configuring STREAMS_QUEUE, CAPTURE, PROPAGATION, and APPLY o Adding Another Apply Stream to Grab Selective Columns o Transformations!!! APPENDIX o Created Complaints table in SYSDBA instead of HR o Ensure that strmadmin has all the Necessary Privileges o Run the Health Check Script o Things to look out for / Useful stuff: o Mike's Document LowMart Corporation 2 Preparation DISABLE ORACLE DATABASE VAULT In your database, as sysdba, enter: SQL> SELECT * FROM V$OPTION WHERE PARAMETER = 'Oracle Database Vault'; If TRUE, we have Data Vault installed. If FALSE, continue to the next section as it will not disrupt streams implementation. If TRUE, we’ll need to disable it: a. shutdown everything (asm, databases, listener) b. perform (as oracle user): $ cd $ORACLE_HOME/rdbms/lib $ make -f ins_rdbms.mk dv_off ioracle c. restart ASM, databases, listener Create Data Warehouse Database: Ora11b ADDING A WAREHOUSE DATABASE WITH DBCA On the VirtualBox machine where we installed the ASM implementation of Ora11, start dbca (Database Configuration Assistant) and create the database ora11b. LowMart Corporation 3 LowMart Corporation 4 LowMart Corporation 5 The warehouse database (ora11b) has been created. If we omitted the sample schemas option during setup, we’ll have to create the HR user to proceed. SPOTCHECK BEFORE PROCEEDING WITH STREAMS SETUP Let’s doublecheck /etc/oratab so we can start setting . oraenv: Agent has already set this up for us. Wonderful. We should also check our listener and tnsnames (done later). LowMart Corporation 6 Downstream Capture/Apply: Oracle Streams SETTING SYSTEM PARAMETERS, CREATING STREAMS TABLESPACE, AND CREATING STRMADMIN USER On both databases (ora11 and ora11b) set the job_queue_processes = 1000, aq_tm_processes = 4 and add supplemental log data. Be sure to create the complaints table in the HR schema as well. On the production machine (ora11), create the streams_tbs tablespace and create the user strmadmin. Grant access to strmadmin. LowMart Corporation 7 For the Warehousing database, also create the streams_tbs tablespace, the strmadmin user, and grant privileges to strmadmin. Make sure the DATAFILE location is specified correctly. CONFIGURING STREAMS_QUEUE, CAPTURE, PROPAGATION , AND APPLY Now as the strmadmin user on the source database, run the DBMS_STREAMS_ADM.REMOVE_QUEUE procedure to clear out old queue if one exists. Now create a new streams queue in ora11: LowMart Corporation 8 Setup the source database to capture changes from the targeted table: Drop the database link if one exists, then create the link to ora11b: Drop existing propagation, just to be safe. LowMart Corporation 9 And now, (re)create the propagation. MAKE SURE YOU’RE STILL THE STRMADMIN USER: By now, the source database ora11 has been set up to capture changes in the hr.complaints table and send them over to the streams_queue@ora11b_link. Next, we’ll work to start the Apply process on the target database. On the TARGET database ora11b, connect as strmadmin/oracle and remove and recreate the queue. LowMart Corporation 10 Create the apply process: And link the apply process to the queue and table (hr.complaints). Still on the Target database (ora11b) as strmadmin, drop and (re)create the database link to ora11 source database. We tested the link with query below, but complained that the listener was not aware of the service requested. SQL> select * from dual@ora11_link; So, we’ll append to listener.ora under grid_home: LowMart Corporation 11 TNSNAMES.ORA already set correctly. Restart the listener service (lsnrctl stop/start) and retest the link: Awesome! Links are working both to and from ora11/ora11b. While still on the Target machine, get the SCN number and instantiate the table with the SCN number: LowMart Corporation 12 Now instantiate to the newest SCN: Start the Apply process with SQL> EXEC DBMS_APPLY_ADM.START_APPLY('SYNC_APPLY'); From this point on, all changes in source table hr.complaints (ora11) are propagated to the target table hr.complaints (ora11b). To test, we’ll insert a record into source (ora11) hr.complaints and monitor the alert logs of both databases: SQL> insert into hr.complaints values (9,'rogers','roy','phone','does not work','yes'); LowMart Corporation 13 Adding a record to the source db hr.complaints applies it to the target hr.complaints! LowMart Corporation 14 ADDING ANOTHER APPLY STREAM TO GRAB SELECTIVE COLUMNS Now let’s configure for selective columns. We’ll need to create the warehouse table DEFECT_TRENDS with the columns PRODUCT and DEFECT on the target DB (ora11b). Create new Apply process named “DEFECT_APPLY” so we can attach rules to it without breaking our replication to hr.complaints on target. BEGIN dbms_apply_adm.create_apply( queue_name => 'strmadmin.streams_queue', apply_name => 'defect_apply', apply_captured => false); END; / Link the Apply process to the queue and table: BEGIN dbms_streams_adm.add_table_rules( table_name => 'hr.complaints', streams_type => 'APPLY', streams_name => 'DEFECT_APPLY', queue_name => 'strmadmin.streams_queue', include_dml => true, include_ddl => false, include_tagged_lcr => false, source_database => 'ora11'); END; / Find the RULE_NAME to extend with declarative transformations: SQL> select streams_type, streams_name, rule_owner, rule_name from dba_streams_rules; LowMart Corporation 15 We see that we’ll need to extend STRMADMIN.COMPLAINTS25 since it belongs to the DEFECT_APPLY stream. TRANSFORMATIONS !!! We’ve done our sleuth-work. Now it’s time to add transformations on the rows being sent to us. BEGIN DBMS_STREAMS_ADM.RENAME_TABLE( rule_name => ‘STRMADMIN.COMPLAINTS25’, from_table_name => ‘hr.complaints’, to_table_name => ‘hr.defect_trends’, step_number => 0, operation => 'ADD'); END; / BEGIN DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'CUSTNO', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'CUSTLNAME', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'CUSTFNAME', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( LowMart Corporation 16 rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'COMPLAINT', value_type => '*', step_number => 1, operation => 'ADD'); END; / To display declarative transformations, run: COLUMN RULE_OWNER HEADING COLUMN RULE_NAME HEADING COLUMN DECLARATIVE_TYPE HEADING COLUMN PRECEDENCE HEADING COLUMN STEP_NUMBER HEADING SELECT FROM WHERE 'Rule Owner' 'Rule Name' 'Declarative|Type' 'Precedence' 'Step Number' FORMAT A15 FORMAT A15 FORMAT A15 FORMAT 99999 FORMAT 99999 RULE_OWNER, RULE_NAME, DECLARATIVE_TYPE, PRECEDENCE, STEP_NUMBER DBA_STREAMS_TRANSFORMATIONS TRANSFORM_TYPE = 'DECLARATIVE TRANSFORMATION'; Match up the rows: LowMart Corporation 17 Start DEFECT_APPLY with BEGIN DBMS_APPLY_ADM.START_APPLY( apply_name => 'defect_apply'); END; / Doesn’t appear like it’s working yet, though… getting error: Due to the step numbers assigned, turns out the ordering of my declarative rule-based transformations changed the name of the table first, and then tried to delete columns off the original name, which no longer exists. Removing the DELETE_COLUMN transformations and adding them back again with the proper table name (hr.defect_trends) worked great! BEGIN DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints ', column_name => 'CUSTNO', value_type => '*', step_number => 1, operation => 'REMOVE'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'CUSTLNAME', value_type => '*', step_number => 1, operation => 'REMOVE'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'CUSTFNAME', value_type => '*', step_number => 1, operation => 'REMOVE'); DBMS_STREAMS_ADM.DELETE_COLUMN( LowMart Corporation 18 rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.complaints', column_name => 'COMPLAINT', value_type => '*', step_number => 1, operation => 'REMOVE'); END; / Then finally: BEGIN DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.defect_trends', column_name => 'CUSTNO', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.defect_trends', column_name => 'CUSTLNAME', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.defect_trends', column_name => 'CUSTFNAME', value_type => '*', step_number => 1, operation => 'ADD'); DBMS_STREAMS_ADM.DELETE_COLUMN( rule_name => 'STRMADMIN.COMPLAINTS25', table_name => 'hr.defect_trends', column_name => 'COMPLAINT', value_type => '*', step_number => 1, operation => 'ADD'); END; / Inserting records to ora11 hr.complaints now successfully ships data to ora11b hr.complaints AND hr.defect_trends. LowMart Corporation 19 For good measure, run the streams health check script after uploading it to /tmp: $ sqlplus /nolog SQL> spool /tmp/mysite.html SQL> @/tmp/streams_hc_11_2_0_3.sql SQL> exit And open the generated mysite.html for review. LowMart Corporation 20 Fin. Done. <High Five> LowMart Corporation 21 Appendix CREATED COMPLAINTS TABLE IN SYSDBA INSTEAD OF HR Pretty self-explanatory… create the complaints table in HR schema instead. ENSURE THAT STRMADMIN HAS ALL THE NECESSARY PRIVILEGES SQL> connect / as sysdba Connected. SQL> GRANT SELECT ON CHANGE_SETS to strmadmin; Grant succeeded. SQL> GRANT EXECUTE on DBMS_CDC_PUBLISH to strmadmin; Grant succeeded. SQL> GRANT SELECT ANY TABLE to strmadmin; Grant succeeded. SQL> GRANT CREATE SESSION to strmadmin; Grant succeeded. SQL> GRANT CREATE TABLE to strmadmin; Grant succeeded. SQL> GRANT CREATE TABLESPACE to strmadmin; Grant succeeded. SQL> GRANT UNLIMITED TABLESPACE to strmadmin; Grant succeeded. SQL> GRANT SELECT_CATALOG_ROLE to strmadmin; Grant succeeded. SQL> GRANT EXECUTE_CATALOG_ROLE to strmadmin; Grant succeeded. SQL> GRANT CREATE SEQUENCE to strmadmin; Grant succeeded. SQL> GRANT DBA to strmadmin; Grant succeeded. SQL> GRANT EXECUTE on DBMS_CDC_PUBLISH to strmadmin; Grant succeeded. SQL> GRANT SELECT ANY TABLE to strmadmin; Grant succeeded. SQL> EXECUTE DBMS_STREAMS_AUTH.GRANT_ADMIN_PRIVILEGE(GRANTEE => 'strmadmin'); PL/SQL procedure successfully completed. LowMart Corporation 22 RUN THE HEALTH CHECK SCRIPT View the output: At this point, it seems my capture setup is not correct. STRMADMIN_CAPTURE. Is this on Prod? Or Warehouse? LowMart Corporation 23 THINGS TO LOOK OUT FOR/USEFUL STUFF : DBA_ALERT_HISTORY exec dbms_streams_adm.remove_streams_configuration; select propagation_name, error_message from dba_propagation; COLUMN OWNER HEADING 'Owner' FORMAT A10 COLUMN NAME HEADING 'Queue Name' FORMAT A28 COLUMN QUEUE_TABLE HEADING 'Queue Table' FORMAT A22 COLUMN USER_COMMENT HEADING 'Comment' FORMAT A15 SELECT q.OWNER, q.NAME, t.QUEUE_TABLE, q.USER_COMMENT FROM DBA_QUEUES q, DBA_QUEUE_TABLES t WHERE t.OBJECT_TYPE = 'SYS.ANYDATA' AND q.QUEUE_TABLE = t.QUEUE_TABLE AND q.OWNER = t.OWNER; Owner Queue Name Queue Table Comment --------------------------------------------------------------------------------------------------------------------------STRMADMIN AQ$_STREAMS_QUEUE_TABLE_E STREAMS_QUEUE_TABLE exception queue STRMADMIN STREAMS_QUEUE STREAMS_QUEUE_TABLE exec DBMS_STREAMS_ADM.REMOVE_QUEUE(‘STREAMS_QUEUE’,TRUE); Alert log of ora11b: LowMart Corporation 24 On target: SQL> select apply_name, queue_name from dba_apply; APPLY_NAME QUEUE_NAME -------------------------------------------------------------------------SYNC_APPLY STREAMS_QUEUE BEGIN DBMS_APPLY_ADM.ALTER_APPLY( apply_name => 'SYNC_APPLY', apply_user => 'HR' ); END; / MIKE’S DOCUMENT In your database, as sysdba, enter: SELECT * FROM V$OPTION WHERE PARAMETER = 'Oracle Database Vault'; If TRUE, you have DV installed. If FALSE, you are good and no concern with data vault in the process of implementing streams If TRUE, you need to disable that before proceeding with implementing streams. Here is how to do that: a. shutdown everything (asm, databases, listener) b. perform (as oracle user): $ cd $ORACLE_HOME/rdbms/lib $ make -f ins_rdbms.mk dv_off ioracle c. restart ASM, databases, listener You will also need to deal with log miner. the docs will tell you more on that but in general, streams plsql procedures will call upon logminer procedures and if not there, streams procedures will fail. create table complaints ( custno number(3) not null, custlname varchar2(20), custfname varchar2(20), product varchar2(10), complaint varchar2(100), defect varchar2(30)); 1. created a 2nd database called ora11b, used regular datafiles (non-asm database). Created an HR user in ora11b, granted connect, resource to HR. 2. ensured that the listener (which is controlled by the grid infrastructure) could listen for both ora11 and ora11b 3. saved off the entire server to ensure i had a good fail back server to go to after steps 1 and 2 were done. 4. setup streams requirements for the source database (ora11): setup streams admin user, streams ts, queues, etc. (per oracle documentation) 5. setup streams requirements for the target database (ora11b): same as above, per oracle documentation 6. much trial and error. Oracle documentation talks about using global names, logminer, etc. to set up the streams. That is true but you can also avoid global names altogether (smart idea) and use db_links as the method of moving replicated streams data over from source to target table. That is the route I suggest to consider. Here is a sample insert record I used for the source table (hr.complaints): insert into hr.complaints values (1,'johnston','dave','phone','does not work','yes'); The idea is to have this newly inserted record in the hr.complaints table replicate over to the ora11b database and into the hr.complaints table there. That's the goal. Sounds easy enough right? Hmmm... LowMart Corporation 25 for the new tablespace needed, as well as the streams administrator user, here is some handy code for that purpose: for source db (ora11 on asm) CREATE TABLESPACE streams_tbs DATAFILE '+DATA/ora11/datafile/streams' SIZE 25M AUTOEXTEND ON MAXSIZE 256M; CREATE USER strmadmin IDENTIFIED BY oracle DEFAULT TABLESPACE streams_tbs QUOTA UNLIMITED ON streams_tbs; grant connect, dba, imp_full_database, exp_full_database, aq_administrator_role to stradmin; for target db (ora11b: non asm): CREATE TABLESPACE streams_tbs DATAFILE '/u02/oradata/ora11b/streams.dbf' SIZE 25M AUTOEXTEND ON MAXSIZE 256M; CREATE USER strmadmin IDENTIFIED BY oracle DEFAULT TABLESPACE streams_tbs QUOTA UNLIMITED ON streams_tbs; grant connect, dba, imp_full_database, exp_full_database, aq_administrator_role to stradmin; At this point, you must setup queues, capture rules, propagation rules on the source database then do the same on the target database (per the documentation). Once done, and database links are built and working correctly, starting the apply process should render decent results (one hopes). This is usually where the troubleshooting begins and the time required to get this working gets more involved. To setup and test your database links from each database, you might like using this code: -- link in source database to connect to the target database CREATE DATABASE LINK ora11b_link CONNECT TO strmadmin IDENTIFIED BY oracle USING 'ora11b'; -- test link select * from dual@ora11b_link; Well, since I spent around 8 hours straight today working on the solution for the project, I will assume many of you did not do the same. So I will let you continue the effort given the above information as a decent road map on this project and see how you fair with it. http://blog.nominet.org.uk/tech/2011/06/03/how-to-corrupt-your-data-dictionary-with-oracle-streams/ I hit this very same issue yesterday as I was working on the implementation. This is a classic case of 'listening' to the documentation and then getting burned by accident. Obviously in this case, I reloaded my server and started the process again but the 2nd time around, I left those logminer objects right where they always live. In fact, since I mentioned to you about using database links to move the data, the logminer facility is not even used to accomplish the replication of the table data. So avoid, at all costs, the part where Oracle tells you to create a logminer tablespace and then move logminer objects into it. If you should ever drop that tablespace, you will wipe your database out pretty nicely. Not a good thing, as the blogger suggested, if you were implementing a streams solution on a production database.