Download Incremental ETL Using CDC for SQL and Oracle with SQL Server

Processing Time vs. Data Volume
-- enable CDC on the database
exec sys.sp_cdc_enable_db
-- enable CDC on a table
exec sys.sp_cdc_enable_table
@source_schema = N'dbo',
@source_name = N'MyTable',
@role_name = N'cdc_admin',
@supports_net_changes = 1
Prepare database
Get last date
Read all from source
Read changes
Write to destination
Process changes
Record date
Save current date
Scenario 1
Mark Initial Load Start
Bulk load from source to destination
Mark Initial Load End
Scenario 2
SCN
LSN
cdc_states
__$reprocessing
True
special handling
All
All vs. Net
All
Net
All
Operation
Insert
Update
Insert
Delete
Update
Delete
Key
50
50
60
60
10
20
Name
Smith
Smyth
Jones
Jones
Williams
Martin
All
All with Old Values
Operation
Insert
Update Old
Update
Insert
Delete
Update Old
Update
Delete
__$update_mask
0xFF
0x80
0x80
0xFF
0xFF
0x80
0x80
0xFF
Key
50
50
50
60
60
10
10
20
Name
Smith
Smith
Smyth
Jones
Jones
William
Williams
Martin
All with Old Values
Net
Operation
Insert
Update
Delete
Key
50
10
20
Name
Smyth
Williams
Martin
Net
Net with Update Mask
Operation
Insert
Update
Delete
Key
50
10
20
__$Key_Changed
True
False
True
Name
Smyth
Williams
Martin
__$Name_Changed
True
True
True
Net with Update Mask
Net with Merge
Operation
Update
Update
Delete
Key
50
10
20
Name
Smyth
Williams
Martin
Net with Merge
CDC Instances
db_owner
sys.sp_replincrementlsn
10% overhead
http://msdn.microsoft.com/en-us/library/hh231087.aspx
http://msdn.microsoft.com/en-us/library/dd266396.aspx
http://msdn.microsoft.com/en-us/library/cc280519.aspx
DBI310: EIM: Bringing Together SSIS, MDS and DQS
DBI24-HOL: Exploring Microsoft SQL Server Integration Services
Exam 70-463: Implementing a Data Warehouse with Microsoft SQL Server 2012
[email protected]
Matt
Masson
mattmasson.com
@mattmasson
mva
http://northamerica.msteched.com
www.microsoft.com/learning
http://microsoft.com/technet
http://microsoft.com/msdn