Download of changed data

IBM Mainframe-Integration
Mainframe Change Data Capture
Presented by
The current situation:
Mainframe
Applications
+
Databases
Workstation/
Client-Server
Internet/Web
Applications
Mobile
Applications
+
Applications
+
Databases
+
Databases
Databases
The current situation:
Mainframe
Applications
+
Databases
• ADABAS
• IDMS/DB
• DB/2
• IMS and DL/I
• VSAM
• DATACOM/DB
• other…
The current situation:
Workstation
Client/Server
Applications
+
Databases
• SQL Server
• Oracle
• DB/2 UDB
• MS Access
• other…
The current situation:
Internet/Web
Applications
+
Databases
• Sybase
• MySQL
• DB/2 UDB
• ORACLE
• other…
The current situation:
Mobile
Applications
+
Databases
• MS SQL
• MS Access
• DB/2 UDB
• ORACLE
• other…
The current situation – Data exchange:
Data exchange problems
• Different data formats
• Different data models
• Large data volumes
• Limited batch window
• Requirement for up-to-dateness of
information
tcVISION – Mainframe Change Data Capture
The Solution
Moving data...
→ as much as needed
→ as little...
→ as transparent...
→ as flexible...
→ as secure...
AS POSSIBLE
tcVISION – Mainframe Change Data Capture
The new data exchange generation
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Basic questions:
1. How current must the information be?
Data exchange between source- and targetsystem should be ...
• continuous / in realtime
• cyclic / interval based
• event based
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Basic questions:
2. What are the attributes of the datasource?
The datasource attributes influence the choice for the
data capture method ...
• Size of datasource
• Format of datasource (Structure, DBMS)
• Change frequency
• Amount of changes and
• available resources (CPU-power, network
bandwidth)
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
• Usage of the DBMS logging capabilities
•IMS/DB, VSAM, DB/2, DL/I, ADABAS, IDMS, DATACOM
• Transfer of changed data in scheduled time frame
• Best for batch window
• Best for processing right after logfile creation
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
• Efficient transfer of entire databases
• Analyzis for data consistancy
• Best for „Initial Load“ prior to log processing
• Best for periodic mass data transfer
• One step data transfer
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
• Comparison of data snapshots
• Efficient transfer of changed data since last processing
• IMS/DB, DL/I, VSAM, DB/2, ADABAS, IDMS, DATACOM,
Sequential files
• Flexible processing options (SORT etc.)
• Automatic creation of deltas by tcVISION
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
• Realtime capture of changed data
• Changes directly obtained from DBMS
• IMS/DB, VSAM, DB/2, ADABAS, IDMS, DATACOM
• Secure data storage even across DBMS restart
• Flexible propagation methods
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Up-To-Dateness of changed data
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Data volume
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Steps:
•
Extraction of database logs
•
Process meta information
•
Process changes of selected
tables
•
Propagate changes
•
Implementation into target
database
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Steps:
•
Read datasource
•
Process meta
information
•
Selective and
structured transfer
•
Implementation into
target databse
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Steps:
•
Read datasource
•
Read
corresponding
snapshot
•
Determine changes
•
Process meta
information
•
Propagate changed
data
•
Implementation into
target database
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Steps:
•
Capture changes in DBMS
•
Transfer changes into „Pool
storage“
•
Parallel processing of changed
data from the pool
•
Propagation of data to the target
system
•
Implementation into target
database
tcVISION – Mainframe Change Data Capture
The tcVISION Data Capture Methods
Summary
Every method has been designed to meet
special needs and requirements. An
efficient and secure data synchronization is
guaranteed with all methods.
tcVISION – Mainframe Change Data Capture
Transfer and Propagation
Propagation to other mainframe systems
Propagation to Internet-databases
Propagation to mobile systems
Propagation to client server databases
tcVISION – Mainframe Change Data Capture
Transfer and Propagation
1.
Propagation independent of data
origin
2.
Propagation to different target
systems (even for the same
data)
tcVISION – Mainframe Change Data Capture
Examples
Case 1: DB/2 Log processing and
implementation into DB/2 UDB
tcVISION – Mainframe Change Data Capture
Examples
Case 2: DL/I Real-Time Capturing and
implementation into a DB/2 UDB
and VSAM
tcVISION – Mainframe Change Data Capture
Examples
Case 3:
IDMS- Realtime Capturing
and bi-directional synchronization with
ORACLE, DB2/UDB and MS-SQL
tcVISION – Mainframe Change Data Capture
Examples
DL/I
IDMS
SQLServer
All cases are Customer implementations,
that are currently in production.
tcVISION – Mainframe Change Data Capture
Examples
Steps:
1.
Import of the
metadata
2.
Assignment of target
tables
3.
Definition of
processing script
4.
Test and
implementation
tcVISION – Mainframe Change Data Capture
Examples
Import of meta data
The tcVISION Manager
publishes the layout of the logrecord for every table  for
later use by the processing
scripts
tcVISION – Mainframe Change Data Capture
Examples
Assignment of target
tables
Creation of target tables
or
Assignment of existing
tables in the target system.
tcVISION – Mainframe Change Data Capture
Examples
Definition of a
processing script
Use the tcVISION script wizard
to create the script
Definition of all required
parameter: Input and output,
LUW processing etc.
tcVISION – Mainframe Change Data Capture
EXAMPLES
Test and
implementation
Start and test the script
Implement script execution into
scheduler and automatic
processing
tcVISION – Mainframe Change Data Capture
Example
Technical aspects:
Standard data exchange was not
possible due to extremely large
amount of data.
• 760 tables
• 200.000 changes per day are
processed
• Propagation to DB2 UDB
under SUN Solaris
• Processing time with tcVISION
per day less than 30 minutes
tcVISION – Mainframe Change Data Capture
Examples
Summary
Just a few steps to implement
a data synchronization process
Solid monitoring and logging
High stability
Changes applied to structures
of source databases are
automatically processed
tcVISION – Mainframe Change Data Capture
Examples
Summary
Just a few steps to implement
a data synchronization process
Solid monitoring and logging
High stability
Changes applied to structures
of source databases are
automatically processed
tcVISION – Mainframe Change Data Capture
Examples
Steps:
1.
Define a collector
and pool storage
+
Definition of a
processing script
2.
Creation of target
table
3.
Test and
implementation
DL/I
tcVISION – Mainframe Change Data Capture
Examples
Creation of a collector/pool
combination for the DL/I
database
Definition of a processing
script
tcVISION – Mainframe Change Data Capture
Examples
Define Collector as DBMS Extension
Define Pool to buffer the changes
Define processing script to implement into
the target database
Use project wizard to specify the work-flow
tcVISION – Mainframe Change Data Capture
Examples
Test and implementation
Start the project using the
tcVISION Control Board
tcVISION – Mainframe Change Data Capture
Examples
Technical aspects:
Standard data exchange was not
possible due to extremely large
amount of data.
• 40 DLI segments
• 200.000.000+ records
• 150.000+ changes per day
• Implementation: Realtime
using DBMS Extension
Capturing of changes performed
by Online- and Batch-programs.
tcVISION – Mainframe Change Data Capture
Examples
Additional
Implementation:
If the target DB/2 is not
available, changes are
automatically saved into a
VSAM KSDS
 Highest possible recovery
Customer performs a
mainframe to mainframe
synchronization
tcVISION – Mainframe Change Data Capture
Examples
Steps:
Step 1:
1.
Create a
collector/pool
combination
+
Definition of a
processing script
2.
Creation of target
table
3.
Test and
implementation
IDMS
SQLServer
tcVISION – Mainframe Change Data Capture
Examples
Import of the SCHEMA
definitions for the IDMS
database
SQLServer
tcVISION – Mainframe Change Data Capture
Examples
Define Collector as DBMS Extension
Define Pool to buffer the changes
Define processing scripts to
implement the changes
Use the project wizard to define the
necessary steps
SQLServer
tcVISION – Mainframe Change Data Capture
Examples
Test and implementation
Start the project using the
tcVISION Control Board
tcVISION – Mainframe Change Data Capture
Examples
Technical aspects:
Conventional synchronization not
possible
• Bi-directional updates must be
supported
• Updates in DB2/UDB, ORACLE
and MS-SQL are captured by
TRIGGERS
• TRIGGERS invoke tcACCESS/ODBC
to perform IDMS updates
• tcVISION recognizes updates
from tcACCESS and prevents
a loop-back
SQLServer
tcVISION – Mainframe Change Data Capture
Mainframe
Applications
+
Databases
Workstation/
Client-Server
Internet/Web
Applications
Mobile
Applications
+
Applications
+
Databases
+
Databases
Databases
tcVISION – Mainframe Change Data Capture
Mainframe
Applications
+
Databases
Workstation/
Client-Server
Internet/Web
Applications
Mobile
Applications
+
Applications
+
Databases
+
Databases
Databases
tcVISION – Mainframe Change Data Capture
Data exchange challenges:
• Different data formats
• Different data models
• Large data volumes
• Limited batch window
• Requirement for
up-to-dateness of
information
tcVISION – Mainframe Change Data Capture
Data exchange challenges:
• Automatic data conversion into the
correct target format
• Different data models
• Large data volumes
• Limited batch window
• Requirement for
up-to-dateness of
information
tcVISION – Mainframe Change Data Capture
Data exchange challenges:
• Automatic data conversion into the
correct target format
• Adaptation of the data structure during
transfer
• Large data volumes
• Limited batch window
• Requirement for
up-to-dateness of
information
tcVISION – Mainframe Change Data Capture
Data exchange challenges:
• Automatic data conversion into the
correct target format
• Adaptation of the data structure during
transfer
• Only transfer changed data
• Limited batch window
• Requirement for
up-to-dateness of
information
tcVISION – Mainframe Change Data Capture
Data exchange challenges:
• Automatic data conversion into the
correct target format
• Adaptation of the data structure during
transfer
• Only transfer changed data
• Batch-Compare with operational database
• Requirement for
up-to-dateness of
information
tcVISION – Mainframe Change Data Capture
Data exchange challenges:
• Automatic data conversion into the
correct target format
• Adaptation of the data structure during
transfer
• Only transfer changed data
• Batch-Compare with operational database
• Real-Time Capturing meets maximum
requirements for up-to-date information
tcVISION – Mainframe Change Data Capture
The state-of-the-art demand for
Mainframe data exchange:
• Automatic data conversion into the
correct target format
• Adaptation of the data structure during
transfer
• Only transfer changed data
• Batch-Compare with operational database
• Real-Time Capturing meets maximum
requirements for up-to-date information
tcVISION – Mainframe Change Data Capture
YOU will profit from:
• Efficiency
• Transparency
• Auditability
• Up-to-dateness
www.treehouse.com