Download 1 Chapter Overview Transferring and Transforming Data Introducing

Chapter Overview
•
Transferring and Transforming Data
•
Introducing Microsoft Data Transformation
Services (DTS)
•
Transferring and Transforming Data with DTS
Graphical Tools
• Working with DTS Packages
•
Using the Bulk Copy Program (Bcp) and the
BULK INSERT Transact-SQL Statement
1
Data Import Preparation
Tasks
•
•
•
Verify internal consistency of data at data source.
•
Determine whether existing data columns need to be
aggregated or separated.
•
Determine whether data import is a one-time task or
a regularly occurring task.
•
Determine access rights to the source data.
Determine whether additional columns are needed.
Determine whether data format of source data
requires modification.
2
Data Transformations
•
Can data transformations be made in the data
source?
•
Should temporary tables and Transact-SQL
statements be used to scrub and cleanse the data
after import into Microsoft SQL Server?
•
Can Data Transformation Services be used to make
changes to the data during the import process itself?
•
Will transformations be applied to a regularly
scheduled import or is the import a one-time task?
3
Data Transfer Tools
•
DTS
•
Bcp
• BULK INSERT statement
4
DTS Process
5
DTS Connections
6
DTS Tasks
•
Discrete unit of work within a DTS package.
•
Can execute sequentially or in parallel.
• Available through DTS Import/Export wizard or
DTS Designer.
•
The DTS Parallel Data Pump task can only be
accessed programmatically.
7
DTS Tasks That Copy Data
and Data Objects
Bulk Insert task
Execute SQL task
Copy SQL Server Objects task
Transfer Database task
Transfer Error Messages task
Transfer Logins task
Transfer Jobs task
Transfer Master Stored
Procedures task
8
DTS Tasks That Transform
Data
•
Transform Data task
•
Data Driven Query task
9
DTS Tasks That Function
as Jobs
ActiveX Script task
Dynamic Properties task
Execute Package task
Execute Process task
File Transfer Protocol task
Send Mail task
10
DTS Transformations
Copy Column
ActiveX Script
Date Time String
Lowercase String
Uppercase String
Middle of String
Trim String
Read File
Write File
11
DTS Package Workflow
12
DTS Package Storage
•
Microsoft SQL Server 2000
•
Meta Data Services
• Microsoft Visual Basic file
•
Structured storage file
13
DTS Tools
•
DTS Import/Export wizard
•
DTS Designer
• Visual Basic/Microsoft Visual C++
•
DTS Run utility and Dtsrun command
14
DTS Import/Export Wizard
•
Define data to be imported, using a TransactSQL query to limit data if necessary.
• Select or create the destination tables if
necessary.
•
Change default column mappings if necessary.
• Choose objects to transfer if applicable.
•
Save and/or schedule the DTS package.
15
DTS Designer
16
Additional DTS Package
Functionality
•
Transaction support
•
Message Queue task
• Send Mail task
•
Programming templates
17
Storage in the msdb
Database
•
DTS packages can be saved in the msdb
database on any SQL Server instance.
• Each DTS package version is saved.
•
DTS packages saved to the msdb database can
be secured using an owner password and a user
password.
18
Storage Using Meta Data
Services
•
DTS packages can be saved to SQL Server Meta
Data Services on any SQL Server instance.
• Allows tracking of package version, meta data,
and data lineage.
•
Saved meta data includes data transformations
and data sources.
•
You must enable the tracking of data lineage
and create a column in the data destination for
the tracking information.
19
Storage Using a Structured
Storage File
•
DTS packages can be saved to an operating
system file.
• DTS packages saved to a structured storage file
can be secured using an owner password and a
user password.
•
Allows the DTS package to be moved, copied, or
mailed across the network.
•
Open files in SQL Server Enterprise Manager, or
use command-prompt utilities to execute them.
20
DTS Package Execution
Utilities
•
DTS Run utility (Dtsrunui.exe).
•
Dtsrun command.
• Packages run in the security context of the
logged-in user.
•
Scheduled packages run in the security context
of the SQL Server Agent.
21
Additional Options When
Working with DTS Packages
•
Use DTS package logs to record information
regarding the success or failure of a DTS
package.
•
Use DTS exception logs to record error
information about uncopied rows.
•
Performing disconnected edits is possible if one
or more of the data sources or data destinations
is unavailable.
22
Bulk Copying of Data Using
Text Files
•
Use either the Bcp command-prompt utility or
the BULK INSERT Transact-SQL statement.
• Used for importing large quantities of data at
high speed with minimal transformation.
•
Format files can be used to specify the format of
the data being imported.
•
DTS can be used with the BULK INSERT task to
handle formatting issues.
23
Optimization of Bulk Copy
Operations
•
Use Bulk-Logged Recovery during importing of a
large amount of data.
•
Use the TABLOCK hint and a large batch size when
importing a large amount of data from a single client
into an empty table.
•
It is generally faster to drop nonclustered indexes
before importing large amounts of data (and then
re-create the indexes).
•
If a table has a clustered index, it is generally faster
to order the data in the text file, if possible, and
then specify the ORDER hint.
24
Chapter Summary
•
Analyze your data before importing it to determine
the type of transformations, if any, that are required.
•
Use the DTS Import/Export wizard for simple
transformations and copying database objects.
•
Use DTS Designer for complex transformations and
workflow constraints.
•
Choose a DTS package storage format based on
usage of DTS packages.
•
Use the BULK INSERT task rather than creating
format files manually.
25