Download Introduction

Data Warehousing & Business Intelligence
Introduction
What do you think of when you hear the
words Data Warehousing ?
Prithwis Mukerjee, Ph.D.
Conceptual DW Definition
Data warehousing is a
program dedicated to
the delivery of
information which
advances decision
making, improves
business practices, and
empowers workers.
© Prithwis Mukerjee
2
The Knowledge Management Framework
© Prithwis Mukerjee
3
How it all fits in ..
Transactional Systems
ERP : Enterprise Resource Planning
SCM :
Supply Chain Management
CRM :
Customer Relationship Management
Data Warehouse
Database
© Prithwis Mukerjee
4
Typical Business Uses of the Data Warehouse
Management
Reporting
Predict Customer
Behavior
Target
Advertising
campaigns
Profitability
Analysis
Strategic Initiatives
Market Basket
Analysis
Customer
Acquisition
and Retention
Business Processes
Determine
Customer
Lifetime Value
Human Resources
Management
© Prithwis Mukerjee
Functions
Category
Management
Just-in-Time
Inventory
Product
Pricing
Cross-selling
and upgrade
selling
5
Benefits of the Data Warehouse Program
Improves the way we do business and the bottom line
Performance
Analysis
Decision
Making
Market
Response
Competitive
advantage
Revenue Stimulation & Revenue Protection
Cost Reduction and Cost Avoidance
Productivity Improvement
Profitability Enhancement
© Prithwis Mukerjee
6
Non-integrated Decision Support Architecture
DSSs,Report writers,
Excel, databases, etc.
Inventory System
Budgeting
Order System
Analysis
Procurement
System
Accounting
System
© Prithwis Mukerjee
Sales Forecasting
Data Feeds
7
Basic Data Warehouse Architecture
Subject oriented Data
Warehouses or Data Marts
Inventory System
Order System
Enterprise
DW/ODS
Procurement
System
Fewer Data Feeds
Accounting
System
© Prithwis Mukerjee
One Stop
Data Shopping
8
Performance Measures : Definition & Examples
Carefully selected set of
measures derived from
strategies, goals and
objectives that
represents a tool to
communicating strategic
direction to the
organization for
motivating change.
These form the basis to
plan, budget, structure
the organization and to
control results.
© Prithwis Mukerjee
Customer
Measures
% Sales of New Products
Customers Acquired
Customer Satisfaction
Internal
Process
Measures
Product Time to Market
Unit Manufacturing Cost
Days Supply to inventory
Financial
Measures
Market Share
ROI and ROA
Revenue Growth
Innovation &
Learning
Measures
New Product Introduction
Management Skills
Employee Turnover
9
Differences between OLTP and DW
Data Access, Manipulation and Use
Data Organisation and Integration
Time Handling
Usage
Data Structures and Schemas
Explanations ..
© Prithwis Mukerjee
10
Data access, manipulation and use
Data Entry
Transaction Oriented
Consistent use patterns
Data retrievals are lookups of
single records
Users deal with one record at
a time
Performance is critical
Reporting is generally table
lists
OLTP
© Prithwis Mukerjee
Data Query
Bulk data oriented
Spiked, uneven use patterns
Queries are unpredictable,
they change continuously
Data retrievals are summary
and sorts of millions of
records
Performance is relaxed
(sec/min)
Reporting is primary activity
(on line, presented in small
chunks)
Differences between OLTP and DW
DW
11
Data Organisation And integration
Organized around applications
Unintegrated data
Different key structures
Different naming conventions
Different file formats
OLTP
© Prithwis Mukerjee
Organized around subject
areas
Integrated data
Standardized key structures
Standardized naming
conventions
Standardized file formats
Differences between OLTP and DW
DW
12
Time Handling
No time series analysis
Data relationships constantly
change
Changes are instantaneous
Limited history, 60-90 days
Twinkling Database ….
OLTP
© Prithwis Mukerjee
Time series analysis
Data is static over time
Series of data snapshots
Snapshots create historical
database, often greater than
two years
Quiet database
Differences between OLTP and DW
DW
13
Usage
Place an order for a product
Look up price for a product
Apply discount
Assign shipper
Trigger inventory pick-list
Verify shipment of product
Create invoice for the product
Apply credit to sales
representative
Essential to RUN the company
What type of customers are
ordering this product?
Who are my top 10% accounts? By
name, by revenue, by profitability,
by region?
How are these different by
customer segments? By sales rep?
By store?
Which shippers have the best on
time delivery records ?
How does this vary by shipment
size? By season of year?
Essential to WATCH the company
OLTP
© Prithwis Mukerjee
Differences between OLTP and DW
DW
14
Data Structures & Schemas
Drives out all data redundancy

Improves performance
Divides data into many
discrete entities
Tables are symmetrical

Can’t tell most important,
largest, which hold measures,
which are static descriptors
Lots of connection paths
between tables

prefers to use tables
individually or in pairs
Too complex for users to
understand
OLTP
© Prithwis Mukerjee
Data redundancy is
encouraged

Improves table browsing
Subject area oriented. Groups
data into categories of
business measure and
characteristics
Tables are symmetrical

Large dominant tables
Clearly defined connection
paths for table joins
Simple for users to
understand and navigate
Differences between OLTP and DW
DW
15
Basic Datawarehousing Topics
The Four Building Blocks
DW Definition
DW Usage and Benefits
DW Vs. the non-integrated
DSS environment
Performance Measures
© Prithwis Mukerjee
Dimensional Modeling
Technical Infrastructure
Knowledge Mgmt.
Architecture
IT and Business Perspectives
DW Methodology
16
Dimensional Data Modeling
Dimensional Data Modeling techniques organize the
content of the data warehouse. It structures the data
according to the way users ask business questions.
© Prithwis Mukerjee
17
The Technical Infrastructure
A technical infrastructure provides the physical
framework to support data acquisition, storage, access,
and data management. It involves development and
integration of hardware and software components.
© Prithwis Mukerjee
18
Knowledge Management Architecture
Metadata
Source Data
Invoicing
Systems
Purchasing
Systems
General
Ledger
Other
Internal
Systems
External Data
Sources
Extract
ODS
Purchasing
Data
Extraction
Integration
and
Cleansing
Marketing
and
Sales
Corporate
information
Product
Line
Processes
Location
Transform Data Warehouse Applications
Custom
Developed
Applications
Translate
Segmented
Attribute
Data
Subsets
Calculate
Synchronize
Summarized
Data
Data Resource Management And Quality Assurance.
© Prithwis Mukerjee
Statistical
Packages
Query
Access
Tools
Derive
Summarize
Data Mining
Data
Marts
19
The Business and The IT Perspective
Information
Technology
Business
Data
Warehouse
What will it do?
What value will it bring?
© Prithwis Mukerjee
How is it built?
How does it work?
20
The Business Perspective of the Data Warehouse
It takes forever to get the information I need to do my
job
When I do get it, it’s wrong
We have mountains of data, but I can’t figure out
what’s important
It takes so long to get the data that I don’t have any
time left over to analyze it
I want it to be easy. Just let me point and click my way
to an answer
I want to see my data in every possible combination
Data is scattered everywhere across our organization.
Where do I look ?
I want a historical view of the business
I want to predict the future
© Prithwis Mukerjee
Focuses on needs and usage
21
The IT Perspective of the Data Warehouse
Organizes and stores data by subject area rather than
application
Extracts and integrates data from multiple source
systems into a single database
Provides data cleansing, summarization, and calculation
User does not create, update, or delete data
Provides snapshots of data over periods of time
Supports analytical processing, not transactional
processing
Builds a technology infrastructure to support data
acquisition, data storage, data access, and metadata
capture
Focuses on database, technology, organizational features
© Prithwis Mukerjee
22
DW Methodology
The methodology provides a detailed roadmap to organize
and perform the tasks required in building the data
warehouse
© Prithwis Mukerjee
23
Data Warehouse System Development Life Cycle
ANALYSI
S
DESIGN
CONSTRUCTION
IMPLEMENTATION
Business Architecture
PLANNING
Data Architecture
MANAGING
Technology Architecture
Management Infrastructure
© Prithwis Mukerjee
24
stop
© Prithwis Mukerjee
25