Download Chapter 1

Data Warehouse Fundamentals
Chapter 3
Trends In Data Warehousing
Paul K Chen
1
Data Warehousing is Becoming
Mainstream
In the early stages, four significant factors drove many
companies to move into data warehousing:




Fierce competition
Government deregulation
Need to revamp internal processes
Imperative for customized marketing
Walmart vs. Amazon.com

Walmart is the US company most quoted for the
successful application & deployment of Data
Warehousing technology.

Walmart filed lawsuit against Amazon.com for its
unlawful way of pirating its DW technology by hiring
away its DA personnel by offering hefty stock option
to these people.
Significant Factors
These significant factors reflect the new trends in data
warehousing:









Multiple Data Types
Data Visualization
Parallel Processing
Query Tools
Browser Tools
Data Fusion
Multidimensional Analysis
Agent Technology
E-Business- ERP, KM, CRM
Decision Making
and Data Warehousing
“A data warehouse is the data, processes, tools, and
facilities to manage and deliver complete, timely, accurate,
and understandable business information to authorized
individuals for effective decision making.”


Structured Data
– Includes traditional relational databases
– Typically internal and enterprise-owned
– Predetermined
Unstructured Data
– Includes articles, reports, images, and videos
– Utilizes external data and expert opinion
– Ad hoc
3
Decision Making
and Data Warehousing

Management Systems
– Extend relational databases to store and support multimedia
– User-defined types (UDT) and functions (UDF) in SQL-3

Specialized Servers
– Used for data which is incompatible with relational databases
(e.g., Streaming video servers)
– Objects may be linked to a relational database

Search Engines
– Query by Image Content (shape, color, texture, etc)
– Text retrieval on free-text documents
– Audio and video searching
Decision Making and Data Warehousing
~ The trend is toward unstructured data and ad hoc
warehouses. ~
~ Trend toward multimedia. ~
4
Types of Decision Support Tools





Data Inquiry
– A request for a set of data based on some search criteria
Data Interpretation
– Manipulation and visualization of a set of data (statistical
analysis)
Multidimensional Analysis (OLAP)
– n-dimensional spreadsheet analysis
Information Discovery
– Pattern recognition, trends
Browsers
– Search metadata catalogs
– Search information object lists
– Launch analysis tools
5
File-based Processing
6
Types of Decision Support Tools
~ Trend toward utilization of the Web, facilitated by Java. ~
7
Data Warehouse Architectures

Single Level
– Decision support tools access operational data directly
– Feasible only with “clean” data
– Valid for unstructured data

Two Level Reconciled
– Scrubbed operational data supporting ad hoc queries

Two Level Derived
– Summarized data

Three Level
– Maintains both scrubbed operational data, and summarized
data.
10
Data Warehouse Architectures
~ Trend toward multidimensional data. ~
11
Data Stores and Access Enablers




Specialized Multidimensional Databases
– Data is peregrinated and loaded into multidimensional
databases
– Long loading times but quick response
Relational-like Stores
– Indexing is used to proved pseudo-multidimensional
functionality
Relational Data Stores
– An extra semantic layer generates multidimensional data on
the fly
Hybrids
– Details are stored in a traditional relational format
– A subset is cached in a multidimensional data structure
12
Database Management System (DBMS)
13
Data Stores and Access Enablers

~ Trend toward multidimensional data.
14
Metadata




Integrated Components
– All components (sources, stores, etc) use a common metadata
repository to maintain their metadata
Standardized Metadata Interchange
– Components keep their own metadata
– Components use a common interchange information model
and syntax to share metadata
Synchronized Metadata Interchange
– Metadata changes are updated automatically across all
components
Building of Business Metadata
– Manually entered, free-text, plain language descriptions
Metadata
~ Trend toward better metadata, exchanged between systems. ~
Middleware - Gluing the Warehouse
Together

Definition: software that shields users and developers
from differences in services and resources used by
applications

Data warehouses often have heterogeneous databases,
operating systems, networks, hardware, applications
Business Issues for Middleware


Role of middleware
– Assist developer in data extraction/transformation
and populating DW
– Assist business user in accessing DW
– Therefore needed at different points in life cycle
Types
– Copy management: data extraction,
transformation, replication, and propagation
– Gateways: DB and independent gateways
– Program-to program: RPCs, TP monitors, ORBs
– Message-oriented
Data Quality



Preprocessing Ownership
– Source application owners know their data
– Warehouse owners still must integrate the entire system
Automated Preprocessing Tools
– Specialized packages
– Generalized tools using pattern processing, lexical analysis,
and statistical matching to reconcile a wide range of data
sources
– Custom programming
Reliability and Credibility of External Data
– Quality ratings
– Posted statistical meta-information (sample size, randomness,
etc)
15
Data Quality
Trend toward better understanding of data quality. ~
16
Significant Trends- Multiple Data Types
Image
Spatial
Structured Numeric
Video
Structured Text
Data Warehouse
Repository
Audio
17
Significant Trends- Data Visualization




More Chart Types-Pie chart, scatter plot
Interactive Visualization
Chart Manipulation
Drill Down
Significant Trends- Parallel Processing

Aims to solve decision-support problems using multiple nodes
working on the same problem.

Performs many database operations simultaneously, splitting
individual tasks into smaller parts so that tasks can be spread
across multiple processors.

Parallel DBMSs must be capable of running parallel queries,
parallel data loading, table scanning, and data archiving, and back
up.
Significant Trends- Parallel Processing
Shared memory architecture (SMP)
– All the servers share all the data
 Shared nothing architecture (MPP)
– Each server has its own partition of data

Significant Trends- Query Tools, Browse
Tools







Flexible Presentation –online results and report
generator
Aggregate Awareness
Crossing Subject Areas
Multiple Heterogeneous Sources
Integration
Overcoming SQL Limitations
Data Fusion
Significant Trends- Integrating ERP and
Data Warehouse



Option 1: Companies implement the data warehouse
solutions of the ERP vendor with the currently
available functionality and await the enhancements.
Option 2: Companies implement customized data
warehouse and use third-party tools to extract data
from the ERP datasets. Retrieving and loading data
from the proprietary ERP datasets is not easy.
Option 3: It is a hybrid approach that combines the
functionalities provided by the vendor’s data
warehouse with additional functionalities from thirdparty tools.
Significant Trends- Integrating KM and
Data Warehouse
What’s KM?

It is a systematic process for capturing, integrating,
organizing, and communicating knowledge
accumulated by employees.

It is a vehicle to share corporate knowledge so that
employees may be more more effective and be
productive in their work.

A knowledge management system must store all such
knowledge in a knowledge repository.
Significant Trends- Integrating KM and
Data Warehouse
A specific corporate scenario:

Sales have dropped in the South region.

Your marketing VP is able to discern this from your data
warehouse by running some queries and doing some preliminary
analysis. If he or she has access to a document prepared by an
analyst explaining why the sales are low and suggesting remedial
action.

Knowledge must be linked to the sales result to provide context to
the sales numbers from the data warehouse.
Significant Trends- Integrating KM and
Data Warehouse
An airplane sales scenario: The following information is essential
For a successful pitch for airplane sales.




Model configuration
Production schedule (Delivery schedule)
Part replacement
Warranty
Knowledge obtained from the knowledge management
system can provide context to the information received from
the data warehouse to understand the story behind the above
information.
Summary of Trends








Ad Hoc Questions
Multidimensional Analysis (OLAP)
Web-Enabled Data Warehouse
Multimedia
Middleware
Metadata Interchange
Integrating ERP with Data Warehouse
Integrating KM with Data Warehouse
Complete E-Business Suite– A Review
ERP
Marketing
EAI
Sales
Projects
Financial
Services
Order Mgt
One Database
Human
Resources
Customer
Manufacturing
Relationship(CRM)
Procurement
Supply Chain (SCM)
Information System Categories
Information System Categories
Data Warehouse & ERP
– ERP = Enterprise Resource Planning
– A software solution that addresses enterprise needs
taking the process view of an organization to meet the
organization goals tightly integrating all the functions
of an organization.
-- It integrates all the departments and functions across
a company into a single computer system that can
serve all those different departments’ particular
needs.
WHY ERP?

Business
Customer satisfaction
Business development – new areas, products and
services
Ability to face competition
Efficient processes required for company’s growth

IT
Present software does not met business needs.
Legacy systems difficult to maintain
Obsolete hardware/software difficult to maintain
How ERP?

ERP Combines various department systems into a
single, integrated software program that runs off a
single database so that the various departments can
more easily share information and communicate with
each other.

The best part of ERP is the way in which it improves
the order fulfillment process that is taking the
customer order and process it into an invoice and
revenue.

It doesn’t handle the front-end that is handled by
CRM (Customer Relationship Management).
How ERP?(cont’d)

When a customer service representative enters a customer order
to an ERP system, he has all the information necessary to
complete the order such as customer’s credit rating and order
history from the finance module, the company’s inventory levels
from the warehouse module and the shipping dock’s trucking
schedule from the logistics module.

How it’s being done: It integrates the financial information and
customer order information . It does so by integrating the
following:
Database
Application
Interfaces
Tools
BPR
How ERP? (cont’d)

It standardizes and speeds up the manufacturing
process. This saves time, increases productivity and
reduces head count.

It reduces the inventory. Due to the information
available about all the orders it helps to maintain the
right level of stock and smoothes the manufacturing
process.
Data Warehouse & EAI

What is EAI? EAI refers to Enterprise Application Integration.
EAI is the merging of applications and data from various new
and legacy systems within a business. Various means are
employed to accomplish EAI, including middleware, in order to
unify IT resources, maximize new ERP investments, diminish
errors and get everyone on the same page. EAI enables
companies to link their existing software applications with each
other and with portals. EAI provide the ability to get their
applications to exchange critical data. EAI is usually close to the
top of any CIO's list of concerns. There are different approaches
to EAI. Some rely on linking specific applications with tailored
code, but most rely on generic solutions, typically called
middleware. XML, combined with SOAP and UDDI is a kind of
middleware.
E-Business
~ Trend toward better understanding as well as consolidation
of internal processes and data ~
~ Trend toward web-enabled data warehouse. ~
Tugas
 Bagaimana
agar data yang ada menghasilkan
informasi yang berguna dalam pengambilan
keputusan. Jelaskan
 Bila Anda sebagai Database Administrator apa
yang akan Anda lakukan terhadap data yang ada?