Download from data - VideoLectures.NET

ACAI’05/SEKT’05
ADVANCED COURSE ON KNOWLEDGE DISCOVERY
Data Mining and
Decision Support
Integration
Marko Bohanec
1
Jožef Stefan Institute
Department of Knowledge Technologies
&
University of Ljubljana
Faculty of Administration
Data Mining vs. Decision Support
knowledge discovery
from data
Use of models:
• classification
Data Mining
• clustering
model
data
• evaluation
• analysis
• visualization
modeling
• explanation
Decision Support
2
decision makers+
experts+
decision analysts
• ...
model
Overview
1. Decision Support:
–
–
–
–
–
Decision problem
Decision-making
Decision support
Decision analysis
Multi-attribute modeling
2. Decision Support and Data Mining
– How to combine and integrate DS and DM?
•
•
•
•
•
DS for DM
DM for DS
DM, then DS
DS, then DM
DM and DS
– DS for DM: ROC space
– DM and DS: Combining DEX and HINT
3
Literature
Part I: Basic Technologies
– Chapter 3: Decision Support
– Chapter 4: Integration of
Data Mining and Decision Support
Part II: Integration Aspects of DM and DS
– Chapter 7: DS for DM: ROC Analysis
Part III: Applications of DM and DS
– Chapter 15: Five Decision Support Applications
– Chapter 16: Large and Tall Buildings
– Chapter 17: Educational Planning
4
1. Decision Support
Decision Problem
Decision-Making
Decision Support
Decision Analysis
Multi-Attribute Modeling
5
Chapter 3 – M. Bohanec: Decision Support
Decision-Making
Decision:
The choice of one among a number of alternatives
Decision-Making:
A process of making the choice that includes:
• Assessing the problem
• Collecting and verifying information
• Identifying alternatives
• Anticipating consequences of decisions
• Making the choice using sound and logical
judgment based on available information
• Informing others of decision and rationale
• Evaluating decisions
6
Decision Problem
options
(alternatives)
goals
• FIND the option that best satisfies the
goals
• RANK options according to the goals
7
• ANALYSE, JUSTIFY, EXPLAIN, …,
the decision
Types of Decisions
• Easy (routine, everyday) vs. Difficult
(complex)
• One-Time vs. Recurring
• One-Stage vs. Sequential
• Single Objective vs. Multiple Objectives
• Individual vs. Group
• Structured vs. Unstructured
• Tactical, Operational, Strategic
8
Characteristics of Complex
Decisions
• Novelty
• Unclearness: Incomplete knowledge
about the problem
• Uncertainty: Outside events that cannot
be controlled
• Multiple objectives (possibly conflicting)
• Group decision-making
• Important consequences of the decision
• Limited resources
9
Decision-Making
10
Human DM
Machine DM
Decision Sciences
Decision Systems
• Switching circuits
• Processors
• Computer programs
• Systems for routine DM
• Autonomous agents
• Space probes
Decision-Making
Decision Sciences
Decision Systems
Normative
Descriptive
Decision Theory
Utility Theory
Game Theory
Theory of Choice

Cognitive Psychology
Social and Behavioral Sciences

11
Decision Support
Decision Support
Decision Support:
Methods and tools for supporting people involved in the decisionmaking process
Central Disciplines:
• Operations Research and Management Sciences
• Decision Analysis
• Decision Support Systems
Contributing and Related Disciplines:
• Decision Sciences (other than DS itself)
• Statistics, Applied Mathematics
• Computer Sciences:
Information Systems, Databases, Data Warehouses, OLAP
• Artificial Intelligence: Expert Systems, ML, NN, GA
• Knowledge Discovery from Databases and Data Mining
Other Methods and Tools:
• Representation and visualization tools
• Methods and tools for organizing data, facts, thoughts, ...
• Communication technology
• Mediation systems
12
Decision-Making
Decision Sciences
Normative Descriptive
13
OR/MS
Decision
trees
Decision Systems
Decision Support
DA
DSS
Influence Multi-attribute
diagrams
models
Other
Decision Analysis
Decision Analysis: Applied Decision Theory
Provides a framework for analyzing decision
problems by
• structuring and breaking them down into more
manageable parts,
• explicitly considering the:
–
–
–
–
possible alternatives,
available information
uncertainties involved, and
relevant preferences
• combining these to arrive at optimal (or "good")
decisions
14
The Decision Analysis
Process
15
Identify decision situation
and understand objectives
Identify alternatives
Decompose and model
• problem structure
• uncertainty
• preferences
Sensitivity Analyses
Choose best alternative
Implement Decision
Evaluation Models
options
16
EVALUATION
EVALUATION
MODEL
ANALYSIS
Types of Models in Decision
Analysis
Decision Trees
17
Multi-Attribute
Utility Models
Succeed
Invest
Fail
Investment
Do not invest
Influence Diagrams
Invest?
Costs
Risks
Results
Success?
Analytic Hierarchy Process
Return
Multi-Attribute Models
cars
18
buying
maint
PRICE
safety
CAR
doors
TECH
pers
COMF
lug
problem decomposition
Tree of Attributes
Decomposition of the problem to sub-problems
("Divide and Conquer!")
CAR
PRICE
BUYING
MAINTEN
The most difficult stage!
19
TECH.CHAR.
SAFETY
COMFORT
Utility Functions
(Aggregation)
Aggregation: bottom-up aggregation of
attributes’ values
CAR
PRICE
75%
BUYING
20
TECH.CHAR.
25%
MAINTEN
SAFETY
COMFORT
SAFETY
COMFORT
TECH.CH.
low
exc
unacc
high
low
unacc
med
accept
accept
high
good
exc
Evaluation and Analysis
21
•
EVALUATION
•
•
buying
maint
PRICE
safety
CAR
doors
pers
lug
TECH
COMF
direction: bottom-up
(terminal  root attributes)
result: each option evaluated
inaccurate/uncertain data?
Evaluation and Analysis
22
ANALYSIS
•
•
•
•
buying
maint
PRICE
safety
CAR
doors
pers
lug
TECH
COMF
interactive inspection
“what-if” analysis
sensitivity analysis
explanation
DEXi: Computer Program for
Multi-Attribute Decision Making
•
•
•
•
Creation and editing of
–
–
–
–
model structure (tree of attributes)
value scales of attributes
decision rules (incl. using weights)
options and their descriptions (data)
–
–
tables
charts
Evaluation of options
(can handle missing values)
“What-if” analysis
Reporting:
23
http://www-ai.ijs.si/MarkoBohanec/dexi.html
Some Application Areas
1.
INFORMATION TECHNOLOGY
•
•
•
2.
evaluation of computers
evaluation of software
evaluation of Web portals
•
evaluation of projects
evaluation of proposal and
investments
product portfolio evaluation
COMPANIES
•
•
24
business partner selection
performance evaluation of companies
PERSONNEL MANAGEMENT
•
•
•
•
PROJECTS
•
•
3.
4.
5.
MEDICINE and HEALTH-CARE
•
•
6.
personnel evaluation
selection and composition of
expert groups
evaluation of personal applications
educational planning
risk assessment
diagnosis and prognosis
OTHER AREAS
•
•
•
•
assessment of technologies
assessments in ecology and
environment
granting personal/corporate loans
choosing sports
Allocation of Housing Loans
25
Ownership
Present
Suitability
Solving
Housing
Stage
Work stage
Advantages
Earnings
Priority
Status
Maint/Employ
Health
Family
Soc-Health
Social
Age
Children
Medicine:
Breast Cancer Risk Assessment
RISK
Personal
characteristics
Hormonal
circumstances
Menstrual
cycle
Fertility
Oral
contracept.
Quetel's
index
Other
Cancerog.
exposure
Fertility
duration
Age
Family
history
Physical
factors
Reg. and
stab. of men.
First delivery
Menopause
Chemical
factors
26
Demograph.
circumstance
# deliveries
Bohanec, M., Zupan, B., Rajkovič, V.: Applications of qualitative multiattribute decision models in health care, International Journal of Medical
Informatics 58-59, 191-205, 2000.
Evaluation and Analysis of
Options
BREAST CANCER RISK
Hormonal circumstances
Menstrual cycle
Fertility duration
Reg./stab. menstruation
Fertility
Age
First delivery
# deliveries
Oral contraceptives
Personal characteristics
Quetel’s index
Family history
Menopause
Other
Cancerogenic exposure
Physical factors
Chemical factors
Demographical circumstances
27
Basic
evaluation
Missing data
“What-if”
analysis
3
2
moderate risk
average
R29+
moderate risk
over 40
29 or younger
up to 4
no
1
29+
no
no
high risk
high risk
higher
no
high risk
3
3/0,5,2/0,5
moderate risk
average
R29+
moderate risk
over 40
29 or younger
up to 4
*
1
29+
no
no
high risk
high risk
higher
*
high risk
2
2
moderate risk
average
R29+
moderate risk
over 40
29 or younger
up to 4
no
1
29+
no
no
moderate risk
moderate risk
lower
no
moderate risk
Selective Explanation of
Options
Reasons FOR higher risk
Age
over 40
Quetel’s index
29+
Other
high risk
Cancerogenic exposure high risk
Physical factors
higher
Demographic circumst.
high risk
28
Reasons AGAINST higher risk
Personal characteristics
1
Family history
no
Menopause
no
First delivery
29 or younger
Oral contraceptives
no
Chemical factors
no
Diabetic Foot Risk
Assessment
Who:
• General Hospital Novo Mesto, Slovenia
• IJS
• Infonet, d.o.o.
Why:
• Reduce the number of amputations
• Improve the risk assessment methodology
• Improve the DSS module of clinical information system
How:
• Develop multi-attribute risk assessment model
• Evaluate it on patient data (about 3400 patients)
• Integrate into the clinical information system
29
Chapter 15 – M. Bohanec, V. Rajkovič, B. Cestnik: 5 DS Applications
Diabetic Foot Risk Assessment
30
Model Structure
RISK
History
Ulcers
Amputations
Present status
Symptoms
Deformities
Tests
Other
changes
Loss of prot.
sensation
Absence
of pulse
2. Combining Data Mining and
Decision Support
31
How to combine DS and DM?
DS for DM: ROC space
DM and DS: Combining DEX and HINT
Chapter 4 – N. Lavrač, M. Bohanec: Integration of DM and DS
Data Mining vs. Decision Support
knowledge discovery
from data
Use of models:
• classification
Data Mining
• clustering
model
data
32
• evaluation
• analysis
• visualization
modeling
• explanation
Decision Support
decision makers+
experts+
decision analysts
• ...
model
DM + DS Integration ?
33
Data Mining
Decision Support
?
DM + DS Integration !
“DS for DM”
DM”
Data Mining
“DM and DS”
DS”
“DM for DS”
DS”
Decision Support
Data Mining
Through Model Development
Decision Support
Expertise
Data
Data Mining
Introducing DM methods into the DS process:
Decision support within the DM process
e.g., ROC curves
– MS SQL Server - Analysis Services
– model revision
Model
Model
Marko Bohanec
Marko Bohanec
Sequential Application:
“First DM,
DM, then DS”
Data
Mining
Sequential Application:
“First DS, then DM”
DM”
Decision
Support
Decision
Support
Model11
Model
34
Decision Support
Model22
Model
Parallel Applications:
Multiple DM models, then DS
Data
Mining
Model11
Model
Marko Bohanec
Data
Mining
Model22
Model
Model11
Model
Decision
Support
Model33
Model
Model22
Model
Marko Bohanec
Marko Bohanec
Marko Bohanec
Combining DM and DS
• “DS for DM”:
– ROC methodology
– meta-learning
• “DM for DS”:
– MS Analysis Services
– model revision (from data)
• “DM, then DS” (sequential application):
– Decisions-At-Hand approach
• “DS, then DM” (sequential application):
– using models in data pre-processing for DM
• “DM and DS” (parallel application):
35
– combining through models, e.g., DEXi and HINT
– considering different problem dimensions
“DS for DM”
36
Data Mining
Decision Support
Decision support within the DM process
e.g., ROC curves
ROC space
Classifier 1
• True positive rate =
#true pos. / #pos.
Predicted positive
Predicted negative
40
10
50
10
40
50
Positive examples
Negative examples
– TPr1 = 40/50 = 80%
– TPr2 = 30/50 = 60%
Positive examples
Negative examples
• False positive rate =
#false pos. / #neg.
• ROC space has
– FPr on X axis
– TPr on Y axis
37
Classifier 2
Predicted positive
Predicted negative
30
0
30
20
50
70
50
50
100
100%
80%
True posit ive rat e
– FPr1 = 10/50 = 20%
– FPr2 = 0/50 = 0%
50
50
100
60%
classifier 1
classifier 2
40%
20%
0%
0%
Chapter 7 – Slides by Peter Flach
20%
40%
60%
False posit ive rat e
80%
100%
The ROC convex hull
100%
true positive rate
80%
38
60%
40%
Confirmation rules
WRAcc
CN2
20%
0%
0%
20%
40%
60%
false positive rate
80%
100%
The ROC convex hull
100%
true positive rate
80%
39
60%
40%
20%
0%
0%
20%
40%
60%
false positive rate
80%
100%
Choosing a classifier
100%
FPcost 1
 2
FNcost
true positive rate
80%
40
60%
Neg
4
Pos
40%

20%
0%
0%
20%
40%
60%
80%
slope  4 2  2
100%

false positive rate

Choosing a classifier
100%
FPcost 1
 8
FNcost
true positive rate
80%
41
60%
Neg
4
Pos
40%

20%
0%
0%
20%
40%
60%
80%
slope  4 8  .5
100%

false positive rate

“DM for DS”
Data Mining
Introducing DM methods into the DS process:
42
– MS SQL Server - Analysis Services
– model revision
Decision Support
“DM for DS”: Model Revision
43
Sequential Application:
“First DS, then DM”
Decision
Support
44
Data
Mining
Model 1
Model 2
“First DS, then DM”
in Data Pre-Processing
RISK
History
Ulcers
45
Amputations
Present status
Symptoms
Deformities
Tests
Other
changes
Input attributes
Loss of prot.
sensation
Absence
of pulse
Generated attributes
Sequential Application:
“First DM, then DS”
Data
Mining
46
Decision
Support
Model 1
Model 2
Decisions-At-Hand Schema
Decision Support Shells …
… on Palm
Data Mining
(Model Construction)
47
Decision Model
in XML
(Synchronization or Upload)
Blaž Zupan et al.: http://www.ailab.si/app/palm/
… on the Web
“DM and DS”
Through Model Development
Requirements
Data
48
Data Mining
Decision Support
Model
Chapter 4 + references
Expertise
Common modeling formalism
Multi-Attribute Decision Models
Expertise
Data
49
Data Mining
Decision Support
Model
Qualitative Hierarchical
Multi-Attribute Decision Models
Model
1. Qualitative Multi-Attribute
Models
• Decomposition of the problem to less
complex subproblems
• Qualitative attributes
• Decision rules
safety COMFORT TECH
low
low
unacc
low
high
unacc
med
low
unacc
med
med
acc
med
high
good
high
low
unacc
high
high
exc
CAR
PRICE
buying
50
maint
TECH
safety
doors
COMFORT
pers
lug
Expertise
2. Expertise
Understanding of the decision problem and
ways for its solving by:
• Decision owner(s)
• Expert(s)
• Decision analyst(s)
• User(s)
3. Data
Data
Previously solved decision problems
• Attribute-value representation
51
4. DEX
"An Expert System Shell for Multi-Attribute Decision Making"
Functionality:
1. Acquisition of attributes and their hierarchy.
2. Acquisition and consistency checking of decision rules.
3. Description, evaluation and analysis of options.
4. Explanation of evaluation results.
Over 50 real-life applications:
•
Health-care
•
Education
•
Industry:
•
•
•
52
Land-use planning
Ecology
Evaluation of enterprises, products, projects, investments, ...
53
5. HINT
Hierarchy INduction Tool:
Automated development of hierarchical models from data
based on Function Decomposition
y
y
x1
lo
lo
lo
lo
lo
lo
med
med
hi
hi
x1
x2
lo
med
med
med
hi
hi
med
hi
lo
hi
x2
x3
lo
lo
lo
hi
lo
hi
lo
hi
lo
lo
y
lo
lo
lo
med
lo
hi
med
hi
hi
hi
x3
x1
lo
lo
lo
med
med
hi
c
1
2
3
1
3
1
y
lo
med
hi
med
hi
hi
x1
c
x2
lo
lo
med
med
hi
hi
x2
x3
lo
hi
lo
hi
lo
hi
c
1
1
1
2
1
3
x3
HINT: Further Information
54
http://magix.fri.uni-lj.si/hint/
HINT Implementation: In ORANGE
55
http://magix.fri.uni-lj.si/orange/
Application: Housing Loan
Allocation
•
•
•
User: Housing Fund of the Republic of Slovenia
Task: Allocating available funds to applicants for housing loans
Method:
Using a multi-attribute model for priority evaluation of applications
•
Supported by a DSS since 1991:
• Completed floats of loans: 21
• Applications: 44378 received, 27813 approved
• Allocated loans: 254 million € (2/3 of housing loans in Slovenia)
56
Modes of Operation
1. DEX only: from expertise
2. HINT only: from data
3. Supervised: from data under expert supervision
4. Serial: HINT-developed model subsequently refined by the expert
5. Parallel: parallel development of model(s) by DEX and HINT
6. Combined: combining sub-models developed in different ways
57
1. DEX-Only Mode
housing
house
stage
ownership
58
present
suitab
status
solving
earnings
cult_hist
Health
(1) normal
(1) normal
(1) normal
(2) priority
(2) priority
(2) priority
advantage
employed
soc_health
children
fin_sources
Social
(1) normal
(2) priority
(3) high_priority
(1) normal
(2) priority
(3) high_priority
health
social
family
Soc-Health
(1) normal
(2) priority
(3) high_priority
(3) high_priority
(3) high_priority
(3) high_priority
age
2. HINT-Only Mode (1 of 2)
Reconstruction of the original model from unstructured data:
• Real-life data from one float in 1994
• 1932 applications
• 12 attributes (2 to 5 values)
• 722 unique examples
• 3.7% coverage of the attribute space
• unsupervised decomposition
59
2. HINT-Only Mode (2 of 2)
Results:
• Relatively good overall structure
• Inappropriate structure around c3
• Excellent classification accuracy:
•
•
HINT:
C4.5:
94.7 ± 2.5 %
88.9 ± 3.9 %
housing
c7
c5
ownership
60
suitab
c8
c4
stage
c6
advantage
earnings
c1
children
c3
family
c2
employed
age
health
3. Supervised Mode (1 of 4)
Unstructured dataset:
housing
stage
own
sui
cult
adv
fin
earn
employ
child
Redundant: cult_hist, fin_sources
61
health
family
age
3. Supervised Mode (2 of 4)
All partitions with b=3 and minimal  (=3) [11 of 120]
suitab
advantage
advantage
advantage
earnings
earnings
advantage
stage
employed
employed
employed
employed
employed
employed
health
family
children
health
earnings
earnings
employed
employed
employed
employed
children
children
health
health
family
health
health
family
age
New concept: status
housing
stage
62
ownership
suitab
advantage
earnings
status
employed
health
children
family
age
3. Supervised Mode (3 of 4)
All partitions with b=3 and minimal  ( =4) [3 of 56]
ownership suitab
advantage
suitab
advantage stage
health
family
age
New concepts: social and then present
housing
stage
present
ownership
63
suitab
advantage
earnings
status
employed
social
children
health
family
age
3. Supervised Mode (4 of 4)
Final structure
housing
status
house
stage
present
ownership
advantage
earnings
employed
suitab
Results:
• Expert sastified with the structure
• Improved classification accuracy:
64
•
•
supervised:
unsupervised:
97.8 ± 1.8 %
94.7 ± 2.5 %
social
children
health
family
age
4. Serial Mode
1. Develop an initial model by HINT from data
2. Extend/enhance the model "manually" using
DEX
For example:
1. Take the model developed by HINT in supervised
mode
2. Add the attributes cult-hist and finsources:
65
– Extend the model structure
– Define the corresponding decision rules
5. Parallel Mode
Develop two or more independent models by HINT
and DEX for:
• comparison
• "second opinion"
• flexibility
For example, in this research we developed:
1. one DEX model
2. two HINT models: in supervised and unsupervised
mode
66
6. Combined Mode
Develop a single model using sub-models
developed
• by different methods and
• from different sources
Hypothetical example:
1. Develop subtree for status by HINT
2. Develop soc-health by HINT from a different
data set
3. A real-estate expert develops the house subtree
using DEX
4. All three models "glued" together in DEX by a
loan-allocation expert
67
DEX and HINT: Results
•
•
Integration of DM and DS for model-based problem solving
Requirements:
•
Offers a multitude of method combinations:
•
Specific schema:
•
68
–
–
–
common model representation
expertise and data (possibly partial)
methods for "automatic" (DM) and "manual" (DS) model
development
–
independent, serial, parallel, combined, …
–
–
–
qualitative hierarchical multi-attribute models
DEX as a DS method
HINT as a DM method
–
–
Application of DEX-only, HINT-only, supervised and parallel modes
Integration of DS and DM through HINT improved both the
classification accuracy and comprehensibility of the model
Real-world application: Housing loan allocation
Parallel Applications:
Multiple DM models, then DS
Data
Mining
69
Model 1
Decision
Support
Model 3
Model 2
Problem: Prediction of Academic
Achievement
Primary School
1
High School
7 8
1
2
3
4
...
5: graduates: 4 or 5
4: graduates: 2 or 3
3: prolonged
2: fails soon
Prediction
70
Chapter 17 – S. Gasar, M. Bohanec, V. Rajkovič
1: fails late
DM + DS Integration:
Academic Achivement
Data
DM: Weka
GA 1st grade
<=1
>1
LEGEND:
2
Slovene
GA 1st grade - general achievement of the first high
school grade
<= 3
>3
Slovene - mark of subject Slovene language
History - mark of subject History
GA 1st grade
Physics - mark of subject Physcis
<= 2
>2
age enrol - age at enrolment (in months)
unex ab 3rd sem - unexcused absence in the third
History
4
semester (hours)
<= 2
4
<= 1
>1
age enrol
<= 180
4
71
DS: DEXi
>2
Physics
unex abs 3rd sem
> 180
<= 0
1
4
5
>0
2
DM: HINT
DEXi
Drevo kriterijev
Kriterij
Opis
final achievement
c5
c1
for lang 8th grade
gen ach 7th grade
c2
regular enrol
for lang
c7
c3
citizenship
birth state
c6
gen ach prim sch
c4
math 8th grade
phys 8th grade
14.8.02
Parallel Application:
EC Harris
72
Models for
Client Value
Building Construction
Project Attributes
Decision Support
Models for
Building Feasibility
Data Mining
Building
Designs to
maximise
Client Value
Feasible
Building
Designs
Feasible
Zone
Shape
Value
Zone
Size
lity
ua
Q
Chapter 16 – Steve Moyle, Marko Bohanec, Eric Ostrowski
Conclusion
• DM & DS approaches are:
– complementary
– supplementary
• New and developing research area
• Typical combinations:
–
–
–
–
–
DS for DM
DM for DS
DM, then DS
DS, then DM
DM and DS
• Open questions:
– formalization (framework) of DM&DS integration
– common methodologies and approaches
– standardization
73