Download Predictive Modeling in Automotive Direct Marketing: Tools

Predictive Modeling in Automotive Direct Marketing: Tools,
Experiences and Open Issues
Wendy Gersten, Rüdiger Wirth, Dirk Arndt
DaimlerChrysler AG, Research & Technology
Data Mining Solutions, FT3/AD
PO BOX 2360
89013 Ulm Germany
{wendy.gersten, ruediger.wirth, dirk.arndt} @
daimlerchrysler.com
projects with different departments within our company. We
present tools we developed to support knowledge transfer and
organizational learning.
ABSTRACT
Direct marketing is an increasingly popular application of data
mining. In this paper we summarize some of our own experiences
from various data mining application projects for direct
marketing. We focus on a particular project environment and
describe tools which address issues across the whole data mining
process. These tools include a Quick Reference Guide for the
standardization of the process and for user guidance and a library
of re-usable procedures in the commercial data mining tool
Clementine. We report experiences with these tools and identify
open issues requiring further research. In particular, we focus on
evaluation measures for predictive models.
In the next section, we introduce our application scenario with a
focus on technical challenges and requirements for solutions.
Then we describe tools we developed while working with several
different marketing departments. First, we present a Quick
Reference Guide for predictive modeling tasks in marketing
applications. This guide standardizes the process and guides end
users through data mining projects. The second set of tools
consists of re-usable procedures in the commercial data mining
tool Clementine. We present practical solutions to common data
mining problems like unbalanced training data and the evaluation
of models. For each of these tools we discuss our experience and
outline open issues for further research.
Categories and Subject Descriptors
H.2.8 Database Applications
Data mining
2. APPLICATION SCENARIO
Keywords
For the purpose of this paper we restrict ourselves to acquisition
campaigns. The challenge is to select prospects from an address
list who are likely to buy a Mercedes for the first time. We try to
derive profiles of Mercedes customers and use these profiles to
identify prospects. Although this sounds like a textbook data
mining application, there are many complications.
Clementine, CRISP-DM, Data Mining Process, Direct Marketing,
Evaluation Measures.
1. INTRODUCTION
There are many opportunities for data mining in automotive
industry. DaimlerChrysler's Data Mining Solutions department
works on different projects in fields like finance, manufacturing,
quality assurance, and marketing. Here, we focus on direct
marketing. Several authors [1, 10, 11] described problems and
solutions for data mining in direct marketing. In this paper we
reflect our own experience working in the automotive industry.
One of the major challenges is the fact that the buying decision
takes a long time and the relationship marketing action can only
contribute a small part. This makes the assessment of its success
and, consequently, the formulation of the proper prediction task
difficult. Furthermore, the process is not as stable and predictable
as one might expect. Typically, we cannot develop models of the
customers behavior based on previous examples of the same
behavior because in many cases we simply lack the data.
For various reasons, modeling the response to direct marketing
actions in the automotive industry is more difficult than for mail
order businesses. In this paper, we describe some results from
A more fundamental problem is that it is not clear what behavior
we want to model. Ideally, we want to increase the acquisition rate
not just the response rate. However, the acquisition rate is difficult
to handle. It is not easy to measure, it is captured fairly late (up to
9 months after modeling), and the impact of the predictive model
is difficult to assess.
Permission to make digital or hard copies of part or all of this work or
personal or classroom use is granted without fee provided that copies are not
made or distributed
for profit
or commercial
advantage
and that copies bear
LEAVE
THIS
TEXT BOX
IN PLACE
this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers, orAND
to redistribute
BLANKto lists, requires prior specific
permission and/or a fee.
KDD 2000, Boston, MA USA
© ACM 2000 1-58113-233-6/00/08 ...$5.00
Regardless of the success criterion, we face a technical problem
during training. Usually, measures like accuracy or lift are used to
398
evaluate models [3], implying that the higher the measure the
better the model. But this is not true, a model with a perfect lift is
not necessarily the best one (see section 4.4.1).
Furthermore, it points to a stream library, i.e., executable and
adaptable procedures in Clementine. Some of them are meant to
be used by marketing people, others facilitate the work of data
mining specialists. The experiences during the process are
archived for later reuse in an experienced base.
In addition, there are many other factors, like the available data or
the market situation, that make every predictive modeling project
unique in a certain sense. This gets even worse if we want the
process to be applicable in different countries with differences in
language, culture, laws, and competitive situation.
In the following sections we describe the Quick Reference Guide
and the Clementine stream library. We specify our particular
requirements, discuss the state of the art, outline our solutions and
identify open issues requiring new research.
Nevertheless, we aim for a standard approach and a set of tools
which should be usable by marketing people with only little data
mining skills. Although the projects differ in details there is a
large set of common issues which come up in most of these
projects. In order to support the construction of a common
experience base, the solutions and the lessons learned need to be
documented and made available throughout the organization.
3. QUICK REFERENCE GUIDE
Basically, the Quick Reference Guide is a specification of the
predictive modeling process with detailed advice for various tasks.
As such it should ensure a basic level of quality, guide
inexperienced users through the predictive modeling process, help
them using the data mining tool, and support the evaluation and
interpretation of results.
There are two groups of people involved. First, there are
marketing people who will do the address selection three to four
times a year and must be able to perform this process reliably.
They have little time and insufficient skills to experiment with
different approaches. Therefore, they need guidance and easy to
use software tools. Second, there is a data mining team which
does the initial case studies, develops and maintains the process,
trains the marketing people, and which will later support the
marketing people with more challenging application scenarios
outside the standard procedure. The data mining team needs
sophisticated data mining tools and a common framework for
documentation and knowledge transfer.
In the remainder of this section, we explain how we developed
this guide and our experience to date.
The Quick Reference Guide is based on CRISP-DM, a recently
proposed standard data mining process model [4]. CRISP-DM
breaks a data mining process down in phases and tasks. The
phases are shown in Figure 2.
To fulfill the needs of these two groups of people we developed
various tools (see Figure 1). This toolbox is organized along the
CRISP-DM data mining process model. We created a Quick
Reference Guide which is supposed to guide the marketing people
through a data mining project. The guide contains specific advice
for various tasks of the process and points to appropriate tools,
like questionnaires for problem understanding and check lists for
the evaluation of data providers.
Business
Understanding
Data
Understanding
Data
Preparation
Deployment
Data
Data
Data
Modelling
CRISP-DM Phases, Steps and Tasks
Data Understanding
Approach Definition
1
2
3
4
5
6
Guides
Evaluation
Figure 2. CRISP-DM Process Model
Experiences
• Quick Reference Guide
•
Questionnaires
• Advice Book
Templates
Project Documentation
Data Source Reports
At the task level, CRISP-DM distinguishes between a generic and
a specific process model (see Figure 3). The generic process
model is supposed to cover any data mining project. As such it
provides an excellent framework and is useful for planning the
projects, for communication both within and outside the project
team, for documentation of results and experiences, and for
quality assurance. However, it is too abstract to describe
repeatable processes for the marketing people. Such processes
typically take place in a certain context and under specific
constraints. To cover this, CRISP-DM allows for instantiations of
the generic process model. The resulting specific process models
Stream Library
Streams for
• data understanding
• data preparation
• modeling
• evaluation
• scoring
Figure 1. Toolbox
399
are meant to reflect the context, constraints and terminology of the
application. The Quick Reference Guide is a specific CRISP-DM
process model.
When developing the Quick Reference Guide, we got into the
following dilemma: On the one hand it is obviously impossible and after all even not desirable - to give a detailed and exhaustive
process description. This is due to the complexity resulting from
many unknown factors and unforeseeable situations that will
always remain. We experienced that even from campaign to
campaign in the same country the circumstances may differ
fundamentally. On the other hand, the opposite extreme, a very
high level description like the generic CRISP-DM process model,
is not a solution either. While it covers the whole process and is
useful for experienced people, it is not suitable for the kind of
users one is confronted with when moving into normal business
processes. The resulting process description should guide the user
as much as possible but, at the same time, enable him to handle
difficult unexpected situations. See [12] for more experiences
with using CRISP-DM in this application.
Phases
CRISP
Process Model
Generic Tasks
Mapping
Specialized Tasks
A major challenge is to put a project management framework
around this highly iterative, creative process with many parallel
activities. There must be firm deadlines for a task to be completed
to ensure timely completion and proper usage of resources. But
when is a task complete in a data mining project? This is a
question that cannot be answered easily. So far, we have not yet a
complete, satisfactory and operational set of criteria in our
application. One approach to define key performance indicators is
described in [6].
CRISP
Process
Process Instances
Figure 3. The four levels of the CRISP-DM methodology
Business
Understanding
Inventory
of
Resources
Situation
Assessment
Specify
Response
Modeling
Goals
Initial
Project
Plan
Data
Understanding
Initial
Data
Collection
Report
Import
Data into
Clementine
Data
Description
Data
Quality
Verification
Select
Working
Data
Data
Preperation
Select
Attributes
and
Datasets
Data
Cleaning
Derive
New
Attributes
Integrate
Data
Sources
Adjust
Modeling &
Scoring
Data
Review
Modeling
Approach
Generate
Test
Design
Set Up
Modeling
Stream(s)
Assess
First
Model
Results
Fine-Tune
Model
Parameters
Final Model
Assessment
Review
Process
Plan
Evaluation
Evaluate
Results
Quality
Assurance
Determine
Next
Steps
Plan
Scoring
Plan
Monitoring&
and
Maintenance
Apply
Predictive
Model
Run
Campaign
Evaluate
Outcome of
Campaign
Produce
Final
Report
Review
Project
Modeling
Understand
Planned
Marketing
Action
Deployment
The starting point for the Quick Reference Guide was the Generic
CRISP-DM User Guide. We deleted tasks and activities that were
not relevant for our application, renamed tasks to make them more
concrete, and added a few tasks at various places. The additions,
however, were not due to omissions in the generic model. It was
more like splitting one abstract task into several more concrete
tasks. The main work was to generate check lists of specific
activities for each task. There, the generic check lists were an
excellent guideline. Figure 4 shows the specific tasks of the Quick
Reference Guide.
Apart from the specification of the process, the Quick Reference
Guide contains pointers to examples from case studies,
documented experiences, and to the library of executable
procedures described in the next section.
So far, the experience with the Quick Reference Guide was mostly
positive. We expected it to be useful for planning, documentation,
and quality assurance, and this turned out to be the case.
However, its use for communication both within and outside the
project was much more advantageous than we originally
anticipated. Presenting the project plan and status reports in terms
of the process model and, of course, the fact that we followed a
process, inspired a lot of confidence in users and sponsors. It also
facilitated status meetings because the process model provided a
clear reference and a common terminology.
In our first case studies, we encountered an unexpected difficulty
with the process model. Although we stated frequently that the
phases and tasks are not supposed to be strictly sequential, the
clear and obvious presentation of the process model inevitably
created this impression in decision makers. Despite our
arguments, we found ourselves forced to very tight deadlines,
which in the end lead to sub-optimal solutions. On the other hand,
the process model gave us a structured documentation which
allowed us to justify how much effort was spent in which task.
This made it fairly easy to argue for more realistic resources in
later applications.
Develop
Initial
Modeling
Approach
Based on this experience, we advise in the Quick Reference Guide
to plan for iterations explicitly. As a rough guide we suggest to
plan for three iterations where each iteration is allocated half the
time of the preceding one. It is also stressed that it is never the
case that a phase is completely done before the subsequent phase
starts. The relation between phases is such that a phase cannot
start before the previous one has started. From this, it follows that
the distinction between phases is not sharp. But this is a rather
Figure 4. Specific Tasks from the Quick Reference Guide
400
academic question. In practice, all we need is a sensible grouping
of tasks.
Logical errors concern the content and meaning of data.
Typically, they are caused by false documentation, wrong
encoding or human misunderstanding. For instance, in one project
we used an attribute (nondependent variable) named
DealerDistance. This attribute was supposed to measure the
distance from a buyer's home to the closest car dealer. First
models included rules like “If DealerDistance is larger than
120 miles then the person is a car buyer.” These rules contradicted
the business knowledge. Closer inspection revealed, that for
buyers DealerDistance contained the distance between their
home and the dealer who sold the car. This is not necessarily the
nearest one.
4. CLEMENTINE STREAM LIBRARY
4.1 Overview
As mentioned above, we do different projects in the field of
marketing. However, there are always some business related
problems as well as repeatable tasks in common. Additionally, we
go through several loops between single tasks within one project
as shown in Figure 2. Therefore, we want to automate parts of the
data mining process. The basic idea is to develop tailored software
procedures supporting special tasks of the process. These
procedures should be modular, easy to handle and simple to
modify. Although they mostly assist within single tasks they must
have the ability to interact automatically in order to avoid human
errors.
Technical errors pertain to the form of data. They are usually
caused by false technical processing like wrong data formats,
incorrect merging of data sources or false SQL statements. For
instance, once we received data from two sources, the customer
data base and an external provider, both with the same file
structure. The modeling algorithm generated rules like “If
Wealth-Poverty-Factor is larger than 4.99 and smaller
than 5.01 then the person is a car owner.” Looking for an
explanation, we detected that the export from the data sources was
inconsistent. Wealth-Poverty-Factor contained real
numbers when exported from the customer database and integers
otherwise.
For our applications, we chose the data mining tool Clementine
from SPSS, mainly because its breadth of techniques, its process
support, and its scripting facilities. Using a visual programming
interface, Clementine offers rich facilities for exploring and
manipulating data. It also contains several modeling techniques
and offers standard graphics for visualization. The single
operations are represented by nodes which are linked on a
workspace to form a data flow, a so called stream.
Such errors are not easy to identify. On the one hand, we need
strong business knowledge to detect logical errors like the one
described above. On the other hand, one needs to be familiar with
data mining tools and techniques. Therefore, data mining
specialists and marketing people need to work closely together
during data understanding. Examples like the ones above are
contained in our toolbox.
Clementine is geared towards interactive, explorative data mining.
A scripting language allows the automation of repetitive tasks,
such as repeated experiments with different parameter settings.
Therefore, the tool serves our needs to automate the data mining
process very well. Nevertheless, there are also disadvantages.
Some useful functionality like gain charts or various basic
statistics are laborious to realize.
The examples also illustrate the usefulness of interpretable
models. Typically, we build a few models early in the process
because they are good indicators for data errors. When we are
fairly confident about the correctness of our data, we use
automation streams for mass modeling, a procedure described
below. Each automation stream can load defined files
automatically and save the results for the next step in process.
We built a library of streams supporting the process as described
in the Quick Reference Guide. The library contains three kinds of
streams:
Sample streams: These streams aim to show marketing people
how to implement important procedures in Clementine. They
illustrate basic techniques on the basis of tasks common in direct
marketing projects.
Automation streams: These streams help to automate the data
mining process. They are used to repeat one task very often under
different conditions, for example, to train a model with changing
parameter settings. They can be used by both marketing people
and data mining specialists.
Functionality streams: Some functionality is used in most of our
projects (e.g. score calculation) but not implemented in
Clementine from the outset. So we added these functions for easy
re-use.
The first two CRISP-DM phases supported are data understanding
and data preparation. Since these phases are highly explorative,
we mostly use sample and functionality streams. These phases are
the most time consuming ones. There are many traps one can
avoid using systematical discovery procedures as provided
through the stream library. In our projects we found two main
kinds of errors in data.
Figure 5. Numerical Model Evaluation Stream
401
systematically and saved for modeling by another stream from the
library.
Figure 5 shows an example of an automation stream used for
numerical model evaluation. The starlike icons are so-called
“Super Nodes” and represent procedures from our library (in this
case numerical quality measures for modeling (see 4.4.3)).
In our projects, we noticed that different training sets strongly
affect the model quality. But we did not yet discover a direct
correlation between properties of the training sets and the
algorithms or their parameter settings. Since we cannot narrow
down the most promising training sets early in the process, we
have to handle a large number of possible sets while modeling.
During model evaluation the situation gets even worse, as we will
discuss. Here, we need systematical tests in order to develop
approaches for fine-tuning training sets regarding properties of
input data and modeling algorithms (including their various
parameter settings) before the learning itself begins.
In the following sections, we will have a closer look at the process
from the modeling to the deployment phase. Therefore, we pick
some of the most important issues for predictive modeling in
direct marketing. Each issue is introduced first. Afterwards, we
discuss approaches, technical solutions, experiences, and the
needs for further research.
4.2 Creation of training and test subsets
Before modeling, we have to define a test design. The most
common approach is to split the data into training and test sets.
The training set is used to generate the models and the test set
serves to measure their quality on unseen data. An alternative
approach is to use resampling techniques, especially for small data
sets. Here we only discuss the former approach because its lower
computational requirements make it more efficient for mass
modeling (see section 4.3).
4.3 Mass Modeling
Today there are lots of techniques and software implementations
to built effective predictors. The question is not only to choose
between different techniques (like decision trees, rule sets, neural
networks, regressions, etc.) but also to choose between several
algorithms (e.g., CART, CHAID, C5.0, etc.). One also has to
consider the parameter settings of the algorithms and the various
training sets as discussed before. It is well known that there is no
generally best technique or algorithm. The choice depends on
many factors like input data, target variable, costs, or output
needed. Up to date we have no practicable approach to select even
the most promising combinations early in the process.
The first decision is to choose the sizes of the sets. In the best
case, we have enough data to create sufficiently large training and
test sets. Practically, we are often limited somehow. In case of
small data sets we do not have a general suggestion how large the
test set should be. It depends on the total number of records
available as well as on the number of records with positive target
variable. A large training set results in more stable models but
there have to be enough records in the test set to evaluate the
model quality. Normally, we reserve at least 20% of the sample
for testing. The remaining data are used to create various training
sets.
Fine-tuning just one modeling parameter after another makes it
really hard to consider all cross influences. Besides, the basic
functionality of some algorithms is not published sufficiently.
Then, it is difficult to judge the real influence of some parameters.
For these reasons we think it is necessary to experiment with
different combinations and, consequently, generate a whole
palette of possible models.
A major challenge in our marketing applications is the highly
unbalanced distribution of the target variable. Typically, we have
an overbalance of records with negative target variable (i.e., nonbuyers) [5, 10]. This makes it difficult to build good models that
find characteristics of the minority class. One way to approach
this problem is to create more or less balanced training sets.
According to our experiences, a ratio between 35:65 and 65:35 of
buyers to non-buyers is most effective [10]. But it differs between
the several projects and can change dramatically under some
circumstances. Of course, the distribution in the test set should be
representative for the universe and will not be changed.
For tuning the parameters, we need to automate the modeling
process. Especially, if we also take the various training sets into
consideration. For this purpose, we built several streams to
generate large numbers of predictive models. Each of the streams
deals with one technique and generates models with a whole range
of sensible parameter combinations.
Clementine offers several decision trees, rule sets, neural networks
and linear regression as techniques to build predictors. The
streams can be run independently of each other. All of them load
different training sets, train the algorithm with changing parameter
combinations on each set and save the generated models. This
way hundreds or thousands of models are generated in
multifarious facets, taking all reasonable possibilities into
consideration. We call this procedure mass modeling. Because of
the automatism we save time (computers work at night), avoid
human mistakes and secure best practice model quality.
There are two ways to change the balance within the training set.
We can either reduce the negative records or boost the positive
ones [8, 9]. This will affect the modeling results because we vary
amount and content of information for learning. To date, we have
not yet sufficiently explored the impact of the two alternatives.
The corresponding stream in our library is designed to handle
these issues flexibly. At first, it allows to choose the percentage of
data to be randomly assigned to the test set. Implicitly, we assume
that the sample is representative for the universe. This operation is
done only once and the same test set serves to evaluate all models.
The remaining data is used for generating multiple training sets,
where the stream automatically balances the data according to
different properties. With this stream, we set only some
parameters and generate multiple training sets with different sizes,
ratios and boosted or reduced records. All the subsets are named
Mass modeling is a pragmatic approach yielding good results in
real-life projects. But there are also some practical problems.
Dealing with such a large number of models places high
requirements on hard- and software and demands careful
preparation. As we will see later, it is also very hard to compare
that many models and pick the optimal one reliably. To improve
the process we need to develop procedures to focus only on the
most promising algorithms and parameter setting combinations.
402
Combinations not fitting to the overall project goal, input data and
other framework should be excluded safely and as soon as
possible.
4.4.2 Approaches to measure model quality
Histogram of scores shows the distribution of all records
according to their score overlaid with their real target value
(Mercedes buyers or non-buyers). The diagram gives evidence
how strong the model assigns records to one of the two classes
(see Figure 7 in section 4.5). In case of many records with
medium scores, the model’s discriminating power is very low.
However, the histogram tells us nothing about stability and
plausibility.
4.4 Evaluation
After models are generated through mass modeling they must be
evaluated.
4.4.1 Requirements of measures
The cumulative gain chart relates the number of objects with a
real positive target value, e.g., Mercedes buyers, to all objects in
the data base (buyers and non-buyers). The objects are sorted
descendingly according to their score. Ideally, all true positive
values should be situated at the left of the graph (see Figure 6).
[7] contains a thorough discussion of requirements and properties
of evaluation measures. In our projects we found the following
characteristics to be practically relevant for model quality:
Predictive accuracy: How accurate is the prediction of the
model? The predictive accuracy is high, if the model assigns a
score of 1 to objects belonging to class 1 and a score of 0 to
objects belonging to class 0.
Discriminatory power: This characteristic indicates the model's
ability to assign objects clearly to one of the two classes, even if
it assigns them to the wrong class. Discriminatory power is high if
the model assigns mostly high or low scores to the records, but
rarely medium scores. Assigning mostly medium scores indicates
a lack of discriminating information or mistakes in data
preparation phase.
Number of objects with
positive target value
(Mercedes-buyers)
Selection with Predictive Model
2500
Advantage
of
Predictive
Model
2000
1500
Random Selection
Stability: Stability indicates how models vary when generated
from different training sets. It implies three facets, overall
stability, stability per object and stability of the output form.
Overall stability means that models with identical parameter
settings generated on different training sets produce similar
results. Stability per object requires that the score for every object
is similar for models generated on different training sets. Under
stability of the output form we understand that the form of the
models is similar, e.g. different rule sets contain similar rules.
1000
500
0
0
10
20
30
40
50
60
70
80
90
100
% of objects (Mercedesbuyers and non-buyers)
Plausibility: Plausibility of models means that they do not
contradict business knowledge and other expectations. It requires
the possibility to interpret the results and is necessary for two
reasons. First, it increases the acceptance of predictive modeling
results and second, it facilitates individual and organizational
learning by recognizing mistakes and inconsistencies (e.g., logical
and technical errors). As such, it is a sort of quality assurance.
Plausibility can also be enhanced by univariate analyses of
important attributes and segments with high scores.
Figure 6. Cumulative Gain Chart
Only taking into account objects with positive target value
presupposes that it does not cause costs to predict a non-buyer as
buyer. This is not in concordance with reality. Furthermore the
measure says nothing about discriminatory power, plausibility and
stability of the model.
Histogram of scores and cumulative gain charts give visually a
good insight in model performance. However, it is not feasible to
compare hundreds of models. We overcome this disadvantage
using numerical indicators.
Accuracy and discriminatory power are often meant to be the most
important objectives of a good model. Although both properties
are important, they are not sufficient. In our setting, there are
many non-customers who resemble our customers but, for various
reasons, have not yet got around to buy our products.
Nevertheless, they are worth contacting to stimulate a purchase.
Therefore, models which perfectly separate customers and noncustomers are not sensible.
The quadratic predictive error gives evidence of the predictive
accuracy of all records and is calculated as sum of squared
deviations between real target value and predicted target value
[7]. By squaring the deviations, a high predictive error contributes
disproportionally more to the resulting measure.
Now the challenge consists in finding one or several measures to
operationalize these characteristics. In our toolbox, we use four
measures to evaluate the quality of the generated models. After
describing their functionality we discuss their weaknesses and
strengths.
A strength of this indicator is that false positive and false negative
predictive errors are taken into account and that deviations will be
registered exactly. But false positive and false negative predictive
errors are not weighted differently, even if – in practice – the error
403
to assign a Mercedes buyer as non-buyer is worse than the other
way round. Therefore, we also consider the squared errors of
positive and negative cases separately. Furthermore, there is no
benchmark what is a good or a bad predictive error.
attributes. Of course, the evaluation process is not always as linear
as described here.
The weighted lift [10] pursues the same goal as the lift curve. It
indicates whether the model assigns high scores to real positive
targets. But in contrast to the graphics this measure weighs true
positive targets with a high score much higher than those with a
low score. This has high practical relevance because mostly only
records with a high score are selected for actions. The maximum
value for the weighted lift is 100. A weighted lift of 50 (in case of
infinite number of partitions) corresponds to random selection.
None of the presented measures fulfills accuracy, discriminatory
power, stability and plausibility at once. Only a combination of
various numerical and graphical measures will give us a complete
picture of the models' characteristics.
4.4.4 Summary of evaluation
Accuracy is well represented in its diverse facets by all measures.
Discriminatory power cannot be calculated as indicator yet. Only
the histogram of scores gives us an impression of the separability.
But in order to compare hundreds of models easily there is a
strong need of an proper indicator of separability. Until now, we
are not able to measure stability and plausibility sufficiently
exactly. Concerning the stability of models we will try to measure
the confidence and the variability of results and integrate these as
confidence bounds in the weighted lift and the predictive error.
This probably allows us to decide whether it makes sense to try to
improve the model.
One problem is the fact, that weighted lifts based on different
number of partitions cannot be compared. The maximum lift of
100 can only be reached if all positive targets could theoretically
fall in the first partition. For instance, if there are 20% positive
targets in the set, there should not be more than 5 partitions. But
the lower the number of partitions, the higher the random lift and
the lower the advantage over random selection. In case of 5
partitions, the random weighted lift is 60. Besides, the weighted
lift tells us nothing about the discriminatory power of the model,
its stability or plausibility. Furthermore, there is the open
question, how many partitions to use. In our applications, we use
as many partitions as possible, as long as the size of a partition is
larger than the number of true targets, usually between 5 and 10.
In summary, we judge the weighted lift as a useful heuristics but
not more.
So far, we ignored costs of misclassifications. Even if costs are
relevant there exist high practical barriers to integrate them in
model evaluation. The main problem is that they are mostly
impractical to estimate reliably and consequently their impact can
only be guessed.
In general, we consider the evaluation of models in settings like
ours to be a largely open and challenging research issue. As we
mentioned at various places, there is no measure that tells us how
a good predictive model will perform in the field.
4.4.3 Realization
We have shown a few measures for model quality. They are
applied in different stages of the process. In the early phase, we
build a few models and inspect them carefully regarding potential
data errors. For this purpose, we use rule sets and judge their
plausibility. Accuracy and discriminatory power are assessed
visually. Only when the obvious data errors are repaired, we start
mass modeling.
4.5 Benefits
Before applying a promising model for the selection of addresses
to be included in the direct marketing campaign, its benefit must
be evaluated. This will decide whether to deploy the model or to
go back to a previous step. Furthermore, illustrating the benefit
when using data mining enhances acceptance and further
distribution of data mining within the enterprise. But what is the
benchmark to calculate the benefit of predictive modeling?
Since it is infeasible to compare hundreds of models visually, we
use the numerical model evaluation stream as shown in Figure 5.
There, we calculate weighted lift, quadratic predictive error and in
addition positive and negative quadratic predictive error. The
stream loads all models and calculates the measures automatically.
The output is a sorted list of model names and corresponding
indicators.
Comparing the model to random selection - as it is usually done is practically not very relevant. This would mean that no sensible
selection of prospects is done. But typically, some intelligent
selection according to common business sense is already
performed in the enterprise. It is often based on human experience
and two or three attributes (like high income, less than two
children and middle-aged). This intelligent selection should be
chosen as benchmark. The aim is to find out which objects are
assigned to the same class by both predictive model and
intelligent selection, and which objects are assigned to different
classes. Mainly the latter must be examined.
This way, many models can be compared. How many models can
be dealt depends on the actual computer. The performance of
Clementine Version 5.1 - the version we currently use - is mainly
limited by physical memory. This means that the whole process
must be broken down in steps so that Clementine can handle it
efficiently.
We built a Clementine stream to compare the predictive model
with such an intelligent selection. Figure 7 shows the results from
one of our business projects. Instead of 328 buyers correctly
recognized by intelligent selection, our predictive models assigned
a high score to 491 buyers. The advantage of the predictive model
over the intelligent selection increases when only objects with a
score higher than 0.8 are contacted.
Then, we pick only the best models for closer inspection with
streams for histogram of scores and cumulative gain chart.
Finally, we test the stability of the best few models using
resampling (10-fold cross-validation) and, in case of decision
trees and rule sets, we check the plausibility of the results by
inspection and with univariate analyses of the most relevant
404
The toolbox we use for our data mining applications in direct
marketing is pragmatically useful but not yet complete. Currently,
we are putting toolbox and experience documentation in a
hypermedia experience base to make storage and retrieval more
flexible and comfortable [2]. We are also developing additional
tools and addressing the important open issues of evaluation.
However, in this area, we require a lot more basic research which
also takes the business constraints into account.
Intelligent Selection
191 buyers
1373 Non-Buyers
328 Buyers
1273 Non-Buyers
Predictive Model
28 Buyers
1624 Non-Buyers
Another blank area is the question of what potential is really
hidden in the data. [11] describes an interesting heuristics for
estimating campaign benefits. As the authors note, the estimation
is valid only for applications where customer behavior is
predicted using previous examples of exactly the same customer
behavior. However, in our applications this assumption is rarely
met, especially not for acquisition campaigns. In our acquisition
campaigns we rarely know more than simple socio-demographic
facts. Estimating the expected lift is a promising approach.
However, the particular approach of [11] does not fit exactly our
experiences and needs. There is more work needed along theses
lines.
491 Buyers
1022 Non-Buyers
Figure 7. Benefit of Predictive Modeling compared to
Intelligent Selection
At this stage, plausibility of the results comes into play. Although
neural nets produce comparable results in terms of evaluation
measures discussed above, we favor decision trees or rules sets.
For them, it is relatively easy to demonstrate to business people
why they perform better than the common sense benchmark
(besides of helping us finding errors). Let us illustrate this
statement with an example from one of our applications. Usually,
we achieve good results with C5.0 using boosting. Typically, the
first classifier is very close to common sense selection, e.g., these
are rules referring to specific income ranges or socio-demographic
types, which are common selection criteria for manual scoring.
The subsequent classifiers typically are specializations of these
initial rules, e.g., socio-demographic types with additional
conditions. This way the boosted model confirms the common
sense selection and refines it.
As we tried to illustrate in this paper, it is not clear what kind of
behavior we need to model in automotive direct marketing.
Ideally, we want to model both dialogue affinity and propensity to
buy. But we usually cannot get hold of data containing the
necessary information. Typically, we have to make
approximations. What we then end up with is that we learn a
particular behavior of one population and try to adapt this to a
different population. It is an open issue how to evaluate models in
this setting.
Although this issue is of immediate practical importance,
Clementine does not sufficiently support the illustration of
boosted rules, requiring rather tedious manual post-processing of
the rules. There is certainly room for improvement.
[2] Bartlmae, K., A KDD Experience Factory: Using Textual
5. CONCLUSIONS
[3] Berry, M. J. A. and G. Linoff, Data Mining Techniques –
6. REFERENCES
[1] Bhattacharyya, S., Direct Marketing response models using
Genetic algorithms, Proceedings of the 4th International
Conference on Knowledge Discovery & Data Mining, pp.
144-148 (1998).
CBR for Reusing Lessons Learned, Proceedings of the 11th
International Conference on Database and Expert Systems
Applications (2000).
For Marketing, Sales and Customer Support, New York et
al. (1997).
A data mining process is a living process which will be changed
by future experiences. Therefore, all documentation, process
models and software tools must be flexible and living as well. The
Quick Reference Guide is an excellent framework for knowledge
transfer, documentation, communication, and quality assurance. It
structures the process while allowing for the necessary flexibility.
Having a standardized process helps to realize economies of scale
and scope, supports individual and organizational learning, and
speeds up the learning curve.
[4] CRISP-DM: Cross Industry Standard Process Model for
Data Mining, http://www.crisp-dm.org/home.html (2000).
[5] Fawcett, T. and F. Provost: Combining data mining and
machine learning for effective user profile. Proceedings. of
the Second International Conference on Knowledge and
Data Mining, pp. 8-13 (1996).
[6] Gersten, W., Einbindung des Predictive Modeling-Prozesses
When choosing a data mining tool you have to consider the users
of the tool. In our case, we chose Clementine because of its
process support, its various techniques and its ability to automate
streams via scripting. It was mainly meant to be used by data
mining experts. Although Clementine is targeted at business end
users, it requires quite skilled users. However, our stream library
aims to reduce the skills necessary to use the tool and, thus,
supports both data mining experts and marketing people.
in den Customer Relationship Management-Prozess:
Formulierung von Key Performance Indicators zur
Steuerung und Erfolgsmessung der Database MarketingAktionen eines Automobilproduzenten, Diploma Thesis,
University of Dresden (1999).
405
[7] Hand, D. J., Construction and Assessment of Classification
[11] Piatetsky-Shapiro, G. and B. Masand, Estimating Campaign
Rules, Chichester (1997).
Benefits and Modeling Lift, Proceedings of the 5th
International Conference on Knowledge Discovery & Data
Mining, pp. 185-193 (1999).
[8] Kubat, M. and S. Matwin, Addressing the curse of
unbalanced training sets, Proceedings of the 14th
International Conference on Machine Learning (1997).
[12] Wirth, R. and J. Hipp, CRISP-DM: Towards a Standard
Process Model for Data Mining, Proceedings of the 4th
International Conference on the Practical Application of
Knowledge Discovery and Data Mining, pp. 29-39 (2000).
[9] Lewis, D. and J. Catlett, Heterogeneous uncertainty sampling
for unsupervised learning, Proceedings of the 11th
International Conference on Machine Learning, pp. 148-156
(1994).
[10] Ling, C. X. and C. Li, Data Mining for Marketing: Problems
and Solutions, Proceedings of the 4th International
Conference on Knowledge Discovery & Data Mining, pp.
73-79 (1998).
406