Download K-means

Chapter 5
Clustering Analysis
Minimize distance
But to Centers of Groups
結束
Contents
Describes the concept of clustering
Demonstrates representative clustering
algorithms on simple cases
Reviews real applications of clustering models
Shows the application of clustering models to
larger data sets
Demonstrates web plots through Tanagra
5-2
結束
Clustering
Clustering analysis is usually used as an initial analytic tool to
identify general groupings in the data and provide a numeric
means to describe underlying pattern.
Clustering analysis is a type of model, usually applied in the
process of data understanding.
Clustering involves the identification of groups of observations
measured over variables.
Discriminant analysis: where the groups are given as part of the
data, and the point is to predict group membership.
Cluster analysis: where the clusters are based on the data, and
not predetermined – the point being to find items that belong
together rather to predict group membership.
5-3
結束
Clustering
Clustering analysis allows the algorithm to determine
the number of clusters. The number of cluster may be
pre-specified.
First need to identify clusters
 Can be done automatically
 Often clusters determined by problem
Then simple matter to measure distance from new
observation to each cluster
 Use same measures as with memory-based reasoning
5-4
結束
Description of cluster analysis
The idea of clustering is to identify the average
characteristics of available measures for data in groups,
or clusters. New data can then be measured by their
distance from each of the cluster averages.
If data are totally independent Many clusters
If data are totally dependent  could be one cluster
5-5
結束
Clustering Uses
Segment customers
Find profitability of each, treat accordingly
Star classification:
Red giants, white dwarfs, normal
Brightness & temperature of starts used to
classify
U.S. Army
Identify sizes needed for female soldiers
(males – one size fits all)
5-6
結束
A clustering algorithm – K-means
1. Select the desired number of cluster k (or 2 to max)
2. Select k initial observations as seeds (arbitrary)
3. Calculate average cluster values over each variable
(for the initial iteration, this will simply be the initial
seed observations)
4. Assign each of the other training observations to the
closest cluster, as measured by squared distance)
5. Recalculate cluster averages based on the assignments
from step 4.
6. Iterate between steps 4 and 5 until the same set of
assignments are obtained.
5-7
結束
A clustering algorithm – K-means
K-means does not guarantee the same result no mater
what the initial seeds are.
Drawback of K-means:
The data needs to be standardized to get rid of the
differences in scale. (variables are equally important)
If there are some variables more important than others,
weights can be used in the distance calculation, but
determining these weights is another source of
uncertainty.
5-8
結束
Insurance Fraud Data
Claimant Age
52
38
21
36
19
41
38
33
18
26
Gender
Male
Male
Female
Female
Male
Male
Male
Female
Female
Male
Claim Amount Tickets Prior Claims
2000
0
1
1800
0
0
5600
1
2
3800
0
1
600
2
2
4200
1
2
2700
0
0
2500
0
1
1300
0
0
2600
2
0
Attorney
Jones
None
Simth
None
Adams
Simth
None
None
None
None
Outcome
OK
OK
Fraudulent
OK
OK
Fraudulent
OK
Fraudulent
OK
OK
Claimant Age
23
31
28
19
41
Gender
Male
Female
Male
Male
Male
Claim Amount Tickets Prior Claims
2800
1
0
1400
0
0
4200
2
3
2800
0
1
1600
0
0
Attorney
None
None
Simth
None
Henry
Outcome
OK
OK
Fraudulent
OK
OK
Test data set
5-9
結束
Standardizations,
examples
5-10
結束
Standardization and clustering
Age < 20 score=0.0
20 to 40  (age020)/20
.
See page 75
Less distance
See page. 77
5-11
結束
Standardization and clustering
Cluster #1: Cases 1, 2, 6, 7, 10
Cluster #2: Cases 3, 4, 5, 8, 9
Recalculate the cluster center
Age: (1+0.9+1+0.9+0.3)/5=0.82
Cluster #1: Cases 1, 2, 6, 7, 10
Cluster #2: Cases 3, 4, 5, 8, 9
5-12
結束
Standardization and clustering
Cluster #1: Cases 1, 2, 6, 7, 10
Cluster #2: Cases 3, 4, 5, 8, 9
Claimant Age
52
38
21
36
19
41
38
33
18
26
Gender
Male
Male
Female
Female
Male
Male
Male
Female
Female
Male
: higher age, all male, less fraudulent
Cluster #1
Claim Amount Tickets Prior Claims
2000
0
1
1800
0
0
5600
1
2
3800
0
1
600
2
2
4200
1
2
2700
0
0
2500
0
1
1300
0
0
2600
2
0
Attorney
Jones
None
Simth
None
Adams
Simth
None
None
None
None
Outcome
OK
OK
Fraudulent
OK
OK
Fraudulent
OK
Fraudulent
OK
OK
5-13
結束
Clustering
Cluster #1 and Cluster #2 Centers.
Test cases
The distance from Cluster 1 to Test case 1 would be:
(0.82-0.15)2+(1-1)2+(0.468-0.44)2 …=0.74
see page. 79.
Test cases 1, 3, 4, and 5 were all closer to Cluster #1.
Test case 2 was closer to Cluster #2.
5-14
結束
Weighted Distance Cluster model
0.01
0.2
0.2
0.1
0.29
0.2
Table 5.2
5-15
結束
Varying the No. of Clusters
Cluster analysis usually assumes the general situation
of an unknown optimal No. of clusters.
They could be a simple binary division, or it could
divide into as many groups as there are observations.
The best fit would be with the largest No. of divisions,
but they would not provide useful or actionable
information.
5-16
結束
Varying the No. of Clusters
5-17
結束
Standardization
5-18
結束
Standardized table
STD Table See Table 5.12
Case #3 (low expenditure on groceries) and Case #6 (high
expenditure on groceries) are the seeds at the first iteration.
The first iteration shown in Table 5.13, below:
5-19
結束
Clustering – two clusters
Case #1 to Cluster #1 and #2.
Treat other cases
5-20
結束
Clustering – 2nd iteration
2nd iteration
Cases 1, 2, 3  cluster #1
Cases 4 to 10  cluster #2
Center
5-21
結束
Clustering – 3rd iteration
Cluster #1  cases 1, 3, 8
Cluster #2  cases 2, 4, 5, 6, 7, 9, 10
Cluster #1  cases 1, 2, 3, 8
Cluster #2  cases 4, 5, 6, 7, 9, 10
5-22
結束
Clustering – 4th iteration
The same cluster groupings were obtained, the algorithm
stopped.
Cluster# 1 spends less on groceries, has a much lower married
status, fewer years on the job, fewer years of education, a much
lower driver’s license score, and much lower credit card user
and churn.
5-23
結束
The Three-Cluster model
Seeds: case 3 (low grocery expenditure), case 6 (high
grocery expenditure), and case 4 (a middle grocery
expenditure).
5-24
結束
The Three-Cluster model – 2nd iteration
Cluster #1: case 3
Cluster #2: cases 5, 6, 7, 8
Cluster #3: cases 1, 2, 9, 10
See page. 83 for more explanations
5-25
結束
Applications of Cluster analysis
Cases:
Using the data mining to monitor accounts for
difficulties in repayment.
Applying knowledge discovery using cluster analysis as
part of a data mining analysis of insurance fraud
detection and customer retention. Decision trees and
neural network algorithms were also used in the study.
See pages 84 and 85 for more detailed information
5-26
結束
Clustering Methods used in Software
Hierarchical clustering
 Number of clusters unspecified a priori
 Can not always separate overlapping clusters
 Two-step a form of hierarchical clustering
Bayesian clustering:
 based on probabilities, constructed with nodes representing
outcomes, and decision tree.
K-means clustering
Self-organizing maps
 Using Neural network to convert many dimensions into a small
number of dimensions, which has the benefit of eliminating
possible data flaws (noise, spurious relationship), outliers, or
missing values.
5-27
結束
Hybrids combine methods
K-means methods have been combined with selforganizing maps as well as genetic algorithms to
improve clustering performance.
Two-step clustering first compresses data into
subclusters, and then applies a statistical clustering
method to merge sub-clusters into larger clusters until
the desired No. of clusters is reached.
K-means algorithms work by defining a fixed No. of
clusters, and iteratively assigning records to clusters. In
each iteration, the cluster centers are redefined. As
demonstrations earlier.
5-28
結束
Kohonen self-organizing maps
Kohonen self-organizing maps (SOM), are neural
network applications to clustering. Input observations
are connected to a set of output layers, with each
connection having a strength (weight). A general
four-step process is applied:
1. Initialize map: a map with initialized reference vectors
is created, and algorithm parameters such as
neighborhood size and learning rate are set.
2. Determine wining node: for each inputs, select the best
matching node by minimizing the distance to an input
vector. The Euclidean normal is usually used.
5-29
結束
Kohonen self-organizing maps
3. Update reference vector: Reference vectors and its
neighborhood nodes are updated based on the learning
rule.
4. Iterate return to step 2 until the selected number of
epochs is reached, adjusting the neighborhood size.
Small maps (a few hundred nodes or less) are best.
Large neighborhood sizes and learning rates are
recommended initially, but can be decreased.
SOM is a useful tool for machine learning as applied
to cluster analysis.
5-30
結束
Application of methods to larger data sets
The business purpose is to identify the type of loan
applicants least likely to have repayment problems.
On-time is good. Late for bad.
Transformation
5-31
結束
Application of Loan
400 observations were identified by averaging the
attribute values for each given cluster). Sorting the 400
training cases into on-time and late categories and then
identifying the average performance by variable for
each group.
Younger
Worse risk
/credit
5-32
結束
Application of Loan
K-means algorithm is used to the application. The
outcome variable is output.
Paid on
-time
Better rate
Tested using the last 100 observations
5-33
結束
Application of Insurance Fraud
The business purpose is to identify the characteristics
of automobile insurance claims that are more likely to
be fraudulent.
4000 observations for training and 1000 for testing.
The transformed data illustrated in Table 5.1.
4000 observations were sorted by outcome, and
averages were later obtained for each group.
The two-step model was applied.
5-34
結束
Two-step modeling
The cases involving tickets, prior claims, or an attorney
are in Cluster #2.
Cluster #1 is older with little difference in gender or
claim amount.
Applying k-means algorithm, varying the No. of
clusters from two to five.
5-35
結束
Two-step modeling
The cluster #1 was similar for both algorithms, but
differences existed in Cluster #2 with respect to age,
prior claims, and tickets.
The point is that different algorithms yield slightly
different results, yet often yield similar conclusions.
5-36
結束
K-means – three clusters
All of the cases involving an attorney were assigned to
the cluster #3. This cluster had much higher prior
claims and tended to have higher late outcomes.
Cluster #1 and #2 were very similar, except for age.
5-37
結束
K-means – four clusters
All of the cases involving an attorney were assigned to
Cluster #4, involved much higher prior claims.
Interestingly, the presence of an attorney also seems to be
associated with lower claim amounts.
Cluster #3 and #4 are very similar, except for gender.
Cluster #1 is distinguished by a much younger average
membership, with higher ticket rate and higher prior claims.
5-38
結束
K-means – five clusters
Involving an attorney were found in Cluster #5 (had worst
outcome results.)
Cluster #1 had many males, and younger with high ticket
experience.
Cluster #2 was differentiated from Cluster #1 by age, all
female, with very low ticket experience and low prior
claims.
5-39
結束
British Credit Card Company
Compared clustering approaches with pattern
detection method
Used medians rather than centroids
More stable
Partitioned data
Clustering useful for general profile behavior
Pattern search method sought local clusters
Unable to partition entire data set
Identified a few groups with unusual behavior
5-40
結束
Insurance Claim Application
Large data warehouse of financial transactions &
claims
Customer retention very important
Recent heavy growth in policies
Decreased profitability
Used clustering to analyze claim patterns
Wanted hidden trends & patterns
5-41
結束
Insurance Claim Mining
Number of clusters
Too few – no discrimination – best here was 50
Used k-means algorithm to minimize least squared
error
Identified a few cluster with high claims frequency,
unprofitability
Compared 1998 data with 1996 data to find trends
Developed model to predict new policy holder
performance
Used for pricing
5-42
結束
Expenditure Data- two-step clustering
5-43
結束
Expenditure Data- Correlation Matrix
5-44