Download Data Mining

Databases:
Visualization, Data Mining,
New DB Paradigms
Thomas Weik
FH Münster
9. Basic Mining Strategies
9.0 References
9.1 Motivation
9.2 Classification
9.3 Clustering
9.4 Association Rule Discovery
9.5 Challenges of Data Mining
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
2
9.0 References: Books
Books:
—  Witten, Eibe, Hall: Data Mining – Practical Machine Learning Tools
and Techniques; 3rd Edition, Morgan Kaufman 2011
—  Han et al.: Data Mining – Concepts and Techniques, Morgan
Kaufman 2011
—  North: Data Mining for the Masses: http://docs.rapid-i.com/files/
DataMiningForTheMasses.pdf
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
3
9.0 References: Software
Software:
—  WEKA: http://www.cs.waikato.ac.nz/ml/weka/
—  Rapid Miner: http://www.rapidminer.com
—  Manual: http://docs.rapid-i.com/files/rapidminer/rapidminer-5.0manual-english_v1.0.pdf
—  KNIME (Konstanz Information Miner): http://www.knime.org
—  R: CLI for Statistical Computing, Graphics and Data Mining:
http://www.r-project.org/
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
4
9.1 Why Mine Data?
—  There is often information “hidden” in the data that is
not readily evident
—  Human analysts may take weeks to discover useful information
—  Much of the data is
never analyzed at all
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
5
9.1 Orders of Magnitude
—  1 PB is enough to store the DNA of every person in the US
– with cloning it twice ...
—  AT&T transfers 30 PB of data through its network per day.
—  Until July 2012 CERN amassed about 200 PB of data
about 800 trillion collisions in search for the Higgs boson.
—  1 PB of MP3 encoded music plays continously for about
2000 years.
—  IDC: Total amount of global data was expected to grow to
2.7 ZB in 2012, which is an increase of 48% from 2011.
—  Whistleblower: NSA's Utah Data Center will have a capacity
of about 5 ZB when completed.
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
6
9.1 Orders of Magnitude
—  According to an IDC paper sponsored by EMC Corporation,
161 exabytes of data were created in 2006, "3 million times
the amount of information contained in all the books ever
written", with the number expected to hit 988 exabytes in
2010.
(Wikipedia.org)
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
7
9.1 Why Mine Data? Commercial Viewpoint
—  Lots of data is being collected
and warehoused
—  Web data, e-commerce
—  purchases at department/
grocery stores
—  Bank/Credit Card
transactions
—  Computers have become cheaper and
more powerful
—  Competitive Pressure is Strong
—  Provide better, customized services for an edge (e.g. in Customer Relationship
Management)
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
8
9.1 Why Mine Data? Scientific Viewpoint
—  Data collected and stored at
enormous speeds (GB/hour)
—  remote sensors on a satellite
—  telescopes scanning the skies
—  microarrays generating gene
expression data
—  scientific simulations
generating terabytes of data
—  Traditional techniques infeasible for raw data
—  Data mining may help scientists
—  in classifying and segmenting data
—  in Hypothesis Formation
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
9
9.1 What is (not) Data Mining?
●  What
is not Data
Mining?
●  What
is Data Mining?
–  Look up phone
number in phone
directory
–  Certain names are more
–  Query a Web
search engine for
information about
“Amazon”
–  Group together similar
documents returned by search
engine according to their
context (e.g. Amazon
rainforest, Amazon.com,)
Thomas Weik: DWH and Data Mining
prevalent in certain US
locations (O’Brien, O’Rurke,
O’Reilly… in Boston area)
WS 2014 / 2015
10
9.1 Origins of Data Mining
—  Draws ideas from machine learning/AI, pattern
recognition, statistics, and database systems
—  Traditional Techniques
may be unsuitable due to
—  Enormity of data
—  High dimensionality
of data
—  Heterogeneous,
distributed nature
of data
Statistics/
AI
Machine Learning/
Pattern
Recognition
Data Mining
Database
systems
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
11
9.1 What is Data Mining?
—  Many Definitions
—  Non-trivial extraction of implicit, previously unknown and potentially
useful information from data
—  Exploration & analysis, by automatic or
semi-automatic means, of
large quantities of data
in order to discover
meaningful patterns
—  Data Mining needs a process!
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
12
9.2 Classification: Definition
—  Given a collection of records (training set )
—  Each record contains a set of attributes, one of
the attributes is the class.
—  Find a model for class attribute as a function
of the values of other attributes.
—  Goal: previously unseen records should be
assigned a class as accurately as possible.
—  A test set is used to determine the accuracy of
the model. Usually, the given data set is divided
into training and test sets, with training set used
to build the model and test set used to validate
it.
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
13
9.2 Illustrating
Classification Task
Tid
Attrib1
Attrib2
Attrib3
Class
1
Yes
Large
125K
No
2
No
Medium
100K
No
3
No
Small
70K
No
4
Yes
Medium
120K
No
5
No
Large
95K
Yes
6
No
Medium
60K
No
7
Yes
Large
220K
No
8
No
Small
85K
Yes
9
No
Medium
75K
No
10
No
Small
90K
Yes
Learning
algorithm
Induction
Learn
Model
Model
10
Training Set
Tid
Attrib1
Attrib2
Attrib3
11
No
Small
55K
?
12
Yes
Medium
80K
?
13
Yes
Large
110K
?
14
No
Small
95K
?
15
No
Large
67K
?
Apply
Model
Class
Deduction
10
Test Set
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
14
9.2 Example of a Decision
Tree
Tid Refund Marital
Status
Taxable
Income Cheat
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
60K
Splitting Attributes
Refund
Yes
No
NO
MarSt
Single, Divorced
TaxInc
< 80K
NO
Married
NO
> 80K
YES
10
Training Data
Thomas Weik: DWH and Data Mining
Model: Decision Tree
WS 2014 / 2015
15
9.2 Classification
Techniques
—  Decision Tree based Methods
—  Rule-based Methods
—  Memory based reasoning
—  Neural Networks
—  Naïve Bayes and Bayesian Belief Networks
—  Support Vector Machines
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
16
9.2 Ex. for Classification
—  Sky Survey Cataloging
—  Goal: To predict class (star or galaxy) of sky objects,
especially visually faint ones, based on the telescopic
survey images (from Palomar Observatory).
—  3000 images with 23,040 x 23,040 pixels per image.
—  Approach:
—  Segment the image.
—  Measure image attributes (features) - 40 of them per
object.
—  Model the class based on these features.
—  Success Story: Could find 16 new high red-shift quasars,
some of the farthest objects that are difficult to find!
Thomas Weik: DWH and Data Mining
From [Fayyad, et.al.] Advances in Knowledge Discovery and Data Mining, 1996
WS 2014 / 2015
17
9.2 Classifying Galaxies
Early
Class:
•  Stages of Formation
Attributes:
•  Image features,
•  Characteristics of light
waves received, etc.
Intermediate
Late
Data Size:
•  72 million stars, 20 million galaxies
•  Object Catalog: 9 GB
•  Image Database: 150 GB
Courtesy: http://aps.umn.edu
Thomas Weik: Data Mining
WS 2014 / 2015
18
9.2 Examples of
Classification
—  Predicting tumor cells as benign or malignant
—  Classifying credit card transactions
as legitimate or fraudulent
—  Classifying secondary structures of protein
as alpha-helix, beta-sheet, or random
coil
—  Categorizing news stories as finance,
weather, entertainment, sports, etc
—  Gene defect analysis
—  Customer Rating
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
19
9.2 Constructing Decision
Trees: Another Example
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
20
9.2 Constructing Decision
Trees: Generic Algorithm
Generic recursive algorithm:
—  Select an attribute to place at the root node
—  Make one branch for every possible value
—  Thus the example set is split up into subsets
—  One for every value of the attribute
—  Repeat this process recursively for each branch
—  Use only instances that actually reach this branch
—  If all instances at a node have the same class value, then stop
developing that part of the tree
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
21
9.2 Constructing Decision
Trees: Problem
Which attribute
should we split on??
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
22
9.2 Constructing Decision
Trees: Information I
—  Any leaf with only one class will not have to be split
further
—  Of course we seek „small“ trees
—  Solution: Measure for „Purity“ of a node
—  Choose attribute which produces the purest daughter
nodes
—  Measure of Purity: Information (unit: bits)
—  For each node: expected amount of information to
classify a new instance („yes“ or „no“)
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
23
9.2 Constructing Decision
Trees: Information II
—  Calculation based on # of „yes“ and „no“ classes at
node
—  # of „yes“ and „no“ at the leaf nodes??
Required properties of Information:
—  When # of „yes“ or „no“ is zero, Information should be
0
—  When # of „yes“ and „no“ is equal, Information reaches
a max value
—  Measure should be applicable in multiclass situations
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
24
9.2 Expanded Tree Stumps
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
25
9.2 Resulting Decision Tree
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
26
9.2 The Measure: Entropy
—  Only one function satisfies all these properties:
—  Information value or entropy
—  entropy(p1, …, pn) = -p1 log p1 - … - pn log pn
—  Thus:
—  Info ([2,3]) = -2/5 x log 2/5 – 3/5 x log 3/5 = 0.971
bits
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
27
9.2 Improvements
—  This algorithm: ID3
—  Very robust
—  Later variant: C4.5
— 
— 
— 
— 
Also numeric attributes
Missing values
Noisy data
Generating rules from trees
—  Commercial version: C5.0
—  Some differences
—  Negligible improvements over C4.5
—  We used J48:
—  Implements C4.5 Rev. 8
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
28
9.3 Clustering: Application 1
—  Market Segmentation:
—  Goal: subdivide a market into distinct subsets of
customers where any subset may conceivably be
selected as a market target to be reached with a
distinct marketing mix.
—  Approach:
—  Collect different attributes of customers based on their
geographical and lifestyle related information.
—  Find clusters of similar customers.
—  Measure the clustering quality by observing buying
patterns of customers in same cluster vs. those from
different clusters.
Thomas Weik: Data Mining
WS 2014 / 2015
29
9.3 Clustering: Application 2
—  Document Clustering:
—  Goal: To find groups of documents that are similar to each
other based on the important terms appearing in them.
—  Approach: To identify frequently occurring terms in each
document. Form a similarity measure based on the
frequencies of different terms. Use it to cluster.
—  Gain: Information Retrieval can utilize the clusters to relate
a new document or search term to clustered documents.
Thomas Weik: Data Mining
WS 2014 / 2015
30
9.3 Illustrating Document
Clustering
—  Clustering Points: 3204 Articles of Los Angeles Times.
—  Similarity Measure: How many words are common in these documents
(after some word filtering).
Category
Total
Articles
Correctly
Placed
555
364
Foreign
341
260
National
273
36
Metro
943
746
Sports
738
573
Entertainment
354
278
Financial
Thomas Weik: Data Mining
WS 2014 / 2015
31
9.4 Association Rule
Discovery: Definition
—  Given a set of records each of which contain some number of items
from a given collection;
—  Produce dependency rules which will predict occurrence of an item based
on occurrences of other items.
TID
Items
1
2
3
4
5
Bread, Coke, Milk
Beer, Bread
Beer, Coke, Diaper, Milk
Beer, Bread, Diaper, Milk
Coke, Diaper, Milk
Thomas Weik: Data Mining
Rules Discovered:
{Milk} --> {Coke}
{Diaper, Milk} --> {Beer}
WS 2014 / 2015
32
9.4 Association Rule
Discovery: An Application
—  Marketing and Sales Promotion:
—  Let the rule discovered be
{Bagels, … } --> {Potato Chips}
—  Potato Chips as consequent => Can be used to
determine what should be done to boost its sales.
—  Bagels in the antecedent => Can be used to see which
products would be affected if the store discontinues
selling bagels.
—  Bagels in antecedent and Potato chips in consequent
=> Can be used to see what products should be sold
with Bagels to promote sale of Potato chips!
Thomas Weik: Data Mining
WS 2014 / 2015
33
9.4 Association Rule
Discovery: An Application II
—  Supermarket shelf management.
—  Goal: To identify items that are bought together by
sufficiently many customers.
—  Approach: Process the point-of-sale data collected with
barcode scanners to find dependencies among items.
—  A classic rule -—  If a customer buys diapers and milk, then he is very likely to
buy beer.
—  So, don’t be surprised if you find six-packs stacked next to
diapers!
Thomas Weik: Data Mining
WS 2014 / 2015
34
9.4 Constructing a Data Warehouse
for Breast Cancer Screening
—  One of the most widespread kinds of cancer
—  55,000 new cases / year in Germany (80,000,000
inhabitants)
—  Decreasing mortality, yet 18,000 deaths / year
—  Mammography-Screening state-sponsored program for early
detection through x-ray of breasts
—  Introduction of one Screening Unit per 1,000,000 inhabitants
—  Invitation of women by centralised unit
—  Reference Center for North-Rhine Westphalia: University
Clinique in Münster
Thomas Weik: Data Mining
WS 2014 / 2015
35
9.4 Goals of Screening
—  The earlier detected, the better the chances of cure
—  Small rate of false positives and false negatives
—  Decrease fear of x-ray (e.g. Tchernobyl)
—  Decrease of mortality
—  Rating of analogous and digital screening systems
—  Analysis of
—  screening participation trends and patterns,
—  applied x-ray doses
—  diagnoses
Thomas Weik: Data Mining
WS 2014 / 2015
36
9.4 Data Sources
—  Data of Central Invitation Units
—  No. of invitations, no. of acceptance, data about screening units
—  Data of medical doctors
—  No. of invitations / acceptance
—  Detected cases tumors
—  Results of biopsies (malign tumors, non-malign tumors, false
positives)
—  Distribution of tumor stages
—  Data about x-ray exposition of used screening equipment
—  Regional distribution etc.
—  Data about used screening technologies
—  Data about acceptance tests of used systems
Thomas Weik: Data Mining
WS 2014 / 2015
37
9.4 Extraction, Transform.
and Analysis
—  Legal Problems
—  Standard transformation problems
—  Large number of pre-defined reports
—  Opportunity for ad-hoc reports and queries
—  Data Mining and additional data sources will be added
in further project
—  Conclusions:
—  Functioning system created in limited time
—  Realisation of legal requirements and much more
—  Good extensibility
Thomas Weik: Data Mining
WS 2014 / 2015
38
9.5 Challenges of Data
Mining
—  Scalability
—  Dimensionality
—  Complex and Heterogeneous Data
—  Data Quality
—  Data Ownership and Distribution
—  Privacy Preservation
—  Streaming Data
Thomas Weik: DWH and Data Mining
WS 2014 / 2015
39