Download Steven F. Ashby Center for Applied Scientific Computing

Introduction to Data Mining
(these slides are based on a variety of sources)
CISC 4631: Data Mining Fall 2010
1
Let’s Start By Seeing What you Know

Quick Quiz
Do you know what Data Mining is?
 Do you know of any examples of Data Mining?

CISC 4631: Data Mining
2
What is Data Mining?
 Data
Mining has many definitions
 Non-trivial
extraction of implicit, previously
unknown and potentially useful information from
data
 Exploration & analysis, by automatic or
semi-automatic means, of large quantities of data
in order to discover meaningful patterns
CISC 4631: Data Mining
3
Alternative Names

Data Mining also known as or related to:

Knowledge discovery in databases (KDD)

Knowledge extraction

Data/pattern analysis

Data archeology, data dredging, information
harvesting, business intelligence, etc.
CSRU4631: Data Mining
4
Some Examples


Netflix and Amazon use data mining to
recommend products (recommender systems)
Companies use data mining for marketing
Who should be mailed a catalog
 Who should see what online ads (Google Adwords)


My WISDM project uses data mining to
determine (from your cell phone accelerometer
data) who you are and what you are doing
CISC 4631: Data Mining
5
Why Data Mining and Why Now?


Data Mining was not very popular until the last
10 years or so.
Quick Quiz:
Why is it data mining popular now?
 What changed?

CISC 4631: Data Mining
6
Why Mine Data?

There are now tremendous amounts of data that
are automatically collected and warehoused.
What are some examples?
Web data, e-commerce
 Store purchases
 Bank/Credit Card transactions
 Cell phone GPS information

CISC 4631: Data Mining
7
Why Mine Data?

What technological changes have helped make
data mining so prevalent now?

Computers: cheaper and more powerful

Smaller mobile devices are exploding in popularity
Disk and other storage: greater capacity and cheaper
 RFID (radio frequency IDs), bar codes, etc
 Increased use of on-line resources and Internet

CISC 4631: Data Mining
8
Why Mine Data?

In business, competitive pressure is strong
Provide better, customized services for an edge (e.g.
in Customer Relationship Management)
 CRM is a relatively big deal now

How do we get the most out of the customer over the
long run
 Example: Customer Churn Analysis

CISC 4631: Data Mining
9
Scientific Viewpoint

Data collected at enormous speeds
remote sensors on satellite
 telescopes scanning the skies
 microarrays generating gene
expression data
 scientific simulations



Traditional techniques infeasible
Data mining may help scientists
in classifying and segmenting data
 in hypothesis formation

CISC 4631: Data Mining
10
Mining Large Data Sets - Motivation
Often information “hidden” in data that is not evident
 Human analysts may take weeks to discover useful info
 Much of the data is never analyzed at all

4,000,000
3,500,000
The Data Gap
3,000,000
2,500,000
2,000,000
1,500,000
Total new disk (TB) since 1995
1,000,000
500,000
# of analysts
0
1995
1996
1997
1998
1999
From: R. Grossman, C. Kamath, V. Kumar, “Data Mining for Scientific and Engineering Applications”
CISC 4631: Data Mining
11
Mining Large Data Sets - Motivation






AT&T’s 26TB call detail database (2003)
Ebay 6PB, IRS 150TB data warehouse
Yahoo has a 2PB DB to analyze behavior of ½
billion web visitors/month (24 billion events/day)
Wal-Mart has a 583 TB database (2006)
Indexed web contains about 20 Billion pages
Sites like Facebook, Flicker & Twitter contain lots
of data
CISC 4631: Data Mining
12
Data Deluge
CISC 4631: Data Mining
13
Amount of Data Created in One Year

Humans created/copied 161/281 Exabytes in 06/07 (IDC)





1 Exabyte = 1018
12 stacks of books stretching from Earth to Sun
3 million times the books ever written
In 2010 will be 988 Exabytes
Not all data stored at once


UC Berkeley 2003 estimate:


Much only temporarily
5 Exabytes of new data created in 2002
US produces ~40% of new stored data worldwide
CISC 4631: Data Mining
14
Data Growth Rate

Twice as much information was created in 2002 as in
1999 (~30% growth rate)

Other growth rate estimates even higher

Very little data will ever be looked at by a human

Knowledge Discovery is NEEDED to make sense and
use of data

Moore’s Law:


The information density on silicon-integrated circuits doubles
every 18 to 24 months.
Parkinson’s Law:

Work expands to fill the time available for its completion
CISC 4631: Data Mining
15
Origins of Data Mining


Draws ideas from machine learning/AI, pattern
recognition, statistics, and database systems
Traditional techniques
Artificial Intelligence
may be unsuitable due to
Statistics / Machine Learning/
Enormity of data
 High dimensionality
of data
 Heterogeneous,
distributed nature
of data
Pattern
Recognition

CISC 4631: Data Mining
Data Mining
Database
systems
16
Origins of Data Mining: My view

Biggest contributor is Machine Learning, which is a
subfield of Artificial Intelligence




Data Mining is a subset of machine learning and focuses on
practical problems of learning from data
Unlike machine learning, ultimate goal is not to build
something that can learn as flexibly as a human
Does include other data analysis methods, like statistics
Databases do not play a central role in data mining.


Most DM does not occur on data in a conventional database,
but rather extracts it to a flat file.
Data Mining methods do not work while data in a
conventional (relational) database.
CISC 4631: Data Mining
17
Statistics & Machine Learning vs. Data Mining

When compared to Data Mining:
 Statistics
is:
 more
theory-based/based on mathematics as
opposed to heuristic methods
 more focused on testing hypotheses
 makes
more assumptions about the data
 Machine
learning is:
 focused
on improving performance of a learning
agent in an environment
CISC 4631: Data Mining
18
The KDD (Data Mining) Process
Data Mining is a process, sometimes referred to as a knowledge
discovery process. In this process there is a data mining step that
applies data mining algorithms to extract knowledge. About 80% of
our class in on the data mining step.
CISC 4631: Data Mining
19
Back to “What is a Data Mining”?

My opinion:
 Before
determining whether something is
data mining need to consider:
 Is
it a DM task?
 Is it implemented using a DM method?
 Ideally, both parts will use data mining but may
be considered DM even if only is used for one.

We now will list the key DM tasks
 The
course is organized around these tasks
CISC 4631: Data Mining
20
Second Part of Introduction:
DATA MINING TASKS
CISC 4631: Data Mining
21
2 Top Level Data Mining Tasks

At highest level, data mining tasks can be
divided into:
 Prediction
Tasks
 Use
some variables to predict unknown or future
values of other variables
 Description
Tasks
 Find
human-interpretable patterns that describe
the data
CISC 4631: Data Mining
22
Key Data Mining Tasks

Overview of major data mining tasks:
 Predictive
 Classification
 Regression
 Deviation/Anomaly
Detection
 Descriptive
 Clustering
 Association
Rule Discovery
 Sequential Pattern Discovery
CISC 4631: Data Mining
23
Classification: Definition

Given a collection of records (training set )


Find a model for class attribute as a function
of the values of other attributes.


Model maps record to a class value
Goal: previously unseen records should be
assigned a class as accurately as possible.


Each record contains a set of attributes, one of the
attributes is the class, which is to be predicted.
A test set is used to determine the accuracy of the model.
Can you think of examples of classification
tasks? We will see several shortly.
CISC 4631: Data Mining
24
Classification Example
Task: Predict if someone cheats on their taxes
10
Tid Refund Marital
Status
Taxable
Income Cheat
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
60K
Refund Marital
Status
Taxable
Income Cheat
No
Single
75K
?
Yes
Married
50K
?
No
Married
150K
?
Yes
Divorced 90K
?
No
Single
40K
?
No
Married
80K
?
10
Training
Set
CISC 4631: Data Mining
Learn
Classifier
Test
Set
Model
25
Classification: Application 1

Direct Marketing


Goal: Reduce cost of mailing by targeting a set of consumers
likely to buy a new cell-phone product.
Approach:
Use the data for a similar product introduced before.
 We know which customers decided to buy and which decided
otherwise. This {buy, don’t buy} decision forms the class attribute
 Collect various demographic, lifestyle, and companyinteraction related information about all such customers.



Type of business, where they stay, how much they earn, etc.
Use this info as input attributes to learn a classifier model
CISC 4631: Data Mining
26
Classification: Application 2

Fraud Detection


Goal: Predict fraudulent cases in credit card transactions
Approach:
 Use credit card transactions and info on account-holders as
attributes

When and what does customer buy, how often pays on time, etc
Label past transactions as fraud or fair transactions. This
forms the class attribute.
 Learn a model for the class of the transactions.
 Use this model to detect fraud by observing credit card
transactions on an account.

CISC 4631: Data Mining
27
Classification: Application 3

Customer Attrition/Churn:
Goal: To predict whether a customer is likely to be
lost to a competitor.
 Approach:


Use detailed record of transactions with each of the past
and present customers, to find attributes.

How often the customer calls, where he calls, what time-of-the
day he calls most, his financial status, marital status, etc.
Label the customers as loyal or disloyal.
 Find a model for loyalty.

CISC 4631: Data Mining
28
Classification: Application 4

Sky Survey Cataloging

Goal: To predict class (star or galaxy) of sky objects,
especially visually faint ones, based on the telescopic
survey images (from Palomar Observatory).


3000 images with 23,040 x 23,040 pixels per image.
Approach:
Segment the image.
 Measure image attributes (features) - 40 of them per
object.
 Model the class based on these features.
 Success Story: Could find 16 new high red-shift quasars,
some of the farthest objects that are difficult to find!

From [Fayyad, et.al.] Advances in Knowledge Discovery and Data Mining, 1996
CISC 4631: Data Mining
29
Classifying Galaxies
Courtesy: http://aps.umn.edu
Early
Class:
• Stages of Formation
Attributes:
• Image features,
• Characteristics of light
waves received, etc.
Intermediate
Late
Data Size:
• 72 million stars, 20 million galaxies
• Object Catalog: 9 GB
• Image Database: 150 GB
CISC 4631: Data Mining
30
Regression



Predict a value of a given continuous valued variable
based on the values of other variables, assuming a
linear or nonlinear model of dependency.
Greatly studied in statistics, neural network fields.
Examples:



Predicting sales amounts of new product based on
advertising expenditure.
Predicting wind velocities as a function of temperature,
humidity, air pressure, etc.
Time series prediction of stock market indices.
CISC 4631: Data Mining
31
Clustering

Given a set of data points, each having a set of
attributes, and a similarity measure among them,
find clusters such that
Data points in one cluster are similar to one another
 Data points in different clusters are not (less) similar


Similarity Measures:
Euclidean distance if attributes are continuous
 Problem-specific measures


Can you think of any applications of clustering?
CISC 4631: Data Mining
32
Illustrating Clustering
Euclidean Distance Based Clustering in 3-D space.
Intracluster distances
are minimized
Intercluster distances
are maximized
CISC 4631: Data Mining
33
Clustering: Application 1

Market Segmentation:
Goal: subdivide a market into distinct subsets of
customers where any subset may conceivably be
selected as a market target to be reached with a
distinct marketing mix.
 Approach:

Collect different attributes of customers based on their
geographical and lifestyle related information.
 Find clusters of similar customers.
 Measure the clustering quality by observing buying
patterns of customers in same cluster vs. those from
different clusters.

CISC 4631: Data Mining
34
Clustering: Application 2

Document Clustering:
Goal: To find groups of documents that are similar
to each other based on the important terms
appearing in them.
 Approach: To identify frequently occurring terms in
each document. Form a similarity measure based on
the frequencies of different terms. Use it to cluster.
 Gain: Information Retrieval can utilize the clusters
to relate a new document or search term to clustered
documents.

CISC 4631: Data Mining
35
Association Rule Discovery

Given a set of records each of which contain
some number of items from a given collection

Produce dependency rules which will predict
occurrence of an item based on occurrences of other
items.
TID
Items
1
2
3
4
5
Bread, Coke, Milk
Beer, Bread
Beer, Coke, Diaper, Milk
Beer, Bread, Diaper, Milk
Coke, Diaper, Milk
Rules Discovered:
{Milk} --> {Coke}
{Diaper, Milk} --> {Beer}
CISC 4631: Data Mining
36
Association Rule Discovery: Application 1

Marketing and Sales Promotion:





Let the rule discovered be
{Bagels, … } --> {Potato Chips}
Potato Chips as consequent => Can be used to determine
what should be done to boost its sales.
Bagels in the antecedent => Can be used to see which
products would be affected if the store discontinues selling
bagels.
Bagels in antecedent and Potato chips in consequent => Can
be used to see what products should be sold with Bagels to
promote sale of Potato chips!
Can help determine where to position store items
CISC 4631: Data Mining
37
Association Rule Discovery: Application 2

Supermarket shelf management
Goal: Identify items that are bought together by
many customers
 Approach: Process the point-of-sale data collected
with barcode scanners to find item dependencies
 A “classic” rule -

If a customer buys diaper and milk, then he is very likely
to buy beer.
CISC 4631: Data Mining
38
Sequential Pattern Discovery: Definition

Given is a set of objects, with each object associated with
its own timeline of events, find rules that predict strong
sequential dependencies among different events.
(A B)

(C)
(D E)
Rules are formed by first disovering patterns. Event
occurrences in the patterns are governed by timing
constraints.
CISC 4631: Data Mining
39
Sequential Pattern Discovery: Examples

In telecommunications alarm logs,


(Inverter_Problem Excessive_Line_Current)
(Rectifier_Alarm) --> (Fire_Alarm)
In point-of-sale transaction sequences,


Computer Bookstore:
(Intro_To_Visual_C) (C++_Primer) -->
(Perl_for_dummies,Tcl_Tk)
Athletic Apparel Store:
(Shoes) (Racket, Racketball) --> (Sports_Jacket)
CISC 4631: Data Mining
40
Deviation/Anomaly Detection


Detect significant deviations from normal behavior
Applications:

Credit Card Fraud Detection

Network Intrusion
Detection
Typical network traffic at University level may reach over 100 million connections per day
CISC 4631: Data Mining
41
Challenges of Data Mining







Scalability
Dimensionality
Complex and Heterogeneous Data
Data Quality
Data Ownership and Distribution
Privacy Preservation
Streaming Data
CISC 4631: Data Mining
42
What is (and is not) Data Mining?

Based on the definitions of data mining,
are these DM or not?

Finding a phone number in a directory


Grouping related documents returned by search engine


Is data mining
Identifying who has a disease based on symptoms


Not data mining (trivial)
Is data mining (not trivial)
Web search on keyword using search engine

May be data mining**
** More of an information retrieval task than data mining task, but since a
search engine like Google does more than just keyword matching– it
decides which web pages are important or not (a classification task that is
part of DM) in order to get good results, the answer is not clear.
CISC 4631: Data Mining
43
If you are Interested in Data Mining

Visit kdnuggets, an online newsletter and more




ACM SIGKDD is the professional organization
associated with data mining



http://www.kdnuggets.com
You can arrange to have newsletter emailed to you
Also includes job openings
ACM Special Interest Group (SIG) on data mining
Can join SIGKDD for $22 or for $54 can also join ACM as
student member
Conferences

KDD, ICDM, DMIN, …
CISC 4631: Data Mining
44
Course Projects

Projects must involve data mining

May be research related


May be application oriented




Solve a realistic, complex, problem
May be a combination of both


Examine some aspect of data mining
Most problems involve some interesting aspect
In some cases can be a survey/analysis paper (i.e., just a
report), but this will be atypical
Can be done individually or in teams of 2
Ideally some projects can be published in a workshop
or conference
CISC 4631: Data Mining
45
Course Projects

Output

A written report, similar to a workshop or conference
paper

Two example workshop papers from last time course
offered :



For more examples:



http://storm.cis.fordham.edu/~gweiss/papers/ubdm05-mccarthy.pdf
http://storm.cis.fordham.edu/~gweiss/papers/ubdm05-ciraco.pdf
http://storm.cis.fordham.edu/~gweiss/publications.html and look
at the various workshop/conference papers
A presentation in class near end of semester
Stretch goal: submit paper to a workshop or conference

I can help you
CISC 4631: Data Mining
46
Course Projects


The sooner you start the better
Think about:






What you know about
What data you have access to
What type of problems you are interested in
Who you want to work with
I will provide some specific project ideas
Areas include:


Classification, clustering, association rules
Web and link mining, text mining, social network analysis
CISC 4631: Data Mining
47