Download Business Intelligence and Data Mining - Hui Xiong

Learning Objectives
Business Intelligence and Data Mining
• Understand the need for business intelligence systems.
• Know the characteristics of reporting systems.
Dr. Hui Xiong
g
Rutgers University
• Know the purpose and role of data warehouses and data marts.
• Understand fundamental data‐mining techniques.
U d
df d
ld
i i
h i
• Know the purpose, features, and functions of knowledge management systems. The Need for Business Intelligence Systems
• According to a study done at the University of California at Berkeley, a total of 403 petabytes of new data were created.
• 403 petabytes is roughly the amount of all printed material ever written.
– The printed collection of the Library of Congress is .01 petabytes.
– 400 petabytes equals 40,000 copies of the print collection of the Library of Congress.
Figure 9‐1 How big is an Exabyte?
Source: Used with permission of Peter Lyman and Hal R. Varian, University of California at Berkeley.
The Need for Business Intelligence Systems (Continued)
• The generation of all these data has much to do with Moore’s Law.
• The capacity of storage devices increases as thei o t de ea e
their costs decrease.
• Today, storage capacity is nearly unlimited.
• We are drowning in data and starving for information.
Figure 9‐2 Hard‐Disk Storage Capacity
Source: Used with permission of Peter Lyman and Hal R. Varian, University of California at Berkeley.
1
Business Intelligence Tools
Business Intelligence Tools
• Tools for searching business data in an attempt to find patterns is called business intelligence (BI) tools.
• The processing of data is simple:
– Data are sorted and grouped.
– Simple totals and averages are calculated.
• Reporting tools are programs that read data f
from a variety of sources, process that data, i t f
th t d t
produce formatted reports, and deliver those reports to the users who need them.
• Reporting tools are used primarily for assessment
– They are used to address questions like:
•What has happened in the past?
•What is the current situation?
•How does the current situation compare to the past?
Business Intelligence Tools (Continued)
Business Intelligence Systems
• Data‐mining tools process data using statistical techniques, many of which are sophisticated and mathematically complex.
• Data mining involves searching for patterns and relationships among data.
• In
In most cases, data‐mining tools are used to make most cases data mining tools are used to make
predictions.
• For example, we can use one form of analysis to compute the probability that a customer will default on a loan. • Another way to distinguish the differences of reporting tools and data‐mining tools is :
– Reporting tools use simple operations like sorting, grouping, and summing.
– Data‐mining tools use sophisticated techniques.
• An information system is a collection of hardware, software, data, procedures, and people.
• The purpose of a business intelligence (BI) system is to provide the right information, to is to provide the right information to
the right user, at the right time.
• BI systems help users accomplish their goals and objectives by producing insights that lead to actions.
Business Intelligence Systems (Continued)
Reporting Systems
• A reporting tool can generate a report that shows a customer has canceled an important order.
• The purpose of a reporting system is to create meaningful information from disparate data sources and to deliver that information to the proper user on a timely basis.
• A reporting system, however, alerts that customer’s salesperson with this unwanted news, and does so in time for the salesperson to try to alter the customer’s decision.
decision
• A data‐mining tool can create an equation that computes the probability that a customer will default on a loan.
• A data‐mining system uses that equation to enable banking personnel to assess new loan applications.
• Reporting
Reporting systems generate information from systems generate information from
data as a result of four operations:
– Filtering data
– Sorting data
– Grouping data
– Making simple calculations on the data
2
Figure 9‐3 Trade Data for NDX.X (NASDAQ 100)
Components of Reporting Systems
Figure 9‐4 Report Based on Trade Data in Figure 9‐3
Figure 9‐5 Components of a Reporting System
• A reporting system maintains a database of reporting metadata.
• The metadata describes the reports, users, groups, roles, events, and other entities involved in the reporting activity.
• The reporting system uses the metadata to prepare and deliver reports to the proper users on a timely basis.
Figure 9‐6 Summary of Report Characteristics
Report Type
• In terms of a report type, reports can be static or dynamic.
• Static reports are prepared once from the underlying data, and they do not change.
– Example, a report of past year’s sales
p ,
p
p
y
• Dynamic reports: the reporting system reads the most current data and generates the report using that fresh data.
– Examples are: a report on sales today and a report on current stock prices 3
Report Type (Continued)
Report Media
• Query reports are prepared in response to data entered by users.
• Reports are delivered via many different report media or channels.
• Online analytical processing (OLAP) reports allow the user to dynamically change the report grouping structures.
i
• Some reports are printed on paper, and others are created in a format like PDF whereby they a e p i e o ie e e e o i a y
can be printed or viewed electronically.
• Other reports are delivered to computer screens.
• Companies sometimes place reports on internal corporate Web sites for employees to access.
Report Media (Continued)
Report Media (Continued)
• Another report medium is a digital dashboard, which is an electronic display customized for a particular user.
• Other dashboards are particular to an organization.
– Vendors like Yahoo! and MSN provide common p
examples.
– Users of these services can define content they want‐
say, a local weather forecast, a list of stock prices, or a list of news sources.
– The vendor constructs the display customized for each user.
Figure 9‐7 Digital Dashboard Example
– The organization might have a dashboard that shows up‐to‐the‐
minute production and sales activities.
• Alerts are another form of report.
– Users can declare that they wish to receive notifications of events say via email or on their cell phones
events, say, via email or on their cell phones.
• Reports can be published via a Web service.
– The Web service produces the report in response to requests from the service‐consuming application.
Report Mode
• The report mode can be either push report or pull report.
• Organizations send a push report to users according to a preset schedule.
– Users receive the report without any activity p
y
y
on their part.
• Users must request a pull report.
– To obtain a pull report, a user goes to a Web portal or digital dashboard and clicks a link or button to cause the reporting system to produce and deliver the report.
4
Functions of Reporting Systems
Report Management
• Three functions of reporting systems are:
– Authoring
– Management
– Delivery
• The purpose of report management is to define who receives what reports, when, and by what means.
• Report authoring involves connecting to data sources, creating the reporting structure, and formatting the report.
• Reports that have been created using the report‐
authoring system are assigned groups and users.
Report Management (Continued)
• Assigning reports to groups saves the administrator work. – When a report is created, changed, or removed, the administrator need only change the report assignments to the group.
– All of the users in the group will inherit the changes.
g
p
g
• Metadata also indicates what channel is to be used and whether the report is to be pushed or pulled.
– If the report is to be pushed, the administrator declares whether the report is to be generated on a regular schedule or as an alert.
Report Delivery (Continued)
• Most report‐management systems allow the report administrator to define user accounts and user groups and to assign particular users to particular groups.
Report Delivery
• The report‐delivery function of a reporting system pushes reports or allows them to be pulled according to report‐management metadata.
• Reports can be delivered via an email server, Web site, XML Web services, or by other program‐specific means. means
• The report‐delivery system uses the operating system and other program security components to ensure that only authorized users receive authorized reports.
Online Analytical Processing
• The report‐delivery system also ensures that push reports are produced at appropriate times.
• Online analytical processing (OLAP) provides the ability to sum, count, average, and perform other simple arithmetic operations on groups of data.
• For query reports, the report‐delivery system serves as an intermediary between the user and
serves as an intermediary between the user and the report generator.
– It receives user query data, such as item numbers in an inventory query, passes the query data to the report generator, receives the resulting report, and delivers the report to the user.
• The remarkable characteristics of OLAP reports is that ey a e y a i
they are dynamic.
• The viewer of the report can change the report’s format, hence, the term online.
5
Online Analytical Processing
• An OLAP report has measures and dimensions.
• A measure is the data item of interest.
– It is the item that is to be summed or averaged or otherwise processed in the OLAP report.
• A dimension
A dimension is a characteristic of a measure.
is a characteristic of a measure
– Purchase data, customer type, customer location, and sales region are all examples of dimension.
Online Analytical Processing (Continued)
• With an OLAP report, it is possible to drill down into the data.
– This term means to further divide the data into more detail.
• Special‐purpose products called OLAP servers have been developed to perform OLAP analysis.
• A
An OLAP server reads data from an operational O A
f
database, performs preliminary calculations, and stores the results of those operations in an OLAP database.
Figure 9‐13 OLAP Family and Store Location by Store Type
Figure 9‐14 Role of OLAP Server and OLAP Database
Data Warehouses and Data Marts
Data Warehouses and Data Marts (Continued)
• Basic reports and simple OLAP analyses can be made directly from operational data.
• For the most part, such reports display the current state of the business; and if there are a few missing values or small inconsistencies with the data, no one is too concerned
too concerned.
• Operational data are unsuited to more sophisticated analyses, particularly, data‐mining analyses that require high‐quality input for accurate and useful results.
• Many organizations choose to extract operational data into facilities called data warehouses and data marts, both of which are facilities that prepare, store, and manage data specifically for data mining and other analyses.
• Programs read operational data and extract, clean, and g
p
,
,
prepare that data for BI processing.
• The prepared data are stored in a data‐warehouse database using data‐warehouse DBMS, which can be different from the organization’s operational DBMS.
6
Data Warehouses and Data Marts
Figure 9‐15 Components of a Data Warehouse
• Data warehouses include data that are purchased from outside sources.
• Metadata concerning the data, its source, its format, its assumptions and constraints, and other facts about the data is kept in a data‐warehouse metadata database.
p
• The data‐warehouse DBMS extracts and provides data to business intelligence tools such as data‐mining programs.
Figure 9‐16 Consumer Data Available for Purchase from Data Vendors
Problems with Operational Data (Continued)
• Inconsistent data are particularly common for data that have been gathered over time.
– When an area code changes, for example, the phone number for a given customer before the change will not match the customer’s number after the change.
• Some data inconsistencies occur from the nature of the business activity
business activity.
• Nonintegrated data can cause problems when data comes from different management information systems.
Figure 9‐17 Problems of Using Transaction Data for Analysis and Data Mining
Data Warehouses Versus Data Marts
• The data warehouse takes data from the data manufacturers (operational systems and purchased data), cleans and processes the data, and locates the data on the shelves, so to speak, of the data warehouse.
• A data mart is a data collection, smaller than the data warehouse, that addresses a particular component or functional area of the business.
7
Data Warehouse Versus Data Marts (Continued)
Figure 9‐18 Data Mart Examples
• The data warehouse is like the distributor in the supply chain and the data mart is like the retail store in the supply chain.
• Users in the data mart obtain data that pertain to a p
particular business function from the data warehouse.
• It is expensive to create, staff, and operate data warehouses and data marts.
Data Mining and Business Intelligence
Knowledge Discovery in Data
Dr Hui Xiong
Dr.
Rutgers University
Why Mine Data? Commercial Viewpoint
• Lots of data is being collected and warehoused – Web data, e‐commerce
– purchases at department/
grocery stores
– Bank/Credit Card transactions
• Computers have become cheaper and more powerful
• Competitive Pressure is Strong – Provide better, customized services for an edge (e.g. in Customer Relationship Management)
Why Mine Data? Scientific Viewpoint
• Data collected and stored at enormous speeds (GB/hour)
– remote sensors on a satellite
– telescopes scanning the skies
– microarrays generating gene p
expression data
– scientific simulations generating terabytes of data
• Traditional techniques infeasible for raw data
• Data mining may help scientists – in classifying and segmenting data
– in Hypothesis Formation
8
Mining Large Data Sets ‐ Motivation
• There is often information “hidden” in the data that is not readily evident
• Human analysts may take weeks to discover useful information
• Much of the data is never analyzed at all
4,000,000
3 500 000
3,500,000
The Data Gap
3,000,000
2,500,000
Scale of Data
Organization
Walmart
Google
Yahoo
NASA satellites
NCBI GenBank
France Telecom
UK Land Registry
AT&T Corp
Scale of Data
~ 20 million transactions/day
~ 8.2 billion Web pages
~10 GB Web data/hr
~ 1.2 TB/day
~ 22 million genetic sequences
29 2 TB
29.2
18.3 TB
26.2 TB
2,000,000
1,500,000
1,000,000
Total new disk (TB) since
1995
500,000
Number of
analysts
0
1995
1996
1997
1998
1999
From: R. Grossman, C. Kamath, V. Kumar, “Data Mining for Scientific and Engineering Applications”
Why Do We Need Data Mining ?
• Leverage organization’s data assets
– Only a small portion (typically ‐ 5%‐10%) of the collected data is ever analyzed
“The great strength of computers is that
they can reliably manipulate vast amounts
of data very quickly. Their great weakness is
that they don’t have a clue as to what any
Why Do We Need Data Mining?
• As databases grow, the ability to support the decision support process using traditional query languages becomes infeasible
– Data that may never be analyzed continues to be g
p
collected, at a great expense, out of fear that something which may prove important in the future is missing.
– Many queries of interest are difficult to state in a query language (Query formulation problem)
query language (Query formulation problem)
– “find all cases of fraud”
– “find all individuals likely to buy a FORD expedition”
– Growth rates of data precludes traditional “manually intensive” approach
– “find all documents that are similar to this customers problem”
(Latitude, Longitude)1
What is Data Mining?
• Many Definitions
– Non‐trivial extraction of implicit, previously unknown and potentially useful information from data
– Exploration & analysis, by automatic or semi‐automatic means, of large quantities of data in order to discover meaningful patterns
What is (not) Data Mining?
z
What is not Data Mining?
– Look up phone number in phone directory – Check the dictionary for the meaning of a word
z
What is Data Mining?
– Certain names are more prevalent in certain US locations (O’Brien, O’Rurke, O’Reilly… in Boston area)
– Group together similar documents returned by search engine according to their context (e.g. Amazon rainforest, Amazon.com,)
9
Data Mining: Confluence of Multiple Disciplines
?
20x20 ~ 2^400 ≈ 10^120 patterns
Data Mining Applications
• Market analysis
• Risk analysis and management
• Fraud detection and detection of unusual p
patterns (outliers)
(
)
• Text mining (news group, email, documents) and Web mining
• Stream data mining
• DNA and bio‐data analysis
Fraud Detection & Mining Unusual Patterns
• Approaches: Clustering & model construction for frauds, outlier analysis
• Applications: Health care, retail, credit card service, …
– Auto insurance: ring of collisions – Money laundering: suspicious monetary transactions – Medical insurance
Medical insurance
• Professional patients, ring of doctors, and ring of references
• Unnecessary or correlated screening tests
– Telecommunications: phone‐call fraud
• Phone call model: destination of the call, duration, time of day or week. Analyze patterns that deviate from an expected norm
– Retail industry
• Analysts estimate that 38% of retail shrink is due to dishonest employees
– Anti‐terrorism
Data Mining and Business Intelligence
Data Mining Tasks …
Data
Tid Refund Marital
Status
Taxable
Income Cheat
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
60K
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
11
No
Married
60K
No
12
Yes
Divorced 220K
No
13
No
Single
85K
Yes
14
No
Married
75K
No
15
No
Single
90K
Yes
10
Milk
10
Clustering
Applications of Cluster Analysis
• Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups
• Understanding
Intraa
cluster
distances
are
minimize
d
Intercluster
distances
are
maximize
d
Clustering: Application 1
• Market Segmentation:
– Goal: subdivide a market into distinct subsets of customers where any subset may conceivably be selected as a market target to be reached with a distinct marketing mix.
– Approach:
• Collect different attributes of customers based on their Collect different attributes of customers based on their
geographical and lifestyle related information.
• Find clusters of similar customers.
• Measure the clustering quality by observing buying patterns of customers in same cluster vs. those from different clusters. What is not Cluster Analysis?
Discovered Clusters
– Group related documents for browsing – Group genes and proteins that have similar functionality
– Group stocks with similar p
price fluctuations
1
2
3
4
Applied-Matl-DOWN,Bay-Network-Down,3-COM-DOWN,
Cabletron-Sys-DOWN,CISCO-DOWN,HP-DOWN,
DSC-Comm-DOWN,INTEL-DOWN,LSI-Logic-DOWN,
Micron-Tech-DOWN,Texas-Inst-Down,Tellabs-Inc-Down,
Natl-Semiconduct-DOWN,Oracl-DOWN,SGI-DOWN,
Sun-DOWN
Apple-Comp-DOWN,Autodesk-DOWN,DEC-DOWN,
ADV-Micro-Device-DOWN,Andrew-Corp-DOWN,
Computer-Assoc-DOWN,Circuit-City-DOWN,
Compaq-DOWN, EMC-Corp-DOWN, Gen-Inst-DOWN,
Motorola-DOWN,Microsoft-DOWN,Scientific-Atl-DOWN
Industry Group
Technology1-DOWN
Technology2-DOWN
Fannie-Mae-DOWN,Fed-Home-Loan-DOWN,
MBNA-Corp-DOWN,Morgan-Stanley-DOWN
Baker-Hughes-UP,Dresser-Inds-UP,Halliburton-HLD-UP,
Louisiana-Land-UP,Phillips-Petro-UP,Unocal-UP,
Schlumberger-UP
Financial-DOWN
Oil-UP
Oil
UP
• Summarization
– Reduce the size of large data sets
Use of K‐means to partition Sea Surface Temperature (SST) and Net Primary Production (NPP) into clusters that reflect the Northern and Southern Hemispheres. Clustering: Application 2
• Document Clustering:
– Goal: To find groups of documents that are similar to each other based on the important terms appearing in them.
– Approach: To identify frequently occurring terms in each document. Form a similarity measure based on the frequencies of different terms. Use it to cluster.
Notion of a Cluster can be Ambiguous
• Simple segmentation
– Dividing students into different registration groups alphabetically, by last name
• Results of a query
– Groupings are a result of an external specification
Groupings are a result of an external specification
– Clustering is a grouping of objects based on the data
How many clusters?
Six Clusters
Two Clusters
Four Clusters
• Supervised classification
– Have class label information
• Association Analysis
– Local vs. global connections
11
Types of Clusterings
Partitional Clustering
• A clustering is a set of clusters
• Important distinction between hierarchical
and partitional sets of clusters • Partitional Clustering
– A division data objects into non‐overlapping subsets (clusters) such that each data object is in exactly one subset
• Hierarchical clustering
– A set of nested clusters organized as a hierarchical tree Hierarchical Clustering
Original Points
A Partitional Clustering
Other Distinctions Between Sets of Clusters
• Exclusive versus non‐exclusive
– In non‐exclusive clusterings, points may belong to multiple clusters.
– Can represent multiple classes or ‘border’ points
p1
p3
p4
p2
p1 p2
Traditional Hierarchical Clustering
p3 p4
Traditional Dendrogram
• Fuzzy versus non‐fuzzy
– In fuzzy clustering, a point belongs to every cluster with some weight between 0 and 1
weight between 0 and 1
– Weights must sum to 1
– Probabilistic clustering has similar characteristics
• Partial versus complete
p1
p3
p4
– In some cases, we only want to cluster some of the data
p2
p1 p2
Non-traditional Hierarchical Clustering
p3 p4
Non-traditional Dendrogram
Types of Clusters
• Well‐separated clusters
• Center‐based clusters
• Heterogeneous versus homogeneous
– Clusters of widely different sizes, shapes, and densities
Types of Clusters: Well‐Separated
• Well‐Separated Clusters: – A cluster is a set of points such that any point in a cluster is closer (or more similar) to every other point in the cluster than to any point not in the cluster. • Contiguous clusters
• Density‐based clusters
b d l
• Property or Conceptual
• Described by an Objective Function
3 well-separated clusters
12
Types of Clusters: Center‐Based
Types of Clusters: Contiguity‐Based
• Center‐based
• Contiguous Cluster (Nearest neighbor or Transitive)
– A cluster is a set of objects such that an object in a cluster is closer (more similar) to the “center” of a cluster, than to the center of any other cluster – The center of a cluster is often a centroid, the average of all the points in the cluster, or a medoid,
average of all the points in the cluster, or a medoid, the most “representative” point of a cluster – A cluster is a set of points such that a point in a cluster is closer (or more similar) to one or more other points in the cluster than to any point not in the cluster.
4 center-based clusters
Types of Clusters: Density‐Based
• Density‐based
– A cluster is a dense region of points, which is separated by low‐density regions, from other regions of high density. – Used when the clusters are irregular or intertwined, and when noise and outliers are present
and when noise and outliers are present. 8 contiguous clusters
Types of Clusters: Conceptual Clusters
• Shared Property or Conceptual Clusters
– Finds clusters that share some common property or represent a particular concept.
6 density-based clusters
Characteristics of the Input Data Are Important
2 Overlapping Circles
Data Mining Tasks …
• Type of proximity or density measure
– This is a derived measure, but central to clustering • Sparseness
Data
– Dictates type of similarity
– Adds to efficiency
Tid Refund Marital
Status
• Attribute type
– Dictates type of similarity
• Type of Data
– Dictates type of similarity
– Other characteristics, e.g., autocorrelation
• Dimensionality
• Noise and Outliers
• Type of Distribution
Taxable
Income Cheat
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
60K
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
11
No
Married
60K
No
12
Yes
Divorced 220K
No
13
No
Single
85K
Yes
14
No
Married
75K
No
15
No
Single
90K
Yes
10
Milk
13
Association Rule Discovery: Definition
Association Analysis: Applications
• Given a set of records each of which contain some number of items from a given collection
• Market‐basket analysis
– Rules are used for sales promotion, shelf management, and inventory management
– Produce dependency rules which will predict occurrence of an item based on occurrences of other items.
TID
Items
1
2
3
4
5
Bread, Coke, Milk
Beer, Bread
Beer, Coke, Diaper, Milk
Beer, Bread, Diaper, Milk
Coke, Diaper, Milk
• Telecommunication alarm diagnosis
– Rules are used to find combination of alarms that occur together frequently in the same time period
Rules Discovered:
• Medical Informatics
{Milk} --> {Coke}
{Diaper, Milk} --> {Beer}
– Rules are used to find combination of patient symptoms and complaints associated with certain diseases
Data Mining Tasks …
Application Deployment Challenge
Data
Tid Refund Marital
Status
Taxable
Income Cheat
1
125K
Yes
Single
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
7
Yes
Divorced 220K
60K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
11
No
Married
60K
No
12
Yes
Divorced 220K
No
13
No
Single
85K
Yes
14
No
Married
75K
No
15
No
Single
90K
Yes
10
Milk
Predictive Modeling: Classification
Classification Example
• Find a model for class attribute as a function of the values of other attributes Model for predicting credit
Tid Employed
worthiness
1
Employed
Tid Employed
1
Yes
Graduate
# years at
present
address
5
2
Yes
High School
2
No
3
No
Undergrad
1
No
Tid Employed
Level of
Education
Credit
Worthy
Yes
No
Yes
4
Yes
High School
10
Yes
…
…
…
…
…
No
Education
Graduate
{ High school,
Undergrad }
Level of
Education
1
Yes
Graduate
2
Yes
High School
# years at
present
address
5
2
Credit
Worthy
Yes
Yes
Undergrad
# years at
present
address
7
Level of
Education
No
Graduate
3
3
Yes
High School
2
?
…
…
…
…
…
< 3 yr
> 7 yrs
< 7 yrs
Yes
No
Yes
No
?
10
No
3
No
Undergrad
1
No
4
Yes
High School
10
Yes
…
…
…
…
…
10
Test
Set
Number of
years
> 3 yr
?
2
10
Number of
years
Credit
Worthy
Training
Set
Learn
Classifier
Model
14
Examples of Classification Task
Classification: Application 1
• Predicting tumor cells as benign or malignant
• Fraud Detection
– Goal: Predict fraudulent cases in credit card transactions.
– Approach:
• Use credit card transactions and the information on its account‐holder as attributes.
– When does a customer buy, what does he buy, how often he pays on time, etc
• Label past transactions as fraud or fair transactions. This forms the class attribute.
• Learn a model for the class of the transactions.
• Use this model to detect fraud by observing credit card transactions on an account.
• Classifying credit card transactions as legitimate or fraudulent
• Classifying secondary structures of protein as alpha‐helix, beta‐sheet, or random coil
• Categorizing news stories as finance, weather, entertainment, sports, etc
• Identifying intruders in the cyberspace
Classification: Application 2
Classification: Application 3
• Churn prediction for telephone customers
• Sky Survey Cataloging
– Goal: To predict whether a customer is likely to be lost to a competitor.
– Approach:
• Use detailed record of transactions with each of the past and present customers, to find attributes.
– How often the customer calls, where he calls, what time‐of‐the day he calls most, his financial status, marital status, etc.
• Label the customers as loyal or disloyal.
• Find a model for loyalty.
From [Berry & Linoff] Data Mining Techniques, 1997
Classifying Galaxies
• Stages of Formation
Late
Data Size:
• 72 million stars, 20 million galaxies
• Object Catalog: 9 GB
• Image Database: 150 GB
From [Fayyad, et.al.] Advances in Knowledge Discovery and Data Mining, 1996
Classification Techniques
Class:
Intermediate
• Success Story: Could find 16 new high red‐shift quasars, some of the farthest objects that are difficult to find!
Attributes:
• Image features,
• Characteristics of light
waves received, etc.
Early
– Goal: To predict class (star or galaxy) of sky objects, especially visually faint ones, based on the telescopic survey images (from Palomar Observatory).
– 3000 images with 23,040 x 23,040 pixels per image.
– Approach:
• Segment the image. • Measure image attributes (features) ‐ 40 of them per object.
• Model the class based on these features.
• Base Classifiers
– Decision Tree based Methods
– Rule‐based Methods
– Nearest‐neighbor
– Neural Networks
N
lN
k
– Naïve Bayes and Bayesian Belief Networks
– Support Vector Machines
• Ensemble Classifiers
– Boosting, Bagging, Random Forests
15
Example of a Decision Tree
Another Example of Decision Tree
MarSt
Splitting Attributes
ID
Home
Owner
Marital
Status
Annual Defaulted
Income Borrower
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
60K
ID
Home
Owner
Yes
No
NO
MarSt
Married
Single, Divorced
Income
< 80K
NO
> 80K
YES
NO
Home
Owner
Marital
Status
Annual Defaulted
Income Borrower
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
60K
Married
Single,
Divorced
NO
Yes
Home
Owner
NO
No
Income
< 80K
> 80K
Yes
6
No
Married
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
YES
NO
No
There could be more than one tree that
fits the same data!
10
10
Model:
Decision
Tree
Training
Data
Decision Tree Classification Task
Tid
Attrib1
Attrib3
Class
1
Yes
Large
Attrib2
125K
No
2
No
Medium
100K
No
3
No
Small
70K
No
4
Yes
Medium
120K
No
5
No
Large
95K
Yes
6
No
Medium
60K
No
7
Yes
Large
220K
No
8
No
Small
85K
Yes
9
N
No
M di
Medium
75K
N
No
10
No
Small
90K
Yes
Apply Model to Test Data
Start from the
root of tree.
Home
Owner
Yes
Learn
Model
NO
Tid
Attrib1
Attrib3
Class
11
No
Small
Attrib2
55K
?
12
Yes
Medium
80K
?
13
Yes
Large
110K
?
14
No
Small
95K
?
15
No
Large
67K
?
No
Marital
Status
Married
Annual Defaulted
Income Borrower
80K
?
10
No
MarSt
Single, Divorced
10
Decision
Tree
Apply
Model
Test Data
Home
Owner
Income
< 80K
Married
NO
> 80K
YES
NO
10
Apply Model to Test Data
Test
Home
Marital
Data
Owner Status
No
Home
Owner
Yes
NO
Defaulted
Borrower
Home
Owner
?
No
Home
Owner
Yes
MarSt
Income
NO
Annual
Income
80K
10
No
Single, Divorced
< 80K
Married
Apply Model to Test Data
NO
Married
YES
Annual Defaulted
Income Borrower
80K
?
10
No
MarSt
Single, Divorced
Income
NO
> 80K
Marital
Status
Married
< 80K
NO
Married
NO
> 80K
YES
16
Apply Model to Test Data
Home
Owner
No
Home
Owner
Yes
NO
Apply Model to Test Data
Marital
Status
Married
Annual Defaulted
Income Borrower
Home
Owner
80K
No
?
Yes
MarSt
Single, Divorced
Income
< 80K
NO
Marital
Status
Married
Annual Defaulted
Income Borrower
80K
?
10
MarSt
Income
< 80K
NO
YES
Decision Tree Classification Task
No
Single, Divorced
NO
> 80K
NO
No
Home
Owner
Married
NO
> 80K
Married
Income
< 80K
Apply Model to Test Data
Yes
MarSt
NO
Home
Owner
Defaulted
Borrower
?
10
Single, Divorced
YES
Annual
Income
80K
No
NO
Married
> 80K
NO
Home
Owner
10
No
Marital
Status
Married
Assign
Defaulted
to “No”
Tid
Attrib1
1
Yes
Large
125K
No
2
No
Medium
Attrib2
100K
Attrib3
No
Class
3
No
Small
70K
No
4
Yes
Medium
120K
No
5
No
Large
95K
Yes
6
No
Medium
60K
No
7
Yes
Large
220K
No
8
No
Small
85K
Yes
9
No
Medium
75K
No
10
No
Small
90K
Yes
Learn
Model
10
YES
Tid
Attrib1
11
No
Small
Attrib2
55K
Attrib3
?
12
Yes
Medium
80K
?
13
Yes
Large
110K
?
14
No
Small
95K
?
15
No
Large
67K
?
Apply
Model
Class
Decision
Tree
10
Decision Tree Induction
Data Mining Tasks …
• Many Algorithms:
– Hunt’s Algorithm (one of the earliest)
– CART
– ID3, C4.5
ID3 C4 5
– SLIQ,SPRINT
Data
Tid Refund Marital
Status
Taxable
Income Cheat
1
Yes
Single
125K
No
2
No
Married
100K
No
3
No
Single
70K
No
4
Yes
Married
120K
No
5
No
Divorced 95K
Yes
6
No
Married
No
60K
7
Yes
Divorced 220K
No
8
No
Single
85K
Yes
9
No
Married
75K
No
10
No
Single
90K
Yes
11
No
Married
60K
No
12
Yes
Divorced 220K
No
13
No
Single
85K
Yes
14
No
Married
75K
No
15
No
Single
90K
Yes
10
Milk
17
Deviation/Anomaly Detection
• Detect significant deviations from normal behavior
• Applications:
– Credit
Credit Card Fraud Card Fraud
Detection
– Network Intrusion Detection
Anomaly Detection
• Challenges
– How many outliers are there in the data?
– Method is unsupervised
• Validation can be quite challenging (just like for clustering)
– Finding needle in a haystack
Finding needle in a haystack
• Working assumption
– There are considerably more “normal” observations than “abnormal” observations (outliers/anomalies) in the data
Anomaly Detection Schemes Graphical Approaches
• General Steps
• Boxplot (1‐D), Scatter plot (2‐D), Spin plot (3‐D)
• Limitations
– Build a profile of the “normal” behavior
• Profile can be patterns or summary statistics for the overall population
– Use the “normal” profile to detect anomalies
• Anomalies are observations whose characteristics
A
li
b
ti
h
h
t i ti
differ significantly from the normal profile
– Time consuming
– Subjective
• Types of anomaly detection schemes
– Graphical & Statistical‐based
– Distance‐based
– Model‐based
Statistical Approaches
• Assume a parametric model describing the distribution of the data (e.g., normal distribution) • Apply a statistical test that depends on – Data distribution
– Parameter of distribution (e.g., mean, variance)
Parameter of distribution (e g mean variance)
– Number of expected outliers (confidence limit)
Limitations of Statistical Approaches
• Most of the tests are for a single attribute
• In many cases, data distribution may not be known
• For high dimensional data, it may be For high dimensional data it may be
difficult to estimate the true distribution
18
Distance‐based Approaches
• Data is represented as a vector of features
• Three major approaches
– Nearest‐neighbor based
– Density based
– Clustering based
Density‐based: LOF approach
• For each point, compute the density of its local neighborhood
• Compute local outlier factor (LOF) of a sample p as the average of the ratios of the density of sample p and the density of its nearest neighbors
p
g
• Outliers are points with largest LOF value
×
p2
×
p1
In the NN
approach, p2 is not
considered as
outlier, while LOF
approach find both
p1 and p2 as outliers
KDD Process
Nearest‐Neighbor Based Approach
• Approach:
– Compute the distance between every pair of data points
– There are various ways to define outliers:
• Data points for which there are fewer than p
Data oi t fo hi h the e a e fe e tha
neighboring points within a distance D
• The top n data points whose distance to the kth nearest neighbor is greatest
• The top n data points whose average distance to the k nearest neighbors is greatest Clustering‐Based
• Basic idea:
¾ Cluster the data into groups of different density
¾ Choose points in small cluster as candidate l t
did t
outliers
¾ Compute the distance between candidate points and non‐candidate clusters. ‐ If candidate points are far from all other non‐candidate points, they are outliers
KDD Process
• Develop an understanding of the application domain – Relevant prior knowledge, problem objectives, success criteria, current solution, inventory resources, constraints, terminology, cost and benefits
• Selection of data mining task
• Create target data set
Collect initial data, describe, focus on a subset of variables,
– Collect initial data, describe, focus on a subset of variables, verify data quality
• Select data mining approach • Data mining to extract patterns or models
D
i i
d l
• Interpretation and evaluation of patterns/models
• Consolidating discovered knowledge
• Data cleaning and preprocessing
– Remove noise, outliers, missing fields, time sequence information, known trends, integrate data
• Data Reduction and projection
– Feature subset selection, feature construction, discretizations, aggregations
– Classification, segmentation, deviation detection, link analysis
19
Challenges of Data Mining
Knowledge Discovery
•
•
•
•
•
•
•
•
Similarities Between Data Miners and Doctors Scalability
Dimensionality
Complex and Heterogeneous Data
Data Quality
Data Ownership and Distribution
Privacy Preservation
Streaming Data
Data from Multi‐Sources Commercial and Research Tools
WEKA: http://www.cs.waikato.ac.nz/ml/weka/
SAS: http://www.sas.com/
Clementine:
Clementine: http://www.spss.com/spssbi/clementine/
Data Characteristics
Intelligent Miner http://www‐3.ibm.com/software/data/iminer/
Insightful Miner http://www.insightful.com/products/product.asp?PID=26
Data Mining Techniques
Textbooks
Medical Devices
Knowledge Management
• Knowledge management systems concern the sharing of knowledge that is already known to exist, either in libraries of documents, in the heads of employees, or in other known sources.
• Knowledge management (KM) is the process of creating value from intellectual capital and sharing i
l f
i ll
l
i l d h i
that knowledge with employees, managers, suppliers, customers, and others who need that capital.
20
Knowledge Management (Continued)
• Knowledge management is a process that is supported by the five components of an information system.
– Its emphasis is on people, their knowledge, and effective means for sharing that knowledge with others.
• The benefits of KM concern the application of knowledge to enable employees and others to leverage organizational knowledge to work smarter.
Content Management Systems
• Content management systems are information systems that track organizational documents, Web pages, graphics, and related materials.
• Such systems differ from operational document systems in that they do not directly support business operations.
i
• KM content management systems are concerned with the creation, management, and delivery of documents that exist for the purpose of imparting knowledge.
• KM preserves organizational memory by capturing and storing the lessons learned and best practices of key employees.
Content Management Systems (Continued)
• Typical users of content management systems are companies that sell complicated products and want to share their knowledge of those products with employees and customers.
• The basic functions of content management systems are the same as for report management systems: author, h
f
h
manage, and deliver.
• The only requirement that content managers place on document authoring is that the document has been created in a standardized format.
Content Management Problems
• Documents may refer to one another or multiple documents may refer to the same product or procedure.
– When one of them changes, others must change as well.
– Some content management systems keep semantic g
g
linkages among documents so that content dependencies can be known and used to maintain document consistency.
• Document contents are perishable.
– Documents become obsolete and need to be altered, removed, or replaced.
• Multinational companies have to ensure document language translations.
Figure 9‐23 Document Management at Microsoft.com (as of December 2003)
Source: microsoft.com/backstage/inside.htm (accessed February 2004). © 2003 Microsoft Corporation. All rights reserved.
Figure 9‐24 Reporting Services: United States
Source: Used with permission of Tom Rizzo of Microsoft Corporation.
21
Figure 9‐25 Reporting Services: China
Content Delivery
• Almost all users of content management systems pull the contents.
• Users cannot pull content if they do not know it exists.
– The content must be arranged and indexed, and a facility for searching the content devised.
searching the content devised.
• Documents that reside behind a corporate firewall, however, are not publicly accessible and will not be reachable by Google or other search engines.
– Organizations must index their own proprietary documents and provide their own search capability for them.
Source: Used with permission of Tom Rizzo of Microsoft Corporation.
KM Systems to Facilitate the Sharing of Human Knowledge
• Nothing is more frustrating for a manager to contemplate than the situation in which one employee struggles with a problem that another employee knows how to solve easily.
• KM systems are concerned with the sharing not only of content, but also with the sharing of knowledge among humans.
– How can one person share her knowledge with another?
– How can one person learn of another person’s great idea?
Figure 9‐26 Technology Support of Sharing Human Knowledge
KM Systems to Facilitate the Sharing of Human Knowledge (Continued)
• Three forms of technology are used for knowledge‐ sharing among humans:
– Portals, discussion groups, and email
– Collaborations systems
Collaborations systems
– Expert systems
Portals
– Employees can share ideas by posting knowledge on a Web portal whereby managers and employees can pull the knowledge from the portal.
KM Systems to Facilitate the Sharing of Human Knowledge (Continued)
Discussion Groups
– Discussion groups allow employees or customers to post questions and queries seeking solutions to problems they have.
– Oracle, IBM, PeopleSoft, and other vendors support Oracle IBM PeopleSoft and other vendors support
product discussion groups where users can post questions and where employees, vendors, and other users can answer them.
– Later, the organization can edit and summarize the questions from such discussion groups into frequently asked questions (FAQs).
22
KM Systems to Facilitate the Sharing of Human Knowledge (Continued)
Discussion groups (continued)
– Basic email can also be used for knowledge‐sharing, especially if email lists have been constructed with KM in mind.
– Two human factors inhibit knowledge‐sharing.
• Employees can be reluctant to exhibit their ignorance.
• Competition exists between employees.
– A KM application may be ill‐suited to a competitive group.
• The company may be able to restructure rewards and incentives to foster sharing of ideas among employees.
Figure 9‐27 Net Meeting Graphic
KM Systems to Facilitate the Sharing of Human Knowledge (Continued)
Collaboration Systems
– Collaboration systems are information systems that enable people to work together more effectively.
– The Internet can be used as a broadcast medium for speeches, panel discussion, and other types of meetings.
– Web broadcasts, because they are digital, can be readily saved and replayed at the viewer’ss convenience.
and replayed at the viewer
convenience
– Web broadcasts can also be made interactive by combining them with discussion group bulletin boards that are live during the broadcast.
– Video conferencing is another popular form of IT‐supported meetings.
• Video‐conferencing equipment is expensive and normally is located in selected sites in the organization.
KM Systems to Facilitate the Sharing of Human Knowledge (Continued)
Expert Systems
– Expert systems are created by interviewing experts in a given business domain and codifying the rules stated by those experts.
– Many expert systems were created in the late 1980s Many expert systems were created in the late 1980s
and 1990s, and some of them have been successful.
– Expert systems suffer from three major disadvantages.
• They are difficult and expensive to develop.
• They are difficult to maintain.
• They were unable to live up to the high expectations set by their name.
23