Download Data Mining - Universität Wien

Data Mining und Data Warehousing
Prof. Dr. Peter Brezany
Institut für Softwarewissenschaften
Universität Wien
Tel. 4277 38825
Sprechstunde: Di, 12.30-13.30
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Outline
Business Intelligence and its components
Knowledge discovery in databases
Data mining techniques
- description
- classification
- prediction
- clustering
- neural networks
Commercial data mining systems (Demo of the SAS
Enterprise Miner)
Data webhousing
Advanced topics: parallel and distributed data analysis
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 1
2
Literature
Mark and Mary Whitehorn: Business Intelligence:
The IBM Solution. Springer-Verlag, 2000.
R. Kimball: The Data Warehouse Toolkit. John Willey, 1996.
J. Han, M. Kamber: Data Mining. Concepts and Techniques
Morgam Kaufmann Publishers, 2000.
M. Ester, J. Sander: Knowledge Discovery in Databases.
Springer-Verlag, 2000.
I.H. Witten, E. Frank: Data Mining. (Practical Machine
Learning Tools and Techniques with Java Implementations).
Morgam Kaufmann Publishers, 2000.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
3
Business Intelligence
Definition:
Business Intelligence is an umbrella term, broadly covering the
processes involved in extracting valuable business information
and knowledge from the mass of data that exists within a typical
enterprise.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 2
4
Business Intelligence Tools
• Data warehouses
• OLAP (On-Line Analytical Processing) tools
• Data mining tools
• Text mining tools
the focus
of our
lectures
• Web mining tools
• Data joiners
• Business Intelligence portals, etc.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
5
Business Intelligence Tools (cont.)
• Data warehouse - a repository of multiple heterogeneous data
sources, organized under a unified schema at a single site in order
to facilitate management decision making.
• OLAP – analysis techniques with functionlities such as summarization, consolidation, and aggregation, as well as the ability to view
information from different angles.
• Data mining – extracting or “mining“ knowledge from large data sets.
• Text mining – “mining“ large textual (document) databases.
• Web mining – discovering knowledge from hypertext data.
• Data joiner - working with data from disparate, heterogeneous data
sources
• Business Intelligence portal – a Web site designed to be the first
point of entry for visitors to information about a company. With help
of the portal´s personalising functions, the use can choose information sources that he needs for performing a specific task. The portal
allows problemlos access to valuable information and data analyses;
so, the basis for competent decisions is optimized.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 3
6
DATA MINING
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
7
Introduction
• This lecture topic is about the theme which has come to be
known as data mining and knowledge discovery in large
databases, data warehouses, and other massive information
repositories.
• Data mining emerged during the late 1980s; has made great
strides during the late 1990s, and is expected to continue to
flourish into the new millennium.
• The implementation methods discussed are particularly oriented
towards the development of scalable and efficient data mining
tools.
• We introduce interesting data mining techniques and systems,
and discuss applications and research directions.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 4
8
What Motivated Data Mining? Why
Is It Important?
• There is the wide availability of huge amounts of data and the
imminent need for turning such data into useful information
and knowledge.
• Applications ranging from business management, production
control, and market analysis, to engineering design and science
exploration.
• Data mining can be viewed as a result of the natural evolution
of information technology - including database technology,
artificial intelligence, machine learning, neural networks,
statistics, pattern-recognition, knowledge-based systems,
high-performance computing, and data visualization.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
9
Collecting Data
Laboratories
Satellites
(microscopes,
MRI/CT scanners, ...)
Business
Data Repositories
Experiments
Computer simulations
(high energy physics,...)
P.Brezany
Analysis
Institut für Softwarewissenschaften – Universität Wien
Page 5
10
CERNs challenge
• Starting point
– New detector LHC
» Large Hadron Collider, 14 TeV
» Goals: Search for Higgs Boson and
Graviton (and others)
– Start 2006
• Challenges
– Data are accessed worldwide
» CERN and Regional Centers (Europe, Asia, America)
» 2000 users
– Huge data volumes
– Data semantics
– Performance and throughput
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
The LHC Detectors
11
CMS
ATLAS
LHCb
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 6
12
Multi-Tier Model
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
13
The Evolution of Database Technology
Data Collection and Database Creation (1960s and earlier)
- Primitive file processing
Database Management Systems (1970s-early 1980s)
- Hierarchical, network and relational DB systems
- Query languages (SQL, etc), query optimization
- Transaction management, concurrency control, recovery
- Data modeling tools
Advanced Database Systems
(mid-1980s-present)
object-oriented, object-relational,
spatial, temporal, multimedia
Web-based Database Systems
(1990s-present)
- XML-based DB systems,
- Web mining
Data Warehousing and Data Mining (late 1980s-present)
- Data warehouse and OLAP technology
- Data mining and knowledge discovery
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 7
14
Database Querying and Data Mining
Query languages like SQL are standardized and powerful, but for not skilled
users are they too difficult.
OLAP Tools allow flexible multidimensional queries. Their methods are querycentric.
Data Warehouse
Query languages like SQL
P.Brezany
OLAP Tools
Institut für Softwarewissenschaften – Universität Wien
Data Mining Tools
15
We Are Data Rich, But Information Poor
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 8
16
So, What Is Data Mining?
Data mining – searching for knowledge (interesting patterns)
in your data.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
17
Data Mining As a Step in the
Process of Knowledge Discovery
• Many people treat data mining as a synonym for the
term Knowledge Discovery in Databases, or KDD.
• Alternative view: data mining as an step in KDD:
– 1, Data cleaning (to remove noise and inconsistent data)
– 2. Data integration (where multiple data sources may be combined)
– 3. Data selection (where data relevant to the analysis task are
retrieved from the database)
– 4. Data transformation (where data are transformed or consolidated
into forms appropriate for mining by performing summary or
aggregation operations, for instance)
– 5. Data mining (an essential process where intelligent methods are
applied in order to extract patterns)
– 6. Pattern evaluation (to identify the truly interesting patterns
representing knowledge based on some interestingness measures)
– 7. Knowledge presentation to the user
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 9
18
Data Mining in Knowledge Discovery
P.Brezany
19
Institut für Softwarewissenschaften – Universität Wien
Architecture of a Data Mining System
Graphical user interface
Pattern evaluation
Knowledge
base
Data mining engine
Database or
data warehouse server
Data cleaning, data integration
Filtering
Database
P.Brezany
Data
warehouse
Institut für Softwarewissenschaften – Universität Wien
Page 10
20
Architecture of a Data Mining System (2)
Database, data warehouse, or other information repository:
One or a set of databases, data warehouses, spreadsheets, etc.
Database or data warehouse server: responsible for fetching
the relevant data, based on the user’s data mining request.
Knowledge base: domain knowledge that is used to guide the
search, or evaluate the interestingness of resulting patterns.
Such knowledge can include concept hierarchies, used to organize attribute values into different levels of abstraction.
Data mining engine: essential to the data mining system; ideally
consists of a set of functional modules for tasks such as characterization, association, classification, cluster analysis, and evolution and deviation analysis.
21
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Architecture of a Data Mining System (3)
Pattern evaluation module: This component typically employs
interestingness measures and interacts with the data mining so
as to focus the search towards interesting patterns. It may use
interestingness thresholds to filter out discovered patterns.
Graphical user interface: This module communicates between
users and the data mining system allowing the user
• to specify a data mining query or task
• provide information to help focus the search
• perform exploratory data mining based on the intermediate
data mining results
• browse database and data warehouse schemas or data structures
• evaluate mined patterns
• visualize the patterns in different forms.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 11
22
Data Mining vs. Other Disciplines
From a data warehouse perspective, data mining can be
viewed as an advance stage of on-line analytical processing
(OLAP). However, data mining goes far beyond OLAP.
There may be many “data mining systems” on the market not all of them can perform true data mining. They should be
more appropriately categorized as
• machine learning systems
• statistical data analysis tools
• experimental system prototypes
• deductive database systems
Data mining integrates techniques from multiple disciplines:
database technology, statistics, machine learning, high-performance computing, neural networks, pattern recognition, visualiz.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
23
Data Mining - On What Kind of Data?
•
•
•
•
Relational Databases
Data Warehouses
Transactional Databases
Advanced Database Systems and Advanced Database
Applications
–
–
–
–
–
–
–
P.Brezany
Object-oriented databases
Object-relational databases
Spatial databases
Temporal databases and time-series databases
Text databases and multimedia databases
Heterogeneous databases and legacy databases
The World Wide Web
Institut für Softwarewissenschaften – Universität Wien
Page 12
24
Relational Database - An Example
• A database system, also called a database management system
(DBMS), consists of a collection of interrelated data, known as a
database, and a set of software programs to manage and access
the data.
• A relational database is a collection of tables, each of which is
assigned a unique name. Each table consists of a set of attributes
(columns or fields) and usually stores a large set of tuples
(records or rows). Each tuple represents an object identified by a
unique key.
• Relational data can be accessed by database queries written in a
relational query language, such as SQL.
• Using data mining, one can search for trends or data patterns in
relational databases.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
25
Relational Databases - Example
The AllElectronics company is described by the following
table: customer, item, employee, and branch. Fragments of
these tables are shown on the next slide; the attribute that
represents the key or composite key component is underlined.
•The relation customer consists of a set of attributes, including a unuque customer identity number (cust_ID), and so on.
•Tables can also be used to represent the relationships between or among multiple relational tables. E.g., these include
purchases (customer purchases items, creating a sales transaction that is handled by an employee), items_sold (lists the
items sold in the given transaction), and works_at (employee
works at a branch of AllElectronics).
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 13
26
Fragments of Relations from AllElectronics DB
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
27
Data Warehouses
A data warehouse is a repository of information collected from
multiple sources, stored under a unified schema, and which
usually resides at a single site.
Data warehouses are constructed via a process of data cleaning,
data transformation, data integration, data loading and periodic
data refreshing.
Figure on the next slide shows the basic architecture of a data
warehouse for AllElectronics.
In order to facilitate decision making, the data in a data warehouse are organized around major subjects, such as customer,
item, supplier, and activity. The data are stored from a historical perspective and are typically summarized.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 14
28
Architecture of a Data Warehouse
Client
Data source in Ch.
Data source in NY
Clean
Transform
Integrate
Load
Data
warehouse
Query and
analysis tools
Client
Data source in T.
Data source in Vancouver
P.Brezany
Remarks: Ch - Chicago, NY - New York, T - Toronto
Institut für Softwarewissenschaften – Universität Wien
29
Modeling a Data Warehouse
A data warehouse is usually modeled by a multidimensional
database structure, where each dimension corresponds to an
attribute in the schema, each cell stores the value of some
aggregate measure, such as count or sales_amount.
The actual physical structure of a data warehouse may be a
relational data store or a multidimensional data cube. It
provides a multidimensional view of data and allows the
precomputation and fast accessing of summarized data.
Example: A data cube for summarized sales data of
AllElectronics is presented in the next slide.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 15
30
A Multidimensional Data Cube
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
31
Modeling a Data Warehouse (2)
Data warehouse vs. Data mart: A data warehouse collects
information about subjects and span an entire organization,
and thus its scope is enterprise wide. A data mart is
a department-wide.
Data warehouse systems are well suited for On-Line Analytical
processing, or OLAP.
OLAP operations allow the presentation of data at different
levels of abstractions.
Examples of OLAP operations include drill-down and roll-up,
which allow the user to view the data at different degrees of
summarization as illustrated in the previous slide.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 16
32
Transactional Databases
A transactional database consists of a file where each record
represents a transaction.
A transaction includes a unique transaction identity number
(trans_id), and a list of the items making up the transaction
(such as items purchased in a store).
The transactional database may have additional tables associated
with it, which contain other information regarding the sale, such
as the date of the transaction, the custommer ID number, the ID
number of the sales person, etc.
Example: Transactions can be stored in a table, with one
record per transaction. A fragment of a transactional database
for AllElectronics is shown in the next slide.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
33
Transactional Databases (2)
Trans_id
list of item_Ids
T100
...
I1, I3, I8, I16
...
The transactional database is usually either stored in a flat file
in a format similar to that of the above table, or unfolded into
a standard relation in a format similar to that of the
items_sold table in slide no. 18.
A regular data retrieval system is not able to answer queries
like “Which items sold well together?”
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 17
34
Advanced Database Systems and
Database Applications
Relational DB systems have been widely used in business applications.
The new database applications include handling
• spatial data (e.g. maps)
• engineering design data (e.g., the design of buildings or
integrated circuits)
• hypertext and multimedia data (text, image, video, audio data)
• time-related data (e.g. stock exchange data)
• World Wide Web (a huge, widely distributed information repository made available by the Internet)
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
35
Data Mining Functionalities - What
Kinds of Patterns Can be Minded?
• Data mining functionalities are used to specify the kind
of patterns that can be found in data mining tasks.
• Data mining tasks can be classified into 2 categories:
– Descriptive - they characterize the general properties of the data in
the database.
– Prescriptive - they perform inference on the current data in order to
make predictions.
• In some cases, users may have no idea which kinds of
patterns may be interesting => searching for several
different kinds of patterns in parallel.
• Data mining systems should be able to discover
patterns at various granularities (abstraction levels).
• Specifying hints to guide or focus the search.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 18
36
Concept/Class Description:
Characterization and Discrimination
• Data can be associated with classes and concepts.
• E.g., in the AllElectronics store,
– classes of items for sale include computers and printers, and
– concepts of customers include bigSpenders and budgetSpenders.
• It can be useful to describe individual classes and concepts in
summarized, concise, and yet precise terms => class/concept
descriptions.
• These descriptions can be derived via
– !. Data characterization, by summarizing the data of the class under study
(often called the target class) in general terms, or
– 2. Data discrimination, by comparison of the target class with one or a set of
comparative classes (often called contrasting classes), or
– 3. Both data characterization and discrimination.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
37
Concept/Class Description - Examples
• Example 1 A description summarizing the characteristics of customers who spent more than $1000 a year at
AllElectronics. The result could be a general profile of
the customers, such qs they are 40-50 years old,
employed, and have excellent credit ratings.
• Example 2 A comparison of 2 custommer groups, such
as those who shop for computer products regularly
versus those who rarely shop for such products. The
resulting description could be a general comparative
profile of the customers, such as 80% of the customers
who frequently purchase computer products are
between 20 and 40 years old and have a university
education, whereas 60% of the customers who
infrequently buy such products are either seniors, and
have no university degree.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 19
38
Association Analysis
• Association analysis is the discovery of association rules
showing attribute-value conditions that occur frequently
in a given set of data.
• The association rule X => Y is interpreted as “database
tuples that satisfy the conditions in X are also likely to
satisfy the conditions in Y.”
• Example A data mining system may find in AllElectronics:
age(X, “20..29”) and income(X, “20K..29K”) => buys(X,”CD
player”) [support = 2%, confidence = 60%]
• X is a variable representing a customer. The rule indicates
that of the customers under study, 2% are 20 to 29 years
of age with an income of 20K to 29K and have purchased a
CD player. There is a 60% probability that a customer in
this age and incomegroup will purchase a CD player.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
39
Association Analysis (Cont.)
• We would like to determine which items are frequently
purchased together within the same transactions. E.g.,
contains(T, “computer”) => contains(T, “software”)
[support = 1%, confidence = 50%]
• Explanation: if a transaction, T, contains “computer”,
there is a 50% chance that it contains “software” as
well, and 1% of all of the transactions contain both.
• This rule involves a single attribute or predicate (i.e.
contains) => single-dimensional association rule. It can
be written simpy as “computer => software {1%,50%]”
Remark: On the last slide, we have: multi-dimensional assoc. rule.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 20
40
Classification and Prediction
• Classification is the process of finding a set of models
(or functions) that describe and distinguish data
classes or concepts, for the purpose of being able to
use the model to predict the class of objects whose
class label is unknown. The derived model is based on
the analysis of a training data (i.e., data objects whose
class label is known),
• “How is the derived model presented?”
– Classification (IF-THEN) rules
– Mathematical formulae
– Decision tree - it is a flow-chart-like tree structure, where each
node denotes a test on an attribute value, each branch represents an
outcome of the test, and the tree leaves represent classes or class
distributions.
– Neural networks - a collection of neuron-like processing units with
weighted connections between the units.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
41
Classification and Prediction (Cont.)
• Prediction - in many applications, users may wish to
predict some missing or unavailable data values rather
then class labels. The predicted values are usually
numerical data.
• Prediction also encompasses the identification of
distribution trends based on the available data.
• Classification and prediction may need to be preceded
by relevance analysis, which attempts to identify
attributes that do not contribute to the classification
or prediction process. These attributes can then be
excluded.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 21
42
Classification and Prediction (Cont.)
• Example Classification of a large set of items in the
store based on three kinds of responses to a sales
campaign: good response, mild response, and no response. We would like to derive a model for each of these
classes based on the descriptive features of the items,
such as price, brand, place_made, type, and category.
• The resulting classification should maximally distinguish
each class from the others, presenting an organized
picture of the data set.
• A decision tree, for instance, may identify price as
being the single factor that best distinguishes the
three classes. Such a decision tree may help us
understand the impact of the given campaign and design
a more effective campaign for the future.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
43
Cluster Analysis
• Clustering analyzes data objects without consulting a
known class label.
• Clustering can be used to generate such labels.
• The objects are clustered or grouped based on the
principle of maximizing the intraclass similarity and
minimizing the interclass similarity.
• Each cluster can be viewed as a class of objects, from
which rules can be derived.
• Example Cluster analysis can be performed on AllElectronics customer data in order to identify homogeneous subpopulations of customers. These clusters may
represent individual target groups for marketing. (Figure
on the next slide shows a 2-D plot of customers with respect to
customer locations in a city).
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 22
44
Cluster Analysis - Example
A 2-D plot of customer data with respect to customer locations
In a city, showing 3 data clusters. Each cluster „center“ is marked with a „+“.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
45
Outlier Analysis
• A database may contain data objects that do not
comply with the general behaviour or model of the
data. These data objects are outliers,
• Most data mining methods discard outliers as noise or
exceptions.
• In some applications such as fraud detection, the rare
events can be more interesting than the more regularly
occuring ones,
• Example Outlier analysis may uncover fraudulent usage
of credit cards by detecting purchases of extreemly
large amounts for a given account number in comparison
to regular charges incurred by the same account.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 23
46
Evolution Analysis
• It describes and models regularities or trends for
objects whose behavior changes over time.
• It includes time-series data analysis.
• Example Suppose that we have the major stock market
(time-series) data of the last several years available
from the New York Stock Exchange and we would like
to invest in shares of high-tech industrial companies. A
data mining study of stock exchange data may identify
stock evolution regularities for overall stocks and for
the stocks of particular companies, Such regularities
may help predict future trends in stock market prices.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
47
Are All of the Patterns Interesting?
• A data mining system has the potential to generate
thousands or even millions of patterns, or rules.
• Only a small fraction of the patterns potentially
generated would actually be of interest to any given
user.
• Questions: What makes a pattern interesting? Can a
data mining system generate all of the interesting
patterns? Can a data mining system generate only
interesting patterns?
• A pattern is interesting if
–
–
–
–
P.Brezany
(1) it is easily understood by humans,
(2) valid on new or test data with some degree of certainty.
(3) potentially useful , and
(4) novel
Institut für Softwarewissenschaften – Universität Wien
Page 24
48
Interestingness of Patterns (Cont.)
• A pattern is also interesting if it validates a hypothesis
that the user sought to confirm.
• An interesting pattern represents knowledge.
• Objective measures of pattern interestingness these are based on the structure of discovered
patterns and the statistics underlying them.
– An objective measure for association rules X => Y is rule
support,representing the percentage of transactions from a transaction base that the given rule satisfies. This is taken to be the probability P(X U Y), where X U Y indicates that a transaction contains
both X and Y, that is, the union of item sets X and Y.
– Another objective measure for association rules is confidence, which
assesses the degree of certainty of the detected association. This is
taken to be the conditional probability P(X | Y), that is, the
probability that a transaction containing X also contains Y.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
49
Interestingness of Patterns (Cont.)
• Each interestingness measure is associated with a
threshold, which can be controlled by the user.
– For example, rules that do not satisfy a confidence threshold of, say,
50% can be considered uninteresting. Rules below the threshold likely
reflect noise, exceptions, or minority cases and are probably of less
value.
• Objective measures are insufficient unless combined
with subjective measures which reflect the needs and
interests of a particular user.
• Many patterns that are interesting by objective standards may represent common knowledge and, therefore, are actually uninteresting.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 25
50
Interestingness of Patterns (Cont.)
• Subjective interestingness measures are based on
user beliefs in the data. These measures find patterns
interesting if they are unexpected or offer strategic
information on which the user can act.
• Patterns that are expected can be interesting if they
confirm a hypothesis that the user wished to validate,
or resemble a user’s hunch.
P.Brezany
Institut für Softwarewissenschaften – Universität Wien
Page 26
51