Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Data
Document related concepts
Transcript
Introduction to
Knowledge Discovery
and Data Mining
TuBao Ho (HieuChi Dam)
School of Knowledge Science
Japan Advanced Institute of Science and Technology
19 November 2005
The lecture aims to …
Provide basic concepts and
techniques of knowledge
discovery and data mining (KDD).
Emphasize on different kinds of
data, different tasks to do with
the data, and different methods
to do the tasks.
Emphasize on the KDD process
and important issues when using
data mining methods.
19 November 2005
Outline
1. Why knowledge discovery and data mining?
2. Basic concepts of KDD
3. KDD techniques: classification, association,
clustering, text and Web mining
4. Challenges and trends in KDD
5. Case study in medicine data mining
19 November 2005
Much more data around us than before
We are living in the most exciting of times:
Computer and computer networks
19 November 2005
Astronomical data
Astronomy is facing a major data avalanche:
Multi-terabyte sky surveys and archives (soon:
multi-petabyte), billions of detected sources,
hundreds of measured attributes per source …
19 November 2005
Astronomical data
Multi-wavelength data paint a more complete
(and a more complex!) picture of the universe
Infrared emission from
interstellar dust
Smoothed galaxy
density map
19 November 2005
Earthquake data
Japanese
earthquakes
1961-1994
1932-1996
04/25/92 Cape
Mendocino, CA
19 November 2005
Predict earthquakes
9 August 2004 (AP): Swedish geologists
may have found a way to predict
earthquakes weeks before they happen
(current accurate warnings only come
seconds before a quake).
Water samples taken 4,900 feet beneath the ground in
northern Iceland show the content of several metals
increased dramatically a few weeks before a magnitude
5.8 earthquake struck.
"We need a database over other earthquakes." The
bedrock at the test site is basalt, which is also found in
other earthquake-prone areas like Hawaii and Japan.
19 November 2005
Finance: the market data
Data on price fluctuation throughout the day in the market
19 November 2005
Ishikawa’s monthly industrial data
250,000
200,000
150,000
100,000
50,000
大型小売店売上高
0
1
2
3
4
5
6
7
19 November 2005
Explosion of biological data
10,267,507,282
bases in
9,092,760
records.
19 November 2005
How biological data look like?
A portion of the DNA sequence, consisting of 1.6 million
characters, is given as follows (about 350 characters,
4570 times smaller):
…TACATTAGTTATTACATTGAGAAACTTTATAATTAAAAAAGATTCATGTAA
ATTTCTTATTTGTTTATTTAGAGGTTTTAAATTTAATTTCTAAGGGTTTGC
TGGTTTCATTGTTAGAATATTTAACTTAATCAAATTATTTGAATTTTTGAAA
ATTAGGATTAATTAGGTAAGTAAATAAAATTTCTCTAACAAATAAGTTAAAT
TTTTAAATTTAAGGAGATAAAAATACTACTCTGTTTTATTATGGAAAGAAA
GATTTAAATACTAAAGGGTTTATATATATGAAGTAGTTACCCTTAGAAAAAT
ATGGTATAGAAAGCTTAAATATTAAGAGTGATGAAGTATATTATGT…
19 November 2005
Text: huge sources of knowledge
Approximately 80% of the world’s data is held in
unstructured formats (source: Oracle Corporation)
Web sites, digital libraries, … increase the volume of
textual data
例：JAISTの図書館
オンライン参照可能なジャーナル数：4700 （280000論文/年）
1%（＝2820論文）を読むには、8論文／日がノルマ.
1960s:
Easy
1980s
Time-consuming
2000s
Difficult
Soon: impossible
19 November 2005
MEDLINE: a medical text database
The world's most comprehensive source of life sciences
and biomedical bibliographic information, with nearly
eleven million records (http://medline.cos.com)
About 40,000 abstracts on hepatitis (concerning our
research project), below is one of them
36003: Biomed Pharmacother. 1999 Jun;53(5-6):255-63.
Pathogenesis of autoimmune hepatitis.
Institute of Liver Studies, King's College Hospital, London, United Kingdom.
Autoimmune hepatitis (AIH) is an idiopathic disorder affecting the hepatic parenchyma. There are no morphological
features that are pathognomonic of the condition but the characteristic histological picture is that of an interface hepatitis
without other changes that are more typical of other liver diseases. It is associated with hypergammaglobulinaemia, high
titres of a wide range of circulating auto-antibodies, often a family history of other disorders that are thought to have an
autoimmune basis, and a striking response to immunosuppressive therapy. The pathogenetic mechanisms are not yet
fully understood but there is now considerable circumstantial evidence suggesting that: (a) there is an underlying genetic
predisposition to the disease; (b) this may relate to several defects in immunological control of autoreactivity, with
consequent loss of self-tolerance to liver auto-antigens; (c) it is likely that an initiating factor, such as a hepatotropic viral
infection or an idiosyncratic reaction to a drug or other hepatotoxin, is required to induce the disease in susceptible
individuals; and, (d) the final effector mechanism of tissue damage probably involves auto-antibodies reacting with liverspecific antigens expressed on hepatocyte surfaces, rather than direct T-cell cytotoxicity against hepatocytes.
19 November 2005
Web server access logs data
Typical data in a server access log
looney.cs.umn.edu han - [09/Aug/1996:09:53:52 -0500] "GET mobasher/courses/cs5106/cs5106l1.html HTTP/1.0" 200
mega.cs.umn.edu njain - [09/Aug/1996:09:53:52 -0500] "GET / HTTP/1.0" 200 3291
mega.cs.umn.edu njain - [09/Aug/1996:09:53:53 -0500] "GET /images/backgnds/paper.gif HTTP/1.0" 200 3014
mega.cs.umn.edu njain - [09/Aug/1996:09:54:12 -0500] "GET /cgi-bin/Count.cgi?df=CS home.dat\&dd=C\&ft=1 HTTP
mega.cs.umn.edu njain - [09/Aug/1996:09:54:18 -0500] "GET advisor HTTP/1.0" 302
mega.cs.umn.edu njain - [09/Aug/1996:09:54:19 -0500] "GET advisor/ HTTP/1.0" 200 487
looney.cs.umn.edu han - [09/Aug/1996:09:54:28 -0500] "GET mobasher/courses/cs5106/cs5106l2.html HTTP/1.0" 200
...
...
...
19 November 2005
Web link data
Internet Map
[lumeta.com]
Friendship Network
[Moody ’01]
Food Web
[Martinez ’91]
19 November 2005
What do we want from the data?
Much more data of different kinds were collected
than before.
Want to exploit the data, to extract new and useful
information/knowledge in the data, such as
Which phenomenon can be seen from data when a disease
occurred?
What are properties of several metals in 4,900 feet
beneath the ground?
Is Japan stock market rising this week?
How other researchers talked about “interferon effect”?
etc.
Want to draw valid conclusions from data.
19 November 2005
What statistics usually does?
Statistics provides principles and methodology for designing the
process of
( 統計学は、下記プロセスを設計する際の原理や方法論の基礎を提供する):
Data collection
データ収集
Summarizing and
Interpreting the
data
データ要約と解釈
Drawing
conclusions or
generalities
結論と一般性の記
述
19 November 2005
Evolution of data processing
Evolutionary Step
Business Question
Enabling Technology
Data Collection
(1960s)
“What was my total
revenue in the last five
years?”
computers, tapes, disks
Data Access (1980s)
“What were unit sales in
Korea last March?”
faster and cheaper computers with
more storage, relational databases,
structured query language (SQL), etc.
Data Warehousing
and Decision
Support
“What were unit sales in
Korea last March? Drill
down to Hokkaido”.
faster and cheaper computers with
more storage, On-line analytical
processing (OLAP), multidimensional
databases, data warehouses
Data Mining
“What’s likely to happen
to Hokkaido unit sales
next month? Why?
faster and cheaper computers with
more storage, advanced algorithms,
massive databases
Drill down: To move from summary information to the detailed data that created it （集計データを集計の元と
なる明細データへと掘り下げる操作). For example, adding totals from all the orders for a year creates gross
sales for the year. Drilling down would identify the types of products that were most popular.
19 November 2005
KDD: Convergence of thee technologies
Increasing
computing power
Statistical
and learning
algorithms
KDD
Improved data
collection and
management
19 November 2005
Increasing computing power
1.6 meters
1966
30MB
JAIST’s CRAY XT3
計算ノード：
CPU: AMD Opteron150
2.4GHz ×4×90
メモリ: 32GB ×90 ＝ 2.88TB
CPU間接続: 3Dトーラス結合
帯域幅: CPU-CPU間
7.68GB/s(双方向)
Lab PC cluster: 16
nodes dual Intel
Xeon 2.4 GHz
CPU/512 KB cache
19 November 2005
How much information is there?
Yotta
Soon everything can be
recorded and indexed
Most bytes will never
be seen by humans
What will be key
technologies to deal
with huge volumes of
information sources?
Everythin
g
Recorded
All Books
MultiMedia
Tera
.
Movie
[This page adapted from the
invited talk of Jim Gray (Microsoft)
at KDD’2003]
Exa
Peta
All books
(words)
20 TB contains
20 M books in LC
Zetta
A Photo
A Book
Giga
Mega
Kilo
19 November 2005
Relational databases
A relational database is a collection of tables, each of which is assigned a
unique name, and consists of a set of attributes * and a set of tuples**.
*: 顧客IDなどのデータ項目 **：データレコード(行)
customer
Cust-ID
C1
name
Smith, Sandy
…
item
Item-ID
I3
I8
…
…
name
high-res-TV
multidiscCDplayer
brand
Toshiba
Sanyo
…
address
5463 E Hasting, Burnaby
BC V5A 459, Canada
…
category
high resolution
multidisc
…
age
21
…
type
TV
CD player
…
price
$988.00
$369.00
…
income
$27000
credit-info
1
…
.
…
…
place-made
Japan
Japan
…
…
supplier
NIkoX
MusicFont
…
cost
$600.00
$120.00
…
employee
Emp-ID
E35
…
branch
Branch-ID
B1
…
name
Jones, Jane
…
name
City square
…
category
home entertainment
…
group
manager
…
address
369 Cambie St., Vancouver, BC V5L 3A2, Canada
…
purchases
Trans-ID
T100
…
cust-ID
C1
. …
empl-ID
B55
…
data
01/21/98
…
time
15:45
…
method-paid
Visa
…
amount
$1357.00
…
salary
$18,000
…
commision
2%
…
Item-sold
works-at
Trnas-ID item-ID sty
Empl-ID branch-ID
T100
T100
…
E55
…
I3
I8
…
1
2
…
B1
…
19 November 2005
Data warehouses
A data warehouse is a repository of information collected from multiple
resources, stored under a unified schema, and which is usually resides
at a single site. 複数のデータ資源からの情報を収集し、形式を統一して一箇所にある単一
のデータ倉庫に格納
Data source
in Hokkaido
Data source
in Kanazawa
Data source
in Busan
Data source
in Hongkong
client
Clean
Transform
Integrate
Load
Data
warehouse
Query and
analysis tool
client
19 November 2005
Transactional databases
A transactional database consists of a file where
each record represents a transaction.
A transaction typically includes a unique
transaction identity number (trans_ID), and list of
the items making up the transaction.
Trans_ID
T100
T200
T300
T400
T500
list of item_ID
beer, cake, onigiri
beer, cake
beer, onigiri
beer, onigiri
cake
19 November 2005
Advanced database systems
Object-Oriented Databases
Object-Relational Databases
Spatial Databases
Temporal Databases and
Time-Series Databases
Text Databases and Multimedia
Databases
Heterogeneous Databases and
Legacy Databases
The World Wide Web
19 November 2005
Spatial databases
Spatial databases contain
spatial-related information:
geographic databases, VLSI
chip design databases, medical
and satellite image databases
etc.
Data mining may uncover
patterns describing the content
of several metals in specific
location when earthquakes
happen, the climate of
mountainous areas located at
various altitudes, etc.
Japanese
earthquakes
1961-1994
19 November 2005
Temporal and time-series databases
They store time-related data. A
temporal database stores relational
data that include time-related
attributes (timestamps with different
semantics). A time-series database
stores sequences of values that
change with time (stock exchange)
Data mining finds the characteristics
of object evolution, trend of change
for objects: e.g., stock exchange
data can be mined to uncover trends
in investment strategies
Spatial-temporal databases
19 November 2005
Text and multimedia databases
Text databases contain
documents, usually highly
unstructured or semi-structured.
To uncover general descriptions of
object classes, keywords, content
associations, clustering behavior
of text objects, etc.
Multimedia databases store
image, audio, and video data:
picture content-based retrieval,
voice-email systems, video-ondemand-systems, speech-based
user interface, etc.
19 November 2005
The world wide web
The Web provides an enormous source of
explicit and implicit knowledge that people
can navigate and search for what they need.
Example: When examining the data
collected from Internet Mart, heavily
trodden paths gave BT hints to regions of the
site which were of key interest to its
visitors.
19 November 2005
Statistical and learning algorithms
Techniques have often been waiting for computing
technology to catch up
Development and improvement of statistical and learning
algorithms during last decades: support vector machine
and kernel methods, multi-relational data mining, graphbased learning, finite state machines, etc.
1960s
t-1
2000s
t
t+1
Ot -1
Ot
t-1
...
...
observations
...
...
...
transitions
Ot+1
HMMs (Hidden Markov
Models: directed graph,
joint, generative)
t
t+1
transitions
...
St-1
St
St+1
...
observations
Ot -1
Ot
Ot+1
MEMMs (Maximum Entropy
Markov Models: directed graph,
conditional, discriminative)
Ot-1
Ot
Ot+1
...
CRFs (Conditional Random
Fields: undirected graph,
conditional, discriminative)
19 November 2005
独立成分分析(ICA) vs. 主成分分析(PCA)
Principal Component Analysis
(PCA) finds directions of
maximal variance in Gaussian
data (second-order statistics).
主成分分析(PCA)： ガウス分布
データにおいて分散が最大となる
方向の発見 (一次統計).
Independent Component
Analysis (ICA) finds directions
of maximal independence in
non-Gaussian data (higherorder statistics).
独立成分分析 (ICA)：非ガウス分
布データにおいて独立性が最大
となる方向の発見 (高次統計).
19 November 2005
ICA: 複数センサで取得した信号データの分離
Perform ICA
Mic 1
Mic 2
Mic 3
Mic 4
Terry
Te-Won
Play Mixtures
Scott
Tzyy-Ping
Play Components
19 November 2005
Need of powerful tools to analyze data
People gathered and stored so much data because they
think some valuable assets are implicitly coded within it.
Its true value depends on the ability to extract useful
information.
How to acquire knowledge for knowledge–based systems
remains as the main difficult and crucial artificial
intelligence problem.
19 November 2005
Outline
1. Why knowledge discovery and data mining?
2. Basic concepts of KDD
3. KDD techniques: classification, association,
clustering, text and Web mining
4. Challenges and trends in KDD
5. Case study in medicine data mining
19 November 2005
Data, information, and knowledge
Metaphor:
Data: rock; knowledge: ore.
Miner?
事実および関係などからな
る総合的な情報 (“検証され
た真実”)
(E = mc2)
意味をもつデータ
(object’s mass: measure of
an object's resistance to
changes in either the speed
or direction of its motion)
未解釈の信号
25.1 27.3 21.6 …
19 November 2005
Data, information, and knowledge
US$ K
Mean of Debt = 18.4, Mean of Income = 34.5
Debt
(information)
Have defaulted
on the loan
Good status
with the bank
0
33
Income
(knowledge)
“if income < $33K, then the person has defaulted on the loan”
(income, debt)
1. ( 5.6, 8.5)
2. ( 6.0, 13.0)
3. (11.0, 12.0)
4. (11.0, 19.0)
5. (13.5, 10.0)
6. (16.5, 20.0)
7. (17.5, 15.0)
8. (17.5, 5.0)
9. (22.5, 25.0)
10. (26.0, 7.5)
11. (30,0, 9.0)
12. (30.0, 18.0)
13. (30.0, 30.0)
14. (31.0, 14.0)
15. (32.5, 25.0)
16. (38.0, 12.0)
17. (41.0, 9.0)
18. (41.0, 22.0)
19. (43.5, 12.5)
20. (44.0, 27.5)
21. (45.0, 22.5)
22. (48.0, 28.0)
23. (52.5, 21.0)
24. (53.5, 32.0)
25. (54.0, 27.5)
26. (57.5, 18.0)
27. (59.0, 18.0)
28. (62.5, 32.5)
29. (63.0, 18.0)
34.5, 18.4
19 November 2005
知識発見とデータマイニング
(KDD: Knowledge Discovery and Data Mining)
the automatic extraction of non-obvious,
hidden knowledge from large volumes of data
大量データに潜在する未発見の知識の自動抽出
106-1012 bytes:
データセット全体の把握・
コンピュータメモリへの展開が
困難な大規模データベース
どのデータマイニング
アルゴリズムを適用するか？
どんな知識か？
どう表現するか？
19 November 2005
From data to knowledge
Meningitis data, Tokyo Med. & Dental Univ., 38 attributes
...
10, M, 0, 10, 10, 0, 0, 0, SUBACUTE, 37, 2, 1, 0,15,-,-, 6000, 2, 0, abnormal, abnormal,-, 2852,
2148, 712, 97, 49, F,-,multiple,,2137, negative, n, n, ABSCESS, VIRUS
12, M, 0, 5, 5, 0, 0, 0, ACUTE, 38.5, 2, 1, 0,15, -,-, 10700,4,0,normal, abnormal, +, 1080, 680, 400,
71, 59, F,-,ABPC+CZX,, 70, negative, n, n, n, BACTERIA, BACTERIA
15, M, 0, 3, 2, 3, 0, 0, ACUTE, 39.3, 3, 1, 0,15, -, -, 6000, 0,0, normal, abnormal, +, 1124, 622, 502,
47, 63, F, -,FMOX+AMK, , 48, negative, n, n, n, BACTE(E), BACTERIA
16, M, 0, 32, 32, 0, 0, 0, SUBACUTE, 38, 2, 0, 0, 15, -, +, 12600, 4, 0,abnormal, abnormal, +, 41,
39, 2, 44, 57, F, -, ABPC+CZX, ?, ?, negative, ?, n, n, ABSCESS, VIRUS
...
numerical
categorical
missing
class attribute
IF cell_poly <= 220 AND Risk = n AND Loc_dat = + AND
Nausea > 15 THEN Prediction = VIRUS [confidence = 87,5%]
19 November 2005
KDD: An interdisciplinary field
Statistics
Infer information from data
(deduction and induction,
mainly numeric data)
KDD
Databases
Store, access, search,
update data (deduction)
Machine Learning
Computer algorithms that improve
automatically through experience
(mainly induction, symbolic data)
also Algorithmics, Visualization, Data warehouses, OLAP, etc.
19 November 2005
KDD: New and fast growing area
KDD’95, 96, 97, 98, …, 04, 05 (ACM, America)
PAKDD’97, 98, 99, 00, …, 04, 05 (Pacific & Asia)
http://www.jaist.ac.jp/PAKDD-05
PKDD’97, 98, 99, 00, …, 04, 2005 (Europe)
ICDM’01, 02,…, 04, 05 (IEEE), SDM’01, …, 04, 05 (SIAM)
Industrial Interest: IBM, Microsoft, Silicon Graphics,
Sun, Boeing, NASA, SAS, SPSS, …
Japan: FGCS Project focus on logic programming and
reasoning; attention has been paid on knowledge
acquisition and machine learning. Projects “Knowledge
Science”, “Discovery Science”, and “Active Mining
Project” (2001-2004)
19 November 2005
The KDD process
5
a step in the KDD process
consisting of methods that
produce useful patterns or
models from the data
Maybe 70-90%
of effort and
cost in KDD
2
1
3
4
Putting the results
in practical use
Interpret and Evaluate
discovered knowledge
Data Mining
Extract Patterns/Models
Collect and
Preprocess Data
Understand the domain
and Define problems
KDD is inherently
interactive and iterative
19 November 2005
Common tasks in the KDD process
1
Data organized by function
Create/select
target database
Data warehousing
Select sampling
technique and
sample data
2
3
5
Supply missing
values
Eliminate
noisy data
Normalize
values
Transform
values
Create derived
attributes
Select DM
task (s)
Select DM
method (s)
Extract
knowledge
Transform to
different
representation
Query & report generation
Aggregation & sequences
Advanced methods
Find important
attributes &
value ranges
Test
knowledge
Refine
knowledge
4
19 November 2005
Data schemas vs. mining methods
Types of data
Flat data tables
Relational database
Temporal & Spatial
Transactional databases
Multimedia data
Genome databases
Materials science data
Textual data
Web data
etc.
Mining tasks and methods
Classification/Prediction
Decision trees
Neural network
Rule induction
Support vector machines
Hidden Markov Model
etc.
Description
Association analysis
Clustering
Summarization
etc.
Different data schemas
19 November 2005
Dataset: cancerous and healthy cells
H1
Unsupervised data
H2
color
H3
C1
C3
H4
C2
C4
#nuclei
Supervised data
#tails
class
H1
light
1
1
healthy
H2
dark
1
1
healthy
H3
light
1
2
healthy
H4
light
2
1
healthy
C1
dark
1
2
cancerous
C2
dark
2
1
cancerous
C3
light
2
2
cancerous
C4
dark
2
2
cancerous
19 November 2005
What to do? Primary tasks of KDD
Predictive mining tasks perform inference
on the current data in order to make
prediction or classification. （予測的マイニン
グの課題は，未知のデータの予測を目的として，現
在のデータに関する推論を行うことである）
Ex. If “color = dark” and “#nuclei =2”
then cancerous
Descriptive mining tasks characterize the
properties of the data in the database.
（記述的マイニングの課題は，データベース中のデ
ータの全般的特性を特徴付ける記述を与えることで
ある）
Ex. “Healthy cells mostly have one nuclei
while cancerous ones have two”
19 November 2005
What to find? Patterns and models
Patterns:
局所的な関係の要約 K417\K417\D2MS\D2MS.exe
A pattern is a low level summary of a relationship,
perhaps which holds only for a few records or for
only a few variables (local). パターンとは，任意の関係に
関する低次の要約であり，１つのパターンがカバーするデータ件
数あるいは変数の数は少数である（局所的）
Ex. If “color = dark” and “#nuclei =2” then cancerous
Models:データセットに関する大域的な記述
A model is a global description of a data set, a high
level population or large sample perspective モデルとは
，データセット，高次元の母集団，あるいは大多数のデータを視野
に入れたに関する大域的な記述である．
A model tells us about correlation between variables
(regression), about hierarchies of clusters (clustering),
a neural network, etc. モデルは，変数間の相関（回帰式）,ク
ラスタ階層（クラスタリング），神経回路網，などを用いて表現する．
19 November 2005
Classification/Prediction
Classification/prediction is the process of finding a set of
models (or patterns) that describe and distinguish data
classes or concepts, for the purpose of being able to use
the model to predict the class of objects whose class label
is unknown
（分類*/予測**とは，データのクラス/概念を説明し識別する一連のモデルを
見つけ出すプロセスであり，このモデルを用い，クラス情報が不明なデータの
分類を予測する *カテゴリデータの場合 **数値データの場合）
Decision trees
IF-THEN rules
Neural networks
Support vector machines
etc.
19 November 2005
Classification—A two-step process
Model construction
H1
H2
H3
H4
Model usage
Classification
Algorithms
Classifier
(model)
C1
training data
Unknown case
Cancerous?
C2
If
and
Then
color = dark
# tails = 2
cancerous cell
19 November 2005
Criteria for classification methods
Predictive accuracy（予測精度）: the ability
of the classifier to correctly predict unseen
data
Speed: refers to computation cost
Robustness（頑健性）: the ability of the
classifier to make correctly predictions
given noisy data or data with missing values
Scalability（拡張性）: the ability to construct
the classifier efficiently given large
amounts of data
Interpretability（解釈容易性）: the level of
understanding and insight that is provided
by the classifier
19 November 2005
Description
Description is the process of finding a set of patterns or
models that describe properties of data (essentially the
summary of data), for the purpose of understanding the
data.
（記述とは、データの理解を目的とし、そのデータを要約するような性質を表
現するパターンあるいはモデル集合を発見する過程である）
Clustering
Association mining
Summarization
Trend detection
etc.
19 November 2005
Criteria for description methods
Interestingness: overall measure
combining novelty, utility,
simplicity, reliability and validity
of discovered patterns/models.
Speed: refers to computation cost
Scalability（拡張性）: the ability to
construct the classifier efficiently
given large amounts of data
Interpretability（解釈容易性）: the
level of understanding and insight
that is provided by the classifier
19 November 2005
Outline
1. Why knowledge discovery and data mining?
2. Basic concepts of KDD
3. KDD techniques: classification, association,
clustering, text and Web mining
4. Challenges and trends in KDD
5. Case study in medicine data mining
19 November 2005
Decision tree learning
Learn to generate classifiers in the form of decision
trees from supervised data.
19 November 2005
Mining with decision trees
A decision tree is a flow-chart-like tree structure:
each internal node denotes a test on an attribute
each branch represents an outcome of the test
leaf nodes represent classes or class distributions
The top-most node in a tree is the root node
#nuclei
1
{H1, H2, H3, H4,
C1, C2, C3, C4}
2
{H1, H2, H3, C1}
{H4, C1, C3, C4}
color?
#tails
light
dark
{H1, H3}
H
1
H
{H2}
1
2
{H2, C2}
{H4}
{C1, C3, C4}
#tails
H
C
2
{C2}
C
19 November 2005
Decision tree induction (DTI)
Decision tree generation
consists of two phases
Tree construction（決定木構築）
Partition examples recursively
based on selected attributes
At start, all the training objects
are at the root
Tree pruning （構築した木の枝刈）
Identify and remove branches
that reflect noise or outliers
K417\K417\D2MS\D2MS.exe
Use of decision tree: Classify
unknown objects（新事例の分類）
Test the attribute values of the
object against the decision tree
19 November 2005
DTI general algorithm
Two steps: recursively generate the
tree (1-4), and prune the tree (5)
1.
At each node, choose the “best”
attribute by a given measure
for attribute selection
2.
Extend tree by adding new
branch for each value of the
attribute
3.
Sorting training examples to
leaf nodes
4.
If examples in a node belong to
one class Then Stop Else Repeat
steps 1-4 for leaf nodes
5.
Prune the tree to avoid overfitting
#nuclei
1
{H1, H2, H3, H4,
C1, C2, C3, C4}
2
{H1, H2, H3, C1}
{H4, C2, C3, C4}
color?
#tails
light
dark
{H1, H3}
H
1
H
{H2}
1
2
{H2, C2}
{H4}
{C1, C3, C4}
#tails
H
C
2
{C2}
C
19 November 2005
Measures for attribute selection
19 November 2005
Training data for concept “play-tennis”
Days
D1
D2
D3
D4
D5
D6
D7
D8
D9
D10
D11
D12
D13
D14
Outlook
sunny
sunny
overcast
rain
rain
rain
overcast
sunny
sunny
rain
sunny
overcast
overcast
rain
Temperature
hot
hot
hot
mild
cool
cool
cool
mild
cool
mild
mild
mild
hot
mild
Humidity
high
high
high
high
normal
normal
normal
high
normal
normal
normal
high
normal
high
Wind
weak
strong
weak
weak
weak
strong
strong
weak
weak
weak
strong
strong
weak
strong
Class
N
N
Y
Y
Y
N
Y
N
Y
Y
Y
Y
Y
N
A typical dataset in
machine learning
14 objects belonging to
two class {Y, N} are
observed on 4
properties.
Dom(Outlook) = {sunny,
overcast, rain}
Dom(Temperature) =
{hot, mild, cool}
Dom(humidity) = {high,
normal}
Dom(Wind) = {weak,
strong}
19 November 2005
A decision tree for playing tennis
temperature
cool
hot
{D5, D6, D7, D9}
{D1, D2, D3, D13}
outlook
sunny
rain
{D9}
{D5, D6}
yes
true
true
{D2}
{D7}
yes no
false
{D5}
{D6}
no
yes
sunny
high
false
sunny
{D8, D11}
normal
true false
{D11}
yes
o’cast
yes
rain
{D4, D10,D14}
yes
wind
{D3}
o’cast
{D12}
humidity
outlook
rain
outlook
{D1, D3, D13}
{D1, D3}
{D1}
no
{D4, D8, D10, D11,D12, D14}
wind
o’cast
wind
mild
humidity
high
{D8}
{D4, D14}
no
wind
true
{D14}
normal
{D10}
yes
false
{D4}
{D3}
null
yes
no
yes
19 November 2005
A simple decision tree for playing tennis
outlook
sunny
{D1, D2, D8
D9, D11}
o’cast
{D3, D7, D12, D13}
humidity
high
{D1, D2, D8}
no
rain
{D4, D5, D6, D10, D14}
wind
yes
normal
{D9, D10}
yes
true
{D6, D14}
false
{D4, D5, D10}
no
yes
This tree is much simpler as “outlook” is selected at the root.
How to select good attribute to split a decision node?
19 November 2005
Which attribute is the best?
The “playing-tennis” set S contains 9 positive objects
(+) and 5 negative objects (-), denote by [9+, 5-]
If attributes “humidity” and “wind” split S into subnodes with proportions of positive and negative
objects as below, which attribute is better?
[9+, 5-]
normal
[6+, 1-]
A1 = humidity
high
[3+, 4-]
[9+, 5-]
weak
[6+, 2-]
A2 = wind
strong
[4+, 2-]
19 November 2005
Entropy
Entropy characterizes the impurity (purity) of an
arbitrary collection of objects (不純度をあらわす).
S
is the collection of positive and negative objects
p
is the proportion of positive objects in S
p
is the proportion of negative objects in S
In
the play-tennis example, these numbers are 14, 9/14
and 5/14, respectively
Entropy is defined as follows
Entropy(S) = −p log2p − p log2p
19 November 2005
Entropy
The entropy function
relative to a Boolean
classification, as the
proportion p of
positive objects varies
between 0 and 1.
c
Entropy(S) pilog2pi
i1
19 November 2005
Example
From 14 examples of Play-Tennis, 9 positive and 5
negative objects (denote by [9+, 5-] )
Entropy([9+, 5-] ) = − (9/14)log2(9/14)
= 0.940
−
(5/14)log2(5/14)
Notice:
1. Entropy is 0 if all members of S belong to the same class. For
example, if all members are positive (p = 1), then p is 0, and
Entropy(S) = − 1. log2(1) − 0. log2 (0) = − 1.0 − 0 . log2 (0) = 0.
2. Entropy is 1 if the collection contains an equal number of
positive and negative examples. If these numbers are unequal, the
entropy is between 0 and 1.
19 November 2005
Information gain measures
the expected reduction in entropy
We define a measure, called information gain, of the
effectiveness of an attribute in classifying data. It is
the expected reduction in entropy caused by
partitioning the objects according to this attribute
Gain(S, A) Entropy(S)
vV alue(A )
Sv
S
Entropy(S v )
where Value(A) is the set of all possible values for
attribute A, and Sv is the subset of S for which A has
value v.
19 November 2005
Information gain measures
the expected reduction in entropy
Values(Wind) = {Weak, Strong}, S = [9+, 5-]
Sweak , the subnode with value “weak”, is [6+, 2-]
Sstrong , the subnode with value “strong”, is [3+, 3-]
Gain(S, Wind) = Entropy(S) −
v{Weak, Strong}
Sv
S
Entropy(S v )
= Entropy(S) − (8/14)Entropy(Sweak)
− (6/14)Entropy(SStrong)
= 0.940 − (8/14)0.811 − (6/14)1.00
= 0.048
19 November 2005
Which attribute is the best classifier?
S:[9+, 5-]
E = 0.940
S:[9+, 5-]
E = 0.940
Humidity
Wind
High
Normal
Weak
Strong
[3+, 4-]
E = 0.985
[6+, 1-]
E = 0.592
[6+, 2-]
E = 0.811
[3+, 3-]
E = 1.00
Gain(S, Humidity)
= .940 − (7/14).985 − (7/14).592
= .151
Gain(S, Wind)
= .940 − (8/14).811 − (6/14)1.00
= .048
19 November 2005
Information gain of all attributes
Gain (S, Outlook) = 0.246
Gain (S, Humidity) = 0.151
Gain (S, Wind) = 0.048
Gain (S, Temperature) = 0.029
19 November 2005
Next step in growing the decision tree
{D1, D2, ..., D14} [9+, 5-]
Outlook
Sunny
Overcast
Rain
{D1, D2, D8, D9, D11}
[2+, 3-]
{D3, D7, D12, D13}
[4+, 0-]
{D4, D5, D6, D10, D14}
[3+, 2-]
?
Yes
?
Which attribute should be tested here?
Ssunny = {D1, D2, D3, D9, D11}
Gain(Ssunny, Humidity) = .970 - (3/5)0.0 - (2/5)0.0 = 0.970
Gain(Ssunny, Temperature) = .970 - (2/5)0.0 - (2/5)1.0 - (1/5)0.0 = 0.570
Gain(Ssunny, Wind) = .970 - (2/5)1.0 - (3/5)0.918 = 0.019
19 November 2005
Stopping condition
1. Every attribute has already been included along
this path through the tree
2. The training objects associated with each leaf
node all have the same target attribute value
(i.e., their entropy is zero)
Notice: Algorithm ID3 uses Information Gain and C4.5,
its successor, uses Gain Ratio (a variant of Information
Gain)
19 November 2005
Over-fitting in decision trees
The generated tree may overfit the training data
Too
many branches, some may reflect anomalies due to noise or
outliers
Result
is in poor accuracy for unseen objects
Two approaches to avoid overfitting
Prepruning:
Halt tree construction early—do not split a node if
this would result in the goodness measure falling below a
threshold
Difficult
to choose an appropriate threshold
Postpruning:
Remove branches from a “fully grown” tree—get a
sequence of progressively pruned trees
Use
a set of data different from the training data to decide
which is the “best pruned tree”
19 November 2005
Converting a tree to rules
outlook
sunny
o’cast
humidity
high
no
rain
wind
yes
normal
yes
true
no
false
yes
IF
(Outlook = Sunny) and (Humidity = High)
THEN PlayTennis = No
IF
(Outlook = Sunny) and (Humidity = Normal)
THEN PlayTennis = Yes
19 November 2005
Attributes with many values
If attribute has many values (e.g., days of the month), ID3
will select it
C4.5 uses GainRatio instead
GainRatio(S, A)
Gain(S, A)
SplitInformation(S, A)
c
SplitInformation(S, A)
i1
Si
S
log2
Si
S
where Si is subset of S with A has value v i
19 November 2005
Bayesian classification
Learning statistical classifiers from supervised data
basing on Bayes theorem and assumptions on
independence/dependence of the data.
19 November 2005
What are Bayesian classification?
Bayesian classifiers are statistical classifiers.
Bayesian classification is based on Bayes theorem.
Naïve Bayesian classifiers assume that the effect of
an attribute value on a given class is independent of
the values of the other attributes.
Bayesian belief networks are graphical models
allow the representation of dependencies among
subsets of attributes.
19 November 2005
Bayes theorem
Let X be an object whose class label is
unknown
Let H be some hypothesis, such as that X
belongs to class C.
For classification, we want to determine the
posterior probability P(H|X), of H conditioned
on X.
Example: Data object consists of fruits,
described by color and shape
Suppose X is red and round, and H is the
hypothesis that X is an apple.
P(H|X) reflects our confidence that X is an
apple given that we have seen that X is red
and round.
P(apple | red and round) = ?
19 November 2005
Bayes theorem
In contrast, P(H) is the prior probability of H.
In our example, P(H) is the probability that
any given data object is an apple, regardless
of how the data sample looks (independent of
X).
P(X|H) is the likelihood of X given H, that is
the probability that X is red and round given
that we know that it is true that X is an apple.
P(X), P(H), and P(X|H) may be estimated from
the given data. Bayes theorem allows us to
calculate P(H|X)
P( H | X )
P( X | H ) P( H )
P( X )
P(apple | red round )
P(red round | apple) P(apple)
P(red round )
19 November 2005
Naïve Bayesian classification
Suppose X = (x1, x2, …, xn), attributes A1, A2, …, An
There are m classes C1, C2, …, Cm
P(Ci|X) denotes probability that X is classified to class Ci.
Example:
P(class = N | outlook=sunny, temperature=hot, humidity=high, wind=strong)
Idea: assign to object X the class label Ci such that
P(Ci|X) is maximal, i.e., P(Ci|X) > P(Cj|X), j, ij.
Ci is called the maximum posterior hypothesis.
19 November 2005
Estimating a posteriori probabilities
Bayes theorem:
P(X | Ci )P(C i )
P(Ci | X)
P(X)
P(X) is constant. Only need maximize P(X|Ci) P(Ci)
Ci such that P(Ci |X) is maximum =
Ci such that P(X| Ci)·P(Ci) is maximum
If prior probability is unknown, commonly assumed that
P(C1) = P(C2) = … = P(Cm), and we would maximize P(X|Ci)
Otherwise, P(Ci) = relative frequency of class Ci = Si/S
Problem: computing P(X|Ci) is unfeasible!
19 November 2005
Naïve Bayesian classification
Naïve assumption: We have P(X|Ci) = P(x1,…,xn|Ci), if attributes are
independent then P(X|Ci) = P(x1|Ci) x … x P(xn|Ci)
If Ak is categorical then P(xk|Ci) = Sik/Si where Sik is the number of
training objects of class Ci having the value xk for Ak, and Si is the
number of training objects belonging to Ci.
If Ak is continuous then the attribute is typically assumed to have a
Gaussian distribution, so that
1
P( xk | Ci )
e
2 Ci
( xk Ci ) 2
2 C2i
To classify an unknown object X, P(X|Ci)P(Ci) is evaluated for each
class Ci. X is then assigned to the class Ci if and only if
P(X|Ci)P(Ci) > P(X|Cj)P(Cj), for 1 j m, j i.
19 November 2005
Play-tennis example: estimating P(xk|Ci)
Days
D1
D2
D3
D4
D5
D6
D7
D8
D9
D10
D11
D12
D13
D14
Outlook
sunny
sunny
overcast
rain
rain
rain
overcast
sunny
sunny
rain
sunny
overcast
overcast
rain
Temperature
hot
hot
hot
mild
cool
cool
cool
mild
cool
mild
mild
mild
hot
mild
Humidity
high
high
high
high
normal
normal
normal
high
normal
normal
normal
high
normal
high
Wind
weak
strong
weak
weak
weak
strong
strong
weak
weak
weak
strong
strong
weak
strong
Class
N
N
Y
Y
Y
N
Y
N
Y
Y
Y
Y
Y
N
outlook
P(sunny|Y) = 2/9
P(sunny|N) = 3/5
P(overcast|Y) = 4/9
P(overcast|N) = 0
P(rain|Y) = 3/9
P(rain|N) = 2/5
temperature
P(hot|Y) = 2/9
P(hot|N) = 2/5
P(mild|Y) = 4/9
P(mild|N) = 2/5
P(cool|Y) = 3/9
P(cool|N) = 1/5
humidity
P(high|Y) = 3/9
P(high|N) = 4/5
P(Y) = 9/14
P(normal|Y) = 6/9
P(normal|N) = 2/5
P(N) = 5/14
windy
P(strong|Y) = 3/9
P(strong|N) = 3/5
P(weak|Y) = 6/9
P(weak|N) = 2/5
19 November 2005
Play-tennis example: classifying X
outlook
P(sunny|Y) = 2/9
P(sunny|N) = 3/5
P(overcast|Y) = 4/9
P(overcast|N) = 0
P(rain|Y) = 3/9
P(rain|N) = 2/5
An unseen object
X = <rain, hot, high, weak>
P(X|Y)P(Y) =
P(rain|Y)P(hot|Y)
P(high|Y)P(weak|Y)P(Y)
= 3/9 2/9 3/9 6/9 9/14
= 0.010582
P(X|N)P(N) =
P(rain|N) P(hot|N)
P(high|N) P(weak|N) P(N)
= 2/5 2/5 4/5 2/5 5/14
= 0.018286
Object X is classified in class N
(don’t play)
temperature
P(hot|Y) = 2/9
P(hot|N) = 2/5
P(mild|Y) = 4/9
P(mild|N) = 2/5
P(cool|Y) = 3/9
P(cool|N) = 1/5
humidity
P(high|Y) = 3/9
P(high|N) = 4/5
P(normal|Y) = 6/9
P(normal|N) = 2/5
windy
P(true|Y) = 3/9
P(true|N) = 3/5
P(false|Y) = 6/9
P(false|N) = 2/5
19 November 2005
The independence hypothesis
It
makes computation possible
It
yields optimal classifiers when independence satisfied
But
it is seldom satisfied in practice, as attributes
(variables) are often correlated
Attempts
to overcome this limitation, among others:
Bayesian networks, that combine Bayesian reasoning with
causal relationships between attributes
Decision trees, that reason on one attribute at the time,
considering most important attributes first
19 November 2005
Other classification methods
Neural Networks
Instance-based Classification
Genetic Algorithms
Rough Set Approach
Statistical Approaches
Support Vector Machines
etc.
19 November 2005
Mining with neural networks
H1
H2
color = dark
H3
H4
# nuclei = 1
C1
C2
# tails = 2
C3
C4
Healthy
Cancerous
19 November 2005
Mining with neural networks
Advantages
prediction accuracy is generally high
robust, works when training examples contain errors
output may be discrete, real-valued, or a vector of
several discrete or real-valued attributes
fast evaluation of the learned target function
Criticism
long training time
difficult to understand the learned function (weights)
not easy to incorporate domain knowledge
19 November 2005
Instance-based classification
Instance-based classification
Using most similarity individual instances known in
the past to classify a new instance
Typical approaches
k-nearest neighbor approach
Locally weighted regression
Instances as points in an Euclidean space
Constructs local approximation
Case-based reasoning
Uses symbolic representations and knowledge-based
inference
19 November 2005
Genetics algorithms (GA)
EVOLUTION
PROBLEM
SOLVING
Generate initial population
do
Calculate the fitness of each member
Environment
Individual
Fitness
Problem
Candidate
Solution
Quality
Fitness chances for
survival and reproduction
Quality chance for
seeding new solutions
..\form\CSD\Cryst.exe
// simulate another generation
do
1. Select parents from current population
2. Perform crossover to add offspring
to the new population
while new population is not full
1. Merge new population into the current
population
2. Mutate current population
while not converged
19 November 2005
Rough set approach
Rough sets are used to approximately or “roughly”
define equivalent classes
X
A rough set for a class C is approximated by two sets:
A lower approximation (certain to be in C)
A upper approximation (possible to be in C)
Finding the minimal subsets (reducts) of attributes,
dependencies in data, rules, etc.
U { ,
Rough sets and Data
Mining, T.Y. Lin, N.
Cercone (eds.), Kluwer
Academic Pub., 1997)
Rough Sets in Knowledge
Discovery, L. Polkowski, A.
Skowron (eds.), PhysicaVerlag, 1998.
,
,
,
Equivalence classes
,} R {color , shape}
Eq. class / color {{ , , }, { , }}
Eq. class / shape {{ , }, { , }, { }}
Eq. class {{ , }, { }, { }, { }}
X {
,
}, X * {
*
X
}, {
,
,
}
19 November 2005
Association rule mining
Description learning that aims to find all
possible associations from data.
19 November 2005
Market basket analysis
Analyzes customer buying habits by finding associations
between the different items that customers place in their
“shopping baskets”
Helps develop marketing strategies by gaining insight into
which items are frequently purchased together by customers
How often people
buy onigiri and
beer together?
19 November 2005
Rule measures: support and confidence
Customer buys both
Customer
buys beer
Customer
buys onigiri
Transaction ID Items Bought
2000
1000
4000
5000
A,B,C
A,C
A,D
B,E,F
Association rule X
Y
support s = probability that a
transaction contains
X and Y
confidence c = conditional
probability that a transaction
having X also contains Y
If minimum support 50%,
minimum confidence 50%:
A C (s=50%, c=66.6%)
C A (s=50%, c=100%)
19 November 2005
Basic concepts
The rule X Y holds in the transaction set D with confidence c
if c% of transactions in D that contain X also contain Y.
The rule X Y has support s in the transaction set D if s% of
transactions in D contain X Y.
Confidence denotes the strength of implication and support
indicates the frequencies of the occurring patterns in the rule
support(X Y) P(X and Y)
# trans. containing both X and Y
# all trans. in the database
# trans. containing X and Y
confidence (X Y) P(Y | X) P(Xand Y)/P(X)
# trans. containing X
19 November 2005
Association mining: Apriori algorithm
It is composed of two steps:
1.
Find all frequent itemsets: By definition, each of
these itemsets will occur at least as frequently as
a pre-determined minimum support count
2.
Generate strong association rules from the
frequent itemsets: By definition, these rules must
satisfy minimum support and minimum confidence
19 November 2005
Association mining: Apriori principle
Transaction ID
2000
1000
4000
5000
For rule A C
Items Bought
A,B,C
A,C
A,D
B,E,F
Min. support 50%
Min. confidence 50%
Frequent Itemset Support
{A}
75%
{B}
50%
{C}
50%
{A,C}
50%
support = support({A and C}) = 50%
confidence = support({A and C})/support({A}) = 66.6%
The Apriori principle:
Any subset of a frequent itemset must be frequent
(if an itemset is not frequent, its supersets are not)
19 November 2005
The Apriori algorithm
1.
Find the frequent itemsets: the sets of items that have
support higher than the minimum support
A subset of a frequent itemset must also be a frequent itemset
i.e., if {AB} is a frequent itemset, both {A} and {B} should be a
frequent itemset
Iteratively find frequent itemsets Lk with cardinality from 1 to k
(k-itemset) by from candidate itemsets Ck (Lk Ck)
C1 … Li-1 Ci Li Ci+1 … Lk
2.
Use the frequent itemsets to generate association
rules.
19 November 2005
Apriori algorithm: Finding frequent itemsets
using candidate generation
Mining frequent itemsets for Boolean association rules
It employs an iterative approach to level-wise search
where k-itemsets are used to explore (k+1)-itemsets
First, the set of frequent 1-itemsets L1 is found.
L1 is used to find the set of 2-itemsets L2,
then L2 is used to find L3,
and so on, until no more frequent k-itemsets can be found.
To improve the efficiency of the level-wise generation of
frequent itemsets, the important Apriori property is
used to reduce the search space.
19 November 2005
Apriori algorithm: Finding frequent itemsets
using candidate generation
If an itemset l does not satisfy the minimum support
threshold min_sup, then l is not frequent, that is,
P(l)<min_sup
If an item A is added to the itemset l, then the resulting
itemset (i.e., l A) cannot occur more frequently than l.
Therefore, l A is not frequent either, that is,
P(l A) < min_sup
This is the anti-monotone property:
if a set cannot pass a test, all of its
supersets will fail the same test as well
19 November 2005
Apriori algorithm: join step to find Ck
To join Lk-1 with itself to generate a set Ck of candidates kitemsets, and to use it to find Lk
Given l1, l2 Lk-1, the notation li[j] refers to the jth item in li
Apriori assumes that items within a transaction or itemset are
sorted in lexicographic order
The join, Lk-1 Lk-1, is performed where members of Lk-1 are
joinable if their first (k-2) items are in common. That is, l1, l2
Lk-1 are joined if
(l1[1] = l2[1]) (l1[2]= l2[2])… (l1[k-2]= l2[k-2]) (l1[k-1]<l2[k-1])
The resulting itemset by joining l1 and l2 is
l1[1] l1[2] … l1[k-2] l1[k-1] l2[k-1]
19 November 2005
Apriori algorithm: The prune step
Ck is a superset of Lk, i.e. its members may or may not be
frequent, but all frequent k-itemsets are included in Ck (Lk Ck)
A scan of the database to determine the count of each candidate
in Ck would result in the determination of Lk (all candidates
having a count no less than the min_sup_count are frequent and
therefore belong to Lk)
Ck can be huge. To reduce the size of Ck: use apriori property
Any (k-1)-itemset that is not frequent cannot be a subset of a
frequent k-itemset
Hence, if any (k-1)-subset of a candidate k-itemset is not in Lk-1,
then the candidate cannot be frequent either and so can be
removed from Ck (can be tested quickly by a hash tree of frequent
itemsets)
19 November 2005
Example (min_sup_count = 2)
Transactional data
TID
List of items_IDs
T100
T200
T300
T400
T500
T600
T700
T800
T900
I1,
I2,
I2,
I1,
I1,
I2,
I1,
I1,
I1,
I2,
I4
I3
I2,
I3
I3
I3
I2,
I2,
I5
I4
I3, I5
I3
Scan D for
count of
each
candidate
Compare
candidate
support count
with minimum
support count
C1
L1
Itemset Sup.Count
Itemset Sup.Count
{I1}
{I2}
{I3}
{I4}
{I5}
6
7
6
2
2
{I1}
{I2}
{I3}
{I4}
{I5}
6
7
6
2
2
19 November 2005
Example (min_sup_count = 2)
C2
Itemset
Generate
{I1, I2}
candidates {I1, I3}
C2 from L1
{I1, I4}
using
{I1, I5}
Apriori
{I2, I3}
principle
{I2, I4}
{I2, I5}
{I3, I4}
{I3, I5}
{I4, I5}
C2
Scan D for
count of
each
candidate
Generate
candidates
C3 from L2
Itemset
using
Apriori
{I1, I2, I3}
principle
{I1, I2, I5}
Compare
candidate
support
count with
minimum
support
count
Itemset S.count
{I1, I2}
4
{I1, I3}
4
{I1, I4}
1
{I1, I5}
2
{I2, I3}
4
{I2, I4}
2
{I2, I5}
2
{I3, I4}
0
{I3, I5}
1
{I4, I5}
0
Scan D for
count of
each
candidate
C3
Itemset
Sc
{I1, I2, I3} 2
{I1, I2, I5} 2
Compare
candidate
support
count with
minimum
support
count
L2
Itemset S.count
{I1, I2}
4
{I1, I3}
4
{I1, I5}
2
{I2, I3}
4
{I2, I4}
2
{I2, I5}
2
L3
Itemset
Sc
{I1, I2, I3} 2
{I1, I2, I5} 2
19 November 2005
Example (min_sup_count = 2)
1.
Join C3 = L2 L2 =
{{I1, I2}, {I1, I3}, {I1, I5}, {I2, I3}, {I2, I4}, {I2, I5}}
{{I1, I2}, {I1, I3}, {I1, I5}, {I2, I3}, {I2, I4}, {I2, I5}}
= {{I1, I2, I3}, {I1, I2, I5}, {I1, I3, I5}, {I2, I3, I4}, {I2, I3, I5}, {I2, I4, I5}}
2. Prune using the Apriori property: all nonempty subsets of a frequent
itemset must also be frequent. Any candidate has a subset that is not
frequent?
2-item subsets of {I1, I2, I3} are {I1, I2}, {I1, I3}, {I2, I3} which are all
members of L2. Therefore, keep {I1, I2, I3} in C3
2-item subsets of {I1, I2, I5} are {I1, I2}, {I1, I5}, {I2, I5} which are all
members of L2. Therefore, keep {I1, I2, I5} in C3
2-item subsets of {I1, I3, I5} are {I1, I3}, {I1, I5}, {I3, I5}. Subset {I3, I5} is not
a member of L2. Therefore, remove {I1, I3, I5} from C3, and so on
3. Therefore, C3 = {{I1, I2, I3}, {I1, I2, I5}}
19 November 2005
Cluster analysis
Description learning that aims to detect and
groups of similar objects from unsupervised
data.
19 November 2005
What is cluster analysis?
A cluster is a collection of
data objects satisfying
Objects
in this cluster are
similar to one another
Objects
in this cluster are
dissimilar to the objects in
other clusters
The process of grouping
objects into clusters is
called clustering
19 November 2005
Mining with clustering
Clustering analyzes data objects
without consulting a known class
label.
The objects are clustered or
grouped based on the principle
of maximizing the between-class
similarity and minimizing the
within-class similarity
Partition-based clustering for large
sets of numerical data.
Hierarchical clustering with at
least O(n2) time complexity seems
not be suitable for very large
datasets
19 November 2005
What is good clustering?
A good clustering method will produce high quality
clusters with
high
intra-class similarity (within a class)
low inter-class similarity (between classes)
The quality of clustering basically depends on the
similarity measure and the cluster representative
used by the method
New forms of clustering require different criteria
of quality.
19 November 2005
Clustering in different fields
Statistics: since many years, focus on distancebased clustering (S-Plus, SPSS, SAS)
Machine learning: unsupervised learning. In
conceptual clustering, a group of objects forms a
class only if it is described by a concept
KDD: Efficient and effective clustering of large
databases: scalability, complex shapes and types of
data, high dimensional clustering, mixed numerical
and categorical data
19 November 2005
Typical requirements of clustering
Scalability
Ability to deal with different types of attributes
Discovery of clusters with arbitrary shape
Minimal requirements for domain knowledge to
determine input parameters
Ability to deal with noisy data
Insensitivity to the order of input records
High dimensionality
Constraint-based clustering
Interpretability and usability
19 November 2005
Clustering methods in KDD
Partitioning Methods
Hierarchical Methods
Density-Based Methods
Grid-Based Methods
Model-Based Methods
19 November 2005
Partitioning methods
Given n objects and k as number of clusters to form.
A partitioning algorithm organizes the objects into
a partition of k clusters
The clusters are formed to optimize an objective
partitioning criterion so that the objects within a
cluster are “similar” , whereas the objects of
different classes are “dissimilar”
C:\STAT\STA_WIN.EXE
19 November 2005
K-means algorithm (K=2)
Two
centers
selected
randomly
from n
objects
Reform
two new
clusters
Calculate
two new
centers
1
Form two
clusters by
assigning
each object to
its nearest
center
2
Calculate
two new
centers
3
Repeat
step 2
and 3
until
the
stopping
conditions
hold
4
19 November 2005
Partitioning methods
The k-means algorithm is sensitive to outliers
The k-medoids method uses medoid (the most
centrally located object in a cluster)
The EM (Expectation Maximization) algorithm:
assigns to a cluster according to a weight
representing the probability of membership.
PAM (Partitioning Around Medoids)
From k-Medoids to CLARA (Clustering LARge
Applications)
From CLARA to CLARANS (Clustering LARge
Applications based on RANdomized Search)
19 November 2005
Hierarchical methods
Partition Q is nested into partition P if every
component of Q is a subset of a component
of P.
P {x1 , x4 , x7 , x9 , x10},{x2 , x8},{x3 , x5 , x6 }
Q {x1 , x4 , x9 },{x7 , x10},{x2 , x8},{x5},{x3 , x6 }
A hierarchical clustering is a sequence of
partitions in which each partition is nested
into the next (previous) partition in the
sequence.
C:\Documents and Settings\Ho Tu Bao\Desktop\STATISTICA.lnk
19 November 2005
Hierarchical clustering: chameleon
Chameleon: A Hierarchical Clustering
Algorithm Using Dynamic Modeling.
Clusters are merged if the
interconnectivity and closeness
between them are highly related to the
internal interconnectivity and
closeness of objects within the
clusters.
19 November 2005
Density-based methods
Typically regards clusters as
dense regions of objects in
the data space that are
separated by regions of low
density
DBSCAN: Based on
Connected Regions with
Sufficiently High Density
(Nearest Neighbor
Estimation)
DBSCAN result for DS2 with MinPts at 4
and Eps at (a) 5.0, (b) 3.5 and (c) 3.0
DENCLUE: Based on Density
Distribution Functions
(Kernel Estimation)
19 November 2005
Text mining
Finding unknown useful information from huge
collection of textual data.
19 November 2005
What is text mining?
“The non trivial extraction of implicit, previously
unknown, and potentially useful information from (large
amount of) textual data”.
“An exploration and analysis of textual (natural
language) data by automatic and semi automatic means
to discover new knowledge”.
A branch of data mining,
targets discovering and
extracting knowledge
from text documents
19 November 2005
テキストマイニング: 研究事例
生物医学文献タイトルからの科学的根拠の抽出 (Swanson &
Smalheiser, 1997)
“stress is associated with migraines” “ストレスは片頭痛を伴う”
“stress can lead to loss of magnesium”
“ストレスはマグネシウム損失の原因となる”
“calcium channel blockers prevent some migraines”
“カルシウム拮抗薬は片頭痛を予防することがある”
“magnesium is a natural calcium channel blocker”
“マグネシウムは天然のカルシウム拮抗薬である”
抜粋した文の断片を人間の医学専門知識を使って組合せ，
文献にない新しい仮説を導き出す
Magnesium deficiency may play a role in some kinds of
migraine headache
マグネシウムはある種の片頭痛に関与するらしい
19 November 2005
Motivation for text mining
Approximately 80% of the world’s data is held in
unstructured formats (source: Oracle Corporation)
Information intensive business processes demand that
we transcend from simple document retrieval to
“knowledge” discovery.
20%
80%
Structured Numerical or Coded
Information
Unstructured or Semi-structured
Information
19 November 2005
Disciplines that influence text mining
Computational linguistics (NLP)
Information extraction
Information retrieval
Web mining
Regular data mining
Text Mining = Data Mining (applied to text data)
+ Language Engineering
19 November 2005
Computational linguistics
Goal: automated language understanding
this isn’t possible
instead, go for subgoals of text analysis, e.g.,
word sense disambiguation
phrase recognition
semantic associations
Common current approach (trend):
statistical analyses over very large text collections
Consider a word like "string" or "rope." No computer today has any way to understand
what those things mean. For example, you can pull something with a string, but you
cannot push anything. You can tie a package with string, or fly a kite, but you cannot
eat a string or make it into a balloon. In a few minutes, any young child could tell you a
hundred ways to use a string or not to use a string but no computer knows any of
this.
19 November 2005
Computational linguistics
Lexical / Morphological Analysis
Tagging
text
Shallow parsing
The woman will give Mary a book
POS tagging
Chunking
Syntactic Analysis
Grammatical Relation Finding
The/Det woman/NN will/MD give/VB
Mary/NNP a/Det book/NN
chunking
Named Entity Recognition
Word Sense Disambiguation
[The/Det woman/NN]NP [will/MD give/VB]VP
[Mary/NNP]NP [a/Det book/NN]NP
Semantic Analysis
Reference Resolution
relation finding
subject
[The woman] [will give] [Mary] [a book]
Discourse Analysis
meaning
i-object
object
19 November 2005
Archeology of computational linguistics
1990s–2000s:
Statistical learning
algorithms, evaluation, corpora
Trainable parsers
1980s:
Standard resources and tasks
Penn Treebank, WordNet, MUC
Trainable FSMs
1970s:
Kernel (vector) spaces
clustering, information retrieval (IR)
1960s:
Representation Transformation
Finite state machines (FSM) and Augmented
transition networks (ATNs)
1960s:
Representation—beyond the word level
lexical features, tree structures, networks
19 November 2005
Information retrieval
Given:
Documents
source
A
source of textual
documents
A
user query (text
based)
Find:
Query
E.g. migraines
causes
A set (ranked) of
documents that are
relevant to the query
IR
System
Document
Ranked
Documents
Document
Document
19 November 2005
Evaluation measures
retrieved
relevant
retrieved relevant
retrieved relevant
, recall
retrieved
relevant
( 2 1) PR
F
,
P precision , R recall
2P R
precision
19 November 2005
Information extraction vs. information
retrieval: Finding “things” but not “pages”
the process of
extracting text
segments of semistructured or free text
to fill data slots in a
predefined template
foodscience.com-Job2
JobTitle: Ice Cream Guru
Employer: foodscience.com
JobCategory: Travel/Hospitality
JobFunction: Food Services
JobLocation: Upper Midwest
Contact Phone: 800-488-2611
DateExtracted: January 8, 2004
Source: www.foodscience.com/jobs_midwest.html
OtherCompanyJobs: foodscience.com-Job1
19 November 2005
What is information extraction?
Given:
A source of textual documents
A well defined limited query
(text based)
Find:
Sentences with relevant
information (i.e., identify specific
semantics elements such as
entities, properties, relations)
Extract the relevant information
and ignore non-relevant information
(important!)
Link related information and output
in a predetermined format
19 November 2005
Example: template extraction
Salvadoran President-elect Alfredo Cristiania condemned the terrorist killing of
Attorney General Roberto Garcia Alvarado and accused the Farabundo Marti
Natioanl Liberation Front (FMLN) of the crime …. Garcia Alvarado, 56, was killed
when a bomb places by urban guerillas on his vehicle explored as it came to a halt
at an intersection in downtown San Salvador … According to the police and Garcia
Alvarado’s driver, who escaped unscathed, the attorney general was traveling with
two bodyguards. One of them was injured.
Incident: Date
Incident: Location
Incident: Type
Perpetrator: Individual ID
Perpetrator: Organization ID
Human target: Name
19 Apr 89
El Salvador: San Salvador
Bombing
“urban guerillas”
“FMLN”
“Roberto Garcia Alvarado”
19 November 2005
Zipf’s “law” and its consequence
One of curiosities in text
analysis and mining (1935)
The product of the frequency
of words (f) and their rank (r)
is approximately constant
f = C * 1/r
C N/10
Rank = order of words’
frequency of occurrence
Always a few very frequent tokens
that are not good discriminators.
Called “stop words” in Information
Retrieval
Usually correspond to linguistic
notion of “closed-class” words
English examples: to, from, on, and,
the, ...
Grammatical classes that don’t take
on new members.
Typically
A few very common words
A middling number of medium
frequency words
A large number of very infrequent
words
Medium frequency words most
descriptive
19 November 2005
Text mining process
Text preprocessing
Syntactic/Semantic
analysis
Features
Generation
Bag of words
Features
text
- Term frequency
- Document frequency
- Term proximity
- Document length
- etc.
Selection
Simple counting
Statistics
Text/Data Mining
Classification
Clustering
Summarization
Etc.
Analyzing results
19 November 2005
Typical issues and techniques
Text Categorization (text classification)
Text Clustering
Text Summarization
Trend Detection
Relationship Analysis
Information Extraction
Question-Answering
Text Visualization
etc.
19 November 2005
Text categorization (classification)
Task: Assignment of one or more labels from a
defined set to a document
Example category set
MeSH medical hierarchy
JAIST’s library, Library of Congress subject headings
Idea: Content vs. External Meta-Data
Techniques: Supervised classification
pre-
Decision trees
Naïve Bayesian classification
Support Vector Machines
etc.
19 November 2005
Text clustering
Task: Detecting topics within a document
collection, assigning documents to those topics,
and labeling these topic clusters
Scatter/Gather clustering
Cluster sets of documents into general “themes”, like
a table of contents
Display the contents of the clusters by showing
typical terms and typical titles
User chooses subsets of the clusters and re-clusters
the documents within
Resulting new groups have different “themes”
Techniques: Different clustering techniques
(similarity between texts, texts and word
densities, etc.)
19 November 2005
Text summarization
A text is entered into the computer and a
summarized text is returned, which is a
non redundant extract from the original
text.
A process of text summarization
Sentence extraction: Find a set of
important sentences that covers the
gist meaning of the text document
Sentence reduction: Convert a long
sentence to a short one without losing the
meaning
Sentence combination: Combine
sentences to make a text.
19 November 2005
Emerging trend detection
Task: Detecting topic areas that are growing in interest and utility
over time (emerging trends)
Example:
Year
Number of
documents
1994
3
“Find sales trends by product and correlate
with occurrences of company name in
business news articles”
1995
1
1996
8
1997
10
KDD’03 challenges: From arXiv (since 1991)
500,000 articles on High Energy Particle
Physics, predict, says, # citation in a period of
a given articles, etc.
1998
170
1999
371
INSPEC®[INS] database search
on keyword “XML”
COE project: Can we detect emerging trends in materials science,
information science, and biology?
19 November 2005
Question Answering
Task: Give answer to question
(document retrieval: find documents relevant to query)
Example:
Who
invented the telephone?
Alexander Graham Bell
When
was the telephone invented?
1876
(Buchholz & Daelemans, 2001)
Imagine how to automatically answer such a
question?
19 November 2005
Text visualization
Network Maps
http://www.lexiquest.com
Landscapes
http://www.aurigin.com
19 November 2005
Web mining
Finding unknown useful information from the
World Wide Web.
19 November 2005
Data Mining and Web Mining
Data mining turns data into knowledge.
Web mining is to apply data mining
techniques to extract and uncover
knowledge from web documents and
services.
19 November 2005
WWW specifics
Web:
A
huge, widely-distributed
Highly
heterogeneous
Semi-structured
Hypertext/hypermedia
Interconnected
information repository
Web is a huge collection of documents plus
Hyper-link
Access
information
and usage information
19 November 2005
Web user tasks
Finding relevant information
The user usually uses a simple keyword query and receive a list of ranked
pages
Current problem: low precision (irrelevance of search result), and low recall
(inability to index all the information on the Web)
Creating new knowledge over the existing data
Personalizing the information
Want to extract knowledge from Web data (assuming to have it)
People differ in the contents and presentations they prefer while
interacting with the Web
Learning about consumers and individual users
Knowing what the customers do and want
19 November 2005
Web mining taxonomy
Web Content Mining
Discovery of information from Web contents (various types of
data such as textual, image, audio, video, hyperlinks, etc.)
Web Structure Mining
Discovery of the model underlying the link structures of the
Web. The model is based on the topology of the hyperlinks
with or without the description of the links.
Web Usage Mining
Discovery of information from web users’ sessions and
behaviors (secondary data derived from the interactions of
the users while interacting with the Web).
19 November 2005
Web mining taxonomy
Webmining
Mining
Web
Web Content
Mining
Web Page
Content Mining
Web Structure
Mining
Search Result
Mining
Web Usage
Mining
General Access
Pattern Tracking
Customized
Usage Tracking
19 November 2005
Web mining taxonomy
Web Mining
Web
mining
Web Content
Mining
Web Structure
Mining
Web Page Content Mining
Web Page Summarization
• WebLog (Lakshmanan et.al. 1996)
• WebSQL(Mendelzon et.al. 1998) …: Web
structuring query languages; Can identify
information within given web pages
• Ahoy! (Etzioni et.al. 1997): Uses heuristics to
distinguish personal home pages from other
web pages
• ShopBot (Etzioni et.al. 1997): Looks for
product prices within web pages
Search Result
Mining
Web Usage
Mining
General Access
Pattern Tracking
Customized
Usage Tracking
19 November 2005
Web mining taxonomy
Web Mining
Web
mining
Web Content
Mining
Web Structure
Mining
Web Usage
Mining
Web Page
Content Mining
Search Result Mining
Search Engine Result Summarization
Clustering Search Result (Leouski and
Croft, 1996; Zamir and Etzioni, 1997):
Categorizes documents using phrases in
titles and snippets
General Access
Pattern Tracking
Customized
Usage Tracking
19 November 2005
Web mining taxonomy
Web Mining
Web
mining
Web Content
Mining
Search Result
Mining
Web Page
Content Mining
Web Structure Mining
Using Links
• PageRank (Brin and Page, 1996), HITS (Kleinberg,
1996)
• CLEVER (Chakrabarti et al., 1998)
Use interconnections between web pages to give weight
to pages
Web Communities
• Communities crawling (Kumar et al, 1999), etc.
Using Generalization
• MLDB (1994), VWV (1998)
Uses a multi-level database representation of the Web.
Counters (popularity) and link lists are used for capturing
structure.
Web Usage
Mining
General Access
Pattern Tracking
Customized
Usage Tracking
19 November 2005
Web mining taxonomy
Web Mining
Web
mining
Web Content
Mining
Web Page
Content Mining
Search Result
Mining
Web Structure
Mining
Web Usage
Mining
General Access Pattern Tracking
Customized
Usage Tracking
• Web Log Mining (Zaïane, Xin and Han, 1998)
Uses KDD techniques to understand general
access patterns and trends.
19 November 2005
Web mining taxonomy
Web mining
Web Content
Mining
Web Page
Content Mining
Search Result
Mining
Web Structure
Mining
General Access
Pattern Tracking
Web Usage
Mining
Customized Usage Tracking
• Adaptive Sites (Perkowitz and Etzioni, 1997)
Analyses access patterns of each user at a time.
Web site restructures itself automatically by
learning from user access patterns.
19 November 2005
Web mining process
1.
Resource finding: The task of
retrieving intended Web
documents
2.
Information selection and
pre-processing: Automatically
selecting and preprocessing
specific information from
retrieved Web resources
3.
Generalization: automatically
discovers general patterns at
individual Web sites as well as
across multiple sites
4.
Analysis: validation and/or
interpretation of the mined
patterns
19 November 2005
Data and knowledge visualization
Sunday
11-12 PM
Fisheye view
Tree map
Hyperbolic tree
Lunch time
MagicLens
Cone tree
19 November 2005
KDD products and tools
Salford Systems
SPSS
Silicon Graphics
SAS
IBM
RuleQuest Research (C4.5)
19 November 2005
Outline
Why knowledge discovery and data mining?
Basic concepts of KDD
KDD techniques: classification, association,
clustering, text and Web mining
Challenges and trends in KDD
Case study in medicine data mining
19 November 2005
Challenges of KDD
Large data sets (106-1012 bytes) and high
dimensionality (102-103 attributes)
[Problems: efficiency, scalability?]
Different types of data in different forms
(mixed numeric, symbolic, text, image, voice,…)
[Problems: quality, effectiveness?]
Data and knowledge are changing
Human-Computer Interaction and Visualization
19 November 2005
Large datasets and high dimensionality
H1
H2
H3
H4
C1
C2
C3
C4
3 attributes each has 2 values:
#instances = 23 = 8
#patterns =27
What if #attributes increases?
Size of instance space and pattern
space increased exponentially
p attributes each has d values,
size of instance space is dp
38 attributes each has 10 values:
#instances = 1038
19 November 2005
Possible solutions
Scalable and efficient algorithms
(scalable: given an amount of main memory, its
runtime increases linearly with the number of input
instances)
Sampling (instance selection)
Dimensionality reduction (feature selection)
Approximation methods
Massively parallel processing
Integration of machine learning and database
management
19 November 2005
Numerical vs. symbolic data
Combinatorial search in hypothesis spaces (machine learning)
Attribute
Numerical
No structure
Ordinal
structure
Ring
structure
Age,
Temperature,
Taste,
Income,
Length
Symbolic
Places,
Color
Nominal
(categorical)
Rank,
Resemblance
Ordinal
Measurable
Often matrix-based computation (multivariate data analysis)
19 November 2005
Mining with decision trees
Attribute selection
Pruning trees
From trees to rules (high cost of pruning)
Visualization
Data access: recent development on very large
training sets, fast, efficient and scalable (inmemory and secondary storage)
(well-known systems: C4.5 and CART)
19 November 2005
Scalable decision tree induction methods
SLIQ (Mehta et al., 1996)
builds an index for each attribute and only class list and the
current attribute list reside in memory
SPRINT (J. Shafer et al., 1996)
constructs an attribute list data structure
PUBLIC (Rastogi & Shim, 1998)
integrates tree splitting and tree pruning: stop growing the tree
earlier
RainForest (Gehrke, Ramakrishnan & Ganti, 1998)
separates the scalability aspects from the criteria that
determine the quality of the tree
builds an AVC-list (attribute, value, class label)
19 November 2005
Mining with neural networks
Effectively address the weakness of the symbolic
AI approach in knowledge discovery (grow of the
hypothesis space)
Extracting or making sense of numeric weights
associated with the interconnections of neurons
to come up with a higher level of knowledge has
been and will continue to be a challenge problem
19 November 2005
Mining with association rules
Improving the efficiency
Database
scan reduction: partitioning (Savaseve
95), hashing (Park 95), sampling (Toivonen 96),
dynamic itemset counting (Brin 97), find nonredundant rules (3000 times less, Zaki KDD’2000)
Parallel
mining of association rules
New measures of association
Interestingness
Generalized
and exceptional rules
and multiple-level rules
19 November 2005
Mining scientific data
Data Mining in Bioinformatics
Data Mining in Astronomy and Earth
Sciences
Mining Physics and Chemistry data
Mining Large Image Databases
etc.
19 November 2005
Solutions to mining huge datasets
Scalable and efficient algorithms
scalable: given an amount of main memory, its
runtime increases linearly with the number of input
instances
Massively parallel processing
Data-parallel vs. Control-parallel Data Mining
Client/Server Frameworks for Parallel Data
Mining
Mining Very Large Databases With Parallel Processing
Alex A. Freitas & Simon H. Lavington, Kluwer Academic Publishers, 1998
19 November 2005
Example of a scalable algorithm
Mixed Similarity Measures (MSM):
Goodall
(1966) time O(n3), Diday and Gowda
(1992),
Ichino
Li
and Yaguchi (1994),
& Biswas (1997) Time O(n2logn2), Space O(n2):
P̂ij
*
ˆ
New and Efficient MSM (Binh & Bao, 2000): Pij
ˆ 1 Pˆ *
Time and Space O(n): P
ij
ij
19 November 2005
Comparative results
US Census database 33 sym + 8 num attributes, Alpha 21264, 500 MHz, RAM 2 GB,
Solaris OS (Nguyen N.B. & Ho T.B., PKDD 2000)
＃cases
500
(0.2M)
1.000
(0.5M)
1.500
(0.9M)
2.000
(1.1M)
5.000
(2.6M)
10.000
(5.2M)
199.523
(102M)
# values
497
992
1.486
1.973
4.858
9.651
97.799
time of LiBis
O(n2logn2)
67.3s
26m6.2 1h46m31s 6h59m45s >60h
not app
not app
Time of OURS
O(n)
0.1s
0.2s
9.2s
36m26s
Memory of LiBis
O(n2)
Memory of OURS
O(n)
5.3M
Preprocessing
0.3s
0.5s
2.8s
20.0M
44.0M
77.0M
455.0M not app
not app
0.5 M
0.7M
0.9M
1.1M
2.1M
3.4M
64.0M
0.1s
0.1s
0.2s
0.5s
0.9s
6.2s
127.2s
19 November 2005
High performance computing for NLP
Experimental environments
Massively parallel computer (Cray XT3)
Linux OS, using C/C++ and MPI library
Experiments for noun phrase chunking
Cross-validation (CV) on 25 sections of Penn
TreeBank: about 1,000,000 words (more
than 40,000 English sentences)
Highest F1 score: 96.32%
Method
Precisi
on
Recall
F1measure
On single CPU (estimated): 2183.25 minutes
( 36.38 hours) for each fold 909.69
hours ( 37.9 days) for all 25 folds of CVtest
[KM01]
95.62%
95.93%
95.77%
[TKS00]
95.04%
94.75%
94.90%
[TV99]
93.71%
93.90%
93.81%
On 45 parallel processes of Cray XT3 system:
51.5 minutes for each fold 21.47 hours
( 1 day) for all 25 folds of CV-test
[RM95]
93.10%
93.50%
93.30%
Ours
96.42%
96.23%
96.32%
19 November 2005
Outline
Why knowledge discovery and data mining?
Basic concepts of KDD
KDD techniques: classification, association,
clustering, text and Web mining
Challenges and trends in KDD
Case study in medicine data mining
19 November 2005
Background
HBV, HCV are viruses which both cause continuous
inflammation of the liver (chronic hepatitis).
The inflammation results in liver fibrosis, and
finally liver cirrhosis (LC) after 20 to 30 years
which is diagnosed by liver biopsy.
In addition, the cirrhosis patients have a highly
potential risk of hepatocellular carcinoma (HCC).
Physicians can treat viral hepatitis with interferon
(IFN). However IFN is not always effective, with
severe side effects.
19 November 2005
The natural course of hepatitis
Hepatitis
Fibrosis stage
LC
HCC
20-30 years
F4
F3
F2
The
course
of HCC?
F1
F0
onset of infection
time
19 November 2005
The effect of interferon therapy
IFN
Fibrosis stage
HCC
F4
LC
F3
F2
F1
Effectiveness
of interferon?
F0
onset of infection
time
19 November 2005
時系列データ
提供元：千葉大学医学部
第一内科
約800人の患者の20年間
にわたるデータ
データの特徴
大規模な未整備時系列データ
検査項目数が非常に多い
検査時期により各検査項目の
値やその精度が異なる，欠損
値が多い
医者によるバイアスが存在
19 November 2005
Example of the hepatitis dataset
Sequences
of length
179 for
MID1
88 for
MID2, and
they are
irregular
19 November 2005
Problems under consideration
P1. Differences in temporal patterns between
hepatitis B and C? (HBV, HCV)
P2. Evaluate whether laboratory examinations can
be used to estimate the stage of liver fibrosis?
(F0, F1, F2, F3, F4)
P3. Evaluate whether the interferon therapy is
effective or not?
(Response, Partial response, Aggravation,
No response)
19 November 2005
Why data abstraction?
Medical knowledge is
usually expressed in a
form of symbolic
statements which is
as general as possible.
Patient data mostly
comprise numeric
measurements of various
parameters at different
points in time.
To perform any kind of medical problem solving,
patient data have to be “matched” against medical
knowledge.
19 November 2005
What is temporal abstraction?
ZTT: H>NS
ZTT
ZTT first was
increasingly high
then changed to the
normal region and
stable
normal region
Idea:
Convert time-stamped points to symbolic intervalbased representation of data
Characteristic: No detail but essence of trend and state
change of patients
19 November 2005
Research objectives
1. Develop temporal
abstraction methods
for describing
hepatitis data
appropriately for
each problem.
1
2
2. Using data mining
methods to solve
problems from the
abstracted data.
19 November 2005
Key issues in temporal abstraction
Tasks
Requirement
1. Define a description
language for
abstraction patterns
Simple but rich enough to
describe abstract patterns
2. Determine the basic
abstraction patterns
Be typical and significant
primitives needed for
analysis purpose
3. Transform each
sequence into temporal
abstraction patterns
Efficiently characterize the
trends and changes in the
temporal data
ZTT: H>NS
19 November 2005
Typical tests by physicians
Short-term changed tests
600
Up:
GPT, GOT, TTT, ZTT
500
400
300
200
100
2/19/2001
2/19/1999
2/19/1997
2/19/1995
2/19/1993
2/19/1991
2/19/1989
2/19/1987
2/19/1985
0
2/19/1983
Concerning inflammation, changed
quickly in days or weeks
Can be much higher (even 40 times)
than normal range with many peaks
2/19/1981
Long-term changed tests
Down: T-CHO, CHE, ALB, TP (liver products)
PLT, WBC, HGB.
Up:
T-BIL, D-BIL, I-BIL, AMONIA, ICG-15.
Concerning liver status, changed slowly
in months or years
Do not much exceed the normal range
19 November 2005
Observation of temporal sequences
Make a tool in
Matlab to
visualize temporal
sequences
Observe a large
number of
temporal
sequences for
different patients
and tests
19 November 2005
Ideas of basic patterns
Short-term changed tests
600
500
Up:
GPT, GOT, TTT, ZTT
400
300
200
100
2/19/2001
2/19/1999
2/19/1997
2/19/1995
2/19/1993
2/19/1991
2/19/1989
2/19/1987
2/19/1985
0
2/19/1983
base state and peaks
2/19/1981
Idea:
Long-term changed tests
Down: T-CHO, CHE, ALB, TP (liver products)
PLT, WBC, HGB.
Up:
T-BIL, D-BIL, I-BIL, AMONIA, ICG-15.
Idea:
change of states
(compactly capture both state
and trend of the sequence)
19 November 2005
時系列抽象化アプローチ
発見アルゴリズムを用い，各検
査項目の変化特性を「長期変化
項目」「短期変化項目」に分類
長期変化項目に関する21のパターン
短期変化項目に関する8の主要パターン
19 November 2005
Two temporal abstraction methods
Abstraction pattern
extraction (APE)
Temporal relation
extraction (TRE)
Mapping each given
temporal sequence of
fixed length into one
of pre-defined
temporal patterns
Detect temporal
relations between
basic patterns, and
extract rules using
temporal relations
(2001~)
(2004~)
19 November 2005
Data and knowledge visualization
Simultaneously view the data in different
forms: top-left is original data; top-right is
histogram of attributes; lower-left is view
by parallel coordinates; and lower-right is
relations between a conjunction of
attribute-value pairs and the class labels.
View an individual rule in D2MS: top-left
window shows the list of discovered
rules, the middle-left and the top-right
windows show a rule under inspection,
and bottom window displays the
instances covered by that rule.
19 November 2005
LC vs. non-LC
19 November 2005
Effectiveness of interferon (LUPC rules)
GOT & GPT
occurred as VH or
EH in no_response
rules
CHE occurred as
N/L or L-D in
partial_response
rules
D-BIL occurred as
N/H, H>N, H>N in
no_response and
partial_response
rules.
19 November 2005
Temporal relations
A
B
A is equal to B
B is equal to A
A
B
A
B
A
B
A
B
Relations between
two basic patterns
each happens in a
period of time,
says, “ALB get down
right after heavy
inflammation
finished”.
Updating a graph of
temporal relations
(Allen, 1983) and
temporal logic
(Allen, 1984)
A is before B
B is after A
A meets B
B is met by A
A overlaps B
B is overlapped by A
A starts B
B is started by A
B
A finishes B
B is finished by A
A
B
A is during B
B contains A
A
(Allen’s Temporal Logic, 1984)
19 November 2005
Findings for HBV and HCV
Findings are different from general medical observations
(no clear distinction between type B and C)
R#13
(HBV): “ALP changed from low to normal state” AFTER
“LDH changed from low to normal state” (supp. count = 21, conf. = 0.71)
R#5
(HCV): “ALP changed from normal to high state” AFTER
“LDH changed from normal to low state” (supp. count = 60, conf. = 0.80)
“Quantitatively” confirm findings in medicine (Medline
abstracts)
R#53 (HCV): “ALB changed from normal to low state” BEFORE “TTT in high
state with peaks” AND “ALP from normal to high state” BEFORE “TTT in
high state with peaks” (supp. count = 10, conf. = 1.00)
Murawaki et al. (2001): the main difference between HBV and HCV is that
the base state of TTT in HBV is normal, while that of HCV is high.
19 November 2005
Findings for LC and non-LC
Typical relations in non-LC rules
“GOT
in high or very high states with peaks”
BEFORE “TTT in high state with peaks” (20
rules contain this temporal relation)
Typical relations in LC rules
“GOT
in high or very high states with peaks”
AFTER “TTT in high or very high states with
peaks” (10 rules contain this temporal relation).
19 November 2005
多種情報源による医療データマイニング
肝炎患者履歴データ
に対する組合せアプ
ローチ
データマイニング
Medlineからのテキス
トマイニング
専門家の評価と提案
(プロジェクト期間 2004-2007)
19 November 2005
Conclusion
Temporal abstraction shown to be a good
alternative approach to hepatitis study.
The results be comprehensive to physicians, and
significant rules founds.
Much work to be done for solving three
problems, including:
Integrating
qualitative and quantitative information
Combining
with text mining techniques
Domain
expert knowledge
19 November 2005
Summary
KDD is motivated by the wide availability of huge
amounts of data and the imminent need for turning
such data into useful information and knowledge.
There are different methods to find predictive or
descriptive models that strongly depend on the data
schemes.
The KDD process is necessarily interactive and
iterative, and requires human participation in all
steps of the process.
19 November 2005
Recommended references
http://www.kdnuggets.com
David J. Hand, Heikki Mannila and Padhraic Smyth,
Principles of Data Mining, MIT Press, 2000
Jiawei Han, Micheline Kamber, Data Mining : Concepts
and Techniques, Morgan Kaufmann, 2000
U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, R.
Uthurusamy, editors, Advances in Knowledge Discovery
and Data Mining, AAAI/MIT Press, 1996.
19 November 2005
Acknowledgments
Profs. S. Ohsuga, M. Kimura, H. Motoda, H.
Shimodaira, Y. Nakamori, S. Horiguchi, T. Mitani,
T. Tsuji, K. Satou, among others
Nguyen N.B., Nguyen T.D., Kawasaki S., Huynh V.N.,
Dam H.C., Tran T.N., Pham T.H., Nguyen D.D.,
Nguyen L.M., Phan X.H., Le M.H., Hassine B.A.,
Le S.Q., Zhang H., Nguyen C.H., Nagai K., Nguyen
C.H., Nguyen T.P., Tran D.H.
19 November 2005
