Download Course on Data Mining: Seminar Meetings

Course on Data Mining (581550-4):
Seminar Meetings
P
Ass. Rules
16.11.
P
Clustering
02.11.
M
23.11.
Episodes
P
KDD Process
09.11.
M
Text Mining
30.11.
M Seminar by Mika
P Seminar by Pirjo
Home Exam
Course on Data Mining: Seminar Meetings
Page
1/17
Course on Data Mining (581550-4):
Seminar Meetings
Today 16.11.2001
• R. Feldman, M. Fresko, H. Hirsh, et.al.: "Knowledge
Management: A Text Mining Approach", Proc of the
2nd Int'l Conf. on Practical Aspects of Knowledge
Management (PAKM98), 1998
• B. Lent, R. Agrawal, R. Srikant: "Discovering Trends
in Text Databases", Proc. of the 3rd Int'l Conference
on Knowledge Discovery in Databases and Data
Mining, 1997.
Course on Data Mining: Seminar Meetings
Page
2/17
Course on Data Mining (581550-4):
Seminar Meetings
Good to Read as Background
• Both papers refer to the Agrawal and Srikant paper
we had last week:
Rakesh Agrawal and Ramakrishnan Srikant: Mining
Sequential Patterns. Int'l Conference on Data
Engineering, 1995.
Course on Data Mining: Seminar Meetings
Page
3/17
Knowledge Management:
A Text Mining Approach
R. Feldman, M. Fresko, H. Hirsh, et.al
Bar-Ilan University and Instict Software, ISRAEL; Rutgers University,
USA; LIA-EPFL, Switzerland
Published in PAKM'98 (Int'l Conf. on Practical Aspects of Knowledge
Management)
Data Mining course Autumn 2001/University of Helsinki
Summary by Mika Klemettinen
Course on Data Mining: Seminar Meetings
Page
4/17
KM: A Text Mining Approach
• Basic idea (see selected phases on the next slides):
1. Get input data in SGML (or XML) format
Select only the contents of desired elements! (title, abstract, etc.)
2. Do linguistic preprocessing:
2.1
Term extraction (use linguistic software for this)
2.2
Term generation (combine adjacent terms to morphosyntactic patterns like "noun-noun", "adj.-noun", etc. by
association coefficients)
2.3
Term filtering (select only the top M most frequent ones)
3. Create taxonomies (there is a tool for this)
4. Generate associations (you may constrain the creation)
5. Visualize/explore the results
Course on Data Mining: Seminar Meetings
calculating
Page
5/17
2.1: Term Extraction
Course on Data Mining: Seminar Meetings
Page
6/17
3: Taxonomy Construction
Course on Data Mining: Seminar Meetings
Page
7/17
4: Association Rule Generation
Course on Data Mining: Seminar Meetings
Page
8/17
4: Association Rule Generation
Course on Data Mining: Seminar Meetings
Page
9/17
5.1: Visualization/Exploration
Course on Data Mining: Seminar Meetings
Page
10/17
5.2: Visualization/Exploration
Course on Data Mining: Seminar Meetings
Page
11/17
Discovering Trends in Text Databases
Brian Lent, Rakesh Agrawal and Ramakrishnan Srikant
IBM Almaden Research Center, USA
Published in KDD'97
Data Mining course Autumn 2001/University of Helsinki
Summary by Mika Klemettinen
Course on Data Mining: Seminar Meetings
Page
12/17
Discovering Trends in Text Databases
• Basic ideas:
• Identify frequent phrases using sequential patterns mining (see the slides & summaries
from the Agrawal et. al paper "Mining Sequential Patterns" (MSP))
• Generate histories of phrases
• Find phrases that satisfy a specified trend
• Definitions:
• Phrase: phrase p is  (w1)(w2) … (wn ), where w is a word
• 1-phrase:  (IBM) (data)(mining) 
• 2-phrase:  (IBM) (data)(mining)   (Anderson) (Consulting) 
(decision)(support) 
• Itemset, sequence, is contained, etc.: as in MSP paper
Course on Data Mining: Seminar Meetings
Page
13/17
Discovering Trends in Text Databases
Gaps: Minimum and maximum gaps between adjacent words: identify relations of
words/phrases inside sentences/paragraphs, between words/phrases in different
paragraphs, between words/phrases in different sections, etc.
• Sentence boundary: 1000
• Paragraph boundary: 100.000
• Section boundary: 10.000.000
• Phases:
• Partition data/documents based on their time stamps, create phrases for each partition
(Lent & al. have patent data documents)
• Select the frequent phrases and save their frequences
• Define shape queries using SDL (Shape Definition Language)
•
Course on Data Mining: Seminar Meetings
Page
14/17
Discovering Trends in Text Databases
Course on Data Mining: Seminar Meetings
Page
15/17
Discovering Trends in Text Databases
Course on Data Mining: Seminar Meetings
Page
16/17
Discovering Trends in Text Databases
Course on Data Mining: Seminar Meetings
Page
17/17