Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Data Mining: Current Status and Research Directions Jiawei Han Intelligent Database Systems Research Lab School of Computing Science Simon Fraser University, Canada http://www.cs.sfu.ca/~han 2017년 5월 22일 Data Mining: Status and Directions 1 Outline Why is data mining hot? Current status: Major technical progress Is data mining flying high, or not? How to fly data mining high?— Research directions on data mining 2017년 5월 22일 Data Mining: Status and Directions 2 Why Is Data Mining Hot? Data mining (knowledge discovery in databases) Extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) information (knowledge) or patterns from data in large databases or other information repositories Necessity is the mother of invention Data is everywhere—data mining should be everywhere, too! Understand and use data—an imminent task! 2017년 5월 22일 Data Mining: Status and Directions 3 Data, Data, Everywhere!! Relational database—A commodity of every enterprise Huge data warehouses are under construction POS (Point of Sales): Transactional DBs in terabytes Object-relational databases, distributed, heterogeneous, and legacy databases Spatial databases (GIS), remote sensing database (EOS), and scientific/engineering databases Time-series data (e.g., stock trading) and temporal data Text (documents, emails) and multimedia databases WWW: A huge, hyper-linked, dynamic, global information system 2017년 5월 22일 Data Mining: Status and Directions 4 Data Mining Is Everywhere, too!—A Multi-Dimensional View of Data Mining Databases to be mined Relational, transactional, object-relational, active, spatial, timeseries, text, multi-media, heterogeneous, legacy, WWW, etc. Knowledge to be mined Characterization, discrimination, association, classification, clustering, trend, deviation and outlier analysis, etc. Techniques utilized Database-oriented, data warehouse (OLAP), machine learning, statistics, visualization, neural network, etc. Applications adapted Retail, telecommunication, banking, fraud analysis, DNA mining, stock market analysis, Web mining, Weblog analysis, etc. 2017년 5월 22일 Data Mining: Status and Directions 5 Data Mining: Confluence of Multiple Disciplines Database Technology Machine Learning (AI) Information Science 2017년 5월 22일 Statistics Data Mining Visualization Other Disciplines Data Mining: Status and Directions 6 Data Mining—One Can Trace Back to Early Civilization Most scientific discoveries involve “data mining” Kepler’s Law, Newton’s Laws, periodic table of chemical elements, …, from “big bang” to DNA Statistics: A discipline dedicated to data analysis Then why data mining? What are the differences? Huge amount of data—in giga to tera bytes Fast computer—quick response, interactive analysis Multi-dimensional, powerful, thorough analysis High-level, “declarative”—user’s ease and control Automated or semi-automated—mining functions hidden or built-in in many systems 2017년 5월 22일 Data Mining: Status and Directions 7 A Brief History of Data Mining Activities 1989 IJCAI Workshop on Knowledge Discovery in Databases 1991-1994 Workshops on Knowledge Discovery in Databases Advances in Knowledge Discovery and Data Mining (U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, 1996) 1995-1998 International Conferences on Knowledge Discovery in Databases and Data Mining (KDD’95-98) Knowledge Discovery in Databases (G. Piatetsky-Shapiro and W. Frawley, 1991) Journal of Data Mining and Knowledge Discovery (1997) 1998 ACM SIGKDD, SIGKDD’1999-2001 conferences, and SIGKDD Explorations More conferences on data mining PAKDD, PKDD, SIAM-Data Mining, (IEEE) ICDM, DaWaK, SPIE-DM, etc. 2017년 5월 22일 Data Mining: Status and Directions 8 Research Progress in the Last Decade Multi-dimensional data analysis: Data warehouse and OLAP (on-line analytical processing) Association, correlation, and causality analysis Classification: scalability and new approaches Clustering and outlier analysis Sequential patterns and time-series analysis Similarity analysis: curves, trends, images, texts, etc. Text mining, Web mining and Weblog analysis Spatial, multimedia, scientific data analysis Data preprocessing and database compression Data visualization and visual data mining Many others, e.g., collaborative filtering 2017년 5월 22일 Data Mining: Status and Directions 9 Multi-Dimensional Data Analysis Data warehousing: integration from heterogeneous or semi-structured databases Multi-dimensional modeling of data: star & snowflake schemas Efficient and scalable computation of data cubes or iceberg cubes OLAP (on-line analytical processing): drilling, dicing, slicing, etc. Discovery-driven exploration of data cubes From OLAP to OLAM: A multi-dimensional view for on-line analytical mining 2017년 5월 22일 Data Mining: Status and Directions 10 Association and Frequent Pattern Analysis Efficient mining of frequent patterns and association rules: Apriori and FP-growth algorithms Multi-level, multi-dimensional, quantitative association mining From association to correlation, sequential patterns, partial periodicity, cyclic rules, ratio rules, etc. Query and constraint-based association analysis 2017년 5월 22일 Data Mining: Status and Directions 11 Classification: Scalable Methods and Handling of Complex Types of Data Classification has been an essential theme in machine learning, and statistics research Decision trees, Bayesian classification, neural networks, k-nearest neighbors, etc. Tree-pruning, Boosting, bagging techniques Efficient and scalable classification methods Exploration of attribute-class pairs SLIQ, SPRINT, RainForest, BOAT, etc. Classification of semi-structured and non-structured data Classification by clustering association rules (ARCS) Association-based classification Web document classification 2017년 5월 22일 Data Mining: Status and Directions 12 Clustering and Outlier Analysis Partitioning methods k-means, k-medoids, CLARANS Hierarchical methods: micro-clusters Birch, Cure, Chameleon Density-based methods: DBSCAN and OPTICS, DENCLU Grid-based methods STING, CLIQUE, WaveCluster Outlier analysis: statistics-based, distance-based, deviation-based Constraint-based clustering COD (Clustering with Obstructed Distance) User-specified constraints 2017년 5월 22일 Data Mining: Status and Directions 13 Sequential Patterns and TimeSeries Analysis Trend analysis Trend movement vs. cyclic variations, seasonal variations and random fluctuations Similarity search in time-series database Handling gaps, scaling, etc. Indexing methods and query languages for time-series Sequential pattern mining Various kinds of sequences, various methods From GSP to PrefixSpan Periodicity analysis Full periodicity, partial periodicity, cyclic association rules 2017년 5월 22일 Data Mining: Status and Directions 14 Similarity Search: Similar Curves, Trends, Images, and Texts Various kinds of data, various similarity mining methods Discovery of similar trends in time-series data Data transformation & high-dimensional structures Finding similar images based on color, texture, etc. Content-based vs. keyword-based retrieval Color histogram-based signature Multi-feature composed signature Finding documents with similar texts Similar keywords (synonymy & polysemy) Term frequency matrix Latent semantic indexing 2017년 5월 22일 Data Mining: Status and Directions 15 Spatial, Multimedia, Scientific Data Analysis Multi-dimensional analysis of spatial, multimedia and scientific data Geo-spatial data cube and spatial OLAP The curse of dimensionality problem Association analysis A progressive refinement methodology Micro-clustering can be used for preprocessing in the analysis of complex types of data Classification Association-based for handling high-dimensionality and sparse data 2017년 5월 22일 Data Mining: Status and Directions 16 Data Mining Industry and Applications From research prototypes to data mining products, languages, and standards IBM Intelligent Miner, SAS Enterprise Miner, SGI MineSet, Clementine, MS/SQLServer 2000, DBMiner, BlueMartini, MineIt, DigiMine, etc. A few data mining languages and standards (esp. MS OLEDB for Data Mining). Application achievements in many domains Market analysis, trend analysis, fraud detection, outlier analysis, Web mining, etc. 2017년 5월 22일 Data Mining: Status and Directions 17 Is Data Mining Flying? Or Not?? Data mining is flying R & D have been striding forward greatly Applications have been broadened substantially But not as high as some may have hoped. Why not? Hope to see billions of $’s within years? Not bread-and-butter but value-added service DBMS, WWW, and other information systems will still be a “data mining” aircraft-carrier Not on-the-shelf in nature A young and coming technology, not a hype! Need training, understanding, and customizing (re-develop.) Young technology—need much R&D to fly high 2017년 5월 22일 Much research, development, and real problem solving! Data Mining: Status and Directions 18 How to Fly Data Mining High?— Research Directions Web mining Towards integrated data mining environments and tools “Vertical” (or application-specific) data mining Invisible data mining Towards intelligent, efficient, and scalable data mining methods 2017년 5월 22일 Data Mining: Status and Directions 19 Web Mining: A Fast Expanding Frontier in Data Mining Mine what Web search engine finds Automatic classification of Web documents Discovery of authoritative Web pages, Web structures and Web communities Meta-Web Warehousing: Web yellow page service Web usage mining 2017년 5월 22일 Data Mining: Status and Directions 20 Mine What Web Search Engine Finds Current Web search engines: A convenient source for mining keyword-based, return too many, often low quality answers, still missing a lot, not customized, etc. Data mining will help: coverage: “Enlarge and then shrink,” using synonyms and conceptual hierarchies better search primitives: user preferences/hints linkage analysis: authoritative pages and clusters Web-based languages: XML + WebSQL + WebML customization: home page + Weblog + user profiles 2017년 5월 22일 Data Mining: Status and Directions 21 Discovery of Authoritative Pages in WWW Page-rank method ( Brin and Page, 1998): Rank the "importance" of Web pages, based on a model of a "random browser." Hub/authority method (Kleinberg, 1998): Prominent authorities often do not endorse one another directly on the Web. Hub pages have a large number of links to many relevant authorities. Thus hubs and authorities exhibit a mutually reinforcing relationship: Both the page-rank and hub/authority methodologies have been shown to provide qualitatively good search results for broad query topics on the WWW. 2017년 5월 22일 Data Mining: Status and Directions 22 Automatic Classification of Web Documents Web document classification: Good human classification: Yahoo!, CS term hierarchies These classifications can be used as training sets to build up learning model Key-word based classification is different from multidimensional classification Association or clustering-based classification is often more effective Multi-level classification is important 2017년 5월 22일 Data Mining: Status and Directions 23 A Multiple Layered Meta-Web Architecture Layern More Generalized Descriptions ... Layer1 Generalized Descriptions Layer0 2017년 5월 22일 Data Mining: Status and Directions 24 Web Yellow Page Service: A MultiLayer, Meta-Web Approach XML: facilitates structured and meta-information extraction Automatic classification of Web documents: based on Yahoo!, etc. as training set + keyword-based correlation/classification analysis (IR/AI assistance) Automatic ranking of important Web pages authoritative site recognition and clustering Web pages Generalization-based multi-layer meta-Web construction With the assistance of clustering and classification analysis Meta-Web can be warehoused and incrementally updated Querying and mining can be performed on or assisted by meta-Web 2017년 5월 22일 Data Mining: Status and Directions 25 Importance of Constructing Multi-Layer Meta Web Benefits of Multi-Layer Meta-Web: Multi-dimensional Web info summary analysis Approximate and intelligent query answering Web high-level query answering (WebSQL, WebML) Web content and structure mining Observing the dynamics/evolution of the Web Is it realistic to construct such a meta-Web? It benefits even if it is partially constructed The benefit may justify the cost of tool development, standardization, and partial restructuring 2017년 5월 22일 Data Mining: Status and Directions 26 Web Usage (Click-Stream) Mining Weblog provides rich information about Web dynamics Multidimensional Weblog analysis: Plan mining (mining general Web accessing regularities): Web cashing, prefetching, swapping Trend analysis: Web linkage adjustment, performance improvements Web accessing association/sequential pattern analysis: disclose potential customers, users, markets, etc. Dynamics of the Web: what has been changing? Customized to individual users 2017년 5월 22일 Data Mining: Status and Directions 27 Towards Integrated Data Mining Environments and Tools OLAP Mining: Integration of Data Warehousing and Data Mining Querying and Mining: An Integrated Information Analysis Environment Basic Mining Operations and Mining Query Optimization “Vertical” (or application-specific) data mining Invisible data mining 2017년 5월 22일 Data Mining: Status and Directions 28 OLAP Mining: An Integration of Data Mining and Data Warehousing Data mining systems, DBMS, Data warehouse systems coupling On-line analytical mining data No coupling, loose-coupling, semi-tight-coupling, tight-coupling integration of mining and OLAP technologies Interactive mining multi-level knowledge Necessity of mining knowledge and patterns at different levels of abstraction by drilling/rolling, pivoting, slicing/dicing, etc. Integration of multiple mining functions Characterized classification, first clustering and then association 2017년 5월 22일 Data Mining: Status and Directions 29 An OLAM Architecture Mining query Mining result Layer4 User Interface User GUI API OLAM Engine OLAP Engine Layer3 OLAP/OLAM Data Cube API Layer2 MDDB MDDB Meta Data Filtering&Integration Database API Filtering Layer1 Data cleaning Databases 2017년 5월 22일 Data Data integration Warehouse Data Mining: Status and Directions Data Repository 30 Querying and Mining: An Integrated Information Analysis Environment Data mining as a component of DBMS, data warehouse, or Web information system Integrated information processing environment MS/SQLServer-2000 (Analysis service) IBM IntelligentMiner on DB2 SAS EnterpriseMiner: data warehousing + mining Query-based mining Querying database/DW/Web knowledge Efficiency and flexibility: preprocessing, on-line processing, optimization, integration, etc. 2017년 5월 22일 Data Mining: Status and Directions 31 Basic Mining Operations and Mining Query Optimization Relational databases: There are a set of basic relational operations and a standard query language, SQL E.g., selection, projection, join, set difference, intersection, Cartesian product, etc. Are there a set of standard data mining operations, on which optimizations can be done? Difficulty: different definitions on operations Importance: optimization can be performed on them systematically, standardization to facilitate information exchange and system interoperability 2017년 5월 22일 Data Mining: Status and Directions 32 “Vertical” Data Mining Generic data mining tools? —Too simple to match domainspecific, sophisticated applications Expert knowledge and business logic represent many years of work in their own fields! Data mining + business logic + domain experts A multi-dimensional view of data miners Complexity of data: Web, sequence, spatial, multimedia, … Complexity of domains: DNA, astronomy, market, telecom, … Domain-specific data mining tools Provide concrete, killer solution to specific problems Feedback to build more powerful tools 2017년 5월 22일 Data Mining: Status and Directions 33 Invisible Data Mining Build mining functions into daily information services Web search engine (link analysis, authoritative pages, user profiles)—adaptive web sites, etc. Improvement of query processing: history + data Making service smart and efficient Benefits from/to data mining research Data mining research has produced many scalable, efficient, novel mining solutions Applications feed new challenge problems to research 2017년 5월 22일 Data Mining: Status and Directions 34 Towards Intelligent Tools for Data Mining Integration paves the way to intelligent mining Smart interface brings intelligence One picture may worth 1,000 words Easy to use, understand and manipulate Visual and audio data mining Human-Centered Data Mining Towards self-tuning, self-managing, selftriggering data mining 2017년 5월 22일 Data Mining: Status and Directions 35 Integrated Mining: A Booster for Intelligent Mining Integration paves the way to intelligent mining Data mining integrates with DBMS, DW, WebDB, etc Integration inherits the power of up-to-date information technology: querying, MD analysis, similarity search, etc. Mining can be viewed as querying database knowledge Integration leads to standard interface/language, function/process standardization, utility, and reachability Efficiency and scalability bring intelligent mining to reality 2017년 5월 22일 Data Mining: Status and Directions 36 One Picture May Worth 1000 Words! Visual Data Mining Visualization of data Visualization of data mining results Visualization of data mining processes Interactive data mining: visual classification One melody may worth 1000 words too! Audio data mining: turn data into music and melody! Uses audio signals to indicate the patterns of data or the features of data mining results 2017년 5월 22일 Data Mining: Status and Directions 37 Visualization of data mining results in SAS Enterprise Miner: scatter plots 2017년 5월 22일 Data Mining: Status and Directions 38 Visualization of association rules in MineSet 3.0 2017년 5월 22일 Data Mining: Status and Directions 39 Visualization of a decision tree in MineSet 3.0 2017년 5월 22일 Data Mining: Status and Directions 40 Visualization of Data Mining Processes by Clementine 2017년 5월 22일 Data Mining: Status and Directions 41 Interactive Visual Mining by Perception-Based Classification (PBC) 2017년 5월 22일 Data Mining: Status and Directions 42 Human-Centered Data Mining Finding all the patterns autonomously in a database? — unrealistic because the patterns could be too many but uninteresting Data mining should be an interactive process User directs what to be mined Users must be provided with a set of primitives to be used to communicate with the data mining system — using a data mining query language User should provide constraints on what to be mined System should use such constraints to guide the mining process (constraint-based mining or mining query optimization) 2017년 5월 22일 Data Mining: Status and Directions 43 Constraint-Based Mining What kinds of constraints can be used in mining? Knowledge type constraint: classification, association, etc. Data constraint: SQL-like queries Find products sold together in Vancouver in Feb.’01. Dimension/level constraints: in relevance to region, price, brand, customer category. Rule constraints: small sales (price < $10) triggers big sales (sum > $200). Interestingness constraints: E.g., strong rules (min_support 3%, min_confidence 60%, min_lift > 3.0). 2017년 5월 22일 Data Mining: Status and Directions 44 Rule Constraints: A Classification Succinctness Anti-monotonicity Monotonicity Convertible constraints Inconvertible constraints 2017년 5월 22일 Data Mining: Status and Directions 45 Constraint-Based Clustering Analysis User-specified constraints: no cluster has less than 1000 gold customers Resource allocation (clustering) with obstacles 2017년 5월 22일 Data Mining: Status and Directions 46 Towards Automated Data Mining? It is not realistic to automatically find all the knowledge in a large database Thus we promote human-centered, constraint-based mining However, to achieve genuine intelligent data mining, data mining process should be self-tuning, self-managing, self-triggering Functions should be developed to achieve such performance 2017년 5월 22일 Data Mining: Status and Directions 47 Conclusions Data mining—A promising research frontier Data mining research has been striding forward greatly in the last decade However, data mining, as an industry, has not been flying as high as expected Much research and application exploration are needed Web mining Towards integrated data mining environments and tools Towards intelligent, efficient, and scalable data mining methods 2017년 5월 22일 Data Mining: Status and Directions 48 http://www.cs.sfu.ca/~han http://db.cs.sfu.ca Thank you !!! 2017년 5월 22일 Data Mining: Status and Directions 49 References J. Han and M. Kamber, Data Mining: Concepts and Techniques, Morgan Kaufmann, 2001. J. Han, L. V. S. Lakshmanan, and R. T. Ng, "Constraint-Based, Multidimensional Data Mining", COMPUTER (special issues on Data Mining), 32(8): 46-50, 1999. 2017년 5월 22일 Data Mining: Status and Directions 50