Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Data Mining Course Chapter 1 Introduction to Data Mining Data Mining Course, Sharif University of Technology 1 Introduction to Data Mining • Examples of Data Mining – – – – – Bank of America 13 million contact Bank of America’s call center each month In past, customers listened to same marketing message Whether relevant to customer or not Kelly, VP Database Marketing states, “...we want to be as relevant as possible to each customer” – Customer profiles available to service representatives – May suggest applicable products or services – Data mining helps identify marketing approach based on customer’s profile Data Mining Course, Sharif University of Technology 2 Introduction to Data Mining (cont’d) – Homeland Security – Shortly after 9/11/2001 events, FBI announced identification of five terrorists in consumer database records – One had 30 credit cards with $250,000 debt – Another had 12 different addresses – Former President Clinton concluded data should be proactively searched – Clinton said, “...they have 12 homes, they’re either really rich or up to no good...shouldn’t be hard to figure out which.” Data Mining Course, Sharif University of Technology 3 Introduction to Data Mining (cont’d) – – – – – – Gene Expression Database In children, brain tumors represent deadly form of cancer 3,000 cases diagnosed per year Children’s Memorial Hospital building gene expression database Goal is developing more effective treatment Bremer, Director of Brain Research, uses Clementine as initial step in tumor identification – Classification identifies one of 12 different tumor types Data Mining Course, Sharif University of Technology 4 Trends leading to Data Flood • More data is generated: – Bank, telecom, other business transactions ... – Scientific data: astronomy, biology, etc – Web, text, and ecommerce Data Mining Course, Sharif University of Technology 5 Big data examples • Europe's Very Long Baseline Interferometry (VLBI) has 16 telescopes, each of which produces 1 Gigabit/second of astronomical data over a 25-day observation session – storage and analysis a big problem • AT&T handles billions of calls per day – so much data, it cannot be all stored -- analysis has to be done “on the fly”, on streaming data Data Mining Course, Sharif University of Technology 6 Largest databases in 2003 • Commercial databases: – Winter Corp. 2003 Survey: France Telecom has largest decision-support DB, ~30TB; AT&T ~ 26 TB • Web – – – – Alexa internet archive: 7 years of data, 500 TB Google searches 4+ Billion pages, many hundreds TB IBM WebFountain, 160 TB (2003) Internet Archive (www.archive.org),~ 300 TB Data Mining Course, Sharif University of Technology 7 Data growth rate • Twice as much information was created in 2002 as in 1999 (~30% growth rate) • Other growth rate estimates even higher • Very little data will ever be looked at by a human • Knowledge Discovery is NEEDED to make sense and use of data. Data Mining Course, Sharif University of Technology 8 What is Data Mining? • “…the process of discovering meaningful new correlations, patterns, and trends by sifting through large amounts of data…” (Gartner Group) • “…the analysis of observational data sets to find unsuspected relationships and to summarize data in novel ways…” (Hand et al.) • “Extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) patterns or knowledge from huge amount of data”, (Han et al.) • “…is an interdisciplinary field bringing together techniques from machine • learning, pattern recognition, statistics, databases, and visualization…” (Cabana et al.) Alternative names: – Knowledge discovery (mining) in databases (KDD), knowledge extraction, data/pattern analysis, data archeology, data dredging, information harvesting, business intelligence, etc. Data Mining Course, Sharif University of Technology 9 What is Data Mining? (cont’d) • Data mining chosen as one of top 10 emerging technologies (MIT Technology Review) • “Data mining expertise is most sought after...” (1999 Information Week Survey) • Brown, from BridgeGate LLC said, “Many companies have implemented a data warehouse...starting to look at what they can do with all that data” Data Mining Course, Sharif University of Technology 10 What is Data Mining? (cont’d) • How widespread is data mining? – – – – – – – – Boston Celtics listed employment position in 12/2003 Statistics Intern: Work with Basketball Operations “Responsibilities include: ...data mining, etc.” New York Nicks already using IBM’s Advanced Scout data mining software Software includes NBA’s game data in form of “events” Each game includes statistics such as shots, passes, points, rebounds, etc. Against Chicago Bulls, software discovered pattern coaching staff missed 16 of 29 NBA teams have turned to Advanced Scout to mine play-by-play data Data Mining Course, Sharif University of Technology 11 Why Data Mining? • “...we are drowning in information but starved for • • knowledge.” (Naisbitt, author Megatrends) Not enough trained analysts available to translate data into knowledge Data mining fueled by several factors – – – – Explosive growth in data collection The storage of enterprise-wide data in data warehouses Increased availability of Web clickstream data The tremendous growth in computing power and storage capacity – Development of off-the-shelf commercial data mining software products Data Mining Course, Sharif University of Technology 12 Potential applications • Data analysis and decision support – Market analysis and management • Target marketing, customer relationship management (CRM), market basket analysis, cross selling, market segmentation • Forecasting, customer retention, improved underwriting, quality control, competitive analysis – Fraud detection and detection of unusual patterns • Other applications – Text mining (news group, email, documents) and Web mining – Stream data mining – DNA and bio-data analysis Data Mining Course, Sharif University of Technology 13 Market Analysis and Management • Where does the data come from? – • • Target marketing – Find clusters of “model” customers who share the same characteristics: interest, income level, spending habits, etc. – Determine customer purchasing patterns over time Cross-market analysis – • • Associations/co-relations between product sales, & prediction based on such association Customer profiling – • Credit card transactions, loyalty cards, discount coupons, customer complaint calls, plus (public) lifestyle studies What types of customers buy what products (clustering or classification) Customer requirement analysis – identifying the best products for different customers – predict what factors will attract new customers Provision of summary information – multidimensional summary reports – statistical summary information (data central tendency and variation) Data Mining Course, Sharif University of Technology 14 Corporate Analysis & Risk Management • Finance planning and asset evaluation – cash flow analysis and prediction – contingent claim analysis to evaluate assets – cross-sectional and time series analysis (financial-ratio, trend analysis, etc.) • Resource planning – summarize and compare the resources and spending • Competition – monitor competitors and market directions – group customers into classes and a class-based pricing procedure – set pricing strategy in a highly competitive market Data Mining Course, Sharif University of Technology 15 Fraud Detection & Mining Unusual Patterns • Approaches: Clustering & model construction for frauds, outlier analysis • Applications: Health care, retail, credit card service, telecomm. – Auto insurance: ring of collisions – Money laundering: suspicious monetary transactions – Medical insurance • Professional patients, ring of doctors, and ring of references • Unnecessary or correlated screening tests – Telecommunications: phone-call fraud • Phone call model: destination of the call, duration, time of day or week. Analyze patterns that deviate from an expected norm – Retail industry • Analysts estimate that 38% of retail shrink is due to dishonest employees – Anti-terrorism Data Mining Course, Sharif University of Technology 16 Other Applications • Sports – IBM Advanced Scout analyzed NBA game statistics (shots blocked, assists, and fouls) to gain competitive advantage for New York Knicks and Miami Heat • Astronomy – JPL and the Palomar Observatory discovered 22 quasars with the help of data mining • Internet Web Surf-Aid – IBM Surf-Aid applies data mining algorithms to Web access logs for marketrelated pages to discover customer preference and behavior pages, analyzing effectiveness of Web marketing, improving Web site organization, etc. Data Mining Course, Sharif University of Technology 17 Data Mining: A KDD Process Pattern Evaluation – Data mining—core of knowledge discovery Data Mining process Task-relevant Data Data Warehouse Selection Data Cleaning Data Integration Databases Data Mining Course, Sharif University of Technology 18 Steps of a KDD Process • Learning the application domain • • • • • • • • – relevant prior knowledge and goals of application Creating a target data set: data selection Data cleaning and preprocessing: (may take 60% of effort!) Data reduction and transformation – Find useful features, dimensionality/variable reduction, invariant representation. Choosing functions of data mining – summarization, classification, regression, association, clustering. Choosing the mining algorithm(s) Data mining: search for patterns of interest Pattern evaluation and knowledge presentation – visualization, transformation, removing redundant patterns, etc. Use of discovered knowledge Data Mining Course, Sharif University of Technology 19 The Need for Human Direction of Data Mining – Some early data mining definitions described process as “automatic” – “…this has misled many people into believing data mining is product that can be bought rather than a discipline that must be mastered.” (Berry, Linoff) – Automation no substitute for human input – Data mining is easy to do badly – Understanding statistical and mathematical model structures of underlying software required – Humans need to be actively involved in every phase of data mining process – Task of data mining should be integrated into human process of problem solving Data Mining Course, Sharif University of Technology 20 Cross Industry Standard Process: CRISP-DM • Cross-Industry Standard Process for Data Mining (CRISPDM) developed in 1996 – – – – – Contributors include DaimlerChrysler, SPSS, and NCR Developed to fit data mining into general business strategy Process vendor and tool-neutral Non-proprietary and freely available Data mining projects follow iterative, adaptive life cycle consisting of 6 phases – Phase sequences are adaptive – Next, Figure 1.1 illustrates CRISP-DM lifecycle Data Mining Course, Sharif University of Technology 21 Cross Industry Standard Process: CRISP-DM (cont’d) – Iterative CRIP-DM process shown in outer circle – Most significant dependencies between phases shown Business / Research Understanding Phase Deployment Phase – Next phase depends on results from preceding phase Evaluation Phase Data Understanding Phase Data Preparation Phase Modeling Phase – Returning to earlier phase possible before moving forward Data Mining Course, Sharif University of Technology 22 Cross Industry Standard Process: CRISP-DM (cont’d) • (1) Business Understanding Phase – Define business requirements and objectives – Translate objectives into data mining problem definition – Prepare initial strategy to meet objectives • (2) Data Understanding Phase – Collect data – Assess data quality – Perform exploratory data analysis (EDA) • (3) Data Preparation Phase – Cleanse, prepare, and transform data set – Prepares for modeling in subsequent phases – Select cases and variables appropriate for analysis Data Mining Course, Sharif University of Technology 23 Cross Industry Standard Process: CRISP-DM (cont’d) • (4) Modeling Phase – Select and apply one or more modeling techniques – Calibrate model settings to optimize results – If necessary, additional data preparation may be required • (5) Evaluation Phase – Evaluate one or more models for effectiveness – Determine whether defined objectives achieved – Make decision regarding data mining results before deploying to field Data Mining Course, Sharif University of Technology 24 Cross Industry Standard Process: CRISP-DM (cont’d) • (6) Deployment Phase – Make use of models created – Simple deployment: generate report – Complex deployment: implement additional data mining effort in another department – In business, customer often carries out deployment based on model • See http://www.crisp-dm.org for more information Data Mining Course, Sharif University of Technology 25 Case Study 1 • Analyzing Automobile Warranty Claims – Business Understanding – Objectives include improving customer satisfaction and reducing costs – Manufacturing engineers consulted to formulate business problems – Data mining techniques used to uncover possible issues: • Warranty claim interdependencies? • Past claims associated with future claims? • Association between claim and repair facility? Data Mining Course, Sharif University of Technology 26 Case Study 1 (cont’d) – Data Understanding – 40GB QUIS database containing 7 million vehicle records used – Vehicle records include manufacturing location, warrant claims, and additional codes – Database unintelligible to non-domain experts – Costly effort to consult with domain experts from different departments – Data Preparation – QUIS discovered to have limited SQL access – Cases and variables manually extracted – Additional variables derived for modeling phase Data Mining Course, Sharif University of Technology 27 Case Study 1 (cont’d) – – – – – Proprietary data mining software used Data format requirements varied for different algorithms Resulted in exhaustive pre-processing of data Modeling Phase Applied Bayesian networks and association rules to uncover dependencies between warranty claims – Discovered specific combination of construction specifications doubles probability of electrical cable claim – Investigated whether some garages had more claims than others – Remaining results confidential Data Mining Course, Sharif University of Technology 28 Case Study 1 (cont’d) – – – – – – – Evaluation Researchers disappointed in results Association rules could not be generalized Rules “not interesting” according to domain experts Data models fell short of business objectives Legacy databases not suited to data mining Proposal suggested database redesign for future data mining efforts – Deployment – Foregoing effort identified as pilot project, models not deployed – Future data mining efforts planned to integrate more closely to database systems at DaimlerChrysler Data Mining Course, Sharif University of Technology 29 Case Study 1 (cont’d) – – – – – – – – Summary Uncovering hidden nuggets very difficult During each phase, researchers encountered roadblocks Applying new data mining effort problematic Data mining effort requires management support Substantial human participation required at every stage Installation, configuration, and data mining modeling not magic Wrong analysis leads to possibly expensive policy recommendations – No guarantee data mining effort delivers actionable results – However, used properly, data mining may provide profitable results Data Mining Course, Sharif University of Technology 30 Fallacies of Data Mining • Four Fallacies of Data Mining (Louie, Nautilus Systems, Inc.) – – – – – – – Fallacy 1 Set of tools can be turned loose on data repositories Finds answers to all business problems Reality 1 No automatic data mining tools solve problems Rather, data mining is process (CRISP-DM) Integrates into overall business objectives – Fallacy 2 – Data mining process is autonomous – Requires little oversight Data Mining Course, Sharif University of Technology 31 Fallacies of Data Mining (cont’d) – – – – Reality 2 Requires significant intervention during every phase After model deployment, new models require updates Continuous evaluative measures monitored by analysts – – – – – Fallacy 3 Data mining quickly pays for itself Reality 3 Return rates vary Depending on startup, personnel, data preparation costs, etc. – Fallacy 4 – Data mining software easy to use Data Mining Course, Sharif University of Technology 32 Fallacies of Data Mining (cont’d) – Reality 4 – Ease of use varies across projects – Analysts must combine subject matter knowledge with specific problem domain • Two Additional Fallacies (Larose) – – – – – Fallacy 5 Data mining identifies causes of business problems Reality 5 Knowledge discovery process uncovers patterns of behavior Humans interpret results and identify causes Data Mining Course, Sharif University of Technology 33 Fallacies of Data Mining – – – – – – (cont’d) Fallacy 6 Data mining automatically cleans data in databases Reality 6 Data mining often uses data from legacy systems Data possibly not examined or used in years Organizations starting data mining efforts confronted with huge data preprocessing task Data Mining Course, Sharif University of Technology 34 What Tasks Can Data Mining Accomplish? • Six common data mining tasks – – – – – – Description Estimation Prediction Classification Clustering Association • (1) Description – Describes patterns or trends in data – For example, pollster may uncover patterns suggesting those laid-off less likely to support incumbent – Descriptions of patterns, often suggest possible explanations Data Mining Course, Sharif University of Technology 35 What Tasks Can Data Mining Accomplish? (cont’d) – For example, those laid-off now less financially secure; therefore, prefer alternate candidate – Data mining models should be transparent – That is, results should be interpretable by humans – Some data mining methods more transparent than others – For example, Decision Trees (transparent) <-> Neural Networks (opaque) – High-quality description accomplished using Exploratory Data Analysis (EDA) – Graphical method of exploring patterns and trends in data Data Mining Course, Sharif University of Technology 36 What Tasks Can Data Mining Accomplish? (cont’d) • (2) Estimation – Similar to Classification task, except target variable numeric – Models built from complete data records – Records include values for each predictor field and numeric target variable in training set – For new observations, estimate of target variable made – For example, estimate a patient’s systolic blood pressure, based on patient’s age, gender, body-mass index, and sodium levels – Here, estimation model built from training set records – Model then estimates value for new case Data Mining Course, Sharif University of Technology 37 What Tasks Can Data Mining Accomplish? (cont’d) – Estimation Tasks in Business and Research: – Estimate amount of money, family of four will spend on back-toschool shopping – Estimate percentage decrease in rotary movement sustained to NFL player with knee injury – Estimate number of points basketball player scores when double-teamed in playoffs – Estimate GPA of graduate student, based on student’s undergraduate GPA Data Mining Course, Sharif University of Technology 38 What Tasks Can Data Mining Accomplish? (cont’d) – Figure 1.2 shows scatter plot of graduate GPA against undergraduate GPA – Linear regression finds line (blue) best approximating relationship between two variables – Regression line estimates student’s graduate GPA based on their undergraduate GPA Data Mining Course, Sharif University of Technology 39 What Tasks Can Data Mining Accomplish? (cont’d) – Minitab statistical software produces regression equation ŷ = 1.24 + 0.67x – Therefore, estimated student’s graduate GPA = 1.24 plus 0.67 times their undergraduate GPA – – – – For example, suppose student’s undergraduate GPA = 3.0 According to estimation model Estimated student’s graduate GPA = 1.24 + 0.67(3.0) = 3.25 Point (x = 3.0, ŷ = 3.25) lies on regression line – Statistical Analysis uses several estimation methods: point estimation, confidence interval estimation, linear regression and correlation, and multiple regression Data Mining Course, Sharif University of Technology 40 What Tasks Can Data Mining Accomplish? (cont’d) • (3) Prediction – Similar to classification and estimation, except results lie in the future – Prediction Tasks in Business and Research: – Predict price of stock 3 months into future, based on past performance ? Stock Price ? ? Q1 Q2 Q3 Q4 Data Mining Course, Sharif University of Technology 41 What Tasks Can Data Mining Accomplish? (cont’d) – Predict percentage increase in traffic deaths next year, if speed limit increased – Predict whether molecule in newly discovered drug leads to profitable pharmaceutical drug – Methods used for classification and estimation applicable to prediction – Includes point estimation, confidence interval estimation, linear regression and correlation, and multiple regression Data Mining Course, Sharif University of Technology 42 What Tasks Can Data Mining Accomplish? (cont’d) • (4) Classification – Classification requires categorical target variable such as Income Bracket – Three values include “High”, “Middle”, “Low” – Data model examines records containing input fields and target field – Table shows several records from data set Subject Age Gender Occupation Income Bracket 001 47 F Software Engineer High 002 28 M Marketing Consultant Middle 003 35 M Unemployed Low … … … … … Data Mining Course, Sharif University of Technology 43 What Tasks Can Data Mining Accomplish? (cont’d) – Records of persons in data set used to “train” classification model – First, Model built from data records, where value of categorical target variable (Income Bracket) already known – Algorithm “first learns about” which combinations of input fields are associated with Income Bracket values in training set – For example, algorithm may determine that older females associated with high income – Next, trained model examines new records – Information regarding Income Bracket not available Data Mining Course, Sharif University of Technology 44 What Tasks Can Data Mining Accomplish? (cont’d) – Based on classifications in training set, new records classified – For example, 63-year old female professor might be classified in “High” income bracket – Classification Tasks in Business and Research: – Determine whether credit card transaction fraudulent – Assessing mortgage application to determine “good” or “bad” credit risk – Diagnosing whether particular disease present Data Mining Course, Sharif University of Technology 45 What Tasks Can Data Mining Accomplish? (cont’d) – Determine if will written by deceased, or fraudulently by someone else – Identify whether certain financial behavior represents terrorist threat – Scatter plot shows Na/K ratio against Age for 200 patients – For example, classify drug type to prescribe based on patient’s age and sodium/potassium ratio Data Mining Course, Sharif University of Technology 46 What Tasks Can Data Mining Accomplish? (cont’d) – Actual drug type prescribed symbolized by shade (light, medium, dark) of points – Suppose prescription of new patient based on this data set? – Prescribe which drug for young patient with high Na/K ratio? – Young patients plotted on left – High Na/K plotted on upper-half – Quadrant of graph shows light points – Recommended drug = Y (corresponds to light points) – Prescribe which drug for older patient with low Na/K ratio? – Lower-right half of graph shows patients prescribed different drug types Data Mining Course, Sharif University of Technology 47 What Tasks Can Data Mining Accomplish? (cont’d) – Definitive classification cannot be made – More information required to make decision – Examples show graphs are helpful for understanding twodimensional data – However, classification often requires many input attributes – More sophisticated methods of classification required – Commonly used algorithms for classification include k-Nearest Neighbor, Decision Trees, and Neural Networks Data Mining Course, Sharif University of Technology 48 What Tasks Can Data Mining Accomplish? (cont’d) • (5) Clustering – Refers to grouping records into classes of similar objects – Clustering algorithm seeks to segment data set into homogeneous subgroups – Where similarity of records in clusters maximized, and similarity to records outside clusters minimized – Target variable not specified – For example, Claritas, Inc. PRIZM software clusters demographic profiles for different geographic areas according to zip code Data Mining Course, Sharif University of Technology 49 What Tasks Can Data Mining Accomplish? (cont’d) – Table shows 62 distinct “lifestyle” types used by PRIZM 01 Blue Blood Estates 02 Winner's Circle 03 Executive Suites 04 Pools & Patios 05 Kids & Cul-de-Sacs 06 Urban Gold Coast 07 Money & Brains 08 Young Literati 09 American Dreams 10 Bohemian Mix 11 Second City Elite 12 Upward Bound 13 Gray Power 14 Country Squires 15 God's Country 16 Big Fish, Small Pond 17 Greenbelt Families 18 Young Influentials 19 New Empty Nests 20 Boomers & Babies 21 Suburban Sprawl 22 Blue-Chip Blues 23 Upstarts & Seniors 24 New Beginnings 25 Mobility Blues 26 Gray Collars 27 Urban Achievers 28 Big City Blend 29 Old Yankee Rows 30 Mid-City Mix 31 Latino America 32 Middleburg Managers 33 Boomtown Singles 34 Starter Families 35 Sunset City Blues 36 Towns & Gowns 37 New Homesteaders 38 Middle America 39 Red, White & Blues 40 Military Quarters 41 Big Sky Families 42 New Eco - topia 43 River City, USA 44 Shotguns & Pickups 45 Single City Blues 46 Hispanic Mix 47 Inner Cities 48 Smalltown Downtown 49 Hometown Retired 50 Family Scramble 51 Southside City 52 Golden Ponds 53 Rural Industria 54 Norma Rae-Ville 55 Mines & Mills 56 Agri - Business 57 Grain Belt 58 Blue Highways 59 Rustic Elders 60 Back Country Folks 61 Scrub Pine Flats 62 Hard Scrabble Data Mining Course, Sharif University of Technology 50 What Tasks Can Data Mining Accomplish? (cont’d) – What do the clusters mean? – According to PRIZM, Clusters for Beverly Hills, CA 90210 include: • • • • Cluster Cluster Cluster Cluster 01: 10: 02: 08: Blue Blood Estates Bohemian Mix Winner’s Circle Young Literati – Description of Cluster 01, “...’old money’ heirs that live in America’s wealthiest suburbs...accustomed to privilege and live luxuriously...” Data Mining Course, Sharif University of Technology 51 What Tasks Can Data Mining Accomplish? (cont’d) – Clustering Tasks in Business and Research: – Target marketing niche product for small business that does not have large marketing budget – For accounting purposes, segment financial behavior into benign and suspicious categories – Use as dimensionality-reduction tool for data set having several hundred inputs – Determine gene expression clusters, where many genes exhibit similar behavior or characteristics – Clustering often used as preliminary step in data mining – Resulting clusters used as input to different technique downstream, such as neural networks Data Mining Course, Sharif University of Technology 52 What Tasks Can Data Mining Accomplish? (cont’d) • (6) Association – Find out which attributes “go together” – Market Basket Analysis commonly used in business applications – Quantify relationships in the form of Rules IF antecedent THEN consequent – Rules measured using support and confidence – For example, discover which items in supermarket are purchased together – Thursday night 200 of 1,000 customers bought diapers, and of those buying diapers, 50 purchased beer – Association Rule: “IF buy diapers, THEN buy beer” – Support = 200/1,000 = 5%, and confidence = 50/200 = 25% Data Mining Course, Sharif University of Technology 53 What Tasks Can Data Mining Accomplish? (cont’d) – Association Tasks in Business and Research: – Investigating proportion of subscribers to cell phone plan responding positively to service upgrade offer – Predicting degradation in telecommunication networks – Discovering which items in supermarket purchased together – Determining proportion of cases where administering new drug exhibits serious side effects – Two commonly-used algorithms for generating association rules – A Priori and Generalized Rule Induction (GRI) Data Mining Course, Sharif University of Technology 54 Case Study 2 • Predicting Abnormal Stock Market Returns – Business Understanding – Alan Safer reports stock market trades by insiders have abnormal returns – Profits by outsiders can be increased, by using legal insider trading information – Safer attempts to predict abnormal stock price returns – Data Preparation – Rank of company insiders not considered important – Also, company insiders omitted where not involved in company decisions Data Mining Course, Sharif University of Technology 55 Case Study 2 – – – – (cont’d) Modeling Data split training = 80% and validation = 20% Neural Network model uncovered results: (a) Several groups had most predictable abnormal returns: • • • • Electronic equipment, excluding computer equipment Chemical products Transportation equipment Business services – (b) Predictions looking farther into future increased ability to predict unusual variations – (c) Abnormal stock returns of small companies easier to predict Data Mining Course, Sharif University of Technology 56 Case Study 2 (cont’d) – Evaluation – Multivariate Adaptive Regression Spline (MARS) also applied to data – Uncovered similar findings to Neural Network model – Confluence of results is powerful method of evaluating validity of model – This increases confidence in results – Deployment – Safer published findings in Intelligent Data Analysis Data Mining Course, Sharif University of Technology 57 Case Study 3 • Mining Association Rules from Legal Databases – Business Understanding – Ivkovic, Yearwood, and Stranieri mine association rules from large database of applicants for government-funded legal aide in Australia – Legal data highly unstructured – Goal is to improve delivery of legal services – Data Understanding – Data provided by Victoria Legal Aid (VLA) – Contains 380,000 applications with 300 attributes Data Mining Course, Sharif University of Technology 58 Case Study 3 (cont’d) – Domain experts consulted in effort to reduce dimensionality – Researchers selected seven of the most important inputs for inclusion in data set – Data Preparation – VLA data set relatively clean – VLA database administration system responsible for high-quality data – Modeling – Rules restricted to having both single antecedent and consequent – Many interesting and uninteresting rules uncovered Data Mining Course, Sharif University of Technology 59 Case Study 3 (cont’d) – Researchers adopted premise that interesting rules spawn interesting hypotheses – For example, possible reasons for rule “If place of birth = Vietnam, then law type = criminal law” include: • Vietnamese applicants applied for criminal law assistance only • Vietnamese applicants committed more crimes than other groups • Perhaps high proportion of males applied, and males more closely • associated with criminal activity Vietnamese didn’t have access to VLA promotional material – Researchers concluded first hypothesis most likely – Note intense human activity required for data mining process Data Mining Course, Sharif University of Technology 60 Case Study 3 (cont’d) – Evaluation – Three domain experts estimated confidence level of 144 rules – These estimates compared to confidence level reported by software – Deployment – Web-based application developed (WebAssociator) – Non-specialists able to access rule-building engine – Researchers suggest WebAssociator deployment to enhance judicial system – May identify unjust processes Data Mining Course, Sharif University of Technology 61 Case Study 4 • Predicting Corporate Bankruptcies using Decision Trees – Business Understanding – Recent economic crisis has spawned many corporate bankruptcies in East Asia – Sung, Chang, and Lee developing models predicting bankruptcies that maximize interpretability of results – Therefore, decision trees used for modeling – Data Understanding – Data included two groups of Korean companies – Those that went bankrupt 1991-1995 during stable period – Companies that went bankrupt during crisis years 1997-1998 Data Mining Course, Sharif University of Technology 62 Case Study 4 (cont’d) – – – – – – – – 29 firms identified, mostly manufacturing Financial data collected from Korean Stock Exchange Data Preparation 56 financial ratios identified through literature search 16 dropped due to duplication Measures included growth, profitability, etc. Modeling Separate decision tree models were applied under “normal” and “crisis” conditions – Normal-conditions rules uncovered: • If productivity of capital > 19.65, then predict non-bankrupt with 86% confidence Data Mining Course, Sharif University of Technology 63 Case Study 4 (cont’d) • If productivity of capital <= 19.65 and ratio of cash flow to total assets <= -5.65, then predict bankrupt with 84% confidence – Crisis-conditions rules uncovered: • If productivity of capital > 20.61, predict non-bankrupt with 95% • confidence If ratio of cash flow to liabilities > 2.54, predict non-bankrupt with 85% confidence – “Cash flow” and “productivity of capital” important predictors, regardless of economic conditions – Evaluation – Panel of domain experts concluded “productivity of capital” most important attribute for differentiating firms at risk – Domain experts verified results of decision tree Data Mining Course, Sharif University of Technology 64 Case Study 4 (cont’d) – Group confirmed results would generalize to population of Korean manufacturing firms – Discriminant analysis determined many of the 40 financial ratios were important predictors – Deployment – No specific deployment took place – However, financial institutions in Korea became more aware of important predictors of bankruptcy Data Mining Course, Sharif University of Technology 65 Case Study 5 • Profiling the Tourism Market using k-Means Clustering Analysis – Business Understanding – Hudson and Richie were interested in studying intra-province tourism behavior in Alberta, Canada – Goal was development of marketing campaign for tourism in Alberta (sponsored by Travel Alberta) – Models created in effort to quantify factors for choosing vacations in Alberta – Data Understanding – Data collected from 13,445 Albertans using phone survey in 1999 – Only 3,071/13,445 records included in modeling Data Mining Course, Sharif University of Technology 66 Case Study 5 (cont’d) – Data Preparation – One question asked respondents to indicate which of 13 factors most influenced their travel decisions – Factors included accommodations, weather conditions, etc. – Modeling – Between two and six clusters explored with k-Means – Five-cluster solution chosen with profile names: • • • • • Young Urban Outdoor Market Indoor Leisure Traveler Market Children-first Market Fair-weather-friends Market Older, Cost-conscious Traveler Market Data Mining Course, Sharif University of Technology 67 Case Study 5 (cont’d) – – – – – Evaluation Discriminant analysis verified “reality” of clusters Classified 93% correctly Deployment Findings resulted in launch of new campaign, “Alberta, Made to Order” – More than 80 projects launched – Travel Alberta found increase of 20% in number of Albertans considering Alberta “top-of-the-mind” travel destination Data Mining Course, Sharif University of Technology 68 Summary • Data mining: discovering interesting patterns from large amounts of data • A natural evolution of database technology, in great demand, with wide applications • A KDD process includes data cleaning, data integration, data selection, transformation, data mining, pattern evaluation, and knowledge presentation • Mining can be performed in a variety of information repositories • Data mining functionalities: characterization, discrimination, association, classification, clustering, outlier and trend analysis, etc. • Data Mining tasks • Case studies Data Mining Course, Sharif University of Technology 69 Readings • Chapter 1 Larose book • Chapters 1 Han and Kamber book • Intro to Data Mining and Knowledge Discovery article • DataMining-1.html • From Data Mining to Knowledge Discovery in Databases article • Data Mining an overview article Data Mining Course, Sharif University of Technology 70 A Brief History of Data Mining Society • 1989 IJCAI Workshop on Knowledge Discovery in Databases (Piatetsky-Shapiro) – Knowledge Discovery in Databases (G. Piatetsky-Shapiro and W. Frawley, 1991) • 1991-1994 Workshops on Knowledge Discovery in Databases – Advances in Knowledge Discovery and Data Mining (U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, 1996) • 1995-1998 International Conferences on Knowledge Discovery in Databases and Data Mining (KDD’95-98) – Journal of Data Mining and Knowledge Discovery (1997) • 1998 ACM SIGKDD, SIGKDD’1999-2001 conferences, and SIGKDD Explorations • More conferences on data mining – PAKDD (1997), PKDD (1997), SIAM-Data Mining (2001), (IEEE) ICDM (2001), etc. Data Mining Course, Sharif University of Technology 71 Where to Find References? • Data mining and KDD (SIGKDD: CDROM) – Conferences: ACM-SIGKDD, IEEE-ICDM, SIAM-DM, PKDD, PAKDD, etc. – Journal: Data Mining and Knowledge Discovery, KDD Explorations • Database systems (SIGMOD: CD ROM) – Conferences: ACM-SIGMOD, ACM-PODS, VLDB, IEEE-ICDE, EDBT, ICDT, DASFAA – Journals: ACM-TODS, IEEE-TKDE, JIIS, J. ACM, etc. • AI & Machine Learning – Conferences: Machine learning (ML), AAAI, IJCAI, COLT (Learning Theory), etc. – Journals: Machine Learning, Artificial Intelligence, etc. • Statistics – Conferences: Joint Stat. Meeting, etc. – Journals: Annals of statistics, etc. • Visualization – Conference proceedings: CHI, ACM-SIGGraph, etc. – Journals: IEEE Trans. visualization and computer graphics, etc. Data Mining Course, Sharif University of Technology 72 Recommended Reference Books • R. Agrawal, J. Han, and H. Mannila, Readings in Data Mining: A Database Perspective, Morgan Kaufmann (in preparation) • U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy. Advances in Knowledge Discovery and Data Mining. AAAI/MIT Press, 1996 • U. Fayyad, G. Grinstein, and A. Wierse, Information Visualization in Data Mining and Knowledge Discovery, Morgan Kaufmann, 2001 • J. Han and M. Kamber. Data Mining: Concepts and Techniques. Morgan Kaufmann, 2001 • D. J. Hand, H. Mannila, and P. Smyth, Principles of Data Mining, MIT Press, 2001 • T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer-Verlag, 2001 • T. M. Mitchell, Machine Learning, McGraw Hill, 1997 • G. Piatetsky-Shapiro and W. J. Frawley. Knowledge Discovery in Databases. AAAI/MIT Press, 1991 • S. M. Weiss and N. Indurkhya, Predictive Data Mining, Morgan Kaufmann, 1998 • I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, Morgan Kaufmann, 2001 Data Mining Course, Sharif University of Technology 73