Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Scalable Benchmarks and Kernels for Data Mining and Analytics Vipin Kumar University of Minnesota [email protected] www.cs.umn.edu/~kumar Joint work with Alok Choudhary and Gokhan Memik (Northwestern) and Michael Steinbach (University of Minnesota) Research funded by NSF Need for High Performance Data Mining Today’s digital society has seen enormous data growth in both commercial and scientific databases Biomedical Data Data Mining is becoming a commonly used tool to extract information from large and complex datasets Advances in computing capabilities and technological innovation needed to harvest the available wealth of data Homeland Security Internet Geo-spatial data Sensor Networks Computational Simulations Data Mining for Climate Data NASA ESE questions: How is the global Earth system changing? What are the primary forcings? How does Earth system respond to natural & human-induced changes? What are the consequences of changes in the Earth system? How well can we predict future changes? Global snapshots of values for a number of variables on land surfaces or water NPP . Pressure NPP . Pressure . Precipitation Precipitation SST SST Latitude grid cell Longitude NASA DATA MINING REVEALS A NEW HISTORY OF NATURAL DISASTERS NASA is using satellite data to paint a detailed global picture of the interplay among natural disasters, human activities and the rise of carbon dioxide in the Earth's atmosphere during the past 20 years…. http://www.nasa.gov/centers/ames/news/releases/2003/03_51AR.html Detection of Ecosystem Disturbances: This interactive module displays the locations on the earth surface where significant disturbance events have been detected. Disturbance Viewer Time zone High Resolution EOS Data: • EOS satellites provide high resolution measurements • Finer spatial grids • 1 km 1 km grid produces 694,315,008 data points • Going from 0.5º 0.5º degree data to 1 km 1 km data results in a 2500fold increase in the data size • More frequent measurements • Multiple instruments • High resolution data allows us to answer more detailed questions: • Detecting patterns such as trajectories, fronts, and movements of regions with uniform properties • Finding relationships between leaf area index (LAI) and topography of a river drainage basin • Finding relationships between fire frequency and elevation as well as topographic position • Leads to substantially high computational and memory requirements Data Mining for Cyber Security 120000 • Due to proliferation of Internet, more and more organizations are becoming vulnerable to sophisticated cyber attacks Traditional Intrusion Detection Systems (IDS) have well-known limitations • – – – • Too many false alarms Unable to detect sophisticated and novel attacks Unable to detect insider abuse/ policy abuse 100000 80000 60000 40000 20000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Data Mining is well suited to address these challenges MINDS – Minnesota Intrusion Detection System Large Scale Data Analysis is needed for • Correlation of suspicious events across network sites – • Incorporated into Interrogator architecture at ARL Center for Intrusion Monitoring and Protection (CIMP) • • Helps analyze data from multiple sensors at DoD sites around the country Routinely detects Insider Abuse / Policy Violations / Worms / Scans • Helps detect sophisticated attacks not identifiable by single site analyses Analysis of long term data (months/years) – Uncover suspicious stealth activities (e.g. insiders leaking/modifying information) Data Mining for Biomedical Informatics Recent technological advances are helping to generate large amounts of both medical and genomic data • High-throughput experiments/techniques - Gene and protein sequences - Gene-expression data - Biological networks and phylogenetic profiles • Electronic Medical Records - IBM-Mayo clinic partnership has created a DB of 5 million patients - NIH Roadmap Data mining offers potential solution for analysis of large-scale data • • • Automated analysis of patients history for customized treatment Design of drugs/chemicals Prediction of the functions of anonymous genes Protein Interaction Network Role of Benchmarks in Architecture Design Benchmarks guide the development of new processor architectures in addition to measuring the relative performance of different systems • SPEC: General purpose architecture (“Advances in the microprocessor industry would not have been possible without the SPEC benchmarks” - David Patterson) • TPC: Database Systems • SPLASH: Parallel machine architectures • Mediabench: Media and Communication Processors • NetBench: Network/Embedded processors Do We Need Benchmarks Specific to Data Mining? Performance metrics of several benchmarks gathered from Vtune • Cache miss ratios, Bus usage, Page faults etc. 11 10 9 8 7 6 5 4 3 2 1 0 SPEC INT SPEC FP MediaBench TPC-H MineBench gcc bzip2 gzip mcf twolf vortex vpr parser apsi art equake lucas mesa mgrid swim wupwise rawcaudio epic encode cjpeg mpeg2 pegwit gs toast Q17 Q3 Q4 Q6 apriori bayesian birch eclat hop scalparc kMeans fuzzy rsearch semphy snp genenet svm-rfe Cluster Number Benchmark applications were grouped using Kohenen clustering to spot trends: Reference: [Pisharath J., Zambreno J., Ozisikyilmaz B., Choudhary A., 2006] Recently funded NSF project: Scalable Benchmarks, Software and Data for Data Mining, Analytics and Scientific Discoveries PIs: A. Choudhary and Gokhan Memik (NW) , V. Kumar and M. Steinbach (UM) Motivate the development of new processor architectures and system design for data mining Scalability (data-level, processor) Motivate the implementation of more sophisticated data mining algorithms that can work with the constraints imposed by current architecture designs Improvement the productivity of scientists and engineers using data mining application in a wide variety of domains Performance (execution time, cache behavior, …) Profiling Types of storage (memory, disks, …) Types of data (streaming, file I/O) Goal: Establish a comprehensive benchmarking suite for data mining applications. Types of applications (scientific, bioinformatics, security, …) Data Mining Tasks … Data 10 Milk Tid Refund Marital Status Taxable Income Cheat 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes 11 No Married 60K No 12 Yes Divorced 220K No 13 No Single 85K Yes 14 No Married 75K No 15 No Single 90K Yes 60K Key Data Mining Algorithms Clustering • • • K-means, EM, SOM Single link / Group Average hierarchical clustering DBSCAN, SNN Classification • • • Bayes SVM Decision trees, Rule based systems Association Rule Mining • Apriori, FP-Growth Anomaly Detection • • • Statistical methods Distance-based Clustering-based Preprocessing • SVD, PCA Major Data Mining Kernels Counting • Given a set of data records, count types of different categories to build a contingency table • Count the occurrence of a set of items in a set of transactions Pairwise computations • Given a set of data records, perform pairwise distane/similarity computations Linear Algebra operations • SVD, PCA General Characteristics of Data Mining Algorithms Dense/Sparse data Hash table / Hash tree Linked Lists Iterative nature Data often too large to fit in main memory • Spatial locality is critical Constructing a Decision Tree Employed 10 10 1 Yes Graduate # years at present address 5 2 Yes High School 2 No 3 No Undergrad 1 No 4 Yes High School 10 Yes 5 Yes Graduate 2 No 6 No High School 2 No 7 Yes Undergrad 3 No 8 Yes Graduate 8 Yes 9 Yes High School 4 Yes 10 No Graduate 1 No Tid Employed Level of Education Credit Worthy Yes Yes Worthy: 4 Not Worthy: 3 No Education Worthy: 0 Not Worthy: 3 Graduate Worthy: 2 Not Worthy: 2 Not Worthy Worthy Key Computation Employed = Yes 4 3 Employed = No 0 3 High School/ Undergrad Worthy: 2 Not Worthy: 4 Constructing a Decision Tree 10 Tid Employed 10 Tid Employed Level of Education # years at Credit present Worthy address 5 Yes 1 Yes Graduate 2 Yes High School 2 No 3 No Undergrad 1 No 4 Yes High School 10 Yes 5 Yes Graduate 2 No 6 No High School 2 No 7 Yes Undergrad 3 No Employed = Yes Level of Education # years at Credit present Worthy address 5 Yes 1 Yes Graduate 2 Yes High School 2 No 4 Yes High School 10 Yes 5 Yes Graduate 2 No 7 Yes Undergrad 3 No 8 Yes Graduate 8 Yes 9 Yes High School 4 Yes 10 8 Yes Graduate 8 Yes 9 Yes High School 4 Yes 10 No Graduate 1 No Tid Employed Employed = No # years at Credit present Worthy address Undergrad 1 No Level of Education 3 No 6 No High School 2 No 10 No Graduate 1 No Constructing a Decision Tree in Parallel m categorical attributes n records Yes No Yes No Yes No Worthy Not Worthy 4 0 3 3 Worthy Not Worthy 2 1 5 2 Worthy Not Worthy 6 1 1 2 Partitioning of data only – global reduction per node is required – large number of classification tree nodes gives high communication cost Constructing a Decision Tree in Parallel Partitioning of classification tree nodes 10,000 training records 7,000 records 2,000 3,000 records 5,000 2,000 1,000 – natural concurrency – load imbalance – the amount of work associated with each node varies – limited concurrency on the upper portion of the tree – child nodes use the same data as used by parent node – loss of locality – high data movement cost Speedup Comparison of the Three Parallel Algorithms Data set used in SLIQ paper (Ref: Mehta, Agrawal and Rissanen, 1996) IBM SP2 with 128 processors 0.8 million examples hybrid Data partitioning Tree partitioning 1.6 million examples hybrid Data partitioning Tree partitioning Dynamic load balancing inspired by parallel sparse Cholesky factorization and parallel tree search Speedup of the Hybrid Algorithm with Different Size Data Sets Hash Table Access • Some efficient decision tree algorithms require random access to large data structures. • Example: SPRINT (Ref: Shafer, Agrawal, Mehta, 1996) Hash Table 10 ID 0 Income ID 2 25K Age ID 0 25 Left/ Right Left Processor 2 P0 28K 5 31 1 Left 8 30K 8 33 2 Right 4 30K 1 37 3 Right Processor 5 P1 35K 3 41 4 Right 1 50K 6 52 5 Left 3 52K 4 55 6 Right Level of Education Tid Employed 1 Yes Graduate 2 Yes High School 7 10 55K 70K 10 7 60 0 61 7 Left 8 Left 2 No 3 No Undergrad 1 No 4 Yes High School 10 Yes 5 Yes Graduate 2 No 6 No High School 2 No 7 Yes Undergrad 3 No 8 Yes Graduate 8 Yes 10 No Graduate 1 No Left Right 10 Processor 6 P2 # years at Credit present Worthy address 5 Yes 10 Tid Employed Level of Education # years at Credit present Worthy address 2 No 5 Yes Graduate 6 No High School 2 7 Yes Undergrad 8 Yes Graduate 10 No Graduate Tid Employed # years at Credit present Worthy address High School 2 No Level of Education 6 No No 7 Yes Undergrad 3 No 3 No 8 Yes Graduate 8 Yes 8 Yes 10 No Graduate 1 No 1 No 10 Storing the entire has table on one processor makes the algorithm unscalable ScalParC (Ref: Joshi, Karypis, Kumar, 1998) ScalParC is a scalable parallel decision tree construction algorithm • Scales to large number of processors • Scales to large training sets ScalParC is memory efficient • The hash-table is distributed among the processors ScalParC performs minimum amount of communication This ScalParC Design is Inspired by.. Communication Structure of Parallel Sparse Matrix-Vector Algorithms Processor P0 Processor P1 Processor P2 Hash Table Entries Parallel Runtime (Ref: Joshi, Karypis, Kumar, 1998) Runtime (seconds) 120 100 0.2M 0.4M 0.8M 1.6M 3.2M 6.4M 80 60 40 20 0 0 50 100 Processors 128 Processor Cray T3D 150 Computing Association Patterns 1. Market-basket transactions TID Items 1 Bread, Diaper,Milk 2 3 4 5 Beer, Diaper, Bread, Eggs Beer, Coke, Diaper, Milk Beer, Bread, Diaper, Milk Coke, Bread, Diaper, Milk 3. Generate association rules 2. Find item combinations (itemsets) that occur frequently in data Item Combination Bread Coke Milk … Bread & Coke Bread & Milk … Bread & Milk & Diaper … Count 4 2 4 … 1 3 … 3 … {Diaper, Milk } {Beer} {Bread} {Diaper} Counting Candidates Frequent Itemsets are found by counting candidates Simple way: • Search for each candidate in each transaction Transactions Candidates Count N ABC D ACE BCD ABD E BCE BD M AB AC A AD D A AE E BC BC BD BD ABE ABE BCD BCD ABDE AABBCDDEE ABCDE 2 0 1 1 2 0 1 2 0 0 1 2 0 1 3 0 1 4 0 0 2 0 1 2 0 0 1 0 0 0 Naïve approach requires O(NM) comparisons Reduce the number of comparisons (NM) by using hash tables to store the candidate itemsets Parallel Association Rules: Scaleup Results (100K,0.25%) (Ref: Han, Karypis, and Kumar, 2000) DD (Agrawal & Shafer, 1996) Efficient implementation of collective communication IDD (Han, Karypis, Kumar, 2000) Dynamic restructuring of computation HD (Han, Karypis, Kumar, 2000) Candidates for MineBench Category Algorithms Description PCA Preprocessing Principal component analysis ABB LVF Normalization ScalParC Preprocessing Preprocessing Preprocessing Predictive Modeling Naïve Bayesian Predictive Modeling RIPPER SVMlight K-means Bisecting K-means Fuzzy K-means EM Clustering MAFIA(N) BIRCH AHC DBSCAN HOP LOF Outlier Detection Predictive Modeling Predictive Modeling Clustering Automatic Branch and Bound A probabilistic feature selection algorithm Variable transformation Decision tree classifier Statistical classifier based on class conditional independence Rule-based predictive modeling Support Vector Machines Partitioning method Clustering Partitioning method Clustering Clustering Clustering Clustering Clustering Clustering Clustering Anomaly Detection Anomaly Detection Apriori ARM MAFIA(C) ARM Eclat ARM FP-growth ARM Fuzzy logic based K-means Partitioning method Multidimensional Clustering Hierarchical method Agglomerative Hierarchical Clustering Density-based method Density-based method Local Outlier Factor Distance-based outlier detection Horizontal database, level-wise mining based on Apriori property Maximal frequent itemset mining Vertical database, break large search space into equivalence classes Encodes database into a compact FP-tree Lang. C/C++/ FORT. C/C++ C/C++ C/C++ C Parallel C++ N C/C++ C/C++ C Y N Y C Y C C/C++ C C++ C/C++ C/C++ C C/C++ C/C++ Y Y Y N N Y Y Y Y C/C++ Y C/C++ N C++ N C/C++ N Y N N Y Y Analysis of Benchmark Algorithms Explore the bottlenecks associated with the current general purpose sequential and parallel machines Explore how different architectural features impact the performance of data mining algorithms Preliminary Evaluation of Some Sample Data Sets Example small (S), medium (M), and large (L) data set Dataset Small Medium Large Classification DB Size(MB) Parameter F26-A32-D125K 27 F26-A32-D250K 54 F26-A64-D250K 108 Association Rule Mining (ARM) Parameter DB Size(MB) T10-I4-D1000K 47 T20-I6-D2000K 175 T20-I6-D4000K 350 Execution time for some algorithms in the MineBench suite. Program Data set = S Data set = M Data set = L P1 P4 P8 P1 P4 P8 P1 P4 P8 HOP 6.3 1.8 1.2 52.7 27.4 18.7 435.3 128.0 81.5 K-means 5.7 2.0 1.3 12.9 3.3 2.6 - - - Fuzzy K-means BIRCH ScalParC Bayesian Apriori Eclat 164.1 3.5 51.0 12.6 6.1 11.8 54.6 13.5 3.0 - 26.4 10.4 2.6 - 146.8 31.7 110.6 25.1 102.7 81.5 42.7 28.5 38.6 - 27.1 21.6 30.5 - 172.6 225.9 51.5 200.2 127.8 56.2 72.6 - 36.5 63.0 - Reference: [Liu Y., Pisharath J., Liao W., Memik G., Choudhary A., Dubey P., 2004] Designing Efficient Kernels for Data Mining Understanding of the bottlenecks in executing DM algorithms on current architectures will help design new, more efficient algorithms Focus will be on design frequently used kernels that dominates the execution time of most DM algorithms Application kMeans Both sequential and parallel versions will be developed Top 3 Kernels (%) Sum Kernel 1 (%) Kernel 2 (%) Kernel 3 (%) % distance(68%) center(21%) minDist(10%) 99 Fuzzy kMeans center(58%) distance(39%) fuzzySum(1%) 98 BIRCH HOP distance(54%) variance(22%) redist.(10%) 86 density(39%) search(30%) 92 gather(23%) Naïve Bayesian probCal(49) variance(38%) dataRead(10%) 97 ScalParC Apriori Eclat classify(37%) giniCalc(36%) compare(24%) 97 subset(58%) dataRead(14%) increment(8%) 80 intersect(39%) addClass(23%) invertC (10%) 72 Frequency of Kernel Operations in Representative Applications Reference: [Pisharath J., Zambreno J., Ozisikyilmaz B., Choudhary A., 2006] Conclusions Data mining applications are becoming increasingly important Current systems design approach not adequate for DM applications MineBench – a new benchmark suite which encompasses many algorithms found in data mining Initial findings: • Data mining applications are unique in terms of performance characteristics • There exists much room for optimization with regards to data mining workloads Bibliography • • • Introduction to Data Mining, Pang-Ning Tan, Michael Steinbach, Vipin Kumar, Addison-Wesley April 2005 Introduction to Parallel Computing, (Second Edition) by Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar. Addison-Wesley, 2003 Data Mining for Scientific and Engineering Applications, edited by R. Grossman, C. Kamath, W. P. Kegelmeyer, V. Kumar, and R. Namburu, Kluwer Academic Publishers, 2001 J. Han, R. B. Altman, V. Kumar, H. Mannila, and D. Pregibon, "Emerging Scientific Applications in Data Mining", Communications of the ACM Volume 45, Number 8, pp 54-58, August 2002 C. Potter, P. Tan, M. Steinbach, S. Klooster, V. Kumar, R. Myneni, V. Genovese, Major Disturbance Events in Terrestrial Ecosystems Detected using global Satellite Data Sets, Global Change Biology 9 (7), 1005-1021, 2003 Vipin Kumar, “Parallel and Distributed Computing for Cyber Security". An article based on the keynote talk by the author at 17th International Conference on Parallel and Distributed Computing Systems (PDCS-2004). DS Online Journal, OLUME 6, NUMBER 10, October 2005 Ying Liu, Jayaprakash Pisharath, Wei-keng Liao, Gokhan Memik, Alok Choudhary, and Pradeep Dubey. Performance Evaluation and Characterization of Scalable Data Mining Algorithms. In Proceedings of the 16th International Conference on Parallel and Distributed Computing and Systems (PDCS), November 2004. Joseph Zambreno, Berkin Ozisikyilmaz, Jayaprakash Pisharath, Gokhan Memik, and Alok Choudhary. Performance Characterization of Data Mining Applications using MineBench. In Proceedings of the 9th Workshop on Computer Architecture Evaluation using Commercial Workloads (CAECW-9), February 2006. Jayaprakash Pisharath, Joseph Zambreno, Berkin Ozisikyilmaz, and Alok Choudhary. Accelerating Data Mining Workloads: Current Approaches and Future Challenges in System Architecture Design. In Proceedings of the 9th International Workshop on High Performance and Distributed Mining (HPDM), April 2006