Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Temporal Relationship Among Clusters for Data Streams Margaret H. Dunham, Michael Hahsler, Doug Raiford Students: Yu Meng, Donya Quick, Jie Huang, Charlie Isaksson, Mallik Kotamarti CSE Department Southern Methodist University Dallas, Texas 75275 [email protected] This material is based upon work supported by the National Science Foundation under Grant No IIS-0948893. 10/26/09, Wilfrid Laurier University 1 Objectives/Outline Traditional Clustering of Data Streams Ignores one of the most Salient Features of Streams: Ordering Introduction Background TRAC-DS TRAC-DS Applications Conclusions/Future Work 10/26/09, Wilfrid Laurier University 2 Objectives/Outline Introduction Stream Data Motivation Background TRAC-DS TRAC-DS Applications Conclusions/Future Work 10/26/09, Wilfrid Laurier University 3 Stream Data A growing number of applications generate streams of data. Computer network monitoring data Call detail records in telecommunications Highway transportation traffic data Online web purchase log records Sensor network data Stock exchange, transactions in retail chains, ATM operations in banks, credit card transactions. Clustering techniques play a key role in modeling and analyzing this data. 10/26/09, Wilfrid Laurier University 4 Stream Data Format Events arriving in a stream At any time, t, we can view the state of the problem as represented by a vector of n numeric values: Vt = <S1t, S2t, ..., Snt> V1 S1 S2 … Sn S11 S21 … Sn1 V2 S12 S22 … Sn2 … … … … … Vq S1q S2q … Snq Time 10/26/09, Wilfrid Laurier University 5 Data Stream Modeling Single pass: Each record is examined at most once Bounded storage: Limited Memory for storing synopsis Real-time: Per record processing time must be low Summarization (Synopsis )of data Use data NOT SAMPLE Temporal and Spatial Dynamic Continuous (infinite stream) Learn Forget Sublinear growth rate - Clustering 10/26/09, Wilfrid Laurier University 6 6 Traditional Clustering 10/26/09, Wilfrid Laurier University 7 TRAC-DS 10/26/09, Wilfrid Laurier University 8 Motivation Temporal Ordering is a major feature of stream data. Many stream applications depend on this ordering Prediction of future values Anomaly (rare event) detection Concept drift 10/26/09, Wilfrid Laurier University 9 Objectives/Outline Introduction Background Clustering Stream Data Extensible Markov Model - EMM TRAC-DS TRAC-DS Applications Conclusions/Future Work 10/26/09, Wilfrid Laurier University 10 Stream Clustering Requirements Dynamic updating of the clusters Identify outliers Barbara [2]: compactness fast incremental processing 10/26/09, Wilfrid Laurier University 11 Stream Clustering Algorithms LOCALSEARCH [4] Partitions stream into segments Clusters each segment individually by solving the kmedians problem Iteratively reclusters the resulting centers CluStream [1] Micro-clusters represented by summary statistics. Micro-clusters are handled online Micro-clusters merged offline MONIC [13] Evolution of clusters over time Cluster transitions over time 10/26/09, Wilfrid Laurier University 12 MM A first order Markov Chain is a finite or countably infinite sequence of events {E1, E2, … } over discrete time points, where Pij = P(Ej | Ei), and at any time the future behavior of the process is based solely on the current state A Markov Model (MM) is a graph with m vertices or states, S, and directed arcs, A, such that: S ={N1,N2, …, Nm}, and A = {Lij | i 1, 2, …, m, j 1, 2, …, m} and Each arc, Lij = <Ni,Nj> is labeled with a transition probability Pij = P(Nj | Ni). 10/26/09, Wilfrid Laurier University 13 Extensible Markov Model (EMM) Time Varying Discrete First Order Markov Model Nodes are clusters of real world states. Learning continues during application phase. Learning: Transition probabilities between states(clusters) State labels (Cluster summary) State are modified as clusters are 10/26/09, Wilfrid Laurier University 14 EMM for TRAC-DS Modeling <18,10,3,3,1,0,0> <17,10,2,3,1,0,0> <16,9,2,3,1,0,0> <14,8,2,3,1,0,0> 2/3 2/3 2/21 2/3 1/1 1/2 1/2 N3 N1 1/3 N2 1/1 1/2 1/1 <14,8,2,3,0,0,0> <18,10,3,3,1,1,0.> 10/26/09, Wilfrid Laurier University 15 Objectives/Outline Introduction Background TRAC-DS Definition Relationship to Traditional Clustering Operations TRAC-DS Applications Conclusions/Future Work 10/26/09, Wilfrid Laurier University 16 TRAC-DS NOTE TRAC-DS is not: Another stream clustering algorithm TRAC-DS is: A new way of looking at clustering Built on top of an existing clustering algorithm TRAC-DS may be used with any stream clustering algorithm 10/26/09, Wilfrid Laurier University 17 TRAC-DS Overview 10/26/09, Wilfrid Laurier University 18 Data Stream Clustering At each point in time a data stream clustering ζ is a partitioning of D', the data seen thus far. Instead of the whole partitions C1, C2,..., Ck only synopses Cc1,Cc2,...,Cck are available and k is allowed to change over time. The summaries Cci with i =1, 2,...,k typically contain information about the size, distribution and location of the data points in Ci. 10/26/09, Wilfrid Laurier University 19 TRAC-DS Definition Given a data stream clustering ζ, a temporal relationship among clusters (TRAC-DS) overlays a data stream clustering ζ with a EMM M, in such a way that the following are satisfied: (1) There is a one-to-one correspondence between the clusters in ζ and the states S in M. (2) A transition aij in the EMM M represents the probability that given a data point in cluster i, the next data point in the data stream will belong to cluster j with i; j = 1; 2; : : : ; k. (3) The EMM M is created online together with the data stream clustering 10/26/09, Wilfrid Laurier University 20 Clustering Operations A clustering operation is a function q : ζ × x → ζ which is used by the data stream clustering algorithm to update the clustering ζ given some additional information x which either is a new data point or other information (e.g., the number of the cluster to be deleted to be simplified the clustering). 10/26/09, Wilfrid Laurier University 21 TRAC-DS Operations A TRAC-DS operation is a function r : M × sc × y → M × sc that updates the temporal relationship among clusters represented by the EMM M with states S given a current state sc ∈ S and additional information y and returns an updated EMM and possibly a new current state. In order to be able to dynamically update the EMM M we need to store a transition count matrix C. The count cij in C contains the number of times we observed a new point being assigned by the clustering algorithm to cluster i followed by a point being assigned to cluster j. 10/26/09, Wilfrid Laurier University 22 Stream Clustering Operations * qassign point(ζ,x): Assigns the new data point x to an existing cluster. qnew cluster(ζ,x): Create a new cluster. qremove cluster(ζ,x): Removes a cluster. Here x is the cluster, i, to be removed. In this case the associated summary Cci is removed from ζ and k is decremented by one. qmerge clusters(ζ,x): Merges two clusters. qfade clusters(ζ,x): Fades the cluster structure. qsplit clusters(ζ,x): Splits a cluster. * Inspired by MONIC [?] 10/26/09, Wilfrid Laurier University 23 TRAC-DS Operations rassign point(M,sc,y): Assigns the new data point to the state representing an existing cluster rnew cluster(M,sc,y): Create a state for a new cluster. rremove cluster(M,sc,y): Removes state. rmerge clusters(M,sc,y): Merges two states. rfade clusters(M,sc,y): Fades the transition probabilities using an exponential decay f(t)=2−λt rsplit clusters(M,sc,y): Splits states. Y clustering operations. 10/26/09, Wilfrid Laurier University 24 TRAC-DS Example 10/26/09, Wilfrid Laurier University 25 TRAC-DS Advantages Dynamic Flexible – Use any Clustering Algorithm Supports and clustering operations Scalable Merges Clustering & Markov Modeling 10/26/09, Wilfrid Laurier University 26 Objectives/Outline Introduction Background: TRAC-DS TRAC-DS Applications Anomaly Detection Bioinformatics Conclusions/Future Work 10/26/09, Wilfrid Laurier University 27 What is Anomaly in Stream Data? Rare - Anomalous – Surprising Out of the ordinary Not outlier detection No knowledge of data distribution Data is not static Must take temporal and spatial values into account May be interested in sequence of events Ex: Snow in upstate New York is not an anomaly Snow in upstate New York in June is rare Rare events may change over time 10/26/09, Wilfrid Laurier University 28 TRAC-DS Approach to Detect Anomalies By learning what is normal, the model can predict what is not Normal is based on likelihood of occurrence Use TRAC-DS to build clusters and behavior between clusters We view a rare event as: Unusual event Transition between events states which does not frequently occur. Continue learning 10/26/09, Wilfrid Laurier University 29 Determining Rare Occurrence Frequency (OFi) of an EMM state Si is normalized count of state: OFi ni / ni i Normalized Transition Probability (NTPmn), from one state, Sm, to another, Sn, is a normalized transition Count: NTPm, n (Cm, n) /( ni ) i 10/26/09, Wilfrid Laurier University 30 Datasets/Anomalies MnDot – Minnesota Department of Transportation Automobile Accident Ouse and Serwent – River flow data from England Flood Drought KDD Cup 1999 & 2000 http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html Intrusion Cisco VoIP – VoIP traffic data obtained at Cisco Unusual Phone Call 10/26/09, Wilfrid Laurier University 31 EMM Sublinear Growth Servent Data 10/26/09, Wilfrid Laurier University 32 TRAC-DS River Prediction 8 7 Water Level (m) 6 5 4 3 2 1 0 1 48 95 142 189 236 283 330 377 424 471 518 565 612 659 Input Time Series RLF Prediction 10/26/09, Wilfrid Laurier University EMM Prediction Observed 33 TRAC-DA Rare Event Detection Detected unusual weekend traffic pattern Weekdays Weekend 10/26/09, Wilfrid Laurier University Minnesota DOT Traffic Data 34 TRAC-DS Intrusion Detection DARPA 1999/2000 Synthetic Dataset MIT Lincoln Lab The DARPA 1999 dataset which is free of attacks for two weeks (1st week and 3rd week) is used as training data DARPA 2000 dataset which contains DDoS attacks is used a test data. 10/26/09, Wilfrid Laurier University 35 Table 8. EMM detection and false positive rates. TRAC-DS Intrusion Detection Thresh old DARPA 1999, and 2000 Detection False Positive Rate Rate 0.9 6% 94% 0.8 20% 80% 0.7 50% 50% 0.6 100% 0% 10/26/09, Wilfrid Laurier University 36 TRAC-DS & Bioinformatics Analysis DNA/RNA Sequences Applications: Classification Differentiation 16s RNA 1542 nt rRNA Highly conserved across species miRNA Short (20-25nt) sequence of noncoding RNA Known since 1993 but significance not widely appreciated until 2001 Impact / Prevent translation of mRNA 10/17/06 37 First – Convert Sequence to NSV acgtgcacgtaactgattccggaaccaaatgtgcccacgtcga Moving Window Pos 0-8 Pos 1-9 A 2 1 C 3 3 G 3 3 T 1 2 4 2 1 … Pos 34-42 2 10/17/06 38 Next – Apply TRAC-DS 10/26/09, Wilfrid Laurier University 39 TRAC-DS Predictionwith miRNA Positive Data Model Cutoff Probability = 0.3 False Positive Rate = 0% True Positive Rate = 66% Test results could be improved by meta classifiers combining multiple positive and negative classifiers together. 10/17/06 41 Profile EMMs •Examples of three different Profile EMMs constructed for 16S data from 3 different bacteria families 10/26/09, Wilfrid Laurier University 42 Profile EMMs for Organism Classification 10/26/09, Wilfrid Laurier University 43 16S Classification Accuracy Classification accuracy using different scoring metrics on 16S rRNA data from NCBI. We learned 31 classification models (at the phylogenetic class level) from 98 organisms and tested with 23 randomly chosen organisms. The Profile EMM approach was able to achieve classification of more than 90% after tuning the resolution settings. 10/26/09, Wilfrid Laurier University 44 TRAC-DS and Bioinformatics Efficient Alignment free sequence analysis Clustering reduces size of model Flexible Any sequence Applicability to Metagenomics Scoring based on similarity between EMMs or EMM and input sequence Applications Classification Differentiation 10/26/09, Wilfrid Laurier University 45 Objectives/Outline Introduction Background TRAC-DS TRAC-DS Applications Conclusions/Future Work 10/26/09, Wilfrid Laurier University 46 TRAC-DS Ongoing/Future Create online tool suite Improve TRAC algorithms: Aging Delete state Merge states Split states Apply to Image Recognition Bioinformatics Build Profile EMM database of NCBI 16S Bacteria Data Perform classification using Metagenomic Data collected from Yellowstone National Park 10/26/09, Wilfrid Laurier University 47 10/26/09, Wilfrid Laurier University 48 Bibliography 1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11) 12) 13) C. C. Aggarwal, J. Han, J. Wang, and P. S. Yu. A framework for clustering evolving data streams. Proceedings of the International Conference on Very Large Data Bases (VLDB), pp 81-92, 2003. D. Barbara, “Requirements for clustering data streams,” SIGKDD Explorations, Vol 3, No 2, pp 23-27, 2002. Margaret H. Dunham, Donya Quick, Yuhang Wang, Monnie McGee, Jim Waddle, “Visualization of DNA/RNA Structure using Temporal CGRs,”Proceedings of the IEEE 6th Symposium on Bioinformatics & Bioengineering (BIBE06), October 16-18, 2006, Washington D.C. ,pp 171-178. S. Guha, A. Meyerson, N. Mishra, R. Motwani, and L. O'Callaghan, “Clustering data streams: Theory and practice,” IEEE Transactions on Knowledge and Data Engineering, Vol 15, No 3, pp 515-528, 2003. Michael Hahsler and Margaret H. Dunham, “TRACDS: Temporal Relationship Among Clusters for Data Streams,” October 2009, submitted to SIAM International Conference on Data Mining. Jie Huang, Yu Meng, and Margaret H. Dunham, “Extensible Markov Model,” Proceedings IEEE ICDM Conference, November 2004, pp 371374. Charlie Isaksson, Yu Meng, and Margaret H. Dunham, “Risk Leveling of Network Traffic Anomalies,” International Journal of Computer Science and Network Security, Vol 6, No 6, June 2006, pp 258-265. Charlie Isaksson and Margaret H. Dunham, “A Comparative Study of Outlier Detection,” July 2009, Proceedings of the IEEE MLDM Conference, pp 440-453. Mallik Kotamarti, Douglas W. Raiford, M. L. Raymer, and Margaret H. Dunham, “A Data Mining Approach to Predicting Phylum for Microbial Organisms Using Genome-Wide Sequence Data,” Proceedings of the IEEE Ninth International Conference on Bioinformatics and Bioengineering, pp 161-167, June 22-24 2009. Yu Meng and Margaret H. Dunham, “Efficient Mining of Emerging Events in a Dynamic Spatiotemporal,” Proceedings of the IEEE PAKDD Conference, April 2006, Singapore. (Also in Lecture Notes in Computer Science, Vol 3918, 2006, Springer Berlin/Heidelberg, pp 750-754.) Yu Meng and Margaret H. Dunham, “Mining Developing Trends of Dynamic Spatiotemporal Data Streams,” Journal of Computers, Vol 1, No 3, June 2006, pp 43-50. MIT Lincoln Laboratory.: DARPA Intrusion Detection Evaluation. http://www.ll.mit.edu/mission/communications/ist/corpora/ideval/index.html, (2008) M. Spiliopoulou, I. Ntoutsi, Y. Theodoridis, and R. Schult. MONIC: Modeling and monitoring cluster transitions. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, Philadelphia, PA, USA, pages 706–711, 2006. 10/26/09, Wilfrid Laurier University 49