Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Vertical Set Inner Product (VSIP) Technology with Predicate-trees William Perrizo Computer Science Department North Dakota State University Fargo, ND, 58105, USA [email protected] ABSTRACT In this paper, a Set Inner Product construct is used to identify high quality centroids for partitioning clustering methods such as kmeans or k-medoids clustering (or as a very fast clustering method in and of itself). A strong advantage is that number of centroids (k) need not be pre-specified, but is determined effectively within the algorithm. The method can also be used to identify outliers. The method is fast and scales well to very large data sets. The method applies to vertical data sets and uses vertical structures called Predicate-trees or P-trees. We show that the method can identify high-quality centroids (which may need no further refinement) and outliers. It is also shown that the method is fast and scalable. General Terms Algorithms, Performance. Keywords Clustering, Vertical Set Inner Products, P-tree. (n is large); the third is that the initial points are chosen randomly. In case of random selection, if the points selected are far away from the mean or the real centroid points, both the quality of the clusters and efficiency of the process deteriorate significantly (t is large). In this paper, we introduce the concept of the Set Inner Product to cope with these problems. Our new approach is founded on the vertical construction of the Set Inner Product. In our approach, through construction of the Set Inner Product, high quality centroids are identified for the partitioning clustering methods. Therefore, users do need to pre-define k ahead at all. The second advantage of our method is that the construct of the Set Inner Product itself is a process of clustering and the process can detect outliers as well. The calculation of the Set Inner Product is based on a vertical structure, P-Trees, which makes our method work efficiently and scale well with very large datasets. We successfully applied our algorithms to several real-world datasets, showing the validity of producing high-quality centroids and its superior performance over existing approaches in term of efficiency and scalability. 1. INTRODUCTION One of the primary data mining tasks is clustering, which intends to discover and understand the natural structure or group in a data set [8]. The goal of clustering is to collect similar objects mutually exclusively and collectively exhaustively, achieving minimal dissimilarity within one cluster and maximal dissimilarity among clusters [4]. Many useful clustering methods, such as partitioning, hierarchical, density-based, grid-based, and model-based methods, were proposed in the last decade [9][4]. This paper focuses on partitioning clustering methods. In a partitioning clustering problem, the aim is to partition a given set of n points in m dimensional space into k groups, called clusters, so that points within each cluster are near each other. The most well known portioning methods are k-means and k-medoids, and their variants. These methods are successful when the clusters are compact clouds that are rather well separated from one another. However, this approach has three main shortcomings. The first shortcoming is the necessity for users to specify k, the number of clusters, in advance, which is not applicable for many real application because users may have no prior knowledge of the distribution of the datasets; the other disadvantage is that the computation complexity of the methods are O (nkt), where n is the number of objects, k is the number of clusters, and t is the number of iterations, which are not efficient and scalable for large datasets The remainder of the paper is organized as follows. Section 2 presents a brief introduction to Predicate Trees (P-Trees). Section 3 presents inner product formulas and examples. Section 4 presents an algorithm for calculating the Set Inner Product. Set Inner Products are very useful in determining the local density of points and therefore in choosing centroids and identifying outliers. The Set Inner Product calculation is very fast and scalable, using vertical data structures. Section 5 discusses performance evaluation. Section 6 presents conclusions and future work. 2. PREDICATE TREES (P-TREES) The P-Tree is a vertically partitioned, lossless compressed representation of binary data [2] [7] that is well suited for representing data sets that are normally stored in a horizontal fashion. As seen in the references [5][6] P-Trees have been used successfully in data mining applications for K-Nearest Neighbor classification. Any data set can be represented as a relation R with attributes A1 thru An and denoted R(A1, A2, … An) with the key normally being A1. In this explanation data will be byte based and the P-tree will convert this into a bit based representation. It should be noted that there is no limitation on the data type size, 8 bits is used for simplicity and can be expanded to multiple bytes without loss of generality. Each 8 bit byte will be converted to 8 P-trees that will recursively give the count of one bits in that bit position in the attribute. For a table with 3 attributes that each have 8 bits of data we will generate 24 P-trees. A P-tree will recursively sub-divide the bit band into halves until we reach the bit level (purity) and counts the number of ones bits in that attribute and considers the predicate all bits are 1. This is also called bit sequential format (bSQ) which is vertically partitioning all the bits in a byte into separate bit vectors which are represented by level 0 of a P-tree. The following example will illustrate this idea for one bit slice of an attribute Ai with bits numbered from 7 down to 0 with bit 7 being most significant. 3. VERTICAL SET INNER PRODUCT Vertical set inner products primarily measures a total variation of set of points in class X about a point. The formula is defined as follows: X a X a x a x a xX n x xX i 1 ai i n n 2 xi ai ai2 i 1 i 1 n x xX i 1 2 i n n n xi2 2 xi ai ai2 xX i 1 Given A17, which we would read as, attribute 1 bit slice 7 Figure 1 shows the conversion to P-Tree. A17 2 xX i 1 T1 T2 T3 where n 1 P17 0 L3 T1 xi2 xX i 1 1 0 1 0 L2 n = 0 0 2j rc ( PX Pij ) i 1 j b 1 1 0 0 0 2 L1 k rc( PX Pij Pil ) k ( j*2)( j 1)&& j 0 l ( j 1)0&& j 0 0 0 0 2 1 0 0 1 L0 1 n T2 2 xi a i x X i 1 n Figure 1 P-Tree of attribute A17 2 xX i 1 n We can see that at Level 3 (L3) the tree represents the predicate all 1 and since the root is 0 we conclude the bit slice is not all 1’s. We denote a half as pure when it contains 2level 1-bits with the root having the highest level and the deepest leaf having level 0. We can see the compression at L1 where there are two branches that are not expanded. When a sub-tree is pure we do not need to expand that branch. Reading the leaves of the tree we see that we have 11100001 at the leaf level and this is recovered by the formula (leaf value) * 2L and read from left to right. P-trees allow for efficient processing of data at the bit level and can be used to perform fast and, or, and exclusive or operations on data. Several optimization techniques have been developed and can be found in the references. 0 2 2 j b 1 0 2 j i 1 j b 1 xX n 2 0 j xij 0 n i 1 0 2 j b 1 j j b 1 j j a ij aij 0 2 j b 1 rc( PX Pij ) n 2 i xX i i 0 2 j aij xX i 1 j b 1 n 0 2 2 j rc( PX Pij ) 2 ai a 2 j b 1 i 1 j b 1 T3 x ij 2 j aij xX i 1 0 rc( PX ) 2 j aij i 1 j b 1 n a)o(X-a) are further examined using the other columns of the table to determine their nature. 2 the total number of points in X, essentially measures total variation of a set of points in class X about a. The table can be sorted on the Set Inner Product column. Then the table can be used to quickly reveal a set select a high-quality value for k (the number of centroids). Those centroids can be used to cluster the data set (putting each non-centroid point with it’s closest centroid point) or, if we wish to perform k-means clustering (with, likely very few iterations), the method can be used as a pre-processing k-means or k-medoid clustering step. The method can also be used to identify outliers (contents of the disks centered at the bottom rows in the table – until ~ 98.5% of the set has been identified as outliers), or detect cluster boundary points by examining the outer disk where the variation changes. 4. ALGORITHM DESIGN 4.1 Algorithm n rc ( PX ) a i2 i 1 The X a X a measures the sum of vector length connecting X and a. Thus, X a X a where N refers to N The Set Inner Product [1] algorithm produces a table of highquality information, which can be used for clustering and outlier analysis. Creating this table is fast and efficient using vertical data structuring. Our choice of vertical data structure, the Predicate-Tree or P-tree [2][7], allows us to build out rectangular neighborhoods of increasing radius until an upturn or downturn in density is discovered and to do so in a scalable manner. A vector, a, is first, randomly selected from the space, X. The Set Inner Product of X about a is then calculated and inserted into a table. The table contains columns for the selected points, a, the Set Inner Product about each a (which we denote as (X-a)o(X-a) for reasons which will become clear later), the radii to which the point, a, can be built-out before the local density begins to change significantly, those local build-out densities, and the direction of change in that final local build-out density (up or down). A very important part of this algorithm is the pruning of X before iteratively picking the next value, a (prune off the maximum builtout disk for which the density remains roughly constant). This pruning step facilitates the building of a table containing the essential local density information we need, without requiring a full scan of X. The table structure is as follows: 1. 2. 3. 4. 5. 6. 7. Select an a X. build out disk D(a,r) of radius r about a, increasing r until the density changes significantly (this significance level is an input parameter). Calculate the Set Inner Product about a, the build-out radii r, the build out densities, and the direction of change of the final build-out density. Enter all of these into the table. Prune all points in the largest disk of common density (these points have approximately the same local density as a). Select another point from what remains of X, Repeat from step 1 until X is empty. Sort the table on Set Inner Product column. O ut lie r Deep Cluster Data Set Figure 2. An Example of the Algorithm. Table 1. Local density information table. a a7 a3 a101 (X-a)o(X-a) 38 38 38 44 44 46 46 46 Build-out Radius 5 10 15 5 10 5 10 15 Build-out Density 10 11 3 9 1 18 17 38 +/variation - + Lower values of (X-a)o(X-a) are associated with points, a, that are “deep cluster points” (deep within clusters), while higher values of (X-a)o(X-a) are associated with outliers. Middle values of (X- Figure 2 shows an example of the algorithm pictorially. The clusters will be built based on the total variation. The disks will be built based on the density threshold with increasing radius r. As we build out the disks, when a major change in density of the disks is observed, the information is added to the table and the disk is pruned. For example, when there is a data point in the first disk and as we build out the disks by increasing the radius r, if there are no data points in the next disk, then we consider that point as an outlier, store the information in the table and prune the disk. We do not further build out disks around that point. In case of boundary points, we store the information in the table and prune the points based on the change in variation of the disk. 4.2 ??? In this section we report the experiments we conducted to evaluate the algorithm. It can be clearly observed from the algorithm that most of the compute time for the algorithm is spent on computing the set inner product. In this section we show that the use of the vertical P-tree data structure for computing the set inner product will scale with respect to the data size. We compare the execution time for the calculation of the set inner product employing a vertical approach (vertical data structure and horizontal bitwise AND operation) with horizontal approach (horizontal data structure and vertical scan operation). We show the results of experiments of execution time with respect to the size of the data set. Performance of both algorithms was observed under different machine specifications, including an SGI Altix CC-NUMA machine. Table 2 summarizes the different types of machines used for the experiments. Table 2. The specification of machines used. Machine AMD1GB P42GB SGI Altix Specification AMD Athlon K7 1.4GHz, 1GB RAM Intel P4 2.4GHz processor 2GB RAM SGI Altix CC-NUMA 12 processor shared memory (12 x 4 GB RAM). 24 : Out of memory 28.80 0.00010 Time to Compute Set Inner Product 35.00 Time (Seconds) 5. EXPERIMENTAL RESULTS 30.00 25.00 20.00 15.00 10.00 5.00 0.00 0 2 4 6 8 10 12 14 16 18 20 22 24 26 Data Set Size (x1024^2) Horz-AMD-1G Horz-P4-2G Virt-AMD-1G out of Mem Horz-SGI-48G Figure 3. Average time running under different machines. The experimental data was generated based on a set of aerial photographs from the Best Management Plot (BMP) of Oakes Irrigation Test Area (OITA) near Oakes, North Dakota. Latitude and longitude are 970 42'18"W, taken in 1998. The image contains three bands: red, green, and blue reflectance values. We use the original image of size 1024x1024 pixels (having cardinality of 1,048,576). Corresponding synchronized data for soil moisture, soil nitrate and crop yield were also used for experimental evaluations. Combining of all bands and synchronized data, we obtained a dataset with 6 dimensions. Additional datasets with different sizes were synthetically generated based on the original data sets to study the timing and scalability. We observe the timing with respect to scalability when executing on machines with different hardware configurations. We were forced to increase the hardware configuration to accommodate the horizontal version of the set inner product calculation on the larger data sets. We were are able to compute the set inner product for the largest data set using the vertical ptree data structure on the smallest machine (AMD Athlon with 1 GB memory). Table 3 presents the average time to compute the set inner product using the two different techniques under different machines and figure further illustrates performance with respect to scalability. Table 3. Average time different hardware configurations. Dataset Size in 10242 1 2 4 8 16 Average Time to Compute Inner Product (Seconds) Horizontal Vertical AMDP4SGI Altix AMD1GB 2GB 12x4GB 1GB 0.55 0.46 1.37 0.00008 1.10 0.91 2.08 0.00008 2.15 1.85 3.97 0.00010 3.79 8.48 0.00010 16.64 0.00010 As the figure shows the horizontal approach is very sensitive to the available memory in the machine. For the vertical approach, it only requires 0.0001 seconds on average to complete the calculation on all data sets datasets, very much less than Horizontal approach. This significant improvement in computation time is due to the use of similar root count values, pre-computed P-trees creation. Although various vectors for different data points are fed during calculation the pre-computed root counts can be used repeatedly. This allows us to pre-compute these once and use their values repeatedly regardless how many inner product calculations are computed as long as the dataset does not change. Notice also that Vertical approach tend to have a constant execution time even though datasets size is expanded. One may argue that pre-calculation of root count makes this comparison fallacious. However, notice the time required for loading vertical data structure to memory and one time root count operations for vertical approach, and loading horizontal records to memory given on table 4. The performance with respect to time of the vertical approach is comparable to the horizontal approach. There is a slight increase for time required to load horizontal records than to load P-trees and to compute root counts as presented in table 4. This illustrates the ability of the P-tree data structure to efficiently load and compute the simple counts. These timing were tested on a P4 with 2GB of memory. Table 4. Time for computing root count and loading dataset. Dataset Size Size in 10242 1 2 4 8 Time (Seconds) Vertical Horizontal Root Count PreHorizontal Dataset Computation and Loading P-trees Loading 3.900 4.974 8.620 10.470 18.690 19.914 38.450 39.646 6. CONCLUSION ???. [6] Perera, A., Denton, A., Kotala,P., Jockheck,W., Granda, W. V., and Perrizo, W., (2002). P-tree Classification of Yeast Gene Deletion Data. SIGKDD Explorations, 4(2): 108-109. [7] Perrizo, W. (2001). Peano Count Tree Technology, Technical Report NDSU-CSOR-TR-01-1. 7. REFERENCES [1] Abidin, T. and Perrizo, W., Vertical Set Inner Products Formula. http://midas.cs.ndsu.nodak.edu/ ~abidin/research/PSIPs.pdf [2] Ding, Q., Khan, M., Roy, A., and Perrizo, W., (2002). The Ptree Algebra, Proceedings of the ACM Symposium on Applied Computing. 426-431. [3] Eric W. Weisstein et al. Total Variation. From MathWorld – A Wolfram Web Resource. http://mathworld.wolfram.com/TotalVariation.html [4] Han, J., and Kamber, M. (2001). Data Mining: Concepts and Techniques. San Francisco, CA., Morgan Kaufmann. [5] Khan, M., Ding, Q., and Perrizo, W., (2002). K-Nearest Neighbor Classification of Spatial Data Streams using Ptrees, Proceedings of the PAKDD. 517-528. [8] Baumgartner, C., Plant, C., Kailing, K., Kriegel, H., Kroger, P., (2004) Subspace Selection for Clustering HighDimensional Data, proceedings of 4th IEEE International Conference on Data Mining. [9] Bohm,C., Kailing, K., Kriegel,H., Kroger,P., Density Connected Clustering with Local Subspace Preferences, proceedings of 4th IEEE International Conference on Data Mining. [10] Tomas,F., Greene,D., “Optimal Algorithm for Approximate Clustering”, the proceeding of the 20th ACM Symp. Theory of computing, pages 434--444, Chicago, Illinois, 1988. [11]