Download Case Studies in Data Mining

Case Studies in Data Mining
András Fülöp, e-Ventures Ltd. <[email protected]>
László Gonda, University of Debrecen <[email protected]>
Dr.. Márton Ispány, University of Debrecen <[email protected]>
Dr.. Péter Jeszenszky, University of Debrecen <[email protected]>
Dr.. László Szathmáry, University of Debrecen <[email protected]>
Created by XMLmind XSL-FO Converter.
Case Studies in Data Mining
by András Fülöp, László Gonda, Dr.. Márton Ispány, Dr.. Péter Jeszenszky, and Dr.. László Szathmáry
Publication date 2014
Copyright © 2014 Faculty of Informatics, University of Debrecen
A tananyag a TÁMOP-4.1.2.A/1-11/1-2011-0103 azonosítójú pályázat keretében valósulhatott meg.
Created by XMLmind XSL-FO Converter.
Table of Contents
Preface ................................................................................................................................................ ii
1. How to use this material ....................................................................................................... iv
I. Data mining tools ............................................................................................................................ 6
1. Commercial data mining softwares ....................................................................................... 7
2. Free and shareware data mining softwares .......................................................................... 11
II. RapidMiner .................................................................................................................................. 13
3. Data Sources ....................................................................................................................... 16
1. Importing data from a CSV file ................................................................................. 16
2. Importing data from an Excel file .............................................................................. 17
3. Creating an AML file for reading a data file ............................................................. 19
4. Importing data from an XML file .............................................................................. 21
5. Importing data from a database ................................................................................. 23
4. Pre-processing ..................................................................................................................... 25
1. Managing data with issues - Missing, inconsistent, and duplicate values ................. 25
2. Sampling and aggregation ......................................................................................... 27
3. Creating and filtering attributes ................................................................................. 31
4. Discretizing and weighting attributes ........................................................................ 35
5. Classification Methods 1 ..................................................................................................... 41
1. Classification using a decision tree ............................................................................ 41
2. Under- and overfitting of a classification with a decision tree .................................. 46
3. Evaluation of performance for classification by decision tree ................................... 51
4. Evaluation of performance for classification by decision tree 2 ................................ 55
5. Comparison of decision tree classifiers ..................................................................... 58
6. Classification Methods 2 ..................................................................................................... 65
1. Using a rule-based classifier (1) ................................................................................ 65
2. Using a rule-based classifier (2) ................................................................................ 66
3. Transforming a decision tree to an equivalent rule set .............................................. 68
7. Classification Methods 3 ..................................................................................................... 71
1. Linear regression ....................................................................................................... 71
2. Osztályozás lineáris regresszióval ............................................................................. 73
3. Evaluation of performance for classification by regression model ............................ 76
4. Evaluation of performance for classification by regression model 2 ......................... 79
8. Classification Methods 4 ..................................................................................................... 84
1. Using a perceptron for solving a linearly separable binary classification problem ... 84
2. Using a feed-forward neural network for solving a classification problem ............... 85
3. The influence of the number of hidden neurons to the performance of the feed-forward
neural network ............................................................................................................... 87
4. Using a linear SVM for solving a linearly separable binary classification problem .. 88
5. The influence of the parameter C to the performance of the linear SVM (1) ............ 90
6. The influence of the parameter C to the performance of the linear SVM (2) ............ 93
7. The influence of the parameter C to the performance of the linear SVM (3) ............ 95
8. The influence of the number of training examples to the performance of the linear SVM
97
9. Solving the two spirals problem by a nonlinear SVM ............................................. 100
10. The influence of the kernel width parameter to the performance of the RBF kernel SVM
101
11. Search for optimal parameter values of the RBF kernel SVM .............................. 103
12. Using an SVM for solving a multi-class classification problem ............................ 105
13. Using an SVM for solving a regression problem ................................................... 106
9. Classification Methods 5 ................................................................................................... 110
1. Introducing ensemble methods: the bagging algorithm ........................................... 110
2. The influence of the number of base classifiers to the performance of bagging ...... 111
3. The influence of the number of base classifiers to the performance of the AdaBoost method
...................................................................................................................................... 113
4. The influence of the number of base classifiers to the performance of the random forest
115
iii
Created by XMLmind XSL-FO Converter.
Case Studies in Data Mining
10. Association rules .............................................................................................................
1. Extraction of association rules .................................................................................
2. Asszociációs szabályok kinyerése nem tranzakciós adathalmazból ........................
3. Evaluation of performance for association rules .....................................................
4. Performance of association rules - Simpson's paradox ............................................
11. Clustering 1 .....................................................................................................................
1. K-means method ......................................................................................................
2. K-medoids method ..................................................................................................
3. The DBSCAN method .............................................................................................
4. Agglomerative methods ...........................................................................................
5. Divisive methods .....................................................................................................
12. Clustering 2 .....................................................................................................................
1. Support vector clustering .........................................................................................
2. Choosing parameters in clustering ...........................................................................
3. Cluster evaluation ....................................................................................................
4. Centroid method ......................................................................................................
5. Text clustering .........................................................................................................
13. Anomaly detection ..........................................................................................................
1. Searching for outliers ...............................................................................................
2. Unsupervised search for outliers .............................................................................
3. Unsupervised statistics based anomaly detection ....................................................
III. SAS® Enterprise Miner™ .......................................................................................................
14. Data Sources ...................................................................................................................
1. Reading SAS dataset ...............................................................................................
2. Importing data from a CSV file ...............................................................................
3. Importing data from a Excel file ..............................................................................
15. Preprocessing ..................................................................................................................
1. Constructing metadata and automatic variable selection .........................................
2. Vizualizing multidimensional data and dimension reduction by PCA ....................
3. Replacement and imputation ...................................................................................
16. Classification Methods 1 .................................................................................................
1. Classification by decision tree .................................................................................
2. Comparison and evaluation of decision tree classifiers ...........................................
17. Classification Methods 2 .................................................................................................
1. Rule induction to the classification of rare events ...................................................
18. Classification Methods 3 .................................................................................................
1. Logistic regression ...................................................................................................
2. Prediction of discrete target by regression models ..................................................
19. Classification Methods 4 .................................................................................................
1. Solution of a linearly separable binary classification task by ANN and SVM ........
2. Using artificial neural networks (ANN) ..................................................................
3. Using support vector machines (SVM) ...................................................................
20. Classification Methods 5 .................................................................................................
1. Ensemble methods: Combination of classifiers .......................................................
2. Ensemble methods: bagging ....................................................................................
3. Ensemble methods: boosting ...................................................................................
21. Association mining ..........................................................................................................
1. Extracting association rules .....................................................................................
22. Clustering 1 .....................................................................................................................
1. K-means method ......................................................................................................
2. Agglomerative hierarchical methods .......................................................................
3. Comparison of clustering methods ..........................................................................
23. Clustering 2 .....................................................................................................................
1. Clustering attributes before fitting SVM .................................................................
2. Self-organizing maps (SOM) and vector quantization (VQ) ...................................
24. Regression for continuous target .....................................................................................
1. Logistic regression ...................................................................................................
2. Prediction of discrete target by regression models ..................................................
3. Supervised models for continuous target .................................................................
25. Anomaly detection ..........................................................................................................
iv
Created by XMLmind XSL-FO Converter.
118
118
121
126
131
135
135
137
140
142
144
148
148
151
154
159
161
165
165
167
171
176
178
178
180
183
185
185
188
191
196
196
200
208
208
212
212
217
221
221
225
232
240
240
244
249
256
256
260
260
267
271
278
278
284
289
289
294
297
304
Case Studies in Data Mining
1. Detecting outliers ..................................................................................................... 304
Bibliography ................................................................................................................................... 307
v
Created by XMLmind XSL-FO Converter.
List of Figures
3.1. Metadata of the resulting ExampleSet. ...................................................................................... 16
3.2. A small excerpt of the resulting ExampleSet. ............................................................................ 17
3.3. Metadata of the resulting ExampleSet. ...................................................................................... 18
3.4. A small excerpt of the resulting ExampleSet. ............................................................................ 18
3.5. The resulting AML file. ............................................................................................................. 20
3.6. A small excerpt of The World Bank: Population (Total) data set used in the exepriment. ........ 21
3.7. Metadata of the resulting ExampleSet. ...................................................................................... 22
3.8. A small excerpt of the resulting ExampleSet. ............................................................................ 22
3.9. Metadata of the resulting ExampleSet. ...................................................................................... 23
3.10. A small excerpt of the resulting ExampleSet. .......................................................................... 24
4.1. Graphic representation of the global and kitchen power consumption in time .......................... 25
4.2. Possible outliers based on the hypothesized hbits of the members of the household ................ 26
4.3. Filtering of the possible values using a record filter .................................................................. 26
4.4. Selection of aggregate functions for attributes .......................................................................... 28
4.5. Preferences for dataset sampling ............................................................................................... 29
4.6. Preferences for dataset filtering ................................................................................................. 29
4.7. Resulting dataset after dataset sampling .................................................................................... 30
4.8. Resulting dataset after dataset filtering ...................................................................................... 30
4.9. Defining a new attribute based on an expression relying on existing attributes ........................ 32
4.10. Properties of the operator used for removing the attributes made redundant .......................... 33
4.11. Selection of the attributes to remain in the dataset with reduced size ...................................... 33
4.12. The appearance of the derived attribute in the altered dataset ................................................. 34
4.13. Selection of the appropriate discretization operator ................................................................ 36
4.14. Setting the properties of the discretization operator ................................................................ 37
4.15. Selection of the appropriate weighting operator ...................................................................... 37
4.16. Defining the weights of the individual attributes ..................................................................... 38
4.17. Comparison of the weighted and unweighted dataset instances .............................................. 39
5.1. Preferences for the building of the decision tree ........................................................................ 41
5.2. Preferences for splitting the dataset into training and test sets .................................................. 42
5.3. Setting the relative sizes of the data partitions ........................................................................... 42
5.4. Graphic representation of the decision tree created ................................................................... 43
5.5. The classification of the records based on the decision tree ...................................................... 44
5.6. Setting a threshold for the maximal depth of the decision tree .................................................. 46
5.7. Graphic representation of the decision tree created ................................................................... 47
5.8. Graphic representation of he classification of the records based on the decision tree ............... 47
5.9. Graphic representation of the decision tree created with the increased maximal depth ............. 48
5.10. Graphic representation of he classification of the records based on the decision tree with increased
maximal depth .................................................................................................................................. 48
5.11. Graphic representation of the decision tree created with the further increased maximal depth 49
5.12. Graphic representation of he classification of the records based on the decision tree with further
increased maximal depth .................................................................................................................. 50
5.13. Preferences for the building of the decision tree ...................................................................... 52
5.14. Graphic representation of the decision tree created ................................................................. 52
5.15. Graphic representation of he classification of the records based on the decision tree ............. 53
5.16. Performance vector of the classification based on the decision tree ........................................ 53
5.17. The modification of preferences for the building of the decision tree. .................................... 53
5.18. Graphic representation of the decision tree created with the modified preferences ................. 54
5.19. Performance vector of the classification based on the decision tree created with the modified
preferences ........................................................................................................................................ 54
5.20. Settings for the sampling done in the validation operator ........................................................ 56
5.21. Subprocesses of the validation operator .................................................................................. 56
5.22. Graphic representation of the decision tree created ................................................................. 56
5.23. Performance vector of the classification based on the decision tree ........................................ 57
5.24. Settings of the cross-validation operator .................................................................................. 57
5.25. Overall performance vector of the classifications done in the cross-validation operator ........ 58
vi
Created by XMLmind XSL-FO Converter.
Case Studies in Data Mining
5.26. Overall performance vector of the classifications done in the cross-validation operator in the leaveone-out case ...................................................................................................................................... 58
5.27. Preferences for the building of the decision tree based on the Gini-index criterion ................ 59
5.28. Preferences for the building of the decision tree based on the gain ratio criterion .................. 60
5.29. Graphic representation of the decision tree created based on the gain ratio criterion .............. 61
5.30. Performance vector of the classification based on the decision tree built using the gain ratio criterion
........................................................................................................................................................... 62
5.31. Graphic representation of the decision tree created based on the Gini-index criterion ............ 62
5.32. Performance vector of the classification based on the decision tree built using the Gini-index
criterion ............................................................................................................................................. 62
5.33. Settings of the operator for the comparison of ROC curves .................................................... 62
5.34. Subprocess of the operator for the comparison of ROC curves ............................................... 63
5.35. Comparison of the ROC curves of the two decision tree classifiers ........................................ 63
6.1. The rule set of the rule-based classifier trained on the data set. ................................................. 65
6.2. The classification accuracy of the rule-based classifier on the data set. .................................... 65
6.3. The rule set of the rule-based classifier. .................................................................................... 67
6.4. The classification accuracy of the rule-based classifier on the training set. .............................. 67
6.5. The classification accuracy of the rule-based classifier on the test set. ..................................... 67
6.6. The decision tree built on the data set. ....................................................................................... 68
6.7. The rule set equivalent of the decision tree. .............................................................................. 69
6.8. The classification accuracy of the rule-based classifier on the data set. .................................... 69
7.1. Properties of the linear regression operator ............................................................................... 71
7.2. The linear regression model yielded as a result ......................................................................... 72
7.3. The class prediction values calculated based on the linear regression model ............................ 72
7.4. The subprocess of the classification by regression operator ...................................................... 74
7.5. The linear regression model yielded as a result ......................................................................... 74
7.6. The class labels derived from the predictions calculated based on the regression model .......... 75
7.7. The subprocess of the classification by regression operator ...................................................... 77
7.8. The linear regression model yielded as a result ......................................................................... 77
7.9. The performance vector of the classification based on the regression model ............................ 78
7.10. The subprocess of the cross-validation by regression operator ............................................... 80
7.11. The subprocess of the classification by regression operator .................................................... 80
7.12. The linear regression model yielded as a result ....................................................................... 80
7.13. The customizable properties of the cross-validation operator ................................................. 81
7.14. The overall performance vector of the classifications done using the regression model defined in the
cross-validation operator .................................................................................................................. 82
7.15. The overall performance vector of the classifications done using the regression model defined in the
cross-validation operator for the case of using the leave-one-out method ........................................ 82
8.1. A linearly separable subset of the Wine data set [UCI MLR] used in the experiment (2 of the total of 3
classes and 2 of the total of 13 attributes was selected). ................................................................... 84
8.2. The decision boundary of the perceptron. .................................................................................. 85
8.3. The classification accuracy of the perceptron on the data set. ................................................... 85
8.4. The classification accuracy of the neural network on the data set. ............................................ 86
8.5. The average classification error rate obtained from 10-fold cross-validation against the number of
hidden neurons. ................................................................................................................................. 87
8.6. A linearly separable subset of the Wine data set [UCI MLR] used in the experiment (2 of the total of 3
classes and 2 of the total of 13 attributes was selected). ................................................................... 89
8.7. The kernel model of the linear SVM. ........................................................................................ 89
8.8. The classification accuracy of the linear SVM on the data set. ................................................. 90
8.9. A subset of the Wine data set used in the experiment (2 of the total of 3 classes and 2 of the total of 13
attributes was selected). Note that the classes are not linearly separable. ........................................ 91
8.10. The classification error rate of the linear SVM against the value of the parameter C. ............. 91
8.11. The number of support vectors against the value of the parameter C. ..................................... 92
8.12. The average classification error rate of the linear SVM obtained from 10-fold cross-validation
against the value of the parameter C, where the horizontal axis is scaled logarithmically. ............... 94
8.13. The classification error rate of the linear SVM on the training and the test sets against the value of
the parameter C. ................................................................................................................................ 95
8.14. The number of support vectors against the value of the parameter C. ..................................... 96
8.15. The classification error rate of the linear SVM on the training and the test sets against the number of
training examples. ............................................................................................................................. 98
vii
Created by XMLmind XSL-FO Converter.
Case Studies in Data Mining
8.16. The number of support vectors against the number of training examples. .............................. 98
8.17. CPU execution time needed to train the SVM against the number of training examples. ....... 98
8.18. The Two Spirals data set ........................................................................................................ 100
8.19. The R code that produces the data set and is executed by the Execute Script (R) operator of the
R Extension. ................................................................................................................................... 100
8.20. The classification accuracy of the nonlinear SVM on the data set. ....................................... 101
8.21. The classification error rates of the SVM on the training and the test sets against the value of RBF
kernel width parameter. .................................................................................................................. 102
8.22. The optimal parameter values for the RBF kernel SVM. ...................................................... 104
8.23. The classification accuracy of the RBF kernel SVM trained on the entire data set using the optimal
parameter values. ............................................................................................................................ 104
8.24. The kernel model of the linear SVM. .................................................................................... 105
8.25. The classification accuracy of the linear SVM on the data set. ............................................. 106
8.26. The optimal value of the gamma parameter for the RBF kernel SVM. ................................... 107
8.27. The average RMS error of the RBF kernel SVM obtained from 10-fold cross-validation against the
value of the parameter gamma, where the horizontal axis is scaled logarithmically. ....................... 107
8.28. The kernel model of the optimal RBF kernel SVM. .............................................................. 108
8.29. Predictions provided by the optimal RBF kernel SVM against the values of the observed values of
the dependent variable. ................................................................................................................... 108
9.1. The average classification error rate of a single decision stump obtained from 10-fold crossvalidation. ....................................................................................................................................... 110
9.2. The average classification error rate of the bagging algorithm obtained from 10-fold cross-validation,
where 10 decision stumps were used as base classifiers. ................................................................ 110
9.3. The average classification error rate obtained from 10-fold cross-validation against the number of
base classifiers. ............................................................................................................................... 112
9.4. The average classification error rate obtained from 10-fold cross-validation against the number of
base classifiers. ............................................................................................................................... 114
9.5. The average error rate of the random forest obtained from 10-fold cross-validation against the number
of base classifiers. ........................................................................................................................... 116
10.1. List of the frequent item sets generated ................................................................................. 118
10.2. List of the association rules generated ................................................................................... 119
10.3. Graphic representation of the association rules generated ..................................................... 120
10.4. Operator preferences for the necessary data conversion ........................................................ 121
10.5. Converted version of the dataset ............................................................................................ 122
10.6. List of the frequent item sets generated ................................................................................. 122
10.7. List of the association rules generated ................................................................................... 123
10.8. Operator preferences for the appropriate data conversion ..................................................... 123
10.9. The appropriate converted version of the dataset .................................................................. 124
10.10. Enhanced list of the frequent item sets generated ................................................................ 124
10.11. List of the association rules generated ................................................................................. 125
10.12. Graphic representation of the association rules generated ................................................... 125
10.13. Operator preferences for the necessary data conversion ...................................................... 127
10.14. Label role assignment for performance evaluation .............................................................. 128
10.15. Prediction role assignment for performance evaluation ....................................................... 128
10.16. Operator preferences for the data conversion necessary for evaluation ............................... 129
10.17. Graphic representation of the association rules generated regarding survival ..................... 129
10.18. List of the association rules generated regarding survival ................................................... 130
10.19. Performance vector for the application of association rules generated ................................ 130
10.20. List of the association rules generated regarding survival ................................................... 131
10.21. Performance vector for the application of association rules generated ................................ 132
10.22. Contingency table of the dataset .......................................................................................... 132
10.23. Record filter usage ............................................................................................................... 132
10.24. Removal of attributes that become redundant after filtering ................................................ 133
10.25. List of the association rules generated for the subset of adults ............................................ 133
10.26. Performance vector for the application of association rules generated regarding survival for the
subset of adults ............................................................................................................................... 133
10.27. List of the association rules generated for the subset of children ........................................ 133
10.28. Performance vector for the application of association rules generated regarding survival for the
subset of children ............................................................................................................................ 133
11.1. The 7 separate groups ............................................................................................................ 135
viii
Created by XMLmind XSL-FO Converter.
Case Studies in Data Mining
11.2. Clustering with default values ...............................................................................................
11.3. Set the distance function. .......................................................................................................
11.4. Clustering with Mahalanobis distance function .....................................................................
11.5. The dataset .............................................................................................................................
11.6. Setting the parameters of the clustering .................................................................................
11.7. The clusters produced by the analysis ...................................................................................
11.8. The groups with varying density ...........................................................................................
11.9. The results of the method with default parameters ................................................................
11.10. The 15 group ........................................................................................................................
11.11. The resulting dendrogram ....................................................................................................
11.12. The clustering generated from dendrogram .........................................................................
11.13. The 600 two-dimensional vectors ........................................................................................
11.14. The subprocess ....................................................................................................................
11.15. The report generated by the clustering .................................................................................
11.16. The output of the analysis ....................................................................................................
12.1. The two groups ......................................................................................................................
12.2. Support vector clustering with polynomial kernel and p=0.21 setup .....................................
12.3. Unsuccessful clustering .........................................................................................................
12.4. Clustering with RBF kernel ...................................................................................................
12.5. More promising results ..........................................................................................................
12.6. The two groups containing 240 vectors .................................................................................
12.7. The subprocess of the optimalization node ............................................................................
12.8. The parameters of the optimalization ....................................................................................
12.9. The report generated by the process ......................................................................................
12.10. Clustering generated with the most optimal parameters ......................................................
12.11. The 788 vectors ...................................................................................................................
12.12. The evaluating subprocess ...................................................................................................
12.13. Setting up the parameters .....................................................................................................
12.14. Parameters to log .................................................................................................................
12.15. Cluster density against k number of clusters .......................................................................
12.16. Item distribution against k number of clusters .....................................................................
12.17. The vectors forming 31 clusters ...........................................................................................
12.18. The extracted centroids ........................................................................................................
12.19. The output of the k nearest neighbour method, using the centroids as prototypes ..............
12.20. The preprocessing subprocess .............................................................................................
12.21. The clustering setup .............................................................................................................
12.22. The confusion matrix of the results .....................................................................................
13.1. Graphic representation of the possible outliers ......................................................................
13.2. The number of outliers detected as the distance limit grows .................................................
13.3. Nearest neighbour based operators in the Anomaly Detection package ................................
13.4. Settings of LOF. ....................................................................................................................
13.5. Outlier scores assigned to the individual records based on k nearest neighbours ..................
13.6. Outlier scores assigned to the individual records based on LOF ...........................................
13.7. Filtering the records based on their outlier scores .................................................................
13.8. The dataset filtered based on the k-NN score ........................................................................
13.9. The dataset filtered based on the LOF score ..........................................................................
13.10. Global settings for Histogram-based Outlier Score .............................................................
13.11. Column-level settings for Histogram-based Outlier Score ..................................................
13.12. Scores and attribute binning for fixed binwidth and arbitrary number of bins ....................
13.13. Graphic representation of outlier scores ..............................................................................
13.14. Scores and attributes binning for dynamic binwidth and arbitrary number of bins .............
13.15. Graphic representation of the enhanced outlier scores ........................................................
14.1. The metadata of the dataset ...................................................................................................
14.2. Setting the Sample operator ...................................................................................................
14.3. The metadata of the resulting dataset and a part of the dataset ..............................................
14.4. The list of file in the File Import operator .........................................................................
14.5. The parameters of the File Import operator .......................................................................
14.6. A small portion of the dataset ................................................................................................
14.7. The metadata of the resulting dataset .....................................................................................
14.8. A small portion of the resulting dataset .................................................................................
ix
Created by XMLmind XSL-FO Converter.
135
136
136
138
138
139
140
141
142
143
143
145
146
146
147
148
149
149
150
150
151
152
152
153
153
155
155
156
156
157
157
159
160
160
162
162
163
165
166
168
168
168
169
169
170
170
171
172
172
173
174
175
178
178
179
180
181
181
182
183
Case Studies in Data Mining
15.1. Metadata produced by the DMDB operator .............................................................................. 185
15.2. The settings of Variable Selection operator .................................................................... 186
15.3. List of variables after the selection ........................................................................................ 186
15.4. Sequential R-square plot ........................................................................................................ 187
15.5. The binary target variables in a function of the two most important input attributes after the variable
selection ......................................................................................................................................... 187
15.6. Displaying the dataset by parallel axis ................................................................................... 189
15.7. Explained cumulated explained variance plot of the PCA ..................................................... 190
15.8. Scatterplit of the Iris dataset using the first two principal components ................................. 191
15.9. The replacement wizard ......................................................................................................... 192
15.10. The output of imputation ..................................................................................................... 193
15.11. The relationship of an input and the target variable before imputation ............................... 193
15.12. The relationship of an input and the target variable after imputation .................................. 194
16.1. The settings of dataset partitioning ........................................................................................ 196
16.2. The decision tree .................................................................................................................... 197
16.3. The response curve of the decision tree ................................................................................. 198
16.4. Fitting statistics of the decision tree ...................................................................................... 198
16.5. The classification chart of the decision tree ........................................................................... 198
16.6. The cumulative lift curve of the decision tree ........................................................................ 199
16.7. The importance of attributes .................................................................................................. 200
16.8. The settings of parameters in the partitioning step ................................................................ 201
16.9. The decision tree using the chi-square impurity measure ...................................................... 202
16.10. The decision tree using the entropy impurity measure ........................................................ 202
16.11. The decision tree using the Gini-index ................................................................................ 203
16.12. The cumulative response curve of decision trees ................................................................. 204
16.13. The classification plot .......................................................................................................... 205
16.14. Response curve of decision trees ......................................................................................... 205
16.15. The score distribution of decision trees ............................................................................... 206
16.16. The main statistics of decision trees .................................................................................... 206
17.1. The misclassification rate of rule induction ........................................................................... 208
17.2. The classification matrix of rule induction ............................................................................ 208
17.3. The classification chart of rule induction ............................................................................... 209
17.4. The ROC curves of rule inductions and decision tree ........................................................... 209
17.5. The output of the rule induction operator .............................................................................. 210
18.1. Classification matrix of the logistic regression ...................................................................... 212
18.2. Effects plot of the logistic regression ..................................................................................... 213
18.3. Classification matrix of the stepwise logistic regression ....................................................... 213
18.4. Effects plot of the stepwise logistic regression ...................................................................... 214
18.5. Fitting statistics for logistic regression models ...................................................................... 214
18.6. Classification charts of the logistic regression models .......................................................... 215
18.7. Cumulativ lift curve of the logistic regression models .......................................................... 215
18.8. ROC curves of the logistic regression models ....................................................................... 216
18.9. Classification matrix of the logistic regression ...................................................................... 217
18.10. The classification chart of the logistic regression ................................................................ 218
18.11. The effects plot of the logistic regression ............................................................................ 219
19.1. A linearly separable subset of the Wine dataset .................................................................... 221
19.2. Fitting statistics for perceptron .............................................................................................. 222
19.3. The classification matrix of the perceptron ........................................................................... 222
19.4. The cumulative lift curve of the perceptron ........................................................................... 222
19.5. Fitting statistics for SVM ....................................................................................................... 223
19.6. The classification matrix of SVM .......................................................................................... 223
19.7. The cumulative lift curve of SVM ......................................................................................... 223
19.8. List of the support vectors ..................................................................................................... 224
19.9. Fitting statistics of the multilayer perceptron ........................................................................ 225
19.10. The classification matrix of the multilayer perceptron ........................................................ 226
19.11. The cumulative lift curve of the multilayer perceptron ....................................................... 226
19.12. Weights of the multilayer perceptron .................................................................................. 227
19.13. Training curve of the multilayer perceptron ........................................................................ 227
19.14. Stepwise optimization statistics for DMNeural operator ...................................................... 228
19.15. Weights of neural networks AutoNeural operátorral kapott háló neuronjainak súlyai ....... 228
x
Created by XMLmind XSL-FO Converter.
Case Studies in Data Mining
19.16. Fitting statistics of neural networks ..................................................................................... 229
19.17. Classification charts of neural networks .............................................................................. 230
19.18. Cumulative lift curves of neural networks ........................................................................... 230
19.19. ROC curves of neural networks ........................................................................................... 231
19.20. Fitting statistics for linear kernel SVM ................................................................................ 233
19.21. The classification matrix of linear kernel SVM ................................................................... 233
19.22. Support vectors (partly) of linear kernel SVM .................................................................... 233
19.23. The distribution of Lagrange multipliers for linear kernel SVM ......................................... 234
19.24. The parameters of polynomial kernel SVM ......................................................................... 234
19.25. Fitting statistics for polynomial kernel SVM ....................................................................... 235
19.26. Classification matrix of polynomial kernel SVM ................................................................ 235
19.27. Support vectors (partly) of polynomial kernel SVM ........................................................... 236
19.28. Fitting statistics for SVM's .................................................................................................. 236
19.29. The classification chart of SVM's ........................................................................................ 237
19.30. Cumulative lift curves of SVM's ......................................................................................... 237
19.31. Comparison of cumulative lift curves to the baseline and the optimal one ........................ 238
19.32. ROC curves of SVM's ......................................................................................................... 238
20.1. Fitting statistics of the ensemble classifier ............................................................................ 240
20.2. The classification matrix of the ensemble classifier .............................................................. 240
20.3. The cumulative lift curve of the ensemble classifier ............................................................. 241
20.4. Misclassification rates of the ensemble classifier and the SVM ............................................ 242
20.5. Classification matrices of the ensemble classifier and the SVM ........................................... 242
20.6. Cumulative lift curves of the ensemble classifier and the SVM ............................................ 242
20.7. Cumulative lift curves of the ensemble classifier, the SVM and the best theoretical model . 243
20.8. ROC curves of the ensemble classifier and the SVM ............................................................ 243
20.9. The classification matrix of the bagging classifier ................................................................ 245
20.10. The error curves of the bagging classifier ............................................................................ 246
20.11. Misclassification rates of the bagging classifier and the decision tree ................................ 247
20.12. Classification matrices of the bagging classifier and the decision tree ................................ 247
20.13. Response curves of the bagging classifier and the decision tree .......................................... 247
20.14. Response curves of the bagging classifier and the decision tree comparing the baseline and the
optimal classifiers ........................................................................................................................... 248
20.15. ROC curves of the bagging classifier and the decision tree ................................................. 248
20.16. The classification matrix of the boosting classifier ............................................................. 250
20.17. The error curve of the boosting classifier ............................................................................ 251
20.18. Misclassification rates of the boosting classifier and the SVM ........................................... 252
20.19. Classification matrices for the boosting classifier and the SVM ......................................... 252
20.20. Cumulative response curves of the boosting classifier and the SVM .................................. 252
20.21. Response curves of the boosting classifier and the SVM comparing the baseline and the optimal
classifiers ........................................................................................................................................ 253
20.22. ROC curves of the boosting classifier and the SVM ........................................................... 254
21.1. List of items ........................................................................................................................... 256
21.2. The association rules as a function of the support and the reliability .................................... 257
21.3. Graph of lift values ................................................................................................................ 257
21.4. List of association rules ......................................................................................................... 258
22.1. The Aggregation dataset. ....................................................................................................... 260
22.2. The setting of the Cluster operator. ..................................................................................... 261
22.3. The result of K-means clustering when K=7 ......................................................................... 262
22.4. The setting of the MacQueen clustering ................................................................................ 263
22.5. The result of the MacQueen clustering .................................................................................. 263
22.6. The result of the clustering with 8 clusters ............................................................................ 264
22.7. The result display of the Cluster operator ......................................................................... 265
22.8. Scatterplot of the cluster means ............................................................................................. 265
22.9. The decision tree of the clustering ......................................................................................... 266
22.10. Scatterplot of the Maximum Variance (R15) dataset ........................................................... 267
22.11. The result of the average linkage hierarchical clustering ..................................................... 268
22.12. Evaluating of the clustering by 3D bar chart ....................................................................... 269
22.13. The result of Ward clustering .............................................................................................. 269
22.14. CCC plot of automatic clustering ........................................................................................ 270
22.15. Proximity graph of the automatic clustering ........................................................................ 270
xi
Created by XMLmind XSL-FO Converter.
Case Studies in Data Mining
22.16. The Maximum Variance (D31) dataset ................................................................................
22.17. The result of automatic clustering ........................................................................................
22.18. The CCC plot of automatic clustering .................................................................................
22.19. Az automatikus klaszterezés proximitási ábrája ..................................................................
22.20. The result of K-means clustering .........................................................................................
22.21. The proximity graph of K-means clustering ........................................................................
22.22. The profile of the segments (clusters) ..................................................................................
23.1. The dendrogram of attribute clustering ..................................................................................
23.2. The graph of clusters and attributes .......................................................................................
23.3. The cluster membership .........................................................................................................
23.4. The correlation plot of the attributes ......................................................................................
23.5. The correlation between clusters and an attribute ..................................................................
23.6. Classification charts of SVM models ....................................................................................
23.7. The response curve of SVM models ......................................................................................
23.8. Az SVM modellek kumulatív lift függvényei ........................................................................
23.9. The ROC curves of SVM models ..........................................................................................
23.10. The scatterplot of the Maximum Variance (R15) dataset .....................................................
23.11. The result of Kohonen's vector quantization .......................................................................
23.12. The pie chart of cluster size .................................................................................................
23.13. Statistics of clusters .............................................................................................................
23.14. Graphical representation of the SOM ..................................................................................
23.15. Scatterplot of the result of SOM ..........................................................................................
24.1. Classification matrix of the logistic regression ......................................................................
24.2. Effects plot of the logistic regression .....................................................................................
24.3. Classification matrix of the stepwise logistic regression .......................................................
24.4. Effects plot of the stepwise logistic regression ......................................................................
24.5. Fitting statistics for logistic regression models ......................................................................
24.6. Classification charts of the logistic regression models ..........................................................
24.7. Cumulativ lift curve of the logistic regression models ..........................................................
24.8. ROC curves of the logistic regression models .......................................................................
24.9. Classification matrix of the logistic regression ......................................................................
24.10. The classification chart of the logistic regression ................................................................
24.11. The effects plot of the logistic regression ............................................................................
24.12. Statistics of the fitted models on the test dataset .................................................................
24.13. Comparison of the fitted models by means of predictions ...................................................
24.14. The observed and predicted means plot ...............................................................................
24.15. The model scores .................................................................................................................
24.16. The decision tree for continuous target ................................................................................
24.17. The weights of neural network after traning ........................................................................
25.1. Statistics before and after filtering outliers ............................................................................
25.2. The predicted mean based on the two decision trees .............................................................
25.3. The tree map of the best model ..............................................................................................
25.4. Comparison of the two fitted decision trees ..........................................................................
xii
Created by XMLmind XSL-FO Converter.
272
273
273
273
274
274
275
278
279
279
280
281
282
282
282
283
284
285
285
286
286
287
289
290
290
291
291
292
292
293
294
295
296
297
298
298
299
300
301
304
305
305
306
Colophon
TODO
1
Created by XMLmind XSL-FO Converter.
Preface
The data mining is an interdisciplinary area of information technology, one of the most important parts of the
so-called KDD (Knowledge Discovery from Databases) process. It consists of such computationally intensive
algorithms, methods which are capable to explore patterns from relatively large datasets that represent wellinterpretable information for further use. The applied algorithms originate from a number of fields, namely,
artificial intelligence, machine learning, statistics, and database systems. Moreover, the data mining combine the
results of these areas and it evolves in interaction of them today too. In contrast to focus merely to data analysis,
see, for example, the statistics, data mining contains a number of additional elements, including the
datamanegement and data preprocessing, moreover, such post-processing issues as the interesting metrics or the
suitable visualization of the explored knowledge.
The use of the word data mining has become very fashionable and many people mistakenly use it for all sorts of
information processing involving large amounts of data, e.g., simple information extraction or data warehouse
building), but it also appears in the context of decision support systems. In fact, the most important feature of
data mining is the exploration or the discovery that is to produce something new, previously unknown and
useful information for the user. The term of data mining has emerged in the '60s, when the statisticians used it in
negative context when one analyzes the data without any presupposition. In the information technology it
appeared first in the database community in the '90s in the context of describing the sophisticated information
extraction. Although, the termof data mining is spread in the business, more synonym exists, for example,
knowledge discovery. It is important to distinguish between data mining and the challenging Big Data problems
nowadays. The solution of Big Data problems usually does not require the development of new theoretical
models or methods, the problem is rather that the well-working algorithms of data mining softwares hopelessly
slow down when you want to process a really large volumes of data as a whole instead of a reasonable sample
size. This obviously requires a special attitude and IT infrastructure that is outside the territory of the present
curriculum.
The data mining activity, in automatic or semi-automatic way, is integrated into the IT infrastructure of the
organization which applies it. This means that we can provide newer and newer information for the users by
data mining tools from the ever-changing data sources, typically from data warehouses, with relatively limited
human intervention. The reason is that because the (business) environment is constantly changing, following the
changes of the data warehouse which collects the data from the environment. Hence, the previously fitted data
mining models lose their validity, new models may need to model the altered data. Data mining softwares
increasingly support this approach in such a way that they are able to operate in very heterogeneous
environments. The collaboration between the information service and the supporting analytics nowadays allows
the development of real-time analytics based online systems, see, e.g., the recommendation systems.
The data mining is organized around the so-called data mining process, followed by the majority of data mining
softwares. In general, this is a five-step process where the steps are as follows:
• Sampling, data selection;
• Exploring,preprocessing;
• Modifying, transforming;
• Modelling;
• Interpreting, assessing.
The data mining softwares provide operators for these steps and we can carry out certain operations with them,
for example, reading an external file, filtering outliers, or fitting a neural network model. Representing the data
mining process by a diagram in a graphical interface, there are nodes corresponding to these operators.
Examples of this process are the SEMMA methodology of the SAS Institute Inc.® which is known about its
information delivery softwares and the widely used Cross Industry Standard Process for Data Mining (CRISPDM) methodology, which has evolved by cooperating of many branch of industry, e.g., finance, automotive,
information technology, etc.
During the sampling the target database is formed for the data mining process. The source of the data in most
cases is an enterprise (organization) data warehouse or its subject-oriented part, a so-called datamart. Therefore,
ii
Created by XMLmind XSL-FO Converter.
Preface
the data obtained from here have gone through a pre-processing phase in general, when they move from the
operational systems into the data warehouse, and thus they can be considered to be reliable. If this is not the case
then the used data mining software provides tools for data cleaning, which, in this case, can already be
considered as the second step of the process. Sampling can generally be done using an appropriate statistical
method, for example, simple random or stratified sampling method. Also in this step the dataset is partitioned to
training, validating, and testing set. On the training dataset the data mining model is fitted and its parameters
isestimated. The validating dataset is used to stop the convergence in the training process or compare different
models. By this method, using independent dataset from the training dataset, we obtain reliable decision where
to stop the training. Finally, on the test data set generalization ability of the model can be measured, that is how
it will be expected to behave in case of new records.
The exploring step means the acquaintance with the data without any preconception if it is possible. The
objective of the exploring step is to form hypotheses to establish in connection with the applicable procedures.
The main tools are the descriptive statistics and graphical vizualization. A data mining software has is a number
graphical tools which exceed the tools of standard statistical softwares. Another objective of the exploring is to
identify any existing errors (noise) and to find the places of missing data.
The purposes of modifying is the preparation of the data for fitting a data mining model. There may be several
reasons for this. One of them is that many methods require directly the modification of the data, for example, in
case of neural networks the attributes have to be standardized before the training of the network. An other one is
that if a method does not require the modification of the data, however a better fitting model is obtained after
suitable modification. An example is the normalization of the attributes by suitable transformations before
fitting a regression model in order the input attributes wiil be close to the normal distribution. The modification
can be carried out at multiple levels: at the level of entire attributes by transforming whole attributes, at the level
of the records, e.g., by standardizing some records or at the level of the fields by modifying some data. The
modifying step also includes the elimination of noisy and missing data, the so-called imputation as well.
The modeling is the most complex step of the data mining process and it requires the largest knowledge as well.
In essence, we solve here, after suitable preparation, the data mining task. The typical data mining tasks can be
divided into two groups. The first group is known as supervised data mining and supervised training. In this
case, there is an attribute with a special role in the dataset which is called target. The target variable should be
indicated in the used data mining software. Our task then is to describe this target variables by using the other
variables as well as we can. The second group is known as unsupervised data mining or unsupervised learning.
In this case, there is no special attributes in the analyzed dataset, where we want to explore hidden patterns.
Within the data mining, 6 task types can be defined, from which the classification and the regression are
supervised data mining and the segmentation, association, sequential analysis, and anomaly detection are
unsupervised data mining.
• Classification: modelling known classes (groups) for generalization purpose in order to apply the built model
for new records. Example: filtering emails by classifying them for spam and no-spam classes.
• Regression: building a model which approximates a continuous target by a function of input attributes such
that the error of this approximation is as small as possible. Example: estimating customer value by current
demographic and historical data.
• Segmenting, clustering: finding in a sense similar groups in the data without taking into account any known
existing structure. A typical example is the customer segmentation when a bank or insurance company is
looking for groups of clients behaving similarly.
• Association: searching relationships between attributes. Typical example is the basket analysis, when we look
at what goods are bought by the customers in the stores and supermarkets.
• Anomaly detection: identifying such records, which may be interesting or require further investigation due to
a mistake. Example is searching extremely behaved clients, users.
• Sequential analysis: searching temporal and spatial relationships between attributes. For example, in which
order are the services took by the customers or examining gene sequences.
The assessing of the results is the last step in the data mining process, which objective is to decide whether truly
relevant and useful knowledge is reached by the data mining process. Namely, it often happens that such model
is produced by the improper use of the data mining which has weak generalization ability and the model works
iii
Created by XMLmind XSL-FO Converter.
Preface
very poorly on new data. This is the so-called overfitting. In order to avoid the overfitting we should lean on the
training, validating, and testing dataset. At this step, we can also compare our fitted models if there are more
than one. In the comparison various measurements, e.g., misclassification rate, mean square error, and graphical
tools, e.g., lift curve, ROC curve, can be used.
This electronic curriculum aims at providing an introduction to data mining applications, so that it shows these
applications in practice through the use of data mining softwares. Problems requiring data mining can be
encountered in many fields of life. Some of these are listed below, datasets used in the course material also came
from these areas.
• Commercial data mining. One of the main driving forces behind the development and application of data
mining. Its objective is to analyze the static, historical business data stored in data warehouses in order to
explore hidden patterns and trends. Besides the standard way of collecting data, companies found out several
other ways to build more reliable data mining models, for example, this is also one of the main reasons behind
the spreading of loyalty cards. Among a number of specific application areas we emphasize the customer
relationship management (CRM): who are our customers and how to deal with them, the churn analysis:
which customers are planning to leave us, the cross-selling: what products should be offered together. The
algorithms of market basket analysis have been born in solving a business problem.
• Scientific data mining. The other main driving force behind the development of data mining. Many data
mining methods, for example, neural networks and self-organizing map, have been developed for solving a
scientific problem and they became a method of data mining only years later. The application areas are
ranging from the astronomy (galaxy classification and processing various radiation detected in space),
chemistry (forecasting, the properties of artificial molecules), and engineering sciences (material science,
traffic management) to biology (bioinformatics, drug discovery, and genetics). Data mining can help in areas
where the problem of data gap appears, i.e., far more data is generated than the scientist is able to process.
• Mining medical data. The development of health information technology makes possible for doctors to share
their diagnostic results with each other and thus it enables not to repeate doing an examination several times.
Moreover, by collecting the diagnosis as the results of examination in a common data warehouse, it will be
possible to develope new medical procedures by means of data mining techniques. Data mining is also likely
to play an important role in personalized medicine as well.
• Spatial data mining. Analysis of spatial data with data mining methods, the extension of the traditional
geographic information systems (GIS) with data mining tools. Application areas: climate research, the spread
of epidemics, customer analysis of large multinational companies taking into account the spatial dimension.
An important area in the future will be the processing of data generated in sensor networks, e.g., pollution
monitoring on an area.
• Multimédia adatbányászat. Analysis of audio, image and video files by data mining tools. Data mining can
help to find similarities between songs in order to decide copyright issues more objectively. Another
application is finding copyright law conflicting or illegal contents in file-sharing systems and in multimedia
service providers.
• Web mining. Analysis of web data. Three types of web mining problems are distinguished: web structure
mining, web content mining, and web usage mining. The web structure mining means the examination of the
structure of the Web, i.e., the examination of the web-graph, where the set of vertices consists of the sites and
the set of edges consists of the links between the sites. The web content mining means the retrieval of useful
information from the contents on the web. The well-known web search engines (Google, AltaVista, etc) also
carry out this thing. The web usage mining deals with examining what users are searching for on the Internet,
using the the data gathered by Web servers and application servers. These areas are strongly related to Big
Data problems because it is needed to work many times over an Internet-scale infrastructure.
• Text mining. Mining of unstructured or semi-structured data. Under unstructured data we mean continuous
texts (sequences of strings), which may be connected by a theme (e.g., scienctific), by a field (e.g., sport), or
they can be customer's sentiments at a customer service. Semi-structured data are typically produced by
computers or files produced for computers, for example, in XML or JSON format. Some specific
applications: data mining for security reasons, e.g., searching for terrorists, analytical CRM, sentiment
analysis, and academic applications (plagiarism investigation).
1. How to use this material
iv
Created by XMLmind XSL-FO Converter.
Preface
The RapidMiner and SAS® Enterprise Miner™ workflows presented in this course material are contained in the
file resources/workflows.zip.
Fontos
Data files used in the experiments must be downloaded by the user from the location specified in the
text. After importing a workflow file paths must be set to point to the local copies of data files
(absolute paths are required).
v
Created by XMLmind XSL-FO Converter.
Part I. Data mining tools
Introduction
In this part, data mining tools and softwares are overviewed. There are three necessary conditions of the
succesful data mining. First, we need an appropriate data sets to perform data mining. In practice, this is often a
task-oriented data-mart generated from the enterprise data warehouse. In the education, and so in this
curriculum, the datasets are taken from widely used data repository. All datasets are attached to this material.
Another important condition to data mining is the data mining expert. We hope that this curriculum will be able
to contribute to the education of this professionals. Finally, the key is the software with which data mining is
performed. They can be classified on the basis of several criteria, e.g., business or free, self-contained or
integrated, general or specific, theme-oriented or not. The most up-to-date information on this topic can be
found on the website KDnuggets™. The reader can get fresh information on current job openings, courses,
conferences etc. from here as well.
In the curriculum two softwares are discussed in detail: a leading one from the free data mining softwares,
RapidMiner 3.5 and one of the most widely used commercial data mining softwares, SAS® Enterprise Miner™
Version 7.1. The list of the data mining softwares below is based on the KDnuggets™ portal.
Created by XMLmind XSL-FO Converter.
Chapter 1. Commercial data mining
softwares
• AdvancedMiner Professional , (formerly Gornik), provides a wide range of tools for data transformations,
Data Mining models, data analysis and reporting.
• Alteryx, offering Strategic Analytics platform, including a free Project Edition version.
• Angoss Knowledge Studio, a comprehensive suite of data mining and predictive modeling tools;
interoperability with SAS and other major statistical tools.
• BayesiaLab, a complete and powerful data mining tool based on Bayesian networks, including data
preparation, missing values imputation, data and variables clustering, unsupervised and supervised learning.
• BioComp i-Suite, constraint-based optimization, cause and effect analysis, non-linear predictive modeling,
data access and cleaning, and more.
• BLIASoft Knowledge Discovery, for building models from data based mainly on fuzzy logic.
• CMSR Data Miner, built for business data with database focus, incorporating rule-engine, neural network,
neural clustering (SOM), decision tree, hotspot drill-down, cross table deviation analysis, cross-sell analysis,
visualization/charts, and more.
• Data Applied, offers a comprehensive suite of web-based data mining techniques, an XML web API, and rich
data visualizations.
• Data Miner Software Kit, collection of data mining tools, offered in combination with a book: Predictive Data
Mining: A Practical Guide, Weiss and Indurkhya.
• DataDetective, the powerful yet easy to use data mining platform and the crime analysis software of choice
for the Dutch police.
• Dataiku, a software platform for data science, statistics, guided machine learning and visualization
capabilities, built on Open Source, Hadoop integration.
• DataLab, a complete and powerful data mining tool with a unique data exploration process, with a focus on
marketing and interoperability with SAS.
• DBMiner 2.0 (Enterprise), powerful and affordable tool to mine large databases; uses Microsoft SQL Server
7.0 Plato.
• Delta Miner, integrates new search techniques and "business intelligence" methodologies into an OLAP frontend that embraces the concept of Active Information Management.
• ESTARD Data Miner, simple to use, designed both for data mining experts and common users.
• Exeura Rialto™ , provides comprehensive support for the entire data mining and analytics life-cycle at an
affordable price in a single, easy-to-use tool.
• Fair Isaac Model Builder, software platform for developing and deploying analytic models, includes data
analysis, decision tree and predictive model construction, decision optimization, business rules management,
and open-platform deployment.
• FastStats Suite (Apteco), marketing analysis products, including data mining, customer profiling and
campaign management.
• GainSmarts, uses predictive modeling technology that can analyze past purchase, demographic, and lifestyle
data, to predict the likelihood of response and develop an understanding of consumer characteristics.
• Generation5 GenVoy, On-Demand Consumer Analytics.
7
Created by XMLmind XSL-FO Converter.
Commercial data mining softwares
• GenIQ Model, uses machine learning for regression task; automatically performs variable selection, and new
variable construction, and then specifies the model equation to "optimize the decile table".
• GhostMiner, complete data mining suite, including k-nearest neighbors, neural nets, decision tree,
neurofuzzy, SVM, PCA, clustering, and visualization.
• GMDH Shell, an advanced but easy to use tool for predictive modeling and data mining. Free trial version is
available.
• Golden Helix Optimus RP, uses Formal Inference-based Recursive Modeling (recursive partitioning based on
dynamic programming) to find complex relationships in data and to build highly accurate predictive and
segmentation models.
• IBM SPSS Modeler, (formerly Clementine), a visual and powerful data mining workbench.
• Insights, (formerly KnowledgeMiner) 64-bit parallel software for autonomously building reliable predictive
analytical models from high-dimensional noisy data using outstanding self-organizing knowledge mining
technologies. Model export to Excel. Localized for English, Spanish, German. Free trial version.
• JMP, offers significant visualization and data mining capabilities along with classical statistical analyses.
• K. wiz, from thinkAnalytics - massively scalable, embeddable, Java-based real-time data-mining platform.
Designed for Customer and OEM solutions.
• Kaidara Advisor, (formerly Acknosoft KATE), Case-Based Reasoning (CBR) and data mining engine.
• Kensington Discovery Edition, high-performance discovery platform for life sciences, with multi-source data
integration, analysis, visualization, and workflow building.
• Kepler, extensible, multi-paradigm, multi-purpose data mining system.
• Knowledge Miner, a knowledge mining tool that works with data stored in Microsoft Excel for building
predictive and descriptive models. (MacOS, Excel 2004 or later).
• Kontagent kSuite DataMine, a SaaS User Analytics platform offering real-time behavioral insights for Social,
Mobile and Web, offering SQL-like queries on top of Hadoop deployments.
• KXEN (SAP company), providing Automated Predictive Analytics tools for Big Data.
• LIONsolver (Learning and Intelligent OptimizatioN) Learning and Intelligent OptimizatioN: modeling and
optimization with "on the job learning" for business and engineering by Reactive Search SrL.
• LPA Data Mining tools, support fuzzy, bayesian and expert discovery and modeling of rules.
• Magnify PATTERN, software suite, contains PATTERN:Prepare for data preparation; PATTERN:Model for
building predictive models; and PATTERN:Score for model deployment.
• Mathematica solution for Data Analysis and Mining, from Wolfram.
• MCubiX, a complete and affordable data mining toolbox, including decision tree, neural networks,
associations rules, visualization, and more.
• Microsoft SQL Server, empowers informed decisions with predictive analysis through intuitive data mining,
seamlessly integrated within the Microsoft BI platform, and extensible into any application.
• Machine Learning Framework, provides analysis, prediction, and visualization using fuzzy logic and ML
methods; implemented in C++ and integrated into Mathematica.
• Molegro Data Modeller, a cross-platform application for Data Mining, Data Modelling, and Data
Visualization.
• Nuggets, builds models that uncover hidden facts and relationships, predict for new data, and find key
variables (Windows).
8
Created by XMLmind XSL-FO Converter.
Commercial data mining softwares
• Oracle Data Mining (ODM), enables customers to produce actionable predictive information and build
integrated business intelligence applications.
• Palisade DecisionTools Suite, complete risk and decision analysis toolkit.
• Partek, pattern recognition, interactive visualization, and statistical analysis, modeling system.
• Pentaho, open-source BI suite, including reporting, analysis, dashboards, data integration, and data mining
based on Weka.
• Polyanalyst, comprehensive suite for data mining, now also including text analysis, decision forest, and link
analysis. Supports OLE DB for Data Mining, and DCOM technology.
• Powerhouse Data Mining, for predictive and clustering modelling, based on Dorian Pyle's ideas on using
Information Theory in data analysis. Most information is in Spanish.
• Predictive Dynamix, integrates graphical and statistical data analysis with modeling algorithms including
neural networks, clustering, fuzzy systems, and genetic algorithms.
• Previa, family of products for classification and forecasting.
• Quadstone DecisionHouse, an agile analytics solution with a complete suite of capabilities to support end-toend data mining cycle.
• RapAnalyst™ , uses advanced artificial intelligence to create dynamic predictive models, to reveal
relationships between new and historical data.
• Rapid Insight Analytics, streamlines the predictive modeling and data exploration process, enabling users of
all abilities to quickly build, test, and implement statistical models at lightning speed.
• Reel Two, real-time classification software for structured and unstructured data as well entity extraction.
From desktop to enterprise.
• Salford Systems Data Mining Suite, CART Decision Trees, MARS predictive modeling, automated
regression, TreeNet classification and regression, data access, preparation, cleaning and reporting modules,
RandomForests predictive modeling, clustering and anomaly detection.
• SAS® Enterprise Miner™ , an integrated suite which provides a user-friendly GUI front-end to the SEMMA
(Sample, Explore, Modify, Model, Assess) process.
• SPAD, provides powerful exploratory analyses and data mining tools, including PCA, clustering, interactive
decision trees, discriminant analyses, neural networks, text mining and more, all via user-friendly GUI.
• Statistica Data Miner, a comprehensive, integrated statistical data analysis, graphics, data base management,
and application development system.
• Synapse, a development environment for neural networks and other adaptive systems, supporting the entire
development cycle from data import and preprocessing via model construction and training to evaluation and
deployment; allows deployment as .NET components.
• Teradata Warehouse Miner and Teradata Analytics, providing analytic services for in-place mining on a
Teradata DBMS.
• thinkCRA , from thinkAnalytics, an integrated suite of Customer Relationship Analytics applications
supporting real-time decisioning.
• TIBCO Spotfire Miner, combining Spotfile visualization, Insightful Miner, S+ with intuitive drag-and-drop
user interface.
• TIMi Suite: The Intelligent Mining machine, a family of stand-alone, automated, user-friendly GUI tools for
prediction, segmentation and data preparation, with high scalability, speed, ROI, prediction accuracy (a
recurrent top winner at KDD cups).
9
Created by XMLmind XSL-FO Converter.
Commercial data mining softwares
• Viscovery data mining suite, a unique, comprehensive data mining suite for business applications with
workflow-guided project environment; includes modules for visual data mining, clustering, scoring,
automation and real-time integration.
• Xeno, InfoCentricity powerful, user-friendly online analytic platform, supporting segmentation, clustering,
exploratory data analysis, and the development of highly predictive models.
• XLMiner, Data Mining Add-In For Excel.
• Xpertrule Miner, (Attar Software) features data transformation, Decision Trees, Association Rules and
Clustering on large scale data sets.
10
Created by XMLmind XSL-FO Converter.
Chapter 2. Free and shareware data
mining softwares
• Alteryx Project Edition, free version of Alteryx, delivers the data blending, analytics, and sharing capabilities
of Alteryx with just enough allowed runs of your workflow to solve one business problem or to complete one
project.
• AlphaMiner, open source data mining platform that offers various data mining model building and data
cleansing functionality.
• CMSR Data Miner, built for business data with database focus, incorporating rule-engine, neural network,
neural clustering (SOM), decision tree, hotspot drill-down, cross table deviation analysis, cross-sell analysis,
visualization/charts, and more. Free for academic use.
• CRAN Task View: Machine Learning and Statistical Learning, machine learning and statistical packages in
R.
• Databionic ESOM Tools, a suite of programs for clustering, visualization, and classification with Emergent
Self-Organizing Maps (ESOM).
• ELKI: Environment for DeveLoping KDD-Applications Supported by Index-Structures, a framework in Java
which includes clustering, outlier detection, and other algorithms; allows user to evaluate the combination of
arbitrary algorithms, data types, and distance functions.
• Gnome Data Mining Tools, including apriori, decision trees, and Bayes classifiers.
• jHepWork, an interactive environment for scientific computation, data analysis and data visualization
designed for scientists, engineers and students.
• KEEL, includes knowledge extraction algorithms, preprocessing techniques, evolutionary rule learning,
genetic fuzzy systems, and more.
• KNIME, extensible open source data mining platform implementing the data pipelining paradigm (based on
eclipse).
• Machine Learning in Java (MLJ), an open-source suite of Java tools for research in machine learning. (The
software will not be developed further.)
• MiningMart, a graphical tool for data preprocessing and mining on relational databases; supports
development, documentation, re-use and exchange of complete KDD processes. Free for non-commercial
purposes.
• ML-Flex, an open-source software package designed to enable flexible and efficient processing of disparate
data sets for machine-learning (classification).
• MLC++, a machine learning library in C++.
• Orange, open source data analytics and mining through visual programming or Python scripting. Components
for visualization, rule learning, clustering, model evaluation, and more.
• PredictionIO, an open source machine learning server for software developers and data engineers to create
predictive features, such as personalization, recommendation and content discovery.
• RapidMiner , a leading open-source system for knowledge discovery and data mining.
• Rattle, a data mining suite based on open source statistical language R, includes graphics, clustering,
modeling, and more.
• TANAGRA, offers a GUI interface and methods for data access, statistics, feature selection, classification,
clustering, visualization, association and more.
11
Created by XMLmind XSL-FO Converter.
Free and shareware data mining
softwares
• Weka, collection of machine learning algorithms for solving real-world data mining problems. It is written in
Java and runs on almost any platform.
12
Created by XMLmind XSL-FO Converter.
Part II. RapidMiner
Created by XMLmind XSL-FO Converter.
Table of Contents
3. Data Sources ................................................................................................................................. 16
1. Importing data from a CSV file ........................................................................................... 16
2. Importing data from an Excel file ....................................................................................... 17
3. Creating an AML file for reading a data file ....................................................................... 19
4. Importing data from an XML file ....................................................................................... 21
5. Importing data from a database ........................................................................................... 23
4. Pre-processing .............................................................................................................................. 25
1. Managing data with issues - Missing, inconsistent, and duplicate values ........................... 25
2. Sampling and aggregation ................................................................................................... 27
3. Creating and filtering attributes ........................................................................................... 31
4. Discretizing and weighting attributes .................................................................................. 35
5. Classification Methods 1 .............................................................................................................. 41
1. Classification using a decision tree ..................................................................................... 41
2. Under- and overfitting of a classification with a decision tree ............................................ 46
3. Evaluation of performance for classification by decision tree ............................................ 51
4. Evaluation of performance for classification by decision tree 2 ......................................... 55
5. Comparison of decision tree classifiers ............................................................................... 58
6. Classification Methods 2 .............................................................................................................. 65
1. Using a rule-based classifier (1) .......................................................................................... 65
2. Using a rule-based classifier (2) .......................................................................................... 66
3. Transforming a decision tree to an equivalent rule set ........................................................ 68
7. Classification Methods 3 .............................................................................................................. 71
1. Linear regression ................................................................................................................. 71
2. Osztályozás lineáris regresszióval ....................................................................................... 73
3. Evaluation of performance for classification by regression model ..................................... 76
4. Evaluation of performance for classification by regression model 2 .................................. 79
8. Classification Methods 4 .............................................................................................................. 84
1. Using a perceptron for solving a linearly separable binary classification problem ............. 84
2. Using a feed-forward neural network for solving a classification problem ........................ 85
3. The influence of the number of hidden neurons to the performance of the feed-forward neural
network ................................................................................................................................... 87
4. Using a linear SVM for solving a linearly separable binary classification problem ........... 88
5. The influence of the parameter C to the performance of the linear SVM (1) ...................... 90
6. The influence of the parameter C to the performance of the linear SVM (2) ...................... 93
7. The influence of the parameter C to the performance of the linear SVM (3) ...................... 95
8. The influence of the number of training examples to the performance of the linear SVM . 97
9. Solving the two spirals problem by a nonlinear SVM ...................................................... 100
10. The influence of the kernel width parameter to the performance of the RBF kernel SVM 101
11. Search for optimal parameter values of the RBF kernel SVM ........................................ 103
12. Using an SVM for solving a multi-class classification problem ..................................... 105
13. Using an SVM for solving a regression problem ............................................................ 106
9. Classification Methods 5 ............................................................................................................ 110
1. Introducing ensemble methods: the bagging algorithm .................................................... 110
2. The influence of the number of base classifiers to the performance of bagging ............... 111
3. The influence of the number of base classifiers to the performance of the AdaBoost method 113
4. The influence of the number of base classifiers to the performance of the random forest 115
10. Association rules ....................................................................................................................... 118
1. Extraction of association rules .......................................................................................... 118
2. Asszociációs szabályok kinyerése nem tranzakciós adathalmazból .................................. 121
3. Evaluation of performance for association rules ............................................................... 126
4. Performance of association rules - Simpson's paradox ..................................................... 131
11. Clustering 1 .............................................................................................................................. 135
1. K-means method ............................................................................................................... 135
2. K-medoids method ............................................................................................................ 137
3. The DBSCAN method ...................................................................................................... 140
4. Agglomerative methods .................................................................................................... 142
14
Created by XMLmind XSL-FO Converter.
RapidMiner
5. Divisive methods ...............................................................................................................
12. Clustering 2 ..............................................................................................................................
1. Support vector clustering ..................................................................................................
2. Choosing parameters in clustering ....................................................................................
3. Cluster evaluation .............................................................................................................
4. Centroid method ................................................................................................................
5. Text clustering ...................................................................................................................
13. Anomaly detection ....................................................................................................................
1. Searching for outliers ........................................................................................................
2. Unsupervised search for outliers .......................................................................................
3. Unsupervised statistics based anomaly detection ..............................................................
15
Created by XMLmind XSL-FO Converter.
144
148
148
151
154
159
161
165
165
167
171
Chapter 3. Data Sources
1. Importing data from a CSV file
Description
The process demonstrates how to import data from CSV files using the Read CSV and the Open File operators.
In the experiment we use a real-time earthquake data feed provided by USGS in CSV format. First, we
download the feed to able to import it into RapidMiner using the Import Configuration Wizard of the Read
CSV operator. The wizard guides the user step by step through the import process and helps him to set the
parameters of the operator correctly. After the local copy of the feed is successfully imported into RapidMiner,
we can switch to the live feed adding the Open File to the process.
Input
The United States Geological Survey (or USGS for short) provides real time earthquake data feeds at the
Earthquake Hazards Program website. Data is available in various formats, including CSV. The experiment uses
the feed of the magnitude 1+ earthquakes in the past 30 days in CSV format from the
URLhttp://earthquake.usgs.gov/earthquakes/feed/v1.0/csv.php. The feed is updated in every 15 minutes.
Output
An ExampleSet that contains data imported from the CSV feed.
Figure 3.1. Metadata of the resulting ExampleSet.
16
Created by XMLmind XSL-FO Converter.
Data Sources
Figure 3.2. A small excerpt of the resulting ExampleSet.
Interpretation of the results
Each time the process is run, it will read live data from the web.
Video
Workflow
import_exp1.rmp
Keywords
importing data
CSV
Operators
Open File
Read CSV
2. Importing data from an Excel file
17
Created by XMLmind XSL-FO Converter.
Data Sources
Description
The process demonstrates how to import data from Excel files using the Read Excel operator. The Concrete
Compressive Strength data set is used in the experiment. Parameters of the Read Excel operator are set via its
Import Configuration Wizard.
Input
Concrete Compressive Strength [UCI MLR] [Concrete]
Output
An ExampleSet that contains data imported from the Excel file.
Figure 3.3. Metadata of the resulting ExampleSet.
Figure 3.4. A small excerpt of the resulting ExampleSet.
18
Created by XMLmind XSL-FO Converter.
Data Sources
Interpretation of the results
Video
Workflow
import_exp2.rmp
Keywords
importing data
Excel
Operators
Guess Types
Open File
Read Excel
Rename by Generic Names
3. Creating an AML file for reading a data file
19
Created by XMLmind XSL-FO Converter.
Data Sources
Description
The process demonstrates how to create an AML file using the Read AML operator for reading a data file. AML
files are XML documents that provide metadata about attributes including their names, datatypes, and roles.
Once the AML file is created it can be used to read the underlying data file properly.
Input
Pima Indians Diabetes [UCI MLR]
Output
An AML file in the file system and an ExampleSet that contains data imported from the data file using the AML
file.
Figure 3.5. The resulting AML file.
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<attributeset default_source="pima-indians-diabetes.data" encoding="UTF-8">
<attribute name="Number of times pregnant " sourcecol="1" valuetype="integer"/>
<attribute name="Plasma glucose concentration" sourcecol="2" valuetype="integer"/>
<attribute name="Diastolic blood pressure" sourcecol="3" valuetype="integer"/>
<attribute name="Triceps skin fold thickness" sourcecol="4" valuetype="integer"/>
<attribute name="2-Hour serum insulin" sourcecol="5" valuetype="integer"/>
<attribute name="Body mass index" sourcecol="6" valuetype="real"/>
<attribute name="Diabetes pedigree function" sourcecol="7" valuetype="real"/>
<attribute name="Age" sourcecol="8" valuetype="integer"/>
<label name="Class" sourcecol="9" valuetype="binominal">
<value>1</value>
<value>0</value>
</label>
</attributeset>
Interpretation of the results
The resulting AML file is intended to be distributed together with the data file.
Video
Workflow
20
Created by XMLmind XSL-FO Converter.
Data Sources
import_exp3.rmp
Keywords
importing data
AML
Operators
Read AML
4. Importing data from an XML file
Description
The process demonstrates how to import data from XML documents using the Read XML operator. Parameters
of the Read XML operator are set via its Import Configuration Wizard. Attribute values are extracted from
the XML document using XPath location paths.
Input
The experiment uses population data in XML from the World Bank Open Data website. The data set is available
at http://data.worldbank.org/indicator/SP.POP.TOTL in various formats, including XML.
Figure 3.6. A small excerpt of The World Bank: Population (Total) data set used in the
exepriment.
<?xml version="1.0" encoding="utf-8"?>
<Root xmlns:wb="http://www.worldbank.org">
<data>
<record>
<field name="Country or Area" key="ABW">Aruba</field>
<field name="Item" key="SP.POP.TOTL">Population (Total)</field>
<field name="Year">1960</field>
<field name="Value">54208</field>
</record>
<record>
<field name="Country or Area" key="ABW">Aruba</field>
<field name="Item" key="SP.POP.TOTL">Population (Total)</field>
<field name="Year">1961</field>
<field name="Value">55435</field>
</record>
<record>
21
Created by XMLmind XSL-FO Converter.
Data Sources
<field name="Country or Area" key="ABW">Aruba</field>
<field name="Item" key="SP.POP.TOTL">Population (Total)</field>
<field name="Year">1962</field>
<field name="Value">56226</field>
</record>
<!-- ... -->
</Root>
Output
An ExampleSet that contains data imported from the XML document.
Figure 3.7. Metadata of the resulting ExampleSet.
Figure 3.8. A small excerpt of the resulting ExampleSet.
Interpretation of the results
Video
22
Created by XMLmind XSL-FO Converter.
Data Sources
Workflow
import_exp4.rmp
Keywords
importing data
XML
Operators
Read XML
5. Importing data from a database
Description
The process demonstrates how to connect to an Oracle database and execute an SQL query.
Input
The experiment uses a local database server running at the Faculty of Informatics, University of Debrecen. Note
that connection to the database server requires authentication.
Output
An ExampleSet that contains the result of the SQL query.
Figure 3.9. Metadata of the resulting ExampleSet.
23
Created by XMLmind XSL-FO Converter.
Data Sources
Figure 3.10. A small excerpt of the resulting ExampleSet.
Interpretation of the results
Video
Workflow
import_exp5.rmp
Keywords
importing data
database
SQL
Operators
Read Database
24
Created by XMLmind XSL-FO Converter.
Chapter 4. Pre-processing
1. Managing data with issues - Missing, inconsistent,
and duplicate values
Description
The process shows, using a sample of the Individual household electric power consumption data set, how to
manage such data sets in which duplicate, inconsistent, and/or duplicate values are present. The missing values
can be substituted with a default value, or one computed based on the other instances of the field, or if it is
necessary, the records belonging to them can be deleted. After defining the inconsistent values, the records
belonging to these can also be filtered out, but however, to define these, it is usually necessary to have some
background knowledge or knowledge in the given field. On the contrary, filtering duplicate values is a rather
automated task, the data identical to each other can be filtered out easily.
Input
Individual household electric power consumption [UCI MLR]
Output
The dataset used here is a sample from the original dataset that encompasses a longer time period; it only
contains the energy consumption data from January 2007. Normally the dataset contains a measurement for
every minute, but if a given measurement was not executed for some reason, the timestamp is present without
data. Such missing values can be substituted with defined values, e.g. with the average of the values present in
the given attribute, or a decision to leave out the records connected to them can be made. However, dealing with
inconsistent values is a more complex issue. In given fields, intervals can be defined into which the values of the
attributes can fall, but in other cases, one must relay on other kinds of background information. For example, let
us suppose that the members of the household, in which the measurements took place, do not fly with the owls.
Based on this, consider the following representation of the data:
Figure 4.1. Graphic representation of the global and kitchen power consumption in time
25
Created by XMLmind XSL-FO Converter.
Pre-processing
On the figure, colors are assigned based on the values of the variable Sub_metering_1, which represents the
energy consumption of the tools in the kitchen, and as the x axis is time, it can be seen that some of the
outstanding kitchen consumption values have been measured late at night. This can also be seen in the data
view, if the data are ordered according to the kitchen consumption values:
Figure 4.2. Possible outliers based on the hypothesized hbits of the members of the
household
If the members of the household are indeed not flying with the owls, these data can be considered inconsistent
based on our background knowledge, and they can be filtered out as follows. Formulating our condition, let us
assume that if the values of a measurement in the kitchen exceeds 50 Wh at a point of time after 10 p.m., this is
considered a piece of inconsistent data. Based on this, the filtering condition can easily be defined, but first, the
time attribute has to be converted, as by default, it is stored in a nominal variable, in the format hh:mm:ss, which
can only be compared for equality. Using the appropriate operators, it can be splitted into the components hour,
minute, and second, interpreted as numbers. The filtering condition can be defined using the Time_1 variable
from among these, which contains the hour component:
Figure 4.3. Filtering of the possible values using a record filter
26
Created by XMLmind XSL-FO Converter.
Pre-processing
Interpretation of the results
Using such filters, the records belonging to inconsistent data can be filtered out, and also, by using the
appropriate operator, duplicate records can be filtered out as well - it can also be defined based on the equality
of which attribute set they are to be considered duplicate - from the dataset, and after this, the actual processing
of the filtered and/or refined records can begin.
Video
Workflow
preproc_exp1.rmp
Keywords
missing data
inconsistent data
data transformation
duplicate data
Operators
Filter Examples
Parse Numbers
Read CSV
Remove Duplicates
Replace Missing Values
Split
2. Sampling and aggregation
27
Created by XMLmind XSL-FO Converter.
Pre-processing
Description
The process shows, using a sample of the Individual household electric power consumption dataset, how the
data can be summed up using aggregation or sampled if not all of the individual records are required during the
given process. Aggregation can be used if the individual data are not necessary, but the values computed from
the whole of the dataset are required, and sampling can be done if generally only a fraction of the dataset is
required, and conclusions are to be derived based on this subset of the data.
Input
Individual household electric power consumption [UCI MLR]
Output
When aggregating, all the aggregate functions available in SQL can be used, and using these, basic statistics can
easily be computed for the data of the given dataset.
Figure 4.4. Selection of aggregate functions for attributes
28
Created by XMLmind XSL-FO Converter.
Pre-processing
If sampling is done on the dataset, this can be done by explicitly specifying the size of the sample, or based on
probability, and also a filter can be used, in the case when the parts of the dataset are not to be represented
proportionally in the sample, rather a given subset of the original dataset is necessary for the process. For
example, filtering for the records belonging to a given time on every day can be done as follows:
Figure 4.5. Preferences for dataset sampling
Figure 4.6. Preferences for dataset filtering
29
Created by XMLmind XSL-FO Converter.
Pre-processing
Interpretation of the results
After performing the aggregation or sampling, the received dataset will only consist of the aggregate values
emerging as a result of the specified operations, or the records that fulfil the specified conditions, respectively:
Figure 4.7. Resulting dataset after dataset sampling
Figure 4.8. Resulting dataset after dataset filtering
30
Created by XMLmind XSL-FO Converter.
Pre-processing
Video
Workflow
preproc_exp2.rmp
Keywords
aggregation
summation
sampling
data filtering
Operators
Aggregate
Filter Examples
Multiply
Read CSV
Sample
3. Creating and filtering attributes
31
Created by XMLmind XSL-FO Converter.
Pre-processing
Description
The process shows, using The Insurance Company Benchmark (COIL 2000) dataset, how new, computed
attributes can be created based on existing data, in case the attributes are not appropriate in their original form,
or some data derived from them is required. Furthermore, how the individual attributes can be removed from the
dataset can also be seen in the process, as in these cases, if the raw data that form the basis of the calculation are
not necessarily required later on, these columns can be removed from the dataset. Naturally, other columns can
be filtered out as well, if they are not required for the solution of the given task, or if their disturbing effects are
needed to be filtered out.
Input
The Insurance Company Benchmark (COIL 2000) [CoIL Challenge 2000]
Output
In the attributes of the dataset which begin with the letter m, the demographic data of the region belonging to the
zip-code of the given potential client are present; among others, the distribution of individual income groups in
the given region. If, for some reason, the original representation is to be compressed, it is possible to create a
derived field based on these income attributes using a given formula, based on some heuristic, for example as
follows:
Figure 4.9. Defining a new attribute based on an expression relying on existing
attributes
32
Created by XMLmind XSL-FO Converter.
Pre-processing
After the appropriate computed field has been created, based on the given case, it can be decided whether the
original fields used during the computation are required later on or not. It has to be considered whether these
original data could be important for the creation of models in the future, or whether they could have some
disturbing effect. The attributes of the raw data used for the computation, or any other arbitrarily selected
attributes can be removed from the original dataset as follows:
Figure 4.10. Properties of the operator used for removing the attributes made
redundant
Figure 4.11. Selection of the attributes to remain in the dataset with reduced size
33
Created by XMLmind XSL-FO Converter.
Pre-processing
Interpretation of the results
After executing these steps, all records will appear in the modified dataset, but with a modified attribute set.
After the computed field has been created, this new attribute appears in every record, while the attributes filtered
out disappear:
Figure 4.12. The appearance of the derived attribute in the altered dataset
34
Created by XMLmind XSL-FO Converter.
Pre-processing
Video
Workflow
preproc_exp3.rmp
Keywords
derived attribute
attribute creation
attribute removal
attribute subset
Operators
Generate Attributes
Read AML
Select Attributes
4. Discretizing and weighting attributes
35
Created by XMLmind XSL-FO Converter.
Pre-processing
Description
The process shows, using a sample of the Individual household electric power consumption dataset, how an
attribute that takes its values from an interval of real numbers can be discretized, i.e. converted to discrete values
that represent defined subintervals of the real interval. Furthermore, it can also be seen in the process, how
weights can be added to the individual data columns, if when using a data minding procedure, it is necessary to
distinguish between the individual data regarding their importance, and not to allow all attributes take part in the
data mining algorithm, and in the conclusions based on it with equal weights.
Input
Individual household electric power consumption [UCI MLR]
Output
In the dataset, the usage of discretization is shown using the variable Global_active_power. This variable
represents the total energy consumption in the whole of the household at a given moment, so following the
changes of the times of day, these values also change in a cyclic fashion. Thus if the total consumption is to be
represented with discrete values, and not real numbers, in order to be used in a given method, then this column
can be properly discretized. Discretization can be done using different operators, by defining the size of the
categories (the number of elements in them), or the number of categories, and based on this number, either
categories of equal size, or ones of equal element numbers can be created, for example as follows:
Figure 4.13. Selection of the appropriate discretization operator
36
Created by XMLmind XSL-FO Converter.
Pre-processing
Figure 4.14. Setting the properties of the discretization operator
Furthermore, using given methods, to receive a result or decision that is appropriate for the requirements later
on, it has to be defined, which attributes have what level of importance - the simplest way to do this is
weighting. In order to be able to weight the attributes, the weights themselves have to be created first, and then
they have to be applied to the dataset. For example, such weights can be set manually for this dataset, whit
which it can be indicated that the globally measured values are of most importance, the submeterings are of less
importance, and the date and time values are of the least importance, as follows:
Figure 4.15. Selection of the appropriate weighting operator
37
Created by XMLmind XSL-FO Converter.
Pre-processing
Figure 4.16. Defining the weights of the individual attributes
38
Created by XMLmind XSL-FO Converter.
Pre-processing
Interpretation of the results
After executing these steps, the value of the variable Global_active_power will be modified in all the records
of the dataset. Here it can be seen that the division into intervals has been done, but behind the discrete values,
the interval the values falling into which are corresponding to the given value are displayed as well. In addition,
by comparing the original and the modified dataset (the weighted dataset can be seen on the left, and the
unweighted dataset can be seen on the right, in their state after the discretization has been executed), it can also
be seen that the numeric values in the individual columns have been altered according to the weighting (as the
normalize weights option is turned on, the greatest weight is considered 1, thus the values of the columns to
which this weight has been assigned are not subject to change, and the values of the columns to which smaller
weights have been assigned decrease proportionally):
Figure 4.17. Comparison of the weighted and unweighted dataset instances
39
Created by XMLmind XSL-FO Converter.
Pre-processing
Video
Workflow
preproc_exp4.rmp
Keywords
attribute discretization
attribute weighting
weighting
discretization
Operators
Discretize by Binning
Multiply
Read CSV
Scale by Weights
Weight by User Specification
40
Created by XMLmind XSL-FO Converter.
Chapter 5. Classification Methods 1
Decision Trees
1. Classification using a decision tree
Description
The process shows, using the Wine dataset, how classification can be executed by building a decision tree. To
build the decision model, first the dataset has to be split into training and testing sets. After this, the splitting
rules are ordered into a decision tree based on the training set, and afterwards, the model created in such a way
will be used on the test set. Later on, it can be checked what decision conditions the model consists of, based on
the training set, and to which class the records of the test set have been assigned based on these decisions.
Input
Wine [UCI MLR]
Output
The decisions about the individual splits can be done based on measures such as the Gini-index or information
gain. For these, and for the confidence level of splits, different parameter values can be defined when creating
the decision tree model. Furthermore, the stop conditions of splitting can also be defined either by specifying the
minimal size of record sets that can be split further, or by specifying the maximal depth of the tree.
Figure 5.1. Preferences for the building of the decision tree
41
Created by XMLmind XSL-FO Converter.
Classification Methods 1
When splitting the dataset, various sampling methods can be defined, and the ratio in which it should be split
into training and test sets can also be specified. Splitting can be done simply based on the order of the records,
randomly, or attending to it that records belonging to each class occur in the same ratio as in the original dataset
in the training and test sets.
Figure 5.2. Preferences for splitting the dataset into training and test sets
Figure 5.3. Setting the relative sizes of the data partitions
42
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Interpretation of the results
After it has been built, the model itself can also be directed to the output, thus it can be checked what decision
tree has been built based on the data of the training set. Based on this, incidental erroneous decisions can be
filtered out using background information or domain knowledge, if any of these is available. If such decisions
are found, the process of building the model can be tuned further. Besides this, by applying the model to the test
set, it can also be seen which classes the records of the test set have been assigned to based on the model trained
using the training set.
Figure 5.4. Graphic representation of the decision tree created
43
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Figure 5.5. The classification of the records based on the decision tree
44
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Video
Workflow
dtree_exp1.rmp
Keywords
classification
decision tree
splitting
Operators
Apply Model
45
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Decision Tree
Multiply
Read AML
Split Data
2. Under- and overfitting of a classification with a
decision tree
Description
The process shows, using the Zoo dataset, under which conditions can under- and oversampling appear when
performing classification using decision trees. If the decision tree which provides the model has a depth that is
too small, it can occur that it cannot explore the structure of the training set in its entirety, thus it is inappropriate
to carry out the classification properly. This is a case of undersampling. However, if the records are split more
that required, such conclusions can be drawn along the decisions that are not true anymore, and following this
excess of splitting rules, inappropriate decisions can be made - for example in the case of irregular records. This
is considered a case of oversampling.
Input
Zoo [UCI MLR]
Output
In this process, operators building similar decision trees are used for the same training set, and in them, only the
stop condition defining the maximal depth of the tree is different. The value of the maximal depth is 3, 6, and 9
respectively.
Figure 5.6. Setting a threshold for the maximal depth of the decision tree
46
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Interpretation of the results
In accordance with this, decision trees of different depths are created, which thus contain different amounts of
splitting conditions, based on which the records of the test set will be classified differently by the different
models. If the value of the maximal value is 3, the following decision tree is received as a result:
Figure 5.7. Graphic representation of the decision tree created
Figure 5.8. Graphic representation of he classification of the records based on the
decision tree
47
Created by XMLmind XSL-FO Converter.
Classification Methods 1
It can be seen here that based on using 2 rules, the 7 possible classes cannot be separated by the model, so this is
a clear case of undersampling. If the value of the maximal value is 3, the following decision tree is received as a
result:
Figure 5.9. Graphic representation of the decision tree created with the increased
maximal depth
Figure 5.10. Graphic representation of he classification of the records based on the
decision tree with increased maximal depth
48
Created by XMLmind XSL-FO Converter.
Classification Methods 1
In this case, only 3 of the records are classified differently from their original labels. However, if the threshold
for the maximal depth is increased further, the result will not be better, rather, it will worsen, as the additional
rules lead to inappropriate consequences, i.e. this is a case of oversampling. If the value of the maximal value is
3, the following decision tree is received as a result:
Figure 5.11. Graphic representation of the decision tree created with the further
increased maximal depth
49
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Figure 5.12. Graphic representation of he classification of the records based on the
decision tree with further increased maximal depth
50
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Video
Workflow
dtree_exp2.rmp
Keywords
classification
decision tree
overfitting
underfitting
Operators
Apply Model
Decision Tree
Multiply
Read AML
Split Data
3. Evaluation of performance for classification by
decision tree
Description
The process shows, using the Congressional Voting Records dataset, how the quality of a given classification
can be evaluated. After the decision tree has been built based on the training set, and the test set has been
classified using it, the quality of the classification executed can be examined. Using the evaluation received this
way, it can be decided whether the resulting classification is appropriate for the goals of the process, the existing
model should be improved further, or the existing model is of such poor quality that using a completely new
model is necessary.
51
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Input
Congressional Voting Records [UCI MLR]
Output
The decision tree is built based on the data set using the following settings in the process:
Figure 5.13. Preferences for the building of the decision tree
In this case, the following decision tree emerges:
Figure 5.14. Graphic representation of the decision tree created
Interpretation of the results
52
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Using the decision tree created, the records of the test set can be classified, and after the classification of the
records, the original class labels can be compared to those assigned based on the decision tree, e.g. using the
following figure:
Figure 5.15. Graphic representation of he classification of the records based on the
decision tree
Examining the performance of the classifier, the number of records classified appropriately and inappropriately
can be obtained, and the precision of the classification done by the model can be seen as well, displayed in
percentages for the individual classes, and overally:
Figure 5.16. Performance vector of the classification based on the decision tree
The question can also be raised in this case whether the performance of the model can be increased further. For
example, the minimal required confidence for splits can be raised as follows:
Figure 5.17. The modification of preferences for the building of the decision tree.
53
Created by XMLmind XSL-FO Converter.
Classification Methods 1
In this case, as a result of the raised value of the required confidence, the structure of the decision tree will be
completely different from that of the original one, and this leads to a change in the numbers and distribution of
the records classified appropriately and inappropriately as well. This model yields a better performance than the
original one, which can also be seen in the figure:
Figure 5.18. Graphic representation of the decision tree created with the modified
preferences
Figure 5.19. Performance vector of the classification based on the decision tree created
with the modified preferences
54
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Video
Workflow
dtree_exp3.rmp
Keywords
classification
decision tree
performance
evaluation
Operators
Apply Model
Decision Tree
Performance (Classification)
Read AML
Split Data
4. Evaluation of performance for classification by
decision tree 2
Description
The process shows, using the Congressional Voting Records dataset, how the quality of a given classification
can be evaluated. After the decision tree has been built based on the training set, and the test set has been
classified using it, the quality of the classification executed can be examined. In some cases, more advanced
levels of validation may be necessary; in these cases, e.g. random subsampling, cross-validation, or a special
case of the latter, the leave-one-out method can be used. Using the evaluation received this way, it can be
decided whether the resulting classification is appropriate for the goals of the process, the existing model should
55
Created by XMLmind XSL-FO Converter.
Classification Methods 1
be improved further, or due to the poor performance of the existing model, it should be replaced with a
completely new model.
Input
Congressional Voting Records [UCI MLR]
Output
Evaluation can be done by using a complex validation operator as well instead of separate operators. In this
case, the split ratio of the dataset, and the form of sampling can be specified:
Figure 5.20. Settings for the sampling done in the validation operator
This is a complex operator that consists of two subprocesses, which can be defined as follows:
Figure 5.21. Subprocesses of the validation operator
Interpretation of the results
This case is completely identical to the process in the previous example, the split into training and test sets is
done, the decision tree built using the training set is applied to the test set, and then its efficiency is evaluated.
Here, the following decision tree emerges, which classifies the records of the test set with the following results:
Figure 5.22. Graphic representation of the decision tree created
56
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Figure 5.23. Performance vector of the classification based on the decision tree
If a deeper examination of the given classifier is necessary, subprocesses identical to the ones above can be
defined in the operator responsible for cross-validation as well. The operator can be tuned using the following
preferences:
Figure 5.24. Settings of the cross-validation operator
Here, it can be defined how many cross-validation iterations should be executed. The dataset is split into a many
subsets of equal size as the number of iterations. Then, each of these splits is selected to be the test set of an
iteration, and the union of all other subsets will serve as the training set of the given iteration. A special case of
this is the leave-one-out method, which can be used by ticking the appropriate checkbox (leave-one-out). When
using this, an iteration is run for each record, in which the given record serves as the test set, and the training set
consists of all other records. As can be seen on the figure, the following average performance values are yielded
by cross-validation with 10 iterations:
57
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Figure 5.25. Overall performance vector of the classifications done in the crossvalidation operator
The following average performance values are yielded by the leave-one-out method:
Figure 5.26. Overall performance vector of the classifications done in the crossvalidation operator in the leave-one-out case
Note that in this case, the standard deviation of the precision values of the leave-one-out method are remarkably
higher than those of standard cross-validation. This might indicate that such irregular records are present the
classification os which is not necessarily accurate, even after learning on all other records.
Video
Workflow
dtree_exp4.rmp
Keywords
classification
decision tree
performance
random subsampling
cross-validation
Operators
Apply Model
Decision Tree
Multiply
Performance (Classification)
Read AML
Split Validation
X-Validation
5. Comparison of decision tree classifiers
58
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Description
The process shows, using the Spambase dataset, how the quality of multiple classifiers, the efficiency of
multiple classifiers can be compared. After the decision trees of the classifiers have been built based on the
training set, the test set can be classified using them, and the quality of the individually executed classifications
can be examined. This can be done separately, measuring the precision of the classifiers one by one, or the
analyses can be merged, and the ROC curves of the individual classifiers can be represented on a common
figure for a better picturing of the differences between the results. Based on the thus received evaluation, it can
be decided which classifier suits the requirements of the process, whether a given model should be improved, or
whether a given model has to be replaced or removed due to its poor performance.
Input
Spambase [UCI MLR]
Output
Let us create the following two decision tree classifiers based on the training set of the data set:
Figure 5.27. Preferences for the building of the decision tree based on the Gini-index
criterion
59
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Figure 5.28. Preferences for the building of the decision tree based on the gain ratio
criterion
60
Created by XMLmind XSL-FO Converter.
Classification Methods 1
The classifier using the gain ratio builds the following decision tree:
Figure 5.29. Graphic representation of the decision tree created based on the gain ratio
criterion
When applied to the test set, this decision tree yields the following performance values:
61
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Figure 5.30. Performance vector of the classification based on the decision tree built
using the gain ratio criterion
On the contrary, the classifier using the Gini-index builds the following decision tree:
Figure 5.31. Graphic representation of the decision tree created based on the Gini-index
criterion
When applied to the test set, this decision tree yields the following performance values:
Figure 5.32. Performance vector of the classification based on the decision tree built
using the Gini-index criterion
Interpretation of the results
It can be seen that the performance of the classifier utilizing the Gini-index is better than that of the classifier
based on the gain ratio. However, on one hand, the difference between individual models is not this obvious in
all cases, and on the other hand, for the sake of simplification, or to avoid the differences caused by sampling,
the evaluation of the individual models can be merged into a complex operator, and thus, the ROC curves of
their precision can also be displayed on a single figure, for example as follows:
Figure 5.33. Settings of the operator for the comparison of ROC curves
62
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Figure 5.34. Subprocess of the operator for the comparison of ROC curves
In this case, an arbitrary number of the model building operators can be placed in the complex operator, thus the
precision of this arbitrary number of models can be examined for the same data set at once. However, it is
advisable to use a local random seed to ensure that the comparison will be repeatable, as this ensures that for any
execution, the records will be split into training and test sets in the same manner.
Figure 5.35. Comparison of the ROC curves of the two decision tree classifiers
63
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Based on the ROC curves, it is obvious that the classifier based on the Gini-index has a much higher precision
than that of the classifier using the gain ratio, as its ROC curve is curved more in the direction of the point (0,1)
from the diagonal between the points (0,0) and (1,1).
Video
Workflow
dtree_exp5.rmp
Keywords
classification
decision tree
performance
comparison
ROC curve
Operators
Apply Model
Compare ROCs
Decision Tree
Multiply
Performance
Read AML
Split Data
64
Created by XMLmind XSL-FO Converter.
Chapter 6. Classification Methods 2
Rule-Based Classifiers
1. Using a rule-based classifier (1)
Description
In this experiment a rule-based classifier is trained on the Zoo data set.
Input
Zoo [UCI MLR]
Output
Figure 6.1. The rule set of the rule-based classifier trained on the data set.
Figure 6.2. The classification accuracy of the rule-based classifier on the data set.
65
Created by XMLmind XSL-FO Converter.
Classification Methods 2
Interpretation of the results
The second figure shows that the rule-based classifier perfectly classifies all training examples.
Video
Workflow
rules_exp1.rmp
Keywords
rule-based classifier
supervised learning
classification
Operators
Apply Model
Map
Performance (Classification)
Read AML
Rule Induction
Subprocess
2. Using a rule-based classifier (2)
Description
66
Created by XMLmind XSL-FO Converter.
Classification Methods 2
This experiment investigates the performance of the rule-based classifier on the Zoo data set. The data set is split
into a training and a test set, half of the examples are used to form a training set, and the rest are for testing. The
classification accuracies on both the training and the test sets are determined for the rule-based classifier.
Input
Zoo [UCI MLR]
Output
Figure 6.3. The rule set of the rule-based classifier.
Figure 6.4. The classification accuracy of the rule-based classifier on the training set.
Figure 6.5. The classification accuracy of the rule-based classifier on the test set.
Interpretation of the results
The second figure shows that the rule-based classifier perfectly classifies all training examples.
The third figure shows that the rule-based classifier perfectly classifies all but 6 of the 50 test examples.
Video
67
Created by XMLmind XSL-FO Converter.
Classification Methods 2
Workflow
rules_exp2.rmp
Keywords
rule-based classifier
supervised learning
classification
Operators
Apply Model
Map
Performance (Classification)
Read AML
Rule Induction
Split Data
Subprocess
3. Transforming a decision tree to an equivalent rule
set
Description
The process demonstrates the use of the Tree to Rules operator that transforms a decision tree to an
equivalent rule based classifier. The experiment uses a decision tree built on the Zoo data set.
Input
Zoo [UCI MLR]
Output
Figure 6.6. The decision tree built on the data set.
68
Created by XMLmind XSL-FO Converter.
Classification Methods 2
Figure 6.7. The rule set equivalent of the decision tree.
Figure 6.8. The classification accuracy of the rule-based classifier on the data set.
Interpretation of the results
It is apparent in the first and second figures that each rule in the rule set corresponds to a branch of the decision
tree from the root to a leaf node.
The third figure shows that the rule based classifier (and thus also the decision tree) perfectly classifies all
examples.
Video
69
Created by XMLmind XSL-FO Converter.
Classification Methods 2
Workflow
rules_exp3.rmp
Keywords
decision tree
rule-based classifier
supervised learning
classification
Operators
Apply Model
Decision Tree
Map
Multiply
Performance (Classification)
Read AML
Subprocess
Tree to Rules
70
Created by XMLmind XSL-FO Converter.
Chapter 7. Classification Methods 3
Regression
1. Linear regression
Description
The process shows, using the Wine dataset, how a regression model can be fitted to a given dataset.
Classification can also be done based on a regression model, but however, this process shows that creating the
regression model itself is insufficient to perform this. Based on the regression model, approximate values for
numerical labels can be defined, but these values are not assigned to concrete class labels. Apart from this, it can
be stated that similarly to other classification methods, the data set has to be split into training and test sets, and
the regression model created using the training set is to be applied to the test set.
Input
Wine [UCI MLR]
Output
When creating the regression model, it can be chosen from among various types of regression, such as linear
regression or logistic regression. From these, linear regression is utilized in the process. In this form, for
example, it can be defined which method should be used for attribute selection, or what the level of minimal
tolerance should be. The thus created linear regression model can be applied to the test set.
Figure 7.1. Properties of the linear regression operator
71
Created by XMLmind XSL-FO Converter.
Classification Methods 3
The following regression model is created based on the data of the training set:
Figure 7.2. The linear regression model yielded as a result
Interpretation of the results
Using the regression model created based on the records of the training set on the test set, approximate values
can be calculated for values of the labels of the individual test records. These approximate values can be seen in
the labelled data set yielded by the model application:
Figure 7.3. The class prediction values calculated based on the linear regression model
72
Created by XMLmind XSL-FO Converter.
Classification Methods 3
It can be seen that most of the approximate values yield a rather good estimation, and take a value that is close
to the original label, but this by itself is insufficient to complete the classification task. In order to be able to
classify records based on a regression model, its estimation have to be assigned to class labels.
Video
Workflow
regr_exp1.rmp
Keywords
classification
regression
Operators
Apply Model
Linear Regression
Read AML
Split Data
2. Osztályozás lineáris regresszióval
73
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Description
The process shows, using the Wine dataset, how a regression model can be fitted to a given dataset, and then
how can a classification task be completed based on the received estimates. Classification can also be done
based on a regression model; in this case, approximate values for numerical labels can be defined based on the
regression model,and afterwards, these values can be assigned to concrete class labels. Similarly to other
classification methods, the data set has to be split into training and test sets, and the regression model created
using the training set is to be applied to the test set.
Input
Wine [UCI MLR]
Output
When creating the regression model, it can be chosen from among various types of regression, such as linear
regression or logistic regression. From these, linear regression is utilized in the process. In order to be able to
use this for classification, it has to be placed into an operator that implements regression-based classification.
Identically to when the operator is used by itself, it can be defined for example which method should be used for
attribute selection, or what the level of minimal tolerance should be. The thus created linear regression model
can be applied to the test set.
Figure 7.4. The subprocess of the classification by regression operator
The following regression model is created based on the data of the training set:
Figure 7.5. The linear regression model yielded as a result
74
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Interpretation of the results
Using the regression model created based on the records of the training set on the test set, confidence values can
be calculated regarding the probabilities of the individual test records belonging to the given groups. These
confidence values, and the class assignments created based on these can be seen in the labelled data set yielded
by the model application:
Figure 7.6. The class labels derived from the predictions calculated based on the
regression model
75
Created by XMLmind XSL-FO Converter.
Classification Methods 3
It can be seen that based on the approximate values, the assignments are done correctly, and are equal to the
original labels in most cases.
Video
Workflow
regr_exp2.rmp
Keywords
classification
regression
Operators
Apply Model
Classification by Regression
Linear Regression
Read AML
Split Data
3. Evaluation of performance for classification by
regression model
76
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Description
The process shows, using the Spambase dataset, how the quality, the precision of a given classification that is
created based on a regression model fitted to a given data set can be evaluated. After the regression model has
been built based on the training set, and the test set has been classified using it, the quality of the classification
executed can be examined. Using the evaluation received this way, it can be decided whether the resulting
classification is appropriate for the goals of the process, the existing model should be improved further, or the
existing model is of such poor quality that using a completely new model is necessary.
Input
Spambase [UCI MLR]
Output
After creating the regression model, in order to be able to use it for classification, it has to be placed into an
operator that implements regression-based classification. Similarly to when using the operator individually, it
can be defined for example which method should be used for attribute selection, or what the level of minimal
tolerance should be. The thus created linear regression model can be applied to the test set.
Figure 7.7. The subprocess of the classification by regression operator
The following regression model is created based on the data of the training set:
Figure 7.8. The linear regression model yielded as a result
77
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Interpretation of the results
Using the regression model created based on the records of the training set on the test set, confidence values can
be calculated regarding the probabilities of the individual test records belonging to the given groups. Based on
these confidence values, class assignments are assigned to the individual records of the test set. Corresponding
to this, it can be evaluated how many records have been classified successfully based on the regression model:
Figure 7.9. The performance vector of the classification based on the regression model
78
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Video
Workflow
regr_exp3.rmp
Keywords
classification
regression
performance
evaluation
Operators
Apply Model
Classification by Regression
Linear Regression
Performance (Classification)
Read AML
Split Data
4. Evaluation of performance for classification by
regression model 2
Description
The process shows, using the Wine dataset, how the quality, the precision of a given classification that is created
based on a regression model fitted to a given data set can be evaluated. After the regression model has been built
79
Created by XMLmind XSL-FO Converter.
Classification Methods 3
based on the training set, and the test set has been classified using it, the quality of the classification executed
can be examined. In some cases, more advanced levels of validation may be necessary; in these cases, e.g.
random subsampling, cross-validation, or a special case of the latter, the leave-one-out method can be used.
Using the evaluation received this way, t can be decided whether the resulting classification is appropriate for
the goals of the process, the existing model should be improved further, or the existing model is of such poor
quality that using a completely new model is necessary.
Input
Wine [UCI MLR]
Output
Evaluation can be done by using a complex validation operator as well instead of separate operators. In this
case, as the regression model has to be placed into an operator that implements regression-based classification,
and this operator has to be placed into the operator of complex evaluation, the result is a process that contains
embedded operators on multiple levels:
Figure 7.10. The subprocess of the cross-validation by regression operator
Figure 7.11. The subprocess of the classification by regression operator
Similarly to when using the operator individually, it can be defined for example which method should be used
for attribute selection, or what the level of minimal tolerance should be. The thus created linear regression
model can be applied to the test set. The following regression model is created based on the data of the training
set:
Figure 7.12. The linear regression model yielded as a result
80
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Interpretation of the results
If a deeper examination of the given classifier is necessary, subprocesses identical to the ones above can be
defined in the operator responsible for cross-validation as well. The operator can be tuned using the following
preferences:
Figure 7.13. The customizable properties of the cross-validation operator
81
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Here, it can be defined how many cross-validation iterations should be executed. The dataset is split into a many
subsets of equal size as the number of iterations. Then, each of these splits is selected to be the test set of an
iteration, and the union of all other subsets will serve as the training set of the given iteration. A special case of
this is the leave-one-out method, which can be used by ticking the appropriate checkbox (leave-one-out). When
using this, an iteration is run for each record, in which the given record serves as the test set, and the training set
consists of all other records. As can be seen on the figure, the following average performance values are yielded
by cross-validation with 10 iterations:
Figure 7.14. The overall performance vector of the classifications done using the
regression model defined in the cross-validation operator
The following average performance values are yielded by the leave-one-out method:
Figure 7.15. The overall performance vector of the classifications done using the
regression model defined in the cross-validation operator for the case of using the leaveone-out method
Note that in this case, the standard deviation of the precision values of the leave-one-out method are remarkably
higher than those of standard cross-validation. This might indicate that such irregular records are present the
classification os which is not necessarily accurate, even after learning on all other records.
Video
Workflow
regr_exp4.rmp
82
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Keywords
classification
regression
performance
cross-validation
Operators
Apply Model
Classification by Regression
Linear Regression
Performance (Classification)
Read AML
X-Validation
83
Created by XMLmind XSL-FO Converter.
Chapter 8. Classification Methods 4
Neural Networks and Support Vector Machines
1. Using a perceptron for solving a linearly separable
binary classification problem
Description
In this experiment a perceptron is trained on a linearly separable two-dimensional data set consisting of two
classes, that is a subset of the Wine data set. The classification accuracy of the perceptron is determined on the
data set.
Input
Figure 8.1. A linearly separable subset of the Wine data set [UCI MLR] used in the
experiment (2 of the total of 3 classes and 2 of the total of 13 attributes was selected).
Output
84
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 8.2. The decision boundary of the perceptron.
Figure 8.3. The classification accuracy of the perceptron on the data set.
Interpretation of the results
The second figure shows that the perceptron perfectly classifies all training examples.
Video
Workflow
ann_exp1.rmp
Keywords
perceptron
supervised learning
classification
Operators
Apply Model
Filter Examples
Perceptron
Performance (Classification)
Read CSV
Remove Unused Values
2. Using a feed-forward neural network for solving a
classification problem
85
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Description
In this experiment a two-layer feed-forward neural network with 2 hidden neurons is trained on the Sonar,
Mines vs. Rocks data set.
Input
Sonar, Mines vs. Rocks [UCI MLR]
Output
Figure 8.4. The classification accuracy of the neural network on the data set.
Interpretation of the results
The figure shows that the neural network perfectly classifies all but one of the training examples.
Video
Workflow
ann_exp2.rmp
Keywords
feed-forward neural network
supervised learning
classification
Operators
Apply Model
86
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Neural Net
Performance (Classification)
Read CSV
3. The influence of the number of hidden neurons to
the performance of the feed-forward neural network
Description
In this experiment two-layer feed-forward neural networks with different number of hidden neurons are trained
on the Sonar, Mines vs. Rocks data set. The average classification error rate from 10-fold cross-validation is
determined for each neural network.
The main contribution of the experiment is that it shows how to change the value of a list type parameter of an
operator (in our case, the hidden layers parameter of the Neural Net operator) in loops using a macro.
To obtain a reasonable execution time only neural networks with the following number of hidden neurons are
considered: 1, 2, 4, 8, 16.
Input
Sonar, Mines vs. Rocks [UCI MLR]
Output
Figure 8.5. The average classification error rate obtained from 10-fold cross-validation
against the number of hidden neurons.
87
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Interpretation of the results
The figure shows that the best average classification error rate (14.5%) is achieved when the number of hidden
neurons is 8.
Video
Workflow
ann_exp3.rmp
Keywords
feed-forward neural network
supervised learning
error rate
classification
cross-validation
Operators
Apply Model
Guess Types
Log
Log to Data
Loop Values
Neural Net
Performance (Classification)
Print to Console
Provide Macro as Log Value
Read CSV
X-Validation
Execute Script (R) [R Extension]
4. Using a linear SVM for solving a linearly separable
binary classification problem
88
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Description
In this experiment a linear SVM is trained on a linearly separable two-dimensional data set consisting of two
classes, that is a subset of the Wine data set. The classification accuracy of the linear SVM is determined on the
data set.
Input
Figure 8.6. A linearly separable subset of the Wine data set [UCI MLR] used in the
experiment (2 of the total of 3 classes and 2 of the total of 13 attributes was selected).
Output
Figure 8.7. The kernel model of the linear SVM.
89
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 8.8. The classification accuracy of the linear SVM on the data set.
Interpretation of the results
The figure shows that the linear SVM perfectly classifies all training examples.
Video
Workflow
svm_exp1.rmp
Keywords
SVM
supervised learning
classification
Operators
Apply Model
Filter Examples
Performance (Classification)
Read CSV
Remove Unused Values
Support Vector Machine (LibSVM)
5. The influence of the parameter C to the
performance of the linear SVM (1)
90
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Description
The process demonstrates the influence of the parameter C on performance of the linear SVM. Linear SVMs are
trained on a subset of the Wine data set while the value of the parameter C is increased from 0.001 to 100. The
classification error rate on the training set and also the number of support vectors are determined for each SVM.
Input
A subset of the Wine data set [UCI MLR].
Figure 8.9. A subset of the Wine data set used in the experiment (2 of the total of 3
classes and 2 of the total of 13 attributes was selected). Note that the classes are not
linearly separable.
Output
Figure 8.10. The classification error rate of the linear SVM against the value of the
parameter C.
91
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 8.11. The number of support vectors against the value of the parameter C.
Interpretation of the results
The first figure shows that the classification error rate quickly falls below 6% as the value of the parameter C is
increased and then it remains constant.
The second figure shows that the number of support vectors decreases similarly with the increasing value of the
parameter C, although not so rapidly as the classification error rate.
Video
Workflow
svm_exp2.rmp
Keywords
92
Created by XMLmind XSL-FO Converter.
Classification Methods 4
SVM
supervised learning
error rate
classification
Operators
Apply Model
Filter Examples
Log
Loop Parameters
Normalize
Performance (Classification)
Performance (Support Vector Count)
Read CSV
Remove Unused Values
Support Vector Machine (LibSVM)
6. The influence of the parameter C to the
performance of the linear SVM (2)
Description
The process demonstrates the influence of the parameter C on the average classification error rate of the linear
SVM in the case of the Heart Disease data set. We consider linear SVMs with different C values, each of which
is an integer power of 2: C = 2^n, where -13 <= n <= 6. The average classification error rate from 10-fold crossvalidation is determined for each linear SVM.
Input
Heart Disease [UCI MLR]
Note
The data set was donated to the UCI Machine Learning Repository by R. Detrano[Detrano et al.].
Output
93
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 8.12. The average classification error rate of the linear SVM obtained from 10fold cross-validation against the value of the parameter C, where the horizontal axis is
scaled logarithmically.
Interpretation of the results
The figure shows that the average classification error rate is minimal when the value of the parameter C is 2^-8.
Larger values of C result in a slightly worse average classification performance. However, values closer to zero
give worse result.
Thus, the performance of the linear SVM seems not to be sensitive to the value of the parameter C in this case.
Video
Workflow
svm_exp3.rmp
Keywords
SVM
supervised learning
error rate
classification
cross-validation
Operators
Apply Model
Filter Examples
Log
Loop Parameters
Map
Normalize
Performance (Classification)
Read CSV
Support Vector Machine (LibSVM)
94
Created by XMLmind XSL-FO Converter.
Classification Methods 4
X-Validation
7. The influence of the parameter C to the
performance of the linear SVM (3)
Description
In this experiment linear SVMs are trained on the Spambase data set while the value of the parameter C is
varied. We will use integer powers of 2 as value of the parameter: C = 2^n, where -8 <= n <= 5. The data set is
split into a training and a test set, 60% of the examples are used to form a training set, and the rest are for
testing. The classification error rates on both the training and the test sets and also the number of support vectors
are determined for each SVM.
Input
Spambase [UCI MLR]
Output
Figure 8.13. The classification error rate of the linear SVM on the training and the test
sets against the value of the parameter C.
95
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 8.14. The number of support vectors against the value of the parameter C.
Interpretation of the results
The first figure shows that the classification error rate on the training set is decreases with the increase of value
of the parameter C. As the value of the parameter C is increased, the error rate on the test set also decreases, until
C reaches 2. However, further increase of the value of the parameter causes a slight increase in the test error.
The second figure shows that the number of support vectors falls by about 50% while the value of the parameter
C is increased from 2^-8 to 8. Further increase of the value of the parameter causes a slight increase in the
number of support vectors.
Video
96
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Workflow
svm_exp4.rmp
Keywords
SVM
supervised learning
error rate
classification
Operators
Apply Model
Log
Log to Data
Loop Parameters
Normalize
Performance (Classification)
Performance (Support Vector Count)
Read CSV
Split Data
Support Vector Machine (LibSVM)
8. The influence of the number of training examples to
the performance of the linear SVM
Description
The process demonstrates the influence of the number of training examples on the performance of the linear
SVM in the case of the Adult (LIBSVM) data set. The number of training examples is increased in the
experiment, and an SVM is trained in each step. The following performance characteristics are determined for
each of the SVMs:
• the classification error rate on the training set,
• the classification error rate on the corresponding test set,
• the number of support vectors,
• the CPU execution time needed to train the linear SVM.
97
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Input
A discretized and binarized version of the Adult data set [UCI MLR] available at the LIBSVM website
[LIBSVM].
Output
Figure 8.15. The classification error rate of the linear SVM on the training and the test
sets against the number of training examples.
Figure 8.16. The number of support vectors against the number of training examples.
Figure 8.17. CPU execution time needed to train the SVM against the number of
training examples.
98
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Interpretation of the results
The first figure shows that the classification error on the training and test sets are roughly the same,
independently of the number of training examples.
The second and the third figures show that both the number of support vectors and the CPU execution time
increase linearly with the number of training examples.
Video
Workflow
svm_exp5.rmp
Keywords
SVM
supervised learning
error rate
classification
cross-validation
Operators
Apply Model
Extract Macro
Generate Attributes
Log
Log to Data
Loop Files
Normalize
Parse Numbers
Performance (Classification)
Performance (Support Vector Count)
Provide Macro as Log Value
Read Sparse
Remove Duplicates
99
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Sort
Support Vector Machine (LibSVM)
9. Solving the two spirals problem by a nonlinear SVM
Description
In this experiment a nonlinear SVM is trained to solve the Two Spirals problem, that is a linearly non-separable
classification problem developed for benchmarking neural networks. The classification accuracy of the
nonlinear SVM is determined on the data set.
Input
Two Spirals [Two Spirals]
Figure 8.18. The Two Spirals data set
Figure 8.19. The R code that produces the data set and is executed by the
(R) operator of the R Extension.
100
Created by XMLmind XSL-FO Converter.
Execute Script
Classification Methods 4
i <- 0:96
angle <- i * pi / 16
radius <- 6.5 * (104 - i) / 104
x <- radius * cos(angle);
y <- radius * sin(angle);
spirals <- data.frame(
rbind(
cbind(x, y, 0),
cbind(-x, -y, 1)
)
)
names(spirals) <- c("x", "y", "class")
spirals <- transform(spirals, class = factor(class))
spirals.label <- "class"
Note
The R code that produces the data set is based on a SAS code snippet in [Neural Network FAQ].
Output
Figure 8.20. The classification accuracy of the nonlinear SVM on the data set.
Interpretation of the results
The figure shows that the nonlinear SVM perfectly classifies all training examples.
Video
Workflow
svm_exp6.rmp
Keywords
SVM
supervised learning
linearly non-separable
classification
Operators
Apply Model
Performance (Classification)
Support Vector Machine (LibSVM)
Execute Script (R) [R Extension]
10. The influence of the kernel width parameter to the
performance of the RBF kernel SVM
101
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Description
In this experiment RBF kernel SVMs are trained on the Pima Indians Diabetes data set with different kernel
width parameter (gamma) values. The value of this parameter is increased from 0.001 to 5 while the value of the
parameter C is fixed to 1 to obtain comparable results. The data set is split into a training and a test set, 75% of
the examples are used to form a training set, and the rest are for testing. The classification error rates on both the
training and the test sets are determined for each SVM.
Input
Pima Indians Diabetes [UCI MLR]
Output
Figure 8.21. The classification error rates of the SVM on the training and the test sets
against the value of RBF kernel width parameter.
Interpretation of the results
The value of the RBF kernel width parameter can be chosen such that the SVM will perfectly classify all
training examples. Unfortunately, the model does not perform well on the test data. Apparently, overfitting
102
Created by XMLmind XSL-FO Converter.
Classification Methods 4
occurs here. It should be noted that the linear SVM does not perform so well on the training set, its classification
error rate is around 20%.
Video
Workflow
svm_exp7.rmp
Keywords
SVM
supervised learning
error rate
classification
Operators
Apply Model
Log
Loop Parameters
Normalize
Performance (Classification)
Read CSV
Split Data
Support Vector Machine (LibSVM)
11. Search for optimal parameter values of the RBF
kernel SVM
Description
In this experiment RBF kernel SVMs are trained on the Ionosphere data set while the value of the parameter
gamma of the RBF kernel and also the value of the parameter C are changed. The average classification error rate
from 10-fold cross-validation is determined for each SVM. As a result, the values yielding the best average
103
Created by XMLmind XSL-FO Converter.
Classification Methods 4
classification error rate will be returned. The following parameter values will be considered for C and gamma: C
= 2^n, where -5 <= n <= 6, gamma = 2^m, where -10 <= m <= 4. Thus, the total number of parameter value
combinations considered is 180.
Input
Ionosphere [UCI MLR]
Output
Figure 8.22. The optimal parameter values for the RBF kernel SVM.
Figure 8.23. The classification accuracy of the RBF kernel SVM trained on the entire
data set using the optimal parameter values.
Interpretation of the results
The first figure shows that the best average classification error rate is achieved when the value of the parameter
C is 16 and the value of the parameter gamma is 0.015625. Note that these parameter values can not be
considered as the global optimum of the average classification error rate, since they were obtained by
performing a grid search that examines only a few points of the search space.
The second figure shows that the RBF kernel SVM trained on the entire data set using the optimal parameter
values performs very well.
Video
Workflow
svm_exp8.rmp
Keywords
104
Created by XMLmind XSL-FO Converter.
Classification Methods 4
SVM
supervised learning
error rate
classification
cross-validation
parameter optimization
Operators
Apply Model
Log
Multiply
Normalize
Optimize Parameters (Grid)
Performance (Classification)
Read CSV
Set Parameters
Support Vector Machine (LibSVM)
X-Validation
12. Using an SVM for solving a multi-class
classification problem
Description
In this experiment a linear SVM is trained on a data set consisting of three classes. The classification accuracy
of the linear SVM is determined on the data set.
Input
Wine [UCI MLR]
Output
Figure 8.24. The kernel model of the linear SVM.
105
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 8.25. The classification accuracy of the linear SVM on the data set.
Interpretation of the results
The second figure shows that the linear SVM perfectly classifies all training examples.
Video
Workflow
svm_exp9.rmp
Keywords
SVM
supervised learning
classification
Operators
Apply Model
Normalize
Performance (Classification)
Read CSV
Support Vector Machine (LibSVM)
13. Using an SVM for solving a regression problem
106
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Description
The process demonstrates how to use an SVM for solving a regression problem. In this experiment RBF kernel
SVMs are trained on the Concrete Compressive Strength data set while the value of the parameter gamma of the
RBF kernel is changed. To obtain comparable results the value of the parameter C is fixed to 10. The average
RMS error from 10-fold cross-validation is determined for each SVM. As a result, the gamma value yielding the
best average RMS error will be returned. Using this value for the parameter gamma an RBF kernel SVM is
trained on the entire data set that is referred to as the “optimal RBF kernel SVM” below.
Input
Concrete Compressive Strength [UCI MLR] [Concrete]
Output
Figure 8.26. The optimal value of the gamma parameter for the RBF kernel SVM.
Figure 8.27. The average RMS error of the RBF kernel SVM obtained from 10-fold
cross-validation against the value of the parameter gamma, where the horizontal axis is
scaled logarithmically.
107
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 8.28. The kernel model of the optimal RBF kernel SVM.
Figure 8.29. Predictions provided by the optimal RBF kernel SVM against the values of
the observed values of the dependent variable.
108
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Interpretation of the results
The first figure shows that the best average RMS error is achieved when the value of the parameter gamma is 2^2 = 0.25.
The third figure shows that the average RMS error decreases with the increasing value of the parameter gamma
until it reaches its minimum. However, further increase of the value of the parameter gamma results in the
degradation of the performance, i.e., model overfitting occurs.
Video
Workflow
svm_regr_exp1.rmp
Keywords
SVM
supervised learning
RMS error
regression
cross-validation
parameter optimization
Operators
Apply Model
Log
Multiply
Normalize
Optimize Parameters (Grid)
Performance (Regression)
Read Excel
Set Parameters
Support Vector Machine (LibSVM)
X-Validation
109
Created by XMLmind XSL-FO Converter.
Chapter 9. Classification Methods 5
Ensemble Methods
1. Introducing ensemble methods: the bagging
algorithm
Description
The experiment introduces the use of ensemble methods, featuring the Bagging operator. The average
classification error rate from 10-fold cross-validation on the Heart Disease data set is compared for a single
decision stump and an ensemble of 10 decision stumps trained by bagging. The impurity measure used for the
decision stumps is the gain ratio.
Input
Heart Disease [UCI MLR]
Note
The data set was donated to the UCI Machine Learning Repository by R. Detrano[Detrano et al.].
Output
Figure 9.1. The average classification error rate of a single decision stump obtained
from 10-fold cross-validation.
Figure 9.2. The average classification error rate of the bagging algorithm obtained from
10-fold cross-validation, where 10 decision stumps were used as base classifiers.
110
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Interpretation of the results
An ensemble of 10 decision stumps trained by bagging gives an average classification error rate that is about 7%
better that those of a single decision stump.
Video
Workflow
ensemble_exp1.rmp
Keywords
bagging
ensemble methods
supervised learning
error rate
cross-validation
classification
Operators
Apply Model
Bagging
Decision Stump
Map
Multiply
Performance (Classification)
Read CSV
X-Validation
2. The influence of the number of base classifiers to
the performance of bagging
111
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Description
This process demonstrates the influence of the number of base classifiers on the classification error rate of
bagging in the case of the Heart Disease data set. The base classifiers are decision stumps and the impurity
measure used is the gain ratio. The number of base classifiers are increased from 1 to 20 in the experiment, and
the average classification error rate of bagging from 10-fold cross-validation is determined in each step.
Input
Heart Disease [UCI MLR]
Note
The data set was donated to the UCI Machine Learning Repository by R. Detrano[Detrano et al.].
Output
Figure 9.3. The average classification error rate obtained from 10-fold cross-validation
against the number of base classifiers.
112
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Interpretation of the results
The figure shows that the best average classification error rate (21.4%) is achieved when the number of base
classifiers is 14.
Video
Workflow
ensemble_exp2.rmp
Keywords
bagging
ensemble methods
supervised learning
error rate
cross-validation
classification
Operators
Apply Model
Bagging
Decision Stump
Log
Loop Parameters
Map
Performance (Classification)
Read CSV
X-Validation
3. The influence of the number of base classifiers to
the performance of the AdaBoost method
Description
113
Created by XMLmind XSL-FO Converter.
Classification Methods 5
The process demonstrates the influence of the number of base classifiers on the classification error rate of the
AdaBoost method in the case of the Heart Disease data set. The base classifiers are decision stumps and the
impurity measure used is the gain ratio. The number of base classifiers are increased from 1 to 20 in the
experiment, and the average classification error rate of the AdaBoost method from 10-fold cross-validation is
determined in each step.
Note
The experiment is the same as the previous one, the only difference is that the AdaBoost operator is
used instead of the Bagging operator.
Input
Heart Disease [UCI MLR]
Note
The data set was donated to the UCI Machine Learning Repository by R. Detrano[Detrano et al.].
Output
Figure 9.4. The average classification error rate obtained from 10-fold cross-validation
against the number of base classifiers.
Interpretation of the results
The figure shows that the best average classification error rate (22.7%) is achieved when the number of base
classifiers is 3. It is also apparent that increasing the number of base classifiers does not result in the degradation
of the performance, that remains constant instead. Thus, model overfitting surprisingly does not occur.
Note that the best performance obtained is almost identical to those of bagging, but requires lesser base
classifiers. Moreover, performance behaves more predictable than in the case of bagging.
Video
Workflow
114
Created by XMLmind XSL-FO Converter.
Classification Methods 5
ensemble_exp3.rmp
Keywords
AdaBoost
ensemble methods
supervised learning
error rate
cross-validation
classification
Operators
AdaBoost
Apply Model
Decision Stump
Log
Loop Parameters
Map
Performance (Classification)
Read CSV
X-Validation
4. The influence of the number of base classifiers to
the performance of the random forest
Description
The process demonstrates the influence of the number of base classifiers on the classification error rate of the
random forest in the case of the Heart Disease data set. The number of base classifiers (i.e., decision trees) are
increased from 1 to 20 in the experiment, and the average classification error rate of the random forest from 10fold cross-validation is determined in each step. The impurity measure used for the decision trees is the gain
ratio.
Note
The experiment is the same as the previous two, the only difference is that the Random Forest
operator is used here instead of the Bagging and the AdaBoost operators.
Input
115
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Heart Disease [UCI MLR]
Note
The data set was donated to the UCI Machine Learning Repository by R. Detrano[Detrano et al.].
Output
Figure 9.5. The average error rate of the random forest obtained from 10-fold crossvalidation against the number of base classifiers.
Interpretation of the results
The figure shows that the best average classification error rate (19.1%) is achieved when the number of base
classifiers is 10.
Note that the best performance obtained is slightly better than those of AdaBoost (22.7%), but requires more
base classifiers. Moreover, the performance of AdaBoost behaves more predictable than those of the random
forest.
Video
Workflow
ensemble_exp4.rmp
Keywords
random forest
ensemble methods
supervised learning
error rate
cross-validation
classification
Operators
Apply Model
116
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Log
Loop Parameters
Map
Performance (Classification)
Random Forest
Read CSV
X-Validation
117
Created by XMLmind XSL-FO Converter.
Chapter 10. Association rules
1. Extraction of association rules
Description
The process shows, using the Extended Bakery dataset, how association rules can be extracted from a
transactional dataset. The emphasis is on the items that are present in the transactional datasets from the possible
items, i.e. on the items which form part of the given transaction, and not those which are missing from it. If such
a transactional dataset is in an uncompressed sparse matrix representation, so all records contain a binomial
value for each of the possible items, the extraction of association rules can be executed without any complex
transformation, the only thing that has to be kept in mind is that the attributes representing the individual items
should be of a binomial type. Using these, the frequent item sets can be extracted, and based on these, the
association rules valid for the dataset can be extracted.
Input
Extended Bakery [Extended Bakery]
Output
Using the FP-Growth algorithm on the version of the dataset that contains 20000 records, the following frequent
item sets are created:
Figure 10.1. List of the frequent item sets generated
118
Created by XMLmind XSL-FO Converter.
Association rules
Interpretation of the results
Based on these frequent item sets, the appropriate association rules can be created. It can be set the rules
meeting what kind of criteria should be considered valid - by default, a required level of confidence can be set,
but filtering can be done based on other values as well. Using the emerging rules, deeper conclusions can be
drawn regarding the connections between the data. Among other things, the table representation of the rules can
aid this, as in this representation, different kinds of filters can be utilized to filter out the rules considered
interesting, for example by outcome or by confidence level:
Figure 10.2. List of the association rules generated
119
Created by XMLmind XSL-FO Converter.
Association rules
Besides the table representation, a graphic representation can also be used, with available filtering conditions
that are similar to those of the former:
Figure 10.3. Graphic representation of the association rules generated
Video
Workflow
assoc_exp1.rmp
Keywords
frequent item sets
association rules
transactional data
binomial attributes
Operators
120
Created by XMLmind XSL-FO Converter.
Association rules
Create Association Rules
FP-Growth
Numerical to Binominal
Read AML
2. Asszociációs szabályok kinyerése nem tranzakciós
adathalmazból
Description
The process shows, using the Titanic dataset, how association rules can be extracted from a non-transactional
dataset. In order to obtain association rules from such a dataset, it first has to be transformed into a transactional
dataset. In these cases, it depends on the structure of the original database whether the emphasis is only on the
items that are present in the transactional dataset from the possible items, or the 0 values of the variables also
have to be interpreted. These datasets have to be transformed into an uncompressed sparse matrix
representation, in which all records contain a binomial value for each of the possible items. After this, the
extraction of association rules can be executed without any complex transformation. The frequent item sets
occurring in the dataset can be extracted, and based on these, the association rules valid for the dataset can be
extracted as well.
Input
Titanic [Titanic]
Output
Using this dataset, it can be examined whether the age, sex, and class of the passengers of the Titanic had any
influence on their survival chances. As the Class variable is not of a binomial type, this has to be converted into
binomial form first, before the frequent item sets could be extracted:
Figure 10.4. Operator preferences for the necessary data conversion
121
Created by XMLmind XSL-FO Converter.
Association rules
Figure 10.5. Converted version of the dataset
Based on these, the frequent item sets can now be acquired, from which the association rules valid for the
dataset can be generated:
Figure 10.6. List of the frequent item sets generated
122
Created by XMLmind XSL-FO Converter.
Association rules
Figure 10.7. List of the association rules generated
Interpretation of the results
Looking at the frequent item sets and association rules created, it is obvious that the handling of the dataset is
inappropriate. In can be seen in the documentation of the dataset that for each variable, including the binomial
variables, 0 values have a separate meaning (e.g. this represents children at the age variable, or belonging to the
crew at the class variable). In accordance with this, to acquire the appropriate transactional records, these
variables will also have to be split into two separate variables that represent the presence or absence of the two
possible values. In this case, the following dataset is yielded as a result:
Figure 10.8. Operator preferences for the appropriate data conversion
123
Created by XMLmind XSL-FO Converter.
Association rules
Figure 10.9. The appropriate converted version of the dataset
Based on these, the frequent item sets, and using those, the appropriate association rules can be extracted. Using
these emerging rules, deeper conclusions can be drawn regarding the connections between the data, and the
factors influencing the survival chances of the passengers can be filtered out. Among other things, the table
representation of the rules can aid this, as in this representation, different kinds of filters can be utilized to filter
out the rules considered interesting, for example by outcome or by confidence level:
Figure 10.10. Enhanced list of the frequent item sets generated
124
Created by XMLmind XSL-FO Converter.
Association rules
Figure 10.11. List of the association rules generated
Besides the table representation, a graphic representation can also be used, with available filtering conditions
that are similar to those of the former:
Figure 10.12. Graphic representation of the association rules generated
125
Created by XMLmind XSL-FO Converter.
Association rules
Video
Workflow
assoc_exp2.rmp
Keywords
frequent item sets
association rules
non-transactional data
binomial attributes
data transformation
Operators
Create Association Rules
FP-Growth
Multiply
Nominal to Binominal
Read AML
3. Evaluation of performance for association rules
126
Created by XMLmind XSL-FO Converter.
Association rules
Description
The process shows, using the Titanic dataset, how the usability and efficiency of association rules can be
checked if association rules are being extracted for a given dataset. After extracting the association rules, their
support can be evaluated, and similarly to classification tasks, it can be checked to what extent the original
values of the dataset can be predicted based on the rules created. Based on these types of evaluation, conclusions
can be drawn based on which it can be decided whether the resulting association rules are appropriate for the
goals of the process, the existing rules should be improved further, or the existing rules have revealed such poor
connections that using a completely new approach is necessary.
Input
Titanic [Titanic]
Output
Using this dataset, it can be examined whether the age, sex, and class of the passengers of the Titanic had any
influence on their survival chances. After the appropriate conversion of the variables, the dataset can be split
into a training set and a test set, and then, by applying the association rules deduced based on the training set to
the test set, it can be defined to what extent the rules are usable. In order to render the attributes created during
the conversion referable, the following parameter has to be used:
Figure 10.13. Operator preferences for the necessary data conversion
127
Created by XMLmind XSL-FO Converter.
Association rules
After this, in order to evaluate the efficiency of applying the rules using the general performance evaluation
operator, the original and predicted values of the attribute of interest (in this case, the variable Surived_1, which
indicates that the given passenger has survived the shipwreck) have to be converted to nominal types, and also,
it also has to be ensured that their values are coded using the same values:
Figure 10.14. Label role assignment for performance evaluation
Figure 10.15. Prediction role assignment for performance evaluation
128
Created by XMLmind XSL-FO Converter.
Association rules
Figure 10.16. Operator preferences for the data conversion necessary for evaluation
Interpretation of the results
After setting the appropriate roles, the performance measurement operator automatically performs the
comparisons, and based on these, it evaluates the efficiency of the application of the rules. Running the process
yields the following rules regarding the survival of the passengers as a result:
Figure 10.17. Graphic representation of the association rules generated regarding
survival
129
Created by XMLmind XSL-FO Converter.
Association rules
Figure 10.18. List of the association rules generated regarding survival
It can be seen here that although many conclusions have been drawn regarding the survival of the passengers,
the support of the rules is rather low. This leads to the conclusion that the rules can be applied in relatively
special cases, and not generally, thus in some cases, no decision will be possible based on them. This can be
illustrated by the low value appearing in the evaluation of performance as well:
Figure 10.19. Performance vector for the application of association rules generated
One of the reasons for this could be that during the extraction of the association rules, some other factor, that
might affect the connections disclosed by the association rules, was not taken into consideration. After the
discovery of these, a better result might be obtainable in some cases.
Video
Workflow
assoc_exp3.rmp
Keywords
frequent item sets
association rules
performance
support
Operators
Apply Association Rules
Create Association Rules
Discretize by User Specification
FP-Growth
130
Created by XMLmind XSL-FO Converter.
Association rules
Multiply
Nominal to Binominal
Performance
Read AML
Set Role
Split Data
4. Performance of association rules - Simpson's
paradox
Description
The process shows, using the Titanic dataset, how the usability and efficiency of association rules can be
enhanced if association rules are being extracted for a given dataset, by creating subsets of the dataset based on
the connections between its data, and then creating association rules separately for its subsets. After extracting
the association rules, their support can be evaluated, and similarly to classification tasks, it can be checked to
what extent the original values of the dataset can be predicted based on the rules created. If these values fail to
reach the expected levels, one of the reasons for this can be the so-called Simpson's paradox, which means that
due to some hidden factors, the connections between the variables can weaken, disappear, or even turn in the
opposite direction. If such factors are discovered, splitting the dataset along these can enhance the performance
of the association rules.
Input
Titanic [Titanic]
Output
Using this dataset, it can be examined whether the age, sex, and class of the passengers of the Titanic had any
influence on their survival chances. After the appropriate conversion of the variables, the dataset can be split
into a training set and a test set, and then, by applying the association rules deduced based on the training set to
the test set, it can be defined to what extent the rules are usable. However, if we do this based on the whole of
the dataset, relatively poor results emerge for support and, rooting from this, for performance as well:
Figure 10.20. List of the association rules generated regarding survival
131
Created by XMLmind XSL-FO Converter.
Association rules
Figure 10.21. Performance vector for the application of association rules generated
But considering the contingency table of the dataset, for example split by the age of the passengers, and by their
class, the conclusion can be drawn that the individual variables have such a strong influence on the value of the
variable of interest - survival - that these effects of the individual classes can neutralize each other in the whole
of the dataset, so it can be more advantageous to split the dataset along these variables, and extract the
association rules separately in the individual subsets:
Figure 10.22. Contingency table of the dataset
In order to do this, for example if the dataset is to be split based on the age of the passengers, first the
appropriate records have to be filtered out, then the variables used as filtering conditions can also be removed,
as in the subsets, they carry information that can now be considered redundant:
Figure 10.23. Record filter usage
132
Created by XMLmind XSL-FO Converter.
Association rules
Figure 10.24. Removal of attributes that become redundant after filtering
Interpretation of the results
After this, the training and test sets are created, the association rules concerning them are extracted, and their
efficiency is evaluated for the separate datasets of adults and children. The subset of adults yielded the following
results:
Figure 10.25. List of the association rules generated for the subset of adults
Figure 10.26. Performance vector for the application of association rules generated
regarding survival for the subset of adults
The subset of children yielded the following results:
Figure 10.27. List of the association rules generated for the subset of children
Figure 10.28. Performance vector for the application of association rules generated
regarding survival for the subset of children
It can be seen that performance can be increased remarkably by such splits of datasets, as by doing this, the
interference between the effects of groups can be neutralized. For the group of children, the enhancement in
performance is much smaller, but this can be explained with the much smaller record count of the subset.
133
Created by XMLmind XSL-FO Converter.
Association rules
Video
Workflow
assoc_exp4.rmp
Keywords
frequent item sets
association rules
performance
support
Simpson's paradox
Operators
Apply Association Rules
Create Association Rules
Discretize by User Specification
Filter Examples
FP-Growth
Multiply
Nominal to Binominal
Performance
Read AML
Select Attributes
Set Role
Split Data
134
Created by XMLmind XSL-FO Converter.
Chapter 11. Clustering 1
Standard methods
1. K-means method
Description
The process demonstrates, using the Aggregation dataset, how the K-means clustering algorithm works. Also, it
also shows the importance of choosing the distance function.
Input
Aggregation [SIPU Datasets] [Aggregation]
The dataset consists of 788 two-dimensional vectors, which form 7 separate groups. The task is to discover these
groups - clusters. The difficulty of the task is in the alignment of the points, as smaller and larger clouds of
points are present with different distances in space between them.
Figure 11.1. The 7 separate groups
Output
Figure 11.2. Clustering with default values
135
Created by XMLmind XSL-FO Converter.
Clustering 1
After reading the data, the node of the K-means method is connected, and the algorithm is set to search for 7
clusters, then the process is initiated. The result is that the discovery of the upper and right side point clouds is
successful, however, the algorithms performed poorly on the lower point cloud.
Figure 11.3. Set the distance function.
Let us try out another distance function, the Mahalanobis distance.
Figure 11.4. Clustering with Mahalanobis distance function
136
Created by XMLmind XSL-FO Converter.
Clustering 1
It can be seen that by minor sacrifices, but the result has become more precise; the clustering of the lower point
cloud is now nearing a perfect solution.
Interpretation of the results
It can be seen that even the simplest clustering algorithms can discover basic connections, and if the distance
function is chosen correctly, the results can even be made more precise.
Video
Workflow
clust_exp1.rmp
Keywords
K-means method
distance functions
cluster analysis
Operators
k-Means
Read CSV
2. K-medoids method
137
Created by XMLmind XSL-FO Converter.
Clustering 1
Description
The process shows, using the Maximum Variance (R15) dataset, how the K-medoids method can be used.
Input
Maximum Variance (R15) [SIPU Datasets] [Maximum Variance]
The dataset contains 600 two-dimensional vectors, which are concentrated into 15 clusters. The points are
aligned around a center with the coordinates (10,10), in increasing distances from each other as they get further
from the center. This is the difficulty of the task, as the clusters near the center are close to blending into each
other.
Figure 11.5. The dataset
Output
Figure 11.6. Setting the parameters of the clustering
138
Created by XMLmind XSL-FO Converter.
Clustering 1
The difference of the K-medoids method from the K-means method is that the centers of the clusters have to be
existing points. After setting the distance function and the number of clusters k, and then running the process, it
can be seen that even though a more sophisticated distance function has been chosen, the alignment of the data
did not make the precise analysis of the central clusters possible.
Figure 11.7. The clusters produced by the analysis
Interpretation of the results
The process has shown that not all datasets provide a chance for the usage of arbitrary cluster analysis.
Video
139
Created by XMLmind XSL-FO Converter.
Clustering 1
Workflow
clust_exp2.rmp
Keywords
K-medoids method
dataset properties
cluster analysis
Operators
k-Medoids
Read CSV
3. The DBSCAN method
Description
The process shows, using the Compound dataset, the advantages of using density based clustering by using the
DBSCAN clustering algorithm.
Input
Compound [SIPU Datasets] [Compound]
The dataset consist of 399 two-dimensional vectors belonging to six groups. The groups differ in the density of
their points, and each set can encompass another point set as well.
Figure 11.8. The groups with varying density
140
Created by XMLmind XSL-FO Converter.
Clustering 1
Output
Figure 11.9. The results of the method with default parameters
The process yields remarkable results even with default settings, only one out of the six clusters contains an
error. Using the parameters epsilon and min points, results can be refined further, but as the points in the
above-mentioned cluster are of different densities, a perfect solution cannot be reached.
Interpretation of the results
The operation of the DBSCAN algorithm has been demonstration on a dataset consisting of groups with
different densities. The deficiencies of the algorithm have been shows, i.e. that if points of different densities
can be found in a cluster, the precise operation of the algorithm cannot be guaranteed.
141
Created by XMLmind XSL-FO Converter.
Clustering 1
Video
Workflow
clust_exp3.rmp
Keywords
DBSCAN method
density function
cluster analysis
Operators
DBSCAN
Read CSV
4. Agglomerative methods
Description
The process shows, using the Maximum Variance (R15) dataset, how to define the appropriate number of
clusters, and the agglomerative hierarchical clustering method.
Input
Maximum Variance (R15) [SIPU Datasets] [Maximum Variance]
The dataset contains 600 two-dimensional vectors, which form 15 separate groups. The task is to define the
cardinality of the groups, and to discover them.
Figure 11.10. The 15 group
142
Created by XMLmind XSL-FO Converter.
Clustering 1
Output
Figure 11.11. The resulting dendrogram
The result of aggregation clustering is a so-called dendrogram, which is such a tree structure the leaves of which
are the points themselves, and the intermediate nodes (the clusters) result from agglomerating two points or
subtrees (clusters). The method always contracts the two points (or clusters) closest to each other, thus building
up the tree, which will contain all the points by the end of the process. The length of the edges in the finished
dendrogram is proportional to the distance between the clusters, thus the number of edges on the appropriate
level defines the ideal number of clusters. So, at the beginning of the process, each point forms a cluster on its
own, while by the end of the process, all points get to be put into one single cluster.
Figure 11.12. The clustering generated from dendrogram
143
Created by XMLmind XSL-FO Converter.
Clustering 1
By using the Flatten clustering operator, the dendrogram can also be used for clustering, manually stating the
number of clusters as a single parameter. The figure shows the result of this cluster analysis.
Interpretation of the results
It could be seen that based on the dendrogram created, the ideal number of clusters can be defined, and then,
based on this, the cluster analysis can be performed as well.
Video
Workflow
clust_exp4.rmp
Keywords
Agglomerative method
agglomerative hierarchical clustering
cluster analysis
Operators
Agglomerative Clustering
Flatten Clustering
Multiply
Read CSV
5. Divisive methods
144
Created by XMLmind XSL-FO Converter.
Clustering 1
Description
The process shows, using the Maximum Variance (R15) dataset, how divisive hierarchical cluster analysis can
be done.
Input
Maximum Variance (R15) [SIPU Datasets] [Maximum Variance]
The dataset contains 600 two-dimensional vectors, which form 15 separate groups. The task is to define the
ideal cardinality of the groups, and to discover them.
Figure 11.13. The 600 two-dimensional vectors
145
Created by XMLmind XSL-FO Converter.
Clustering 1
Output
Figure 11.14. The subprocess
In order to perform divisive clustering, an arbitrary clustering method with which the division can be performed
is necessary. In the initial state, all points belong to the same cluster, then, the method continuously divides the
points into multiple groups, until finally, all points are placed into separate clusters. The operator determines the
ideal number of clusters, and assigns the points to the clusters as well.
Figure 11.15. The report generated by the clustering
146
Created by XMLmind XSL-FO Converter.
Clustering 1
In the present case, the procedure determined the number of cluster as 63 groups.
Figure 11.16. The output of the analysis
Then the points are assigned to these groups.
Interpretation of the results
It can be seen that indeed, the method has created a larger number of clusters, but due to this, the central clusters
can be separated better from each other.
Video
Workflow
clust_exp5.rmp
Keywords
Divisive method
divisive hierarchical clustering
cluster analysis
Operators
k-Means
Read CSV
Top Down Clustering
147
Created by XMLmind XSL-FO Converter.
Chapter 12. Clustering 2
Advanced methods
1. Support vector clustering
Description
The process shows, using the Jain dataset, how support vector clustering can be used, and what the effects of its
parameters are.
Input
Jain [SIPU Datasets] [Jain]
The dataset contains 373 two-dimensional vectors, which are organized into 2 groups. The challenge posed by
the point set is that clouds of points are aligned closely to each other, and they have non-spherical shapes.
Figure 12.1. The two groups
Output
During support vector clustering, data are transformed using kernel functions, and then, a circle is enlarged until
a state in which all points are located within the circle. Finally, the thus created boundary curve is transformed
148
Created by XMLmind XSL-FO Converter.
Clustering 2
back into real space along with the data, and thus the clusters are created. The kernel functions are identical to
the functions described at the support vector machines, their parameters are the same as well. Support vector
clustering has a unique parameter r, with which the radius of the circle in the transformed space can be defined.
Figure 12.2. Support vector clustering with polynomial kernel and p=0.21 setup
Firstly, let us test the polynomial kernel, letting the points reach over the boundary curve.
Figure 12.3. Unsuccessful clustering
149
Created by XMLmind XSL-FO Converter.
Clustering 2
It can be seen that the result is rather disappointing, the resulting cluster are extending into each other, and the
second cluster is considered to be noise by the method.
Figure 12.4. Clustering with RBF kernel
Switching to the RBF kernel, and not allowing the points to reach over the boundary curve, the result is much
more promising. However, the upper cluster is split into multiple clusters, but the lower one remains in one
piece, and is separated from the other clusters.
Figure 12.5. More promising results
Interpretation of the results
150
Created by XMLmind XSL-FO Converter.
Clustering 2
Just like when using support vector machines, when using SVC, the factors that influence the efficiency of the
method the most are choosing the appropriate kernel function, and finding the ideal value of the ability to
generalize.
Video
Workflow
clust2_exp1.rmp
Keywords
Support vector clustering
SVC
cluster analysis
kernel functions
Operators
Read CSV
Support Vector Clustering
2. Choosing parameters in clustering
Description
The process shows, using the Flame dataset, how the ideal parameters can be found automatically.
Input
Flame [SIPU Datasets] [Flame]
The dataset consists of 240 two-dimensional vectors, that belong to two clusters. The clusters are aligned close
to each other, and one of the clusters has a non-spherical shape.
Figure 12.6. The two groups containing 240 vectors
151
Created by XMLmind XSL-FO Converter.
Clustering 2
Output
Figure 12.7. The subprocess of the optimalization node
To perform parameter optimization, a performance operator is required, which, in this case, will be the node
measuring cluster distance.
Figure 12.8. The parameters of the optimalization
152
Created by XMLmind XSL-FO Converter.
Clustering 2
The parameters to be optimized, and their possible values are chosen in the parameter optimization operator, and
then, it is confided to the system to choose the ideal values.
Figure 12.9. The report generated by the process
In the present case, the best result was yielded by partitioning the task into 10 clusters, and defining the distance
between them with the Euclidean distance.
Figure 12.10. Clustering generated with the most optimal parameters
153
Created by XMLmind XSL-FO Converter.
Clustering 2
Interpretation of the results
For many parameterized clustering methods, it can be ideal to confide the determination of the appropriate
number of clusters to a performance measurement operator, and then run the clustering with the obtained values.
Video
Workflow
clust2_exp2.rmp
Keywords
Support vector clustering
SVC
cluster analysis
kernel functions
Operators
Cluster Distance Performance
k-Means
Optimize Parameters (Grid)
Read CSV
3. Cluster evaluation
154
Created by XMLmind XSL-FO Converter.
Clustering 2
Description
The process shows, using the Aggregation dataset, how to gather and display cluster metrics.
Input
Aggregation [SIPU Datasets] [Aggregation]
The dataset contains 788 two-dimensional vectors, which form 7 separate groups. In the present case, the aim is
to evaluate the clusters created.
Figure 12.11. The 788 vectors
Output
Figure 12.12. The evaluating subprocess
155
Created by XMLmind XSL-FO Converter.
Clustering 2
After reading the data, an agglomerative clustering is run with different parameters, and then using this, clusters
can be created. A similarity function is created to measure cluster density, and then, the results of the
measurements are saved for each parameter setting.
Figure 12.13. Setting up the parameters
60 different settings are tested, the number of clusters ranging from 2 to 20, and all three of the agglomeration
strategies of the agglomerative clustering are tried out.
Figure 12.14. Parameters to log
156
Created by XMLmind XSL-FO Converter.
Clustering 2
The cluster sizes, the cluster densities, the distribution of the points, and the agglomeration strategy are saved
for each setting.
Figure 12.15. Cluster density against k number of clusters
Figure 12.16. Item distribution against k number of clusters
157
Created by XMLmind XSL-FO Converter.
Clustering 2
The final result can be acquired by reading the log.
Interpretation of the results
The final result shows that the increase in the number of clusters leads to the increase of cluster densities, and
the decrease of point distribution in different paces for the tree different strategies. However, the single link
strategy falls a bit behind compared to the complete link and average link methods.
Video
Workflow
clust2_exp3.rmp
Keywords
cluster evaluation
agglomerative clustering
single link
complete link
average link
point density
point distribution
Operators
Agglomerative Clustering
Cluster Density Performance
Data to Similarity
Flatten Clustering
Item Distribution Performance
Log
Log to Data
Loop Parameters
158
Created by XMLmind XSL-FO Converter.
Clustering 2
Multiply
Read CSV
4. Centroid method
Description
The process shows, using the Maximum Variance (D31) dataset, that cluster centers are suitable for representing
even the whole of their clusters.
Input
Maximum Variance (D31) [SIPU Datasets] [Maximum Variance]
The dataset contains 3100 two-dimensional vectors, which are concentrated into 31 clusters. Using this dataset,
it is to be illustrated the generalization power centroids possess.
Figure 12.17. The vectors forming 31 clusters
159
Created by XMLmind XSL-FO Converter.
Clustering 2
Output
Figure 12.18. The extracted centroids
Centroids are obtained after the cluster analysis of the data, and then, to illustrate their representative power,
they are utilized as training data for a k-NN classifier.
Figure 12.19. The output of the k nearest neighbour method, using the centroids as
prototypes
160
Created by XMLmind XSL-FO Converter.
Clustering 2
The efficiency of the k-NN classification method primarily depends on the prototypes selected. Based on the
result, it can be seen that the well-chosen points have aided the classification.
Interpretation of the results
It can be seen that clustering can be a good starting point for the extraction of the prototypes of a dataset, which
can make the cut-back of the training dataset possible.
Video
Workflow
clust2_exp4.rmp
Keywords
centroids
X-means method
k-NN
Operators
Apply Model
Extract Cluster Prototypes
k-NN
Multiply
Read CSV
Set Role
X-Means
5. Text clustering
161
Created by XMLmind XSL-FO Converter.
Clustering 2
Description
The process shows, using the Twenty Newsgroups dataset, how the clustering of documents can be performed.
Input
A subset of the Twenty Newsgroups dataset [UCI MLR].
Note
The data set was donated to the UCI Machine Learning Repository by Tom Mitchell.
The dataset contains about 20000 news articles belonging to 20 topics. The subset of this dataset utilized here
contains only three of the topics, which are concerned with cars, electronics, and everyday politics.
Output
Figure 12.20. The preprocessing subprocess
The data are read by topic, transformed to lower-case, tokenized, stemmed, and then stop words are filtered out.
After this, the only thing that needs to be done is to cluster the TF-IDF vectors by document.
Figure 12.21. The clustering setup
162
Created by XMLmind XSL-FO Converter.
Clustering 2
The distance between the document vectors can be measured using the cosine similarity. The cluster labels are
transformed into class labels, and then, it is checked to what extent the clusters cover the individual topics.
Figure 12.22. The confusion matrix of the results
Interpretation of the results
The results show that cars have severely blended with electronics, which is possibly not too far from reality, as
there are numerous common points in the two professions.
Video
Workflow
clust2_exp5.rmp
Keywords
K-means method
cosine similarity
text clustering
text mining
Operators
k-Means
Map Clustering on Labels
Performance (Classification)
163
Created by XMLmind XSL-FO Converter.
Clustering 2
Filter Stopwords (English) [Text Mining Extension]
Process Documents from Files [Text Mining Extension]
Stem (Snowball) [Text Mining Extension]
Tokenize [Text Mining Extension]
Transform Cases [Text Mining Extension]
164
Created by XMLmind XSL-FO Converter.
Chapter 13. Anomaly detection
1. Searching for outliers
Description
The workflow, using the Wisconsin Diagnostic Breast Cancer (WDBC) dataset, shows how to find outliers
based on the distances measured between the data. This can be done either by measuring their distance from
their k nearest neighbours, or by checking whether their distance from some data object is above a given
threshold. The definition of an outlier is relative, it can always be defined in comparison with the distances
between the data objects. Thus if the distances between the data objects are basically great, a high threshold has
to be set for outliers.
Input
Wisconsin Diagnostic Breast Cancer (WDBC) [UCI MLR]
Output
It can be seen that outliers can be filtered out using the appropriate settings. As for example even differences
that range in the hundreds occur between the individual values of the represented attribute area, thus this result
can be obtained by setting the threshold for the Euclidean distance to the value 500.
Figure 13.1. Graphic representation of the possible outliers
165
Created by XMLmind XSL-FO Converter.
Anomaly detection
Interpretation of the results
Note that due to the existing great distances between the data objects, the number of outliers detected will only
decrease to its true level if the threshold is incremented up to 500, and under a certain value, way too many data
objects would be identified as outliers.
Figure 13.2. The number of outliers detected as the distance limit grows
Video
Workflow
anomaly_exp1.rmp
166
Created by XMLmind XSL-FO Converter.
Anomaly detection
Keywords
outliers
data preprocessing
data cleansing
Operators
Detect Outlier (Densities)
Detect Outlier (Distances)
Multiply
Read AML
2. Unsupervised search for outliers
Description
The process shows, using a sample of the Individual household electric power consumption dataset, how
incidentally occurring outliers, anomalies can be found in a dataset with unsupervised methods. Several methods
can be used for unsupervised anomaly detection, e.g., in general cases, methods based on k nearest neighbours,
which assign an outlier indicator value to each element based on its distance from its k nearest neighbours. The
higher this indicator value is, the value of the given item will be more of an outlier, and more likely to be a
potential anomaly. However, this scoring may vary depending on the dataset and the method utilized, so the
threshold from which a given element is to be considered an outlier should be set according to the distances
between the data and the methods used.
Input
Individual household electric power consumption [UCI MLR]
Output
167
Created by XMLmind XSL-FO Converter.
Anomaly detection
The Anomaly Detection extension that can be installed in Rapid Miner proposes several possible methods for
detecting anomalies, for example the method based on k nearest neighbours, and the LOF metric based on this,
which relies on the k nearest neighbours method, but it also takes density into consideration.
Figure 13.3. Nearest neighbour based operators in the Anomaly Detection package
Figure 13.4. Settings of LOF.
These methods assign different scores to the elements, based on which it can be seen which elements are
outliers. The k nearest neighbours method assigns the following scores to the elements of the dataset:
Figure 13.5. Outlier scores assigned to the individual records based on k nearest
neighbours
168
Created by XMLmind XSL-FO Converter.
Anomaly detection
The LOF method assigns the following scores to the elements of the dataset:
Figure 13.6. Outlier scores assigned to the individual records based on LOF
Interpretation of the results
Based on the results received, it can be decided which score should be considered as a threshold above which an
element is considered an anomaly, and the elements with scores above this threshold, i.e. outliers can also be
immediately filtered out of the dataset, or a separate dataset can be formed from them:
Figure 13.7. Filtering the records based on their outlier scores
169
Created by XMLmind XSL-FO Converter.
Anomaly detection
For example, using the k-NN method, the following dataset appears as a result after removing the values rated
as outliers:
Figure 13.8. The dataset filtered based on the k-NN score
Furthermore, the set of elements rated as outlier based on the LOF is the following:
Figure 13.9. The dataset filtered based on the LOF score
Video
Workflow
anomaly_exp2.rmp
Keywords
outliers
anomaly detection
k nearest neighbours
Operators
170
Created by XMLmind XSL-FO Converter.
Anomaly detection
Filter Examples
Multiply
Read CSV
k-NN Global Anomaly Score [Anomaly Detection]
Local Outlier Factor (LOF) [Anomaly Detection]
3. Unsupervised statistics based anomaly detection
Description
The process shows, using the Flame dataset, how incidentally occurring outliers, anomalies can be found in a
dataset using statistics based unsupervised methods. Several methods can be used for unsupervised anomaly
detection, e.g., a statistics based, histogram based method. In this case, groups of values are defined for each
attribute with a histogram, and based on the deviation from these can the given value in the given column be
considered an outlier. After this, using these scores is the overall outlier score of the records defined. The higher
this indicator value is, the value or record will be more of an outlier, and more likely to be a potential anomaly.
However, this scoring may vary depending on the dataset and the method utilized, so the threshold from which a
given element is to be considered an outlier should be set according to the distances between the data and the
methods used. At the same time, due to this fact, it can be more illustrative to use colors instead of only values
to indicate the outlier scores, which is done automatically be the histogram based method.
Input
Flame [SIPU Datasets] [Flame]
Output
The Anomaly Detection extension that can be installed in Rapid Miner proposes several possible methods for
detecting anomalies, for example the histogram based method, which defines the outlier score of the individual
values in each column of the dataset, and based on these, it calculates the final score of the records. This method
can be refined with multiple settings, either on operator level, or on column level as well:
Figure 13.10. Global settings for Histogram-based Outlier Score
171
Created by XMLmind XSL-FO Converter.
Anomaly detection
Figure 13.11. Column-level settings for Histogram-based Outlier Score
Based on the settings, the operator splits the set of values in the individual columns into either a pre-defined or
an arbitrary number of bins which are either equal or variable in length. Based on these, it assigns color codes,
and calculates the record level score based on the scores of the column values as well. Using a fixed binwidth,
and an arbitrary number of bins, the following values are returned as a result:
Figure 13.12. Scores and attribute binning for fixed binwidth and arbitrary number of
bins
172
Created by XMLmind XSL-FO Converter.
Anomaly detection
Interpretation of the results
Based on the results received, it can be decided which score should be considered as a threshold above which an
element is considered an anomaly. In this case, however, a more detailed examination is possible as well, as
based on the colour codes set, it can be viewed how much the probability of the individual attributes containing
outliers is, and if these coincide with outlier values of other columns. Based on this, on one hand, the model can
be refined if necessary, and on the other hand, in some cases, it can be easier to define which values should be
considered an anomaly. By checking the graphic representation of the model built based on the scores, it can be
seen that there are sightly outlying values that have not been assigned a high score:
Figure 13.13. Graphic representation of outlier scores
173
Created by XMLmind XSL-FO Converter.
Anomaly detection
Based on this, it might be advisable to alter the model, for example to split the attributes into dynamically sized
bins. This enhances the performance of outlier detection, as can be seen in the following results:
Figure 13.14. Scores and attributes binning for dynamic binwidth and arbitrary
number of bins
174
Created by XMLmind XSL-FO Converter.
Anomaly detection
Figure 13.15. Graphic representation of the enhanced outlier scores
Video
Workflow
anomaly_exp3.rmp
Keywords
outliers
anomaly detection
statistics based anomaly detection
histogram based anomaly detection
bin size
Operators
Read CSV
Histogram-based Outlier Score (HBOS) [Anomaly Detection]
175
Created by XMLmind XSL-FO Converter.
Part III. SAS® Enterprise Miner™
Created by XMLmind XSL-FO Converter.
Table of Contents
14. Data Sources .............................................................................................................................
1. Reading SAS dataset .........................................................................................................
2. Importing data from a CSV file .........................................................................................
3. Importing data from a Excel file .......................................................................................
15. Preprocessing ............................................................................................................................
1. Constructing metadata and automatic variable selection ..................................................
2. Vizualizing multidimensional data and dimension reduction by PCA ..............................
3. Replacement and imputation .............................................................................................
16. Classification Methods 1 ..........................................................................................................
1. Classification by decision tree ...........................................................................................
2. Comparison and evaluation of decision tree classifiers ....................................................
17. Classification Methods 2 ..........................................................................................................
1. Rule induction to the classification of rare events .............................................................
18. Classification Methods 3 ..........................................................................................................
1. Logistic regression ............................................................................................................
2. Prediction of discrete target by regression models ............................................................
19. Classification Methods 4 ..........................................................................................................
1. Solution of a linearly separable binary classification task by ANN and SVM ..................
2. Using artificial neural networks (ANN) ............................................................................
3. Using support vector machines (SVM) .............................................................................
20. Classification Methods 5 ..........................................................................................................
1. Ensemble methods: Combination of classifiers ................................................................
2. Ensemble methods: bagging .............................................................................................
3. Ensemble methods: boosting .............................................................................................
21. Association mining ...................................................................................................................
1. Extracting association rules ...............................................................................................
22. Clustering 1 ..............................................................................................................................
1. K-means method ...............................................................................................................
2. Agglomerative hierarchical methods .................................................................................
3. Comparison of clustering methods ....................................................................................
23. Clustering 2 ..............................................................................................................................
1. Clustering attributes before fitting SVM ...........................................................................
2. Self-organizing maps (SOM) and vector quantization (VQ) .............................................
24. Regression for continuous target ..............................................................................................
1. Logistic regression ............................................................................................................
2. Prediction of discrete target by regression models ............................................................
3. Supervised models for continuous target ..........................................................................
25. Anomaly detection ....................................................................................................................
1. Detecting outliers ..............................................................................................................
177
Created by XMLmind XSL-FO Converter.
178
178
180
183
185
185
188
191
196
196
200
208
208
212
212
217
221
221
225
232
240
240
244
249
256
256
260
260
267
271
278
278
284
289
289
294
297
304
304
Chapter 14. Data Sources
1. Reading SAS dataset
Description
The experiment illustrates how existing SAS data sets can be made available to the SAS® Enterprise Miner™
by the Input Data operator. In the experiment, an earlier prepared SAS dataset will be read. A SAS dataset can
be created by using the SAS® System or the SAS® Enterprise Guide™. In order to load a SAS file that we
would like to use we need to know the path to the file. The file may be on the local machine, but also can be on
a remote SAS server. The SAS file can be read by using a wizard that guides you through the entire process.
Then, the original dataset is sampled by the Sample operator where a part of the relatively large data file is
selected.
Figure 14.1. The metadata of the dataset
Input
Individual household electric power consumption [UCI MLR]
Output
A dataset which contains the 10 percent of the original dataset. At the sampling, absolute and relative sample
size can be chosen as well. It is also possible to set the Random Seed parameter which controls the cycle of the
pseudo-random number generator. If the same value is set to on different machines we get the same random
sample. We also set the method of sampling, e.g. simple random, clustered or stratified.
Figure 14.2. Setting the Sample operator
178
Created by XMLmind XSL-FO Converter.
Data Sources
Figure 14.3. The metadata of the resulting dataset and a part of the dataset
179
Created by XMLmind XSL-FO Converter.
Data Sources
Interpretation of the results
Whenever we rerun the process, the current state of the data set will be imported to the system, so the Input
Data operator can be used to retrieve data files and to rerun the data mining process based on them, which are
updated constantly by other SAS based systems.
Video
Workflow
sas_import_exp1.xml
Keywords
reading SAS dataset
sampling
Operators
Data Source
Sample
2. Importing data from a CSV file
Description
The process demonstrates how to import data from CSV datasets by the File Import operator. In the
experiment, the Bodyfat dataset of the StatLib data repository is used. In order to open the dataset we would like
to use, we need to know the path to this file which can be on the local machine or on a remote SAS server as
well. This path can be assigned step by step in a menu.
Figure 14.4. The list of file in the File
Import
operator
180
Created by XMLmind XSL-FO Converter.
Data Sources
Input
Bodyfat [StatLib]
Note
The dataset was donated by Roger W. Johnson to the StatLib.
The process of import can be parametrized in the File Import operator. We can set the maximal number of
records, the maximal number of attributes and the separator character. It is also possible to define the number of
rows which determines the file structure.
Figure 14.5. The parameters of the File
Import
operator
Output
A datatset which consists of the imported data.
Figure 14.6. A small portion of the dataset
181
Created by XMLmind XSL-FO Converter.
Data Sources
Figure 14.7. The metadata of the resulting dataset
Interpretation of the results
Whenever we rerun the process, the current state of the data set will be imported to the system, so the Input
Data operator can be used to reload data files and to rerun the data mining process based on them, which are
updated constantly by other SAS based systems.
Video
Workflow
sas_import_exp2.xml
Keywords
importing data
CSV file
Operators
File Import
182
Created by XMLmind XSL-FO Converter.
Data Sources
Graph Explore
Statistic Explore
3. Importing data from a Excel file
Description
The process illustrates how to import data from an Excel dataset by the help of the File Import operator. In
the experiment the Zoo dataset is used, which was saved previously as an Excel file. In order to open the file that
we would like to use, we need to know the path to this file which can be on the local computer or on a remote
SAS server as well. This path can be assigned step by step by going through the directory tree.
Input
Zoo [UCI MLR]
The process of import can be parametrized in the File Import operator. We can set the maximal number of
records, the maximal number of attributes. It is also possible to define the number of rows which determines the
file structure.
Output
A dataset which contains the imported data.
Figure 14.8. A small portion of the resulting dataset
Interpretation of the results
Whenever we rerun the process the system will import the newest version of the dataset. Thus the File Import
operator can be used to load such datasets and to rerun data mining processes based on them which are updated
by operative systems. The import procedure works only for Excel 97-2003 files with xls extension.
183
Created by XMLmind XSL-FO Converter.
Data Sources
Video
Workflow
sas_import_exp3.xml
Keywords
importing data
Excel
Operators
File Import
Graph Explore
184
Created by XMLmind XSL-FO Converter.
Chapter 15. Preprocessing
1. Constructing metadata and automatic variable
selection
Description
The process illustrates, by using the Spambase dataset, how to generate the metadata of a dataset by the DMDB
operator, then how automatic variable selection can be obtained by the Variable Selection operator. The
Spambase dataset contains 58 attributes, one of which is the binary target. In order to visualize a dataset, it may
be necessary to determine the most important input attributes which can be used in the graphical representation.
Input
Spambase [UCI MLR]
Output
The DMDB operator produces such metadata (descriptive statistics) as mean, variance, minimum, maximum,
skewness, and kurtosis. In case of discrete attributes these are complemented by the mode.
Figure 15.1. Metadata produced by the DMDB operator
185
Created by XMLmind XSL-FO Converter.
Preprocessing
The default settings of the Variable Selection operator are applied except that the minimum R-square is
increased in order to filter the unnecessary attributes.
Figure 15.2. The settings of Variable
Selection
operator
The result on the one hand will be a list which contains the decision about the variables, i.e., whether it remains
or not in the data mining process, on the other hand, a few graphs of the importance of the variables.
Figure 15.3. List of variables after the selection
186
Created by XMLmind XSL-FO Converter.
Preprocessing
Figure 15.4. Sequential R-square plot
In view of the important variables a number of graphical tools of the Enterprise Miner ™ can be used to display
the records.
Figure 15.5. The binary target variables in a function of the two most important input
attributes after the variable selection
187
Created by XMLmind XSL-FO Converter.
Preprocessing
Interpretation of the results
The experiment shows how metadata can be extracted from SAS datasets which we can then transmit to other
operators. Moreover, we demonstrated how can variable selection be performed in case of large number of
attributes and how can we be working with the important attributes.
Video
Workflow
sas_preproc_exp1.xml
Keywords
variable selection
metadata
Operators
Data Source
Data Mining DataBase
Graph Explore
Variable Selection
2. Vizualizing multidimensional data and dimension
reduction by PCA
188
Created by XMLmind XSL-FO Converter.
Preprocessing
Description
The experiment presents vizualization and dimension reduction methods by the help of the Fisher-Anderson Iris
dataset. Multidimensional datasets can be vizualized by the Graph Explore operator. Dimension reduction can
be performed by the Principal Components operator. After the dimension reduction, it becomes much easier
to display multi-dimensional datasets in the space of principal components.
Input
Fisher-Anderson Iris
Output
The Graph Explore operator provides several graphical tools for displaying multi-dimensional datasets, which
plays a key role in the preprocessing step of data mining. Some of these are extensions of well-known tools such
as two- and three-dimensional scatterplots and bar charts supplemented by a number of options such as the use
of colors and symbols. Other techniques such as parallel axis or the radar plot, however, are only characteristics
of data mining software tools.
Figure 15.6. Displaying the dataset by parallel axis
189
Created by XMLmind XSL-FO Converter.
Preprocessing
The Pricipal Components Analysis (PCA) can be performed by the Principal Components operator. In the
operator the following settings can be defined: the dependency structure as covariance or correlation, the cut-off
condition as the number of eigenvalues or the cumulative eigenvalue ratio.
Figure 15.7. Explained cumulated explained variance plot of the PCA
190
Created by XMLmind XSL-FO Converter.
Preprocessing
The main result of principal component analysis is the principal component coordinates of individual records,
which can be used in the further data analysis and visualization.
Figure 15.8. Scatterplit of the Iris dataset using the first two principal components
Interpretation of the results
The experiment shows that how we can display high-dimensional data sets and perform dimension reduction. In
our experiment, the original 4-dimensional data set that can not be displayed using the standard scatterplot, is
managed to reduce to 2 dimensions such that the 95 percent of the information contained in the data is
preserved.
Video
Workflow
sas_preproc_exp2.xml
Keywords
principal components analysis (PCA)
parallel axis
Operators
Data Source
Graph Explore
Principal Components
3. Replacement and imputation
191
Created by XMLmind XSL-FO Converter.
Preprocessing
Description
In this experiment, we demonstrate by the help of the Congressional Voting Records dataset how to modify the
values of attributes by the Replacement operator and then how to impute the missing value by the Impute
operator. The replacement of missing values for each variable can be carried out independently of the others and
to interact with the target variable by fitting a decision tree.
Input
Congressional Voting Records [UCI MLR]
Output
By the Replacement operator we can set the substitution of discrete and continuous variables separately.
Figure 15.9. The replacement wizard
192
Created by XMLmind XSL-FO Converter.
Preprocessing
A number of imputation methods can be choosen in the Impute operator. We may fill in the missing values by a
constant value, but also can use distribution-based value, where a random value is generated by the system, or
decision tree based method.
Figure 15.10. The output of imputation
The results of the imputation correlated by the target variable are shown in the following two bar charts.
Figure 15.11. The relationship of an input and the target variable before imputation
193
Created by XMLmind XSL-FO Converter.
Preprocessing
Figure 15.12. The relationship of an input and the target variable after imputation
Interpretation of the results
The experiment shows that if the method of imputation is chosen in appropriate way the values obtained in place
of the missing data values is not very distorted and thus, on a larger dataset, we can perform a more reliable
fitting of the model.
Video
Workflow
194
Created by XMLmind XSL-FO Converter.
Preprocessing
sas_preproc_exp3.xml
Keywords
replacement
imputation
Operators
Data Source
Graph Explore
Impute
Replacement
195
Created by XMLmind XSL-FO Converter.
Chapter 16. Classification Methods 1
Decision trees
1. Classification by decision tree
Description
The process demonstrates how to classify by the Decision Tree operator in the case when the target is a
nominal attribute. In this case the Wine dataset is used and the target variable has three values. In order to build
a decision tree classifier it is worth to divide the dataset to training and validation datasets. Then the current best
splitting rule is found by the algorithm on the training set, but the growth of the tree is stopped by using the
validation dataset when the algorithm does not find a significant split. In partitioning step, a test dataset can be
separated too in order to measure the generalization ability of the resulting tree, but now this is not
recommended due to the limited size of the data set. The decision tree as the result of the process can be
displayed where we can see the decisions at the splittings of the model. Using the principle of majority voting
the algorithm decides which class label should assign to each leaf (terminal nodes).
Input
Wine [UCI MLR]
Output
In case of nominal target variable, we can decide about the execution of each split on the basis of various
impurity measures such as the chi-square, the Gini index, or the entropy. For these, and for the reliability of
splitting, depending on the choosen measure, a parameter value can be specified. In addition, the stopping
condition of splitting can be determined by the way that we give the minimum size set of records can be divided
even further, or the maximal depth of the tree. Also we may set the maximum number of branches of a tree. The
default is 2, that is, the algorithm builds a binary tree. It is also possible to decide if we wish to use the missing
values in splitting used as possible value. We can also decide that the input attributes are used only once or
several times when the decision tree is produced.
Figure 16.1. The settings of dataset partitioning
196
Created by XMLmind XSL-FO Converter.
Classification Methods 1
In the partition of dataset different sampling methods can be choosen and the proportion can be determined
among the training, validation, and test dataset. This partitioning can be carried out simply by considering the
order of records, randomly, or stratifying with respect to the target variable. The stratified sampling ensures the
same proportion of each class in the training, validation, and test set.
Figure 16.2. The decision tree
The results of the classification can be seen in the decision tree for the training and validation dataset as well,
including the number of records in each vertex of the tree according to each class, respectively. On the edges
between vertices the variables that define the splittings and their splitting values are presented. The thickness of
the lines is proportional to the number of concerned records.
197
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Interpretation of the results
The evaluation of the resulting decision tree is supported by numerous statistical indicators and graphical tools.
Among them, the most important ones are displayed by multiple windows at a time where we can do
comparisons. These windows can also be opened one by one by the view menu. By the help of this tools wrong
decisions can be filtered out and the modeling process can be tuned by using further background information or
domain knowledge. In this process, an interactive tree building process also helps.
Figure 16.3. The response curve of the decision tree
On the response curve above it can be seen, for the training and validation dataset, based on the ranking of
records according to their goodness how many percent of the records are classified correctly. The curve is
generally monotonically decreasing.
Figure 16.4. Fitting statistics of the decision tree
In the Fit Statistics table different indicators can be seen on the fiiting of the decision tree classifier produced by
the algorithm. The simplest and most important one among them is the misclassification rate in the red circle,
which shows the proportion of the wrong classification.
Figure 16.5. The classification chart of the decision tree
198
Created by XMLmind XSL-FO Converter.
Classification Methods 1
On the classification bar chart we can look at the details which classes work well or poorly the model.
Figure 16.6. The cumulative lift curve of the decision tree
On the figure, it can be concluded how relates the resulting decision tree to the best possible model based on the
cumulative lift value.
199
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Figure 16.7. The importance of attributes
The variable importance measure table shows which variables and with what importance involved in the
decision of the decision tree. This is a useful tool for the users who possess some speciality knowledge.
Video
Workflow
sas_dtree_exp1.xml
Keywords
classification
decision tree
cutting
response curve
misclassification rate
Operators
Data Source
Decision Tree
Data Partition
2. Comparison and evaluation of decision tree
classifiers
200
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Description
The process illustrates how to fit decision trees by using different impurity measures and then how to compare
these models on the basis of the Congressional Voting Records dataset. After the decision tree is built based on
the training and the validation dataset, the best model is selected by the model comparison operator ( Model
Comparison) using the validation dataset during the decision process. Finally, the quality of the performed
classification can be studied on the test dataset. Using the resulting model we can perform the scoring, which is
the evaluating of the test set or a data set where we do not know the value of the target variable.
Input
Congressional Voting Records [UCI MLR]
Output
In the process we set the following parameters during the partitioning step.
Figure 16.8. The settings of parameters in the partitioning step
201
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Using the chi-square impurity measure we obtain the following decision tree.
Figure 16.9. The decision tree using the chi-square impurity measure
Using the entropy impurity measure we obtain the following decision tree.
Figure 16.10. The decision tree using the entropy impurity measure
202
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Using the Gini impurity measure we obtain the following decision tree.
Figure 16.11. The decision tree using the Gini-index
203
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Interpretation of the results
The first of the three resulting decision tree is the simplest and its fitting is the worst. The other two ones are
fairly similar to each other, they use the same input variables in the splits, only the splitting values differ a little
bit. The three decision trees can be compared in many ways by using graphical tools and statistics. For example,
the following chart shows that, on the basis of the cumulative response curve, decision tree given by the Gini
index is the best model if we want to build a model until the first few deciles.
Figure 16.12. The cumulative response curve of decision trees
204
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Looking at the efficiency of the classifier the number of the correctly or incorrectly classified records can be
obtained with respect to the two parties that can be represented by a bar chart.
Figure 16.13. The classification plot
A detailed comparison is possible by using the response curve, the lift curve, and their variants. The following
figure shows the (non-cumulative) response curve for the three different datasets and the three types of models.
It can be seen how to relate the response curve to the best possible and the baseline one. The bottom right figure
shows that the decision tree based on the Gini index is close to the optimal one on the test dataset.
Figure 16.14. Response curve of decision trees
205
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Another possibility is the examination of the score distributions. The model is fitting well if the red and the blue
line is a mirror image of each other and their gradients are high. In virtue of this indicator the entropy-based
decision tree is the best.
Figure 16.15. The score distribution of decision trees
The three built trees can be improved further by changing the level of significance. As a consequence, the
decision tree will be built in a different way comparing to the original one, and as a result of this the number of
correctly and incorrectly classified records and their distribution will be different. The performance of the
resulting models can be read in the following figure, on which the incorrect classification rate was underlined as
one of the most important indicators.
Figure 16.16. The main statistics of decision trees
206
Created by XMLmind XSL-FO Converter.
Classification Methods 1
Video
Workflow
sas_dtree_exp2.xml
Keywords
classification
decision tree
performance
evaluation
Operators
Data Source
Decision Tree
Model Comparison
Data Partition
Score
207
Created by XMLmind XSL-FO Converter.
Chapter 17. Classification Methods 2
Rule induction for rare events
1. Rule induction to the classification of rare events
Description
In this experiment, using the Spambase dataset, we show how can a baseline classifier be improved for a binary
classification task with rare events by Rule Induction operator.
Input
Spambase [UCI MLR]
The corresponding input is prepared using the Sample operator. The number of records are deposited on the top
of the dataset is choosen such that the proportion of the cases to be 5 percent. Then, we partition the dataset in
the usual way.
Output
Two rule induction models are fitted to the dataset. The former is based on decision tree model, the latter is
based on logistic regression model. The fitted models are compared to a baseline decision tree classifier. The
figures below show the goodness of fit.
Figure 17.1. The misclassification rate of rule induction
Figure 17.2. The classification matrix of rule induction
208
Created by XMLmind XSL-FO Converter.
Classification Methods 2
Figure 17.3. The classification chart of rule induction
On the left side of the model comparison figure a perfect ROC curve can be seen. This curve clearly shows that
the fitting is perfect on the training dataset in case of the second rule induction model.
Figure 17.4. The ROC curves of rule inductions and decision tree
209
Created by XMLmind XSL-FO Converter.
Classification Methods 2
On the next output window, the number of wrongly classified cases can also be seen as the output of the Rule
Induction operator, besides the usual information. This is a very important information in this case.
Figure 17.5. The output of the rule induction operator
Interpretation of the results
The experiment shows that, when the class is very uneven, i.e. one class frequency is very low, compared to the
traditional classification models, significant improvement can be achieved by the rule induction method.
Video
Workflow
210
Created by XMLmind XSL-FO Converter.
Classification Methods 2
sas_rules_exp1.xml
Keywords
rule induction
supervised learning
classification
Operators
Data Source
Decision Tree
Model Comparison
Data Partition
Rule Induction
Sample
211
Created by XMLmind XSL-FO Converter.
Chapter 18. Classification Methods 3
Logistic Regression
1. Logistic regression
Description
The process shows, using the Spambase dataset, how can a regression model be fitted to a dataset which has
binary target. The conventional linear regression are not suitable for this task even though the Regression
operator offers this option. Instead, we must use the logistic regression method which is the default option of
this operator. We can choose between the following link functions: logit, which takes the name of the procedure,
probit and complementary logit. There is no significant difference among these link functions. The Enterprise
Miner™ gives an other operator for fitting regression. By the Dmine Rgeression operator forward stepwise
regression can be fitted. In each step, an input variable is selected that contributes most significantly to the
variability of the target.
Input
Spambase [UCI MLR]
Output
After fitting the logistic regression, standard statistics and graphs are obtained similarly to the binary
classification tasks. Here, only the confusion matrix is shown, the rest of comparison tools is left at the and of
this experiment.
Figure 18.1. Classification matrix of the logistic regression
212
Created by XMLmind XSL-FO Converter.
Classification Methods 3
In addition to the usual tools, the regression operators, using the effect plot, also show the importance of the
input variables in the regression model which were built during the process.
Figure 18.2. Effects plot of the logistic regression
In addition to the traditional regression analysis Enterprise Miner ™ yields another operator to fit forward
stepwise regression. This is the Dmine Rgeression operator. The results can be seen in the figures below.
Figure 18.3. Classification matrix of the stepwise logistic regression
213
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Figure 18.4. Effects plot of the stepwise logistic regression
The two regressions can be compared by the usual way with the Model Comparison operator. The results of
this comparison are presented in the following figures.
Figure 18.5. Fitting statistics for logistic regression models
214
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Figure 18.6. Classification charts of the logistic regression models
Figure 18.7. Cumulativ lift curve of the logistic regression models
215
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Figure 18.8. ROC curves of the logistic regression models
Interpretation of the results
The fit statistics and ROC curves clearly show on the test set that the logistic regression model is better than the
stepwise logistic regression model.
Video
Workflow
sas_regr_exp1.xml
216
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Keywords
classification
binary target
logistic regression
Operators
Data Source
Dmine Regression
Model Comparison
Data Partition
Regression
2. Prediction of discrete target by regression models
Description
The process shows, using the Wine dataset, that how can we fit a regression model to a dataset containing
discrete but non-binary target variable. Moreover, how can we performe a classification task using the parameter
estimates obtained from the model. The type of the fitted regression model depends on the measurement scale of
the discrete target variable. If the target is nominal then the Regression operator fits binary logistic models
separately, so that a selected event of the target is compared to the class of the other values of the discrete target
variable. On the other hand, if the target is ordinal then a common logistic regression model are fitted, wherein
only the constant parameters differ, but the parameters of the input variables are shared. (As opposed to the
nominal case where these coefficients are different.)
Input
Wine [UCI MLR]
Output
A number of models can be choosen to fit a regression model, e.g., linear or logistic regression. Among them,
the logistic regression used in the process. It is not needed to set in, the software recognizes the right type of the
regression using the metadata on the target. Of course, it is possible to override this option and to enforce linear
regression, but this does not make sense because in this case the Model Comparison operator can not be used to
compare this model to other discrete supervised models. The fit of the model can be tested by the well-known
statistics and charts.
Figure 18.9. Classification matrix of the logistic regression
217
Created by XMLmind XSL-FO Converter.
Classification Methods 3
The classification chart shows that the fitted model is perfect on the training dataset and has small error on the
validation dataset.
Figure 18.10. The classification chart of the logistic regression
218
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Besides the standard goodness-of-fit tests the Regression operator presents a bar graph showing the
importance of the input variables in the regression models. The higher the coefficient of an input variable, the
more is its explanation power with respect to the target variable. In case of ordinal target, there is only one,
while, in case of the nominal target, the number of different values of the target variable minus 1 bar graphs are
created. Since the Class variable has three values there are two such bar graphs below.
Figure 18.11. The effects plot of the logistic regression
Interpretation of the results
Using the regression model created on the training set for the validation dataset the above results show that a
model can be built with relatively high accuracy for multiple discrete target variable value by the Regression
operator. It is noted that for the problem discussed here the Dmine Regression operator can not be applied.
Video
Workflow
sas_regr_exp2.xml
219
Created by XMLmind XSL-FO Converter.
Classification Methods 3
Keywords
classification
nominal and ordinal target
logistic regression
Operators
Data Source
Data Partition
Regression
220
Created by XMLmind XSL-FO Converter.
Chapter 19. Classification Methods 4
Neural networks and support vector machines
1. Solution of a linearly separable binary classification
task by ANN and SVM
Description
In this experiment, we teach a perceptron as a simple artificial neural network (ANN) and a support vector
machine ( SVM) on a linearly separable two dimensional dataset with two classes. The dataset is a subset of the
Wine dataset. adatállomány egy részhalmaza. The classification accuracy of the classifiers is determined on the
dataset.
Input
Wine [UCI MLR]
Figure 19.1. A linearly separable subset of the Wine dataset
221
Created by XMLmind XSL-FO Converter.
Classification Methods 4
In order to apply the dataset for the experiment it have to go through a significant preprocessing. This involves
choosing 2 attributes from the total 13 by using the Drop operator, then deleting the second class of the Class
attribute and changing the measure level to binary by Metadata operator.
Output
The goodness of fit of the perceptron model can be checked by the usual statistics (misclassification rate, the
number of incorrectly classified cases) and graphics (response and lift curve).
Figure 19.2. Fitting statistics for perceptron
Figure 19.3. The classification matrix of the perceptron
Figure 19.4. The cumulative lift curve of the perceptron
222
Created by XMLmind XSL-FO Converter.
Classification Methods 4
In the case of support vector machines (SVM), besides the above mentioned diagnostic tools, more details are
given about the support vectors, namely the number of support vectors, the size of the margin and the list of
support vectors.
Figure 19.5. Fitting statistics for SVM
Figure 19.6. The classification matrix of SVM
Figure 19.7. The cumulative lift curve of SVM
223
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 19.8. List of the support vectors
Interpretation of the results
The figures and statistics show that the perceptron and support vector machine classify perfectly all of the
traning cases as well.
Video
Workflow
sas_ann_svm_exp1.xml
Keywords
224
Created by XMLmind XSL-FO Converter.
Classification Methods 4
perceptron
supervised learning
classification
Operators
Data Source
Drop
Filter
Graph Explore
Metadata
Neural Network
Support Vector Machine
2. Using artificial neural networks (ANN)
Description
In this experiment, a number of algorithms that can be used for tranining artificial neural networks are compared
for binary classification task. In the experiment the Spambase dataset is used. The classification accuracy of the
resulting classifier is determined and the interpretation of related graphs is reviewed.
Input
Spambase [UCI MLR]
Before fitting the models the dataset is partitionated by the Data Partition operator according to the rates
60/20/20 among the training, validatation and test datasets.
Output
Firstly, a standard artificial neural network is fitted by the NeuralNetwork operator where the network topology
of a multilayer perceptron is defined as 3 hidden neuron in one hidden layer. The goodness-of-fit of the resulting
model can be verified using standard statistics (misclassification rate, the number of incorrectly classified cases)
and graphics (response and lift curve).
Figure 19.9. Fitting statistics of the multilayer perceptron
225
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 19.10. The classification matrix of the multilayer perceptron
Figure 19.11. The cumulative lift curve of the multilayer perceptron
226
Created by XMLmind XSL-FO Converter.
Classification Methods 4
In addition to the standard goodness-of-fit tests we get results which have meaning for artificial neural networks
only. These results involve the graph of the weights of the neurons, the graph of the history of the teaching
where the misclassification rate can be seen as the function of the iteration for training and validation datasets.
Figure 19.12. Weights of the multilayer perceptron
Figure 19.13. Training curve of the multilayer perceptron
227
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Similar graphs were obtained for the other two neural network fitting operator, namely for the DMNeural
operator and for the AutoNeural operator. At the first, the exception is the following stepwise optimization
statistics.
Figure 19.14. Stepwise optimization statistics for DMNeural operator
Figure 19.15. Weights of neural networks
neuronjainak súlyai
AutoNeural
228
Created by XMLmind XSL-FO Converter.
operátorral kapott háló
Classification Methods 4
Finally, the three models can be compared by the Model Comparison operator. As a result, we obtain the
following statistics and graphs.
Figure 19.16. Fitting statistics of neural networks
229
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 19.17. Classification charts of neural networks
Figure 19.18. Cumulative lift curves of neural networks
230
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 19.19. ROC curves of neural networks
Interpretation of the results
The above statistics and figures clearly show that the best model is the first artificial neural network with multilayer perceptron architecture, where there is one hidden layer with 3 neurons.
231
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Video
Workflow
sas_ann_svm_exp2.xml
Keywords
artificial neural network
supervised learning
classification
Operators
Data Source
Model Comparison
Neural Network
Data Partition
3. Using support vector machines (SVM)
Description
In this experiment, support vector machines (SVM - Support Vector Machine) are fitted to solve the binary
classification task using the Spam Database dataset. The aim of this experiment is the comparison of different
kinds of SVM defined by linear, polynomial etc. kernel functions. The classification accuracy of the resulting
classifiers are determined and we review the interpretation of statistics and graphs related to support vector
machines. The model fitting is carried out by the SVM operator.
Input
Spambase [UCI MLR]
Before fitting the models the dataset is partitionated by the Data Partition operator according to the rates
60/20/20 among the training, validatation and test datasets.
Output
Firstly, a support vector machine is fitted using linear kernel. The goodness-of-fit of the resulting model can be
checked by standard statistics (e.g. misclassification rate, the number of incorrectly classified cases) and graphs
(e.g. response and lift curve). These tools will be discussed later during the comparison of the two models.
Besides these results the SVM operator provides such additional statistics as goodness-of-fit measures and list of
support vectors which have meaning only for support vector machines.
232
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 19.20. Fitting statistics for linear kernel SVM
Figure 19.21. The classification matrix of linear kernel SVM
Figure 19.22. Support vectors (partly) of linear kernel SVM
233
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 19.23. The distribution of Lagrange multipliers for linear kernel SVM
Secondly, a polynomial kernel SVM is fitted to the dataset and compared to the previous model which one is the
best. The parametrization of the SVM operator can be seen below.
Figure 19.24. The parameters of polynomial kernel SVM
234
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 19.25. Fitting statistics for polynomial kernel SVM
Figure 19.26. Classification matrix of polynomial kernel SVM
235
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 19.27. Support vectors (partly) of polynomial kernel SVM
The support vector machines with two different kernels (linear and polynomial) can be compared by the usual
statistical and graphical tools.
Figure 19.28. Fitting statistics for SVM's
236
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 19.29. The classification chart of SVM's
Figure 19.30. Cumulative lift curves of SVM's
237
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Figure 19.31. Comparison of cumulative lift curves to the baseline and the optimal one
Figure 19.32. ROC curves of SVM's
238
Created by XMLmind XSL-FO Converter.
Classification Methods 4
Interpretation of the results
The above figures and statistics clearly show that the polynomial kernel support vector machine can improve the
fit of the model against the linear kernel one. The misclassification rate is improved by 2 per cent and the lift
and ROC curves also show a significant improvement. The cumulative lift curve shows a better model at second
to third deciles, while the ROC curve also show an improvement if the specificity is very close to 1.
Video
Workflow
sas_ann_svm_exp3.xml
Keywords
support vector machine (SVM)
supervised learning
classification
Operators
Data Source
Model Comparison
Data Partition
Support Vector Machine
239
Created by XMLmind XSL-FO Converter.
Chapter 20. Classification Methods 5
Ensemble methods
1. Ensemble methods: Combination of classifiers
Description
This experiment presents the ensemble classification method by using the Ensemble operator. By the help of
this operator a better model can be built from separate supervised data mining models. In the experiment, an
ensemble classifier is constructed from a decision tree, a logistic regression and a neural network classifer using
the average voting method. The resulted ensemble model is compared with a polynomial kernel SVM derived
by the SVM operator. For the evaluation the misclassification rate is applied on the Spambase dataset.
Input
Spambase [UCI MLR]
Before fitting the models, in the preprocessing step the dataset is partitionated by the Data Partition operator
according to the rates 60/20/20 for training, validatation and test dataset.
Output
The ensemble classifiers can be evaluated by similar tools as other supervised data mining models: statistics like
number of the incorrectly classified cases and the misclassification rate and graphs like like lift and response
curves.
Figure 20.1. Fitting statistics of the ensemble classifier
Figure 20.2. The classification matrix of the ensemble classifier
240
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Figure 20.3. The cumulative lift curve of the ensemble classifier
241
Created by XMLmind XSL-FO Converter.
Classification Methods 5
The resulted ensemble classifier is compared with a baseline polynomial kernel SVM. The statistics and graphs
of this comparison are summarized below.
Figure 20.4. Misclassification rates of the ensemble classifier and the SVM
Figure 20.5. Classification matrices of the ensemble classifier and the SVM
Figure 20.6. Cumulative lift curves of the ensemble classifier and the SVM
242
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Figure 20.7. Cumulative lift curves of the ensemble classifier, the SVM and the best
theoretical model
Figure 20.8. ROC curves of the ensemble classifier and the SVM
243
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Interpretation of the results
The experiment shows that by combining simple classifiers we can obtain a competitive model against such
supervised model as the polynomial kernel SVM. The classification matrix clearly shows that the ensemble
classification model is better than the SVM, especially at the false positive cases. The cumulative lift curves
slightly favor the combined model, and the ROC curve of the combined model passes over the SVM's a little.
Video
Workflow
sas_ensemble_exp1.xml
Keywords
ensemble method
supervised learning
SVM
misclassification rate
ROC curve
classification
Operators
Data Source
Decision Tree
Ensemble
Model Comparison
Neural Network
Data Partition
Regression
Support Vector Machine
2. Ensemble methods: bagging
244
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Description
This experiment shows the combined method of bagging. In this method, a better fitting model can be built from
supervised data mining models using bootstrap aggregation. Bagging is sampling the original training dataset
obtaining several subsamples by the bootstrap method. On these subsamples supervised models (decision treee
in this experiment) are fitted, respectively, and a new model is obtained by aggregation of the models that are
obtained. In the experiment, the bagging cycle has set 10, i.e., 10 pieces of decision trees are fitted on 10
different subsamples. The results are compared with a simple decision tree, which is fitted to the entire training
dataset. In the bagging method the basic classifier is determined by the Ensemble operator which is straddled
between the Start Groups and End Groups operators. The size of the bagging cycle is set in the Start
Groups operator.
Input
Spambase [UCI MLR]
In the preprocessing step the dataset is partitionated by the Data Partition operator according to the rates
60/20/20 for training, validatation and test dataset.
Output
There are similar tools to evaluate bagging classifiers which are available for other supervised data mining
models: statistics (number of incorrectly classified cases, misclassification rate) and graphs (response and lift
curves). The only additional graph can be seen on the second figure below, where the errors of the 10 classifiers
obtained in the consecutive bagging cycle are plotted.
Figure 20.9. The classification matrix of the bagging classifier
245
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Figure 20.10. The error curves of the bagging classifier
246
Created by XMLmind XSL-FO Converter.
Classification Methods 5
The obtained bagging classifier is compared with a reference decision tree that we fit on the whole training
dataset. The statistical and graphical results obtained are shown below.
Figure 20.11. Misclassification rates of the bagging classifier and the decision tree
Figure 20.12. Classification matrices of the bagging classifier and the decision tree
Figure 20.13. Response curves of the bagging classifier and the decision tree
247
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Figure 20.14. Response curves of the bagging classifier and the decision tree comparing
the baseline and the optimal classifiers
Figure 20.15. ROC curves of the bagging classifier and the decision tree
248
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Interpretation of the results
The experiment shows that a better working model can be obtained by taking a bagging classifier than a simple
decision tree if the models are compared on the first deciles. This is clear considering the classification matrix,
the response and the ROC curve.
Video
Workflow
sas_ensemble_exp2.xml
Keywords
ensemble method
supervised learning
bagging
mi9sclassification rate
ROC curve
classification
Operators
Data Source
Decision Tree
End Groups
Model Comparison
Data Partition
Start Groups
3. Ensemble methods: boosting
249
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Description
The experiment shows the combined method of boosting. In this method, a better fitting model can be built from
supervised data mining models. The method is based on the repeated reweighting of the records and the
classifiers in such a way that the wrongly classified cases are gaining more and more importance and we try to
classify them to the right class. In the boosting method a basic classifier is selected which can be a decision tree,
a logistic regression, or a neural networks etc, of which several copies, given by the boosting cycle, are built up.
In this experiment, this basic classifier is a decision tree. In the experiment, the boosting cycle has set to 20, i.e.
20 pieces of decision trees are fitted on the whole training dataset. The result is compared to a polynomial kernel
support vector machine (SVM), which is recognized as an effective method for binary classification tasks. In the
boosting method the basic classifier is determined by the Ensemble operator which is straddled between the
Start Groups and End Groups operators. The size of the boosting cycle is set in the Start Groups operator.
Input
Spambase [UCI MLR]
In the preprocessing step the dataset is partitionated by the Data Partition operator according to the rates
60/20/20 for training, validatation and test dataset.
Output
There are similar tools to evaluate boosting classifiers which are available for other supervised data mining
models: statistics (number of incorrectly classified cases, misclassification rate) and graphs (response and lift
curves). The only additional graph is the second figure, where the error of the resulting classifiers can be seen
which are 20 decision trees in our case.
Figure 20.16. The classification matrix of the boosting classifier
250
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Figure 20.17. The error curve of the boosting classifier
251
Created by XMLmind XSL-FO Converter.
Classification Methods 5
The obtained boosting classifier is compared with a reference polynomial kernel SVM that we fit on the whole
training dataset. The statistical and graphical results obtained are shown below.
Figure 20.18. Misclassification rates of the boosting classifier and the SVM
Figure 20.19. Classification matrices for the boosting classifier and the SVM
Figure 20.20. Cumulative response curves of the boosting classifier and the SVM
252
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Figure 20.21. Response curves of the boosting classifier and the SVM comparing the
baseline and the optimal classifiers
253
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Figure 20.22. ROC curves of the boosting classifier and the SVM
Interpretation of the results
The experiment shows that a classifier obtained by the boosting method is competitive even comparing with a
polynomial kernel support vector machine classifier in the sense that, although the misclassification rate is
worse it has higher accuracy in the first few deciles. This can be seen clearly on the response and the ROC
curves.
Video
254
Created by XMLmind XSL-FO Converter.
Classification Methods 5
Workflow
sas_ensemble_exp3.xml
Keywords
ensemble method
supervised learning
boosting
ROC curve
classification
Operators
Data Source
Decision Tree
End Groups
Model Comparison
Data Partition
Start Groups
Support Vector Machine
255
Created by XMLmind XSL-FO Converter.
Chapter 21. Association mining
1. Extracting association rules
Description
The process presents, in the case of Extended Bakery dataset, how can association rules be obtained from a
transaction dataset. In transaction datasets, items from the possible ones are emphasized that are part of the
transaction, rather than those that are lacking. In the Enterprise Miner™, such dataset must be defined as
transaction dataset and it must include a ID variable and a Target target variables that must be nominal. The
separate market baskets can be formed by ID variable. The association rule mining (also called market basket
analysis) can be carried out by the Market Basket operator. First, the frequent itemsets are extracted, then, the
significant association rules are discovered based on these itemsets.
Input
Extended Bakery [Extended Bakery]
Output
Applying the Market Basket operator on the dataset of 20000 records the following results are given.
Figure 21.1. List of items
256
Created by XMLmind XSL-FO Converter.
Association mining
Figure 21.2. The association rules as a function of the support and the reliability
Figure 21.3. Graph of lift values
257
Created by XMLmind XSL-FO Converter.
Association mining
Interpretation of the results
The significant association rules can be discovered on the basis of the frequent itemsets. We can select which
criteria we would like to consider in order to find the appropriate rules. The defaults is the confidence level of
the rules, but other criteria can also be applied to filter the discovered association rules. Based on the established
rules deeper conclusions can be drawn from the relationships between the data. To support this inference several
tools can be applied, e.g. the table of the association rules where we may filter the relevant rules by choosing
different evaluation metric like support, confidence or lift value.
Figure 21.4. List of association rules
Video
Workflow
sas_assoc_exp1.xml
258
Created by XMLmind XSL-FO Converter.
Association mining
Keywords
frequent itemset
association rule
transaction data
Operators
Data Source
Market Basket
259
Created by XMLmind XSL-FO Converter.
Chapter 22. Clustering 1
Standard methods
1. K-means method
Description
The process shows the usage of the K-means clustering algorithm and illustrates the importance of the choice of
its various parameters on the Aggregation dataset. This clustering algorithm can be fitted using the Cluster
operator.
Input
Aggregation [SIPU Datasets] [Aggregation]
The dataset consists of 788 two-dimensional vectors, which form 7 separate groups. The task is to discover
these groups which are called clusters. The difficulty of the task is in the alignment of the points, as smaller and
larger clouds of points are present with different distances in space between them. The visualization is done by
the Graph Explore operator.
Figure 22.1. The Aggregation dataset.
260
Created by XMLmind XSL-FO Converter.
Clustering 1
Output
After reading the dataset, we drag and drop the Cluster operator and take the following setting. The user
specify cluster number is chosen and, by the above scatterplot, the number of clusters is defined as 7.
Figure 22.2. The setting of the Cluster operator.
261
Created by XMLmind XSL-FO Converter.
Clustering 1
The result can also be displayed by the Graph Explore operator. One can see that the algorithm found the
upper and right clusters but it has weak performance on the points below.
Figure 22.3. The result of K-means clustering when K=7
262
Created by XMLmind XSL-FO Converter.
Clustering 1
Let's try a different parameter settings where the choice of the initial cluster centers was the MacQueen method
and the minimum value of the distances between the cluster centers minimum is choosen as 9.
Figure 22.4. The setting of the MacQueen clustering
One can see that the result is better using this parameter setting, only for the left set of points below This result
can be improved by significantly sophisticated clustering method. Láthatjuk hogy ezzel a paraméterezéssel az
eredmény pontosabb lett, egyedül a baloldali alsó ponthalmaznál láthatunk még nagyobb hibát. Ezt már csak
jóval haladottabb módszerrel tudnánk korrigálni.
Figure 22.5. The result of the MacQueen clustering
263
Created by XMLmind XSL-FO Converter.
Clustering 1
Finally, let us look at what happens if we take a slightly larger number of clusters to be produced, which is let's
say 8 . In this case, the only changes is that the algorithm finds the two small clusters in the bottom left, but it
cuts the major cluster beside them into three parts and cuts the upper right cluster.
Figure 22.6. The result of the clustering with 8 clusters
Interpretation of the results
Based on the experiment, we can see that the simplest clustering algorithm such as the K-means method is able
to extract the simple relationships. Moreover, if we choose the parameters of the algorithm well then the
264
Created by XMLmind XSL-FO Converter.
Clustering 1
accuracy of the results can be increased. In addition, the Cluster operator provides several visualization
functions that help to evaluate the results.
Figure 22.7. The result display of the
Cluster
operator
After selecting the menu Result the figure above shows the emerging main windows where you can see the
results summary. On the left top at the cluster (segment) plot can be seen in the function of the input attributes.
On the left bottom a pie chart shows the size of the clusters. On the right top the cluster statistics can be read,
while on the right bottom the output list is shown. These windows can be enlarged individually. Among the
many other tools available we want to point out the following two.
Figure 22.8. Scatterplot of the cluster means
The figure above shows the centers of the clusters along with the total average of the attributes. Finally, the
figure below shows the decision tree constructed by the clusters, which can be obtained in such way that the
resulting cluster variable will be a classification target variable and a classification task is solved by fitting of a
decision tree.
265
Created by XMLmind XSL-FO Converter.
Clustering 1
Figure 22.9. The decision tree of the clustering
Video
Workflow
sas_clust_exp1.xml
Keywords
K-means
unsupervised learning
clustering
Operators
266
Created by XMLmind XSL-FO Converter.
Clustering 1
Cluster
Data Source
Graph Explore
2. Agglomerative hierarchical methods
Description
The process shows, using the Maximum Variance (R15) dataset, how agglomerative hierarchical clustering
algorithms work. These clustering algorithms can be run by the Cluster operator.
Input
Maximum Variance (R15) [SIPU Datasets] [Maximum Variance]
The dataset contains 600 two-dimensional vectors, which are concentrated into 15 clusters. The points are
aligned around a center with the coordinates (10,10), in increasing distances from each other as they get further
from the center. This is the difficulty of the task, as the clusters near the center are close to blending into each
other.
Figure 22.10. Scatterplot of the Maximum Variance (R15) dataset
267
Created by XMLmind XSL-FO Converter.
Clustering 1
Output
Firstly, the average linkage hierarchical method is applied. In this case, the distance between the clusters is
calculated as the average of the pairwise distance of cluster elements by the algorithm. The results are shown in
the following figure.
Figure 22.11. The result of the average linkage hierarchical clustering
The goodness of clustering can be measured so that the original grouping Class attribute and the Segment
attribute which contains the cluster membership obtained after clustering are plotted by a spatial bar chart. It can
be seen that, apart from a permutation, there is a one-to-one correspondance between the lines and the columns
except two records.
268
Created by XMLmind XSL-FO Converter.
Clustering 1
Figure 22.12. Evaluating of the clustering by 3D bar chart
An other hierarchical clustering method is the Ward method. Using this, we obtain the following results.
Figure 22.13. The result of Ward clustering
Interpretation of the results
269
Created by XMLmind XSL-FO Converter.
Clustering 1
The process demonstrated that if the number of possible clusters is relatively large then it is worth choosing one
of the automatic clustering procedures. In the SAS® Enterprise Miner™, the hierarchical clustering is available
for this purpose in several different ways. The experiment also shows that the choice of the agglomerative
method does not always affect the resulting clusters. The SAS proposes the cluster number by investigating the
CCC graph, see figure below.
Figure 22.14. CCC plot of automatic clustering
In addition, a schematic display on the location of the clusters, the so-called proximity diagram is also obtained
which is clearly similar to the previously obtained scatterplot on the clusters.
Figure 22.15. Proximity graph of the automatic clustering
270
Created by XMLmind XSL-FO Converter.
Clustering 1
Video
Workflow
sas_clust_exp2.xml
Keywords
hierarchical methods
average linkage
Ward method
CCC graph
clustering
Operators
Cluster
Data Source
Graph Explore
3. Comparison of clustering methods
271
Created by XMLmind XSL-FO Converter.
Clustering 1
Description
The experiment presents the difference between the automatic clustering and the clustering where the number of
clusters is specified by the user on the Maximum Variance (D31) dataset. In the experiment the Cluster
operator is used.
Input
Maximum Variance (D31)
The dataset consists of 3100 two-dimensional vectors, which are grouped around 31 clusters.
Figure 22.16. The Maximum Variance (D31) dataset
Output
272
Created by XMLmind XSL-FO Converter.
Clustering 1
Firstly, an automatic clustering is performed where the Class attribute is ignored. The algorithm finds 31
clusters which aggres to the original number of clusters. The resulting clusters are shown in the following figure.
Figure 22.17. The result of automatic clustering
The correctness of the resulted cluster number is clearly shown by the CCC plot.
Figure 22.18. The CCC plot of automatic clustering
The scematic arrangement of the clusters is shown by the proximity graph below.
Figure 22.19. Az automatikus klaszterezés proximitási ábrája
273
Created by XMLmind XSL-FO Converter.
Clustering 1
You can try a cluster model based on the CCC chart, which has 9 clusters. This can be done by the Ward's
version of the K-means algorithm. As a result, the scatterplot and the proximity graph of the clusters are shown
in the following two figures.
Figure 22.20. The result of K-means clustering
Figure 22.21. The proximity graph of K-means clustering
274
Created by XMLmind XSL-FO Converter.
Clustering 1
Then, by the so-called segment profiling, the resulted clusters can be investigated from the point of view that
how the input variables determine the clusters.
Figure 22.22. The profile of the segments (clusters)
275
Created by XMLmind XSL-FO Converter.
Clustering 1
Interpretation of the results
The experiment shows that the automatic clustering is able to find the correct number of clusters in case of a
relatively large number of closely spaced, but spherical groups. If this we put down it too big so this number can
be reduced to a reasonable size by analyzing of the CCC graph and searching a suitable breakpoint.
Video
Workflow
sas_clust_exp3.xml
Keywords
automatic clustering
K-means
cluster profiling
CCC graphs
clustering
276
Created by XMLmind XSL-FO Converter.
Clustering 1
Operators
Cluster
Data Source
Graph Explore
MultiPlot
Segment Profile
277
Created by XMLmind XSL-FO Converter.
Chapter 23. Clustering 2
Advanced methods
1. Clustering attributes before fitting SVM
Description
The process demonstrates how to cluster the attributes by using the Variable Clustering operator when there
are a number of attributes in the dataset. The process uses the Spambase dataset. After clustering the attributes,
further supervised data mining methods can be applied, e.g., we may classify the e-mails into spam and nonspam classes.
Input
Spambase [UCI MLR]
The dataset contains 4601 records and 58 attributes. The records are classified into 2 groups by the Class
variable, which identifies the spam e-mails, i.e., its value equals to 1 if the record is spam and 0 otherwise. The
challenge in the dataset is that there are relatively large number of attributes which slow down the training
process. The experiment points out that a competitive model can be obtained after a suitable clustering of the
attributes to such models which are fitted on the whole dataset.
Output
During the attribute clustering the columns of the dataset are clustered by a hierarchical method to reduce the
dimension of the dataset. The most important parameter of the Variable Clustering operator is the Maximum
Cluster, which can be used to adjust the maximal number of clusters. Similar parameters are the maximal
number of eigenvalues and the explained variance. You can also choose between the correlation and the
covariance matrix in the analysis. One of the most important results is the dendrogram which visualizes the
process of the hierarchical clustering.
Figure 23.1. The dendrogram of attribute clustering
278
Created by XMLmind XSL-FO Converter.
Clustering 2
The relationship between the original attributes and the obtained clusters is depicted on the following graph.
Figure 23.2. The graph of clusters and attributes
The list of cluster membership, i.e., the set of attributes belonging to clusters, respectively, can be seen on the
following figure.
Figure 23.3. The cluster membership
279
Created by XMLmind XSL-FO Converter.
Clustering 2
To create clusters the correlation (covariance) between the original attributes plays the most important role.
Those attributes will be in one cluster which have high correlation with each other. This displays in the
following figure.
Figure 23.4. The correlation plot of the attributes
280
Created by XMLmind XSL-FO Converter.
Clustering 2
It can also be investigated how high is the correlation between each variable and the new cluster variables
obtained. The following figure shows the correlation bar chart of the variable representing the special character
dollar.
Figure 23.5. The correlation between clusters and an attribute
281
Created by XMLmind XSL-FO Converter.
Clustering 2
After the attribute clustering, SVM model was fitted to the Class binary variable by using the obtained 19 new
cluster attributes. Then, the results obtained in this way were compared with a similar model fitted to the
original 58 attributes directly. The results below show that the models obtained have similar performance. The
classification bar charts show a similar classification matrix.
Figure 23.6. Classification charts of SVM models
The response curve behaves better in some places on the clustered attributes than on the original ones.
Figure 23.7. The response curve of SVM models
If the cumulative lift functions are compared considering the baseline and the best lift functions, similar
behavior can be seen.
Figure 23.8. Az SVM modellek kumulatív lift függvényei
282
Created by XMLmind XSL-FO Converter.
Clustering 2
Finally, the ROC curves are very similar to each other.
Figure 23.9. The ROC curves of SVM models
Interpretation of the results
If there are very much input attributes in teaching a supervised data mining model, which makes teaching so
very slow, then it worth reducing the dimension by clustering the input attributes. The explanatory power of the
resulting model is usually not much worse than the one fitted to the original attributes.
Video
Workflow
283
Created by XMLmind XSL-FO Converter.
Clustering 2
sas_clust2_exp1.xml
Keywords
attibute clustering
dendrogram
hierarchical methods
clustering
ROC curve
SVM
Operators
Data Source
Model Comparison
Support Vector Machine
2. Self-organizing maps (SOM) and vector
quantization (VQ)
Description
The process presents the Kohonen's vector quantization (VQ) and the self-organizing map (SOM) algorithms
using the Maximum Variance (R15) dataset. These algorithms can be fitted by the SOM/Kohonen operator.
Input
Maximum Variance (R15) [SIPU Datasets] [Maximum Variance]
The dataset consists of 600 two-dimensional records, which are grouped into 15 groups. The points are located
around the point with coordinates (10, 10) and they are farther from each other as they are far from the center.
The difficulty of the task is that the groups which are around the center almost fuse. In the figure below these
points are depicted by coloring the different groups.
Figure 23.10. The scatterplot of the Maximum Variance (R15) dataset
284
Created by XMLmind XSL-FO Converter.
Clustering 2
Output
First, the method of Kohonen's vector quantization is used. By this method we got 10 clusters. The results can
be seen on the figure below.
Figure 23.11. The result of Kohonen's vector quantization
The size of clusters can be depicted by a simple pie chart.
Figure 23.12. The pie chart of cluster size
285
Created by XMLmind XSL-FO Converter.
Clustering 2
A table displays all the statistics which characterize the clusters, among others the frequency of clusters, the
standard deviation of clusters, the maximum distance from the center of clusters, and the number of the adjacent
cluster with the distance between them.
Figure 23.13. Statistics of clusters
Then, the method of batch SOM algorithm is applied for the same dataset. In this case, the numbers of row and
column segments should be defined where 6 was chosen. The results are shown in the following two figures.
The first one is the schematic graph of the SOM/Kohonen operator on the resulting net where the coloring
shows the frequency of each cell.
Figure 23.14. Graphical representation of the SOM
286
Created by XMLmind XSL-FO Converter.
Clustering 2
The second figure is a scatterplot which displays the resulting clusters in the coordinate system of original input
attributes.
Figure 23.15. Scatterplot of the result of SOM
Interpretation of the results
The experiment shows how to use two unsupervised data mining techniques such as vector quantization and
self-organizing maps. The two methods are particularly effective for examining 2-dimensional data. However,
being important prototype methods, they can greatly simplify the further analysis in higher dimension too.
Video
Workflow
sas_clust2_exp2.xml
287
Created by XMLmind XSL-FO Converter.
Clustering 2
Keywords
vector quantization (VQ)
self-organizing map (SOM)
clustering
Operators
Data Source
Graph Explore
Self-organizing Map
288
Created by XMLmind XSL-FO Converter.
Chapter 24. Regression for
continuous target
1. Logistic regression
Description
The process shows, using the Spambase dataset, how can a regression model be fitted to a dataset which has
binary target. The conventional linear regression are not suitable for this task even though the Regression
operator offers this option. Instead, we must use the logistic regression method which is the default option of
this operator. We can choose between the following link functions: logit, which takes the name of the procedure,
probit and complementary logit. There is no significant difference among these link functions. The Enterprise
Miner™ gives an other operator for fitting regression. By the Dmine Rgeression operator forward stepwise
regression can be fitted. In each step, an input variable is selected that contributes most significantly to the
variability of the target.
Input
Spambase [UCI MLR]
Output
After fitting the logistic regression, standard statistics and graphs are obtained similarly to the binary
classification tasks. Here, only the confusion matrix is shown, the rest of comparison tools is left at the and of
this experiment.
Figure 24.1. Classification matrix of the logistic regression
289
Created by XMLmind XSL-FO Converter.
Regression for continuous target
In addition to the usual tools, the regression operators, using the effect plot, also show the importance of the
input variables in the regression model which were built during the process.
Figure 24.2. Effects plot of the logistic regression
In addition to the traditional regression analysis Enterprise Miner ™ yields another operator to fit forward
stepwise regression. This is the Dmine Rgeression operator. The results can be seen in the figures below.
Figure 24.3. Classification matrix of the stepwise logistic regression
290
Created by XMLmind XSL-FO Converter.
Regression for continuous target
Figure 24.4. Effects plot of the stepwise logistic regression
The two regressions can be compared by the usual way with the Model Comparison operator. The results of
this comparison are presented in the following figures.
Figure 24.5. Fitting statistics for logistic regression models
291
Created by XMLmind XSL-FO Converter.
Regression for continuous target
Figure 24.6. Classification charts of the logistic regression models
Figure 24.7. Cumulativ lift curve of the logistic regression models
292
Created by XMLmind XSL-FO Converter.
Regression for continuous target
Figure 24.8. ROC curves of the logistic regression models
Interpretation of the results
The fit statistics and ROC curves clearly show on the test set that the logistic regression model is better than the
stepwise logistic regression model.
Video
Workflow
sas_regr_exp1.xml
293
Created by XMLmind XSL-FO Converter.
Regression for continuous target
Keywords
classification
binary target
logistic regression
Operators
Data Source
Dmine Regression
Model Comparison
Data Partition
Regression
2. Prediction of discrete target by regression models
Description
The process shows, using the Wine dataset, that how can we fit a regression model to a dataset containing
discrete but non-binary target variable. Moreover, how can we performe a classification task using the parameter
estimates obtained from the model. The type of the fitted regression model depends on the measurement scale of
the discrete target variable. If the target is nominal then the Regression operator fits binary logistic models
separately, so that a selected event of the target is compared to the class of the other values of the discrete target
variable. On the other hand, if the target is ordinal then a common logistic regression model are fitted, wherein
only the constant parameters differ, but the parameters of the input variables are shared. (As opposed to the
nominal case where these coefficients are different.)
Input
Wine [UCI MLR]
Output
A number of models can be choosen to fit a regression model, e.g., linear or logistic regression. Among them,
the logistic regression used in the process. It is not needed to set in, the software recognizes the right type of the
regression using the metadata on the target. Of course, it is possible to override this option and to enforce linear
regression, but this does not make sense because in this case the Model Comparison operator can not be used to
compare this model to other discrete supervised models. The fit of the model can be tested by the well-known
statistics and charts.
Figure 24.9. Classification matrix of the logistic regression
294
Created by XMLmind XSL-FO Converter.
Regression for continuous target
The classification chart shows that the fitted model is perfect on the training dataset and has small error on the
validation dataset.
Figure 24.10. The classification chart of the logistic regression
295
Created by XMLmind XSL-FO Converter.
Regression for continuous target
Besides the standard goodness-of-fit tests the Regression operator presents a bar graph showing the
importance of the input variables in the regression models. The higher the coefficient of an input variable, the
more is its explanation power with respect to the target variable. In case of ordinal target, there is only one,
while, in case of the nominal target, the number of different values of the target variable minus 1 bar graphs are
created. Since the Class variable has three values there are two such bar graphs below.
Figure 24.11. The effects plot of the logistic regression
Interpretation of the results
Using the regression model created on the training set for the validation dataset the above results show that a
model can be built with relatively high accuracy for multiple discrete target variable value by the Regression
operator. It is noted that for the problem discussed here the Dmine Regression operator can not be applied.
Video
Workflow
sas_regr_exp2.xml
296
Created by XMLmind XSL-FO Converter.
Regression for continuous target
Keywords
classification
nominal and ordinal target
logistic regression
Operators
Data Source
Data Partition
Regression
3. Supervised models for continuous target
Description
The process presents, using the dataset Concrete Compressive Strength, that how can we fit supervised data
mining models for datasets with continuous target. We can use for this task the Regression operator, the
Decision Tree operator, and the Neural Network operator, as well. By Regression operator linear
regression model can be fitted. Decision trees can be given by the Decision Tree operator. Finally, the result
of a Neural Network operator is a neural network that minimizes a predefined error function on the validation
dataset. In each case, continuous target variable with continuous level metadata must be selected.
Input
Concrete Compressive Strength [UCI MLR] [Concrete]
In the preprocessing step, the dataset is partitioned for training, validation and test datasets with percent
60/20/20, respectively.
Output
A comparison of the resulting models are significantly different from the tools applied for discrete or binary
target. Among the statistical indicators, we can use different information criterion (e.g. AIC, SBC) or the mean
square error and the square root of the it. In this case there aro no such graphical tools as lift curve and
classification chart. Instead, we can use the graphs showing averages of forecasts.
Figure 24.12. Statistics of the fitted models on the test dataset
297
Created by XMLmind XSL-FO Converter.
Regression for continuous target
The figure below shows that the neural network and the linear regression behave fairly similarly.
Figure 24.13. Comparison of the fitted models by means of predictions
Figure 24.14. The observed and predicted means plot
298
Created by XMLmind XSL-FO Converter.
Regression for continuous target
The following curves is good if we are closer to the diagonal straight line.
Figure 24.15. The model scores
299
Created by XMLmind XSL-FO Converter.
Regression for continuous target
In addition to the comparisons, one by one, we can examine the individual models. Below you can see the
constructed decision tree, which is created by using F-statistics.
Figure 24.16. The decision tree for continuous target
300
Created by XMLmind XSL-FO Converter.
Regression for continuous target
For the neural network model the weights of neurons can be visualized.
Figure 24.17. The weights of neural network after traning
301
Created by XMLmind XSL-FO Converter.
Regression for continuous target
Interpretation of the results
Both the statistics and graphics show that the neural network model fits the best.
Video
Workflow
sas_regr_exp3.xml
Keywords
supervised learning
continuous target
decision tree
linear regression
neural network
Operators
302
Created by XMLmind XSL-FO Converter.
Regression for continuous target
Data Source
Decision Tree
Model Comparison
Neural Network
Data Partition
Regression
303
Created by XMLmind XSL-FO Converter.
Chapter 25. Anomaly detection
1. Detecting outliers
Description
The process presents, using the Concrete Compressive Strength dataset, that how can we filter outliers
considering various criteria by the Filter operator. The applied criteria involve the mean square deviation from
the mean or the mean absolute deviation. Other possibility is the modal center. In the experiment, the records
which differ from the mean by twice standard deviation are filetered.
Input
Concrete Compressive Strength [UCI MLR] [Concrete]
Output
As shown, the above setting can filter out a significant number of outliers.
Figure 25.1. Statistics before and after filtering outliers
304
Created by XMLmind XSL-FO Converter.
Anomaly detection
Figure 25.2. The predicted mean based on the two decision trees
Figure 25.3. The tree map of the best model
305
Created by XMLmind XSL-FO Converter.
Anomaly detection
Interpretation of the results
The following comparison shows that after the filtering the error of the fitted decision tree is significantly
smaller than the error of the decision tree fitted on the full dataset. Thus, in suitable cases, the removal of
outliers is able to improve the efficiency of supervised models.
Figure 25.4. Comparison of the two fitted decision trees
Video
Workflow
sas_anomaly_exp1.xml
Keywords
outliers
preprocessing
data cleaning
Operators
Data Source
Decision Tree
Filter
Graph Explore
Model Comparison
306
Created by XMLmind XSL-FO Converter.
Bibliography
Data Sets
[Aggregation] Aristides Gionis. Heikki Mannila. Panayiotis Tsaparas. “Clustering Aggregation”. 4:1–4:30.
ACM Transactions on Knowledge Discovery from Data (TKDD). 1. 1. 2007.
[Compound] C. T. Zahn. “Graph-Theoretical Methods for Detecting and Describing Gestalt Clusters”. 68–86.
IEEE Transactions on Computers. 20. 1. 1971.
[CoIL Challenge 2000] Peter van der Putten. Maarten van Someren. CoIL Challenge 2000: The Insurance
Company Case . 2000. Sentient Machine Research. Amsterdam. Also a Leiden Institute of Advanced
Computer Science. Technical Report 2000-09.
[Concrete] I-Cheng Yeh. “Modeling of strength of high performance concrete using artificial neural networks”.
1797–1808. Cement and Concrete Research. 28. 12. 1998.
[Detrano et al.] R. Detrano. A. Jánosi. W. Steinbrunn. M. Pfisterer. J. Schmid. S. Sandhu. K. Guppy. S. Lee. V.
Froelicher. “International application of a new probability algorithm for the diagnosis of coronary
artery disease”. 304–310. American Journal of Cardiology. 64. 5. 1989.
[Extended Bakery] Alex Dekhtyar. Jacob Verburg. Extended Bakery Dataset . 2009.
[Flame] Limin Fu. Enzo Medico. “ FLAME, a novel fuzzy clustering method for the analysis of DNA
microarray data ”. BMC Bioinformatics. 8. 3. 2007.
[Jain] Anil K. Jain. Martin H.C. Law. “Data Clustering: A User's Dilemma”. 1–10. Lecture Notes in Computer
Science Volume. 3776. Springer. 2005.
[Maximum Variance] C. J. Veenman. M. J. T. Reinders. E. Backer. “A Maximum Variance Cluster Algorithm”.
1273–1280. IEEE Transactions on Pattern Analysis and Machine Intelligence. 24. 9. 2002.
[SIPU Datasets] Clustering datasets . Speech and Image Processing Unit, School of Computing, University of
Eastern Finland.
[StatLib] Mike Meyer. Pantelis Vlachos. StatLib—Datasets Archive . 2005.
[Titanic] Robert J. MacG. Dawson. “ The “Unusual Episode” Data Revisited ”. Journal of Statistics Education.
3. 3. 1995.
[Two Spirals] Kevin J. Lang. Michael J. Witbrock. “Learning to Tell Two Spirals Apart”. 52–59. David
Touretzky. Geoffrey Hinton. Terrence Sejnowski. Proceedings of the 1988 Connectionist Models
Summer School. Morgan Kaufmann. 1988.
[UCI MLR] K. Bache. M. Lichman. UCI Machine Learning Repository . University of California, Irvine,
School of Information and Computer Sciences. 2013.
Other
[DMBOOK] Pang-Ning Tan. Michael Steinbach. Vipin Kumar.
Wesley. 2005.
Introduction to Data Mining . Addison-
[LIBSVM] Chih-Chung Chang. Chih-Jen Lin. “LIBSVM: A Library for Support Vector Machines”. 27:1–
27:27. ACM Transactions on Intelligent Systems and Technology. 2. 3. 2011. Software available at
http://www.csie.ntu.edu.tw/~cjlin/libsvm/.
[Neural Network FAQ] Warren S. Sarle. Neural Network FAQ, part 3 of 7: Generalization . 1997. Periodic
posting to the Usenet newsgroup comp.ai.neural-nets.
307
Created by XMLmind XSL-FO Converter.
Bibliography
[RapidMiner Manual] RapidMiner 5.0 Manual . Rapid-I GmbH. 2010.
[SAS Enterprise Miner] Getting Started with SAS®Enterprise Miner™ . SAS Institute Inc.. 2011.
[SAS Enterprise Miner Ref] SAS® Enterprise Miner™: Reference Help . SAS Institute Inc.. 2011.
308
Created by XMLmind XSL-FO Converter.