Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Data Mining with Decision Trees and Decision Rules C. Apte and S.M. Weiss Future Generation Computer Systems November 1997 Data Mining with Decision Trees and Decision Rules Chidanand Apte Sholom Weiss T.J. Watson Research Center IBM Research Division Yorktown Heights, NY 10598 Department of Computer Science Rutgers University New Brunswick, NJ 08903 Abstract acceptably accurate learning method, it is possible to develop predictive applications, i.e., using the solution to predict the expected value of the eld of interest, given all the others. For example, a credit company may use its customer data base to model delinquency. Given a new application prole, this solution may be used to predict the likelihood that the applicant may default. Classication and regression are critical types of prediction problems. When the goal of prediction is discrete valued, a classication solution is developed. When the goal is numerical and continuous, a regression solution is developed. This type of data analysis has been an active area of research in many scientic areas for quite some time. These domains, ranging from medicine and geology to astronomy and physics, have relied upon the investigation of data gathered from experiments and observations to formulate predictive models. These solutions are essentially approximation functions for describing the behavior of a response or objective variable of interest in terms of independent or input variables or features. Modeling methods have been developed drawing upon techniques from statistics, pattern recognition, and machine learning [18, 21, 27]. A classical example of a predictive modeling application in medicine is the gathering of extensive observations of patients with and without diagnosis of a particular disease. The modeling task then attempts to formulate a model that describes the likelihood of a patient having the disease as a function of all the independent observations about the patient. Once an accurate predictive model is formulated, it serves as an additional piece of diagnostic machinery that may be used for predicting the likelihood of a disease given a patient's data. Until recently, these techniques had been restricted in their applications, given the volume of data and computational resource that was required for the modeling. However, with the increasing availability of high This paper describes the use of decision tree and rule induction in data mining applications. Of methods for classication and regression that have been developed in the elds of pattern recognition, statistics, and machine learning, these are of particular interest for data mining since they utilize symbolic and interpretable representations. Symbolic solutions can provide a high degree of insight into the decision boundaries that exist in the data, and the logic underlying them. This aspect makes these predictive mining techniques particularly attractive in commercial and industrial data mining applications. We present here a synopsis of some major state-of-the-art tree and rule mining methodologies, as well as some recent advances. Keywords Decision Tree, Rule Induction, Data Mining 1 Introduction The use of computer technology in decision support is now widespread and pervasive across a wide range of business and industry. This has resulted in the capture and availability of data in immense volume and proportion. There are many examples that can be cited. Point of sale data in retail, policy and claim data in insurance, medical history data in health care, nancial data in banking and securities, are some instances of the types of data that is being collected. The data are typically a collection of records, where each individual record may correspond to a transaction or a customer, and the elds in the record correspond to attributes. Very often, these elds are of mixed type, with some being numerical (continuous valued, e.g. age) and some symbolic (discrete valued, e.g. color). Given this layout of data, one may model a eld in terms of all the other elds in the data. With an 1 volume data in business and industry, and the signicant drop in computational cost, many of these techniques are beginning to be applied in commercial applications, with many demonstrated successes. There are many issues related to the problem of formulating a predictively accurate model. These have mainly to with the nature of the data and the representation language for the solution. Data characteristics generally dictate the complexity of the mining task. The data may be noisy, incomplete, or incorrect. The representation language for the solution will usually limit the scope of the functions that can be formulated. Classication and regression learning has been evolving over a considerable period of time, with contributions coming from statistics, pattern recognition, and more recently, the eld of machine learning. One of the earliest methods developed for classication modeling was the technique of linear discriminants [14]. An early technique that came into existence for regression was linear regression [23]. Each has its own limitations. Since then, a slew of techniques and methods have been developed, including k-nearestneighbor, decision tree, rule induction, neural networks, etc. [27]. For the remaining part of this paper, the problem of classication modeling will be examined from a decision tree and rule induction perspective. Tree and rule based regression modeling will then be briey introduced, and nally the conclusion will discuss general issues and directions for symbolic solution mining. Figure 1: Peg Data 2 illustrates a decision tree that corresponds to the partitions shown in Figure 1. At the top level of the tree, there is the root node, at which the classication process begins. The test at the root node tests all example instances for Length 0:75. Examples that satisfy this test are passed down the left (TRUE) arc to a leaf node, indicating that all examples belong to a single class (SQUARE) and no more tests are needed. The right (FALSE) arc from the root node receives all examples that fail the test at the root node. These examples are not yet purely from one class, so further testing is required at this intermediate node. The test at this node is for Diameter 3:00. Examples that satisfy this test are all in one class (STAR) and those that don't are also all in one class (DIAMOND), and so both arcs from this node lead to leaf nodes, and the decision tree solution is complete. Note that this example illustrates a binary tree, where each intermediate node can split into at most two sub-trees. Decision trees may be non-binary also, where each node may split into more than two sub-trees, by performing tests that result in more than two outcomes (.e.g. subset membership or interval membership tests). Closely related to decision tree solutions are rule based solutions. A rule may be constructed by forming a conjunct of every test that occurs on a path between the root node and a leaf node of a tree. The collection of all such rules obtained by traversing every unique path from root node to leaf node is a corresponding rule based solution for classication. For example, for the peg data that was used to illustrate the decision 2 Symbolic methods for classication modeling The problem of classication modeling will be examined here through a simple, hypothetical example. Assume that length and diameter data is available for a variety of pegs that are being manufactured on an assembly line. Pegs are either square, star, or diamond shaped. A classication solution that characterizes the peg variety as a function of the length and diameter can be useful in understanding how these varieties dier, for designing visual inspection systems and automated sorting machinery. Figure 1 illustrates this data. The gure also shows the two axis parallel lines, one at Length = 0:75, and the second at Diameter = 3:00, that seem to completely partition the three peg varieties into three dierent sub-areas. Decision tree solution methods provide automated techniques for discovering these types of axis parallel partitions, in their maximally general forms. Figure 2 non-mutually exclusive decision rules. These rules essentially correspond to decision regions that overlap each other in the data space. Once a decision tree or decision rule solution is generated from data, it can be used for estimating or predicting the response or class variable for a new case. The application of a decision tree to a data example is a straightforward top-down decision process, controlled by evaluating the tests and taking the appropriate branch, beginning at the root, and terminating when a leaf node is reached. The process of applying decision rules to data examples is determined by the style in which the rules were generated. A rule generation algorithm may induce ordered rule sets, i.e., induced by ordering all the classes which are present, and then using a xed sequence, such as the smallest to the largest class, with rules for every class being discovered under the assumption that the only classes that need to be discriminated between are the ones that are remaining in the sequence. With this rule set, the rules have to be applied to a new data example in exactly the same sequence as they were generated. A rule generation algorithm may also induce un-ordered rule sets, in which rules for every class are generated under the assumption that all other classes that are present in the data need to be discriminated against during the induction process. With this type of a rule set, application to a new data example can be order independent and more exible in creating dierent rule application strategies. Decision tree and decision rule solutions oer a level of interpretability that is unique to symbolic models. The solutions may be directly inspected to understand the decision surfaces that exist in the data. This particular aspect, which makes these solutions easy to digest even for a non-technical end-user, makes these techniques very appealing in decision support related data mining activities, where insight and explanations are of critical importance. What makes this approach technically viable is the fact that most modern symbolic modeling methodologies succeed in formulating solutions that are also competitive in predictive accuracy when compared to more non-intuitive or quantitative techniques, such as neural networks. This is an important reason for the increased attention to and use of decision rule modeling techniques that generate rules directly from data [9, 1, 2, 8]. Classication modeling algorithms are designed with several objectives. Perhaps the most well known criteria by which these algorithms are evaluated is accuracy, speed, and interpretability. Solutions derived using dierent approaches can thus be compared in terms of their predictive accuracy on unseen data, on Length .le. 0.75 True False Square Diameter .le. 3.00 True Star False Diamond Figure 2: Classifying Pegs with a Decision Tree tree, an equivalent rule solution is: If (Length <= 0.75) Then Square If (Not (Length <= 0.75)) & (Diameter <= 3.0) Then Star If (Not (Length <= 0.75)) & (Not (Diameter <= 3.0)) Then Diamond Some rule induction programs are add-ons to decision tree solutions, whereby a tree is rst generated, and then translated into a set of rules. However, techniques that directly generate rules from data are also available, which overcome some of the drawbacks of decision tree modeling. Often, in more complex data sets, disjuncts of rules form the description for a class, and hence the rule based solutions are more generally identied by the term DNF (Disjunctive Normal Form), or decision rules. Rules that are created by translating a decision tree into DNF expressions are typically mutually exclusive in nature, since a decision tree essentially partitions a data space into distinct disjoint regions via axis parallel surfaces created by its top-down sequence of decisions. For certain data spaces, this nature of partitioning may not always be capable of producing compact solutions. For example, decision trees cannot easily model simple exclusive-or functions. On the other hand, if algorithms are employed that directly generate DNF expressions from data, it is possible to create rules that capture such decision surfaces, via 3 the computational cost involved in generating the solution, and the level of understanding and insight that is provided by the solution. Both decision tree and decision rule modeling systems score high on interpretability. Accuracy and speed vary from algorithm to algorithm, and in most instances these two issues are coupled, i.e., improving predictive accuracy tends to require increased computational eort. Learning algorithms take a variety of factors into account while computing the classication solution. There is inherent noise in real-world data that needs to be handled. The prior distributions of classes in the training set may aect the solution generation. There may be explicit penalties associated with misclassication that need factoring in. These and related issues will be described by going through brief descriptions of some actual decision tree and decision rule modeling algorithms in the following sections. and test partitions. The training partition is used to generate an over-tted model, while the test partition is used to generalize this model to the best possible derivative. An averaging process across the many different train-test combinations (hence the term crossvalidation) is used to select the nal RSbest from the many candidates that are available. When the data set that is available is large, a single train-test partition is sucient for evaluation and to select RSbest using the pruning approach. 3 Decision Tree Modeling Decision trees are generated from training data in a top-down, general-to-specic direction. The initial state of a decision tree is the root node that is assigned all the examples from the training set. If it is the case that all examples belong to the same class, then no further decisions need to be made to partition the examples, and the solution is complete. If examples at this node belong to two or more classes, then a test is made at the node that will result in a split. The process is recursively repeated for each of the new intermediate nodes until a completely discriminating tree is obtained. A decision tree at this stage is potentially an over-tted solution, i.e., it may have components that are too specic to noise and outliers that may be present in the training data. To relax this over-tting, most decision tree methods go through a second phase called pruning that tries to generalize the tree by eliminating sub-trees that seem too specic. Error estimation techniques play a major role in tree pruning. Most modern decision tree modeling algorithms are a combination of a specic type of a splitting criterion for growing a full tree, and a specic type of a pruning criterion for pruning tree. 2.1 Error estimation and evaluation criteria Estimating the true accuracy of a decision tree or rule model is one of the most important aspects of the modeling process. A solution generated from a set of training examples will almost always be highly accurate on the same data, but far less accurate on new data. A sample of cases contains noise and vary from other samples, leading a learning method astray in its predictions. To handle this shortcoming, most modeling techniques employ a two-fold strategy in the model generation process, where the rst step involves the generation of the model from the training data, and the second step involves testing the proposed solution on independent cases, sometimes pruning it to compensate for the over-tting of the rst step. The essential problem to be solved in the pruning step may be specied as follows: given a set of sample examples, S, where each example is composed of observed features and the class labels, the problem is to nd the best model RSbest such that the error rate on new examples, Errtrue(RSbest), is minimum. Given an over-tted model RS on a set of examples S, a derivative of RS needs to be determined that satises the above criteria. Several pruning techniques have been devised that t the above paradigm, and will be explained in more detail in the following sections. Broadly speaking, these techniques usually either employ the crossvalidation approach or the train-and-test approach. Cross-validation is usually preferred when the modeling is being done with small samples, so one repeatedly breaks up the data into dierent combinations of train 3.1 Growing a Full Tree CART [4] is a binary decision tree modeling algorithm that has been in extensive use. The evaluation function used for splitting in CART is the GINI index. For a given current P node t, this index is dened as Gini(t) = 1 ; i p2i where pi is the probability of class i in t. For each candidate split, the impurity (as dened by the GINI index) of all the sub-partitions is summed and the split that causes the maximumreduction in impurity is chosen. For the candidate splits, CART considers all possible splits in the sequence of values for continuous valued attributes ((n ; 1) splits for n values) and all possible subset splits for categorical attributes ((2n;1 ; 1) splits for n distinct values) 4 if n is small, and equivalence splits for categorical attributes (n splits for n distinct values) if n is large. At each node, CART determines the best split for each attribute and then selects the winner from this short list, utilizing the GINI index. C4.5 [20] is another popular decision tree modeling system, a variant and extension of an earlier well known decision tree modeling system, ID3. ID3 utilizes entropy criteria for splitting nodes. Given a criterion used is Entropy(t) = Pnodei ;pt,i logthepisplitting , where pi is the probability of class i within node t. An attribute and split are selected that minimize entropy. Splitting a node produces two or more direct descendants. Each child has a measure of entropy. The sum of each child's entropy is weighted by its percentage of the parent's cases in computing the nal weighted entropy used to decide the best split. In C4.5, given a node t, the splitting criterion used is the GainRatio(t) = gain(t)=SplitInformation(t). This ratio expresses the proportion of information generated by a split that is helpful for developing the classication, and may be thought of as a normalized information gain or entropy measure for the test. A test is selected that maximizes this ratio, as long as the numerator (the information gain) is larger than the average gain across all tests. The numerator in this ratio is the standard information entropy difference achieved at node t, expressed P as gain(t) = k Ci info(T);info (T), where info(T ) = ; t i=1 CT , and P s Ti infot (T ) = i=1 T info(Ti ) Bayes' classication rule assigns an example to the class with the highest conditional probability. This rule is known theoretically to be the optimum, i.e., it minimizes the total classication error. Formally, if there are C classes, then Bayes' rule is to assign an example to class i where P (Cijx) > P (Gj jx)8j 6= i. The success of this rule is in the underlying fact that all information that can be had about class membership is contained in the set of conditional probabilities. In practice, it is not possible to compute conditional probabilities for high dimensional data sets, for an enormous number of examples will be required to correctly assess conditional probabilities of the type P(C jx). Limited practical derivations of Bayes' rule exist, which include linear discrimination, and more recently, Bayes' tree [5]. Bayes' tree requires knowledge of prior class probabilities (empirically derivable from class proportions in the training data). Associated with a tree is a posterior probability of correct classication. The decision to grow a tree from a node is based upon increasing the posterior probability of the resulting tree. Of all the candidate splits that can be made, the one chosen is the one that causes the maximum increase to the posterior probability. SLIQ [16], a recent decision tree building system, pays attention to scalability issues, utilizing data structures and processing methods that allow applications to very large data sets are not required to be memory resident. In the tree building phase, SLIQ utilizes the Gini index, as in the CART system. For more discussions on splitting criteria, see [15] in this issue. 3.2 Pruning a Full Tree CART's pruning mechanism is an eort to get to the right sized tree that minimizes the true misclassication error estimate. A fully grown decision tree will have an apparent error rate of zero or close to zero on the training data from which the tree was built. However, its true error rate, measured by evaluating the misclassications when the tree is applied to a test data set, may be much higher. The goal of the pruning process is to nd that sub-tree that produces the least true error, taking into account the size (complexity) of the tree. Utilizing a formulation for the cost complexity of a tree, which is a function of the misclassication of the tree on the training data and the size (e.g. total number of leaves), one can derive a sequence of trees of decreasing cost complexity, starting from the fully grown tree. This sequence is recursively created by picking the last tree in the sequence (initially, the full tree), examining each of its non-leaf sub-trees, and picking the one with the least cost-complexity metric, and making that the next sub-tree in the sequence. The process stops when the nal sub-tree is just the root node. Once this sequence of decreasing cost-complexity sub-trees is produced, their individual true error rates can be determined by applying each sub-tree to a holdout data set. Typically, it is observed that initially, as the cost-complexity decreases, so does the true error rate, until one reaches a minima. Beyond that, as the cost-complexity decreases, the true error starts increasing again. Obviously, one chooses the sub-tree corresponding to the minimum true error rate as the nal pruned version. This is similar to the process described in [13]. In contrast to the cost-complexity pruning of CART, in which the true error rate of a tree and its subtrees is predicted from a separate set of examples that are distinct from the training examples, C4.5 uses a signicance test that compares a parent node to its children. Starting with a fully grown tree, when the results of the children are not found signicantly dier5 ent than the parent, the children are pruned. Not requiring independent holdout cases, a signicance test can make this comparison. For example, a signicance test at two standard errors is usually eective. However, holdout cases can improve pruning performance when pruning is performed at varying signicance levels and the most predictive solution is selected. Pruning Bayes' tree also relies upon enhancing posterior probability of the resulting tree. A test set is used for determining the probabilities. Of all trees resulting from pruning a node from a tree, the one that results in maximum posterior probability is chosen. SLIQ employs an alternate scheme for decision tree pruning, the MDL (minimum description length) principle. In the tree pruning phase, the MDL methodology is employed. As per the MDL principle, the total cost of description is the sum of the cost of describing a model, and the cost of describing data that are exceptions to this model. Given alternate models, the MDL principle further states that the best model is the one with the least description cost. In the case of decision trees, the alternate models may be viewed as the set of sub-trees made available as a result of pruning, and the data is the set of examples from which the full tree is initially built. SLIQ utilizes the classication error as the cost for encoding data, given a tree. The cost for describing a tree is formulated as a recursive combination of the cost of encoding a node and the cost of encoding the split at that node. The total cost at each node in a fully grown tree is then used to prune the node back to a leaf node, or to prune its left sub-tree, or right sub-tree, or leave it unchanged. SLIQ employs a twophase pruning strategy; the rst phase does a balanced pruning in which internal nodes either get fully converted to leaf nodes or left unchanged. In the second phase the sub-tree obtained from the rst phase is reexamined to prune back nodes by eliminating partial (either left or right) sub-trees. modeling systems employ a search process to evolve this set of highly specic and individual instances to more general rules. This search process is iterative, and usually terminates when rules can no longer be generalized, or some other alternate stopping criteria satised. As in the case of decision tree building, noise in the data may lead to over-tted decision rules, and various pruning mechanisms have been developed to deal with over-tted decision rule solutions. Rule induction methods attempt to nd a compact \covering" rule set that completely partitions the examples into their correct classes. The covering set is found by heuristically searching for a single \best" rule that covers cases for only one class. Having found a \best" conjunctive rule for a class C, the rule is added to the rule set, and the cases satisfying it are removed from further consideration. The process is repeated until no cases remain to be covered. The AQ [17] family of algorithms is inuenced and motivated by methods used in electrical engineering for simplifying logic circuits. Using the AQ terminology, a test on an attribute is called a selector, a conjunct of tests is called a complex, and a disjunction of complexes is called a cover. If a rule satises an example, it is called a cover for the example. Initially, every example is itself a complex in the model. Complexes are then examined, and selectors are then dropped as long as the resulting complex remains consistent (matching only examples of the same class and none of any other class). Complexes are thus produced, one at a time. Combining generalized complexes produces covers that are also complete (all examples of a class are covered). In the search process for creating complexes, an evaluation function is used for ordering and determining which selectors to drop (or generalize). Although this evaluation function can be set by an external entity (such as an end-user or calling function), the one that is normally used is the ratio of examples correctly classied by a complex to the total examples classied by that complex. CN2 [6] may be regarded as a system that extends AQ in terms of its ability to deal with noise in the data. Specically, CN2 retains a set of complexes during its search that are deemed to be statistically covering a large number of instances of a class, even if they also cover instances of other classes. Additionally, CN2 executes a general to specic search, as opposed to AQ's strict specic to general approach. Each specialization step either add new selectors to a complex, or removes an entire complex. CN2 employs two types of heuristics in the search for the best complexes, signicance and goodness. Sig- 4 Decision Rule Induction Decision rules, in disjunctive normal form (DNF), may be induced from training data in a bottom-up specic-to-general style, or in a top-down general-tospecic style, as in decision tree building. This section will highlight methodologies dealing with bottom-up specic-to-general approaches to rule induction. The initial state of a decision rule solution is indeed the collection of all individual instances or examples in a training data set, each of which may be thought of as a highly specialized decision rule. Most decision rule 6 nicance is the threshold such that any complexes below this threshold will not be considered for selecting the best complex. To test signicance, P CN2 uses the entropy statistic. This is given by 2 ni=1 pi log(pi =qi ), where the distribution p1 ; :::; pi is the observed frequency distribution of examples among classes satisfying a given complex and q1; :::; qi is the expected frequency distribution of the same number of examples under the assumption that the complex selects examples randomly. This statistic provides an informationtheoretic measure of the distance between the two distributions. Any complex whose entropy statistic falls below a pre-specied threshold is rejected. Goodness is a measure of the quality of the complex that is used for ordering complexes that are candidates for inclusion in the nal cover. The commonly used measure of goodness in CN2 is the Laplacian error estimate (n ; nc + k ; 1)=(n + k) where n is the total number of examples covered by the rule, nc is the number of positive examples covered by the rule, and k is the number of classes in the data being modeled. ITRULE [24] also employs a bottom-up search process to formulate rules directly from data. The algorithm generates a set of R rules, where R is a user-dened parameter. This set of rules is considered to be the R most informative rules as dened by the J-measure. The J-measure evaluates the average information content of a rule, and can be used for both generalization and specialization of individual rules until they reach optimal information content level. The algorithm proceeds by rst nding R rules, calculating their J-measures, and then iterating through a process whereby rules with J-measures higher than the rule with the least J-measure are introduced into the list at the expense of the latter. The J-measure and its use is analogous to the entropy statistic and its use in CN2, although theoretically shown to be more robust. Swap-1 [25] uses local optimization techniques to dynamically revise and improve its covering set. Once a covering set is found that separates the classes, the induced set of rules is further rened. Using train and test evaluation methods, the initial covering rule set is scaled back to the most statistically accurate subset of rules. Rules for two classes can potentially be satised simultaneously. Such conicts are resolved by inducing rules for each class according to a class priority ordering, with the last class considered a default class. Unlike the 1-level lookahead employed in constructing tests (such as Gini and entropy based methods), Swap-1 constantly looks back to see whether any improvement can be made before adding a new test. The following steps are taken to form the single best rule: (a) Make the single best swap from among all possible rule component swaps, including deleting a component; (b) If no swap is found, add the single best component to the rule, where \best" is evaluated as predictive value, i.e. percentage correct decisions by the rule. For equal predictive values, maximum case coverage is a secondary criterion. Swapping and component addition terminate when 100% predictive value is reached. Finding the optimal combination of attributes and values for even a single xed-size rule is a complex task. However, there are other optimization problems, such as the traveling salesman problem, where local swapping nds excellent approximate solutions. Given a set of samples S, and a covering rule set RS, RS can be progressively weakened so that it becomes increasingly less complex, though decreasing in accuracy. The objective is to select rule set RSbest from fRS1,...RSi,...RSng, a collection of rule sets in decreasing order of complexity, such that RSbest will make the fewest errors on new cases T. In practice, the optimal solution can usually not be found because of incomplete samples and limitations on search time. It is not possible to search over all possible rule sets of complexity Cx(RSi ), where Cx is some appropriate complexity t measure, such as the number of components in the rule set. If the set fRS1,...RSi,...RSng is ordered by some complexity measure Cx(RSi ), then the best one is selected by min[Err(RSi)]. Thus to solve this problem in practice, a method must induce and order fRSi g by Cx(RSi ) and estimate each rule set's error rate, Err(RSi ). A rule set's error rate is dened as the fraction of misclassied cases to the total classied cases as a result of applying the rule. Pruning methods adapted to rule induction can be used to prune a rule set and form fRSi g. Let the rule set RS1 be the covering rule set. Each subsequent RSi+1 can be found by pruning RSi at its weakest link. A rule set can be pruned by deleting single rules or single components. The application of a form of pruning known as weakest-link pruning results in an ordered series of decreasing complexity rule sets, fRSi g. The RAMP rule generation system [12] generates \minimal" classication rules from tabular data sets where one of the columns is a \class" variable and the remaining columns are \explanatory" features. The data set is completely discretized (i.e., continuous valued features are discretized into a nite set of discrete values, categorical features are left untouched) by an optimal numerical discretization step prior to rule generation. While the RAMP approach to generating classication rules is similar to techniques that directly gen7 erate rule from data, it's primary goal is to strive for a \minimal" rule set that is complete and consistent with the training data. Completeness implies that the rules cover all of the examples in the training data while consistency implies that the rules cover no counter-examples for their respective intended classes. The RAMP system utilizes a logic minimization methodology, called R-Mini, to generate \minimal" complete and consistent rules. This technique was rst developed for programmable logic array circuit minimization (MINI), and is considered to be one of the best known 2-level logic minimization techniques. The merits of minimality have been well discussed [22]. The principal hypothesis here is that a simpler solution tends to have higher accuracy. Thus, if two dierent solutions (in the same representation) both describe a particular data set, the less complex of the two will be more accurate in its description. Complexity is measured dierently for dierent modeling techniques. For decision rules, it would be total number of rules and total number of tests in all the rules. A smaller description will tend to be better in its predictive accuracy, and this has been borne out in our extensive evaluations. A data set with N features may be thought as a collection of discrete points (one per example) in an N-dimensional space. A classication rule is a hypercube (a \complex" in AQ terminology) in this space that contains one or more of these points. When there is more than one cube for a given class, all the cubes are Or-ed to provide a complete logical classication function for the class. Within a cube the conditions for each part are And-ed, thereby giving the DNF representation for the overall classication solution. The size of a cube indicates its generality, i.e., the larger the cube, the more vertices it contains, and potentially cover more example-points. RAMP's minimality objective is rst driven by the minimal number of cubes, and then the most general cubes. The most general cubes are prime cubes that cannot be further generalized without violating the consistency of that cube. The minimality objective translates to nding a minimal number of prime cubes that cover all the example-points of a class and not cover any examplepoints of any counter-class. This objective is similar to many switching function minimization algorithms. The core heuristics used in the RAMP rule generation system for achieving minimality consists of iterating (for a reasonable number of rounds) over two key sub-steps: 1. Generalization step, R-EXPAND, which takes each rule in the current set (initially each exam- ple is a rule) and opportunistically generalizes it to remove other rules that are subsumed. 2. Specialization/Reformulation, R-REDUCE, which takes each rule in the current set and specializes it to the most specic rule necessary to continue covering only the unique examples it covers. Redundant cubes disappear during this step. This annealing-like approach to rule generation (via iterative improvements) may be potentially indenite. A limit is used that controls how long the system should keep iterating without observing a reduction. If no reduction takes place within this limit, the minimization process may be stopped. In practice, it has been observed that RAMP rule generation satisfactorily converges the rule set given that it has gone through at least 5-7 iterations without performing a reduction. RAMP takes a a slightly dierent approach to overcoming the overtting bias in the modeling process. Instead of pruning a solution, RAMP generates multiple solutions. The heuristics that control the rule generation process in RAMP use randomization, and therefore multiple generations from the same training set can result in solutions that model the decision surface using dierent combinations of rules. Although each individual solution models some regions correctly and some incorrectly (due to overtting), the union of multiple solutions enhances the correctness, and smooths out the overtting. Benchmark tests have indicated that combining up to ve solutions produces solutions with very competitive predictive accuracies. 5 Tree and Rule-based Regression Regression is the problem of approximating the values of a continuous variable. Given samples of output (response) variable y and input (predictor) variables x = fx1:::xng, the regression task is to nd a mapping y = f(x). The classical approach to the problem is linear least-squares regression [23]. Linear regression has proven quite eective for many real-world applications. However, the simple linear model has its limits, and more complex models often t the data better. Nonlinear regression models have been explored and many new eective methods have emerged, including projection pursuit and MARS. Neural networks trained by back-propagation is another alternate nonlinear regression model. An overview of many dierent regression models, with application to classica8 tion models as well, is available [21]. Most of these methods produce solutions in terms of weighted models. The CART program induces both classication and regression trees. These regression trees are strictly binary trees. In terms of performance, regression trees often are competitive in performance to other regression methods [4]. Regression trees are noted to be particularly strong when there are many higher order dependencies among the input variables. The advantages of the regression tree solution are similar to the advantages enjoyed by classication trees over other models. On the negative side, decision trees cannot represent compactly many simple functions, for example linear functions. A second weakness is that the regression tree solution is discrete, yet predicts a continuous variable. For function approximation, the expectation is a smooth continuous function, but a decision tree provides discrete regions that are discontinuous at the boundaries. All in all though, regression trees often produce strong results, and for many applications their advantages strongly outweigh their potential disadvantages. x13 ! y=10 x21 ! y=2 Otherwise y=5 Figure 3: Example of Regression Rules recursively pruned so that the ratio delta/n is minimized, where n is the number of pruned nodes and delta is the increase in error. Weakest link pruning has several desirable characteristics: (a) it prunes by training cases only, so that the remaining test cases are relatively independent (b) it is compatible with resampling. An interesting extension to regression trees is exemplied in [19], wherein the tree may be terminated at each of its leaf nodes by a linear regression model. Thus the linearity in a decision surface is modeled at the leaves, while the non-linearity in the decision surface is modeled by the actual tree. 5.2 Regression by Rule Induction 5.1 Regression by Tree Induction Both tree and rule induction models nd solutions in disjunctive normal form, and the model of equation 1 is applicable to both. Each rule in a rule-set represents a single partition or region Ri. However, unlike the tree regions, the regions for rules need not be disjoint. With non-disjoint regions, several rules may be satised for a single sample. Some mechanism is needed to resolve the conicts in ki , the constant values assigned, when multiple rules, Ri regions, are invoked. One standard model is to order the rules, as in Figure 3. Such ordered rule-sets have also been referred to as decision lists. The rst rule that is satised is selected, as in equation 2. Like classication trees, regression trees are induced by recursive partitioning. The solution takes the form of equation 1, where Ri are disjoint regions, ki are constant values, and yji refers to the y-values of the training cases that fall within the region Ri. if x Ri then f(x) = ki = medianfyji g (1) Regression trees have the same representation as classication trees except for the terminal nodes. The decision at a terminal node is to assign a case a constant y value. The single best constant value is the median of the training cases falling into that terminal node because for a partition, the median is the minimizer of mean absolute distance. For regression tree induction, the minimized function, i.e. absolute distance, is a satisfactory splitting criteria for growing the tree. At each node, the single best split that minimizes the mean absolute distance is selected. Splitting continues until fewer than a minimum number of cases are covered by a node, or until all cases within the node have the identical value of y. The pruning strategies employed for classication trees are equally valid for regression trees. Like the covering procedures, the only substantial dierence is that the error rate is measured in terms of mean absolute distance. For weakest-link pruning, a tree is if i < j and x both Ri and Rj then f(x) = ki (2) Given this model of regression rule sets, the problem is to nd procedures that eectively induce solutions. For rule-based regression, a covering strategy analogous to the classication tree strategy could be specied. A rule could be induced by adding a single component at a time, where each added component is the single best minimizer of distance. As usual, the constant value ki is the median of the region formed by the current rule. As the rule is extended, fewer cases are covered. When fewer than a minimal number of cases are covered, rule extension terminates. The covered cases are removed and rule induction can continue 9 1. Generate a set of Pseudo-classes. 2. Generate a covering rule-set for the transformed classication problem using a rule induction method such as Swap-1. 3. Initialize the current rule set to be the covering rule set and save it. 4. If the current rule set can be pruned, iteratively do the following: a) Prune the current rule set. b) Optimize the pruned rule set and save it. c) Make this pruned rule set the new current rule set. 5. Use test cases or cross-validation to pick the best of the saved rule sets. Figure 4: Swap-1R Method for Learning Regression Rules on the remaining cases. This is also the regression analogue of rule induction procedures for classication. The Swap-1R [26] system for inducing decision regression rules works by mapping the regression problem into a classication problem. Let fCig be a set consisting of an arbitrary number of classes, each class containing approximately equal values of fyi g. To solve a classication problem, the classes are expected to be dierent from each other, and it is assumed that rules can be found to distinguish these classes. Classes formed by an ordering and discretization of fyi g form the classication problem. In practice, one learning model is not always superior to others, and a learning strategy that examines the results of dierent models may do better. Moreover, by combining dierent models, enhanced results may be achieved. A general approach to combining learning models is a scheme referred to as stacking [28]. The models could be completely dierent, such as combining decision trees with linear regression models. Dierent models are applied independently to nd solutions, and in a subsequent layer yet another model is used for combining the solutions into a single solution. This layer may be a simple weighted vote, as per equation 3, or something more sophisticated. This method of model combination is in contrast to the usual approach to evaluation of dierent models, where the single best performing model is selected. 10 y= K X w M (x) k=1 k k (3) While stacking has been shown to give improved results, a major drawback is that properties of the combined models are not retained. Thus when interpretable models are combined, the result may not be interpretable at all. It is also not possible to compensate for weaknesses in one model by introducing another model in a controlled fashion. A modied technique for combination of alternate solutions is to retain the interpretable nature of rules, while at the same time addressing the problem of symbolic regression solutions assigning a constant value as the predictor, once a region is identied. The following strategy is used to determine the y-value of a case x that falls in region Ri , instead of assigning a single constant value ki for region Ri, where ki is determined by the median y value of training cases in the region, i (x), the mean of the k-nearest (training assign yknn set) instances of x in region Ri . An interesting aspect of this strategy is that knearest neighbor results need only be considered for the cases covered by a particular partition. This hybrid approach alleviates the weakness of partitions being assigned single constant values. Moreover, some of the global distance measure diculties of the k-nn methods may also be relieved because the table lookup is reduced to partitioned and related groupings. Another decision rule based regression approach is exemplied by the RAMP system. This system also supports post-processing for regression, by computing additional metrics for the rules as they are generated on a pre-transformed pseudo classication problem, based upon example data-points that are covered by each rule in the training data. Three parameters are attached to each rule, , the mean of all the original class values of training examples covered by that rule; , the standard deviation of these values; and N, the total number of training examples covered by that rule. For regression estimation, two straightforward averaging approaches are made available, the simple and weighted approach. In the simple averaging approach, the simple average of of all rules that cover an example is computed as its predicted value. Therefore, for each example in the test data, if M is the total number of rulesPthat cover the example, its predicted M i class value is iM=1 . In the weighted average approach, there are several options available for weighting the rules. One of these options is to compute and assign a prediction of the weighted average, e.g., P P p Ni M i=1 pi i Ni M i=1 i . In general, weighted averaging usually leads to smoother correlations between predicted and actual values. It has been observed that no unique combination of weighting and error estimation seems to be uniformly applicable. The user has the ability to evaluate these combinations on test data, determine which one tends to be most accurate, and then utilize that metric for ne tuning the solution. Data Warehouse Sample Rule/Tree solution Sample Rule/Tree solution Sample Rule/Tree solution Sample Rule/Tree solution New example 6 Discussion The rise in attention and focus on decision support solutions using data mining techniques has refueled a big interest in classication and regression modeling, particularly symbolic techniques [10]. This paper has attempted to provide the reader with the key issues of decision tree and decision rule modeling techniques, two important approaches to symbolic modeling. While some aspects of this technology have reached maturity and become stable, there are also many aspects that remain open. Symbolic modeling approaches that remain consistently robust across a wide variety of data sets are not yet well understood. Additionally, though some of these techniques are conceptually robust and elegant, they prove to be computationally challenging when applied to large scale business and industrial data sets. Many new approaches are under active research and development. It is well recognized that these techniques cannot be applied in a black box mode to data. Intensive application analysis and knowledge engineering continues to play an important role. Techniques such as preprocessing of raw data and feature extraction contribute greatly to improving the accuracy of the symbolic modeling process. Techniques to handle missing and erroneous values in data are also critical. Techniques and methods for feature selection [15] play a useful role in pre-pruning the search space. Characteristics of data, such as too many categories for a variable, or extreme bias in class proportions, or hierarchies in attributes, can signicantly aect the modeling process, and a catalog of methodologies to address these issues is also slowly emerging. Scalability is emerging as a key factor in coupling these techniques to the extremely large volumes of data that are becoming available today. Systems and techniques that focus on resolving this key problem are beginning to emerge [7, 16]. Techniques such as stacking [28], bagging [3], and boosting [11], also prove to be extremely useful in improving the modeling process, by using hybrid ap- Averaging or Voting Mechanism New example classification or regression result Figure 5: Multiple Solutions for Maximizing Accuracy proaches that either combine solutions from multiple approaches, or multiple solutions using the same approach but on dierent training samples. Utilizing the bagging approach, predictive performance can be improved, sometimes very substantially, by nding many solutions on dierent random samples taken from a large data warehouse. For classication, the many answers for a new case are voted or for regression they are averaged. This process is illustrated in Figure 5. N cases may be randomly drawn from a large data base or even simulated by resampling from a smaller dataset. If a learning method is fast, it is not dicult to generate new solutions for each new sample of N cases. The most obvious candidate for this approach to learning is the decision tree. In the boosting approach, error cases are sampled with greater frequency, in subsequent modeling iterations. For a single solution and sample, a decision tree usually yields good results for most learning problems. Solutions are found quickly, typically much faster than most other learning methods. The predictive performance of decision trees is often weaker than some other learning methods, such as neural nets. However, studies show that when answers for solutions found on many random samples are voted, the resulting performance can approach optimal predictive performance [3]. The predictive performance is often signicantly increased, though the clarity of presentation may be sometimes compromised. Finally, an open issue that continues to be explored is characterization of datasets, using either simple measures, statistical measures, or information theoretic measures, that will allow an educated mapping of the most appropriate mining technique to a dataset 11 for maximizing the accuracy of the resulting solution. [13] J. Hosking, E. Pednault, and M. Sudan. A Statistical Perspective on Data Mining. 1997. in this issue. [14] M. James. Classication Algorithms. John Wiley & Sons, 1985. [15] I. Kononenko and S.J. Hong. Attribute Selection for Modelling. 1997. in this issue. [16] M. Mehta, R. Agrawal, and J. Rissanen. SLIQ: A Fast Scalable Classier for Data Mining. In References [1] C. Apte, F. Damerau, and S. Weiss. Automated Learning of Decision Rules for Text Categorization. ACM Transactions on Information Systems, 12(3):233{251, July 1994. [2] C. Apte and S.J. Hong. Predicting Equity Returns from Securities Data with Minimal Rule Generation. In Advances in Knowledge Discovery, pages 541{560. AAAI Press / The MIT Press, 1995. Proceedings of the Fifth International Conference on Extending Database Technology, 1996. [17] R. Michalski, I. Mozetic, J. Hong, and N. Lavrac. The Multi-Purpose Incremental Learning System AQ15 and its Testing Application to Three Medical Domains. In Proceedings of the AAAI-86, pages 1041{1045, 1986. [18] D. Michie, D. Spiegelhalter, and C. Taylor. Ma- [3] L. Breiman. Bagging predictors. Machine Learning, 24:123{140, 1996. [4] L. Breiman, J. Friedman, R. Olshen, and C. Stone. Classication and Regression Trees. Wadsworth, Monterrey, CA., 1984. chine Learning, Neural and Statistical Classication. Ellis Horwood, 1994. [19] J. Quinlan. Combining instance-based and model-based learning. In International Conference on Machine Learning, pages 236{243. Morgan Kaufmann, 1993. [20] J.R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993. [21] B. Ripley. Pattern Recognition and Neural Networks. Cambridge University Press, 1996. [22] J. Rissanan. Stochastic Complexity in Statistical Inquiry. World Scientic Series in Computer Science, 15, 1989. [23] H. Schee. The Analysis of Variance. John Wiley & Sons, 1959. [24] P. Smyth and R. Goodman. An Information Theoretic Approach to Rule Induction from Databases. IEEE Transactions on Knowledge and Data Engineering, 4(4):301{316, August 1992. [25] S. Weiss and N. Indurkhya. Optimized Rule Induction. IEEE EXPERT, 8(6):61{69, December 1993. [26] S. Weiss and N. Indurkhya. Rule-based Machine Learning Methods for Functional Prediction. Journal of Articial Intelligence Research, 3:383{403, 1995. [27] S. Weiss and C.A. Kulikowski. Computer Systems That Learn. Morgan Kaufmann, 1991. [5] W. Buntine. Learning Classication Trees. Statistics and Computing, 2:63{73, 1992. [6] P. Clark and T. Niblett. The CN2 Induction Algorithm. Machine Learning, 3:261{283, 1989. [7] W. Cohen. Fast eective rule induction. In The XII International Conference on Machine Learning, pages 115{123, 1995. [8] M. Craven and J. Shavlik. Using Neural Networks for Data Mining. 1997. in this issue. [9] U. Fayyad, S.G. Djorgovski, and N. Weir. Automating the Analysis and Cataloging of Sky Surveys. In Advances in Knowledge Discovery, pages 471{493. AAAI Press / The MIT Press, 1995. [10] U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, editors. Advances in Knowledge Discovery and Data Mining. AAAI Press / The MIT Press, 1995. [11] Y. Freund and R. Schapire. Experiments with a New Boosting Algorithm. In Proceedings of the International Machine Learning Conference, pages 148{156. Morgan Kaufmann, 1996. [12] S.J. Hong. R-MINI: An Iterative Approach for Generating Minimal Rules from Examples. IEEE Transactions on Knowledge and Data Engineering, 1997. to appear. 12 [28] D. Wolpert. Stacked generalization. Neural Networks, 5:241{259, 1992. 13