Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
A Genetic Programming System for Building Block Analysis to Enhance Data Analysis and Data Mining Techniques Christoph F. Eick, Walter D. Sanz, and Ruijian Zhang www.cs.uh.edu/~ceick/matta.html University of Houston Organization 1. Introduction 2. General Approach and Corresponding Methodology 3. Wols --- A Genetic Programming System to Find Building Blocks 4. Experimental Results 5. Summary and Conclusion SPIE Data Mining Conference, Orlando, April 1999 1. Introduction Regression analysis, which is also know as curve fitting, may be broadly defined as the analysis of relationships among variables. The actual specification of the form of the function weather it be linear, quadratic, or polynomial, is left up entirely to the user. Symbolic regression is the process of discovering both the functional form of a target function and all of its necessary coefficients. Symbolic regression involves finding a mathematical expression in symbolic form, that provides a good, best or perfect fit between a finite sampling of values of the input variables and the associated values of the dependent variables. Genetic Programming simulates the Darwinian evolutionary process and naturally occurring genetic operations on computer programs. Genetic programming provides a way to search for the fittest computer program in a search space which consists of all possible computer programs composed of functions and terminals appropriate to the problem domain. SPIE Data Mining Conference, Orlando, April 1999 Example -- Evolving Syntax Trees with Genetic Programming: * A*B= A B Population = 4 Best So Far - * cos * + Geneartion 0: A B C D D C 5 0.1 0.2 0.5 0.9 * - cos * A B 0.1 cos Geneartion 1: A C B 0.07 D 0.7 A / 5 E crossover B 0.1 / 5 E copying 0.05 mutation 0.05 SPIE Data Mining Conference, Orlando, April 1999 Motivating Example Task: Approximate boundaries between classes with rectangles C1 C2 Easy! C1 C2 Difficult! Idea: Transform the representation space so that the approximation problem becomes simpler (e.g. instead (x,y), just use (x-y)). Problem: How do we find such representation space transformations? SPIE Data Mining Conference, Orlando, April 1999 Building Blocks Definition: Building Blocks are patterns which occur with a high frequency in a set of solutions. Example: (- (* C D) (+ A B)) (- C (+ B A)) (- (+ A B) (COS A)) (- (+ C (/ E (SQRT D))) (+ A B)) Leaf Building Block: (+ A B) Root Originating Building Block: (- ? (+ A B)) FUNC1 = (+ A B) SPIE Data Mining Conference, Orlando, April 1999 Constructive Induction Definition: Constructive Induction is a general approach for coping with inadequate attributes, redundant, and useless attributes found in original data. Constructive Induction explores various representation spaces. Building block analysis and constructive induction are used in this research to transform the original data by adding new attributes and by dropping useless ones. Example: Primative Attributes Cases A B C #1 0.8 0.1 0.2 #2 0.4 0.5 0.3 Primative Attributes and New Attributes Cases A B C A-B A-C B-C #1 0.8 0.1 0.2 0.82 0.84 0.28 #2 0.4 0.5 0.3 0.7 0.58 0.65 SPIE Data Mining Conference, Orlando, April 1999 2. General Approach and Corresponding Methodology Primary Learning System: the one we use for the task at hand (e.g. C5.0) Secondary Learning System: is used to find building blocks General Steps of the Methodology: 0. Run the primary learning system for the data collection. Record prediction performance and good solutions. 1. Run the genetic programming/symbolic regression/building block analysis system (secondary learning system) several times for the data collection with a rich function set. Record prediction performance and good solutions. 2. Analyze the good solutions found by the secondary learning system for the occurrence of building blocks. 3. Using the results of step 2, transform the data collection by generating new attributes. 4. Redo steps 0-3 for the modified data collection, as long as a significant improvement occurs SPIE Data Mining Conference, Orlando, April 1999 3. Wols --- A Genetic Programming System that finds Bulding Blocks SRGP Front End Interface Executable SRGP CODE Building Block Detection Tool SPIE Data Mining Conference, Orlando, April 1999 Wols’ Major Inputs Example: Attributes Cases A1 A2 B #1 0.8 0.1 0.2 #2 0.4 0.3 0.3 B = f(A1, A2) #3 0.3 0.4 0.4 Function Set: *, +, SIN #4 0.2 0.5 0.1 B = A1 * SIN(A2 + A1) SPIE Data Mining Conference, Orlando, April 1999 Features of the Wols System The WOLS system is basically a modified version of the standard genetic program model. A standard genetic program model employs crossover and mutation operators that evolve solutions (here approximation functions) relying on the principles of the survival of the fittest: better solutions reproduce with a higher probability. The WOLS system employs the traditional “sub-tree swapping” crossover operator, but provides a quite sophisticated set of 7 mutation operators, that apply various random changes to solutions. The goal of the WOLS system is to find good approximations for an unknown function ƒ, and to analyze the decomposition of good approximations with the goal to identify building blocks. The front end of the WOLS system takes a data collection DC=(A1,…,An, B) as its input and tries to find good approximations h, where B=h(A1,…,An), of the unknown function f with respect to a given function set and a given error function. The symbolic regression genetic programming component (SRGP) of the WOLS system includes an interactive front end interface which allows a user to easily generate a genetic program that employs symbolic regression and is tailor made to the problem he wishes to solve. SPIE Data Mining Conference, Orlando, April 1999 Features of the Wols System (cont.) The system run starts by asking the user interactively for parameters needed to run the symbolic regression system. The information the user provides is then used to generate the functions for the specific problem he wishes to solve. These problem specific functions are then merged with a set of general genetic programming functions to form a complete problem specific genetic program. The SRGP component allows the user to select between three different error functions: • Manhattan error function (minimizes the absolute error in the output variable) • Least Square error function (minimizes the squared error in the output variable) • Classification error function (minimizes the number of misclassifications) The WOLS system is capable of solving both regression analysis and classification problems. In regression analysis problems the goal is to minimize the error, while classification problems the goal is to minimize the number of misclassifications. SPIE Data Mining Conference, Orlando, April 1999 Features of the Wols System (cont.) The goal of the decision block analysis tool is to identify frequently occurring patterns in sets of solutions. In our particular approach, building blocks are expressions that involve functions symbols, variable symbols, as well as the ‘?’ (“don’t care”) symbol. For example, if DC=(A1, A2, A3, B) is a data collection the variable A1, the expression (A2 * A3), and expression ((sin A1) + (A2 * ?)) are all potential building blocks. In general we are interested in finding patterns that occur in about 85% or more of the set of analyzed approximations. The identification of building blocks is useful for both improving the attribute space and in determining which part of the search space has been explored. SPIE Data Mining Conference, Orlando, April 1999 Decision Block Analysis Functions Root Analysis: Finds common root patterns among a population of solution trees. General Analysis: Finds common patterns anywhere in a solution tree. Leaf Analysis: Finds common patterns at the leaf nodes of solution trees. Frequency Analysis: Finds the total number of times a specific operator,variable, or constant is used in the population of solution trees. SPIE Data Mining Conference, Orlando, April 1999 4. Experimental Results Experiment 1- 5 variable problem with 30 points training data Experiment 2 - 5 variable problem with 150 points training data Experiment 3 - Simple Linear Regression Experiment 4 - Time for a Hot Object To Cool Experiment 5 - Time for a Hot Object To Cool Using Constructive Induction Experiment 6 - Glass Classification Experiment 7 - Learning An Approximation Function SPIE Data Mining Conference, Orlando, April 1999 Experiment 7 - Learning An Approximation Function f(a, b, c, d, e) = |b-|d-0.44||*|log(a+1)| + a*b*(1-log(2)) Random values were plugged into function f in order to generate test cases of the form (a, b, c, d, e, f(a, b, c, d, e)) training data set of 150 test cases, testing data set size 150 test cases Goal was to find a function h(a, b, c, d, e) that best approximates function f. function set: +, *, loga(x):= |log(x)|, cosa(x):=|cos(x)|, sqrta(x):=|sqrt(x)|, por(a,b):= a+b-a*b, diva(a/b):= |a/b|, aminus(a,b):= |a-b|, expa(x):= emin(x 300), sin100a:= |sin(100*x)| variable set: a, b, c, d, e SPIE Data Mining Conference, Orlando, April 1999 Experiment 7 - continued Fitness Function: Average Absolute Manhattan Distance n i i where: i 1 n n = total number of test cases xi = output generated by the solution found through genetic programming yi = output generated by the known function (desired output) x y RUN 1 - Direct Approach RUN 2 - Using Leaf Building Blocks RUN 3 - Using Leaf Building Blocks and Reduced Operator and Variable Sets For each run a population size of 200 was used. Each run was terminated after 100 generations. SPIE Data Mining Conference, Orlando, April 1999 RUN 1 - Direct Approach Average Absolute Manhattan Distance Error Training: 0.01102 Testing: 0.01332 Leaf Building Blocks In The Top 10% Of The Population Pattern (* B A) occurred 25 times in 100.0 % of the solutions. Pattern (AMINUS 0.41999999999999998 D) occurred 32 times in 95.0 % of the solutions. Pattern (EXPA (AMINUS 0.41999999999999998 D)) occurred 28 times in 95.0 % of the solutions. Frequency Analysis In The Top 10% Of The Population + was used 9 times in 25.0% of population aminus was used 56 times in 100.0% of population * was used 176 times in 100.0% of population / was used 75 times in 100.0% of population por was used 111 times in 100.0% of population sqrta was used 63 times in 100.0% of population SPIE Data Mining Conference, Orlando, April 1999 Frequency Analysis (continued) sin100a was used 3 times in 10.0% of population cosa was used 1 times in 5.0% of the population loga was used 2 times in 5.0% of population expa was used 33 times in 100.0% of population A was used 76 times in 100.0% of population B was used 132 times in 100.0% of population C was used 9 times in 35.0% of population D was used 57 times in 100.0% of population E was used 21 times in 40.0% of population CONSTANT was used 152 times in 100.0% of population Leaf Building Blocks: (* B A), (AMINUS 0.41999999999999998 D), and (EXPA (AMINUS 0.41999999999999998 D)) Since variables C and E are the only two variables not present in 100% of the solutions analyzed they will be eliminated in RUN 3 along with the least used operator cosa. Also notice that variables C and E were not actually present in function f and were successfully filtered out. SPIE Data Mining Conference, Orlando, April 1999 RUN 2 - Using Leaf Building Blocks Solution (* ADF1 (SQRTA (+ (SQRTA (AMINUS (SQRTA (POR B (SQRTA (AMINUS (* (* B (POR 0.93000000000000005 (AMINUS (+ ADF1 ADF1) ADF1))) (POR (AMINUS (POR (SQRTA (AMINUS (* 0.91000000000000003 (POR A (EXPA (AMINUS (POR ADF2 (POR (SQRTA ADF2) (+ (DIVA 0.80000000000000004 B) (SQRTA (AMINUS (SQRTA (AMINUS B ADF2)) ADF2))))) ADF2)))) ADF2)) (LOGA (DIVA 0.80000000000000004 B))) ADF1) ADF2)) ADF2)))) ADF2)) 0.029999999999999999))) Average Absolute Manhattan Distance Error Training: 0.00886 Testing: 0.01056 In RUN 2 there is a 19.6% increase in accuracy in training and a 20.7% improvement in testing over RUN 1. Where: ADF1 = (* B A) ADF2 = (AMINUS 0.41999999999999998 D) ADF3 = (EXPA (AMINUS 0.41999999999999998 D)) SPIE Data Mining Conference, Orlando, April 1999 Best Solution Run3 Solution (DIVA ADF1 (EXPA (* (LOGA ADF3) (AMINUS (+ (DIVA (POR 0.23999999999999999 A) (POR B (LOGA (POR ADF1 (POR (POR (EXPA A) (EXPA (LOGA (POR (DIVA ADF3 B) (AMINUS ADF3 B))))) (+ (EXPA 0.95000000000000007) A)))))) ADF2) ADF1)))) Where: ADF1 = (* B A) ADF2 = (AMINUS 0.41999999999999998 D) ADF3 = (EXPA (AMINUS 0.41999999999999998 D)) Variables C and E and operator cosa were removed from the variable and operator sets. SPIE Data Mining Conference, Orlando, April 1999 RUN 3 - Using Leaf Building Blocks and Reduced Operator and Variable Sets Solution (DIVA ADF1 (EXPA (* (LOGA ADF3) (AMINUS (+ (DIVA (POR 0.23999999999999999 A) (POR B (LOGA (POR ADF1 (POR (POR (EXPA A) (EXPA (LOGA (POR (DIVA ADF3 B) (AMINUS ADF3 B))))) (+ (EXPA 0.95000000000000007) A)))))) ADF2) ADF1)))) Where: ADF1 = (* B A) ADF2 = (AMINUS 0.41999999999999998 D) ADF3 = (EXPA (AMINUS 0.41999999999999998 D)) Average Absolute Manhattan Distance Error Training: 0.00825 Testing: 0.00946 In RUN 3 there is a 7.0% increase in accuracy in Variables C and E and operator cosa were removed from the variable and operator sets. training and a 10.4% improvement in testing over RUN 2 and a 25.1% increase in accuracy in training and a 29.0% improvement in testing over RUN 1. SPIE Data Mining Conference, Orlando, April 1999 5. Summary and Conclusion The WOLS system is a tool whose objective is to find good approximations for an unknown function , and to analyze the decomposition of good approximations with the goal to identify building blocks. Our experiments demonstrated that significant improvements in the predictive accuracy of the function approximations obtained can be achieved by employing building block analysis and constructive induction. However, more experiments are needed to demonstrate the usefulness of the WOLSsystem. The symbolic regression/genetic programming front-end (SRGP) did quite well for a number of regression system benchmarks. This interesting observation has to be investigated in more detail in the future. The WOLS system was written in GCL Common LISP Version 2.2. All Experiments were run on an Intel 266 MHZ Pentium II with 96 MB RAM. SPIE Data Mining Conference, Orlando, April 1999 Future Work Improving the WOLS system’s performance in the area of classification problems. Conduct more experiments that use the WOLS system as a secondary learning system, whose goal is to search for good building blocks useful for improving the attribute space for other inductive learning methods such as C4.5. Construct a better user interface, such as a graphical user interface. Although the empirical results obtained from the large number of experiments involving regression were encouraging more work still needs to be done in this area as well. SPIE Data Mining Conference, Orlando, April 1999 Experiment 3 - Simple Linear Regression In this experiment the WOLS system, NLREG, and Matlab were all used to obtain an approximation function for a simple linear regression problem. NLREG, like most conventional regression analysis packages, is only capable of finding the numeric coefficients for a function whose form (i.e. linear, quadratic, or polynomial) has been prespecified by the user. A poor choice, made by the user, of the functions form will in most cases lead to a very poor solution which would not be truly representative of the programs ability to solve the problem. In order to avoid this problem, for this experiment both the data set and the actual functional form, from one of the example problems provided with this software, were used. In Matlab there are basically two different approaches available to perform regression analysis. The quickest method, which was used in this experiment, is to use the command polyfit(x, y, n), which instructs Matlab to fit an nth order polynomial to the data. The second method is similar to NLREG and requires the functional form to be specified by the user. training data set of 30 test cases; testing data set size 15 test cases The following parameters were used for the WOLS system: Population Size: 500 Generations: 200 function set: +, *, loga(x):= |log(x)|, cosa(x):=|cos(x)|, sqrta(x):=|sqrt(x)|, por(a,b):= a+b-a*b, diva(a/b):= |a/b|, aminus(a,b):= |a-b|, expa(x):= emin(x 300), sin100a:= |sin(100*x)| variable set: a SPIE Data Mining Conference, Orlando, April 1999 NLREG Solution 1: Title "Linear equation: y = p0 + p1*x"; 2: Variables x,y; // Two variables: x and y 3: Parameters p0,p1; // Two parameters to be estimated: p0 and p1 4: Function y = p0 + p1*x; // Model of the function Stopped due to: Both parameter and relative function convergence. Proportion of variance explained (R^2) = 0.9708 (97.08%) Adjusted coefficient of multiple determination (Ra^2) = 0.9698 (96.98%) ---- Descriptive Statistics for Variables ---Variable ---------x y Minimum value Maximum value --------------------------1 8 6.3574 22.5456 Mean value --------------4.538095 14.40188 Standard dev. ------------2.120468 4.390238 Standard error ----------------5.14417276 0.06681681 t ----------------0.3336883 30.53 Prob(t) --------15.42 0.00001 Mean Square --------932.15 0.5821475 F value ------0.00001 ---- Calculated Parameter Values ---Parameter ---------p0 p1 Initial guess 1 Final estimate --------------1 2.0399976 ------0.00001 ---- Analysis of Variance ---- Source DF Sum of Squares --------------------------------------Regression 1 542.6514 542.6514 Error 28 16.30013 Total 29 558.9515 y = 5.14417276 + 2.0399976x The average absolute Manhattan distance error Training: 0.6361 Testing: 0.7438 Prob(F) SPIE Data Mining Conference, Orlando, April 1999 Matlab Solution 1st order polynomial(n = 1) y = 2.0400x + 5.1442 Training: 0.6361 Testing: 0.7438 5th order polynomial(n = 5) y = 0.0070x5 - 0.1580x4 + 1.3365x3 - 5.2375x2 + 11.5018x - 1.0632 Training: 0.5838 Testing: 0.8034 10th order polynomial(n = 10) y = 0.0004x10 - 0.0187x9 + 0.3439x8 - 3.5856x7 + 23.2763x6- 97.5702x5 + 265.0925x4 456.1580x3 + 469.7099x2 - 254.6558x + 59.9535 Training: 0.4700 Testing: 0.8669 In viewing the results above, one the first things that is noticed is that as the order of the polynomial increases, the training performance also increases, but the testing performance decreases. This is because the higher order polynomials tend emulated the training set data too closely which in return causes a poorer performance when the testing data set values are applied. SPIE Data Mining Conference, Orlando, April 1999 WOLS Solution (+ (POR (+ X X) 0.080000000000000002) (+ (+ (DIVA 0.95000000000000007 0.38) (+ (DIVA (LOGA (* (POR (EXPA (POR 0.94000000000000006 (LOGA (COSA (+ (DIVA (+ (EXPA (DIVA (+ (EXPA (+ 0.68000000000000005 0.13)) 0.51000000000000001) 0.38)) 0.51000000000000001) 0.38) X))))) 0.42999999999999999) (POR X (LOGA (SIN100A (LOGA (* (POR 0.98999999999999999 0.42999999999999999) (POR X 0.040000000000000001)))))))) 0.38) (COSA (LOGA (* (POR (EXPA (POR 0.46000000000000002 1.0)) 0.46000000000000002) (POR (SIN100A (COSA X)) (+ (DIVA (DIVA 0.83000000000000007 0.17999999999999999) 0.38) X))))))) (SIN100A (* X 0.26000000000000001)))) Average Absolute Manhattan Distance Training: 0.3788 Testing: 0.6906 Runtime wise the WOLS system cannot compete with NLREG or Matlab each of which found a solution in around 1 second, while the WOLS system spent nearly 4 hours generating its solution. The WOLS system produced a solution that had 40% better training performance and a 7 % better testing performance then both the NLREG and Matlab solutions. This solution was obtained without using building block analysis and constructive induction. SPIE Data Mining Conference, Orlando, April 1999