Download 2004 - NEO

Genetic and Evolutionary Computation COnference 2004
1/13
2004
ANNs
Algorithms
Experiments
Training Neural Networks
with GA Hybrid
Algorithms
Conclusions &
Future Work
Enrique Alba and J. Francisco Chicano
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
2/13
Artificial Neural Networks
• An ANN is a structured pool of Artificial Neurons:
2004
ANNs
Algorithms
Experiments
Conclusions &
Future Work
• The architecture determines how the neurons are connected:
• Feedforward networks (used here)
• Recurrent networks
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
3/13
Multilayer Perceptron
• We use Multilayer Perceptrons
2004
ANNs
Algorithms
Experiments
Conclusions &
Future Work
• Parameters: number of layers, neurons per layer, activating functions
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
4/13
Training
• The training process consists in adjusting the network weights:
2004
• Supervised learning
• Non-supervised learning
ANNs
Algorithms
Experiments
• Supervised training process consists in adjusting the weights to get
the desired output for the present input patterns
Conclusions &
Future Work
• To measure de quality of the network several options exist:
• SEP
• CEP: percentage of incorrectly classified patterns
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
5/13
Backpropagation
• BP is a gradient-based method
2004
learning rate
ANNs
Algorithms
BP
LM
Hybrids
Experiments
Conclusions &
Future Work
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
6/13
Levenberg-Marquardt
2004
• LM is a “quasi” second order method
variable parameter Jacobian
ANNs
Algorithms
BP
LM
Hybrids
Experiments
Conclusions &
Future Work
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
7/13
Hybrid Algorithms
• Hybridization: Inclusion of problem knowledge into the algorithm
2004
ANNs
Algorithms
BP
LM
Hybrids
• Two possible classes of hybrid algorithms:
• Strong: Specific representation and operators
• Weak: Combination of several algorithms (cooperation)
GA main loop
GABP main loop
GALM main loop
Selection
Selection
Selection
Recombination
Recombination
Recombination
Mutation
Backpropagation
Levenberg-Marquardt
Replacement
Replacement
Replacement
Experiments
Conclusions &
Future Work
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
8/13
Experiments
• Three classification problems from PROBEN1 benchmark
2004
ANNs
Algorithms
Experiments
Results
Conclusions &
Future Work
• Multilayer perceptrons with one output neuron per class
• Six neurons in the hidden layer
• Sigmoid activating function
• In GA and hybrids, fitness = the inverse of SEP
• Pattern sets: Training (75%) and Test (25%)
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
9/13
Results
• GA gets higher CEP than BP and LM (expected result)
2004
ANNs
• BP more accurate than LM
• GALM more accurate than GABP
Algorithms
Experiments
BP
60
Conclusions &
Future Work
50
GA
GABP
GALM
50 independent runs
21.76
25.77
36.46
36.46
28.29
27.41
34.73
41.50
54.30
22.66
40
Diabetes
Heart
30
20
10
0
0.91
3.17
16.76
1.43
0.02
CEP (%)
Results
LM
Cancer
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
10/13
Results
• GA gets higher CEP than BP and LM (expected result)
2004
ANNs
• BP more accurate than LM
• GALM more accurate than GABP
unexpected behavior!
Algorithms
Experiments
BP
60
Conclusions &
Future Work
50
GA
GABP
GALM
50 independent runs
21.76
25.77
36.46
36.46
28.29
27.41
34.73
41.50
54.30
22.66
40
Diabetes
Heart
30
20
10
0
0.91
3.17
16.76
1.43
0.02
CEP (%)
Results
LM
Cancer
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
11/13
Results
2004
ANNs
Algorithms
Experiments
• LM is the fastest algorithm in minimizing the SEP
• GA evolution is accelerated with the inclusion of BP and LM
• BP and LM over-train the ANN to the used pattern set
50 independent runs
Results
Conclusions &
Future Work
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
12/13
Conclusions & Future Work
Conclusions
2004
• BP and LM outperform GA
ANNs
Algorithms
Experiments
• Hybrid algorithms outperform pure ones
• GALM hybrid found the best CEP (known to our knowledge) for
Cancer (0.02%) and Heart (22.66%)
Conclusions &
Future Work
Future Work
• Study other algorithms for these problems (ES, ESBP, ESLM)
• Solve additional instances (Gene, Soybean, Thyroid, …)
Seattle, Washington (USA), June 26-30, 2004
Genetic and Evolutionary Computation COnference 2004
13/13
THE END
2004
Thanks for your attention !!!
ANNs
Algorithms
Experiments
Conclusions &
Future Work
Seattle, Washington (USA), June 26-30, 2004