Download Evolutionary algorithms

http://lamda.nju.edu.cn
演化学习
基于演化优化处理机器学习问题
俞扬
南京大学
软件新技术国家重点实验室
机器学习与数据挖掘研究所
Evolutionary algorithms
Genetic Algorithms
[J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press,
1975.]
Evolutionary Strategies
[I. Rechenberg. Evolutionstrategie: Optimierung Technisher Systeme nach
Prinzipien des Biologischen Evolution. Fromman-Hozlboog Verlag, Stuttgart, 1973.]
Evolutionary Programming
[L. J. Fogel, A. J. Owens, M. J. Walsh. Artificial Intelligence through Simulated
Evolution, John Wiley, 1966.]
and many other nature-inspired algorithms ...
problem-independent
reproduction
for binary vector:
archive
new
solutions
evaluation
& selection
2015.11.23, CAAMAS：演化学习
mutation: [1,0,0,1,0] → [1,1,0,1,0]
crossover: [1,0,0,1,0] + [0,1,1,1,0]
→ [0,1,0,1,0] + [1,0,1,1,0]
for real vector:
mutation:
.nju.edu.cn
Evolutionary algorithms
Genetic Algorithms
[J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press,
1975.]
Evolutionary Strategies
[I. Rechenberg. Evolutionstrategie: Optimierung Technisher Systeme nach
Prinzipien des Biologischen Evolution. Fromman-Hozlboog Verlag, Stuttgart, 1973.]
Evolutionary Programming
[L. J. Fogel, A. J. Owens, M. J. Walsh. Artificial Intelligence through Simulated
Evolution, John Wiley, 1966.]
and many other nature-inspired algorithms ...
problem-independent
reproduction
archive
new
solutions
evaluation
& selection
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Evolutionary algorithms
Genetic Algorithms
[J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press,
1975.]
Evolutionary Strategies
[I. Rechenberg. Evolutionstrategie: Optimierung Technisher Systeme nach
Prinzipien des Biologischen Evolution. Fromman-Hozlboog Verlag, Stuttgart, 1973.]
Evolutionary Programming
[L. J. Fogel, A. J. Owens, M. J. Walsh. Artificial Intelligence through Simulated
Evolution, John Wiley, 1966.]
and many other nature-inspired algorithms ...
problem-independent
reproduction
archive
new
solutions
evaluation
& selection
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Evolutionary algorithms
Genetic Algorithms
[J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press,
1975.]
Evolutionary Strategies
[I. Rechenberg. Evolutionstrategie: Optimierung Technisher Systeme nach
Prinzipien des Biologischen Evolution. Fromman-Hozlboog Verlag, Stuttgart, 1973.]
Evolutionary Programming
[L. J. Fogel, A. J. Owens, M. J. Walsh. Artificial Intelligence through Simulated
Evolution, John Wiley, 1966.]
and many other nature-inspired algorithms ...
problem-independent
reproduction
archive
new
solutions
evaluation
& selection
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Evolutionary algorithms
Genetic Algorithms
[J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press,
1975.]
Evolutionary Strategies
[I. Rechenberg. Evolutionstrategie: Optimierung Technisher Systeme nach
Prinzipien des Biologischen Evolution. Fromman-Hozlboog Verlag, Stuttgart, 1973.]
Evolutionary Programming
[L. J. Fogel, A. J. Owens, M. J. Walsh. Artificial Intelligence through Simulated
Evolution, John Wiley, 1966.]
and many other nature-inspired algorithms ...
problem-independent
reproduction
archive
new
solutions
evaluation
& selection
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Evolutionary algorithms
Genetic Algorithms
[J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press,
1975.]
Evolutionary Strategies
[I. Rechenberg. Evolutionstrategie: Optimierung Technisher Systeme nach
Prinzipien des Biologischen Evolution. Fromman-Hozlboog Verlag, Stuttgart, 1973.]
Evolutionary Programming
[L. J. Fogel, A. J. Owens, M. J. Walsh. Artificial Intelligence through Simulated
Evolution, John Wiley, 1966.]
and many other nature-inspired algorithms ...
problem-independent
reproduction
archive
new
solutions
evaluation
& selection
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Evolutionary algorithms
Genetic Algorithms
[J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press,
1975.]
Evolutionary Strategies
[I. Rechenberg. Evolutionstrategie: Optimierung Technisher Systeme nach
Prinzipien des Biologischen Evolution. Fromman-Hozlboog Verlag, Stuttgart, 1973.]
Evolutionary Programming
[L. J. Fogel, A. J. Owens, M. J. Walsh. Artificial Intelligence through Simulated
Evolution, John Wiley, 1966.]
and many other nature-inspired algorithms ...
problem-independent
reproduction
archive
new
solutions
evaluation
& selection
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Evolutionary algorithms
Genetic Algorithms
[J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press,
1975.]
Evolutionary Strategies
[I. Rechenberg. Evolutionstrategie: Optimierung Technisher Systeme nach
Prinzipien des Biologischen Evolution. Fromman-Hozlboog Verlag, Stuttgart, 1973.]
Evolutionary Programming
[L. J. Fogel, A. J. Owens, M. J. Walsh. Artificial Intelligence through Simulated
Evolution, John Wiley, 1966.]
and many other nature-inspired algorithms ...
problem-independent
reproduction
archive
new
solutions
evaluation
& selection
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Evolutionary algorithms
Genetic Algorithms
[J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press,
1975.]
Evolutionary Strategies
[I. Rechenberg. Evolutionstrategie: Optimierung Technisher Systeme nach
Prinzipien des Biologischen Evolution. Fromman-Hozlboog Verlag, Stuttgart, 1973.]
Evolutionary Programming
[L. J. Fogel, A. J. Owens, M. J. Walsh. Artificial Intelligence through Simulated
Evolution, John Wiley, 1966.]
and many other nature-inspired algorithms ...
problem-independent
reproduction
archive
new
solutions
evaluation
& selection
only need to evaluate solutions ⇒ calculate f(x) !
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Applications
Series 700
Series N700
this nose ... has been newly developed ... using the
latest analytical technique (i.e. genetic algorithms)
N700 cars save 19% energy ... 30% increase in the output...
This is a result of adopting the ... nose shape
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Applications
human designed
QHAs(human designed)
38% efficiency
2015.11.23, CAAMAS：演化学习
evolved antennas
93% efficiency
.nju.edu.cn
Evolutionary optimization v.s. machine learning
[Computing machinery and intelligence. Mind 49: 433-460, 1950.]
“We cannot expect to find a good child machine at the first attempt. One must
experiment with teaching one such machine and see how well it learns. One can then
try another and see if it is better or worse. There is an obvious connection between
this process and evolution, by the identifications
“Structure of the child machine = Hereditary material
Alan Turing
1912-1954
“Changes of the child machine = Mutations
“Judgment of the experimenter = Natural selection”
[Genetic algorithms and machine learning.
Machine Learning, 3:95-99, 1988.]
[Evolvability.
Journal of the ACM, 56(1), 2009.]
D. E. Goldberg
J. H. Holland
Leslie Valiant
ACM/Turing Award
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Evolutionary optimization v.s. machine learning
Selective ensemble:
[Z.-H. Zhou, J. Wu, and W. Tang. Ensembling neural networks: Many could be better
than all. Artificial Intelligence, 2002]
The GASEN approach
minimize the estimated generalization error
directly by a genetic algorithm
regression
classification
figures from
[Zhou et al., AIJ’02]
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Fundamental Problems Unsolved
When get the result?
How is the result?
Which operators affect the result?
...
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
A summary of our work
When get the result:
we developed a general tool for the first hitting time analysis [AIJ’08]
and disclosed a general performance bound
How is the result:
we derived a general approximation performance bound [AIJ’12]
and disclosed that EAs can be the best-so-far algorithm with practical advantages
Which operators affect the result:
we proved the usefulness of crossover for multi-objective
optimization. [AIJ’13]
on synthetic as well as NP-hard problems
And more: optimization with noisy [Qian et al., PPSN’14],
class-wise analysis [Qian, Yu and Zhou, PPSN’12], statistical view [Yu and Qian, CEC’14] ...
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Outline
- Approximation ability
- Pareto v.s. Penalty
- Evolutionary learning
- Pareto Ensemble Pruning
- Pareto Sparse Regression
- Conclusion
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
In applications...
“...save 19% energy ... 30%
increase in the output...”
“...38% efficiency ... resulted
in 93% efficiency...”
“... roughly a fourfold
improvement...”
A simple EA takes exponential time to find
an optimal solution, but
time to
find a
-approximate solution
Maximum Matching: find the largest
possible number of non-adjacent edges
2015.11.23, CAAMAS：演化学习
[Giel and Wegener, STACS’03]
.nju.edu.cn
Exact v.s. approximate
approximate optimization: obtain good enough solutions
f(x)
with a close-to-opt.
objective value
x*
x
measure of the goodness:
(for minimization)
is called the approximation ratio of x
x is an r-approximate solution
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Previous studies
On Minimum Vertex Cover (MVC)
(NP-Hard problem)
[Friedrich et al., ECJ’10]
⇒
performance
a simple EA: arbitrarily bad!
Pareto optimization: good
C
B
A
better price: A
better performance: A
price
problem-independent
reproduction
solution
multi-objective
evolutionary algorithm
a special case or a general principle?
2015.11.23, CAAMAS：演化学习
archive
new
solutions
evaluation &
selection
.nju.edu.cn
Our work
[Y. Yu, X. Yao, and Z.-H. Zhou. On the approximation ability of evolutionary
optimization with application to minimum set cover. Artificial Intelligence, 2012.]
We propose the SEIP framework for analysis approximation ratios
isolation function: isolates the competition among solutions
Only one isolation ⇒ the bad EA
Properly configured isolation ⇒ the
multi-objective reformulation
Partial ratio:
measures infeasible solutions
feasible
2015.11.23, CAAMAS：演化学习
infeasible
.nju.edu.cn
Our work
[Y. Yu, X. Yao, and Z.-H. Zhou. On the approximation ability of evolutionary
optimization with application to minimum set cover. Artificial Intelligence, 2012.]
Theorem
SEIP can find
-approximate solutions in
time
number of isolations
best conditional partial ratio in
isolations
size of an isolation
bad EAs ⇒ only one isolation ⇒ c is very large
multi-objective EAs ⇒ balance q and c
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Our work
[Y. Yu, X. Yao, and Z.-H. Zhou. On the approximation ability of evolutionary
optimization with application to minimum set cover. Artificial Intelligence, 2012.]
On minimum set cover problem
a typical NP-hard problem for approximation studies
For unbounded minimum set cover problem:
SEIP finds
-approximate
solutions in
time
n elements in E
m weighted sets in C
k is the size of the
largest set
For minimum k-set cover problem:
SEIP finds
solutions in
-approximate
time
EAs can be the best-so-far approximation algorithm
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Our work
[Y. Yu, X. Yao, and Z.-H. Zhou. On the approximation ability of evolutionary
optimization with application to minimum set cover. Artificial Intelligence, 2012.]
Greedy algorithm: bad! no better than
SEIP: for
,
time
(anytime algorithm)
approximate ratio
solutions in
-approximate
time
EAs can be the best-so-far approximation algorithm,
with practical advantages
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Outline
- Approximation ability
- Pareto v.s. Penalty
- Evolutionary learning
- Pareto Ensemble Pruning
- Pareto Sparse Regression
- Conclusion
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
For constrained optimizations
⇒
Constrained optimization:
Penalty method:
Pareto method (multi-objective reformulation):
when Pareto method is better?
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Problem Class 1
[C. Qian, Y. Yu and Z.-H. Zhou. On Constrained
Boolean Pareto Optimization. IJCAI’15]
Minimum Matroid Problem
matroid
rank:
minimum matroid optimization
given a matroid (U,S), let x be the subset indicator vector of U
e.g. minimum spanning tree, maximum bipartite matching
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Problem Class 1
[C. Qian, Y. Yu and Z.-H. Zhou. On Constrained
Boolean Pareto Optimization. IJCAI’15]
Minimum Matroid Problem
- the worst problem-case average-runtime complexity
- solve optimal solutions
For the Penalty Function Method
For the Pareto Optimization Method
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Problem Class 2
[C. Qian, Y. Yu and Z.-H. Zhou. On Constrained
Boolean Pareto Optimization. IJCAI’15]
Minimum Cost Coverage
Monotonic submodular function
minimum cost coverage problem
given U, let x be the subset indicator vector of U, given a
monotone and submodular function f , and some value
e.g. minimum submodular cover, minimum set cover
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Problem Class 2
[C. Qian, Y. Yu and Z.-H. Zhou. On Constrained
Boolean Pareto Optimization. IJCAI’15]
Minimum Matroid Problem
- the worst problem-case average-runtime complexity
- solve Hq-approximate solutions
For the Penalty Function Method
For the Pareto Optimization Method
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Outline
- Approximation ability
- Pareto v.s. Penalty
- Evolutionary learning
- Pareto Ensemble Pruning
- Pareto Sparse Regression
- Conclusion
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Previous approaches
Previous approaches
Ordering-based methods
error minimization [Margineantu and Dietterich, ICML’97]
OEP
diversity-like criterion maximization [Banfield et al., Info Fusion’05]
[Martínez-Munõz, Hernańdez-Lobato, and Suaŕez TPAMI’09]
combined criterion [Li, Yu, and Zhou, ECML’12]
Optimization-based methods
semi-definite programming [Zhang, Burer and Street, JMLR’06]
quadratic programming [Li and Zhou, MCS’09]
genetic algorithms [Zhou, Wu and Tang, AIJ’02]
SEP artificial immune algorithms [Castro et al., ICARIS’05]
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Back to selective ensemble
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
multi-objective reformulation
reduce error
reduce size
selective ensemble can be divided into two goals
Pareto Ensemble Pruning (PEP):
1. random generate a pruned ensemble, put it into the archive
2. loop
| 2.1 pick an ensemble randomly from the archive
| 2.2 randomly change it to make a new one
| 2.3 if the new one is not dominated
| | 2.3.1 put it into the archive
| | 2.3.2 put its good neighbors into the archive
3. when terminates, select an ensemble from the archive
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Back to selective ensemble
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
Pareto Ensemble Pruning (PEP):
1. random generate a pruned ensemble, put it into the archive
2. loop
| 2.1 pick an ensemble randomly from the archive
| 2.2 randomly change it to make a new one
| 2.3 if the new one is not dominated
| | 2.3.1 put it into the archive
| | 2.3.2 put its good neighbors into the archive
3. when terminates, select an ensemble from the archive
reproduction:
archive
new
solutions
evaluation
& selection
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Theoretical advantages
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
Can we have theoretical comparisons now?
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Theoretical advantages
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
Can we have theoretical comparisons now?
✓
PEP is at least as good as ordering-based methods
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Theoretical advantages
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
Can we have theoretical comparisons now?
✓
✓
PEP is at least as good as ordering-based methods
PEP can be better than ordering-based methods
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Theoretical advantages
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
Can we have theoretical comparisons now?
✓
✓
✓
PEP is at least as good as ordering-based methods
PEP can be better than ordering-based methods
PEP/ordering-based methods can be better than the direct use of
heuristic search
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Theoretical advantages
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
Can we have theoretical comparisons now?
✓
✓
✓
PEP is at least as good as ordering-based methods
PEP can be better than ordering-based methods
PEP/ordering-based methods can be better than the direct use of
heuristic search
For the first time
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Empirical comparison
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
Pruning bagging base learners with size 100
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Empirical comparison
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
Pruning bagging base learners with size 100
on error
2015.11.23, CAAMAS：演化学习
on size
.nju.edu.cn
Empirical comparison
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
On pruning different bagging size
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Application
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Ensemble Pruning. AAAI’15]
mobile human activity recognition
3 times more than the runner-up
saves more than 20% storage and
testing time than the runner-up
Compared with previous overall accuracy 89.3% [Anguita et al., IWAAL’12],
we achieves 90.2%
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Outline
- Approximation ability
- Pareto v.s. Penalty
- Evolutionary learning
- Pareto Ensemble Pruning
- Pareto Sparse Regression
- Conclusion
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Sparse regression
Regression:
Sparse regression (sparsity k):
denotes the number of non-zero elements in w
Previous methods
Greedy methods [Gilbert et al.,2003; Tropp, 2004]
Forward (FR)
Current best approximation ratio:
on R2 [Das and Kempe, ICML’11]
Forward-Backward (FoBa), Orthogonal Matching Pursuit (OMP) ...
Convex relaxation methods [Tibshirani, 1996; Zou & Hastie, 2005]
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Our approach
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Optimization for Subset Selection. NIPS’15]
multi-objective reformulation:
reduce MSE
reduce size
sparse regression can be divided into two goals
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Our approach
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Optimization for Subset Selection. NIPS’15]
reproduction
archive
new
solutions
evaluation
& selection
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Theoretical advantages
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Optimization for Subset Selection. NIPS’15]
Is POSS as good as the previously best method (FR) ?
✓
Yes, POSS can achieve the same approximation ratio
Can POSS be better ?
✓
Yes, POSS can solve exact solutions on problem subclasses, while
FR cannot
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Empirical comparison
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Optimization for Subset Selection. NIPS’15]
select 8 features, report R2 (the larger the better), average over 100 runs
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Empirical comparison
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Optimization for Subset Selection. NIPS’15]
Comparison optimization performance with different sparsities
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Empirical comparison
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Optimization for Subset Selection. NIPS’15]
Comparison test error with different sparsities
with L2 regularization
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Empirical comparison
[C. Qian, Y. Yu and Z.-H. Zhou. Pareto
Optimization for Subset Selection. NIPS’15]
POSS running time v.s. performance
best greedy performance
theoretical running time
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Outline
- Approximation ability
- Pareto v.s. Penalty
- Evolutionary learning
- Pareto Ensemble Pruning
- Pareto Sparse Regression
- Conclusion
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Conclusion
Motivated by the hard optimizations in machine learning
We did research on the theoretical foundation of
evolutionary algorithms
With the leveraged optimization power, we can solve learning
problems better
convex formulations are limited
non-convex formulations are rising
2015.11.23, CAAMAS：演化学习
.nju.edu.cn
Thank you!
[email protected]
http://cs.nju.edu.cn/yuy
Collaborators
钱超
2015.11.23, CAAMAS：演化学习
Prof. Xin Yao
周志华教授
.nju.edu.cn