Download Marjan Mernik

Exploration and exploitation in EA:
Do we completely understand
these processes
Marjan Mernik
University of Maribor, Slovenia
Shih-Hsi Liu, Barrett R. Bryant
The University of Alabama at Birmingham, USA
UNIVERSITY OF MARIBOR
AVN-8, Ljubljana, May 28, 2007
FACULTY OF ELECTRICAL ENGINEERING
AND COMPUTER SCIENCE
1/36
Outline of the Presentation
• Motivation
• Background
• How exploration and exploitation are
achieved in EAs?
• How and when exploration and exploitation
should be controlled?
• Conclusion
AVN-8, Ljubljana, May 28, 2007
2/36
Motivation
1) Misunderstanding or common beliefs
about exploration and exploitation in
EAs.
The common opinion about EAs is that they
explore the search space by the genetic
search operators, while exploitation is done
by selection. This opinion is, however,
questionable [Eiben, Schippers 1998]
AVN-8, Ljubljana, May 28, 2007
3/36
Motivation
2) To provide more complete treatment on
evolutionary exploration and exploitation.
3) Better understanding of exploration and
exploitation might help us to develop
better EAs (selection, crossover/mutation
operators and to better collaborate these
processes among each other,…).
AVN-8, Ljubljana, May 28, 2007
4/36
Background
To explore or to exploit?
This is the question now.
(anonymous EA agent)
Exploration is a process of visiting entirely new
regions of a search space.
Exploitation is a process of visiting regions of
a search space based on previously visited
points (neighborhood).
AVN-8, Ljubljana, May 28, 2007
5/36
Background
• To be successful a search algorithm needs to find
a good balance between exploration and
exploitation.
• Many researchers believe that EAs are effective
because of a good ratio between exploration and
exploitation.
GAs efficiently combine the exploration/exploitation
sense of the search so as to avoid getting trapped
into suboptimal local minima. [San Jose-Revuelta
2007]
AVN-8, Ljubljana, May 28, 2007
6/36
Background
• Balancing between exploration and
exploitation should be intelligent.
• The more intelligent the balance is, the
better results can be expected, and/or the
faster the algorithm will converge.
• One common belief is that in the beginning
EAs should start with exploration which
should gradually change into exploitation.
AVN-8, Ljubljana, May 28, 2007
7/36
How exploration and exploitation are
achieved in EAs?
• Common opinion about EAs is that they explore
the search space by crossover/mutation
operators, while exploitation is done by selection.
• In many papers too simplistic view on this subject
is perceived.
In order to optimize the efficiency and effectiveness, GAs
must maintain a balance between the exploitation of
beneficial aspects of existing solutions (by crossover) in
order to improve them, and the exploration of the solution
space (by mutation) so as to increase the probability of
finding the optimal solution [Wong 2003]
AVN-8, Ljubljana, May 28, 2007
8/36
How exploration and exploitation are
achieved in EAs?
• Selection
– A selection process drives searching towards the
regions of the best individuals. Hence, selection can be
mainly seen as exploitation operator.
– However, [Bäck 1994] showed that selection process
can control the level of exploration or exploitation by
varying selection pressure.
– Higher selection pressure pushes the search towards
more of exploitation, and lower selection pressure urges
the search towards more exploration.
AVN-8, Ljubljana, May 28, 2007
9/36
How exploration and exploitation are
achieved in EAs?
• Mutation
– A mutation operator randomly modifies individuals, with
a given probability, and thus increases the structural
diversity of a population. From this point of view, a
mutation operator is more an exploration operator.
– Conversely, a mutation operator can also be seen as an
exploitation operator, because most of the genetic
materials are preserved.
– The role of mutation is slightly different in different EAs,
too. In ES mutation is more exploration operator, while
in GA is more exploitation.
AVN-8, Ljubljana, May 28, 2007
10/36
How exploration and exploitation are
achieved in EAs?
• Crossover
– A crossover operator combines two or more parents to
generate better offspring.
– Such a combination can be derived from the idea that
the exchange of information between good individuals
will generate even better offspring.
– From this perspective, a crossover operator is more an
exploitation operator.
– However, a good crossover operator should also
generate individuals in the exploration zone.
AVN-8, Ljubljana, May 28, 2007
11/36
How exploration and exploitation are
achieved in EAs?
• As can be seen from above discussion
exploration and exploitation are achieved
by selection, mutation and crossover.
• But, it is difficult to delimit exploration
from exploitation in these processes. The
line between exploration and exploitation
is blurred!
AVN-8, Ljubljana, May 28, 2007
12/36
How exploration and exploitation are
achieved in EAs?
• But, these are not the only factors.
• Do not forget the role of population size in
evolutionary algorithms.
– With bigger population size, the search space is
more explored than with smaller population size.
• Another tricky point is representation! Is
search in EA independent of representation?
– It is important to know at what levels operators
work: at individual, sub-individual or gene level.
AVN-8, Ljubljana, May 28, 2007
13/36
How exploration and exploitation are
achieved in EAs?
• Up to now, achieving the balance
between exploration and exploitation is
managed by proper control parameter setting.
• If crossover and mutation rates are very high,
much of the space will be explored, but there
is a high probability of losing good solutions
and of failing to exploit existing schema. In
such case EA heads towards random search.
AVN-8, Ljubljana, May 28, 2007
14/36
How exploration and exploitation are
achieved in EAs?
• If crossover and mutation rates are low, the
search space is not explored, at all. In such
case EA is more close to hill climbing
algorithms.
• What control parameter settings are most
likely to produce the best results is the
question that every EA developer/user has
faced.
AVN-8, Ljubljana, May 28, 2007
15/36
How exploration and exploitation are
achieved in EAs?
• In EA community the following approaches
have been tried in the past:
– trial-and-error (time-consuming and tedious
task),
– follow general guidelines [deJong, Schaffer,
Harik] (often not applicable for specific case)
– use parameterless EA [Harik] (robust but not
efficient)
AVN-8, Ljubljana, May 28, 2007
16/36
How exploration and exploitation are
achieved in EAs?
– use experience from previous similar
applications (not applicable when problem is
tackled for the first time or such experience is
not available)
– use mathematical models (while important,
often too simple to be realistic or hard to
understand by ordinary users)
– meta-evolutionary approaches (lets evolution
find good parameters; evolution of evolution;
too slow for most of the problems)
AVN-8, Ljubljana, May 28, 2007
17/36
How exploration and exploitation are
achieved in EAs?
• To make the problem even harder, different
values of control parameters might be
optimal at different stages of the evolutionary
process.
• In [Eiben 1999] an excellent overview of this
problem has been given, where the authors
distinguish between:
– parameter tuning, and
– parameter control (deterministic, adaptive, selfadaptive)
AVN-8, Ljubljana, May 28, 2007
18/36
How exploration and exploitation are
achieved in EAs?
• It is important to note that with different
setting of control parameters (before or
during the run) we just explicitly control
specific processes (selection, mutation, and
crossover) but only implicitly exploration
and exploitation.
AVN-8, Ljubljana, May 28, 2007
19/36
How exploration and exploitation are
achieved in EAs?
• Regarding the question how the balance
between exploration and exploitation is
achieved we classified approaches into:
– uni-process driven approach, and
– multi-process driven approach.
AVN-8, Ljubljana, May 28, 2007
20/36
How exploration and exploitation are
achieved in EAs?
• In the uni-process driven approaches a
good balance between exploration and
exploitation is achieved separately in each
operator (e.g., selection, mutation,
crossover).
• Each process is responsible for this balance
and there is no coordination between
processes to achieve the balance.
• Most work was done in this area!
AVN-8, Ljubljana, May 28, 2007
21/36
How exploration and exploitation are
achieved in EAs?
• In the multi-process driven approaches
the balance is coordinated among different
processes (e.g., with high selection pressure
such mutation/crossover operator is chosen
that is biased more to exploration).
– An early example of multi-process driven
approach is CHC [Eshelman 1991] which
combines selection that always preserve the best
individuals so far (exploitation) with crossover
operator that produces maximally different
individuals (exploration).
AVN-8, Ljubljana, May 28, 2007
22/36
How exploration and exploitation are
achieved in EAs?
– Another good example is reported in [Fonseca
1995] where authors found that proper
collaboration among selection (fitness sharing)
and crossover (mating restriction) significantly
improves performance of multiobjective GA
(MOGA).
AVN-8, Ljubljana, May 28, 2007
23/36
How and when exploration and exploitation
should be controlled?
• Balance should be controlled during the run
(due to success of parameter control over
parameter tuning), but the tricky issue is on
what occasions.
– Here again some deterministic schemas (e.g.,
every k generations, more often at the later
stages) or
– adaptive schemas (e.g., when best fitness did
not change for several generations, when
diversity of population drop under some
threshold value) can be applied.
AVN-8, Ljubljana, May 28, 2007
24/36
How and when exploration and exploitation
should be controlled?
• In what order the phases of exploration
and exploitation should appear?
• Intuitively, phase of exploration should be
followed by exploitation.
• But, currently this is not under user control
and phases of exploration and exploitation
are interleaved (an exception is work of
[Ursem 2002]).
AVN-8, Ljubljana, May 28, 2007
25/36
How and when exploration and exploitation
should be controlled?
• But, before control we need to know how
to measure it.
• How to measure exploration and
exploitation is an open question in EAs, as
far we are aware of.
AVN-8, Ljubljana, May 28, 2007
26/36
How and when exploration and exploitation
should be controlled?
• What measures (direct or indirect) can be
used for exploration and exploitations?
– diversity (a good indirect measure most of the
researchers used today)
– entropy, which is very succinct measure for
diversity; distribution of values is also included
in this measure (precise distribution is not
known)
– search space maps (direct, but currently almost
not existent)
AVN-8, Ljubljana, May 28, 2007
27/36
How and when exploration and exploitation
should be controlled?
• What is diversity?
• Diversity refers to differences among
individuals, which can be at genotype or
phenotype level.
• It is widely accepted in EA community that
diversity of the population greatly
contributes to EA performance.
AVN-8, Ljubljana, May 28, 2007
28/36
How and when exploration and exploitation
should be controlled?
• Usual genotype measures:
– Euclidean distance [de Jong 1975]
– Hamming distance [Whitley 1990]
– bias of population [Grafenstette 1991]
– subtree variety (in GP) [Kaijzer 1996]
– number of different genotypes [Langdon 1998]
– nodes and subtrees sharing [McPhee 1999]
– edit distance [de Jong 2001]
– distance to average point [Ursem 2002]
AVN-8, Ljubljana, May 28, 2007
29/36
How and when exploration and exploitation
should be controlled?
• Usual phenotype measures:
– number of different phenotypes [Rosca]
– behaviour signatures [D'haeseleer 1994]
– number of individuals in the neighborhood
hypercube [Tsutsui 1997 – Forking GA]
– average distance between the non-dominated
solutions to the true Pareto front [Zitzler 2000]
AVN-8, Ljubljana, May 28, 2007
30/36
How and when exploration and exploitation
should be controlled?
• We classified current EAs into following
classes:
– Maintaining diversity
– Guided by Diversity
– Learning (Tracking History of) Diversity
AVN-8, Ljubljana, May 28, 2007
31/36
How and when exploration and exploitation
should be controlled?
• Maintaining diversity
– Fitness sharing [Holland 1975, Goldberg &
Richardson 1989]
– Crowding [DeJong 1975, Mahfoud 1992],
– neighbourhoods [Collins 1992] and islands
[Martin 2000]
– mating restrictions [Deb&Goldberg 1989]
– Forking GA [Tsutsui 1997]
– Bi-population GA (bGA) [Tsutsui 1997]
AVN-8, Ljubljana, May 28, 2007
32/36
How and when exploration and exploitation
should be controlled?
• Guided by Diversity
– Adaptive genetic algorithm (AGA) [Srinivas
1994]
– DCGA [Ursem 2002]
– The PRAM algorithm [Wong 2003]
– Diversity-guided micro-genetic algorithm
(DGGA) [San Jose-Revuelta 2007]
– Entropy-based exploration and exploitation
[Liu, Mernik, Bryant 2007]
AVN-8, Ljubljana, May 28, 2007
33/36
How and when exploration and exploitation
should be controlled?
• Learning (Tracking History of) diversity
– stimulus-response map [D'haeseleer 1994]
– genetic lineages [McPhee 1999] where number of
individuals in the initial population contributing genetic
material to the final population are tracked
– GASOP (GA using Self-Organizing Maps) [Amor 2005]
– learning approach for classifying clusters in fitness class
identification [Liu, Mernik, Bryant 2007]
– Some new ideas (identifying not-yet-explored regions
and appropriately adopt crossover/mutation and
selection to explore these regions; tracking history of
explored regions to avoid visiting them again)
– ???
AVN-8, Ljubljana, May 28, 2007
34/36
Conclusion
• Exploration and exploitation are
fundamental concepts of any search
algorithms and it is surprisingly that these
concepts are not better understand in EA.
AVN-8, Ljubljana, May 28, 2007
35/36
Conclusion
• Overall, many questions remain unanswered:
– Should these phases alternate in each generation?
– In what order and how often they should appear in
consequence.
– Should they be present in each generation?
– Should all parameters and whole population be
preserved when we switch from one phase to the
other?
– Are different factors (e.g., diversity) equally important
in both phases?
– How different is the role of crossover/mutation
regarding exploitation/exploration in different EAs.
AVN-8, Ljubljana, May 28, 2007
36/36