Download Evolutionary Learning

Evolutionary Computation
and beyond
Elhanan Borenstein
May. 2004
www.cs.tau.ac.il/~borens
Evolution vs. Learning
Evolution
Learning

Process:
Random (?)
Directional (?)

Level:
Population
Individual

Entity:
Genotype
Phenotype

Time Scale:
Generations
Lifetime

Operators:
Mutation, Crossover
Gradient Descent

Search:
Global
Local

Problems:
Slow / Random
Initial conditions
Talk Outline

Evolution – An Inevitable Process

Genetic Algorithms


Motivation

Techniques & Pseudo Code
Evolution as a Stochastic Search Process

Fitness Landscapes

Enhanced Genetic Algorithms

Artificial Life and Evolutionary Autonomous Agents

Life as it could be…

Definition & Motivation

Putting is All Together

Experiments Demo
Evolution – An inevitable Process
Evolution – An inevitable Process

Any system, that embeds the following three processes,
will inevitably undergo evolution:

Reproduction
(inheritance)

Initial Population
Next
Generation
Selection

Variation
Reproduction

Selection
Variation
A process of selective reproduction and substitution in a
population of individuals.
Evolution – An inevitable Process
Just a few examples may include:
Domestic
Animals
Breeding

Life on Earth

Strategies
(Financial…)

Ideas, Beliefs,
knowledge

Evolutionary
Game Theory

culture

memes

Evolutionary Computation

If it works so well in nature… why not using it to
solve computational problems…

Many problems in computer science (and mathematics)
have the following characteristics:


We are searching for the optimal solution…

The number of possible solutions (search space) is huge.

We do not have an analytical method to reach the optimal
solution…

but… given a solution, we can evaluate its quality.
Evolution is a search process !!!
For Example …
Genetic Algorithms

Each possible solution will be represented (coded) as a
string of characters (genome)
For example:
LPMRSBN (London Paris  Madrid  Rome  Lisbon  Berlin  Monaco)

Generate a random collection (population) of solutions:
LPMRSBN, MSBRNLP, NBSMPLR, MRLNSBP, …

Evaluate the quality (fitness) of each solution

Generate the next generation of solutions:


Delete (kill) all bad solutions.

Duplicate (reproduce) a number of copies of each remaining
(surviving) solution.

Insert random modifications (mutations) to the newly create solutions.
Repeat (life goes on) until the optimal solution was found.
Genetic Algorithm – Pseudo Code
Initialize population: P  Generate p hypotheses at random
Evaluate: For each h in P, compute Fitness(h)
While [max Fitness(h) < threshold] do
Select: Probabilistically select p/2 pairs of hypotheses from P.
Pr(hi) = Fitness(hi) / Σ Fitness(hi)
Reproduction: Produce two offspring for each pair. Add to Ps
With probability pc apply crossover operator
mutate: For each gene, with probability pm, apply mutation operator
Update: P  Ps
Evaluated: For each h in P, compute Fitness(h)
return
Genetic Algorithm – Pseudo Code
Initialize population: P  Generate p hypotheses at random
Evaluate: For each h in P, compute Fitness(h)
While [max Fitness(h) < threshold] do
Select: Probabilistically select p/2 pairs of hypotheses from P.
Pr(hi) = Fitness(hi) / Σ Fitness(hi)
Reproduction: Produce two offspring for each pair. Add to Ps
With probability pc apply crossover operator
mutate: For each gene, with probability pm, apply mutation operator
Update: P  Ps
Evaluated: For each h in P, compute Fitness(h)
return
Population

Genotype vs. Phenotype

Representation
2.31

1.02
-0.28
-7.93
6.32
Coding

Direct

Developmental

Stochastic
1.87
-1.72
2.46
0.45
011
101
100
111
001
010
011
111
110
Genetic Algorithm – Pseudo Code
Initialize population: P  Generate p hypotheses at random
Evaluate: For each h in P, compute Fitness(h)
While [max Fitness(h) < threshold] do
Select: Probabilistically select p/2 pairs of hypotheses from P.
Pr(hi) = Fitness(hi) / Σ Fitness(hi)
Reproduction: Produce two offspring for each pair. Add to Ps
With probability pc apply crossover operator
mutate: For each gene, with probability pm, apply mutation operator
Update: P  Ps
Evaluated: For each h in P, compute Fitness(h)
return
Evaluation – The fitness function

Reliable assessment

Average fitness

Scaling (rank-based, linear, quadratic, Baltzmann)

Tournament
Genetic Algorithm – Pseudo Code
Initialize population: P  Generate p hypotheses at random
Evaluate: For each h in P, compute Fitness(h)
While [max Fitness(h) < threshold] do
Select: Probabilistically select p/2 pairs of hypotheses from P.
Pr(hi) = Fitness(hi) / Σ Fitness(hi)
Reproduction: Produce two offspring for each pair. Add to Ps
With probability pc apply crossover operator
mutate: For each gene, with probability pm, apply mutation operator
Update: P  Ps
Evaluated: For each h in P, compute Fitness(h)
return
Termination Criteria

Maximal Fitness

Convergence of the population

No improvement

Maximal # of generations
Genetic Algorithm – Pseudo Code
Initialize population: P  Generate p hypotheses at random
Evaluate: For each h in P, compute Fitness(h)
While [max Fitness(h) < threshold] do
Select: Probabilistically select p/2 pairs of hypotheses from P.
Pr(hi) = Fitness(hi) / Σ Fitness(hi)
Reproduction: Produce two offspring for each pair. Add to Ps
With probability pc apply crossover operator
mutate: For each gene, with probability pm, apply mutation operator
Update: P  Ps
Evaluated: For each h in P, compute Fitness(h)
return
Selection (survival of the fittest)

A stochastic process !!!

Roulette wheel selection

Tournament

Competition

Elitism
Genetic Algorithm – Pseudo Code
Initialize population: P  Generate p hypotheses at random
Evaluate: For each h in P, compute Fitness(h)
While [max Fitness(h) < threshold] do
Select: Probabilistically select p/2 pairs of hypotheses from P.
Pr(hi) = Fitness(hi) / Σ Fitness(hi)
Reproduction: Produce two offspring for each pair. Add to Ps
With probability pc apply crossover operator
mutate: For each gene, with probability pm, apply mutation operator
Update: P  Ps
Evaluated: For each h in P, compute Fitness(h)
return
Reproduction

Sexual vs. Asexual

Haploid vs. Diploid
Genetic Algorithm – Pseudo Code
Initialize population: P  Generate p hypotheses at random
Evaluate: For each h in P, compute Fitness(h)
While [max Fitness(h) < threshold] do
Select: Probabilistically select p/2 pairs of hypotheses from P.
Pr(hi) = Fitness(hi) / Σ Fitness(hi)
Reproduction: Produce two offspring for each pair. Add to Ps
With probability pc apply crossover operator
mutate: For each gene, with probability pm, apply mutation operator
Update: P  Ps
Evaluated: For each h in P, compute Fitness(h)
return
Genetic Operators - Variation

Mutation:
011 101 100 111 001 010 011 111 110
011 101 101 111 001 010 011 111 110

Crossover (one point/two points):
011 101 100 111 001 010 011 111 110
110 100 110 000 100 101 100 111 001
011 101 100 111 001 010 100 111 001
110 100 110 000 100 101 011 111 110
Genetic Algorithm – Pseudo Code
Determining which of the numerous
extensions and variations of Genetic
Algorithms matches a specific task is
still very much a matter of art and
experience.
Genotype/Phenotype Coding


A few notes to bear in mind about genetics and
evolutionary computation:

Most evolutionary computation applications, implement some sort
of distinction between genotypes and phenotypes, in an
analogous manner to biological evolution.

While the genotype is the one that transferred across
generations, it is the phenotype that goes trough the selection
process.

We work mostly in a strict Darwinian framework, where lifetime
adaptations are not inherited.
The Genotype to Phenotype coding may
have a dramatic effect on the
evolutionary search !!!
Evolution – A Stochastic Search
The Fitness Landscape
Fitness Value
Optimal Solution
Evolution – A Stochastic Search
Fitness Value
Illustration – Initial Population
Evolution – A Stochastic Search
Fitness Value
Illustration – Natural Selection
Evolution – A Stochastic Search
Fitness Value
Illustration – Reproduction & Variation
Evolution – A Stochastic Search
Fitness Value
Illustration – The Next Generation
Evolution – A Stochastic Search

Which properties of the fitness landscape affect the
evolutionary search?

Size!!!

Incremental

Ruggedness

Local optima

Size of basins of attraction
B
A
Genetic Configuration
C
Genetic Configuration
D
Genetic Configuration
Genetic Configuration
Evolution – A Stochastic Search

Can we control these properties?

Yes (to some degree) !!!



Genetic Representation

Dimensionality

Neighborhood function

Building blocks

Stochastic encoding
Fitness Function

Smooth

Monotone / incremental

Resolution
Much is still dependent on the Task !!!
Genetic Configuration
Enhancing Genetic Algorithms

How can we push the evolutionary process forward?


Taking another lesson from nature
Co-evolution:

host-parasite / predator-pray

Red queen effect (“You have to run faster and faster just to stay
in the same place!”).

Incremental evolution

Lifetime learning:

Darwinian framework vs Lamarckian framework.

Social Learning

Cultural evolution
Life As It Could Be
A Short Introduction to
Artificial Life
and
Evolutionary Autonomous Agents
A Multidisciplinary Research
Biology
Computer Science
Neuroscience
Mathematics
Physics
Robotics
Artificial
Life
Psychology
Animal Behavior
Medical Research
Artificial Life (ALife) - Definition
“Artificial Life (ALife) is the study of man-made
systems that exhibit behaviors characteristic of
natural systems.
It complements the traditional biological science concerned with the analysis
of living organisms by attempting to synthesize lifelike behaviors within
computers and other artificial media.
…by extending the empirical foundation upon which biology is based beyond
the carbon-chain life that has evolved on Earth…
ALife can contribute to theoretical biology by
locating life-as-we-know-it within the large picture
of life-as-it-could-be.”
Chris Langton
Artificial Life (ALife) - Motivation
ALife motivation is twofold:
Computational
Create better
artificial systems
that are inspired by
biological systems
Biological
Gain a better
understanding of
biological systems
Modeling Complex Systems


What advantages are there in artificial models?

Full access to all data (all vs. tens, all vs. fossil record)

Time resolution (milliseconds vs. years)

Intervention (all and any vs. few)
But…will our findings in such models (systems) apply to
real natural systems?

How accurate should our model be?

Which parameters should be included?
:‫על הדיוק במדע‬
‫ שמפתו של מחוז אחד‬,‫המפות למידת שלמות כזו‬-‫ באימפריה זו הגיעה אומנות כתיבת‬...
‫ ברבות‬.‫השתרעה על פני עיר שלמה ואילו מפתה של האימפריה כולה – על פני המחוז כולו‬
‫ענק אלה את התושבים וועדות כותבי המפות החלו להכין את‬-‫הימים לא סיפקו עוד מפות‬
...‫ והיא זהה עמה בכל נקודה ונקודה‬,‫מפת האימפריה שגודלה כגודל האימפריה עצמה‬
‫חורחה לואיס בורחס‬
Artificial Neural Networks (ANN)

Artificial Neural Network:
w6
w1
w21
w2
Input
w3
Output
w8

Such neural network can be thought of as a “black-box”
that compute some function of the input. Different
assignment of the synaptic weights will result in a
different function.
Autonomous Agents
w
w
2 w
1 2
6
w
1
w
3
w
8

Such artificial creatures may be either real autonomous
robots or simulated autonomous agents.
Developing the Agent’s Brain

So…we have manage to construct a good model for a
neural-network brain, controlling an artificial agent…

But…we are back to square one!!! How will we develop
the agent’s brain and get “working” and interesting
brain models?

There are two approaches we can take (both inspired
again by biological process).

Lifetime Learning

“Genetic” Evolution
Evolutionary Autonomous Agents
Create a first
random generation
Insert the “brain”
into an agent and
let it operate in
the world
100 brains
(random genome)
Evaluate each
“brain” in the
(virtual) world
fitness value
Allow the
successful agent to
reproduce
mutations,
crossovers
A new generation is
created
Evolutionary Autonomous Agents
Agents developed by
evolution and imitation
>>
Evolutionary Autonomous Agents
Agent in a fire-fighting contest
Evolutionary Autonomous Agents

One of the main objectives of our study is to develop
methodologies for neuroscience research.

We thus need to develop more complex / interesting
brains (neural networks) !!!
Complex Task
Evolutionary Autonomous Agents
Evolutionary Autonomous Agents

How would you
solve this task?
Evolutionary Autonomous Agents
S10: a forager-explorer agent
S10: a forager-explorer agent
Generation 200
Generation 900
>>
>>
Emergent Phenomena !!!
Evolutionary Autonomous Agents

Analyzing the emergent neural network:
S10: a forager-explorer agent
Command neuron clamp to 0
>>

Much work is done to put forward analytical methods
that could be applied to biological neuroscience
research !!
Summary

Evolutionary computation is a powerful tool that can be
used to “solve” a wide range of tasks.

It is still hard to provide rules of thumb to match the
numerous extensions and variations of Genetic
Algorithms to a specific task.

Artificial Life and biological system modeling (in-silico
biology) is a growing and exiting field that can shed new
light on natural biological systems.
Questions
?