Download Computational Intelligence and Games

International Journal of Automation and Computing
05(1), January 2008, 45-57
DOI: 10.1007/s11633-008-0045-8
Computational Intelligence and Games:
Challenges and Opportunities
Simon M. Lucas
Centre for Computational Intelligence, Department of Computing and Electronic Systems, University of Essex, Colchester CO4 3SQ, UK
Abstract: The last few decades have seen a phenomenal increase in the quality, diversity and pervasiveness of computer games. The
worldwide computer games market is estimated to be worth around USD 21bn annually, and is predicted to continue to grow rapidly.
This paper reviews some of the recent developments in applying computational intelligence (CI) methods to games, points out some of
the potential pitfalls, and suggests some fruitful directions for future research.
Keywords:
1
Games, machine learning, evolution, temporal difference learning (TDL), neural networks, n-tuple systems.
Introduction
Games have long been seen as an ideal test-bed for the
study of artificial intelligence (AI). Early pioneers of computing and information theory such as Alan Turing and
Claude Shannon were intrigued by the idea that computers
might one day play grand-master level chess, and sketched
out ideas of how this might be achieved. Much of the academic work on AI and games has focused on traditional
board games until recently, with the challenge being to beat
humans at a variety of challenging games. This strand of
work has been successful in achieving this goal for many
complex games, such as checkers and chess. Checkers is now
a solved game[1] , the result being a draw if both players play
optimally. Chess is far from being solved, but computers
play at world-leading level. Checkers and chess playing algorithms depend on efficient game-tree search coupled with
a heuristic board evaluation function, which may be handprogrammed or learned. In AI research, the emphasis is on
producing apparently intelligent behaviour using whatever
techniques are appropriate for a given problem. In computational intelligence (CI) research, the emphasis is placed
on intelligence being an emergent property. Examples of
CI methods include evolutionary computation, where intelligence may arise from due to principles of natural selection,
and neural networks, where intelligence may arise from the
collective connected behaviour of many simple neurons.
Both checkers and chess have seen the successful application of CI methods, with Fogel and Chellapilla0 s famous work on Blondie 24[2] (checkers) and Fogel et al.[3] on
Blondie 25 (chess). Both used a fairly standard minimax engine, but instead of using a hand-coded heuristic evaluation
function, used a multi-layer perceptron (MLP). The MLP
had specially designed input layers with spatially arranged
inputs covering all possible sub-squares of the 8 × 8 board.
While the architecture of the MLP was fixed, the weights
were evolved using co-evolution. Essentially, a (15 + 15)
evolution strategy (ES) was used, but the fitness function
was based on how well each network played against a set
of opponent networks drawn from the current population.
Manuscript received September 24, 2007; revised November 7, 2007
E-mail address: [email protected]
The Blondie systems achieved high levels of play both at
checkers and at chess, and have acted as a great inspiration
to many researchers. However, due to the way the MLP
was embedded in a minimax engine, it is unclear how much
of the playing ability comes from the neural network, and
how much from the minimax search.
Other games such as Go have been less susceptible to
this combination of game tree search and heuristic position
evaluation, due to the large branching factor and the difficulty of designing a useful position evaluation function. Go
remains a grand challenge for computer science research,
though very recently Monte Carlo methods have shown
great promise, especially for smaller versions of the game
such as 9 × 9 (standard Go is played on a 19 × 19 board).
While traditional game-tree methods search broad but shallow, Monte Carlo methods are relatively narrow but deep.
A position is evaluated by playing it many thousands of
times using random moves until the end of each game is
reached (at which point the true value of that particular
line of play is known for sure). Each position is then characterized by the probability of winning from that position,
based on those random play outs. Successful Go programs
such as MoGo[4, 5] use a more sophisticated version of this
called UCT Monte Carlo (UCT stands for upper confidence
bounds applied to trees). As each games is played out, common initial moves are shared in a tree structure so that each
line of play updated the statistics for all the tree-nodes that
led to it. More attention is given to playing out promising
nodes in the tree, but this is also balanced by the need for
exploration. UCT Monte Carlo has achieved remarkable results in Go, and the method is a testament to the power of
statistical techniques in general. It is reminiscent of the success of statistical language processing methods compared to
their more grammar-based alternatives[6] . It is likely that
UCT Monte Carlo also has significant potential to be applied to other games. While the branching factor of Go is
much larger than chess, it is still of a different nature to that
encountered in video games, with their virtually continuous
state-spaces.
While the traditional focus of academic research has been
on board games, and this research continues to flourish,
there is also a growing body of work on applying AI and
46
International Journal of Automation and Computing 05(1), January 2008
CI to video games, and many techniques are applicable to
any game regardless of genre, though may require careful
application to get best results. For example, it seems likely
that UCT Monte Carlo may have important application to
control non-player characters (NPCs) in video games, but
although the concept is quite simple, getting the best out of
the method has been the subject of intensive research over
the last few years. In the UK, the Engineering and Physical
Sciences Research Council (EPSRC) has just funded a network grant to foster closer collaboration between industry
and academia in the area of AI and games[7] .
Commercial games are based on increasingly sophisticated simulated environments, and present human players
with world views approaching photo-realism. As virtual
worlds become more complex, it becomes harder to NPCs
with sufficiently interesting and believable behaviour. Current NPC control techniques mostly involve methods such
as finite state machines, scripting, and search. Most of the
behaviour is hand-programmed, involving much human effort to code and to test. Although games companies have
occasionally used AI techniques such as genetic algorithms
and neural networks[8, 9] , the methods used in games often lag behind the state of the art in machine learning.
Similarly, machine learning and AI have much to benefit
from games by applying and extending algorithms to more
challenging environments than are usually encountered[4] .
Furthermore, as games utilise more realistic physics engines, there is scope for greater synergy between games and
robotics research: the control of a humanoid robot, or of
an NPC may eventually share much in common. One challenge for research in this area is to find sufficient common
ground to make collaboration attractive for both parties.
This is because while academics may be most interested in
how smart an NPC is, or how well it learns from its environment, a games company is in the business of selling games,
and is therefore likely to be most interested in how intelligent NPC behavior can be channeled into making games
more enjoyable. Many academics are now eager to test their
methods in the complex virtual worlds of modern console
games, and guided by industrial collaboration. This will
bring great benefits both to the research community and to
the games industry.
Peter Molyneux OBE, an internationally renowned game
designer recently commented: “AI is certainly the undiscovered country of games design. Any game genre – from
hardcore shooters to the most story-driven adventure game
– would be truly revolutionized by AI driving plot, characters and scenarios.”
On the other hand, there is still some resistance to the
more open-ended game play that advanced AI and CI techniques could facilitate. This comment on a blog site1 expresses a view held by many games industry insiders: “I
think the most drastic different in the mind set of an AI
programmer/designer in big budget game and an academic
is that in a real game project you really need to think about
the production issues (anything unpredictable is no-no) . . .”
By their very nature, however, next generation video
games will be more open-ended, and therefore unpredictable. This is in the same way that multi-player on1
Memoni, http://realtimecollisiondetection.net/blog/?p=36, October 13, 2007
line games are more open-ended and less predictable due to
the intelligence and variable behaviour of the other human
players.
The views of game-industry insiders such as these cannot
be simply dismissed. However, it is likely that their objection to more open-ended AI is based on what seems to be
currently achievable, rather than what might one day be
achieved. Hence it is not unpredictable behaviour that is
unacceptable, but annoying or uninteresting behaviour.
As NPC behaviour becomes more intelligent, and NPCs
are better able to learn, this opens the way for new genres
of game. For instance, the NERO game[10] involves players
pre-training characters to act according to a desired tactical
doctrine. This is achieved by NERO using a neuro-evolution
approach internally to evolve soldier behaviours that satisfy
the given training tasks. Fig. 1 shows a screenshot from the
game.
Fig. 1
A screenshot from the NERO game
NERO presents an example of the open-ended game-play
that CI methods enable, and one can expect to see many
new games that exploit CI in various ways.
Games have also been studied from a social, economics,
and financial perspective. A classic social game is the iterated prisoner0 s dilemma (IPD)[11] , which has led to many CI
studies of evolving and co-evolving strategies for this game.
CI methods have also been applied to many aspects of finance, such as supply-chain management[9] and sequential
bargaining[12] .
CI methods have been used not only to optimize agent
strategy, but also to design marketplace rules. A good example of this can be found in the work of Dave Cliff. He
noted that Gode and Sunder[13] had found that stochastic agents with zero intelligence (ZI) could approximate
the equilibrium price when trading in a suitably structured
market. Cliff[13] showed that the ZI agents could vary significantly from this performance when the market structure
was varied from the Gode and Sunder[14] setup. To overcome this, Cliff added a small amount of intelligence to the
agents (hence ZI plus (ZIP)), endowing each one with a
simple learning rule that adjusted their behaviour based on
observations of the last transaction. The ZIP agents were
shown to better approximate human behaviour than the ZI
ones.
S. M. Lucas / Computational Intelligence and Games: Challenges and Opportunities
Cliff then reasoned that if such simple agents could perform well, they could be used within a fitness function for
an evolutionary algorithm where the aim was to optimize
the design of the marketplace[15] . This is a great example of
the creative application of CI. In game terms, this is using
evolution to design the game rather than agents to play the
game.
2
Playing games with CI
This section describes the two main ways in which a neural network or fuzzy system can be interfaced to a game.
These are as an action selector, or as a value function.
As an example, we consider the Mountain Car task introduced by Moore[16] , and also described in [17, pp. 214].
The problem is illustrated in Fig. 2. The task is to drive a
car to reach a goal at the top of the hill on the right (the
rightmost vertical line), but the engine force is weaker than
gravity. There is a barrier to the left (the leftmost vertical line) which the car stops dead at. At each time step,
there are three possible actions: accelerate left (at = −1),
accelerate right (at = 1), or apply no force (at = 0). The
car engine has insufficient power to overcome gravity, so the
problem is a bit deceptive: to reach the goal as quickly as
possible, it may be necessary to accelerate (apply a force)
away from the goal in some circumstances. The following
equations specify the update rules for position (s) and velocity (v).
47
generate a set of future states. Each future state is then
evaluated by the value function. The action is then chosen
that would lead to the state with the highest value. Fig. 3
shows that for this example, the state reached by taking
the accelerate right action has the highest value, and so
that would be the chosen action. Value function methods
are very flexible because more computation can be applied
to look further into the future, and hence get a truer picture
of which actions lead to the best states in the long run.
Fig. 3 Illustration of the value function applied to the Mountain
Car problem
To interface an action selection network to this, the inputs to the network are the current state of the system
(defined by the car0 s current position and velocity), and
the output must be interpretable as one of the possible actions. For complex action sets, one might use one output
per possible action, and take the action associated with the
highest output value as in Fig. 4.
Fig. 2
The Mountain Car problem
st+1
=
bounds (st + vt+1 )
(1)
vt+1
=
boundv (vt + 0.001at − 0.0025cos(3st )). (2)
The bounds for s and v are as follows:
−1.2
≤ st+1
≤ 0.5
(3)
−0.07
≤ vt+1
≤ 0.07.
(4)
Each epoch (trial) starts with the car having a random
position and velocity drawn uniformly within the above
bounds. Each trial terminates when the car reaches the
goal, or after 2 500 steps. The best solutions for this particular configuration of the problem solve the task in approximately 52 steps on average (each step has a cost of 1, so
this is referred to as a score of −52).
The value function method has been a popular way to
solve this, and is depicted in Fig. 3. All possible actions
are applied to a model of the current state of the car to
Fig. 4 Illustration of the action selection method applied to the
Mountain Car problem
For this case, there are three possible outputs that have
a naturally ordered relationship, so a natural way to code is
to have a single output unit, and define the output intervals
to represent left, neutral, and right.
48
International Journal of Automation and Computing 05(1), January 2008
The mountain problem was considered to be reasonably
hard for neural networks to solve, and was used as a benchmark by Whiteson and Stone[18] as an initial test for their
NEAT+Q evolutionary/temporal difference hybrid learning algorithm, where NEAT is an abbreviation for neuroevolution with augmenting topologies, and +Q denotes the
addition of Q-learning, a particular type of temporal difference learning (TDL).
Running TDL with a MLP (or a CMAC architecture)
may produce value functions that have a meaningful interpretation, in this case, the value of each state is the negative of the number of steps that the agent should take
to reach the goal from that state. Whiteson and Stone[18]
solved the problem with a combination of neuro-evolution
and Q-learning (a kind of TDL). They noted that the value
functions produced by NEAT+Q were very different from
the TDL ones, yet the performance of these functions was
very similar.
Either TDL or evolution can train a network to perform
well at this task, and take hundreds (or even thousands) of
epochs or fitness evaluations. The value function learned by
TDL using a grid-based decomposition of the input space
and after 3000 epochs is shown in Fig. 5. It is shown that
the value had an average score of -61. The performance
is similar but inferior to the results reported for CMAC
architectures which use perturbed overlapping grids, but
illustrates how TDL attempts to learn a meaningful value
function.
interpretation of the output explained above, optimal solutions could be tanh(wv) where w is a single weight with a
high value such as 10, enough to make the neuron behave
like a sign function sgn(v).
For the value function, abs(v) is near-optimal, as shown
in Fig. 6, which assigns the value of a state to the absolute
value of the velocity in that state. It is also worth noting
that genetic programming would be expected to produce
solutions for this problem with relative ease, especially if
an abs(·) function was included in the terminal set.
Fig. 6
There are two important points raised here. First, the
problem setup is critical. The problem can be made easy or
hard depending on the details of the setup (input coding,
action versus state value function). The second point that
leads from this is that one should be wary of drawing any
conclusions from tests on problems that have such simple
solutions.
3
Fig. 5
A value function learned with TDL after 3000 epochs
However, as we can see, the problem can be made trivial depending on how it is presented to the learner. While
experimenting on this problem, it was observed that the
value function could look very different from run to run,
while generally converging to networks with high performance. Furthermore, it was observed that when the inputs
used were the square of the velocity and the position, it
was easily solved by a single layer perceptron network, with
all the best solutions tending to ignore the position, and
strongly weigh the velocity. This led to the hypothesis that
only the absolute value of the velocity matters for this problem, and this was confirmed empirically, with abs(v) having
an average score of -51.97 (standard error = 0.09) based on
100 000 trials. This compares favourably with the Whiteson and Stone[18] quoted averages of 52.75 (NEAT+Q) and
52.02 (Sarsa + CMAC), though NEAT+Q took hundreds
of thousands of episodes to reach that performance.
This leads to two very simple near optimal solutions, depending on whether a state value or direct action method
is used. For an action selection network with the ordered
A near-optimal solution
Learning game strategy
For many real-world game applications, it is vital that
learning can happen quickly. The main methods for game
strategy learning are TDL and evolution (including coevolution). From previous works of the author and others, it seems that these methods have significantly different
strengths and weaknesses. When it works, TDL often learns
much more quickly than evolution. This is not surprising
in one sense, since TDL learns a value function that is constantly updated during game-play. However, note that for
the most part this is done in an unsupervised way; it simply learns that game states that are close in time should
have similar values. At certain points in the game (which
points depends on the game; typically the end of the game
in the case of board games), the true game-state is known,
and over the course of training, this ties the value of states
close to those states, and so on. The problem is that for
non-trivial games, and for complex function approximators
such as MLPs, there is no guarantee that the value function
will converge to a reasonable value (let alone optimal).
Until recently, there had been very little work comparing
TDL with evolution. With recognition of the importance
of such comparisons, the body of work is growing steadily.
Previous comparisons deal mostly with learning evaluation
functions for board games.
An influential early experiment on using TDL to learn
board game evaluation functions is due to Tesauro[19] ,
who achieved world-class performance when training neural
network-based Backgammon evaluation functions with self-
S. M. Lucas / Computational Intelligence and Games: Challenges and Opportunities
play. Pollack and Blair[20] tried training evaluation functions using the same game and function representation, but
instead of TDL they used the simplest possible form of evolutionary algorithm (EA), a hill-climber where a single individual was repeatedly mutated, and the mutation was kept
only if it won a number of games over the non-mutated individual. The algorithm worked, but its end results were
inferior to those of Tesauro[21] .
Darwen[22] did a set of similar comparisons for backgammon, and found that a population-based EA eventually outperformed TDL when training a linear board evaluation
function, even though the EA was much slower. However,
when training a nonlinear evaluation function, board evaluators trained by the EA never reached the performance of
those trained by TDL for the simple reason that the EA
took too long; to be effective, the EA needed to evaluate
the same pair of individuals many times.
Runarsson and Lucas[23] investigated TDL versus coevolutionary learning (CEL) for small-board Go strategies.
There it was found that TDL learned faster, but that with
careful tuning, CEL eventually learned better strategies.
In particular, with CEL it was necessary to use parentoffspring weighted averaging in order to cope with the effects of noise. This effect was found to be even more pronounced in a follow-up paper by Lucas and Runarsson[24] ,
comparing the two methods for learning an Othello position
value function.
Kotnik and Kalita[25] found that evolution outperformed
TDL at learning to play the card-game rummy, which unlike the board games in the above studies is not a game of
perfect information.
There have also been some comparisons using controlbased problems. Taylor et al.[26] compared TDL and the
NEAT neuro-evolution algorithm for ‘keep-away’ robocup
soccer, and found that evolution could learn better policies, though it took more evaluations to do so. Results also
showed that TDL learned better policies when the task was
fully observable and NEAT learned faster when the task
was deterministic.
Gomez et al.[27] investigated an impressive range of reinforcement techniques, including several versions of neuroevolution and TDL, on four increasingly difficult versions of
the benchmark pole-balancing problem, ranging from simple to very hard. In striking contrast to some other studies,
the best evolutionary methods universally outperformed the
best temporal difference-based (TD-based) methods, both
in terms of learning speed and in terms of which methods could solve the harder versions of the problem at all.
Further, there was significant differences between different
neuro-evolutionary and TD-based methods, with the best
TD-based techniques sometimes outperforming some evolutionary techniques. The relative ordering of the algorithms
was similar across the different versions of the problem. One
feature of all their pole balancing problems was that they
could be solved by relatively small neural networks.
Some general results are beginning to emerge, but there is
still much to be learned. One tendency that can be observed
in many (though not all) of the observed studies is that TDL
learns faster than evolution, but evolution eventually learns
better strategies. Under which conditions this is true is an
important question.
4
49
TDL versus evolution for simulated
car racing
The choice of game to use for benchmarking CI methods
is important. On one hand, it is preferable for it to be
simple and fast to compute, since some learning algorithms
may require millions of simulated game steps to converge.
If it is too simple, however (as could be said of Mountain
Car), then it is questionable what can be learned from it.
Lucas and Togelius[28] compared TDL and evolution for
a simplified car racing game. In that study, they used a
point-to-point car racing game, i.e., the objective being to
drive a car to visit as many way-points as possible within a
given number time steps (they used 500). The way-points
were randomly generated, and had to be visited in order. At
any time the agent could only see the next three way-points.
The cars also had simple underlying models. This made the
simulation very fast to compute, allowing millions of gamesteps per second to be calculated (though the actual speed
would then depend on the complexity of the controller).
Fig. 7 shows a sample run for a hand-designed heuristic state value function, illustrating that the controller got
stuck orbiting a way-point. The heuristic was to value proximity to the next way-point, while punishing excess speed,
which can lead to oscillations, or the orbiting shown here
(for more details see [28]).
Fig. 7 A naive controller that became stuck after orbiting 5 waypoints
Because each trial or episode (i.e., a run of 500 time
steps) is randomly generated, there is a natural random
variation in the scores obtained by a particular agent. For
this task, average scores of about 20 are reasonable, above
30 are good, and above 40 are very good. Two types of car
were used, normal and holonomic. The normal car was a
simplified model of a bang-bang type radio controlled toy
car, which at each time step had choices: accelerate forward, accelerate backward, steer left, and steer right. The
holonomic car was modelled as a point mass. Again the
controller had 5 choices at each time step: apply no force,
or apply a force in one of the four compass point directions.
Each network had three inputs: the square of the velocity,
the distance to the next waypoint, and the angular differ-
50
International Journal of Automation and Computing 05(1), January 2008
ence between the current heading of the car and a straight
line connecting the car with the next waypoint.
Overall, the best results were obtained by evolving state
value MLPs. However, when TDL worked, it could learn
these state values much more quickly.
For the normal car, TDL was competitive with evolution,
with the very best learned controllers having very similar
performance. While evolution also worked well for the holonomic car, TDL failed badly for this case. For the normal
car, when TDL did learn it often achieved high performance
within the first 10 epochs, and in these cases offered much
faster learning than evolution. A sample successful run is
shown in Fig. 8. Note that that score is plotted against
epoch for TDL rather than against generation (used for evolution). An epoch is approximately equivalent to a single
fitness evaluation in terms of the computation required.
One of the most surprising conclusions of this work was
just how sensitive the results were to details of the setup
that might have hitherto seemed unimportant. It was observed that the MLP failed to learn a good controller for
the holonomic car when all three inputs were used. The
holonomic car does not have a heading in the sense of a
normal car, since it can be accelerated in any of the four
directions independently of its current direction of travel.
Hence, the heading information seems to act as a spurious
input. Evolution failed to learn to ignore this, but was much
more successful when using only the two-input version, as
can be seen in Tables 1–2.
Regarding the learning speed of each technique, some
sample runs demonstrate this. Fig. 9 shows an average of
20 runs of the evolutionary algorithm, evolving an MLP for
the normal car (might replace with a perceptron).
Table 1
Statistics for 20 learning runs of the normal car
Method
Mean
Standard error
EVO-MLP
35.0
0.24
EVO-perceptron
30.2
0.33
TDL-perceptron
26.2
1.4
Heuristic
18.8
1.0
Table 2
Statistics for 20 runs of the holonomic car
Method
Mean
Standard error
EVO-MLP-2
32.2
0.06
EVO-perceptron-2
26.7
0.01
EVO-perceptron-3
18.5
2.1
EVO-MLP-3
11.3
0.7
Heuristic
26.8
0.21
The extent to which results from one domain (e.g.
rummy) can be used to make predictions for another domain, such as car racing or Othello is unclear. One of the
significant findings of the current work is that even when
tasks are very similar (and all to do with a simple car racing problem), minor changes in the problem setup can introduce unpredictable biases resulting in significant changes in
the relative performance of each method. There currently
seems to be very little theoretical research that can help us
here, and a great need for empirical investigation.
Fig. 8 Learning a perceptron-based state-controller with TDL
(a successful run)
Fig. 9
5
Evolving a state-value MLP for controlling a normal car
Othello, and the choice of architecture
The first strong learning Othello program developed was
Bill[29, 30] . Later, the first program to beat a human champion was Buro and Logistello[31] , the best Othello program
from 1993–1997. Logistello also uses a linear weighted evaluation function but with more complex features than just
the plain board. Nevertheless, the weights are tuned automatically using self-play. Logistello also uses an opening
book based on over 23 000 tournament games and fast game
tree search[8] .
More recently, Chong et al.[32] co-evolved a spatially
aware MLP for playing Othello. Their MLP was similar to
the one used by Fogel and Chellapilla[2] for playing checkers, and had a dedicated input unit for every possible subsquare of the board. Together with the hidden layers, this
led to a network with 5 900 weights, which they evolved
with around 100 000 games. The weighted piece counter
(WPC) used in the current paper has only 64 weights. The
results below show that optimal tuning of such WPCs can
take hundreds of thousands of games, and relies heavily on
parent-child averaging. These considerations suggest that
further improvement in the performance of evolved spatial
MLPs should be possible.
In this section we review recent results on learning a position value function for Othello.
Othello is a challenging unsolved game, where the best
computer players already exceed human ability. It is an
51
S. M. Lucas / Computational Intelligence and Games: Challenges and Opportunities
interesting benchmark for CI methods, and the author has
been running a series of competitions to find the best neural
network (or other function approximation architecture) for
this game. First, we give a brief explanation of the game.
Othello is played on an 8 × 8 board between two players,
black and white (black moves first). At each turn, a counter
must be placed on the board if there are any legal places to
play, else the player must pass. At each move, the player
must place a counter on an empty board square to “pincer”
one or more opponent counters on a line between the new
counter and an old counter. All opponent counters that are
pincered in this way are flipped over to the color of the current player. The initial board has four counters (two of each
color) with black to play first. This is shown in Fig. 10, with
the open circles representing the possible places that black
can play (under symmetry, all opening moves for black are
identical). The game terminates when there are no legal
moves available for either player, which happens when the
board is full (after 60 moves, since the opening board already has four counters on it), or when neither player can
play. The winner is the player with the most pieces of their
color at the end of the game.
the site. When a network is uploaded, it is played against
the standard heuristic WPC for many games (initially 1 000,
but this has been reduced to 100 to reduce load), and this
gives it a ranking in the trial league. Then, for particular
competition events, entrants are allowed to nominate two of
their best networks to participate in a round-robin league.
For the competitions, and for the results in this paper,
all play is based on value functions evaluated at 1-ply. All
games are played with a 10% chance of a forced random
move. Hence, this is no longer strictly speaking Othello,
but it is a better benchmark for our purpose. When evaluating two feed-forward neural networks on the true game of
Othello, there are only two possible outcomes, depending
only on who plays first.
The best network found in this way so far was an MLP.
Co-evolution finds it hard to learn MLPs for this task, and
for a long time the best network was a TDL-trained MLP.
For the 2006 IEEE Congress on Evolutionary Computation
(CEC) Othello competition, however, a new champion was
developed by Kyung-Joon Kim and Sung-Bae Cho. They
seeded a population with small random variations of the
previous best MLP, and then ran co-evolution for 100 generations. This was able to produce a champion that performed in the round-robin league significantly better than
the other players, and than the TDL-trained MLP that it
was developed from. This points toward the value of TDL
/ evolution hybrids.
5.1
Fig. 10
The opening board for Othello
As play proceeds, the piece difference tends to oscillate
wildly, and some strategies aim to have few counters during
the middle stages of the game to limit possible opponent
moves. Fig. 11 shows how piece difference can change during
the course of a game.
Fig. 11 Typical volalite trajectory of piece difference during the
game of Othello
The author has been running an Othello neural network
web server for the past two years. During that time, well
over one thousand neural networks have been uploaded to
Parent-child averaging
A surprising result obtained by Runarsson and Lucas[23]
for small-board Go, and validated even more dramatically
for Othello[24] , is the way that standard co-evolution can
fail to learn very well. In both these studies, forced random
moves were used to get a more robust evaluation of playing ability. To get good performance from co-evolution,
they found it essential to use parent-child averaging. This
is a technique that had been previously used in evolution
strategies, and was also used by Pollack and Blair[20] with
their (1 + 1) ES for back-gammon. Runarsson and Lucas[23]
experimented with a wide variety of settings for the ES,
but unless parent child weighted averaging was used, all
performed poorly. The following equation explains the process. The weights of the parent neural network are in vector
w0 and the weights of the best child network are in vector
w. The β factor controls the weighting between parent and
child. With β = 1.0 we get a standard ES. With β = 0,
no information from the child is used at all, so no evolution occurs. Remarkably, they found that small values of
β worked best. The results plotted in Fig. 12 (from [24])
show how standard co-evolution (i.e. a standard ES with
relative fitness function) fails to learn much, but learning
does occur with β = 0.05. Overall, they found that coevolution with weighted parent-child averaging performed
better than TDL, and was able to learn a WPC that can
outperform the standard heuristic weights. TDL was not
able to match this, and while it learned more quickly, it was
unable to match the standard heuristic weights.
w
~ 0 ← βw
~ + β(w
~i − w
~ 0 ).
(5)
52
International Journal of Automation and Computing 05(1), January 2008
Fig. 12 Co-evolutionary learning average performance (probability of a win) versus the heuristic player, plotted against generation (The 1/45000 term indicates the scaling on this axis; multiple each value by 45000 to find the number of games played.
The grey lines indicate one standard deviation from the mean,
and this run used a (1, 10) ES[24] .)
5.2
n-tuple architectures
Given that background, this now brings us on to the subject of architecture. In the field of CI and games, by far
the most popular choice of architecture is the neural network. This comes in various forms, but perceptrons and
multi-layer perceptrons are the most common. Also popular are more flexibly structured networks such as those
created with NEAT, where there is no distinct layering and
connections may evolve between any neurons, and the number of neurons is also not fixed.
Very recently, the author has experimented with using ntuple networks for this task. n-tuple networks date back to
the late 1950s with the optical character recognition work
of Bledsoe and Browning[33] . More detailed treatments of
standard n-tuple systems can be found in [34, 35]. They
work by randomly sampling input space with a set of n
points. If each sample point has m possible values, then the
sample point can be interpreted as an n digit number in base
m, and used as an index into an array of weights. The value
function for a board is then calculated by summing over all
table values indexed by all the n-tuples. The n-tuple works
in a similar way to the kernel trick used in support vector
machines (SVMs), and is also related to Kanerva0 s sparse
distributed memory model. The low dimensional board is
projected into a high dimensional sample space by the ntuple indexing process.
This work is still in its initial stages, but has already
proved to be remarkably successful. An n-tuple network
trained with a few hundred of self-play games was able to
significantly outperform the 2006 CEC champion.
Fig. 13 illustrates the system architecture but shows only
a single n-tuple. Each n-tuple specifies a set of n board
locations, but samples them under all equivalent reflections
and rotations. Fig. 13 shows a single 3-tuple, sampling 3
squares along an edge into the corner.
Fig. 13 The system architecture of the n-tuple-based value function, showing a single 3-tuple sampling at its eight equivalent
positions (equivalent under reflection and rotation)
5.3
How it works
Fig. 13 illustrates the operation of a single n-tuple. The
value function for a board is simply the sum of the values
for each n-tuple. For convenient training with error backpropagation, the total output is put through a tanh function. Each n-tuple has an associated look-up table (LUT).
The output for each n-tuple is calculated by summing the
LUT values indexed by each of its equivalent sample positions (eight in the example). Each sample position is simply interpreted as an n digit ternary (base three) number,
since each square has three possible values (white, vacant,
or black). The board digit values were chosen as (white=0,
vacant=1, black=2). By inspecting the board in Fig. 13,
it can be seen that each n-tuple sample point indexes the
look-up table value pointed to by the arrow. These table
values are shown after several hundred self-play games of
training using TDL. The larger the black bar for a LUT
entry, the more positive the value (the actual range for this
figure was between about +/ − 0.04). Some of these table entries have obvious interpretations. Good for black
means more positive, good for white means more negative.
The LUT entry for zero corresponds to all sampled squares
being white: this is the most negative value in the table.
The LUT entry for twenty six corresponds to all sampled
squares being black: this is the most positive value in the
table. This can be expressed in the following equation.
v(b) =
X
l(d)
(6)
d∈D(b)
where b is the board, v(b) is the value of the calculated value
of the board, d is a sampled n digit number in the set D(b)
of samples, given the n-tuple, and l is the vector of values
in the LUT.
Given the explanation above for how the value function is
calculated, the l entries of LUT can be seen as the weights of
a single layer perceptron. The indexing operation performs
a non-linear mapping to high-dimensional feature space, but
that mapping is fixed for any particular choice of n-tuples.
53
S. M. Lucas / Computational Intelligence and Games: Challenges and Opportunities
Since a linear function is being learned, there are no local
optima to contend with.
The TDL training process, as shown in Algorithm, is very
simple, and can be explained in two parts.
The first is how it is interfaced to the Othello game. The
game engine calls a TDL update method for any TDL player
after each move has been made. It calls “inGameUpdate”
during a game, or “terminalUpdate” at the end of a game.
To show just how simple the process is, the code of these
algorithm is shown as below.
Algorithm. The main two methods for TDL learning in Othello.
public void inGameUpdate(double[ ] prev, double[ ] next)
and taking a random walk from that point. At each step
of the walk, the next square is chosen as one of the eight
immediate neighbours of the current square. Each walk was
for six steps, but only distinct squares are retained. So each
randomly constructed n-tuple had between 2 and 6 sample
points. The results in this paper are based on 30 such ntuples. One would expect some n-tuples to be more useful
than others, and there should be scope for evolving the ntuples sample points while training the look-up table values
using TDL.
Fig. 14 shows how performance improves with the number of self-play games. After every 25 self play games, performance was measured by playing 100 games against the
standard heuristic player (50 each as black and white).
{
double op = tanh(net.forward(prev));
double tg = tanh(net.forward(next));
double delta = alpha · (tg − op) · (1 − op · op);
net.updateWeights(prev, delta);
}
public void terminalUpdate(double[ ] prev, double tg)
{
double op = tanh(net.forward(prev));
double delta = alpha · (tg − op) · (1 − op · op);
net.updateWeights(prev, delta);
}
The variables are as follows: op is the output of the network; tg is the target value; alpha is the learning rate (set
to 0.001); delta is the back error term; prev is the previous
state of the board; next is the current state of the board;
“net” is an instance variable bound to some neural network
type of architecture (an n-tuple system in this case).
The n-tuple system implements the “net” interface, and
an instance of one is bound to the “net” in the code. The
forward method calculates the output of the network given
a board as input. The “updateWeight” method propagates
an error term, and makes updates based on this in conjunction with the board input. For the n-tuple system the
update method is very simple. While the value function was
calculated by summing over all LUT entries indexed by the
current board state, the update rule simply adds the error
term δ to all LUT entries indexed by the current board:
l(d) = l(d) + δ,
∀d ∈ D(b)
(7)
One of the best features of an n-tuple system is how it
scales with size. Because of the indexing operation, it is
independent of the size of the LUT. Therefore, although
the LUT size grows exponentially with respect to n, the
speed remains almost constant, and linear in the number of
n-tuples. Hence, n-tuple value functions with hundreds of
thousands of weights can be calculated extremely quickly.
5.4
Choosing the sample points
The n positions can be arranged in a straight line, in a
rectangle, or as random points scattered over the board.
The results in this paper are based on random snakes:
shapes constructed from random walks. Each n-tuple is
constructed by choosing a random square on the board,
Fig. 14 Variation in win ratio against a heuristic player (each
sample point based on 100 games, 50 each as black and white)
(The x-axis, nGames/25, indicates the scaling; multiply each
value by 25 to find the number of games played. The y-axis
shows the ratio of the number of wins to the total number of
games played)
Table 3 shows how performance against the 2006 CEC
champion varies with the number of self play games (in this
case playing 200 games against the champion (100 each as
black and as white), where nsp denotes the number of selfplay games. After the first 500 self-play games have been
played, the champion is defeated in nearly 70% of games.
Table 3 Performance of TDL n-tuple player versus 2006 CEC
champion over 200 games, sampled after varying nsp
nsp
Won
Drawn
Lost
250
89
5
106
500
135
6
59
750
142
5
53
1000
136
2
62
1250
142
5
53
Not only has the n-tuple based player reached a higher
level of performance than any player to date, it has also
done so much more quickly. While n-tuple systems are well
known to a small base of appreciative users for their high
speed and reasonable accuracy, their accuracy is usually
not quite as high as the best neural networks or SVMs for
many pattern recognition tasks. Game strategy learning
is different to pattern recognition in important ways: the
data is acquired through active exploration, and the data is
typically much noisier. This might be the reason why they
54
International Journal of Automation and Computing 05(1), January 2008
perform so well on this task.
6
Future directions
6.1
Robotics and games
The simulation middleware that underlies many video
games enables the development of more lifelike games to be
developed with greater ease. With high quality middleware
to support game development, developers can concentrate
their creativity on creating fun environments with appropriate interest and challenge, with the game-play arising
naturally from the situation rather than having to be preprogrammed. For example, until recently, the graphics for
explosions would have to be designed with a great deal of
manual effort. With modern physics engines such as Ageia0 s
PhysX, it can be simulated as a particle system; the graphics then arise directly from the physical model. The explosion can then depend very naturally on the amount of fuel
in the tank, for example.
A major interest of our research group has been in car
racing challenges. Competitive car driving is a problem of
great practical importance, and has received some attention
from the CI community. Most often researchers have used
various learning methods for developing controllers for car
racing simulations or games[36, 37] . However, CI techniques
have also been applied to physical car racing, famously by
Thrun in the DARPA Grand Challenge[38] (where DARPA
is defense advanced research projects agency), but also
by e.g. Tanev et al.[39] who evolved controllers for radiocontrolled toy cars.
The author has recently been involved with work developing a robotic car racing platform2 . Robotic car racing
offers the same type of challenge whether done on full size
cars or model cars, but model cars are much cheaper. If
the research can be done in simulation, then of course it
is cheaper still, though ultimately less convincing, at least
until transferred back to the real world.
As a starting point, take the challenge of driving quickly
along a path or road using a computer vision system as
the main source of input. The vision problem is of similar
complexity whether doing it on a real-world car, or using
the video output of a modern console or personal computer
(PC) racing game. This can be seen from the examples.
Fig. 15 shows an image captured by the web-cam on our
robotic model car (while in motion at about 10 miles per
hour). Fig. 16 shows a screen shot taken from Sega Rally
running on a PC.
While the vision problems are of similar complexity (depending on the details of the environment or track), the
game version typically offers much more forgiving physics
than the real world, especially when it comes to car damage!
We are currently investigating running a car racing competition using an on-line commercial gaming lobby such as
XBox live, where all the competitors are software agents
processing the real-time video from the console output to
drive the tracks as competitively as possible.
2
http://dces.essex.ac.uk/staff/lucas/roborace/roborace.html
Fig. 15
An image captured on-board our autonomous model car
Fig. 16
6.2
A screen shot from Sega Rally
Direct video input mode
Direct video input mode has much wider application than
car racing. The idea is that the input to the software agent
is simply the real-time video output from the game (audio
could also be included).
One of the main difficulties for academic research on CI
methods for video games is the time taken to learn the application programming interface (API) to interact with a
complex game. While we may look forward to more standardization of NPC APIs in the future, the possibility of
circumventing API issues by giving the software agent exactly the same view as a human player offers tremendous
challenges, but for some types of game (especially car racing
games) it may be possible to make rapid progress. Indeed,
results have already been obtained for a relatively simple
3D car racing game using CI methods[37] .
More recently, this screen capture mode has been used
to enable software agents to play Ms. Pac-Man without requiring any access to details of the software.
Fig. 17 shows an agent under test controlling the Ms.
Pac-Man character using the direct screen capture mode of
operation. The Ms. Pac-Man game is shown to the right,
the extracted game objects are shown to the left, and the
top window shows the direction key currently selected by
the agent. The screen is captured approximately 15 times
per second. The exact number depends on the speed of
S. M. Lucas / Computational Intelligence and Games: Challenges and Opportunities
the computer and the amount of processing performed by
the agent control algorithm. After each screen is captured,
some basic image processing routines (e.g. connected component analysis) are then run to extract the main game
objects, such as the Ms. Pac-Man agent, the ghosts, the
power pills, and the food pills.
Fig. 17 An agent under test controlling the Ms. Pac-Man character via direct screen capture
These extracted objects are then given as input to the
agent under test. In response to this, the agent generates a
key event (one of the cursor keys) to control the movement
of the Ms. Pac-Man agent.
Because of the inherent delays in the screen capture process and the image processing operations, the state of the
game that the agent sees will often be out of date at the
point of delivery, and any further processing done by the
agent only adds to the delay in sending the key event to
the operating system to control the game. This delay is
typically variable, but is often between 50 ms and 100 ms.
Naive agent controllers that ignore this effect often spend
much of their time oscillating around a junction while missing the turning! From a CI viewpoint this merely adds to
the challenge of developing high quality controllers. This
has been run as a competition for the 2007 IEEE CEC, and
will also be run for 2008 IEEE World Congress on Computational Intelligence (WCCI). So far, the leading entry
has been one called Pacool, supplied by Abbas Mehrabian
and Arian Khosravi, which has scored over 17 000 points
when run in this mode. The current human world champion record stands at 933 580 (by Abdner Ashman, who
cleared 133 screens in the process), so there is some way
to go yet until software agents can compete with the best
humans at this game.
6.3
Competitions
There are many game-related competitions, and this is a
great way to drive research forward. One of the most active
fields of game research is Go, and the dedicated Go community is also one of the best at running regular competitions.
Without this, radical new approaches such as MoGo,
which really was a major departure from conventional com-
55
puter Go wisdom, would have most likely taken much longer
to become established.
Well-designed competitions are simply the best way to
establish which techniques work best. Some games require
interesting AI rather than competitive AI. Interesting behaviour is in the eye of the beholder: it is subjective and
naturally hard to measure, though statistically there may
be much agreement about what is interesting or not. Other
games (especially real-time strategy) do require smarter AI,
and any games company wishing to discover the best AI for
their game might find that publishing the appropriate API
and running open competitions offered a very cost-effective
way to do this.
This can be done for the entire behaviour of an agent,
or to measure performance over certain tasks. This also
enables a rapid transfer of technology between academia
and the games industry. As for some standard tasks, it will
be relatively easy to test the various performance aspects
of a given algorithm or component. The use of standard
benchmarks also makes entry into this research area attractive for an academic. This has already been seen for simpler
game environments such as the Stanford general game playing competition3 , the Essex Othello neural network server4 ,
project Hoshimi5 and simulated Car racing6 .
There are many interesting challenges and opportunities
involved in making these competitions work for more complex games, including the standardization of sensory inputs,
body parts, control systems and actuators.
There are also great opportunities for companies wishing to outsource advanced software agent development work
where it is unclear who may be the best providers for the
required system. In such cases, a competition can be run
where the reward goes to the entrants who provide the best
solutions. This is essentially what DARPA did by organizing the grand challenge, but it can also be done using the
web for smaller scale challenges.
7
Conclusions
There exists a diverse range of game genres on which to
test and apply CI methods, and this paper has mentioned
only a few examples. CI methods can be used to successfully develop competitive agents for these games to control
non-player characters. However, the study of the Mountain
Car problem, point-to-point car racing, and Othello led to
some remarkable observations. In the case of the Mountain
Car problem, we showed how certain problem setups could
render the problem trivial, so that running an evolutionary algorithm often revealed near-optimal solutions in the
initial randomly constructed population.
In point-to-point car racing we observed that learning
performance could be severely impaired by the presence of
an unnecessary input.
In the case of Othello, there were two main findings.
First, standard co-evolution failed to learn good quality
WPCs. However, by using parent-child weighted averaging
we were able to learn high quality WPCs, the best of which
3
http://games.stanford.edu
http://algoval.essex.ac.uk:8080/othello/html/Othello.html
5
http://www.project-hoshimi.com
6
http://julian.togelius.com/cig2007competition
4
56
International Journal of Automation and Computing 05(1), January 2008
even outperformed the standard heuristic weights. Second,
by adopting a radically different type of neural network, an
n-tuple system, we were able to easily outperform the best
performing MLPs for this problem.
In summary, much experimentation and perhaps even
rather novel techniques may be necessary to get best performance from CI methods. This makes the area very interesting and challenging to research.
References
[1] J. Schaeffer, N. Burch, Y. Björnsson, A. Kishimoto, M.
Müller, R. Lake, P. Lu, S. Sutphen. Checkers Is Solved.
Science, vol. 317, no. 5844, pp. 1518–1522, 2007.
[2] K. Chellapilla, D. B. Fogel. Evolving an Expert Checkers
Playing Program without Using Human Expertise. IEEE
Transactions on Evolutionary Computation, vol. 5, no. 4,
pp. 422–428, 2001.
[3] D. B. Fogel, T. J. Hays, S. L. Hahn, J. Quon. An Evolutionary Self-learning Chess Program. Proceedings of the IEEE,
vol. 92, no. 12, pp. 1947–1954, 2004.
[4] S. Gelly, Y. Wang, R. Munos, O. Teytaud. Modification of
UCT with Patterns in Monte-Carlo Go, Technical Report
6062, INRIA, France, 2006.
[5] Y. Wang, S. Gelly. Modifications of UCT and Sequence-like
Simulations for Monte-Carlo Go. In Proceedings of IEEE
Symposium on Computational Intelligence and Games,
IEEE Press, pp. 175–182, 2007.
[6] E. Charniak. Statistical Language Learning, MIT Press,
Cambrige, Massachusetts, USA, 1996.
[7] S. Colton, P. Cowling, S. M. Lucas. An Industry/Academia
Research Network on Artificial Intelligence and Games
Technologies, Technical Report EP/F033834, EPSRC,
Swindon, 2007.
[8] M. Buro. ProbCut: An Effective Selective Extension of the
Aalpha-Beta Algorithm. ICCA Journal, vol. 18, no. 2, pp.
71–76, 1995.
[15] D. Cliff. Explorations in Evolutionary Design of Online
Auction Market Mechanisms. Journal of Electronic Commerce Research and Applications, vol. 2, no. 2, pp. 162–175,
2003.
[16] A. Moore. Efficient Memory-based Learning for Robot Control, Ph. D. dissertation, University of Cambridge, UK,
1990.
[17] R. Sutton and A. Barto. Introduction to Reinforcement
Learning, MIT Press, Cambridge, MA, USA, 1998.
[18] S. Whiteson, P. Stone. Evolutionary Function Approximation for Reinforcement Learning. Journal of Machine Learning Research, vol. 7, pp. 877–917, 2006.
[19] G. Tesauro. Temporal Difference Learning and TDgammon. Communications of the ACM, vol. 38, no. 3, pp.
58–68, 1995.
[20] J. B. Pollack, A. D. Blair. Co-evolution in the Successful
Learning of Backgammon Strategy. Machine Learning, vol.
32, no. 3, pp. 225–240, 1998.
[21] G. Tesauro. Comments on “Co-Evolution in the Successful Learning of Backgammon Strategy”. Machine Learning,
vol. 32, no. 3, pp. 241–243, 1998.
[22] P. J. Darwen. Why Co-evolution Beats Temporal Difference
Learning at Backgammon for a Linear Architecture, but not
a Non-linear Architecture. In Proceedings of Congress on
Evolutionary Computation, IEEE Press, vol. 2, pp. 1003–
1010, 2001.
[23] T. P. Runarsson, S. M. Lucas. Co-evolution versus Selfplay Temporal Difference Learning for Acquiring Position
Evaluation in Small-board Go. IEEE Transactions on Evolutionary Computation, vol. 9, no. 6, pp. 628–640, 2005.
[24] S. M. Lucas, T. P. Runarsson. Temporal Difference Learning versus Co-evolution for Acquiring Othello Position Evaluation. In Proceedings of IEEE Symposium on Computational Intelligence and Games, Reno/Lake Tahoe, USA, pp.
53–59, 2006.
[9] T. Gosling, N. Jin, E. Tsang. Games, Supply Chains and
Automatic Strategy Discovery Using Evolutionary Computation. Handbook of Research on Nature-inspired Computing for Economics and Management, J. P. Rennard (ed.),
vol. 2, pp. 572–588, 2007.
[25] C. Kotnik, J. Kalita. The Significance of Temporaldifference Learning in Self-play Training: TD-rummy versus EVO-rummy. In Proceedings of the International Conference on Machine Learning, Washington D.C., USA, pp.
369–375. 2003.
[10] K. O. Stanley, B. D. Bryant, R. Miikkulainen. Real-time
Neuroevolution in the NERO Video Game. IEEE Transactions on Evolutionary Computation, vol. 9, no. 6, pp.
653–668, 2005.
[26] M. E. Taylor, S. Whiteson, P. Stone. Comparing Evolutionary and Temporal Difference Methods in a Reinforcement
Learning Domain. In Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, Seattle,
Washington, USA, pp. 1321–1328, 2006.
[11] R. M. Axelrod. The Evolution of Cooperation, Basic Books
Inc., New York, USA, 1984.
[12] N. Jin, E. Tsang. Co-adaptive Strategies for Sequential Bargaining Problems with Discount Factors and Outside Options. In Proceedings of Congress on Evolutionary Computation, Vancouver, BC, Canada, pp. 2149–2156, 2006.
[13] D. Cliff. Minimal-intelligence Agents for Bargaining Behaviors in Market-based Environments, Technical Report HPL97-91 970811, Hewlett Packard Laboratories, USA, 1997.
[14] D. K. Gode, S. Sunder. Allocative Efficiency of Markets
with Zero-intelligence Traders: Market as a Partial Substitute for Individual Rationality. The Journal of Political
Economy, vol. 101, no. 1, pp. 119–137, 1993.
[27] F. Gomez, J. Schmidhuber, R. Miikkulainen. Efficient Nonlinear Control through Neuroevolution. In Proceedings of
the European Conference on Machine Learning, Lecture
Notes in Computer Science, vol. 4212, pp. 654–662, 2006.
[28] S. M. Lucas, J. Togelius. Point-to-point Car Racing: An Initial Study of Evolution versus Temporal Difference Learning. In Proceedings of IEEE Symposium on Computational
Intelligence and Games, Westin Harbour Castle Toronto,
Ontario Canada, pp. 260–267, 2007.
[29] K.-F. Lee, S. Mahajan. A Pattern Classification Approach
to Evaluation Function Learning. Artificial Intelligence, vol.
36, no. 1, pp. 1–25, 1988.
S. M. Lucas / Computational Intelligence and Games: Challenges and Opportunities
[30] K.-F. Lee, S. Mahajan. The Development of a World Class
Othello Program. Artificial Intelligence, vol. 43, no. 1, pp.
21–36, 1990.
[31] M. Buro. LOGISTELLO – A Strong Learning Othello Program. NEC Research Institute, Princeton, NJ,
1997, [Online], Available: http://www.cs.ualberta.ca/˜
mburo/ps/log-overview.ps.gz.
[32] S. Y. Chong, M. K. Tan, J. D. White. Observing the Evolution of Neural Networks Learning to Play the Game of
Othello. IEEE Transactions on Evolutionary Computation,
vol. 9, no. 3, pp. 240–251, 2005.
[33] W. W. Bledsoe, I. Browning. Pattern Recognition and
Reading by Machine. In Proceedings of the Eastern Joint
Computer Conference, pp. 225–232. 1959.
[34] J. Ullman. Experiments with the n-tuple Method of Pattern
Recognition. IEEE Transactions on Computers, vol. 18, no.
12, pp. 1135–1137, 1969.
[35] R. Rohwer, M. Morciniec. A Theoretical and Experimental
Account of n-tuple Classifier Performance. Neural Computation, vol. 8, no. 3, pp. 629–642, 1996.
[36] B. Chaperot, C. Fyfe. Improving Artificial Intelligence in a
Motocross Game. In Proceedings of IEEE Symposium on
Computational Intelligence and Games, Reno/Lake Tahoe,
USA, pp. 181–186, 2006.
[37] D. Floreano, T. Kato, D. Marocco, E. Sauser. Coevolution
of Active Vision and Feature Selection. Biological Cybernetics, vol. 90, no. 3, pp. 218–228, 2004.
[38] S. Thrun, M. Montemerlo, H. Dahlkamp, D. Stavens, A.
Aron, J. Diebel, P. Fong, J. Gale, M. Halpenny, G. Hoffmann, K. Lau, C. Oakley, M. Palatucci, V. Pratt, P. Stang,
S. Strohband, C. Dupont, L.-E. Jendrossek, C. Koelen, C.
Markey, C. Rummel, J. van Niekerk, E. Jensen, P. Alessandrini, G. Bradski, B. Davies, S. Ettinger, A. Kaehler, A.
Nefian, and P. Mahoney. The Robot that Won the DARPA
Grand Challenge. Journal of Field Robotics, vol. 23, no. 9,
pp. 661–692, 2006.
57
[39] I. Tanev, M. Joachimczak, H. Hemmi, K. Shimohara. Evolution of the Driving Styles of Anticipatory Agent Remotely
Operating a Scaled Model of Racing Car. In Proceedings of
IEEE Congress on Evolutionary Computation, vol. 2, pp.
1891–1898, 2005.
Simon M. Lucas received the B. Sc. degree in computer systems engineering from
the University of Kent, UK, in 1986, and
the Ph. D. degree from the University of
Southampton, UK, in 1991. He has worked
a year as a research engineer for GEC
Avionics. After a one-year post doctoral
research fellowship (funded by British Telecom), he was appointed to a lectureship at
the University of Essex in 1992 and is currently a reader in computer science.
He was chair of IAPR Technical Committee 5 on Benchmarking and Software. He is the inventor of the scanning n-tuple
classifier, a fast and accurate OCR method. He was appointed
inaugural chair of the IEEE CIS Games Technical Committee in
July 2006, and has been competitions chair for many international conferences, and co-chaired the first IEEE Symposium on
Computational Intelligence and Games in 2005. He was program
chair for IEEE Congress on Evolutionary Computation (CEC) in
2006, and program co-chair for IEEE Symposium on Computational Intelligence and Games (CIG) in 2007, and will be program
co-chair for International Conference on Parallel Problem Solving from Nature in 2008. He is an associated editor of IEEE
Transactions on Evolutionary Computation, and the Journal of
Memetic Computing. He was invited keynote speaker at IEEE
Congress on Evolutionary Computation in 2007.
His research interests include evolutionary computation,
games, and pattern recognition, and he has published widely
in these fields with over 120 refereed papers, mostly in leading
international conferences and journals.