Download Computational Intelligence and Game Design

Computational Intelligence and Game Design
Julian Togelius and Jürgen Schmidhuber
Abstract. This short paper is an attempt to broaden the scope of CIG
research, proposing that we optimize other things than policies and for
other objectives than winning a game. It is at least as interesting to
optimize environments and rules with the objective of creating fun games.
1
Introduction
The field of computational intelligence and games is mostly concerned with learning to play particular games as well as possible. There is nothing wrong with
this type of research; indeed, games provide the type of deep but accessible reinforcement learning problems that we badly need in order to develop better CI
algorithms and controller representations. However, by single-mindedly focusing
on learning to play well, we risk overlooking a number of research directions that
are equally interesting from a scientific perspective, and much more interesting
for commercial game developers.
To start with the commercial perspective: game developers are, in general,
not interested in ways of making NPCs (non-playing characters, e.g. opponents)
play better. This is because it is usually possible to make them play well enough
using just a tiny bit of (virtually undetectable) cheating, such as giving the NPC
more information than the human player, or slightly superior abilities. Further,
having NPCs that play as well as possible is no end in itself; if the opponents are
too hard to beat, the game quickly becomes uninteresting for human players.
Of course, there are exceptions. In particular, many strategy games (such as
Civilization or Starcraft ) are so complicated that it’s hard to come up with NPC
AI that plays competitively against good human players, without resorting to
most blatant cheating. In such cases, CI techniques for beating the game better
might contribute to the quality of the game. And let us not forget that many
games are representations of real-life problems (e.g. battle tactics or vehicle
control), and advances in learning how to play games might be transferable
to related real-life problems. But for most game genres, such as racing games,
platformers, puzzle games or first person shooters, there does not seem to be
any interest from game developers in learning to play the game better per se.
However, learning to play a particular game well is a necessary first step for
some of the more industrially relevant research directions described below.
From a scientific perspective there’s plenty of uncovered ground, some formidable
challenges which call for design of new algorithms, and potential for interdisciplinary research involving both computational intelligence and social sciences.
The rest of the paper will assume that we are using some kind of CI optimization algorithm, loosely defined. This could be evolutionary computation,
2
some form of swarm intelligence, or something similar. The next section discusses
what objective we should optimize for, if not necessarily playing well, and the
section after that discusses what we should optimize, if not necessarily a policy.
A concluding section discusses how these ideas could be combined.
2
The objective: for what should we optimize?
We should optimize aspects of games for entertainment value. But how do we do
this? Can we measure entertainment in a quantitative way that does not involve
using human subjects, so that it can be incorporated into a fitness function? To
start with, we need some sort of theory about what makes a game entertaining.
Fortunately, there are a number of theories from psychology and game studies
that might be of help here, many of them focusing on the challenge a human
player experiences when playng the game. One of the oldest and best known of
these is Csikszentmihalyi’s concept of flow, the “optimal experience” [1]. When
in the state of flow, a human is performing a task (not necessarily a game; the
theory was developed with e.g. the creative process of artists in mind) such that
he is fully concentrated on the task and loses his sense of self but has a sense of
full control. One of the key prerequisites for reaching flow is that the amount of
challenge of the task is just about right: not too hard, and not too easy.
Malone [2] distinguishes between three factors that make games engaging:
challenge, fantasy and curiosity. While the right level of challenge is a key component of the flow concept as well, the other two factors are more game-specific;
fantasy refers to how well the game evokes a sense of being somewhere else or
doing something exotic, whereas curiosity means that environments should have
an “optimal level of informational complexity”: novel but not incomprehensible.
Sweetser and Wyeth proposed GameFlow as a an adaptation of the flow
concept specifically to games [3]. GameFlow offers a way of evaluating how entertaining a game is based on a number of criteria: concentration, challenge,
player skills, control, clear goals, feedback, immersion and social interaction.
Koster’s “theory of fun for game design” also focuses on the challenge of the
game, but in a different way. Koster sees the learning of how to play a game as
its main source of entertainment; playing is learning, and learning is fun [4]. For
Koster, well-designed games are games that are easy to start playing, but where
the difficulty curve is such that the player continuously learns something new
while playing. I.e., games that take a minute to learn, but a lifetime to master.
A theory which has not so far been explored in the context of game design
is Schmidhuber’s theory of artificial curiosity [5]. Unlike the theories above, it
does not focus on the challenge of the task/game, but instead focuses on the
predictability of environments. Agents prefer environments with some sort of
algorithmic regularity that is not yet known but expected to be quickly learnable.
A curious agent is rewarded whenever a separate adaptive predictor learns to
predict the incoming data stream better than before. Thus the agent is motivated
to explore those parts of the world that yield temporarily ”interesting” input
sequences, maximizing the derivative of the subjective predictability.
3
3
The substrate: what should we optimize?
When learning to play a game as well as possible, what is optimized is a policy,
represented e.g. as a state value function or a controller that maps game states
and state histories to actions. In other words, what is learned is an AI for the
game. However, there are many other things than policies that can be optimized.
3.1
Environments
In the development of a commercial game, a very large part of the effort is spent
in designing the environments in which the player and NPCs move around.
Depending on the game type, these environments can take the form of levels,
dungeons, race tracks, cities etc. As the capabilities of graphics and storage hardware grow, players place ever higher demands on game environments, meaning
ever larger parts of development efforts being spent on them. It would thus be of
great value to the games industry to have CI techniques assist with environment
development. Such techniques could operate either offline (during development
of the game) or online (during game play, based on the actions the player takes).
There are various demands on the environments: they should be aesthetically
pleasing, fun to play in, varied, fit with the story arc of the game, etc. Most of
these are optimizable. Likewise, there are many different ways the environments
could be represented, partly dependent on the type of environment. While a racing track could be represented in a direct fashion with a linear genome (perhaps
specifying widths and turning angles for each segment), a complete city for use
in an open-world game would have to be represented in a procedural or otherwise
generative way, with a sublinear genome. Fortunately there are many such encodings available “off the shelf” in the fields of evolutionary art and evolutionary
design, including representations for plants, cities, buildings etc.
3.2
Rules
But we need not stop with environments. It is entirely plausible that we could
use CI methods to generate the games in themselves. The core of any game is its
rules, and good rulesets are something that evolutionary methods find efficiently.
Rules define a number of things in a game: under what conditions the game
is won, what the player’s agent can and can’t do, what the NPCs can do, and
how various agents interact (e.g. what happens when the player’s agent bumps
into a particular type of NPC). In addition to discrete rules, there are any number of continuous parameters that evolutionary or other algorithms could tune:
movement speeds, time limits, energy levels, forces of attraction or repulsion etc.
It would certainly be possible to construct a rule space that had the rules
(but not graphics) of Asteroids in one corner and Defender in another, and an
astronomical number of possible games in between. Most of these games would
be either impossible or very boring, but a well-designed rule space should include
gradients that could be followed towards new fun games. In the same manner, it
seems doable to construct another rule space that includes both Othello and Go.
4
Search in rule and parameter space could be used both for creating completely
new games, and for fine-tuning games based on an existing core mechanic.
4
Bringing it all together
In [6] we present an approach to evolving interesting tracks for a driving game,
for which we have previously evolved well-performing neural network-based players. We let a human player drive a test track that contained different sorts of
driving challenges, and measured aspects of the driving. We then further evolved
a previously evolved controller to conform with these measures, creating an artificial player that drove similarly to the modelled human. The fitness function
for the track was derived from Malone’s factor. The fitness of a track depended
on how the “human-like” controller performed on the track. The objectives were
for the car to have the right (not too high, not too low) average speed, high
maximum speed, and high variability in performance between trials, indicating
driving “on the edge”. Tracks were represented as b-splines. Evolution found
tracks whose level of challenge matched the proficiency of the modelled players,
and possibly more fun for the modelled players in other respects as well.
This general technique probably carries over to other definitions of fun and
types of evolvable substrate as well. In particular, having access to a good artificial player that plays the game in a way representative either of a particular
human or of humans in general is a prerequisite for being able to judge the “fun”
of a new evolved environment or ruleset. Fortunately, learning to play games is
what most CIG researchers are working on. The learning could even be built
in to the fitness function; maybe the best way of capturing Koster’s notion of
learning as fun is to optimize for evolvability, so that fitness depends on how fast
an agent learns to win in in the new environment or under the new rules. Many
fitness functions and substrate representations will probably be specific to particular game genres; multiobjective optimization will probably also be necessary,
as some definitions of fun can be decomposed into several conflicting objectives.
In sum, the field of computational intelligence and game design is wide open
and unexplored, and you are very welcome to dive in!
References
1. Csikszentmihalyi, M.: Flow: the Psychology of Optimal Experience. (1990)
2. Malone, T.W.: What makes things fun to learn? heuristics for designing instructional
computer games. In: Proceedings of the 3rd ACM SIGSMALL symposium and the
first SIGPC symposium on Small systems. (1980) 162–169
3. Sweetser, P., Wyeth, P.: Gameflow: A model for evaluating player enjoyment in
games. ACM Computers in Entertainment (2005)
4. Koster, R.: A theory of fun for game design. Paraglyph press (2005)
5. Schmidhuber, J.: Developmental robotics, optimal artificial curiosity, creativity,
music, and the fine arts. Connection Science 18 (2006) 173–187
6. Togelius, J., De Nardi, R., Lucas, S.M.: Towards automatic personalised content
creation in racing games. In: Proceedings of the IEEE Symposium on Computational
Intelligence and Games. (2007)