Download Project overview

Institute of Arts, Media and Computer Games
Dynamic artificial intelligence systems with online learning
Kenneth D. Mackinnon
0804731
BSc (Hons) Computer Game Applications Development
2011
Kenneth D. Mackinnon
1
0804731
Table of Contents
ABSTRACT ................................................................................................................................................................... 3
INTRODUCTION .......................................................................................................................................................... 3
RESEARCH QUESTION ................................................................................................................................................. 4
AIM ................................................................................................................................................................................. 4
OBJECTIVES........................................................................................................................................................................ 4
LITERATURE REVIEW ................................................................................................................................................... 5
SCRIPTING ......................................................................................................................................................................... 5
CASE-BASED REASONING ...................................................................................................................................................... 5
MACHINE LEARNING ............................................................................................................................................................ 6
DYNAMIC AI ...................................................................................................................................................................... 7
PROJECT OVERVIEW ................................................................................................................................................... 7
RESEARCH METHODS .................................................................................................................................................11
REQUIRED RESOURCES........................................................................................................................................................ 11
PROJECT TASKS & PROJECT PLAN .......................................................................................................................................... 11
FALL BACK AIM.................................................................................................................................................................. 12
ADDITIONAL TASKS ............................................................................................................................................................ 12
TESTING AND GATHERING DATA ............................................................................................................................................ 12
RELEVANCE OF PROJECT ............................................................................................................................................13
CONCLUSION .............................................................................................................................................................13
Kenneth D. Mackinnon
2
0804731
Abstract
The tactics available for use by an artificial intelligence (AI) player, in a real time strategy game,
are limited by design, technology and time. The aim of this project is therefore to analyse a variety
of techniques available to increase the number of strategies which can be employed by AI players in
real time strategy games, and to outline a system which could extend upon some of these
techniques. This project will test whether human players find a dynamic AI player more enjoyable
and challenging to play against than a statically scripted AI player. In conclusion, this project hopes
to analyse current AI techniques, building upon previous work to allow an AI player to enhance the
gameplay for human players. The project also addresses the question of whether it is computational
efficient or commercial viable to use these new techniques.
Introduction
Real time strategy games have existed in various forms since 1982, offering players the opportunity
to fight virtual opponents in real time. As technology has improved over the years, processing
power has increased exponentially. A large amount of processing power is dedicated to graphics,
leaving the AI to share the remaining processing time with input and game logic amongst other
things. The key problem this creates is that visually vivid, realistic environments and characters can
be created, however the lack of intelligence of entities within these environments has the potential
to break all immersion. This is due to the fact that the common approach to AI within games is to
statically script entities, so that they are functional enough to play with or against. However,
humans have the innate ability to recognise patterns and as such can easily find ways to bypass
static AI entities. This is a particular problem for real time strategy games, whereby in order to
create difficult opponents, AI is often simply given extra resources or construction speed is
increased.
Dynamic AI systems, which can employ a variety of different strategies, improve upon static
scripting systems by allowing the entity to choose a strategy from a database of different cases. This
allows a dynamic AI to change tactics based on the current situation, making it appear more
intelligent, and less rigid, than a static AI. Dynamic AI goes a long way towards removing the
problems of static AI, however eventually players will find a way to circumvent the system's
intelligence just as they can with static AI. This is because Dynamic AI is still limited in its options,
and realistically cannot be instructed to deal with every possibility. Hoekstra (Unpublished, pg.1)
considered this point, noting that “Dynamic scripting fails at some points because the scripts are
Kenneth D. Mackinnon
3
0804731
made from a rule database hard-coded by the developers”.
A possible solution to the problem with these systems is a Dynamic AI system with online learning
(henceforth referred to as DAL). A DAL system is an extension of dynamic AI which can quickly
generate new rules during game play and cease to use rules that have proven redundant. This
proposal outlines an experiment to compare a static AI system, a dynamic system and a DAL
system, in order to determine which system human participants prefer. This data will then be used to
determine whether it could be beneficially for developers to implement a DAL system.
Research question
Is the game play experience of a real time strategy game improved when using a DAL system rather
than a static or a dynamic system?
Aim
The present project aims to develop a system in order to compare and contrast a static scripted
model and the dynamic scripted model of AI with a DAL model, to test the hypothesis that a DAL
system will improve the game playing experience in a real time strategy game.
Objectives
-
Review current trends in AI in real time strategy games.
-
Identify problems associated with each technique.
-
Construct a test system and obtain performance data of the techniques versus each other and
human participants.
-
Critically analyse data and determine the pros and cons of each technique, including
computational efficiency.
-
Make recommendations on the commercial viability of each technique.
Kenneth D. Mackinnon
4
0804731
Literature review
This section aims to explain the focus of the project in more detail as well as talking more about the
background of the areas involved.
Scripting
Scripting is a method employed in games which can be utilized by the source code to enact actions
and set variables, amongst other functions. Commonly it is used in games to help create a desired
environment within a game world. Scripting can be used for anything from AI entity control to
configuration of weapons. The definition of a script is [searchenterpriselinux, 2000] “A program or
sequence of instructions that is interpreted or carried out by another program rather than by the
computer processor”. Modern game engines, such as Unity and the Unreal Development Kit have
scripting build in as an integral part of using the functionality. One key reason scripts are useful is
because scripted languages are easier to read and this allows non-programmers in a development
team to adjust aspects of a game through said scripts without detailed knowledge of the source
code. This project will utilize a custom scripting language and interpreter to run scripts to control
both the static, dynamic and DAL AI.
Case-based Reasoning
On the topic of Case-based reasoning and referencing (Kolodner, 1992) “In case-based reasoning, a
reasoner remembers previous situations similar to the current one and uses them to help solve the
new problem.” Case-based reasoning (CBR) can be used to effectively analyse a problem and
compare it to previous solutions in a database. CBR is becoming popular in games development as
a method for allowing AI entities different ways to do things based on the circumstances. E.g. if
cost, speed and size were factors, there could be a case for when cost is high, then spend less, or if
cost is low, speed is low and size is low then perform an action. What makes case based reasoning
different is that is requires the AI entity to assess the environment for relevant variables which could
affect its decision on which case to use. Because a CBR approach can have varying results
(although limited to the number of cases in the database) it can often appear to be intelligent and
can offer substantial re playability for players.
Kenneth D. Mackinnon
5
0804731
Machine learning
Machine learning is a technique used to allow AI agents to learn new behaviours, actions or
methods of using its domain of knowledge. “It seems likely also that the concepts and techniques
being explored by researchers in machine learning may illuminate certain aspects of biological
learning.” (Nilsson, 1998) Current methods in machine learning allow agents to resemble biological
learning abilities, the purpose of this is so that agents can adapt to their environment, which
biological life is adept at doing.
An example of machine learning in a commercial video game is Black and White (Lionhead games,
2001) which uses machine learning in combination with reinforcement learning to allow the player
to teach their 'creature' (see figure 1) how to behave and interact with objects. Partly because of the
'intelligence' of the creature the game garnered good reviews from video game reviewers such as
metacritic, gamespot, IGN, amongst others, mostly scoring 90% or above. “reinforcement learning
concentrates on modelling reward-based learning in animals” (Barnes, Hutchens, 2002,) This AI
architecture resembles reinforcement techniques often used with animals to teach behaviour,
language and even perform tricks.
Figure 1: The creature character from the video game Black-and-White developed by Lionhead
games.
Kenneth D. Mackinnon
6
0804731
Dynamic AI
In the paper Adaptive Game AI with Dynamic Scripting [Spronck et al,2006] is the specification of
an adaptive AI system which uses reinforcement learning to create an AI agent which can learn
during a game. The learning process involves running scripts from a predefined rule base based on
their weights, their effectiveness is measured based on the success/failure rate and therefore
adjusted accordingly. This leads to AI which can perform well against many opponents because
tactics which fail are used less often and successful tactics are used more often, much like humans
would perform in the same circumstances. The experiment discussed within the paper shows that
this system of dynamic scripting can effectively be used to overcome the intelligence of statically
scripted agents and suggests that it could be used against human opponents qualifying it for
implementation in a relevant commercial computer game.
The Dynamic AI system is further improved in the paper Rapid and reliable adaptation of video
game AI. “We define case-based adaptive game AI as an approach to game AI where domain
knowledge is gathered automatically by the game AI” [Bakkes, S. Spronck, P. van den Herik, J.,
2009]. This paper extends dynamic scripting by using case-based reasoning in conjunction with
online learning to create an AI agent which can consider strategies of previous games. This
approach also focuses on analysing every aspect of the opponent to determine what kind of
opponent it is facing and how to counteract it. This system does not actively change the database of
cases during play, but searches through it to find and use a similar situation to the current game. The
system also has offline learning (done out-with play) which allows the agent to index its
performance and match variables for future use in the database. The experiment in this paper uses
the open source game Spring [http://springrts.com/], to demonstrate the system and through
experiments proved that the method can be used to greatly increase the strategic effectiveness of an
AI player against static AI players without being computationally expensive.
Project overview
This project proposes to build upon the Dynamic scripting method and extend it to include
reinforcement machine learning. This system we will call a DAL system. The hypothesize is that if
a case-based reasoning (CBR) agent is unable to find a case within its database which can
appropriately be used to adapt to the situation, then according to the method described in Rapid and
reliable adaptation of video game AI the entity will be unable to improve itself and could possibly
perform irrational or irrelevant actions in an attempt to perform effectively until the agent can
perform offline learning. If online learning can be implemented to allow an agent to learn a new
Kenneth D. Mackinnon
7
0804731
strategy during play due to a lack of relevant cases or an opponent using a surprising new tactic then
immediately the agent can counteract providing a greater challenge to any opponent.
Figure 2: Architecture overview
Figure 2 shows how the architecture of the DAL system should work. Black arrows show which
modules of the code will run every frame. The green arrow shows that the case database module
will only be used if the objective has changed or is being failed. The red arrow shows that the
learning module only becomes active when the case database cannot provide an acceptable case.
Orange arrows indicate when the application can return to the game loop, this occurs when a
module successfully completes its task. E.g. the case database module finds a suitable case.
Kenneth D. Mackinnon
8
0804731
Table 1: Structure of data for each rule in the action database.
Table 1 gives an example snippet of a database, rules will be ordered by an ID, have success rates,
number of units and types of units associated with each rule. For example if the system decided that
it needs to harass the opposing player, then it could choose between rule 6 & 7. Which rule is
chosen will depend on current availability of specified units, success rate of each rule and whether
there is enough resources to build enough units.
Figure 3: Hierarchy of command
Kenneth D. Mackinnon
9
0804731
Figure 3 shows that objectives will be handled by distributing orders to units. This is how all three
AI players will function, allowing individual unit AI to be separate from the overarching player AI.
Each unit will have a task string; if this string is assigned then the unit will follow routines relevant
to its task. For example a unit that has a task ‘Defend My Base’ will move to somewhere around his
base. Then it will fight any enemy units that come in range of it.
The test bed application will be a two dimensional top down perspective real time strategy game
built using Microsoft's XNA framework. The application is intentionally not as complex as a full
scale commercial game. As such, there will only be four types of units, player’s bases, and
necessary terrain to distinguish levels and resources which can be captured. Figure 4 illustrates the
three key objects in the game. Circles are player bases which must be destroyed to achieve victory,
squares are units and triangles are resource points.
Figure 4: Application environment layout
In order to effectively measure the computational expense of online learning, the DAL system will
be run in a separate thread from the rest of the source code, allowing calculations to be quick and
not frame dependant. However if time permits then a non-threaded version of the source code will
be constructed to more thoroughly evaluate the performance overhead of the DAL system.
Kenneth D. Mackinnon
10
0804731
Research Methods
Required resources
A computer with Microsoft visual studio 2008 and the XNA game studio installed.
Project Tasks & project plan
This section contains the tasks of this project and a Gantt chart (Figure 5) which shows a projected
time frame for each task. The project plan leaves plenty of room for error and does suggest a fallback aim should time, technology or circumstances become problematic. A breakdown of the
documentation task, giving time scales for each task within it is displayed in figure 6.
1) Review of techniques
2) Construction of application
3) Creation of static AI player
4) Dynamic scripted AI
5) Development of DAL system
6) Testing and optimisation
7) Documentation (of which includes):
Introduction

Methods

Literature review

Development documentation

Experiments

Results/ Analysis

Conclusion
Kenneth D. Mackinnon
11
0804731
Figure 5: Gantt charts of projected time frames
Figure 6: Time scales of documentation tasks
Fall back aim
If the DAL system development task is not completed then the dynamic scripted system can be
tested for the purposes of answering the question of whether human participants prefer a changing
AI player over a static AI player.
Additional tasks
If time permits then several static AI agents will be constructed which use classic strategies for RTS
applications such as Rushing, Stream rolling, Raiding, amongst others. The reason for this would be
to help train the DAL system and could give more valuable data for analysis and comparison.
Testing and gathering data
Test data will be gather in two ways. Firstly the static AI agent will be matched against the DAL
agent and compared to establish whether the system can adapt to static opponents and its
effectiveness. Secondly human players will be tested against both static and DAL players across a
Kenneth D. Mackinnon
12
0804731
selection of maps. Human players will then answer a questionnaire about their experience with this
genre of game, how challenging each AI was and which AI was more enjoyable as an opponent. See
appendix A for the proposed questionnaire.
Relevance of project
The relevance of the project is that it seeks to establish whether online learning combined with
dynamic scripting can be used to provide a truly adaptive AI which would improve the user
experience of the players. In addition to this the project will provide an insight into the problems
associated with this genre of games AI design. Furthermore among concerns for the games industry
is that research and development of new and novel approaches are often expensive and high risk
resulting in incremental improvements of systems over multiple titles. If the DAL system is
efficient, performs better than a static scripted system and provides interesting game play then this
system could be considered for a commercial application. In addition if the DAL system is effective
in proving an adaptive AI then it could be applied to many other areas such as military war
simulation, telecoms-medicine and advanced metrics software. But the main point of this project is
to look at ways to allow real time strategy game AI to adapt to game play and players.
Conclusion
To conclude, this project aims to analyse current trends of AI used in real time strategy games, build
a test system which implements a DAL system, contrast and compare this technique to static
scripting from a commercial and computational approach and finally, discover which human
participants prefer.
Kenneth D. Mackinnon
13
0804731
Appendix A - Questionnaire
Dynamic artificial intelligence systems with online learning
Thank you for taking part in this project. For this project you will be required to play a real time
strategy game and will face three different artificial intelligence controlled players, then answer
questions based on the experience.
You are under no obligation to complete any testing or questionnaires and may leave at any time.
Demographic data
Please tick the appropriate box
What is your experience of real time
strategy games?
How many hours per month do you play real time
strategy games?
 No experience of playing
 Occasional player
 regular player
0
 1-5
 5-10
 10+
AI behaviour
Please tick one box for the answer that you feel is most relevant
Strongly agree Agree
Neutral Disagree Strongly Disagree
I found that player A was more
challenging than player B.





I found that player C was more
challenging than player B.





I found that player B was more
challenging than player A.





I found that player B was more
challenging than player C.





I found that player A was more
challenging than player C.





I found that player C was more
challenging than player A.





I found player A enjoyable as an
opponent.





I found player B enjoyable as an
opponent.
I found player C enjoyable as an
opponent.










Kenneth D. Mackinnon
14
0804731
References
Bakkes, S. Spronck, P. van den Herik, J. , 2009, Rapid and reliable adaptation of video game AI
[Online]. Available at:<http://ticc.uvt.nl/~pspronck/pubs/BakkesTCIAIG.pdf>[accessed 12
Oct 2011]
Barnes, J. , Hutchens J. , 2002, Testing Undefined behaviour as a result of learning. In S. Rabin ed.
2002, AI game programming wisdom. Massachusetts, USA, Charles River Media, ch.11.7
Definition of scripts, 2000, www.searchenterpriselinux.com[online] Available at:
<http://searchenterpriselinux.techtarget.com/definition/script>[Accessed 15 October 2011].
Hoekstra, C, Unpublished. Adaptive Artificially Intelligent Agents in Video Games: A Survey,
[online]Available at: <http://citeseerx.ist.psu.edu/viewdoc/download?
doi=10.1.1.137.4978&rep=rep1&type=pdf>[Accessed 15 October 2011].
Kolodner, J.L. ,1992. An Introduction to Case-Based Reasoning, [e-journal], pp.2, Available
through: <http://web.media.mit.edu/~jorkin/generals/papers
/Kolodner_case_based_reasoning.pdf> [Accessed 16 October 2011].
Nilsson, N.J., 1998, Introduction to machine learning[pdf] Available at:
<http://ai.stanford.edu/~nilsson/MLBOOK.pdf>
Spronck, P., et al, 2006. Adaptive Game AI with Dynamic Scripting. [Online]. Available at:
<http://ticc.uvt.nl/~pspronck/pubs/DynamicScripting.pdf>[accessed 13 Oct 2011]
Kenneth D. Mackinnon
15
0804731
Bibliography
Bourg, D. M. & Seemann G., 2004. AI for Game Developers. Sebastopol: O'Reilly Media Inc. Ch.
15
Buckland, M., 2004. Building Better Genetic Algorithms. In Rabin, S., ed. AI Game Programming
Wisdom 2. Massachusetts: Charles River Media. Ch. 11.4
Champandard, A. J. , January 4, 2011,This Year in Game AI: Analysis, Trends from 2010 and
Predictions for 2011<http://aigamedev.com/open/editorial/2010-retrospective/ >
Champandard, A. J., June 20th 2008.The Anatomy of an AI Architecture for Game
Characters<http://aigamedev.com/open/article/anatomy-ai-architecture/>
Hammond, K. , December 4, Adaptative AI engine for RTS Games, [online]
<http://rtsairesearch.wordpress.com/>
Kasavin, G. ,31 March 2001, Black & White Review [online]
<http://uk.gamespot.com/pc/strategy/black-and-white/review.html>
Lindh, D. , Adaptive combat AI in strategy games - An approach based on Dynamic Scripting,
Diploma, Lund University <http://fileadmin.cs.lth.se/ai/xj/DavidLindh/report.pdf>
Metacritic, Review: Black & White [online]<http://www.metacritic.com/game/pc/black-white>
Pfeifer, B., 2004. Narrative Combat: Using AI to Enhance Tension in an Action Game.
Schaeffer, J., 2001, ‘A Gamut of Games’. Artificial Intelligence Magazine 22(3),
29–46. <https://www.aaai.org/ojs/index.php/aimagazine/article/view/1570/1469>
Van Lent, M. et al, Increasing Replayability with Deliberative and Reactive Planning
<http://www.dtic.mil/cgi-bin/GetTRDoc?Location=U2&doc=
GetTRDoc.pdf&AD=ADA459203>
Kenneth D. Mackinnon
16
0804731