Download Dynamic Learning Approach to Artificial Intelligence Based

International Journal of Emerging Technology and Advanced Engineering
Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 4, Issue 4, April 2014)
Dynamic Learning Approach to Artificial Intelligence Based
Generated Combat Game Tactics
Baljinder Singh1, Bhawna Arora2, Simranjit Singh3, Barinder Singh4
1,2
Assistant Professor, Department of Computer Applications, SSCET, Amritsar, Punjab
3,4
B.Tech (CSE), SSCET, Amritsar, Punjab
As the current emphasis is on the games that get better
the longer we play them. So, Adaptive Game AI with
online learning has capabilities of automatically adapting
the behavior of Computer Controlled Opponent, to keep
game interesting for the Human-Player. By constantly
learning new strategies Adaptive Game AI can
potentially increase the quality of game AI.
Since the start of game development, game developers
have always used game AI for game characters to make
them appear intelligent. It can be ghosts in the classic
game of PAC man or sophisticated bots in first_person
shooter game Half_life. Aim to create AI entities that
reach human_level intelligence to outperform human
beings. But this high_level of intelligence is not
acceptable in game AI because such high_level of
intelligence might not give human players a chance to
win. So, if a player cannot advance in levels or if game
play does not progress eventually a player will lose
interest in the game. So, one can formulate that the goal
of game AI is to be knocked out by human player in most
challenging, impressive and unprecedented manner.
Abstract-- Game (Artificial Intelligence) is defined as the
decision making process of computer-controlled opponent
in video games. To improve the entertainment value of
games, Adaptive game AI can be used to allowing
computer-controlled opponents to fix weaknesses
automatically in the game AI, and to respond to changes in
human-player tactics. Dynamic Scripting is a reinforcement
learning technique succeeds in endowing computercontrolled opponents with adaptive performance. Dynamic
Scripting gives illusion of Human_like behavior; it plays the
game for a while, evaluates its own behavior, and builds up
a tactic from behaviors that it has been successful with so
far. If it starts to lose, it quickly adapts and discards current
tactics, trying to learn how to deal with the new threat. In
Dynamic Scripting rule_bases are manually designed using
domain_specific knowledge. Manually designing these
knowledge bases may be time_intensive, and risks errors in
analysis and encoding. In this paper we generate the tactics
for the knowledge rule of Dynamic Scripting fully
automatically by using Evolutionary algorithm technique.
Keywords- Artificial Intelligence, Dynamic Scripting,
Reinforcement learning, First Person Shooter, Finite State
Machine, Zone Impact, Self Defense, Impact Factor,
Intelligence Factor, Fire Action Time, Application
Programming Interface, Real Time Strategy Game,
Computer Role Play Games, Artificial Neural Network.
Current Usage of AI in Games
Here in this paper we study the level of AI currently
used in video games. So, we selected a few of the most
popular games and examined them. It turns out that the
majority of AI is scripted or hard coded in logic without
using any academic AI techniques.
I. INTRODUCTION
Popularity of commercial computer games is directly
related to their entertainment value. Games pose
challenges for the human to overcome, usually in the
form of combat. And the decision making process of
computer controlled opponent in combat is refers as
Game AI. The goal of computer controlled opponent to
get defeated by its human-player in more challenging,
impressive and unpredictable manner. So if the Game AI
manages to keep players indulge in the game, we call it
“entertaining”. However, the time allocated to develop
game AI is typically short; but with the growing power of
CPUs, and as computers becoming cheaper, it's time for
game developers to dedicate significant CPU cycles to
employ academic AI techniques. A typical approach for
creating AI components in a game is using Finite State
Machine. But after some time Human Players can
observe predictable and deterministic Game Character
and exploit weaknesses in the Character behavior and the
very next, game loses its appeal.
Age of Empires
Age of Empires is a real-time strategy game. In Age of
Empires a player builds kingdoms and armies with a goal
to survive through ages, defend his own kingdom and
conquer other kingdoms. A player starts with few
workers and his/her primary building and then slowly
builds a complete town and many buildings. AI of Age of
Empires is smart enough to find and revisit other
kingdoms and their buildings, i.e. path finding, and also
start attacking them. One of the major drawbacks with
this game is that AI is too predictable.
Half-life
Half-life is a science fiction first-person shooter game
(FPS). Half-life takes shooter games to the next level of
intelligent behavior of enemy monsters and won 35
„game of the year‟ awards in 2004.
66
International Journal of Emerging Technology and Advanced Engineering
Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 4, Issue 4, April 2014)
Half-life is a more or less a traditional game in which
a player advances by killing creatures coming in his way.
Monsters in this game can hear, see, and track human
players
Path finding is necessary for almost every video game,
for navigating the computer player in a complex
environment. Fortunately, an effective solution to this
problem is provided by A* algorithm which is one of the
most commonly used algorithms by game programmers
(J. Matthews, 2002)( D. Higgins, 2002).
Formally, A* is a best_first graph search algorithm to
finds the path with least_cost between given initial node
and to one goal node using distance_plus_cost heuristic
function (f(x)). A* uses two functions to compute
distance_plus_cost function:
• Path_cost_function(g(x)): calculates the cost from
start note to current node.
• Heuristic estimate (h(x)): a heuristic function to
estimate the cost from current node to goal node.
So, f(x) = g(x) + h(x) is used to determine a promising
path.
Fuzzy logic is derived from fuzzy set theory. It is a
method of dealing with reasoning using approximate
values rather than precise crisp values. On the contrary to
conventional set theory, fuzzy theory uses membership
values in the range between 0 to 1 instead of using 1 and
0. These membership values are used to represent
linguistic variables like short, average and tall. The main
idea behind fuzzy logic is to represent a computer
problem in the way a human being would. Consider an
example of Combat Bot, which got various inputs like
Hit Point, Enemy Unit, etc. Based on these parameters a
bot needs to decide whether to Attack or not. So, a
membership function for this type of system is shown in
table 1.
Bots in Computer Game
Bots in Counter-strike, also called NPCs, are used to
replace human players. Bots play as a part of the team
and achieve goals similar to humans. Bots simulate
human players and are aimed to give game players the
`illusion' of playing against actual human players.
Currently, bots used in Counter-strike are programmed to
find path, attack opponent players, or run away from the
site if they have heavy retaliation, providing an illusion
that they are intelligent. Similar species of bots are used
in many other FPS games, with similar methods of
programming. Bots are usually preprogrammed
according to the requirements of a game and play for or
against human players. Based on the method of
programming, there can be two styles of bots (Patel, P.,
2009).
1. Static: Static bots are static in the levels and maps
have already been processed. This means that they
need to have all information about the maps and
level prior to the start of game.
2. Dynamic: Dynamic bots learn as they go through
the level. They can be played at any level while
static bots cannot.
An FSM for Bot of Combat Game is quite simple like
firstly making initial decision i.e. game strategies,
selecting weapon etc. and the start searching for enemy.
After the enemy is spotted, it makes a transition to attack
state where it fires bullets at enemy.
Table 1
Fuzzified Bot's State
Table 2:
Fuzzy Rules
Figure 1: FSM for Bots in Combat Game.
67
International Journal of Emerging Technology and Advanced Engineering
Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 4, Issue 4, April 2014)
This kind of fuzzy system model reduces the time and
effort game programmers require for configuring crisp
values. Also, the number of rules needed in a Bot can be
significantly reduced using a fuzzy system.
III. PROBLEM AND APPROACH
In the past mainly focused on sound and graphics. But
by setting aside old „Finite State Machine‟ and using
advanced Adaptive Game AI techniques, the trend is
shifting and AI is often assigned an equal priority to
graphics and sound. So the next breakthrough for Game
Industry will be with Computer Controlled Opponent
character behave realistically and that can learn and adapt
rather than just more resolution textures and more
frames-per-second. Major problem area was to produce
an AI which learned online i.e. allowing the game to
adapt its fighting style mid-game if a Human-Player
decided to try a new technique. A novel technique called
Dynamic Scripting successfully endowed Computer
Controlled Opponent with Adaptive performance.
Dynamic Scripting has already been shown to be
successful for CRPGs (Spronck, 2003), but not yet for
Fighting games. As we expect that it is applicable to
Fighting games, we decide to focus on these for our
research.
II. BACKGROUND STUDY
Spronck et al. (2005) evaluates Dynamic scripting
performance in experiments in which Adaptive game AI
agents are pitted against a collective of manually
designed tactics in a simulated computer Role Playing
Game. With the enhancement, dynamic scripting meets
all computational and functional requirements. (Spronck,
P., 2005).
Szita, I., et al. (2008) described learning methods
which can increase adaptively, and diversity, while
retaining playing strength. In practice, random actions are
also often applied to make AI more diverse and
unpredictable. However, the style of game AI still
remains predictable because it is rarely affected by this
kind of randomness. (Szitz, I., 2008).
Andrade, G. et al. (2005) provide mechanisms to
dynamically balance the difficulty level of the games in
order to keep the user interested in playing. First, the
game must, as quickly as possible, identify and adapt
itself to the human player‟s initial level, which can vary
widely from novices to experts. Second, the game must
track as close and as fast as possible the evolutions and
regressions in the player‟s performance. Third, in
adapting itself, the behavior of the game must remain
believable, since the user is not meant to perceive that the
computer is somehow facilitating things. (Andrade, G.,
2005).
Spronck et al. (2004) experimental result indicated
that, the time needed by Dynamic Scripting to generate
effective opponents becomes unacceptably long.
(Spronck, P., 2004).
The performance of Adaptive Game AI, by using an
offline evolutionary algorithm to evolve new tactics that
are able to deal with optimized tactics which Dynamic
Scripting cannot defeat using its original rule base. But
the translation of the evolved solutions to improvements
in domain knowledge was done manually (Ponsen, M.,
2004). Few attempts have been made to automatically
create game AI.
Madeira et al. (2004) focused on a new approach to
the semiautomatic generation of a game AI using
Machine Learning techniques i.e. used reinforcement
learning as a basic technique, and bootstrapping the
learning process by letting the learning Game AI play
against another AI to get more and more control of the
decision-making. To the very best Ponsen et al. (2005)
introduced a novel methodology that uses an
evolutionary algorithm to automatically generate highquality domain knowledge used by dynamic scripting
(Ponsen, M., 2005).
Problem statement: “To What Extent Dynamic
Scripting Technique Can Be Improved, In Order to
Improve Adaptive Behavior of Computer-Player in
Combat Games”.
Player is the term used to point human being playing
the computer game.
Avatar is the game‟s representation of a player.
Game World is the virtual universe where the game
takes place.
Game Objects are the objects that appear in computer
game. Game objects can either be active or reactive.
Active means that the object can actively conduct actions
upon itself or upon the game world. Reactive means that
the object cannot by itself perform actions, but only react
as an effect of an action.
Characters are active game objects situated in the
game world.
Non-Player Characters are those characters who
participate in the game‟s theme. They are considered as
side character in game.
Participants are active game objects which assume
the role of a player.
Support Routine is software that assists the player in
playing the game.
Game AI is the term used to describe the commercial
game industry‟s methods for controlling active game
objects. Game AI includes methods which create
interesting auto-generated worlds and create opposing
participants and characters for the player to face in the
game. In our search for an appropriate environment, we
took the following list of requirements into account:
68
International Journal of Emerging Technology and Advanced Engineering
Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 4, Issue 4, April 2014)
1. The game environment required being easily accessible
and changeable.
2. The game environment should include a scripting
language, preferably with a sophisticated AI API, able
to support learning techniques.
3. Preferably experiments in the environment should be
fast.
4. The game environment must be state-of-the-art in
terms of game play, meaning that game will have to
incorporate non-trivial AI. Here for this research we
opted to build own game in java for implementing
Dynamic Scripting in Fighting games.
3.1 Design description of Battle City Game
Aim of the proposed system is to successfully
implement reinforcement learning using Dynamic
Scripting in Java Game specifically created to show the
learning process.
Figure 3: Transitions between the various game states
Java Battle City Game is quite simple like firstly start
the battle game then the player enters the decision stage
i.e. game strategies, explosion etc. and the start searching
for enemy. After the enemy is spotted, it makes a
transition to attack state where it fires bullets at enemy.
Figure 2: Battle City Game
3.2 Battle City Game Terminology
In this Java Game we have three enemy tanks that
move in their specified zone and only attack when the
Player tank enters their zone. The Game consists of three
Stages. In first stage the game EnemyTank learn about
Explosions Position and next time when the player
detected at that position, the EnemyTank attack the
Player.
Figure 4: Analyzer Thread Activity Diagram
69
International Journal of Emerging Technology and Advanced Engineering
Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 4, Issue 4, April 2014)
The Activity Analyzer generates rules based on the
explosion i.e. it sense the position at which the enemy
was killed as that particular position including margin is
added to the dangerous Zone and that specific Rule is
added to the Rule-List.
It maintains several rule-bases, one for each opponent
type in the game. These rule-bases are used to create new
scripts that control opponent behavior every time a new
opponent is generated. The probability that a rule is
selected for a script is influenced by a weight value that
is associated with each rule. The release adapts by
changing the weight values to reflect the success or
failure rate of the corresponding rules in scripts. The size
of the weight changes is determined by a weight-update
function. Figure 6 describe the dynamic scripting for a
Combat game.
Figure 6. Flow of Dynamic Scripting
IV. EXPERIMENTAL RESULTS
In this Paper we have compared the Simple game with
Dynamically generating Script Game AI in Battle City
Java game. Key issues with any learning techniques is
setting various parameters, which in case of
Reinforcement-learning are Zone Impact( ), Self
Defense( ), and Impact Factor( ) . Zone Impact
calculated to determine the strength of attack.
Figure 5: Learning Process State diagram
3.3 Automatically Generated Tactics in Battle City
Reinforcement algorithm technique is used for
automatically generating knowledge bases for dynamic
scripting. Reinforcement algorithm searches for
strategies which can defeat specific opponent strategies.
Reinforcement Learning is a type of Machine
Learning that allows machines to automatically
determine the ideal Strategy, i.e. to maximize the
performance of the system. The feedback process is
necessary to use in this type of learning because the agent
have to learn its behavior, and this is basically known as
the reinforcement signal.
Like in AI games robot player repeatedly observes the
state of its environment, and then chooses and performs
an action in the standard framework of reinforcement
learning. The respective action performed can changes
the state of the world, and the agent also obtains an
immediate numeric result. Positive outputs are rewards
and negative outputs are punishments. The agent must
learn to choose actions so as to maximize the future
results it will receive. Like in AI gaming the Bot player
will be able to choose preventive actions as it learns from
the agent's behavior.
Dynamic scripting is an unsupervised reinforcement
online learning technique for commercial computer
games.
Algorithm 1: Zone Impact Factor Calculation Algorithm
1.
Set Count=0;
2.
For i=1 to N
3.
4.
If x <= xi+10 or x >= xi-10
Count++;
5.
End if
6.
End for
7.
impactfactor = Count;
8.
return impactfacot
Self Defense automatically fire the bullet when the
Player enters the dangerous Zone calculated thought
reinforcement learning.
Algorithm 2: Self Defense Automated
1.
Set inDanger = false;
2.
For i=1 to N
3.
70
If x <= x+10 or x >= x-10
International Journal of Emerging Technology and Advanced Engineering
Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 4, Issue 4, April 2014)
4.
inDanger=true;
5.
Break;
6.
Here, Impact Factor is incremented as the Player
learns during its hitting the enemy. As Impact Factor is
more than zero the Player learns and FireAction is
performed automatically when the Player enter the
Danger Zone.
End if
7.
End for
8.
If(inDanger)
9.
ImpactFactor = calculateImapact(x,y)
10.
Fire(Impact Factor)
Dynamic Experiment Result
Here the Player learns smartly through Reinforcement
Learning and the Player gets better as it moves to next
Stage. So in case of Dynamic Script Experiment
IntelligenceFactor is more than Zero as the Game is
learning new tactics and FireActionTime will be small
as Player need not to manually fire when it enters the
dangerous zone beside fire is automatically fire when AI
sense that the Player is entered the dangerous zone.
11. End if
Impact Factor Respective Firing
Algorithm 3: Fire (ImpactFactor)
1.
For i=1 to ImpactFactor
2.
fire;
3.
End for
Firstly in Simple Non-AI Game the player is learning
nothing and as it moves to next stage, the intelligence of
the player is same and the chances of losing the game
with proceeding further is increased. And due to that the
Player is easily get defeated.
Static Experiment
The static AI is used in game with no OnlineLearning. In static Script case all levels are priory set and
maps have already been processed. During the game
three playing factors are taken into consideration i.e.
Movement, Direction, and Fire. So Enemy Hit depends
upon on these three factors.
Figure 7. BattleCite Game
Player in South-West Zone:
At start when the game is just started the
FireActionTime is high as the Player has to hit
manually. IntelligenceFactor (IF) will be:
IF= ImpactFactor / 3 ;
Intially ImpactFactor is Zero;
So, IF= 0/3 = 0
Now when the Player Kill One Enemy then,
IF= 1/3 = 0.33
So after killing one Enemy the Rule 1 is generated
with Dangerous Zone detection with position x and y.
Now after killing TWO Enemy then,
ImpactFactor= 2/3 = 0.66
Static Experiment Result
The static scripted Player fought the standard game
enemies. So here we evaluate two factors i.e
IntelligenceFactor and FireActionTime. So in case of
Static Script Experiment IntelligenceFactor is Zero as
the Game is learning nothing new tactics and
FireActionTime will be high as Player has to manually
fire when it enters the dangerous zone.
Dynamic Experiment
In Dynamic game AI the Player Online-learn tactics of
game play. So here the Player learn as they go through
the levels. So IntelligenceFactor is calculated as the
Player learns more and more from its dynamically
created Explosion rules.
IntelligenceFactor = Impact Factor / [ Movemement +
Direction + Fire ]
71
International Journal of Emerging Technology and Advanced Engineering
Website: www.ijetae.com (ISSN 2250-2459, ISO 9001:2008 Certified Journal, Volume 4, Issue 4, April 2014)
REFERENCES
[1]
[2]
[3]
[4]
[5]
Figure 8. Rule Formulation
[6]
So after killing Second Enemy a one more Rule 2 will
be generated describing the Second dangerous Zone. So
the more we kill enemy tank the more our Intelligence
factor increase and thereby more Dangerous Zone is
detected by Player.
[7]
[8]
[9]
[10]
[11]
[12]
[13]
Figure 9. Dangerous Zone Detection
V. CONCLUSION
[14]
From the survey and self observations, it is concluded
that the adaptive AI in Battle City certainly added some
value to the game. It made the Artificial Intelligence
seem more intelligent to the player, and slightly increased
the game-play. One of the goals of this paper was to help
realize the practicality of machine learning in game AI.
The fact that Battle City used four adaptive Artificial
Intelligence models in various ways and that the
Artificial Intelligence system and the game as a whole
were enhanced because of it, shows that machine
learning can, and should be a major consideration in the
design of a video game AI system.
[15]
[16]
[17]
[18]
72
Andrade, G., Ramalho, G., Santana, H., & Corruble, V. (2005),
“Extending Reinforcement Learning to Provide Dynamic Game
Balancing”, Proceedings of the 2005 IJCAI Workshop on
Reasoning, Representation, and Learning in Computer Games,712.
Bakkes, S., Spronck, P., & Herik, J.V.D. (2009), “Rapid and
Reliable Adaptation of Video Game AI”, Ieee Transactions On
Computational Intelligence And Ai In Games,2(1),73-104.
Cheng, D., & Thawonmas, R. (2004). “Case-based plan
recognition for real-time strategy games”. Proceedings of the 5th
International Conference on Intelligent Games and Simulation
(GAME-ON-04), 36–40.
D. Higgins. “Generic A* Path finding.” AI Game Programming
Wisdom by Steve Rabin. Charles River Media, Inc., 2002, pages
122-132.
Demasi, P., & Cruz, A. (2002). “Online co evolution for action
games”, Proceedings of the 3rd International Conference on
Intelligent Games and Simulation (GAMEON-02),113–120.
Graepel, T., Herbrich, R., & Gold, J. (2004, January). “Learning
to fight”. In Proceedings of the International Conference on
Computer Games: Artificial Intelligence, Design and Education”
(pp. 193-200).
J. Matthews. “Basic A* Path finding Made Simple” AI Game
Programming Wisdom by Steve Rabin. Charles River Media, Inc.,
2002, pages 105-113.
Jones, J. (2003). Benefits of Genetic Algorithms in Simulations
for Game Designers. Thesis, School of Informatics, University of
Buffalo, Buffalo, USA.
Laird, J. E. van Lent, M (2000). “Human-Level AI's Killer
Application: Computer Game AI”. Proceedings of AAAI 2000
Fall Symposium on Simulating Human Agents, 80-87.
Lee-Urban, S., Ponsen, M., Muñoz-Avila, H., W. Ah a, D., (2005)
“Stratagus: An Open- Source Game Engine for Research in RealTime Strategy Games”, Proceedings of the 2005 IJCAI Workshop
on Reasoning, Representation, and Learning in Computer Games,
78-83.
Madeira, C., Corruble, V. Ramalho, G. Ratich B.
(2004).“Bootstrapping the Learning Process for the SemiAutomated Design of Challenging Game AI”, Proceedings of the
AAAI-04 workshop on AI in games, San Jose 2004.
McPartland, M., & Gallagher, M. (2008, October). “Learning to
be a bot: Reinforcement learning in shooter games”. In 4th
Artificial Intelligence for Interactive Digital Entertainment
Conference, Stanford, California (pp. 78-83).
Overholtzer, C.A. (2004). “Evolving AI Opponents in a FirstPerson-Shooter Video Game”, Thesis, Computer Science
Department, Washington and Lee University: Lexington, VA.
Patel, P., & Hexmoor, H. (2009, May). “Designing BOTs with
BDI agents”. In Collaborative Technologies and Systems, 2009.
CTS'09. International Symposium on (pp. 180-186). IEEE.
Ponsen, M., & Spronck, P., (2004)”Improving Adaptive Game Ai
With Evolutionary Learning”.
Ponsen, M., Muñoz-Avila, H., Spronck, P., W. Aha, D., (2005)
“Automatically Acquiring Domain Knowledge For Adaptive
Game AI Using Evolutionary Learning”, American Association
for Artificial Intelligence.
Spronck, P., Sprinkhuizen-Kuyper, I., & Postma, E. (2004).
“Online adaptation of game opponent AI with dynamic scripting”,
International Journal of Intelligent Games and Simulation, 3(1),
45–53.
Tozour, P. (2002). “The Perils of AI Scripting.” AI Game
Programming Wisdom, Charles River Media, Hingham,MA, pp.
541–547.