Download Comprehensive Overview of Game AI Competitions

Comprehensive Overview of Game
AI Competitions: History and
Ecosystems
Discussion
Sehar Shahzad Farooq
Group Meeting Feb 03, 14
Game AI Competitions
• Game AI Test bed for AI and CI research
• Competitions Tool to promote research and
education in Challenging problems
• Impact Provide Goals, rules, platforms and
comparisons
Why
• Game AI Competitions provide a fair,
transparent and reusable benchmarking
algorithms
• Allow an easy way to get in research field for
new researchers
• Problem solving
What
• Classical board games Chess, Checkers, Go, Othello
• Strategy games Planet Wars (google competition),
Defcon, ORTS, StarCraft,
• Car Racing Tactics(overtaking), Strategy( damage,
fault management)
• Console Games Ms. Pac-Man, Mario AI
Championship, Geometry Friends, Xpilot
• First Person Shooter Games Unreal Tournament 04,
2K BotPrize, Deathmatch competition,
• Hybrid gamesPhysical TSP (Racing, TSP)
Where
• CIG Conference on Computational Intelligence and
Games
• AIIDE Artificial Intelligence for Interactive Digital
Entertainment
• CEC Congress on Evolutionary Computation
• WCCI World congress on Computational Intelligence
• GECCO Genetic and Evolutionary Computation
Conference
• Evostar European events on evolutionary
computation
• Fuzz IEEE, AJCAI, IJCAI (Angry Bird), SSCAI (StarCraft)
History/ When
•
•
•
•
•
•
•
•
•
•
Car Racing (07-13)
StarCraft(10-13)
Angry Bird (12-13)
Physical TSP(12-13)
Fighting Game AI, Geometry Friends (13)
Super Mario (09-12), Platformer Game AI (13)
Ghosts Challenge (13)
Ms. Pac-Man (07-11), Ghost Vs PaMan(11,12)
Unreal Tournament (08-12)
GO (5-12), Othello (6-8), X-Pilot (07)
2007
2008
2009
WCCI
CEC
O
O
O
O
O
O
O
CIG
O
Fuzz-IEEE
O
CIG
O
O
O
GECCO
O
WCCI
O
O
CEC
O
O
CIG
O
O
GECCO
O
O
O
O
O
O
O
AIIDE
O
WCCI
O
O
O
CIG
O
O
O
Evostar
O
GECCO
O
ICE-GIC
O
O
O
O
O
AI
Platformer game
Physical TSP
AI
Fighting Game
Ghost AI
Angry Bird
Rubik's Cube
O
O
ICE-GIC
2010
Friends
O
O
Geometry
CEC
2006
X-Pilot AI
O
2005
Othello
O
CIG
Go
CEC
Starcraft
2004
Tournament
O
Unreal
Car Racing
CEC
Super Mario
Conference
2003
Ms Pac-Man
Year
Game AI Competitions
CEC
2011
O
O
O
O
AI
Platformer game
Physical TSP
AI
Fighting Game
Ghost AI
Angry Bird
Rubik's Cube
Friends
Geometry
X-Pilot AI
Othello
O
O
O
SSCAI
O
O
Fuzz-IEEE
Evo
Go
O
AIIDE
CIG
Starcraft
Tournament
O
WCCI
GECCO
Unreal
Super Mario
Ms Pac-Man
Car Racing
Conference
Year
Game AI Competitions
O
O
CEC
WCCI
O
O
O
AIIDE
SSCAI
2012
O
O
O
O
CIG
O
GECCO
O
O
O
O
O
AJCAI
O
O
Fuzz-IEEE
Evo
O
CEC
2013
AIIDE
O
SSCAI
O
CIG
GECCO
O
O
O
Fuzz-IEEE
IJCAI
O
O
Game AI Competitions, Organizers,
Participants and Winners (affiliations)
Game
Year
2013
2012
2011
2010
Car Racing
Conference
GECCO
Evostar
GECCO
CIG
Evostar
GECCO
CIG
Evostar
GECCO
WCCI
CIG
GECCO
2009
2008
2007
Tournament Type
Car Setup Optimization
Car Setup Optimization
CEC
CIG
WCCI
CIG
CEC
CIG
Fuzz-IEEE
CIG
Ms Pac-Man vs Gh
osts League
WCCI
Ms Pac-Man vs Gh
osts League
2012
63
55
80
Winner
Mr. Racer
Mr. Racer
Mr. Racer
Mr. Racer
Mr. Racer
Mr. Racer
Mr. Racer
J. Munoz
AUTOPIA
J. Munoz
AUTOPIA
E. Onieva et. al
P. Versari et al.
COBOSTAR
COBOSTAR
M. Simmerson
L. Cardamone
H. D. Thang et al.
P. Burrow
H. D. Thang et al.
Maastricht
Eiisolver
Eiisolver
Eiisolver
Eiisolver
Eiisolver
Nozomu
Screen Capture
5
CIG
CIG
CEC
Pac-Man
Ms Pac-Man vs Gh
osts League
Ghosts
Pac-Man
Ms Pac-Man vs Gh
osts League
Ghosts
Screen Capture
Screen Capture
Screen Capture
15
18
11
8
6
4
5
2008
WCCI
Screen Capture
11
2007
CEC
Screen Capture
4
CIG
Ms. PacMan
Pacman
Ghosts
Combined
Pacman
Ghosts
Combined
# of Entries
8
1
3
4
7
6
7
5
6
7
7
11
3
6
13
5
5
10
6
4
2011
CEC
2010
2009
Spooks
XSL11
James
nqkien
Pac-mAnt
H. Matsumoto et al.
H. Matsumoto et al.
A. Fitzgerald et al
.
Unclaimed
Affiliation
TU Dortmund
TU Dortmund
TU Dortmund
TU Dortmund
TU Dortmund
TU Dortmund
TU Dortmund
Carols III University
CSIC, Madrid
Carols III University
CSIC, Madrid
CSIC, Madrid
Universita Degli Studi Di Parma
University of Wuzburg
University of Wuzburg
Politecnico di Milano
University of Nottingham
University of Essex
University of Nottingham
Maastricth University, Netherlands
Sweden
Sweden
Sweden
Sweden
Sweden
University of Electro -Communications, Tok
yo
Private, United Kingdom
Private, Canada
University of Nottingham, Malaysia
Ritsumeiken University, Japan
Carolos III University
Ritsumeikan University
Ritsumeikan University
University of Southern Maine
Game AI Competitions, Organizers,
Participants and Winners (affiliations)
Game
Year
2012
Conference
Winner
Affiliation
Ya-Hung Chen et al.
Vinay Sudha Ethiraj
B. Weber
S. Bojarski et al.
S. Bojarski et al.
Robin Baumgarten
S.Polikarpov
S. Bojarski et al.
S. Bojarski et al.
B. Weber
R. Baumgarten
S. Polikarpov
R. Baumgarten
R. Baumgarten
Conscious-Robots
SQLiteBot
AMIS
UAlbertaBot
Skynet
Tomas Vajda
Skynet
Skynet
Matej Istenik
Skynet
Skynet
Roman Danielis
FreSCBot
FreSCBot
MimicBot
Overmind
Beau-Rivage
Naïve Agent
BLISS
PurofMovio
Slash
Purofvio
T
National Taiwan Norman University
Blekinge institute of technology
UC Santa Cruz
University Southern Maine
University Southern Maine
University of York, United Kingdom
Australia
University of York, United Kingdom
TAIST-Tokyo Tech, Thailand
CIG
CIBot
Sejong University, South Korea
CIG
Rinat Sadykov and Dexter Ju
ng
CIG
CIG
2011
EvoStar
WCCI
Super Mario
CIG
2010
2009
Unreal Tournament
2010
2009
2008
2013
2012
StarCraft
2011
Angry Bird
Ghost AI
Physical TSP
2010
AIIDE
2013
2012
2013
2013
IJCAI
AJCAI
GEISTER
CIG
WCCI
CIG
2012
Fighting Game AI
2013
Geometry Friends
2013
Plaftormer AI
Evostar
WCCI
ICE-CIG
CIG
CIG
CIG
CIG
AIIDE
CIG
SSCAI
AIIDE
CIG
SSCAI
AIIDE
CIG
SSCAI
2013
Tournament Type
Level Generation
Turing test
Level Generation
Learning
Gameplay
Gameplay
Gameplay
Gamplay
Learning
Level Generation
Gameplay
Gameplay
One VS One
One VS One
One VS One
Micro Management
Small-Scale Combat
Tech-limited Game
Complete Game
Classic Track
Ghost Challenge
MO-PTSP
Human Vs. Bot
Turing Test
# of Entries
5
6
4
7
7
4
6
6
6
9
15
11
10
6
8
8
38
10
10
52
30
10
50
7
2
8
17
19
8
8
8
23
31
10
211
IDSIA
University Southern Maine
University Southern Maine
UC Santa Cruz
Imperial College, London
IDSIA
Imperial College, London
Imperial College, London
Carlos III University
Charles University in Prague
University of Alberta, Canada
freelancer
Comenius University (Slovakia)
freelancer
freelancer
University od Zilina (Slovakia)
freelancer
freelancer
Comenius University, Bratislava
RPI
UC Berkeley
EPFL (Switzerland)
Competitions
20
18
16
# of Competitions
14
12
10
8
6
4
2
0
2003
2004
2005
2006
2007
2008
Year
2009
2010
2011
2012
2013
Paper Publications related to Games
and Competitions
30
Platformer AI
25
Geometry Friends
Fighting Game AI
# of Papers
20
Ghost AI
Angry Bird
15
Physical TSP
StarCraft
10
Unreal Tournament
Super Mario
5
Ms. Pac-Man
Car Racing
0
2005
2006
2007
2008
2009
Year
2010
2011
2012
2013
Phylogeny of Car Racing
• Simulated Car Racing (2007)
– Julian Togelius and Simon Lucas
– Java
– 2D Racing…. Simple Physics
– TORCS [3D Open Source] (2008)
– Java, C++, Python Clients
– Still ongoing
– Warm-up, Qualifying and final stages
Phylogeny of Ms. Pac-Man
• Screen Capture (2007)
–
–
–
–
–
–
–
–
Simon Lucas
Java interface
Goal: High Score
Ms. Pac-Man vs. Ghosts (2010~2012)
Philipp Rohlfshangen, David Robles and Simon Lucas
Java
Goal: High Schore (Pac-Man), Low Score (Ghosts)
Online evaluation
Phylogeny of Super Mario
• Mario AI Championship (2010)
– Game Play(Win level)
– Learning(Evaluation)
• Sergey Karakovskiy, Julian Togelius
– Level Generation(Infinite Mario Bros)
• Noor Shaker, Julian Togelius and Georgios Yannakakis
– Turing Test (2012)
• Noor Shaker, Julian Togelius and Georgios Yannakakis
– Java
– Platformer AI Competition (2013)
• Level Generation and Turing Test Tracks
– Goal: Play in a believable manner
Eco System
Centralized Management
Platform (Suggestion)
Game Ai Competition
Portals
Advertisement
Organizers
Game Ai
Competitions
Participants/
Affiliations
Game AI
Related
Conferences
Attraction/
Rewards
Publications
Education
Free Video
Games
(Benchmark)
Sharing
Winners’
Source Code
Special Publication Journal
(Suggestion)
Game AI Competition Survey
https://docs.google.com/forms/d/1PCYJ5Zs1Lsb
f7ttI7DoL5kpCL2wepF0k7zh09hakSvg/viewform