Download Mateas Statement September 2007 - University of California, Santa

PERSONAL STATEMENT
UNIVERSITY OF CALIFORNIA ACADEMIC PERSONNEL REVIEW
September 14, 2007
Michael Joseph Mateas
Assistant Professor
Department of Computer Science
University of California, Santa Cruz
Expressive AI: AI-based Art and Entertainment
My work focuses on Artificial Intelligence (AI)-based art and entertainment. As computer games are the
predominant AI-intensive art form, much of my research involves pushing the boundaries of game AI, creating new AI architectures and approaches that open up whole new genres and styles of games. To do this it
is not enough to apply existing AI approaches to games. Rather, taking the design of AI-intensive interactive experiences as a first-class agenda opens up new AI research questions; this simultaneous research
agenda and design practice I call Expressive AI. In Expressive AI, science, engineering and design are integrated into research that explores the expressive possibilities of AI architectures while simultaneously
pushing the boundaries of the conceivable and possible in games and other interactive art and entertainment
forms. This inherently interdisciplinary work involves building intelligences that robustly function outside
of the lab to engage human participants in intellectually and aesthetically satisfying interactions, and results
in publications in a variety of fields including AI, game design, and media theory.
The central research question underlying all my work is the problem of computational expression: what
architectures, representational techniques and design methods enable the creation of richly interactive, generative experiences. The problem of computational content is intimately related to research in AI; procedural content requires expressing, in machine manipulable form, knowledge, structures and processes describing cultural artifacts such as characters, stories, rhetoric, visual aesthetics, etc.
Interactive Drama
Interactive Drama (ID), in which a player can interact with rich autonomous characters and experience a
dynamically constructed story that depends on her actions, is one of the holy grails of game design. Building an ID requires solving a number of challenging research questions in areas including autonomous characters, story management and generation, natural language processing in the context of dramatic worlds,
and the design question of how particular authored experiences can be deconstructed and re-expressed
within the multiple AI systems for character, story and language. While there has certainly been prior work
within the individual areas of character, story and language, Façade, released in July 2005, is the first fullyproduced ID to integrate all these capabilities into a downloadable, playable experience [10,11].
By being the first interactive drama to move beyond technology demo and thought experiment, and to thus
be the first fully-formed instance of this new genre, Façade has had significant impact. Since July 2005 it
has been downloaded several hundred thousand times, though the actual number of individual players is far
higher, since Façade has also been included as a free DVD with several game magazines, and has been
made available on the internal servers of game companies such as Electronic Arts. Numerous publications
have written about Façade, including The New York Times (who called it “the future of videogames”),
noted game pundit Ernest Adams (who described it as “the most important game of the last 10 years”), and
Newsweek, who described Façade as a game that has appeal for both female and male players. Additionally, Façade has been discussed and reviewed on hundreds of web sites and blogs, is now included as an
example in a number of classes on games and interactive storytelling, and has been featured in several juried art venues, including ISEA 2004 (the International Symposium of Electronic Arts), Slamdance, a game
competition held concurrently with the Sundance film festival (January 2006), and the Independent Game
Festival held at the Game Developers Conference (GDC 2004).
A research challenge for interactive drama, and for Expressive AI in general, is developing methodologies
for performing useful audience (player) evaluations. The traditional focus in human-computer interaction
on error rates, time on task, and user satisfaction are inappropriate for assessing these systems, especially
when the goal of evaluation is not to simply provide a binary “succeeds/fails” assessment, but rather to assess the relationship between the player’s experience and the underlying technologies. We have been developing qualitative evaluation methodologies for performing these detailed assessments [2,3,12].
Autonomous Characters
The autonomous characters in Façade are implemented in ABL, a reactive planning architecture and language for autonomous characters, which organizes characters as collections of reactive behaviors [9]. These
behaviors dynamically mix over time as a function of
the character’s environment and interactions. Currently, individual behaviors do not adapt and change
over time, requiring the character author to explicitly
specify the complete details for all possible character
behaviors. We seek to relieve some of this authorial
burden, while still supporting authorial control, by extending ABL so as to support dynamic generation and
adaptation of behaviors. Behavior generation uses
transformational planning and execution monitoring
methods to generate new behaviors from existing character behaviors in response to detecting violations of
personality invariants [26]. The goal is to leverage deScreen capture from Façade
clarative knowledge about the communicative style and
Façade (collaboration with Andrew Stern) is an in- intent of behaviors to generate new behaviors that preteractive drama combining autonomous, believable serve the author’s style. In work supported by Sandia,
characters with dynamic plot construction. The player
we are developing a general semantic markup language
has been invited over to a couple’s apartment for
drinks, where she ends up participating in their mari- for2 expressing personality invariants. Adaptive ABL
tal troubles. Façade is available for download at (A BL) aims to integrate statistical machine learning
techniques into ABL as a first class language construct
www.interactivestory.net
[1]. The ultimate goal is to allow everyday programmers, who are not machine learning experts, to easily incorporate adaptation into ABL programs. This
work has been supported by grants from the Intel Corporation, as well as a DARPA contract under the Integrated Learning program.
To support the autonomous character research, we have integrated ABL with the Unreal Tournament game
engine. We are currently integrating ABL with the Torque game engine. Several other research groups are
currently using the ABL/UT infrastructure for their own research, including a group at the Institute for
Creativity Technology at USC [17,18] and a group at Georgia Tech [22].
Drama Management
The most common approach for authoring interactive stories, and still the state of the art in the commercial
game industry, involves specifying story graphs that explicitly represent possible story branches. The combinatorics of manually unwinding all possible branches limits the complexity of interactive stories that can
be authored. The general research problem of drama management (DM) is to replace story graphs with a
story policy – consisting of a library of story “moves”, a model of the desired story structure, and a story
move selection policy – which, given the history of the player’s interaction so far, plus the story model,
selects which story move should happen next in order to maximize future expected story quality. Our current DM research frames this as an optimization problem [4]. Given a player’s current history of story
events, project all possible future histories, executing the story move (world manipulation) that maximizes
the future expected value (story goodness). We’ve experimented with uncached game-tree search [14], reinforcement learning [16], and cached trajectory sampling [19], showing that, with simulated players, DM
does improve the experience quality over the unmanaged case. We’re currently designing a study to validate this effect with real players.
Game Generation
Research in game generation seeks to build AI systems that make design decisions with respect to the rules,
physical layout, and visual representation of a game. The small amount of prior work in this area tends to
be limited to chess-like and tile-based games on the academic side, and relatively shallow level generators
that randomly combine large-grain-scale human-authored pieces on the commercial side. The goal of my
game generation research is not to replace human designers, but rather:
• to move human design up the abstraction hierarchy, and thus get a content multiplier by enabling
authors to specify processes that generate concrete content,
• to facilitate formal game analysis by computationally operationalizing game rules, mechanics, and
representations,
• to enable new game mechanics and game genres where the game dynamically morphs and changes
as a function of player interaction.
Thematic Reasoning
One of the research questions in automated game design is the problem of thematic mapping; how, given
desired game content (e.g. “I want a game with fish” or “I want a game with ducks and shooting”), this
content interacts with and constrains the game mechanics (the possible game states, the goal state, and the
state transitions that can be caused by player and game entity actions). One line of our work is focusing on
the thematic mapping problem for micro-games of the sort found in the commercial Gameboy game Wario
Ware. These micro-games, by focusing on just one or two game verbs and lasting just a few seconds, provide a tractable (but commercially interesting) micro-domain within which to explore the thematic mapping
problem. Our current system performs inferences over the ConceptNet common sense ontology (using
Wordnet to provide generalizations over ConceptNet nodes) to attempt to satisfy role fillers in various abstractly represented game mechanics, given initial human-provided themes. The system generate a sequence of playable micro-games (generating J2ME Java code, runable on cell phones) which are all variations on the provided theme [15]. We are currently modifying the generator with a mixed-initiative authoring interface that allows a human designer to collaborate with the generator, incrementally adding and removing constraints as human and system jointly explore the design space.
Level Generation
In another line of the game generation work we are exploring dynamic level generation for training games.
Our training domain is collapsed structure training, training fire-fighters on the strategic and tactical skills
involved in looking for live victims in collapsed buildings. The current state of the art in level generation is
to essentially randomly place and connect pre-designed rooms. Rather than random generation, our goal is
to develop a generator that accepts specific training goals and strategically generates a scenario (level) designed to exercise those training goals. Our current approach makes use of hierarchically organized “deformation” operators (e.g. bomb blast, structural support failure) applied to a simple qualitative model of a
building; the system searches for a combination of operator applications that results in a scenario with the
desired training properties. This work is supported by a NASA ARP grant.
Formal Game Analysis
The system-building work in game generation is supported by an ontological analysis of games, the Game
Ontology Project (GOP) [23,24,25]. The GOP, consisting of a hierarchy of game-design concepts abstracted from analyses of many different games, is a framework for describing, analyzing and studying
games. The elements are often derived from common game terminology (e.g. level and boss) and then refined by abstracting more general concepts and identifying more precise or specific concepts. GOP borrows
concepts and methods from prototype theory as well as grounded theory to achieve a framework that is
always growing and changing as new games are analyzed or particular research questions are explored. The
top-level concepts of the GOP are available on a public wiki (www.gameontology.org); we have begun
collaborating with multiple scholars in refining and adding concepts, as well as using the GOP in assignments in game design classes, asking students to provide examples of and/or refine existing concepts based
on specific games they analyze in the class. While the GOP is not fully formalized, it provides a semiformal knowledge structure to guide full formalization for game generation research projects.
Digital Art and New Media
An important component of my research practice is participating in the new media theory and art communities. Just as combining AI research and game design opens up new possibilities in games, combining AI
research with digital media art can open up new possibilities in other interactive art forms. This work takes
the form of research in generative ambient displays, new media theory, and participation in art shows.
Generative Ambient Displays
The field of ambient intelligence is concerned with building AI systems that engage in long-term monitoring of environments and dynamically and proactively respond to the monitored activity, often with the goal
of supporting specific tasks. In my group we’re exploring an alternative to this task support model of intelligent ubiquitous computing, instead building ambient computational devices that
open unusual viewpoints onto everyday
human activity, create pleasure, and provide opportunities for contemplation and
wonder. We call this design approach of
combining AI-based interpretation of human activity with expressive, ambient, displays alien presence.
An alien presence provides a dynamic and
autonomous interpretation of a human social context, not with the goal of interpreting the context the way a human would, but
rather to provide an alien interpretation of
the context that encourages participants to
Tableau Machine vision data
reflect on their own activity. Unlike inforThis image depicts social energy, as derived from adjacent frame
mation visualization, which aims to condifference data, across multiple semantically significant image
struct decodeable displays, an alien presregions during a social evening in a home. Visualizations such as
this one are used by designers to tune the Tableau Machine inter- ence creates a sense of a living being with
its own perceptions, desires and underpretation and generation system.
standings. The user and the system together
create a site of cointerpretation: the meaning of the ambient display is produced both by the autonomous
interpretations of the system and by the projections of the user. The system, through its alien interpretation,
makes strange the unreflective activity it observes, providing participants an opportunity to reflect and project new meanings onto their own activity. This work builds on the design insights and strategies developed
for an earlier project, Office Plant #1, desktop robot that responds to the social and emotional meaning of
email received by the user.
Tableau Machine is an ambient display that dynamically generates abstract art (inspired by the style of
Russian Consructivism) in response to the sensed “mood” of a physical place [20,21]. The camera based
sensing system computes three values from simple and robust adjacent-frame difference measurements:
energy (E, the amount of activity in the space), density (D, how clumped together activity is in the space)
and flow (F, how much the energy distribution pattern is spatially shifting). Global plus several region-level
EDF measurements form a multi-dimensional EDF cube; online clustering within this cube discovers activity “modes” in a space. Unique modes are mapped to unique combinations of values for composition attributes (foreground/background ratio, horizontal and vertical balance, delicacy, etc.), which are passed to the
generator. The art generator makes use of design grammars, plus global image-based feature detectors that
capture global composition constraints. By generating unique ambient images correlated with unique
modes of activity detected in a physical space, Tableau Machine helps occupants to notice and reflect on
the different types of activities that take place in their space. In addition to gallery-style showings, Tableau
Machine is currently installed in several homes for long-term longitudinal evaluation. This work is funded
by an NSF grant, Closing the Affective Gap
New Media Theory
In new media theory, I am most involved in software studies and work on AI and culture. Software Studies
is concerned with understanding the culture dimensions of coding practice, how it is that software conveys
meaning, and unpacking the rhetorics and ideology around code. I’m specifically interested in software
studies work that moves beyond treating code metaphorically (ie. using the language of code to engage
traditional cultural studies topics), that actually engages in a computational understanding of media artifacts
and a deep reading of code. Examples of such work include:
• Code semiotics, in which I explore the relationship between code signs and the signs experienced
by an audience [6,13].
• Obfuscated programming and esoteric languages, in which I explore how the notion of code aesthetics plays out in these non-traditional code practices [8].
• Expressive AI, in which I explore how both traditional AI research and traditional art practice are
changed when AI and art become simultaneous, first-class goals [5,7].
Art Shows
The research questions driving my work arise from building real, publicly deployed, interactive experiences
that push the boundaries of the possible in both AI and interactive art. In order to keep my research
grounded by the needs of real, expressive systems, I display my research systems in art settings. Only by
building showable systems can I fully understand and grapple with the real research questions of Expressive AI. Like conference or journal publications, display in these settings is selective; a review committee
chooses which pieces will actually be shown from all of the art submitted to the show. Art shows can provide an opportunity for audience evaluations of the systems. I am beginning to experiment with acquiring
Institutional Review Board clearance for human subject protocols that allow us to observe and interview
gallery and museum patrons as they interact with the system.
Teaching
As I work in an emerging, interdisciplinary area, I have been involved in significant curriculum and course
development. At the Georgia Institute of Technology, I served on the committee that developed the Computational Media (CM) degree, an interdisciplinary undergraduate degree that prepares students for careers
in interactive art and entertainment (the majority of the students in the degree program are interested in
games). CM is jointly administered between computing and the digital media department. At UC Santa
Cruz, I am involved in the continuing development of the Computer Game Design (CGD) degree, an undergraduate computer science-focused degree program that provides students with the technical, scientific
and interdisciplinary design skills to perform hard-core game engineering, while understanding the complex interplay between game engine design and game design. Students graduating from this curriculum are
fully competent computer scientists, while additionally possessing the interdisciplinary training to put their
CS skills to work in the production of novel player experiences.
While at Georgia Tech I developed and taught three new courses: LCC6310: Computation as an Expressive
Medium, LCC6317: Interactive Narrative and CS 2260: Media Device Architectures. LCC 6310 is a graduate-level introduction to computer science and programming for students with art and humanities backgrounds. It introduces computational skills in the context of making interactive art, using the Processing
programming environment. It is now part of the core curriculum of Tech’s digital media degree. LCC 6317
is an introduction to interactive narrative for art and humanities students, focusing on the theory behind
interactive storytelling. CS 2260, a new undergraduate course in the Computational Media degree, is an
introduction to hardware-level systems programming, using the Gameboy Advance game console as a reference architecture.
In my first year at UC Santa Cruz I have developed and taught three new courses: CS 244: Game AI, CS
148/248: Interactive Narrative, and CS 170: Game Design Studio 1 (this last course being taught Fall
2007). CS 244 is a graduate level introduction to game AI. CS 148/248 is an interactive storytelling class,
loosely based on the one I developed at Georgia Tech, but significantly revised for CS and Computer Game
Design students. This class focuses on technologies for interactive storytelling, particularly AI techniques,
and introduces the theoretical background behind the game design debates around storytelling in games. CS
170 is the first course in the year-long capstone studio sequence of the new Computer Game Design degree.
In 170, students are introduced to concept design and rapid prototyping methods for game design, as well
as introduced to a number of innovative game genres to provide inspiration for their own designs.
Bibliography
[1] Bhat, S., Isbell, C., & Mateas, M. (2006). On the Difficulty of Modular Reinforcement Learning for
Real-World Partial Programming. Proceedings of the Twenty-First National Conference on Artificial Intelligence (AAAI), Boston, MA.
[2] Dow, S., Mehta, M., Harmon, E., MacIntyre, B., & Mateas, M. (2007). Presence and engagement in an
interactive drama. Proceedings of the SIGCHI Conference on Human factors in Computing Systems (CHI),
San Jose, California, USA.
[3] Knickmeyer, R., & Mateas, M. (2005). Preliminary Evaluation of the Interactive Drama Façade. Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR.
[4] Lamstein, A., & Mateas, M. (2004). A Search-based Drama Manager. AAAI 2004 Workshop on Challenges in Game AI.
[5] Mateas, M. (2001). Expressive AI: A hybrid art and science practice. Leonardo: Journal of the International Society for Arts, Sciences and Technology, 34(2), 147-153.
[6] Mateas, M. (2003). Expressive AI: A semiotic Analysis of Machinic Affordances. 3rd Conference on
Computational Semiotics and New Media, University of Teeside, UK.
[7] Mateas, M. (2003). Expressive AI: Games and Artificial Intelligence. Level Up: Digital Games Research Conference, Utrecht, Netherlands.
[8] Mateas, M., & Montfort, N. (2005). A Box, Darkly: Obfuscation, Weird Languages, and Code Aesthetics. Digital Arts and Culture: Digital Experience: Design, Aesthetics, Practice (DAC 2005), Copenhagen,
Denmark.
[9] Mateas, M., & Stern, A. (2002). A Behavior Language for Story-based Believable Agents. IEEE Intelligent Systems, 17(4), 39-47.
[10] Mateas, M., & Stern, A. (2003). Integrating plot, character and natural language processing in the
interactive drama Façade. Technologies for Interactive Digital Storytelling and Entertainment (TIDSE),
Darmstadt, Germany.
[11] Mateas, M., & Stern, A. (2005). Structuring Content in the Façade Interactive Drama Architecture.
Artificial Intelligence and Interactive Digital Entertainment (AIIDE 2005), Marina del Rey, CA.
[12] Mehta, M., Dow, S., Mateas, M., & MacIntyre, B. (2007). Evaluating a Conversation-Centered Interactive Drama. Proceedings of the Sixth International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), Honolulu, Hawaii, USA.
[13] Montfort, N., & Mateas, M. (2007 (forthcoming)). Hammurabi's Code, Twenty-First Annual Conference of the Society for Literature, Science and the Arts: Code Portland Maine, USA.
[14] Nelson, M., & Mateas, M. (2005). Search-based Drama Management in the Interactive Fiction Anchorhead. Artificial Intelligence and Interactive Digital Entertainment (AIIDE 2005), Marina del Rey, CA.
[15] Nelson, M., & Mateas, M. (2007). Towards Automated Game Design. Procedings of the 10th Congress
of the Italian Association for Artificial Intelligence (AIIA 2007), Rome, Italy.
[16] Nelson, M., Mateas, M., Roberts, D., & Isbell, C. (2006). Declarative Optimization-Based Drama
Management in the Interactive Fiction Anchorhead. IEEE Computer Graphics and Applications, 26(3), 3241.
[17] Riedl, M., & Stern, A. (2006). Believable Agents and Intelligent Scenario Direction for Social and
Cultural Leadership Training. Proceedings of the 15th Conference on Behavior Representation in Modeling and Simulation, Baltimore, MD, USA.
[18] Riedl, M., & Stern, A. (2006). Believable Agents and Intelligent Story Adaptation for Interactive Storytelling. Proceedings of the 3rd International Conference on Technologies for Interactive Digital Storytelling and Entertainment (TIDSE 2006), Darmstadt, Germany.
[19] Roberts, D., Nelson, M., Isbell, C., Mateas, M., & Littman, M. (2006). Targeting Specific Distributions of Trajectories in MDPs. Proceedings of the Twenty-First National Conference on Artificial Intelligence (AAAI), Boston, MA.
[20] Romero, M., & Mateas, M. (2005). A Preliminary Investigation of Alien Presence. Eleventh International Conference on Human-Computer Interaction (HCII 2005), Las Vegas, NV.
[21] Romero, M., Pousman, Z., & Mateas, M. (in press). Alien Presence in the Home: The Design of Tableau Machine. Personal and Ubiquitous Computing.
[22] Strong, C., Mehta, M., Mishra, K., Jones, A., & Ram, A. (2007). Emotionally Driven Natural Language Generation for Personality Rich Characters in Interactive Games. Third Conference on Artificial
Intelligence for Interactive Digital Entertainment (AIIDE-07), Stanford, CA, USA.
[23] Zagal, J., Fernández-Vara, C., & Mateas, M. (in press). Rounds, Levels, and Waves: The Early Evolution of Gameplay Segmentation. Games and Culture.
[24] Zagal, J., & Mateas, M. (2007 (forthcoming)). Temporal Frames: A Unifying Framework for the
Analysis of Game Temporality. Situated Play (DIGRA 2007), Tokyo, Japan.
[25] Zagal, J., Mateas, M., Fernandez-Vara, C., Hochhalter, B., & Lichti, N. (2005). Towards an Ontological Analysis of Games. Changing Views: Worlds In Play (DIGRA 2005), Vancouver, BC.
[26] Zang, P., Mehta, M., Mateas, M., & Ram, A. (2007). Towards Runtime Behavior Adaptation for Embodied Characters. Proceedings of The International Joint Conference on Artificial Intelligence (IJCAI07), Hyderabad, India.