Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
PERSONAL STATEMENT UNIVERSITY OF CALIFORNIA ACADEMIC PERSONNEL REVIEW September 14, 2007 Michael Joseph Mateas Assistant Professor Department of Computer Science University of California, Santa Cruz Expressive AI: AI-based Art and Entertainment My work focuses on Artificial Intelligence (AI)-based art and entertainment. As computer games are the predominant AI-intensive art form, much of my research involves pushing the boundaries of game AI, creating new AI architectures and approaches that open up whole new genres and styles of games. To do this it is not enough to apply existing AI approaches to games. Rather, taking the design of AI-intensive interactive experiences as a first-class agenda opens up new AI research questions; this simultaneous research agenda and design practice I call Expressive AI. In Expressive AI, science, engineering and design are integrated into research that explores the expressive possibilities of AI architectures while simultaneously pushing the boundaries of the conceivable and possible in games and other interactive art and entertainment forms. This inherently interdisciplinary work involves building intelligences that robustly function outside of the lab to engage human participants in intellectually and aesthetically satisfying interactions, and results in publications in a variety of fields including AI, game design, and media theory. The central research question underlying all my work is the problem of computational expression: what architectures, representational techniques and design methods enable the creation of richly interactive, generative experiences. The problem of computational content is intimately related to research in AI; procedural content requires expressing, in machine manipulable form, knowledge, structures and processes describing cultural artifacts such as characters, stories, rhetoric, visual aesthetics, etc. Interactive Drama Interactive Drama (ID), in which a player can interact with rich autonomous characters and experience a dynamically constructed story that depends on her actions, is one of the holy grails of game design. Building an ID requires solving a number of challenging research questions in areas including autonomous characters, story management and generation, natural language processing in the context of dramatic worlds, and the design question of how particular authored experiences can be deconstructed and re-expressed within the multiple AI systems for character, story and language. While there has certainly been prior work within the individual areas of character, story and language, Façade, released in July 2005, is the first fullyproduced ID to integrate all these capabilities into a downloadable, playable experience [10,11]. By being the first interactive drama to move beyond technology demo and thought experiment, and to thus be the first fully-formed instance of this new genre, Façade has had significant impact. Since July 2005 it has been downloaded several hundred thousand times, though the actual number of individual players is far higher, since Façade has also been included as a free DVD with several game magazines, and has been made available on the internal servers of game companies such as Electronic Arts. Numerous publications have written about Façade, including The New York Times (who called it “the future of videogames”), noted game pundit Ernest Adams (who described it as “the most important game of the last 10 years”), and Newsweek, who described Façade as a game that has appeal for both female and male players. Additionally, Façade has been discussed and reviewed on hundreds of web sites and blogs, is now included as an example in a number of classes on games and interactive storytelling, and has been featured in several juried art venues, including ISEA 2004 (the International Symposium of Electronic Arts), Slamdance, a game competition held concurrently with the Sundance film festival (January 2006), and the Independent Game Festival held at the Game Developers Conference (GDC 2004). A research challenge for interactive drama, and for Expressive AI in general, is developing methodologies for performing useful audience (player) evaluations. The traditional focus in human-computer interaction on error rates, time on task, and user satisfaction are inappropriate for assessing these systems, especially when the goal of evaluation is not to simply provide a binary “succeeds/fails” assessment, but rather to assess the relationship between the player’s experience and the underlying technologies. We have been developing qualitative evaluation methodologies for performing these detailed assessments [2,3,12]. Autonomous Characters The autonomous characters in Façade are implemented in ABL, a reactive planning architecture and language for autonomous characters, which organizes characters as collections of reactive behaviors [9]. These behaviors dynamically mix over time as a function of the character’s environment and interactions. Currently, individual behaviors do not adapt and change over time, requiring the character author to explicitly specify the complete details for all possible character behaviors. We seek to relieve some of this authorial burden, while still supporting authorial control, by extending ABL so as to support dynamic generation and adaptation of behaviors. Behavior generation uses transformational planning and execution monitoring methods to generate new behaviors from existing character behaviors in response to detecting violations of personality invariants [26]. The goal is to leverage deScreen capture from Façade clarative knowledge about the communicative style and Façade (collaboration with Andrew Stern) is an in- intent of behaviors to generate new behaviors that preteractive drama combining autonomous, believable serve the author’s style. In work supported by Sandia, characters with dynamic plot construction. The player we are developing a general semantic markup language has been invited over to a couple’s apartment for drinks, where she ends up participating in their mari- for2 expressing personality invariants. Adaptive ABL tal troubles. Façade is available for download at (A BL) aims to integrate statistical machine learning techniques into ABL as a first class language construct www.interactivestory.net [1]. The ultimate goal is to allow everyday programmers, who are not machine learning experts, to easily incorporate adaptation into ABL programs. This work has been supported by grants from the Intel Corporation, as well as a DARPA contract under the Integrated Learning program. To support the autonomous character research, we have integrated ABL with the Unreal Tournament game engine. We are currently integrating ABL with the Torque game engine. Several other research groups are currently using the ABL/UT infrastructure for their own research, including a group at the Institute for Creativity Technology at USC [17,18] and a group at Georgia Tech [22]. Drama Management The most common approach for authoring interactive stories, and still the state of the art in the commercial game industry, involves specifying story graphs that explicitly represent possible story branches. The combinatorics of manually unwinding all possible branches limits the complexity of interactive stories that can be authored. The general research problem of drama management (DM) is to replace story graphs with a story policy – consisting of a library of story “moves”, a model of the desired story structure, and a story move selection policy – which, given the history of the player’s interaction so far, plus the story model, selects which story move should happen next in order to maximize future expected story quality. Our current DM research frames this as an optimization problem [4]. Given a player’s current history of story events, project all possible future histories, executing the story move (world manipulation) that maximizes the future expected value (story goodness). We’ve experimented with uncached game-tree search [14], reinforcement learning [16], and cached trajectory sampling [19], showing that, with simulated players, DM does improve the experience quality over the unmanaged case. We’re currently designing a study to validate this effect with real players. Game Generation Research in game generation seeks to build AI systems that make design decisions with respect to the rules, physical layout, and visual representation of a game. The small amount of prior work in this area tends to be limited to chess-like and tile-based games on the academic side, and relatively shallow level generators that randomly combine large-grain-scale human-authored pieces on the commercial side. The goal of my game generation research is not to replace human designers, but rather: • to move human design up the abstraction hierarchy, and thus get a content multiplier by enabling authors to specify processes that generate concrete content, • to facilitate formal game analysis by computationally operationalizing game rules, mechanics, and representations, • to enable new game mechanics and game genres where the game dynamically morphs and changes as a function of player interaction. Thematic Reasoning One of the research questions in automated game design is the problem of thematic mapping; how, given desired game content (e.g. “I want a game with fish” or “I want a game with ducks and shooting”), this content interacts with and constrains the game mechanics (the possible game states, the goal state, and the state transitions that can be caused by player and game entity actions). One line of our work is focusing on the thematic mapping problem for micro-games of the sort found in the commercial Gameboy game Wario Ware. These micro-games, by focusing on just one or two game verbs and lasting just a few seconds, provide a tractable (but commercially interesting) micro-domain within which to explore the thematic mapping problem. Our current system performs inferences over the ConceptNet common sense ontology (using Wordnet to provide generalizations over ConceptNet nodes) to attempt to satisfy role fillers in various abstractly represented game mechanics, given initial human-provided themes. The system generate a sequence of playable micro-games (generating J2ME Java code, runable on cell phones) which are all variations on the provided theme [15]. We are currently modifying the generator with a mixed-initiative authoring interface that allows a human designer to collaborate with the generator, incrementally adding and removing constraints as human and system jointly explore the design space. Level Generation In another line of the game generation work we are exploring dynamic level generation for training games. Our training domain is collapsed structure training, training fire-fighters on the strategic and tactical skills involved in looking for live victims in collapsed buildings. The current state of the art in level generation is to essentially randomly place and connect pre-designed rooms. Rather than random generation, our goal is to develop a generator that accepts specific training goals and strategically generates a scenario (level) designed to exercise those training goals. Our current approach makes use of hierarchically organized “deformation” operators (e.g. bomb blast, structural support failure) applied to a simple qualitative model of a building; the system searches for a combination of operator applications that results in a scenario with the desired training properties. This work is supported by a NASA ARP grant. Formal Game Analysis The system-building work in game generation is supported by an ontological analysis of games, the Game Ontology Project (GOP) [23,24,25]. The GOP, consisting of a hierarchy of game-design concepts abstracted from analyses of many different games, is a framework for describing, analyzing and studying games. The elements are often derived from common game terminology (e.g. level and boss) and then refined by abstracting more general concepts and identifying more precise or specific concepts. GOP borrows concepts and methods from prototype theory as well as grounded theory to achieve a framework that is always growing and changing as new games are analyzed or particular research questions are explored. The top-level concepts of the GOP are available on a public wiki (www.gameontology.org); we have begun collaborating with multiple scholars in refining and adding concepts, as well as using the GOP in assignments in game design classes, asking students to provide examples of and/or refine existing concepts based on specific games they analyze in the class. While the GOP is not fully formalized, it provides a semiformal knowledge structure to guide full formalization for game generation research projects. Digital Art and New Media An important component of my research practice is participating in the new media theory and art communities. Just as combining AI research and game design opens up new possibilities in games, combining AI research with digital media art can open up new possibilities in other interactive art forms. This work takes the form of research in generative ambient displays, new media theory, and participation in art shows. Generative Ambient Displays The field of ambient intelligence is concerned with building AI systems that engage in long-term monitoring of environments and dynamically and proactively respond to the monitored activity, often with the goal of supporting specific tasks. In my group we’re exploring an alternative to this task support model of intelligent ubiquitous computing, instead building ambient computational devices that open unusual viewpoints onto everyday human activity, create pleasure, and provide opportunities for contemplation and wonder. We call this design approach of combining AI-based interpretation of human activity with expressive, ambient, displays alien presence. An alien presence provides a dynamic and autonomous interpretation of a human social context, not with the goal of interpreting the context the way a human would, but rather to provide an alien interpretation of the context that encourages participants to Tableau Machine vision data reflect on their own activity. Unlike inforThis image depicts social energy, as derived from adjacent frame mation visualization, which aims to condifference data, across multiple semantically significant image struct decodeable displays, an alien presregions during a social evening in a home. Visualizations such as this one are used by designers to tune the Tableau Machine inter- ence creates a sense of a living being with its own perceptions, desires and underpretation and generation system. standings. The user and the system together create a site of cointerpretation: the meaning of the ambient display is produced both by the autonomous interpretations of the system and by the projections of the user. The system, through its alien interpretation, makes strange the unreflective activity it observes, providing participants an opportunity to reflect and project new meanings onto their own activity. This work builds on the design insights and strategies developed for an earlier project, Office Plant #1, desktop robot that responds to the social and emotional meaning of email received by the user. Tableau Machine is an ambient display that dynamically generates abstract art (inspired by the style of Russian Consructivism) in response to the sensed “mood” of a physical place [20,21]. The camera based sensing system computes three values from simple and robust adjacent-frame difference measurements: energy (E, the amount of activity in the space), density (D, how clumped together activity is in the space) and flow (F, how much the energy distribution pattern is spatially shifting). Global plus several region-level EDF measurements form a multi-dimensional EDF cube; online clustering within this cube discovers activity “modes” in a space. Unique modes are mapped to unique combinations of values for composition attributes (foreground/background ratio, horizontal and vertical balance, delicacy, etc.), which are passed to the generator. The art generator makes use of design grammars, plus global image-based feature detectors that capture global composition constraints. By generating unique ambient images correlated with unique modes of activity detected in a physical space, Tableau Machine helps occupants to notice and reflect on the different types of activities that take place in their space. In addition to gallery-style showings, Tableau Machine is currently installed in several homes for long-term longitudinal evaluation. This work is funded by an NSF grant, Closing the Affective Gap New Media Theory In new media theory, I am most involved in software studies and work on AI and culture. Software Studies is concerned with understanding the culture dimensions of coding practice, how it is that software conveys meaning, and unpacking the rhetorics and ideology around code. I’m specifically interested in software studies work that moves beyond treating code metaphorically (ie. using the language of code to engage traditional cultural studies topics), that actually engages in a computational understanding of media artifacts and a deep reading of code. Examples of such work include: • Code semiotics, in which I explore the relationship between code signs and the signs experienced by an audience [6,13]. • Obfuscated programming and esoteric languages, in which I explore how the notion of code aesthetics plays out in these non-traditional code practices [8]. • Expressive AI, in which I explore how both traditional AI research and traditional art practice are changed when AI and art become simultaneous, first-class goals [5,7]. Art Shows The research questions driving my work arise from building real, publicly deployed, interactive experiences that push the boundaries of the possible in both AI and interactive art. In order to keep my research grounded by the needs of real, expressive systems, I display my research systems in art settings. Only by building showable systems can I fully understand and grapple with the real research questions of Expressive AI. Like conference or journal publications, display in these settings is selective; a review committee chooses which pieces will actually be shown from all of the art submitted to the show. Art shows can provide an opportunity for audience evaluations of the systems. I am beginning to experiment with acquiring Institutional Review Board clearance for human subject protocols that allow us to observe and interview gallery and museum patrons as they interact with the system. Teaching As I work in an emerging, interdisciplinary area, I have been involved in significant curriculum and course development. At the Georgia Institute of Technology, I served on the committee that developed the Computational Media (CM) degree, an interdisciplinary undergraduate degree that prepares students for careers in interactive art and entertainment (the majority of the students in the degree program are interested in games). CM is jointly administered between computing and the digital media department. At UC Santa Cruz, I am involved in the continuing development of the Computer Game Design (CGD) degree, an undergraduate computer science-focused degree program that provides students with the technical, scientific and interdisciplinary design skills to perform hard-core game engineering, while understanding the complex interplay between game engine design and game design. Students graduating from this curriculum are fully competent computer scientists, while additionally possessing the interdisciplinary training to put their CS skills to work in the production of novel player experiences. While at Georgia Tech I developed and taught three new courses: LCC6310: Computation as an Expressive Medium, LCC6317: Interactive Narrative and CS 2260: Media Device Architectures. LCC 6310 is a graduate-level introduction to computer science and programming for students with art and humanities backgrounds. It introduces computational skills in the context of making interactive art, using the Processing programming environment. It is now part of the core curriculum of Tech’s digital media degree. LCC 6317 is an introduction to interactive narrative for art and humanities students, focusing on the theory behind interactive storytelling. CS 2260, a new undergraduate course in the Computational Media degree, is an introduction to hardware-level systems programming, using the Gameboy Advance game console as a reference architecture. In my first year at UC Santa Cruz I have developed and taught three new courses: CS 244: Game AI, CS 148/248: Interactive Narrative, and CS 170: Game Design Studio 1 (this last course being taught Fall 2007). CS 244 is a graduate level introduction to game AI. CS 148/248 is an interactive storytelling class, loosely based on the one I developed at Georgia Tech, but significantly revised for CS and Computer Game Design students. This class focuses on technologies for interactive storytelling, particularly AI techniques, and introduces the theoretical background behind the game design debates around storytelling in games. CS 170 is the first course in the year-long capstone studio sequence of the new Computer Game Design degree. In 170, students are introduced to concept design and rapid prototyping methods for game design, as well as introduced to a number of innovative game genres to provide inspiration for their own designs. Bibliography [1] Bhat, S., Isbell, C., & Mateas, M. (2006). On the Difficulty of Modular Reinforcement Learning for Real-World Partial Programming. Proceedings of the Twenty-First National Conference on Artificial Intelligence (AAAI), Boston, MA. [2] Dow, S., Mehta, M., Harmon, E., MacIntyre, B., & Mateas, M. (2007). Presence and engagement in an interactive drama. Proceedings of the SIGCHI Conference on Human factors in Computing Systems (CHI), San Jose, California, USA. [3] Knickmeyer, R., & Mateas, M. (2005). Preliminary Evaluation of the Interactive Drama Façade. Conference on Human Factors in Computing Systems (CHI 2005), Portland, OR. [4] Lamstein, A., & Mateas, M. (2004). A Search-based Drama Manager. AAAI 2004 Workshop on Challenges in Game AI. [5] Mateas, M. (2001). Expressive AI: A hybrid art and science practice. Leonardo: Journal of the International Society for Arts, Sciences and Technology, 34(2), 147-153. [6] Mateas, M. (2003). Expressive AI: A semiotic Analysis of Machinic Affordances. 3rd Conference on Computational Semiotics and New Media, University of Teeside, UK. [7] Mateas, M. (2003). Expressive AI: Games and Artificial Intelligence. Level Up: Digital Games Research Conference, Utrecht, Netherlands. [8] Mateas, M., & Montfort, N. (2005). A Box, Darkly: Obfuscation, Weird Languages, and Code Aesthetics. Digital Arts and Culture: Digital Experience: Design, Aesthetics, Practice (DAC 2005), Copenhagen, Denmark. [9] Mateas, M., & Stern, A. (2002). A Behavior Language for Story-based Believable Agents. IEEE Intelligent Systems, 17(4), 39-47. [10] Mateas, M., & Stern, A. (2003). Integrating plot, character and natural language processing in the interactive drama Façade. Technologies for Interactive Digital Storytelling and Entertainment (TIDSE), Darmstadt, Germany. [11] Mateas, M., & Stern, A. (2005). Structuring Content in the Façade Interactive Drama Architecture. Artificial Intelligence and Interactive Digital Entertainment (AIIDE 2005), Marina del Rey, CA. [12] Mehta, M., Dow, S., Mateas, M., & MacIntyre, B. (2007). Evaluating a Conversation-Centered Interactive Drama. Proceedings of the Sixth International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), Honolulu, Hawaii, USA. [13] Montfort, N., & Mateas, M. (2007 (forthcoming)). Hammurabi's Code, Twenty-First Annual Conference of the Society for Literature, Science and the Arts: Code Portland Maine, USA. [14] Nelson, M., & Mateas, M. (2005). Search-based Drama Management in the Interactive Fiction Anchorhead. Artificial Intelligence and Interactive Digital Entertainment (AIIDE 2005), Marina del Rey, CA. [15] Nelson, M., & Mateas, M. (2007). Towards Automated Game Design. Procedings of the 10th Congress of the Italian Association for Artificial Intelligence (AIIA 2007), Rome, Italy. [16] Nelson, M., Mateas, M., Roberts, D., & Isbell, C. (2006). Declarative Optimization-Based Drama Management in the Interactive Fiction Anchorhead. IEEE Computer Graphics and Applications, 26(3), 3241. [17] Riedl, M., & Stern, A. (2006). Believable Agents and Intelligent Scenario Direction for Social and Cultural Leadership Training. Proceedings of the 15th Conference on Behavior Representation in Modeling and Simulation, Baltimore, MD, USA. [18] Riedl, M., & Stern, A. (2006). Believable Agents and Intelligent Story Adaptation for Interactive Storytelling. Proceedings of the 3rd International Conference on Technologies for Interactive Digital Storytelling and Entertainment (TIDSE 2006), Darmstadt, Germany. [19] Roberts, D., Nelson, M., Isbell, C., Mateas, M., & Littman, M. (2006). Targeting Specific Distributions of Trajectories in MDPs. Proceedings of the Twenty-First National Conference on Artificial Intelligence (AAAI), Boston, MA. [20] Romero, M., & Mateas, M. (2005). A Preliminary Investigation of Alien Presence. Eleventh International Conference on Human-Computer Interaction (HCII 2005), Las Vegas, NV. [21] Romero, M., Pousman, Z., & Mateas, M. (in press). Alien Presence in the Home: The Design of Tableau Machine. Personal and Ubiquitous Computing. [22] Strong, C., Mehta, M., Mishra, K., Jones, A., & Ram, A. (2007). Emotionally Driven Natural Language Generation for Personality Rich Characters in Interactive Games. Third Conference on Artificial Intelligence for Interactive Digital Entertainment (AIIDE-07), Stanford, CA, USA. [23] Zagal, J., Fernández-Vara, C., & Mateas, M. (in press). Rounds, Levels, and Waves: The Early Evolution of Gameplay Segmentation. Games and Culture. [24] Zagal, J., & Mateas, M. (2007 (forthcoming)). Temporal Frames: A Unifying Framework for the Analysis of Game Temporality. Situated Play (DIGRA 2007), Tokyo, Japan. [25] Zagal, J., Mateas, M., Fernandez-Vara, C., Hochhalter, B., & Lichti, N. (2005). Towards an Ontological Analysis of Games. Changing Views: Worlds In Play (DIGRA 2005), Vancouver, BC. [26] Zang, P., Mehta, M., Mateas, M., & Ram, A. (2007). Towards Runtime Behavior Adaptation for Embodied Characters. Proceedings of The International Joint Conference on Artificial Intelligence (IJCAI07), Hyderabad, India.