* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Artificial Emotion Simulation Techniques for Intelligent Virtual
Survey
Document related concepts
Soar (cognitive architecture) wikipedia , lookup
Perceptual control theory wikipedia , lookup
Existential risk from artificial general intelligence wikipedia , lookup
Agent-based model wikipedia , lookup
History of artificial intelligence wikipedia , lookup
Philosophy of artificial intelligence wikipedia , lookup
Agent (The Matrix) wikipedia , lookup
Ethics of artificial intelligence wikipedia , lookup
Agent-based model in biology wikipedia , lookup
Artificial intelligence in video games wikipedia , lookup
Cognitive model wikipedia , lookup
Transcript
Polytechnics University of Bucharest Faculty of Computers and Automation Department of Computer Science Artificial Emotion Simulation Techniques for Intelligent Virtual Characters Research Report #2 A Model of Affective Behaviour Author: Valentin Lungu, B.Sc. Scientific adviser: Adina Magda Florea, Ph.D. Bucharest January 2012 Abstract The study of what is now called emotional intelligence has revealed yet another aspect of human intelligence. Research in human emotion has provided insight into how emotions influence human cognitive structures and processes, such as perception, memory management, planning, reasoning and behavior. This information also provides new ideas to researchers in the fields of affective computing and artificial life about how emotion simulation can be used in order to improve artificial agent behavior. Virtual characters acting in an intelligent manner based on their goals and desires as opposed to standing still, waiting around for the user and interacting based on pre-scripted dialog trees brings a great deal of depth to the experience. Evolving virtual worlds are by far more interesting than static ones. Applications in which activity independent of the user takes place, such as populating a virtual world with crowds, traffic, virtual humans and nonhumans, behaviour and intelligence modeling [6] are possible by integrating artificial intelligence and virtual reality tools and techniques. Emotion theories attribute several effects of emotions on cognitive, decision and team-working capabilities (such as directing/focusing attention, motivation, perception, behavior towards objects, events and other agents. The main goal of our work is to provide virtual characters with the ability of emotion expression. Our approach is based on the idea that emotions serve a fundamental function in human behavior, and that in order to provide artificial agents with rich believable affective behavior, we need to develop an artificial system that serves the same purpose. We believe that emotions act as a subsystem that enhances human behavior, by stepping up brain activity in arousing circumstances, directing attention and behavior, establishing importance of events and act as motivation. This thesis analyzes important aspects of human emotions and models them in order to create more believable artificial characters and more effective agents and multi-agent systems. This thesis deals with the effects that emotions have on perception, memory, learning and reasoning and their integration into an artificial agent architecture. We show that emotion simulation does not only serve to improve human-computer interaction and behavior analysis, but can also be used in order to improve artificial agent and multi-agent system performance and effectiveness. We study various emotion theories, emotion computational simulation models, and, drawing from their wisdom, we propose a model of artificial agents, which attempts to simulate the effects of human-like emotions on the cognitive and reactive abilities of artificial agents. This thesis summarizes the psychological theories that our computational emotion model is based on, defines the key concepts of the Newtonian emotion model that we developed and describes how they work within an agent architecture. The Newtonian emotion model is a light-weight and scalable emotion representation and evaluation model to be used by virtual agents. We also designed a plug-and-play emotion subsystem to be integrated into any agent architecture. We also describe the first theoretical prototype for our architecture, an artificial agent emotion-driven architecture that not only attempts to provide complex believable behavior and representation for virtual characters, but that attempts to improve agent performance and effectiveness by mimicking human emotion mechanics such as motivation, attention narrowing and the effects of emotion on memory, and explain its shortcomings. We have chosen role-playing gems where virtual characters exist and interact in virtual environments as our application domain for the unique challenges and constraints that arise for artificial intelligence techniques geared towards this type of application, namely, being forced to run alongside CPUintensive processes such as graphics and physics simulations, as well as the fact that role-playing games are unique in the fact that they have social interactions built into the game. We believe that believable emotional expression and behavior will add a new dimension and progress virtual environments beyond visually compelling but essentially empty environments to incorporate other aspects of virtual reality that require intelligent behavior. 2 Chapter 1 Introduction Several beneficial effects on human cognition, decision-making and social interaction have been attributed to emotions, while several studies and projects in the field of artificial intelligence have shown the positive effects of artificial emotion simulation in human-computer interaction and simulation applications. This thesis presents various emotion theories and artificial emotion simulation models, and proposes a model of emotions that tries to reproduce the human-like effects on cognitive structures, processes, patterns and functions in artificial agents through emotion simulation and emotion based memory management, leading thus to more believable agents, while also improving agent performance. Autonomous artificial entities are required to be able to exist in a complex environment and respond to relevant changes in the environment, reason about the selection and application of operators and actions and be able to pursue and accomplish goals. This type of agent has applications in various fields, including character behavior in virtual environments, as used in elearning applications and serious games. Emotion integration is necessary in such agents in order to generate complex believable behavior. It has been shown that emotions influence human cognition, interaction, decision-making and memory management in a beneficial and meaningful way. Several studies in the field of artificial intelligence have also been conducted ([16, 27, 59]) and have shown the necessity of emotion simulation in artificial intelligence, especially when human-computer interaction is involved. Emotions are a part of the human evolutionary legacy [22], serving adaptive ends, acting as a heuristic in time-critical decision-processes (such as fight-or-flight situations); however, emotions have also become an important part of the multi-modal process that is human communication, to the point that its absence is noted and bothersome. 3 Affective computing is the study and development of systems that can recognize, interpret, process and simulate human affects, in order to add this extra dimension into our interaction with machines. Our goal is to create believable intelligent artificial characters capable of displaying affective behavior. To achieve this, we attempt to provide artificial characters with an emotional layer similar to that of humans, serving adaptive ends. The emotion subsystem will influence agent behavior by establishing the importance of events and by influencing knowledge processing, as well as provide the agent with an emotional state that it will be able to express and that will further influence its behavior. We first describe our Newtonian emotion system, that explains the way we represent emotions and how external and internal forces act upon them, based on the work of psychologist R.E. Plutchik [67] and Newton’s laws of motion. Based on the psychological theory of dr. R. Lazarus [46], we devised an emotional subsystem that interacts with and influences an agent architecture. The system also takes into account the effects that emotions have on human perception and cognitive processes. As an application domain, we chose virtual characters in computer role playing games (CRPGs) as they are a type of game meant to simulate all manner of human interactions (be they gentle or violent in nature). This type of game provides us with complex dynamic environments in which modern agents are expected to operate. It also focuses on individual characters and the way they respond to relevent changes in their environment (behavior). Last, but not least, the limited action set available to characters and current generation game engines significantly reduce the overhead of proof-of-concept apps. Applications of the technology are not limited simply to CRPGs, as several types of endeavours find it useful, such as behavior simulation, interactive storytelling, and, indeed, any application involving human-computer interaction (for example, online social sites could recommend events and friends based on mood). 4 Chapter 2 Overview of artificial emotion simulation models We begin this chapter by presenting two popular emotion models with different goals and approaches in emotion simulation, and end by presenting the findings of the PETEEI study that shows the importance of learning and evolving emotions. As shown in this section, there are several key aspects to take into consideration when designing a memory and emotion architecture, and we believe that all are covered by our model. 2.1 Ortony Clore and Colins (OCC) The OCC (Ortony Clore & Collins) model has become the standard model for the synthesis of emotion. The main idea of this model is that emotions are valenced reactions to events, agents or objects (the particular nature being determined by contextual information). This means that the intensity of a particular emotion depends on objects, agents and events present in the environment [59]. In this model, emotions are reactions to perceptions and their interpretation is dependent on the triggering event, for example: an agent may be pleased or displeased by the consequences of an event, it may approve or disapprove of the actions of another agent or like or dislike an object. Another differentiation taken into account within the model is the target of an event or action and the cause of the event/action: events can have consequences for others or for oneself and an acting agent can be another or oneself. These consequences (whether for another or oneself) can be divided into desirable or undesirable and relevant or irrelevant. In the OCC model, emotion intensity is determined by three intensity 5 variables: desirability - reaction towards events and is evaluated with regard to goals, praiseworthiness relates to actions of other agents (and self), evaluated with regard to standards, and appealingness depends on reaction to objects and its value is determined with regard to attitudes. The authors also define a set of intensity variables: sense of reality, proximity, unexpectedness and arousal are the global variables that influence all three emotion categories. There is also a threshold value for each emotion, below which an emotion is not subjectively felt. Once an emotion is felt however, it influences the subject in meaningful ways, such as focusing attention, increasing the prominence of events in memory, and influencing judgments. As shown, according to the OCC model, behavior is a response to an emotional state in conjunction with a particular initiating event [59]. As in the OCC model, in our model, emotions are valenced reactions to events, agents or objects determined by contextual information; in fact we will be using a simplified version of the OCC emotion evaluation process similar to the one used in [27] in early validation of our model. 2.2 Connectionist model (Soar) In this model, emotions are regarded as subconscious signals and evaluations that inform, modify, and receive feedback from a variety of sources including higher cognitive processes and the sensoriomotor system. Because the project focuses on decision-making, the model emphasizes those aspects of emotion that influence higher cognition [36]. The model uses two factors, clarity and confusion (Fig. 2.2), in order to provide an agent with an assessment of how well it can deal with the current situation. Confusion is a sign that the agent’s current knowledge, rule set and abilities are inadequate to handle the current situation, while clarity comes when the agent’s internal model best represents events in the world. Since confusion is a sign of danger it is painful and to be avoided, while clarity, indicating the safety of good decisions, is pleasurable and sought-after. Pleasure/pain, arousal and clarity/confusion form the basis of the emotional system. Instead of creating separate systems for fear, anger, etc., the authors assume that humans attach emotional labels to various configurations of these factors. For example, fear comes from the anticipation of pain. Anxiety is similar to fear, except the level of arousal is lower [36]. The advantage of a generalized system is that it does not require specialized processing. Arousal is the main way that the emotional system interacts with the cognitive system. Memory and attention are most affected by changes in 6 Figure 2.1: Soar 9 system architecture 7 Figure 2.2: Soar emotion subsystem arousal. Highly aroused people tend to rely on well-learned knowledge and habits even if they have more relevant knowledge available. In the Soar implementation of this system, the Soar rules that comprise have special conditions such that different kinds of rules only fire at differing levels of arousal. For example, highly cognitive rules will not fire at high levels of arousal, while more purely emotional rules may only fire at such levels [36]. This allows for a very general approach to emotions. The Soar model emphasizes the effects of emotions on cognition and decision-making, effects considered important and also present in our own model. 2.3 Evolving emotions (PETEEI) Emotion is a complex process often linked with many other processes, the most important of which is learning: memory and experience shape the dynamic nature of the emotional process. PETEEI is a general model for simulating emotions in agents, with a particular emphasis on incorporating various learning mechanisms so that it can produce emotions according to its own experience [27]. The model was also developed with the capability to recognize and react to the various moods and emotional states of the user. In order for an agent to simulate a believable emotional experience, it must adapt its emotional process based on its own experience. The model uses Q-learning with the reward treated as a probability distribution to learn about events and uses a heuristic approach to define a pattern and to further define the probability of an action, a1, to occur given that an action, a0, has occurred [27], thus learning about the user’s patterns of actions. External feedback was used to learn what actions are pleasing or displeasing to the 8 user. The application also uses accumulators to associate objects to emotions (Pavlovian conditioning). The accumulators are incremented by the repetition and intensity of the object-emotion occurrence. An evaluation involving twenty-one subjects indicated that simulating the dynamic emotional process through learning provides a significantly more believable agent [27]. The PETEEI study focuses on the importance of learning agents and evolving emotions. Our memory and emotion architecture is built primarily with these two aspects in mind. 2.4 EBDI The EBDI [64] model proposes an extension to the BDI architecture in which agent behavior is guided not only by beliefs, desires and intentions, but also by emotions in the processes of reasoning and decision-making. The architecture is used to establish some properties that agents under the effect of an emotion should exhibit. The EBDI model describes agents whose behavior is dictated by interactions between beliefs, desires and inteintions (as in the classical BDI arhitecture), but influenced by an emotional component. The emotional component attempts to impose some of the positive aspects that emotions play in reasoning and decision-making in humans into the artifical agent paradigm. EBDI introduces a multi-modal logic for specifying emotional BDI agents. The logic of EBDI The temporal relation is established over a set of elements, called situations. Situations are pairs (world, state), where the world refers to a scenario the agent considers as valid, and the state refers to a particular point in that scenario. Actions are a labelling of the branching time structure and can be either atomic (cannot be subdivided into a combination of smaller actions) or regular (constructions of atomic actions). The model introduces two emotional driving forces: fear and fundamental desire. Fear motivates agent behavior, while the notion of fundamental desire plays a role in properly establishing the notion of fear. The emotions can be triggered either by an action which will lead to some wanted/unwanted situation or triggered by believing that such situations may or will inevitably be true in the future. Bounded reasoning The model also introduces the notions of capabilities and resources. Both Capabilities and resources are prerequisites for successful action-execution. 9 • Resources physical or virtual means which may be drawn in order to enable the agent to execute actions. If the resources for an action do not exist, the action most likely cannot be executed. • Capabilities abstract means that the agent has to change the environment in some way (resemble abstract plans of action). EBDI considers the set of capabilities as a dynamic set of plans which the agent has available to decide what to do in each execution state. Fundamental desires are special desires which are vital desires of the agent, or desires which cannot be failed to achieve, in any condition, since may put in cause the agents own existence. Fundamental desires must always be true and the agent must always do its best to maintain them valid. Fear is one of the most studied emotions.The triggering of this emotion is usually associated with the fact that one or more fundamental desires are at stake. However, this triggering does not always occur in a similar way in all agents. It is based on this differences that different classes of Emotional-BDI agents can be modelled. Fearful agents The fearful agent is an agent that always acts in a very self-preserving way. Negative emotions, such as fear occur when unwanted or uncontrollable facts or events are present in the environment. In its example, EBDI only considers threats and unpleasant facts. Threats are facts or events that directly affect one or more fundamental desire of the agent, putting its self-preservation at stake. • Dangerous a threat is dangerous when the agent believes that some condition leads inevitably to the falsity of a fundamental desire, and also believes that will also be inevitably true in the future • Serious a threat is serious if the same conditions of a dangerous one hold, except that the agent believes that may eventually be true in the future, and not always • Possible a threat is possible if the fundamental desire continues at stake, but the agent believes that it may hold only in the future Unpleasant facts are facts or events that may put one or more desires in a situation of non-achievemnt. The agent will attempt to protext its desires. • Highly Unpleasant something becomes highly unpleasant if the agent believes that the source of the unpleasantness will always occur in the future and will always in the future put in cause some desire 10 • Strongly Unpleasant something becomes strongly unpleasant if the agent believes that the source of the unpleasantness will eventually occur in the future and will always in the future put in cause some desire • Possibly Unpleasant something becomes possibly unpleasant if the agent believes that the source of the unpleasantness will eventually occur in the future and, in the case of occurring, maybe it will put in cause some desire The model also defines the following actions, which represent specific behavior exhibited by the agent under certain conditions. • Self-preservation the self-preservation behaviour is activated when the agent is fearing the failure of some of its fundamental desires. We can see this as atomic action which mainly reacts to threats in a selfprotective way. In EBDI, this special action is represented by spreserv • Motivated Processing Strategy this processing strategy is employed by the agent when some desire which directs its behaviour must be maintained but may be at risk. This strategy is computationally intensive, as its should produce complex data-structures for preserving desires. • Direct Access Strategy this processing strategy relies on the use of fixed preexisting structures/knowledge. It is the simplest strategy and corresponds to a minimization of the computational effort and to fast solutions. A fearful agent is an agent which exhibits a very careful behaviour, considering any threat as a fear factor, and considering all uncomfortable events at the same level. Being careful, the agent also acts towards solving threats and unpleasant facts with the best of its means, and even if the means for the best action are not available, the agent always tries to put itself on a safe, or selfpreservation condition. 2.5 ERIC ERIC is an affective embodied agent for realtime commentary in many domains [86]. The system is able to reason in real time about events in its environment and can produce natural language and non-verbal behavior based on artificial emotion simulation. Efforts have been made to keep the expert system domain indepndent (as a demonstration of this, ERIC was used in two domains: a horse race simulator and a multiplayer tank battle game). 11 The ERIC2 framework (Fig. 2.3) allows for the development of an embodied character that commentates on events happening in its environment. The framework is a collection of Java modules that each encapsulate a Jess rule engine. All cognitive tasks are taken care of in the rule-based expert system paradigm. The ERIC framework also uses template nased natural language generation in order to create a coherent discourse. Knowledge module The main job of the knowledge module is to elaborate information perceived from the environment (throught the domain interface) into a world model that is the basis for the agent’s natural language and emotional state generation. As with all rule based expert systems, logical inference (by applying axioms) is used on perceived in order to elaborate the world model. This world model is the sent to the other ERIC modules in order to generate affect and language. Timestamps are used within the model in order to manage beliefs and allow ERIC to detect patterns. Affective appraisal Emotions are used in order to enhance the character’s believability as well as to convey some information (i.e. excitement means that a a decisive action is being performed). ERIC uses ALMA to model affect [31]. There are three features in the ALMA model: • emotions (short-term) - usually bound to an event, action or object; they decay through time; ERIC uses te OCC model of emotions in order to deduce emotions from stimuli. • moods (medium-term) - generally more stable than emotions; modelled as an average of emotionl states accross time; described by a vector in PAD (pleasure, arousal, dominance) space; intensity is equal to the magnitude of the PAD vector; the characters emotion changes his mood according to the pull and push mood change function. • personality (long-term) - long-term description of an agent’s behavior; modelled along five factors: openness, conscientiousness, extraversion, agreeableness and neuroticism; these traits influence the intensity of the character’s emotional interactions and the decay of its emotional intensities. ERIC makes the appraisals that ALMA requires by comparing events, objects and actions to the agents desires and beliefs. The framework also separates goals and desires from causal relations. The cause-effect relations allow the model to appraise events, actions and objects that are not specified 12 Figure 2.3: ERIC emotion system 13 as goals or desires (the following four relations are used: leadsto, hinders, supports and contradicts). The agent also maintains a set of beliefs about other agant’s goals and desires to enable it to judge the desirability of and event for other agents in the scenario. The agent’s affective state can be expressed through hand and body gestures, facial expression and selection of words and phrasese used to fill in a natural language template. In the ERIC model, affective output is generated from the dynamic appraisal of events, actions and objects related to goals and desires, acting as an indicator of the agent’s progress and its opinion on events. 2.6 Deep The DEEP project specifically designed this model to be used in the video game industry to improve the believability of Non-Player Characters. The emotion dynamics model takes into account the personality and social relations of the character. The model focuses on the influence of personality in triggering emotions and the influence of emotions in inter-character social interactions. Because of the specific nature of video games, the goal is not to reach an optimal behavior, but to produce behaviors that are believable from the player’s standpoint [54]. The proposed model (Fig. 2.4) is an adaptation of the OCC model based on attitudes of NPCs towards actions, objects and other characters of the environment. Each character is defind through its personality, defined by a set of traits, and its social roles, which determine its initial social relation with other characters. Knowledge representation The knowledge representation model used by the DEEP Project as a formal represetnation of the virtual environment that the game takes place in has been kept purosefully simple in order to be easily handled by designers and developers that do not have a background in artificial intelligence. Actions are defined as couples (namea , ef f ecta ). A character is defined by two feature vectors, one of attitudes that express its attitude towards element in the lexicon (each possible element in the world) and one of praises, which is a feature vector of attitudes towards possible actions in the environment. Events are simply represented by a quadruplet (agent, action, patient, certainty), where certainty denotes the degree of certainty of the event. 14 Figure 2.4: Deep emotion system Emotion representation Emotion representation is based on a simpification of the OCC model, in which 10 emotions represent an agent’s emotional experience. Emotions are considered according to triggering conditions: joy (distress) - desirable (undesirable) event; hope (fear) - expectation of a desirable (undesirable) event; relief (disappointment) - non-occurrence of an expected undesirable (desirable) event; pride (shame) -praiseworthy (blameworthy) action done by the agent; admiration (anger) - praiseworthy (blameworthy) action done by another agent. An emotional state is represented as a feature vector of values varying in [0,1]. The emotional stimulus of an event is the emotion that an agent should feel after the event, provided that the agent has a neutral personality. The OCC model posits that the intensity of emotions is directly proportional to the desirabilty/undesirability factor of the event (and its probability of occurence). Personality The personality of a virtual character is represented as an ndimensional feature vector, each value corresponding to a personality trait. The emotion triggerd by an event is computed from the emotional stimulus and the personality trait’s values. When no emotional stimulus occurs, a character’s emotion intensity naturally decays towards the neutral state, by a monotonous function that depends on personality. The model also mentions intensity thresholds that define the minimal value for an emotion to have and impact on a character’s behavior. 15 Social The relationship between two virtual characters is represented as a quadruplet = (liking, dominance, familiarity, solidarity), respectively the degree of liking i has for j at time t; the power that i thinks it can exert on j at time t; the degree of familiarity, i.e. the average degree of confidentiality for i of the information that i has transmitted to j; and the degree of solidarity that represents the degree of like-mindedness or having similar behavioral dispositions that i thinks to have with j. Relationships are not symmetrical. A social role rx /ry is definedby a quadruplet where each element represents the degree of liking, dominance, familiarity and solidarity that a character with role rx feels for a character with a role ry . In order to compute a combination of several roles, using an average of complementary roles. The model has been implemented in Java as a tool for game designers to simulate the impact of a scenario on non player characters. The vocabulary of the game, NPCs personality, social roles nad scenario events need to be defined. The DEEP Project motivates the use of emotionally-enhanced intelligent agents as virtual characters in video games, in order to improve the player’s feeling of immersion and enjoyment. The model takes into account personality and social relations The heart of the model presented is contextindependent. It can be applied to any domain that makes use of intelligent agents with an affective level. This work is part of the DEEP project supported by RIAM (Innovation and research network on audiovisual and multimedia). 2.7 iCat iCat is a system developed in order to study how emotions and expressive behavior in interactive characters can be used to help users better understand the game state in educational games. The emotion model is mainly influenced by the current game state and is mased on the emotivector anticipatory mechanism. The hypothesis is that if a virtual character’s affective behavior reflects the game state, users will be able to better peprceive the game state [61]. Architecture The architecture is composed of three modules: • The game module is made up of the game played by the user and the virtual character. The game module is the main input for the emotion module. 16 • The emotion module receives evaluations from the game module, processes them and update the agent’s emotional state. • The animation module animates the character by setting up a character position or running the character through predetermined animation sequences. The predefined sequences are used to display emtoional reactions (each one of the 9 elements in the emotivector is mapped to a different animation). Mood is reflected asa linear interpolation between the two parametrizations corresponding to the two extremes of the valence space. Emotion The affective state is represented by two components: emotional reactions and mood. Emotional reactions are the immediate emotional feedback received by the character. In order to endow the agent with emotional ractions, the iCat uses the emotivector anticipatory system. The system contains a predictive system of itself and its environment that generates an affective signal resulting from the mismatch of the expected and received values of the sensor. This means that the system is either rewarded or punished if events do not go according to predictions, and this will show up through the displayed animation. Mood is represented as a valence variable that varies between a minimum and a maximum value. The mood magnitude represents the mood intensity. Positive values are associated to good scores, and negative ones to bad ones. Mood works like a background affective state, when other emotions are not occuring. Mood decays towards zero, over time in the absence of new stimuli. Upon receiving a new stimuli, the adjustment is gradual until the desired value is reached. Evaluation The architecture was implemented in a scenario composed by a social robot, the iCat, which acts as the opponent of a user in a chess game. A preliminary experiment was performed, where the iCat played chess against a human player. The results (correlated from a group where the iCat showed the proper emotions, a group where it showed the wrong ones randomly and a control group where it showed no emotions), indicate that users were able to correctly interpret the iCats emotional behaviour and that the correlation between the users own analysis of the game and the actual game state variables is also higher in the emotional condition. The results of a preliminary evaluation suggested that the emotional behaviour embedded in the character indeed helped the users to have a better perception of the game [61]. 17 2.8 Comparison In this section we present a comparison of the emotion simulation models presented relating to features we consider important. • Knowledge whether the model influences agent knowledge • Perception whether the model influences agent perception • Behavior whether the model influences agent behavior • Motivation if emotions are used as agent motivation within the model • Personality if the model is able to generate different types of characters, characters that would act differently, given the same virtual world state • Theory How complex the theory is • Dev how high the design, development and implementation effort would be for someone to use the system • Adoption how easily a system would be adopted by the game industry • Graphics does the model have a way for artificial characters to express their emotions? • Emotion what emotion representation model the system uses • Synthesis what emotion synthesis model the system uses • Expandability how expandable the model is • Scalability how scalable the model is • Flexibility how flexible the model is • Evolving if the emotion process and behavior are adapted based on agent’s experience As can be seen in table 2.1, most systems use the OCC model for emotion expression and synthesis which is difficult to learn and implement, which can be deduced from the fact that the authors have reduced adapted by reducing the number of emotions from 21 to 10, and as can be seen through the effort expanded to formalize it [82, 83, 85]. We can also see from the table that for most models, either the emotion process or behavior are not adapted 18 Name Soar PETEEI EBDI DEEP iCat Knowledge Perception Behavior Motivation yes maybe yes yes yes yes yes yes Name Soar PETEEI EBDI DEEP iCat Theory(cpx) Dev (cpx) Adoption Graphics Emotion simple simple med. Pleasure/Pain high med. med. yes OCC high OCC high yes OCC med. simple med. yes iCat Name Soar PETEEI EBDI DEEP iCat Synthesis Expandability Scalability Flexibility Evolving Soar yes yes med. static OCC low dynamic OCC yes yes high static OCC yes yes high static iCat med. static Table 2.1: Emotion simulation model comparison 19 Personality yes according to the agent’s experiece, and none of them use emotion as a driving force for a character’s actions. We have taken it upon ourselves to design and develop a fast light-weight, easily understood and adopted emotion representation model, the system necessary for its integration into an artificial agent architecture that uses emotion simulation as motivation for the virtual character’s actions, as well as the underlying framework for its integration into our chosen demo application type. 20 Chapter 3 Newtonian Emotion System (NES) This chapter presents our new and improved emotion simulation technique, still based off of Plutchik’s emotion taxonomy. However, the emotion representation scheme has been streamlined, the emotion feature being reduced to a four-dimensional vector, and emotion interactions follow Newtonian interaction laws that are easier to learn and understand. The architecture has also been reduced to the bare minimum necessary for emotion simultaion and expression while still influencing character behavior and memory in the same way. This is because, while we believe that the best way to produce humanlike behavior is to attempt to reproduce thought patterns, game developers, for example, tend to favor finite-state-machine based solutions that are very fast and reduce development complexity. The new architecture allows for any behavior generation technique (belief-desire-intention, expert systems, behavior trees, finite state machines or even backtracking) and any machine learning algorithm (SVM, linear regression, clustering), in essence, allowing designers to develop a system that suits them best, while incoroprating emotion simulation. 3.1 Newtonian emotion space In this section we introduce the Newtonian Emotion Space, where we define concepts that allow emotional states to interact with each other and external factors, as well as the two laws that govern the interaction between an emotional influence and an emotional state. Concepts 21 Definition 1. Position specifies the intersection of an emotional state with each of the four axes Definition 2. Distance (||p~2 − p~1 ||) measures the distance between two emotional positions Definition 3. Velocity (~v = p~2 −t p~1 ) represents the magnitude and direction of an emotion’s change of position within the emotion space over a unit of time Definition 4. Acceleration (~a = v~2 −t v~1 ) represents the magnitude and direction of an emotion’s change of velocity within the emotion space over a unit of time Definition 5. Mass represents an emotional state’s tendency to maintain a constant velocity unless acted upon by an external force; quantitative measure of an emotional object’s resistance to the change of its velocity Definition 6. Force (F~ = m · ~a) is an external influence that causes an emotional state to undergo a change in direction and/or velocity Laws of Emotion Dynamics The following are two laws that form the basis of emotion dynamics, to be used in order to explain and investigate the variance of emotional states within the emotional space. They describe the relationship between the forces acting on an emotional state and its motion due to those forces. They are analogous to Newton’s first two laws of motion. The emotional states upon which these laws are applied (as in Newton’s theory) are idealized as particles (the size of the body is considered small in relation to the distance involved and the deformation and rotation of the body are of no importance in the analysis). Theorem 1. The velocity of an emotional state remains constant unless it is acted upon by an external force. Theorem 2. The acceleration ~a of a body is parallel and directly proportional to the net force F~ and inversely proportional to the mass m: F~ = m · ~a Emotion center and gravity The emotion space has a center, the agent’s neutral state, a point in space to which all of the agent’s emotional states tend to gravitate (usually (0,0,0,0), however, different characters might be predisposed to certain kinds of emotions, thus, we should be able to specify a different emotional centre, and instill a certain disposition, for each character; we also use different emotion state mass to show how easy/hard it is to change 22 an agent’s mood). In order to represent this tendency we use gravitational force: ~ = m · p~ − ~c · kg , G ||~p − ~c|| where p is the current position, c is the center and kg is a gravitational constant. This force ensures that, unless acted upon by other external forces, the emotional state will decay towards the emotion space center. 3.2 Plug-and-play architecture We developed an emotion subsystem architecture (Fig. 3.1) that interposes itself between the agent’s behavior module and the environment. The subsystem models the process described by Lazarus. Events perceived from the environment are first processed by the appraisal module, where an emotional force is associated with it. The resulting list of events is then sorted in descending order according to magnitude and fed into the agent’s perception module. This is done in accordance with the psychological theory of attention narrowing, so that in a limited attention span scenario, the agent will be aware of the more meaningful events. We treat the agent behavior module as a black box that may house any behavior generation technique (rule based expert system, behavior trees, etc.). The interface receives as input events perceived from the environment and produces a set of possible actions. The conflict set module takes this set of rules and evaluates them in order to attach an emotional state to each. Out of these, the action whose emotion vector most closely matches the agent’s current state is selected to be carried out. arg min arccos i∈{conf lictset} eagent · ei ||eagent || · ||ei || Feedback from the environment has an associated emotion force vector that actually affects the agent’s emotional state. Feedback is distributed to the appraisal and conflict set modules as well as the agent’s behavior module. Actions undertaken are temporariliy stored in an action buffer for feedback purposes. 3.3 NES Learning Both appraisal (percept and action appraisal) modules use machine learning techniques in order to learn to predict the outcome of events and actions, 23 Figure 3.1: Emotion system architecture 24 respectively. As there are many viable options for which learning technique to use (hidden markov models, neural nets, q-learning), we use a plug-andplay model where the technique can be replaced. The appraisal module attempts to predict the emotional feedback received from the environment based on characteristics of the event to classify and previous feedback. The goal of the appraisal module is to better label (and thus establish priority of) events for the perception model. The conflict set module works in a similar way based on the characteristics of actions taken. The goal of the conflict set module is to settle conflicts between competing actions by selecting the most appropriate action to be performed. The action that has the greatest Gain − Risk ratio will be chosen. Both modules treat the event, rule respectively, configuration as the observable state of the model, and attempt to predict the next hidden state (the emotional feedback force). Newtonian emotion system interaction is a three stage process. Each inference turn, the behavior module generates a list of possible actions within the given context for the virtual character. Each action is then appraised by the action appraisal module (that is a wrapper interface for a machine learning technique that attempts to learn the feedback function). At this time, the value is and absolute value, that needs to be related to the agent’s current state. This is done by evaluating the impact of an emotional force on the agent’s current state. Based on this impact factor, actions to be carried out are selected and executed. Once executed, these actions receive emotional feedback from the environment and the appraisal module that evaluated them is updated with the new values. Let us take the following example: character A’s behavior module generates as the next action, an attack on character B. The following simplified feature vector for the action is built, that encodes information about the current context (feelings about the action, feelings about our target and action direction). From previous experience, the agent knows that the action would raise it’s aggression level (feelings about the attack action are on the anger and anticipation axes [0,0,-.9,-.3]); we also assume that we are somewhat afraid of the character that we are attacking [0,0,.7,0], and we know that we are performing the action, so action direction is 1. Based on this feature vector, the appraisal module predicts an emotional feedback with a dominant on the anger and anticipation axes of [0,0,-.2,-.2]. We assume this action has a satisfactory Gain/Risk ratio and gets selected and executed. We also assume that it succeeds. According to the environment feedback formula we used, the result would be [0,0,-.2,-.3] which means that the estimation was close, but could be better. The feedback result is added to the learning module data set, and the results are nudged in the direction of [0,0,-.2,-.3] 25 [0,0,-.2,-.2] = [0,0,0,-.1], for a better future prediction. The perception appraisal module works in a similar way to appraise perceived actions, however the percepts are sorted according to a factor based on their distance to the agent’s current emotional state (minimum) or their distance to the average emotional state of events perceived this turn (maximum). This is so that an agent will prioritize events that it resonates with or that stand out. Validation In order to validate the learning module, we randomly generated feature vectors with valid values, used the feedback formula to label them, and separated them into learning and validation data sets. These data sets were used to validate a linear regression learning model. The results can be seen in the following graphs that show the error between training and validation related to number of training examples. A slight oscillation can be seen in convergence due to the fact that some information mey be missing in certain contexts. Figure 3.2: Joy learning curve 26 Figure 3.3: Trust learning curve 27 Figure 3.4: Fear learning curve 28 Figure 3.5: Surprise learning curve 3.4 Personality filter The personality filter allows designers to skew an agent’s perception of events. It defines a perception in intepreting emotional forces when applied to feedback received from the environment. The personal filter consists of a four dimensional vector (each element matching an axis of the Newtonian Emotion representation scheme) with values between -1 and 1 that scales the agent’s emotional feedback according to personal criteria. For example, an agent may feel fear more accutely than joy and as such, its personal filter should reflect this, having a lower scaling factor for joy than for fear. Another agent may be extremely slow to anger, this would be reflected in the personal filter as a very low scaling factor for the anger axis. It would even be possible to change the sign altogether so that some agents may feel anger instead of fear or joy instead of sadness (leaving room for the representation of psychologically damaged individuals). The feedback applied to the agent’s state would then be: F eedback 0 = F ilter · F eedback (3.1) Together with the emotional center and emotion mass, these three would 29 be all the parameters that a character designer needs to interact with in order to generate varied virtual characters capable of rich unique nuanced emotiondriven behavior. As can be seen, the system would be easy to learn and use in the industry thanks to the small number of parameters that influence it and the intuitive nature of these and their interactions within the model. 3.5 Emotion as motivation As mentioned before, the emotional feedback that an agent receives from the environment influences its behavior. This is achieved by selecting an action from the conflict set provided by the behavior model according to the feedback that the agent predicts from the environment. In order to evaluate its effect on the agent’s current state, we put forth the notion of tensor product between two states, because of the information this gives about the influence that one axis has on another. This product has an interpretation as stress produced by an emotional force on the current emotional state, showing how the force is, relative to the current state of the agent. This is called the emotional impact and the tensor product is computed between the two forces acting on the agent: the Gravity force, keeping him in his current state, and the learned action force (estimated feedback). Stress = [F eedback] · [Gravity] fjoy ftrust = ff ear · gjoy gtrust gf ear gsurprise fsurprise (3.2) This allows the assessment of the influence that each emotion axis of the action has on those of the current state. In line with Lazarus’ appraisal theory, emotional evaluation implies reasoning about the significance of the event, as well the determination of the ability to cope with the event. Therefore we define two metrics for assessing emotional experiences: Well-being (Joy and Trust) and Danger (Fear and Surprise). Well-being measures the effect the action has on the positive axes (Joy-Trust) of the current state, thus establishing the gain implied by the action; while Danger describes the risk, as it evaluates the action in terms of its influence on the Fear-Surprise axes. When choosing an action, an agent estimates how much the current wellbeing is influenced by the action in regard to how much danger the action 30 implies, a gain-risk ratio. In terms of the tensor product, this involves a column-wise computation with respect to the Well-being axes of the agent’s current state (for Gain) and a row-wise computation with respect to the Danger axes of the action (for Risk). Gain = fjoy ·(gjoy +gtrust )+ftrust ·(gjoy +gtrust )+ff ear ·(gjoy +gtrust )+fsurprise ·(gjoy +gtrust ) (3.3) Risk = ff ear ·(gjoy +gtrust +gf ear +gsurprise )+fsurprise ·(gjoy +gtrust +gf ear +gsurprise ) (3.4) Thus the Impact factor can be used to assign priorities to all actions in order to resolve the conflict set. Impact = Gain − Risk (3.5) Let’s assume that a character’s current state is [.3,.5,-.1,-.1] (gravity = [-.5,-.8,.16,.16]). We will further assume that the agent finds itself in a conflictual situation surrounded by other agents. We will also assume that the behavior module will only generate options to attack the other characters, and that the appraisal module predicts the correct feedback. In the table, the possible feedback options are various degrees of • fear and surprise (alarm) - the attack fails, but was not expected to • anger and surprise (outrage) - when the attack succeeds, but was not expected to • anger and anticipation (aggression) - when the attack succeeds and was expected to According to our feedback formula, we have the following impact factors: In table 3.1) we can see that the agent will prefer the two options that will increase it’s aggression, will be neutral towards attempting to attack an agent that will suprise it, and will be reluctant to attack when it might fail. As a further example, let us assume that the agent is in the same state and that it is surrounded by objects that it can pick up, the only actions generated are actions to pick up objects and the joy felt when picking up an value(object) . object is equal to value(object)+wealth(agent) As we can see in the second table (Fig. 3.2), for varying item values, we get different priorities. The agent would prefer to pick up the most valuable item first. 31 [-0.5 [-0.5 [-0.5 [-0.5 [-0.5 [-0.5 Gravity -0.83 0.16 -0.83 0.16 -0.83 0.16 -0.83 0.16 -0.83 0.16 -0.83 0.16 0.16] 0.16] 0.16] 0.16] 0.16] 0.16] Feedback Gain [0 0 0.2 0.3] 0.4 [0 0 -0.6 -0.6] 1.56 [0 0 -0.4 0.3] 0.13 [0 0 -1 -1] 2.6 [0 0 1 0.6] -2.08 [0 0 1 0.3] -1.69 Risk Impact 0.3 0.10 1.176 0.38 0.098 0.03 1.96 0.64 -1.568 -0.51 -1.274 -0.42 Table 3.1: Attack impact table [-0.5 [-0.5 [-0.5 [-0.5 [-0.5 [-0.5 Gravity -0.8 0.16 -0.8 0.16 -0.8 0.16 -0.8 0.16 -0.8 0.16 -0.8 0.16 0.16] 0.16] 0.16] 0.16] 0.16] 0.16] Feedback Gain [0.1 0 0 0] -0.13 [0.2 0 0 0] -0.26 [0.3 0 0 0] -0.39 [0.4 0 0 0] -0.52 [0.5 0 0 0] -0.65 [0.6 0 0 0] -0.78 Table 3.2: Take impact table 32 Risk 0 0 0 0 0 0 Impact 0.13 0.26 0.39 0.52 0.65 0.78 Chapter 4 Emotion integration in game design In this chapter we present how the two architectures (the Emotion-Driven Character Behavior Engine, and the Newtonian Emotion System) were integrated to provide non-player character behavior in two games: a fire evacuation scenario serious game named ALICE, meant to teach preteen students proper fire evacuation procedures, and a demo game that showcases the capabilites of the NES, called OrcWorld. 4.1 Artificial intelligence in virtual environments Research into virtual environments on the one hand and artificial intelligence on the other has mainly been carried out by two different groups of people, but some convergence is now apparent between the two fields. One of the great selling points of virtual environments is their immersion factor, therefore researchers are seeking to progress beyond visually compelling but essentially empty environments to incorporate other aspects of virtual reality that require intelligent behavior. Virtual characters acting in an intelligent manner based on their goals and desires as opposed to standing still, waiting around for the user and interacting based on pre-scripted dialog trees brings a great deal of depth to the experience. Evolving virtual worlds are by far more interesting than static ones. Applications in which activity independent of the user takes place, such as populating a virtual world with crowds, traffic, virtual humans and non-humans, behaviour and intelligence modeling are possible by integrating artificial intelligence and virtual reality tools and techniques. 33 This combination of intelligent techniques and tools, embodied in autonomous creatures and agents, together with effective means for their graphical representation and interaction of various kinds, has given rise to a new area at their meeting point, called intelligent virtual environments [6]. 4.1.1 Application domains Games A video game is an electronic game that involves interaction with a user interface to generate visual feedback on a visual device. Video games have been using virtual environments from the start. The popular video game Tron inspired the movie of the same name where a computer programmer is transposed int o the virtual world of Tron. The most popular virtual environments to date belong to video games (World of Warcraft, Second Life, The Sims). The video game industry has also been using artificial intelligence techniques as well, so in essence, the field of intelligent virtual environments has existed for decades, however, video games usually limit themselves to brute force or very time-efficient techniques, however it is our opinion that the video game industry would have much to gain from the finess offered by more refined artificial intelligence techniques. Serious Games Another field that has recently gained momentum is the field of serious games. More and more companies and state agencies are using serious games to train and educate employees or the general public, respectively. A serious game is a game designed for a primary purpose other than entertainment, such as education or training. Although they can be entertaining, serious games are designed in order to solve a problem and may deliberately trade fun and entertainment in order to make a serious point. There are many advantages to using serious games in education. First of all, computer game developers are accustomed to developing games that simulate functional entities quickly and cheaply by using existing infrastructure. Secondly, game developers are experienced at making games fun and engaging, automatically injecting entertainment and playability into their applications, features that serious games, while meant to train or educate the user, could also benefit from. One artificial intelligence paradigm has been used extensively in the field of training and education, namely rule-based expert systems. These have expert knowledge in their domain of expertise and are able to create plans, explain actions and even reason non-monotonously [71]. Affective gaming has received much attention lately, as the gaming community recognizes the importance of emotion in the development of engaging games. Affect plays a key role in the user experience, both in entertainment 34 and in serious games developed for education, training, assessment, therapy or rehabilitation. A significant effort is being devoted to generate affective behaviours in the game characters, player and AI-controlled avatars, in order to enhance their realism and believability [39]. Games Emotion has become increasingly established, as the following examples illustrate: in the SIMS, characters actions are influenced by an emotive state, which results from their personality and events occurring during the game [4]; in Fahrenheit, players are able to influence the emotional state of characters indirectly [2]. There is also an increased interest in affective storytelling, creating stories interactively. E-Learning Education has always seemed a natural application domain for artificial intelligence and virtual reality technologies, because visual representation and social interaction play a large part in learning and human memory. There are a few applications worth mentioning: in the Cosmo System, an artificial agent used emotional means to achieve teaching goals. ITSPOKE [3] tried to register the students emotional states and use them to tailor spoken tutorial dialogues appropriately. FearNot used emotion-driven virtual characters in order to teach lessons about bullying [17]. The growing need for communication, visualization and organization technologies in the field of e-learning environments has led to the application of virtual reality and the use of collaborative virtual environments [70]. Virtual reality, together with emotion-capable artificial cognitive agents can highly broaden the types of interaction between students and virtual tutors. As in conventional learning tasks, the computer can monitor students, responding to questions and offering advice in a friendly manner, while keeping the lessons pace comfortable to the student. Artificial agents are able to fulfill several valuable roles in a (virtual) elearning environment: act as a virtual tutor and explain lessons, give help, advice and feedback to students; act as a students learning peer and/or participate in simulations, especially useful in team training scenarios; an artificial agent may also be used (known or unknown to the student) in order to monitor progress and personalize the students learning experience. An artificial agent present in a virtual environment should dispose of the following capabilities: demonstrate and explain tasks, monitor students and provide assistance when it is needed, provide feedback to the student, ability to explain the reasoning behind its actions and adapt (a plan) to unexpected events and be able to fill the roles of extras [70]. A virtual tutor needs to have a way of storing and manipulating domain specific knowledge in a useful way in order to be able to explain to, and 35 correct students, and justify its actions. In practice, the best way to build such a system is by using a rule-based system together with a justificationbased truth maintenance system. Steve [71] is an animated pedagogical agent for procedural training in virtual environments that teaches students how to perform procedural tasks. Steve is able to demonstrate and explain tasks, and monitor students performing tasks, providing assistance when it is needed, by making use of expert knowledge of the domain stored in the form of hierarchical plans that contain a set of steps (the steps required to complete an action, in order for Steve to demonstrate and explain), a set of ordering constraints (so that Steve is able to provide assistance or intervene at any point during a student’s performance of a task), and a set of causal links (that allow Steve to explain a certain action). 4.1.2 Techniques In this section we present and discuss applications of well-known artificial intelligence paradigms and techniques in virtual environment applications, with a focus on games, serious games and e-learning. Neural networks An artificial neural network, usually called neural network, is a mathematical model or computational model that is inspired by the structure and/or functional aspects of biological neural networks. A neural network consists of an interconnected group of artificial neurons, and it processes information using a connectionist approach to computation. In other words, a neural network is an interconnected assembly of simple processing elements, units or nodes whose functionality is loosely based on the biological neuron. The processing ability of the network is stored in the inter-unit connection strengths or weights obtained by a process of adaptation to or learning from a set of training patterns [35]. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network during the learning phase. Modern neural networks are non-linear statistical data modeling tools. They are usually used to model complex relationships between inputs and outputs or to find patterns in data. Artificial neuron design is based off of the initial model developed by McCullogh-Pitts, where each neuron has several inputs (synapses, x ) and one output (axon, y). Each input has an associated weight (w ), that stores the neuron’s evolution and allows it to adapt to the data that it is being trained on. 36 internal activation potential: ni = n X wij · xi + bi (4.1) j=0 activation of the neuron: yi = ψ(ni ) (4.2) Neural networks are capable of both supervised and unsupervised learning, and several error propagation and weight update algorithms exist. Neural networks are fundamentally different from the processing done by traditional computers and are better suited to problems currently difficult to solve, such as meaning of shapes in vidual images, distinguishing between different kinds of similar objects (classification and association), learning certain functions (including unknown functions). In a MMORPG context, neural networks could act as fast, efficient classifiers (the costly process of training neural networks is usually done offline, while the classification is usually fast) that help artificial agents classify and reason about objects in their environment, including users. For example, based on characteristics exhibited by user or artificial characters, such as faction, class, gender, build, background, an artificial agent could classify another agent as a threat or non-threat, or as an enemy or a foe. Genetic algorithms Genetic algorithms are a search heuristic that mimics the process of natural evolution, routinely used to generate useful solutions to optimization and search problems. In a genetic algorithm, a population of individuals, each of which encode candidate solutions to an optimization problem evolves toward better solutions by using techniques inspired by natural evolution such as inheritance, mutation, selection and crossover. Problems which appear to be particularly appropriate for solution by genetic algorithms include timetabling and scheduling problems. Genetic algorithms are also a good approach to solve global optimization problems, especially domains that have a complex fitness landscape as the combination of mutation and crossover operators is designed to move the population away from local optima. Within the virtual intelligent environment context, genetic algorithms may be used to generate walking methods for computer figures, bounding box generation, as well as any optimization problems artificial agents may require, where an exact solution is not strictly required and strict time constraints are imposed, such as planning. 37 Self organizing systems Self-organization is a process in which structure and functionality at the global level of a system emerge solely from numerous interactions among the lower-level components of a system without any external or centralized control. The system’s components interact in a local context either by means of direct communication of environmental observations without reference to the global pattern. Swarm intelligence is the collective behaviour of decentralized, self-organized systems, natural or artificial. The concept is employed in work on artificial intelligence. Also known as self organizing systems, they are typically made up of a population of simple agents interacting locally with one another and with their environment. The agents follow very simple rules, and although there is no centralized control structure dictating how individual agents should behave, local interactions between such agents lead to the emergence of intelligent global behavior, unknown to the individual agents. Natural examples of swarm intelligence include ant colonies, bird flocking, animal herding, bacterial growth, and fish schooling [42]. Swarm intelligence as a field is based on algorithms developed to simulate this type of complex behavior that emerges in the natural world, and thus, the obvious application of these techniques in virtual environments would be to simulate the behavior of similar creatures that inhabit the environment; however, other uses are possible as well, such as using ant colony optimization in order to compute optimal paths the agent may take in the virtual environment or using particle swarm optimization in order to create more complex and versatile particle systems [53]. Other applications of swarm intelligence are crowd simulation and ant-based routing. Reactive agents Reactive agents are simple processing units that perceive and react to changes in their environment. The architecture consists of a perception component, a decision component and an action component. The behavior is quite simple: in a loop, the agent perceives the state of its environment (which may be complex and may include the states of other agents), selects and action to perform and then performs it, changin its environment’s state. Reactive agents are able to reach their goal only by reacting reflexively to external stimuli. While limited, reactive agents operate in a timely fashion and are able to cope with highly dynamic and unpredictable environments, making them ideal for an artificial environment’s lower, less intelligent artificial life forms, such as animals and unevolved humanoids. Cognitive agents Artificial agents inhabiting a virtual environment, where multiple aspects of the world are currently changing, must be able to infer 38 new knowledge about the world, make plans, accomplish goals and take decisions in uncertain conditions in real-time, in order to challenge human users. This is what cognitive agents are able to do. Several approaches based on the belief-desire-intention model of agency have been desgined and implemented successfully, while other approaches are being tested. The hierarchical goal decomposition used in backward chaining engines is a powerful tool, allowing agents to represent and solve complex problems [49], while forward chaining techniques allow exploration of the knowledge domain. Truth maintenance systems and fuzzy logic are two of the techniques that may be employed in order to reason in uncertain conditions by artificial agents, either by maintaining a coherent knowledge base (handles conflicting knowledge and maintains a consistent set of beliefs), or adding a certainty weight to knowledge items. The cognitive agent model is very similar to the reactive agent model, however, the decision component now handles planning as well (next action), while the interaction component adds a social dimension to agent interaction. We have proposed an architecture that uses forward and backward chaining, truth maintenance and emotion simulation to achieve fast decision-making cognitive agents. Expert systems Another subfield of artificial intelligence, expert systems use a knowledge base of human expertise on a specific problem domain. An expert system uses some knowledge representation structure to capture knowledge, typically gathering from knowledge human experts on the subject matter and codifying it according to the structure. Last but not least, the expert system is placed in a problem-solving situation where it can use its expertise. Successful e-learning applications have been developed in the past [71], where the expert system acts as a virtual tutor. Emotion simulation Affective computing is the study and development of systems and devices that can recognize, interpret, process, and simulate human affects. Emotion integration is necessary in generating complex believable behavior, making the agent decision-making process less predictable and more realistic as well as generating actions in time comparable to human reaction time. Emotion simulation allows virtual environments more realistic simulations of artificial characters, while virtual environments allow artificial agents capable of emotion simulation more expressive freedom because of the additional channels available. There are several motivations for this research, relative to its intended application. One approach is the simulation of emotions in conversational 39 agents in order to enrich and facilitate interactivity between human and machine. In e-learning applications, affective computing can be used to adjust the presentation style of a virtual tutor when a learner is bored, interested, frustrated or pleased. Psychological health services, an application domain for virtual reality as well, can also benefit from affective computing: patients would be able to act in a virtual world populated by interacting artificial agents and be observed and diagnosed by a professional or learn how to correct their own behaviour. Our understanding of human emotion is best within the cognitive modality and most existing models of emotion generation implement cognitive appraisal, which is considered to be best suited for affective modeling in learning and gaming applications. The study of emotional intelligence has revealed an aspect of human intelligence. Emotions were shown to have great influence on many of human activities, including decision-making, planning, communication, and behavior. Emotion theories attribute several effects of emotions on cognitive, decision and team-working capabilities (such as directing/ focusing attention, motivation, perception, behavior towards objects, events and other agents. One of our research papers proposes the use of emotion simulation as an efficiency measure in cognitive agents, focusing attention/perception and managing agent memory, as well as giving the agent a new expressive dimension [50]. Ontologies The term ontology denotes a branch of philosophy that deals with the nature and organisation of reality. Virtual reality could benefit from an ontology. Each object that exists in virtual space could attached a defition using domain-specific ontological terms. This could improve the virtual environment for both human users and artificial ones. The downside is that an ontology would have to be defined and each object in the scene graph would be doubled in an ontolgy graph, taking up more memory. In computer science, an ontology is a formal representation of knowledge as a set of concepts within a domain, and the relationships between those concepts. An ontology is necessary in order for agents to have a shared understanding of a domain of interest, a formal an machine manipulable model of said domain. It is used to reason about the entities within that domain, and greatly enhances symbolical reasoning. Ontologies aid in interagent communication, definitions being exchanged between agents until a consensus is reached. Ontologies are usually defined in such a way as to be easily interpreted by humans as well, and may be used to bridge the gap created by the deficiencies in the field of natural language processing, making 40 interaction between human and artificial agents easier. One may also limit the options human agents have in expressing themselves in such a way so as to be indistinguishable from artificial agents, increasing immersion. However, even without limiting user expresivity, which is not a desired feature in virtual environments, an ontology would still increase human-computer interaction quality. 4.2 Orc World During an internship at the Laboratoire d’Informatique Paris 6, Universite Pierre et Marie Curie, paris, France, we devised and implemented a game on top of the virtual character framework to test the Newtonian Emotion System. In the following sections we go on to define the scenario and game rules, present what actions are available to agents in the game and how their emotional feedback, and detail the agent behavior generation and learning algorithms we used. 4.2.1 Scenario and rules The premise of the game is simple. We created a virtual game world inhabited by orcs. Each orc has a treasure stash that he must increase (in essence hoarding items and valuables). The orcs may do this by either foraging in the wilderness, stealing from other orcs or pillaging other orcs’ stashes, however, in order to do this they must leave their own treasure unprotected. This scenario provides us with the same goals for all virtual characters involved (survive and hoard) and allows us to use the same behavior generation module for all characters, showing the difference in behavior based on personality characteristics and character experience. The scenario also provides us with a balanced mix of social, physical and environment interactions. In order to determine the success or failure of actions, we used a very simplified version of the dungeons and dragons 3.5 edition rules [11]. The main reason behind this was that the dungeons and dragons ruleset contains rules that govern social interactions as well as physical ones. Alg. 1 describes the turn sequence. 4.2.2 Actions and feedback Possible actions within the game world and their associated effects are: • Attack The character attacks another character with the goal of disabling it. 41 Figure 4.1: Round sequence diagram 42 Algorithm 1 Turn sequence for all character ∈ characters characterinitiative = roll(d20) end for for all character ∈ characters Update(characterpercepts ) Generate(characteractions ) Appraise(characteractions ) Run(characteractions ) end for for all character ∈ characters Generate(characterf eedback ) Process(characterf eedback ) end for do {determine turn order} do do – success: the attack hits, inflicting damage (decreasing the health) of the opposing character – failure: the attack misses, however the target is aware of the attack – feedback: this action influences the anger and anticipation axes (aggression) • Social The character attempts to impose its will that another character perform an action through social interaction. – success: the interaction succeeds and the target performs the desired action next turn – failure: the interaction fails and the target is aware of the attempt – feedback: ∗ Bluff the target is tricked into performing the action; feedback is on the trust and joy axes (disgust) ∗ Diplomacy the target is lead to believe that it is in its best interest to pursue the action; feedback is on the positive side of the joy and trust axes (love) ∗ Intimidate the target is bullied into carrying out the action; feedback is on the trust and anger axes (dominance) • Move The character moves to a new destination. – success: always succeeds 43 – feedback: this action receives feedback either according to the area the character is moving to (which is an average of the feedback of all the actions that a character has experienced in the area), or on the joy and surprise axes if it is a new area • Take The character attempts to take an item from a container. – success: The character succeeds in taking the item – failure: The action fails if the container is locked, the character is caught stealing or if the container does not contain the item – feedback: emotional feedback is on the joy axis and is directly proportional to the acquired object’s value scaled by the character’s current wealth • Lock The character attempts to lock or unlock a container (fails or succeeds based on the character’s skills; no explicit feedback ). • Equip The character equips an item (fails if the character is unable to equip the item, or not in possession of the item; no explicit feedback ). • Sneak The character enters or exits sneak mode (in sneak mode a character’s is less likely to be detected and able to steal items using the take action, however, its speed is halved; no explicit feedback ). The Sneak, Equip and Lock actions do not receive feedback for being executed; however, all feedback is distributed to previous actions based on a geometric progression, so that if, for example, a character would unlock a container and then receive a valuable item, the joy felt from the item would also influence the previous Lock action, making it pleasurable as well, thus acknowledging its contribution in the action sequence that led to acquiring the item. For example, let’s assume that an agent executes the following action sequence {Sneak, Move, Unlock, Take(item)}, and ends up picking up an item that increases it’s joy by [.32,0,0,0]. The first three action (Sneak, Move and Unlock ) do not receive any feedback for being done, however, they will receive a fraction of the feedback for the Take action, based on the assumption that the sequence led to it, and so, for example, Unlock will receive [.16,0,0,0], Move - [.8,0,0,0] and Sneak - [.4,0,0,0], to show the synergy existing between these actions. Emotional feedback for the Attack, Social and Move actions is generated according to the following formula, where success and direction ∈ [-1,1]: 44 f eedback = success · (direction · actionemotion + X paramemotion ) (4.3) params The success parameter is ignored when an agent is convinced the action is in the best interest of both parties involved (for example in the case of diplomacy), assuming that the proposing agent meant well, regardless of result (the result may still be skewed disfavourably by the backwards feedback distribution). Table 4.1 presents the base emotion force feedback used for actions available to agents in OrcWorld (column S denotes if the action was successful and D its directionality, either source or destination), while table 4.2 presents some examples of action instances, their computed feedback and its meaning (where the S column denotes the action’s succes (1 for success, -1 for failure), and the D column its directionality (1 for outgoing, -1 for incoming)). For example, if an agent uses the Intimidate action (that has feedback on the Trust and Fear Axes) on an agent that makes it feel proud, in order to scare it away and steal its treasure, and succeeds, the resulting emotion felt, according to the feedback equation, would be a combination of pride and dominance. The virtual character on the other end of the action, based on its feelings towards the acting agent would be despair, shame or remorse. Perception module A character in the virtual environment can perceive other characters, actions and environment objects through the perception module. A virtual character can have several senses that it uses to make sense of its surroundings. Fig. 4.2 shows a character that has two senses (virtual sensors): sight and hearing; the area each of them covers and how they overlap. The character gathers new knowledge through a process called multi-source information fusion. Fig. 4.3 presents the information fusion process, showing how multiple knowledge items (each from different sensors) representing the same virtual environment object are integrated into a consistent representation. In essence, when new information about an object is perceived, the knowledge item pertaining to it is updated. 4.2.3 Learning module Feedback from the environment has an associated emotion force vector that affects the agent’s state. The agent attempts to maximize its Gain (joy and truth axes) and minimize its Risk (fear and surprise axes), and thus will 45 Action Attack Dir Success Joy Trust − src src success failure dst dst success failure src src dst dst success failure success failure +0.1 -0.1 -0.1 +0.1 -0.1 +0.1 Diplomacy src src dst dst success failure success failure +0.1 -0.1 +0.1 -0.1 +0.1 -0.1 +0.1 -0.1 Intimidate src src dst dst success failure success failure Bluff Move 10 Surprise − damage −1 health 10 +0.1 +0.1 damage −1 health damage −1 health 10 10 -0.1 -0.1 -0.1 +0.1 +0.1 -0.1 -0.1 +0.1 -0.1 +0.1 +0.1 -0.1 as the area the agent is moving into +0.1 +0.1 Take Action Fear damage −1 health Dir value success + wealth value failure − wealth Success Joy Trust Fear Table 4.1: Action feedback table 46 Surprise Action Attack (0,0,-1,-1) Diplomacy (1,1,0,0) Intimidate (0,1,-1,0) Action S D Param Name Result Name 1 1 1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 (1,0,-1,0) (1,0,1,0) (1,0,0,1) (1,0,-1,0) (1,0,1,0) (1,0,0,1) (1,0,-1,0) (1,0,1,0) (1,0,0,1) (1,0,-1,0) (1,0,1,0) (1,0,0,1) pride guilt delight pride guilt delight pride guilt delight pride guilt delight 1 1 1 -1 -1 -1 -1 1 1 1 1 1 1 1 1 -1 -1 -1 -1 -1 (1,0,-1,0) pride (1,0,1,0) guilt (1,0,0,1) deligth (1,0,-1,0) pride (1,0,1,0) guilt (1,0,-1,0) pride (1,0,1,0) guilt (1,0,-1,0) pride (1,0,1,0) guilt (1,0,0,1) deligth (2,1,-1,0) (2,1,1,0) (2,1,0,1) (0,-1,-1,0) (0,-1,1,0) (0,-1,-1,0) (0,-1,1,0) (2,1,-1,0) (2,1,1,0) (2,1,0,1) love, pride love, guilt love, delight contempt shame contempt shame love, pride love, guilt love, delight 1 1 1 1 -1 -1 -1 -1 1 1 -1 -1 1 1 -1 -1 (1,0,-1,0) (1,0,1,0) (1,0,-1,0) (1,0,1,0) (1,0,-1,0) (1,0,1,0) (1,0,-1,0) (1,0,1,0) pride guilt pride guilt pride guilt pride guilt (1,1,-2,0) (1,1,0,0) (-1,-1,2,0) (-1,-1,0,0) (1,-1,0,0) (1,-1,2,0) (-1,1,0,0) (-1,1,0,0) pride, dominance love despair, shame remorse morbidness guilt, shame sentimentality envy, dominance S D Param Name Result Name (1,0,-2,-1) pride, aggression (1,0,0,-1) delight (1,0,-1,0) pride (1,0,0,1) delight (1,0,2,1) guilt, alarm (1,0,1,2) delight, alarm (-1,0,0,-1) pessimism (-1,0,-2,-1) envy, aggression (-1,0,-1,-2) pessimism, aggression (-1,0,2,1) despair, alarm (-1,0,0,1) disappointment (-1,0,1,0) despair Table 4.2: Action feedback generation example table 47 Figure 4.2: Character perception sensors choose actions that increase one while decreasing the other; however, the agent does not know what the feedback will be. It is the learning’s module task to attempt to predict it, based on the current context. The better the prediction, the better informed a decision the agent can make. We chose to use linear regression to predict feedback in our application. Linear regression is a way of modelling the relationship between a dependent variable and one or more explanatory variables. The data is modelled using a linear combination of the set of coefficients and explanatory variables. Linear regression is a supervised machine learning technique that estimates unknown model parameters from the data. In our case, the algorithm attempts to learn the feedback method explained. The dependent variable is the emotional force received by the agent, while the explanatory variable is a feature vector constructed based on the action’s context (Fig. 4.4). The structure of this array is made up of three segments: 1. Base Contains the basic information about the action. (a) Action The emotion force associated with the action (b) Action direction Denotes whether the action is incoming or outgoing (c) Area The emotion force associated with the area that the action takes place in 2. Agents Contains information about the agents involved in the action (a) Multiple Number of agents taking part in the action 48 Figure 4.3: Information fusion (b) Agents Average emotion force associated with the agents taking part in the action (c) Coherence Coherence of the emotion forces associated with the 1 ) agents taking part in the action (coherence = variance (d) Equipment Average emotion force associated with the equipment carried by the agents 3. Objects Contains information about the objects (other than equipment, i.e. containers, traps) involved in the action (a) Multiple Number of objects involved in the action (b) Objects Average emotion force associated with the agents taking part in the action (c) Locked Number of locked containers taking part in the action (d) Coherence Coherence of the emotion forces associated with the 1 agents taking part in the action (coherence = variance ) (e) Blocked Average emotion force for all agents blocking containers involved in the action The algorithm then receives the correct feedback from the environment which modifies the weights in the algorithm according to the difference between the estimated feedback force and the actual feedback force. Following Lazarus’ synthesis, emotions are integrated in the game mechanism as a 3-phase process, by which the agent interacts with the current 49 Figure 4.4: Context feature vector game context in terms of emotions. Every action is first estimated, by means of Machine Learning, to produce the value of emotional force it implies. At this point, no reference is made to his current state, so it represents a quantitative exploration of his action possibilities, and can be viewed as the Agent’s emotional knowledge. A qualitative evaluation follows, where the Agent determines the effect each action will have on his current state and chooses one accordingly. He sets priorities to the actions he is able to perform at the given time, therefore the outcomes of this phase can be viewed as emotional meaning of the actions in respect to his current state. Upon choosing an action he is confronted with the real force which determines his next state. This last phase has a normative role on the Agent’s performance, as the new state is also influenced by the quality of the estimation. 4.2.4 Behavior module For the behavior generation module, we have chosen to use Behavior Trees. Behavior Trees are a framework for game AI that model character behaviors extremely well, being simple and highly customizable at the same time. Behavior trees are one abstraction layer higher than Finite State Machines and combine the best features of linear scripts and reactive state machines while at the same time allowing purposeful behavior similar to planning techniques [12–14]. A behavior tree is a form of hierarchical logic, made up of behavior nodes. That means that the overall behavior is made up of a task that breaks down recursively into subtasks, which in turn would break down into even lower level tasks until it reaches the leaves of the tree. The leaves provide the interface between the artificial intelligence logic and the game engine. We only use the Decorator leaf node behavior, that checks a condition, and if this condition is true it attempts and action that makes changes in the environment. Complexity is built up in the branches of the tree. The branches are responsible for managing the child nodes. This is done using Composite tasks, which are tasks made up of multiple child tasks. We use four types of Composite tasks: Sequence, Selector, Loop and Concurrent. Fig. 4.6 shows the behavior tree management system hierarchy. 50 Figure 4.5: Action evaluation 51 Figure 4.6: Behavior tree Sequences are responsible for running their child tasks in sequence one by one. If one child fails, the whole sequence fails. Selector nodes run all child nodes until one either succeeds or returns a running state. Concurrent nodes visit all of their children during each traversal. Loop nodes behave as sequences, however they loop around when reaching their last child. To update a behavior tree, it’s always traversed beginning with its root node in depth first order. Algorithm 2 Behavior if status == INVALID then Initialize() end if status = Update() if status != RUNNING then Terminate(status) end if return status Example Let’s assume we have the following behavior tree for OrcWorld: 1. Root (Selector ) (a) Guard Treasure (Decorator ) i. Is enemy near treasure? (Condition) 52 Algorithm 3 Decorator if Condition() then return Action() else return FAILURE end if ii. Intimidate enemy! (Action) (b) Pillage Treasure (Sequence) i. ii. iii. iv. Know any treasure? (Condition) Go to treasure (Action) Intimidate guard (Action) Steal treasure (Action) (c) Idle (Decorator ) All nodes start out as ready. At the first tree update, the ready Root node is run. The Selector node checks all children from first to last until a child node returns a success or running state. It fails if no such child can be found. The Guard Treasure node is run, which is a Decorator made up of a condition and an action. The Is enemy near trasure? condition fails because there is no enemy visible and thus the whole Guard Treasure node fails. The next node to be run is the Pillage Treasure node. The sequence node starts by running its Know any treasure? node. A target trasure is chosen successfully and the node returns success. The next node in the sequence is the Go to treasure node. As the treasure is far, it can’t be reached in this step, so the node returns running. Its parent, Pillage treasure also returns running. Algorithm 4 Selector while true do status = current.Tick() if status != FAILURE then return status end if if Current.MoveNext() == false then return FAILURE end if end while 53 For the next iteration, all nodes that are not currently running are set to ready. The Guard treasure node is visited and fails again, however, when the Pillage treasure node is run again, it picks up in the Go to treasure node, where it left off in the sequence. In OrcWorld, we use a behavior tree for each character in order to generate all possible actions, which are then added to an action module, where they ratio and added to an are appraised and prunned according to their Gain Risk execution queue. Since the nature of the game is turn-based, all actions are run until completion. In order to select an appropriate action, the agent computes the resulting state of the Action Force on his Initial State and compares the Impact of all actions. He is then given feedback for the action he selected, representing the real force of the action. Confronting this force to what he estimated via Machine Learning results in a Regret Force which is then applied to the real Force in order to compute the Current State of the Agent (Fig. 4.7). Algorithm 5 Sequence while true do status = Current.Tick() if status != SUCCESS then return status end if if Current.MoveNext() == false then return SUCCESS end if end while 54 Figure 4.7: Decision flow 55 Chapter 5 Conclusion and future work This section presents our conclusions, a summary of the contributions and interesting future research directions. 5.1 Conclusion The main goal of this work has been to develop an emotion simulation model and agent architecture that provides artificial characters in virtual environments with believable behavior in order to enhance virtual environment experiences for human users; this means that emotions should act as motivation for agent behavior, influence and improve perception, reasoning and decisionmaking. The main objective of this project has been the proposal of an original architectural solution emotion representation and virtual character framework that possesses the following characteristics • easily expandable; independent of the AI techniques used to endow the virtual characters with behavior • simple, few algorithm parameters, easy to learn, use and implement • accomplishes all effects of emotion on behavior according to theory, use emotions as motivation • easily integrated with a game engine We first put forth the Emotion-Driven Character Reasoning Engine, a Belief-Desire-Intension inspired architecture that replicates our understanding of the functioning of the human mind at an abstract level. We use Plutchik’s emotion taxonomy and Lazarus’ appraisal model to endow virtual 56 characters with an emotional layer. The architecture gives an in-depth specification of how emotions interact with and influence other cognitive processes in the BDI architecture, such as perception, memory management, fact recall and rule application. The main contributions of this model are that emotion is not only tied to the agent’s learning and reasoning process, but it is a key component, providing agents with the underlayer for the simulated emotion, making the reasoning process more akin to humans, achieving complex, believable behavior for virtual agents. Another important contribution of this model is the improvement of agent performance and effectiveness through emotion simulation by speeding up relevant rule and fact recall, as it is currently believed emotions do in humans. Further on, we refined the emotion representation scheme used in the Emotion-Driven Chracter Reasoning Engine into the Newtonian Emotion System, a representation model vectorial in nature, easily integrated into computational environments and firmly grounded in Pluthcik’s theory of emotion. We have also developed laws that dictate how emotions interact with each other and external emotional forces. This representation scheme is easily understood by technical minded people and should pose no problems during implementation or authoring, helping with its adoption by developers and designers. The NES is fast and light-weight and we have also shown that it is very flexible, being used to represent emotional states, moods and personality. The model is also expandable, allowing for example to create a more complex personality description through several personality filters, each applied in certain cases. Our NES is fast and lightweight, without sacrificing any expressitivity. The artificial intelligence technique used needs to match the agent’s, game’s and environment’s capabilities. This is why we took a step back from the BDI paradigm, as not all applications require the same complex and in-depth reasoning process, and developed the Plug-and-Play architecture as a generic means of providing an emotion layer to any virtual character architecture, as an interface between the character’s behavior layer and the environment. The system still functions according to psychological theory, influencing the way that a character perceives the environment. The system does not influence reasoning, however it does influence the agent’s actions through the motivation mechanism. A generic interface architecture frees up dvelopers and designers to create the rest of the architecture as simple or as complex as necessary. The behavior generation module may be as simple as a finite state machine or as complicated as the belief desire architecture, with everything from decision trees to behavior trees in between. The appraisal module is again left to the developer, the only requierement being that it attepmt to predict and action’s feedback or evaluate a percept correctly based 57 on experience. We have thought of several machine learning techniques that can be used, depending on the constraints imposed, such as linear regression, support vector machines and clustering. By using the Plug-and-Play interface proposed in this thesis, the game logic can be kept as simple as possible, as there is no need to adapt the game engine architecture to accomodate computationally expensive artificial intelligence techniques. To demonstrate the system, we developed a computer game artificial intelligence framework using the Unity 3d Game Engine. The framework provides the underlying mechanisms for a character’s existence and interaction with a game world, including a behavior module. To ease behavior management and generation, we implemented a behavior tree library in c# to be used with Unity. we provide and example of how to integrate the Newtonian Emotion System and the Plug-and-Play subsystem with the mentioned framework in a computer role-playing game. We chose computer role-playing games as an application domain due to personal preference, however, the system can be used in any type of application that requires emotion simulation. Moreover, other applications may not have the processing power restrictions that running along side CPU intensive special effects and physics simulations impose, and would leave more compuring power for the artificial intelligence module. We have provided virtual characters with the means of artifical expression and methods for its integration into intelligent artificial agent architectures. Both systems are firmly grounded in theory, respecting PLutchik’s taxonomy, Lazarus’ appraisal model, replicating the way emotions influence human perception and motivating a character’s actions in the environment. We have developed a game artificial intelligence framework and demonstrated how easily the Plug-and-play architecture is integrated with the AI framework as well as the game engine used. We have also succeeded in keeping the system complexity down in order to make the system more accessible and easy to use and adopt. Last, but not least, the Plug-and-Play system is scalable (as the system only intervenes within the agent’s context, the overhead is linear) and both the New Newtonian Emotion System and the Plug-and-Play interface are easily expandable. 5.2 Contribution The main contributions of this thesis are as follows: • We have conceived a model of emotional BDI (Belief-Desire-Intention) agents to be used as the basis for the construction of believable virtual characters. 58 • We have conceived and developed the emotion-driven virtual character BDI reasoning engine that replicates the way I understand and believe human cognition works and is influenced by emotions. • We have proposed the Newtonian Emotion model and the associated Newtonian Emotion System, an emotion representation scheme easily integrated and used in a computational environment, firmly grounded in Plutchik’s theory of emotion. • We have conceived and implemented an artificial emotion appraisal process grounded in the Lazarus’ leading theory in the field that was used in tailoring the behavior of believable virtual characters. • We have designed and implemented a Plug-and-play architecture for the integration of affective computing into intelligent artificial agent systems. • We have proposed a framework for the integration of intelligent believable behaviour into computer games (specifically computer role-playing games). • We have conceived and implemented a behavior tree framework that manages the execution of behavior generation nodes for virtual characters. • We have developed an action module as part of a demo, that resolves a virtual character’s actions and their consequences and feedback. • We have conceived and validated a model in which emotions act as motivation for virtual character actions and in which emotions are used as heuristics in order to speed up event evaluation and decision-making. • We have developed an emotion simulation module where a character’s personality and past experience (through machine learning) influence its reasoning and decision-making. • We have designed and developed a generic, domain-independent artificial emotion simulation system, which is simple, easy to learn, adopt and integrate. • We have proposed a set of methods for the graphical representation of artificially generated emotions. 59 • We have investigated several general theories of emotions and computational theories of affective reasoning, we have compared and discussed their merits and drawbacks. Publications • V. Lungu and A. Baltoiu. Using Emotion as Motivation in the Newtonian Emotion System for Intelligent Virtual Characters. In Buletin Universitatea Politehnica Bucuresti, Bucuresti, submitted 2012. • V. Lungu. Newtonian Emotion System. In Proceedings of the 6th International Symposium on Intelligent Distributed Computing - IDC 2012, Springer Series on Intelligent Distributed Computing, vol. VI, pages 307-315, Calabria, Italy, 2012. • V. Lungu. Artificial Emotion Simulation Model and Agent Architecture. In Advances in Intelligent Control Systems and Computer Science, Springer Series in Advances in Intelligent Systems and Computing, vol. 187, pages 207-221, 2013. • V. Lungu and A. Sofron. Using Particle Swarm Optimization for Particle Systems. In The 18th International Conference on Control Systems and Computer Science, vol. 2, pages 750-754, Bucharest, Romania, 2011. • V. Lungu. Artificial Emotion Simulation Model. In Agents for Complex Systems Workshop (ACSYS 2010) held in conjuction with 7th Workshop on Agents for Complex Systems - SYNASC 2010, Timisoara, Romania, 2010. • V. Lungu. Rule-based System for Emotional Decision-Making Agents. In Distributed Systems Conference, Suceava, Romania, 2009. 5.3 Future work There are several improvements that can be made to the model in order to make it more flexible, and better able to suit the motivation needs for virtual characters, as well as interesting new research directions that lead to new application domains for the model. The main issue pointed out about the model is its inability to scale incoming emotions according to action class. For example, greed is considered a driving force for humans, however, greed is not considered an emotion within 60 Plutchik’s taxonomy. This can be achieved by rewarding a character acquiring wealth o power through Joy emotion (which is a driving force, that the character attempts to maximize, within our model). However, in its current state, the model is unable to create a very greed driven character. One solution would be to add an amplifier filter (similar to the personality filter) that only applies in certain cases (for a certain class of actions, in our example, those that have as a result increases in wealth and/or power; in our scenario this would be the Take action). We would also like to mention an implementation issue here, as we feel it is important, namely that the method we used in our scenario to provide emotional feedback gives out inconclusive results in some border cases (namely when the feedback is distributed evenly accross the four axes). Fortunately, these are rare (only when the feedback is distributed evenly accross the four axes), however we feel a more refined feedback generation method could solve this issue. Another interesting issue that has been pointed out in discussion, that is not currently being handled by the model, is the cold start problem, how the system behaves when there is no data available upon initialization. Currently, the system can generate data sets randomly and label them using the formula. Agents that do not do this, are considered blank slate newborns, and will only learn from experience (we believe that having your world views skewed by personal experience is a boon for human behavior simulation). As mentioned before, there are several application domains that the model can easily expand to. The first among these would be as a trust and reputation model. Emotions play such a role in humans. The process is called reciprocal altruism, an emergent norm in human systems that states that one person will help another, seemingly without any possibility of reciprocation, in the hopes that the initial actor will be helped themselves at a later date - in artificial intelligence terms, this means that an agent would act in a manner that temporarily reduces its fitness while increasing another agent’s fitness, with the expectation that the other agent will act in a similar manner at a later time. Emotions serve as an evaluator to help spot cheaters that try to abuse the system. We could use the Trust-Disgust axis in order to quantify how a given agent performs when a contract has been agreed upon, and use the Surprise-Anticipation axis in order to evaluate how an agent conforms to emergent but non-contractual norms. In order for the system to be useful in spotting cheaters, we would also need to achieve a level of information fusion (integrating emotion information about an agent in relation to norms from multiple sources - i.e. all other agents that have come into contact); this can be achieved in a centralized or distributed manner, depending on goals. Another possible extension of the model would be in the field of multi 61 agent systems, endowing them with collective emotions, where based on the law of universal attraction, agents would be able to influence each other’s states through proximity. This would work to give an overall state of the system. For example, if one or a few agents are damaged or not functioning properly, it would go unnoticed, however, if there is something wrong with the network, the overall emotion displayed would be poor. This could be used in self-organizing systems (such as ambient intelligence applications) in order to give a non-invasive diagnosis of the system (both for the system overall, as well as for individual devices); different emotions could be used to make users aware of different classes of problems. The same system could be used by artificial agents part of the same network to signal states among themselves. It could also serve as an indicator of how devices interact and influence each other (if emotion influence happens during direct interaction, and not through the dissemination proposed earlier). Yet another application is the integration of the model, together with a natural language processing unit capable of extracting emotion from text, into social networks. This would serve to analize how the mood of a message influences a person’s emotional state and how emotional states diffuse themselves in human networks, granting new insight into how emotions work. The model could also be used in the psychological domain, for example allow medical professionals to play out different scenarios for patients in a virtual reality environment in order to study their reaction and prescribe treatment, as well as enable them to create scenarios that teach people how to express their emotions by means of example. As emotions have been shown to establish the importance of events and details around us, and we have shown that the same can be applied to artificial systems, the Newtonian Emotion System could be used in time critical information fusion applications, in order to establish information integration priority (or belief maintenance in artificial agents). It has been show that mood is contagious in humans, so a large number of applications will probably want to take advantage of this, and attempt to influence users in order to get them into a more agreeable mood or ensure that they are happy or enjoying themselves using a certain product or service. The technology could further be integrated with ambient intelligence applications to influence people’s mood in order to make them more productive/happier/more accepting. Coupled with an emotion reading system, the model could serve as an indicator for the state of an area (i.e. mental health or quality of life), as well as give an idea on how neighboring areas interact and influence each other. 62 Appendix A Plutchik emotion model details This chapter presentes two figures that give some details about Pluthik’s emotion model (the model that the Newtonian Emotion System presented in this thesis is based off of). Dyads In our emotion representation model, emotions are created as a mix of the eight basic emotions. Each pair of two (dominant) emotions gives out a new emotion. These pairs are called dyads. Figure A.11 presents the names given to dyads in Plutchik’s emotion model: first, second and third-degree dyads are shown. 1 Image dyads.jpg source: http://danspira.files.wordpress.com/2010/12/plutchik-emotional- 63 Figure A.1: Plutchik extended dyads 64 Appraisal Figure A.22 presents some examples of event appraisals within the Plutchik emotion theory. The chart shows what the triggering event is (stimulus event), what the significance of the event is to the subject (cognitive appraisal), the emotion felt (subjective reaction), what the subject’s ensuing behavior should be (behavioral reaction) and the function it serves (function). Figure A.2: Plutchik appraisal examples 2 Image source: http://thumbnails.visually.netdna-cdn.com/robert-plutchiks-psychoevolutionary-theory-of-basic-emotions 50290c02eeb7e w587.jpg 65 Bibliography [1] Soar: Along the frontiers. Online, available at www.soartech.com, 2002. Soar technology. [2] Fahrenheit video game. http://en.wikipedia.org/wiki/ 2009. Online, available at October Fahrenheit (video game), [3] Itspoke: An intelligent tutoring spoken dialogue system. Online, available at http://www.cs.pitt.edu/ litman/itspoke.html, October 2009. [4] The sims video game. Online, available at http://thesims.ea.com, October 2009. [5] M. Ben Ari. Principles of concurrent and distributed programming. Prentice Hall, N.Y., 1990. [6] Ruth Aylett and Michael Luck. Applying artificial intelligence to virtual reality: Intelligent virtual environments. In Applied Artificial Intelligence 4, pages 3–32, 2000. [7] Christoph Bartneck. Integrating the occ model of emotions in embodied characters. Design, pages 1–5, 2002. [8] J. Bates. The role of emotion in believable agents. In Commun, pages 122–125. ACM 37, 1994. [9] Ronald Brachman and Hector Levesque. Knowledge Representation and Reasoning. Morgan Kaufman, 2004. [10] A. Burke, F. Heuer, and D. Reisberg. Remembering emotional events. Memory and Cognition, 20:277–290, 1992. [11] Jans W. Carton. The hypertext d20 system reference document. Online, 2012. available at http://d20srd.org. 66 [12] Alex J. Champandard. Behavior trees for next-gen game ai (video, part 1). Online, December 2007. http://aigamedev.com/open/article/behavior-trees-part1/. [13] Alex J. Champandard. Understanding behavior trees. Online, September 2007. http://aigamedev.com/open/article/bt-overview/. [14] Alex J. Champandard. Behavior trees for next-gen game ai. Online, December 2008. http://aigamedev.com/insider/presentations/behaviortrees/. [15] Alex J. Champandard. Trends and highlights in game ai for 2011. Online, February 2012. http://aigamedev.com/insider/discussion/2011trends-highlights. [16] Eric Chown, Randolph M. Jones, and Amy E. Henninger. An architecture for emotional decision-making agents. In Proceedings of the first international joint conference on Autonomous agents and multiagent systems, pages 352–353, Bologna, Italy, July 2002. [17] R. Cowie. Emotion oriented computing: State of the art and key challenges. Online, available at http://emotionresearch.net/projects/humaine/aboutHUMAINE/ HUMAINE%20white%20paper.pdf, October 2010. Humaine Network of Excellence. [18] A.D. Craig. Interoception: The sense of the physiological condition of the body. Current Opinion in Neurobiology, 13:500–505, 2003. [19] Michael David and Chen Sande. Serious games: Games that educate, train, and inform, 2006. [20] Sara de Freitas and Steve Jarvis. A framework for developing serious games to meet learner needs. In Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), 2006. [21] Sara de Freitas and Steve Jarvis. Serious games engaging training solutions: A research and development project for supporting training needs. British Journal of Educational Technology, 38(3):523–525, May 2007. [22] Ronald de Sousa. Emotion. Online, available at http://plato.stanford.edu/entries/emotion/, September 2010. Stanford Encyclopedia of Philosophy. 67 [23] A. S. Drigas, L. G. Koukianakis, and J. G. Glentzes. Knowledge engineering and a virtual lab for hellenic cultural heritage. In Proceedings of the 5th WSEAS Int. Conf. on Artificial Intelligence, 2006. [24] Oliver Duvel and Stefan Zerbst. 3D Game Engine Programming. Course Technology PTR, 2004. [25] David Eberly. The 3D Game Engine Architecture: Engineering Realtime Applications with Wild Magic. Morgan Kaufmann, 2005. [26] David Eberly. 3D Game Engine Design: A Practical Approach to RealTime Computer Graphics. Morgan Kaufmann, 2006. [27] Magy Seif El-Nasr, Thomas R. Ioerger, and John Yen. Peteei: a pet with evolving emotional intelligence. In Proceedings of the third annual conference on Autonomous Agents, pages 9–15, Seattle, Washington, United States, April 1999. [28] Wolfgang f. Engel. Shader X2 Introductions and Tutorials with DirectX 9. Wordware Publishing Inc., 2004. [29] Randima Fernando. GPU Gems: Programming Techniques, Tips and Tricks for Real-Time Graphics. Pearson Education, 2004. [30] Adina Magda Florea and Adrian George Boangiu. Bazele logice ale inteligentei artificiale. Universitatea Politehnica Bucuresti, 1994. [31] Patrick Gebhard. Alma - a layered model of affect. In 4th International Joint Conference of Autonomous Agents and Multi-Agent Systems (AAMAS 05), 2005. [32] M.P. Georgeff, B. Pell, M.E. Pollack, , M. Tambe, and M. Wooldridge. The belief-desire-intention model of agency. In Proceedings of the 5th international Workshop on intelligent Agents V, Agent theories, Architectures, and Languages, 1999. [33] Aron Granberg. A* pathfinding project. Online, 2012. available at http://www.arongranberg.com/unity/a-pathfinding. [34] Jason Gregory. Game Engine Architecture. A.K. Peters, 2009. [35] Kevin Gurney. An Introduction to Neural Networks. CRC Press, 1997. 68 [36] A.E. Henninger, R.M. Jones, and E. Chown. Behaviors that emerge from emotion and cognition: implementation and evaluation of a symbolicconnectionist architecture. In Proceedings of the Second international Joint Conference on Autonomous Agents and Multiagent Systems, pages 321–328, Melbourne, Australia, 2003. ACM, New York, NY. [37] Amy E. Henninger, Randolph M. Jones, and Eric Chown. Behaviors that emerge from emotion and cognition: A first evaluation. In Interservice/Industry Training Systems and Education Conference(I/ITSEC), 2002. [38] Obe Hostetter. Video games - the necessity of incorporating video games as part of constructivist learning. James Madison University, 2002. Department of Educational Technology. [39] E. Hudlicka. Affective computing for game design. In Proceedings of the 4th Intl. North American Conference on Intelligent Games and Simulation (GAMEON-NA), pages 5–12, 2008. [40] HUMAINE Network of Excellence. Humaine emotion annotation and representation language (earl). Online, 2009. available at http://emotion-research.net/projects/humaine/earl/index html/ ?searchterm=emotion%20markup%20language. [41] R. M. Jones. An introduction to cognitive architectures for modeling and simulation. In Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), 2004. [42] James Kennedy and Russell C. Eberhart. Swarm Intelligence. Morgan Kaufmann, 2001. [43] E.A. Kensinger. Remembering emotional experiences: The contribution of valence and arousal. Reviews in the Neurosciences, 15:241–251, 2004. [44] J.E. Laird, A. Newell, and P.S. Rosenbloom. Soar: An architecture for general intelligence. Artificial Intelligence, 33:1–64, 1987. [45] Pat Langley, John E. Laird, and Seth Rogers. Cognitive architectures: Research issues and challenges. Technical report, University of Michigan, 2002. [46] R.S. Lazarus. Emotion and Adaptation. Oxford University Press, New York, NY, 1991. 69 [47] Jill Fain Lehman, John Laird, and Paul Rosenbloom. A gentle introduction to soar, an architecture for human cognition. In S. Sternberg and D. Scarborough (Eds). MIT Press, 1996. Invitation to Cognitive Science. [48] Iolanda Leite, Carlos Martinho, André Pereira, and Ana Paiva. icat: an affective game buddy based on anticipatory mechanisms. In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems - Volume 3, AAMAS ’08, pages 1229–1232, Richland, SC, 2008. International Foundation for Autonomous Agents and Multiagent Systems. [49] Valentin Lungu. Rule-based system for emotional decision-making agents. In Sisteme distribuite ISSN 2067-5259, Suceava, 2009. online. [50] Valentin Lungu. Artificial emotion simulation model. In Agents for Complex Systems Workshop, Timisoara, 2010. SYNASC 2010. [51] Valentin Lungu. Newtonian emotion system. In Proceedings of the 6th International Symposium on Intelligent Distributed Computing - IDC 2012, volume VI, pages 307–315. Springer Series on Intelligent Distributed Computing, September 2012. [52] Valentin Lungu. Artificial emotion simulation model and agent architecture. Advances in Intelligent Control Systems and Computer Science, 187:207–221, 2013. [53] Valentin Lungu and Angela Şofron. Using particle swarm optimization to create particle systems. Proceedings of CSCS-18 18th International Conference on Control Systems and Computer Science, 2:750–754, May 2011. [54] Vincent Corruble Magalie Ochs, Nicolas Sabouret. Simulation of the dynamics of non-player characters’ emotions and social relations in games. IEEE Transactions on Computational Intelligence and AI in Games, 1(4):281–297, December 2009. [55] Xiao Mao and Zheng Li. Implementing emotion-based user-aware elearning. In CHI 2009, Spotlight on Works in Progress, 2009. [56] D. Moura and E. Oliveira. Fighting fire with agents: an agent coordination model for simulated firefighting. In Proceedings of the 2007 Spring Simulation Multiconference, pages 71–78, San Diego, CA, 2007. Society for Computer Simulation International. Volume 2. 70 [57] D. Nardi and R. Brachman. An introduction description logics. Online, 2010. available http://www.inf.unibz.it/ franconi/dl/course/dlhb/dlhb-01.pdf. to at [58] E. Oliveira and L. Sarmento. Emotional advantage for adaptability and autonomy. In Proceedings of the Second international Joint Conference on Autonomous Agents and Multiagent Systems, pages 305–312, Melbourne, Australia, 2003. ACM, New York, NY. [59] A. Ortony, G. Clore, and A. Collins. The cognitive structure of emotions. Cambridge University Press, Cambridge, 1988. [60] H. Van Dyke Parunak, R. Bisson, S. Brueckner, R. Matthews, and J. Sauter. A model of emotions for situated agents. In Proceedings of the Fifth international Joint Conference on Autonomous Agents and Multiagent Systems, pages 993–995, New York, NY, 2006. ACM. [61] André Pereira, Carlos Martinho, Iolanda Leite, and Ana Paiva. icat, the chess player: the influence of embodiment in the enjoyment of a game. In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems - Volume 3, AAMAS ’08, pages 1253– 1256, Richland, SC, 2008. International Foundation for Autonomous Agents and Multiagent Systems. [62] D. Pereira, E. Oliveira, and N. Moreira. Formal modelling of emotions in bdi agents. Computational Logic in Multi-Agent Systems: 8th international Workshop, CLIMA VIII, 2008. [63] David Pereira, Eugnio Oliveira, and Nelma Moreira. Modelling emotional bdi agents. In Workshop on Formal Approaches to Multi-Agent Systems (FAMAS 2006), 2006. [64] David Pereira, Eugnio Oliveira, Nelma Moreira, and Lus Sarmento. Towards an architecture for emotional bdi agents. In EPIA 05: Proceedings of 12th Portuguese Conference on Artificial Intelligence, 2005. [65] David Pereira, Eugnio Oliveira, Nelma Moreira, and Lus Sarmento. Towards an architecture for emotional bdi agents. In EPIA 05: Proceedings of 12th Portuguese Conference on Artificial Intelligence, pages 40–47. Springer, 2005. [66] Matt Pharr. GPU Gems 2: Programming Techniques for HighPerformance Graphics and General-Purpose Computation. Pearson Education, 2005. 71 [67] R. Plutchik. Emotion: Theory, research, and experience: Vol. 1. Theories of emotion. New York: Academic, New York, NY, 1990. [68] David Poole, Alain Mackworth, and Randy Goebel. Computational Intelligence: a Logical Approach. Oxford University Press, 1998. [69] Craig Reynolds. Steering behaviors for autonomous characters. Online, 1999. available at http://www.red3d.com/cwr/steer/. [70] J. Rickel and W.L. Johnson. Animated agents for procedural training in virtual reality: Perception, cognition, and motor control. In Applied Artificial Intelligence 13, pages 343–382, 1999. [71] Jeff Rickel and W. Lewis Johnson. Steve: An animated pedagogical agent for procedural training in virtual environments (extended abstract). SIGART Bulletin, 8:16–21, 1997. [72] Jeff Rickel and W. Lewis Johnson. Steve: An animated pedagogical agent for procedural training in virtual environments (extended abstract). SIGART Bulletin, 8:16–21, 1997. [73] Stuart J. Russell and Peter Norvig. Artificial Intelligence – A Modern Approach. Prentice Hall, Englewood Cliffs, New Jersey 07632, 1995. [74] S. Schachter and J. Singer. Cognitive, social, and physiological determinants of emotional state. Psychological Review, 69:379–399, 1962. [75] K. Scherer. What are emotions and how can they be measured? Social Science Information, 44(4):695–729, 2005. [76] Silvia Schiaffino, Analı́a Amandi, Isabela Gasparini, and Marcelo S. Pimenta. Personalization in e-learning: the adaptive system vs. the intelligent agent approaches. In Proceedings of the VIII Brazilian Symposium on Human Factors in Computing Systems, IHC ’08, pages 186– 195, Porto Alegre, Brazil, Brazil, 2008. Sociedade Brasileira de Computação. [77] M. Schrder, H. Pirker, and M. Lamolle. First suggestions for an emotion annotation and representation language. In Proceedings of LREC’06 Workshop on Corpora for Research on Emotion and Affect, pages 88– 92, Genoa, Italy, 2006. [78] M. Schroder, L. Devillers, K. Karpouzis, J. Martin, C. Pelachaud, C. Peter, H. Pirker, J. Schuller, J. Tao, and I. Wilson. What should a generic 72 emotion markup language be able to represent? In Second International Conference on Affective Computing andIntelligent Interaction: Lecture Notes in Computer Science. Springer, 2007. [79] T. Sharot and E.A. Phelps. How arousal modulates memory: Disentangling the effects of attention and retention. Cognitive, Affective and Behavioral Neuroscience, 4:294–306, 2004. [80] M.P. Singh. Agent communication languages: Rethinking the principles. Computer 31, 12:40–47, 1998. [81] Bas R. Steunebrink, Mehdi Dastani, and John-Jules Ch. Meyer. Towards a quantitative model of emotions for intelligent agents. [82] Bas R. Steunebrink, Mehdi Dastani, and John-Jules Ch. Meyer. A logic of emotions for intelligent agents. In Proceedings of the 22nd national conference on Artificial intelligence - Volume 1, pages 142–147. AAAI Press, 2007. [83] Bas R. Steunebrink, Mehdi Dastani, and John-Jules Ch. Meyer. The occ model revisited. In Proceedings of the 4th Workshop on Emotion and Computing – Current Research and Future Impact, 2009. [84] Bas R. Steunebrink, Mehdi Dastani, and John-Jules Ch. Meyer. The occ model revisited. In Proceedings of the 4th Workshop on Emotion and Computing, 2009. [85] B.R. Steunebrink, M.M. Dastani, and J-J.Ch. Meyer. Emotions as heuristics in multi-agent systems. In Proceedings of the 1st Workshop on Emotion and Computing - Current Research and Future Impact, pages 15–18, 2006. [86] Martin Strauss and Michael Kipp. ERIC : A Generic Rule-based Framework for an Affective Embodied Commentary Agent. pages 97–104, 2008. [87] Arges Systems. Unitysteer - steering components for unity. Online, 2000. [88] Unity Technologies. Unity 3d game engine reference manual. Online, 2012. available at http://docs.unity3d.com/Documentation/Components/index.html. [89] Gerard Tel. Introduction to Distributed Algorithms. Cambridge University Press, 1994. 73 [90] W3C Incubator Group. Elements of an emotionml 1.0. Online, 2008. available at http://www.w3.org/2005/Incubator/emotion/XGRemotionml/. [91] Michael Wooldridge and Nicholas R. Jennings. Pitfalls of agent-oriented development. In Proceedings of the Second International Conference on Autonomous Agents, pages 385 – 391. ACM Press, 1998. [92] Nick Yee. The labor of fun: How video games blur the boundaries of work and play. Games and Culture, 1:68–71, 2006. 74