Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
The Display of Emotion Brent Lance CS 543 Lecture 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 1 Overview Motivation Psychological View of Nonverbal Behavior Encoding Vs. Decoding Nonverbal Behavior Expression of Emotion through Nonverbal Behavior Artistic View of Emotional Expression Realizing Emotional Expression 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 2 Motivation What is interesting about the display of emotion? The development of Virtual characters Believable Capable of open-ended interaction Engaging For this, a character must be able to signal its internal state through its external behavior 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 3 Virtual Characters Keyframe Animation Classic technique from 2D cell animation Skilled animator develops key frames Interpolation between them creates movement Slow, iterative process Produces most realistic behavior Takes very long time to develop Animations are not reusable 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 4 KeyFrame Animation 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 5 Motion Capture Motion Capture Animation Cameras record movement of sensors placed on an actor Software reconstructs movement and approximation of character’s anatomy Allows for quicker generation of movement Requires expensive hardware and analysis tools Captured sequences not reusable Can be difficult to retarget capture motion to 3D model being animated 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 6 Motion Capture 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 7 Procedural Animation Generation of movement without human oversight Virtual Agents Video Game Characters Can use libraries of keyframe or motion capture animation Allow for dynamic or interactive environments Do not have the quality of previous two methods 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 8 Procedural Animation 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 9 Motivation - Revisited Hopefully, knowledge of how emotion is expressed through nonverbal behavior can improve interactive virtual agents to the point where they are as engaging as traditionally animated characters 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 10 Overview Motivation Psychological View of Nonverbal Behavior Encoding Vs. Decoding Nonverbal Behavior Expression of Emotion through Nonverbal Behavior Artistic View of Emotional Expression Realizing Emotional Expression 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 11 Nonverbal Behavior Encoding The display of information through external behavior Decoding The comprehension of information through observation of external behavior 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 12 Encoding Types of encoding [Ekman & Friesen, 1969] define three types of coding Arbitrary No relation between coding and meaning Iconic Coding is representation of meaning Intrinsic Coding is meaning 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 13 Complications Encoding may not be deliberate Encoding may be used to mask actual information Encoding may be idiosyncratic Men and women encode & decode differently Different cultures have different coding rules 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 14 Realistic vs. Believable Decoding research is easier than Encoding research Generating Believable expressive behavior is easier than generating Realistic expressive behavior “Realistic” -> behavior as a human would have performed it “Believable” -> behavior understandable by an observer 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 15 Overview Motivation Psychological View of Nonverbal Behavior Encoding Vs. Decoding Nonverbal Behavior Expression of Emotion through Nonverbal Behavior Artistic View of Emotional Expression Realizing Emotional Expression 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 16 Nonverbal Communication Technically, communication without words Extremely broad field [Knapp & Hall, 1997] Includes: Environment Appearance Dress, grooming Height, weight Race & gender None of these really express emotion However, they may affect appraisal ratings 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 17 Overview Motivation Psychological View of Nonverbal Behavior Encoding Vs. Decoding Nonverbal Behavior Expression of Emotion through Nonverbal Behavior Artistic View of Emotional Expression Realizing Emotional Expression 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 18 Expression of Emotion What nonverbal behaviors display emotion? Torso Posture Head Position Facial Expression Hand Gestures Gaze Behavior Movement Expression of Emotion in Man and Animals [Darwin, 1872]. 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 19 Posture - Demonstration 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 20 Posture - Demonstration 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 21 Posture - Demonstration 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 22 Torso Posture Torso Posture is affected less by emphasis and other speech-related behaviors [Bull & Connelly, 1985] Positive and Negative emotions are both easily distinguished both from static body postures [Coulson, 2004], [Schouwstra & Hoogstraten, 1995], and from dynamic trunk movement [de Meijer, 1989]. 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 23 Expression of Emotion What nonverbal behaviors display emotion? Torso Posture Head Position Facial Expression Hand Gestures Gaze Behavior Movement 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 24 Head Position 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 25 Head Position Movement of the head is tightly connected to speech [Hadar et al., 1985], [McClave, 1999]. Static posture of the head is still useful for the display of emotions Head which is tilted upwards is viewed as more dominant, and displaying a more positive than one tilted downwards [Kappas et al., 1994], [Mignault & Chaudhuri, 2003]. 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 26 Expression of Emotion What nonverbal behaviors display emotion? Torso Posture Head Position Facial Expression Hand Gestures Gaze Behavior Movement 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 27 Facial Expression Facial expression is one of the stronger channels for emotional expression Paul Ekman Performed many of early experiments on facial expression Found that facial expression was highly cross cultural Developed the Facial Action Coding System (FACS) 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 28 Happiness 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 29 Anger 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 30 Sadness 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 31 Fear 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 32 Disgust 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 33 Facial Expression Other researchers have shown that Children who are Deaf/Blind from birth share many of the same facial expressions As do very young infants As well as monkeys [Knapp & Hall, 1997] 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 34 Expression of Emotion What nonverbal behaviors display emotion? Torso Posture Head Position Facial Expression Hand Gestures Gaze Behavior Movement 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 35 Hand Gestures Much more research has been done on the association of hand gestures with the communication of concepts However, research has demonstrated that level of arousal, and to a lesser extent the valence of emotion can be determined through arm movement [Pollick et al., 2001] 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 36 Expression of Emotion What nonverbal behaviors display emotion? Torso Posture Head Position Facial Expression Hand Gestures Gaze Behavior Movement 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 37 Gaze Behavior What do we mean by gaze behaviors? Eye shape and eyebrow position are part of facial expression Length of gaze & mutual gaze Gaze attraction vs. aversion 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 38 Non-Emotional Gaze Gaze is closely linked to speech-related behaviors Patterns of gaze for individuals speaking are often different from patterns of gaze for listeners [Kleinke, 1986]. Attention and Saliency also control eye movement [Argyle & Cook, 1973]. 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 39 Gaze and Dominance Gaze is very important in signaling dominance [Exline, 1971] However this relationship is not fully understood Seen in the way monkeys perform dominance displays when looked at by experimenters Highly dominant individuals look less while listening, and more while speaking than do individuals of low dominance [Knapp, 1997]. 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 40 Gaze and Arousal Gaze is also closely related to physiological arousal [Argyle & Cook, 1973] Receiving gaze and engaging in mutual gaze increases arousal Gaze is used as a regulation for physiological arousal Increased blinks, and increased pupil dilation are also strong signals of arousal 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 41 Gaze and Preference Gaze also has a drastic effect on liking Individuals who gaze very little at people they interact with are rated lower on scales of preference, credibility, attractiveness, and are less likely to be hired [Burgoon et al., 1985], [Argyle & Cook, 1973]. Individuals are also more likely to look at people that they like [Argyle & Cook, 1973] 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 42 Expression of Emotion What nonverbal behaviors display emotion? Torso Posture Head Position Facial Expression Hand Gestures Gaze Behavior Movement 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 43 Movement & Locomotion Emotion can be recognized through how people walk But further, emotion can be recognized through the movement of simple geometric shapes [Rime et al., 1985] 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 44 Movement 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 45 Locomotion 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 46 Synchrony The interactions of all of these behaviors is as important as any single behavior If not, emotional signals lack coherency The ability of users to recognize emotion decreases Believability and engagement decrease 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 47 Conclusion – Nonverbal While there are many different ways emotion can be signaled through nonverbal behavior Only facial expression (and possibly prosody) can clearly differentiate between individual emotional categories Much nonverbal behavior research done using dimensional models of emotion that can be applied to factor analyses of nonverbal behavior results 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 48 Overview Motivation Psychological View of Nonverbal Behavior Encoding Vs. Decoding Nonverbal Behavior Expression of Emotion through Nonverbal Behavior Artistic View of Emotional Expression Realizing Emotional Expression 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 49 Art & Emotion Looking at three artistic views of emotional expression Delsarte’s Structural Acting System Laban Movement Analysis, a system derived from choreography Emotion in 2D Disney Animation 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 50 Structural Acting System Francois Delsarte (1811-1871) French opera singer who damaged his voice Turned to the expression of emotion through acting Compiled extremely extensive library of movements, and the meanings of those movements However, this library has not been empirically verified 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 51 Delsarte Recently, [Marsella et al., 2006] performed an evaluation of some of Delsarte’s work on hand gestures “Delsarte’s cube” Small imaginary cube directly in front of chest Placement of hands on cube sends different signals i.e. hands placed on inside surface of near face of cube to reveal possession 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 52 Laban Movement Analysis Rudolph Laban (1879-1958) Hungarian Dancer and Choreographer Developed Labanotation, a notation for recording dance Rendered mostly obsolete through widespread distribution of video recording technology Developed, in conjunction with several of his students, Laban Movement Analysis, a taxonomy for describing movement 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 53 Laban Movement Analysis Qualitative, hierarchical taxonomy Effort Space Indirect/Direct Time Sustained/Sudden Weight Light/Strong Flow Free/Bound Shape 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 54 EMOTE [Chi et al., 2000] used Laban parameters Effort and Shape to animate emotion Theory was that instead of specifying emotional state, would instead specify LMA parameters to animate a character Extended by [Zhao & Badler, 2005] which learned LMA parameters from motion capture or video recording 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 55 Disney Animation Principles of Disney animation laid out in [Thomas & Johnston, 1981], extended to 3D animation in [Lasseter, 1994]. 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 56 Keyframe Animation Skilled animators drew key frames 2nd Tier drew the “inbetweens” Developed principles such as “Squash & Strech”, “Anticipation”, and “Follow Through” 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 57 2-D Animation Extremely slow process Highly iterative Highly intuitive Very difficult to clearly explain Requires great deal of training, and a certain amount of talent Results are excellent 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 58 Analyzing 2D Animation [Lance et al., 2004], and [Chafai et al., 2006] were attempts to analyze 2D animation for information on nonverbal behavior for gaze and gesture, respectively Moderately successful Extremely labor intensive 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 59 Overview Motivation Psychological View of Nonverbal Behavior Encoding Vs. Decoding Nonverbal Behavior Expression of Emotion through Nonverbal Behavior Artistic View of Emotional Expression Realizing Emotional Expression 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 60 Realizing Behavior 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 61 Animation Curves 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 62 Animation of Behavior Qualitative descriptions of nonverbal behavior need to be transformed into quantitative changes in animation Two Choices: Directly manipulate the animation curves yourself Optimization Develop and utilize an abstraction, such as EMOTE or the Body Markup Language (BML) 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 63 Conclusion Many sources of information for the expression of emotion through human behavior Lots of them are qualitative Or not experimentally verified Or describe more subtle relationships than direct emotional state -> behavior Picture is still very incomplete Still lots of good information for a starting point! 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 64 References Argyle, M., Cook, M. Gaze and Mutual Gaze. Cambridge University Press. 1976. Bull, P., Connelly, G. Body Movement and Emphasis in Speech. Journal of Nonverbal Behavior, 9(3). Fall 1985. Burgoon, J., Manusov, V., Mineo, P., Hale, J. Effects of Gaze on Hiring, Credibility, Attraction and Relational Message Interpretation. Journal of Nonverbal Behavior. 9(3). Fall 1985. Chafai, N., Pelachaud, C., Pele, D., Breton, G. Gesture Expressivity Modulations in an ECA Application. Proceedings of the Intelligent Virtual Agents IVA 2006 Conference. 2006. Coulson, M. Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence. Journal of Nonverbal Behavior. 28(2). Summer 2004. Chi, D., Costa, M., Zhao, L., Badler, N. The EMOTE Model for Effort and Shape. Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. ACM Press/Addison-Wesley Publishing, New York, NY. 2000. Darwin, C., The Expression of Emotions in Man and Animals. 1872. Ekman P., Friesen, W.V., The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding. Semiotica. 1, 49-98. 1969. Exline, R. Visual Interaction: The Glances of Power and Preference. In Weitz, S. (ed), Nonverbal Communication: Readings with Commentary. Oxford University Press, 1974. Hadar, U., Steiner, T. J., Rose, F. C. Head Movement During Listening Turns in Conversation. Journal of Nonverbal Behavior, 9(4). Winter 1985. Kappas, A., Hess, U., Barr, C., Kleck, R. Angle of Regard: The Effect of Vertical Viewing Angle on the Perception of Facial Expressions. Journal of Nonverbal Behavior. 18(4). Winter 1994. Kleinke, C. Gaze and Eye Contact: A Research Review. Psychological Bulletin. v. 100, n. 1. 1986. 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 65 References (Cont.) Knapp, M., Hall, J. Nonverbal Communication in Human Interaction. Fourth Edition. Harcourt Brace Publishers. 1997. Lance, B., Marsella, S., Koizumi, D. Towards Expressive Gaze Manner in Embodied Virtual Agents. AAMAS Workshop on Empathic Agents. 2004. Lasseter, J. Principles of Traditional Animation Applied to 3D Computer Animation. Computer Graphics. v. 21, n 4. July 1987. Marsella, S., Carnicke, S. M., Gratch, J., Okhmatovskaia, A., Rizzo, A. An Exploration of Delsarte’s Structural Acting System. Proceedings of the Intelligent Virtual Agents IVA 2006 Conference. 2006. McClave, E. Linguistic Functions of Head Movements in the Context of Speech. Journal of Pragmatics. v. 32, 2000. de Meijer, M. The Contribution of General Features of Body Movement to the Attribution of Emotions. Journal of Nonverbal Behavior. 13(4). Winter 1989. Mignault, A., Chaudhuri, A. The Many Faces of a Neutral Face: Head Tilt and Perception of Dominance and Emotion. Journal of Nonverbal Behavior. 27(2). Summer 2003. Pollick, F., Paterson, H., Bruderlin, A., Sanford, A. Perceiving Affect from Arm Movement. Cognition. i 82, B51-B61. 2001. Rime, B., Boulanger, B., Laubin, P., Richir, M., Stroobantst, K. The Perception of Interpersonal Emotions Originated by Patterns of Movement. Motivation and Emotion. v. 9, n. 3. 1985. Schouwstra, S., Hoogstraten, J. Head Position and Spinal Position as Determinants of Perceived Emotional State. Perceptual and Motor Skills. v. 81, 1995. Thomas, F., Johnston, O. The Illusion of Life: Disney Animation. Walt Disney Productions. 1981. Zhao, L., Badler, N. Acquiring and Validating Motion Qualities from Live Limb Gestures. Graphical Models. v. 67 i. 1. January, 2005. 5/23/2017 7:32 PM Copyright 2005, USC/ISI. All rights reserved. 66