Download Involuntary Leakage in Deceptive Facial Expressions as a Function

J Nonverbal Behav (2012) 36:23–37
DOI 10.1007/s10919-011-0120-7
ORIGINAL PAPER
Secrets and Lies: Involuntary Leakage in Deceptive
Facial Expressions as a Function of Emotional Intensity
Stephen Porter • Leanne ten Brinke • Brendan Wallace
Published online: 9 October 2011
Springer Science+Business Media, LLC 2011
Abstract Darwin (1872) hypothesized that some facial muscle actions associated with
emotion cannot be consciously inhibited, particularly when the to-be concealed emotion is
strong. The present study investigated emotional ‘‘leakage’’ in deceptive facial expressions
as a function of emotional intensity. Participants viewed low or high intensity disgusting,
sad, frightening, and happy images, responding to each with a 5 s videotaped genuine or
deceptive expression. Each 1/30 s frame of the 1,711 expressions (256,650 frames in total)
was analyzed for the presence and duration of universal expressions. Results strongly
supported the inhibition hypothesis. In general, emotional leakage lasted longer in both the
upper and lower face during high-intensity masked, relative to low-intensity, masked
expressions. High intensity emotion was more difficult to conceal than low intensity
emotion during emotional neutralization, leading to a greater likelihood of emotional
leakage in the upper face. The greatest and least amount of emotional leakage occurred
during fearful and happiness expressions, respectively. Untrained observers were unable to
discriminate real and false expressions above the level of chance.
Keywords
Universal emotions Facial expression Deception Emotional intensity
Introduction
During his 1938 Munich meeting with Hitler, British Prime Minister Neville Chamberlain
scrutinized Hitler’s face as he swore that he would not invade Czechoslovakia. After the
meeting, Chamberlain infamously reported, ‘‘I got the impression that here was a man who
could be relied upon when he had given his word’’ (see Porter and ten Brinke 2010, Ekman
1985/2001). How could he have made such a cataclysmic error in misreading Hitler’s
deceptive face?
S. Porter (&) L. ten Brinke B. Wallace
Centre for the Advancement of Psychological Science and Law, Department of Psychology,
University of British Columbia, Kelowna, BC V1V 1V7, Canada
e-mail: [email protected]
123
24
J Nonverbal Behav (2012) 36:23–37
The complex musculature of the human face and its direct relation with affective
processes of the brain make it a rich canvas upon which humans communicate their
emotional states and from which we infer those of others. Accordingly, in daily life we
read the faces of intimates and strangers to make inferences about their emotions and
intentions, and adopt expressions ourselves to communicate sincerely or insincerely how
we are feeling. Accurately assessing the nature and veracity of others’ facial expressions is
a serious business in everyday life (e.g., Hess and Bourgeois 2010) and applied settings. In
the context of personal relationships, the presence of concealed but subtly communicated
negative emotional expressions is highly predictive of divorce (Gottman et al. 2001).
Further, the identification of falsified emotional information is crucial in many applied
contexts in society, including the courts, parole hearings, politics, and corporations.
The Supreme Court of Canada (R. v. B. (K. G.) 1993) concluded that judges and jurors
must view a witness to ‘‘adequately evaluate body language, facial expressions and other
indicators of credibility’’ and that credibility assessment is ‘‘common sense.’’ In a recent
landmark Canadian case R v N.S. (2010), the Ontario Court of Appeal—in deciding
whether to permit a Muslim complainant to wear her face-covering niqab during testimony—concluded that: ‘‘Covering the face of a witness may impede cross-examination in
two ways. First, it limits the trier of fact’s ability to assess the demeanour of the witness.
Demeanour is relevant to the assessment of the witness’s credibility and the reliability of
the evidence given by that witness. Second, witnesses do not respond to questions by words
alone. Non-verbal communication can provide the cross-examiner with valuable insights’’
(p. 54). In sentencing and parole hearings, the perceived credibility of an offender’s
emotional display/remorse informs decision-making (ten Brinke et al. 2011). Further,
travelers’ faces are scrutinized by airport security staff trained in detecting concealed
emotions and intentions (see Porter and ten Brinke 2010).
The foundation of this evaluative process originates in our evolutionary past; the basic
discrimination of friend and foe likely was one of the earliest interpersonal judgments to
evolve (e.g., Williams and Mattingley 2006). Sometimes emotion is ‘‘written all over the
face’’, a salient and powerful representation of one’s affective state (Matsumoto 2007;
Matsumoto and Willingham 2009). However, the evolutionary development of interpersonal deception—partially accomplished by altering or inhibiting an expression normally
accompanying the emotion—magnified the complexity of interpreting facial expressions.
There are three major ways in which emotional facial expressions are intentionally
manipulated (Ekman and Friesen 1975): an expression is simulated when it is not
accompanied by any real emotion, masked when the expression corresponding to the felt
emotion is replaced by a falsified expression corresponding to a different emotion, or
neutralized when the expression of a genuine felt emotion is inhibited while the face
remains neutral. These deceptive displays can be highly convincing; observers often are
unable to discriminate genuine versus faked expressions (Porter and ten Brinke 2008;
Porter et al. 2010), despite high self-reported confidence in their assessments (Vrij and
Mann 2001).
Thus, as with Chamberlain, mistakes in reading faces occur frequently and sometimes
with major consequences. In corporate settings, white-collar criminals such as Bernard
Madoff often find easy victims by appearing trustworthy, empathetic, and displaying
convincing emotional masks. Porter et al. (2009) found that psychopaths were 2.5 times as
likely as their counterparts to be granted parole in National Parole Board hearings, a
pattern the authors attributed in part to their convincing emotional performances. Indeed,
psychopaths seem particularly proficient at stifling emotional facial inconsistencies (Porter
et al. 2011).
123
J Nonverbal Behav (2012) 36:23–37
25
Although emotional performances often are thoroughly misinterpreted, emotional
deception may be expressed in subtle but reliable and perceptible ways that are sometimes
missed by untrained observers. This idea first was studied by Duchenne (1862/1990), who
examined the muscle actions of the smile. He noted that the common conceptualization of
a happiness expression is the contraction of the zygomatic major muscle, which upturns the
mouth corners. But when he electrically stimulated this muscle, the resulting smile did not
seem ‘‘genuine.’’ As it turned out, real happiness also involves the orbicularis oculi surrounding the eyes, which pull the cheek up while slightly lowering the brow (e.g., Ekman
et al. 1990).
Darwin (1872), of course, was interested in emotional expressions (Hess and Thibault
2009) and observed that, ‘‘A man when moderately angry, or even when enraged, may
command the movements of his body, but…those muscles of the face which are least
obedient to the will, will sometimes alone betray a slight and passing emotion’’ (p. 79). He
hypothesized that some facial muscle actions associated with strong emotion are beyond
voluntary control and cannot be completely inhibited. Further, he proposed that certain
facial muscles cannot be intentionally engaged during emotional simulation. Collectively,
these two propositions form the inhibition hypothesis (Ekman 2003a, 2009), a proposal
with major theoretical and applied implications but one that has hardly been examined. A
derivative idea long advocated by Ekman (Ekman 1985/2001; Ekman and O’Sullivan
2006; Haggard and Isaacs 1966) is that when an emotion is concealed or masked, the true
emotion is manifested as a micro-expression, a fleeting expression suppressed in 1/25–1/5
of a second making it difficult to perceive with the naked eye. This hypothesis has been
accepted somewhat non-critically in the scientific community and media (e.g., Henig 2006;
Lipton 2006). In fact, although the validity of these hypotheses, and more generally the
idea that the face involuntarily communicates covert emotional states, are widely assumed,
they have been subjected to relatively little direct empirical scrutiny (Ekman et al. 1988,
1991; Porter and ten Brinke 2008; Stewart et al. 2009).
One might argue that the face is largely uncharted terrain in terms of the richness of the
information it may communicate relative to the empirical attention it has received. Porter
and ten Brinke (2008) investigated the nature of facial expressions accompanying genuine
and false/concealed emotions. They found that the involuntary leakage of emotions was, in
a way, ubiquitous—no participant was able to falsify emotions without such betrayals on at
least one occasion. Emotional arousal associated with this form of deception also was
revealed by changes in blink rate; masking one’s true feelings with a false emotion led to
increases in blink rate while holding a neutral ‘‘poker face’’ in response to emotional
stimuli led to decreased blink rate, relative to genuine expressions. Despite the presence of
cues to deceit indicated by these analyses, naı̈ve judges could only discriminate genuine
versus deceptive expressions at a level slightly above chance (above chance for happiness
and disgust but at chance for sad and fearful expressions; see also Ekman et al. 1988 and
Frank et al. 1993 for discrimination of genuine vs. deceptive smiles). Similarly, Warren
et al. (2009)—utilizing Ekman and Friesen’s (1974) emotional fabrication paradigm—
found that performance for emotional lie detection was slightly but significantly above
chance (mean of 64.4%), while that for unemotional lie detection was significantly below
chance (36.1%). Indeed, this may have been related to the leakage of discordant emotions
during fabricated, emotional lies. Other lie scenario variables too seem to influence ability
to detect deceit; police officers are better at detecting high stakes lies, relative to low
(O’Sullivan et al. 2009). In general, however, while lie scenario or individual differences
may produce slight advantages over chance, deception detection accuracy remains sub-par
without empirically-based training.
123
26
J Nonverbal Behav (2012) 36:23–37
Darwin’s (1872) hypothesis stated only that strong emotions would be difficult to
conceal or fabricate. No research has examined directly the intensity of emotion required
for ‘‘leakage’’ to occur in facial communication. Does emotional leakage occur only when
concealing powerful emotions, or is ‘‘more leakage’’ exhibited during such emotions
relative to lower intensity emotions? We hypothesize that powerful emotional states will be
relatively unique in leading to leakage in involuntary facial expressions, similar to an
involuntary reflex of the knee requiring contact with a minimum level of force. Emotional
leakage during high intensity deception specifically may relate in part to a reduction in
attentional resources. Gable and Harmon-Jones (2010) found that low intensity emotion
increases the breadth of one’s attention whereas high intensity narrows one’s attentional
focus. It is possible that experiencing and trying to inhibit or mask powerful emotions—
relative to weaker emotional states—require so much competing cognitive, physiological,
and affective effort that one’s voluntary control is compromised or eliminated resulting in
leakage.
The Present Study
While the interpretation of emotional facial expressions has been complicated by the
evolutionary development of interpersonal deception, it appears that certain aspects of the
face cannot be controlled. As such, the face has the potential to reveal hidden emotions
with objective analysis. However, little research has been conducted that investigates the
inhibition hypothesis and the secrets of the human face. A key remaining question is does
emotional ‘‘leakage’’ occur only for powerful emotions, or is there ‘‘more leakage’’ in such
emotions versus lower intensity emotions? The present study was the first to directly test
the proposition that intensity has an impact on the presence and duration of emotional
leakage during simulated, masked, and neutralized expressions. We tested the hypothesis
that concealed felt emotions must be relatively potent in order for ‘‘leakage’’ to appear
involuntarily on the face, and that less powerful emotional states will be easily concealed
or masked (i.e., under voluntary control).
Specifically, we predicted that more inconsistent emotional expressions would appear in
deceptive versus honest expressions, and a veracity by intensity interaction such that
inconsistent emotion would be least prevalent in high intensity genuine expressions and
most prevalent in high intensity masked expressions (replacing one’s emotion with a
different opposing emotion) followed by low intensity masked expressions, high intensity
neutralized expressions, and low intensity simulated expressions. Further, it was expected
that neutralizing high intensity emotions (trying to appear as though one is feeling no
emotion) would lead to greater emotional leakage than neutralizing low intensity emotion.
Because emotions generally involve muscles in both the upper and lower face, but some
muscles may be less controllable than others, emotional expression in the upper (eyes and
forehead) and lower (nose and mouth) face were coded separately to allow for accurate
facial coding when, for example, the eyebrows lowered as in disgust, but the mouth leaked
genuine happiness (Rinn 1984).
We also examined observers’ accuracy at identifying high versus low intensity
emotional deception. Given our prediction that more leakage will occur in high intensity
emotional deception, it was hypothesized that observers would perform somewhat better
at detecting such deception (slightly better than chance as found in Porter and ten
Brinke 2008) and around the level of chance/guessing with low intensity emotional
deception.
123
J Nonverbal Behav (2012) 36:23–37
27
Method
Participants
The primary sample was comprised of N = 59 participants (19 male, 40 female;
M age = 20.26 years, SD = 3.62) attending a Western Canadian university. An additional
28 (21 female, 7 male; M age = 19.14 years, SD = 1.14) participants participated as
untrained observers of the facial expressions; they were asked to assess the veracity of the
participants’ emotional expressions in real-time and it was anticipated that their presence
during the emotional displays would increase the realism of the experiment and the
motivation of the primary participants.
Materials
The images used to evoke emotion in this study were chosen from the International
Affective Picture System (IAPS; Lang et al. 1999). These images have been normed on
emotional valence, arousal, and the discrete emotion type that they evoke in viewers
(Mikels et al. 2005); images were selected based on these ratings. Images normed by Mikels
et al. (2005) as primarily evoking sadness, disgust, fear, contentment, and amusement (low
and high happiness, respectively) were considered for use as stimuli (i.e., images evoking a
combination of emotions, like sadness and disgust, were removed from the pool of potential
stimuli). High and low intensity emotional images then were chosen based on IAPS
(Libkuman et al. 2007) arousal norms (rated on a 1–9 Likert scale). The images were
selected to evoke either high or low intensity happiness, fear, disgust, or sadness, and
neutral/no emotion.1 For example, images included a smiling baby in the high-intensity
happy condition to a severed hand as a high-intensity disgusting image. Analysis of these
images revealed significant differences in arousal ratings of high (M = 5.89, SD = 0.75)
and low (M = 4.40, SD = 0.86) intensity images, t(22) = -4.54, p \ .01. Positive (high
and low intensity happiness) (M = 7.58, SD = 0.46), negative (high and low intensity
sadness, fear and disgust) (M = 3.27, SD = 0.84) and neutral (M = 4.82, SD = 0.21)
images were all significantly different on valence ratings, F(2,29) = 137.54, p \ .01.
Procedure
A 27-inch monitor was used to display the images. While viewing emotional images,
participants were recorded using a 30-frame per second Sony HD camcorder. The design of
the room was such that the primary sample participants sat approximately one meter away
from the display on which they viewed a timed Powerpoint presentation of images. The
camcorder was situated directly behind the display to record the participant’s face with a
direct field of view. The observer participant sat behind the camcorder and to the participant’s right field of view.
Participants were run individually and presented with a slideshow of images that were
high or low in emotional intensity and varied in emotional valence. They were instructed to
1
Mean arousal ratings for each emotion, by intensity, were as follows: high happiness (M = 5.40), low
happiness (M = 4.03), high sad (M = 5.86), low sad (M = 3.94), high fear (M = 6.61), low fear
(M = 6.17), high disgust (M = 6.46), low disgust (M = 4.25), and neutral (M = 3.04). Mean valence
ratings were: high happiness (M = 7.56), low happiness (M = 7.60), high sad (M = 2.30), low sad
(M = 4.00), high fear (M = 3.90), low fear (M = 4.03), high disgust (M = 2.31), low disgust (M = 3.28),
and neutral (M = 4.82).
123
28
J Nonverbal Behav (2012) 36:23–37
produce facial expressions that were genuine (felt emotion will be expressed), simulated
(expressed emotion with no emotion felt), masked (felt emotion will be covered by
opposing emotional expression), or neutralized (despite presence of a felt emotion, no
emotion will be expressed on the face) while attending to the emotional image on the
screen. Participants were monitored by the naı̈ve observers—who recorded dichotomously
whether they believed each expression observed was sincere or insincere—and were
videotaped (at a rate of 30 frames per second) for later analysis.
The emotional image slideshow consisted of 29 images. Each was displayed for 5 s with
a 5 s break between images to allow the participants to return to a neutral face. Images were
divided into sets, which were prefaced with instructions about how to respond to the
subsequently presented images. Five images were presented in each emotion set, excluding
the neutral set, which consisted of nine images. Participants were instructed to respond with
the same facial expression to each of the images in that set, while maintaining their gaze at
the screen. In each of the four 5-image sets, there was a high and low intensity emotional
image that was consistent with the expressed emotion (i.e., resulting in genuine expressions), a high and low intensity emotional image evoking an emotion opposite to what was
expressed (i.e., resulting in masked expressions), and a neutral image (i.e., resulting in a
simulated expression). For example, in the happiness set a participant would see an initial
screen stating ‘‘Please respond to the following five images with the expression of happiness’’. Participants then viewed images of high and low intensity happy (genuine) and sad
(masked) images, along with one neutral (simulated) image. Participants were asked to
conceal their felt emotions by adopting a neutral face in response to each of the nine images
(one high and low intensity happy, sad, fearful and disgusting image, in addition to one
neutral image) comprising the neutral set (i.e., neutralized expressions). The order in which
the sets were presented, and the order of the images within each set, was randomized.
Further, to confirm that the normed emotional stimuli indeed elicited the intended (genuine)
emotions, participants rated their own emotional experiences (valence and arousal) for each
image following the presentation of the image sets using 1 (not at all) to 7 (highly) scales.
Emotion Expression Analysis
Coding of the emotional expressions was conducted using a highly reliable coding procedure developed for Porter and ten Brinke’s (2008) study, founded on the Facial Action
Coding System (FACS; Ekman et al. 2002), with particular attention to upper and lower
face action units associated with each emotion (Emotion Facial Action Coding System;
EMFACS) and Pictures of Facial Affect which served as prototypical examples for each
emotion (POFA; a set of photographs of expressions depicting the universal emotions;
Ekman and Friesen 1976). For facial coding, each 1/30th second frame of the videotaped
clips was analyzed (150 frames/each 5-s clip) for the presence and duration (from onset to
offset) of the universal emotional expressions in the upper and lower facial regions (by two
trained ‘‘blind’’ analysts). The upper facial region corresponds to the eye and forehead
regions, and the muscles underlying the upper-face action units in the FACS; these muscles
include the frontalis, corrugator, orbicularis oculi, and procerus. The lower facial region
corresponds to the nose, mouth, cheek, and chin areas; the muscles involved include the
risorius, orbicularis oris, zygomatic major, and mentalis). Coders were blind to the veracity
of the emotions they were analyzing (i.e., whether participants were displaying genuine,
simulated, masked, or neutralized expressions), but aware of the emotions participants
intended to portray. They coded a total of 256,650 frames twice each (once focused on
upper face emotion and again for the lower face) in 1,711 expressions. Coding required
123
J Nonverbal Behav (2012) 36:23–37
29
classifying the emotion exhibited in each facial region and recording the frame/time at
which these expressions began and ended. Inconsistent emotions lasting from 1/25 to 1/5th
of a second were recorded as ‘‘Ekman (1985/2001) micro-expressions’’.
Training in this method involves facial musculature recognition, memorization of facial
action units associated with universal emotions, and identification of the seven universal
emotions. One of the analysts was previously trained for the Porter and ten Brinke (2008)
study. The second analyst was trained for two weeks in the coding procedure. Both coders
studied facial musculature, facial action units associated with the universal emotions, and
the identification of universal emotional expressions extensively. To facilitate training, we
have created a detailed reference guide that includes numerous examples of each emotion
and the main muscle movements involved. Training with this reference guide was complemented by detailed study of the POFA and by practice/expertise with the FACS,
although the FACS was not formally used for coding. In addition, the coders reviewed
studies investigating facial actions involved in the universal emotional expressions (Kohler
et al. 2004; Suzuki and Naitoh 2003).
So that we could assess the coders’ knowledge level after training, coders viewed a slide
show of 50 faces from the POFA database and classified the emotion expressed in each
image. Additionally, they viewed 48 videos in a micro-expression task similar to that used
by Frank and Ekman (1997). Each of these videos included a 1/25-s glimpse of one still
picture of facial affect embedded within another, different expression, and coders were
required to classify the emotion in the micro-expression image. The two coders obtained
accuracy rates of 100 and 96% on the POFA task, and 98 and 96% on the microexpression-identification task. Finally, they practiced frame-by-frame video analysis of
emotional facial expressions by coding the video of a sample participant until they were
able to attain nearly perfect reliability.
Coding Reliability
To examine inter-rater reliability statistically, we had both coders analyze the complete
videos of nine participants (261 expressions or 39,150 frames, each coded twice—once for
the upper, and once for lower face emotional expression). Inter-rater reliability was at least
‘‘good’’ (as defined, e.g., by Cicchetti and Sparrow 1981, and Fleiss 1981) on all indices.
The coders demonstrated good reliability in coding the presence of inconsistent emotions
(i.e., any emotional expression discordant with the intended expression, not including
neutral/no emotion) and the duration of the displays, Kappa = .70, p \ .001, 87.3%
agreement, and r(520) = .75, p \ .001, respectively. The raters averaged 92.18%
(SD = 17.79) agreement on the number of inconsistent frames per expression. Disagreement in coding a frame as inconsistent was infrequent, occurring for an average of 12.28
(SD = 27.24) frames for the upper face and 11.14 (SD = 26.14) frames for the lower face,
out of 150 frames per expression (i.e., the coders agreed on an average of 137.72 and
138.86 frames for the upper and lower face, respectively).
Results
Ensuring that Participants Experienced the Intended Emotion
Because participants were instructed to exhibit particular expressions in response to the
normed stimuli, we wanted to ensure that the intended genuine emotions were experienced.
123
30
J Nonverbal Behav (2012) 36:23–37
As such, we had participants rate their own emotional reactions (valence and arousal) to
the images on 7 point scales. Indeed, analyses of mean ratings of emotional valence and
arousal replicated those conducted using IAPS norms. Analysis of participant ratings (on a
1–7 Likert scale) revealed significant differences in arousal ratings of high (M = 5.16,
SD = 0.50) and low (M = 4.37, SD = 0.52) intensity images, t(22) = -3.78, p \ .01.
Positive (high and low intensity happiness) (M = 5.71, SD = 0.51), negative (high and
low intensity sadness, fear and disgust) (M = 2.30, SD = 0.65), and neutral (M = 4.06,
SD = 0.05) images were all significantly different on valence ratings, F(2,29) = 137.54,
ps \ .01.
Testing the Inhibition Hypothesis
Due to the within-subjects study design and the greater number of images in neutral,
relative to happy, sad, fear or disgust sets, the inhibition hypothesis was examined in two
independent sets of statistical tests. Specifically, the impact of emotional intensity on the
frequency and duration of emotional leakage was examined in genuine versus masked
expressions and in genuine neutral versus neutralized expressions, separately. Potential
differences in blink rate also were examined in genuine versus masked expressions and in
genuine neutral versus neutralized expressions, separately.
Inconsistent Emotional Leakage
Impact of Emotional Intensity in Genuine Versus Masked Expressions
First, a 4 (expressed emotion: happy, sad, fear disgust) 9 2 (veracity: genuine, masked) 9
2 (intensity: high, low)2 MANOVA was conducted to evaluate the effect of emotional
intensity on the presence of inconsistent emotion in the upper and lower face. A significant
multivariate main effect of expressed emotion was revealed, F(6,53) = 28.28, p \ .01,
g2p = .76. This effect was present for the both upper (F(3,174) = 19.68, p \ .01, g2p = .25)
and lower face (F(3,174) = 40.99, p \ .01, g2p = .41). An examination of means suggests
that inconsistencies were most likely to occur in fearful expressions (upper face: M = 0.48,
SE = 0.05; lower face: M = 0.53, SE = 0.05) and least likely to occur during expressions
of happiness (upper face: M = 0.08, SE = 0.02; lower face: M = 0.03, SE = 0.01).
A significant expressed emotion 9 veracity interaction also was present at the multivariate
level, F(6,53) = 2.70, p \ .05, g2p = .23. Follow-up univariate analyses revealed that this
effect held only for the upper face, F(3,174) = 4.05, p \ .01, g2p = .07. While veracity did
not affect the presence of inconsistent emotions for either fearful or disgusting expressions
(ps [ .05), emotional leakage was more likely to occur in masked (M = 0.10, SE = 0.04)
than genuine (M = 0.05, SE = 0.02) expressions of happiness and in genuine (M = 0. 37,
SE = 0.05) than masked (M = 0.24, SE = 0.05) displays of sadness. Lastly, while the
intensity 9 veracity interaction only approached significance, F(2,57) = 1.81, p = .17,
g2p = .06, there was a significant multivariate expressed emotion 9 intensity 9 veracity
interaction, F(6,53) = 4.28, p [ .01, g2p = .33, which held for both the upper and lower
face, ps \ .01. In general, an examination of means revealed that masking high intensity
emotions resulted in a greater likelihood of inconsistent emotion for expressions of
2
In all analyses, the order of images to which the participants was assigned was included as a betweensubjects variable. The effect of this variable was never statistically significant, p’s [ .05, and was dropped
from subsequent analyses.
123
J Nonverbal Behav (2012) 36:23–37
31
happiness and sadness, but did not lead to greater emotional leakage during masks of fear
or disgust.
In addition, the impact of veracity and emotional intensity on duration of inconsistent
emotional expressions was examined using a 4 (expressed emotion) 9 2 (veracity) 9 2
(intensity) MANOVA. Duration of inconsistent emotional expressions in the upper and
lower face, respectively, served as dependent variables. A main effect of expressed
emotion was present at the multivariate level (F(6,53) = 15.73, p \ .01, g2p = .64) and for
the upper and lower face independently, p’s \ .01. In general, the greatest amount of
emotional leakage occurred during fearful expressions (upper face: M = 35.09,
SE = 4.83; lower face: M = 46.10, SE = 6.14) while happiness expressions were the least
likely to include inconsistent emotions (upper face: M = 2.99, SE = 1.00; lower face:
M = 0.62, SE = 0.28). As the inhibition hypothesis would predict, our analysis revealed a
significant Intensity 9 Veracity interaction, F(2,57) = 4.28, p \ .05, g2p = .13. Further,
this interaction was significant in both the upper (F(1,58) = 6.74, p \ .05, g2p = .10) and
lower (F(1,58) = 4.49, p \ .05, g2p = .07) face. In the upper face in particular, there was
significantly longer emotional leakage in the high intensity masked relative to the high
intensity genuine condition, p \ .05. Further, the difference between duration of emotional
inconsistency during high intensity masks (M = 23.67; 95% CI [17.89, 29.45]) and low
intensity masks (M = 18.56; 95% CI [12.59, 24.30]) approached significance (see Fig. 1).
This interaction is qualified by an expressed emotion 9 intensity 9 veracity interaction,
F(6,53) = 5.65, p \ .01, g2p = .39. In general, emotional leakage was greater in both the
upper and lower face during high-intensity masked, relative to low-intensity masked,
expressions for all emotions except disgust.
Impact of Emotional Intensity in Neutralized Expressions
The impact of veracity and emotional intensity on duration of inconsistent emotional
expressions during emotion neutralization was examined using 4 (felt emotion) 9 2
(intensity) MANOVAs. First, the presence of inconsistent emotional expression in the
upper and lower face were the dependent variables. Although the main effect of intensity
did not reach statistical significance at the multivariate level, F(2,57) = 2.06, p = .14,
g2p = .07, post-hoc univariate follow-up analyses revealed that this effect was significant in
the upper face, F(1,58) = 4.11, p \ .05, g2p = .07. Emotional inconsistencies were more
Fig. 1 Duration (in 1/30thsecond frames) of inconsistent
emotion in the upper face as a
function of intensity and veracity
Genuine
Genuine, Low,
23.2
Masked
Masked, Low,
18.56
Masked, High,
23.67
Genuine, High,
17.68
123
32
J Nonverbal Behav (2012) 36:23–37
likely to occur during high (M = 0.14, SE = 0.03) relative to low (M = 0.09, SE = 0.02)
intensity neutralized expressions. Second, duration of inconsistent emotional expression
(measured in 1/30th-second frames) in the upper and lower face, respectively, served as
dependent variables. The analyses did not reveal any significant main effects or interaction. However, there was a trend for high intensity emotions (M = 7.0, SE = 2.14) to be
revealed by emotional leakage of a longer duration than low intensity emotions (M = 5.13,
SE = 1.95) in the upper face, F(1,58) = 3.34, p = .07.
Ekman Micro-Expressions
No complete micro-expressions (1/25th–1/5th of a second) involving both the upper and
lower halves of the face simultaneously (as described by Ekman and Friesen 1975) were
detected in any of the 1,711 analyzed expressions. However, 15 participants exhibited 18
partial micro-expressions; 10 in the upper and 8 in the lower facial region. Of these, seven
occurred during the presentation of high intensity emotional images, and 11 during low
intensity emotional images. Six of the 10 micro-expressions that occurred during masked
and neutralized emotional portrayals were congruent with the deceivers’ felt emotion.
Blink Rate
Impact of Emotional Intensity in Genuine Versus Masked Expressions
A 4 (expressed emotion: happy, sad, fear, disgust) 9 2 (veracity: genuine, masked) 9 2
(intensity: high, low) repeated measures ANOVA was conducted to evaluate the effect of
veracity and emotional intensity on the number of blinks in each 5-second expression.
There was a significant main effect of expressed emotion, F(3,174) = 5.41, p \ .05,
g2p = .09, such that the greatest number of blinks occurred during sad (M = 1.71,
SE = 0.19), followed by fearful (M = 1.49, SE = 0.16), happy (M = 1.28, SE = 0.16),
and disgust (M = 1.21, SE = 0.17) expressions. However, effects of veracity and intensity
were non-significant, ps [ .05.
Impact of Emotional Intensity in Neutralized Expressions
The impact of veracity and emotional intensity on blink rate during emotion neutralization
was examined using 4 (felt emotion) 9 2 (intensity) repeated measures ANOVA. A significant effect of intensity was revealed, F(1,58) = 5.61, p \ .05, g2p = .09, with high
intensity neutralized expressions being associated with fewer blinks (M = 1.25,
SE = 0.16) than low intensity (M = 1.44, SE = 0.15).
Accuracy of the Observers in Judging the Veracity of Expressions
A final question was whether the observers could identify deceptive facial expressions with
the naked eye. A 4 (expressed emotion: happy, sad, fear, disgust) 9 2 (intensity: high vs.
low) 9 2 (veracity: genuine vs. masked) was conducted to test the hypothesis that high
intensity deceptive emotions would be easier for naı̈ve observers to detect than low
intensity emotional deception. However, this analysis did not yield any significant main
effects or interactions, all ps [ .05. In general, judges achieved an overall mean accuracy
of 54.82%, which was not significantly above chance, t(27) = 1.93, p = .06. Judges
123
J Nonverbal Behav (2012) 36:23–37
33
performed above the level of chance in detecting deception in happiness, t(27) = 2.56,
p \ .05, whereas their accuracy in judging the veracity of sad, disgust and fearful
expressions did not differ from chance (ps [ .05).
Discussion
The widespread notion that the human face and its expressions can betray covert emotional
states originates in the inhibition hypothesis (Ekman et al. 1988, 1991). Darwin (1872)
postulated that powerful emotions in particular could not be completely inhibited nor
fabricated because of the involuntary nature of emotional expressions. A century later,
Haggard and Isaacs (1966) and then Ekman and colleagues (e.g., Ekman 1985/2001;
Ekman 2006; Frank and Ekman 1997) argued that extremely brief ‘‘flashes’’ of an individual’s true emotion appear on the face uncontrollably as ‘‘micro-expressions.’’ If confirmed, these hypotheses potentially offer critical insights into the nature of human
communication.
The present study was a comprehensive investigation relating to these ideas. It also was
one of the most thorough studies of human facial expressions yet conducted, with a
database of 1,711 expressions comprising a manual analysis of 256,650 video frames of
facial behavior and four of the universal emotions. Results were consistent with the
proposition that facial communication is not always under conscious control in that
emotional leakage was essentially ubiquitous, occurring in 98.3% of participants at least
once. Most importantly, the findings strongly supported the inhibition hypothesis; high
intensity deceptive emotions were associated with substantially more emotional leakage
than low intensity deceptive emotions. High intensity emotion was more difficult to
conceal than low intensity emotion during emotional masking (replacing a felt emotion
with another, false emotional expression), and leakage was especially likely to manifest in
the muscles of the upper face (see also Rinn 1984). As in Porter and ten Brinke (2008), it
was more difficult for participants to mask an emotion than to neutralize one (maintain a
neutral face in the presence of felt emotion). However, there were trends toward more and
longer inconsistencies in high intensity neutralization relative to low intensity neutralization, consistent with the inhibition hypothesis. On the other hand, some leakage did
occur in low intensity deceptive emotions, suggesting the possibility that leakage may
occur on a continuum of emotional intensity as opposed to being an ‘‘all or none’’ phenomenon (as we predicted), occurring only in falsified emotions of a particular high
intensity. Future research could clarify this issue by using a greater number of levels of
emotional intensity and establish the intensity threshold required for leakage to occur.
Inconsistencies were more likely to occur during particular emotions relative to others.
Specifically, in both the upper and lower face, inconsistencies were most likely to occur in
fearful expressions and least likely to occur during expressions of happiness, which may
relate to level of experience; Somerville and Whalen (2006) found that happiness is the
most commonly experienced expression whereas fearful expressions are exhibited the
least. Further, the impact of emotional intensity on the likelihood that one’s emotional
deception would be revealed by leakage of emotional inconsistencies varied by emotion.
Considering high intensity emotion specifically, masking resulted in a greater likelihood of
inconsistent emotion for expressions of happiness and sadness, but did not lead to greater
emotional leakage during masks of fear or disgust. The data suggest that one reason for this
is a ‘‘ceiling’’ effect such that fear and disgust were associated with a high level of leakage
for both low and high intensity emotions.
123
34
J Nonverbal Behav (2012) 36:23–37
This support for the inhibition hypothesis, specifically in the upper face, during falsification of happiness may reflect the incomplete fabrication of deceptive, negative emotions, and the leakage of genuine, positive emotion in falsified emotion (and vice versa for
the falsification of sadness). As in a recent study of falsified remorse (ten Brinke et al.
2011), this finding may be due specifically to the uncontrollability of the medial portion of
the frontalis muscles. Whereas most people can easily raise the eyebrows (frontalis muscle,
Action Units 1 & 2; Ekman et al. 2002), it is difficult for most people to engage only the
medial portion of the frontalis muscle (Action Unit 1; Ekman et al. 2002) spontaneously,
which brings the inner eyebrows upward to mimic the upper face activation associated with
a genuine sad face (Ekman 2003b). Thus, the deceiver activates the complete (inner and
outer) frontalis muscle, appearing surprised and leaking inconsistent emotion during falsified sadness. Alternatively, while genuinely experiencing sadness, the deceiver may not
be able to suppress medial frontalis action, leading to leakage of a raised brow during a
falsified display of happiness.
The notion of the 1/25th–1/5th of a second full-face micro-expression long advocated
by Ekman was not supported here. No complete micro-expressions involving both the
upper and lower halves of the face simultaneously (as described by Ekman and Friesen
1975) were detected in any of the 1,711 analyzed expressions, suggesting that they may not
exist or that they are exceedingly rare. However, there were rare occurrences of ‘‘partial’’
fleeting micro-expressions manifesting in either the upper or lower face. Fifteen (25.4%)
participants exhibited 18 partial micro-expressions: ten in the upper and eight in the lower
facial region. From an applied perspective, this offers good and bad news. The good news
is that most instances of emotional leakage lasted longer than 1/5th of a second, and
frequently lasted for closer to a full second. This means that ‘‘subtle’’ deceptive expressions occur frequently (especially in high intensity emotional displays) and should be
relatively perceptible to the trained eye. On the other hand, both Ekman (1985/2001)
micro-expressions and (the far more common) longer-lasting emotional leakage we
observed typically are manifested in one region of the face (the latter more common in the
upper face) making them more ‘‘subtle’’ than traditionally believed. Clearly, relative to the
traditional view of micro-expressions, a better way to conceptualize involuntary facial
communication is as subtle emotional leakage lasting long enough for the trained eye to
detect and more often occurring in the upper face and for high intensity emotions. Related
‘‘real-life’’ findings were recently generated in study of extremely high-stakes emotional
deception (ten Brinke and Porter 2011); deceptive individuals (killers) ‘‘pleading’’ for the
return of a missing relative (versus sincere pleaders) exhibited similar patterns of emotional leakage in their facial expressions.
Despite the perceptible leakage that appeared on deceptive emotional faces, observers
generally were unable to discriminate sincere and insincere emotions, performing at the
level of chance for all negative emotions and only slightly above chance for happiness
expressions (see also Ekman et al. 1988). Although objectively there was more leakage in
high intensity expressions, observers did no better at identifying high intensity versus low
intensity emotional deception. It should be noted that when producing the expressions,
participants were asked to look at the screen. Thus, it is possible that observers had
difficultly detecting deceptive expressions because all participants were looking away from
them (e.g., Adams and Kleck 2003). However, the findings are consistent with the general
findings on deception detection (see Vrij et al. 2011) and are similar to the findings of Hess
and Kleck (1994) and Porter and ten Brinke (2008) relating to emotional deception.
Generally, these findings suggest that if observers are not sophisticated in evaluating
emotional facial expression, the increased likelihood and duration of diagnostic emotional
123
J Nonverbal Behav (2012) 36:23–37
35
leakage are of little assistance to naı̈ve judges. However, research does suggest that short
training programs that focus on emotional cues to deceit can drastically improve the
detection of high-stakes, high-intensity emotional deception from the level of chance to
80% accuracy (Shaw et al. 2011).
This exciting scientific area promises to provide greater illumination of a topic pioneered by Duchenne, Darwin, and Ekman: that of human facial communication and the
universal emotions. Arguably, many aspects of human facial communication are uncharted
scientific territory, despite long-standing ideas and assumptions about the involuntary
nature of face communication. Research such as the current study is firmly establishing that
the human face can reveal the presence of a contradictory underlying emotion, particularly
when the deceiver is experiencing a powerful emotional state at odds with the expressed
emotion. Ultimately, this knowledge could improve the identification of emotional
deception in legal and security environments where such assessments carry enormous
consequences.
References
Adams, R. B., Jr., & Kleck, R. E. (2003). Perceived gaze direction and the processing of facial displays of
emotion. Psychological Science, 14, 644–647. doi:10.1046/j.0956-7976.2003.psci_1479.x.
Cicchetti, D., & Sparrow, S. (1981). Developing criteria for establishing interrater reliability of specific
items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86(2),
127–137.
Darwin, C. (1872). The expression of the emotions in man and animals. Chicago: University of Chicago
Press.
Duchenne, G. B. (1990). The mechanism of human facial expression. New York: Cambridge University
Press. (Original work published 1862).
Ekman, P. (1985/2001). Telling lies: Clues to deceit in the marketplace, politics, and marriage. New York:
Norton.
Ekman, P. (2003a). Darwin, deception and facial expression. In P. Ekman, R. J. Davidson, & F. De Waals
(Eds.), Annals of the New York Academy of Sciences. Emotions inside out: 130 years after Darwin’s
The Expression of the Emotions in Man and Animals (Vol. 1000, pp. 205–221). New York: New York
Academy of Sciences.
Ekman, P. (2003b). Emotions revealed: Recognizing faces and feelings to improve communication and
emotional life. New York, NY, US: Times Books/Henry Holt and Co.
Ekman, P. (2006, October 29). How to spot a terrorist on the fly. Washington Post. Retrieved from
http://www.washingtonpost.com.
Ekman, P. (2009). Telling lies: Clues to deceit in the marketplace, politics, and marriage. New York, NY,
US: Norton.
Ekman, P., Davidson, R., & Friesen, W. (1990). The Duchenne smile: Emotional expression and brain
physiology: II. Journal of Personality and Social Psychology, 58(2), 342–353. doi:10.1037/
0022-3514.58.2.342.
Ekman, P., & Friesen, W. (1974). Detecting deception from the body or face. Journal of Personality and
Social Psychology, 29, 288–298. doi:10.1037/h0036006.
Ekman, P., & Friesen, W. V. (1975). Unmasking the face: A guide to recognizing emotions from facial clues.
Oxford, England: Prentice-Hall.
Ekman, P., & Friesen, W. V. (1976). Pictures of facial affect. Palo Alto, CA: Consulting Psychologists
Press.
Ekman, P., Friesen, W. V., & Hagar, J. C. (2002). Facial action coding system. Salt Lake City, UT: Network
Information Research. (Original work published 1976).
Ekman, P., Friesen, W. V., & O’Sullivan, M. (1988). Smiles when lying. Journal of Personality and Social
Psychology, 54(3), 414–420. doi:10.1037/0022-3514.54.3.414.
Ekman, P., & O’Sullivan, M. (2006). From flawed self-assessment to blatant whoppers: The utility of
voluntary and involuntary behavior in detecting deception. Behavioral Sciences and the Law, 24(5),
673–686. doi:10.1002/bsl.729.
123
36
J Nonverbal Behav (2012) 36:23–37
Ekman, P., O’Sullivan, M., Friesen, W. V., & Scherer, K. R. (1991). Invited article: Face, voice, and body in
detecting deceit. Journal of Nonverbal Behavior, 15(2), 125–135. doi:10.1007/BF00998267.
Fleiss, J. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological
Measurement, 5(1), 105–112. doi:10.1177/014662168100500115.
Frank, M. G., & Ekman, P. (1997). The ability to detect deceit generalizes across different types of highstake lies. Journal of Personality and Social Psychology, 72, 1429–1439.
Frank, M. G., Ekman, P., & Friesen, W. V. (1993). Behavioral markers and recognizability of the smile of
enjoyment. Journal of Personality and Social Psychology, 64(1), 83–93. doi:10.1037/0022-3514.
64.1.83.
Gable, P., & Harmon-Jones, E. (2010). The blues broaden, but the nasty narrows: Attentional consequences
of negative affects low and high in motivational intensity. Psychological Science, 21(2), 211–215. doi:
10.1177/0956797609359622.
Gottman, J., Levenson, R., & Woodin, E. (2001). Facial expressions during marital conflict. Journal of
Family Communication, 1(1), 37–57. doi:10.1207/S15327698JFC0101_06.
Haggard, E. A., & Isaacs, K. S. (1966). Micromomentary facial expressions as indicators of ego mechanisms
in psychotherapy. In L. A. Gottschalk & A. H. Auerbach (Eds.), Methods of research in psychotherapy
(pp. 154–165). New York: Appleton Century Crofts.
Henig, R. M. (2006, February 5). Looking for the lie. New York Times. Retrieved December 1, 2010, from
www.nytimes.com.
Hess, U., & Bourgeois, P. (2010). You smile–I smile: Emotion expression in social interaction. Biological
Psychology, 84, 514–520. doi:10.1016/j.biopsycho.2009.11.001.
Hess, U., & Kleck, R. (1994). The cues decoders use in attempting to differentiate emotion-elicited and
posed facial expressions. European Journal of Social Psychology, 24(3), 367–381. doi:10.1002/
ejsp.2420240306.
Hess, U., & Thibault, P. (2009). Darwin and emotion expression. American Psychologist, 64, 120–128. doi:
10.1037/a0013386.
Kohler, C., Turner, T., Stolar, N., Bilker, W., Brensinger, C., Gur, R., et al. (2004). Differences in facial
expressions of four universal emotions. Psychiatry Research, 128(3), 235–244. doi:10.1016/j.psychres.
2004.07.003.
Lang, P., Bradley, M., & Cuthbert, B. N. (1999). International Affective Picture System (IAPS): Instruction
manual and affective ratings (Tech. Rep. No. A-4). Gainesville: University of Florida, Center for
Research in Psychophysiology.
Libkuman, T. M., Otani, H., Kern, R., Viger, S. G., & Novak, N. (2007). Multidimentional normative ratings
for the International Affective Picture System. Behaviour Research Methods, 39(2), 326–334. doi:
10.3758/BF03193164.
Lipton, E. (2006, August 17). Threats and responses: Screening; faces, too, are searched as U.S. airports try
to spot terrorists. New York Times. Retrieved December 1, 2010, from www.nytimes.com.
Matsumoto, D. (2007). Emotion judgments do not differ as a function of perceived nationality. International
Journal of Psychology, 42(3), 207–214. doi:10.1080/00207590601050926.
Matsumoto, D., & Willingham, B. (2009). Spontaneous facial expressions of emotion of congenitally and
non-congenitally blind individuals. Journal of Personality and Social Psychology, 96(1), 1–10. doi:
10.1037/a0014037.
Mikels, J. A., Fredrickson, B. L., Larkin, G. R., Lindberg, C. M., Magilo, S. J., & Reuter-Lorenz, P. A.
(2005). Emotional category data on images from the International Affective Picture System. Behavior
Research Methods, 37(4), 626–630.
O’Sullivan, M., Frank, M. G., Hurley, C. M., & Tiwana, J. (2009). Police lie detection accuracy: The effect
of lie scenario. Law and Human Behavior, 33(6), 530–538. doi:10.1007/s10979-008-9166-4.
Porter, S., Juodis, M., ten Brinke, L., Klein, R., & Wilson, K. (2010). Evaluation of the effectiveness of a
brief deception detection training program. Journal of Forensic Psychiatry and Psychology, 21(1),
66–76. doi:10.1080/14789940903174246.
Porter, S., & ten Brinke, L. (2008). Reading between the lies: Identifying concealed and falsified emotions in
universal facial expressions. Psychological Science, 19(5), 508–514. doi:10.1111/j.1467-9280.2008.
02116.x.
Porter, S., & ten Brinke, L. (2010). The truth about lies: What works in detecting high-stakes deception?
Legal and Criminological Psychology, 15(1), 57–75. doi:10.1348/135532509X433151.
Porter, S., ten Brinke, L., Baker, A., & Wallace, B. (2011, in press). Would I lie to you? ‘‘Leakage’’ in
deceptive facial expressions relates to psychopathy and emotional intelligence. Personality and
Individual Differences. doi:10.1016/j.paid.2011.03.031.
123
J Nonverbal Behav (2012) 36:23–37
37
Porter, S., ten Brinke, L., & Wilson, K. (2009). Crime profiles and conditional release performance of
psychopathic and non-psychopathic sexual offenders. Legal and Criminological Psychology, 14(1),
109–118. doi:10.1348/135532508X284310.
R. v. B. (K.G.), 1 S.C.R. 740 (Supreme Court of Canada, 1993).
R v N.S., ONCA 670 (Court of Appeal for Ontario, 2010).
Rinn, W. E. (1984). The neuropsychology of facial expression: A review of the neurological and psychological mechanisms for producing facial expressions. Psychological Bulletin, 95(1), 52–77. doi:
10.1037/0033-2909.95.1.52.
Shaw, J., Porter, S., & ten Brinke, L. (2011, under review). Catching liars: Training mental health and legal
professionals to detect extremely high-stakes lies.
Somerville, L. H., & Whalen, P. J. (2006). Prior experience as a stimulus category confound: An example
using facial expressions of emotion. Social, Cognitive, and Affective Neuroscience, 1, 271–274. doi:
10.1093/scan/nsl040.
Stewart, P., Waller, B., & Schubert, J. (2009). Presidential speechmaking style: Emotional response to
micro-expressions of facial affect. Motivation and Emotion, 33(2), 125–135. doi:10.1007/s11031-0099129-1.
Suzuki, K., & Naitoh, K. (2003). Useful information for face perception is described with FACS. Journal of
Nonverbal Behavior, 27(1), 43–55. doi:10.1023/A:1023666107152.
ten Brinke, L., MacDonald, S., Porter, S., & O’Connor, B. (2011). Crocodile tears: Facial, verbal and body
language behaviours associated with genuine and fabricated remorse. Law and Human Behavior,
published online February 8, 2011. doi:10.1007/s10979-011-9265-5..
ten Brinke, L., & Porter, S. (2011, under review). Darwin the detective: Behavioural consequences of
extremely high-stakes lies.
Vrij, A., Granhag, P. A., & Porter, S. (2011). Pitfalls and opportunities in nonverbal and verbal lie detection.
Psychological Science in the Public Interest, 11, 89–121. doi:10.1177/1529100610390861.
Vrij, A., & Mann, S. (2001). Who killed my relative? Police officers’ ability to detect real-life high-stake
lies. Psychology, Crime and Law, 7(2), 119–132. doi:10.1080/10683160108401791.
Warren, G., Schertler, E., & Bull, P. (2009). Detecting deception from emotional and unemotional cues.
Journal of Nonverbal Behavior, 33(1), 59–69. doi:10.1007/s10919-008-0057-7.
Williams, M. A., & Mattingley, J. B. (2006). Do angry men get noticed? Current Biology, 16, R402–R404.
doi:10.1016/j.cub.2006.05.018.
123