Download Facial expression and piano performance

Alma Mater Studiorum University of Bologna, August 22-26 2006
Facial expression and piano performance
Luisa Bonfiglioli
Roberto Caterina
Iolanda Incasa
Mario Baroni
Dept. of Psychology,
University of Bologna.
Bologna, Italy
Dept. of Psychology,
University of Bologna.
Bologna, Italy
Dept. of Psychology,
University of Bologna.
Bologna, Italy
Dept. of Music,
University of Bologna.
Bologna, Italy
email
[email protected]
eyebrow raising occurred mainly when playing pianissimo,
staccato, and high pitch melodies, whereas frowning occurred mainly when playing forte or fortissimo, during a
fast tempo, and low register melodies. In accordance with
human ethological studies (Costa & Ricci Bitti 2003), eyebrow raising is associated with the expression of surprise,
and in the musical field, it mainly occurs whenever there is
a violation of expectations. Frowning is associated with the
expression of negative emotions such as fear, anger and
disgust, and in music performance it generally occurs
when musical features express these negative emotions,
such as during dissonant passages and
ABSTRACT
Previous studies (Caterina et al. 2004) carried out on the
body and facial expressions of pianists during their performances, have shown that there is a specific relation between non verbal expressions and music structure.
We tried to compare the value of single facial action unit
movements– such as eyebrow raising and frowning – found
in our observations, with the concurring musical structure.
Videorecording of a group of professional pianists performing different repertoires were observed. Two independent expert judges analysed the facial expressions using
Ekman and Friesen FACS. Facial action unit movements
were examined together with main musical features, in
order to better understand the meaning of the most frequent facial signals.
The first results of our analyses indicated that eyebrow and
head movements, and mouth opening-closing are the most
frequent non verbal signals in pianists’ performances. On
a musical level, eyebrow raising and lowering can be seen
as perceptual cues, since they often separate one musical
phrase-segment from another.
We also defined different categories concerning the meaning of eyebrow movement. With regard to music structure,
low register melodies, and when playing forte or fortissimo.
Eyebrow raising and frowning are the most frequent
movements during piano performances. Their occurrence
is associated with specific musical features and is related
to performers’ self monitoring activities (sensomotory control and emotional control), to the cognitive processes involved in musical analysis, to the memorization of the music that is being played, and to the performer’s communication with the listeners.
In: M. Baroni, A. R. Addessi, R. Caterina, M. Costa (2006) Proceedings
of the 9th International Conference on Music Perception & Cognition
(ICMPC9), Bologna/Italy, August 22-26 2006.©2006 The Society for
Music Perception & Cognition (SMPC) and European Society for the
Cognitive Sciences of Music (ESCOM). Copyright of the content of an
individual paper is held by the primary (first-named) author of that paper. All rights reserved. No paper from this proceedings may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information
retrieval systems, without permission in writing from the paper's primary
author. No other part of this proceedings may be reproduced or transmitted in any form or by any means, electronic or mechanical, including
photocopying, recording, or by any information retrieval system, without
permission in writing from SMPC and ESCOM.
ISBN 88-7395-155-4 © 2006 ICMPC
Keywords
Piano performance; Facial expression.
INTRODUCTION
There is little literature regarding studies about musicians’
facial expressions. The main obstacles to this type of research are most probably due to interindividual variables
between performers and the fact that, with regard to many
performances executed by great pianists, it is impossible to
know exactly what kind of formal analysis the musician
had in mind when performing the piece.
1355
ICMPC9 Proceedings
However, it is important to point out that while a performers’ movements can be observed directly during a
concert, when an audio system is employed, mental representation are evoked by the sound alone.
Research on the involvement of the body during musical
execution has entailed the use of both non verbal index
observation grids (Poggi, 2001) and computational models
for real time analysis of movement (Camurri, Mazzarino &
Volpe, 2003).
Finally, it is also important to highlight that while musical
gestures are indeed directly related to certain sounds, it is
also true that musicians’ movements depend on the interpretation and specific stylistic rules which performers
chooses to adopt.
Existing research shows that facial expressions are linked
to strategies of memorization of musical pieces (RubinRabson, 1939) and to the preparation that leads up to musical performances (Miklaszewski, 1989).
The ideas expressed by Delalande so far, are mostly based
on the analysis of Glenn Gould’s gestures during the execution of a number of compositions taken from the repertoire of J.S. Bach, which were performed and shot in the
film by Bruno Monsaigeon “Glenn Gould joue Bach”.
Further studies have analysed the relation between violinists and flautists’ posture during performances and the
varying quality of the sounds produced (Ruggieri, Ragni &
Terrezza, 2002).
Studies have also explored the effects that specific movement instructions given to musicians have on their performance (Shove & Repp, 1995).
This analysis defines five “kinds of gestures” (hunched
over the piano, vibrant, fluent, delicate and vigorous),
which include different facial expression configurations,
postures and arm movements that indicate different affective contents. Delalande in fact suggests the term “expressive scheme” meaning that the motor and affective planes
of a performer’s expressive conduct are not to be thought
of as separate aspects but on the contrary, they complement
each other.
Finally, recent research (Davidson, 1995) has shown that
individuals are able to recognize the kind of interpretation
that performers adopt (unexpressive, normal, exaggerated)
based solely on visual information and, therefore, on the
analysis of facial and body expressions (in particular of the
joints and of the head).
The connection between movement and sound production
should not be surprising since music is produced through
human and instrumental movement, and is therefore deeply
affected by its origins, which arise in the body (Clarke,
2002).
However, it is interesting to note that among the gestures
defined in this study, the most significant facial action unit
movement is eyebrow movement. In the study these
movements are divided into the following categories:
The analysis of performers’ movements during executions
of musical pieces, as Delalande highlights (1993), can be
accomplished on three different levels at least.
The first level involves the functional aspect of gesture. In
this case analysis regards so called productive gestures,
which include the movements that performers have to accomplish in order to produce sounds mechanically. These
are linked to sensomotory control aspects of the execution.
fast movements (which correspond to pulsation);
•
movements that accompany melodies;
•
prolonged raising of eyebrows;
•
frowning.
As to the links between musical elements and movements,
Delalande has discovered:
The second level of analysis relates to what are known as
the accompanying gestures, the movements that a pianist
carries out during a performance involving facial expressions, movements of the head, of the bust and of the shoulders. Delalande highlights that it is a simplification to state
that these movements are exclusively connected to motor
coordination. On the contrary, Delalande claims that their
significance should also be ascribed to the mental processes
that govern a performance.
•
eyebrow movements that separate one musical phrase
from another;
•
eyebrow raising linked to pianissimo, staccato and
high register;
•
frowning linked to forte or fortissimo, fast tempo, low
register and dissonant passages.
As regards performers, there are many functions of eyebrow movements because they are linked to the following:
The third and final level of analysis that was determined,
regards the figurative gestures that are evoked in the listener’s mind according to the music that the pianist is
playing. For instance a swaying movement may be evoked
by a certain melody, as well as lunging or leaning etc.
•
sensomotory and emotional control;
•
formal analysis of a work;
•
communication with the listeners.
These observations concerning eyebrow movements should
not be surprising, since numerous studies carried out
among populations in different geographical regions, have
shown that the communicative value of these movements is
common to many cultures (Eibl Eibesfeldt, 2001).
This final level of analysis shows that there is a relation
between the sound produced by performers’ movements
and the mental representations of these movements in the
listeners’ minds.
ISBN 88-7395-155-4 © 2006 ICMPC
•
1356
ICMPC9 Proceedings
The studies particularly highlighted that frowning and eyebrow raising have a wide range of values within social interaction. In fact, frowning can either carry a protective or
an aggressive meaning, and eyebrow raising may express a
greeting, it plays a role in mother and child interaction and
in seduction, and it is also closely linked to verbal discourse and persuasive communication (Costa & Ricci
Bitti, 2003).
by the pianist when the eyebrow movement that we observed, occurred.
Being aware of the many levels on which a score can be
analysed, during this phase of the research it was deemed
necessary to establish very simple criteria.
The criteria we used also aimed at finding music-structural
features that had already been highlighted by Delalande’s
study: the ascending and descending movement of the
notes, the occurrence of a theme or of the repetition of a
theme, the passage between two different themes, the occurrence of ornaments (very brief notes) and prolonged
notes or chords.
While it is true that the decision to analyse great musicians’
performances meant that we could not obtain information
regarding the kind of formal analysis they had been chosen
for the execution of the musical pieces, this method has, on
the other hand, allowed us to gather important information.
METHOD
Vidorecordings of famous performers were analysed in
order to study the relation between specific facial action
unit movements
(eyebrow movement)
during piano
performances and music-structural features.
More specifically the following videorecordings were used:
1) Glenn Gould
Before describing our observation, it is important to specify
that in the grid above, the eyebrow movements we observed were classed as rapid when the eyebrows were
raised and lowered in fast succession, whereas all other
movements were defined as prolonged.
Johann Sebastian Bach, Partita in E minor BWV 830 (duration 26 min.). Taken from: “Glenn Gould, The Alchemist”,
film by Bruno Monsaingeon (Toronto, 1972).
2) Vladimir Horowitz
This distinction was considered necessary because while
rapid movements occurred when a precise element was
present (often a single note), prolonged movements were
related to melodic fragments or whole musical phrases.
Fryderych Chopin, Sonata n. 2 in B flat minor op. 35
"Marche funèbre" (duration: 25 minutes). A public concert
that was held at the White House (1931).
3) Bruno Canino
Claude Debussy, Preludes (“Generale Lavine Eccentric” ,
“Le feés sont d’exquises danseuses” -2° Vol.- and “La fille
aux cheveux de lin”- 1° Vol.) (duration: 15 min). This
videorecording was filmed during two different concerts: in
a public concert which was broadcast by Italian Television
(first performance), and in a second performance during
concert trials that were held in the Bossi Hall in the Conservatoire of Bologna (2004).
RESULTS
The following four tables show the correspondence between eyebrow raising and musical features for each performer.
Videorecording didn’t allow us to collect data systematically, thus these results are preliminary and have a qualitative value.
4) Daniel Barenboim
The following table (Tab.1) shows the correspondence between eyebrow raising and musical features in Gould’s
performance.
Claude Debussy, Preludes (1° Livre) (duration: 59 minutes). Taken from : “Entre Quatre-z-Yeux”, film by Paul
Smaczny (1999).
Table 1. Glenn Gould: correspondence between eyebrow raising and musical elements (J. S. Bach, Partita
in E min. BWV 830)
The video analysis was carried out by two independent
judges according to a grid that had been specifically designed for the purpose.
The grid contained some information regarding eyebrow
movement. More specifically the following characteristics
were indicated in the grid:
Eyebrow
Movements
• Eyebrow movements (frowning or raising);
• Types of eyebrow movements (rapid or prolonged).
Furthermore, in the grid we also noted
down
some
characteristics of the execution (articulation and dynamics)
and some characteristics of the score (register) that coincided with the eyebrow movements that we observed.
Music-structural
features
Raising
Prolonged
Finally, in the last column of the grid, we indicated some of
the music-structural features of the score that were played
ISBN 88-7395-155-4 © 2006 ICMPC
Articulation
Dynamic
Register
Prolonged
1357
Clearly
articulated
mf
Medium register
Staccato
mf
Descending
gression
pro-
Ascending notes
Ornaments
ICMPC9 Proceedings
Prolonged
Prolonged
Rapid
Rapid
Two rapid
raising in
succession
Medium-high
register
Legato
mf
Medium-high
register
Legato
p
Medium-.low
register
Legato
mf
Medium register
Clearly
articulated
f
Medium-high
register
Staccato
p
Medium-high
register
Table 3. Bruno Canino: correspondence between eyebrow
raising and musical elements (C. Debussy, three Preludes,
1° and 2° Vol.)
Articulation
Eyebrow
Music-structural
Dynamic
features
movements
Register
Prolonged high note
Prolonged chord
Raising
Chord (ascending
arpeggio)
Legato
Prolonged
Higher note of
ascending arpeggio
Prolonged
End of trill
Prolonged
Two
ascending
notes of melodic
fragment
mf
Medium-low
register
Legato
mf
Medium- high
register
Staccato
p
Medium register
Repetitions of descending melodic
fragments
Ascending chords
Fast repetition of
chords
Fast notes
The following table (Tab.4) shows the correspondence between eyebrow raising and musical features in Barenboim’s performance.
The following table (Tab.2) shows the correspondence between eyebrow raising and musical features in Horowitz’s
performance.
Table 4. Daniel Barenboim: correspondence between
eyebrow raising and musical elements (C. Debussy,
Preludes , 1°Vol.)
Articulation
Eyebrows
Music-structural
Dynamic
movements
features
Register
Raising
Legato
Repetition of the
mf
theme
Prolonged
Medium-high
Prolonged notes
register
Legato
mf
High register note
Prolonged
Medium-high
(prolonged)
register
Legato
Fast
ascending
mf
scale
Prolonged
Medium-high
register
Tab. 2. Vladimir Horowitz: correspondence between
eyebrow raising and musical elements (F. Chopin, Sonata n. 2 in B flat minor op. 35 "Marche funèbre")
Articulation
Eyebrow
Music-structural
Dynamic
movements
features
Register
Raising
Legato
mf
Second repetition
Prolonged
Medium-high
of the theme
register
Staccato
p
Caesura between
Prolonged
Medium-low
two themes
register
Legato
mf
Higher note of
Rapid
Medium-high
melodic fragment
register
The following four tables show the correspondence between frowning and musical features for each performer.
The following table (Tab.3) shows the correspondence between eyebrow raising and musical features in Canino’s
performance.
ISBN 88-7395-155-4 © 2006 ICMPC
The following table (Tab.5) shows the correspondence between eyebrow frowning and musical features in Gould’s
performance.
1358
ICMPC9 Proceedings
Table 5. Glenn Gould : correspondence between eyebrow frowning and musical elements (J. S. Bach, Partita in E min. BWV 830)
Articulation
Eyebrow
Music-structural
Dynamic
features
Movements
Register
Frowning
Clearly
articulated
Left hand: chords
Prolonged
f
(ascending
Medium-low
arpeggios)
register
Staccato
End of melodic
mf
fragment
Prolonged
Medium-low
(notes with legato )
register
Register
Frowning
Legato
Prolonged
Medium-high register
Table 8. Daniel Barenboim: correspondence between
eyebrow frowning and musical elements (C. Debussy,
Preludes , 1° Livre)
Articulation
Eyebrow
Music-structural
Dynamic Regfeatures
movements
ister
Frowning
Prolonged note
Legato
Lower note of demf
Prolonged
scending
melodic
Medium–low fragment
register
Prolonged chord
Staccato
f
Prolonged
Fast notes
Medium-low
register
The following table (Tab.6) shows the correspondence between eyebrow frowning and musical features in
Horowitz’s performance.
One result which is particularly worth highlighting, is the
connection between eyebrow raising and register. In fact,
eyebrow raising generally appears related to medium and
medium-high registers.
Tab. 6. Vladimir Horowitz: correspondence between
eyebrow frowning and musical elements (F. ChopinSonata n. 2 in B flat minor op. 35 "Marche funèbre")
Articulation
Eyebrow
Music-structural
Dynamic
movements
features
Register
Frowning
Legato
Repetition of the
mf
theme “Andante”
Prolonged
(with
harmonic
Medium register
modulation)
Legato
Last presentation
P and crescendo of first funereal
Prolonged
Medium-low
theme
register
Legato
Note in suspense
P
(with
dissonant
Rapid
Medium-high
chord )
register
With regard to dynamics, we see that eyebrow raising occurs very rarely in correspondence with a forte, while there
seems to be a stronger link with piano and mezzo forte.
On the other hand, there appears to be no particular correlation between the articulation performers adopt and eyebrow
raising.
With regard to the musical elements we noted, eyebrow
raising seems more strongly correlated with ascending melodic lines, specific highlighting of high pitch notes, ornaments and prolonged notes and chords that are
characterized, from an expressive point of view, by a feeling of suspense and awaiting.
As regards frowning, our observations show that there is a
correlation between this movement and medium or medium-low registers.
While concerning dynamics, contrary to eyebrow raising,
there appears to be a more frequent correspondence with
mezzo forte and forte.
Another interesting fact is that legato articulation generally
appears linked to frowning.
In the performances we analysed, from an expressive point
of view, the musical characteristics we have just mentioned
conveyed a sense of power, energy and sometimes of pain.
In addition, a very interesting fact that emerged from our
observations is that, as the above tables show, eyebrow
Finally, the following tables (Tab.7, Tab. 8) show the correspondence between eyebrow frowning and musical features in Canino’s and Barenboim’s performances.
Table 7. Bruno Canino: correspondence between eyebrow frowning and musical elements (C. Debussy,
three Preludes, 1° and 2° Vol.)
Eyebrow
Articulation
movements
Dynamic
ISBN 88-7395-155-4 © 2006 ICMPC
f
Descending
melodic fragments of
the theme
Music-structural
features
1359
ICMPC9 Proceedings
movements are sometimes connected to the articulation of
a musical phrase.
This is especially noticeable in Glenn Gould’s performance
when he plays Bach’s Sixth Partita, a piece of music that is
characterized by counterpointing.
This link suggests an analogy between the expressive function of a performer’s eyebrows during a performance and
their significance in the spoken language.
Specific studies have in fact shown that eyebrow movements are not only used in persuasive communication, but
in every day conversations, too, for instance to underline
semantically relevant word, or before speaking, in order to
attract the listener’s attention.
Costa, M., & Ricci Bitti, P.E. (2003). Il linguaggio delle
sopracciglia. Psicologia contemporanea, 176, 38-47.
Davidson, J. W. (1995). What does the visual information
contained in music performances offer the observer? Some
preliminary thoughts. In R. Steinberg (Ed.), Music and the
mind machine. Psychophysiology and psychopathology of
the sense of music. Berlin: Springer.
Delalande, F. (1993). Le condotte musicali: Comportamenti e motivazioni del fare e ascoltare musica. Bologna:
CLUEB.
Eibl- Eibesfeldt, I. (2001). Etologia umana, le basi
culturali e bioloogiche del comportamento. Torino:
Boringhieri.
CONCLUSION
Ekman, P., & Friesen, W. (1978). Facial Action Coding
System. Palo Alto: Consulting Psychologist Press, Inc.
To sum up, we should point our that some characteristics of
the material employed in the study (stylistic differences of
the pieces performed, monitoring of the footage on which
the observations were carried out, and each pianist’s personal interpretation) do not allow for generalisations.
Miklaszewski, K. (1989). A case study of a pianist
preparing a musical performance. Psychology of Music, 17,
95-109.
Poggi, I. (2001). The lexicon of the conductor’s face. In P.
McKevitt, S. Nuallàin, & C. Mulvihill (Eds.), Language,
Vision and Music. Amsterdam: John Benjamins.
Ruggieri, V., Ragni, S., & Terrezza, L. (2002). Psychopsychological mechanismus of artistic expression and musical performance: A psycho-physiological inquiry. In J.
Fachner, & D. Aldridge (Eds.), Dialogue and debate. Proceedings of the 10th World Congress on Music Therapy,
Oxford, UK, July 23-28, pp.1358-1399.
However, even if this study is a preliminary one, the tables
indicate interesting relationships between eyebrow movements and the musical elements that are executed in concurrence with these movements.
Future development of this study will involve the integration of the data so far gathered regarding eyebrow movement, with other significant non verbal indexes of pianists’
bodies and faces, with the aim of verifying the existence of
expressive configurations or gesture types, in accordance
with Delalande’s theories.
Rubin-Rabson, G. (1939). Studies in the psychology of
memorizing piano music: A comparison of the unilateral
and the coordinated approaches. Journal of Educational
Psychology, 30, 321-345.
ACKNOWLEDGMENTS
We thank Dr. Mariateresa Storino for her precious collaboration in the video analysis of the performances. We are
also especially grateful to Prof. Marco Costa
for offered support.
Shove, P., & Repp, B. (1995). Musical motion and
performance: theoretical and empirical perspectives. In J.
Rink (Ed.), The practice of the performance, (pp. 55-83).
Cambridge: Cambridge University Press.
REFERENCES
Caterina, R., Bonfiglioli, L., Baroni, M, & Addessi, A.
(2004). Mimic expression and piano performance. In S.D.
Lipscomb, R. Ashley, R.O. Gjerdingen, & P. Webster. Proceedings of 8th International Conference on Music Perception and Cognition, Evanston, August 3-7, pp.258-261.
Camurri, A., Mazzarino, B., & Volpe, G. (2003). Analysis
of expressive gestures in human movement: The Eyesweb
Expressive Gesture Processing Library. Proceedings of the
XIV Colloquim on Musical Informatics, Firenze, Italy,
May 8-10, pp.54-58.
Clarke, E. (2002). Processi cognitivi nell’esecuzione musicale. In J.J. Nattiez, M. Bent, R. Dalmonte, & M. Baroni
(Eds.), Enciclopedia della musica. Il sapere musicale
(pp.288-304). Torino: Einaudi.
ISBN 88-7395-155-4 © 2006 ICMPC
1360