Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Alma Mater Studiorum University of Bologna, August 22-26 2006 Facial expression and piano performance Luisa Bonfiglioli Roberto Caterina Iolanda Incasa Mario Baroni Dept. of Psychology, University of Bologna. Bologna, Italy Dept. of Psychology, University of Bologna. Bologna, Italy Dept. of Psychology, University of Bologna. Bologna, Italy Dept. of Music, University of Bologna. Bologna, Italy email [email protected] eyebrow raising occurred mainly when playing pianissimo, staccato, and high pitch melodies, whereas frowning occurred mainly when playing forte or fortissimo, during a fast tempo, and low register melodies. In accordance with human ethological studies (Costa & Ricci Bitti 2003), eyebrow raising is associated with the expression of surprise, and in the musical field, it mainly occurs whenever there is a violation of expectations. Frowning is associated with the expression of negative emotions such as fear, anger and disgust, and in music performance it generally occurs when musical features express these negative emotions, such as during dissonant passages and ABSTRACT Previous studies (Caterina et al. 2004) carried out on the body and facial expressions of pianists during their performances, have shown that there is a specific relation between non verbal expressions and music structure. We tried to compare the value of single facial action unit movements– such as eyebrow raising and frowning – found in our observations, with the concurring musical structure. Videorecording of a group of professional pianists performing different repertoires were observed. Two independent expert judges analysed the facial expressions using Ekman and Friesen FACS. Facial action unit movements were examined together with main musical features, in order to better understand the meaning of the most frequent facial signals. The first results of our analyses indicated that eyebrow and head movements, and mouth opening-closing are the most frequent non verbal signals in pianists’ performances. On a musical level, eyebrow raising and lowering can be seen as perceptual cues, since they often separate one musical phrase-segment from another. We also defined different categories concerning the meaning of eyebrow movement. With regard to music structure, low register melodies, and when playing forte or fortissimo. Eyebrow raising and frowning are the most frequent movements during piano performances. Their occurrence is associated with specific musical features and is related to performers’ self monitoring activities (sensomotory control and emotional control), to the cognitive processes involved in musical analysis, to the memorization of the music that is being played, and to the performer’s communication with the listeners. In: M. Baroni, A. R. Addessi, R. Caterina, M. Costa (2006) Proceedings of the 9th International Conference on Music Perception & Cognition (ICMPC9), Bologna/Italy, August 22-26 2006.©2006 The Society for Music Perception & Cognition (SMPC) and European Society for the Cognitive Sciences of Music (ESCOM). Copyright of the content of an individual paper is held by the primary (first-named) author of that paper. All rights reserved. No paper from this proceedings may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information retrieval systems, without permission in writing from the paper's primary author. No other part of this proceedings may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information retrieval system, without permission in writing from SMPC and ESCOM. ISBN 88-7395-155-4 © 2006 ICMPC Keywords Piano performance; Facial expression. INTRODUCTION There is little literature regarding studies about musicians’ facial expressions. The main obstacles to this type of research are most probably due to interindividual variables between performers and the fact that, with regard to many performances executed by great pianists, it is impossible to know exactly what kind of formal analysis the musician had in mind when performing the piece. 1355 ICMPC9 Proceedings However, it is important to point out that while a performers’ movements can be observed directly during a concert, when an audio system is employed, mental representation are evoked by the sound alone. Research on the involvement of the body during musical execution has entailed the use of both non verbal index observation grids (Poggi, 2001) and computational models for real time analysis of movement (Camurri, Mazzarino & Volpe, 2003). Finally, it is also important to highlight that while musical gestures are indeed directly related to certain sounds, it is also true that musicians’ movements depend on the interpretation and specific stylistic rules which performers chooses to adopt. Existing research shows that facial expressions are linked to strategies of memorization of musical pieces (RubinRabson, 1939) and to the preparation that leads up to musical performances (Miklaszewski, 1989). The ideas expressed by Delalande so far, are mostly based on the analysis of Glenn Gould’s gestures during the execution of a number of compositions taken from the repertoire of J.S. Bach, which were performed and shot in the film by Bruno Monsaigeon “Glenn Gould joue Bach”. Further studies have analysed the relation between violinists and flautists’ posture during performances and the varying quality of the sounds produced (Ruggieri, Ragni & Terrezza, 2002). Studies have also explored the effects that specific movement instructions given to musicians have on their performance (Shove & Repp, 1995). This analysis defines five “kinds of gestures” (hunched over the piano, vibrant, fluent, delicate and vigorous), which include different facial expression configurations, postures and arm movements that indicate different affective contents. Delalande in fact suggests the term “expressive scheme” meaning that the motor and affective planes of a performer’s expressive conduct are not to be thought of as separate aspects but on the contrary, they complement each other. Finally, recent research (Davidson, 1995) has shown that individuals are able to recognize the kind of interpretation that performers adopt (unexpressive, normal, exaggerated) based solely on visual information and, therefore, on the analysis of facial and body expressions (in particular of the joints and of the head). The connection between movement and sound production should not be surprising since music is produced through human and instrumental movement, and is therefore deeply affected by its origins, which arise in the body (Clarke, 2002). However, it is interesting to note that among the gestures defined in this study, the most significant facial action unit movement is eyebrow movement. In the study these movements are divided into the following categories: The analysis of performers’ movements during executions of musical pieces, as Delalande highlights (1993), can be accomplished on three different levels at least. The first level involves the functional aspect of gesture. In this case analysis regards so called productive gestures, which include the movements that performers have to accomplish in order to produce sounds mechanically. These are linked to sensomotory control aspects of the execution. fast movements (which correspond to pulsation); • movements that accompany melodies; • prolonged raising of eyebrows; • frowning. As to the links between musical elements and movements, Delalande has discovered: The second level of analysis relates to what are known as the accompanying gestures, the movements that a pianist carries out during a performance involving facial expressions, movements of the head, of the bust and of the shoulders. Delalande highlights that it is a simplification to state that these movements are exclusively connected to motor coordination. On the contrary, Delalande claims that their significance should also be ascribed to the mental processes that govern a performance. • eyebrow movements that separate one musical phrase from another; • eyebrow raising linked to pianissimo, staccato and high register; • frowning linked to forte or fortissimo, fast tempo, low register and dissonant passages. As regards performers, there are many functions of eyebrow movements because they are linked to the following: The third and final level of analysis that was determined, regards the figurative gestures that are evoked in the listener’s mind according to the music that the pianist is playing. For instance a swaying movement may be evoked by a certain melody, as well as lunging or leaning etc. • sensomotory and emotional control; • formal analysis of a work; • communication with the listeners. These observations concerning eyebrow movements should not be surprising, since numerous studies carried out among populations in different geographical regions, have shown that the communicative value of these movements is common to many cultures (Eibl Eibesfeldt, 2001). This final level of analysis shows that there is a relation between the sound produced by performers’ movements and the mental representations of these movements in the listeners’ minds. ISBN 88-7395-155-4 © 2006 ICMPC • 1356 ICMPC9 Proceedings The studies particularly highlighted that frowning and eyebrow raising have a wide range of values within social interaction. In fact, frowning can either carry a protective or an aggressive meaning, and eyebrow raising may express a greeting, it plays a role in mother and child interaction and in seduction, and it is also closely linked to verbal discourse and persuasive communication (Costa & Ricci Bitti, 2003). by the pianist when the eyebrow movement that we observed, occurred. Being aware of the many levels on which a score can be analysed, during this phase of the research it was deemed necessary to establish very simple criteria. The criteria we used also aimed at finding music-structural features that had already been highlighted by Delalande’s study: the ascending and descending movement of the notes, the occurrence of a theme or of the repetition of a theme, the passage between two different themes, the occurrence of ornaments (very brief notes) and prolonged notes or chords. While it is true that the decision to analyse great musicians’ performances meant that we could not obtain information regarding the kind of formal analysis they had been chosen for the execution of the musical pieces, this method has, on the other hand, allowed us to gather important information. METHOD Vidorecordings of famous performers were analysed in order to study the relation between specific facial action unit movements (eyebrow movement) during piano performances and music-structural features. More specifically the following videorecordings were used: 1) Glenn Gould Before describing our observation, it is important to specify that in the grid above, the eyebrow movements we observed were classed as rapid when the eyebrows were raised and lowered in fast succession, whereas all other movements were defined as prolonged. Johann Sebastian Bach, Partita in E minor BWV 830 (duration 26 min.). Taken from: “Glenn Gould, The Alchemist”, film by Bruno Monsaingeon (Toronto, 1972). 2) Vladimir Horowitz This distinction was considered necessary because while rapid movements occurred when a precise element was present (often a single note), prolonged movements were related to melodic fragments or whole musical phrases. Fryderych Chopin, Sonata n. 2 in B flat minor op. 35 "Marche funèbre" (duration: 25 minutes). A public concert that was held at the White House (1931). 3) Bruno Canino Claude Debussy, Preludes (“Generale Lavine Eccentric” , “Le feés sont d’exquises danseuses” -2° Vol.- and “La fille aux cheveux de lin”- 1° Vol.) (duration: 15 min). This videorecording was filmed during two different concerts: in a public concert which was broadcast by Italian Television (first performance), and in a second performance during concert trials that were held in the Bossi Hall in the Conservatoire of Bologna (2004). RESULTS The following four tables show the correspondence between eyebrow raising and musical features for each performer. Videorecording didn’t allow us to collect data systematically, thus these results are preliminary and have a qualitative value. 4) Daniel Barenboim The following table (Tab.1) shows the correspondence between eyebrow raising and musical features in Gould’s performance. Claude Debussy, Preludes (1° Livre) (duration: 59 minutes). Taken from : “Entre Quatre-z-Yeux”, film by Paul Smaczny (1999). Table 1. Glenn Gould: correspondence between eyebrow raising and musical elements (J. S. Bach, Partita in E min. BWV 830) The video analysis was carried out by two independent judges according to a grid that had been specifically designed for the purpose. The grid contained some information regarding eyebrow movement. More specifically the following characteristics were indicated in the grid: Eyebrow Movements • Eyebrow movements (frowning or raising); • Types of eyebrow movements (rapid or prolonged). Furthermore, in the grid we also noted down some characteristics of the execution (articulation and dynamics) and some characteristics of the score (register) that coincided with the eyebrow movements that we observed. Music-structural features Raising Prolonged Finally, in the last column of the grid, we indicated some of the music-structural features of the score that were played ISBN 88-7395-155-4 © 2006 ICMPC Articulation Dynamic Register Prolonged 1357 Clearly articulated mf Medium register Staccato mf Descending gression pro- Ascending notes Ornaments ICMPC9 Proceedings Prolonged Prolonged Rapid Rapid Two rapid raising in succession Medium-high register Legato mf Medium-high register Legato p Medium-.low register Legato mf Medium register Clearly articulated f Medium-high register Staccato p Medium-high register Table 3. Bruno Canino: correspondence between eyebrow raising and musical elements (C. Debussy, three Preludes, 1° and 2° Vol.) Articulation Eyebrow Music-structural Dynamic features movements Register Prolonged high note Prolonged chord Raising Chord (ascending arpeggio) Legato Prolonged Higher note of ascending arpeggio Prolonged End of trill Prolonged Two ascending notes of melodic fragment mf Medium-low register Legato mf Medium- high register Staccato p Medium register Repetitions of descending melodic fragments Ascending chords Fast repetition of chords Fast notes The following table (Tab.4) shows the correspondence between eyebrow raising and musical features in Barenboim’s performance. The following table (Tab.2) shows the correspondence between eyebrow raising and musical features in Horowitz’s performance. Table 4. Daniel Barenboim: correspondence between eyebrow raising and musical elements (C. Debussy, Preludes , 1°Vol.) Articulation Eyebrows Music-structural Dynamic movements features Register Raising Legato Repetition of the mf theme Prolonged Medium-high Prolonged notes register Legato mf High register note Prolonged Medium-high (prolonged) register Legato Fast ascending mf scale Prolonged Medium-high register Tab. 2. Vladimir Horowitz: correspondence between eyebrow raising and musical elements (F. Chopin, Sonata n. 2 in B flat minor op. 35 "Marche funèbre") Articulation Eyebrow Music-structural Dynamic movements features Register Raising Legato mf Second repetition Prolonged Medium-high of the theme register Staccato p Caesura between Prolonged Medium-low two themes register Legato mf Higher note of Rapid Medium-high melodic fragment register The following four tables show the correspondence between frowning and musical features for each performer. The following table (Tab.3) shows the correspondence between eyebrow raising and musical features in Canino’s performance. ISBN 88-7395-155-4 © 2006 ICMPC The following table (Tab.5) shows the correspondence between eyebrow frowning and musical features in Gould’s performance. 1358 ICMPC9 Proceedings Table 5. Glenn Gould : correspondence between eyebrow frowning and musical elements (J. S. Bach, Partita in E min. BWV 830) Articulation Eyebrow Music-structural Dynamic features Movements Register Frowning Clearly articulated Left hand: chords Prolonged f (ascending Medium-low arpeggios) register Staccato End of melodic mf fragment Prolonged Medium-low (notes with legato ) register Register Frowning Legato Prolonged Medium-high register Table 8. Daniel Barenboim: correspondence between eyebrow frowning and musical elements (C. Debussy, Preludes , 1° Livre) Articulation Eyebrow Music-structural Dynamic Regfeatures movements ister Frowning Prolonged note Legato Lower note of demf Prolonged scending melodic Medium–low fragment register Prolonged chord Staccato f Prolonged Fast notes Medium-low register The following table (Tab.6) shows the correspondence between eyebrow frowning and musical features in Horowitz’s performance. One result which is particularly worth highlighting, is the connection between eyebrow raising and register. In fact, eyebrow raising generally appears related to medium and medium-high registers. Tab. 6. Vladimir Horowitz: correspondence between eyebrow frowning and musical elements (F. ChopinSonata n. 2 in B flat minor op. 35 "Marche funèbre") Articulation Eyebrow Music-structural Dynamic movements features Register Frowning Legato Repetition of the mf theme “Andante” Prolonged (with harmonic Medium register modulation) Legato Last presentation P and crescendo of first funereal Prolonged Medium-low theme register Legato Note in suspense P (with dissonant Rapid Medium-high chord ) register With regard to dynamics, we see that eyebrow raising occurs very rarely in correspondence with a forte, while there seems to be a stronger link with piano and mezzo forte. On the other hand, there appears to be no particular correlation between the articulation performers adopt and eyebrow raising. With regard to the musical elements we noted, eyebrow raising seems more strongly correlated with ascending melodic lines, specific highlighting of high pitch notes, ornaments and prolonged notes and chords that are characterized, from an expressive point of view, by a feeling of suspense and awaiting. As regards frowning, our observations show that there is a correlation between this movement and medium or medium-low registers. While concerning dynamics, contrary to eyebrow raising, there appears to be a more frequent correspondence with mezzo forte and forte. Another interesting fact is that legato articulation generally appears linked to frowning. In the performances we analysed, from an expressive point of view, the musical characteristics we have just mentioned conveyed a sense of power, energy and sometimes of pain. In addition, a very interesting fact that emerged from our observations is that, as the above tables show, eyebrow Finally, the following tables (Tab.7, Tab. 8) show the correspondence between eyebrow frowning and musical features in Canino’s and Barenboim’s performances. Table 7. Bruno Canino: correspondence between eyebrow frowning and musical elements (C. Debussy, three Preludes, 1° and 2° Vol.) Eyebrow Articulation movements Dynamic ISBN 88-7395-155-4 © 2006 ICMPC f Descending melodic fragments of the theme Music-structural features 1359 ICMPC9 Proceedings movements are sometimes connected to the articulation of a musical phrase. This is especially noticeable in Glenn Gould’s performance when he plays Bach’s Sixth Partita, a piece of music that is characterized by counterpointing. This link suggests an analogy between the expressive function of a performer’s eyebrows during a performance and their significance in the spoken language. Specific studies have in fact shown that eyebrow movements are not only used in persuasive communication, but in every day conversations, too, for instance to underline semantically relevant word, or before speaking, in order to attract the listener’s attention. Costa, M., & Ricci Bitti, P.E. (2003). Il linguaggio delle sopracciglia. Psicologia contemporanea, 176, 38-47. Davidson, J. W. (1995). What does the visual information contained in music performances offer the observer? Some preliminary thoughts. In R. Steinberg (Ed.), Music and the mind machine. Psychophysiology and psychopathology of the sense of music. Berlin: Springer. Delalande, F. (1993). Le condotte musicali: Comportamenti e motivazioni del fare e ascoltare musica. Bologna: CLUEB. Eibl- Eibesfeldt, I. (2001). Etologia umana, le basi culturali e bioloogiche del comportamento. Torino: Boringhieri. CONCLUSION Ekman, P., & Friesen, W. (1978). Facial Action Coding System. Palo Alto: Consulting Psychologist Press, Inc. To sum up, we should point our that some characteristics of the material employed in the study (stylistic differences of the pieces performed, monitoring of the footage on which the observations were carried out, and each pianist’s personal interpretation) do not allow for generalisations. Miklaszewski, K. (1989). A case study of a pianist preparing a musical performance. Psychology of Music, 17, 95-109. Poggi, I. (2001). The lexicon of the conductor’s face. In P. McKevitt, S. Nuallàin, & C. Mulvihill (Eds.), Language, Vision and Music. Amsterdam: John Benjamins. Ruggieri, V., Ragni, S., & Terrezza, L. (2002). Psychopsychological mechanismus of artistic expression and musical performance: A psycho-physiological inquiry. In J. Fachner, & D. Aldridge (Eds.), Dialogue and debate. Proceedings of the 10th World Congress on Music Therapy, Oxford, UK, July 23-28, pp.1358-1399. However, even if this study is a preliminary one, the tables indicate interesting relationships between eyebrow movements and the musical elements that are executed in concurrence with these movements. Future development of this study will involve the integration of the data so far gathered regarding eyebrow movement, with other significant non verbal indexes of pianists’ bodies and faces, with the aim of verifying the existence of expressive configurations or gesture types, in accordance with Delalande’s theories. Rubin-Rabson, G. (1939). Studies in the psychology of memorizing piano music: A comparison of the unilateral and the coordinated approaches. Journal of Educational Psychology, 30, 321-345. ACKNOWLEDGMENTS We thank Dr. Mariateresa Storino for her precious collaboration in the video analysis of the performances. We are also especially grateful to Prof. Marco Costa for offered support. Shove, P., & Repp, B. (1995). Musical motion and performance: theoretical and empirical perspectives. In J. Rink (Ed.), The practice of the performance, (pp. 55-83). Cambridge: Cambridge University Press. REFERENCES Caterina, R., Bonfiglioli, L., Baroni, M, & Addessi, A. (2004). Mimic expression and piano performance. In S.D. Lipscomb, R. Ashley, R.O. Gjerdingen, & P. Webster. Proceedings of 8th International Conference on Music Perception and Cognition, Evanston, August 3-7, pp.258-261. Camurri, A., Mazzarino, B., & Volpe, G. (2003). Analysis of expressive gestures in human movement: The Eyesweb Expressive Gesture Processing Library. Proceedings of the XIV Colloquim on Musical Informatics, Firenze, Italy, May 8-10, pp.54-58. Clarke, E. (2002). Processi cognitivi nell’esecuzione musicale. In J.J. Nattiez, M. Bent, R. Dalmonte, & M. Baroni (Eds.), Enciclopedia della musica. Il sapere musicale (pp.288-304). Torino: Einaudi. ISBN 88-7395-155-4 © 2006 ICMPC 1360