A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances CHIKA OSHIMA National Institute of Information and Communications Technology and ATR Cognitive Information Science Labs KAZUSHI NISHIMOTO Japan Advanced Institute of Science and Technology and NORIHIRO HAGITA ATR Media Information Science Laboratories In this article, we propose “Family Ensemble,” a piano duo support system for a musically inept parent and his/her child who is a beginner at playing the piano. The system makes it easier for parents to correctly reproduce a given sequence of pitches along with the child’s performance by using score tracking and note-replacement functions. The experiments with this support system showed that the parents can immediately participate in the piano duo. Furthermore, we found that during joint practices using Family Ensemble some subjects discussed musical ideas that they would not have talked about without using the system. Categories and Subject Descriptors: H.5.5 [Information Interfaces and Presentation]: Sound and Music Computing—Systems; K.3.1 [Computers and Education]: Computer Uses in Education—Collaborative learning General Terms: Experimentation Additional Key Words and Phrases: Piano duo, musical expression, score tracking, entertainment, support system ACM Reference Format: Oshima, C., Nishimoto, K., and Hagita, N. 2007. A piano duo support system for parents to lead children to practice musical performances. ACM Trans. Multimedia Comput. Commun. Appl. 3, 2, Article 9 (May 2007), 21 pages. DOI = 10.1145/1230812.1230815 http://doi.acm.org/10.1145/1230812.1230815 1. INTRODUCTION In this article, we propose a collaborative musical performance support system named “Family Ensemble” (FE) [Oshima et al. 2004], which allows musically inept parents to play piano duos with their children who are beginners at the piano. Nishimoto’s work was partially supported by the Ministry of Education, Science, Sports, and Culture, Grant-in-Aid for Scientific Research (C), 16500580, 2004, and was partially supported by Hayao Nakayama Foundation for Science and Technology and Culture. Hagita’s work was supported by the National Institute of Information and Communications Technology (NICT). Authors’ addresses: C. Oshima, ATR Cognitive Information Science Labs. 2-2-2, Hikaridai, Keihan’na Gakken Toshi, Kyoto, JP 619-0288; email: [email protected]; K. Nishimoto, Japan Advanced Institute of Science and Technology 1-1, Asahidai, Nomi, Ishikawa, JP 923-1292; N. Hagita, ATR Media Information Science Labs. 2-2-2, Hikaridai, Keihan’na Gakken Toshi, Kyoto, JP 619-0288. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or [email protected] c 2007 ACM 1551-6857/2007/05-ART9 $5.00 DOI 10.1145/1230812.1230815 http://doi.acm.org/10.1145/1230812.1230815 ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. 2 • C. Oshima et al. Generally, the child has an opportunity to play the piano duo with her teacher or friends, since the efficacy of duo play has been recognized in piano playing education. However, in practice, most children can play piano duos only in school since often parents cannot play. Using FE, the parent can become a lesson mate at home rather than a proxy teacher, encouraging the child to practice more enjoyably and more musically. By compensating for the difference between the parent and child, FE not only provides entertainment, but also promotes enthusiastic discussion of musical expression. The parents are readily able to always output correct notes by hitting any keys (we call it a notereplacement function). Moreover, FE always tracks the position where the child is playing at each point in the score and automatically synchronizes the position where the parents should perform. Therefore, the parents are readily able to output notes that correspond to the position where the child is (unexpectedly) performing now (we call it a score tracking function). FE is an entertainment system that supports the parent by allowing him/her to play piano duos with his/her child at the same level of piano playing as the child. FE only supports the parent, by providing a note-replacement function for the parent’s performance and a score tracking function for the child’s performance. The note-replacement function only replaces the pitches of notes performed by the parent to the correct pitches that should be output. However, this function does not modify any other elements, for example, when and how strongly the keys are hit and when and how quickly the keys are released. As a result, while parents become readily able to always output correct notes by hitting any keys, even with only one finger, they can also fully control the other elements to achieve their own musical expression. Thus, this function supports parent’s ability while preserving sufficient room for musical expression. Similar systems have already been proposed (e.g., Subotnick  and Takeuchi and Katayose ) and some of them have already been on the market (e.g., Casio ). Though these systems aimed at mainly supporting solo performances, we applied a similar mechanism for supporting piano duo performances. Regarding duos with the child, however, it is inadequate to merely sequentially replace the pitches along with the parent’s score. When the child makes a mistake and goes back to an earlier point in the music, the parent gets lost and they have difficulty getting synchronized again. To solve this problem, we implemented a score tracking function that always tracks the position where the child is playing at each point on the score. We discovered that there are four types of mistakes in children’s piano performances through analysis of the features of the actual performances of children. Based on these results, we implemented a robust score tracking algorithm that can quickly “catch up” with a child’s performance even if he/she makes mistakes. As a result, even if the child suddenly makes mistakes, it is not necessary to stop to synchronize both performance positions again. They can keep on performing and then the note-replacement function can correctly work again after minimum unavoidable errors.1 Thus, by combining these two functions, FE allows the parent to readily perform duos with his/her child at home at almost an equal level as the child. Moreover, since FE only helps parents reproduce the correct sequence of pitches, they can enjoy externalizing their musical expression as well as readily playing the music. Thereby, the child and parent are encouraged to discuss music and to collaboratively create a duo performance. In an ensemble, each performer learns the partner’s good expression and finds his/her mistakes or poor expression by comparing his/her own performance with a partner’s. Piano students make suggestions to each other for improvement in ensemble practice [Lyke 2004], discussing musical expression to explore an ideal goal [Oshima and Shimojima 2003]. Technologies for automatically tracking human musical performances have been studied mainly in terms of automatic accompaniment systems. Dannenberg  developed an algorithm for tracking a 1 Many piano teachers pointed out that the performer in an ensemble should keep on going, even if he/she misses a note or a beat even in a practice [Zeigler and Ostromencki 2004]. ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances • 3 score based on pitch sequence data by applying a DP-matching method. Vercoe  developed a score tracking method that referred not only to the pitch sequence but also to the time each note sounded. In addition, a human cognition model for estimating musical performance in an ensemble has been considered. Studies of an automatic jam session system attempted to follow the emotive transition of human performer(s) [Goto et al. 1996]. Although FE’s score tracking function is an extension of Dannenberg’s algorithm, as described below, all of these previous studies aimed to enable ensemble play by advanced human performers and machines. In contrast, FE seeks to enable ensemble play by human pairs with little and no musical experience. Regarding the study of interactive musical systems, the Continuator [Pachet and Addessi 2003] system supports users in learning musical styles. When the user plays a musical sequence of any kind (i.e., a phrase), the Continuator in turn generates a musical phrase in response to the user’s phrase based on a Markov model of the musical phrase. Experimental results showed that participants played the system in a self-motivated way. The Jum-O-Drum [Blain and Perkis 2000] is a system that allows up to six players to practice collaborative musical improvisation. FE helps to fill a gap in mechanical skills—reproducing correct pitches—between the members so that the performer who has no experience of playing the piano can participate in a piano duo as well as discuss musical expressions with others. In the next section, we give an overview of related works. Section 3 describes the setup of FE in detail. Section 4 describes the note-replacement function, and Section 5 describes the score tracking function. Section 6 illustrates experiments for evaluating FE efficacy and shows the results. In Section 7, we discuss the FE feature, which can encourage players to richer, more expressive performances. Section 8 concludes this article and outlines the future works. 2. RELATED WORKS Recently, in piano playing education the efficacy of duo play, which is a kind of ensemble, in the learning process has come to be widely recognized [Agay 1987; Kihara 1991; Miyoshi 1977]. Practicing the piano alone is painful, and raising a child’s motivation to practice is difficult. However, by incorporating piano duos into lessons, children come to practice more enthusiastically and perform more musically [Miyoshi 2000]. This is because, first of all, an ensemble performance is essentially fun entertainment. In addition, practice and rehearsal time with other musicians involves a very rewarding experience [Zeigler and Ostromencki 2004]. Such ensembles can be regarded as collaborative learning methods as well as edutainment methods. In general collaborative learning, learners address an issue together to acquire knowledge while encouraging and stimulating each other. Teachers use ensemble music as a tool to improve the students’ rhythmic and harmonic awareness [Everett 1999]. A performer who plays in an ensemble must perfectly synchronize his/her playing with a partner by listening to the partner’s performance. The performer learns to respond spontaneously to another person’s musical cues or flubs [Zeigler and Ostromencki 2004]. Throughout the ensemble performance, such musical expressions as tempo, dynamics, duration, and so on, must be consonantly controlled. As a solution to lacking partners outside of lessons, various “minus-one” software applications exist in which all parts except for the pupil’s are recorded [Kihara 1991]. Regrettably, music professors also comment that minus-one software prevents pupils from acquiring the cooperative performance skills essential for ensembles [Miyoshi 2000]. Moreover, they cannot perform subtle changes in tempo and dynamics, because minus-one plays continuously independently of the child’s performance. Another solution, automatic musical accompaniments [Casio 2000; Dannenberg 1984; Raphael 2001; Vercoe 1984], which allow a computer to function as a musical accompanist, have been studied. They follow the human performance even if he/she makes little mistakes or subtle changes in tempo and dynamics. ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. 4 • C. Oshima et al. Fig. 1. Family Ensemble Setup. However, since these systems just track the performance position of the human performer, the performer cannot learn how to play cooperatively. Moreover, the human player cannot discuss music with a machine. pianoFORTE [Smoliar et al. 1995] records and displays several aspects of a user’s performance. It aims to have teachers and students discuss the distinctions between the art of playing the piano and the technique of playing the correct notes. DrumSteps [McCarthy et al. 2005] is a dynamic graphical interface that enables children to create, manipulate, edit, and save original pieces of percussion music. Toy Symphony [Jennings 2003] is an international music performance and education project that allows children to learn by playing alongside some of the world’s most accomplished musicians. Several interesting tools or instruments have been developed in this project. For example, Music Toys are specially developed musical instruments that allow anybody to easily enjoy musical performances. Hyperscore [Farbood et al. 2004], a tool that enables children to “compose by drawing,” is designed to intuitively and dynamically introduce children to musical composition and creativity. FE provides no special tools to learners, unlike these systems. 3. SYSTEM SETUP OF FAMILY ENSEMBLE Figure 1 shows the setup of Family Ensemble (FE). FE directly supports the parent’s part in playing a piano duo with his/her child. We call the parent part “Secondo,” that is, the part that performs the lower register of a duo. On the other hand, the child’s part is called “Primo,” the part that takes charge of the upper register of the duo. Primo’s performance is output as Musical Instrument Digital Interface (MIDI) data and directly input to a sound generator as the notes are played. At the same time, Primo’s performance data are input to the score tracking module, which determines Primo’s current position in the score. The obtained current position data are input to a note-number-acquisition module that compares Primo’s current position with Secondo’s score. As a result, the MIDI note number(s) that Secondo should play at that moment is (are) acquired. The acquired MIDI note numbers are input to the note-number-replacement module, which replaces the MIDI note number(s) that Secondo played ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances • 5 with the acquired correct MIDI note number(s), while preserving other characteristics of the performed note(s): loudness and length of sound performed. Consequently, by using any keys, Secondo can perform the correct notes with his/her own expression, alongside Primo’s performance. If Secondo momentarily stops playing while Primo continues performing, Secondo can immediately resume the performance from the correct place in the score since FE continues to track Primo’s position regardless of the progress of Secondo’s performance. On the other hand, if Primo suddenly stops playing while Secondo continues performing, Secondo cannot proceed beyond the point where Primo stopped. When Primo resumes the performance, Secondo resumes, too. Even if Primo returned several notes back when resuming, Secondo can quickly catch up. Additionally, users can arbitrarily decide the boundary between Primo’s and Secondo’s regions on a keyboad. Thus FE includes two important supporting functions: a note-replacement function in the notenumber-replacement module and a score tracking function in the score tracking module. The notereplacement function allows Secondo to readily reproduce an accurate sequence of pitches, which is necessary, and one of the most significant initial barriers for novices. Additionally, it is important that Secondo can still freely and fully express his/her musical intention and emotion even though being supported. Section 4 describes the note-replacement function in detail. The score tracking function always tracks a position on the score where the child is playing to obtain pitches that Secondo should play at that moment. Section 5 describes the score tracking function in detail, wherein even if the child makes some mistakes, the piano duo continues. Note that Primo, his/her child, plays the piano normally since these functions do not support Primo’s part. 4. NOTE-REPLACEMENT FUNCTION In this section, we will demonstrate that the note-replacement function is effective in readily playing and externalizing the performer’s musical intention and emotion. 4.1 Expressive and Nonexpressive Elements Generally speaking, from the performer’s perspective a musical performance consists of controlling two kinds of elements: non-expressive and expressive. In performing “reproduction type music,” such as classical music, performers must accurately reproduce the pitches and the sequence of pitches along given scores; modifying them is prohibited. The performers cannot demonstrate their own expression by the pitches and their sequences. Therefore, these elements are nonexpressive elements. On the other hand, each note has many other characteristics, for example, loudness, and length of sound. Individual performer expression is reflected in the control of these elements. Therefore, these elements are expressive elements. The difficulty of dealing with nonexpressive elements causes cognitive and physical overload for performers and prevents them from concentrating on expressing individual musical intentions and emotions. Furthermore control of nonexpressive elements is usually unavoidable before tackling the control of expressive elements when using conventional keyboards. Therefore, allowing performers to directly control expressive elements facilitates the demonstration of their musical expression (see Figure 2). 4.2 Effectiveness of the Note-Replacement Function Based on the aforementioned idea, several piano performance supporting systems have been developed [Subotnick 1997; Takeuchi and Katayose 1995; Casio 2000]. These systems employed note-replacement functions that alleviate the difficulties of controlling nonexpressive elements in the performance of reproduction type music, the accurate reproduction of pitch sequence. We implemented a similar system to support the musically inept performers. We illustrate that the musically inept performers and ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. 6 • C. Oshima et al. ~Attribute~ ~Support~ Volume Duration Expressive elements Non-support These elements are input concurrently* Attenuate Pitches, the sequence of pitches Non-expressive elements Computer supports *In a step-by step input system, a user also can input pitches individually. However, he cannot input the expressive elements concurrently like the conventional piano. Fig. 2. Basic idea for supporting reproduction type music performance. beginners can immediately and accurately reproduce the given sequence of pitches (melody) by the notereplacement function, and also sufficiently express their musical intentions and emotions(see Sections 4.2.1 and 4.2.2). Furthermore, we also show that the note-replacement function provides the performer with enough room for musical growth(see Section 4.2.3). 4.2.1 Experiments. In this section, we estimate the effectiveness of the note-replacement function. We compared the performance by using the note-replacement function (we call this way of performance “NR method,” hereafter) with the normal performance without using any supporting functions (we call this way of performance a “normal method,” hereafter) . We recorded all performances by both methods as MIDI sequence data. In the NR method, at first, without considering any expressive elements, the performer inputs the simple sequence of MIDI note numbers as numerical values in a computer. Then using these data, the performer plays the piece on a keyboard to concurrently input all the expressive elements. In the normal method, sequence data is basically input by normally performing a musical instrument that outputs MIDI data. Therefore, all of the elements, both expressive and nonexpressive, are concurrently input. We conducted experiments using subjects as performers to compare the NR method with the normal method. We employed eighteen undergraduate and graduate school students with little or no experience playing the keyboard. The subjects were asked to play “Akatonbo (a red dragonfly),” which is a very famous Japanese children’s song, by the two methods. Since this piece is made in a range of eleven notes, performers have to change their finger positions and pass the thumb through the other fingers along the way. Therefore, it is a difficult piece for beginners to play. After the subjects listened to “Akatonbo” sung by a professional vocalist [Toyoda 1995], they were asked to record a performance of “Akatonbo”as MIDI sequence data by the two methods as expressively as possible, emulating the professional vocalist. Before the experiments we explained both methods and had them practice the methods with another small piece. Each subject provided a subjective evaluation for each method after recording by both of the methods. The subjects evaluated their degree of satisfaction (1: unsatisfied to 5: very satisfied) with each performance and the difficulty (1: very easy to 5: very difficult) of controlling four elements: melody, rhythm, agogics, and dynamics, by each method. Here, melody means the sequence of pitches, rhythm and agogics are shown by the length of each note, and dynamics means the change of volume. ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances • 7 Table I. Average Evaluation Values for Two Methods Elements melody rhythm agogic dynamics emotional burden Evaluation difficult satisfied difficult satisfied difficult satisfied difficult satisfied NR (ave.) 1.89 3.22 2.22 3.39 2.50 3.17 2.44 3.28 2.28 Normal (ave.) 3.44 2.56 3.67 2.83 3.44 2.50 3.56 2.56 3.22 t-value 3.50∗∗∗ 2.13∗∗ 3.71∗∗∗ 1.82∗ 2.88∗∗ 2.06∗ 2.60∗∗ 2.06∗ 2.12∗∗ In difficulty, 1: very easy to 5: very difficult. In satisfaction, 1: unsatisfied to 5: very satisfied. (∗∗∗ ) means a significant difference at 1%. (∗∗ ) means a significant difference at 5%. (∗ ) means a significant difference at 10%. 4.2.2 Results. Table I shows the average evaluation values for the NR method and normal method. The results of a t-test indicate that the NR method is significantly easier than the normal method and significantly lighter than the normal method in cognitive burden. Moreover, the subjects tended to be more satisfied with the performances using the NR method than those with the normal method. A feature of the NR method is that nonexpressive elements can be input as numerical values separated from expressive elements. This allows people to readily reproduce the correct sequence of pitches without worrying about expression. Therefore they can achieve better expression than with the normal method because the nonessential load for accurate reproduction of melody is alleviated. As a matter of fact, the results of the satisfaction rate with expressive items: rhythm, agogics, and dynamics, using the NR method are higher than the normal method. On the other hand, all of the expressive elements are tightly related; balance among them is very important for achieving good musical expression. Therefore, all of the expressive elements should be concurrently controlled, as they are in the normal method. 4.2.3 Evaluation at a Contest. To investigate whether the note-replacement function has enough space for musical growth, we entered the performance data performed by the NR method in a contest for performance rendering systems named “Rencon [Rencon 000 ].2 ” This international event aims to establish evaluation methods for performance rendering systems that automatically generate expressive musical performance. We employed one subject as a performer, a forty-year-old male who studied electric organ for one year. He can read a score. However, he had never heard the set piece, “Waltz Op.69-2” by F. Chopin. Of course, he could not play this piece on a conventional piano (i.e., the normal method) even after the experiment. The first author input to the system the sequence of pitches of the right-hand part from the 98th bar to the 145th bar of the piece in advance. Then, we asked the subject to play the set piece with the NR method on the piano using the pitch sequence data, and we recorded the performance as MIDI sequence data. After glancing over the score, he played the first performance. We asked him to practice untill he was satisfied with his performance. He practiced the set piece for a total of forty minutes over two days. He was satisfied with the 13th performance. We recorded all of the performances and submitted the 1st performance (called “CiP3 -1”) and the 13th (final) performance (called “CiP-2”) to the Rencon contest. 2 FIT 2002 Rencon Workshop: Toward the Number One Pianist in the World, June 30 2002, in Japan. In the contest, we did not need to compose the sequence data on the spot. Recorded data were evaluated. 3 “CiP” means “Coloring-in Piano,” which was our system’s name at Rencon. ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. 8 • C. Oshima et al. A total of ten systems entered Rencon. Sixty-three judges (not professional musicians) evaluated the ten performances for naturalness and subjective preferences. One of the semi-automatic rendering systems won 1st prize and two of the fully automatic rendering systems won 2nd and 3rd prizes. CiP-2 got 4th prize and CiP-1 8th place. CiP-2 was evaluated much higher than CiP-1, showing that the note-replacement function provides the performer with enough room for musical growth and that the performer can quickly improve his/her performance in a short period. 4.3 Concluding This Section From the results illustrated in 4.2.2 and 4.2.3, we can conclude that supporting the control of nonexpressive elements alleviated the performers’ cognitive load when playing the piano, as well as provided enough room for the performers to grow musically. Eventually, the note-replacement function raised the musically inept performer’s ability to play reproduction type music. Therefore, by applying this function to support the parent who has no experience with playing the piano, he/she becomes an equal partner of the child and can start to discuss musical ideas during joint practice. 5. SCORE TRACKING FUNCTION The note-replacement function can make even a musically inept parent a partner with the child. However, it is not sufficient to merely provide this function to parents for supporting duo performances. The replacement function monotonously replaces the performed pitch by the parent with the correct pitch in the note sequence data one by one. Therefore, when Primo, who is using a normal piano, and Secondo, who is supported by the note-replacement function, play an ensemble together, Primo can never make mistakes; if Primo makes mistakes, for example, some missing notes or extra notes, the playing position immediately becomes inconsistent. Accordingly a support function for solving this problem called the “score tracking function,” always determines where Primo is playing on the score. The parents are able to output the notes that correspond to the position where Primo is even unexpectedly, performing now. As a result, Secondo is always able to consistently perform with Primo even if Primo suddenly makes a mistake. As for our objective in this article, the child is Primo, using a normal piano. Therefore, in 5.1, we analyze the features of the novice child’s errors. Then, we describe the details of the score tracking function in 5.2. 5.1 Analysis of Beginners’ Errors Several score tracking systems [Dannenberg 1984; Grubb and Dannenberg 1997; Orio et al. 2003] have dealt with three types of performance errors, as defined by Bloch and Dannenberg . (1) extra: An extra note that does not match any notes in the score is performed. (2) wrong: Although a performed note matches a note in the score, its pitch is incorrect. (3) missing: A note in the score is not performed. They assume that their target users are experienced performers whose performances closely match the scores. Therefore, when the performers make an error, the system only has to check one or a few note(s) before and after the position where the performer made the error. On the other hand, our system, FE, has to be adapted to undertrained performers: children who are beginners. To examine the kinds of errors that beginners are apt to make, we conducted experiments with two children studying the piano. Subject A had studied the piano for four years. She is a fourth year elementary school student. Subject B had studied the piano for one year. She is in nursery school and the sister of Subject A. Subject A was asked to perform seventeen new pieces in four, thirty-minute ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances • 9 Table II. Error Frequency extra wrong missing replaying Sum Place —– —– —– Start of piece Start of column Start of phrase Alternation 1st beat 2nd downbeat Last beat 1st beat (pre-bar) Last beat (pre-bar) —– A 1 0 3 7 1 5 4 315 17 0 26 2 381 B 6 37 0 3 1 0 0 141 2 6 25 3 224 Sum 7 37 3 10 2 5 4 456 19 6 51 5 605 % 1.2 6.1 0.5 1.7 0.3 0.8 0.7 75.4 3.1 1.0 8.4 0.8 100.0 experiments. Subject B was asked to perform thirteen new pieces in four, thirty-minute experiments. They were asked to perform the pieces from one to five times according to the level of difficulty. Moreover, they were asked to perform each piece from the beginning to the end since we wanted to create a situation similar to duo play. It is not simple solo practice but similar to the duo practice where they have to aim to play through the entire piece. We recorded the sounds and utterances of these experiments on a DVD-RAM disk. From the results of the experiments, although it is not new to piano teachers at all, we found it necessary to add a fourth type of error in addition to the aforementioned three types: (4) replaying: An extra note sequence that matches a part of the score is performed. In other words, an undirected refrain is performed. It has been ignored in the conventional score-tracking systems up to now. In solo practice, we usually can see such partial practices as “replaying errors.” However, in our experiments, we asked the subjects to play through the pieces in advance. Consequently, “replaying” is not considered partial practice. In replaying, the number of notes that the performer played is more than the notes in the score, as in the case of extra error. However, the surplus notes of replaying errors are part of melody in the score, while those of extra errors are not found in the score. Table II shows the frequency of errors: extra, wrong, missing, and replaying. As for replaying, the places where the subject restarted as well as the frequency of each restart are shown. “Column” means a group of bars forming a line on a printed score although this category does not include the notes in “start of piece.” “Start of phrase” denotes the start of the slur mark, although this category does not include the notes in “start of piece” and “column.” “Alternation” denotes where a performer needs to alternate the right hand with left hand in one phrase, although this category does not include the notes in these categories. “1st beat” means that the subject restarted at the 1st beat of the same bar where the subject stopped her performance. “2nd downbeat” means the 2nd beat of a binary measure, the 3rd beat of a quadruple measure, and so on. “Last beat” means the 4th beat of a quadruple measure although it does not include the 2nd beat of a binary measure. “Pre-bar” means that the subject restarted at the bar preceding where the subject stopped performing. “1st beat,” “2nd downbeat,” and “last beat” do not include the notes in “start of piece,” “column,” “start of phrase,” and “alternation.” Subjects A and B made 381 and 224 mistakes, respectively. Surprisingly, the cases of replaying were 377 times in the 381 errors in Subject A’s performances, and 181 times in the 224 errors in Subject B’s. Moreover, “1st beat” accounted for about 75% of all errors. Thus, we need to consider not only extra, ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. 10 • C. Oshima et al. wrong, missing but also replaying errors for dealing with the novice childrens’ performances in the score tracking function. 5.2 Details of Score Tracking Function We designed the score tracking function in response to the results of Section 5.1. The score tracking module in FE (Figure 1) consists of this function. Let the number of notes in Primo’s score be N , where a chord is regarded as a note and is represented by its highest note, and let “performance time” be t j where j is the number of notes performed by Primo. Now, assuming that the current position at performance time t j −1 was decided at note Si (1 ≤ i ≤ N ) in the score, Primo performed note P j at performance time t j . In this case, our score tracking function determines the correspondence between the note in the score and the performed note P j based on the following algorithm: (1) First, Pitch(P j )—the pitch of the performed note P j , is compared with Pitch(Sk ) of each note Sk (1 ≤ k ≤ N ) in the score. Then, the weight of each note Sk at performance time t j , W (Sk , t j ), is calculated as follows: (a) if Pitch(Sk ) = Pitch(P j ), then W (Sk , t j ) := W (Sk−1 , t j −1 ) + 1 (b) else W (Sk , t j ) := W (Sk−1 , t j −1 ) − 1, where if W (Sk−1 , t j −1 ) − 1 < 0 then W (Sk , t j ) := 0 (2) If W (Si+1 , t j ) = W (Si , t j −1 ) + 1, then an error is detected. In this case the following correction algorithm is applied. (a) To correct a replaying error, let W (Sh , t j ) := W (Si+1 , t j ) − m, where Sh is the point in the score to which the beginner is likely to return when he/she stops playing, as enumerated in Section 5.1. At present, m is empirically set to 2. (b) To correct extra, wrong, and missing errors, let W (S g , t j ) := W (Si+1 , t j ), where S g is all notes from Si−r to Si+r . At present, r is empirically set to 2. (3) Finally, the score tracking module chooses Sk as the note whose weight W (Sk , t j ) is the largest in the current position at performance time t j . If there are multiple notes whose weights are equally large, the module chooses one note as follows: (a) If W (Si+1 , t j ) is the largest, then Si+1 is chosen as the current position. (b) If W (Si+1 , t j ) is not the largest, then: (i) Note Sk whose weight is the largest and nearest to note Si is chosen as the current position. (ii) If two notes Si−d and Si+d have the same (largest) weight and are both nearest to note Si , then Si−d is chosen as the current position. Figure 3 illustrates our score tracking algorithm. This algorithm achieves robust score tracking even for performances by children with little musical experience, because the function copes with replaying errors as well as the three ordinary errors. Figure 4 shows sample processing situations of the score tracking function. (a) shows results in cases in which the system corrects replaying errors. (b) shows results in cases in which the system does not correct replaying errors. Columns correspond to notes described in the score. Rows correspond to notes performed by Primo, P j . The hatched part of the performed notes shows where Primo replayed. Primo ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances • 11 Start Primo performed a note Pj at performance time tj k 1 Yes k > N? N is the number of notes in Primo’s score No Yes No Pitch(Sk) = Pitch(Pi) ? Sk is the kth note in the score Yes The weight of each note Sk at performance time tj W(Sk, tj) W(Sk-1, tj-1) + 1 The weight of each note Sk at performance time tj W(Sk, tj) No W(Sk, tj) 0 k+1 W(Si+1, tj) = W(Si, tj-1) + 1 ? where Si is the estimated performed note at performance time tj-1 Yes Si+1 is determined as the performed note at performance time tj W(Sh, tj) W(Si+1, tj) – m for each note Sh to which the beginners are likely to return. (m is empirically set to 2) W(Sg, tj) W(Si+1, tj) for each note Sg that is included between Si+r and Si+r (r is empirically set to 2) Are there multiple notes whose weights W(Sk, tj) (k=1~N) are the largest? The weight of each note Sk at performance time tj W(Sk-1, tj-1) - 1 k No W(Sk-1, tj-1) – 1 > 0 ? Yes Are there multiple notes Sk whose weights W(Sk, tj) are the largest and whose distances from the note Si are the nearest? No Yes Sk(k<i) is determined as the performed note at performance time tj Sk is determined as the performed note at performance time tj No Sk whose weight W(Sk, tj) is the largest is determined as the performed note at performance time tj End Fig. 3. Score tracking algorithm. ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. 12 • C. Oshima et al. Fig. 4. Results of score tracking. replayed “sol, la, sol, mi” although she should have played “mi, sol, sol, sol.” “mi” is the 14th note in the column. Furthermore, cells that include outline characters show the estimated current position determined by the score tracking function. Cells surrounded by a black border show the actual correct notes that Primo is playing at each point. In Figure 4(a), we can see that notes “sol” and “do” have weight “11” at the start of replaying. Beforehand, we had appointed these two notes to correspond to the start of bars. Accordingly, the system could follow Primo’s performance after mistakes of only three notes. On the other hand, in Figure 4(b) the system mistook thirteen notes before correctly following Primo’s performance. Thus, we see that our score tracking algorithm, which is added to correct replaying errors, is practical. ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances • 13 This algorithm is sufficiently robust as for usual musical pieces. In some extreme cases, however, it might not be able to recover after an error. For example, if one makes a mistake during the first eight bars of “One Note Samba” composed by A. C. Jobim, which consists only of notes of an identical pitch (F), it is impossible to find the correct position. The time values of notes should be considered when tracking such a piece. It is also difficult to track a piece that only consists of an identical motif, for example, a simple repeat of “do, re, mi, re” over and over. As for the processing time of this algorithm, although it scans the entire piece each time, pieces that beginners perform are usually short and simple. Practically, therefore, the total processing time can be ignored. 6. TRIAL EXPERIMENTS OF FAMILY ENSEMBLES To investigate the effects of FE for ensemble systems, we asked five pairs to practice duo play with/ without FE. 6.1 Purpose of Experiments The purpose of our experiments was to clarify following two questions: (1) Can even a musically inept performer who takes charge of the Secondo part play a duo with a beginner who takes charge of the Primo part? (2) Do even a musically inept performer and a beginner consider their musical expressions? We conducted experiments with five pairs and analyzed the effects of FE from the number of performances and dialogues in joint practices and the results of questionnaires. 6.2 Methods 6.2.1 Subjects. We employed the following five pairs of subjects: Pair A Primo was a female junior high school student who had studied piano for eight years (intermediate). Secondo was her father who had never played the piano, and so it was impossible for him to read the scores (musically inept performer). Pair B Primo was a female elementary school student who had played piano for four years (beginner). Secondo was her mother, who had studied the electronic organ for two years (extreme beginner). Pair C Primo was a nursery school child, the other daughter of the Secondo of Pair B. She had been studying piano for one year (extreme beginner). Pair D Primo was a male student at our institute who had studied the electronic organ for eight years (intermediate). Secondo was a male student at the same institute who had never played the piano, and so it was impossible for him to read the scores (musically inept performer). Pair E Primo was the same as Pair D. Secondo was another male student at the same institute who had never played the piano, and so it was impossible for him to read the scores (musically inept performer). 6.2.2 Setting and Set Pieces. Our experiments were conducted on an acoustic grand piano, YAMAHA Silent Ensemble Grand Piano Proffessional Model C5 that output MIDI data in a soundproof room at the second author’s institute. The piano was connected to a personal computer on which the FE system operated and to a sound generator. In both situations, with and without FE, we used the sound generator and the sound was output from speakers placed under the piano. In joint practice with FE, Secondo was allowed to use any keys below C3, which Primo did not need to use for playing the set pieces. Moreover, Secondo used a special score, a so-called “rhythm score” (Figure 5), in which a chord is represented by piled note symbols in numbers that correspond to the ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. 14 • C. Oshima et al. 2 4 Primo 2 4 Secondo 2 4 Fig. 5. Example of score for Family Ensemble. notes in the chord. Secondo can play a chord by hitting either only one key or multiple keys. By hitting multiple keys to play a chord, Secondo can separately control the volume and length of each constituent note in the chord. We prepared three set pieces for the duo play that the subjects all knew well: “Prelude Opus28-7” and “Waltz Opus18” by Frederic Chopin, and “Takibi” (“Bonfire”), a popular Japanese children’s song. According to each Primo’s ability, we decided which set piece to ask them to play, and asked each Primo to practice the given set pieces beforehand until they could perform them with few mistakes. We also asked the subject who played Primo for Pairs D and E to try to use minus-one software which is a prerecorded performance data of Secondo’s part. We prepared minus-one software for two set pieces, “Takibi” and “Prelude.” The set piece, “Takibi,” was prepared in three tempo patterns: 44 (slow), 60 (medium), and 72 (fast) beats per minute. Another set piece “Prelude” was also prepared in three tempo patterns: 72 (slow), 88 (medium), and 104 (fast) beats per minute. 6.2.3 Procedure of Experiments. We asked each pair to jointly practice the duos, both with and without the use of FE. We conducted the experiments without FE before the experiments with FE in all pairs. With FE, the mental load on Secondo’s playing is less than without FE. We wondered if Secondo tried joint practice without FE after that with FE, the difficulty of the performance on a conventional keyboard would be exaggerated. Accordingly, all pairs were first asked to try joint practice without FE. First, Pairs A, B, and E had joint practice for fifteen minutes to perform “Prelude,” and Pairs C and D had joint practice to perform “Takibi” without FE. Then we explained how to use FE and let Secondo try it, using a short trial piece. Next, all pairs had joint practice for fifteen minutes with FE using the same set piece as the former joint practice without FE. After that, Pair A had two more joint practice sessions with FE for ten minutes since Primo of Pair A could play all of the set pieces. Last, the Primo of Pairs D and E tried to perform “Takibi” and “Prelude” with minus-one software in three different tempos for each piece. 6.2.4 Questionnaires. After recording the joint practice with and without FE, the subjects answered the following questions. To both Primo and Secondo Question 1 Is it difficult for Secondo to perform a set piece without FE? (“1” is very easy, “5” is very difficult) Question 2 Why do you think so? (Optional) Question 3 Is it difficult for Secondo to perform the set pieces with FE? (“1” is very easy, “5” is very difficult) ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances • 15 Table III. Number of Performances in Joint Practice Sessions Without and With FE Pair A Pair B Pair C Pair D Pair E ∗ Piece Prelude Prelude Takibi Takibi Prelude Sum Without FE Duo Solo 0 0 5∗ 0 0 0 0 0 0 0 5 0 With FE Duo Solo 13 1 (Primo) 14 0 (Primo) 2 4 (Primo) 5 7 (Primo) 9 1 (Primo) 43 13 The Secondo played the right-hand part. Table IV. Answers to Questions 1, 3, 6, and 8 Pair A B C D E ∗ Part Primo Secondo Primo Secondo Primo Secondo Primo Secondo Primo Secondo Q1: Without FE Prelude 5 5 Prelude 4 5 Takibi 5 5 Takibi 5 5 Prelude 5 5 Takibi 1 2 1 1 5 2 2 2 2 2 Q3: With FE Prelude Waltz 1 3 4 2 4 — 2 — — — — — — — — — 2 — 2 — Q6: Use again? Yes Yes Yes Yes Yes — Yes Yes — Yes Q8: with whom? Father Daughter, Wife Sister Daughter Sister — Girlfriend Friend — Friend 1: very easy to 5: very difficult. Question 4 Why do you think so? (Optional) Question 5 How do you feel about Secondo being able to perform with only one finger? (Optional) Question 6 Do you want to play a duo with FE again? (“Yes,” “Maybe,” and “No”) Question 7 Why? (Optional) Question 8 Who do you want to play a duo with? Question 9 Please comment on any expected improvements, and so forth. (Optional) About minus-one software to the Primo of Pairs D and E Question 10 Could you perform the set pieces with the correct melody and rhythm? Question 11 Could you express agogics and fermata (subtle changes in tempo)? Question 12 Could you express dynamics? 6.3 Comparison of the Number of Performances, and Results of Questionnaires Table III shows the number of subjects playing duo and solo fifteen minutes in joint practice with and without FE. We counted the frequency of a pair or one of the performers playing from start to end even if there were many errors. Consequently, even if Secondo could only perform with his/her right hand in a duo, we considered it one duo, as shown in Pair B. Table IV shows the results of Questions 1, 3, 6, and 8, and Table V shows the results of Questions 10, 11, and 12 concerning the minus-one software. We see that none of the pairs could complete the duo in the joint practice without FE. On the other hand, with FE they immediately performed the duo more than ten times in the fifteen-minute session (see Table III). In the joint practice sessions of Pairs A, B, D, and E without FE, Primo was quite occupied telling Secondo which key(s) to hit. According to the answers to Question 2, especially the Secondos of Pairs A, D, and E did not know how to read notes or the correct position corresponding to each note in ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. 16 • C. Oshima et al. Table V. Answers to Questions 11, 12, and 13: with Minus-One Software Tempo Slow Medium Fast Q10: Melody, Rhythm No Marginal No Q11: Agogic No No No Q12: Dynamics No No No the score. Hence, the results of Question 1 in Table IV show that most Primos and Secondos thought it was very difficult to play a duo without FE (answers with “5”). Moreover, notice the increase in the number of solo practice sessions by the Primos in joint practice with FE (see Table III). In the answer to Question 9, the Primo of Pair D wrote that FE prompted him to practice playing the piano. Since Primos’ performances became poorer than those of Secondos supported by FE, Primos wanted to recover domination by practice. The Primo of Pair A told us that she felt frustrated that she could not blame her father when the performance was not perfect (see answers to Question 5). This result suggests that FE can encourage children who are studying the piano to practice. The subject who played the Primo of Pairs D and E tried to use the minus-one software in “Takibi” and “Prelude.” Table V shows that he could almost perform it in medium tempo. However, he answered that he could not perform the subtle changes in tempo and dynamics, the musicality, in any tempo (see answers to Questions 11 and 12). According to the answers to Questions 4 and 7, the Secondos of Pairs D and E felt that it was wonderful that they could perform the pieces from the start to the end. According to the answers to Question 5, none of the Primos in any of the pairs felt envious of the FE support for their partner. According to the answers to Question 6, all subjects want to use FE again. The subjects of Pairs D and E wanted to use FE with their friends for fun (see answers to Question 8). On the other hand, the Secondo of Pairs B and C, who had some experience playing the organ, answered that she hopes to be able to play the piano more naturally. In the session of Pair B without FE, she tried to perform only the right-hand part. The rate of correct notes that she played was 76.7% of all right part notes on the score in fifteen minutes. In her answer to Question 9, she proposed a means of support such that the Secondo can practically perform the right-hand part with the left-hand part supported by FE. Table VI shows the results of score-tracking in joint practice with FE. We counted the number of erroneous notes classified as: extra, wrong, and missing. In the case of replaying, we counted the number of notes between the current position and the start position of replaying. The 4th column indicates the number of wrong notes before FE could track a correct position of Primo’s playing. We could not count the notes when they finished the performance just after Primo’s error before the system synchronized again. This is listed as “unknown.” “Combination” means that Secondo made errors in succession. The average number of incorrectly estimated current positions before synchronizing again is 2.77 (the “unknown” cases are not included in this calculation): the average number for extra, wrong and missing types of error is 2.28, and that for replaying type of error is 3.83. Even a human usually needs about 3 notes to synchronize again when the partner suddenly returns to somewhere before. Therefore, we can say that this performance is acceptable. The worst result is “10 notes” till FE could find Primo’s correct current position again, although Primo made mistakes on only two notes. This is inevitable, however, since two wrong notes also could indicate another position in the score by accident. Primo played “ra, fa, sol, do, do” although she should have played “ra, fa, sol, mi, mi.” On the other hand, the phrase “ra, fa, sol, do, do” coincides with another position. As a result, it took 10 notes till FE found Primo’s current position. It is even difficult for a human to find the right position quickly in such a case. ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. • A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances 17 Table VI. Results of Score-Tracking 1 1 2 1 The Number of Incorrectly Estimated Current Positions Before Synchronizing Again 1 2 4 6 3 1 2 4 unknown 1 2 3 4 5 10 unknown 4 5 6 4 1 5 unknown 1 2 unknown 4 6 1 unknown 1 4 8 unknown 6 unknown 2 5 Number 1 1 1 2 1 26 2 1 4 2 2 1 1 2 1 2 1 1 1 1 1 1 2 1 1 5 2 1 1 8 1 1 1 8 1 1 2 1 1 4 1 1 3 1 1 4 1 1 5 1 1 6 1 1 4 1 2 5 2 2 3 2 extra The Number of Error Notes 1 Number 5 wrong 3 1 1 33 2 11 3 3 1 2 1 1 3 1 (back) 2 7 2 (back) 3 3 (back) 9 4 (back) 11 6 (back) 8 (back) 12 (back) 22 (back) 2 4 (back) 1 2 (back) 1 2 (back) 2 3 (back) 1 (back) 1 (back) 3 (back) 3 (back) 3 (back) 3 (back) 1 1 (back) 1 (back) 1 2 (back) 2 2 (back) missing replaying combination combination combination combination combination combination combination combination wrong replaying wrong replaying wrong replaying missing replaying replaying replaying replaying replaying replaying replaying missing replaying replaying wrong replaying wrong replaying ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. 18 6.4 • C. Oshima et al. Example of Subjects’ Dialogues About FE In this section, we show two effects of FE from examples of subjects’ dialogue about joint practice with FE. One is that FE prompts Primos to practice more enthusiastically. The other is that Primo and Secondo practiced to increase their musically expressive duo play. 6.4.1 Motivating Children to Practice More. In joint practice with FE, the following dialogue took place between Pair A child and her father: Father & Child: Duo Play (Made a bad start and stopped immediately.) Father: omae mazu rensyu sei! (Do your practice first!) Child: un rensyu suruwa (I see. I am going to practice.) Child: Solo Play Child: yosya ikou (OK! Let’s go!) Father: eeka? (Are you ready?) Child: otosan to hikuto yarinikuiyona (It is tricky to play with you, dad.) [ Father: nandeyanen (Why?) Father & Child: Duo Play By using FE, the parent and child are on almost equal ground concerning reproducing the correct pitches. Consequently, if the child cannot perform the piece perfectly because of a lack of practice, the parent can achieve the rank of the child. In this dialogue, the child said, “It’s tricky to play with you, dad.” showing that she is impatient with her incomplete performance. After all the experiments, her comment relates to the flawed performance of the Primo part, and reproducing correct pitches (answer to Question 5). Therefore, she is frustrated that she could not blame her father when the performance was not perfect. However a child’s impatience may promote more enthusiastic practice. The results of the experiments show that in joint practice using FE, the Primos practiced by themselves (Table III). 6.4.2 Collaboration for Improved Musical Performances. The fact that the parents could play their part correctly and easily with the help of FE also had a desirable effect on another aspect of musicality. The following dialogue took place between Pair A father and child in joint practice with FE: Child: Father: Child: Child: ano ko tan tan tan tte hikuto oto togireru wakeyo= (Well, your key touch is like: tan tan tan, a way of touch that indicates disconnected tapping sounds.) =un un un (yeah, yeah, yeah) dakara motyotto sa (So, a bit more like this) Solo Play ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances Father: Child: • 19 un motto ko nobasette iuno? = (You mean I should sustain the sounds, right?) =so (Right!) In other words, the child wanted her father to play his part more legato. Such conversations about musical expression were never found in the joint practice of any of the pairs without FE, the Secondo considered nothing other than performing his/her part. Moreover though the father of Pair A was a musically inept performer, he was able to learn that their timing was off at the dotted notes: Child: Father: Child: Child: Father: o dekitayan= (Oh! We finished!) =a koko aitaine (Well, I want the timing in this part to be perfect.) Circling the dotted notes with his finger ta-n ta tan (ta-n ta tan) Playing the dotted notes [ so so ta-n ta tan un kokodake rensyusuru (Yeah yeah, ta-n ta tan. O.K. Let’s do it from this part?) Primarily, ensemble practice should be a time when the players construct their own performance by conveying their musical ideas and performance plans to each other. Without FE, such musical expression discussions could not be achieved because of the knowledge gap between the Primo and Secondo (e.g., the Secondo cannot even read a normal score) and because of the Secondo’s lack of technical ability. 7. DISCUSSION Table III shows the first effect of FE: even a totally inexperienced performer can quickly play a duo with a beginner. The number of duo plays in joint practice with FE is much more than in joint practice without FE. Table III also reveals a second effect of FE: Primo (the child) begins to practice by herslf. The number of Primo’s solo plays in joint practice with FE is more than in joint practice without it. We argue that this result is not just caused by the decrease of Secondo’s solo play time but due to support from FE. When Secondo is supported by FE, it is clear that the errors in their duo performances were caused by Primo. Moreover, both Primo and Secondo clearly noticed these facts. Consequently, Primo lost her superiority over Secondo. Primo’s willingness to practice had to rise. Primo’s utterance, “It’s tricky to play with you, Dad (Section 6.4.2),” also shows her irritation. The dialogues indicate that they were talking about musical expression, even though Secondo has no experience with playing the piano. Such conversations about musical expression were never found in the joint practice of any of the pairs without FE: Secondo considered nothing other than performing his/her part. Both Primo and Secondo played music with as good musical expression as they wanted through practice. We believe there are two reasons why they immediately aimed for richer musical expression, that is, talking about musical expression despite their being a musically inept performer and a beginner. First, FE supports Secondo in outputting correct notes to be played at each point. By easing the cognitive load in the performance due to nonexpressive elements, by providing accurate reconstruction of the sequence of notes, FE allows Secondo to externalize more of his/her own musical expression (Section 4.2.2). Consequently, the technical gap between Primo, the beginner, and Secondo, the musically inept ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. 20 • C. Oshima et al. performer, has narrowed. Additionally, since Primo does not need to teach Secondo how to play the correct notes, they can both afford to create their own musical expressions. Second, FE does not intervene in Secondo’s musical expression at all. Expressive elements such as the volume and the length of each note are output as Secondo controls them. Accordingly, even the supported Secondo has sufficient room for musical expression (Section 4.2.3). Therefore, Secondo can also contribute to creating the pair’s musical expression in a piano duo. Note that although FE can help Secondo externalize the musical expressions that already exist in his/her mind, it can neither create nor add new musical expression for him/her. We should mention that the keyboard for the parent’s part of FE is not a piano but another “new” musical instrument: the way to perform this part is seemingly similar but actually very different from a conventional piano. This is based on a practical decision that the parent need not master the piano: the one who should master it is the child. Therefore, we neither modify the child’s musical instrument nor directly support the child’s part. 8. CONCLUSION We proposed “Family Ensemble” (FE), which allows a musically inept parent to enjoy accompanying their child’s piano practice, as an alternative to a completely automated accompaniment. FE allows the parent to output the correct sequence of pitches along with the child’s performance by support from note-replacement and score tracking functions even if he/she touches keys with just one finger. Although we implemented the score tracking function based on Dannenberg’s score tracking algorithm, which can deal with the extra, wrong, and missing types of errors, we extended it to cope with a peculiar error of beginners: replaying. As a result, even if the child makes many errors and even if his/her parent has little or no experience playing the piano, with FE they can immediately play a piano duo. The results of our experiments show that FE allows even performers with no experience playing the piano to begin to play a duo with the beginner. Furthermore, we show that Secondos, for example, the parents, can sufficiently externalize their musical expressions as well as maintain enough space to musically grow, even though being supported to reproduce correct sequence of pitches by FE. Eventually, we found that in joint practices using FE some subjects discussed musical ideas to achieve richer musical expression. We would like to conduct additional experiments with more subjects and more musical pieces at homes and at piano schools to assess the long-term usefulness of this system. ACKNOWLEDGMENTS We would like to thank the subjects. REFERENCES AGAY, D. 1987. Learning to Play Piano: Book 1. Yorktown Music Press. BLAIN, T. AND PERKIS, T. 2000. The jum-o-drum interactive music system: A study in interaction design. In Proceedings of the Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques. 165–173. BLOCH, J. AND DANNENBERG, R. B. 1985. Real-time computer accompaniment of keyboard performance. In Proceedings of the International Computer Music Conference (ICMC). International Computer Music Association (ICMA). 279–289. CASIO. 2000. LK-55 Full Size Keyboard with Key Lighting System. Casio, http://www.casio.com/index.cfm?fuseaction= products.detail&product=LK%2D55. DANNENBERG, R. B. 1984. An online algorithm for real-time accompaniment. In Proceedings of the International Computer Music Conference (ICMC). International Computer Music Association (ICMA). 193–198. EVERETT, C. 1999. Piano Education Page—Artist/Educator Archive Interview—. Zeigler, J. M. and Ostromencki, N. L., http://pianoeducation.org/pnoevret.html. ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007. A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances • 21 FARBOOD, M., PAZSTOR, E., AND JENNINGS, K. 2004. Hyperscore, a graphical approach to composing music. IEEE Computer Graphics and Applications, Special Issue on Emerging Technologies, 50–54. GOTO, M., HIDAKA, I., MATSUMOTO, H., KURODA, Y., AND MURAOKA, Y. 1996. A jazz session system for interplay among all players— virja session (virtual jazz session system)—. In Proceedings of the International Computer Music Conference (ICMC). International Computer Music Association (ICMA). 346–349. GRUBB, L. AND DANNENBERG, R. B. 1997. A stochastic method of tracking a vocal performer. In Proceedings of the International Computer Music Conference (ICMC). International Computer Music Association (ICMA). 301–308. JENNINGS, K. 2003. Toy symphony: An international music technology project for children. Music Education International 2, 3–21. KIHARA, R. 1991. Piano Land, vol. 1–5, with music data (an educational material in Japan). Ongakunotomo-sya. LYKE, J. 2004. Piano Education Page—Artist/Educator Archive Interview—. Zeigler, J. M. and Ostromencki, N. L., http://pianoeducation.org/pnojlyke.html. MCCARTHY, C., BLIGH, J., JENNINGS, K., AND TANGNEY, B. 2005. Virtual collaborative learning environments for music: Networked drumsteps. Computers & Education 44, 2 (February), 173–195. MIYOSHI, A. 1977. Miyoshi Piano Method, vol. 1–8 (an educational material in Japan). Kawai. MIYOSHI, A. 2000. Piano duo sympathetic world outside language. MUSICA NOVA 31, 9, 30–32. ORIO, N., LEMOUTON, S., AND SCHWARZ, D. 2003. Score following: State of the art and new development. In Proceedings of the New Interface for Musical Expression (NIME). 36–41. OSHIMA, C., NISHIMOTO, K., AND SUZUKI, M. 2004. Family ensemble: A collaborative musical edutainment system for children and inexperienced parents. In Proceedings of the ACM Multimedia (MM’04) and Co-Located Workshops. 556–563. OSHIMA, C. AND SHIMOJIMA, A. 2003. Grounding function of instrument plays in dialogue: A case-study of piano duos in joint practice. In Proceedings of the 25th Annual Conference of the Cognitive Science Society. PACHET, F. AND ADDESSI, A. 2003. When children reflect on their playing style: Experiments with the continuator and children. ACM Computers in Entertainment 2, 1, Article 05. A probabilistic expert system for automatic musical accompaniment. J. Comput. Graph. Stats. 10, 3, RAPHAEL, C. 2001. 487–512. RENCON. 2000. Rencon (performance RENdering piano Contest). http://shouchan.ei.tuat.ac.jp/∼rencon/index.shtml. SMOLIAR, S. W., WATERWORTH, J. A., AND KELLOCK, P. R. 1995. pianoforte: A system for piano education beyond notation literacy. In Proceedings of Multimedia. ACM. SUBOTNICK, M. 1997. ghost. http://www.mortonsubotnick.com/about.html. TAKEUCHI, Y. AND KATAYOSE, H. 1995. Representation of music expression with two finger piano. In IPSJ SIGNotes Music and Computer. IPSJ, 37–44. TOYODA, K. S. 1995. Compact Disc, Japanese Song Without Instrumental Accompaniment. Victor Entertainment, VICC-169. VERCOE, B. 1984. The synthetic performer in the context of live performance. In Proceedings of the International Computer Music Conference (ICMC). International Computer Music Association (ICMA) 301–308. ZEIGLER, J. AND OSTROMENCKI, N. 2004. Piano Education Page—Tips Performance—. http://pianoeducation.org/pnotperf.html. Received April 2005; revised October 2005, February 2006; accepted May 2006 ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.