A Piano Duo Support System for Parents to Lead Children to Download

Transcript
A Piano Duo Support System for Parents to Lead Children
to Practice Musical Performances
CHIKA OSHIMA
National Institute of Information and Communications Technology and
ATR Cognitive Information Science Labs
KAZUSHI NISHIMOTO
Japan Advanced Institute of Science and Technology
and
NORIHIRO HAGITA
ATR Media Information Science Laboratories
In this article, we propose “Family Ensemble,” a piano duo support system for a musically inept parent and his/her child who is
a beginner at playing the piano. The system makes it easier for parents to correctly reproduce a given sequence of pitches along
with the child’s performance by using score tracking and note-replacement functions. The experiments with this support system
showed that the parents can immediately participate in the piano duo. Furthermore, we found that during joint practices using
Family Ensemble some subjects discussed musical ideas that they would not have talked about without using the system.
Categories and Subject Descriptors: H.5.5 [Information Interfaces and Presentation]: Sound and Music Computing—Systems; K.3.1 [Computers and Education]: Computer Uses in Education—Collaborative learning
General Terms: Experimentation
Additional Key Words and Phrases: Piano duo, musical expression, score tracking, entertainment, support system
ACM Reference Format:
Oshima, C., Nishimoto, K., and Hagita, N. 2007. A piano duo support system for parents to lead children to practice musical performances. ACM Trans. Multimedia Comput. Commun. Appl. 3, 2, Article 9 (May 2007), 21 pages. DOI = 10.1145/1230812.1230815
http://doi.acm.org/10.1145/1230812.1230815
1.
INTRODUCTION
In this article, we propose a collaborative musical performance support system named “Family Ensemble” (FE) [Oshima et al. 2004], which allows musically inept parents to play piano duos with their
children who are beginners at the piano.
Nishimoto’s work was partially supported by the Ministry of Education, Science, Sports, and Culture, Grant-in-Aid for Scientific
Research (C), 16500580, 2004, and was partially supported by Hayao Nakayama Foundation for Science and Technology and
Culture. Hagita’s work was supported by the National Institute of Information and Communications Technology (NICT).
Authors’ addresses: C. Oshima, ATR Cognitive Information Science Labs. 2-2-2, Hikaridai, Keihan’na Gakken Toshi, Kyoto,
JP 619-0288; email: [email protected]; K. Nishimoto, Japan Advanced Institute of Science and Technology 1-1, Asahidai, Nomi,
Ishikawa, JP 923-1292; N. Hagita, ATR Media Information Science Labs. 2-2-2, Hikaridai, Keihan’na Gakken Toshi, Kyoto,
JP 619-0288.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided
that copies are not made or distributed for profit or direct commercial advantage and that copies show this notice on the first
page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, to redistribute
to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be
requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481,
or [email protected]
c 2007 ACM 1551-6857/2007/05-ART9 $5.00 DOI 10.1145/1230812.1230815 http://doi.acm.org/10.1145/1230812.1230815
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
2
•
C. Oshima et al.
Generally, the child has an opportunity to play the piano duo with her teacher or friends, since the
efficacy of duo play has been recognized in piano playing education. However, in practice, most children
can play piano duos only in school since often parents cannot play. Using FE, the parent can become a
lesson mate at home rather than a proxy teacher, encouraging the child to practice more enjoyably and
more musically. By compensating for the difference between the parent and child, FE not only provides
entertainment, but also promotes enthusiastic discussion of musical expression.
The parents are readily able to always output correct notes by hitting any keys (we call it a notereplacement function). Moreover, FE always tracks the position where the child is playing at each
point in the score and automatically synchronizes the position where the parents should perform.
Therefore, the parents are readily able to output notes that correspond to the position where the child
is (unexpectedly) performing now (we call it a score tracking function). FE is an entertainment system
that supports the parent by allowing him/her to play piano duos with his/her child at the same level of
piano playing as the child.
FE only supports the parent, by providing a note-replacement function for the parent’s performance
and a score tracking function for the child’s performance. The note-replacement function only replaces
the pitches of notes performed by the parent to the correct pitches that should be output. However,
this function does not modify any other elements, for example, when and how strongly the keys are
hit and when and how quickly the keys are released. As a result, while parents become readily able to
always output correct notes by hitting any keys, even with only one finger, they can also fully control the
other elements to achieve their own musical expression. Thus, this function supports parent’s ability
while preserving sufficient room for musical expression. Similar systems have already been proposed
(e.g., Subotnick [1997] and Takeuchi and Katayose [1995]) and some of them have already been on the
market (e.g., Casio [2000]). Though these systems aimed at mainly supporting solo performances, we
applied a similar mechanism for supporting piano duo performances.
Regarding duos with the child, however, it is inadequate to merely sequentially replace the pitches
along with the parent’s score. When the child makes a mistake and goes back to an earlier point in the
music, the parent gets lost and they have difficulty getting synchronized again. To solve this problem,
we implemented a score tracking function that always tracks the position where the child is playing
at each point on the score. We discovered that there are four types of mistakes in children’s piano
performances through analysis of the features of the actual performances of children. Based on these
results, we implemented a robust score tracking algorithm that can quickly “catch up” with a child’s
performance even if he/she makes mistakes. As a result, even if the child suddenly makes mistakes, it
is not necessary to stop to synchronize both performance positions again. They can keep on performing
and then the note-replacement function can correctly work again after minimum unavoidable errors.1
Thus, by combining these two functions, FE allows the parent to readily perform duos with his/her
child at home at almost an equal level as the child. Moreover, since FE only helps parents reproduce
the correct sequence of pitches, they can enjoy externalizing their musical expression as well as readily
playing the music. Thereby, the child and parent are encouraged to discuss music and to collaboratively
create a duo performance. In an ensemble, each performer learns the partner’s good expression and
finds his/her mistakes or poor expression by comparing his/her own performance with a partner’s. Piano
students make suggestions to each other for improvement in ensemble practice [Lyke 2004], discussing
musical expression to explore an ideal goal [Oshima and Shimojima 2003].
Technologies for automatically tracking human musical performances have been studied mainly in
terms of automatic accompaniment systems. Dannenberg [1984] developed an algorithm for tracking a
1 Many piano teachers pointed out that the performer in an ensemble should keep on going, even if he/she misses a note or a beat
even in a practice [Zeigler and Ostromencki 2004].
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances
•
3
score based on pitch sequence data by applying a DP-matching method. Vercoe [1984] developed a score
tracking method that referred not only to the pitch sequence but also to the time each note sounded.
In addition, a human cognition model for estimating musical performance in an ensemble has been
considered. Studies of an automatic jam session system attempted to follow the emotive transition
of human performer(s) [Goto et al. 1996]. Although FE’s score tracking function is an extension of
Dannenberg’s algorithm, as described below, all of these previous studies aimed to enable ensemble
play by advanced human performers and machines. In contrast, FE seeks to enable ensemble play by
human pairs with little and no musical experience.
Regarding the study of interactive musical systems, the Continuator [Pachet and Addessi 2003]
system supports users in learning musical styles. When the user plays a musical sequence of any kind
(i.e., a phrase), the Continuator in turn generates a musical phrase in response to the user’s phrase
based on a Markov model of the musical phrase. Experimental results showed that participants played
the system in a self-motivated way. The Jum-O-Drum [Blain and Perkis 2000] is a system that allows
up to six players to practice collaborative musical improvisation. FE helps to fill a gap in mechanical
skills—reproducing correct pitches—between the members so that the performer who has no experience
of playing the piano can participate in a piano duo as well as discuss musical expressions with others.
In the next section, we give an overview of related works. Section 3 describes the setup of FE in detail.
Section 4 describes the note-replacement function, and Section 5 describes the score tracking function.
Section 6 illustrates experiments for evaluating FE efficacy and shows the results. In Section 7, we
discuss the FE feature, which can encourage players to richer, more expressive performances. Section
8 concludes this article and outlines the future works.
2.
RELATED WORKS
Recently, in piano playing education the efficacy of duo play, which is a kind of ensemble, in the learning
process has come to be widely recognized [Agay 1987; Kihara 1991; Miyoshi 1977]. Practicing the piano
alone is painful, and raising a child’s motivation to practice is difficult. However, by incorporating
piano duos into lessons, children come to practice more enthusiastically and perform more musically
[Miyoshi 2000]. This is because, first of all, an ensemble performance is essentially fun entertainment.
In addition, practice and rehearsal time with other musicians involves a very rewarding experience
[Zeigler and Ostromencki 2004].
Such ensembles can be regarded as collaborative learning methods as well as edutainment methods.
In general collaborative learning, learners address an issue together to acquire knowledge while encouraging and stimulating each other. Teachers use ensemble music as a tool to improve the students’
rhythmic and harmonic awareness [Everett 1999]. A performer who plays in an ensemble must perfectly
synchronize his/her playing with a partner by listening to the partner’s performance. The performer
learns to respond spontaneously to another person’s musical cues or flubs [Zeigler and Ostromencki
2004]. Throughout the ensemble performance, such musical expressions as tempo, dynamics, duration,
and so on, must be consonantly controlled.
As a solution to lacking partners outside of lessons, various “minus-one” software applications exist
in which all parts except for the pupil’s are recorded [Kihara 1991]. Regrettably, music professors also
comment that minus-one software prevents pupils from acquiring the cooperative performance skills
essential for ensembles [Miyoshi 2000]. Moreover, they cannot perform subtle changes in tempo and
dynamics, because minus-one plays continuously independently of the child’s performance. Another
solution, automatic musical accompaniments [Casio 2000; Dannenberg 1984; Raphael 2001; Vercoe
1984], which allow a computer to function as a musical accompanist, have been studied. They follow
the human performance even if he/she makes little mistakes or subtle changes in tempo and dynamics.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
4
•
C. Oshima et al.
Fig. 1. Family Ensemble Setup.
However, since these systems just track the performance position of the human performer, the performer cannot learn how to play cooperatively. Moreover, the human player cannot discuss music with a
machine.
pianoFORTE [Smoliar et al. 1995] records and displays several aspects of a user’s performance. It
aims to have teachers and students discuss the distinctions between the art of playing the piano and
the technique of playing the correct notes. DrumSteps [McCarthy et al. 2005] is a dynamic graphical interface that enables children to create, manipulate, edit, and save original pieces of percussion
music. Toy Symphony [Jennings 2003] is an international music performance and education project
that allows children to learn by playing alongside some of the world’s most accomplished musicians.
Several interesting tools or instruments have been developed in this project. For example, Music Toys
are specially developed musical instruments that allow anybody to easily enjoy musical performances.
Hyperscore [Farbood et al. 2004], a tool that enables children to “compose by drawing,” is designed to
intuitively and dynamically introduce children to musical composition and creativity. FE provides no
special tools to learners, unlike these systems.
3.
SYSTEM SETUP OF FAMILY ENSEMBLE
Figure 1 shows the setup of Family Ensemble (FE). FE directly supports the parent’s part in playing
a piano duo with his/her child. We call the parent part “Secondo,” that is, the part that performs the
lower register of a duo. On the other hand, the child’s part is called “Primo,” the part that takes charge
of the upper register of the duo. Primo’s performance is output as Musical Instrument Digital Interface
(MIDI) data and directly input to a sound generator as the notes are played. At the same time, Primo’s
performance data are input to the score tracking module, which determines Primo’s current position
in the score. The obtained current position data are input to a note-number-acquisition module that
compares Primo’s current position with Secondo’s score. As a result, the MIDI note number(s) that
Secondo should play at that moment is (are) acquired. The acquired MIDI note numbers are input to
the note-number-replacement module, which replaces the MIDI note number(s) that Secondo played
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances
•
5
with the acquired correct MIDI note number(s), while preserving other characteristics of the performed
note(s): loudness and length of sound performed. Consequently, by using any keys, Secondo can perform
the correct notes with his/her own expression, alongside Primo’s performance. If Secondo momentarily
stops playing while Primo continues performing, Secondo can immediately resume the performance
from the correct place in the score since FE continues to track Primo’s position regardless of the progress
of Secondo’s performance. On the other hand, if Primo suddenly stops playing while Secondo continues
performing, Secondo cannot proceed beyond the point where Primo stopped. When Primo resumes
the performance, Secondo resumes, too. Even if Primo returned several notes back when resuming,
Secondo can quickly catch up. Additionally, users can arbitrarily decide the boundary between Primo’s
and Secondo’s regions on a keyboad.
Thus FE includes two important supporting functions: a note-replacement function in the notenumber-replacement module and a score tracking function in the score tracking module. The notereplacement function allows Secondo to readily reproduce an accurate sequence of pitches, which is
necessary, and one of the most significant initial barriers for novices. Additionally, it is important that
Secondo can still freely and fully express his/her musical intention and emotion even though being
supported. Section 4 describes the note-replacement function in detail. The score tracking function
always tracks a position on the score where the child is playing to obtain pitches that Secondo should
play at that moment. Section 5 describes the score tracking function in detail, wherein even if the child
makes some mistakes, the piano duo continues. Note that Primo, his/her child, plays the piano normally
since these functions do not support Primo’s part.
4.
NOTE-REPLACEMENT FUNCTION
In this section, we will demonstrate that the note-replacement function is effective in readily playing
and externalizing the performer’s musical intention and emotion.
4.1
Expressive and Nonexpressive Elements
Generally speaking, from the performer’s perspective a musical performance consists of controlling two
kinds of elements: non-expressive and expressive. In performing “reproduction type music,” such as
classical music, performers must accurately reproduce the pitches and the sequence of pitches along
given scores; modifying them is prohibited. The performers cannot demonstrate their own expression
by the pitches and their sequences. Therefore, these elements are nonexpressive elements. On the
other hand, each note has many other characteristics, for example, loudness, and length of sound.
Individual performer expression is reflected in the control of these elements. Therefore, these elements
are expressive elements.
The difficulty of dealing with nonexpressive elements causes cognitive and physical overload for
performers and prevents them from concentrating on expressing individual musical intentions and
emotions. Furthermore control of nonexpressive elements is usually unavoidable before tackling the
control of expressive elements when using conventional keyboards. Therefore, allowing performers
to directly control expressive elements facilitates the demonstration of their musical expression (see
Figure 2).
4.2
Effectiveness of the Note-Replacement Function
Based on the aforementioned idea, several piano performance supporting systems have been developed
[Subotnick 1997; Takeuchi and Katayose 1995; Casio 2000]. These systems employed note-replacement
functions that alleviate the difficulties of controlling nonexpressive elements in the performance of
reproduction type music, the accurate reproduction of pitch sequence. We implemented a similar system to support the musically inept performers. We illustrate that the musically inept performers and
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
6
•
C. Oshima et al.
~Attribute~
~Support~
Volume
Duration
Expressive
elements
Non-support
These elements are
input concurrently*
Attenuate
Pitches,
the sequence of pitches
Non-expressive
elements
Computer
supports
*In a step-by step input system, a user also can input pitches individually. However,
he cannot input the expressive elements concurrently like the conventional piano.
Fig. 2. Basic idea for supporting reproduction type music performance.
beginners can immediately and accurately reproduce the given sequence of pitches (melody) by the notereplacement function, and also sufficiently express their musical intentions and emotions(see Sections
4.2.1 and 4.2.2). Furthermore, we also show that the note-replacement function provides the performer
with enough room for musical growth(see Section 4.2.3).
4.2.1 Experiments. In this section, we estimate the effectiveness of the note-replacement function.
We compared the performance by using the note-replacement function (we call this way of performance
“NR method,” hereafter) with the normal performance without using any supporting functions (we call
this way of performance a “normal method,” hereafter) .
We recorded all performances by both methods as MIDI sequence data. In the NR method, at first,
without considering any expressive elements, the performer inputs the simple sequence of MIDI note
numbers as numerical values in a computer. Then using these data, the performer plays the piece on
a keyboard to concurrently input all the expressive elements. In the normal method, sequence data is
basically input by normally performing a musical instrument that outputs MIDI data. Therefore, all of
the elements, both expressive and nonexpressive, are concurrently input.
We conducted experiments using subjects as performers to compare the NR method with the normal
method. We employed eighteen undergraduate and graduate school students with little or no experience
playing the keyboard. The subjects were asked to play “Akatonbo (a red dragonfly),” which is a very
famous Japanese children’s song, by the two methods. Since this piece is made in a range of eleven
notes, performers have to change their finger positions and pass the thumb through the other fingers
along the way. Therefore, it is a difficult piece for beginners to play.
After the subjects listened to “Akatonbo” sung by a professional vocalist [Toyoda 1995], they were
asked to record a performance of “Akatonbo”as MIDI sequence data by the two methods as
expressively as possible, emulating the professional vocalist. Before the experiments we explained
both methods and had them practice the methods with another small piece. Each subject provided a
subjective evaluation for each method after recording by both of the methods. The subjects evaluated
their degree of satisfaction (1: unsatisfied to 5: very satisfied) with each performance and the difficulty
(1: very easy to 5: very difficult) of controlling four elements: melody, rhythm, agogics, and dynamics,
by each method. Here, melody means the sequence of pitches, rhythm and agogics are shown by the
length of each note, and dynamics means the change of volume.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances
•
7
Table I. Average Evaluation Values for Two Methods
Elements
melody
rhythm
agogic
dynamics
emotional burden
Evaluation
difficult
satisfied
difficult
satisfied
difficult
satisfied
difficult
satisfied
NR (ave.)
1.89
3.22
2.22
3.39
2.50
3.17
2.44
3.28
2.28
Normal (ave.)
3.44
2.56
3.67
2.83
3.44
2.50
3.56
2.56
3.22
t-value
3.50∗∗∗
2.13∗∗
3.71∗∗∗
1.82∗
2.88∗∗
2.06∗
2.60∗∗
2.06∗
2.12∗∗
In difficulty, 1: very easy to 5: very difficult.
In satisfaction, 1: unsatisfied to 5: very satisfied.
(∗∗∗ ) means a significant difference at 1%.
(∗∗ ) means a significant difference at 5%.
(∗ ) means a significant difference at 10%.
4.2.2 Results. Table I shows the average evaluation values for the NR method and normal method.
The results of a t-test indicate that the NR method is significantly easier than the normal method and
significantly lighter than the normal method in cognitive burden. Moreover, the subjects tended to be
more satisfied with the performances using the NR method than those with the normal method.
A feature of the NR method is that nonexpressive elements can be input as numerical values separated from expressive elements. This allows people to readily reproduce the correct sequence of pitches
without worrying about expression. Therefore they can achieve better expression than with the normal
method because the nonessential load for accurate reproduction of melody is alleviated. As a matter of
fact, the results of the satisfaction rate with expressive items: rhythm, agogics, and dynamics, using the
NR method are higher than the normal method. On the other hand, all of the expressive elements are
tightly related; balance among them is very important for achieving good musical expression. Therefore,
all of the expressive elements should be concurrently controlled, as they are in the normal method.
4.2.3 Evaluation at a Contest. To investigate whether the note-replacement function has enough
space for musical growth, we entered the performance data performed by the NR method in a contest for
performance rendering systems named “Rencon [Rencon 000 ].2 ” This international event aims to establish evaluation methods for performance rendering systems that automatically generate expressive
musical performance.
We employed one subject as a performer, a forty-year-old male who studied electric organ for one
year. He can read a score. However, he had never heard the set piece, “Waltz Op.69-2” by F. Chopin.
Of course, he could not play this piece on a conventional piano (i.e., the normal method) even after the
experiment. The first author input to the system the sequence of pitches of the right-hand part from the
98th bar to the 145th bar of the piece in advance. Then, we asked the subject to play the set piece with
the NR method on the piano using the pitch sequence data, and we recorded the performance as MIDI
sequence data. After glancing over the score, he played the first performance. We asked him to practice
untill he was satisfied with his performance. He practiced the set piece for a total of forty minutes
over two days. He was satisfied with the 13th performance. We recorded all of the performances and
submitted the 1st performance (called “CiP3 -1”) and the 13th (final) performance (called “CiP-2”) to the
Rencon contest.
2 FIT 2002 Rencon Workshop: Toward the Number One Pianist in the World, June 30 2002, in Japan. In the contest, we did not
need to compose the sequence data on the spot. Recorded data were evaluated.
3 “CiP” means “Coloring-in Piano,” which was our system’s name at Rencon.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
8
•
C. Oshima et al.
A total of ten systems entered Rencon. Sixty-three judges (not professional musicians) evaluated
the ten performances for naturalness and subjective preferences. One of the semi-automatic rendering
systems won 1st prize and two of the fully automatic rendering systems won 2nd and 3rd prizes. CiP-2
got 4th prize and CiP-1 8th place.
CiP-2 was evaluated much higher than CiP-1, showing that the note-replacement function provides
the performer with enough room for musical growth and that the performer can quickly improve his/her
performance in a short period.
4.3
Concluding This Section
From the results illustrated in 4.2.2 and 4.2.3, we can conclude that supporting the control of nonexpressive elements alleviated the performers’ cognitive load when playing the piano, as well as provided
enough room for the performers to grow musically. Eventually, the note-replacement function raised
the musically inept performer’s ability to play reproduction type music. Therefore, by applying this
function to support the parent who has no experience with playing the piano, he/she becomes an equal
partner of the child and can start to discuss musical ideas during joint practice.
5.
SCORE TRACKING FUNCTION
The note-replacement function can make even a musically inept parent a partner with the child. However, it is not sufficient to merely provide this function to parents for supporting duo performances. The
replacement function monotonously replaces the performed pitch by the parent with the correct pitch in
the note sequence data one by one. Therefore, when Primo, who is using a normal piano, and Secondo,
who is supported by the note-replacement function, play an ensemble together, Primo can never make
mistakes; if Primo makes mistakes, for example, some missing notes or extra notes, the playing position
immediately becomes inconsistent. Accordingly a support function for solving this problem called the
“score tracking function,” always determines where Primo is playing on the score. The parents are able
to output the notes that correspond to the position where Primo is even unexpectedly, performing now.
As a result, Secondo is always able to consistently perform with Primo even if Primo suddenly makes
a mistake.
As for our objective in this article, the child is Primo, using a normal piano. Therefore, in 5.1, we
analyze the features of the novice child’s errors. Then, we describe the details of the score tracking
function in 5.2.
5.1
Analysis of Beginners’ Errors
Several score tracking systems [Dannenberg 1984; Grubb and Dannenberg 1997; Orio et al. 2003] have
dealt with three types of performance errors, as defined by Bloch and Dannenberg [1985].
(1) extra: An extra note that does not match any notes in the score is performed.
(2) wrong: Although a performed note matches a note in the score, its pitch is incorrect.
(3) missing: A note in the score is not performed.
They assume that their target users are experienced performers whose performances closely match
the scores. Therefore, when the performers make an error, the system only has to check one or a few
note(s) before and after the position where the performer made the error. On the other hand, our system,
FE, has to be adapted to undertrained performers: children who are beginners.
To examine the kinds of errors that beginners are apt to make, we conducted experiments with
two children studying the piano. Subject A had studied the piano for four years. She is a fourth year
elementary school student. Subject B had studied the piano for one year. She is in nursery school and
the sister of Subject A. Subject A was asked to perform seventeen new pieces in four, thirty-minute
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances
•
9
Table II. Error Frequency
extra
wrong
missing
replaying
Sum
Place
—–
—–
—–
Start of piece
Start of column
Start of phrase
Alternation
1st beat
2nd downbeat
Last beat
1st beat (pre-bar)
Last beat (pre-bar)
—–
A
1
0
3
7
1
5
4
315
17
0
26
2
381
B
6
37
0
3
1
0
0
141
2
6
25
3
224
Sum
7
37
3
10
2
5
4
456
19
6
51
5
605
%
1.2
6.1
0.5
1.7
0.3
0.8
0.7
75.4
3.1
1.0
8.4
0.8
100.0
experiments. Subject B was asked to perform thirteen new pieces in four, thirty-minute experiments.
They were asked to perform the pieces from one to five times according to the level of difficulty. Moreover,
they were asked to perform each piece from the beginning to the end since we wanted to create a situation
similar to duo play. It is not simple solo practice but similar to the duo practice where they have to
aim to play through the entire piece. We recorded the sounds and utterances of these experiments on
a DVD-RAM disk.
From the results of the experiments, although it is not new to piano teachers at all, we found it
necessary to add a fourth type of error in addition to the aforementioned three types:
(4) replaying: An extra note sequence that matches a part of the score is performed. In other words, an
undirected refrain is performed.
It has been ignored in the conventional score-tracking systems up to now. In solo practice, we usually
can see such partial practices as “replaying errors.” However, in our experiments, we asked the subjects
to play through the pieces in advance. Consequently, “replaying” is not considered partial practice. In
replaying, the number of notes that the performer played is more than the notes in the score, as in the
case of extra error. However, the surplus notes of replaying errors are part of melody in the score, while
those of extra errors are not found in the score.
Table II shows the frequency of errors: extra, wrong, missing, and replaying. As for replaying, the
places where the subject restarted as well as the frequency of each restart are shown. “Column” means
a group of bars forming a line on a printed score although this category does not include the notes in
“start of piece.” “Start of phrase” denotes the start of the slur mark, although this category does not
include the notes in “start of piece” and “column.” “Alternation” denotes where a performer needs to
alternate the right hand with left hand in one phrase, although this category does not include the notes
in these categories. “1st beat” means that the subject restarted at the 1st beat of the same bar where the
subject stopped her performance. “2nd downbeat” means the 2nd beat of a binary measure, the 3rd beat
of a quadruple measure, and so on. “Last beat” means the 4th beat of a quadruple measure although
it does not include the 2nd beat of a binary measure. “Pre-bar” means that the subject restarted at the
bar preceding where the subject stopped performing. “1st beat,” “2nd downbeat,” and “last beat” do not
include the notes in “start of piece,” “column,” “start of phrase,” and “alternation.”
Subjects A and B made 381 and 224 mistakes, respectively. Surprisingly, the cases of replaying were
377 times in the 381 errors in Subject A’s performances, and 181 times in the 224 errors in Subject B’s.
Moreover, “1st beat” accounted for about 75% of all errors. Thus, we need to consider not only extra,
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
10
•
C. Oshima et al.
wrong, missing but also replaying errors for dealing with the novice childrens’ performances in the
score tracking function.
5.2
Details of Score Tracking Function
We designed the score tracking function in response to the results of Section 5.1. The score tracking
module in FE (Figure 1) consists of this function.
Let the number of notes in Primo’s score be N , where a chord is regarded as a note and is represented
by its highest note, and let “performance time” be t j where j is the number of notes performed by Primo.
Now, assuming that the current position at performance time t j −1 was decided at note Si (1 ≤ i ≤ N )
in the score, Primo performed note P j at performance time t j . In this case, our score tracking function
determines the correspondence between the note in the score and the performed note P j based on the
following algorithm:
(1) First, Pitch(P j )—the pitch of the performed note P j , is compared with Pitch(Sk ) of each note Sk (1 ≤
k ≤ N ) in the score. Then, the weight of each note Sk at performance time t j , W (Sk , t j ), is calculated
as follows:
(a) if Pitch(Sk ) = Pitch(P j ),
then W (Sk , t j ) := W (Sk−1 , t j −1 ) + 1
(b) else W (Sk , t j ) := W (Sk−1 , t j −1 ) − 1,
where if W (Sk−1 , t j −1 ) − 1 < 0
then W (Sk , t j ) := 0
(2) If W (Si+1 , t j ) = W (Si , t j −1 ) + 1, then an error is detected. In this case the following correction
algorithm is applied.
(a) To correct a replaying error, let
W (Sh , t j ) := W (Si+1 , t j ) − m,
where Sh is the point in the score to which the beginner is likely to return when he/she stops
playing, as enumerated in Section 5.1. At present, m is empirically set to 2.
(b) To correct extra, wrong, and missing errors, let
W (S g , t j ) := W (Si+1 , t j ), where S g is all notes from Si−r to Si+r . At present, r is empirically set
to 2.
(3) Finally, the score tracking module chooses Sk as the note whose weight W (Sk , t j ) is the largest in
the current position at performance time t j . If there are multiple notes whose weights are equally
large, the module chooses one note as follows:
(a) If W (Si+1 , t j ) is the largest, then Si+1 is chosen as the current position.
(b) If W (Si+1 , t j ) is not the largest, then:
(i) Note Sk whose weight is the largest and nearest to note Si is chosen as the current position.
(ii) If two notes Si−d and Si+d have the same (largest) weight and are both nearest to note Si ,
then Si−d is chosen as the current position.
Figure 3 illustrates our score tracking algorithm.
This algorithm achieves robust score tracking even for performances by children with little musical
experience, because the function copes with replaying errors as well as the three ordinary errors.
Figure 4 shows sample processing situations of the score tracking function. (a) shows results in cases
in which the system corrects replaying errors. (b) shows results in cases in which the system does not
correct replaying errors. Columns correspond to notes described in the score. Rows correspond to notes
performed by Primo, P j . The hatched part of the performed notes shows where Primo replayed. Primo
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances
•
11
Start
Primo performed a note Pj at performance time tj
k
1
Yes
k > N?
N is the number of notes in Primo’s score
No
Yes
No
Pitch(Sk) = Pitch(Pi) ?
Sk is the kth note in the score
Yes
The weight of each note Sk
at performance time tj
W(Sk, tj)
W(Sk-1, tj-1) + 1
The weight of each note Sk
at performance time tj
W(Sk, tj)
No
W(Sk, tj)
0
k+1
W(Si+1, tj) = W(Si, tj-1) + 1 ?
where Si is the estimated performed note
at performance time tj-1
Yes
Si+1 is determined as the
performed note
at performance time tj
W(Sh, tj)
W(Si+1, tj) – m
for each note Sh to which the beginners
are likely to return. (m is empirically set to 2)
W(Sg, tj)
W(Si+1, tj)
for each note Sg that is included between
Si+r and Si+r (r is empirically set to 2)
Are there multiple notes
whose weights W(Sk, tj) (k=1~N)
are the largest?
The weight of each note Sk
at performance time tj
W(Sk-1, tj-1) - 1
k
No
W(Sk-1, tj-1) – 1 > 0 ?
Yes
Are there multiple
notes Sk whose weights W(Sk, tj) are
the largest and whose distances from
the note Si are the nearest?
No
Yes
Sk(k<i) is determined as
the performed note
at performance time tj
Sk is determined as
the performed note
at performance time tj
No
Sk whose weight W(Sk, tj) is the largest
is determined as the performed note
at performance time tj
End
Fig. 3. Score tracking algorithm.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
12
•
C. Oshima et al.
Fig. 4. Results of score tracking.
replayed “sol, la, sol, mi” although she should have played “mi, sol, sol, sol.” “mi” is the 14th note in
the column. Furthermore, cells that include outline characters show the estimated current position
determined by the score tracking function. Cells surrounded by a black border show the actual correct
notes that Primo is playing at each point.
In Figure 4(a), we can see that notes “sol” and “do” have weight “11” at the start of replaying. Beforehand, we had appointed these two notes to correspond to the start of bars. Accordingly, the system
could follow Primo’s performance after mistakes of only three notes. On the other hand, in Figure 4(b)
the system mistook thirteen notes before correctly following Primo’s performance. Thus, we see that
our score tracking algorithm, which is added to correct replaying errors, is practical.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances
•
13
This algorithm is sufficiently robust as for usual musical pieces. In some extreme cases, however, it
might not be able to recover after an error. For example, if one makes a mistake during the first eight
bars of “One Note Samba” composed by A. C. Jobim, which consists only of notes of an identical pitch
(F), it is impossible to find the correct position. The time values of notes should be considered when
tracking such a piece. It is also difficult to track a piece that only consists of an identical motif, for
example, a simple repeat of “do, re, mi, re” over and over. As for the processing time of this algorithm,
although it scans the entire piece each time, pieces that beginners perform are usually short and simple.
Practically, therefore, the total processing time can be ignored.
6.
TRIAL EXPERIMENTS OF FAMILY ENSEMBLES
To investigate the effects of FE for ensemble systems, we asked five pairs to practice duo play with/
without FE.
6.1
Purpose of Experiments
The purpose of our experiments was to clarify following two questions:
(1) Can even a musically inept performer who takes charge of the Secondo part play a duo with a
beginner who takes charge of the Primo part?
(2) Do even a musically inept performer and a beginner consider their musical expressions?
We conducted experiments with five pairs and analyzed the effects of FE from the number of performances and dialogues in joint practices and the results of questionnaires.
6.2
Methods
6.2.1 Subjects. We employed the following five pairs of subjects:
Pair A Primo was a female junior high school student who had studied piano for eight years (intermediate). Secondo was her father who had never played the piano, and so it was impossible for him to
read the scores (musically inept performer).
Pair B Primo was a female elementary school student who had played piano for four years (beginner).
Secondo was her mother, who had studied the electronic organ for two years (extreme beginner).
Pair C Primo was a nursery school child, the other daughter of the Secondo of Pair B. She had been
studying piano for one year (extreme beginner).
Pair D Primo was a male student at our institute who had studied the electronic organ for eight years
(intermediate). Secondo was a male student at the same institute who had never played the piano, and
so it was impossible for him to read the scores (musically inept performer).
Pair E Primo was the same as Pair D. Secondo was another male student at the same institute who
had never played the piano, and so it was impossible for him to read the scores (musically inept performer).
6.2.2 Setting and Set Pieces. Our experiments were conducted on an acoustic grand piano, YAMAHA
Silent Ensemble Grand Piano Proffessional Model C5 that output MIDI data in a soundproof room at
the second author’s institute. The piano was connected to a personal computer on which the FE system
operated and to a sound generator. In both situations, with and without FE, we used the sound generator
and the sound was output from speakers placed under the piano.
In joint practice with FE, Secondo was allowed to use any keys below C3, which Primo did not need
to use for playing the set pieces. Moreover, Secondo used a special score, a so-called “rhythm score”
(Figure 5), in which a chord is represented by piled note symbols in numbers that correspond to the
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
14
•
C. Oshima et al.
2
4
Primo
2
4
Secondo
2
4
Fig. 5. Example of score for Family Ensemble.
notes in the chord. Secondo can play a chord by hitting either only one key or multiple keys. By hitting
multiple keys to play a chord, Secondo can separately control the volume and length of each constituent
note in the chord.
We prepared three set pieces for the duo play that the subjects all knew well: “Prelude Opus28-7”
and “Waltz Opus18” by Frederic Chopin, and “Takibi” (“Bonfire”), a popular Japanese children’s song.
According to each Primo’s ability, we decided which set piece to ask them to play, and asked each Primo
to practice the given set pieces beforehand until they could perform them with few mistakes.
We also asked the subject who played Primo for Pairs D and E to try to use minus-one software
which is a prerecorded performance data of Secondo’s part. We prepared minus-one software for two
set pieces, “Takibi” and “Prelude.” The set piece, “Takibi,” was prepared in three tempo patterns: 44
(slow), 60 (medium), and 72 (fast) beats per minute. Another set piece “Prelude” was also prepared in
three tempo patterns: 72 (slow), 88 (medium), and 104 (fast) beats per minute.
6.2.3 Procedure of Experiments. We asked each pair to jointly practice the duos, both with and
without the use of FE. We conducted the experiments without FE before the experiments with FE in all
pairs. With FE, the mental load on Secondo’s playing is less than without FE. We wondered if Secondo
tried joint practice without FE after that with FE, the difficulty of the performance on a conventional
keyboard would be exaggerated. Accordingly, all pairs were first asked to try joint practice without FE.
First, Pairs A, B, and E had joint practice for fifteen minutes to perform “Prelude,” and Pairs C and
D had joint practice to perform “Takibi” without FE. Then we explained how to use FE and let Secondo
try it, using a short trial piece. Next, all pairs had joint practice for fifteen minutes with FE using the
same set piece as the former joint practice without FE. After that, Pair A had two more joint practice
sessions with FE for ten minutes since Primo of Pair A could play all of the set pieces. Last, the Primo
of Pairs D and E tried to perform “Takibi” and “Prelude” with minus-one software in three different
tempos for each piece.
6.2.4 Questionnaires. After recording the joint practice with and without FE, the subjects answered
the following questions.
To both Primo and Secondo
Question 1 Is it difficult for Secondo to perform a set piece without FE? (“1” is very easy, “5” is very
difficult)
Question 2 Why do you think so? (Optional)
Question 3 Is it difficult for Secondo to perform the set pieces with FE? (“1” is very easy, “5” is very
difficult)
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances
•
15
Table III. Number of Performances in Joint
Practice Sessions Without and With FE
Pair A
Pair B
Pair C
Pair D
Pair E
∗
Piece
Prelude
Prelude
Takibi
Takibi
Prelude
Sum
Without FE
Duo
Solo
0
0
5∗
0
0
0
0
0
0
0
5
0
With FE
Duo
Solo
13
1 (Primo)
14
0 (Primo)
2
4 (Primo)
5
7 (Primo)
9
1 (Primo)
43
13
The Secondo played the right-hand part.
Table IV. Answers to Questions 1, 3, 6, and 8
Pair
A
B
C
D
E
∗
Part
Primo
Secondo
Primo
Secondo
Primo
Secondo
Primo
Secondo
Primo
Secondo
Q1: Without FE
Prelude
5
5
Prelude
4
5
Takibi
5
5
Takibi
5
5
Prelude
5
5
Takibi
1
2
1
1
5
2
2
2
2
2
Q3: With FE
Prelude
Waltz
1
3
4
2
4
—
2
—
—
—
—
—
—
—
—
—
2
—
2
—
Q6: Use again?
Yes
Yes
Yes
Yes
Yes
—
Yes
Yes
—
Yes
Q8: with whom?
Father
Daughter, Wife
Sister
Daughter
Sister
—
Girlfriend
Friend
—
Friend
1: very easy to 5: very difficult.
Question 4 Why do you think so? (Optional)
Question 5 How do you feel about Secondo being able to perform with only one finger? (Optional)
Question 6 Do you want to play a duo with FE again? (“Yes,” “Maybe,” and “No”)
Question 7 Why? (Optional)
Question 8 Who do you want to play a duo with?
Question 9 Please comment on any expected improvements, and so forth. (Optional)
About minus-one software to the Primo of Pairs D and E
Question 10 Could you perform the set pieces with the correct melody and rhythm?
Question 11 Could you express agogics and fermata (subtle changes in tempo)?
Question 12 Could you express dynamics?
6.3
Comparison of the Number of Performances, and Results of Questionnaires
Table III shows the number of subjects playing duo and solo fifteen minutes in joint practice with and
without FE. We counted the frequency of a pair or one of the performers playing from start to end even
if there were many errors. Consequently, even if Secondo could only perform with his/her right hand
in a duo, we considered it one duo, as shown in Pair B. Table IV shows the results of Questions 1, 3, 6,
and 8, and Table V shows the results of Questions 10, 11, and 12 concerning the minus-one software.
We see that none of the pairs could complete the duo in the joint practice without FE. On the other
hand, with FE they immediately performed the duo more than ten times in the fifteen-minute session
(see Table III). In the joint practice sessions of Pairs A, B, D, and E without FE, Primo was quite occupied
telling Secondo which key(s) to hit. According to the answers to Question 2, especially the Secondos of
Pairs A, D, and E did not know how to read notes or the correct position corresponding to each note in
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
16
•
C. Oshima et al.
Table V. Answers to Questions 11, 12, and 13: with
Minus-One Software
Tempo
Slow
Medium
Fast
Q10: Melody, Rhythm
No
Marginal
No
Q11: Agogic
No
No
No
Q12: Dynamics
No
No
No
the score. Hence, the results of Question 1 in Table IV show that most Primos and Secondos thought it
was very difficult to play a duo without FE (answers with “5”).
Moreover, notice the increase in the number of solo practice sessions by the Primos in joint practice
with FE (see Table III). In the answer to Question 9, the Primo of Pair D wrote that FE prompted
him to practice playing the piano. Since Primos’ performances became poorer than those of Secondos
supported by FE, Primos wanted to recover domination by practice. The Primo of Pair A told us that she
felt frustrated that she could not blame her father when the performance was not perfect (see answers
to Question 5). This result suggests that FE can encourage children who are studying the piano to
practice.
The subject who played the Primo of Pairs D and E tried to use the minus-one software in “Takibi”
and “Prelude.” Table V shows that he could almost perform it in medium tempo. However, he answered
that he could not perform the subtle changes in tempo and dynamics, the musicality, in any tempo (see
answers to Questions 11 and 12).
According to the answers to Questions 4 and 7, the Secondos of Pairs D and E felt that it was wonderful
that they could perform the pieces from the start to the end. According to the answers to Question 5,
none of the Primos in any of the pairs felt envious of the FE support for their partner. According to the
answers to Question 6, all subjects want to use FE again. The subjects of Pairs D and E wanted to use
FE with their friends for fun (see answers to Question 8).
On the other hand, the Secondo of Pairs B and C, who had some experience playing the organ,
answered that she hopes to be able to play the piano more naturally. In the session of Pair B without
FE, she tried to perform only the right-hand part. The rate of correct notes that she played was 76.7% of
all right part notes on the score in fifteen minutes. In her answer to Question 9, she proposed a means
of support such that the Secondo can practically perform the right-hand part with the left-hand part
supported by FE.
Table VI shows the results of score-tracking in joint practice with FE. We counted the number of
erroneous notes classified as: extra, wrong, and missing. In the case of replaying, we counted the
number of notes between the current position and the start position of replaying. The 4th column
indicates the number of wrong notes before FE could track a correct position of Primo’s playing. We
could not count the notes when they finished the performance just after Primo’s error before the system
synchronized again. This is listed as “unknown.” “Combination” means that Secondo made errors in
succession. The average number of incorrectly estimated current positions before synchronizing again
is 2.77 (the “unknown” cases are not included in this calculation): the average number for extra, wrong
and missing types of error is 2.28, and that for replaying type of error is 3.83.
Even a human usually needs about 3 notes to synchronize again when the partner suddenly returns
to somewhere before. Therefore, we can say that this performance is acceptable. The worst result is
“10 notes” till FE could find Primo’s correct current position again, although Primo made mistakes on
only two notes. This is inevitable, however, since two wrong notes also could indicate another position
in the score by accident. Primo played “ra, fa, sol, do, do” although she should have played “ra, fa, sol,
mi, mi.” On the other hand, the phrase “ra, fa, sol, do, do” coincides with another position. As a result,
it took 10 notes till FE found Primo’s current position. It is even difficult for a human to find the right
position quickly in such a case.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
•
A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances
17
Table VI. Results of Score-Tracking
1
1
2
1
The Number of Incorrectly Estimated Current
Positions Before Synchronizing Again
1
2
4
6
3
1
2
4
unknown
1
2
3
4
5
10
unknown
4
5
6
4
1
5
unknown
1
2
unknown
4
6
1
unknown
1
4
8
unknown
6
unknown
2
5
Number
1
1
1
2
1
26
2
1
4
2
2
1
1
2
1
2
1
1
1
1
1
1
2
1
1
5
2
1
1
8
1
1
1
8
1
1
2
1
1
4
1
1
3
1
1
4
1
1
5
1
1
6
1
1
4
1
2
5
2
2
3
2
extra
The Number of Error Notes
1
Number
5
wrong
3
1
1
33
2
11
3
3
1
2
1
1
3
1 (back)
2
7
2 (back)
3
3 (back)
9
4 (back)
11
6 (back)
8 (back)
12 (back)
22 (back)
2
4 (back)
1
2 (back)
1
2 (back)
2
3 (back)
1 (back)
1 (back)
3 (back)
3 (back)
3 (back)
3 (back)
1
1 (back)
1 (back)
1
2 (back)
2
2 (back)
missing
replaying
combination
combination
combination
combination
combination
combination
combination
combination
wrong
replaying
wrong
replaying
wrong
replaying
missing
replaying
replaying
replaying
replaying
replaying
replaying
replaying
missing
replaying
replaying
wrong
replaying
wrong
replaying
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
18
6.4
•
C. Oshima et al.
Example of Subjects’ Dialogues About FE
In this section, we show two effects of FE from examples of subjects’ dialogue about joint practice with
FE. One is that FE prompts Primos to practice more enthusiastically. The other is that Primo and
Secondo practiced to increase their musically expressive duo play.
6.4.1 Motivating Children to Practice More. In joint practice with FE, the following dialogue took
place between Pair A child and her father:
Father & Child: Duo Play
(Made a bad start and stopped immediately.)
Father: omae mazu rensyu sei!
(Do your practice first!)
Child:
un rensyu suruwa
(I see. I am going to practice.)
Child:
Solo Play
Child:
yosya ikou
(OK! Let’s go!)
Father: eeka?
(Are you ready?)
Child:
otosan to hikuto yarinikuiyona
(It is tricky to play with you, dad.)
[
Father:
nandeyanen
(Why?)
Father & Child: Duo Play
By using FE, the parent and child are on almost equal ground concerning reproducing the correct
pitches. Consequently, if the child cannot perform the piece perfectly because of a lack of practice,
the parent can achieve the rank of the child. In this dialogue, the child said, “It’s tricky to play with
you, dad.” showing that she is impatient with her incomplete performance. After all the experiments,
her comment relates to the flawed performance of the Primo part, and reproducing correct pitches
(answer to Question 5). Therefore, she is frustrated that she could not blame her father when the
performance was not perfect. However a child’s impatience may promote more enthusiastic practice.
The results of the experiments show that in joint practice using FE, the Primos practiced by themselves
(Table III).
6.4.2 Collaboration for Improved Musical Performances. The fact that the parents could play their
part correctly and easily with the help of FE also had a desirable effect on another aspect of musicality.
The following dialogue took place between Pair A father and child in joint practice with FE:
Child:
Father:
Child:
Child:
ano ko tan tan tan tte hikuto oto togireru wakeyo=
(Well, your key touch is like: tan tan tan,
a way of touch that indicates disconnected tapping sounds.)
=un un un
(yeah, yeah, yeah)
dakara motyotto sa
(So, a bit more like this)
Solo Play
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances
Father:
Child:
•
19
un motto ko nobasette iuno? =
(You mean I should sustain the sounds, right?)
=so
(Right!)
In other words, the child wanted her father to play his part more legato. Such conversations about
musical expression were never found in the joint practice of any of the pairs without FE, the Secondo
considered nothing other than performing his/her part. Moreover though the father of Pair A was a
musically inept performer, he was able to learn that their timing was off at the dotted notes:
Child:
Father:
Child:
Child:
Father:
o dekitayan=
(Oh! We finished!)
=a koko aitaine
(Well, I want the timing in this part to be perfect.)
Circling the dotted notes with his finger
ta-n ta tan
(ta-n ta tan)
Playing the dotted notes
[
so so ta-n ta tan un kokodake rensyusuru
(Yeah yeah, ta-n ta tan. O.K. Let’s do it from this part?)
Primarily, ensemble practice should be a time when the players construct their own performance by
conveying their musical ideas and performance plans to each other. Without FE, such musical expression
discussions could not be achieved because of the knowledge gap between the Primo and Secondo (e.g.,
the Secondo cannot even read a normal score) and because of the Secondo’s lack of technical ability.
7.
DISCUSSION
Table III shows the first effect of FE: even a totally inexperienced performer can quickly play a duo
with a beginner. The number of duo plays in joint practice with FE is much more than in joint practice
without FE. Table III also reveals a second effect of FE: Primo (the child) begins to practice by herslf.
The number of Primo’s solo plays in joint practice with FE is more than in joint practice without it. We
argue that this result is not just caused by the decrease of Secondo’s solo play time but due to support
from FE. When Secondo is supported by FE, it is clear that the errors in their duo performances were
caused by Primo. Moreover, both Primo and Secondo clearly noticed these facts. Consequently, Primo
lost her superiority over Secondo. Primo’s willingness to practice had to rise. Primo’s utterance, “It’s
tricky to play with you, Dad (Section 6.4.2),” also shows her irritation.
The dialogues indicate that they were talking about musical expression, even though Secondo has
no experience with playing the piano. Such conversations about musical expression were never found
in the joint practice of any of the pairs without FE: Secondo considered nothing other than performing
his/her part. Both Primo and Secondo played music with as good musical expression as they wanted
through practice.
We believe there are two reasons why they immediately aimed for richer musical expression, that is,
talking about musical expression despite their being a musically inept performer and a beginner. First,
FE supports Secondo in outputting correct notes to be played at each point. By easing the cognitive
load in the performance due to nonexpressive elements, by providing accurate reconstruction of the
sequence of notes, FE allows Secondo to externalize more of his/her own musical expression (Section
4.2.2). Consequently, the technical gap between Primo, the beginner, and Secondo, the musically inept
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
20
•
C. Oshima et al.
performer, has narrowed. Additionally, since Primo does not need to teach Secondo how to play the
correct notes, they can both afford to create their own musical expressions. Second, FE does not intervene
in Secondo’s musical expression at all. Expressive elements such as the volume and the length of each
note are output as Secondo controls them. Accordingly, even the supported Secondo has sufficient room
for musical expression (Section 4.2.3). Therefore, Secondo can also contribute to creating the pair’s
musical expression in a piano duo.
Note that although FE can help Secondo externalize the musical expressions that already exist in
his/her mind, it can neither create nor add new musical expression for him/her. We should mention that
the keyboard for the parent’s part of FE is not a piano but another “new” musical instrument: the way
to perform this part is seemingly similar but actually very different from a conventional piano. This is
based on a practical decision that the parent need not master the piano: the one who should master it is
the child. Therefore, we neither modify the child’s musical instrument nor directly support the child’s
part.
8.
CONCLUSION
We proposed “Family Ensemble” (FE), which allows a musically inept parent to enjoy accompanying
their child’s piano practice, as an alternative to a completely automated accompaniment. FE allows the
parent to output the correct sequence of pitches along with the child’s performance by support from
note-replacement and score tracking functions even if he/she touches keys with just one finger. Although
we implemented the score tracking function based on Dannenberg’s score tracking algorithm, which
can deal with the extra, wrong, and missing types of errors, we extended it to cope with a peculiar error
of beginners: replaying. As a result, even if the child makes many errors and even if his/her parent has
little or no experience playing the piano, with FE they can immediately play a piano duo.
The results of our experiments show that FE allows even performers with no experience playing
the piano to begin to play a duo with the beginner. Furthermore, we show that Secondos, for example,
the parents, can sufficiently externalize their musical expressions as well as maintain enough space to
musically grow, even though being supported to reproduce correct sequence of pitches by FE. Eventually,
we found that in joint practices using FE some subjects discussed musical ideas to achieve richer musical
expression.
We would like to conduct additional experiments with more subjects and more musical pieces at
homes and at piano schools to assess the long-term usefulness of this system.
ACKNOWLEDGMENTS
We would like to thank the subjects.
REFERENCES
AGAY, D. 1987. Learning to Play Piano: Book 1. Yorktown Music Press.
BLAIN, T. AND PERKIS, T. 2000. The jum-o-drum interactive music system: A study in interaction design. In Proceedings of the
Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques. 165–173.
BLOCH, J. AND DANNENBERG, R. B. 1985. Real-time computer accompaniment of keyboard performance. In Proceedings of the
International Computer Music Conference (ICMC). International Computer Music Association (ICMA). 279–289.
CASIO. 2000.
LK-55 Full Size Keyboard with Key Lighting System. Casio, http://www.casio.com/index.cfm?fuseaction=
products.detail&product=LK%2D55.
DANNENBERG, R. B. 1984. An online algorithm for real-time accompaniment. In Proceedings of the International Computer
Music Conference (ICMC). International Computer Music Association (ICMA). 193–198.
EVERETT, C. 1999.
Piano Education Page—Artist/Educator Archive Interview—. Zeigler, J. M. and Ostromencki, N. L.,
http://pianoeducation.org/pnoevret.html.
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.
A Piano Duo Support System for Parents to Lead Children to Practice Musical Performances
•
21
FARBOOD, M., PAZSTOR, E., AND JENNINGS, K. 2004. Hyperscore, a graphical approach to composing music. IEEE Computer
Graphics and Applications, Special Issue on Emerging Technologies, 50–54.
GOTO, M., HIDAKA, I., MATSUMOTO, H., KURODA, Y., AND MURAOKA, Y. 1996. A jazz session system for interplay among all players—
virja session (virtual jazz session system)—. In Proceedings of the International Computer Music Conference (ICMC). International Computer Music Association (ICMA). 346–349.
GRUBB, L. AND DANNENBERG, R. B. 1997. A stochastic method of tracking a vocal performer. In Proceedings of the International
Computer Music Conference (ICMC). International Computer Music Association (ICMA). 301–308.
JENNINGS, K. 2003. Toy symphony: An international music technology project for children. Music Education International 2,
3–21.
KIHARA, R. 1991. Piano Land, vol. 1–5, with music data (an educational material in Japan). Ongakunotomo-sya.
LYKE, J. 2004.
Piano Education Page—Artist/Educator Archive Interview—. Zeigler, J. M. and Ostromencki, N. L.,
http://pianoeducation.org/pnojlyke.html.
MCCARTHY, C., BLIGH, J., JENNINGS, K., AND TANGNEY, B. 2005. Virtual collaborative learning environments for music: Networked
drumsteps. Computers & Education 44, 2 (February), 173–195.
MIYOSHI, A. 1977. Miyoshi Piano Method, vol. 1–8 (an educational material in Japan). Kawai.
MIYOSHI, A. 2000. Piano duo sympathetic world outside language. MUSICA NOVA 31, 9, 30–32.
ORIO, N., LEMOUTON, S., AND SCHWARZ, D. 2003. Score following: State of the art and new development. In Proceedings of the
New Interface for Musical Expression (NIME). 36–41.
OSHIMA, C., NISHIMOTO, K., AND SUZUKI, M. 2004. Family ensemble: A collaborative musical edutainment system for children
and inexperienced parents. In Proceedings of the ACM Multimedia (MM’04) and Co-Located Workshops. 556–563.
OSHIMA, C. AND SHIMOJIMA, A. 2003. Grounding function of instrument plays in dialogue: A case-study of piano duos in joint
practice. In Proceedings of the 25th Annual Conference of the Cognitive Science Society.
PACHET, F. AND ADDESSI, A. 2003. When children reflect on their playing style: Experiments with the continuator and children.
ACM Computers in Entertainment 2, 1, Article 05.
A probabilistic expert system for automatic musical accompaniment. J. Comput. Graph. Stats. 10, 3,
RAPHAEL, C. 2001.
487–512.
RENCON. 2000. Rencon (performance RENdering piano Contest). http://shouchan.ei.tuat.ac.jp/∼rencon/index.shtml.
SMOLIAR, S. W., WATERWORTH, J. A., AND KELLOCK, P. R. 1995. pianoforte: A system for piano education beyond notation literacy.
In Proceedings of Multimedia. ACM.
SUBOTNICK, M. 1997. ghost. http://www.mortonsubotnick.com/about.html.
TAKEUCHI, Y. AND KATAYOSE, H. 1995. Representation of music expression with two finger piano. In IPSJ SIGNotes Music and
Computer. IPSJ, 37–44.
TOYODA, K. S. 1995. Compact Disc, Japanese Song Without Instrumental Accompaniment. Victor Entertainment, VICC-169.
VERCOE, B. 1984. The synthetic performer in the context of live performance. In Proceedings of the International Computer
Music Conference (ICMC). International Computer Music Association (ICMA) 301–308.
ZEIGLER, J. AND OSTROMENCKI, N. 2004. Piano Education Page—Tips Performance—. http://pianoeducation.org/pnotperf.html.
Received April 2005; revised October 2005, February 2006; accepted May 2006
ACM Transactions on Multimedia Computing, Communications and Applications, Vol. 3, No. 2, Article 9, Publication date: May 2007.