Download Bridging the gap between performers and the audience using

Bridgingthegapbetweenperformersandtheaudience
usingnetworkedsmartphones:
thea.belsystem
1,2
1
1,2
1,2
AlexandreResendeClément ,FilipeRibeiro ,RuiRodrigues ,andRuiPenha 1
INESCTEC,Porto,Portugal
2
FEUP,Porto,Portugal
[email protected]
Abstract.Thea.belprojectaimstoprovideartistswithawaytoeasilyinteractwiththeiraudience,makinguseoftheir
participationtoeffectivelycraftuniqueperformances.Thispapergivesanoverviewofthea.belsystemanddetailsthe
developmentofasuiteoftools(aswellasitsintegrationintomobileapplications)withwhichmultimediaartistscan
easilycreateanddistributeinteractivecontentuntomobiledevices.
Keywords:music,interaction,interactiveart,collaborativesystem,smartphones,audienceparticipation
Introduction
Theuseofsmartphoneshasbeensteadilyincreasingduringthelastdecadeand2015markedtheyearwhenmost
WesternEuropeancountriessurpassed50%smartphonepenetrationamongthetotalpopulation(Google2016).These
devicescombineagrowingclusterofsensors,networkingtechnologiesandhighqualitymultimediacapabilities.Yet,we
stillmostlyusethesedevicesaspersonalcomputingdevices,usingthemtocommunicateaswedidwith2Gmobile
phones(i.e.voicecallsandSMS),toplayorworkindividually(e.g.readingthenews,playingsingleusergames,listening
tomusic,editingdocuments),andtoaccessservicesthroughtheweb(e.g.socialnetworks,internetbrowsing,video
calls).Webelievethatsmartphonescanalsobeusedtofosterreal-timeinteractionbetweenuserssharingthesame
physicallocation,andthusweproposeasolutiontoexplorethelargescaleuseofsmartphonesinamusicconcert
settingtobridgethegapbetweenperformersandtheaudience.
Background
Digitalorchestras
Eventhoughtheuseoflaptopsandmobilephonesinnetworkedlivemusicalperformancesystemsisnotnew,these
devices’rolehasmostlyremainedasthatofaninstrumentdedicatedtoperformeruse.StanfordandPrinceton
universities,forexample,havebothimplementednetworkedsystemsforliveperformanceusinglaptopcomputers,
whichbecamethestapleformostlaptop-orchestraimplementations(TruemanandCook2006;Wangetal.2009).Each
ofthelaptopsisconnectedtoacommonnetwork,communicatingwithacentralcomputerserver,andisoperatedbya
performertoexecuteagivenmusicalpiece.Mobilephonesalsohavebeenusedinasimilarway,takingtheplaceof
laptopsinsimilarlyorganizedmobilephoneorchestras(Wang,Essl,andPenttinen2008;Oh,Herrera,andBryan2010).
Elementsinteractwitheachothertocreateaperformancewheretheindividualcomposer/performerisresponsiblefor
theestablishmentoftherules,muchlikeacomposerwouldgoaboutcreatingamusicalpieceforaconventional
orchestra.
Audienceparticipationusingportabledevices
GolanLevin’s2001Dialtones(ATelesymphony)madeuseoftheaudience’scellphoneringtonesasaninstrument(Levin
2001),namelybyuploadingringtonestothecellphonesandcallingeachoneatagiventime.Thisapproachisclosely
relatedtotheonewepropose,asitmakestheaudienceapartoftheperformance,evenifinaratherpassiverole.
Since2012theBritishbandColdplayhasbeenusingadevicecalledXylobandsto“engageaudiencesandmakethem
partoftheshow”(Xylobands2016).AsimilartechnologyisusedbyPixMobtocreate“interactive,immersiveand
unforgettablecrowdexperiences”(PixMob2016).Thesearesomeexamplesofthegrowingwillingnessofpopularmusic
actstoadoptnovelwaystoincreaseinteractionwiththeiraudience.Infact,theresponsiveness,precision,reliability,
andtimingrequirementsofamusicalperformanceprovideaninterestinganddemandingtestingenvironmentforboth
novelinterfacesandcomputernetworksinthecontextofmusicalinteraction.
a.bel
Itisnowadaysincreasinglysafetoassumethatasignificantpartoftheaudienceinaconcertsettingwillbecarryinga
smartphoneintheirpocket.Thebridgingthegapbetweenperformersandtheaudienceusingnetworkedsmartphones
project,henceforthreferencedinthisdocumentbyitsworkingnamea.bel,aimstoprovideasolutiontotheuseof
networkedsmartphonestopromoteinteractioninmusicalconcerts,proposingaframeworkthatenablesthe
bidirectionalinteractionbetweenperformersandaudience,inaperformer-system-audienceinteractivesystem
(Bongers2000).Someexamplesofthecreativepotentialofthisnovelframeworkare:thedistributionofsound
emanatingfromtheaudience,enablingthecreationoflargediffusesoundfields,soundpathsandmusicalresponses
thatemergefromtheaudience;thedistributionofvideosignalsamongstthedevices,usingtheirscreensasasmallpart
ofalargerscreen;thenetworkcontroloversomemusicalparametersofeachdevice(e.g.currentharmonyortimbre),
thusfacilitatingthemusicalintegrationofeachperson’scontribution.Allofthisaspartof“asituation[...]wherethe
audienceandperformermeet,eachinfluencingtheother,asifconversing,whilemaintainingthequalityofthe
performanceatahighlevel.”(Bongers2000,49).Thepotentialofthissystem,however,isnotlimitedtomusicor
concertsettings:thesameframeworkcouldalsocontributetothedevelopmentoflocation-basedandtime-sensitive
networkedintelligentsystemsusingsmartphones,suchaslocation-basedmulti-usergames.
Sincethemainobjectiveofthisprojectwastoallowmultimediaartists,inparticularmusicalcomposers,tomakeuseof
audiencemembers’mobiledevicesaspartoftheirperformance,twomainissueswereaddressed:
•
•
Thedevelopmentofamulti-platform(iOSandAndroid)mobileapplicationthatprovides,totheartistsandfrontenddevelopers,aneasywaytodevelop,testanddistributecontentbuiltwithastandarddevelopment
environmentforinteractivemedia;andtotheuser,atransparentwaytoparticipateinliveperformancesettings.
Thedevelopmentofasolutiontoconnecthundredsofmobiledevicestoanetworkinaconcertenvironment
thatisreliable,cost-effective,andeasytodeployandoperate.
Thispaperfocusesonthefirstissueanddescribestheimplementationofthea.belapp,availableforiOSandAndroid,
withthenetworkimplementationdetailsgoingbeyonditsparticularscope.
ProposedFramework
Thea.belframeworkcanbeseparatedin3distinctoperativeblocks(asillustratedinfigure1).Themainapplication
consistsoftheperformancematerial,developedbythecomposerusingPureData(Puckette1997),andthenetworking
managementapplication,whichusestheimplementednetworkinfrastructuretosenddatatothemobileapplication.
ThedataissentfromtheMainApplicationtotheMobileApplication,whereittriggersaspecificeventorcreatesand
controlsanopportunityforuserinteraction.Thereturnoftheaudienceinteractionisthussolelybasedontheirsonic
andvisualcontribution,asthecommunicationfrommobiledevicestotheserverisnotcurrentlyimplemented.This
intrinsiclimitationoftheframeworkprovedtobe,however,averyinterestingchallengeforthecomposers,asthey
wereforcedtobuildself-containedmusical/sonicinstrumentsfortheaudience,eveniftheirbehaviourcanbealtered
viathenetwork.Thisway,eachaudiencememberhasalocalfeedback,andthusincreasedawareness,ofhisorher
contribution.
Figure1.Globalsystemstructure
Audience-sideassetsforthemobileapplicationcanalsobedevelopedusingPureData,asthemobileapplicationscarry
thelibPDwrapperlibrary(Brinkmannetal.2011),andaredistributedusingsimpleQR-codescanning.Schiemer’sPocket
Gamelan(SchiemerandHavryliv2005)implementsatoolallowingformusicalperformanceonmobilephonesusing
PureDataaswell,althoughitmakesuseofPD2j2meinordertoconvertPureDatapatchesintojavacode,allowing
developedassetstorunonthephones.ByusinglibPD,webypasstheneedofthisconversion,makingitsothemobile
applicationscanloadandrunthePDpatchesdirectly.Atoolsetofexternalobjectswasdeveloped,inordertogive
composerseasyaccesstothesystem’sdatacommunication,tothefunctionalitiesoftheaudiencemembers’devices
(sensordataoruserinterfacefeedback,suchastouchingorwavingthedevice),andallothersystemfunctionalities.The
applicationwasdesignedtooperateseamlessly,withtheembeddedpatchestakingcareofallaudiovisualcontent
associatedwiththeperformance,usingtheinformationgatheredbothfromcomposer/performanceprovideddata
(throughnetworkcommunication)andfromthedevice’sownsensorstopromoteinteractivity.1Wedecidedearlyon
that,inordertotakethedeviceoutofitsordinaryroleasapersonaldevicedestinedforindividualuse,thescreen
shouldbecompletelyfilledbyasingle,patch-controllablecolourwheninperformancemode,showingnouserinterface
elementsotherthananoptionaltextmessage.Thisdecisionprovedtobeaneffectivewayoftransfiguringthevery
personaldeviceasamusicalinstrumentduringtheperformance,andaudiencememberswereoftenseenwavingthe
devicesintheair,sometimesperformingimpromptugroupchoreographies.Theseinteractionsmightbeachievedby
meansofOSCcontrollermobileapplications,butthesearen’t,bythemselves,toolsformusicalexpressionor
collaborationtools,butmerelyawayofcontrollingremotesystems.
PureDatalibrary
LibPDisbasedonPd-Vanilla(Brinkmannetal.2011),whichmeansthatitonlyfeaturesalimitedsetofobjectsthatlack
functionalitiesthatwerecriticaltoachieveourgoals.Whilstsomeoftheseneedswerealreadycoveredviathird-party
objects,theneedtohavefinecontroloversomeofthenetworksettingsandaneasytouseplatformforthe
developmentofmobileassetsledustothedevelopmentofasuiteofPureDataobjects.Thistoolkitisembeddedinthe
mobilea.belapp,beingalsoavailablefordevelopmentusingMacOsX,WindowsandLinux.Table1givesanoverviewof
thecreatedtoolkit.
1
Examplesofusercontrolofmusicalfeaturesusingthea.belappcanbeseenat
http://vimeo.com/142726571andhttp://vimeo.com/142726073.
Category
Description
Function
Datacommunication
One-waydatatransmissionusingWi-Fi
and/ordataoversound
8-bitvaluetransmission,
performancecontrolmessages
Deviceinput
Devicesensorreadingbasedonuserinput
Accelerometer,touch,swipe
Deviceoutput
Devicevisualoutput
Screencolourchange,text
messageson-screenoutput
Auxiliary
Processstandardizationandsystem-specific
functionalities
Scaleremapping,datamessage
targeting
Table1.a.beltoolkitcomponents
Datacommunication
Tocopewiththeparticularfeaturesofthenetworkmodel,aparticularcommunicationprotocolbetweenmainand
mobileapplicationwasdeveloped.OurfirstapproachwastouseaWi-Finetwork,whichprovedareliablewayof
connectinghundredsofdeviceswithlowlatencybutrequiredaconsiderablenetworkinfrastructuretoguaranteethe
system’sstability.Atthetimeofthiswriting,wehavedevelopedandarecurrentlytestinganadditionalsolutionbased
onultrasoundcommunication.
WhenusingWi-Fi,dataissentandreceivedthroughaseriesofupto40dataslots.Thesedataslotscorrespondtoa
seriesofuser-definablevariables,whichareresponsibleforstoringthedatavaluestobesentvianetworktothemobile
devices.Thedatacommunicationobjectsaccept8-bitintegervalues,whicharegrouped,convertedandsentover
networkintheformofastringblock,andreadbackbytheapplicationuponreception.Thisstringiscomposedof2
differentcharactersforeachdataslotvalue,thefirstcorrespondingtothedataslotindex(forcorrectparsingand
outputintheembeddedpatch)andthesecondcorrespondingtothatslot’ssentvalue.
Whenusingultrasoundcommunication,dataissentusingamplitudemodulationofagivenfrequencyandreceivedby
demodulatingontheclientside.Thisfrequencycanbechosenbythecomposer,butithastotakeintoaccountthe
frequencyrangeofboththeemittingloudspeakersandthereceivingsmartphones.Thismodeofcommunicationonly
allowsfortwo8-bitintegervaluesandisnoticeablyslowerandlessreliablethanWi-Fi.Ithas,however,thepotentialof
enablingthecontroloverabiggernumberofdevicesandofunlockinglocalcommunicationbetweendevices.Itcanuse
theconcertvenue’sexistingsoundreinforcementinfrastructureandcanbeusedforreal-timelocalizationofdevices,
usingtimedifferenceofarrivaltechniques,somethingweplantoimplementinthenearfuture.
Inputmethods
Thetouchscreenisthemostubiquitoussensorinmodernsmartphones.Itallowsformultipleinputtypes,fromsimple
touch-coordinatereadingtocomplexgesturerecognition.Forthea.beltoolset,2interactionmodesareconsidered:
1.
touch:touch/tapeventcoordinatesaremadeavailableinanormalisedformasscreendimension
percentages(e.g.atapatcentrescreencorrespondstoa50/50coordinate);
2.
swipe:4differentswipegesturesaremadeavailable(up,down,left,right)asactiontriggers(triggerupon
gesturedetection);
Anotherverycommonsensorinsmartphonesistheaccelerometer.Thedevelopedobjectgivesthecomposeraccessto
rawaccelerationvalues,aswellascalculatedrollandtiltvalues,overallaccelerationmagnitudeandadeviceshake
actiontrigger.
Outputmethods
Acolourobjectprovidessimplebackgroundcolourchangefunctionality,allowingforindividualmanipulationoftheRed,
GreenandBluecomponentsofthecolour.Atextobjectallowstheartisttopassatextmessagefromthepatchtothe
application,showingiton-screen.Theseobjectscanbeusedtogethertoshow,e.g.asplashscreenshowingrelevant
informationattheonsetofeachpiece.
Auxiliarycomponents
Inadditiontothemainobjects,someadditionalauxiliaryobjectswerecreated,soastoprovidestandardizationofsome
processesandsystem-specificfunctionalities.Theseobjectsarenotasrelevantastheaforementionedonesandwillnot
bediscussedinthispaper.Thetoolkitdownload2includesstandardPureDatahelpfilestodemonstratethe
functionalityofalloftheobjects.
Testingthesystem
a.bel-novamúsicainteractivawastheeventwhichservedthepurposeofbothpresentingandtestingtheproposed
system.IttookplaceonOctober26th,2015atCasadaMúsica’sSalaSuggia,themainconcerthallatPorto,Portugal.It
featuredpiecesbyCarlosGuedes,JoséAlbertoGomes,NeilLeonardandRuiPenha,usingacousticinstruments,
electroacousticdevicesandthea.belsystem.3
Oneofthepieces—cellular,forsopranosaxophone,vibraphoneandliveelectroacoustics,inadditiontotheaudience
members’smartphones—wascomposedbyoneoftheauthors(RuiPenha).Thepiecestartedwithonlytheaudience’s
smartphoneswakingupfromblackscreenstodifferentshadesofevolvinggreenswhileproducingvarioussynthesized
animalandwatersounds,producingadiffuserainforestsoundscape.Thefactthateachsmartphoneproducedan
unique,isolatedandeasilylocatablesound—and,inthecaseofthewaterdrops,changedcolourwhiledoingso—
madethecontributionofeachsmartphoneveryeasytopinpoint.Thissectionhelpedtheaudiencetobeawareofthe
individualimpactoftheirsmartphonetothewholeexperienceandtobuildaserenestartingpointtothesubsequent
shifttowardsamorefreneticinteractiveexperience.
Thesecondsectionofthepieceintroduceduserinteractiontoaccompanyfast-pacedarticulationsofthesaxophone
andvibraphone.Audiencememberscouldinteractwiththemusiciansandwitheachotherbyusingtwomusical
gestures:
•
•
bywavingthesmartphoneintheairtoproduceslowlyevolvingdrones4;
bytappingthescreentoproduceshortnotesfromagivenharmonicseries,whichwerethenautomatically
repeatedwithvariabledelays5;
Theharmony,timinganddelaybehavioursofthesegestureswereautomaticallysynchronizedwiththeoutputfromthe
musiciansonstageviathea.belsystem.
Morethan700peopleattendedtheeventandlogsshowthatroughly70%wereabletoconnecttheirsmartphoneto
thesystemandsuccessfullymadeuseofit.Theremaining30%includepeoplethatdidnotcarryasmartphonetothe
concert,usersofotheroperatingsystems(e.g.WindowsPhoneorBlackberry),usersofincompatibleAndroidversions
(belowversion4.0),usersofincompatibleiOSversions(belowversion6.0)andasmallnumberofusersthatcouldnot
connecttothesystemforotherreasons,mainlyduetoincompatibleAndroidROMs.Duringtheeventnodisruptions
wereobserved,eitherinthenetworkdatastreamingorfromtheaudienceengagement.
2
Tobemadefreelyavailableintimefortheconference.
Anintegralvideooftheconcertcanbeseenathttp://youtu.be/X_YeO3Qi3H4.
4
Anexampleofthisinteractionmodecanbeseenathttps://vimeo.com/142726571.
5
Anexampleofthisinteractionmodecanbeseenathttps://vimeo.com/142726073.
3
Conclusionsandfuturework
Wehavedescribedtheinitialdevelopmentofa.bel,asystemforthedevelopmentofinteractiveperformer-systemaudienceexperiences,andwehaveshownhowitcanbeusedtobridgethegapbetweenperformersandaudiencesin
musicalconcerts.Theaudienceresponseatthesystems’premièreeventwasveryenthusiastic.6Thesystemwasproven
toworkasintendedand,giventheobservedresults,hasshownroomforconcreteimprovement,scalabilityandfurther
development.
Withtheemphasisbeingplacedontheabilitytoreliablyconnecthundredsofsmartphonesandtriggerbehaviourswith
verylowlatency,itwasearlyondecidedtopostponetoalaterstagethecapabilityofsendingdatafromthe
smartphonestothenetwork.Thecyclethatstartswiththeinformationbeingsentfromthestagetothesmartphones
is,thus,completedsolelybyanaudiovisualreturnthatispartiallydependentontheaudience’swillingnesstointeract.
Whilstwestillintendtoenabletheuploadofdatafromthesmartphonestothenetworkinthefuture,webelievethat
theexclusivelylocalaudiovisualfeedback—akintotheoneofatraditionalmusicalinstrument—isinfacta
characteristicthathelpsenduserstobemoreawareoftheirowncontributionthanwhatwouldbepossibleif,e.g.,the
datageneratedbytheirinteractionwastobemingledwithhundredsofothersourcesandusedtodriveanaudiovisual
feedbackcomingfromstage.
Wehaveonlybeguntoscratchthesurfaceofhowthissystemcanbeusedtoenableaudienceparticipationinmusical
concerts.Oneissuethatweneedtoaddressisthefactthattheaudiencehasnorehearsaltimewiththemusicians
beforetheconcert,whichcanlimitthequalityofthemorecomplexinteractions.Asolutionthatweplantotestisto
havetheconcertdividedintotwosections:aninformalrehearsalofallintendedinteractionmodesfollowedbythe
mainperformance.Anotherissueisthefactthatmostconcertvenuesweredesignedwiththeaudiencefacingthestage
and,consequently,theaudiencemembersdonotseeeachothereasily.Atthea.belconcert,thismeantthataudience
membersoccupyingthefrontrowsweresomewhatexcludedfromthecollectiveexperience.InspiredbyGolanLevin’s
Dialtones(ATelesymphony)(Levin2001),wedisplayedtheviewfromthestageovertheaudienceonabigprojection
abovethestage,butthissolutionencouragedsomeoftheaudiencemembers’tofindwaystoidentifythemselveson
theprojection,leadingtoamomentarydetachmentfromtheperformance.Weaimtoexploreadispositionofthe
audiencearoundthemusicians,thuswithaudiencemembersfacingeachother,akintothedispositionusedbyIannis
XenakisinTerretektorth(Xenakis1966).Finally,weaimtoexplorewaystopromotethedirectmusicalinteraction
betweenaudiencemembers,namelybycomposingpiecesthatrelysolelyontheaudience.
Acknowledgements.ThisworkwasfundedwithintheResearchLineFourEyesoftheProject“TEC4Growth-Pervasive
Intelligence,EnhancersandProofsofConceptwithIndustrialImpact/NORTE-01-0145-FEDER-000020”,financedbythe
NorthPortugalRegionalOperationalProgramme(NORTE2020),underthePORTUGAL2020PartnershipAgreement,
andthroughtheEuropeanRegionalDevelopmentFund(ERDF).
References
Bongers,Bert.2000.“PhysicalInterfacesintheElectronicArtsInteractionTheoryandInterfacingTechniquesforRealtimePerformance.”TrendsinGesturalControlofMusic2000(January):41–70.
Brinkmann,Peter,ChrisMccormick,PeterKirn,MartinRoth,andRichardLawler.2011.“EmbeddingPureDatawith
libpd.”InProceedingsoftheFourthInternationalPureDataConvention,291–301.
Google.2016.“ConsumerBarometer:Percentageofpeoplewhouseasmartphone.”https://www.
consumerbarometer.com/en/trending/?countryCode=FR&category=TRN-NOFILTER-ALL.
Levin,Golan.2001.“Dialtones(ATelesymphony).”
6
Mediacoverageoftheconcertcanbeseenathttp://youtu.be/fkCrtEPFTN4,http://youtu.be/07XwuHrHgW4,
http://youtu.be/aEalgQez28M,http://youtu.be/p2iGkpZZXN0andhttp://youtu.be/JofTAb2OIDU.
Oh,Jieun,JorgeHerrera,andNJBryan.2010.“Evolvingthemobilephoneorchestra.”Proceedingsofthe2010
ConferenceonNewInterfacesforMusicalExpression(NIME2010),pp.82-87.
PixMob.2016.“PixMobwirelessLEDtechnology.”http://pixmob.com.
Puckette,Miller.1997.“PureData:anotherintegratedcomputermusicenvironment.”ProceedingsoftheSecond
IntercollegeComputerMusicConcerts:37–41.
Schiemer,G.,andM.Havryliv.2005.“PocketGamelan:aPureDatainterfaceforMobilephones.”ProceedingsofNIME
05,pp.156-159.
Trueman,Daniel,andPerryCook.2006.“PLOrk:thePrincetonlaptoporchestra,year1.”Proceedingsofthe
internationalcomputermusicconference,pp.443-450.
Wang,Ge,NicholasBryan,JieunOh,andRobHamilton.2009.“Stanfordlaptoporchestra(slork).”Proceedingsofthe
InternationalComputerMusicConference(ICMC2009).
Wang,Ge,GeorgEssl,andHenriPenttinen.2008.“Domobilephonesdreamofelectricorchestras.”InProceedingsof
theInternationalComputerMusicConference.2008.
Xenakis,Iannis.1966.Terretektorh.EditionsSalabert.
Xylobands.2016.“RadiocontrolledLEDwristbandsforconcertsandliveevents.”http://www.xylobands.com/
xylobands/xylobands/.