Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Bridgingthegapbetweenperformersandtheaudience usingnetworkedsmartphones: thea.belsystem 1,2 1 1,2 1,2 AlexandreResendeClément ,FilipeRibeiro ,RuiRodrigues ,andRuiPenha 1 INESCTEC,Porto,Portugal 2 FEUP,Porto,Portugal [email protected] Abstract.Thea.belprojectaimstoprovideartistswithawaytoeasilyinteractwiththeiraudience,makinguseoftheir participationtoeffectivelycraftuniqueperformances.Thispapergivesanoverviewofthea.belsystemanddetailsthe developmentofasuiteoftools(aswellasitsintegrationintomobileapplications)withwhichmultimediaartistscan easilycreateanddistributeinteractivecontentuntomobiledevices. Keywords:music,interaction,interactiveart,collaborativesystem,smartphones,audienceparticipation Introduction Theuseofsmartphoneshasbeensteadilyincreasingduringthelastdecadeand2015markedtheyearwhenmost WesternEuropeancountriessurpassed50%smartphonepenetrationamongthetotalpopulation(Google2016).These devicescombineagrowingclusterofsensors,networkingtechnologiesandhighqualitymultimediacapabilities.Yet,we stillmostlyusethesedevicesaspersonalcomputingdevices,usingthemtocommunicateaswedidwith2Gmobile phones(i.e.voicecallsandSMS),toplayorworkindividually(e.g.readingthenews,playingsingleusergames,listening tomusic,editingdocuments),andtoaccessservicesthroughtheweb(e.g.socialnetworks,internetbrowsing,video calls).Webelievethatsmartphonescanalsobeusedtofosterreal-timeinteractionbetweenuserssharingthesame physicallocation,andthusweproposeasolutiontoexplorethelargescaleuseofsmartphonesinamusicconcert settingtobridgethegapbetweenperformersandtheaudience. Background Digitalorchestras Eventhoughtheuseoflaptopsandmobilephonesinnetworkedlivemusicalperformancesystemsisnotnew,these devices’rolehasmostlyremainedasthatofaninstrumentdedicatedtoperformeruse.StanfordandPrinceton universities,forexample,havebothimplementednetworkedsystemsforliveperformanceusinglaptopcomputers, whichbecamethestapleformostlaptop-orchestraimplementations(TruemanandCook2006;Wangetal.2009).Each ofthelaptopsisconnectedtoacommonnetwork,communicatingwithacentralcomputerserver,andisoperatedbya performertoexecuteagivenmusicalpiece.Mobilephonesalsohavebeenusedinasimilarway,takingtheplaceof laptopsinsimilarlyorganizedmobilephoneorchestras(Wang,Essl,andPenttinen2008;Oh,Herrera,andBryan2010). Elementsinteractwitheachothertocreateaperformancewheretheindividualcomposer/performerisresponsiblefor theestablishmentoftherules,muchlikeacomposerwouldgoaboutcreatingamusicalpieceforaconventional orchestra. Audienceparticipationusingportabledevices GolanLevin’s2001Dialtones(ATelesymphony)madeuseoftheaudience’scellphoneringtonesasaninstrument(Levin 2001),namelybyuploadingringtonestothecellphonesandcallingeachoneatagiventime.Thisapproachisclosely relatedtotheonewepropose,asitmakestheaudienceapartoftheperformance,evenifinaratherpassiverole. Since2012theBritishbandColdplayhasbeenusingadevicecalledXylobandsto“engageaudiencesandmakethem partoftheshow”(Xylobands2016).AsimilartechnologyisusedbyPixMobtocreate“interactive,immersiveand unforgettablecrowdexperiences”(PixMob2016).Thesearesomeexamplesofthegrowingwillingnessofpopularmusic actstoadoptnovelwaystoincreaseinteractionwiththeiraudience.Infact,theresponsiveness,precision,reliability, andtimingrequirementsofamusicalperformanceprovideaninterestinganddemandingtestingenvironmentforboth novelinterfacesandcomputernetworksinthecontextofmusicalinteraction. a.bel Itisnowadaysincreasinglysafetoassumethatasignificantpartoftheaudienceinaconcertsettingwillbecarryinga smartphoneintheirpocket.Thebridgingthegapbetweenperformersandtheaudienceusingnetworkedsmartphones project,henceforthreferencedinthisdocumentbyitsworkingnamea.bel,aimstoprovideasolutiontotheuseof networkedsmartphonestopromoteinteractioninmusicalconcerts,proposingaframeworkthatenablesthe bidirectionalinteractionbetweenperformersandaudience,inaperformer-system-audienceinteractivesystem (Bongers2000).Someexamplesofthecreativepotentialofthisnovelframeworkare:thedistributionofsound emanatingfromtheaudience,enablingthecreationoflargediffusesoundfields,soundpathsandmusicalresponses thatemergefromtheaudience;thedistributionofvideosignalsamongstthedevices,usingtheirscreensasasmallpart ofalargerscreen;thenetworkcontroloversomemusicalparametersofeachdevice(e.g.currentharmonyortimbre), thusfacilitatingthemusicalintegrationofeachperson’scontribution.Allofthisaspartof“asituation[...]wherethe audienceandperformermeet,eachinfluencingtheother,asifconversing,whilemaintainingthequalityofthe performanceatahighlevel.”(Bongers2000,49).Thepotentialofthissystem,however,isnotlimitedtomusicor concertsettings:thesameframeworkcouldalsocontributetothedevelopmentoflocation-basedandtime-sensitive networkedintelligentsystemsusingsmartphones,suchaslocation-basedmulti-usergames. Sincethemainobjectiveofthisprojectwastoallowmultimediaartists,inparticularmusicalcomposers,tomakeuseof audiencemembers’mobiledevicesaspartoftheirperformance,twomainissueswereaddressed: • • Thedevelopmentofamulti-platform(iOSandAndroid)mobileapplicationthatprovides,totheartistsandfrontenddevelopers,aneasywaytodevelop,testanddistributecontentbuiltwithastandarddevelopment environmentforinteractivemedia;andtotheuser,atransparentwaytoparticipateinliveperformancesettings. Thedevelopmentofasolutiontoconnecthundredsofmobiledevicestoanetworkinaconcertenvironment thatisreliable,cost-effective,andeasytodeployandoperate. Thispaperfocusesonthefirstissueanddescribestheimplementationofthea.belapp,availableforiOSandAndroid, withthenetworkimplementationdetailsgoingbeyonditsparticularscope. ProposedFramework Thea.belframeworkcanbeseparatedin3distinctoperativeblocks(asillustratedinfigure1).Themainapplication consistsoftheperformancematerial,developedbythecomposerusingPureData(Puckette1997),andthenetworking managementapplication,whichusestheimplementednetworkinfrastructuretosenddatatothemobileapplication. ThedataissentfromtheMainApplicationtotheMobileApplication,whereittriggersaspecificeventorcreatesand controlsanopportunityforuserinteraction.Thereturnoftheaudienceinteractionisthussolelybasedontheirsonic andvisualcontribution,asthecommunicationfrommobiledevicestotheserverisnotcurrentlyimplemented.This intrinsiclimitationoftheframeworkprovedtobe,however,averyinterestingchallengeforthecomposers,asthey wereforcedtobuildself-containedmusical/sonicinstrumentsfortheaudience,eveniftheirbehaviourcanbealtered viathenetwork.Thisway,eachaudiencememberhasalocalfeedback,andthusincreasedawareness,ofhisorher contribution. Figure1.Globalsystemstructure Audience-sideassetsforthemobileapplicationcanalsobedevelopedusingPureData,asthemobileapplicationscarry thelibPDwrapperlibrary(Brinkmannetal.2011),andaredistributedusingsimpleQR-codescanning.Schiemer’sPocket Gamelan(SchiemerandHavryliv2005)implementsatoolallowingformusicalperformanceonmobilephonesusing PureDataaswell,althoughitmakesuseofPD2j2meinordertoconvertPureDatapatchesintojavacode,allowing developedassetstorunonthephones.ByusinglibPD,webypasstheneedofthisconversion,makingitsothemobile applicationscanloadandrunthePDpatchesdirectly.Atoolsetofexternalobjectswasdeveloped,inordertogive composerseasyaccesstothesystem’sdatacommunication,tothefunctionalitiesoftheaudiencemembers’devices (sensordataoruserinterfacefeedback,suchastouchingorwavingthedevice),andallothersystemfunctionalities.The applicationwasdesignedtooperateseamlessly,withtheembeddedpatchestakingcareofallaudiovisualcontent associatedwiththeperformance,usingtheinformationgatheredbothfromcomposer/performanceprovideddata (throughnetworkcommunication)andfromthedevice’sownsensorstopromoteinteractivity.1Wedecidedearlyon that,inordertotakethedeviceoutofitsordinaryroleasapersonaldevicedestinedforindividualuse,thescreen shouldbecompletelyfilledbyasingle,patch-controllablecolourwheninperformancemode,showingnouserinterface elementsotherthananoptionaltextmessage.Thisdecisionprovedtobeaneffectivewayoftransfiguringthevery personaldeviceasamusicalinstrumentduringtheperformance,andaudiencememberswereoftenseenwavingthe devicesintheair,sometimesperformingimpromptugroupchoreographies.Theseinteractionsmightbeachievedby meansofOSCcontrollermobileapplications,butthesearen’t,bythemselves,toolsformusicalexpressionor collaborationtools,butmerelyawayofcontrollingremotesystems. PureDatalibrary LibPDisbasedonPd-Vanilla(Brinkmannetal.2011),whichmeansthatitonlyfeaturesalimitedsetofobjectsthatlack functionalitiesthatwerecriticaltoachieveourgoals.Whilstsomeoftheseneedswerealreadycoveredviathird-party objects,theneedtohavefinecontroloversomeofthenetworksettingsandaneasytouseplatformforthe developmentofmobileassetsledustothedevelopmentofasuiteofPureDataobjects.Thistoolkitisembeddedinthe mobilea.belapp,beingalsoavailablefordevelopmentusingMacOsX,WindowsandLinux.Table1givesanoverviewof thecreatedtoolkit. 1 Examplesofusercontrolofmusicalfeaturesusingthea.belappcanbeseenat http://vimeo.com/142726571andhttp://vimeo.com/142726073. Category Description Function Datacommunication One-waydatatransmissionusingWi-Fi and/ordataoversound 8-bitvaluetransmission, performancecontrolmessages Deviceinput Devicesensorreadingbasedonuserinput Accelerometer,touch,swipe Deviceoutput Devicevisualoutput Screencolourchange,text messageson-screenoutput Auxiliary Processstandardizationandsystem-specific functionalities Scaleremapping,datamessage targeting Table1.a.beltoolkitcomponents Datacommunication Tocopewiththeparticularfeaturesofthenetworkmodel,aparticularcommunicationprotocolbetweenmainand mobileapplicationwasdeveloped.OurfirstapproachwastouseaWi-Finetwork,whichprovedareliablewayof connectinghundredsofdeviceswithlowlatencybutrequiredaconsiderablenetworkinfrastructuretoguaranteethe system’sstability.Atthetimeofthiswriting,wehavedevelopedandarecurrentlytestinganadditionalsolutionbased onultrasoundcommunication. WhenusingWi-Fi,dataissentandreceivedthroughaseriesofupto40dataslots.Thesedataslotscorrespondtoa seriesofuser-definablevariables,whichareresponsibleforstoringthedatavaluestobesentvianetworktothemobile devices.Thedatacommunicationobjectsaccept8-bitintegervalues,whicharegrouped,convertedandsentover networkintheformofastringblock,andreadbackbytheapplicationuponreception.Thisstringiscomposedof2 differentcharactersforeachdataslotvalue,thefirstcorrespondingtothedataslotindex(forcorrectparsingand outputintheembeddedpatch)andthesecondcorrespondingtothatslot’ssentvalue. Whenusingultrasoundcommunication,dataissentusingamplitudemodulationofagivenfrequencyandreceivedby demodulatingontheclientside.Thisfrequencycanbechosenbythecomposer,butithastotakeintoaccountthe frequencyrangeofboththeemittingloudspeakersandthereceivingsmartphones.Thismodeofcommunicationonly allowsfortwo8-bitintegervaluesandisnoticeablyslowerandlessreliablethanWi-Fi.Ithas,however,thepotentialof enablingthecontroloverabiggernumberofdevicesandofunlockinglocalcommunicationbetweendevices.Itcanuse theconcertvenue’sexistingsoundreinforcementinfrastructureandcanbeusedforreal-timelocalizationofdevices, usingtimedifferenceofarrivaltechniques,somethingweplantoimplementinthenearfuture. Inputmethods Thetouchscreenisthemostubiquitoussensorinmodernsmartphones.Itallowsformultipleinputtypes,fromsimple touch-coordinatereadingtocomplexgesturerecognition.Forthea.beltoolset,2interactionmodesareconsidered: 1. touch:touch/tapeventcoordinatesaremadeavailableinanormalisedformasscreendimension percentages(e.g.atapatcentrescreencorrespondstoa50/50coordinate); 2. swipe:4differentswipegesturesaremadeavailable(up,down,left,right)asactiontriggers(triggerupon gesturedetection); Anotherverycommonsensorinsmartphonesistheaccelerometer.Thedevelopedobjectgivesthecomposeraccessto rawaccelerationvalues,aswellascalculatedrollandtiltvalues,overallaccelerationmagnitudeandadeviceshake actiontrigger. Outputmethods Acolourobjectprovidessimplebackgroundcolourchangefunctionality,allowingforindividualmanipulationoftheRed, GreenandBluecomponentsofthecolour.Atextobjectallowstheartisttopassatextmessagefromthepatchtothe application,showingiton-screen.Theseobjectscanbeusedtogethertoshow,e.g.asplashscreenshowingrelevant informationattheonsetofeachpiece. Auxiliarycomponents Inadditiontothemainobjects,someadditionalauxiliaryobjectswerecreated,soastoprovidestandardizationofsome processesandsystem-specificfunctionalities.Theseobjectsarenotasrelevantastheaforementionedonesandwillnot bediscussedinthispaper.Thetoolkitdownload2includesstandardPureDatahelpfilestodemonstratethe functionalityofalloftheobjects. Testingthesystem a.bel-novamúsicainteractivawastheeventwhichservedthepurposeofbothpresentingandtestingtheproposed system.IttookplaceonOctober26th,2015atCasadaMúsica’sSalaSuggia,themainconcerthallatPorto,Portugal.It featuredpiecesbyCarlosGuedes,JoséAlbertoGomes,NeilLeonardandRuiPenha,usingacousticinstruments, electroacousticdevicesandthea.belsystem.3 Oneofthepieces—cellular,forsopranosaxophone,vibraphoneandliveelectroacoustics,inadditiontotheaudience members’smartphones—wascomposedbyoneoftheauthors(RuiPenha).Thepiecestartedwithonlytheaudience’s smartphoneswakingupfromblackscreenstodifferentshadesofevolvinggreenswhileproducingvarioussynthesized animalandwatersounds,producingadiffuserainforestsoundscape.Thefactthateachsmartphoneproducedan unique,isolatedandeasilylocatablesound—and,inthecaseofthewaterdrops,changedcolourwhiledoingso— madethecontributionofeachsmartphoneveryeasytopinpoint.Thissectionhelpedtheaudiencetobeawareofthe individualimpactoftheirsmartphonetothewholeexperienceandtobuildaserenestartingpointtothesubsequent shifttowardsamorefreneticinteractiveexperience. Thesecondsectionofthepieceintroduceduserinteractiontoaccompanyfast-pacedarticulationsofthesaxophone andvibraphone.Audiencememberscouldinteractwiththemusiciansandwitheachotherbyusingtwomusical gestures: • • bywavingthesmartphoneintheairtoproduceslowlyevolvingdrones4; bytappingthescreentoproduceshortnotesfromagivenharmonicseries,whichwerethenautomatically repeatedwithvariabledelays5; Theharmony,timinganddelaybehavioursofthesegestureswereautomaticallysynchronizedwiththeoutputfromthe musiciansonstageviathea.belsystem. Morethan700peopleattendedtheeventandlogsshowthatroughly70%wereabletoconnecttheirsmartphoneto thesystemandsuccessfullymadeuseofit.Theremaining30%includepeoplethatdidnotcarryasmartphonetothe concert,usersofotheroperatingsystems(e.g.WindowsPhoneorBlackberry),usersofincompatibleAndroidversions (belowversion4.0),usersofincompatibleiOSversions(belowversion6.0)andasmallnumberofusersthatcouldnot connecttothesystemforotherreasons,mainlyduetoincompatibleAndroidROMs.Duringtheeventnodisruptions wereobserved,eitherinthenetworkdatastreamingorfromtheaudienceengagement. 2 Tobemadefreelyavailableintimefortheconference. Anintegralvideooftheconcertcanbeseenathttp://youtu.be/X_YeO3Qi3H4. 4 Anexampleofthisinteractionmodecanbeseenathttps://vimeo.com/142726571. 5 Anexampleofthisinteractionmodecanbeseenathttps://vimeo.com/142726073. 3 Conclusionsandfuturework Wehavedescribedtheinitialdevelopmentofa.bel,asystemforthedevelopmentofinteractiveperformer-systemaudienceexperiences,andwehaveshownhowitcanbeusedtobridgethegapbetweenperformersandaudiencesin musicalconcerts.Theaudienceresponseatthesystems’premièreeventwasveryenthusiastic.6Thesystemwasproven toworkasintendedand,giventheobservedresults,hasshownroomforconcreteimprovement,scalabilityandfurther development. Withtheemphasisbeingplacedontheabilitytoreliablyconnecthundredsofsmartphonesandtriggerbehaviourswith verylowlatency,itwasearlyondecidedtopostponetoalaterstagethecapabilityofsendingdatafromthe smartphonestothenetwork.Thecyclethatstartswiththeinformationbeingsentfromthestagetothesmartphones is,thus,completedsolelybyanaudiovisualreturnthatispartiallydependentontheaudience’swillingnesstointeract. Whilstwestillintendtoenabletheuploadofdatafromthesmartphonestothenetworkinthefuture,webelievethat theexclusivelylocalaudiovisualfeedback—akintotheoneofatraditionalmusicalinstrument—isinfacta characteristicthathelpsenduserstobemoreawareoftheirowncontributionthanwhatwouldbepossibleif,e.g.,the datageneratedbytheirinteractionwastobemingledwithhundredsofothersourcesandusedtodriveanaudiovisual feedbackcomingfromstage. Wehaveonlybeguntoscratchthesurfaceofhowthissystemcanbeusedtoenableaudienceparticipationinmusical concerts.Oneissuethatweneedtoaddressisthefactthattheaudiencehasnorehearsaltimewiththemusicians beforetheconcert,whichcanlimitthequalityofthemorecomplexinteractions.Asolutionthatweplantotestisto havetheconcertdividedintotwosections:aninformalrehearsalofallintendedinteractionmodesfollowedbythe mainperformance.Anotherissueisthefactthatmostconcertvenuesweredesignedwiththeaudiencefacingthestage and,consequently,theaudiencemembersdonotseeeachothereasily.Atthea.belconcert,thismeantthataudience membersoccupyingthefrontrowsweresomewhatexcludedfromthecollectiveexperience.InspiredbyGolanLevin’s Dialtones(ATelesymphony)(Levin2001),wedisplayedtheviewfromthestageovertheaudienceonabigprojection abovethestage,butthissolutionencouragedsomeoftheaudiencemembers’tofindwaystoidentifythemselveson theprojection,leadingtoamomentarydetachmentfromtheperformance.Weaimtoexploreadispositionofthe audiencearoundthemusicians,thuswithaudiencemembersfacingeachother,akintothedispositionusedbyIannis XenakisinTerretektorth(Xenakis1966).Finally,weaimtoexplorewaystopromotethedirectmusicalinteraction betweenaudiencemembers,namelybycomposingpiecesthatrelysolelyontheaudience. Acknowledgements.ThisworkwasfundedwithintheResearchLineFourEyesoftheProject“TEC4Growth-Pervasive Intelligence,EnhancersandProofsofConceptwithIndustrialImpact/NORTE-01-0145-FEDER-000020”,financedbythe NorthPortugalRegionalOperationalProgramme(NORTE2020),underthePORTUGAL2020PartnershipAgreement, andthroughtheEuropeanRegionalDevelopmentFund(ERDF). References Bongers,Bert.2000.“PhysicalInterfacesintheElectronicArtsInteractionTheoryandInterfacingTechniquesforRealtimePerformance.”TrendsinGesturalControlofMusic2000(January):41–70. Brinkmann,Peter,ChrisMccormick,PeterKirn,MartinRoth,andRichardLawler.2011.“EmbeddingPureDatawith libpd.”InProceedingsoftheFourthInternationalPureDataConvention,291–301. Google.2016.“ConsumerBarometer:Percentageofpeoplewhouseasmartphone.”https://www. consumerbarometer.com/en/trending/?countryCode=FR&category=TRN-NOFILTER-ALL. Levin,Golan.2001.“Dialtones(ATelesymphony).” 6 Mediacoverageoftheconcertcanbeseenathttp://youtu.be/fkCrtEPFTN4,http://youtu.be/07XwuHrHgW4, http://youtu.be/aEalgQez28M,http://youtu.be/p2iGkpZZXN0andhttp://youtu.be/JofTAb2OIDU. Oh,Jieun,JorgeHerrera,andNJBryan.2010.“Evolvingthemobilephoneorchestra.”Proceedingsofthe2010 ConferenceonNewInterfacesforMusicalExpression(NIME2010),pp.82-87. PixMob.2016.“PixMobwirelessLEDtechnology.”http://pixmob.com. Puckette,Miller.1997.“PureData:anotherintegratedcomputermusicenvironment.”ProceedingsoftheSecond IntercollegeComputerMusicConcerts:37–41. Schiemer,G.,andM.Havryliv.2005.“PocketGamelan:aPureDatainterfaceforMobilephones.”ProceedingsofNIME 05,pp.156-159. Trueman,Daniel,andPerryCook.2006.“PLOrk:thePrincetonlaptoporchestra,year1.”Proceedingsofthe internationalcomputermusicconference,pp.443-450. Wang,Ge,NicholasBryan,JieunOh,andRobHamilton.2009.“Stanfordlaptoporchestra(slork).”Proceedingsofthe InternationalComputerMusicConference(ICMC2009). Wang,Ge,GeorgEssl,andHenriPenttinen.2008.“Domobilephonesdreamofelectricorchestras.”InProceedingsof theInternationalComputerMusicConference.2008. Xenakis,Iannis.1966.Terretektorh.EditionsSalabert. Xylobands.2016.“RadiocontrolledLEDwristbandsforconcertsandliveevents.”http://www.xylobands.com/ xylobands/xylobands/.