Download Games as neurofeedback training for children with FASD

Full Papers
IDC 2013, New York, NY, USA
Games as Neurofeedback Training for Children with FASD
Regan L. Mandryk1, Shane Dielschneider1, Michael R. Kalyn1, Christopher P. Bertram2, Michael
Gaetz2, Andre Doucette1, Brett A. Taylor1, Alison Pritchard Orr2, Kathy Keiver2
1
2
University of Saskatchewan
Saskatoon, SK, Canada, S7N5C9
1-306-966-2327
University of the Fraser Valley
Abbotsford, BC, Canada, V2S 7M8
1-604-504-7441
{firstname.lastname}@usask.ca
{firstname.lastname}@ufv.ca
Figure 1. Columns show low, medium, and high levels of texture-based biofeedback. Rows show customizations of the same effect
for two different games: top) Static Sprite (cracks) over Portal 2, bottom) Static Sprite (mud) over Nail’d.
ABSTRACT
Keywords
Biofeedback games help people maintain specific mental or
physical states and are useful to help children with cognitive
impairments learn to self-regulate their brain function. However,
biofeedback games are expensive and difficult to create and are
not sufficiently appealing to hold a child’s interest over the long
term needed for effective biofeedback training. We present a
system that turns off-the-shelf computer games into biofeedback
games. Our approach uses texture-based graphical overlays that
vary in their obfuscation of underlying screen elements based on
the sensed physiological state of the child. The textures can be
visually customized so that they appear to be integrated with the
underlying game. Through a 12-week deployment, with 16
children with Fetal Alcohol Spectrum Disorder, we show that our
solution can hold a child’s interest over a long term, and balances
the competing needs of maintaining the fun of playing, while
providing effective biofeedback training.
Biofeedback, neurofeedback, games, FASD, ADHD.
1. INTRODUCTION
Fetal alcohol exposure is the most prevalent cause of intellectual
impairment in the western world [17]. An accurate account of the
incidence of fetal alcohol spectrum disorder (FASD) is unknown
but estimates range from 3 per 1000 live births to 10 per 1000
children being affected by prenatal alcohol exposure [10], which
translates to thousands of affected infants born each year in
Western Canada [3]. Children with FASD are often also
diagnosed with Attention Deficit Hyperactivity Disorder (ADHD)
[2]; using biofeedback (BF) to train brain function self-regulation
has been effective at reducing the symptoms of ADHD, and at
reducing differences of ADHD children from normative databases
of elecroencephalography (EEG) [6,9].
Biofeedback training systems encourage a specific mental or
physical state in a user through a closed biofeedback loop. These
systems gather a child’s physiological state through sensing
hardware, integrate this state into a computer-based interactive
system, and present the feedback so that the child can work to
adjust their state. Biofeedback training systems often use games
for interaction because playing games is intrinsically motivating
for most children. Biofeedback games work by altering the game
mechanics (i.e., rules and procedures) based on the child’s
physiology; however, traditionally, biofeedback games have not
been engaging enough to hold a child’s interest over the repeated
sessions needed for effective training [14].
Categories and Subject Descriptors
H5.2 [Information interfaces and presentation]: User Interfaces. Graphical user interfaces.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee. Request permissions
from [email protected].
IDC '13, June 24 - 27 2013, New York, NY, USA
Copyright 2013 ACM 978-1-4503-1918-8/13/06 $15.00.
165
Full Papers
IDC 2013, New York, NY, USA
motivating for many users and will encourage participation,
potentially resulting in improved training compliance [14];
however, many biofeedback games (e.g., [20], [21]) might be
better described as interactive systems because they lack the
uncertain and quantifiable outcome of a game [15].
Creating engaging biofeedback games remains difficult because
the game’s mechanics must be altered to create the biofeedback
loop. This means that each biofeedback game is a custom
creation, which is both expensive and time consuming; choosing
to play off-the-shelf games as biofeedback games is simply not
possible. As a result, biofeedback games have two main problems.
First, they tend to be toy applications that don’t hold a child’s
interest in the long term, which is a problem because biofeedback
training requires repeated exposure to yield successful results
[19]. Second, a child who wants to play a biofeedback game has
little choice over the game genre, and may not be motivated to
play a game from a small selection with little appeal.
In biofeedback games, players must maintain a particular
physiological state to make progress, generally accomplished by
adjusting the game’s mechanics (i.e., the rules and procedures of
play). For example, in a bowling game, the ball might roll toward
the gutter as the player becomes stressed. The player responds by
self-regulating their sensed physiological state in the desired
direction (see Figure 2–left). Note that biofeedback games are not
the same as ‘brain-training games’. Although both claim to help
users improve cognition, biofeedback games require a closed loop
of physiological sensing and real-time feedback; brain-training
games are mental exercises made fun with game-like elements.
In this paper, we present a system that turns off-the-shelf
computer games into biofeedback games. We propose to close the
biofeedback loop by altering display graphics instead of game
mechanics. We present a graphical overlay on top of a running
game that obscures the underlying game when the child is not in
the desired physiological state (see Figure 1). Our system works
with off-the-shelf games, so children can choose games that they
like. The amount of obfuscation varies parametrically in real time,
and is driven by physiological sensors. Our graphical overlays can
be chosen from an all-purpose set or be customized to be
consistent with the visual style, theme, or genre of the game, so
that they appear to be integrated with the underlying game. In
addition, the graphical effects are consistent with current abstract
in-game visualizations that players are already familiar with using
(e.g., tunnel vision representing poor in-game health).
2.1.1 Commercial History
Industry manufacturers have investigated biofeedback gaming
since the 1970s, when Thought Technology created CalmPrix1, a
racing game that came packaged with an Apple II mouse modified
with galvanic skin response (GSR) electrodes. Other biofeedback
gaming systems include the unreleased Atari Mindlink2 in 1983,
The Journey to Wild Divine3 in 2001, and the Nintendo 64
biosensor4 included in the Japanese version of Tetris 64 in 1998.
In 2005, Smart Brain Technologies5 released a home-based EEG
system. Recent biofeedback game devices include two popular
consumer-grade EEG devices – the NeuroSky Mindset6 and the
Emotiv EPOC7. These devices measure EEG via electrodes that
are held in place by a headset; both include a variety of games. In
particular, the NeuroSky features Focus Pocus8 – a 2012
biofeedback game designed specifically for children with ADHD.
Because our solution is a novel approach to biofeedback, we
investigate three main questions surrounding its efficacy. First,
does altering display graphics instead of game mechanics work as
a biofeedback mechanism? Second, do graphical overlays ruin the
fun of playing games? Third, can our system remain motivating
over the long term needed for successful biofeedback training?
We answer these questions by deploying the system over 12
weeks with 16 children with FASD, a target demographic for
brain-based biofeedback training. We conclude by discussing how
our approach can be applied to interaction other than games, to
open biofeedback training to those who are uninterested in games
or who would prefer to integrate it into their day.
2.1.2 Academic History
Early academic research in biofeedback games did not intend to
create therapeutic systems, but to create compelling play
experiences (see [12] for an overview). Sensed physiology can be
indicative of a user’s emotional state, and if used to manipulate a
play experience, could create engaging affective games [5]. Relaxto-Win (2001) [1] involved racing dragons whose speed was
controlled by GSR; BalloonTrip (2003) [16] also used GSR as a
game control; Brainball [8] is a game where a user’s EEG controls
a physical ball rolling on a table. There are also examples of
systems that use software development kits to integrate
biofeedback with existing games. In AlphaWoW [12], players
trigger their shapeshifting ability through EEG; in [4], some game
mechanics in Half-Life 2 (e.g., enemy spawn points, screen
shaking) are controlled with heart rate; and in AffQuake [13], the
player’s avatar jumps when startled and grows with player
excitement. Finally, there are many examples of heart-rate control
in games for the purpose of improving physical fitness (see [18]).
This research presents the first general solution for turning offthe-shelf software into biofeedback systems where the user
chooses which physiological trait to train. We focus on games,
leveraging the millions of dollars and years of development that
go into triple-A titles, and ensuring an engaging play experience
that will hold a child’s interest. Our low-cost system provides the
opportunity for biofeedback training – previously only available
in clinics – to children directly in their homes.
2. RELATED WORK
Biofeedback training has been used to help patients with
Asperger’s Syndrome [20], to reduce the frequency of seizures in
patients with epilepsy [6], and to improve the behaviour of
children with ADHD [9]. There is also evidence of successful
biofeedback training for children with tic disorder, autism,
schizophrenia, and learning disabilities (see [6]). In healthy
individuals, biofeedback has been used to improve working
memory and attention [21].
These systems all use indirect physiological control [11], where
players must work to change their physiological state through a
1
http://www.thoughttechnology.com/thewall2.htm
http://www.atarimuseum.com/videogames/consoles/2600/mindlink.html
3
http://www.wilddivine.com/
4
http://nintendo.wikia.com/wiki/Bio_Sensor
5
http://www.smartbraintech.com/
6
http://store.neurosky.com/products/mindset
7
http://www.emotiv.com/
8
http://ballantinespr.com/News/NeuroSky/NeuroSky_Focus_Pocus.html
2
2.1 Biofeedback Games
Instead of providing biofeedback through simple graphical
feedback, games are used because they are intrinsically
166
Full Papers
IDC 2013, New York, NY, USA
making it less enjoyable to play, and potentially impossible to
progress if there is enough obfuscation. Similar to traditional
approaches, we vary the feedback depending on the user’s state;
the textures have different obscuring parameters (e.g., opacity,
position, coverage) that vary continuously along a scale, providing
varying levels of obfuscation of the game display. Players want to
play with as little obfuscation as possible (see Figure 1),
motivating them to maintain the desired physiological state.
mediating mechanism. For example, to make the dragon race
faster in Relax-to-Win, a user must reduce their GSR by relaxing.
Indirect control is also the principle used in biofeedback games;
however, the games presented here were not developed as
therapy, but as novel systems that provide new forms of play.
2.1.3 Biofeedback Games in Practice
Although biofeedback games used in clinical practice resemble
the simple systems presented here, they have seen some clinical
success. Biofeedback games have been used to help people
manage stress, relax, focus, and manage anxiety see [6], and have
seen particular success when with special populations – for
example, to help children with ADHD and anxiety learn to
manage their symptoms [6], or to help children with pelvic floor
dysfunction improve voiding dysfunction [7].
Figure 2-left shows the traditional biofeedback loop, completed
through modifying game mechanics. This approach requires that
each game be custom designed specifically with the biofeedback
loop in mind. With texture-based biofeedback (see Figure 2-right),
texture overlays obscure the screen to complete the biofeedback
loop. This approach is agnostic to the underlying game, requiring
no access to source code, thus can be used with any off-the-shelf
game. In addition, the user’s physiological state is integrated into
our system rather than into the game itself, so any physiological
system can be trained, regardless of game choice.
In research closer to ours, Pope and Palsson [14] created a
hardware solution5 that worked with Playstation games by altering
the performance of a game controller based on a user’s EEG. In a
study comparing their game-based system to a traditional
biofeedback system with children with ADHD, the authors found
that both approaches resulted in improvements, but that children
and parents were happier with the game system [14]. Like our
research, the authors were interested in leveraging the motivation
provided by off-the-shelf games; however, our approach is not
limited to games played with a controller or on a Playstation, uses
a software solution that is agnostic to the underlying game (their
solution affected performance in the game), and is not limited to a
single physiological sensor (they train a specific band of EEG).
Our system has three main components; the biofeedback game
interface accesses two libraries: one senses a player’s
physiological state and the other renders the textures to the
display. In the following sections, we describe these components.
3.1 Physiological-Sensing System
Current biofeedback games (e.g., Focus Pocus) integrate the
sensor with the game mechanics, leaving users no decision over
game choice or what aspect of their physiology they wish to train.
Our system separates the selections of sensor and game.
3. TEXTURE-BASED BIOFEEDBACK
Our physiological sensing system is managed by a custom library
called SensorLib, which is a multi-threaded library written in C#
that provides an interface for external third-party sensors. It
handles the connection to the sensors, the data-buffering, digital
signal processing, and offers the data through a high-level .NET
interface. SensorLib aggregates third-party software development
kits (SDKs) into a single interface for ease of programming; new
sensors are easily added if the signal can be accessed via an SDK.
Biofeedback systems have two general requirements. First, they
must sense a user’s physiological state; and second, they must
provide this sensed state to the user through a feedback
mechanism. Both of these requirements should occur without
delay, as close to real time as possible. Our biofeedback system
had two additional requirements. First, as we wanted to engage
children over the long term, our system had to work with off-theshelf games. Second, the computational resources needed to run
the system should not affect the performance of the game.
3.2 Texture-Rendering System
Our texture-rendering system (TextureLib) was built in C++ using
Microsoft DirectX 10 graphics libraries, and displays an overlay
window over any application. The overlay window is rendered
over top of other windows even when it does not have focus, is
transparent, and allows keyboard and mouse events to pass
through it so interaction still occurs with the applications running
“behind” the overlay. TextureLib renders visual representations in
the transparent overlay by making use of DirectX resources (DDS
files) and pixel shaders written in High Level Shader Language
(HLSL). Our software requires Windows 7 and a video card that
supports DirectX 10 (common in gaming computers).
3.2.1 Visual Appearance of Textures
The visual representations that we use are made up of an effect (a
pixel shader), and any number of resources (textures and
colormaps) that can be edited using image editors. TextureLib
contains five pre-packaged effects that can be customized.
Figure 2. Biofeedback loops: Left – Traditional loop; Right –
Our revised texture-based loop.
Together, these four requirements inspired our use of texturebased overlays, rendered in real time in a transparent overlay on
top of a user’s primary task of playing a game. Traditionally,
biofeedback games work by not allowing the user to progress
unless they are in the desired physiological state. In our case, the
textures obscure the graphics related to the user’s primary task,
Tunnel Vision creates a semi-transparent texture with a definable
encroachment area on the screen. The location and size of this
area, the fade-in threshold for the texture to become opaque, and
the texture colour can be controlled. Note that the next four effects
can be applied to Tunnel Vision, combining the effects.
167
Full Papers
IDC 2013, New York, NY, USA
Figure 3. Columns show progression of low to medium to high levels of texture-based biofeedback. Rows show customizations of
effects: 1) Tunnel Vision (vines) over Up, 2) Tunnel Vision (veins) over Hulk, 3) Fractal Noise (mist) over Homecoming, 4) Fractal
Noise (fire portal) over World of Warcraft.
with the theme of the underlying game. For example, rain falling
in a golfing game may contribute to a better gameplay experience
than using that same effect in an ice hockey game, where the
concept of rain inside an arena makes little sense. By visually
customizing the effects, our system can appear to be integrated
with the underlying game. However, the appearance of the effects
not need be as literal as falling rain on a golf course to appear to
be integrated with the game. Effects that relate to the theme,
narrative, or world of the game may also contribute to a good
experience. For example, a fiery portal that grows and shrinks to
reveal the underlying display may be effective for use with a
fantasy game, even though it has no literal meaning in the game.
Fractal Noise uses a noise texture to render semi-transparent
textures. Multiple octaves of a noise texture (e.g., Perlin noise) are
used for variation. The colour and the opacity can be controlled.
Waves fills the screen with drops that generate ripples. Generated
with a 2D wave simulation, the resulting height field is rendered
with specular lighting. The size, frequency, and coordinates of
drops and the size and decay of the ripples can be controlled.
Static Sprite renders static 2D image sprites. The number,
starting position (x, y coordinates), speed, acceleration, rotation
speed, and size of the sprites can be controlled. In addition,
particle system parameters for sprites can be specified to create
visual representations such as explosions.
Customizing the textures is a process can be done by a developer,
but also could be done by an end user with no programming
experience. Simply substituting a different image file or editing an
image with standard image editing tools can customize existing
effects. For example, this approach can be used to change an
effect of mud splatters into slime splatters. We customized the
five effects included in TextureLib to demonstrate a wide range of
visual appearances (see Figures 3 and 4). In the following
examples, we used each effect twice, with different graphical
resources, to show how the appearance can be changed.
Animated Sprite renders animated 2D image sprites using a
sprite sheet. The number, starting position, speed, acceleration,
rotation, and size of the sprites can be controlled.
3.2.2 Game-related Texture Customizations
Although our system can use any visual effect to provide
biofeedback, we feel that the experience of biofeedback games
might be improved if the graphical feedback is visually consistent
168
Full Papers
IDC 2013, New York, NY, USA
Figure 4. Columns show progression of low to medium to high levels of texture-based biofeedback. Rows show customizations of
effects: 1) Waves (droplets) over Spearfishing, 2) Waves (frost) over NHL ‘11, 3) Animated Sprites (spiders) over Crysis, 4)
Animated Sprites (particles) over NetRumble.
more generic (e.g., a flowing mist). Generic effects can be treated
as a separate interface element, specific to the biofeedback
system. In this case, the textures would not be customized to
match the game being played; however, our system would still
have the advantage of operating with off-the-shelf games. Casual
games are designed to be quick to set up and fast to play,
involving play times of only a few minutes. With a generic effect,
a user could switch games a number of times within a single
training session using our system; the visual effect would stay
present between games, and the biofeedback experience would be
seamless throughout the training session.
We used the Tunnel Vision effect to create vines, which grew over
the jungle setting in an adventure game, in a fairly literal
customization. We also used the effect to generate pulsing veins,
which we deployed over a game based on the Incredible Hulk, in
a less literal, but still thematically consistent visual representation.
Fractal Noise was used to create a mist effect over a survival
horror game, and a fiery portal over a fantasy roleplaying game.
These effects are visually different, but require only a few
changes to the graphical resources and parameters. We used the
Waves effect to create water droplets over a spearfishing game in
a literal customization of the effect. This effect was also used to
create a frost effect, which we deployed over an ice hockey game.
We used the Static Sprite effect to create cracks that appeared in
the screen over a first-person shooter game with puzzle-solving
aspects, where the world involves a lot of glass walls. This effect
was also used to create mud splatter on the display in an outdoor
racing game. Finally, we used the Animated Sprite effect to render
spiders crawling over the screen during play of a first-person
shooter game that takes place in a jungle setting, and particle
explosions in a 2D space shooter game.
For more advanced users, parameter values of existing effects can
be programmatically changed at run-time by a client application.
For example, this can be used to change a mist effect into a smoke
effect by changing the colour and opacity values. Also, developers
can create new effects by implementing their own shaders.
3.3 Biofeedback Game Interface
The biofeedback game interface gathers the user’s physiological
state from the physiological sensing system (using SensorLib),
and renders textures in the overlay (using TextureLib)
corresponding to the user’s state. Any sensor in SensorLib can be
Some of the effects we created are fairly specific to certain games
or environments (e.g., spiders on the screen), while others are
169
Full Papers
IDC 2013, New York, NY, USA
used to drive the biofeedback training, while any effect that can be
created with TextureLib can be used as feedback. To use a sensor
not currently available, SensorLib would need to integrate the
sensor’s SDK, whereas creating a new visual effect does not
require changing TextureLib, but rather using the built-in
customization tools. To work with off-the-shelf games, our
interface must be agnostic to the user’s underlying task, so the
selection and launching of the game occur outside of our interface
using the standard Windows interface.
Games. We provided a selection of games from which players
could choose. Games were required to run in windowed mode,
maximized to the screen (rather than full screen mode) to allow
the overlays to display properly. In addition, games could not
contain objectionable content, including violence, sexuality, or
harsh language. Games were accessed via Steam, and initially
included NBA 2K10, Osmos, World of Goo, Bejeweled Twist,
and Blur. Part way through the deployment, we added additional
games, including Capsized and Plants vs. Zombies.
We created a simple graphical user interface that allows users
control over their biofeedback training. A commercial system
designed for an end user would hide some features; whereas, one
designed for a trained biofeedback clinician would include more
control. Our interface was developed for use in a series of
experiments, and resembles the features that would be presented
to a clinician. Users can select which aspect of their physiology
they wish to train (e.g., range of EEG), determine normalization
values for the effect from a calibration procedure or accept values
entered manually, preview and choose the effects and the
obfuscation levels, and view the signal strength for the training
hardware. In addition, we included an interface for saving log files
for further processing throughout the evaluation of our system.
Training. Users trained two or three times per week, in a research
lab at the University of the Fraser Valley, with our system for 12
consecutive weeks between October 2011 and April 2012. Each
session lasted about 60 minutes, with 30-45 minutes of game play.
Users chose which games they wanted to play; however, all
players used the mist effect at obfuscation levels set by the
experimenters, based on a standard pre-play calibration procedure.
Obfuscation levels were set at the beginning of each training
session so that thresholds could adjust with the players over time.
Measures. We asked users to fill out a survey with questions
related to the play experience after 12 weeks of training. All
questions were answered using a 3-point scale (yes-maybe-no)
suitable for the age of our users. In addition, we processed the log
files for each training session. EEG data was logged every 500ms,
and these data were aggregated over each session.
4. TEST DEPLOYMENT
Because of the novelty of our approach, we were interested in
answering three questions through a test deployment:
4.2 Results
1. Does altering display graphics instead of game mechanics
work as a biofeedback mechanism in games?
2. Do graphical overlays ruin the fun of playing games?
3. Can our system remain motivating over the long term needed
for successful biofeedback training?
Does altering display graphics instead of game mechanics work
as a biofeedback mechanism in games? Users generally agreed
that they “wanted the mist to go away” (13-yes, 1-maybe, 1-no).
Wanting to play without the obscuring textures is important to
motivate the progression of NF training. Players also agreed that
they were “able to control the mist to make it go away” (12-yes,
3-maybe). That players felt in control over the textures suggests
that altering display graphics works as a biofeedback mechanism.
We chose to evaluate the system with children with FASD, as
fetal alcohol exposure is a prevalent cause of intellectual
impairment in the western world [17], and children with FASD
experience symptoms similar to those with ADHD [2].
Biofeedback training of brain function self-regulation (called
neurofeedback (NF) training) using EEG has been effective at
reducing the symptoms of ADHD [6,9]. Also, differences in EEG
between children with ADHD and normative databases have been
reduced with NF training. Specifically, those with ADHD exhibit
higher power in the Theta band of EEG (related to decreased
attention and less retention of material) and lower power in the
low Beta band of EEG (related to increases in both hyperactivity
and impulsivity) [19]. NF training helped children with ADHD
lower the ratio of Theta/low Beta activity, by either lowering
Theta activity or increasing low Beta activity [6,19].
Users controlled the obfuscation of the mist with their Theta/low
Beta ratio, because children with ADHD have been shown to have
elevated ratios as compared to population norms [19]. Figure 5
presents average ratios for the beginning and end of the
deployment. Because children participated at different times and
for a different number of sessions, we performed a median split on
the number of sessions and classified training as either beginning
sessions or end sessions. Data for two children were removed due
to the connection with the Neurosky mindset being below
threshold (80%) for the majority of the sessions. Figure 5 shows
that players were successful at lowering the ratio in the later
training sessions as compared to the initial sessions. A pairedsamples t-test supports that this difference in average theta/low
beta ratio is significant (T13=2.16, p<0.05). We do not claim that
our results demonstrate successful NF training; however, the
lowered ratios do suggest that altering display graphics based on
physiology has potential as a biofeedback mechanism.
We tested our system with 16 children (9 male) between the ages
of 8 and 17 (median=11), who were all diagnosed with FASD,
and played video games regularly. The children participated as
part of a larger study on the effects of FASD interventions.
4.1 Biofeedback Training during Deployment
Physiological Sensors. We used the Neurosky Mindset7, which
is a single-electrode EEG device, as the input for our system. The
Mindset was chosen for its simplicity of deployment, robustness
of signal, SDK quality, and because it integrates headphones into
the device. We modified the Mindset by moving the electrode
from the forehead to EEG location Cz, on the top of the head,
which produces a better signal for NF training [19]. The reference
electrodes were positioned on the ear via the audio headphones.
Figure 5. Mean±SE Theta/low Beta ratios for the first half of
the sessions as compared to the last half of sessions.
170
Full Papers
IDC 2013, New York, NY, USA
(must be played in windowed mode) and content (no
objectionable content), we also were limited by the timing of the
training sessions. Because users played every few days for half an
hour, we offered casual games. Casual games are those that are
easy to learn with limited instructions and simple rules and
controls; they have short play times and allow users to put the
game on hold, thus lending themselves well to our experiment
protocol [22]. These types of games also lend themselves to our
approach to biofeedback training, as the texture overlays do not
present significant risk to players. For example, if a user lost a
game of Bejeweled during training, she could simply start a new
game. How our system would function with deep narrative-based
games designed to immerse a player, which contain leveling tasks
or boss battles (e.g., Mass Effect, Uncharted) remains to be seen.
Whether our system could work with time-critical collaborative
games (e.g., Team Fortress, World of Warcraft) also is unclear.
That our textures do not break the ‘fun’ offered by the casual
games tested in our deployment does not imply that the system
will work for all games. Commercial games are engineered to
provide a compelling and emotional experience for the player –
our intention is to help users self-regulate. There is a tension
between these goals that requires further investigation.
Do graphical overlays ruin the fun of playing games? Players
agreed that “playing the biofeedback games was fun” (14-yes, 1maybe). The NF textures did not fundamentally alter the
gameplay experience so that it was no longer enjoyable, and fun
games remained fun to play when used as part of our system.
However, the kids also agreed that they “would have preferred to
play the games without the headsets and the mist” (14-yes, 1maybe). That kids wanted to play without NF is expected, and
also fundamental to how the training system works; players need
to prefer to play without obfuscation to help motivate training.
Players did not feel that “playing the games was challenging” (5yes, 1-maybe, 9-no). It is good that the players did not feel
excessively challenged by the games because the challenge of
keeping the textures from obscuring the screens should be the
focus of the NF training activity.
Can our system remain motivating over the long term needed for
successful biofeedback training? Our test deployment shows that
kids enjoyed playing the biofeedback games, even when asked at
the end of the 12-week period. Thus our texture-based
biofeedback system did not break the enjoyment of gameplay. In
fact, after a number of training sessions, players became bored
with the available selection of games and asked that we add more.
Once we did, the kids were again happy to play. As long as a child
is motivated to use the computer (i.e., through gameplay or other
computer-based activities), our system will remain motivating.
5.2 Presentation of the Biofeedback Textures
Our textures were abstract representations of brain state. A user
may understand that the opacity of flowing mist is indicative of
their brain state; however, it is possible that, in some applications,
concrete visual representations may prove superior. Numerical
representations or progress bars are possible using TextureLib.
Also, our textures were agnostic to the underlying interface of the
user’s primary task. Although ours is a general solution that will
operate with off-the-shelf software in a black-box manner,
TextureLib can be aware of interface elements. For example,
animated sprites could be directed toward a screen location, static
sprites could reside over interface elements, or effects could
follow the mouse cursor. Consider a dynamic icon that resides in a
user’s system tray, or as part of a player’s interface in game.
Connecting a texture-based visual representation to an on-screen
or cursor location introduces new possibilities for biofeedback
training without requiring access to underlying applications.
5. DISCUSSION
5.1 Summary of Findings
Our test deployment shows that kids enjoyed playing the
biofeedback games, wanted the mist to go away, and felt able to
control the mist to make it disappear. Our texture-based
biofeedback system did not ruin the fun of games; rather it
provided the opportunity for choice and variety so kids could
maintain enjoyment. When players became bored with the
available selection of games, we added more. By giving choices,
we were able to retain the interest of the players over a long span
of time (3 months). Had players tired of a traditional biofeedback
game, with game mechanics that adjust to physiological state,
there would have been no way to renew their interest in play,
short of building a new game or new game levels. Our test
deployment also showed that obfuscating overlays have potential
as a feedback mechanism for biofeedback training; most users
agreed that they were able to control the overlays. Also, the
lowered ratios of Theta/low Beta activity in later play sessions
suggest that our biofeedback training approach holds promise.
Some games allow limited access to the game state through SDKs
(e.g., Valve Software’s (Portal, Team Fortress, Half-Life) Source
Engine and SDK). With access to game state through an SDK,
biofeedback games could be highly customized to gameplay,
appearing to be an integrated solution presented by the developer.
For example, different overlays could be presented at different
locations (e.g., indoor / outdoor); overlays could be combined
with in-game information (e.g., in-game health pack could reduce
obfuscation); or overlays could follow in-game elements (e.g.,
obscuring text bubble of a non-player character conversation).
5.1.1 Interpretation of Results
Our results indicate the potential of obscuring overlays as a
biofeedback training system; we do not claim that our results
show successful NF training for kids with FASD. The data
gathered in our deployment are preliminary and representative of
a small population. In addition, the EEG data used in our analyses
was gathered during play from a consumer electronics device, not
a multi-channel high-frequency EEG system with a standard
electrode array. To determine whether our approach is successful
in NF training, we are conducting a large-scale study involving
pre- and post-testing using validated outcome measures.
5.3 Interaction Beyond Games
Although we focus on biofeedback games in this paper, our
system is general enough to operate over most computer-based
tasks. The premise of biofeedback games is that the desire to play
motivates users to alter their physiological state in the preferred
direction. For children who do not enjoy playing games, our
system can also be used over web browsers, such as Firefox or
Chrome, and thus function with any task that a user performs in a
web browser (e.g., search, chat, social sites, media viewing).
Alternatively, our system can be used with other off-the-shelf
software (e.g., drawing application). Given that most consumer-
5.1.2 Generalizing Beyond Casual Games
We deployed our system alongside a selection of games from
which players could choose. Limited primarily by technology
171
Full Papers
IDC 2013, New York, NY, USA
[4] Dekker, A. & Champion, E. Please Biofeed the Zombies:
Enhancing the Gameplay and Display of a Horror Game
Using Biofeedback. Proc. of DiGRA'07, 550-558.
level biofeedback training systems are game-based, our approach
creates possibilities for a new population of users by providing an
opportunity for children uninterested in games to participate in
biofeedback training. In addition, our system decouples the
physiological sensing system from the activity being performed
on the computer, so users can choose their training activity
separately from their choice of which physiological trait to train.
So a child can train, for example, their brain activity, or their
anxiety, by playing a game or chatting online with friends.
[5] Gilleade, K., Dix, A., & Allanson, J. Affective Videogames
and Modes of Affective Gaming: Assist Me, Challenge Me,
Emote Me. Proc. DiGRA'05, 2005.
[6] Heinrich, H., Gevensleben, H., & Strehl, U. Annotation:
Neurofeedback — Train your brain to train behaviour. J.
Child Psychology and Psychiatry, 48, (2007), 3!16.
There is still the opportunity for developers who wish to create
integrated biofeedback training solutions – where the sensing is
directly integrated with the application – to do so. Our system
does not eliminate the prospect of integrated solutions; we simply
provide a general-purpose solution that decouples sensing from
activity, appealing to a broad range of tasks, domains, and users.
[7] Herndon, C., Decambre, M., & McKenna, P. Interactive
computer games for treatment of pelvic floor dysfunction.
The Journal of Urology, 166, (2001), 1893–1898.
[8] Hjelm, S. Research + design: the making of Brainball.
interactions 10, 1 (2003), 26-34.
[9] Lubar, J.F., Swartwood, M.O., et al. Evaluation of the
effectiveness of EEG neurofeedback training for ADHD in a
clinical setting as measured by changes in T.O.V.A. scores,
behavioral ratings, and WISC-R performance. Biofeedback
and Self Regulation, 20, (1995), 83–99.
6. FUTURE WORK AND CONCLUSIONS
Our research demonstrates a new approach to biofeedback
training, where the physiological sensing is decoupled from the
user’s primary task. We show that this approach has potential
through a test deployment. To investigate whether our system can
help children with FASD reduce their symptoms related to
ADHD, we are conducting a large-scale study.
[10] May, P., & Gossage, P. (2001). Estimating the prevalence of
Fetal Alcohol Syndrome: A summary. Alcohol Research &
Health, 25(3), 159-167.
We used casual games as the player’s primary task and are
presently exploring how the texture overlay system affects
gameplay with more immersive games with narrative, timecritical tasks, and longer playtimes. As part of this exploration, we
are focusing on how different presentations of the biofeedback
display balance the competing (and perhaps mutually exclusive)
desires for engaging gameplay and effective biofeedback training.
[11] Nacke, L.E., Kalyn, M., et al. Biofeedback game design:
using direct and indirect physiological control to enhance
game interaction. Proc. CHI '11, 103-112.
[12] Nijholt, A., Plass-Oude Bos, D., & Reuderink, B. Turning
shortcomings into challenges: Brain-computer interfaces for
games. Ent. Computing 1, 2 (2009), 85-9.
Biofeedback games have the potential to help children selfregulate their physiological function; neurofeedback games –
those that help users self-regulate their brain function – hold
promise for special populations, such as children with FASD.
However, choice in biofeedback games is limited, they are
difficult to create, and have little depth of play. We present a
solution for turning off-the-shelf games into biofeedback games.
Our texture-based overlay solution for decoupling the sensing
from the game gives kids the choice of what aspect of their
physiology to train and which game to play. Our approach
provides biofeedback training to children previously uninterested
or unable, and creates enough choice in interaction to support the
long-term and repeated use that is necessary for success.
[13] Picard, R.W. Affective Computing, MIT Press, Cambridge,
1997.
[14] Pope, A.T. & Palsson, O.S. Helping video games ‘‘rewire
our minds”, NASA TR (2001).
[15] Salen, K. & Zimmerman, E. Rules of play: Game design
fundamentals. MIT Press, Cambridge, MA, 2003.
[16] Sakurazawa, S., Yoshida, N., Munekata, N. et al. A computer
game using galvanic skin response, Proc. ICEC’03,
Pittsburgh, USA, 2003.
[17] Spohr, H.L., Willms, J., and Steinhausen, H.C. Prenatal
alcohol exposure and long- term developmental
consequences. The Lancet, 341, (1993), 907-910.
7. ACKNOWLEDGEMENTS
[18] Stach, T., Graham, T.C.N., et al.. Heart rate control of
exercise video games. Proc. GI '09, (2009), 125-132.
Thanks to NSERC and the GRAND and NeuroDevNet NCEs for
funding. Thanks to Bassam Khaleel and the UofS Interaction Lab.
[19] Thompson, M. and Thompson L. The Neurofeedback Book:
An Introduction to Basic Concepts in Applied
Psychophysiology. The Association for Applied
Psychophysiology and Biofeedback, Colorado, USA, 2003.
8. REFERENCES
[1] Bersak, D., Mcdarby, G., et al. Intelligent biofeedback using
an immersive competitive environment. Proc. of UBICOMP
2001 Workshop on Ubiquitous Gaming (2001).
[20] Thompson, L., Thompson, M., & Reid, A. Neurofeedback
outcomes in clients with Asperger's Syndrome, App.
Psychophysiology Biofeedback, 35, (2010), 63-81.
[2] Bhatara V, Loudenberg R, & Ellis R. (2006). Association of
attention deficit hyper-activity disorder and gestational
alcohol exposure: An exploratory study. Journal of Attention
Disorders, 9, 515–522.
[21] Vernon, D., Egner, T., Cooper, N., et al. The effect of
training distinct neurofeedback protocols on aspects of
cognitive performance. International Journal of
Psychophysiology, 47, (2003), 75–85.
[3] Clarren, S., & Lutke, J. (2008). Building clinical capacity for
Fetal Alcohol Spectrum Disorder diagnoses in Western and
Northern Canada. Canadian Journal of Clinical
Pharmacology, 15(2), e223-e237.
[22] Casual Game White Papers | IGDA. International Game
Developers Association (2008).
172