Download Thrust 4 Writing Team: Patterson, Harteveld, El

Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
Project Abstract: New tools for rapid recovery from upsets to critical infrastructure are the
focus of the project for this thrust. The goal is to ensure graceful extensibility of key services to
citizens. We will develop two complementary apps for use on a smartphone. The development
process will use participatory design and formative in-situ evaluation mechanisms to allow
tuning the application to the operations in the field. (1) A data mining app useful to decision
makers will harvest data from sources that may change dynamically during an event. It will cope
with sparse and missing data. Different disruption scenarios will be available as standalone
modules for evaluation and testing by stakeholders. After we consult with stakeholders, these are
likely to include: earthquake, multi-day storm, heat wave, brownout or blackout, water supply
disruption, and terror attack (including cyber denial of service). (2) A complementary app,
available to the public, will be developed with a bi-functional use. If the network is still up, it
acts a one-stop shop for information and advice from agencies that is location-aware to the
owner. If the network is down, the app allows smartphones at the neighborhood level to link
together in a resilient ad hoc mesh network through wireless. Participants can share information
in a reddit-style interface organized by critical areas: e.g., health care, water and food access,
local status reports, send text messages to neighbors and emergency managers, and receive
information alerts from emergency personnel passing through the neighborhood via vehicle
patrol, unmanned aerial vehicle, or tethered network balloon.
In order to develop these two apps a mix of various expertises are needed. First, software design
and in-field testing and design tuning is essential to make such apps work. Our team includes
expertise in software design and evaluation. Second, in order to test the app. simulation/gamelike scenarios is required for formative and summative evaluation. For this reason, we have
included a team of game design experts, with specific expertise in evaluation and analytics
techniques. Development of the apps will leverage our team’s expertise in data mining1, game
design2 and game analytics,3 risk-taking under training scenarios4, urban informatics5, complex
systems analysis6, autonomous systems and sensor networks7, and smartphone app development
for citizen connectivity8. We will incorporate insights and features from our ongoing research
into how critical infrastructure broke during past events9, the sociology of neighborhoods
1
Tan, P.-N., M. Steinbach, and V. Kumar. 2005. Introduction to Data Mining. Addison-Wesley, Boston.
Harteveld, C. 2011. Triadic game design: Balancing reality, meaning, and play. Springer, London.
3
Seif El-Nasr, M., A. Drachen, and A. Canossa. 2013. Game Analytics: Maximizing the Value of Player Data.
Springer, London.
4
Harteveld, C. 2012. Making sense of virtual risks: A quasi-experimental investigation of game-based training. IOS
Press, Amsterdam.
5
Offenhuber, D., and C. Ratti (Eds). 2014. Decoding the City: How Big Data Can Change Urbanism. Springer
Birkhäuser, Basel.
6
Amin, S.M, and A.M. Giacomoni. 2012. Smart Grid - Safe, Secure, Self-Healing: Challenges and Opportunities in
Power System Security, Resiliency, and Privacy. IEEE Power & Energy Magazine, pp. 33-40, January/February
2012.
7
Hayes, D., T. Boyd, and M.R. Patterson. 2007. Sensors and instrument requirements for Autonomous Underwater
Vehicles. Proceedings of the Masterclass in AUV Technology for Polar Science at the National Oceanography
Centre, Southampton, 28-30 March 2006, London, Society for Underwater Technology, pp. 39-48.
8
O'Brien, D.T. 2012. Citizens Connect: A smartphone application that reduces "broken windows": Boston Area
Research Initiative, Working Paper #2. Radcliffe Institute for Advanced Study, Harvard University.
9
Flynn, S. 2007. The Edge of Disaster: Rebuilding a Resilient Nation. Random House, New York.
2
1
Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
including response to disruptive events10, citizen science best practices11, and disruption
scenarios from our studies of climate change12, heath care13, and terrorist use of asymmetric
technology like unmanned vehicles14. The design spiral will use iterations of meetings with
stakeholders, beta testing at the neighborhood level and during training exercises of
infrastructure managers in a major metro area, and trial deployment during real events. Success
metrics will include semi-quantitative surveys conducted during trials by end users, post-event
surveys of management personnel and the public, and new metrics for measuring graceful
extensibility15.
Research significance of Project (how it builds on state of art; how work is innovative): Our
proposed research is guided by several important precepts: 1) The public needs and wants to
have more involvement during upsets to critical infrastructure; our citizens are an underutilized
force multiplier that should be part of how we bake resilience into our critical systems. 2) Data
mining, urban informatics, and the sociology and psychology of neighborhoods, are intersecting
in ways that make smartphone app development the logical next step to link citizens together. 3)
Because smartphones are low-power devices, we believe they can be kept alive through solar and
wind harvesting at the local level. 4) Use of smartphones in local mesh networks using 802.11
during grid and/or network failure puts to use assets that will enable greater levels of graceful
extensibility. 5) Managers of critical infrastructure will welcome the democratization of data
mining during a crisis among all players that the management app will provide. This app will
enable them to enjoy more Margin of Maneuver, as event cascades start happening.
Apps for subsets of important data are already available from several federal agencies, but are
usually focused solely on that agency’s mission. FEMA has a disaster app that offers general
advice on specific situations, shelter locations, allows crowdsourcing of disaster photos, and
links to FEMA social media. Amazingly NOAA and USGS do not provide apps for earthquake,
severe weather, or tsunami warnings, and thus there exists a confusing array of third-party
providers for such data. The US Red Cross, a non-government charity, uses data from several
federal agencies websites to field separate apps for tornado, hurricane, general first aid,
earthquakes, floods, shelter locations, and wildfires. The CDC has several emergency healthrelated apps for influenza, blast injury, and patient triage.
O’Brien, D.T. 2013. Custodians and Custodianship in Urban Neighborhoods: A Methodology Using Reports of
Public Issues Received by a City's 311 Hotline. Environment and Behavior, published online 8 September 2013.
DOI: 10.1177/0013916513499585
11
Patterson, M.R., D.H. Niebuhr, and J.A. Elliott. 2012. A STEM Experiment in Informal Science Education: ROVs
and AUVs survey shipwrecks from the American Revolution. Proceedings of the OCEANS 2012 Marine
Technology Society (MTS)/ Institute of Electrical and Electronics Engineers (IEEE) Conference, Virginia Beach,
VA, 8 pp.
12
Ghosh, S., D. Das, S.-C. Kao, and A.R. Ganguly. 2012. Lack of uniform trends but increasing spatial variability in
observed Indian rainfall extremes. Nature Climate Change 2(2): 86-91.
13
Hollnagel, E., J. Braithwaite, and R.L. Wears. 2013. Resilient Health Care. Ashgate Publishing, Surrey.
14
Patterson, M.R., and S.J. Patterson. 2011. Unmanned systems: an emerging threat to waterside security. In
Waterside Security Conference (WSS), 2010 International, Marina di Carrara, Italy, IEEE/Oceanic Engineering
Society. pp. 1-7. DOI: 10.1109/ WSSC.2010.5730271
15
Pariès, J., E. Hollnagel, J. Wreathall, and D. Woods, Eds. 2011. Resilience Engineering in Practice. 2011. Ashgate
Publishing, Surrey.
10
2
Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
In 2010, the US announced the creation of the Open 311 Application Programming Interface
(API) to allow consistent software development of smartphone and computer connectivity to the
311 services prevalent in many metro areas. 311 service calls are intended for non-emergency
municipal services. Boston Citizens Connect is an example of an app that allows the public to
promptly report situations involving degradation of infrastructure including potholes, graffiti,
debris, broken windows, etc. Such apps are proving popular as users often see the municipality
respond quickly to the complaint, and study of complaints has proved useful in understanding the
sociology and connectivity within neighborhoods.
The proposed work is innovative as nothing currently exists with the above functionality. A onestop data-mining app that dynamically harvests information from a plethora of federal and
private sources on critical infrastructure and current conditions does not currently exits. Neither
does an app that would allow a private, neighborhood level mesh network to self-assemble from
willing parties during an infrastructure crisis. These data mining/crowd sourcing tools will be
new territory, but we believe that mindful insightful design will provide great value to stressed
citizens and infrastructure managers.
Project Plan: Over the five-year period, we aim to run the design spiral three times. Scoping
meetings among the Center team are the necessary first step, and at this stage we recruit the best
postdocs and graduate students from a variety of disciplines to work under our guidance.
Prototype GUIs are the next step as modern software design encourages this step first, in
collaboration with the stakeholder end-user. For this, a series of workshops will be conducted at
Northeastern’s Boston and Seattle campuses, both with citizens in targeted test neighborhoods
and with infrastructure managers. Two development teams will be stood up for the two
smartphone apps proposed, but given that code reuse is likely and design attributes are linked, a
common wiki will be maintained during the development process and there will be frequent
meeting between the teams. Strict version control practices will be applied. After the first beta
versions are available, field trials begin, and an informational media campaign begins, with
presentations on our approach at national conferences on resilience. At this point, a separate
group of experts from within our core group will begin the quantitative assessment and feedback
phase. The design spiral then continues for second turn, with modification of the GUI, etc, as
above. Some of the funding during this period will be used to pre-position our tools on secure
servers for authorized beta testers. Provided we can gain cooperation of local and regional
emergency managers in MA and WA near our target neighborhoods, we will be in a position to
trial these products during an actual event. Or if initial non-event beta testing shows great
efficacy, we would work with Center partners in other metro areas to move quickly to make the
apps available when the inevitable stress event arrives.
We will also conduct an analysis of power needs to keep a neighborhood of smartphones alive
when the grid is down, and suggest low-cost best practices to address the power needed from
wind, solar, or other kinetic sources, e.g., bicycles.
Title: Smartphone apps for graceful extensibility: from the local neighborhood to the response
command center
Principal Investigators: This thrust requires a team effort requiring co-principal investigators to
achieve success. Patterson, O’Brien, Offenhuber, and Harteveld will serve as technology
coordinators and lead the development of the smartphone apps. Patterson has expertise in
3
Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
software design, distributed robotics, counter-terrorism for unmanned systems, resilience
infrastructure for coastal ecosystem health and property protection from storms (tide gates), and
networked emergency communications during disasters (Hurricanes Agnes, Allen, Isabel, Loma
Prieta Earthquake, and the invasion of Grenada). O’Brien is expert in the sociology and
psychology of neighborhoods, and emerging tools to assess their status, including smartphone
apps for 311 connectivity. Harteveld is an expert in game design for serious purposes, including
how games can be used to understand risk-related decision-making. Offenhuber is expert on
urban informatics, big data visualization, and design.
The significant partners listed next will play substantial roles as subject matter experts and are
full co-investigators.
Significant Partners and Roles: Additional key members of the team are: Magy Seif El-Nasr,
who is expert in game design and analytics, including data mining from the players themselves;
Robert Wears, University of Florida, an expert in resilient design in health care including patient
and industrial safety, with extensive experience in triage and emergency management scenarios;
Vipin Kumar, data mining expert and computer systems scientist whose work spans climate
issues at the global scale, and Massoud Amin, expert on complex systems and smart grid design,
both from the University of Minnesota. All team members will contribute to the scoping, and
will interact with stakeholders during the design phase. Kumar will be a key person in guiding
the data mining approach, as will Auroop Ganguly, who is working on another thrust of this
proposal. A subset of the team consisting of El-Nasr, Wears, and Amin, will help with the design
of the assessments we will conduct with the public and infrastructure stakeholders. Amin will
also work to help identify off-grid solutions that scale at the neighborhood level to keep
smartphones charged.
Background/Context of Research Problem: Our fourth research thrust is to increase the
resilience of critical infrastructures in a systematic way through the synthesis of new tools.
Innovation, case studies, and modeling from the other thrusts must come together to provide
useful products for the Center’s clients. These clients are varied in their sophistication and
resources and prior experience with resilience. They range from big public utilities to federal
agencies to cooperatives in rural areas to citizens of urban neighborhoods. The products the
Center may be tasked with producing are necessarily driven by the needs of the client. A
decision software tool or a new technology that nobody wants achieves nothing. Research into
stakeholder needs will be key to successful adoption and use of the Center’s products. There will
always be surprises and extreme events that overwhelm even the most thoughtful resilience
designs. Infrastructure systems will periodically fail on a regional scale with the associated
cascading effects. When this happens, public officials and industry leaders will need rapid
recovery tools. When disasters strikes, they also need to learn what went wrong and why. These
“day-after” assessments can inform new best practices.
The research has two core elements: (a) support the development of new social media (e.g.,
Advanced 911/311 through the mesh network proposed at the neighborhood level) and (b) tools
for managers to support rapid recovery of critical infrastructure after a disaster through data
mining of existing online resources, with flexibility to deal with missing or ambiguous data. To
create these tools, we will leverage the state of the art and practice for innovative designs, tools,
technologies, and applications that can be recommended by CIRC as “best practices.”
4
Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
Retrospective case studies are useful, to a point. Modeling is useful, to a point. Some of the
challenges we face in implementing resilience into critical infrastructure stems from failures of
imagination, institutional inertia and culture of key stakeholders, the sociology and psychology
of risk assessment and risk taking by individuals, communities, and corporations, and the twin
black swans of climate change and future terrorism incidents. Climate change and terrorism
provide difficult challenges in part because the risk prediction models useful in other arenas (e.g.,
earthquake engineering for built infrastructure) are largely not applicable. The statistics of
extremes are not applicable for a non-stationary system like a changing climate (although we can
still put some bounds on how bad it can get). Likewise, terrorism shows extreme spikiness as the
destructive power of individuals and small groups increases exponentially because of new
technology.
We are optimistic that the interdisciplinary team we have assembled, working with our extensive
connection of outside experts and stakeholders, will successfully integrate the Center’s work into
useful systems and processes. Our optimism for new methods stems from analysis of current
trends in several arenas. These include:
• Data mining as decision informatics. As the Internet of Things grows and social media use
approaches saturation among younger adults, we have new means of asking questions we
couldn’t ask five minutes ago.
Some examples of the new data, that can be mined to measure and improve resilience,
now, or very soon:
1) Metro heat distribution during a summer heat wave, at the apartment building, floor,
and room level read from the networked HVAC controls.
2) Storm surge estimates at the neighborhood level from DropCams owned by
individuals or businesses.
3) Perception of risk, and actions taken by people in response to that risk, during an event
from Twitter feeds at the regional scale.
4) Structural integrity of roadways or rail-lines from vibration sensors built into nextgen
cars, or citizen science apps on smartphones traveling in those cars.
Data mining of big data sets can now occur at the desktop or smartphone level. An urgent area of
research is how to allow the data miner to pivot to alternate data sets, or work with sparser data,
when components of the base data set are unavailable. Current data mining methods are largely
focused on large data sets that have undergone some level of QA/QC and produce answers to
questions with a time lag that would not be useful for some of society’s future risk scenarios
involving critical infrastructures. Our data mining researchers are eager to work on new
approaches to pivoting when data disappears during a crisis, making their powerful tools useable
by non-experts, and coping with sparseness or incomplete data during crisis management so that
these powerful analysis engines don’t break.
Because people with information feel and act differently during a crisis, the sociology of data
mining is a ripe area for work that would benefit the mission of DHS. At the neighborhood level,
there is big data available by virtue of smartphone connectivity. As discussed next, this provides
5
Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
a threat yet an opportunity to increase resilience at the hyper-local level. The key research area is
how empowerment of individuals or neighborhood groups to mine data themselves can increase
resilience of key infrastructure including services. In other words, should the city manager or the
FEMA control center or the public utility be the only one asking questions of big data? The
sociology of how neighborhoods operate becomes a rich area for work, and by engaging the lay
public in analyzing a complex system, especially when it is stressed, insights will emerge that
might not be found by the experts who normally manage or maintain the infrastructure.
This approach is an extension of citizen science, a new participatory means of engaging the
public in data collection for larger scale, or long term studies, that to now have benefited
scientists. By crowd-sourcing some aspects of critical infrastructure analysis, graceful
extensibility during a crisis might become easier. A key research question is whether public
engagement not only leads to faster solutions to dynamic, fast-moving events, but whether the
perception and experience of the event itself, changes when citizen resilience analysis and coping
discovery becomes part of the resilience equation.
• Cyber fragility as cyber resilience. In many urban areas, data transmission by smartphone now
exceeds bandwidth through fiber or cable. Modern life is now so inextricably linked to Internet
connectivity that when systems go down, companies send people home. Cyber disruption
through absence of access is a difficult resilience issue, and coping with it is a necessary to
achieve graceful extensibility during an infrastructure crisis. Yet modern smartphone hardware
and software are portable supercomputers by the standards of the 1990s. We can leverage this
power to increase resilience.
Some examples of how this could be achieved, and how these products could be used:
1) Solar or small wind turbines can keep smartphones running at the level of an
apartment building for all the owners, even if the main power is absent. A
charged cellphone become as important as a flashlight during an infrastructure
crisis because a prepared citizenry makes their own network, as follows.
2) Absence of the larger cellular network is dealt with by the phones in a hyperlocal area linking together via Bluetooth or 802.11x. An app allows email, phone
calls, and messaging through phones in close proximity, that is building- or
neighborhood-centric. First responders can serially link to neighborhoods for
dissemination of information by driving through, or via unmanned aerial vehicle.
This would allow data harvesting to assess status of far greater numbers of the
public than the conventional approach of knocking on doors. The data mining
tools developed above would be applied to these dynamic data.
3) Health management of issues that might normally show up in the ER could be
managed via this approach. Diabetics, the elderly susceptible to dehydration,
heart patients, could be monitored via their smartphone app, that disseminates
status at the neighborhood or building level so overwhelmed first responders are
relieved by persons already in place, and vested in the “success” of the response.
Although there are HIPAA and privacy issues involved in this approach, the
alternative during an infrastructure crisis (e.g., power failure during summer
months) is death. The tens of thousands that died from heat stroke and
6
Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
dehydration in Europe during a summer heat wave could possibly been saved
with such an approach. It doesn’t take an EMT to discern dehydration during a
stress event. A neighbor with a smartphone app measuring capillary refill of the
fingernail bed can fill the role of first responder.
In addition to the proposed smartphone apps, the most important products of this fourth thrust
will emerge from the interdisciplinary collaboration the Center will foster both internally and
externally.
Goals and Objectives of Research: We aim to create new tools (smartphone apps) for the
public and operators of critical infrastructure to increase resilience, in particular, graceful
extensibility of systems and services, during a stress event.
Theoretical Approach A key concept, developed by David Woods and colleagues, is that of
graceful extensibility, also known as graceful degradation, that brings ‘extra’ adaptive capacity
to bear in the face of potential for surprise. We believe new tools can use data mining to help
inform crisis managers so they better understand what they need for a Margin of Maneuver. We
also believe that empowering the public to help during a stress event for infrastructure is an
underexplored avenue, that is currently limiting the ability of the US to increase the resilience of
its critical infrastructure.
Social media and apps are important new tools that can support post-disaster assessments and
civic engagement in the aftermath of disasters. Understanding the role of these tools such as in
events like the 2013 Boston Marathon bombing can inform how they might be better harnessed
to enhance situational awareness, communicate with the public, and prioritize response and
recovery efforts. Regional scale sector and agent-based computational modeling of
interdependent infrastructure sectors, informed by real-world case studies, can help to identify in
advance of a disaster the potential effects that alternative strategies can play in containing
cascading effects across these interdependent systems. The modeling tools should be designed
for the full spectrum of public sector and industry decision makers to evaluate alternative
emergency policies and actions. This involves the need for visualization tools that can provide
real-time decision support in the aftermath of major disruptive events by integrating and
assimilating multiple types of information, processing that information, and presenting it in a
manner useful to decision makers.
Hypotheses: Our working hypotheses are: (1) Resilience increases in neighborhoods during a
stress event if crowd-sourcing of data about the local conditions are available through a mesh
network that is grid- and Internet-independent that allows citizens to communicate with other. (2)
Resilience of critical infrastructure increases when operators have access to data mining tools
that help them get data about systems that impact their own operations, so that they can maintain
a Margin of Maneuver during event cascades.
Methods: Data gathering on the needs of infrastructure stakeholder and neighborhoods occurs
through face-to-face meetings. Principles and algorithms from urban informatics, data mining
and visualization, complex systems, human machine interface development, and game
development are applied to the development of the software, which is then tested in the design
spiral described above under the Project Plan, and following the timeline below. Existing efforts
to build and test the performance of 802.11 ad hoc mesh networks in South Africa, rural America,
7
Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
metro areas, and in university settings will be analyzed for suitability in our proposed app. We
will necessarily be developing new methods for data mining and portrayal by non-experts, as
well as pushing the envelope of urban informatics and emergency management at the level of the
concerned citizen in a neighborhood.
Metrics for Success: Semi-quantitative and quantitative metrics on user experience, and system
attributes of graceful extensibility will be collected during beta testing and actual use during real
events. Analytics used to analyze gaming, and social networks, will be applied to capture
statistics on penetration of the tools at the neighborhood level by socio-economic status. The
performance of the ad hoc network itself will be assessed secondarily using formal methods that
have emerged since the 90s for the analysis of MANETs. However, our orientation will be on
measuring operational success, not BER, throttling, or other metrics. The deliverable is adequate
network performance that enhances resilience, not vice versa.
Data Collection: Human subject protocols will be followed for in-person interviews, as well as
web-based questionnaires. Data mining will occur from a variety of online resources many of
them maintained by federal and local governments.
Analysis: Our work will be analyzed and reviewed in the peer-reviewed literature, on app stores
by users, and at professional conferences by practitioners. At the end of the last design spiral we
plan to convene a working group of stakeholders and citizen groups to evaluate our work
independently from the creators and designers of the system.
Likely Outputs and Outcomes: We predict widespread adoption of ad hoc mesh networks during
infrastructure events by neighborhoods nationwide. Likewise, we predict that our data mining
decision and dashboard aid for infrastructure management may become a best practice. The
result is increased resilience of critical infrastructure by empowering the public and operators
alike using a pervasive tool of the modern age, a smartphone app.
Project Schedule (year by year description, including key milestones for each year) – Note to
writing team, this part will appear as a table, once we agree that the path proposed above is
viable.
Total Costs (per year) $1,000,000. – Note to writing team, this is a guess, as there are 4
thrusts and $4M/year budget overall nominally. Obviously, $1M/yr does not go far as we
might like, given the number of investigators needed to pull this off. The real promise of
landing this Center is to leverage additional sole source funding from DHS, DoD, NOAA,
FEMA, etc., once we are established. For example, the Stevens Inst. of Technology
leveraged their DHS COE money into another c. $50M of sole-source funding. So as Steve
Flynn has commented, that’s the real carrot after capture.
Question to Team: Do we need a compelling graphic in these 7 pages and if so what should
it convey? I had a couple of thoughts if we go this route: A dummy screen shot of the two
smartphone screens with graphic objects and arrows showing the disciplines feeding into
their creation, the data sources to be harvested for the data mining app, and the
functionality achieved by the citizen app during a crisis. I tend to like conceptual images
myself as an reviewer, if they are well-done. Send me your thoughts, and we can get our
8
Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
graphic resource folks working on this to have it as an option for consideration in a week
or two.
Magy’s comments:
I suggest adding the following foci, some we have already built methods or systems for:




Verification software to verify the data on the apps. This can be another app to
professional staff who can verify the news or data shared with public.
Interactive visualization tools for verification software, data aggregation app., and public
app. These visualizations can be different. The game design group and Dietmar have
expertise in visualizations that are more event and temporal in their nature due to the
game’s nature. Less on social network visualizations. I think Dietmar has more expertise
in that.
Software Design Methodology. Our group has engaged with companies to develop
methodologies that work for in the field testing especially for games that are more on
apps and need continuous asynchronous interaction. We do formative and summative
evaluation through analytics and mixed methods – qualitative and quantitative methods.
Game as a testing method for the app. It is unclear how the app will be tested. I was
thinking perhaps game testing would be good, developing scenarios and seeing them
unfold. We can use past scenarios, which we have access to. Our game design expertise
will be great here as we can develop the game and play it out through the systems and see
how the apps work collecting behavioral and qualitative information as it is tested to
iterate.
Existing work:
1. Formative Evaluation Method:
Pre-Study
Instrumentati
on of the app
data
processing:
abstraction and
visualization
user
registratio
n
behavior data
interpretation
Interview
Question
Construction
Interview
Interview
data analysis
Re-interpret behavioral data based on interview
data
9
Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
Data-Driven Retrospective Interviewing Method Overview – in the field analytics +
interviewing formative evaluation method.
The method has been used for evaluating and designing SpaPlay – commercialized pervasive
health game. Publication of this method is still under review.
Other references on analytics:
Seif El-Nasr, M., Drachen, A., & Canossa, A. (2013). Game analytics: Maximizing the value of player
data. Springer.
Seif El-Nasr, M., Desurvire, H., Aghabeigi, B., & Drachen, A. (2013). Game Analytics for Game User
Research, Part 1: A Workshop Review and Case Study. Computer Graphics and Applications,
IEEE, 33(2), 6-11.
Aghabeigi, B., Calvert, T., Seif El-Nasr, M., & Riedewald, M. (2012, September). Assistive design and
production in computer games: Parametric systems, data mining, visual analytics. In Games Innovation
Conference (IGIC), 2012 IEEE International (pp. 1-4). IEEE.
Desurvire, H., & Seif El-Nasr, M. (2013). Methods for Game User Research: Studying Player Behavior to
Enhance Game Design. Computer Graphics and Applications, IEEE, 33(4), 82-87.
2. Testing Scenarios – A language for quickly developing scenarios.
In order to test the event simulation in a game including all organizations and structures in play,
scenarios need to be developed. In the past we have developed a language that allows us to
author scenarios that unfold over time within a simulation involving virtual characters inhabiting
an environment. This scenario system will need to be revised for the kind of systems we are
working with here, but the language is useful as a starting point for quickly authoring scenarios
for testing purposes.
References
3. Visualization Systems
We have developed various interactive visualization systems in the past, mostly for designers
and business decision makers within a gaming company. But some of the methods used to show
events unfolding over time and behaviors may be useful here.
Pathways
10
Thrust 4 Writing Team: Patterson, Harteveld, El-Nasr, O’Brien, Wears, Offenhuber, Kumar, Amin
A system that shows paths through time that different users (white) took to combat an enemy
(red) in a strategy game. Circles are part of the structure of the environment that the users will
need to get around. The interface includes a time bar, shown above which designers or decision
makers can manipulate to see the behaviors over time. Decision makers can also select specific
users or structural elements in the environment to examine further. The screen is split to show
winning (right) vs loosing (left) strategies.
Dada
This is another interactive visualization that shows multiple information: navigation within the
environment, interaction with maps, objects, and characters within a simulation environment.
Cycles in navigation as well as interaction behaviors are easier to pick up using the figures in the
right, which shows different areas – horizontal dimension, and number of interaction – vertical
dimension. This visualization was used to show aggregate information from multiple players
over time. Data mining was also used to abstract behaviors to handle big data – 60 million
character data for this particular game (Dragon Age, by Bioware).
11