Download The effects of game mechanics and web

The effects of game mechanics and web-based
testing on cognitive test performance and participant
enjoyment
(i.e. Gamification and Cognitive Tests)
Jim Lumsden
[email protected]
School of Experimental Psychology,
University of Bristol,
BS8 1TU,
Bristol, UK
Cognitive Tests
• Cognitive tests are a staple tool in psychological research
• Aim to collect pure measures of cognitive function and are therefore simplistic
and involve lots of repetition
• Cannot produce valid measures if the participant is bored
• They need to be engaging, and gamification may be the answer
Research Questions
1.
Do different game mechanics effect the cognitive data collected?
2.
Do different game mechanics effect subjective ratings of engagement?
3.
Do different game mechanics influence attrition rates?
(Study 2 only)
We ran two studies to look at the effects of game mechanics on
the data collected by two cognitive tasks
Study 1 Overview
(the cowboy study)
• Three variants of a Go-No-Go task
(non-game, points and theme)
• 287 participants (lab: 84, online: 203)
Signup
through
MTurk
Randomly
assigned to
variant
Signup in
person in
Bristol
Complete task
(~ 10min)
Subjective
questionnaire
delivered
Cognitive Task Results
(the cowboy study)
Go RT:
points vs non-game (BF = .157)
Go Accuracy:
points vs non-game (BF = .459)
No-Go Accuracy:
points vs non-game (BF = .253)
Questionnaire Results
(the cowboy study)
Study 2 Overview
(the attrition study)
• Three variants of a Stop Signal Task
(non-game, points and theme)
• Longitudinal (10 days long)
www.prolific.ac
Participants and Procedure
(the attrition study)
• 265 participants total, all online, ~90 in each variant
• Reimbursed between £4 and £7
4 consecutive days of test sessions
Signup through
Prolific
Academic
Randomly
assigned to
variant
Day 2
Day 1
Day 3
Day 4
Reimbursed £4 after all completed
6 optional days of test sessions
Day 5
Day 6
Day 7
Day 8
Reimbursed £0.50 for each completed
Day 9
Day
10
Study ends
Cognitive Task Results
(the attrition study)
Go RT: F[2, 260] = 4.421, p=0.014, η2=0.032
FailedStop RT: F[2, 260] = 5.403, p=0.005, η2=0.040
Go Accuracy: F[2, 260] = 1.053, p=0.350,η2=0.008
Stop Accuracy: F[2, 260] = 4.450, p=0.013, η2=0.033
SSRT: F[2, 260] = 4.747, p=0.009, η2=0.035
SSRT: theme vs non-game (BF = 0.278)
Questionnaire & Participant Attrition
(the attrition study)
Kaplin-Meier
Mantel-Cox Test: X2= 2.323, p = .127
One way ANOVA of condition
F(2, 261) = 1.643, p = 0.195
Bayes T-test
Mean Sessions Completed,
Points vs Non-game: BF = 0.163
Key findings
• Even with very light gamification there were impacts on cognitive measures
•
However, these effects are quite small
• Points had the least impact on performance and increased participant
enjoyment
• Negative effects of Theme variant on data and engagement
•
“looks like a game but doesn’t play like one”
• Study 2: No statistical difference in participant attrition between task variants
•
But, 10% more participants remained in the Theme variant until the end
Conclusions
• Don’t waste time making graphics or complex games: just pointify
• Gamifying cognitive tests can be effective for increasing reported
engagement but it is a trade off because you will impact the data some way.
• Further research is needed to quantify impact on cognitive measures
Acknowledgements
With thanks to my supervisors:
Tobacco and Alcohol Research Group:
Dr Jenny Barnett
Dr David Coyle
Dr Charlotte Housden
Prof Natalia Lawrence
Prof Marcus Munafò
Angela Attwood
Emily Crowe
Kayleigh Easey
Meg Fluharty
Therese Freuler
Suzi Gage
Meryem Grabski
Gemma Hammerton
Eleanor Kennedy
Jasmine Khouja
Glenda Lassi
Rebecca Lawn
Jim Lumsden
Olivia Maynard
Andy Skinner
Alex Board
Amy Taylor
Gemma Taylor
Chris Stone
David Troy
Miriam Cohen
Andy Eastwood
Postdoc
PhD Student
PhD Student
PhD Student
Research Assistant
Postdoc
PhD Student
Postdoc
PhD Student
PhD Student
Postdoc
PhD Student
PhD Student
Postdoc
Postdoc
Administrator
Postdoc
Postdoc
Research Assistant
PhD Student
PhD Student
PhD Student
[email protected]
@jl9937
21 February 2017
Any questions?
14
Questionnaire Results
(the cowboy study)
Questionnaire Results
(the attrition study)
Cognitive Task Results
(the cowboy study)
Go RT:
points vs non-game (BF = .157)
Go Accuracy:
points vs non-game (BF = .459)
No-Go Accuracy:
points vs non-game (BF = .253)
Cognitive Task Results
(the attrition study)
SSRT:
theme vs non-game (BF = .278)