Download Online data collection CHew

Using the Internet to Collect Primary
Research Data
Postgraduate Research Methods Day Conference,
The Open University,
Camden Regional Centre
Claire Hewson,
Department of Psychology,
The Open University
[email protected]
A (very) Brief History of Internet-mediated
Research (IMR)

Pioneers started implementing online data collection
methods around the mid-1990s Bordia (1996); Dillman (1991);
Gaiser (1997); Hewson (1994).

Reports (e.g. validation studies) have since supported the
quality of data that can be obtained in IMR Corley & Scheepers
(2002); Hewson & Charlton (2005).

A new era of IMR (Web 2.0) The web as organic, collaborative,
interactive, ever-changing (e.g. Wikipedia); the rise of social media (e.g.
Facebook, Twitter, YouTube).
IMR Data Collection Methods

Obtrusive Approaches Surveys, experiments, interviews, participant
and / or disclosed observation approaches.

Unobtrusive Approaches Observation using traces and archives, e.g.
discussion group archives, browsing behaviours, social networking links
(SNA); lurking?

'Big Data' Data sets which become so large as to be difficult to manage;
presently a 'hot topic'.
The Visualisation of Complex
Networks
http://www.visualcomplexity.com/vc/
Nike+ involves the
placement of a sensor
underneath the footbed
of your Nike running
shoe in order to collect
data about where
you've run, how long it
took and where you
can improve over time.
Visualisation of a year's worth of runs uploaded to the Nike+ website. It plays back runs
throughout three cities: New York, London and Tokyo. The runs showed tens of thousands
of peoples' runs animating the city and bringing it to life. The software visualizes and
follows individual runs, as well as showing the collective energy of all the runners,
defining the city by the constantly changing paths of the people running in it.
Web-based Surveys and Experiments


There is now a large range of software for helping
implement these, particularly survey packages (which can
also support simple experimental designs), e.g.
'Surveymonkey', 'Qualtrics'.
Dedicated packages to support experimental web-based
studies include: WEXTOR (Reips, wextor.org/wextor/en)
and WEBEXP (Edinburgh webexp.info).
Web-based Surveys and Experiments

New design issues and principles emerge in this novel
context. Useful resources and guidelines include:
Exploring Online Research Methods: www.restore.ac.uk/orm
WebSurveyMethodology: websm.org
Couper (2008). Designing Effective Web Surveys. CUP: New York.
Reips & Krantz (2010). Conducting True Experiments on the Web. In
Gosling & Johnson (eds) Advanced Methods for Conducting Online
Behavioral Research. APA.
Fielding, Lee & Blank (2008). The Sage Handbook of Online Research
Methods. Sage: London.
Recruiting Participants in IMR

Sample Bias Issues?, e.g. Bordia (1997).

True Volunteers, place adverts in public spaces, e.g. newsgroups,
webpages, IMR study clearing houses, Facebook.

List-based Sampling Frame, e.g. email lists, online panel.

Probability Sampling Methods, e.g. random selection from known
list, online probability panel.

General Principle: methods which allow some estimation of the
sampling frame are preferable where measures of response rate, non-response
bias, etc. are important to the research design.
Online Sampling Resources
●
Online Study Clearing Houses, e.g. Online Psychology Research:
http://www.onlinepsychresearch.co.uk
●
Newsgroups, SNSs, e.g. Facebook: facebook.com, Newsgroups (e.g.
http://uk.groups.yahoo.com; https://www.groups.google.com), Mailing Lists
(see the searchable Catalist database: http://www.lsoft.com/catalist.html)
●
Online Panels, e.g. Knowledge Networks:
http://www.knowledgenetworks.com/knpanel

Online workforces, e.g. Mechanical Turk: www.mturk.com

OU Participant Panel, Jim will talk more about this
OU ‘Virtual Participant Panel’

URL:
http://www.open.ac.uk/socialsciences/psychology/cog
psypanel/





People can participate in studies ad hoc and/or…
… can sign up to be contacted for future studies.
Sign-ups give certain information (e.g. sex, age, location,
handedness) which can be used to filter if particular
characteristics are needed in a sample.
Currently has 1,245 signed-up members.
Has a built-in ‘Questionnaire Manager’ to create surveys.
OU ‘Virtual Participant Panel’
Instruction
text
Question
text
Response
text
OU ‘Virtual Participant Panel’
Stimulus
image
(‘question’)
Response
images
(‘answers’
)
Response
buttons
SurveyMonkey Example
●
●


Computer Attitudes
Surveyhttps://www.surveymonkey.com/s/computer_attitudes_study
Qualtrics – site license for all OU social science faculty
students / staff. Visit 'openss.qualtrics.com' and sign up
with a valid OU (open.ac.uk) email address.
Martin discussed using Qualtrics in a radiology expertise study this morning,
and Jim will demonstrate an online experiment implementation using
Qualtrics.
Importing studies from SurveyMonkey into Qualtrics?
Qualtrics
Qualtrics
Stimulus
image
Response
images
Ethics in Online Research

Some ethics issues require special consideration in an IMR
context, the complexities of which are not always obvious
to researchers or participants.
e.g. Data security, traceability (confidentiality, anonymity).

Key feature of IMR: involves acquisition of data from or
about individuals in absence of face-to-face co-presence
COUPLED WITH greater scope for carrying out quite
complex interactive procedures
The New BPS Guidelines (November 2013)


Document aims to outline some of the key ethics principles
for researchers and ethics committees to keep in mind
when designing and / or evaluating an IMR study
Emphasis on how to apply existing ethics principles in an
IMR context
“Thinking is not optional”
The New BPS Guidelines (November 2013)
●
Centred around the four main principles of the BPS 'Code
of Human Research Ethics' (2011).
1. Respect for the autonomy and dignity of persons
2. Scientific value
3. Social responsibility
4. Maximising benefits and minimising harm
Main Ethics Issues to Consider in IMR

Public-private domain distinction

Confidentiality*

Copyright

Consent; withdrawal; debrief*
●
Levels of control*
●
Disruption of social structures
Valid Consent

Verifying characteristics (e.g. age, sex)

Engagement with consent process (check boxes?)


Adequate information – not all risks online are obvious,
e.g. potential data leakage (balance between information
and overload)
Risks will be higher in some contexts than others – key
principle of proportionality of ethics procedures to levels of
risk of harm re-emerges
Withdrawal and Debrief

Robust withdrawal procedures may be more difficult to
implement online (e.g. volunteer web survey)

Detecting withdrawal may be harder (use of partial data?)

Ensuring debrief difficult (e.g. exit by closing browser)

Enhanced risk in cases of deception (no debrief)?

Principle of proportionality of ethics procedures to levels
of risk (of harm) re-emerges, e.g. effects of withdrawal
without debrief (such as in very sensitive research)
Confidentiality, Anonymity
●
Enhanced traceability of data, dissemination implications,
use of quotes.

Other data leakage risks, e.g. email.

Traces on personal computing equipment.


Sensitivity of data is a factor associated with levels of risk
of harm.
Researchers should carefully assess risks, and perhaps
inform participants of these risks.
Data Protection

UK Data Protection Act (1998).

Issue of storing data on non-UK/EU servers?


Popular software packages (e.g. SurveyMonkey, Qualtrics)
are US-based.
'Safe Harbour Scheme'.
Office of National Statistics
SurveyMonkey, Safe Harbour Scheme
Levels of control

Over procedures, who takes part, participant behaviours...

Implications for scientific value (validity)

Implications for levels of harm (monitoring participant
reactions, presenting debrief info., etc.)
Maximising Benefits, Minimising Harm


These considerations permeate the other issues
Careful assessment needed; trade-offs will be inevitable
(e.g. anonymity versus traceability/verifiability; disclosure
versus social disruption/harm)
“Thinking is not optional”
AoIR
http://aoir.org/
Some Examples
●
http://www.socialpsychology.org/expts.htm

http://www.onlinepsychresearch.co.uk

http://psych.hanover.edu/Research/exponnet.html

http://www.wexlist.net