Download CHAPTER 1- PLANNING YOUR EXPERIMENTAL DESIGN

Chapter 1: Planning your experimental design
The aim of this chapter is for you to understand important research design issues and, in
so doing, to be able to design an experiment that minimises the introduction of error into
the results.
Please note that the ‘statistics in action’ questions are not included in this document but
make for excellent class discussion.
Key learning objectives are:
 to be able to state an experimental hypothesis succinctly and completely
 to understand the issues associated with the correct measurement of data and how
these impact the analysis
 to be able to obtain and use appropriate participants for your experiment
 to understand the advantages and disadvantages of particular research designs so
that you can choose the one that best suits your needs
 to understand how ‘error’ creeps into an experiment and so minimise it.
1.2 DEVISING YOUR RESEARCH QUESTION
DISCUSSION QUESTION
1. Formulate an hypothesis for the following experiment:
We believe a new morphine analogue will be particularly effective in controlling
post-operative pain without causing significant unwanted side-effects. In this
way, post-operative subjective wellbeing should be improved in patients taking
the new drug when compared to patients being administered a common analgesic
agent.
SUGGESTED RESPONSE
“That administration of the morphine analogue to post-operative patients will
provide a heightened level of subjective well being when compared to the administration
of a common analgesic agent.
This hypothesis may be considered complete, as it possesses the following
elements: (1) it begins with the word “That…”; (2) it includes both the independent
variable (i.e. drug) and the dependent variable (i.e. subjective wellbeing); (3) it notes the
comparison between the two levels of the independent variable (i.e. morphine analogue
vs. a common analgesic agent); and (4) suggests a relationship between the two levels of
the independent variable.
Instructor Resource Manual t/a Research Design and Statistics by Edwards
1-1
1.4.3 THE INTERVAL AND RATIO SCALES
DISCUSSION QUESTION
1. Which scales of measurement pertain to the following data and why?
(a) velocity measured in a driving simulator
(b) strength of the right arm(c) the number of stem cells that form after the
addition of certain embryonic
factors
(d) memory in trained rats relative to an untrained group
(e) level of depression measured as either: (1) not depressed; (2) mild depression;
(3) moderate depression; (4) severe depression.
SUGGESTED RESPONSE
(a) Ratio scale – For each participant a ‘characteristic’ is being measured. Velocities are
numerically well defined, presented on a continuous scale, with an absolute
zero
(i.e. stopping).
(b) Ratio – For each participant a ‘characteristic’ is being measured. Strength as a force
is numerically well defined, presented on a continuous scale, with an absolute
zero
(i.e. no strength).
(c) Nominal scale – Simply the number of stem cells is being measured
(d) Ratio scale – For each rat a ‘characteristic’ is being measured. Memory scores are
numerically well defined, presented on a continuous scale, with an absolute zero (i.e.
amnesia).
(e) Ordinal scale – For each participant a ‘characteristic’ is being measured. However,
depression scores are subjective and therefore not well defined. Further, the scale on
which the scores are measured has only four discrete values (i.e. 1, 2, 3 & 4).
1.5.1 METHODS OF RANDOM SAMPLING
DISCUSSION QUESTION
1. To understand the outcomes of sampling with replacement vs. sampling without
replacement we can use a ‘lucky dip’ experiment. To do this, label bits of paper
with the name of each classmate and put them in a box. Now choose a sample size
and randomly sample that many names from the box by each of the two methods
discussed above. When sampling with replacement you should note that, some of
the same names will be sampled repeatedly. When sampling without replacement
each name will only be sampled once but the chance of sampling any one
remaining name will increase as the sampling progresses.
SUGGESTED RESPONSE
Students are encouraged to perform this experiment comparing the probability of
sampling names from the box by the two methods using both small and large sample
sizes. In brief, sampling with replacement should provide a constant probability of
choosing any one name from the population no matter the sample size. In addition, if the
Instructor Resource Manual t/a Research Design and Statistics by Edwards
1-2
sample size is large the probability of repeatedly sampling the same names increases.
When using sampling without replacement you cannot sample the same name more than
once. However, as the population shrinks during sampling the remaining names have an
ever greater probability of being sampled. This effect is magnified when the sample size
is large.
1.5.2 HOW TO RANDOMLY SAMPLE
DISCUSSION QUESTION
1. How could you use the random number table shown in Table 1.5 to identify
participants:
(a) from the electoral roll?
(b) from a list of all first-year psychology students at your university?
SUGGESTED RESPONSE
(a) A random number table is a versatile tool. In using it to randomly identify participants
from the electoral roll; you could use the first three digits of each six-digit string to
represent a page number of the electoral roll and the second three digits in a string to
represent a person on a particular page. Therefore, having randomly chosen a starting
number, for example, ‘452023’, from your table you go to page 452 and choose the 23rd
person on that page. The participant has now been sampled. You then move down the
column of numbers in turn. If a string does not make sense with respect to the number of
pages in the electoral roll, or with respect to the number of people on a page, then ignore
it and move to the number below.
(b)A six-digit random number table can also be used to sample a relatively small number
of participants from a relatively small population by discarding part of the string. For
example, having assigned an identifying number to each member of the population (i.e.
the students), refer to the random number table and randomly choose the initial string. If,
for example, the population had only 500 members then only the last three digits of the
string would be required. If the initial string was ‘335422’ then you would have sampled
the 422nd person. Now simply work down the list of random numbers to identify
participants skipping those random numbers that do not relate to specific members of the
population.
Instructor Resource Manual t/a Research Design and Statistics by Edwards
1-3
1.6.1.1 OBSERVATIONAL STUDIES
DISCUSSION QUESTION
1. Using the four examples of a disguised study mentioned above, note the specific
ethical and safety implications of each.
SUGGESTED RESPONSE
As a general guide, ethical implications pertain to issues of informed consent, the privacy
of those under observation and the appropriate use of deception by the researcher. Safety
issues for the researcher may include those of personal safety. In addition, having
disclosed the deception after completing the study some participants may be distressed
which may impact their safety.
1.6.1.4 CORRELATIONAL AND QUASI-EXPERIMENTAL
STUDIES
DISCUSSION QUESTION
1. Even although non-experimental methods may collect qualitative data, lack
control and have no manipulation, provide examples where this sort of research
has nevertheless been of significant importance in understanding the world around
us.
SUGGESTED RESPONSE
Students and instructors should feel free to use their experience and the scientific
literature to answer this question.
1.6.2.5 FACTORIAL DESIGNS
DISCUSSION QUESTION
1. Find and discuss examples of each type of experimental design noted above (i.e.
simple and factorial between-subjects, simple and factorial within-subjects,
simple matched-groups and mixed factorial designs). For an example of a mixed
factorial design you may wish to read:
Blakemore. S. J., den Ouden, H., Choudhury, S., & Frith, C. (2007). Adolescent
development of the neural circuitry for thinking about intentions. Social
Cognitive and Affective Neuroscience, 2(2), 130–139.
For each example, note the benefits of the design to the research question being
investigated.
Instructor Resource Manual t/a Research Design and Statistics by Edwards
1-4
SUGGESTED RESPONSE
Students and instructors should feel free to use their experience and the scientific
literature to answer this question.
1.6.2.6 CONCLUDING REMARKS ABOUT DESIGN CHOICE
DISCUSSION QUESTION
1. Design (a) a between-subjects and (b) a within-subjects experiment for the
following scenario:
We believe a new antipsychotic medication will reduce the level of sedation
experienced by some individuals taking currently available drugs. Therefore,
compare the new drug to a commonly prescribed antipsychotic medication to
determine if it substantially reduces sedation.
SUGGESTED RESPONSE
(a) The between-subjects design uses two groups of participants. One group will be
administered the new antipsychotic drug, while the other group will be administered the
commonly used antipsychotic drug. The level of sedation for participants in each group
will be compared. Specific issues include: (1) Ethical considerations pertaining to the use
of mentally ill participants. (2) The viability of random sampling as opposed to the
random assignment of an opportunistic sample. (3) Determination of drug doses, period
of administration and participant compliance. (4) Students should also seek to determine
how ‘sedation’ (i.e. the dependent variable) will be measured, the reliability and validity
of this method and on what scale of measurement would the data fit.
Advanced students will also note: (1) confounding variables associated with the
participants including age, gender, disease manifestations, disease duration and potential
cause. (2) Confounds pertaining to where the experiment is to occur must be considered
(i.e. sleep laboratory, hospital or home). (3) Issues of external validity given the
participants used must also be addressed.
Students should feel free to note other issues which although not addressed above appear
important given class discussions.
(b) A within-subjects design uses one group of participants whereby each participant will
receive one drug and then the other. The level of sedation for participants in each group
will be compared. Specific issues include: (1) Ethical considerations pertaining to the use
of mentally ill participants. (2) The viability of random sampling as opposed to the use
of an opportunistic sample. (3) The random assignment of treatment order. (4)
Determination of drug doses, period of administration and participant compliance. (5)
Students should also seek to determine how ‘sedation’ (i.e. the dependent variable) will
be measured, the reliability and validity of this method and on what scale of measurement
would the data fit.
Instructor Resource Manual t/a Research Design and Statistics by Edwards
1-5
In addition, within-subjects experiments have specific issues which must be addressed:
(1) How are order effects to be dealt with? (2) How will cumulative effects of the drugs
be dealt with? If a wash-out period is to be used then how long should it be? (3) How
will participant expectations of drug effects be controlled given that the two drugs may
not look alike and that some participants may be familiar with the appearance and effects
of one of the two drugs?
Advanced students will also note: (1) confounding variables associated with the
participants including age, gender, disease manifestations, disease duration and potential
cause. (2) Confounds pertaining to where the experiment is to occur must be considered
(i.e. sleep laboratory, hospital or home). (3) Issues of external validity given the
participants used must also be addressed. (4) The problem of participant loss due to the
potentially lengthy nature of the experiment may also be a significant issue.
Students should feel free to note other issues which although not addressed above appear
important given class discussions.
1.7.1 RANDOM VARIABLES
DISCUSSION QUESTION
1. Identify a variety of random variables that may impact both human and animal
studies.
SUGGESTED RESPONSE
This response should be based upon the experience and interests of the instructor and the
students. Nevertheless, random variables that might impact human studies when
conducted in the laboratory could include:
 unintended interruptions
 unintended machine break-down
 inconsistencies in how instructions and explanations are given by the
experimenter
 participant anxiety if the experimenter is running late to the appointment
 an unexpected building evacuation.
Random variables when human studies are conducted in the home may include:
 distractions pertaining to other family members
 distractions relating to TV, phone calls and kitchen appliances
 situations where the participant has an illness which impedes their ability to take
part in the study
 the use of caffeine during the test session.
Random variables associated with animal studies may include:
Instructor Resource Manual t/a Research Design and Statistics by Edwards
1-6
 the use of different animal handlers
 the changing environment of the animal house
 other experimenters making noise etc. in adjacent labs.
1.7.2 CONFOUNDING VARIABLES
DISCUSSION QUESTION
1. Identify a variety of confounding variables that may impact both human and
animal studies.
SUGGESTED RESPONSE
This response should be based upon the experience and interests of the instructor and the
students. It may be useful to divide general confounds from those specific to withinsubjects designs.
Nevertheless, human studies may suffer from the following general confounds amongst
others:
 progressive tiredness, or boredom of participants
 progressive changes in laboratory temperature
 treatment group non-equivalence brought about by poor planning on the part of
the experimenter.
For animal studies general confounds may include:
 environmental confounds such as progressive tiredness if animals are repeatedly
tested
 confounds due to incorrect group structure brought about by a failure to have
equal genetic diversity across groups
 failure of the experimenter to note the affect of oestrus cycles and circadian
rhythms on the testing of animals.
1.7.3.2 WITHIN-SUBJECTS DESIGNS
DISCUSSION QUESTION
1. Using the scientific literature, what suggestions are made with respect to the
length of wash-out periods for various drugs? To answer this question search for
within-subject studies which test different drugs. You may even like to note how
each drug is metabolized and how this influences the length of the wash-out
period.
SUGGESTED RESPONSE
This is an open-ended question and requires a literature search. Nevertheless, we might
consider wash-out periods to extend from hours, to days, to weeks. Students could also
Instructor Resource Manual t/a Research Design and Statistics by Edwards
1-7
refer to the articles by Coull et al. (1996) and Kennedy et al. (2003) from ‘Statistics in
action’ and listed in the chapter’s bibliography.
DISCUSSION QUESTION (Order effects)
1. Use the scientific literature to identify studies that note potential order effects.
SUGGESTED RESPONSE
This is a good opportunity to discuss the complex nature of order effects by comparing
and contrasting studies from the scientific literature.
CALCULATION QUESTIONS
1. to 3. Given that incomplete within-subjects experiments tend to investigate only a few
treatments, construct a Latin square appropriate to test four, five and six
treatments.
SUGGESTED RESPONSE
Using: 1, 2, N, 3, N−1, 4, N−2, 5, N−3, 6, N−4, 7…
1. Four treatments (N = 4)
order of
treatments
treatment order 1
1
2
4
3
treatment order 2
2
3
1
4
treatment order 3
3
4
2
1
treatment order 4
4
1
3
2
treatments
2. Five treatments (N = 5)
treatment
order of
treatments
treatment order 1
left-hand side of treatment order
right-hand side of treatment order (which
is a reflection of the left-hand side)
4
3
5
2
1
1
2
5
3
4
treatment order 2
2
3
1
4
5
5
4
1
3
2
treatment order 3
3
4
2
5
1
1
5
2
4
3
treatment order 4
4
5
3
1
2
2
1
3
5
4
treatment order 5
5
1
4
2
3
3
2
4
1
5
Instructor Resource Manual t/a Research Design and Statistics by Edwards
1-8
3. Six treatments (N = 6)
order of
treatments
treatment order 1
treatment
1
2
6
3
5
4
treatment order 2
2
3
1
4
6
5
treatment order 3
3
4
2
5
1
6
treatment order 4
4
5
3
6
2
1
treatment order 5
5
6
4
1
3
2
treatment order 6
6
1
5
2
4
3
Instructor Resource Manual t/a Research Design and Statistics by Edwards
1-9