Download exp1

Experimental Design,
Statistical Analysis
CSCI 4800/6800
University of Georgia
Spring 2007
Eileen Kraemer
Terminology
empirical study

based on observations and measurements
probabilistic

based on probabilities, inferences
causal

based on cause-effect relationships
Types of research questions
Descriptive.


designed primarily to describe what is going on or what exists.
For example, a study in which you observe and note the current
practice of users in performing a task of interest
Relational.


designed to look at the relationships between two or more
variables.
For example, a study in which you look at the relationship
between user’s preferred background color and font type
Causal


a study is designed to determine whether one or more variables
(e.g., a program or treatment variable) causes or affects one or
more outcome variables.
For example, a study of the effect of font size on time-tocomplete or error rates in a task of interest
Time in research
cross-sectional versus longitudinal studies.


A cross-sectional study is one that takes place at a single
point in time.
A longitudinal study is one that takes place over time -- we
have at least two (and often more) waves of measurement
in a longitudinal design.
 repeated measures - two or a few waves of measurement
 time series - many waves of measurement over time
 Analysis considerations: Time series analysis requires that you
have at least twenty or so observations. Repeated measures
analyses (like repeated measures ANOVA) aren't often used
with as many as twenty waves of measurement.
Relationships between variables
correlational v. causal
preferred font size
Age v. preferred font size
positive
25
20
15
10
5
0
20
30
40
50
60
70
80
age
Age v. accuracy
1
accuracy
negative (inverse)
0.8
0.6
0.4
0.2
0
20
30
40
50
age
60
70
80
Variables
variable

any entity that can take on different values.
attribute

a specific value of a variable.
independent variable

the thing you change
dependent variable

the thing you expect to change as a result of your manipulation of
the independent variable – the thing you measure
Example: I perform an experiment in which I give apply fertilizer
at concentrations of 5%, 10%, and 20% to some number of
otherwise identical plants. I measure the growth of the plants.
The concentration of fertilizer is the independent variable, the
height of the plant is the dependent variable.
Well-chosen variables/attributes
exhaustive - should include all possible
responses (all possible values of plant height,
for example)
mutually exclusive - no response should be
able to have two attributes simultaneously (24 inches and 4-6 inches shouldn’t be two
possible attributes – exactly 4 in. would fall
into two categories)
Hypotheses
hypothesis

a specific statement of prediction
For the hypothetical-deductive model : two hypotheses


one that describes your prediction (alternative hypothesis) (H1)
one that describes all the other possible outcomes (null hypothesis) (H0)
One-tailed v. two-tailed

One-tailed – the prediction specifies a direction
 H1: increase fertilizer application will increase the height of the plant (H1)
 H0: increased fertilizer application will not increase the height of the plant

Two-tailed – the prediction does not specify a direction
 H1: increased fertilizer application will affect the height of the plant
 H0: increased fertilizer application will not affect the height of the plant
In either case, the goal of testing and analysis is to accept one
hypothesis and reject the other
Sampling
the process of selecting units from a
population of interest in a way that permits
us to study those samples and then
generalize our results back to the population
external validity


the degree to which study conclusions would hold
for other experimenters in other similar studies
the approximate truth of the inferences and
conclusions that result from the study
Sampling Model for generalization
identify the population you wish to
generalize to
draw a fair sample of that population
conduct research with sample
generalize back to population from
which sample is drawn
Sampling terminology
theoretical population

group you wish to generalize to
accessible population

subset of that population that is accessible to the
experimenter
sampling frame

list of the accessible population you’ll draw your
sample from
sample

group selected to be in your study
Selection, assignment
selection

how you draw your sample from a
population
assignment

how you assign members of the sample to
groups in your study
Goal: avoid systematic error, bias
Statistical sampling terminology,
continued
response

specific measurement value that a sampling unit
provides
statistic

calculated across the response from the sample
parameter


calculated across the population
a statistic provides an estimate of a parameter
Terminology, continued
sampling distribution - the distribution of an infinite
number of samples of the same size as the sample in
our study
standard deviation

the spread of scores around the average in a single sample
standard error, sampling error




the spread of averages around the average of averages in a
sample distribution
indicates the precision of our statistical estimate
calculated based on standard deviation of sample
larger sample size -> smaller standard error
Variance
variance – a measure of how spread out a
distribution is. The average squared deviation of
each value from its mean. Example: values = {1, 2,
3}
2 =( (1-2)2 + (2-2)2 + (3-2)2)/3 = 0.667
for a population:


2 = Σ(X – μ)2 / N
μ = mean, N = number of samples
for a sample:


s2 = Σ (X – M)2/(N-1)
M = mean of the sample
Standard deviation
for a population:

 = sqrt(2)
for a sample:

s = sqrt(s2)
Normal distribution:
68-95-99 rule
normal dist = bellshaped curve
68% of cases fall
w/in one S.D.
95% of cases fall
w/in two S.D.
99% fall w/in three
S.D.
Basic sampling concept:
If we had a sampling distribution, we would be able to predict the 68,
95 and 99% confidence intervals for where the population parameter
should be!

We don't actually have the sampling distribution
We do have the distribution for the sample itself.
From that distribution we can estimate the standard error (the
sampling error) because it is based on the standard deviation and we
have that.
We still don't actually know the population parameter value, but we can
use our best estimate for that -- the sample statistic.
Now, we can use


the mean of our sample as the mean of the sampling distribution
standard deviation and sample size to estimate the standard error
 SE = s / sqrt(N)
to estimate confidence intervals for the population parameter.
An example:
We draw a 100 member
sample from a population



M = 3.75
s = 0.25
SE = 0.25 / 10 = 0.025
p(3.725 < μ < 3.775) = 0.68
p(3.700 < μ < 3.800) = 0.95
p(3.675 < μ < 3.825) = 0.99
these are known as
confidence intervals