Download Difference

Resident Research
Preparation Lecture Series
Alexander Villafranca, BESS, MSc..
Lecture 7- Analyzing a research study
Powerpoint Templates
Page 1
Objectives
• “What is the field of statistics and what is
it used for?”
• “What are some basic ideas in
descriptive and inferential statistics I
should know about?”
• “What are some pitfalls in data analysis
and data interpretation?”
• “What resources are available to help me
plan and conduct a statistical analysis?”
Powerpoint Templates
Page 2
Steps to a successful resident research project
Lecture 1- Starting
Lecture 2- Planning
Lecture 3 & 4- Designing
Lecture 5- Proposing
Lecture 6- Conducting
Lecture 7- Analyzing
Lecture 8- Reporting
Powerpoint Templates
Page 3
What is the field of
statistics?
• The study of how to organize, analyze,
and interpret data
• Broad goal- overcome inability of
human to make sense of number lists
visually
• Are different branches of statistics with
different purposes
Powerpoint Templates
Page 4
Q: Why do we use statistics?
Powerpoint Templates
Page 5
Why use statistics?
• Describe a population or compare
population subgroups using a few
summary numbers and/or plots
(Descriptive statistics)
• Use sample descriptors to try to estimate
descriptors of underlying population
(Parameter estimation)
Powerpoint Templates
Page 6
Why use statistics?
•Help create hypotheses for future projects,
or guide/test model selection with summary
numbers and plots (Exploratory statistics)
•make statistical inferences about one or
more populations (Inferential statistics)
using data from representative sample
Powerpoint Templates
Page 7
Q: What kind things would we
want to infer about one or
more populations?
Powerpoint Templates
Page 8
Kinds of inferences
• Determine if differences observed
between samples apply to underlying
populations
• Determine if relationships observed
between predictors and outcomes in a
sample are present in the underlying
population
• Determine if relationships observed
between predictors (interactions) in a
sample are present in the underlying
Powerpoint Templates
population
Page 9
Kinds of inferences
• The inference you want to make may be
framed as a hypothesis (a statement you
are testing)
• Don’t need to use inferential stats if you
have sampled the entire population (e.g.
census, some q/a contexts)
Powerpoint Templates
Page 10
Terminology
• Parameter
– Unknown population descriptor we want to
estimate (e.g. mean body mass of
population)
• Statistic
– Number we calculate based on data from a
sample (e.g. mean body mass of
population)
– Used to estimate a parameter
Powerpoint Templates
Page 11
Terminology
• Outcome
– A.k.a. dependent variable/ criterion measure
– The main variable you want to observe,
predict, or compare across groups
• Predictor of interest
– A.k.a. independent variable
– Variables of interest which we think could
help predict our outcome
• Covariate
– Variable which we think could help predict
our outcome, but is not of interest (already
known, etc…)
Powerpoint Templates
Page 12
• To understand stats, need to
understand frequency
distributions and probability
distributions
Powerpoint Templates
Page 13
Types of frequency distributions
Variable distributions
Distributions of sample
statistics
• Shows how often variable
takes on a specific value
• Shows how often you
(usually in a sample)
would get a statistic of a
given value if you re-ran
• Values in distribution (dataset)
the experiment a bunch of
used to calculate descriptive
&
Powerpoint Templates
Page 14
times
inferential statistics
Probability distributions
Can change frequency distributions into probability (relative fq)
distributions by dividing fq by total number of values (left), or total
number of experiments (right)
• Probability on y axis instead of fq
• Total area of any probability distribution equal to 1
• Same shape as fq distribution
Powerpoint Templates
Page 15
Normal/ Gaussian distribution
• Bell shaped
• Symmetric about
vertical axis
• Type of probability distribution
• Originally thought to be most common, but are
many others
• Both variablePowerpoint
& statistic
distributions
can
be
Templates
Page 16
Gaussian
Non-normal due to skewness
(amount of asymmetry)
Non-symmetric (outliers in one direction)
Left (negative)
skewed
Right (positive) skewed
• Common in medicine
Powerpoint Templates
Page 17
Non-normal due to
different kurtosis
(pointy-ness)
• Platykurtic- flatter and
wider peak, thinner tails
• Leptokurticpronounced peak,
Powerpoint Templates fatter tails
Page 18
Describing variable
distributions (descriptive
statistics)
Already talked about less commonly
reported descriptors of variable
distributions:
• Symmetry of the distribution (Skewness)
• Pointy-ness of the distribution (Kurtosis)
• Others:
Powerpoint Templates
Page 19
Measures of central tendency
within a sample
Mean
Median
Mode
Other- e.g. trimean
Value seen around the centre of the curve
Powerpoint Templates
Page 20
Measures of spread/variability
within a sample
Variance
Standard deviation
Range
IQR
How spread out the data points are from the
distribution centre
Powerpoint Templates
Page 21
Distributions of statistics
Powerpoint Templates
Page 22
How do they come up with
distributions of statistics?
Simulation studies:
• Have a dataset representing single
population
• Pick repeated random samples and
calculate statistic
• Plot outcomes of experiments as histogram
• Fit histogram with a curve (Generate fq
distribution)
• Can create probability distribution graph
Powerpoint Templates
Page 23
What are distributions of
statistics good for?
• Creating probability distribution lets you
calculate probability of seeing a statistic
equal or more extreme then the one
observed, by chance
• Called the p-value (area under the curve
beyond the statistic value observed)
• Smaller the p-value (area), the less likely
effect is due to chance
• Helps decide whether to accept sample
difference, etc… as applying to the
populations
Powerpoint Templates
Page 24
Example: Z-score
•If in a single experiment you got a Z=0.1
difference likely due to chance (probably isn’t
true of population)
•Area between 0.1 and right of curve is big, so p
Powerpoint Templates
Page 25
value would be big as well
E.g. Hypothesis testing
• Let’s look at an example
• Observational study
• Comparing 2 groups
– 50 People who report eating fast food
>1/month vs 50 people who report eating fast
food <1/month
– Outcome is body mass (measured by
experimenter)
– Predictors collected include height, self
reported exercise level, sex, etc…
Powerpoint Templates
Page 26
E.g. Hypothesis testing
• Mean body mass of samples visually
different
– Frequent fast food group- 120 kg (SD-10)
– Infrequent fast food group- 110 kg (SD- 10)
• But do the differences apply to
underlying populations?
Powerpoint Templates
Page 27
E.g. Hypothesis testing
• Start by stating:
1) HA- alternative hypothesis- the
statement we want to test
e.g. There is a significant difference
between the mean mass of two groups
2) H0- null hypothesis- the opposite
statement from HA
e.g. There is no significant difference
between the mean mass of two groups
Powerpoint Templates
Page 28
E.g. Hypothesis testing
• Set alpha- highest probability of saying
there is a significant effect (i.e.
difference), when there is actually NOT
such an effect, which we will accept
• Expressed as probability (e.g.
alpha=0.05 = are willing to accept a 5%
chance of type 1 error)
• Type 1 error- Supporting HA when it
should be rejected (saying diff applies
to population when it doesn’t)
• Alpha cutoffs are arbitrary: tradition
says 0.05 or 0.01
Let’s set ours at 0.01 for the example
Powerpoint Templates
Page 29
E.g. Hypothesis testing
• Pick test (series of mathematical
steps)e.g. Two sample Z-test for our example
• Formalize alpha and test used into a
“decision rule”:
e.g. if Z statistic takes a value
corresponding to a p-value <0.01,
conclude that the difference in the
samples applies to the populations (i.e.
You will accept HA)
Powerpoint Templates
Page 30
E.g. Hypothesis testing
• Calculate stat generated by test (e.g.
Z-statistic)
Z=observed difference- expected
difference/ standard error of difference
Z=(10-0)/2= 5!!!
• Turn statistic value into a p-value
(computer will calculate)
two-tailed p< 0.0001
• Use decision rule to decide whether to
accept or reject HA
Powerpoint Templates
Page 31
E.g. Hypothesis testing
• Use decision rule to decide whether to
accept or reject HA
Since p<0.05, accept HA
Powerpoint Templates
Page 32
Writing it up, part 1
• A two sample z-test demonstrated that
there was a significant difference
between the mean body masses of the
frequent and infrequent fast food
consumption groups (p< 0.0001)
Powerpoint Templates
Page 33
Understanding p-values
• p-value is only indicting risk of type 1
error due to random variation, NOT
indicating how bias could affecting
result
• Don’t be blinded by p value cutoffs:
should put more stock into comparison
with p=0.0001 than comparison with
p=0.044
Powerpoint Templates
Page 34
Understanding statistical
significance
• Formulas for statistics to test differences
between group often have a format
similar to this:
Statistic= Overall effect normalized to
random variability
Powerpoint Templates
Page 35
Understanding statistical
significance
• Different ways to achieve significance:
– Small overall var. & miniscule within groups
var.
– Big overall var. & smaller within groups var.
• Thus:
– Statistically significance not always = to
clinically significance
– Statistically significance not always equal to
a large effect
Powerpoint Templates
Page 36
Recap
• So we’ve decided that the mean mass of
the 2 groups is different
• How do we know the direction of
difference?
– Look at parameter estimates
• i.e. in this case, we need to know diff
between mean mass of populations
• How do we estimate this?
Powerpoint Templates
Page 37
Parameter estimation
• Point estimate approach
– Pick single “best guess” as to what population
parameter would be
– Normally take related sample statistic to be
best guess (e.g. population difference=
sample difference)
– In our example: 120-110= 10kg difference
• Better idea: add interval estimate
– a range of values the true population
parameter is likely to fall within (e.g.
population difference= sample difference +/interval)
– In our Powerpoint
example Templates
10 kg [6.03-13.97] 95% CI
Page 38
Reporting results, part 2
• A two sample z-test demonstrated that
the frequent fast food consumption
group had a significantly higher mean
body mass than the infrequent fast
food consumption group (p< 0.0001,
table 1)
Body Mass
(kg)
Frequent fast Infrequent
food mean
fast food
(SD)
mean (SD)
Difference and
95% CI of the
difference
120
10 kg [6.03-13.97]
110
Powerpoint Templates
Page 39
Interval estimates
• Theory behind interval estimation:
– Estimate averaged from multiple
samples/experiments would be closer to real
value than estimate from single experiment
• Can figure out variability of estimate derived
from single sample without having to
resample, using formulas developed
• Take sampling variabilities and n into
account
Powerpoint Templates
Page 40
Interval estimates
• Different types
– Standard errors (of mean, difference
between means, etc…)
– Confidence intervals
• 95% confidence interval- 95/100 experiments,
difference would fall within this interval thus
interval probably includes true value of
population difference
• Greater the sample size, the narrower
interval estimate (greater precision)
• Great variabilities within groups lead to
wider interval estimates
Powerpoint Templates
Page 41
Distinctions in Inferential
statistics
• Parametric vs Non parametric tests
– Statistical tests can only be used in
circumstances similar to simulation studies
– Assume certain things about your dataset
– Parametric tests assume data has a specific
distribution (e.g. normal distribution)
– Non parametric do not make this assumption
Powerpoint Templates
Page 42
Distinctions in Inferential
statistics
• Paired vs unpaired comparisons
– Some tests make assumptions about how
your data were collected (study designs)
– Between subjects designs unpaired tests
– Within subjects designs paired tests
– Panel designs more complex models
Powerpoint Templates
Page 43
Distinctions in Inferential
statistics
• Univariate Hypothesis tests vs
multivariate models
– Normally single variable tests don’t cut it in
medicine, since there are covariates (unless
strict experimental design)
– Usually need multivariate models- give you
estimates of the effect of a variable,
independent of the other variables entered
into the model (e.g. differences in mass,
independent of sex, height, etc…)
Powerpoint Templates
Page 44
Reporting statistical
results
Normally give:
• Point estimates + Interval estimates
• P-values & alpha
• Maybe effect sizes - give a measure of how
big the effect is
– Based on var. explained, distance btwn means,
etc…
– Many different types:
• Odds ratio
• R
• Others
• R squared
Powerpoint
• Cohen’s
D Templates
Page 45
Reporting results, part 3
• A two sample z-test demonstrated that
the frequent fast food consumption
group had a significantly higher mean
body mass than the infrequent fast
food consumption group (Cohen’s D=
1.0, two tailed p< 0.0001, table 1)
Body
Mass (kg)
Frequent
fast foodmean (SD)
Infrequent
fast foodmean (SD)
Difference
[95% CI of
the
difference]
Effect size
(Cohen’s D)
120
110
10
[6.03-13.97]
1.0
Powerpoint Templates
Page 46
Q: What are some data
analysis pitfalls?
Powerpoint Templates
Page 47
Data analysis pitfalls
(assuming good study design)
• Not having statistical plan before starting
project
• Not testing assumptions of statistical model
– Using non-paired tests with paired data
– Using parametric tests with non-parametric data
• Not thoughtfully dealing with missing data &
outliers
• Assuming there is a “right test” for a
situation, which everyone will accept
Powerpoint Templates
Page 48
Data analysis pitfalls
(assuming good study design)
• Data snooping/massaging/dredging
• Overestimating your expertise in statistics
Powerpoint Templates
Page 49
Data snooping
• Running statistical tests on the same
comparisons a bunch of different ways
until it somehow reaches significance
• Running tests on every variable
collected to get a positive result to
report
• Increases type I error dramatically
• Often accomplished through absurd
post-hoc (after the fact) stratification
Powerpoint Templates
Page 50
Q: What are some data
interpretation pitfalls?
Powerpoint Templates
Page 51
Data interpretation pitfalls
• Not having a basic (conceptual)
understanding of statistics used
• Getting “hung up” on p-value cutoffs- pvalue gives more info than binary
significant/not significant classification
• Confusing correlation with causation
• Not understanding difference between
results (what the numbers are) &
findings (what you think they mean)
Powerpoint Templates
Page 52
Data interpretation pitfalls
• Thinking that every “positive” finding is
really true
• Mistaking statistical significance for
clinical significance
Powerpoint Templates
Page 53
How do I get help with
stats?
• Resources available
– Anesthesia research office: myself,Linda Girling can provide some advice or help
interpret/implement the recommendations of PhD
statisticians
http://umanitoba.ca/faculties/medicine/units/anesth
esia/about/about_hub.html
– Biostatistical consulting unit: PhD statisticians
based out of Community Health Sciences
http://umanitoba.ca/faculties/medicine/units/commu
nity_health_sciences/departmental_units/biostat.
html
Powerpoint Templates
Page 54
Summary
• Should plan on seeking statistical
consultation BEFORE project begins
• Don’t underestimate the time and
expertise needed to do data analysis
• High impact journals often have
statistician reviewers, so best to get
professional advice
• You need to be able to understand the
stats well enough to defend them in
presentations
• Avoid the other pitfalls in data analysis
and interpretation
Powerpoint Templates
Page 55
Questions?
Powerpoint Templates
Page 56
References/ additional
Resources
• Dawson-Saunders, B., Trapp, R. (1994) Basic and
clinical biostatistics, 2nd edition, Appleton and Lange,
Paramount publishing business and professional group.
• Harvey, B.J., Lang, E.S., Frank, J.R. (2011) The
research guide: A primer for residents, other health
trainees, and practitioners. Royal College of Physicians
and surgeons of Canada
• Katz, M.H. (2011) Multivariable Analysis: A guide for
clinicians and public health researchers. Third edition.
Cambridge University Press.
• Alexander M. Strasak, Qamruz Zaman, Karl P. Pfeiffer,
Georg Göbel, Hanno Ulmer, Statistical errors in medical
research – A review of common pitfalls, SWISS MED
WKLY 2007 ; 1 3 7 : 4 4 – 4 9
Powerpoint Templates
Page 57