Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Resident Research Preparation Lecture Series Alexander Villafranca, BESS, MSc.. Lecture 7- Analyzing a research study Powerpoint Templates Page 1 Objectives • “What is the field of statistics and what is it used for?” • “What are some basic ideas in descriptive and inferential statistics I should know about?” • “What are some pitfalls in data analysis and data interpretation?” • “What resources are available to help me plan and conduct a statistical analysis?” Powerpoint Templates Page 2 Steps to a successful resident research project Lecture 1- Starting Lecture 2- Planning Lecture 3 & 4- Designing Lecture 5- Proposing Lecture 6- Conducting Lecture 7- Analyzing Lecture 8- Reporting Powerpoint Templates Page 3 What is the field of statistics? • The study of how to organize, analyze, and interpret data • Broad goal- overcome inability of human to make sense of number lists visually • Are different branches of statistics with different purposes Powerpoint Templates Page 4 Q: Why do we use statistics? Powerpoint Templates Page 5 Why use statistics? • Describe a population or compare population subgroups using a few summary numbers and/or plots (Descriptive statistics) • Use sample descriptors to try to estimate descriptors of underlying population (Parameter estimation) Powerpoint Templates Page 6 Why use statistics? •Help create hypotheses for future projects, or guide/test model selection with summary numbers and plots (Exploratory statistics) •make statistical inferences about one or more populations (Inferential statistics) using data from representative sample Powerpoint Templates Page 7 Q: What kind things would we want to infer about one or more populations? Powerpoint Templates Page 8 Kinds of inferences • Determine if differences observed between samples apply to underlying populations • Determine if relationships observed between predictors and outcomes in a sample are present in the underlying population • Determine if relationships observed between predictors (interactions) in a sample are present in the underlying Powerpoint Templates population Page 9 Kinds of inferences • The inference you want to make may be framed as a hypothesis (a statement you are testing) • Don’t need to use inferential stats if you have sampled the entire population (e.g. census, some q/a contexts) Powerpoint Templates Page 10 Terminology • Parameter – Unknown population descriptor we want to estimate (e.g. mean body mass of population) • Statistic – Number we calculate based on data from a sample (e.g. mean body mass of population) – Used to estimate a parameter Powerpoint Templates Page 11 Terminology • Outcome – A.k.a. dependent variable/ criterion measure – The main variable you want to observe, predict, or compare across groups • Predictor of interest – A.k.a. independent variable – Variables of interest which we think could help predict our outcome • Covariate – Variable which we think could help predict our outcome, but is not of interest (already known, etc…) Powerpoint Templates Page 12 • To understand stats, need to understand frequency distributions and probability distributions Powerpoint Templates Page 13 Types of frequency distributions Variable distributions Distributions of sample statistics • Shows how often variable takes on a specific value • Shows how often you (usually in a sample) would get a statistic of a given value if you re-ran • Values in distribution (dataset) the experiment a bunch of used to calculate descriptive & Powerpoint Templates Page 14 times inferential statistics Probability distributions Can change frequency distributions into probability (relative fq) distributions by dividing fq by total number of values (left), or total number of experiments (right) • Probability on y axis instead of fq • Total area of any probability distribution equal to 1 • Same shape as fq distribution Powerpoint Templates Page 15 Normal/ Gaussian distribution • Bell shaped • Symmetric about vertical axis • Type of probability distribution • Originally thought to be most common, but are many others • Both variablePowerpoint & statistic distributions can be Templates Page 16 Gaussian Non-normal due to skewness (amount of asymmetry) Non-symmetric (outliers in one direction) Left (negative) skewed Right (positive) skewed • Common in medicine Powerpoint Templates Page 17 Non-normal due to different kurtosis (pointy-ness) • Platykurtic- flatter and wider peak, thinner tails • Leptokurticpronounced peak, Powerpoint Templates fatter tails Page 18 Describing variable distributions (descriptive statistics) Already talked about less commonly reported descriptors of variable distributions: • Symmetry of the distribution (Skewness) • Pointy-ness of the distribution (Kurtosis) • Others: Powerpoint Templates Page 19 Measures of central tendency within a sample Mean Median Mode Other- e.g. trimean Value seen around the centre of the curve Powerpoint Templates Page 20 Measures of spread/variability within a sample Variance Standard deviation Range IQR How spread out the data points are from the distribution centre Powerpoint Templates Page 21 Distributions of statistics Powerpoint Templates Page 22 How do they come up with distributions of statistics? Simulation studies: • Have a dataset representing single population • Pick repeated random samples and calculate statistic • Plot outcomes of experiments as histogram • Fit histogram with a curve (Generate fq distribution) • Can create probability distribution graph Powerpoint Templates Page 23 What are distributions of statistics good for? • Creating probability distribution lets you calculate probability of seeing a statistic equal or more extreme then the one observed, by chance • Called the p-value (area under the curve beyond the statistic value observed) • Smaller the p-value (area), the less likely effect is due to chance • Helps decide whether to accept sample difference, etc… as applying to the populations Powerpoint Templates Page 24 Example: Z-score •If in a single experiment you got a Z=0.1 difference likely due to chance (probably isn’t true of population) •Area between 0.1 and right of curve is big, so p Powerpoint Templates Page 25 value would be big as well E.g. Hypothesis testing • Let’s look at an example • Observational study • Comparing 2 groups – 50 People who report eating fast food >1/month vs 50 people who report eating fast food <1/month – Outcome is body mass (measured by experimenter) – Predictors collected include height, self reported exercise level, sex, etc… Powerpoint Templates Page 26 E.g. Hypothesis testing • Mean body mass of samples visually different – Frequent fast food group- 120 kg (SD-10) – Infrequent fast food group- 110 kg (SD- 10) • But do the differences apply to underlying populations? Powerpoint Templates Page 27 E.g. Hypothesis testing • Start by stating: 1) HA- alternative hypothesis- the statement we want to test e.g. There is a significant difference between the mean mass of two groups 2) H0- null hypothesis- the opposite statement from HA e.g. There is no significant difference between the mean mass of two groups Powerpoint Templates Page 28 E.g. Hypothesis testing • Set alpha- highest probability of saying there is a significant effect (i.e. difference), when there is actually NOT such an effect, which we will accept • Expressed as probability (e.g. alpha=0.05 = are willing to accept a 5% chance of type 1 error) • Type 1 error- Supporting HA when it should be rejected (saying diff applies to population when it doesn’t) • Alpha cutoffs are arbitrary: tradition says 0.05 or 0.01 Let’s set ours at 0.01 for the example Powerpoint Templates Page 29 E.g. Hypothesis testing • Pick test (series of mathematical steps)e.g. Two sample Z-test for our example • Formalize alpha and test used into a “decision rule”: e.g. if Z statistic takes a value corresponding to a p-value <0.01, conclude that the difference in the samples applies to the populations (i.e. You will accept HA) Powerpoint Templates Page 30 E.g. Hypothesis testing • Calculate stat generated by test (e.g. Z-statistic) Z=observed difference- expected difference/ standard error of difference Z=(10-0)/2= 5!!! • Turn statistic value into a p-value (computer will calculate) two-tailed p< 0.0001 • Use decision rule to decide whether to accept or reject HA Powerpoint Templates Page 31 E.g. Hypothesis testing • Use decision rule to decide whether to accept or reject HA Since p<0.05, accept HA Powerpoint Templates Page 32 Writing it up, part 1 • A two sample z-test demonstrated that there was a significant difference between the mean body masses of the frequent and infrequent fast food consumption groups (p< 0.0001) Powerpoint Templates Page 33 Understanding p-values • p-value is only indicting risk of type 1 error due to random variation, NOT indicating how bias could affecting result • Don’t be blinded by p value cutoffs: should put more stock into comparison with p=0.0001 than comparison with p=0.044 Powerpoint Templates Page 34 Understanding statistical significance • Formulas for statistics to test differences between group often have a format similar to this: Statistic= Overall effect normalized to random variability Powerpoint Templates Page 35 Understanding statistical significance • Different ways to achieve significance: – Small overall var. & miniscule within groups var. – Big overall var. & smaller within groups var. • Thus: – Statistically significance not always = to clinically significance – Statistically significance not always equal to a large effect Powerpoint Templates Page 36 Recap • So we’ve decided that the mean mass of the 2 groups is different • How do we know the direction of difference? – Look at parameter estimates • i.e. in this case, we need to know diff between mean mass of populations • How do we estimate this? Powerpoint Templates Page 37 Parameter estimation • Point estimate approach – Pick single “best guess” as to what population parameter would be – Normally take related sample statistic to be best guess (e.g. population difference= sample difference) – In our example: 120-110= 10kg difference • Better idea: add interval estimate – a range of values the true population parameter is likely to fall within (e.g. population difference= sample difference +/interval) – In our Powerpoint example Templates 10 kg [6.03-13.97] 95% CI Page 38 Reporting results, part 2 • A two sample z-test demonstrated that the frequent fast food consumption group had a significantly higher mean body mass than the infrequent fast food consumption group (p< 0.0001, table 1) Body Mass (kg) Frequent fast Infrequent food mean fast food (SD) mean (SD) Difference and 95% CI of the difference 120 10 kg [6.03-13.97] 110 Powerpoint Templates Page 39 Interval estimates • Theory behind interval estimation: – Estimate averaged from multiple samples/experiments would be closer to real value than estimate from single experiment • Can figure out variability of estimate derived from single sample without having to resample, using formulas developed • Take sampling variabilities and n into account Powerpoint Templates Page 40 Interval estimates • Different types – Standard errors (of mean, difference between means, etc…) – Confidence intervals • 95% confidence interval- 95/100 experiments, difference would fall within this interval thus interval probably includes true value of population difference • Greater the sample size, the narrower interval estimate (greater precision) • Great variabilities within groups lead to wider interval estimates Powerpoint Templates Page 41 Distinctions in Inferential statistics • Parametric vs Non parametric tests – Statistical tests can only be used in circumstances similar to simulation studies – Assume certain things about your dataset – Parametric tests assume data has a specific distribution (e.g. normal distribution) – Non parametric do not make this assumption Powerpoint Templates Page 42 Distinctions in Inferential statistics • Paired vs unpaired comparisons – Some tests make assumptions about how your data were collected (study designs) – Between subjects designs unpaired tests – Within subjects designs paired tests – Panel designs more complex models Powerpoint Templates Page 43 Distinctions in Inferential statistics • Univariate Hypothesis tests vs multivariate models – Normally single variable tests don’t cut it in medicine, since there are covariates (unless strict experimental design) – Usually need multivariate models- give you estimates of the effect of a variable, independent of the other variables entered into the model (e.g. differences in mass, independent of sex, height, etc…) Powerpoint Templates Page 44 Reporting statistical results Normally give: • Point estimates + Interval estimates • P-values & alpha • Maybe effect sizes - give a measure of how big the effect is – Based on var. explained, distance btwn means, etc… – Many different types: • Odds ratio • R • Others • R squared Powerpoint • Cohen’s D Templates Page 45 Reporting results, part 3 • A two sample z-test demonstrated that the frequent fast food consumption group had a significantly higher mean body mass than the infrequent fast food consumption group (Cohen’s D= 1.0, two tailed p< 0.0001, table 1) Body Mass (kg) Frequent fast foodmean (SD) Infrequent fast foodmean (SD) Difference [95% CI of the difference] Effect size (Cohen’s D) 120 110 10 [6.03-13.97] 1.0 Powerpoint Templates Page 46 Q: What are some data analysis pitfalls? Powerpoint Templates Page 47 Data analysis pitfalls (assuming good study design) • Not having statistical plan before starting project • Not testing assumptions of statistical model – Using non-paired tests with paired data – Using parametric tests with non-parametric data • Not thoughtfully dealing with missing data & outliers • Assuming there is a “right test” for a situation, which everyone will accept Powerpoint Templates Page 48 Data analysis pitfalls (assuming good study design) • Data snooping/massaging/dredging • Overestimating your expertise in statistics Powerpoint Templates Page 49 Data snooping • Running statistical tests on the same comparisons a bunch of different ways until it somehow reaches significance • Running tests on every variable collected to get a positive result to report • Increases type I error dramatically • Often accomplished through absurd post-hoc (after the fact) stratification Powerpoint Templates Page 50 Q: What are some data interpretation pitfalls? Powerpoint Templates Page 51 Data interpretation pitfalls • Not having a basic (conceptual) understanding of statistics used • Getting “hung up” on p-value cutoffs- pvalue gives more info than binary significant/not significant classification • Confusing correlation with causation • Not understanding difference between results (what the numbers are) & findings (what you think they mean) Powerpoint Templates Page 52 Data interpretation pitfalls • Thinking that every “positive” finding is really true • Mistaking statistical significance for clinical significance Powerpoint Templates Page 53 How do I get help with stats? • Resources available – Anesthesia research office: myself,Linda Girling can provide some advice or help interpret/implement the recommendations of PhD statisticians http://umanitoba.ca/faculties/medicine/units/anesth esia/about/about_hub.html – Biostatistical consulting unit: PhD statisticians based out of Community Health Sciences http://umanitoba.ca/faculties/medicine/units/commu nity_health_sciences/departmental_units/biostat. html Powerpoint Templates Page 54 Summary • Should plan on seeking statistical consultation BEFORE project begins • Don’t underestimate the time and expertise needed to do data analysis • High impact journals often have statistician reviewers, so best to get professional advice • You need to be able to understand the stats well enough to defend them in presentations • Avoid the other pitfalls in data analysis and interpretation Powerpoint Templates Page 55 Questions? Powerpoint Templates Page 56 References/ additional Resources • Dawson-Saunders, B., Trapp, R. (1994) Basic and clinical biostatistics, 2nd edition, Appleton and Lange, Paramount publishing business and professional group. • Harvey, B.J., Lang, E.S., Frank, J.R. (2011) The research guide: A primer for residents, other health trainees, and practitioners. Royal College of Physicians and surgeons of Canada • Katz, M.H. (2011) Multivariable Analysis: A guide for clinicians and public health researchers. Third edition. Cambridge University Press. • Alexander M. Strasak, Qamruz Zaman, Karl P. Pfeiffer, Georg Göbel, Hanno Ulmer, Statistical errors in medical research – A review of common pitfalls, SWISS MED WKLY 2007 ; 1 3 7 : 4 4 – 4 9 Powerpoint Templates Page 57