Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
How to Torture Your Statistician: Ben Herman ACRIN Biostatistics Center Brown University Providence, RI ACRIN Fall Meeting – RA Session ACRIN Fall Meeting – RA or Session Title of Presentation Statistical Presentation Conference Here 29 September 2010 September 29,2010 22 May 2017 Acknowledgements: Divider (sub-section) Slide • Although they undoubtedly would like to distance themselves from the title and the rest of this talk, ACRIN receives funding from the National Cancer Institute through the grants U01 CA079778 and U01 CA080098 * Thanks to all the web sites/sources from which I liberally borrowed much of this material (not all of which is cited) ACRIN Fall Meeting – RA Session September 29,2010 Divider (sub-section) Slide Disclaimer: The view expressed in the presentation are wholly my own, they do not necessarily represent ACRIN, ACRIN’s data collection policies, Data Management, real statistics, or the general views of the Biostatistics Center. Shhh… nobody knows I’m giving this talk. ACRIN Fall Meeting – RA Session September 29,2010 Charlatans, Liars & Cheats? And Everybody Else “The group was alarmed to find that if you are a labourer, cleaner or dock worker, you are twice as likely to die than a member of the professional classes”, from The Sunday Times, 31st August 1980. ACRIN TitleFall of Presentation Meeting – RA or Session Conference Here September 22 May 2017 29,2010 Fortune Example • As quickly as you can do the following: • 2+2+2=? • 7+7+7=? • What is the first VEGETABLE that comes to mind? – Tomatoes are a fruit…… – Did you say Carrot? • 98% of normal people do! ACRIN TitleFall of Presentation Meeting – RA or Session Conference Here September 22 May 2017 29,2010 Know your subjects Divider (sub-section) Slide Why are Cooperative Group statisticians so hard to break? Not beholden to the hypothesis No vested interest in the outcome Can make objective assessments &recommendations to the DSMC ACRIN Fall Meeting – RA Session September 29,2010 Know your subjects Normal stressors Opportunities for Mayhem! What does an ACRIN Statistician Do? Divider (sub-section) Slide Study Design and Analysis Plans Monitor Trial Progress Aggregate Information Report to Monitors (DSMC, NCI, CIP, etc.) Data Analyses and Reports/Papers ACRIN Fall Meeting – RA Session September 29,2010 Know your subjects Normal stressors Tools of the trade The soft Underbelly of Statistics Hit ‘em Where it Hurts! Divider (sub-section) Slide Information Information Information ACRIN Fall Meeting – RA Session September 29,2010 Know your subjects Normal stressors Tools of the trade Control information Example • A $50 Million a year company has entry level Divider (sub-section) Slide positions open for people willing to work their way up. The company pays $30M in salary compensation to its 150 employees. Therefore, the Average salary at the company is $200,000/yr should you take a job? Max = $19M Median= $200K Mean = $200K ACRIN Fall Meeting – RA Session Median= $10K Mean = $200K September 29,2010 Know your subjects Normal stressors Tools of the trade Control Information Example Effective Information Control Use general termsDivider like (sub-section) average Slide Never disclose your assumptions and bin your data Need Blinding or other interesting study designs Expect P-values but ignore power(until the end) Refuse to accurately identify the Sample/Population Vigorously deny any potential sources of Bias Explore every possible hypothesis conceivable ACRIN Fall Meeting – RA Session September 29,2010 Examples: Interesting Designs P-values v Power Sample/Population Assumptions/Bias Divider (sub-section) Slide Conclusions As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomized controlled trials. Advocates of evidence based medicine have criticized the adoption of interventions evaluated by We think that everyone might benefit if the most radical protagonists of evidence based medicine organized and participated in a double blind, randomized, placebo controlled, crossover trial of the parachute. using only observational data. ACRIN Fall Meeting – RA Session September 29,2010 Examples: Interesting Designs P-values v Power Sample/Population Assumptions/Bias Power v P: measures of Probability not Pain Divider (sub-section) Slide Null Hypothesis: The effect we are trying to DISPROVE Alpha (a): Probability of being WRONG! Set a priori (Falsely rejecting the Null Hypothesis) Power Probability of being right given the assumptions. (Correctly rejecting the Null Hypothesis) P-Values A measure of evidence under the NULL Hypothesis ACRIN Fall Meeting – RA Session September 29,2010 Examples: Interesting Designs P-values v Power Sample/Population Assumptions/Bias Examples Divider (sub-section) Slide Were gonna be rich! Hypothesis: Roulette table pays off red 20% or more than black Null Hypothesis: Black and Red occur equally (%R-%B<20%) Alpha: reject at the a=5% level (I.e., P-Value <0.05) 80% Power: used to calculate sample size = 100 BAD Unplanned looks at the data /Multiple looks at the data -- If you look at the data 20 times you expect to discover at least 1 false significant result Result: 61 red of 100 cases (P>0.05) "No one can possibly win at roulette unless he steals money from the table while the croupier isn't looking." — Albert Einstein ACRIN Fall Meeting – RA Session September 29,2010 Examples: Interesting Designs P-values v Power Sample/Population Assumptions/Bias Population: The group you want to study (generalize) (sub-section) Slide Sample: The subjects Divider you actually study Population: Americans meeting screening guidelines for CRC Sample: 1600 Asymptomatic Americans >50 yrs old using Protocol specified prep, technique, parameters, etc. To maximize pain at analysis: Refuse to identify any abnormality observed during accrual that might allow the statistician to subset or re-categorize the data. ACRIN Fall Meeting – RA Session September 29,2010 Examples: Interesting Designs P-values v Power Sample/Population Assumptions/Bias Birth Control is 99.9 effective when used according to directions Divider (sub-section) Slide •1 in 1000 fail but what is the population they are talking about Is it a single regimen? Is it a single dose? Is it per person (1 out of every 1000 users)? Who are the Failures? Is there something special about these cases? How do you collect this special information ACRIN Fall Meeting – RA Session September 29,2010 Examples: Interesting Designs P-values v Power Sample/Population Assumptions/Bias Assumptions may introduce bias into the study (sub-section)aSlide -Forms will alwaysDivider be completed certain way -Procedures will always go as planned -Technical data/Lab values are not significant -A Yes/No response will be sufficient to answer the question -All Bias is identifiable Bias: a systematic error that may alter the outcome -Approach only those patient you think will complete -Only help some people complete forms -Unblind readers to results Beware: statisticians have ways to correct/report on some forms of bias if they know about it – So never let them know about it until it is too late! ACRIN Fall Meeting – RA Session September 29,2010 Examples: Interesting Designs P-values v Power Sample/Population Assumptions/Bias The Monty MitchyHall Schnall Problem Problem Divider (sub-section) Slide 1 ACRIN Fall Meeting – RA Session 2 3 September 29,2010 Examples: Interesting Designs P-values v Power Sample/Population Assumptions/Bias The Mitchy Schnall Problem If the Host is biased Switch Divider (sub-section) Slide If the Host is unbiased the probability after switching is 50% 1/3 Chance 2/3 Chance 1 2 3 1/3 1/3 1/3 ACRIN Fall Meeting – RA Session September 29,2010 DON’T ASK, DON’T TELL Divider (sub-section) Slide Why are statisticians so secretive? How do we get those secrets out of them? ACRIN Fall Meeting – RA Session September 29,2010 Don’t Ask: Random Blinding Feedback • Reduce Bias byDivider (sub-section) Slide – Randomization: choose people or treatments at random (in a reproducible manner) – Masking (Blinding): Do not let the patient know their treatment assignment – Double Masking: Don’t let anyone know the treatment assignment – Approach everyone in a predefined systematic manner ACRIN Fall Meeting – RA Session September 29,2010 Don’t Ask: Random Blinding Feedback • Feedback loops (Divider tell me(sub-section) how I’mSlidedoing) – Past events changes current events • Data Mining/ Exploration/Hypothesis generation – Collected data is explored for correlations/associations – A Large number of “Chance” associations must be explored • Hypothesis testing – A Hypothesis is developed – Data is collected to test the Hypothesis ACRIN Fall Meeting – RA Session September 29,2010 Don’t Ask: Random Blinding Feedback • Feedback loops or “Divider Tell us how good are (sub-section) Slide we doing!” – Research is not Training it is hypothesis testing Training and testing data must be kept separate. – Defeats the purpose of masking – Leads to uncorrectable bias and unstable performance – Introduces unknown confounders into the analysis • Data Mining or “Maybe it was this effect!” – At the a=0.05 level, we expect 5% of comparisons to have an effect size that exceeds the threshold. – Associations are not causal relationships ACRIN Fall Meeting – RA Session September 29,2010 Don’t Ask: Random Blinding Feedback Data Mining • DNA testing is 99.99% accurate Divider (sub-section) Slide – It’s wrong in 1 out of 10,000 • If a DNA database has 20,000 individuals – 86% chance of matching a random donor • If a DNA database has 40,000 individuals – 98% chance of matching a random donor • Should we have a national DNA database? ACRIN Fall Meeting – RA Session September 29,2010 KISS (Keep it Simple, Stupid!) Divider (sub-section) Slide The biggest threat to the Primary Aim of a study are the Secondary Aims. After the key elements required for analysis and monitoring, collect whatever is easy and meaningful. ACRIN Fall Meeting – RA Session September 29,2010 Divider (sub-section) Slide WHAT HAPPENS: AFTER YOU HIT ENTER Humans Vs Computers Statisticians Vs Humans Data Vs Information Queries ACRIN Fall Meeting – RA Session September 29,2010 After you hit Enter: Information Flow Garbage collection Goals Misinformation Divider (sub-section) Slide Data Entry (DM/HQ) Data Collection Patient Level CRAs Central Database DM Biostats Analysis Database Web Based DE (CRA/Sites) ACRIN Fall Meeting – RA Session September 29,2010 After you hit Enter: Information Flow Garbage collection Goals Misinformation Lady GIGO Divider (sub-section) Slide Garbage in, Garbage out Computers process numbers Humans interpret everything “5” + “A” Computer sees 53+65=118 (“v”?) “5”+ “Bob” = ??? Minimize errors: Double-data entry Real time data queries ACRIN Fall Meeting – RA Session Range/Logic checks Cross form validation September 29,2010 After you hit Enter: Information Flow Garbage collection Goals Misinformation Lady GIGO (cont) Divider (sub-section) Slide Cooperative Garbage Statisticians don’t like data from cooperative/outside groups Data form out side: Different Aims Not familiar with our study Different priorities Inability to query in a timely manner Long delays in getting data ACRIN Fall Meeting – RA Session No control of data Hard to audit source Hard to assign responsibility for data September 29,2010 After you hit Enter: Information Flow Garbage collection Goals Misinformation •Statisticians Divider (sub-section) Slide Interact with Computers and Data Aggregate Data Don’t generally care about individual cases Want all meaningful data available • RAs/PIs/DM Interact with People and Data Focus on individual cases Deal with individual data elements ACRIN Fall Meeting – RA Session September 29,2010 After you hit Enter: Information Flow Garbage collection Goals Misinformation Nobody wants to manipulate data, Divider (sub-section) Slide so who corrects the data? Current PHS Policies on Research Misconduct (42 CFR Parts 50 and 93) define falsification as, "manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.” • RAs: Want data to be as accurate as possible so put additional information on forms and in comments. • DMs: Want database to accurately reflect what is on the forms • Stats: Want data that is meaningful and analyzable in aggregate ACRIN Fall Meeting – RA Session September 29,2010 Analyze this Let the experts show you how it’s done Divider (sub-section) Slide ACRIN Fall Meeting – RA Session September 29,2010 Analyze this Let the experts show you how it’s done extensive proably invades nipple has 2-3 ductal extensions+5mm satellite 1tiny multifocal nodule inf sag loc 79.3 Divider (sub-section) Slide long ax actuallyAP~60mm nr pectoral musc 22 mm 79/232 long dia = oblique cc,meas on AP/coronal 47 ML, MIP 107 Sag many morphologic patterns 92/224 mass enhance pattern N/A, remove gradual A/P MIP ML MIP, S-I MIP E53=can't assess ActuallySeroma not cyst, measured on MIP add area enhancement w multi lob dom mas don't use case for vol/ser anterolat mass likely = more ca, MIP measurements lge ax. nodes; el 176=1 MIP-surrounded by fluid area of enh.in mass ext. from it is stip ML&APax87.9ser 8/14SIsgse7/14I33/66L90.1 area of lateral enhance.,indeterminate multicentric spiculated masses;e93=severe assoc. field effect superimposed on mass Multifocality & area enhan leads to ty C broad area 6:00, narrow at 12:00 None E93=severe can't evaluate axilla-fatsat failed not enough room for comments dominant mass w/assoc.uncontained enhanc post-surq changes in axilla e174 = unable to assess prior sentinal node biopsy COMMENTS E6=10 E93,94=unknown E93=SEVERE E93=severe ACRIN Fall Meeting – RA Session Pt is post-surg biopsy & ax. LN dissect Q17=3 rim enhancing cyst upper inner quadrant T2ax 19/39, 95/224 September 29,2010 How to Torture Your Statistician: simple ways to maximize pain! ACRIN Fall Meeting – RA Session ACRIN Fall Meeting – RA or Session Title of Presentation Statistical Presentation Conference Here 29 September 2010 September 29,2010 22 May 2017