Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Likelihood based confidence intervals Christine Borgen Linander A huge thank to a former colleague of mine Rune H B Christensen. DTU Compute Section for Statistics Technical University of Denmark [email protected] August 20th 2015 © Christine Borgen Linander (DTU) Likelihood based confidence intervals Sensometrics Summer School 1 / 15 © Christine Borgen Linander (DTU) Outline We are interested in: 2 Examples 3 The likelihood function and confidence intervals Examples revisited 5 Perspectives © Christine Borgen Linander (DTU) 2 / 15 Motivation Motivation 4 Sensometrics Summer School Motivation Outline 1 Likelihood based confidence intervals 1 Which sensory difference (d 0 ) is most supported by the data? 2 Which interval of sensory differences is supported by the data? We usually answer those with: 1 The maximum likelihood estimate (MLE), d̂ 0 2 A confidence interval (CI) for d 0 . . . but there are many ways to compute the CI, and which is best? Likelihood based confidence intervals Sensometrics Summer School 3 / 15 © Christine Borgen Linander (DTU) Likelihood based confidence intervals Sensometrics Summer School 4 / 15 Motivation Examples Problems with standard CIs John and Dorothy’s duo-trio experiment The guessing probability is 1/2 They obtain 13 correct answers to 20 samples Standard (Wald) 95% confidence intervals: John analyzes the probability of a correct answer, pc : p̂c = 0.65(0.11) and CI95% = [0.44; 0.86] which covers pc = 1/2 For binomial probability of a correct answer pc : p̂c ± 1.96 · se(p̂c ) For the Thurstonian δ: δ̂ ± 1.96 · se(δ̂) (Bi et al, 1997) Dorothy analyzes the Thurstonian δ: δ̂ = 1.42(0.63) and CI95% = [0.18; 2.66] (Bi et al, 1997) which does NOT cover δ = 0 Problems and solution: The standard CIs are incompatible and lead to contradictions The standard CIs do not cover the values of δ or pc that are most supported by the data Which method is (most) correct? CIs based on the likelihood function have better properties Should we trust John or Dorothy? How much evidence is there really in the data about a difference between the products? © Christine Borgen Linander (DTU) Likelihood based confidence intervals Sensometrics Summer School 5 / 15 © Christine Borgen Linander (DTU) Examples 6 / 15 Properties of the likelihood function Likelihood function = density: L(δ; x , n) = nx p x (1 − p)n−x , p = fpsy (δ) The guessing probability is 1/3 They obtain 10 correct answers to 20 samples Measures support of values of δ relative to δ̂ Peter analyzes the probability of a correct answer, pc : p̂c = 1/2(0.11) and CI95% = [0.28; 0.72] which covers pc = 1/3 An objective way to measure information in the data about δ Sally analyzes the Thurstonian δ: δ̂ = 1.47(0.59) and CI95% = [0.32; 2.62] (Bi et al, 1997) which does NOT cover δ = 0 duo−trio: x= 13 n= 20 1.0 Should we trust Peter or Sally? The maximum likelihood estimate (MLE) 0.8 Relative likelihood Which method is (most) correct? 0.6 Likelihood CIs are given by horizontal lines 0.4 Symmetric approximation produces standard (Wald) CIs proposed by Bi et al (1997) 0.2 How much evidence is there really in the data about a difference between the products? Likelihood based confidence intervals Sensometrics Summer School The likelihood function and confidence intervals Peter and Sally’s triangle experiment © Christine Borgen Linander (DTU) Likelihood based confidence intervals Sensometrics Summer School 7 / 15 0.0 0 1 ^ δ 2 3 4 δ © Christine Borgen Linander (DTU) Likelihood based confidence intervals Sensometrics Summer School 8 / 15 The likelihood function and confidence intervals The likelihood function and confidence intervals Properties of the likelihood function Properties of the likelihood function Likelihood function = density: L(δ; x , n) = nx p x (1 − p)n−x , p = fpsy (δ) Likelihood function = density: L(δ; x , n) = nx p x (1 − p)n−x , p = fpsy (δ) Measures support of values of δ relative to δ̂ Measures support of values of δ relative to δ̂ An objective way to measure information in the data about δ An objective way to measure information in the data about δ duo−trio: x= 13 n= 20 duo−trio: x= 13 n= 20 1.0 1.0 The maximum likelihood estimate (MLE) 0.6 Likelihood CIs are given by horizontal lines 0.4 0.2 Symmetric approximation produces standard (Wald) CIs proposed by Bi et al (1997) 95% limit 99% limit 0.0 0 1 2 3 0.6 Likelihood CIs are given by horizontal lines 0.4 0.2 Symmetric approximation produces standard (Wald) CIs proposed by Bi et al (1997) 95% limit 99% limit 0.0 4 0 1 2 δ 3 4 δ © Christine Borgen Linander (DTU) Likelihood based confidence intervals Sensometrics Summer School 8 / 15 © Christine Borgen Linander (DTU) Likelihood based confidence intervals Examples revisited Sensometrics Summer School John and Dorothy’s duo-trio example revisited (2) “No difference” between the products has reasonably high likelihood The symmetric approximations are inaccurate An intermediate difference between products is most likely Neither John’s nor Dorothy’s CIs are appropriate A large difference between products is unlikely Likelihood inference for δ and pc is compatible duo−trio: x= 13 n= 20 duo−trio: x= 13 n= 20 0.8 0.8 0.8 0.4 0.2 95% limit Relative likelihood 1.0 Relative likelihood 1.0 0.6 0.4 0.2 0.0 1 2 3 4 0.4 95% limit 0.5 0.6 0.7 Likelihood based confidence intervals 0.8 pc 0.9 John 0.8 0.6 0.4 0.2 95% limit 99% limit 99% limit 0.0 δ © Christine Borgen Linander (DTU) 0.6 99% limit 0.0 0 Dorothy 0.2 95% limit 99% limit duo−trio: x= 13 n= 20 1.0 Relative likelihood duo−trio: x= 13 n= 20 1.0 0.6 8 / 15 Examples revisited John and Dorothy’s duo-trio example revisited Relative likelihood The maximum likelihood estimate (MLE) Dorothy 0.8 Relative likelihood Relative likelihood 0.8 1.0 0.0 0 1 2 3 4 0.5 0.6 0.7 δ Sensometrics Summer School 9 / 15 © Christine Borgen Linander (DTU) Likelihood based confidence intervals 0.8 pc 0.9 1.0 Sensometrics Summer School 10 / 15 Examples revisited Examples revisited Peter and Sally’s triangle example revisited (1) Peter and Sally’s triangle example revisited (2) “No difference” between the products has reasonably high likelihood The likelihood curve tells the full story about the data An intermediate difference between products is most likely The likelihood curve illustrates the effect of confidence level A large difference between products is unlikely triangle: x= 10 n= 20 1.0 0.8 0.8 0.6 0.4 0.2 0.6 0.4 1 2 3 0.8 0.6 0.4 0.0 99% limit 4 95% limit 0.5 0.6 δ 0.7 0.8 0.9 0.4 0.2 95% limit 99% limit 0.0 0 0.4 0.6 99% limit 0.0 0 0.8 95% limit 99% limit 0.0 1.0 0.2 0.2 95% limit triangle: x= 10 n= 20 1.0 Relative likelihood 1.0 Relative likelihood triangle: x= 10 n= 20 Relative likelihood Relative likelihood triangle: x= 10 n= 20 1 2 3 4 0.4 0.5 0.6 δ 1.0 0.7 pc 0.8 0.9 1.0 pc © Christine Borgen Linander (DTU) Likelihood based confidence intervals Sensometrics Summer School 11 / 15 © Christine Borgen Linander (DTU) Examples revisited Likelihood based confidence intervals Sensometrics Summer School 12 / 15 Perspectives Peter and Sally’s triangle example revisited (3) Coverage probability The symmetric approximation for δ is quite inaccurate Sally’s CI is very misleading Boyles (2008) showed that likelihood CIs have the best coverage probability among common CIs for the binomial p. triangle: x= 10 n= 20 triangle: x= 10 n= 20 1.0 Sally 0.6 0.4 0.2 95% limit Peter 0.8 Relative likelihood 0.8 Relative likelihood Coverage probability in % for the binomial p with a nominal level of 95% (Boyles, 2008) n Standard Exact Likelihood 10 76.9 98.4 94.9 50 90.1 96.9 95.0 100 92.2 96.5 95.0 500 94.3 95.7 95.0 1.0 0.6 0.4 0.2 95% limit 99% limit 0.0 99% limit 0.0 0 1 2 3 4 0.4 0.5 0.6 δ © Christine Borgen Linander (DTU) Likelihood based confidence intervals 0.7 pc 0.8 0.9 1.0 Sensometrics Summer School 13 / 15 © Christine Borgen Linander (DTU) Likelihood based confidence intervals Sensometrics Summer School 14 / 15 Perspectives Likelihood methods in discrimination testing Likelihood — a common framework for: Estimation Testing Confidence intervals Maximum likelihood Likelihood ratio test Profile likelihood Gracefully handle boundary cases Likelihood methods extend to complex situations © Christine Borgen Linander (DTU) Likelihood based confidence intervals Sensometrics Summer School 15 / 15