Download Statistical considerations in small proof-of

Statistical considerations in small
proof-of-concept trials, including
crossover designs
Stephen Senn
(c) Stephen Senn 2008
1
• People look down on
marketing men
• It’s not true that they are
not scientists
• They work in sell
biology
• I would like to take this
opportunity to draw your
attention to a book I
rather like
• In
fact I wrote it myself
(c) Stephen Senn 2008
2
Outline
•
•
•
•
•
Decision analysis and proof of concept
Value of information perspective
Place of cross-over trials
Carry-over
The potential for cross-over trials in
studying individual response
(c) Stephen Senn 2008
3



A Model
Probability proof of concept (POC) study successful
Probability proof of efficacy study (POE) successful if POC successful
Probability POE study successful if POC unsuccessful
    1   
Probability POE study successful
CC
Cost of POC including any lost sales through extra delay
CE
Cost of POE study
R
fE
Expected NPV revenue if POE initiated immediately and successful
fC
Value of strategy of POC + POE
Value of strategy of POE only
V  fC  f E
Value of POC study
(c) Stephen Senn 2008
4
Model Continued
 
f E  max 0,   1     R  CE

 
fC  max 0,  max 0,  R  CE    1    max 0,  R  CE    CC
(c) Stephen Senn 2008
5

Example
R  100, CC  5, CE  25,   0.3,   0.25,  
Expected return on two strategies

  
1
0.65
Value of two strategies plotted
against , the probability POE
successful if POC successful
5
0
0.5
1
Prob POE successful if POC success
direct POE
initial POC
(c) Stephen Senn 2008
6
Value of an initial POC trial
10
0.3
0.65
5
0
0.2
0.4
0.6
0.8
5
(c) Stephen Senn 2008
7
Value of Biomarker Information
in Terms of Posterior Variance
• Suppose that over all products for this
indication the correlation of true
therapeutic and biomarker outcomes is 0.9
• Let the prior means be zero in this class
• Let the prior variances be 1
• Let the data variance of a minimal
experiment be also 1
– Implies prior information equivalent to one
minimal experiment
(c) Stephen Senn 2008
8
Posterior variances based on proof of concept trial
Here n is the number of
minimal experiments we
run
0.6
0.5
Of course we expect a
biomarker experiment to
be cheaper than a
therapeutic one
0.4
0.3
Nevertheless note that
fairly rapidly there is no
interest in getting further
biomarker information
0.2
0.1
0.0
0
10
20
30
40
50
n
simulated therapeutic
simulated biomarker
theory therapeutic
theory biomarker
9
A Serious Warning to
Statisticians
In the mathematical formulation of any problem it is necessary to base
oneself on some appropriate idealizations and simplification. This is,
however, a disadvantage; it is a distorting factor which one should always
try to keep in check, and to approach circumspectly. It is unfortunate that
the reverse often happens. One loses sight of the original nature of the
problem, falls in love with the idealization, and then blames reality for not
conforming to it.
De Finetti 1975
‘The only way that human beings could ever have survived as a species
for long as we have is that we’ve developed another kind of decisionmaking apparatus that’s capable of making very quick judgements based
on very little information.
Malcolm Gladwell, Blink, 2005
(c) Stephen Senn 2008
10
My Gloomy Take on This
• We don’t really understand this topic
• There may be less value in proof of
concept studies than we propose
• Therapeutic studies may be valuable even
if they have low power
• There is no point in undertaking POC
studies unless you can see circumstance
under which they would cause you to
cancel projects
(c) Stephen Senn 2008
11
Appropriate Attitudes for Crossover Trials
• They are not suitable for all indications and
questions
• They are extremely valuable for some
indications and questions
• Carry-over has to be dealt with by washout
• Don’t pre-test for carry-over
• Don’t rely on classical statistical approaches to
carry-over
• Cross-over trials have great potential in
investigating individual response
(c) Stephen Senn 2008
12
Carry-over
Definition: Carry-over is the persistence (whether physically or
in terms of effect) of a treatment applied in one period in
a subsequent period of treatment.
If carry-over applies in a cross-over trial we shall, at some
stage, observe the simultaneous effects of two or more
treatments on given patients.
We may, however, not be aware that this is what we are
observing and this ignorance may lead us to make errors in
interpretation.
(c) Stephen Senn 2008
13
The simple carry-over model.
This is a very popular model amongst “applied” statisticians of
a mathematical bent.
The model assumes that if a carry-over effect is present
1) it lasts for one period exactly
2) it depends on the engendering treatment only and not on the
perturbed treatment.
(c) Stephen Senn 2008
14
Three Period Bioequivalence
Designs
• Three formulation designs in six sequences
common.
• Subjects randomised in equal numbers to six
possible sequences.
For example, 18 subjects, three on each of the
sequences ABC, ACB, BAC, BCA, CAB, CBA.
– A = test formulation under fasting conditions,
– B = test formulation under fed conditions
– C = reference formulation under fed conditions.
–
(c) Stephen Senn 2008
15
Weights for the Three Period Design:
not Adjusting for Carry-over
Period
Sequence
1
2
3
ABC
A
0
B
1/6
C
-1/6
ACB
A
0
C
-1/6
B
1/6
BAC
B
1/6
A
0
C
-1/6
BCA
B
1/6
C
-1/6
A
0
CBA
C
-1/6
A
0
B
1/6
CAB
C
-1/6
B
1/6
A
0
(c) Stephen Senn 2008
16
Properties of these weights
• Sum to 0 in any column,
– eliminates the period effect.
• Sum to 0 in any row
– eliminates patient effect
• Sum to 0 over cells labelled A
– A has no part in definition of contrast
• Sum to 1 over the cells labelled B and to -1
over the cells labelled C
– Estimate contrast B-C
(c) Stephen Senn 2008
17
Weights for the Three Period Design:
Adjusting for Carry-over
Period
Sequence
1
2
3
ABC
A
-1/24
Ba
4/24
Cb
-3/24
ACB
A
1/24
Ca
-4/24
Bc
3/24
BAC
B
4/24
Ab
2/24
Ca
-6/24
BCA
B
5/24
Cb
-2/24
Ac
-3/24
CBA
C
-4/24
Ac
-2/24
Ba
6/24
CAB
C
-5/24
Bc
2/24
Ab
3/24
B-C contrast: illustration of treatment effect and elimination of
period and patient effects
(c) Stephen Senn 2008
18
Weights for the Three Period Design:
Adjusting for Carry-over
Period
Sequence
1
2
3
ABC
A
-1/24
Ba
4/24
Cb
-3/24
ACB
A
1/24
Ca
-4/24
Bc
3/24
BAC
B
4/24
Ab
2/24
Ca
-6/24
BCA
B
5/24
Cb
-2/24
Ac
-3/24
CBA
C
-4/24
Ac
-2/24
Ba
6/24
CAB
C
-5/24
Bc
2/24
Ab
3/24
Illustration of elimination of ‘carry-over’ effects
(c) Stephen Senn 2008
19
Have We Got Something for
Nothing?
• Sum of squares weights of first scheme
is 1/3 (or 4/12)
• Sum of squares of weights of second
scheme is 5/12
• Given independent homoscedastic
within- patient errors, there is thus a
25% increase in variance
• Penalty for adjusting is loss of efficiency
(c) Stephen Senn 2008
20
The difference between
mathematical and applied
statistics is that the former is full
of lemmas whereas the latter is
full of dilemmas
(c) Stephen Senn 2008
21
The Dangers of Pre-testing
• Situation with AB/BA design
– Two-stage procedure is very badly biased
– CARRY and PAR are highly correlated
• 1/2 <  < 1
• Three treatment design
– Same problem
– Carry-over and adjusted estimates correlated
•  = 0.45
(c) Stephen Senn 2008
22
The Phoenix Bioequivalence
Trials
•
•
•
•
•
•
Analysed by D’Angelo, Potvin & Turgeon *
20 drug classes
1989-1999
12 or more subjects
96 three period designs
324 two period designs
D'Angelo, G.Potvin, D.Turgeon, J. J Biopharm Stats, 11, 27-36, 2001
(c) Stephen Senn 2008
23
Three Treatment Designs
P-Values for Carry-Over
AUC
0 : 115567899
1 : 01458999
2 : 01225568999
3 : 011335577
4 : 24688
5 : 35667788
6 : 00336667888
7 : 14444566999
8 : 011233468888
9 : 13335667899
Cmax
0 : 223557888
1 : 4677799
2 : 000124566899
3 : 011124689
4 : 01223455799
5 : 00045599
6 : 000166667778
7 : 0345566779
8 : 2345779
9 : 13444556889
“Significant” results in bold
Senn, S. J., G. D'Angelo, et al. (2004). "Carry-over in cross-over trials in
bioequivalence: theoretical concerns and empirical evidence."
Pharmaceutical Statistics 3(2): 133-142.
24
Two Treatment Designs
AUC
0 : 00111111222222234444
0 : 5666777777789999
1 : 00000112222223333
1 : 5556667777899999
2 : 0011112223344444
2 : 555666788899999
3 : 00001112233344
3 : 5556666666777778888899999
4 : 001111112222223334
4 : 55666666777777788999
5 : 00000111222333344444
5 : 566677888899
6 : 000001134
6 : 55666667777888889999
7 : 111233333344
7 : 555556777888899
8 : 0000112234444
8 : 55666778888999
9 : 00011112233334444
9 : 555567777788999
Cmax
0 : 00122222344
0 : 55555556666677999999
1 : 0001122233333344444444
1 : 55566667778888899
2 : 00011111122344
2 : 566667788889999
3 : 111112222233444444
3 : 555566666777778888999
4 : 000001112222333334444
4 : 5557888889999
5 : 00001122233
5 : 5555666678999
6 : 0000111222233334
6 : 55555566677788889999
7 : 000000112223344
7 : 6666777777889
8 : 0122233444
8 : 55666677888899
9 : 1111111222333444
9 : 555555556666677778889999
“Significant” results in bold
25
Test of Uniformity of P-Values
Study
Design
Variable
Total
number
of studies
KS
statistic
pvalue*
2-way
AUC0-t
Cmax
324
324
0.0645
0.0496
0.1354
0.4040
3-way
AUC0-t
Cmax
96
96
0.1048
0.0542
0.2424
0.9407
*
H0: true cdf U[0,1] vs. H1: true cdf NOT U[0,1]
(c) Stephen Senn 2008
26
Galling as this may appear to
statisticians, the cure for carryover is more biological and
pharmacological understanding
not more statistics
(c) Stephen Senn 2008
27
Conclusions
• Distribution of P-values uniform
– no evidence of carry-over
• Carry-over a priori implausible
– presence testable by assay
• No point is testing for it
– leads to bias
• Or adjusting for it
– increased variance
(c) Stephen Senn 2008
28
Possible Strategy
• Run multi-period cross-overs
• Patient by treatment interaction becomes
identifiable
• This provides an upper bound for gene by
treatment interaction
– Because patients differ by more than their
genes
(c) Stephen Senn 2008
29
Second cross-over
Responders NonTotal
Responders
First cross- Responders 24
over
(c) Stephen Senn 2008
0
24
Non0
Responders
8
8
Total
8
32
24
30
Second cross-over
Responders NonTotal
Responders
First cross- Responders 18
over
(c) Stephen Senn 2008
6
24
Non6
Responders
2
8
Total
8
32
24
31
Advantages and Disadvantages
PRO
• Cheap
• Low tech
• Insight into sources
of variation gained
(c) Stephen Senn 2008
CON
• Only suitable for
chronic diseases
• Demanding of
patient’s time
• Unglamorous
• Does not produce
diagnostic patents
32
An Overlooked Source of Genetic
Variability
• Humans may be classified into two important
genetic subtypes.
• One of these suffers from a massive
chromosomal deficiency.
• This is expressed in.
– Important phenotypic differences.
– A massive disadvantage in life expectancy.
• Many treatment strategies take no account of
this.
• The names of these subtypes are...
(c) Stephen Senn 2008
33
Men and Women
(c) Stephen Senn 2008
34
A Difficult
Decision
• You have $100
• Should you spend it
on beer?
– US 20 beers
– UK 15 beers
• Or on books?
• In particular 1 book
• Have I mentioned this
before?
(c) Stephen Senn 2008
35