Download Comparison of outcomes in cancer patients treated within and

ARTICLES
Articles
Comparison of outcomes in cancer patients treated within and
outside clinical trials: conceptual framework and structured
review
Jeffrey M Peppercorn, Jane C Weeks, E Francis Cook, Steven Joffe
Summary
Background Many oncologists believe that patients with
cancer who enrol in clinical trials have better outcomes than
those who do not enrol. We aimed to assess the empirical
evidence that such a trial effect exists.
Methods We developed a conceptual framework for
comparison of trial and non-trial patients. We then did a
comprehensive literature search to identify studies that
compared outcomes between these groups. We critically
evaluated these studies to assess whether they provide valid
and generalisable support for a trial effect.
Findings We identified 26 comparisons, from 24 published
articles, of outcomes among cancer patients enrolled and
not enrolled in clinical trials. 21 comparisons used
retrospective cohort designs. 14 comparisons provided some
evidence that patients enrolled in trials have improved
outcomes. However, strategies to control for potential
confounding factors were inconsistent and frequently
inadequate. Only eight comparisons restricted non-trial
patients to those meeting trial eligibility criteria. Of these,
three noted better outcomes in trial patients than in non-trial
patients. Children with cancer, patients with haematological
malignant disease, and patients treated before 1986 were
disproportionately represented in positive studies.
Interpretation Despite widespread belief that enrolment in
clinical trials leads to improved outcomes in patients with
cancer, there are insufficient data to conclude that such a
trial effect exists. Until such data are available, patients with
cancer should be encouraged to enrol in clinical trials on the
basis of trials’ unquestioned role in improving treatment for
future patients.
Lancet 2004; 363: 263–70
Departments of Medical Oncology (J M Peppercorn MD,
J C Weeks MD), and Pediatric Oncology (S Joffe MD), Dana-Farber
Cancer Institute, Boston, MA, USA; Department of Medicine,
Brigham and Women’s Hospital, Boston (J M Peppercorn,
J C Weeks, Prof E F Cook ScD); Department of Medicine, Children’s
Hospital, Boston (S Joffe); and Harvard School of Public Health,
Boston (E F Cook)
Correspondence to: Dr Steven Joffe, Dana-Farber Cancer Institute,
44 Binney Street, Boston, MA 02115, USA
(e-mail: [email protected])
THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com
Introduction
The belief that clinical trials offer the best treatment for
patients with cancer is widespread in the oncology
community. This claim, motivated partly by aims to
increase accrual1–3 and ensure third-party payment,4,5
appears frequently in pronouncements by professional
organisations and leaders. For example, the American
Federation of Clinical Oncologic Societies maintains that
“treatment in a clinical trial is often a cancer patient’s best
option”.6 Other people argue that “clinical trials are
proven to offer children the best chance of survival”,7 and
that trial access is one of the “basic requirements of
quality cancer care.”5 Such claims suggest that trials are
viewed not only as a way to improve future treatment, but
also as the best treatment for current patients.
The view that trials lead to better outcomes, if correct,
has important implications. First, that more than 95% of
adults and perhaps 40% of children with cancer do not
enrol in trials would constitute evidence of substandard
care. Second, the suggestion that patients benefit directly
by becoming research participants changes the traditional
model of human experimentation. If so, clinicians
arguably should advocate forcefully for enrolment on
grounds of direct benefit, rather than presenting the risks
and benefits for patients to weigh. In the conventional
view, such advocacy might be criticised as misleading or
coercive. Third, acceptance of this view might require
substantial changes in trial financing and organisation,
eligibility criteria, and patient selection. Anything that
might constitute a barrier to participation (including
considerations of scientific validity and integrity8) would
be suspect. We must therefore be confident that trial
participation improves outcomes before using the claim to
inform practice or policy.
Ideally, the statement that trials are the best treatment
option should rest on evidence that trial participants have
better outcomes than similar patients treated off-protocol.
Several studies9–22 have shown such a trial effect, also
sometimes known as an inclusion benefit.23 However,
showing a causal relation between trial participation and
improved outcome is difficult.
Here, we seek to develop a conceptual framework for
assessing the trial effect; describe the methodological
challenges in studying this effect and the hierarchy of
evidence that could be used to support its existence; and
use these insights to assess systematically the quality,
validity, and generalisability of the published work.
Methods
We sought to identify articles that presented primary data
comparing outcomes between trial and non-trial patients
with cancer. As others note,24 there is no obvious set
of terms to capture all relevant reports. We therefore
searched MEDLINE using the terms trial effect, inclusion
benefit, population outcomes, community outcomes, trial
benefit, patient preference trial, and comprehensive
263
For personal use. Only reproduce with permission from The Lancet publishing Group.
ARTICLES
Ref
Pop
Dates
Type Enrolled* Eligible Treatment Potential confounders and methods of control‡
controls similar†
Accounted for
Not accounted for
9
Multiple 1979–85 NE
myeloma
405/164 NA
No
34
High-grade 1983–87 ER
Glioma
55/23
No
29
Trial 2 Localised 1983–89 ER
breast
Trial 3 Localised 1983–89 ER
breast
Yes
473/247 Yes
Yes
199/129 Yes
Yes
No
Menopause, number of
positive axillary nodes,
tumour size, tumour grade,
hormone receptor status
(NBD)
Menopause, number of
positive axillary nodes,
tumour size, tumour grade,
hormone receptor status
(NBD)
Age, sex, stage, histology,
tumour size, TN group,
radiation therapy (MVA)
13
Stage I
NSCLC
1977–79 RC
78/471
33
Gastric
1976–80 RC
217/493 Yes
Yes
Sex, clinical stage (NBD)
15
AML
1975–82 RC
46/84
No
No
WBC, LDH, chemotherapy
dose, platelet count, PS,
receipt of antibiotics,
preleukaemia, fever (MVA)
16
Localised 1973–80 RC
breast
(T)
1980–84
(C)
Hodgkin’s 1978–84 RC
disease
Advanced 1982–88 RC
testicular (T)
1978–84
(C)
352/
1408
No
Yes
Age, tumour size, axillary
nodes, tumour site,
histology (MVA)
Treatment centre (NBD)
Age (MVA)
Sex (NBD)
Extent of disease (SgpA)
18
17
36
SCLC
32
1986–89 RC
Yes
Sex, period of diagnosis,
follow-up year (MVA)
Age, sex, period of
diagnosis (ES)
Age, histology, treatment
centre (NBD)
1106/
No
4807
133/172 No
73/37
No
No
Newly diagnosed vs
relapsed disease, histology,
receipt of chemotherapy (RC)
Age (NBD)
Age, PS, extent of disease
(MVA)
Sex, weight loss, fever, SVC
syndrome, paraneoplastic
syndrome, chest pain, liver/
bone/bone marrow
metastases, treatment
centre (NBD)
T status, N status, hormone
receptor status, hormone
therapy, treatment centre
(MVA)
Stage, type of surgery (StrA)
Age, treatment centre (NBD)
Trial effect observed§
Unadjusted Adjusted
Treatment centre (BD)
SES, PS, comorbidity, stage
(NE)
Yes¶
Yes
PS (BD)
Sex, SES, comorbidity (NE)
No
Not done
Treatment centre (BD)
Age, SES, PS, comorbidity
(NE)
No
Not done
Treatment centre (BD)
Age, SES, PS, comorbidity
(NE)
No
Not done
Treatment centre, county of
residence (BD)
PS, comorbidity, complete
staging, SES (NE)
Age, stage, symptom duration,
tumour site, number, size,
type of surgery, treatment
centre (BD)
SES, PS, comorbidity (NE)
% blasts, other
laboratory studies (NBD)
Age, sex, histological subtype,
year of treatment, treatment
centre (BD)
SES, comorbidity (NE)
Temporal trends, multifocality,
adjuvant chemotherapy (BD)
SES, PS, comorbidity (NE)
Yes
Yes
No
Not done
Yes
Yes
Yes
Mixed||
PS, comorbidity, SES, stage,
treatment centre (NE)
Treatment centre, temporal
trends (BD)
SES, PS, comorbidity (NE)
Not reported Mixed**
Mixed
Mixed††
Brain metastases,
hemoptysis (BD)
SES, comorbidity (NE)
Yes
No
Age, SES, PS, comorbidity,
radiation therapy (NE)
Yes
No
No
No
Localised 1980–90 RC
breast
160/519 No
No
30
Rectal
1987–90 RC
557/798 Yes
Yes
37
SCLC
1987–92 RC
41/40
No
Yes
10
Adult and
paediatric
Hodgkin’s
disease
1988–94 RC
(T)
1969–94
(C)
62/163
No
No
31
Prostate
1990–94 RC
(T)
1994–98
(C)
80/132
No
Yes
PS (MVA)
Age, years since diagnosis
(NBD)
Baseline PSA, lymph node
Mixed***
involvement, hormone
therapy, treatment centre (BD)
SES, comorbidity (NE)
No
38
SCLC
1994–98 RC
60/46
No
Yes
Age, sex, PS, stage,
alkaline phosphatase,
serum sodium, LDH,
respiratory score, site of
tumour, treatment centre
(NBD)
SES, comorbidity (NE)
Not done
Sex, tumour height, radiation No
therapy (BD)
SES, PS, comorbidity (NE)
Age, sex, SES (insurance
PS, previous cancer, weight
Mixed§§
status), stage, LDH (NBD) loss, treatment centre (BD)
Comorbidity (NE)
Age, sex, stage, risk factors, Year of diagnosis, staging
Mixed¶¶
B symptoms, treatment
methods (BD)
mode (MVA)
PS, SES, comorbidity (NE)
Treatment centre (NBD)
No
No‡‡
Not done
No||||
(continues next page)
264
THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com
For personal use. Only reproduce with permission from The Lancet publishing Group.
ARTICLES
(continued from previous page)
Ref
Pop
Dates
Type Enrolled* Eligible Treatment Potential confounders and methods of control‡
controls similar†
Accounted for
Not accounted for
22
12
Adolescent 1984–94
and young
adult
ALL
Adolescent 1984–94
and young
adult
AML
Osteo1982–84
sarcoma
Wilms’
1970–73
tumour
ALL
1970–75
14
ALL
1971–82 RC
2137/
933
No
No
19
AML
1975–88 RC
369/449 No
No
20
NHL
120/42
No
No
21
ALL
1976–91 RC
(T)
1981–91
(C)
1980–94 RC
3759/
1229
No
No
22
35
11
Trial effect observed§
Unadjusted Adjusted
RC
154/263 No
No
Period of diagnosis (NBD)
Age, teaching hospital,
No
hospital volume (BD)
Sex, SES, PS, comorbidity (NE)
Not done
RC
180/282 No
No
Period of diagnosis (SgpA)
Age, teaching hospital,
hospital volume (NBD)
Sex, SES, PS, comorbidity (NE) Yes
Mixed†††
EF.
36/77
Yes
Yes
SES, PS, comorbidity (NE)
Not done
RC
98/104
Yes
No
RC
257/70
No
No
Age, sex, tumour site,
treatment centre (NBD)
Age, stage (StrA)
Treatment centre (SgpA)
Sex, WBC, treatment
centre (MVA)
Age, race, survival for at
least 28 days (RC)
Age, period of diagnosis,
treatment centre (StrA)
Sex, WBC (NBD)
Age (StrA)
Period of treatment,
teaching hospital (SgpA)
Sex (NBD)
No
Radiation dose (BD)
Yes
Sex, SES, PS, comorbidity (NE)
SES, PS, comorbidity (NE)
Yes
Yes
SES, PS, comorbidity (NE)
Yes
Yes
SES, PS, comorbidity,
WBC (NE)
Not reported Yes‡‡‡
Location of initial care (BD)
Mixed§§§
Age, sex, SES, PS, comorbidity
stage, treatment centre (NE)
Period of diagnosis (SgpA)
Age, sex, WBC, immunophenotype, Down’s
syndrome (StrA)
Hospital volume, UKCCSG
membership (BD)
SES, PS, comorbidity (NE)
Yes
Not done
Not reported Mixed¶¶¶
ALL=acute lymphoblastic leukaemia. AML=acute myeloid leukaemia. NHL=non-Hodgkin lymphoma. NSCLC=non-small-cell lung cancer. SCLC=small-cell lung cancer.
NA=not applicable. NE=natural experiment. ER=eligible refuser. RC=retrospective cohort. MVA=multivariable analysis. ES=external standardisation. SgpA=subgroup
analysis. StrA=stratified analysis. RC=restriction of cohort. NBD=no baseline difference. BD=baseline difference recorded. NE=not evaluated. SES=socioeconomic
status. PS=performance status. WBC=white blood cell count. PSA=prostate specific antigen. LDH=lactate dehydrogenase. TN=tumour-node group. SVC=superior vena
cava. *Values are number in trial/number in control or non-trial group. †For randomised controlled trials, refers to similarity between treatment offered on the control
group and that received by non-trial participants. ‡We assessed whether each study attempted to account for possible confounding by age, sex (where applicable), PS,
comorbidity, SES, stage (where applicable), and treatment centre. We also assessed other potential confounding factors, as appropriate to individual studies.
§p<0·05, unless otherwise noted. ¶Both unadjusted and adjusted analyses used survival rates, relative to the expected age-specific, sex-specific, and period-specific
survival in the general population, as the outcome of interest. ||No difference in overall survival or distant recurrence was detected in adjusted analyses; local
recurrence rates were significantly lower in trial patients than in non-trial patients. **Qualitatively, trial effect seen in patients 45 years or older only; authors did not
present results of significance tests. ††In patients with minimum/moderate disease, trial participants had improved survival in all analyses. In patients with advanced
disease, survival was better in trial participants than in non-trial patients (p=0·056 in unadjusted analysis). However, this advantage disappeared after patients with
relapsed disease were eliminated from the trial cohort, and after adjustment was made for possible misclassification bias in staging of non-trial patients (Will-Rogers
phenomenon). ‡‡Trial group includes patients from the control group of the randomised controlled trial only. Participants in the experimental group of the trial had
better outcomes than those on the control group. Therefore, the authors might have recorded a trial effect (due to receipt of experimental therapy) had they included all
randomised controlled trial participants in the trial group. §§Trial patients had improved survival and disease-specific survival, but not disease-free survival (p=0·06),
when compared with non-trial patients. ¶¶In unadjusted analyses, a difference favouring trial patients was apparent in disease-free but not overall survival.
||||In the multivariable model evaluating predictors of disease-free survival, a trend towards improved outcomes in trial participants was noted (p=0·064).
When treatment modality was omitted as a covariate from the multivariable model, disease-free survival was significantly improved in trial patients (p=0·038).
***Unadjusted analysis showed improved survival, but no improvement in PSA response, in trial patients. †††Subgroup analyses showed that there was no difference
between trial and non-trial patients during 1984–88, but trial patients had better outcomes than non-trial patients during 1989–94. ‡‡‡Trial effect was restricted to
patients treated in a teaching hospital between 1975–83. All analyses adjusted for age. §§§One trial group (Pediatric Oncology Group) had better outcomes than the
non-trial group, whereas the other trial group (Swiss Pediatric Oncology Group) did not. ¶¶¶An apparent trial effect was recorded in patients diagnosed in 1985–89 and
1990–94, but not in those diagnosed in 1980–84. The difference between the trial and non-trial groups was no longer significant when patients who died in the first
4 weeks after diagnosis, most of whom were not enrolled in the trials, were excluded from the analysis.
Table 1: Studies comparing cancer outcomes within and outside a trial
cohort trial, cross-referenced with cancer, oncology,
neoplasms, and clinical trials. We also scanned an online
annotated bibliography maintained by researchers at
McMaster University25 and examined the reference lists
of the empirical studies we identified, of two previous
reviews, and of position papers arguing that trial
enrolment is beneficial. We used Science Citation Index
Expanded (ISI, Philadelphia, PA) to locate subsequent
reports that cited the manuscripts identified above.
Two authors (JP, SJ) independently reviewed all
articles identified in our search. We included articles
if they involved cancer-directed treatment, and if the
authors claimed to provide a valid comparison of
outcomes between trial and non-trial patients. We
excluded articles in which the main purpose was to
show the non-representativeness of clinical trial
participants (including two included in previous
THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com
reviews),26,27 unless the authors also provided an analysis
they claimed accounted for baseline differences. We
also omitted one article included in previous reviews
that investigated treatment by a specialist rather than
trial participation.28 We excluded studies of centre
effects, guideline effects, and other elements of
specialised care that did not assess trial entry as an
independent variable.
Using forms that we pilot-tested with four nononcology reports, we recorded study dates, sample sizes,
study design, whether controls were restricted to trialeligible individuals, strategies used to control for
confounding, potential sources of bias, and major
outcomes. Except for one study that reported results
graphically but not statistically,18 we classified studies as
showing a trial effect if outcomes in trial patients were
better with p<0·05. We noted whether studies attempted
265
For personal use. Only reproduce with permission from The Lancet publishing Group.
ARTICLES
Studies (n=26)
Design of trial versus non-trial comparison
Randomised controlled
Natural experiment
Eligible refuser
Prospective cohort
Retrospective cohort
0
1
4
0
21
Type of clinical trial in which patients were participating
Randomised only
20
Other*
6
All patients treated before 1986
Yes
No
10
16
Age-group of patients
Children†
Adult‡
9
17
Type of malignant disease
Haematological
Other
11
15
Baseline differences accounted for§
Age
Sex¶
Performance status
Comorbidity
Socioeconomic status||
Disease-specific prognostic factors**
Treatment centre
19
13
4
0
1
19
14
Type of analysis
Unadjusted only
Adjusted only
Both adjusted and unadjusted
9
3
14
Non-trial patients restricted to those meeting
trial eligibility criteria††
Yes‡‡
No
8
17
*Two studies considered single-group trials only, four considered both singlegroup trials and randomised controlled trials, and type of trial could not be
determined for one study. †Includes two studies involving adolescents and
young adults.22 ‡Includes one study that involved a small proportion of
children.10 §Includes studies that showed no baseline differences between
groups, as well as those that adjusted for observed differences in the analysis.
Nine studies used multivariable techniques to adjust for age, five for sex, three
for performance status, none for socioeconomic status, seven for diseasespecific prognostic factors, and two for treatment centre. Among 17 studies
that did some type of adjusted analysis, the median number of covariates was
4 (IQR 3–6). ¶Denominator includes 20 studies involving cancers that affect
both sexes. ||Insurance status. **Studies that accounted for at least one
disease-specific prognostic factor. Includes stage, where applicable.
††n=25. Excludes one study9 for which restriction to trial-eligible patients was
not applicable. This study compared outcomes among all patients with multiple
myeloma (both trial-eligible and not trial-eligible) between two adjacent
geographical regions, only one of which participated in clinical trials.
‡‡Includes all four studies using eligible refuser designs,29,34,35 and four of
21 retrospective cohort analyses.11,13,30,33
Table 2: Characteristics of studies comparing outcomes in trial
and non-trial patients
to address potential selection differences by age, sex
(if applicable), performance status, comorbidity, stage
(if applicable), socioeconomic status, and treatment
centre. In addition, for each study we recorded other
potential factors that might have affected outcomes, and
whether the analysis attempted to address them. Finally,
we reconciled the two reviews. A third author (JCW)
mediated disagreements. We present descriptive data
only. Because of concerns about systematic biases in the
published work due to inadequate control of selection
factors, we did not undertake a formal meta-analysis.
Studies suggesting a trial effect*
Age-group
Paediatric (n=9)
Adult (n=17)
All patients treated before 1986
Yes (n=10)
No (n=16)
Type of malignant disease
Haematological (n=11)
Other (n=15)
Type of study
Natural experiment (n=1)
Eligible refuser (n=4)
Retrospective cohort (n=21)
All studies (n=26)
7†
7‡
8§
6¶
9||
5**
1
0
13††
14††
*Where studies reported both unadjusted and adjusted analyses, table presents
results of adjusted analyses. †Three studies found evidence for a trial effect in
selected subgroups. ‡Two studies found evidence for a trial effect in selected
subgroups, and two found evidence for a trial effect with respect to selected
endpoints. §One study found evidence for a trial effect in selected subgroups,
and one found evidence for a trial effect with respect to selected endpoints.
¶Four studies found evidence for a trial effect in selected subgroups, and one
found evidence for a trial effect with respect to selected endpoints. ||Four
studies found evidence for a trial effect in selected subgroups. **One study
found evidence for a trial effect in to selected subgroups, and 2 found evidence
for a trial effect with respect to selected endpoints. ††Five studies found
evidence for a trial effect in selected subgroups, and two found evidence for a
trial effect with respect to selected endpoints
Table 3: Relations between characteristics of studies and trial
effects (n=26)
summarises these studies, arranged by population, study
design, and dates of the primary data reported in the
reports. Additional detail is available from the authors.
Study characteristics
Table 2 presents characteristics of the studies. Most
(81%) were retrospective cohorts, and most (77%)
compared non-trial patients with those enrolled in
randomised rather than single-group trials. In 38% of
comparisons, all patients had been treated before 1986,
which is about the midpoint of the available data. A third
of the studies were restricted to children, and 42%
involved haematological malignant diseases.
Control of baseline imbalances
About two-thirds of studies provided some form of
adjusted analysis. They used various strategies, including
multivariable models, stratification (weighted average of
subgroup-specific results), subgroup analysis (without
averaging), matching of trial and non-trial patients on the
basis of important prognostic factors, and restriction
(repeating the main analysis in presumably comparable
subsets of trial and non-trial patients) to exclude
confounding as an alternative explanation for observed
effects. Studies that found no evidence for a trial effect in
unadjusted comparisons generally did not do adjusted
analyses. In addition, some studies investigated baseline
imbalances in prognostic factors and, if none was found,
assumed that they were unlikely to cause confounding.
Table 2 lists the number of studies that used one or more
of these strategies to account for specific confounders,
and table 1 lists the covariates addressed by individual
studies. Excluding the natural experiment,9 eight of
25 comparisons (including only four of 21 retrospective
cohorts) restricted non-trial patients to those who would
have been eligible for the trial.
Results
Inclusion criteria
We identified 24 published articles that met our inclusion
criteria.9–22,29–35 Of these, seven were included in previous
reviews.24,49 Two articles22,29 reported two comparisons
each, thus, there was a total of 26 comparisons. Table 1
266
Trial effects
Of 23 comparisons that reported unadjusted analyses, ten
showed that trial patients had better outcomes than nontrial patients. Two additional comparisons suggested that
outcomes were better in trial participants than in non-
THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com
For personal use. Only reproduce with permission from The Lancet publishing Group.
ARTICLES
Randomisation 1
Group 1
Non-trial
care
Group 2
Trial
care
Randomisation 2
Standard group of
clinical trial
Decline
randomisation
Experimental
group of clinical
trial
Hypothetical randomised controlled trial to assess whether
trial participation improves clinical outcomes
trial participants for selected subgroups,17,20 and three
showed that outcomes were better in trial compared with
non-trial patients for selected endpoints.10,31,37 In seven
unadjusted comparisons, there was no evidence for a trial
effect.
17 of 26 comparisons reported adjusted analyses. These
controlled for a median of four covariates (IQR 3–6). Trial
patients had improved outcomes in seven comparisons.
In four additional comparisons, outcomes were better
among trial compared with non-trial patients for selected
subgroups,17,18,21,22 and in one comparison, outcomes were
improved with respect to selected endpoints.16 Five
adjusted analyses did not find evidence for a trial effect.
Finally, we investigated the eight studies that restricted
non-trial patients to those meeting the trial’s eligibility
criteria, together with the population-based natural
experiment.9 Compared with non-trial patients, trial
patients had better outcomes in three of nine
comparisons.9,11,13
In post-hoc comparisons, the number of analyses
showing a trial effect varied qualitatively according to
study characteristics (table 3). Positive studies were more
likely than negative studies to involve children, patients
treated during the earlier period of the available data (ie,
before 1986), and patients with haematological malignant
diseases. The only natural experiment was positive, none
of the four studies using eligible-refuser designs was
positive, and 13 of 21 retrospective cohorts were positive.
No studies recorded worse outcomes in trial-enrolled
patients than in non-trial controls.
Discussion
In our review of the published work, we found little highquality evidence to support the pervasive belief that cancer
trial participation leads to improved outcomes. Although
about half the studies provided some evidence for a trial
effect, and none found trial participation to be harmful,
methodological difficulties with most studies suggest the
need for cautious interpretation.
There are four possible reasons that trial participants
might be found to have improved outcomes when
compared with non-trial controls. First, there might be an
experimental treatment effect,24 in which the experimental
treatment offered in the trial was better than standard
therapies. Such an effect might result if early-phase
clinical testing or rational drug design reliably identified
therapeutic advances. It is worth noting, however, that in
view of the requirement for equipoise or uncertainty in
randomised controlled trials,44,45 widespread evidence for a
treatment effect would raise ethical issues.
THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com
Second, there might be a participation effect, in which
aspects of trial participation other than exposure to
investigational therapy might cause the improvement. A
participation effect might be concluded if participants in
the control group of a randomised controlled trial reliably
had better outcomes than did non-trial patients.
Braunholtz and colleagues24 further subdivide this effect
into: protocol effect (the way the treatments are
delivered); care effect (incidental aspects of care);
Hawthorne effect (changes in doctor or patient behaviour
on the basis of the knowledge that they are under
observation); and placebo effect (psychologically
mediated benefits from patients’ awareness of trial
participation). Although determining which of these four
effects contributed to any benefit seen from study
participation might be difficult, all are true trial effects
that, if proven, would give patients valid self-interested
reasons to enrol. Also, experimental treatment effects and
participation effects could coexist in the same trial.
Third, the improved outcomes might result from
confounding, or differences in baseline characteristics
(eg, age, sex, ethnic origin, socioeconomic status,46
performance status, comorbidity) that are associated with
both enrolment and outcome, rather than from trial
participation itself. Trial participants are often a
prognostically favourable subset of patients,26,27,41–43 making
consideration of baseline comparability between trial and
non-trial groups essential.47 Differences in treatment or
context associated with, but not caused by, trial
participation (eg, treatment in high-volume centres)48
might also lead to better outcomes. Confounding does not
constitute a true trial effect.
Fourth, the improvement in outcomes might be due to
bias resulting from how the data was gathered. For
example, follow-up might be more complete in trial
participants than in non-trial controls, or non-trial controls
who survive might be more likely to be censored than those
who die (eg, if a death certificate search is done).
It is important to note that bias and confounding can
operate in either direction, creating apparent trial effects
where none exists or obscuring real trial effects.
A further subset of bias is publication bias, which might
result from failure to publish studies reporting negative
trial effects.49,50 Investigators might also be more likely to
study trial effects when they notice an apparent advantage
to trial participation in a particular population, or when an
area of oncology seems to have undergone rapid advance.
Such hindsight bias would result in a systematically
unrepresentative set of studies. Because publication and
hindsight biases can exist even if individual studies are
methodologically sound, they are difficult to detect.
The major challenge in separating true from false trial
effects is to identify an appropriate comparison group.
Ideally, trial patients should differ from non-trial patients
only in exposure to the trial. If baseline comparability
cannot be assumed, then methods such as matching on or
statistical adjustment for prognostic factors are needed.
However, such methods are not ideal, especially because
they cannot control for unmeasured differences.
The best way to ensure baseline comparability would
be to do a randomised controlled trial (eg, figure),
in which patients are randomly assigned (or not) to
be offered trial participation. Ideally, a randomised
controlled trial would be double-blind—neither patients
nor clinicians would be aware of group assignment.
This, of course, would be ethically untenable. Obtaining
consent for the first randomisation might overcome
ethical objections, but would increase the probability of a
biased comparison.
267
For personal use. Only reproduce with permission from The Lancet publishing Group.
ARTICLES
Apart from randomisation and blinding, a randomised
controlled trial should not specify treatment or follow-up
strategies for the non-trial group, since this would involve
intervening in a way that approximates participation.
Second, because exclusions after randomisation can
create baseline imbalances and bias comparisons, data
should be analysed on an intention-to-treat basis.39 All
those assigned to trial care should be analysed with that
group, irrespective of whether they actually enrol. For
valid ethical, political, and logistical reasons, such a study
will probably never be done. However, imagining such a
study provides a methodological paradigm against which
other, more feasible designs can by judged.
Natural experiments, or incidental randomisations are
the next best study design. For example, a clinical trial
programme might be done in one geographic region but
not in a comparable region; a group of institutions might
start a trial programme whereas comparable institutions
do not; or conversely, a group of institutions might stop
doing trials while others continue. Assuming that selection
of trial regions or institutions is unrelated to other factors
likely to affect prognosis (clinical expertise, socioeconomic
status of the population, etc), outcome differences might
reasonably be attributed to the conduct of trials. Natural
experiments are rare but have occurred.9
A third possibility is comparison with people who were
offered trial enrolment, but declined. Like natural
experiments, such studies might be prospective or
retrospective. Prospective studies, however, must be
careful not to intervene in treatment of refusers in ways
that approximate trial entry.
An advantage of this design is that it selects controls
from the same pool as trial patients. Its major limitation is
that refusal might correlate directly with the outcome in
question. For example, patients who enter the trial might
be more adherent.40 Furthermore, in trials for advanced
disease, some patients who decline participation might
reasonably opt for supportive care only, even if anticancer
therapy might extend life. If so, then the predictably
longer survival in trial participants compared with
controls would reflect patient choice rather than trial
effect. Eligible refuser studies that address these
limitations, however, are valuable because they confront
directly a crucial question: should we encourage patients
with cancer to accept entry in a clinical trial on the basis of
self-interest alone?
The patient preference (or comprehensive cohort)
trial,29 in which potential patients who decline
randomisation may request direct assignment to one of
the trial groups, is a variation on the eligible refuser
design. However, since non-trial patients are treated in
accordance with a protocol, such analyses are biased
towards finding no effect.
The fourth possibility is to compare prospectively a
cohort of patients receiving non-trial care with a group of
trial participants. Potential sources of non-trial cohorts
include population-based cancer registries or patients seen
at institutions not participating in the trial. Ideally,
controls should meet all trial eligibility criteria. Analysts
can further reduce confounding by matching on or
adjusting statistically for known prognostic factors. As
with other prospective designs, the study should not
intervene in the care of non-trial patients.
The main advantages of a prospective cohort are the
ability to gather complete information on potential
confounders and outcomes, and, by specifying the study
hypothesis in advance, the avoidance of hindsight bias.
The main disadvantage compared with the designs above
is that, because trial participants and non-participants are
268
selected from different pools, the risk of baseline
differences is great.
The final (and most popular) option is retrospectively
to compare a group of trial participants with a group of
non-trial patients. This study design has important
limitations, including difficulty in controlling for baseline
imbalances between groups (frequently, important
covariates are not recorded in non-trial patients) and the
possibility of hindsight bias. Its main advantage, of course,
is practicality. Retrospective cohort designs can be
strengthened by a systematic method for identifying all
appropriate controls; use of concurrent controls;
restriction of controls to those who would have met
eligibility criteria for the trial; careful adjustment for
potential confounders; and inclusion of several trials and
diseases to minimise the possibility of hindsight bias.
Our results showed that most analyses, including all but
one positive study, used retrospective cohort designs.
Second, despite extensive evidence that trial patients
constitute a prognostically favourable subset of those with
the disease under study,26,27,41–43 few analyses controlled
adequately for covariates that might provide alternative
explanations for improved outcomes. No study controlled
for comorbidity, only one controlled for socioeconomic
status, and only four controlled for performance status.
Third, only four of 21 retrospective cohort analyses
restricted non-trial controls to those meeting trial
eligibility criteria, which we view as a minimum standard
for establishing comparability between groups. Fourth, no
studies found that eligible patients who accepted trial
entry had better outcomes than those who declined.
Finally, studies did not consistently control for treatment
differences, such as hospital volume or care in a cancer
centre, that might correlate with improved outcomes.
In addition to concerns about the validity of individual
comparisons, we found reasons for caution in generalising
from this body of evidence. Almost half the studies
(including eight of 14 positive studies) involved patients
diagnosed and treated in the 1970s and early 1980s,
a time of rapid change in both cancer treatment and
the organisation and delivery of cancer care. These
studies might therefore not be relevant to contemporary
oncology practice. In addition, several reports stated
that better-than-expected outcomes in a particular
population sparked the investigation of trial effects,
thereby explicitly suggesting hindsight bias.13,19,20 Others
retrospectively assessed diseases for which recent
therapeutic progress was well established.11,12,14,21 Few
studies included early-phase and non-randomised trials,
and paediatric trials and haematological malignant
diseases were disproportionately represented, especially in
positive studies.
Two previous reviews have considered trial effects in
oncology. Stiller51 assessed cancer survival in relation to
patterns of health-care delivery. He identified nine articles
that assessed trial entry as an independent variable and,
on the basis of six positive studies, suggested that trial
participation was linked to increased survival. He did not,
however, assess the quality of referenced studies or the
validity of the comparisons.
A more recent systematic review,24 part of a
comprehensive assessment of ethical issues in randomised
controlled trials,52 investigated whether trial participation
is associated with improved outcomes. Ten of 14 articles
in their study involved patients with cancer. We excluded
three of these from our review, either because they did not
compare trial with non-trial patients,28 or because they did
not claim that comparisons between trial and non-trial
patients were fair.26,27 The authors found significant
THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com
For personal use. Only reproduce with permission from The Lancet publishing Group.
ARTICLES
evidence for a trial effect in five of the seven remaining
articles,9,11,13,14,19 and trends towards a trial effect in two.29,33
After acknowledging that there is little good quality
evidence available, they cautiously concluded that there is
(weak) evidence that well conducted trials tend to benefit
the participants and do not seem (on average) to result in
harm.24
There are at least two reasons why our conclusions
differ somewhat from those of Braunholtz and
colleagues.52 First, our study contains 17 new articles
(18 comparisons). Of these, two provided evidence for a
trial effect, seven suggested a trial effect with respect to
selected subgroups or endpoints, and nine were negative.
Second, inadequately controlled baseline differences
between groups undermined our confidence in most
studies. In their comprehensive assessment of randomised
controlled trials,52 Braunholtz and colleagues seem to
agree with these concerns. For example, when discussing
confounding in the seven articles included in both their
report and our own, they rated two as no difference/fully
adjusted, two as small differences/partly adjusted, and
three as no adjustment for large differences.
Our review has several limitations. First, there is no
search strategy that can reliably capture all relevant
publications. As a result, we might have missed one or
more pertinent studies. However, we have incorporated
seven articles from previous reviews, and identified
17 articles that were not included in previous reports. To
ensure completeness, we did a new literature search using
conventional techniques, and searched retrospectively and
prospectively from all primary articles, reviews, and
position papers. Second, there are no accepted standards
for assessing the quality of relevant studies. To limit
subjectivity, we present the primary data concerning
outcomes, study designs, and potential confounders
(table 1). Third, any apparent differences in the
proportion of studies showing a trial effect by age, time
period or type of malignant disease should be viewed as
descriptive rather than as rigorous tests of a priori
hypotheses.
In sum, we found little generalisable evidence to
support the contention that trial participation directly
improves outcomes for cancer patients. Until more
convincing evidence for a trial effect is available,
recruitment messages to patients considering trials should
focus on their contribution to advances in treatment. We
believe that patients, professionals, and third-party payers
can recognise the crucial function of clinical trials in
advancing treatment, and that de-emphasising direct
benefits to patients need not compromise accrual or
coverage. We remain optimistic that strong support for
trials can flourish on the basis of their unquestioned role
in improving options and outcomes for patients with
cancer.
References
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Contributors
J Peppercorn, J Weeks, and S Joffe had the idea for this research.
J Peppercorn, J Weeks, E F Cook, and S Joffe developed the conceptual
model. J Peppercorn and S Joffe abstracted the data, with assistance from
J Weeks. J Peppercorn and S Joffe wrote the report, and J Weeks and
E F Cook revised it for important intellectual content. All authors
approved the final version.
26
27
28
Conflict of interest statement
The authors have no financial or other conflicts of interest with respect to
the contents of this report.
29
Acknowledgments
S Joffe received support from the US National Cancer Institute
(K01 CA96872) during the period of this research. The authors received
no specific funding for the study reported here.
THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com
30
Gelber RD, Goldhirsch A. Can a clinical trial be the treatment of
choice for patients with cancer? J Natl Cancer Inst 1988; 80: 886–87.
Lara PN Jr, Higdon R, Lim N, et al. Prospective evaluation of cancer
clinical trial accrual patterns: identifying potential barriers to
enrollment. J Clin Oncol 2001; 19: 1728–33.
Merz B. NCI seeks to boost study participation. JAMA 1988; 260: 1337.
Antman K, Schnipper LE, Frei E 3rd. The crisis in clinical cancer
research: third-party insurance and investigational therapy.
N Engl J Med 1988; 319: 46–48.
Bailes J. Managed care and oncology: the quality debate. J Clin Oncol
2000; 18: 705–07.
American Federation of Clinical Oncologic Societies. Access to quality
cancer care: consensus statement. J Clin Oncol 1998; 16: 1628–30.
Children’s Oncology Group: About cancer clinical trials.
http://www.childrensoncologygroup.org/ (accessed Dec 11, 2003).
Poisson R. Fraud in breast-cancer trials. N Engl J Med 1994; 330: 1460.
Karjalainen S, Palva I. Do treatment protocols improve end results? A
study of survival of patients with multiple myeloma in Finland.
BMJ 1989; 299: 1069–72.
Greil R, Holzner B, Kemmler G, et al. Retrospective assessment of
quality of life and treatment outcome in patients with Hodgkin’s
disease from 1969 to 1994. Eur J Cancer 1999; 35: 698–706.
Lennox EL, Stiller CA, Jones PH, Wilson LM. Nephroblastoma:
treatment during 1970–73 and the effect on survival of inclusion in the
first MRC trial. BMJ 1979; 2: 567–69.
Meadows AT, Kramer S, Hopson R, Lustbader E, Jarrett P, Evans
AE. Survival in childhood acute lymphocytic leukemia: effect of
protocol and place of treatment. Cancer Invest 1983; 1: 49–55.
Davis S, Wright PW, Schulman SF, et al. Participants in prospective,
randomized clinical trials for resected non-small cell lung cancer have
improved survival compared with nonparticipants in such trials.
Cancer 1985; 56: 1710–18.
Stiller CA, Draper GJ. Treatment centre size, entry to trials, and survival
in acute lymphoblastic leukaemia. Arch Dis Child 1989; 64: 657–61.
Boros L, Chuang C, Butler FO, Bennett JM. Leukemia in Rochester
(NY). A 17-year experience with an analysis of the role of cooperative
group (ECOG) participation. Cancer 1985; 56: 2161–69.
Marubini E, Mariani L, Salvadori B, et al. Results of a breast-cancersurgery trial compared with observational data from routine practice.
Lancet 1996; 347: 1000–03.
Feuer EJ, Frey CM, Brawley OW, et al. After a treatment
breakthrough: a comparison of trial and population- based data for
advanced testicular cancer. J Clin Oncol 1994; 12: 368–77.
Roy P, Vaughan Hudson G, Vaughan Hudson B, Esteve J,
Swerdlow AJ. Long-term survival in Hodgkin’s disease patients: a
comparison of relative survival in patients in trials and those recorded
in population-based cancer registries. Eur J Cancer 2000; 36: 384–89.
Stiller CA, Eatock EM. Survival from acute non-lymphocytic leukaemia,
1971–88: a population based study. Arch Dis Child 1994; 70: 219–23.
Wagner HP, Dingeldein-Bettler I, et al. Childhood NHL in
Switzerland: incidence and survival of 120 study and 42 non-study
patients. Med Pediatr Oncol 1995; 24: 281–86.
Stiller CA, Eatock EM. Patterns of care and survival for children with
acute lymphoblastic leukaemia diagnosed between 1980 and 1994.
Arch Dis Child 1999; 81: 202–08.
Stiller CA, Benjamin S, Cartwright RA, et al. Patterns of care and
survival for adolescents and young adults with acute leukaemia—
a population-based study. Br J Cancer 1999; 79: 658–65.
Lantos JD. The “inclusion benefit” in clinical trials. J Pediatr 1999;
134: 130–31.
Braunholtz DA, Edwards SJ, Lilford RJ. Are randomized clinical trials
good for us (in the short term)? Evidence for a “trial effect”.
J Clin Epidemiol 2001; 54: 217–24.
TROUT Review Group. How do the outcomes of patients treated
within randomised control trials compare with those of similar patients
treated outside these trials? http://hiru.mcmaster.ca/ebm/trout/
(accessed Dec 11, 2003).
Antman K, Amato D, Wood W, et al. Selection bias in clinical trials.
J Clin Oncol 1985; 3: 1142–47.
Bertelsen K. Protocol allocation and exclusion in two Danish
randomised trials in ovarian cancer. Br J Cancer 1991; 64: 1172–76.
Medical Research Council. Duration of survival of children with acute
leukaemia. Report to the Medical Research Council from the
Committee on Leukaemia and the Working Party on Leukaemia in
Childhood. BMJ 1971; 4: 7–9.
Schmoor C, Olschewski M, Schumacher M. Randomized and nonrandomized patients in clinical trials: experiences with comprehensive
cohort studies. Stat Med 1996; 15: 263–71.
Dahlberg M, Glimelius B, Pahlman L. Improved survival and
reduction in local failure rates after preoperative radiotherapy:
269
For personal use. Only reproduce with permission from The Lancet publishing Group.
ARTICLES
31
32
33
34
35
36
37
38
39
40
evidence for the generalizability of the results of Swedish Rectal
Cancer Trial. Ann Surg 1999; 229: 493–97.
Dowling AJ, Czaykowski PM, Krahn MD, Moore MJ, Tannock IF.
Prostate specific antigen response to mitoxantrone and prednisone in
patients with refractory prostate cancer: prognostic factors and
generalizability of a multicenter trial to clinical practice. J Urol 2000;
163: 1481–85.
Mayers C, Panzarella T, Tannock IF. Analysis of the prognostic
effects of inclusion in a clinical trial and of myelosuppression on
survival after adjuvant chemotherapy for breast carcinoma. Cancer
2001; 91: 2246–57.
Ward LC, Fielding JW, Dunn JA, Kelly KA, for the British Stomach
Cancer Group. The selection of cases for randomised trials: a registry
survey of concurrent trial and non-trial patients. Br J Cancer 1992; 66:
943–50.
Winger MJ, Macdonald DR, Schold SC, Jr., Cairncross JG. Selection
bias in clinical trials of anaplastic glioma. Ann Neurol 1989; 26: 531–34.
Link MP, Goorin AM, Miser AW, et al. The effect of adjuvant
chemotherapy on relapse-free survival in patients with osteosarcoma of
the extremity. N Engl J Med 1986; 314: 1600–06.
Cottin V, Arpin D, Lasset C, et al. Small-cell lung cancer: patients
included in clinical trials are not representative of the patient
population as a whole. Ann Oncol 1999; 10: 809–15.
Schea RA, Perkins P, Allen PK, Komaki R, Cox JD. Limited-stage
small-cell lung cancer: patient survival after combined chemotherapy
and radiation therapy with and without treatment protocols. Radiology
1995; 197: 859–62.
Burgers JA, Arance A, Ashcroft L, Hodgetts J, Lomax L,
Thatcher N. Identical chemotherapy schedules given on and off
trial protocol in small cell lung cancer: response and survival results.
Br J Cancer 2002; 87: 562–66.
Fergusson D, Aaron SD, Guyatt G, Hebert P. Post-randomisation
exclusions: the intention to treat principle and excluding patients from
analysis. BMJ 2002; 325: 652–54.
Leventhal H, Nerenz DR, Leventhal EA, Love RR, Bendena LM. The
behavioral dynamics of clinical trials. Prev Med 1991; 20: 132–46.
270
41 Chen CI, Skingley P, Meyer RM. A comparison of elderly
patients with aggressive histology lymphoma who were entered or not
entered on to a randomized phase II trial. Leuk Lymphoma 2000; 38:
327–34.
42 Hjorth M, Holmberg E, Rodjer S, Westin J, for the Myeloma Group
of Western Sweden. Impact of active and passive exclusions on the
results of a clinical trial in multiple myeloma. Br J Haematol 1992; 80:
55–61.
43 Rahman ZU, Frye DK, Buzdar AU, et al. Impact of selection process
on response rate and long-term survival of potential high-dose
chemotherapy candidates treated with standard-dose doxorubicincontaining chemotherapy in patients with metastatic breast cancer.
J Clin Oncol 1997; 15: 3171–77.
44 Freedman B. Equipoise and the ethics of clinical research.
N Engl J Med 1987; 317: 141–45.
45 Peto R, Collins R, Gray R. Large-scale randomized evidence:
large, simple trials and overviews of trials. J Clin Epidemiol 1995; 48:
23–40.
46 Cella DF, Orav EJ, Kornblith AB, et al. Socioeconomic status and
cancer survival. J Clin Oncol 1991; 9: 1500–09.
47 Zelen M. Theory and practice of clinical trials. In: Bast RC,
Pollock RE, Kufe DW, Weichselbaum RR, Frei E, eds. Cancer
Medicine. Hamilton, Ontario: B C.Decker Inc, 2000: 298–313.
48 Hillner BE, Smith TJ, Desch CE. Hospital and physician volume or
specialization and outcomes in cancer treatment: importance in quality
of cancer care. J Clin Oncol 2000; 18: 2327–40.
49 Ioannidis JP. Effect of the statistical significance of results on the time
to completion and publication of randomized efficacy trials.
JAMA 1998; 279: 281–86.
50 Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr.
Publication bias and clinical trials. Control Clin Trials 1987; 8: 343–53.
51 Stiller CA. Centralised treatment, entry to trials and survival.
Br J Cancer 1994; 70: 352–62.
52 Edwards SJ, Lilford RJ, Braunholtz DA, Jackson JC, Hewison J,
Thornton J. Ethical Issues in the design and conduct of randomised
controlled trials. Health Tech Assess 1998; 2: 1–131.
THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com
For personal use. Only reproduce with permission from The Lancet publishing Group.