Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
ARTICLES Articles Comparison of outcomes in cancer patients treated within and outside clinical trials: conceptual framework and structured review Jeffrey M Peppercorn, Jane C Weeks, E Francis Cook, Steven Joffe Summary Background Many oncologists believe that patients with cancer who enrol in clinical trials have better outcomes than those who do not enrol. We aimed to assess the empirical evidence that such a trial effect exists. Methods We developed a conceptual framework for comparison of trial and non-trial patients. We then did a comprehensive literature search to identify studies that compared outcomes between these groups. We critically evaluated these studies to assess whether they provide valid and generalisable support for a trial effect. Findings We identified 26 comparisons, from 24 published articles, of outcomes among cancer patients enrolled and not enrolled in clinical trials. 21 comparisons used retrospective cohort designs. 14 comparisons provided some evidence that patients enrolled in trials have improved outcomes. However, strategies to control for potential confounding factors were inconsistent and frequently inadequate. Only eight comparisons restricted non-trial patients to those meeting trial eligibility criteria. Of these, three noted better outcomes in trial patients than in non-trial patients. Children with cancer, patients with haematological malignant disease, and patients treated before 1986 were disproportionately represented in positive studies. Interpretation Despite widespread belief that enrolment in clinical trials leads to improved outcomes in patients with cancer, there are insufficient data to conclude that such a trial effect exists. Until such data are available, patients with cancer should be encouraged to enrol in clinical trials on the basis of trials’ unquestioned role in improving treatment for future patients. Lancet 2004; 363: 263–70 Departments of Medical Oncology (J M Peppercorn MD, J C Weeks MD), and Pediatric Oncology (S Joffe MD), Dana-Farber Cancer Institute, Boston, MA, USA; Department of Medicine, Brigham and Women’s Hospital, Boston (J M Peppercorn, J C Weeks, Prof E F Cook ScD); Department of Medicine, Children’s Hospital, Boston (S Joffe); and Harvard School of Public Health, Boston (E F Cook) Correspondence to: Dr Steven Joffe, Dana-Farber Cancer Institute, 44 Binney Street, Boston, MA 02115, USA (e-mail: [email protected]) THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com Introduction The belief that clinical trials offer the best treatment for patients with cancer is widespread in the oncology community. This claim, motivated partly by aims to increase accrual1–3 and ensure third-party payment,4,5 appears frequently in pronouncements by professional organisations and leaders. For example, the American Federation of Clinical Oncologic Societies maintains that “treatment in a clinical trial is often a cancer patient’s best option”.6 Other people argue that “clinical trials are proven to offer children the best chance of survival”,7 and that trial access is one of the “basic requirements of quality cancer care.”5 Such claims suggest that trials are viewed not only as a way to improve future treatment, but also as the best treatment for current patients. The view that trials lead to better outcomes, if correct, has important implications. First, that more than 95% of adults and perhaps 40% of children with cancer do not enrol in trials would constitute evidence of substandard care. Second, the suggestion that patients benefit directly by becoming research participants changes the traditional model of human experimentation. If so, clinicians arguably should advocate forcefully for enrolment on grounds of direct benefit, rather than presenting the risks and benefits for patients to weigh. In the conventional view, such advocacy might be criticised as misleading or coercive. Third, acceptance of this view might require substantial changes in trial financing and organisation, eligibility criteria, and patient selection. Anything that might constitute a barrier to participation (including considerations of scientific validity and integrity8) would be suspect. We must therefore be confident that trial participation improves outcomes before using the claim to inform practice or policy. Ideally, the statement that trials are the best treatment option should rest on evidence that trial participants have better outcomes than similar patients treated off-protocol. Several studies9–22 have shown such a trial effect, also sometimes known as an inclusion benefit.23 However, showing a causal relation between trial participation and improved outcome is difficult. Here, we seek to develop a conceptual framework for assessing the trial effect; describe the methodological challenges in studying this effect and the hierarchy of evidence that could be used to support its existence; and use these insights to assess systematically the quality, validity, and generalisability of the published work. Methods We sought to identify articles that presented primary data comparing outcomes between trial and non-trial patients with cancer. As others note,24 there is no obvious set of terms to capture all relevant reports. We therefore searched MEDLINE using the terms trial effect, inclusion benefit, population outcomes, community outcomes, trial benefit, patient preference trial, and comprehensive 263 For personal use. Only reproduce with permission from The Lancet publishing Group. ARTICLES Ref Pop Dates Type Enrolled* Eligible Treatment Potential confounders and methods of control‡ controls similar† Accounted for Not accounted for 9 Multiple 1979–85 NE myeloma 405/164 NA No 34 High-grade 1983–87 ER Glioma 55/23 No 29 Trial 2 Localised 1983–89 ER breast Trial 3 Localised 1983–89 ER breast Yes 473/247 Yes Yes 199/129 Yes Yes No Menopause, number of positive axillary nodes, tumour size, tumour grade, hormone receptor status (NBD) Menopause, number of positive axillary nodes, tumour size, tumour grade, hormone receptor status (NBD) Age, sex, stage, histology, tumour size, TN group, radiation therapy (MVA) 13 Stage I NSCLC 1977–79 RC 78/471 33 Gastric 1976–80 RC 217/493 Yes Yes Sex, clinical stage (NBD) 15 AML 1975–82 RC 46/84 No No WBC, LDH, chemotherapy dose, platelet count, PS, receipt of antibiotics, preleukaemia, fever (MVA) 16 Localised 1973–80 RC breast (T) 1980–84 (C) Hodgkin’s 1978–84 RC disease Advanced 1982–88 RC testicular (T) 1978–84 (C) 352/ 1408 No Yes Age, tumour size, axillary nodes, tumour site, histology (MVA) Treatment centre (NBD) Age (MVA) Sex (NBD) Extent of disease (SgpA) 18 17 36 SCLC 32 1986–89 RC Yes Sex, period of diagnosis, follow-up year (MVA) Age, sex, period of diagnosis (ES) Age, histology, treatment centre (NBD) 1106/ No 4807 133/172 No 73/37 No No Newly diagnosed vs relapsed disease, histology, receipt of chemotherapy (RC) Age (NBD) Age, PS, extent of disease (MVA) Sex, weight loss, fever, SVC syndrome, paraneoplastic syndrome, chest pain, liver/ bone/bone marrow metastases, treatment centre (NBD) T status, N status, hormone receptor status, hormone therapy, treatment centre (MVA) Stage, type of surgery (StrA) Age, treatment centre (NBD) Trial effect observed§ Unadjusted Adjusted Treatment centre (BD) SES, PS, comorbidity, stage (NE) Yes¶ Yes PS (BD) Sex, SES, comorbidity (NE) No Not done Treatment centre (BD) Age, SES, PS, comorbidity (NE) No Not done Treatment centre (BD) Age, SES, PS, comorbidity (NE) No Not done Treatment centre, county of residence (BD) PS, comorbidity, complete staging, SES (NE) Age, stage, symptom duration, tumour site, number, size, type of surgery, treatment centre (BD) SES, PS, comorbidity (NE) % blasts, other laboratory studies (NBD) Age, sex, histological subtype, year of treatment, treatment centre (BD) SES, comorbidity (NE) Temporal trends, multifocality, adjuvant chemotherapy (BD) SES, PS, comorbidity (NE) Yes Yes No Not done Yes Yes Yes Mixed|| PS, comorbidity, SES, stage, treatment centre (NE) Treatment centre, temporal trends (BD) SES, PS, comorbidity (NE) Not reported Mixed** Mixed Mixed†† Brain metastases, hemoptysis (BD) SES, comorbidity (NE) Yes No Age, SES, PS, comorbidity, radiation therapy (NE) Yes No No No Localised 1980–90 RC breast 160/519 No No 30 Rectal 1987–90 RC 557/798 Yes Yes 37 SCLC 1987–92 RC 41/40 No Yes 10 Adult and paediatric Hodgkin’s disease 1988–94 RC (T) 1969–94 (C) 62/163 No No 31 Prostate 1990–94 RC (T) 1994–98 (C) 80/132 No Yes PS (MVA) Age, years since diagnosis (NBD) Baseline PSA, lymph node Mixed*** involvement, hormone therapy, treatment centre (BD) SES, comorbidity (NE) No 38 SCLC 1994–98 RC 60/46 No Yes Age, sex, PS, stage, alkaline phosphatase, serum sodium, LDH, respiratory score, site of tumour, treatment centre (NBD) SES, comorbidity (NE) Not done Sex, tumour height, radiation No therapy (BD) SES, PS, comorbidity (NE) Age, sex, SES (insurance PS, previous cancer, weight Mixed§§ status), stage, LDH (NBD) loss, treatment centre (BD) Comorbidity (NE) Age, sex, stage, risk factors, Year of diagnosis, staging Mixed¶¶ B symptoms, treatment methods (BD) mode (MVA) PS, SES, comorbidity (NE) Treatment centre (NBD) No No‡‡ Not done No|||| (continues next page) 264 THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com For personal use. Only reproduce with permission from The Lancet publishing Group. ARTICLES (continued from previous page) Ref Pop Dates Type Enrolled* Eligible Treatment Potential confounders and methods of control‡ controls similar† Accounted for Not accounted for 22 12 Adolescent 1984–94 and young adult ALL Adolescent 1984–94 and young adult AML Osteo1982–84 sarcoma Wilms’ 1970–73 tumour ALL 1970–75 14 ALL 1971–82 RC 2137/ 933 No No 19 AML 1975–88 RC 369/449 No No 20 NHL 120/42 No No 21 ALL 1976–91 RC (T) 1981–91 (C) 1980–94 RC 3759/ 1229 No No 22 35 11 Trial effect observed§ Unadjusted Adjusted RC 154/263 No No Period of diagnosis (NBD) Age, teaching hospital, No hospital volume (BD) Sex, SES, PS, comorbidity (NE) Not done RC 180/282 No No Period of diagnosis (SgpA) Age, teaching hospital, hospital volume (NBD) Sex, SES, PS, comorbidity (NE) Yes Mixed††† EF. 36/77 Yes Yes SES, PS, comorbidity (NE) Not done RC 98/104 Yes No RC 257/70 No No Age, sex, tumour site, treatment centre (NBD) Age, stage (StrA) Treatment centre (SgpA) Sex, WBC, treatment centre (MVA) Age, race, survival for at least 28 days (RC) Age, period of diagnosis, treatment centre (StrA) Sex, WBC (NBD) Age (StrA) Period of treatment, teaching hospital (SgpA) Sex (NBD) No Radiation dose (BD) Yes Sex, SES, PS, comorbidity (NE) SES, PS, comorbidity (NE) Yes Yes SES, PS, comorbidity (NE) Yes Yes SES, PS, comorbidity, WBC (NE) Not reported Yes‡‡‡ Location of initial care (BD) Mixed§§§ Age, sex, SES, PS, comorbidity stage, treatment centre (NE) Period of diagnosis (SgpA) Age, sex, WBC, immunophenotype, Down’s syndrome (StrA) Hospital volume, UKCCSG membership (BD) SES, PS, comorbidity (NE) Yes Not done Not reported Mixed¶¶¶ ALL=acute lymphoblastic leukaemia. AML=acute myeloid leukaemia. NHL=non-Hodgkin lymphoma. NSCLC=non-small-cell lung cancer. SCLC=small-cell lung cancer. NA=not applicable. NE=natural experiment. ER=eligible refuser. RC=retrospective cohort. MVA=multivariable analysis. ES=external standardisation. SgpA=subgroup analysis. StrA=stratified analysis. RC=restriction of cohort. NBD=no baseline difference. BD=baseline difference recorded. NE=not evaluated. SES=socioeconomic status. PS=performance status. WBC=white blood cell count. PSA=prostate specific antigen. LDH=lactate dehydrogenase. TN=tumour-node group. SVC=superior vena cava. *Values are number in trial/number in control or non-trial group. †For randomised controlled trials, refers to similarity between treatment offered on the control group and that received by non-trial participants. ‡We assessed whether each study attempted to account for possible confounding by age, sex (where applicable), PS, comorbidity, SES, stage (where applicable), and treatment centre. We also assessed other potential confounding factors, as appropriate to individual studies. §p<0·05, unless otherwise noted. ¶Both unadjusted and adjusted analyses used survival rates, relative to the expected age-specific, sex-specific, and period-specific survival in the general population, as the outcome of interest. ||No difference in overall survival or distant recurrence was detected in adjusted analyses; local recurrence rates were significantly lower in trial patients than in non-trial patients. **Qualitatively, trial effect seen in patients 45 years or older only; authors did not present results of significance tests. ††In patients with minimum/moderate disease, trial participants had improved survival in all analyses. In patients with advanced disease, survival was better in trial participants than in non-trial patients (p=0·056 in unadjusted analysis). However, this advantage disappeared after patients with relapsed disease were eliminated from the trial cohort, and after adjustment was made for possible misclassification bias in staging of non-trial patients (Will-Rogers phenomenon). ‡‡Trial group includes patients from the control group of the randomised controlled trial only. Participants in the experimental group of the trial had better outcomes than those on the control group. Therefore, the authors might have recorded a trial effect (due to receipt of experimental therapy) had they included all randomised controlled trial participants in the trial group. §§Trial patients had improved survival and disease-specific survival, but not disease-free survival (p=0·06), when compared with non-trial patients. ¶¶In unadjusted analyses, a difference favouring trial patients was apparent in disease-free but not overall survival. ||||In the multivariable model evaluating predictors of disease-free survival, a trend towards improved outcomes in trial participants was noted (p=0·064). When treatment modality was omitted as a covariate from the multivariable model, disease-free survival was significantly improved in trial patients (p=0·038). ***Unadjusted analysis showed improved survival, but no improvement in PSA response, in trial patients. †††Subgroup analyses showed that there was no difference between trial and non-trial patients during 1984–88, but trial patients had better outcomes than non-trial patients during 1989–94. ‡‡‡Trial effect was restricted to patients treated in a teaching hospital between 1975–83. All analyses adjusted for age. §§§One trial group (Pediatric Oncology Group) had better outcomes than the non-trial group, whereas the other trial group (Swiss Pediatric Oncology Group) did not. ¶¶¶An apparent trial effect was recorded in patients diagnosed in 1985–89 and 1990–94, but not in those diagnosed in 1980–84. The difference between the trial and non-trial groups was no longer significant when patients who died in the first 4 weeks after diagnosis, most of whom were not enrolled in the trials, were excluded from the analysis. Table 1: Studies comparing cancer outcomes within and outside a trial cohort trial, cross-referenced with cancer, oncology, neoplasms, and clinical trials. We also scanned an online annotated bibliography maintained by researchers at McMaster University25 and examined the reference lists of the empirical studies we identified, of two previous reviews, and of position papers arguing that trial enrolment is beneficial. We used Science Citation Index Expanded (ISI, Philadelphia, PA) to locate subsequent reports that cited the manuscripts identified above. Two authors (JP, SJ) independently reviewed all articles identified in our search. We included articles if they involved cancer-directed treatment, and if the authors claimed to provide a valid comparison of outcomes between trial and non-trial patients. We excluded articles in which the main purpose was to show the non-representativeness of clinical trial participants (including two included in previous THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com reviews),26,27 unless the authors also provided an analysis they claimed accounted for baseline differences. We also omitted one article included in previous reviews that investigated treatment by a specialist rather than trial participation.28 We excluded studies of centre effects, guideline effects, and other elements of specialised care that did not assess trial entry as an independent variable. Using forms that we pilot-tested with four nononcology reports, we recorded study dates, sample sizes, study design, whether controls were restricted to trialeligible individuals, strategies used to control for confounding, potential sources of bias, and major outcomes. Except for one study that reported results graphically but not statistically,18 we classified studies as showing a trial effect if outcomes in trial patients were better with p<0·05. We noted whether studies attempted 265 For personal use. Only reproduce with permission from The Lancet publishing Group. ARTICLES Studies (n=26) Design of trial versus non-trial comparison Randomised controlled Natural experiment Eligible refuser Prospective cohort Retrospective cohort 0 1 4 0 21 Type of clinical trial in which patients were participating Randomised only 20 Other* 6 All patients treated before 1986 Yes No 10 16 Age-group of patients Children† Adult‡ 9 17 Type of malignant disease Haematological Other 11 15 Baseline differences accounted for§ Age Sex¶ Performance status Comorbidity Socioeconomic status|| Disease-specific prognostic factors** Treatment centre 19 13 4 0 1 19 14 Type of analysis Unadjusted only Adjusted only Both adjusted and unadjusted 9 3 14 Non-trial patients restricted to those meeting trial eligibility criteria†† Yes‡‡ No 8 17 *Two studies considered single-group trials only, four considered both singlegroup trials and randomised controlled trials, and type of trial could not be determined for one study. †Includes two studies involving adolescents and young adults.22 ‡Includes one study that involved a small proportion of children.10 §Includes studies that showed no baseline differences between groups, as well as those that adjusted for observed differences in the analysis. Nine studies used multivariable techniques to adjust for age, five for sex, three for performance status, none for socioeconomic status, seven for diseasespecific prognostic factors, and two for treatment centre. Among 17 studies that did some type of adjusted analysis, the median number of covariates was 4 (IQR 3–6). ¶Denominator includes 20 studies involving cancers that affect both sexes. ||Insurance status. **Studies that accounted for at least one disease-specific prognostic factor. Includes stage, where applicable. ††n=25. Excludes one study9 for which restriction to trial-eligible patients was not applicable. This study compared outcomes among all patients with multiple myeloma (both trial-eligible and not trial-eligible) between two adjacent geographical regions, only one of which participated in clinical trials. ‡‡Includes all four studies using eligible refuser designs,29,34,35 and four of 21 retrospective cohort analyses.11,13,30,33 Table 2: Characteristics of studies comparing outcomes in trial and non-trial patients to address potential selection differences by age, sex (if applicable), performance status, comorbidity, stage (if applicable), socioeconomic status, and treatment centre. In addition, for each study we recorded other potential factors that might have affected outcomes, and whether the analysis attempted to address them. Finally, we reconciled the two reviews. A third author (JCW) mediated disagreements. We present descriptive data only. Because of concerns about systematic biases in the published work due to inadequate control of selection factors, we did not undertake a formal meta-analysis. Studies suggesting a trial effect* Age-group Paediatric (n=9) Adult (n=17) All patients treated before 1986 Yes (n=10) No (n=16) Type of malignant disease Haematological (n=11) Other (n=15) Type of study Natural experiment (n=1) Eligible refuser (n=4) Retrospective cohort (n=21) All studies (n=26) 7† 7‡ 8§ 6¶ 9|| 5** 1 0 13†† 14†† *Where studies reported both unadjusted and adjusted analyses, table presents results of adjusted analyses. †Three studies found evidence for a trial effect in selected subgroups. ‡Two studies found evidence for a trial effect in selected subgroups, and two found evidence for a trial effect with respect to selected endpoints. §One study found evidence for a trial effect in selected subgroups, and one found evidence for a trial effect with respect to selected endpoints. ¶Four studies found evidence for a trial effect in selected subgroups, and one found evidence for a trial effect with respect to selected endpoints. ||Four studies found evidence for a trial effect in selected subgroups. **One study found evidence for a trial effect in to selected subgroups, and 2 found evidence for a trial effect with respect to selected endpoints. ††Five studies found evidence for a trial effect in selected subgroups, and two found evidence for a trial effect with respect to selected endpoints Table 3: Relations between characteristics of studies and trial effects (n=26) summarises these studies, arranged by population, study design, and dates of the primary data reported in the reports. Additional detail is available from the authors. Study characteristics Table 2 presents characteristics of the studies. Most (81%) were retrospective cohorts, and most (77%) compared non-trial patients with those enrolled in randomised rather than single-group trials. In 38% of comparisons, all patients had been treated before 1986, which is about the midpoint of the available data. A third of the studies were restricted to children, and 42% involved haematological malignant diseases. Control of baseline imbalances About two-thirds of studies provided some form of adjusted analysis. They used various strategies, including multivariable models, stratification (weighted average of subgroup-specific results), subgroup analysis (without averaging), matching of trial and non-trial patients on the basis of important prognostic factors, and restriction (repeating the main analysis in presumably comparable subsets of trial and non-trial patients) to exclude confounding as an alternative explanation for observed effects. Studies that found no evidence for a trial effect in unadjusted comparisons generally did not do adjusted analyses. In addition, some studies investigated baseline imbalances in prognostic factors and, if none was found, assumed that they were unlikely to cause confounding. Table 2 lists the number of studies that used one or more of these strategies to account for specific confounders, and table 1 lists the covariates addressed by individual studies. Excluding the natural experiment,9 eight of 25 comparisons (including only four of 21 retrospective cohorts) restricted non-trial patients to those who would have been eligible for the trial. Results Inclusion criteria We identified 24 published articles that met our inclusion criteria.9–22,29–35 Of these, seven were included in previous reviews.24,49 Two articles22,29 reported two comparisons each, thus, there was a total of 26 comparisons. Table 1 266 Trial effects Of 23 comparisons that reported unadjusted analyses, ten showed that trial patients had better outcomes than nontrial patients. Two additional comparisons suggested that outcomes were better in trial participants than in non- THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com For personal use. Only reproduce with permission from The Lancet publishing Group. ARTICLES Randomisation 1 Group 1 Non-trial care Group 2 Trial care Randomisation 2 Standard group of clinical trial Decline randomisation Experimental group of clinical trial Hypothetical randomised controlled trial to assess whether trial participation improves clinical outcomes trial participants for selected subgroups,17,20 and three showed that outcomes were better in trial compared with non-trial patients for selected endpoints.10,31,37 In seven unadjusted comparisons, there was no evidence for a trial effect. 17 of 26 comparisons reported adjusted analyses. These controlled for a median of four covariates (IQR 3–6). Trial patients had improved outcomes in seven comparisons. In four additional comparisons, outcomes were better among trial compared with non-trial patients for selected subgroups,17,18,21,22 and in one comparison, outcomes were improved with respect to selected endpoints.16 Five adjusted analyses did not find evidence for a trial effect. Finally, we investigated the eight studies that restricted non-trial patients to those meeting the trial’s eligibility criteria, together with the population-based natural experiment.9 Compared with non-trial patients, trial patients had better outcomes in three of nine comparisons.9,11,13 In post-hoc comparisons, the number of analyses showing a trial effect varied qualitatively according to study characteristics (table 3). Positive studies were more likely than negative studies to involve children, patients treated during the earlier period of the available data (ie, before 1986), and patients with haematological malignant diseases. The only natural experiment was positive, none of the four studies using eligible-refuser designs was positive, and 13 of 21 retrospective cohorts were positive. No studies recorded worse outcomes in trial-enrolled patients than in non-trial controls. Discussion In our review of the published work, we found little highquality evidence to support the pervasive belief that cancer trial participation leads to improved outcomes. Although about half the studies provided some evidence for a trial effect, and none found trial participation to be harmful, methodological difficulties with most studies suggest the need for cautious interpretation. There are four possible reasons that trial participants might be found to have improved outcomes when compared with non-trial controls. First, there might be an experimental treatment effect,24 in which the experimental treatment offered in the trial was better than standard therapies. Such an effect might result if early-phase clinical testing or rational drug design reliably identified therapeutic advances. It is worth noting, however, that in view of the requirement for equipoise or uncertainty in randomised controlled trials,44,45 widespread evidence for a treatment effect would raise ethical issues. THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com Second, there might be a participation effect, in which aspects of trial participation other than exposure to investigational therapy might cause the improvement. A participation effect might be concluded if participants in the control group of a randomised controlled trial reliably had better outcomes than did non-trial patients. Braunholtz and colleagues24 further subdivide this effect into: protocol effect (the way the treatments are delivered); care effect (incidental aspects of care); Hawthorne effect (changes in doctor or patient behaviour on the basis of the knowledge that they are under observation); and placebo effect (psychologically mediated benefits from patients’ awareness of trial participation). Although determining which of these four effects contributed to any benefit seen from study participation might be difficult, all are true trial effects that, if proven, would give patients valid self-interested reasons to enrol. Also, experimental treatment effects and participation effects could coexist in the same trial. Third, the improved outcomes might result from confounding, or differences in baseline characteristics (eg, age, sex, ethnic origin, socioeconomic status,46 performance status, comorbidity) that are associated with both enrolment and outcome, rather than from trial participation itself. Trial participants are often a prognostically favourable subset of patients,26,27,41–43 making consideration of baseline comparability between trial and non-trial groups essential.47 Differences in treatment or context associated with, but not caused by, trial participation (eg, treatment in high-volume centres)48 might also lead to better outcomes. Confounding does not constitute a true trial effect. Fourth, the improvement in outcomes might be due to bias resulting from how the data was gathered. For example, follow-up might be more complete in trial participants than in non-trial controls, or non-trial controls who survive might be more likely to be censored than those who die (eg, if a death certificate search is done). It is important to note that bias and confounding can operate in either direction, creating apparent trial effects where none exists or obscuring real trial effects. A further subset of bias is publication bias, which might result from failure to publish studies reporting negative trial effects.49,50 Investigators might also be more likely to study trial effects when they notice an apparent advantage to trial participation in a particular population, or when an area of oncology seems to have undergone rapid advance. Such hindsight bias would result in a systematically unrepresentative set of studies. Because publication and hindsight biases can exist even if individual studies are methodologically sound, they are difficult to detect. The major challenge in separating true from false trial effects is to identify an appropriate comparison group. Ideally, trial patients should differ from non-trial patients only in exposure to the trial. If baseline comparability cannot be assumed, then methods such as matching on or statistical adjustment for prognostic factors are needed. However, such methods are not ideal, especially because they cannot control for unmeasured differences. The best way to ensure baseline comparability would be to do a randomised controlled trial (eg, figure), in which patients are randomly assigned (or not) to be offered trial participation. Ideally, a randomised controlled trial would be double-blind—neither patients nor clinicians would be aware of group assignment. This, of course, would be ethically untenable. Obtaining consent for the first randomisation might overcome ethical objections, but would increase the probability of a biased comparison. 267 For personal use. Only reproduce with permission from The Lancet publishing Group. ARTICLES Apart from randomisation and blinding, a randomised controlled trial should not specify treatment or follow-up strategies for the non-trial group, since this would involve intervening in a way that approximates participation. Second, because exclusions after randomisation can create baseline imbalances and bias comparisons, data should be analysed on an intention-to-treat basis.39 All those assigned to trial care should be analysed with that group, irrespective of whether they actually enrol. For valid ethical, political, and logistical reasons, such a study will probably never be done. However, imagining such a study provides a methodological paradigm against which other, more feasible designs can by judged. Natural experiments, or incidental randomisations are the next best study design. For example, a clinical trial programme might be done in one geographic region but not in a comparable region; a group of institutions might start a trial programme whereas comparable institutions do not; or conversely, a group of institutions might stop doing trials while others continue. Assuming that selection of trial regions or institutions is unrelated to other factors likely to affect prognosis (clinical expertise, socioeconomic status of the population, etc), outcome differences might reasonably be attributed to the conduct of trials. Natural experiments are rare but have occurred.9 A third possibility is comparison with people who were offered trial enrolment, but declined. Like natural experiments, such studies might be prospective or retrospective. Prospective studies, however, must be careful not to intervene in treatment of refusers in ways that approximate trial entry. An advantage of this design is that it selects controls from the same pool as trial patients. Its major limitation is that refusal might correlate directly with the outcome in question. For example, patients who enter the trial might be more adherent.40 Furthermore, in trials for advanced disease, some patients who decline participation might reasonably opt for supportive care only, even if anticancer therapy might extend life. If so, then the predictably longer survival in trial participants compared with controls would reflect patient choice rather than trial effect. Eligible refuser studies that address these limitations, however, are valuable because they confront directly a crucial question: should we encourage patients with cancer to accept entry in a clinical trial on the basis of self-interest alone? The patient preference (or comprehensive cohort) trial,29 in which potential patients who decline randomisation may request direct assignment to one of the trial groups, is a variation on the eligible refuser design. However, since non-trial patients are treated in accordance with a protocol, such analyses are biased towards finding no effect. The fourth possibility is to compare prospectively a cohort of patients receiving non-trial care with a group of trial participants. Potential sources of non-trial cohorts include population-based cancer registries or patients seen at institutions not participating in the trial. Ideally, controls should meet all trial eligibility criteria. Analysts can further reduce confounding by matching on or adjusting statistically for known prognostic factors. As with other prospective designs, the study should not intervene in the care of non-trial patients. The main advantages of a prospective cohort are the ability to gather complete information on potential confounders and outcomes, and, by specifying the study hypothesis in advance, the avoidance of hindsight bias. The main disadvantage compared with the designs above is that, because trial participants and non-participants are 268 selected from different pools, the risk of baseline differences is great. The final (and most popular) option is retrospectively to compare a group of trial participants with a group of non-trial patients. This study design has important limitations, including difficulty in controlling for baseline imbalances between groups (frequently, important covariates are not recorded in non-trial patients) and the possibility of hindsight bias. Its main advantage, of course, is practicality. Retrospective cohort designs can be strengthened by a systematic method for identifying all appropriate controls; use of concurrent controls; restriction of controls to those who would have met eligibility criteria for the trial; careful adjustment for potential confounders; and inclusion of several trials and diseases to minimise the possibility of hindsight bias. Our results showed that most analyses, including all but one positive study, used retrospective cohort designs. Second, despite extensive evidence that trial patients constitute a prognostically favourable subset of those with the disease under study,26,27,41–43 few analyses controlled adequately for covariates that might provide alternative explanations for improved outcomes. No study controlled for comorbidity, only one controlled for socioeconomic status, and only four controlled for performance status. Third, only four of 21 retrospective cohort analyses restricted non-trial controls to those meeting trial eligibility criteria, which we view as a minimum standard for establishing comparability between groups. Fourth, no studies found that eligible patients who accepted trial entry had better outcomes than those who declined. Finally, studies did not consistently control for treatment differences, such as hospital volume or care in a cancer centre, that might correlate with improved outcomes. In addition to concerns about the validity of individual comparisons, we found reasons for caution in generalising from this body of evidence. Almost half the studies (including eight of 14 positive studies) involved patients diagnosed and treated in the 1970s and early 1980s, a time of rapid change in both cancer treatment and the organisation and delivery of cancer care. These studies might therefore not be relevant to contemporary oncology practice. In addition, several reports stated that better-than-expected outcomes in a particular population sparked the investigation of trial effects, thereby explicitly suggesting hindsight bias.13,19,20 Others retrospectively assessed diseases for which recent therapeutic progress was well established.11,12,14,21 Few studies included early-phase and non-randomised trials, and paediatric trials and haematological malignant diseases were disproportionately represented, especially in positive studies. Two previous reviews have considered trial effects in oncology. Stiller51 assessed cancer survival in relation to patterns of health-care delivery. He identified nine articles that assessed trial entry as an independent variable and, on the basis of six positive studies, suggested that trial participation was linked to increased survival. He did not, however, assess the quality of referenced studies or the validity of the comparisons. A more recent systematic review,24 part of a comprehensive assessment of ethical issues in randomised controlled trials,52 investigated whether trial participation is associated with improved outcomes. Ten of 14 articles in their study involved patients with cancer. We excluded three of these from our review, either because they did not compare trial with non-trial patients,28 or because they did not claim that comparisons between trial and non-trial patients were fair.26,27 The authors found significant THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com For personal use. Only reproduce with permission from The Lancet publishing Group. ARTICLES evidence for a trial effect in five of the seven remaining articles,9,11,13,14,19 and trends towards a trial effect in two.29,33 After acknowledging that there is little good quality evidence available, they cautiously concluded that there is (weak) evidence that well conducted trials tend to benefit the participants and do not seem (on average) to result in harm.24 There are at least two reasons why our conclusions differ somewhat from those of Braunholtz and colleagues.52 First, our study contains 17 new articles (18 comparisons). Of these, two provided evidence for a trial effect, seven suggested a trial effect with respect to selected subgroups or endpoints, and nine were negative. Second, inadequately controlled baseline differences between groups undermined our confidence in most studies. In their comprehensive assessment of randomised controlled trials,52 Braunholtz and colleagues seem to agree with these concerns. For example, when discussing confounding in the seven articles included in both their report and our own, they rated two as no difference/fully adjusted, two as small differences/partly adjusted, and three as no adjustment for large differences. Our review has several limitations. First, there is no search strategy that can reliably capture all relevant publications. As a result, we might have missed one or more pertinent studies. However, we have incorporated seven articles from previous reviews, and identified 17 articles that were not included in previous reports. To ensure completeness, we did a new literature search using conventional techniques, and searched retrospectively and prospectively from all primary articles, reviews, and position papers. Second, there are no accepted standards for assessing the quality of relevant studies. To limit subjectivity, we present the primary data concerning outcomes, study designs, and potential confounders (table 1). Third, any apparent differences in the proportion of studies showing a trial effect by age, time period or type of malignant disease should be viewed as descriptive rather than as rigorous tests of a priori hypotheses. In sum, we found little generalisable evidence to support the contention that trial participation directly improves outcomes for cancer patients. Until more convincing evidence for a trial effect is available, recruitment messages to patients considering trials should focus on their contribution to advances in treatment. We believe that patients, professionals, and third-party payers can recognise the crucial function of clinical trials in advancing treatment, and that de-emphasising direct benefits to patients need not compromise accrual or coverage. We remain optimistic that strong support for trials can flourish on the basis of their unquestioned role in improving options and outcomes for patients with cancer. References 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Contributors J Peppercorn, J Weeks, and S Joffe had the idea for this research. J Peppercorn, J Weeks, E F Cook, and S Joffe developed the conceptual model. J Peppercorn and S Joffe abstracted the data, with assistance from J Weeks. J Peppercorn and S Joffe wrote the report, and J Weeks and E F Cook revised it for important intellectual content. All authors approved the final version. 26 27 28 Conflict of interest statement The authors have no financial or other conflicts of interest with respect to the contents of this report. 29 Acknowledgments S Joffe received support from the US National Cancer Institute (K01 CA96872) during the period of this research. The authors received no specific funding for the study reported here. THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com 30 Gelber RD, Goldhirsch A. Can a clinical trial be the treatment of choice for patients with cancer? J Natl Cancer Inst 1988; 80: 886–87. Lara PN Jr, Higdon R, Lim N, et al. Prospective evaluation of cancer clinical trial accrual patterns: identifying potential barriers to enrollment. J Clin Oncol 2001; 19: 1728–33. Merz B. NCI seeks to boost study participation. JAMA 1988; 260: 1337. Antman K, Schnipper LE, Frei E 3rd. The crisis in clinical cancer research: third-party insurance and investigational therapy. N Engl J Med 1988; 319: 46–48. Bailes J. Managed care and oncology: the quality debate. J Clin Oncol 2000; 18: 705–07. American Federation of Clinical Oncologic Societies. Access to quality cancer care: consensus statement. J Clin Oncol 1998; 16: 1628–30. Children’s Oncology Group: About cancer clinical trials. http://www.childrensoncologygroup.org/ (accessed Dec 11, 2003). Poisson R. Fraud in breast-cancer trials. N Engl J Med 1994; 330: 1460. Karjalainen S, Palva I. Do treatment protocols improve end results? A study of survival of patients with multiple myeloma in Finland. BMJ 1989; 299: 1069–72. Greil R, Holzner B, Kemmler G, et al. Retrospective assessment of quality of life and treatment outcome in patients with Hodgkin’s disease from 1969 to 1994. Eur J Cancer 1999; 35: 698–706. Lennox EL, Stiller CA, Jones PH, Wilson LM. Nephroblastoma: treatment during 1970–73 and the effect on survival of inclusion in the first MRC trial. BMJ 1979; 2: 567–69. Meadows AT, Kramer S, Hopson R, Lustbader E, Jarrett P, Evans AE. Survival in childhood acute lymphocytic leukemia: effect of protocol and place of treatment. Cancer Invest 1983; 1: 49–55. Davis S, Wright PW, Schulman SF, et al. Participants in prospective, randomized clinical trials for resected non-small cell lung cancer have improved survival compared with nonparticipants in such trials. Cancer 1985; 56: 1710–18. Stiller CA, Draper GJ. Treatment centre size, entry to trials, and survival in acute lymphoblastic leukaemia. Arch Dis Child 1989; 64: 657–61. Boros L, Chuang C, Butler FO, Bennett JM. Leukemia in Rochester (NY). A 17-year experience with an analysis of the role of cooperative group (ECOG) participation. Cancer 1985; 56: 2161–69. Marubini E, Mariani L, Salvadori B, et al. Results of a breast-cancersurgery trial compared with observational data from routine practice. Lancet 1996; 347: 1000–03. Feuer EJ, Frey CM, Brawley OW, et al. After a treatment breakthrough: a comparison of trial and population- based data for advanced testicular cancer. J Clin Oncol 1994; 12: 368–77. Roy P, Vaughan Hudson G, Vaughan Hudson B, Esteve J, Swerdlow AJ. Long-term survival in Hodgkin’s disease patients: a comparison of relative survival in patients in trials and those recorded in population-based cancer registries. Eur J Cancer 2000; 36: 384–89. Stiller CA, Eatock EM. Survival from acute non-lymphocytic leukaemia, 1971–88: a population based study. Arch Dis Child 1994; 70: 219–23. Wagner HP, Dingeldein-Bettler I, et al. Childhood NHL in Switzerland: incidence and survival of 120 study and 42 non-study patients. Med Pediatr Oncol 1995; 24: 281–86. Stiller CA, Eatock EM. Patterns of care and survival for children with acute lymphoblastic leukaemia diagnosed between 1980 and 1994. Arch Dis Child 1999; 81: 202–08. Stiller CA, Benjamin S, Cartwright RA, et al. Patterns of care and survival for adolescents and young adults with acute leukaemia— a population-based study. Br J Cancer 1999; 79: 658–65. Lantos JD. The “inclusion benefit” in clinical trials. J Pediatr 1999; 134: 130–31. Braunholtz DA, Edwards SJ, Lilford RJ. Are randomized clinical trials good for us (in the short term)? Evidence for a “trial effect”. J Clin Epidemiol 2001; 54: 217–24. TROUT Review Group. How do the outcomes of patients treated within randomised control trials compare with those of similar patients treated outside these trials? http://hiru.mcmaster.ca/ebm/trout/ (accessed Dec 11, 2003). Antman K, Amato D, Wood W, et al. Selection bias in clinical trials. J Clin Oncol 1985; 3: 1142–47. Bertelsen K. Protocol allocation and exclusion in two Danish randomised trials in ovarian cancer. Br J Cancer 1991; 64: 1172–76. Medical Research Council. Duration of survival of children with acute leukaemia. Report to the Medical Research Council from the Committee on Leukaemia and the Working Party on Leukaemia in Childhood. BMJ 1971; 4: 7–9. Schmoor C, Olschewski M, Schumacher M. Randomized and nonrandomized patients in clinical trials: experiences with comprehensive cohort studies. Stat Med 1996; 15: 263–71. Dahlberg M, Glimelius B, Pahlman L. Improved survival and reduction in local failure rates after preoperative radiotherapy: 269 For personal use. Only reproduce with permission from The Lancet publishing Group. ARTICLES 31 32 33 34 35 36 37 38 39 40 evidence for the generalizability of the results of Swedish Rectal Cancer Trial. Ann Surg 1999; 229: 493–97. Dowling AJ, Czaykowski PM, Krahn MD, Moore MJ, Tannock IF. Prostate specific antigen response to mitoxantrone and prednisone in patients with refractory prostate cancer: prognostic factors and generalizability of a multicenter trial to clinical practice. J Urol 2000; 163: 1481–85. Mayers C, Panzarella T, Tannock IF. Analysis of the prognostic effects of inclusion in a clinical trial and of myelosuppression on survival after adjuvant chemotherapy for breast carcinoma. Cancer 2001; 91: 2246–57. Ward LC, Fielding JW, Dunn JA, Kelly KA, for the British Stomach Cancer Group. The selection of cases for randomised trials: a registry survey of concurrent trial and non-trial patients. Br J Cancer 1992; 66: 943–50. Winger MJ, Macdonald DR, Schold SC, Jr., Cairncross JG. Selection bias in clinical trials of anaplastic glioma. Ann Neurol 1989; 26: 531–34. Link MP, Goorin AM, Miser AW, et al. The effect of adjuvant chemotherapy on relapse-free survival in patients with osteosarcoma of the extremity. N Engl J Med 1986; 314: 1600–06. Cottin V, Arpin D, Lasset C, et al. Small-cell lung cancer: patients included in clinical trials are not representative of the patient population as a whole. Ann Oncol 1999; 10: 809–15. Schea RA, Perkins P, Allen PK, Komaki R, Cox JD. Limited-stage small-cell lung cancer: patient survival after combined chemotherapy and radiation therapy with and without treatment protocols. Radiology 1995; 197: 859–62. Burgers JA, Arance A, Ashcroft L, Hodgetts J, Lomax L, Thatcher N. Identical chemotherapy schedules given on and off trial protocol in small cell lung cancer: response and survival results. Br J Cancer 2002; 87: 562–66. Fergusson D, Aaron SD, Guyatt G, Hebert P. Post-randomisation exclusions: the intention to treat principle and excluding patients from analysis. BMJ 2002; 325: 652–54. Leventhal H, Nerenz DR, Leventhal EA, Love RR, Bendena LM. The behavioral dynamics of clinical trials. Prev Med 1991; 20: 132–46. 270 41 Chen CI, Skingley P, Meyer RM. A comparison of elderly patients with aggressive histology lymphoma who were entered or not entered on to a randomized phase II trial. Leuk Lymphoma 2000; 38: 327–34. 42 Hjorth M, Holmberg E, Rodjer S, Westin J, for the Myeloma Group of Western Sweden. Impact of active and passive exclusions on the results of a clinical trial in multiple myeloma. Br J Haematol 1992; 80: 55–61. 43 Rahman ZU, Frye DK, Buzdar AU, et al. Impact of selection process on response rate and long-term survival of potential high-dose chemotherapy candidates treated with standard-dose doxorubicincontaining chemotherapy in patients with metastatic breast cancer. J Clin Oncol 1997; 15: 3171–77. 44 Freedman B. Equipoise and the ethics of clinical research. N Engl J Med 1987; 317: 141–45. 45 Peto R, Collins R, Gray R. Large-scale randomized evidence: large, simple trials and overviews of trials. J Clin Epidemiol 1995; 48: 23–40. 46 Cella DF, Orav EJ, Kornblith AB, et al. Socioeconomic status and cancer survival. J Clin Oncol 1991; 9: 1500–09. 47 Zelen M. Theory and practice of clinical trials. In: Bast RC, Pollock RE, Kufe DW, Weichselbaum RR, Frei E, eds. Cancer Medicine. Hamilton, Ontario: B C.Decker Inc, 2000: 298–313. 48 Hillner BE, Smith TJ, Desch CE. Hospital and physician volume or specialization and outcomes in cancer treatment: importance in quality of cancer care. J Clin Oncol 2000; 18: 2327–40. 49 Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998; 279: 281–86. 50 Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H Jr. Publication bias and clinical trials. Control Clin Trials 1987; 8: 343–53. 51 Stiller CA. Centralised treatment, entry to trials and survival. Br J Cancer 1994; 70: 352–62. 52 Edwards SJ, Lilford RJ, Braunholtz DA, Jackson JC, Hewison J, Thornton J. Ethical Issues in the design and conduct of randomised controlled trials. Health Tech Assess 1998; 2: 1–131. THE LANCET • Vol 363 • January 24, 2004 • www.thelancet.com For personal use. Only reproduce with permission from The Lancet publishing Group.