Download Comparative Effectiveness Research: What Kind of Studies Do We

RESEARCH TOOLS AND ISSUES
Comparative Effectiveness Research: What Kind
of Studies Do We Need?
John Concato, MD, MS, MPH,*þ Peter Peduzzi, PhD,Þ§ Grant D. Huang, PhD,||¶
Timothy J. O’Leary, MD, PhD,||¶ and Joel Kupersmith, MD¶
Abstract: Comparative effectiveness research (CER) is increasingly
popular, yet discussions of its conduct and consequences often overlook
the extensive history of comparing different therapeutic options in
patient-oriented research. In particular, research in the Department of
Veterans Affairs (VA) has included a decades-long focus on generating
information that can enhance medical decision making and improve
health outcomes. Categories of such research include multisite randomized controlled trials (conducted by the Cooperative Studies Program) and observational studies involving either primary or secondary
data collection. As representative examples from cardiology, a landmark
VA clinical trial published in the 1970s evaluated the benefits of coronary
artery bypass grafting surgery among patients with angina; a VA trial
initiated in the 1990s, and identified formally as CER, demonstrated that
percutaneous coronary intervention is not superior to optimal medical
therapy; and a database investigation using information from the VA
electronic medical record system in the 2000s found that use of proton
pump inhibitor medication is associated with the attenuation of the
benefits of clopidogrel among patients hospitalized for acute coronary
syndrome. A review of these (and other) selected projects, based on
their type of study design, serves to highlight the strengths, limitations,
and potential of CER.
Key Words: comparative effectiveness research, randomized
controlled trials, observational studies, database research, meta-analysis,
US veterans
(J Investig Med 2010;58: 764Y769)
C
omparative effectiveness research (CER) evaluates the relative effectiveness of different options for treating a specific
medical condition in a selected population.1Y4 Comparative effectiveness research is not a single entity; it can focus on patientlevel or system-level elements of care, involving medications or
procedures, with or without considering costs. Regardless of its
scope, CER has the potential to generate information that can
From the *Clinical Epidemiology Research Center, and †Cooperative Studies
Program Coordinating Center, Veterans Affairs (VA) Connecticut Healthcare
System, West Haven; ‡Schools of Medicine, and §Public Health, Yale University, New Haven, CT; ||VA Cooperative Studies Program Central Office;
and ¶Office of Research and Development, Veterans Health Administration,
Washington, DC.
Received January 11, 2010, and in revised form March 29, 2010.
Accepted for publication March 29, 2010.
Reprints: John Concato, MD, MS, MPH, Clinical Epidemiology Research
Center, VA Connecticut Healthcare System, 950 Campbell Ave, Mailcode
151B, West Haven, CT 06516. E-mail: [email protected];
[email protected].
The authors have no financial disclosures to report.
The views in this article are those of the authors and do not reflect an
official position or policy of the Department of Veteran Affairs.
Dr. Kupersmith serves on the Federal Coordinating Council for
Comparative Effectiveness Research.
Copyright * 2010 by The American Federation for Medical Research
ISSN: 1081-5589
DOI: 10.231/JIM.0b013e3181e3d2af
764
enhance medical decision making and improve health outcomes,
but concerns exist regarding its conduct and consequences.5,6
Recent attention to CER has discussed mechanisms to
implement findings into practice,7,8 potential impacts on personalized medicine,9 and disagreements regarding funding.5
Less attention has been given to strategies of the design and
conduct of such research. Whereas discussions of nonscientific
issues are ongoing, an internal look at the methods of CER is
also warranted. In addition, selected examples from the Department of Veterans Affairs (VA) can illustrate one organization’s experience in conducting CER.
HISTORICAL OVERVIEW
Comparative effectiveness research can be considered a
subset of patient-oriented research that examines available
therapeutic options among particular patients to determine relevant health outcomes. As such, CER is a relatively recent term,
but the idea of evaluating Breal world[ treatment options dates
back to antiquity. As a centuries-old example, James Lind
compared various treatments for scurvy in the mid-1700s10;
providing citrus fruits led to resolution of symptoms, whereas
vinegar, cider, and other remedies did not. (This example also
demonstrates that valid associations can be identified before a
full explanation of underlying mechanisms; the isolation of vitamin C and a biological understanding of scurvy did not occur
until almost 200 years later.) In the 20th century, the so-called
comparative effectiveness of 2 penicillin treatment schedules in
pneumococcal infections of mice11 was evaluated in 1948, but
the phrase Bdosing study[ might be applied to the same research
today. In the same era, the first rigorous studies of the treatment
of tuberculosis in humans involved the US Veterans Administration, Army, and Navy collaborating to evaluate streptomycin
among veterans and soldiers whose condition was diagnosed
after World War II.12 This observational study12 predated, and
found results similar to, a randomized trial of streptomycin
conducted in the United Kingdom by the Medical Research
Council.13 (The latter study13 is acknowledged as the first randomized trial reported in modern clinical medicine.)
In the last half-century, terminology for research design has
evolved, based specifically on the meaning and application of the
words Befficacy[ and Beffectiveness.[ Attributable in large part
to the passage (in 1962) of the Kefauver-Harris Amendment14 to
the Federal Food, Drug, and Cosmetic Act and developed in the
wake of thalidomide causing phocomelia and other birth defects,
this amendment required proof of efficacy as part of the drug
approval process in the United States. The subsequent adoption
of phase I to III (and later phase 0 and phase IV) schema for
clinical trials emphasizes the evaluation of new drugs versus
placebo rather than with existing agents. This system also places
more emphasis on internal validityVbased on Befficacy trials[
that tend to have restrictive inclusion, exclusion, and treatment
criteriaVthan on external validity derived from Beffectiveness
trials[ that tend to have patient populations and (nonplacebo)
Journal of Investigative Medicine
&
Volume 58, Number 6, August 2010
Copyright © 2010 American Federation for Medical Research. Unauthorized reproduction of this article is prohibited.
Journal of Investigative Medicine
&
Volume 58, Number 6, August 2010
interventions more representative of clinical practice. Implementation of research results after the publication of trials (or
other types of studies) is another relatively recent consideration.
By any reasonable definition, numerous studies of comparative effectiveness have been done in recent decades, but
usually without being identified as such. Interestingly, the formal
term Bcomparative effectiveness research[ has been popularized
only in the past several years, related more to policy15 and
economic16 considerations than to scientific issues themselves.
Whether an article is classified as comparative effectiveness is
variable, depending on the authors and the viewpoint of
reviewers and journal editors. A recent review4 classified 689
studies as CER based on a search of clinicaltrials.gov,17 a database of clinical trials, and HSRProj,18 a database of health services research projects.
The visibility of CER was increased by an appropriation of
$1.1 billion in the American Recovery and Reinvestment Act of
2009.19 Although this amount is less than 1% of the $150 billion
directed toward health care overall, the support represents a
major infusion of funding for CER.19 Even before this new
initiative, however, CER had become a prominent component of
research on health care. As the purposes and implications of
CER are debated in medical and nonmedical venues, a major
unanswered question is what kind of CER will be considered
worthwhile? For example, in the current era of evidence-based
medicine, a hierarchy of research designs might be promoted for
CER that favors randomized trials. Conversely, the relatively
short timeframe to spend Bstimulus funds[ encourages the use of
secondary sources of data that can be analyzed promptly.
CATEGORIES OF RESEARCH DESIGN
From a methodological perspective, randomized controlled
trials can address comparative effectiveness and are arguably
underused for this purpose. Specifically, studies funded by the
pharmaceutical industry represent most trials in the United
States but are usually designed to evaluate efficacy and not effectiveness. As mentioned, randomized trials evaluating efficacy
(or explanatory clinical trials) often compare an intervention to
placebo among patients with little or no comorbid illness to
address whether an intervention can work under ideal circumstances. Early-phase clinical trials and applications to the Food
and Drug Administration for licensure, including studies conducted or supported directly by the pharmaceutical industry, tend
to fall into this category.
Randomized trials evaluating effectiveness (or Bpractical
clinical trials[) determine whether an intervention actually
works under more representative, or real-world, conditions.
These studies compare an intervention to a standard-of-care
among patients representative of a target population. Randomized trials have often been criticized as lacking generalizability,20
but effectiveness trials are more likely to apply to general medical practice than efficacy trials. Specific methodological strategies, such as adaptive designs, represent novel features of trials
that can improve their utility.
Random allocation promotes a balance of prognostic factors among groups being comparedVprecluding a major source
of biasVbut randomized trials are not always feasible, ethical, or
necessary. Well-designed observational cohort studies and casecontrol studies collecting primary data constitute another approach to CER. These studies have enhanced generalizability in
representative patient populations (especially compared with
efficacy trials), yet can produce results similar to randomized
trials when sufficient attention is given to sources of bias.21,22 In
this context, an often-cited example of hormone replacement
* 2010 The American Federation for Medical Research
Comparative Effectiveness Research
therapy and subsequent health outcomes has generated misconceptions regarding the limitations of observational studies. In
reality, when differences in patient populations, treatment regimens, and duration of follow-up are taken into account, data
from randomized and observational studies on this topic (and
others) are quite consistent.23Y25
So-called database research, involving information obtained for clinical or administrative purposes during health
care delivery, is another (observational) method for evaluating
comparative effectiveness. The generalizability of results can
become unrivaled, given assessments of large and representative
populations of health care recipients. Conversely, problems include the lack of randomization and concerns with data quality
(such as BAre diagnoses listed in an administrative database
accurate?[). Carefully validated analytic strategies are therefore critically important; comparison of results with randomized
trials and other observational studies can help to enhance
interpretation.
These categories of research design26 are not exhaustive,
but they do account for common, primary strategies of CER.
Secondary strategies include systematic reviews and statistical
modeling. Rather than obtaining data as observations on
individual participants, systematic reviews (including metaanalyses) use information from previously completed studiesV
but not without limitations.27 BSecondary[ statistical modeling
uses novel techniques to assess cause-effect associations, typically with a data set that had already been collected and analyzed. Although most patient-oriented research investigations
use a statistical model per se, this type of statistical modeling
goes beyond original hypotheses and planned calculations to
explore associations based on statistical theory and corresponding assumptionsVthat may or may not be justified.
EXAMPLES FROM THE DEPARTMENT OF
VETERANS AFFAIRS (VA)
The VA has conducted CER using various methodologies
for many years. For example, the VA Cooperative Studies Program has been recognized for producing Bnumerous studies
designed to support clinical and policy decisions.[20 The Cooperative Studies Program28,29 has generated more than 175
large-scale, multisite investigations, mostly as randomized trials
of comparative effectiveness. Key elements of the Program include methodological, technical, and administrative personnel in
a national infrastructure of specialized centers for conducting collaborative research. The Program also has established
guidelines30 for the design, conduct, and analysis of its multisite
studies.
Selected trials over 4 decades have shown (Table 1) that
coronary artery bypass surgery (vs medical therapy) improves
survival among patients with left main coronary artery disease,31
steroid therapy (added to usual care) does not reduce mortality
from septic shock,32 watchful waiting (vs transurethral resection)
is a safe alternative among men less bothered by symptoms from
benign prostatic hyperplasia,33 and percutaneous coronary intervention (vs optimal medical therapy) does not reduce the risk
of cardiovascular events or death in stable angina.34 These trials
were selected with the main goal of highlighting both a longstanding BVA experience[ and a broad scope of topics and
methods, such as medical versus surgical interventions and usual
care versus optimal therapy as a comparator. In conducting CER,
and looking beyond the strengths or limitations of any individual
study, the Cooperative Studies Program has been described by
non-VA authors as Ban important source of practical clinical
trials in the United States.[20
765
Copyright © 2010 American Federation for Medical Research. Unauthorized reproduction of this article is prohibited.
Concato et al
Journal of Investigative Medicine
&
Volume 58, Number 6, August 2010
TABLE 1. Representative Examples of CER Conducted by VA, Organized According to Type of Study Design
Study Design and Clinical Context
Randomized controlled trials:
Patients with angina pectoris found to have a significant
lesion in the left main coronary artery
Patients with clinical signs of systemic sepsis receiving
intensive care
Ambulatory men with symptoms of benign prostatic hyperplasia
Patients with evidence of myocardial ischemia and stable
coronary artery disease
Observational studies involving primary data collection:
Older patients undergoing major noncardiac surgery
Database (observational) investigations of archival information:
Patients after discharge from a hospital admission for
acute coronary syndrome
Systematic reviews (including meta-analyses):
Patients with coronary artery disease considered eligible for
coronary revascularization
Secondary statistical modeling:
Ambulatory patients receiving treatment for alcohol dependence
Observational studies with primary data collection also
contribute to CER in the VA, originating from the Cooperative
Studies Program (via epidemiology centers), individual investigators, and other sources. An example is the National Surgical
Quality Improvement Program (NSQIP) that systematically
measures and seeks to improve risk-adjusted surgical outcomes
using validated methods. The NSQIP35 is based in the VA Health
Services Research and Development Service and was expanded
to the private sector by the American College of Surgeons.36 In a
collaborative NSQIP project,37 abnormally low or high preoperative hematocrit levels were found to be associated with
increased 30-day mortality and cardiac events after noncardiac
surgery among elderly patients (Table 1). These results inform
CER by identifying a potentially modifiable risk factor for
developing targeted treatments, with a goal of improving
outcomes.
Health services research projects frequently use archived
database information only, as in a recent evaluation38 of an adverse interaction between proton-pump inhibitor medications
and clopidogrel after hospitalization for acute coronary syndrome (Table 1). This study is pertinent in merging and analyzing national-level VA data collected for other purposes, with
careful attention given to possible sources of bias and with
multiple sensitivity analyses. The Department of Veterans
Affairs Health Services Research also supports the Quality Enhancement Research Initiative,39 designed to translate study
findings into clinical practice via implementation research.7,8
As an example of a systematic review (Table 1), a report40
evaluated the comparative effectiveness of percutaneous coronary interventions and coronary artery bypass graft surgery
(CABG); the focus was not limited to VA-based studies. After a
structured literature search, results from 23 randomized trials
were identified, reviewed, and analyzed. The authors concluded
that compared with percutaneous coronary interventions, CABG
was more effective in relieving angina, and led to fewer revascularizations, but had a higher risk for procedural stroke;
Bsurvival to 10 years was similar[ for both procedures.40
766
Information to Improve Patient Care (Year)
Coronary artery bypass grafting surgery improves survival compared
with medical therapy (1976).30
Steroid therapy added to usual care is not effective in reducing
mortality among patients with septic shock (1987).31
Watchful waiting is a safe alternative to surgery among men less
bothered by urinary symptoms (1995).32
Percutaneous coronary intervention is not superior to optimal
medical therapy (2007).33
Abnormally low or high preoperative hematocrit levels increase
mortality after surgery (2007).36
Proton pump inhibitor medications are associated with attenuation of
benefits of clopidogrel (2009).37
Coronary artery bypass surgery improves angina and decreases
revascularizations versus percutaneous intervention, but with
higher risk of stroke and similar survival (2007).39
Naltrexone may reduce levels of Bconsistent[ drinking even if time
to first drink is not improved (2007).42
As an example of Bsecondary[ statistical modeling, data
were obtained from a VA Cooperative Study41 and a similar trial
conducted among nonveterans,42 evaluating naltrexone in the
treatment of alcohol dependence (Table 1). Although the original trials were null in terms of not finding an improvement
in outcomes, such as the time to the first day of drinking a
threshold amount of alcohol, questions arose as to whether the
intervention might have benefited certain participants. A followup study43 used longitudinal models with Bfixed polynomial
trends over time[ and other statistical methods; the analyses
generated estimates of the proportion of participants classified
(on a given day) as abstaining, drinking sporadically, or drinking
consistently. The authors reported that Bnaltrexone may have
a clinically meaningful effect for alcohol-dependent patients[
regarding consistent drinking.43
These VA-based (and other) examples serve collectively
to characterize a methodological spectrum of CER. Within the
Veterans Health Administration, such research has changed
health policy and influenced clinical care,44 as with recommendations and treatment patterns for anemia after a study
comparing subcutaneous and intravenous administration of
erythropoietin.45,46 Despite such examples to support an argument that research can and does directly change practice patterns, a counterargument would include the previously cited
example31 of a randomized trial evaluating CABG surgery for
patients with angina. Approximately 10 years after publication
of the main report, questions still lingered as to whether the study
was valid and should influence the care of patients47; the answers
eventually emerged as Byes[ and Byes,[ respectively.48 Ultimately, the complex issue of how to change physicians’ behavior
is beyond the scope of this report, but conducting and understanding rigorous and relevant CER is an important component
of the process.
A related issue involves the patient population receiving
health care at VA medical facilities, representing the source
population for VA-based research. For example, although the
proportion of women in the VA is increasing, their representation
* 2010 The American Federation for Medical Research
Copyright © 2010 American Federation for Medical Research. Unauthorized reproduction of this article is prohibited.
Journal of Investigative Medicine
&
Volume 58, Number 6, August 2010
still falls far short of the one-to-one balance in the community.
Similarly, veterans who receive VA health care tend to come
from lower socioeconomic strata, often linked to private health
insurance being less available or affordable to them. Thus, a
question of generalizability (external validity) does exist when
VA studies are considered, yet this concern is sometimes extrapolated inappropriately to cast doubt on the internal validity of
a study, without giving due consideration to the methods involved. In other situations, this concern is not applicableYfor
example, when VA studies are done in collaboration with the
National Institutes of Health49 or as international collaborations.34
Overall, if examined as a body of knowledge, VA studies
in CER have been considered reliable and relevant in improving care delivered to the general (nonveteran) population.20
In addition, an informal review (data not shown) indicates that
VA research has contributed to a sizeable majority of the 100
categories listed in the Institute of Medicine’s recent report on
national priorities for CER.50
GENERAL CONSIDERATIONS
The VA is distinctive in supporting virtually all of the types
of research design mentioned in this report and in promoting
implementation of research results. Other organizations and
agencies might embrace a hierarchy of research design so that
randomized trials and meta-analyses are emphasized as the best
forms of evidence. (Under such circumstances, the VA Cooperative Studies Program should be considered an important repository of randomized studies.) At the same time, however, any
approach that imposes a research design hierarchy should be
understood as placing a Bfilter[ between the generation of evidence and its interpretation, bypassing the question of whether
such filtering is appropriate or how well it is done. This problem
can be subtle, as when Bdefinitive randomized controlled trials[
versus Bweaker forms of evidence, such as nested case-control or
well-designed cohort studies[ are presented as self-evident
truths in discussions of CER.51 Conversely, other organizations
and agencies might use only secondary sources of information,
perhaps underestimating the corresponding challenges, for reasons including database research can be done more readily.
A major take-home message of this report is that types of
study design are an important feature of CER, but hierarchies of
research design should not become a distraction in the process
of scientific inquiry. A concurring opinion was articulated as:
BThis is not a plea to abandon RCTs and replace them with
observational studies. [I] Rather, it is a plea to investigators to
continue to develop and improve their methods; to decision
makers to avoid adopting entrenched positions about the nature
of evidence; and for both to accept that the interpretation of
evidence requires judgment.[52 A similar viewpoint appeared
in a report on CER,53 with statements including Btraditional
Fhierarchies of evidence_ are overly simplistic and should not
necessarily guide the implementation of CER[; Ba range of
methods are important in CER[; and Bthe right approach to a
given CER study depends on the circumstances.[ (Of note, a
recent descriptive report54 suggests that randomized trials constitute most of the published CER studies.)
Another take-home message involves the attributes needed
for an organization to conduct CER. The VA is certainly not
alone in its capabilities and commitment to CER,1Y4,55 and it also
is not the largest health research organization in the United
States. Yet, the unique position of VA research in the national
health care environment includes at least 4 elements that have
sustained CER: a comprehensive and relatively stable infrastructure, support for methodological advances, research ideas
* 2010 The American Federation for Medical Research
Comparative Effectiveness Research
arising from clinical and administrative questions, and a commitment to excellence in research on clinical care. Studies of
comparative effectiveness in the VA are also facilitated by an
electronic medical record system, a uniform reimbursement
mechanism, and a national health care delivery network. In
particular, few national-level organizations have a similar ability
to integrate clinical, laboratory, pharmacy, service use, and
outcome data in one health care context.
The VA also serves an example of an organization that can
collaborate with the private sector and yet avoid common conflicts of interest, using a philosophy of Bscience comes first.[ As
exemplified by the Cooperative Studies Program, strict policies
govern the (intramural) design, conduct, interpretation, and
regulatory aspects of a study, whereas unrestricted research
grants (or in-kind support, such as medications) from industry
are acceptable. Cooperative Research and Development Agreements (CRADAs), authorized under the Federal Technology
Transfer Act, represent one strategy used to safeguard the
interests of the VA and veterans while being sensitive to the
perspective of industry regarding commercially valuable information. Research conducted under a Cooperative Research and
Development Agreement concurs with federal statutes, regulations, and policies and, at the same time, protects background
inventions, trade secrets, and confidential information. A representative statement would mention that B[the industry sponsor]
had no role in the design of the study, collection, and analysis of
data, or preparation and approval of the manuscript.[ Given the
paucity of incentives for the pharmaceutical or medical device
industry to conduct CER on their own, this category of research
represents an important opportunity to advance knowledge. Finally, details pertaining to cost-effectiveness analysis are beyond
the scope of this paper, but a decision not to focus on costs in
studies of CER is related more to policy than to science. Interestingly, the integrated nature of the VA health care system is a
potential advantage in terms of the ability to assess the costs of
treatment options accurately on a head-to-head basis.
CONCLUSIONS
If a national-level entity dedicated to CER1,5,15,51,56 is
created, either as a public or private (or hybrid) agency, the
culture of such an organization would likely influence the selection of types of study design, but the methodological issues
raised in this report should be considered. Rather than adhering
to a hierarchy of research design, comparative effectiveness research should be conducted and evaluated using sound clinical,
epidemiological, and statistical judgment. Beyond these attributes of CER, and in the broader context of health care reform,57,58 careful attention should be given to methods of
patient-oriented research. Although not unique to CER, important steps include identifying an important research question;
developing a cogent study design among feasible options; collecting relevant, high-quality data; and performing appropriate
statistical analyses. Ultimately, precise ratings of interventions
based on CER are arguably neither feasible nor desirable, and the
source of CER is less pertinent than creating and disseminating
rigorous research. The utility of a particular CER study should
be judged in a peer-reviewed, scientific context and applied to
individual patients by competent and caring clinicians.
REFERENCES
1. IOM (Institute of Medicine). 2007. Learning What Works Best:
The Nations Need for Evidence on Comparative Effectiveness in
Health Care. Available at: http://www.iom.edu/ebm-effectiveness.
Accessed March 24, 2010.
767
Copyright © 2010 American Federation for Medical Research. Unauthorized reproduction of this article is prohibited.
Concato et al
Journal of Investigative Medicine
&
Volume 58, Number 6, August 2010
2. Kupersmith J, Sung N, Genel M, et al. Creating a new structure for
research on health care effectiveness. J Investig Med. 2005;53:67Y72.
28. Department of Veterans Affairs Cooperative Studies Program.
Available at: www.csp.research.va.gov/. Accessed March 24, 2010.
3. Lauer MS. Comparative effectiveness research: the view from the
NHLBI. J Am Coll Cardiol. 2009;53:1084Y1086.
29. Henderson WG, Lavori PW, Peduzzi P, et al. Biostatistics in clinical
trials. In: Redmond CK, Colton T, eds. Cooperative Studies
Program, US Department of Veterans Affairs. New York:
John Wiley & Sons, Ltd; 2001:99Y115.
4. Hove E, Pittman P. A First Look at the Volume and Cost of Comparative
Effectiveness Research in the United States. Washington, DC:
AcademyHealth; June 2009.
5. Avorn J. Debate about funding comparative-effectiveness research.
N Engl J Med. 2009;360:1927Y1929.
6. Selker HP. Comparative effectiveness research: medical practice,
payments, and politics: the need to retain standards of medical research.
J Gen Int Med. 2009;24:776Y778.
7. Naik AD, Petersen LA. The neglected purpose of
comparative-effectiveness research. N Engl J Med.
2009;360:1929Y1931.
8. Alexander GC, Stafford RS. Does comparative effectiveness have a
comparative edge? JAMA. 2009;310:2488Y2490.
9. Garber AM, Tunis SR. Does comparative-effectiveness research
threaten personalized medicine? N Engl J Med. 2009;360:1925Y1927.
10. Lind J. A Treatise of the Scurvy in Three Parts. London: A. Millar; 1753.
11. Gibson CD. Comparative effectiveness of two penicillin treatment
schedules in pneumococcal infections of mice. Proc Soc Exp Biol Med.
1948;67:278Y280.
12. Streptomycin Committee. The effects of streptomycin on tuberculosis
in man. JAMA. 1947;5135:634Y641.
13. Medical Research Council. Streptomycin treatment of pulmonary
tuberculosis. Br Med J. 1948;2:769Y782.
14. Kefauver-Harris Amendments, 87 U.S. Constitution, Section 2 (1962).
15. Wilensky GR. Developing a center for comparative effectiveness
information. Health Aff. 2006;25:w572Yw585.
16. Neumann PJ, Rosen AB, Weinstein MC. Medicare and
cost-effectiveness analysis. N Engl J Med. 2005;353:1516Y1522.
17. ClinicalTrials.gov: a service of the U.S. National Institutes of Health.
Available at: http://www.clinicaltrials.gov/. Accessed March 24, 2010.
18. HSRProj (Health Services Research Projects in Progress), National
Information Center on Health Services Research and Health Care
Technology (NICHSR). Available at www.nlm.nih.gov/hsrproj/.
Accessed March 24, 2010.
19. Steinbrook R. Health care and the American Recovery and
Reinvestment Act. N Engl J Med. 2009;360:1057Y1060.
20. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing
the value of clinical research for decision making in clinical and
health policy. JAMA. 2003;290:1624Y1632.
21. Concato J, Shaw N, Horwitz RI. Randomized, controlled trials,
observational studies, and the hierarchy of research designs.
N Engl J Med. 2000;342:1887Y1892.
22. Benson K, Hartz AJ. A comparison of observational studies and
randomized, controlled trials. N Engl J Med. 2000;342:1878Y1886.
23. Concato J, Horwitz RI. Beyond randomised versus observational
studies. Lancet. 2004;363:1660Y1661.
24. Grodstein F, Clarkson TB, Manson JE. Understanding the divergent
data on postmenopausal hormone therapy. N Engl J Med.
2003;348:648Y650.
25. Garbe E, Suissa S. Hormone replacement therapy and acute coronary
outcomes: methodological issues between randomized and
observational studies. Hum Reprod. 2004;19:8Y13.
26. Concato J. Overview of research design in epidemiology. Journal of
Law and Policy. 2004;12:489Y507.
27. Feinstein AR. Meta-analysis: statistical alchemy for the 21st century.
J Clin Epidemiol. 1995;48:71Y79.
768
30. Guidelines for the planning and conduct of cooperative studies.
Washington, DC: Office of Research and Development,
Department of Veterans Affairs. May 2007. Available at:
http://www.csp.research.va.gov/update/guide.pdf.
Accessed March 24, 2010.
31. Takaro T, Hultgren HN, Lipton MJ, et al. The VA cooperative
randomized study of surgery for coronary arterial occlusive disease II.
Subgroup with significant left main lesions. Circulation.
1976;54(suppl 6):III107YIII117.
32. The Veterans Administration Systemic Sepsis Cooperative Study
Group. Effect of high-dose glucocorticoid therapy on mortality in
patients with clinical signs of systemic sepsis. N Engl J Med.
1987;317:659Y665.
33. Wasson JH, Reda DJ, Bruskewitz RC, et al. A comparison of
transurethral surgery with watchful waiting for moderate symptoms of
benign prostatic hyperplasia. The Veterans Affairs Cooperative Study
Group on Transurethral Resection of the Prostate. N Engl J Med.
1995;332:75Y79.
34. Boden WE, O’Rourke RA, Teo KK, et al. Optimal medical therapy with
or without PCI for stable coronary disease. N Engl J Med.
2007;356:1503Y1516.
35. Khuri SF, Daley J, Henderson W, et al. The Department of Veterans
Affairs’ NSQIP: the first national, validated, outcome-based,
risk-adjusted, and peer-controlled program for the measurement and
enhancement of the quality of surgical care. National VA Surgical
Quality Improvement Program. Ann Surg. 1998;228:491Y507.
36. History of the ACS NSQIP (American College of Surgeons,
National Surgical Quality Improvement Program). Available at:
http://www.acsnsqip.org/main/about_history.asp.
Accessed March 24, 2010.
37. Wu WC, Schifftner TL, Henderson WG, et al. Preoperative hematocrit
levels and postoperative outcomes in older patients undergoing
noncardiac surgery. JAMA. 2007;297:2481Y2488.
38. Ho PM, Maddox TM, Wang L, et al. Risk of adverse outcomes
associated with concomitant use of clopidogrel and proton pump
inhibitors following acute coronary syndrome. JAMA. 2009;301:937Y944.
39. VA Quality Enhancement Research Initiative. Available at:
http://www.queri.research.va.gov/. Accessed March 24, 2010.
40. Bravata DM, Gienger AL, McDonald KM, et al. Systematic review:
the comparative effectiveness of percutaneous coronary interventions
and coronary artery bypass graft surgery. Ann Intern Med.
2007;147:703Y716.
41. Krystal JH, Cramer JA, Krol WF, et al. Naltrexone in the treatment of
alcohol dependence. N Engl J Med. 2001;345:1734Y1739.
42. O’Malley SS, Sinha R, Grilo CM, et al. Naltrexone and cognitive
behavioral coping skills therapy for the treatment of alcohol drinking
and eating disorder features in alcohol-dependent women: a randomized
controlled trial. Alcohol Clin Exp Res. 2007;31:625Y634.
43. Gueorguieva R, Wu R, Pittman B, et al. New insights into the efficacy of
naltrexone based on trajectory-based reanalyses of two negative
clinical trials. Biol Psychiatry. 2007;61:1290Y1295.
44. Kizer KW, Demakis JG, Feussner JR. Reinventing VA Health Care:
systematizing quality improvement and quality innovation. Med Care.
2000;38(suppl I):7Y16.
45. Kaufman JS, Reda DJ, Fye CL, et al. Subcutaneous compared with
intravenous epoetin in patients receiving hemodialysis.
N Engl J Med. 1998;339:578Y583.
* 2010 The American Federation for Medical Research
Copyright © 2010 American Federation for Medical Research. Unauthorized reproduction of this article is prohibited.
Journal of Investigative Medicine
&
Volume 58, Number 6, August 2010
46. Hynes DM, Stroupe KT, Kaufman JS, et al. Adherence to guidelines
for ESRD anemia management. Am J Kidney Dis.
2006;47:455Y461.
47. Hultgren H, Takaro T, Detre K, et al. Criticisms of the study.
Prog Cardiovasc Dis. 1986;28:391Y396.
48. Yusuf S, Zucker D, Peduzzi P, et al. Effect of coronary artery bypass
graft surgery on survival: overview of 10-year results from randomised
trials by the Coronary Artery Bypass Graft Surgery Trialists
Collaboration. Lancet. 1994;334:563Y570.
49. Palevsky PM, Zhang JH, O’Connor TZ, et al. Intensity of renal
support in critically ill patients with acute kidney injury.
N Engl J Med. 2008;359:7Y20.
50. IOM (Institute of Medicine). Initial National Priorities for Comparative
Effectiveness Research. Washington, DC: The National Academies
Press; 2009.
51. VanLare JM, Conway PH, Sox HC. Five next steps for a new national
program for comparative-effectiveness research. N Engl J Med.
2010;362:970Y973.
* 2010 The American Federation for Medical Research
Comparative Effectiveness Research
52. Rawlins M. De testimonio: on the evidence for decisions about the
use of therapeutic interventions. Lancet. 2008;372:2152Y2161.
53. Tunis ST. Strategies to improve comparative effectiveness research
methods and data infrastructure. In: Implementing Comparative
Effectiveness Research: Priorities, Methods, And Impact. Washington,
DC: Brookings Institution; 2009:35Y53.
54. Hochman M, McCormick D. Characteristics of published comparative
effectiveness studies of medications. JAMA. 2010;303:951Y958.
55. Steinbrook R. Saying no isn’t NICEVthe travails of Britain’s National
Institute for Health and Clinical Excellence. N Engl J Med.
2008;359:1977Y1981.
56. American College of Physicians [Position Paper]. Information on
cost-effectiveness: an essential product of a national comparative
effectiveness program. Ann Intern Med. 2008;148:956Y961.
57. Brook RH. The science of health care reform. JAMA.
2009;301:2486Y2487.
58. Wilensky GR. Health care reformVwhere do we go from here?
N Engl J Med. 2010;362:e36(1Y3).
769
Copyright © 2010 American Federation for Medical Research. Unauthorized reproduction of this article is prohibited.