Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
REVIEW Outcomes Research in Oncology: History, Conceptual Framework, and Trends in the Literature Stephanie J. Lee, Craig C. Earle, Jane C. Weeks “Outcomes research” is a commonly used term, but what does it really mean? In this review article, we 1) briefly review the historic background of outcomes research, paying particular attention to the social and political movements that helped shape the field; 2) present a conceptual framework to help classify the major areas of research and provide a working definition of outcomes research; and 3) review the oncology literature in the English language from 1966 through 1998 to examine temporal trends and characterize the body of work being presented to the practicing oncology community. We conclude that outcomes research is a broad concept, which, in its current usage, describes an array of distinct types of research. However, common themes are apparent when outcomes research is viewed in the context of its historic origins and is contrasted with other established disciplines, especially clinical trials. Our literature review shows that outcomes studies are increasing in absolute numbers, in relative proportion of the oncology literature, and in quality. We suggest that as different branches of investigation develop their own literature and methodology—in effect, outgrowing the generic label of outcomes research— they become identified by separate, more precise terms. [J Natl Cancer Inst 2000;92:195–204] What constitutes “outcomes research”? Although it is increasingly common to hear people refer to outcomes research, there is no consensus on its precise definition. At one extreme, all results are outcomes; thus, all research is outcomes research. However, a narrower body of work with a specific historic context is generally implied. The 1996 Medical Outcomes & Guidelines Sourcebook (1) defines outcomes research in its glossary as “a term originally used to describe a particular line of health services research that focused on identifying variations in medical procedures and associated health outcomes. The term has since been applied to a wide variety of vaguely associated activities and no longer has a clearly identifiable meaning.” Taxonomic evolution is not unique to outcomes research. Parallels can be made with the biologic sciences where the discovery of new species, techniques, and paradigms frequently extends or rearranges nomenclature. In basic science, researchers could have stated 10 years ago that they worked in molecular biology, and that was explanation enough. Now molecular methods are so ubiquitous that further detail is required to characterize a research effort. Bench scientists describe their fields to specific audiences by the topics (signal transduction), methods (genetic engineering), or model systems (zebra fish) that best describe their pursuits. The same phenomenon is now occurring in outcomes research. For some, outcomes research could be identified by a question or topic (quality of care), a method (decision analysis), or a data source (administrative da- tabases). Indeed, the field has evolved such that the term “outcomes research” no longer describes a particular activity, but rather it describes an array of related yet distinct fields of inquiry. To achieve some insight into the current state of the science of outcomes research in oncology, we believe that it is helpful to review three separate but related topics. First, we present the background behind the outcomes movement and the forces that shaped the field and its nomenclature. This historic context helps to explain the current diversity of research topics classified under outcomes research. Second, we present a conceptual framework for classifying outcomes research and highlight areas of the field’s evolution. We struggled to develop a clear, comprehensive, and cohesive definition of outcomes research. Ultimately, we decided that outcomes studies are distinguished both by what they are and what they are not. This definition is presented graphically and developed within the second section. Finally, we review oncology outcomes studies published in selected general, specialty, and methodology journals during the past 33 (1966–1998 inclusive) years. This literature review supports the proposed conceptual framework, examines trends in research emphasis over time, and provides an opportunity to see how outcomes research is being presented to the practicing oncology community. HISTORIC PERSPECTIVE Early Work Medicine has long been cautioned about the need to measure the results of its interventions. Early proponents of outcomes measurement focused on its value as an indicator of quality. In 1914, Ernst A. Codman, a surgeon, noted that hospitals were reporting the number of patients treated but not how many of the patients benefited from treatment. He argued (2) that all hospitals should produce a report “showing as nearly as possible what are the results of the treatment obtained at different institutions. This report must be made out and published by each hospital in a uniform manner, so that comparison will be possible.” Furthermore, he suggested that all worthy products and activities of Affiliations of authors: Center for Outcomes and Policy Research, Department of Adult Oncology, Dana-Farber Cancer Institute, and Department of Medicine, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA. Correspondence to: Stephanie J. Lee, M.D., M.P.H., Center for Outcomes and Policy Research, Dana-Farber Cancer Institute, 44 Binney St., Boston, MA 02115 (e-mail: [email protected]). Reprint requests to: Jane C. Weeks, M.D., M.Sc., Center for Outcomes and Policy Research, Dana-Farber Cancer Institute, 44 Binney St., Boston, MA 02115. See “Notes” following “References.” © Oxford University Press Journal of the National Cancer Institute, Vol. 92, No. 3, February 2, 2000 REVIEW 195 a hospital, from the physicians trained there, to the research conducted, to the care delivered, depended on the premise that ill patients were deriving benefit from the therapies that they received. Unless results were measured, how could one be sure of benefits conferred? The field was relatively quiet for the next several decades. According to published histories of outcomes research (3–5), the next major figure was Avides Donabedian. He returned to Codman’s concept of quality in 1966 (6), coining the term “outcome” as part of his structure, process, and outcome paradigm for quality assessment. Donabedian had a broad definition of outcome: “Although some outcomes are generally unmistakable and easy to measure (death, for example), other outcomes, not so clearly defined, can be difficult to measure. These include patient attitudes and satisfactions, social restoration and physical disability, and rehabilitation.” He believed that “outcomes, by and large, remain the ultimate validators of the effectiveness and quality of medical care.” However, it was financial, political, and social pressures that pushed the outcomes agenda forward. In the decades leading up to the introduction of Medicare in the 1960s, scientific advances occurred at a rapid rate, and health care spending was relatively unscrutinized. However, from the 1970s to mid-1980s, concerns about lack of oversight grew as advanced technology was applied to a greater number of patients, resulting in exploding health care costs. Many writers noted that the increase in costs did not necessarily bring improved patient outcomes. Archie L. Cochrane warned, in his book Effectiveness and Efficiency: Random Reflections on the Health Services, that “cure is rare while the need for care is widespread, and that the pursuit of cure at all costs may restrict the supply of care” [reprinted in (7)]. He noted (7) the “very widespread belief that for every symptom, or group of symptoms there was a bottle of medicine, a pill, an operation, or some other therapy which would at least help.” If all of these therapies were applied without evidence of benefit, the system would become bankrupt. He went on to advocate (7) the randomized clinical trial as the best way of determining effectiveness. In 1973, John Wennberg and Alan Gittelsohn (8) published a seminal article in Science that reported a wide variation in resource utilization, expenditures, and rates of hospitalization and procedures. In particular, they found that rates of tonsillectomy varied dramatically within the state of Vermont without obvious differences in health outcomes. Many people began to question why there were such different patterns of care. Shouldn’t physicians know and basically agree on what worked? If not, they asked, could money be saved by eliminating needless medical interventions? The first large-scale effort to study practice variation in oncology was the radiation therapy-focused Patterns of Care Study (PCS), funded by the National Cancer Institute (NCI), Bethesda, MD, in the early 1970s (9,10). This program was the first systematic attempt in the United States to evaluate the patterns of care, processes, access, and patient outcomes of an entire specialty by surveying radiation therapy practices. Six commonly treated tumors were selected, and outcomes were related to patient and disease characteristics, treatment variables, and practitioner and treatment center information. Other groups, such as the American College of Surgeons, continue similar efforts through the National Cancer Data Base (NCDB), established in 1989 (11). Approximately 50% of U.S. cancer surgical cases are registered in this database, which is used to monitor surgical 196 REVIEW practice variation and provide data for quality-control assessment. Although efforts such as the PCS and NCDB have been useful in supporting epidemiologic research as well as clinical research, they have not proven to be sufficiently comprehensive to support true health services research. It was recognized that, to move from observation on small-area variation to clinically and financially meaningful changes in the practice of medical care, a new research paradigm was needed. In 1986, the National Center for Health Services Research and Health Care (Technology Assessment) (NCHSR) was directed to establish a “patient outcomes assessment research program to promote research with respect to patient outcomes of selected medical treatments and surgical procedures for the purpose of assessing their appropriateness, necessity and effectiveness,” although no funds were appropriated that year (12). When funding was later approved, four Patient Outcomes Assessment Research Program projects were established (for prostate disease, cataracts, myocardial infarction, and low back pain). This effort was the precursor to the Agency for Health Care Policy and Research’s (AHCPR’s) Patient Outcomes Research Teams (PORTs). However, this effort was carried out on a limited scale until 1988, when several key articles in the medical literature again focused public attention on outcomes research as a possible solution to a growing problem. Calls for Assessment and Accountability By 1988, government, other third-party payers, patients, and physicians were all seeking improvements in the delivery of health care and objective evidence of value for money. Paul M. Ellwood (13) wrote about an ambitious plan to link treatment and outcome data in a massive database to facilitate “outcomes management.” He believed that studying the outcomes of large numbers of patients, including standardized survival, disease status, quality of life, and cost information, could improve patient care and inform national policies. Later that same year, William L. Roper and colleagues (14) published an article calling for collaboration among the private and public sectors to improve medical care through an “Effectiveness Initiative.” At the time, Roper was the Administrator of the Health Care Financing Administration (Washington, DC), which oversaw health care coverage for all Medicare beneficiaries. He also had access to a large database of claims data and noted that many results of randomized trials could not be extrapolated to the older persons whom he represented. In an accompanying editorial, Arnold S. Relman (15) labeled “assessment and accountability” the “third revolution in medical care,” following the earlier revolutions of health care expansion and a backlash of costcontainment. Thus, the “outcomes movement,” as defined in 1990 by Arnold M. Epstein (16), undertook efforts to address “the effectiveness of different interventions, the use of this information to make possible better decision making by physicians and patients, and the development of standards to guide physicians and aid third-party payers in optimizing the use of resources.” Mounting concerns over variability in practice patterns, lack of documented benefit from medical interventions, and skyrocketing medical costs had coalesced into a call for standardization of health care based on empirical data. Patient Outcomes Research Teams: Research Funding In many ways, the availability of government funding helped to shape the outcomes movement. The Effectiveness Initiative Journal of the National Cancer Institute, Vol. 92, No. 3, February 2, 2000 evolved into the Medical Treatment Effectiveness Program (MEDTEP), which was charged with carrying out the research effort. By focusing on “the evaluation of outcomes (i.e., what resulted) of health care services, rather than on the processes (i.e., what was done),” there were echoes of Donabedian’s initial formulation of quality (17). In 1988, the NCHSR was dismantled and the AHCPR was created in its place. The AHCPR implemented MEDTEP by funding PORTs. Appropriate topics for funding were decided on the basis of “the number of individuals affected, the extent of uncertainty or controversy with respect to the use of a procedure or its effectiveness, the level of related expenditure, and the availability of data” (17) (Table 1). Prospective, randomized trials were discouraged because the focus was on alternative management strategies already in clinical practice. The following four components were required in all applications for funding and emphasize the goals of the PORTs: 1) literature review and synthesis (on which to base hypotheses for analyzing associations between practice variations and outcomes), 2) analysis of variations in medical practice and associated patient outcomes, 3) dissemination of findings about effectiveness, and 4) evaluation of the effects of dissemination. Given that the funding became available because of an amendment to the Social Security Act, the first projects reflected the needs of the older, Medicarecovered population. Of the first 14 PORT projects funded, only one addressed a cancer diagnosis—localized prostate cancer—as a subsidiary component of the project on benign prostatic hypertrophy. Patient Outcomes Research Team-II Although quality of life, costs, and decision making were features of the original PORTs, they were not emphasized until the concept of “appropriate care” was embraced in the early 1990s. In 1993, AHCPR published its second Request for Proposals for PORT studies, so-called PORT-IIs. These projects differed somewhat from the original PORTs in that they encourTable 1. Example of categories of research topics from the PORT-I Request for Proposals* • Evaluation of procedures or services intended to prevent, diagnose, treat, or manage clinical conditions for which there is controversy and/or variation in approach. • Examination of the factors associated with variations in practice patterns and variations in outcomes, including clinical, socioeconomic, and demographic characteristics; organizational characteristics; structure and processes of care; patient and provider risk behavior and preferences; cost and financing issues and incentives; and geographic location. • Application of health outcomes measures to evaluations of the structure, resource use, and performance of the health care system and its components, including preventive services, primary care, ambulatory surgery, teaching facilities, long-term care, intensive care, and rural hospital care. • Use of claims data to study patient outcomes, including the integration of insurance, hospital, and physician records. • Development and/or validation of sensitive and comprehensive measures of patient outcomes, incorporating functional status, perceived health status, and quality of life. • Development of meaningful criteria for defining the geographic areas and patient populations to be compared. • Analysis of the effects of guidelines, standards, and review criteria on the quality, appropriateness, effectiveness, and costs of health care. aged generation of primary data about the effectiveness of interventions, through either randomized trials or prospective, longitudinal study designs. There was general acknowledgment that administrative databases would only be able to contribute some types of information and that additional data would have to be generated. The format was liberalized so that PORT-IIs could accomplish their goals by whatever means was most efficient, rather than the more rigidly outlined structure of the original PORTs. For example, some study designs funded by the PORTII grants included large, simple trials (to enhance the generalizability of findings) and studies evaluating the impact of PORT-II recommendations. Whereas the common theme running through the original PORTs was variation in patterns of care, the PORTIIs emphasize the influence of patient characteristics and preferences on ideal therapy and cost-effectiveness. A total of 21 PORTs have been funded. Of these, two have addressed prostate cancer and one focuses on breast cancer (18,19) (Table 2). Integration of Outcomes Research Into the Mainstream The most striking shift in outcomes research has been its migration into mainstream clinical research. It is increasingly common to see traditional outcomes research end points and methods used alongside classic trial designs answering biologic questions. In addition, although outcomes research used to be the purview of generalists, specialists and subspecialists are increasingly involved in the conduct of such studies. In oncology, the American Society of Clinical Oncology has a health services research group charged with “the development of practice guidelines, conduct of technology assessment, and the development of standardized parameters and definitions to use in outcomes reTable 2. Projects funded through the Patient Outcomes Research Teams (PORTs)* PORT-I Grants (1989 through 1995) Back pain outcome assessment team Consequences of variation in treatment for acute myocardial infarction Variations in cataract management: patient and economic outcomes Assessing therapies for benign prostatic hypertrophy and localized prostate cancer Assessing and improving outcomes: total knee replacement Variations in the management and outcomes of diabetes Outcome assessment program in ischemic heart disease Outcomes assessment of patients with biliary tract disease Analysis of practices: hip fracture repair and osteoarthritis Assessment of the variation and outcomes of pneumonia Contracts (1990 through 2000) Variations in management of childbirth and patient outcomes Secondary and tertiary prevention of stroke Schizophrenia patient outcomes research team PORT on low birth weight in minority and high-risk women PORT-II Grants (1994 through 2001) PORT-II for prostatic disease Care, costs, and outcomes of local breast cancer Dialysis care: choices, outcomes, costs, and tradeoffs Cardiac arrythmia PORT Safety and effectiveness of homemade cereal oral-based rehydration therapy for infants with diarrhea Value of medical testing prior to cataract surgery Improving the cost-effectiveness of care for depression Effectiveness of outpatient treatment for pelvic inflammatory disease Right heart catheterization: outcomes and effectiveness Strategies for care of the very-low-birth-weight infant Pediatric asthma care PORT-II *From (17). PORT ⳱ Patient Outcomes Research Teams. Journal of the National Cancer Institute, Vol. 92, No. 3, February 2, 2000 *From (18,19). PORT ⳱ Patient Outcomes Research Teams. REVIEW 197 search. It tries to improve the quality and appropriateness of patient care, enhance the physician–patient relationship, and provide credible information to assist third-party payers determine reimbursement policies” (20). The American Society of Therapeutic Radiation Oncologists (21) likewise has an outcomes research committee. Several of the cooperative oncology groups have outcomes and health services research committees charged with integrating outcomes end points into clinical trials. This integration is reflected in general funding sources beyond the AHCPR. For example, while the NCI has a long history of funding large-scale efforts in outcomes research, the agency also cooperates with the AHCPR to review and fund outcomes research using the R01 (i.e., investigator-initiated) grant mechanisms. Although the NCI already had a Health Services and Economics Branch within its Applied Research Program, it also recently established an Outcomes Research Branch, which focuses on “multi-dimensional measures of patient function, quality-of-life and health status, preference-based utility measures, and measures of the economic costs of cancer-specific interventions” (22). This Outcomes Research Branch is also responsible for helping to incorporate a broader array of outcome measures, such as quality of life and costs, into NCI-sponsored trials. In contrast, the AHCPR has been renamed the Agency for Health Care Research and Quality (AHRQ) as of December 6, 1999, and will have additional responsibilities. CONCEPTUAL FRAMEWORK Outcomes Research Versus Health Services Research Although the field was growing, the terminology of outcomes research was not adapted and clarified as its study questions and methods became more sophisticated. The term “outcomes research” is still commonly used, although many investigators in the United States prefer the term “health services research,” and the two often seem to be used interchangeably. The National Library of Medicine, the body responsible for assigning Medical Subject Heading (MeSH) terms to publications, has two separate categories of research that encompass most of the work that is considered to be outcomes research: health services research and outcome assessment (Table 3). The term “health services research” was introduced in 1980 and is defined (23) as “the integration of epidemiologic, sociological, economic, and other analytic sciences in the study of health services. Health services research is usually concerned with relationships between need, demand, supply, use, and outcome of health services. The aim of the research is evaluation, particularly in terms of structure, process, output, and outcome.” In contrast, outcome assessment is defined (23) as “research aimed at assessing the quality and effectiveness of health care as measured by the attainment of a specified end result or outcome, improved health, lowered morbidity or mortality, and improvement of abnormal states (such as elevated blood pressure).” This category was introduced in 1992. The National Institutes of Health (NIH), Bethesda, MD, seems to recognize a difference between outcomes research and health services research. In 1996, the NIH Director’s Panel on Clinical Research (24) wrote that clinical research not only includes “development of new technologies, human disease mechanisms, therapeutic interventions, and clinical trials” but “also includes epidemiologic and behavioral studies and outcomes research and health services research,” thus drawing a 198 REVIEW distinction between outcomes research and health services research. Other organizations appear to equate outcomes research and health services research. The 1999 AHCPR strategic plan (25) states, “The field of health outcomes research studies the end results of the structure and processes of health care on the health and well-being of patients and populations. A unique characteristic of this research is the incorporation of the patient’s perspective in the assessment of effectiveness.” However the AHCPR (26) also states, “health services research addresses issues of organization, delivery, financing, utilization, patient and provider behavior, quality, outcomes, effectiveness and cost. It evaluates both clinical services and the system in which these services are provided. It provides information about the cost of care, as well as its effectiveness, outcomes, efficiency, and quality. It includes studies of the structure, process, and effects of health services for individuals and populations. It addresses both basic and applied research questions, including fundamental aspects of both individual and system behavior and the application of interventions in practice settings.” Thus, the terms “outcomes research” and “health services research” are increasingly synonymous. Those who make a distinction between these terms regard outcomes research as measuring and addressing clinical issues and health services research as responding to policy questions. However, given the ambiguous way in which the terms are applied, we suggest the use, whenever possible, of more precise terminology that better reflects the diversity of study questions, methods, end points, and target audiences. This conceptual framework is presented below. Classification of Studies In the past, outcomes research has often been recognized more by what it is not (phase I, II, or III clinical trials evaluating survival or disease response) than by what it is. As a result, a single defining theme never developed. Classically, the key feature distinguishing clinical research from outcomes research was the emphasis on efficacy (the effect of an intervention measured under controlled circumstances, as in a clinical trial) rather than on effectiveness (the effect of an intervention as applied to broad populations in real practice). However, with the “mainstreaming” of outcomes research during the past decade, many studies, other than those looking at effectiveness, have come to be considered outcomes research. The umbrella term “outcomes research” now loosely covers a broad range of study questions (quality of care, access, decision making, prediction rules, and effectiveness), methods (analysis of administrative databases and decision analysis), and end points (health-related quality of life and costs). Fig. 1 depicts our conceptualization of the relationship between the study questions, end points, analytic methods, and applications that define outcomes research. It is a complicated diagram that reflects the historic development of outcomes research as contrasted to classical clinical trials. The upper section of Fig. 1 represents areas of research, and the lower half gives examples of applications. The upper left portion of Fig. 1 shows phase I, II, and III clinical trials designed to answer efficacy questions. These trials are not outcomes research. The upper right section of Fig. 1 shows the research topics classically considered to be outcomes or health services research: quality of care, access, decision making, prediction rules, and effectiveness. Sometimes, outcomes studies are defined by the source of Journal of the National Cancer Institute, Vol. 92, No. 3, February 2, 2000 Table 3. Medical Subject Headings (MeSH) and definitions related to outcomes research ordered by date of first usage* Date of first usage MeSH and definition No date Economics: The science of utilization, distribution, and consumption of services and materials. No date Decision making. The process of making a selective intellectual judgment when presented with several complex alternatives consisting of several variables and usually defining a course of action or an idea. No date Neoplasms: New abnormal growth of tissue. Malignant neoplasms show a greater degree of anaplasia and have the properties of invasion and metastasis, compared to benign neoplasms. 1966 Costs and cost analysis: Absolute, comparative, or differential costs pertaining to services, institutions, resources, etc., or the analysis and study of these costs. 1968 Quality of health care: The levels of excellence that characterize the health service or health care provided based on accepted standards of quality. 1976 Cost-benefit analysis: A method of comparing the cost of a program with its expected benefits in dollars (or other currency). The benefit-to-cost ratio is a measure of total return expected per unit of money spent. This analysis generally excludes consideration of factors that are not measured ultimately in economic terms. Cost-effectiveness compares alternative ways to achieve a specific set of results. 1977 Quality of life: A generic concept reflecting concern with the modification and enhancement of life attributes, e.g., physical, political, moral, and social environment. 1978 Health services accessibility: The degree to which individuals are inhibited or facilitated in their ability to gain entry to and to receive care and services from the health care system. Factors influencing this ability include geographic, architectural, transportational, and financial considerations, among others. 1980 Health services research: The integration of epidemiologic, sociological, economic, and other analytic sciences in the study of health services. Health services research is usually concerned with relationships between need, demand, supply, use, and outcome of health services. The aim of the research is evaluation, particularly in terms of structure, process, output, and outcome. 1980 Quality assurance (health care): Activities and programs intended to assure or improve the quality of care in either a defined medical setting or a program. The concept includes the assessment or evaluation of the quality of care; identification of problems or shortcomings in the delivery of care; designing activities to overcome these deficiencies; and follow-up monitoring to ensure effectiveness of corrective steps. 1991 Decision support techniques: Mathematical or statistical procedures used as aids in making a decision. They are frequently used in medical decision making. 1992 Outcome assessment (health care): Research aimed at assessing the quality and effectiveness of health care as measured by the attainment of a specified end result or outcome. Measures include parameters such as improved health, lowered morbidity or mortality, and improvement of abnormal states (such as elevated blood pressure). 1993 Guidelines: Directions or principles presenting current or future rules of policy. Guidelines may be developed by government agencies at any level, institutions, professional societies, governing boards, or by the convening of expert panels. The text may be cursive or in outline form, but it is generally a comprehensive guide to problems and approaches in any field of activity. For guidelines in the field of health care and clinical medicine, “practice guidelines” is available. 1993 Practice guidelines: Directions or principles presenting current or future rules of policy for the health care practitioner to assist him in patient care decisions regarding diagnosis, therapy, or related clinical circumstances. The guidelines may be developed by government agencies at any level, institutions, professional societies, governing boards, or by the convening of expert panels. The guidelines form a basis for the evaluation of all aspects of health care and delivery. 1998 Quality indicators (health care): Norms, criteria, standards, and other direct qualitative and quantitative measures used in determining the quality of health care. *From (23). data, regardless of the question. For example, studies using large administrative databases are usually identified as outcomes studies. Some study designs, such as large, simple trials, fall in between the categories of clinical trials and outcomes research. These randomized studies are conducted in real practice settings without the rigid controls of clinical trials; thus, they represent something of a hybrid between clinical trials and outcomes research. In the center of Fig. 1 are the main end points of interest in oncology: overall survival (OS), disease-free survival (DFS), tumor response, symptoms, health-related quality of life (HRQOL), and cost. The outcomes on the left side of Fig. 1, especially OS, DFS, and response, are usually found in classically designed clinical trials that evaluate efficacy. However, if these end points are used in cohort studies, in analyses of administrative databases, or in decision analysis, they would be considered to be outcomes research. In contrast, HRQOL and cost are usually considered to be end points for outcomes research, even if they are collected in phase III clinical trials. Measurement of symptoms blends both efficacy and outcomes end points, and its position between these end points reflects the important contribution of both influences. Methods of secondary analysis, such as meta-analysis (27) and decision analysis (28,29), often use the same data sources and similar techniques. In practice, meta-analysis is much more likely to be considered a branch of clinical trials, whereas decision analysis is often considered to be outcomes research. This distinction is both substantive and historic. The substantive difference is that meta-analysis results are more often used to inform clinical decisions, while decision analysis is associated with policy decisions. The historic background further separates these methods; meta-analysis is a method developed by biostatisticians who are typically associated with the clinical trials world, whereas decision analysis was developed by practitioners seeking to incorporate many variables, whether taken from the literature or estimated, into making the best clinical and policy decisions with the available information. The bottom of Fig. 1 depicts examples of clinical and policy Journal of the National Cancer Institute, Vol. 92, No. 3, February 2, 2000 REVIEW 199 Fig. 1. Conceptual framework. Interaction is shown between research topics, end points, analytic techniques, and applications in defining outcomes research. Depicted in the upper left corner are the classic clinical trials and analytic techniques that are not outcomes research. In the upper right corner are shown the study topics, end points, and analytic techniques that are considered to be outcomes research. Outcomes depicted in the center box may or may not constitute outcomes research, depending on the context. For example, overall survival as measured in a phase III trial is not an outcomes study (efficacy), whereas it is if observed in a large community cohort (effectiveness). Symptoms have both efficacy and outcomes influences. Applications are indicated in italics and may emanate from either clinical trials or outcomes research. See text for further details. OS ⳱ overall survival; DFS ⳱ disease-free survival; HRQOL ⳱ healthrelated quality-of-life. of health care services on the access to, delivery, cost, outcomes and quality of services” including studies of health status, quality of life, and effectiveness. Exclusion criteria include “projects that do not involve some component of research or evaluation, clinical trials looking at efficacy, and studies of animals.” Although this definition is broad, we believe that it most accurately represents the current spectrum of outcomes/health services research. decisions that are guided by the research above. The positioning of clinical decisions under clinical trials and policy decisions under outcomes research reflects the major influences, but this is by no means absolute and cannot be used to define branches of research. Examples of cross-fertilization include clinical decisions that are influenced by outcomes research on decision making, quality of life, or reports of cohort studies and policy decisions that are heavily influenced by the results of randomized clinical trials. It is obvious that this schema will continue to evolve and that others may have developed different, equally effective classification systems. Our goal in presenting this framework is to depict one way of organizing the broad categories of outcomes studies. We hope it also encourages precision in describing one’s work. Stating a research question, method, or data source will convey more meaningful information than the overview category of outcomes research. This may be especially true for quality-of-life end points and decision analysis that have particularly well developed methodologies. To determine how the historic background and conceptual framework of outcomes research have been translated into a body of work, we conducted a structured review of the literature on outcomes research in oncology. This search was designed with the goal of examining how outcomes research is being presented to the community of practitioners. Therefore, we reviewed a broad range of general medicine, subspecialty, and methodology journals likely to be read by practitioners and researchers in the field. Definition Search Paradigm An appropriate definition of outcomes research needs to incorporate many nuances. It must capture the diverse branches of a field that are more united by their historic development than by what they now have in common. It should recognize that research topics, study designs, methods, and data sources must be examined together to correctly identify outcomes research. It must allow outcomes research to be defined both by what it is and by what it is not. Thus, our favorite definition is provided by the Association for Health Services Research (30). They favor the term “health services research” and define it as “a field of inquiry using quantitative or qualitative methodology to examine the impact of the organization, financing, and management A MEDLINE® search was performed to retrieve all articles in English published from 1966 (the first year available in MEDLINE) through 1998 with a keyword of “neoplasms” and the following topics: cost, cost-benefit analysis, economics, health service accessibility, health services research, outcome assessment (health care), quality of life, quality of health care, quality assurance (health care), quality indicators (health care), guidelines, practice guidelines, decision making, and decision support techniques (Table 3). The following journals were included: New England Journal of Medicine, Journal of the American Medical Association, Lancet, Annals of Internal Medicine, Journal of the National Cancer Institute, Journal of Clinical Oncol- 200 REVIEW LITERATURE REVIEW Journal of the National Cancer Institute, Vol. 92, No. 3, February 2, 2000 ogy, Blood, British Journal of Haematology, Annals of Oncology, Cancer, Medical Decision Making, Medical Care, Health Economics, Journal of Health Economics, European Journal of Cancer, British Journal of Cancer, and Bone Marrow Transplantation. Classification The abstracts were independently reviewed by two of the authors (S. J. Lee and C. C. Earle) and classified according to the research topic, study design, and relevant tumor type. Discrepancies were resolved by consensus or mediation by a third party (J. C. Weeks), as necessary. Articles were retrieved as needed for clarification. Clinical reviews, letters, and editorials were not reviewed further. Articles dealing with nonhumans or primarily nononcologic topics were excluded from the analysis. Only primary reports were considered; secondary reports were not included. Research topics/end points were classified as survival, disease status, complications, health-related quality of life, patient preferences, quality of decision making, costs, quality of care, access, differences in practice patterns, physician–patient relationships, appropriate use of diagnostic tests, and guidelines. Often, we found that traditional outcomes or health services research questions were secondary end points in trials primarily analyzing survival. In these cases, the end point was designated secondary. Tertiary end points were those that were not quantified according to current research standards or those that were mentioned but minimally addressed. For example, the sole use of Karnofsky performance status, return to employment, or nonstandardized methods to determine quality of life was considered to be a tertiary end point. Short-term and long-term complications of therapy, such as pain, memory problems, or mucositis, were considered to be clinical manifestations if they were reported as symptoms or as quality-of-life dimensions if they were measured with the use of structured patient self-report surveys. Psychologic, psychiatric, and emotional end points were always considered to be quality-of-life studies. Examination of the impact of guidelines on practice patterns was classified under quality of care. Study design was categorized according to the primary design of the trial, with occasional consideration of the type of statistical analysis performed. Options included the following: phase I, II, or III clinical trial; nonrandomized intervention study; prospective and retrospective cohort studies; case–control study; cross-sectional study; decision analysis; meta-analysis; and systematic review of the literature. Studies that relied on administrative databases, as well as studies with a primary goal of instrument development and assessment, were considered to be distinctive enough to warrant separate classification. Articles Retrieved A total of 92 891 oncology references were published in these journals from 1966 through 1998. Of these, 1591 articles contained the MeSH terms of interest and were retrieved by the search; 678 (43%) met the inclusion criteria for the analysis. Of the 913 (57%) articles initially retrieved by the search but excluded from further review, 360 of the original 1591 (23%) were deemed to be editorials, 170 (11%) letters, 62 (4%) news, 2 (0.1%) comments, 138 (9%) reviews, and 111 (7%) phase I, II or III clinical trials or meta-analyses with survival or disease status end points. An additional 70 (4%) references were excluded because they either did not deal with humans or otherwise could not be classified as having one of the end points of interest. Search Results The distribution of research topics, journals, study designs, and disease sites among the 678 included references is shown in Table 4. The most common topic of these publications was quality of life (209 [31%] references). Effectiveness studies examining clinical end points (excluding quality of life) outside clinical trials accounted for the next largest groups (147 [22%] references). Studies of costs (127 [19%] references), quality of care (101 [15%] references), patient preferences (46 [7%] references), appropriate use of diagnostic tests (40 [6%] references), and guidelines (8 [1%] references) accounted for the remaining studies. General medical journals published 102 (15%) of the articles included in the analysis, specialty journals contributed 497 (73%), and methodology journals contributed 79 (12%). The greatest number of studies were published by Cancer, with 152 (22%) references, and by the Journal of Clinical Oncology, with 112 (17%) references. More articles focused on breast cancer (164 [24%] references) than on other cancer diagnoses, followed by multiple cancer sites (more than two sites), with 156 (23%) references, and hematologic malignancies, with 74 (11%) references. All other sites accounted for less than 10% each. Temporal Trends The number of outcomes studies has grown faster than the number of oncology articles, leading to an increase in the proportion of all cancer articles devoted to outcomes research (Fig. 2). However, the total number of articles is still small. Our search identified no outcomes studies in the first 4 years of the search period (1966–1969). Beginning in 1970, the number of studies meeting our criteria was fewer than 10 per year until 1982. There was then a gradual increase in the number of studies, starting particularly in 1995. Almost half of all of the articles meeting our criteria were published in the last 4 years of the study period (1995–1998; n ⳱ 331 references). Despite this dramatic increase, outcomes research continues to represent a very small proportion of clinical research studies in oncology, making up only 2.6% of the articles published in these journals in 1998. Perhaps the most striking change over time has been an improvement in the quality of outcomes studies being published. We noted greater standardization in definitions and methods, more prospective collection of outcomes end points, and more articles devoted entirely to the results of outcomes research. More and more phase I, II, or III studies combine traditional survival and disease status end points with outcomes research as secondary end points. Of the 108 “traditional” efficacy studies pulled up by the search but excluded from the final analysis, 57 were classified as having a secondary end point consistent with outcomes research (quality of life in 53 studies and costs in four studies). The number of such studies increased with time. Before 1996, there were fewer than six studies per year. After 1996, there were nine or more studies per year. However, the proportion of published clinical trials and reports with secondary outcomes end points is well below the prevalence of these end points in ongoing clinical trials in oncology. It will take several years for this recent investment in outcomes research by the Journal of the National Cancer Institute, Vol. 92, No. 3, February 2, 2000 REVIEW 201 Table 4. Primary research topics, journals, study designs, and disease sites/therapy (1966–1998) Category No. (%*) Research topic Quality of life Effectiveness (excluding quality-of-life end points) Cost Quality of care Patient preferences Appropriate use of diagnostic tests Guidelines 209 (31) 147 (22) 127 (19) 101 (15) 46 (7) 40 (6) 8 (1) Journals General Journal of the American Medical Association Annals of Internal Medicine Lancet New England Journal of Medicine Specialty Cancer Journal of Clinical Oncology European Journal of Cancer British Journal of Cancer Annals of Oncology Journal of the National Cancer Institute Bone Marrow Transplantation Blood British Journal of Hematology Methodology Medical decision making Medical care Health economics Journal of Health Economics 102 (15) 44 (7) 21 (3) 18 (3) 19 (3) 497 (73) 152 (22) 112 (17) 69 (10) 57 (8) 33 (5) 29 (4) 23 (3) 16 (2) 6 (1) 79 (12) 38 (6) 32 (5) 6 (1) 3 (<1) Study design Retrospective cohort Cross-sectional Decision analysis Prospective cohort Instrument design Phase III trial Administrative database Nonrandomized intervention Consensus conference Meta-analysis Phase I or II trial 150 (22) 128 (19) 112 (17) 79 (12) 73 (11) 56 (8) 40 (6) 24 (4) 10 (1) 3 (<1) 3 (<1) Disease site/therapy Breast Hematologic malignancies Colorectal Prostate Lung Pediatric Other gastrointestinal Cervical Testis Head and neck Ovarian Skin High-dose chemotherapy with stem cell support Central nervous system Endocrine Sarcoma Bladder Multiple sites Other 164 (24) 74 (11) 50 (7) 38 (6) 36 (5) 32 (5) 22 (3) 19 (3) 16 (2) 13 (2) 12 (2) 11 (2) 9 (1) 6 (1) 4 (1) 3 (<1) 2 (<1) 156 (23) 11 (2) *May not add to 100% because of rounding error. pharmaceutical industry, cooperative groups, and funding agencies to be reflected in the literature. Comparison of the study questions in reports published before and after 1995 did not reflect any clear shifts in attention. The distribution of studies since 1995 has reflected proportional 202 REVIEW growth across all categories. For example, in 1998, the proportions of studies on each topic resemble the distributions seen across all years combined. However, there have been qualitative shifts in research focus. In the early years of outcomes research, there was confidence that existing databases, especially administrative billing data, could be used to answer many of the primary questions of outcomes research. More recently, there has been a growing recognition that such databases are useful primarily for hypothesis generation and that prospective experimental and longitudinal studies are needed to test these hypotheses, as suggested in the PORT-II Request for Proposals. There has also been an evolution in attitudes toward evidence-based practice guidelines, originally heralded as a panacea that would offer better care at lower costs. Attempts to actually implement clinical practice guidelines have led to greater appreciation for the need to consider patient preferences and to allow for some tailoring of treatment choices to the individual. These considerations are increasingly reflected in published guidelines (31–37). DISCUSSION Outcomes research is a response to concerns raised by many stakeholders in medicine, and our literature review revealed that the number and proportion of published studies are increasing. The field arose from a recognition of variation in medical practice and from efforts to determine effective and appropriate care. However, the term “outcomes research” has been used liberally by anyone with an interest in the practice of medicine. “Good” outcomes support the therapy or policy under evaluation. “Bad” outcomes argue against it. The conclusions of outcomes research have been used by patients, physicians, hospitals, insurance companies and managed care organizations, government, and others with an interest in the appropriate delivery of health care at the lowest cost. In presenting the historic background of outcomes research, offering a conceptual framework to help classify outcomes studies, and reviewing the literature for trends, we tried to summarize concisely a very complicated, frequently used, but poorly defined concept. Considering ourselves to be outcomes researchers, we were encouraged by some findings, such as the noticeable improvement in the quality of studies over time observed in our literature search, and the increasing percentage of the oncology literature representing outcomes work. However, we were surprised by some of our other findings. Although the number of outcomes end points investigated in the context of phase I, II, or III clinical trials seems to be increasing, the literature review did not quite seem to reflect this increase, despite the considerable investment in this area by oncology cooperative groups, pharmaceutical companies, and funding agencies. Possible explanations for the small number of publications describing quality of life and costs collected alongside clinical trials include the following: 1) It is too early for these studies to have made their way into the literature, 2) the results of these studies are not being published, or 3) our literature search algorithm did not successfully identify these publications. The conceptual framework and the definition of health services research put forth by the Association for Health Services Research represent the integration of the historic background and current state of the field. We expect that there will continue to be evolution in the nomenclature. Is a study evaluating quality of life in a phase III randomized trial considered to be outcomes research or not? What about a study reporting the effectiveness Journal of the National Cancer Institute, Vol. 92, No. 3, February 2, 2000 Fig. 2. Outcomes studies (1966 through 1998): selected journals. Both the absolute number of outcomes articles and the proportion of total oncology articles are increasing. of a highly specialized technique, not performed as a part of a clinical trial, but only available in a tertiary care institution? Or what about a decision analysis of survival, using the results of multiple, randomized trials? For the purpose of this review, all of the above studies were considered to be outcomes research. However, as outcomes research continues its diffusion into mainstream clinical investigations, these end points and techniques may no longer automatically classify such studies as outcomes research. Evidence for the Impact of Outcomes Research Very little research has looked at whether outcomes research actually affects practice or policy. Fineberg (38) studied the effects of randomized clinical trials on patterns of care and concluded that this information had little influence on general medical practice. Kosecoff et al. (39) studied the effect of the NIH Consensus Development Conferences and concluded that these conferences also had little effect on medical practice. Although “report cards” giving mortality rates for cardiac surgeons may have improved the results of bypass surgery, patients were generally unaware of them (40). On the other hand, gemcitabine was approved by the U.S. Food and Drug Administration for the treatment of pancreatic cancer largely on the basis of its ability to decrease symptoms. Efforts to assist patients in clarifying their treatment preferences and to assist them in their decision making are proliferating. Cost-minimization studies are used extensively to help organizations respond to budget constraints. Cost-effectiveness studies are used by Australia and the Canadian province of Ontario as part of their drug-approval mechanisms. Algorithms for evaluation of thyroid nodules or abnormal Pap smears, for example, are common in clinical practice. The National Comprehensive Cancer Network (41) has published consensus guidelines for cancer therapy and has created an outcomes database to track adherence to selected guidelines among participating institutions. The Future Outcomes research is fundamentally concerned with improving the practice of medicine as applied to patients treated outside clinical trials. Decreasing the variation in practice patterns was seen as the first solution to the problem. This was followed by a recognition of the need for better information about appropriate treatment algorithms. More recently, there is greater acknowledgment of the need for individualization of therapy. The result is that increasing numbers of variables (both patient characteristics and societal needs) have been added to the process of determining what is the most appropriate therapy for individual patients. This shift reflects a new recognition of the complexity of medical decisions and the myriad influences affecting patient outcomes. As the discipline of outcomes research continues to develop, it is likely that the boundary between this field and classical clinical research will become progressively more blurred. If so, umbrella terms such as “clinical research,” “outcomes research,” and “health services research” may be replaced by more specific descriptors that reflect the type of question being addressed and the methods used to answer it. REFERENCES (1) Youngs MT, Wingerson L. The 1996 medical outcomes & guidelines sourcebook. New York (NY): Faulkner & Gray, Inc.; 1995. (2) Codman EA. The product of a hospital. Surg Gynecol Obstet 1914;18: 491–6. (3) Weeks JC. Outcomes assessment. In: Holland JF, Frei EJ, editors. Cancer medicine. Baltimore (MD): Williams & Wilkins; 1996. p. 1451–8. (4) Weeks J, Pfister DG. Outcomes research studies. Oncology (Huntingt) 1996;10(11 Suppl):29–34. (5) Weeks J. Overview of outcomes research and management and its role in oncology practice. Oncology (Huntingt) 1998;12(3 Suppl 4):11–3. (6) Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q 1966;44:Suppl:166–206. (7) Cochrane AL. Archie Cochrane in his own words. Selections arranged from his 1972 introduction to “Effectiveness and Efficiency: Random Reflections on the Health Services” 1972. Controlled Clin Trials 1989;10:428–33. (8) Wennberg J, Gittelsohn A. Small area variations in health care delivery. Science 1973;182:1102–8. (9) Hanks GE, Kramer S, Diamond JJ, Herring DF. Patterns of care outcome survey: national outcome data for six disease sites. Am J Clin Oncol 1982; 5:349–53. (10) Kramer S, Hanks GE, Diamond JJ, MacLean CJ. The study of the patterns of clinical care in radiation therapy in the United States. CA Cancer J Clin 1984;34:75–85. (11) Steele GD Jr, Winchester DP, Menck HR, Murphy GP. Clinical highlights Journal of the National Cancer Institute, Vol. 92, No. 3, February 2, 2000 REVIEW 203 (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (22) (23) (24) (25) (26) (27) from the National Cancer Data Base: 1993. CA Cancer J Clin 1993;43: 71–82. Agency for Health Care Policy and Research. Report to Congress: progress of research on outcomes of health care services and procedures. Rockville (MD): Agency for Health Care Policy and Research; 1991. Ellwood PM. Shattuck lecture—outcomes management. A technology of patient experience. N Engl J Med 1988;318:1549–56. Roper WL, Winkenwerder W, Hackbarth GM, Krakauer H. Effectiveness in health care. An initiative to evaluate and improve medical practice. N Engl J Med 1988;319:1197–202. Relman AS. Assessment and accountability: the third revolution in medical care [editorial]. N Engl J Med 1988;319:1220–2. Epstein AM. The outcomes movement—will it get us where we want to go? N Engl J Med 1990;323:266–70. Agency for Health Care Policy and Research. Medical treatment effectiveness research. Rockville (MD): Agency for Health Care Policy and Research; 1990. Agency for Health Care Policy and Research. Patient outcomes research teams (PORTs). Available from: http://www.ahcpr.gov/news/ports99.htm Agency for Health Care Policy and Research. Current patient outcomes research teams (PORT-IIs). Available from: http://www.ahcpr.gov/news/ portii99.htm American Society of Clinical Oncology. Committee: Health Services Research (HSR). Available from: http://asco.infostreet.com/prof American Society of Therapeutic Radiation Oncologists. ASTRO Committees. Available from: http://www.astro.org/about National Cancer Institute. Applied research: outcomes research. Available from: http:/www-dccps.ims.nci.nih.gov/ARB National Library of Medicine. Medical subject headings. Available from: http:/www.nlm.nih.gov/mesh Nathan DG. Clinical research: perceptions, reality, and proposed solutions. National Institutes of Health Director’s Panel on Clinical Research. JAMA 1998;280:1427–31. Agency for Health Care Policy and Research. AHCPR strategic plan. Available from: http:/www.ahcpr.gov/about/stratpln.htm Eisenberg JM. Health services research in a market-oriented health care system [published erratum appears in Health Aff (Millwood) 1998;17:230]. Health Aff (Millwood) 1998;17:98–108. The University of York NHS Centre for reviews & dissemination. Undertaking systematic reviews of research on effectiveness. York (U.K.): York Publishing Services Ltd.; 1996. 204 REVIEW (28) Weinstein MC, Fineberg HV, Elstein AS, Frazier HS, Neuhauser D, Neutra RR, et al. Clinical decision analysis. Philadelphia (PA): Saunders; 1980. (29) Sonnenberg FA, Beck JR. Markov models in medical decision making: a practical guide. Med Decis Making 1993;13:322–38. (30) Association for Health Services Research. Definition of health services research. Available from: http://www.ahsr.org/hsrproj/define.htm (31) Cancer Care Ontario. Program in evidence-based care and practice guidelines initiative. Available from: http://www.cancercare.on.ca/ccopgi (32) National Comprehensive Cancer Network. NCCN Oncology practice guidelines. Oncology 1996;10:47–315. (33) National Comprehensive Cancer Network. NCCN Oncology practice guidelines. Oncology 1997;11:25–347. (34) National Comprehensive Cancer Network. NCCN Oncology practice guidelines. Oncology 1998;12:35–271. (35) National Comprehensive Cancer Network. NCCN Oncology practice guidelines. Oncology 1998;12:53–462. (36) National Comprehensive Cancer Network. NCCN Oncology practice guidelines. Oncology 1999;13:35–256. (37) Silver RT, Woolf SH, Hehlmann R, Appelbaum FR, Anderson J, Bennett C, et al. An evidence-based analysis of the effect of busulfan, hydroxyurea, interferon, and allogeneic bone marrow transplantation in treating the chronic phase of chronic myeloid leukemia: developed for the American Society of Hematology. Blood 1999;94:1517–36. (38) Fineberg HV. Clinical evaluation: how does it influence medical practice? Bull Cancer 1987;74:333–46. (39) Kosecoff J, Kanouse DE, Rogers WH, McCloskey L, Winslow CM, Brook RH. Effects of the National Institutes of Health Consensus Development Program on physician practice. JAMA 1987;258:2708–13. (40) Schneider EC, Epstein AM. Use of public performance reports: a survey of patients undergoing cardiac surgery. JAMA 1998;279:1638–42. (41) Weeks JC. Outcomes assessment in the NCCN. Oncology (Huntingt) 1997; 11:137–40. NOTES Supported in part by Public Health Service grant CA75267-01 from the National Cancer Institute, National Institutes of Health, Department of Health and Human Services. We thank Taiye Odedosu for her assistance with the literature review. Manuscript received April 13, 1999; revised October 13, 1999; accepted December 6, 1999. Journal of the National Cancer Institute, Vol. 92, No. 3, February 2, 2000