Download mammography Facilities, Regardless Of Their Size Or

Clinical Practice Guidelines Number 13
Quality Determinants of Mammography
U.S. Dept of Health & Human Services Pages 74-86, October 1994
This document was taken directly from pages 74-86 of Clinical Determinants of Mammography. This
government book was the predecessor to most of our current MQSA Regulations. The FDA left out most
of the audit information. As you’ll see, it’s valuable information… (see Table 10 on page 10)
-Medical Audit-
STRONG RECOMMENDATION: Mammography facilities, regardless of their size or setting,
should perform medical audits at regular intervals, usually annually.
The mammography medical audit is a systematic collection and analysis of mammography results, which
are compared with outcomes data. It is the only way to measure both technical and interpretive aspects of
mammography performance.
Medical audits permit evaluation of the ability to detect very small cancers at the expected rate, which is an
important measure for any mammography practice. Interpreting physicians receive individualized feedback
on performance, which instills confidence if results meet expectations or which identifies a need for
additional training. Longitudinal audit studies may detect causes of false-negative errors, allowing technical
and interpretive shortcomings to be corrected. When audit data are reviewed and acted upon appropriately,
the audit is a powerful education tool. Audit results can indicate acceptable levels of performance to third
party payers and government agencies and assist in legal cases. Timely audits of abnormal mammograms
can result in optima followup of individual patients as a part of risk management program. An audit also
can help monitor patient population outcomes of screening mammography, locally and through a national
database. Yearly review of audit data allows for more timely and meaningful longitudinal evaluation of
individual and group performance.
-Data CollectionRECOMMENDATION: Data should be collected from the appropriate sources for all audits.
Data sources should include the mammography report; information recorded by the scheduler, receptionist,
and technologist; and biopsy results. If biopsy results indicate cancer, information can be found through a
regional or statewide tumor registry. If no such registry is available, a definitive diagnosis of cancer or
benign results can be obtained from, in order of preference, the pathology report, the referring physician or
surgeon, or the patient herself. Anyone who is involved in patient care and has access to biopsy results,
such as surgeons and pathologists, should cooperate in communicating these results to the mammography
facility.
RECOMMENDATION: Regional or national mammography databases should be established with a
major emphasis on the medical audit component.
RECOMMENDATION: A medical audit should use data from regional or national databases to
expedite data collection, ensure completeness, and compare individual medical audit data with those
of a wide variety of other practices.
Collection of some data elements may be difficult because of patient mobility, performance of biopsies off
site, and physicians’ referral patterns and because some mammography facilities only do screening
mammography. For comparisons, a mammography facility needs data that match its patient population and
practice setting. Regional or national databases permit the collection of data from a wide geographic area
and comparison over a wide variety of practices. A regional or national mammography database may be
Page 1 of 10
reproduced by D. Eldridge, M.S.
Clinical Practice Guidelines Number 13
Quality Determinants of Mammography
U.S. Dept of Health & Human Services Pages 74-86, October 1994
implemented as a voluntary, cooperative program linked to standardized cancer registries in collaboration
with State or local health departments.
A regional or national mammography database,(Table 6) is a computerized method of data collection that
includes the collection and analysis of medical audit data from individual practices and may include a
system for patient monitoring and tracking. Such a database may also include a population registry, which
would identify women not in the screening system.
However, before any such system can be implemented, patient confidentiality and protection of medical
audit information from discovery must be insured. Current State peer review statutes protecting peer review
activities, such as collection of medical audit data, apply to inpatient facilities. In many States, there is
virtually no protection from discovery of quality assurance activities in ambulatory and outpatient settings,
where most mammography facilities function.
STRONG RECOMMENDATION: Certain essential raw data and derived data should be obtained
for a meaningful medical audit.
The raw (unprocessed) data recommended as minimum requirements, (Table 7) have been collected in most
major audits. They are necessary for proper calculation of the derived data (calculated from raw data) and
for meaningful analysis of all audit data.
The more complete data (those listed in tables 8 and 9) have been useful as determinants of prevalent vs.
incident cancer rates, predictive value of various mammographic findings, and significance of various risk
factors. It is recommended that these data be collected because they provide information about variables
that can cause considerable fluctuation in audit results. For example, if more initial than follow-up
mammograms are performed, the rate of cancers detected will be higher.
Before any calculations of derived data can be made, every mammogram should be categorized as follows:
*True Positive (TP). Cancer diagnosed within 1 year after biopsy recommendation based on abnormal
mammogram.
*True Negative (TN). No known cancer diagnosed within 1 year of normal mammogram.
*False Negative (FN). Cancer diagnosed within 1 year of a normal mammogram.
definitions of false negative exist, this definition is the most widely applied.
Although other
*False Positive (FP). Three separate definitions are seen in the literature:
(1) Cases recalled for additional imaging evaluation of an abnormal finding on a screening
mammogram in which no cancer was found within 1 year or recalled cases not shown to be
malignant within 1 year.
(2) No known cancer diagnosed within 1 year after an abnormal mammogram and recommendation
for biopsy or surgical consultation.
(3) Benign disease found at biopsy within 1 year after an abnormal mammogram and
recommendation for biopsy or surgical consultation. This last definition of false positive is similar
to the ACR BI-RADS language.
The following derived data can then be calculated and will help to reflect the overall quality of a
mammography practice:
Page 2 of 10
reproduced by D. Eldridge, M.S.
Clinical Practice Guidelines Number 13
Quality Determinants of Mammography
U.S. Dept of Health & Human Services Pages 74-86, October 1994
*Sensitivity. The probability of detecting a cancer when a cancer exists, or the percentage of all patients
found to have breast cancer within 1 year of screening who were correctly diagnosed at the screening
session.
Sensitivity = TP/(TP+FN)
*Positive predictive value (PPV). Three separate definitions may be applied, depending on the practice
conditions and based on the above three definitions of false positive. It is important to know which
definition of PPV is being used to accurately compare audit data from a mammography practice with
published data. For practices performing only screening mammography, only PPV 1 will be of value in
evaluating data. For practices performing both screening and diagnostic mammography, all three
definitions of PPV can be applied.
*PPV1 (Abnormal). The percentage of all screening mammography cases that result in a
diagnosis of cancer based on abnormal screening examination.
PPV1 = TP/(number of abnormal screening exams), or
PPV1 = TP/(TP+FP1).
*PPV2 (Biopsy recommended). The percentage of all screening mammography cases that result in
a diagnosis of cancer based on a recommendation of consideration for biopsy.
PPV2 = TP/
(# of cases recommended for biopsy after abn. screening exams), or
PPV2 = TP/(TP+FP2).
*PPV3 (Biopsy done). The percentage of all screening mammography cases that result in a
diagnosis of cancer based on biopsies performed. This is also known as the biopsy yield of
malignancy, or the positive biopsy rate.
PPV3 = TP/(number of biopsies), or
PPV3 = TP/(TP+FP3).
*Specificity. The probability of a normal mammogram report when no cancer exists, or the percentage of
all patients not found to have breast cancer within 1 year of screening who were correctly identified as
normal at the time of screening.
Specificity = TN/(FP+TN).
Some variation in the range of specificity will exist depending on the definition of false positive
being applied, but the variation will be small because of the very small number of false positives
and the very large number of true negatives in most audit series.
*Cancer detection rate. The number of cancers detected per 1,000 women screened by mammography.
This number should be available in all basic audits.
The following cancer detection rates can be calculated only if the appropriate raw data are collected.
Although not essential to a basic audit, they provide valuable information and should be calculated when
possible:
Page 3 of 10
reproduced by D. Eldridge, M.S.
Clinical Practice Guidelines Number 13
Quality Determinants of Mammography
U.S. Dept of Health & Human Services Pages 74-86, October 1994
1. Prevalent vs. incident cancer rates, that is, rates of cancer in first time vs. follow-up exams.
2. Cancer detection rates in diagnostic exams.
3. Cancer detection rates by age group.
Separate sensitivities, PPV’s, and specificities can be calculated for each of these three subgroups, yielding
yet another stratum of useful audit information.
RECOMMENDATION: To improve their interpretive skills, interpreting physicians should keep
track of the results of the breast biopsies they have recommended. They should attempt to increase
the positive predictive value of biopsied mammographic abnormalities without decreasing the
number of cancers detected.
Excessive biopsies are reflected by a low positive predictive value of biopsied mammographic
abnormalities, which can result in increased costs of screening mammography programs, decreased use of
screening mammography, decreased compliance with recommendations for necessary biopsies by referring
health care providers and women, and avoidable patient anxiety and discomfort or pain. The most
appropriate positive biopsy rate for abnormalities detected by mammography has not been determined.
Reported positive predictive values of biopsied mammographic abnormalities range from 10 to 40 percent,
even for experienced interpreting physicians; positive biopsy rates of 25-40 percent have been
recommended as appropriate.
Interpreting physicians should review mammograms and results of recommended biopsies to determine the
positive predictive value of a biopsied mammographic abnormality for individual radiologists. These rates
would be kept in logs or computer databases to evaluate cumulative results, changes over time, and effects
activities to improve the accuracy of interpretations.
-Data AnalysisRECOMMENDATION: To achieve optimal benefit from a mammography audit, the data should be
analyzed from many perspectives. The analysis should use sensitivity (if and when measurable),
positive predictive value, tumor size, node positivity, cancers found per 1,000 women screened, and
recall rate.
RECOMMENDATION: Audit data should be analyzed not only for a mammography interpreter
group as a whole, but also for each individual within that group. If available, false-negative cases
should be evaluated individually to assess cause, thus further improving future outcomes.
Mammography interpreters should strive to reach the desirable numbers achieved by highly skilled experts
in the field of mammography (shown in table 10). Data must be analyzed to answer the three questions that
most directly measure the quality of a mammographer’s performance:
(1) Is the mammographer finding a high percentage of the cancers that exist in that particular
patient population? That is, what are the sensitivity and number of cancers found per 1,000
women screened?
(2) Are these cancers being found while the numbers of biopsies and requests for further imaging
evaluation of screened cases remain acceptably low? That is, what is the PPV and what is the
recall rate?
Page 4 of 10
reproduced by D. Eldridge, M.S.
Clinical Practice Guidelines Number 13
Quality Determinants of Mammography
U.S. Dept of Health & Human Services Pages 74-86, October 1994
(3) Are a large proportion of the cancers being found still small and confined to the breast? That
is, what are the numbers of minimal (invasive cancer is less than or equal to 1 cm or in situ ductal
carcinoma) and node-negative cancers found?
Derived data are required to answer the above questions. An analysis of derived data found in the scientific
literature yielded the following:
*Sensitivity. The range of sensitivity (as defined above) in most recent mammography audits is 85-90%.
Sensitivity may vary with patient age and appears to decrease in populations with denser breast tissue. It is
among the most difficult data to obtain because their calculation requires knowledge of the number of falsenegatives, information usually obtainable only through a computer linkage with a regional tumor registry. It
may be necessary instead to estimate the number of false-negatives to obtain an approximation of
sensitivity.
*PPV. This number is almost always measurable, using one or more of the above definitions. PPV2 (2540% in most reported series) is most frequently used. Lower PPVs are acceptable when associated with
greater numbers of early-stage disease and node-negative cases. If a facility performs only screening
mammography, PPV1 (5-10% in most reported series) should be applied instead. Facilities performing both
screening and diagnostic mammography may find calculations of both PPV1 and PPV2 to be valuable.
Separate PPV statistics should be maintained for cases where core or fine needle aspiration biopsy is
recommended. PPV is subject to many variables, including age distribution, percentage of palpable
cancers, cancer detection rate, the size and node positivity of cancers found, and sensitivity.
Tumor size. In most series, more than 50% of tumors diagnosed by mammography are stage 0 or 1 and
more than 30% of cancers diagnosed by mammography are minimal cancers-invasive cancer is less than or
equal to 1 cm or in situ ductal carcinoma. Because mortality from breast cancer is directly related to tumor
size, these values should be considered valuable goals. Tumor size varies with the percentage of screening
and diagnostic examinations in a mammography practice; tumors found by diagnostic examinations are
invariably larger.
*Node positivity. Tumor size should be correlated with node positivity, which in most series is less than
25%. Because mortality from breast cancer is directly proportional to node positivity, this value must also
be considered a desirable goal.
*Cancer detection rate. (Cancers found per 1,000 women screened by mammography). Rates of 2-10
cancers per 1,000 have been reported in most series. Variability is due to differing rates of detection in
first-time-screened vs. already-screened women (prevalent vs. incident cancers). Prevalent cancer rates
vary from 6 to 10 per 1,000 and incident cancer rates vary from 2 to 4 per 1,000. The cancer detection rate
also varies in younger vs. older women and in asymptomatic vs. symptomatic patients. Despite these
variables, the cancer rate serves as a relative threshold for abnormal. For example, if an audit shows that
sensitivity and PPV are both acceptably high but the number of cancers found is less than 2 per 1,000, the
sensitivity figure is suspect; the number of cancers eluding detection in that particular population is
probably still too great, and the overall quality of the mammography should be further evaluated.
*Recall rate. The percentage of patients for whom further imaging evaluation (coned compression views,
magnification views, sonography, etc.) is recommended can be used to calculate one of the definitions of
PPV (see above). This is useful for all facilities, especially for those performing screening mammography
only. A disproportionately high recall rate can negatively affect the cost-effectiveness and credibility of
mammography. In most large reported series, the recall rate is 10% or less. This value is desirable and
seems to decrease with increasing experience of the interpreting physician.
Page 5 of 10
reproduced by D. Eldridge, M.S.
Clinical Practice Guidelines Number 13
Quality Determinants of Mammography
U.S. Dept of Health & Human Services Pages 74-86, October 1994
*Specificity. Specificity is a difficult measure of quality to obtain; it is not even calculated in many large
studies. When calculated, it is usually found to be greater than 90%. Calculating specificity is difficult in
that it requires knowledge of all true negatives. The number of true negatives is based on the number of
false negatives, which is the least accessible information in any audit.
Two other important issues regarding analysis of audit data should be noted:
*Group audit vs. individual audit. A group audit gives greater statistical power to results, which enables
comparison with overall expected rates. However, the multiple variables described earlier (prevalent vs.
incident cancers, age of a population, ratio of screening to diagnostic mammograms, etc.) markedly
influence group audit results and may render comparisons with other group audits less valuable than an
intragroup comparison of individual interpreting physicians’ results. A major advantage of an individual
audit is that it provides a valid relative comparison among individual group members. If the performance of
certain group members shows considerable variation in sensitivity and other standards, measures can be
taken to improve their performance and thus improve future outcomes.
Review of false negatives. As noted, these cases may be difficult to identify unless a complete tumor
registry is available. However, if available, false-negative cases should be evaluated thoroughly to assess
cause (technical vs. interpretive error). A thorough and critical review of false negative cases can benefit
groups’ members and improve overall quality.
Medico legal Considerations
RECOMMENDATION: Very few broadly drawn statutes protecting audit information from
discovery are in place at this time. Therefore, it is recommended that complete mammography audits
be maintained primarily as internal audits. Interpreting physicians should not disseminate the data
more widely without being aware of confidentiality legislation in their state.
Mammography facilities and interpreting physicians should be aware of State legislation pertaining to
release of medical audit information. All States have statutes that protect the records of hospital peer
review committees from discovery. However, few statutes exist that protect other information collected for
quality review, including outpatient information. Congress provided protection to participants in quality
control programs and created qualified immunity for quality assurance records in the military health care
system and the Department of Veteran Affairs. Efforts are underway to enact legislation to more fully
protect audit material outside the military setting. The panel urges that these efforts be maximized to
expedite rapid passage of protective legislation. Enactment of such legislation would encourage all
mammography facilities to participate in medical audit without fear of increasing their medico legal liability
and would allow them to more completely fulfill the quality assessment standards requirements of the Joint
Commission for Accreditation of Healthcare Organizations, ACR, and MSQA. It would also enable them
to obtain audit information beneficial to themselves, the medical community, and their patients.
Table 6. Medical audit component of a regional or national mammography database
<>Identification of practice (confidential identification)
<>Anonymous identification number of mammography interpreters
Page 6 of 10
reproduced by D. Eldridge, M.S.
Clinical Practice Guidelines Number 13
Quality Determinants of Mammography
U.S. Dept of Health & Human Services Pages 74-86, October 1994
<>Dates of audit period and number of examinations audited for each practice
<>Patient identification number (confidential identification) compatible with
database(s) enabling linkage, for example, to tumor registry
<>Patient date of birth
<>Type of mammogram (baseline or first-time, routine screen or diagnostic)
<>Interpretation of mammogram
<>Followup recommendation
<>Dates and results of followup studies, if any
<>If abnormality found, palpable vs. nonpalpable
<>Results of cytology, core needle biopsy, or open biopsy
<>Cancer data:
-Size of each breast cancer diagnosed
-Histology of each breast cancer diagnosed
-Lymph node status
-Pathological stage
-False-negative rate
-True-positive biopsy rate
-False positive rate
-Positive predictive value
-Overall cancer detection rate
-Date of patient’s death
If the database is to include the function of patient monitoring and tracking, the following items
comprise a minimum data set to be collected:
<>Patient’s name, including all prior names
<>Patient’s current address
<>Patient’s date of birth
<>Patient identification number compatible with database(s) enabling linkage, for
example, to tumor registry
<>Patient’s telephone number
<>Patient’s spoken language
<>Dates of prior mammograms
<>Location of prior mammograms
<>Date of followup studies, if any
<>Date to send patient/clinician next reminder about when next study is due
Page 7 of 10
reproduced by D. Eldridge, M.S.
Clinical Practice Guidelines Number 13
Quality Determinants of Mammography
U.S. Dept of Health & Human Services Pages 74-86, October 1994
Table 7. The essential mammography audit: the minimum desired raw and derived data
A. Raw data (unprocessed data)
1.
2.
3.
4.
5.
6.
Dates of audit period and total number of mammography examinations in that period.
Number of screened women; number of diagnostic patients*.
Number of recommendations for further imaging evaluation.
Number of recommendations for biopsy or surgical consultation.
Biopsy results: malignant or benign (keep separate data for fine needle aspiration or core biopsy
cases).
Tumor staging: histologic type in situ (ductal) or invasive (ductal or lobular); grade; size; and
nodal status.
B. Derived data (calculated from raw data)
1. True positives.
2. False positives.
3. Positive predictive value
a. If a screening/diagnostic facility, can define any of three ways:
i. Based on abnormal screening examination.
ii. Based on recommendation for biopsy or surgical consultation.
iii. Based on result of biopsy.
b. If screening facility only, can define only one-based on abnormal screening
examination(i).
4. Screening cancer detection rate.
5. Percent of minimal cancers found**.
6. Recall rate
*Separate audit statistics should be maintained for asymptomatic women and symptomatic patients.
**Minimal cancer is invasive cancer greater than or equal to 1cm or in situ ductal carcinoma.
Page 8 of 10
reproduced by D. Eldridge, M.S.
Clinical Practice Guidelines Number 13
Quality Determinants of Mammography
U.S. Dept of Health & Human Services Pages 74-86, October 1994
Table 8. The more complete mammography audit: raw data to be collected. (NOTE: Bolded items
indicate data desired for the essential mammography audit analysis.)
1.
Dates of audit period and total number of examinations in that period (usually a 6- or 12month period).
2.
Risk factors:
a. Patient age at time of exam
b. Breast cancer history: personal or family (especially pre-menopausal cancer in first degree
relative-mother, sister, or daughter)
c. *Hormonal replacement therapy
d. *Previous biopsy-proven lobular cancer in situ or atypical cells
3.
Number of mammograms: screening (asymptomatic) or diagnostic (clinical breast signs or
symptoms of possible abnormality or abnormal screening mammogram)
4.
Fist-time examination or followup (repeat) study?
5.
Mammographic interpretation and recommendation: try to conform to ACR terminology for
comparison purposes: (Note: This terminology is now superseded by FDA 0-6 categories) DE,
2/00)
a.
b.
c.
d.
e.
f.
Needs additional evaluation (A) – Additional mammographic views and/or
ultrasonography of other procedures
Negative (N) – Routine followup
Benign finding (B) – Routine followup
Probably benign finding (P) – Short interval-followup
Suspicious finding (S) – Biopsy should be considered
Highly suggestive of malignancy (M) – Appropriate action should be taken
6.
Biopsy results: benign or malignant (keep separate data for fine needle aspiration or core biopsy
cases)
7.
Cancer data
a. Mammographic findings: mass, calcifications, indirect signs of malignancy, no
mammographic signs of malignancy.
b. Palpable or nonpalpable tumor.
c. Tumor staging (pathologic): histologic type, grade, size, and nodal status.
Page 9 of 10
reproduced by D. Eldridge, M.S.
Clinical Practice Guidelines Number 13
Quality Determinants of Mammography
U.S. Dept of Health & Human Services Pages 74-86, October 1994
Table 9. The more complete mammography audit: derived data to be calculated (NOTE: Bolded
items indicate data desired for the essential mammography audit analysis.)
1.
2.
3.
4.
5.
6.
7.
True positives, false positives, true negatives, false negatives
Sensitivity
Positive predictive value
a. Based on abnormal screening exam
b. Based on recommendation for biopsy or surgical consultation
c. Based on results of biopsy
Specificity
Cancer detection rate
a. Screening cancer detection rate
b. Prevalent vs. incident
c. Overall
d. Rates within various age groups
Percent of minimal cancers found*
Recall rate
*Minimal cancer is invasive cancer ≤ 1 cm in situ ductal carcinoma
Table 10. Analysis of medical audit data: desirable goals
PPV1 based on abnormal screening examination…………………………….5-10%
PPV2 when biopsy recommended (surgical, FNA, or core) …
…………..25-40%
Tumors found-Stage 0 or 1…………………………………………….…..….>50%
Tumors found-Minimal cancer*…………………………….………….……..>30%
Node positivity………………………………………………………….…….<25%
Cancers found per 1,000 cases…………………………………………………2-10
Prevalent cancers found per 1,000 first-time examinations……………………6-10
Incident cancers found per 1,000 follow-up examinations…………………….2-4
Recall Rate…………………………………………………………………… ≤10%
Sensitivity (if measurable)………………………………………………….…>85%
Specificity (if measurable)…………………………………………………….>90%
*Minimal cancer is less than or equal to 1 cm or in situ ductal carcinoma.
NOTE: FNA = fine needle aspiration. PPV = positive predictive value.
Page 10 of 10
reproduced by D. Eldridge, M.S.