Download Worksheet for Critical Appraisal of Systematic Reviews/Meta

Critical Appraisal
Critical Appraisal of a Meta-analysis
Jill Radtke
University of Pittsburgh
1
Critical Appraisal
2
Worksheet for Critical Appraisal of Systematic Reviews/Meta-Analyses
Citation:
Martin, R.M., Gunnell, D., Owen, C.G., & Smith, G.D. (2005). Breast-feeding and
childhood cancer. International Journal of Cancer, 117(6), 1020-31.
What type of article is this (e.g., research/data-based, clinical paper, review,
editorial)?
Research/data-based
If this is a research/data-based article, what makes it this type of article? Identify
2-3 key characteristics of the article.
1. Methodology: The article/study seeks to obtain data in a systematic fashion
(e.g., the comprehensive, systematic search for articles; the choosing of articles
for inclusion, based on criteria determined a priori).
2. Style: The article’s results and discussion are presented in an objective and
frank manner (also discussing the limitations) in order that the reader may judge,
implement, question, and/or disregard the evidence.
3. Original Findings: The study is a meta-analysis, combining results of many
individual studies for new statistics that offer a more comprehensive answer as to
whether breastfeeding is associated with a decreased risk of childhood cancers.
State the research question posed by the authors.
In epidemiologic studies examining the association between breastfeeding and
childhood cancer, selected through systematic review for this study, what is the
association between any breastfeeding and childhood cancer across studies, as
compared to the never-having-been-breastfed child’s development of childhood cancer,
and are these associations different depending upon type of cancer?
What is my clinical question?
In breast milk samples taken weekly in an industrialized area from postpartum women
exclusively breastfeeding starting two days after birth and lasting until 26 weeks
postpartum, is there any biochemical marker or element, in presence, or concentration,
associated with an increased rate of childhood cancer development by 5 years of age in
these women’s offspring, as compared to a similar group of breastfed children in a rural
area?
Using PICO, identify the following if applicable:
P (= population): Children aged 0-19 whose cancer status/mortality was known, and
whose breastfeeding status as an infant could be classified as “never breastfed” or “any
breastfeeding”
I (= intervention): Breastfeeding
C (= comparison group): Children never having been breastfed
Critical Appraisal
3
O (=outcome): Childhood cancer development
Appraisal Guide
COMMENTS
I. Are the results valid/trustworthy?
1. Are the specific
The main research question, whether breast milk is
aims/research questions
associated with a decrease in the risk of childhood
clearly focused for this
cancer, is clear both in the abstract and introduction.
systematic review/metaAdditionally, the specific aims, to determine “the
analysis?
consistency of associations [between childhood
a) What is being reviewed?
cancer development and breastfeeding] across
b) What is the population?
studies and to determine whether associations
c) What are the independent
differed by cancer types” is clear in the introduction.
variables (i.e., factors, exposures, However, the types of childhood cancer the authors
interventions)?
are looking at and how breastfeeding is being
d) What are the dependent
classified (never vs. ever been) is not clearly
variables (i.e., outcomes
delineated. Rather, we are given this information in
responses)?
pieces throughout the article. This may be
e) Were all clinically relevant
appropriate for a meta-analysis, as the type of
(patient-important) outcomes
articles included should provide more direction and
considered?
specificity for specific aims. Nevertheless, giving
this information up front would have made the article
more readable.
This meta-analysis reviewed the Medline database
from 1966 (inception) until June 2004 for articles,
papers, letters, abstracts and review articles
pertaining to infant feeding and cancer. They also
employed the auto-alert system and reviewed
relevant reference lists (manually searched). In the
end, of 1,415 “hits,” 46 articles were selected as
relevant. Of these 46, 26 were selected for study
inclusion (the discarded studies were discarded for a
variety of reasons, including that some articles were
commentary or review articles or did not focus on
childhood cancer). Of these 26 articles, 2 were
cohort/nested case-control studies, and the rest
were case-control studies. All examined the
association between childhood cancer and
breastfeeding. The authors reviewed the individual
studies for quality and characteristics, as shown in
the tables in the article. The types of childhood
cancers reviewed in the 26 studies included
leukemia, acute lymphoblastic leukemia, acute
nonlymphoblastic leukemia, Hodgkin’s disease, nonHodgkin’s lymphoma, central nervous system
cancers, neuroblastoma, malignant germ cell
tumors, juvenile bone tumors, and other solid
cancers. The meta-analysis looked at the
associations of these cancers with the classification
of never having been breastfed as compared to any
or exclusive breastfeeding. The authors employed
Critical Appraisal
4
pooled odds ratios to determine if any breastfeeding
was associated with any of the childhood cancers
mentioned above, as well as childhood cancer in
general. They also looked at heterogeneity for
between study variation among many factors,
chosen a priori, that might have impacted study
results (i.e., study size, study design) using
Cochran’s Q-statistic and the I2 statistic. If too much
heterogeneity were found, it would not have been
meaningful or possibly even valid to pool the odds
ratios between the studies. The authors also looked
at breastfeeding specifically for greater than 6 to 8
months as compared to never having been
breastfed to determine any impact of prolonged
breastfeeding on childhood cancer. Finally, the
authors looked at the Egger test for small study bias,
in order to determine if the small studies utilized in
the review arrived at exaggerated effect sizes.
The actual populations used in the individual articles
included children from 0-19 years whose
breastfeeding status could be classified as “ever” or
“never.” Additionally, their development of childhood
cancer or mortality from cancer was known.
The independent variables are the child’s
breastfeeding classifications (e.g., never, ever,
greater than 6-8 months); this includes the duration
of breastfeeding. Age of the child might be
considered an independent variable, but year of
birth is similar, and this was considered a possible
confounder in the study.
The dependent variable was the rates of childhood
cancer: i.e., the number of children with each type of
cancer.
I believe the relevant outcome is considered in the
context of this study’s goal. We are interested in
childhood cancer development and its association
with breastfeeding in order to ascertain whether
breastfeeding should be recommended to mothers
for this potential benefit. The authors addressed
odds ratios of childhood cancer development for
each major type of childhood cancer and childhood
cancer in general when any or no breastfeeding
occurs. They found that breastfeeding’s effect on
childhood cancer is negligible, but should be
recommended anyway for its other known benefits.
Truly, the most important outcome here is cancer
Critical Appraisal
2. Is the search for studies
comprehensive (i.e., did the
authors look for the
appropriate sort of studies)?
a) What bibliographic databases
were used?
b) What years were searched?
c) What languages were
searched? d) Were the study
inclusion criteria appropriate (i.e.,
study design, participants,
intervention, and outcomes of
interest)?
e) Was the inclusion process
discussed?
f) Did at least 2 individuals review
articles for selection?
g) What process did they use to
reach agreement on article
selection?
h) Was agreement achieved?
i) Were all relevant studies
included in the review?
j) Were any important studies
missed? Explain.
5
development, in order to establish or negate the
relationship between childhood cancer and
breastfeeding, but other outcomes, such as mortality
and age at diagnosis of cancer and its association
with breastfeeding, might have been considered.
The Medline database was used, combining
keywords for infant feeding and cancer.
Additionally, auto-alerts were employed on Medline,
and the authors manually searched the reference
lists of eligible studies and previous meta-analyses
for the specific cancers.
Medline was searched from its inception in 1966 to
June 2004.
It was not mentioned what languages were
searched.
Yes, the study inclusion criteria appeared
appropriate. No specifications of specific study
design were mentioned under “Material and
methods,” however it was noted that several studies
were excluded because they were reviews and
commentaries. Thus, we see that the authors are
ensuring the quality of the meta-analysis by
including studies that are primary sources, and
excluding secondary sources. The age criterion, 019 years, seems appropriate, as we are studying
“childhood” cancer. Finally, the meta-analysis
criterion included studies comparing those ever
breastfed to those never breastfed, and estimates of
the association of having ever been breastfed to
cancer outcomes had to be able to be derived or
available in the study. This is appropriate, given the
main objective of our study: to determine the
association of breastfeeding with the development of
childhood cancer.
The inclusion process was discussed, as noted
above, under “Material and methods,” and reasons
for excluding certain studies were also discussed
under “results” on page 1021.
It was not mentioned how many individuals reviewed
the articles for selection. (Thus, “g” and “h” are n/a.)
There may be some relevant studies not included in
the review. The authors mentioned that they had to
exclude studies matching their inclusion criteria
because they were already mentioned in their article
previously. Additionally, they had to exclude some
Critical Appraisal
6
studies because estimates of the odds ratios could
not be derived. This does not mean, however, that
these studies did not offer valuable insight and
evidence for our research question. For this
particular meta-analysis, though, they did not fit.
Additionally, the authors might have tried to search
other databases, confer with “experts” in the area to
uncover unpublished sources of information, search
other languages, etc. It was not mentioned if these
techniques were employed or not, but they might
have contributed relevant and valuable data to our
meta-analysis.
3. Is the validity of the
included studies adequately
assessed?
a) What criteria were used to
evaluate validity of individual
studies (e.g., use of intention-totreat, if an intervention study;
efforts to obtain missing
information; efforts to minimize
publication bias; assessment of
methodologic quality)?
b) Were these criteria
appropriate? Explain.
c) Did at least 2 reviewers
examine individual studies for
validity?
d) What process did reviewers
use to reach agreement on the
validity of individual studies?
e) Was agreement achieved?
Additionally, I searched OVID Medline for the same
study years as the article’s authors used, using the
keywords, “infant feeding” and “cancer.” I had 15
“hits”, but I did not find any relevant articles fitting
the study’s inclusion criteria that the authors did not
use in their systematic review. However, one study
uncovered did seem to offer an important opposing
viewpoint. It looked at breastmilk samples’
contamination with heavy metals and how this can
contribute to cancer in later life for infants who
breastfeed (Lutter, Lyengar, Barnes, Chuvakova,
Kazbekova, & Sharmanov, 1998). While not meeting
study inclusion criteria completely, it does make an
important and valid point about environmental
influence on breastmilk and subsequent cancer
development in offspring. In my opinion, this should
be considered in future studies looking at childhood
cancer risk associated with breastfeeding.
The article mentions that, unlike randomized control
trials, there are no widely accepted criteria used to
judge the validity and quality of observational
studies (which this meta-analysis used). Therefore,
the study employed quality criteria chosen a priori
from previous meta-analyses and systematic
reviews examining infant feeding. They seem to be
appropriate, as they are chosen a priori. However,
the model criteria were derived from articles whose
titles implied they were looking at the association
between breastfeeding and indicators of overweight.
We are examining the relationship between
breastfeeding and childhood cancer. So, although
the criteria may be easily transferable to our metaanalysis, we cannot be sure. It would have been
appropriate to also consider criteria from systematic
reviews based on childhood cancer.
Some of the criteria used and accounted for by the
authors included study size (less than or equal to
Critical Appraisal
7
500, or greater than 500—cited by the authors as
the number used for effect size in meta-analyses for
breastfeeding), study design (cohorts considered
more robust than case-control), whether
breastfeeding was the main outcome measure,
whether maternal recall was used to assess
breastfeeding after 1 year (memory problems), early
exposure to infectious disease in the child (because
this has been associated with breastfeeding and
cancer, according to the authors), whether effects
estimates controlled for social factors in the child or
adult, reproductive factors, if the study was
population-based (potential for selection bias), if the
control or case response was less than 80%
(selection bias), whether prevalence of
breastfeeding was at least 70% (authors did not
justify this criteria), the region the study was done in
(as different views of breastfeeding, different cancer
rates in different regions), and the year of birth of the
child participants (as breastfeeding is viewed
differently in different generations, and formula has
changed dramatically over the years). This
information can be found on p.1026, Table II.
Throughout the analysis/results, several studies are
mentioned by the authors as having limited validity
for one reason or another, and the analysis is done
with and without these studies for this reason. For
example, one study was only given only in abstract
form and was thus considered poor quality for the
meta-analysis. Removing it changed the
association between Hodgkin’s disease and
breastfeeding quite significantly (p.1026).
It was not mentioned whether more than one
reviewer arrived at the validity criteria or assessed
the studies for validity.
4. What are the study
characteristics?
a) How many individual studies
were included?
b) What types of studies were
included (e.g., randomized
controlled trials, case-control
studies, cohort studies, crosssectional studies)?
c) What were the sample sizes
for the studies? If applicable,
what are the sample sizes for the
26 individual studies were included. These included
2 cohort/nested case-control studies, and 24 casecontrol studies.
The study sample sizes were in a wide range.
Some studies looked at individual cancer types and
gave number of cases with each type. Some
studies looked one cancer type or childhood cancer
in general and gave the total sample size. The
smallest sample was composed of 33 subjects and
looked at childhood cancer in general. The largest
study included over 3,000 subjects, and broke down
Critical Appraisal
study groups?
d) In what countries were the
studies conducted?
e) If an intervention was the
primary independent variable
under investigation: What was
the duration of treatment?
What was the adherence to the
study treatment?
Were adverse results reported?
8
childhood cancer into several specific cancer types
(and included number of subjects with each type of
cancer).
The studies were collected from a wide variety of
countries and areas, including among them, the
United States, England, Ireland, Greece, Australia,
Italy, United Kingdom, United Arab Emirates,
Shanghai, Canada, New Zealand, etc. The metaanalysis was very diverse in study location.
The “intervention” was breastfeeding, and the
classifications used in the meta-analysis were “ever”
or “never” breastfed. Additionally, the authors did a
second small analysis comparing “never” breastfed
to breastfed greater than 6-8 months. This was the
only reference to duration of breastfeeding. We are
not told how long those who “ever” breastfed
nursed. Because these studies were observational
in nature, breastfeeding was not classified whether it
was “adhered” to. Instead, retrospectively, these
children’s parents classified whether they did or did
not ever breastfeed.
5. What are the results of
systematic review/metaanalysis?
a) Were the results in terms of
effect sizes of the individual
studies clearly reported?
b) Were these results consistent
from individual study to individual
study? Explain.
c) Were the results of the
different studies similar (e.g.,
Were tests of homogeneity
conducted? What did they
show?)
d) Were the individual study
findings combined statistically? (If
results were combined, did it
make sense to do so? Explain.
e) If results were not combined,
should they have been?
Explain.)
f) Were the reasons for study
Although some studies showed no significant
decrease in risk for certain childhood cancers
associated with breastfeeding, it was not noted that
breastfeeding was ever positively associated with
childhood cancer significantly.
The odds ratios and 95% confidence intervals (CI’s)
for the individual studies were reported for childhood
cancer (individual types and overall, depending on
study), comparing never breastfed children to
breastfed children. This was shown in Figure 1
(p.1027) in the “Results” section, but was difficult to
read. Some studies’ odds ratios were discussed
individually, but most were not. Additionally, for the
“other” cancers in childhood, such as germ cell
tumors (in which only one study was used), no odds
ratios were provided for any studies. We are simply
told that breastfeeding and that cancer are not
significantly associated.
For childhood cancers as a whole, it was noted that
there was considerable variation in effects estimates
from each individual study comparing all childhood
cancer (this was attributed to large between study
heterogeneity, and one particular study that largely
contributed to the heterogeneity). Judging by Figure
1 and the article’s commentary, studies looking at
acute lymphoblastic leukemia, acute
Critical Appraisal
variations discussed?
9
nonlymphoblastic leukemia, and Hodgkin’s disease
had variable odds ratios and 95% CI’s. However, it
was very difficult to read Figure 1. These results in
table form would have been helpful.
Tests of homogeneity were not used. Instead, tests
of heterogeneity were utilized to detect differences
in between-study quality criteria, as mentioned
previously: the Q-test for heterogeneity and the I2
value (the I2 value was used to supplement the Qstatistic, because its power to detect heterogeneity
is not affected by a small number of studies, which
we have here). These tests showed considerable
heterogeneity among studies looking at all childhood
cancers. Studies looking at all childhood leukemia
showed moderate heterogeneity (I2 = 61%). Studies
looking at acute lymphoblastic leukemia, and
Hodgkin’s disease, and neuroblastoma showed little
between study heterogeneity (I2 =16%, 0%, and 0%,
respectively), however, only 3 studies were chosen
looking at neuroblastoma. Non-Hodgkin’s
lymphoma, other central nervous system cancers,
juvenile bone tumors, and germ cell tumors failed to
establish an association in this meta-analysis with
breastfeeding, and this may be due to the small
number of studies looking at these cancers.
Therefore, estimates of heterogeneity were not as
meaningful in these cases, and were classified as
either 0% or N/A.
The individual study findings were combined into
pooled odds ratios. This made sense, as the
studies were looking at the same variables (ever or
never breastfed and childhood cancer). Studies that
looked at the same childhood cancers were pooled
together, and all noted whether the child was ever or
never breastfed. Additionally, it made sense to pool
to the odds ratios together, as odds ratios were
available or could be derived for each study chosen
for inclusion, per the authors.
6. What are the methodological
The reasons for the study variations (heterogeneity)
are discussed. The authors note that one particular
study, by Smulevitch et al., contributed to much
heterogeneity for several types of childhood cancer
between studies. They state that this is so, because
about 1% of controls were breastfed less than a
month, which contaminates the control group. By
removing the study, between-study heterogeneity for
these types of cancer are reduced.
There are a number of methodological weaknesses.
Critical Appraisal
strengths and weaknesses of
this systematic review/metaanalysis?
10
First, there were a small number of studies analyzed
in the meta-analysis, per each type of childhood
cancer. Thus, the authors comment that significant
associations between breastfeeding and rates of
childhood cancer might not have been found, when
in fact, they do exist. Second, the study is based on
observational studies, which leads to the very real
possibility of confounding. Although the authors
controlled for some of these factors (and found that
the results did not change when controlled for) it
was impossible to rule potential masking of effects,
because the etiology of breast milk’s role in
childhood cancer development is poorly understood.
Randomized control trials in meta-analysis are more
powerful, because they nearly eliminate
confounding.
Another weakness is the possibility of publication
bias. That is, perhaps significant associations
between reduction in certain childhood cancers and
breastfeeding were reflective of the fact that studies
finding no effect were not published. The authors
did not note whether they searched for negative
findings or unpublished literature, which may have
served to ease our discriminating minds. Other
methodological weaknesses include the fact that
control response rates were less than 80% in most
studies included (possible selection bias, as
participating controls are likely to be more socially
affluent and breastfeed), 85% of the studies were
based on maternal recall of infant feeding (possible
recall bias), and most studies did not look at the
timing of breastfeeding initiation, exclusivity of
breastfeeding, and supplementation. Because of
this last point, it was difficult to establish dose
effects of breastfeeding, which would have been
more methodologically sound in tying reductions in
childhood cancer to breastfeeding.
The methodological strengths of this article included
the fact that the authors attempted to control for
many possible confounders (listed in Table II),
electronic literature searches were supplemented
with manual searching of reference lists and autoalerts, a wide range of time was utilized in the
search, most childhood cancers were considered, as
well as childhood cancer as a whole, and the
authors did statistical analysis with and without
studies that accounted for a large amount of
variability or had questionable validity, in order to
ensure that meta-analysis results were more valid
Critical Appraisal
11
and generalizable.
II. What are the results/findings?
1. What are the overall results
A summary measure was given for rates of all
of the systematic review/meta- childhood cancer’s association with breastfeeding:
analysis?
that is, there is a 22% decrease in childhood cancer
a) If a summary measure was
with a 95% CI of 1-39% associated with
given, how large was the overall
breastfeeding. The pooled odds ratio for this is 0.96
effect (e.g., treatment effect,
(95% CI = 0.88-1.04), meaning that there is not
measure of association)?
strong evidence that breastfeeding is associated
b) Are the results clinically
with a decrease in childhood cancer. In fact, the
meaningful? Explain.
number needed to treat is given by the authors on
c) Based on the confidence
p.1030: “breast-feeding would prevent at most 5% of
intervals of the summary
these cases, if the prevalence of breast-feeding was
measure (if provided), how
increased from 50% to 100%.”
precise was the result? Explain.
d) Does the lower limit of the
The authors deny that the results are clinically
confidence interval include
meaningful, as evident in their number needed to
clinical relevant effects?
treat statement above. They state that there is
e) Does the upper limit exclude
moderate evidence that breastfeeding is associated
clinically relevant effects?
with a decrease in acute lymphoblastic leukemia,
Hodgkin’s disease, and neuroblastoma (but the
latter two are based on one study each). The
authors do recommend breastfeeding for other
health benefits, but even state in the abstract, “Even
if causal, the public health importance of these
associations may be small.”
Based on the 95% CI of the odds ratio, (0.88-1.04),
the result was within a narrow window (rather
precise), but precisely non-significant. A CI of an
odds ratio that includes 1 (which 0.88-1.04 does)
cannot claim that decreased childhood cancer rates
are associated with breastfeeding, any more than
not breastfeeding. The confidence interval’s upper
limit probably greatly limits clinical significance,
because it is so close to 1 (1 being equal odds of
developing childhood cancer with or without
breastfeeding). However, it does not exclude
clinical significance, because it is greater than 1
(indicating that breastfeeding, as opposed to not
breastfeeding, may be associated with childhood
cancer). The lower limit does not include clinical
relevance—it is very close to 1, in the opposite
direction, indicating that not breastfeeding may be
associated with decreased rates of childhood cancer
to a greater degree than breastfeeding (the opposite
of our hypothesis), but this was not significant.
III. How can I apply the results/findings?
1. What relevance do the findings As the results did not provide much clinical
have to nursing practice?
relevance in terms of breastfeeding’s association
Critical Appraisal
12
with decreased rates of childhood cancer, nurses
must use caution in advising parents to breastfeed
for this reason. However, the results do show
marginal significance in the association of
breastfeeding with decreased rates of childhood
acute lymphoblastic leukemia, Hodgkin’s disease,
and neuroblastoma. Thus, as nurses we must
consider the risk:benefit ratio. Breastfeeding has
been shown to have many other positive outcomes
in children (decreased allergies, ear infections, etc.),
and this study, while not showing definitive evidence
that breastfeeding decreases childhood cancer,
does not show any increased risk of childhood
cancer associated with breastfeeding. Thus, as
nurses we should recommend breastfeeding for a
myriad of reasons, and for the fact that it may
possibly play a role in decreasing some childhood
cancer.
2. Discuss how the findings can
be applied to practice.
These findings also are relevant in the sense that
we need more research, from nurses and other
disciplines, in the area of breastfeeding’s
association with childhood cancer. Particularly, as
the authors mention, we need studies looking at the
duration and exclusivity of breastfeeding and its
relationship to childhood cancer.
As stated above, these findings suggest that nurses
should continue to encourage and support
breastfeeding for the many health benefits that have
been confirmed in both mother and child. As this
meta-analysis did not show that childhood cancer is
decreased for breastfed children, we should not
advocate breastfeeding to families based on this
reason. To do so would be unethical, as we have
no sound evidence at this point to confirm the
relationship.
Critical Appraisal
13
References
Lutter, C., Lyengar, V., Barnes R., Chuvakova, T., Kazbekova, G., &
Sharmanov, T. (1998). Breast milk contamination in Kazakhstan: implications for
infant feeding. Chemosphere, 37(9-12), 1761-72.