Download Abstracts of the 19th Cochrane Colloquium

Abstracts of
the 19th
Cochrane
Colloquium
Editorial
Abstracts of the 19th Cochrane Colloquium
Resúmenes del 19◦ Colloquium Cochrane
1
2
Oral sessions
Supplement 2011
A. Knowledge translation
Knowledge translation and communicating evidence (A1)
Knowledge translation and communicating evidence (A2)
Knowledge translation and communicating evidence (A3)
Knowledge translation and education (A4)
Knowledge translation and communicating evidence (A5)
Knowledge translation in patient safety (A6)
3
6
8
9
11
13
B. Methods
Diagnostic test accuracy review methods (B1)
Searching and information retrieval (B2)
Searching and information retrieval (B3)
Methods for preparing reviews (non-statistical) (B4)
Methods for preparing reviews (statistical and non-statistical) (B5)
Statistical methods (B6)
Investigating bias (B7)
Investigating bias (B8)
14
16
18
21
22
24
25
27
C. Other topics
Education and training (C1)
Global health and equity (C2)
Consumer issues and shared decision making (C3)
Editorial processes and supporting review authors (C4)
29
34
36
37
Posters sessions
Posters sessions
41
Workshops
Workshops
205
Annex LatinCLEN
Sesiones orales
Sesiones de pósteres
235
241
Annex REDIBAM-GPC and JBI COnNECT+
Talleres REDIBAM-GPC
Taller JBI COnNECT+
249
251
Author Index
Author Index
253
Abstracts available online at www.cochrane.org
EDITORIAL
Abstracts of the 19th Cochrane Colloquium
maintaining some at their own. A similar approach is taken with the
VI International Conference on Patient Safety, although this event only
overlaps with the Colloquium the first day and a half.
Apart from this book, the Colloquium Programme resumes the content
of the different plenaries, which will be recorded and posted in the
web afterwards, and the high number of workshops and meetings that
have been scheduled. All together, it is a good demonstration of the
vitality of the Cochrane Collaboration and the variety of topics that are
being addressed.
Finally, it is necessary to remark and appreciate the partnership of
Wiley & Sons in the publication of this book.
Dear colleagues,
This is the Abstracts book where the oral presentations and posters of
the 2011 Madrid Colloquium are described: there are 72 contributions
that will be presented orally, grouped in 3 parallel sessions and with
four presentations each. The 393 included posters have been assigned
to one of three sessions that will take place during the morning coffee
breaks. We really thank all those who evaluated the abstracts received,
since it was a demanding task given the amount of work and the
quality of most studies.
You also can find an annex with the abstracts to be presented
at the LatinCLEN (Red Latinoamericana de Epidemiologı́a Clı́nica)
independent oral and posters sessions. LatinCLEN and REDIBAM-GPC
(Red Iberoamericana de Guı́as de Práctica Clı́nica) celebrate their
regular meetings as part of the Colloquium, sharing many activities but
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Xavier Bonfill
Director
Iberoamerican Cochrane Centre
1
Cochrane Database Syst Rev Suppl 1–1 (2011)
DOI:10.1002/14651858.CD000003
EDITORIAL
Resúmenes del 19◦ Colloquium Cochrane
Se trata de un enfoque similar con el de la VI Conferencia Internacional
sobre la Seguridad del Paciente, aunque este evento sólo se solapa
con el Colloquium el primer dı́a y medio.
Al margen de este libro, el Programa del Colloquium aporta el
contenido de las sesiones plenarias, que serán grabadas y publicadas
en el web más tarde, y el gran número de talleres y reuniones que
se han organizado. El conjunto de estas actividades es una buena
demostración de la vitalidad de la Colaboración Cochrane y de la
variedad de temas que se abordan.
Por último, es necesario destacar y valorar la contribución de Wiley &
Sons en la publicación de este libro.
Estimados colegas,
Éste es el libro de resúmenes donde se describen las presentaciones
orales y pósteres del Colloquium 2011 de Madrid: serán 72 las
contribuciones que se presentarán oralmente, agrupadas en tres
sesiones paralelas y con cuatro presentaciones en cada una. Los
393 pósteres incluidos han sido asignados a una de las tres sesiones
que tendrán lugar durante las pausas-café de la mañana. Realmente
queremos agradecer a todos aquellos que desde el Comité de Abstracts
han evaluado los resúmenes recibidos, ya que la tarea ha sido difı́cil
por la cantidad de trabajo y la calidad de la mayorı́a de los estudios.
También publicamos un anexo con los resúmenes que se presentarán de
forma independiente en las sesiones orales y de pósteres de LatinCLEN
(Red Latinoamericana de Epidemiologı́a Clı́nica). LatinCLEN y REDIBAM
GPC (Red Iberoamericana de Guı́as de Práctica Clı́nica) celebran sus
reuniones regulares como parte del Colloquium, donde van a compartir
muchas actividades aunque manteniendo algunas de las suyas aparte.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Xavier Bonfill
Director
Centro Cochrane Iberoamericano
2
Cochrane Database Syst Rev Suppl 2–2 (2011)
DOI:10.1002/14651858.CD000003
Abstracts available online at www.cochrane.org
Abstracts of the 19th Cochrane Colloquium
A1O2
Using evidence to stop ineffective practice: The
UK Cochrane Centre (UKCC) and the National
Institute of Health and Clinical Excellence (NICE)
work together
Oral sessions
A1 Knowledge translation and
communicating evidence
A1O1
Presenting the results of a Cochrane review to
policy makers in a video format
Docherty M1 , Garner S2 , Littlejohns P2 , Clarke M3
1 Institute of Psychiatry, Kings College London, UK; 2 NICE, UK; 3 Centre for
Public Health, Queen’s University Belfast, Northern Ireland
Rosenbaum S1 , Glenton C1
1 Norwegian Knowledge Centre for the Health Services, Norway
Background: All healthcare systems face increasing pressure to
control expenditure without compromising quality or patient safety.
Disinvestment in clinically- or cost-ineffective interventions is thought
to be key to transferring resources to higher-value care. Objectives:
NICE and the UKCC undertook a pilot project using Cochrane reviews
as an evidence base to identify low-value interventions for local NHS
disinvestment. Methods: Over six months all Cochrane reviews
concluding that an intervention could not be recommended or should
be used only in research were identified by UKCC and evaluated by
NICE. (Figure 1). The potential impact of stopping the intervention was
assessed against four domains: quality of care, patient/carer experience,
patient safety and productivity savings. (Table 1). Interventions were
excluded if they: were not relevant to UK practice; required additional
investigation; or were unlikely to achieve gains in any domains.
Evaluated topics were published as reports suggesting disinvestment
opportunities for local healthcare providers. (Table 2) Results: Of
the 65 reviews appraised, 43% provided candidate interventions for
local disinvestment. (Table 3) Quantification of potential productivity
savings was difficult due to the absence of NHS usage data. Most
could not be quantified; one suggested potential savings of £2 million.
Many reviews required additional analyses by NICE. Furthermore,
many interventions were no longer used or had insufficient evidence
to refute potential benefits, making disinvestment recommendations
either irrelevant or inappropriate. Conclusions: This is the first
large-scale systematic application of a gold standard evidence base
to the problem of disinvestment. It challenges assumptions that
disinvestment can produce large savings, highlights the importance of
robust processes to evaluate interventions when research evidence is
inadequate, and demonstrates the need for detailed data on healthcare
usage and expenditure. It identifies a translational gap between the
requirements of decision makers and the content of Cochrane reviews,
highlighting a need to improve the consistency and precision of
Cochrane review conclusions.
Background: There is growing attention to how The Cochrane
Collaboration can best present the results of Cochrane reviews in a
manner that is both reliable and accessible. A number of research and
development projects have focused on the development of summary
formats aimed at clinicians, consumers and policy makers. While
these projects have brought us to presentation formats that are both
reliable and accessible, a number of challenges remain, such as readers
sometimes ignoring or misunderstanding important methodological
concepts. Objectives: To develop a video presentation format that
can: • help policy makers understand the results of a specific review
• help policy makers understand general methodological concepts
• attract a wider audience to Cochrane reviews • be easily accessible
in multiple languages • be easily transferable to other reviews with
a minimum of resources Methods: We used the knowledge gained
from our developmental work with Summary of Findings Tables, Plain
language summaries and SUPPORT summaries for policy makers. We
selected the Cochrane review on the effects of lay health workers
for maternal and child health and tuberculosis. We developed a
10-minute video using low-tech, low-cost solutions, and presenting
what we know about the benefits and harms of this intervention,
what we do not know, and how confident we can be in these
results. We pilot tested the video among a relevant audience before
developing a second version of the video and publishing it on YouTube.
Results: Pilot testing led us to increase the information about what
we do not know (i.e. where information was lacking for important
subgroups and outcomes). Responses were largely positive, including
appreciation of narration and use of tables, maps and photos. A few
people were concerned about length. A second round of testing is
being planned. Conclusions: Video is a feasible format for reliable,
accessible and low-cost presentation of Cochrane Reviews. http://
www.youtube.com/watch?v=0zHHhpE5Rb0
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
3
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
4
Figure 1 (A1O2). Flow chart of NICE review process.
UKCC
Manager
Tech lead
Deadline
for report
Tech
support
Send new DNDs
For each DND
QA
Analyse and formulate recommendation
Discuss
recommendation
DND
DND
rejected accepted
Inform of
reason
Costing
support
Request costing and background
Prepare
Prepare
background costing
Prepare Cochrane statements
For QA
Send statements for QA
Sign off
QA statements
Finalise statements and
send for sign off
Sign off statements for publication
Table 1 (A1O2). Cochrane QP topic selection criteria.
Criterion
Examples
Potential productivity and cash-releasing savings
Direct cash savings: substituting prescriptions of minocycline for acne vulgaris with
cheaper tetracyclines.
Releasing human or physical resources: stopping routine fetal and umbilical Doppler
ultrasound in normal pregnancy would free up clinic time for other patients or other
activities.
Potential impact of disinvestment on quality of
clinical care delivered to patients and on clinical
outcomes.
The use of joint lavage alone, without debridement, in patients with osteoarthritis of the
knee has not been shown to result in relevant benefits. Therefore, stopping it will
result in improved quality of care.
Potential impact on patient safety
There is no evidence that selective serotonin reuptake inhibitors are effective as a
treatment for children with autism and there is some evidence they can cause harm.
Stopping the routine use of these medications in children may improve patient safety.
Potential impact on patient and carer experience
Stopping the routine replacement of peripheral venous catheters without clinical
indication after 72–96 hours was predicted to reduce unnecessary patient pain and
discomfort without compromising clinical quality or patient safety.
A1O3
Wording GRADEd evidence statements
and recommendations
Kunnamo I1 , Kattainen A1 , Pelttari H1 , Alenius H1 , Atula S1 , Jousimaa J1 ,
Teikari M1
1 Duodecim Medical Publications Ltd, Finland
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: The GRADE process requires that the quality of evidence
and strength of recommendation are assessed, and the reasons for
downgrading or upgrading the evidence, and for giving strong or
weak recommendations are described transparently. Since 2000, we
have developed over 4000 evidence summaries containing evidence
statements in PICO (Patient, Intervention, Comparison, Outcome)
format, and recently we started to include recommendations. The main
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
5
Table 2 (A1O2). Summary of Cochrane Quality and
Productivity topic reports.
Section
Explanatory notes
Title
Cochrane review title reworded to state
intervention and patient population
included in the Cochrane QP Topic.
Introductory text
A standard description of the Cochrane QP
Topic process.
Summary
• Quotation from Implications for
practice summary of Cochrane review
• NICE summary of review authors’
conclusions
• NICE comment
Details of Cochrane
review
• Cochrane review title and hyperlink
• Citation
• Date review content was assessed as
up-to-date
• QIPP category
• Relevant codes: ICD 10, programme
budget.
Background
• Information about the condition and
intervention
Evidence
• Summary of evidence (Information
usually includes number of trials,
outcomes measures used, inclusion
criteria, etc.)
• Relevant NICE guidance
• Other NHS Evidence accredited
guidance
Potential impact on
quality of NHS care
• Impact on clinical quality
• Impact on patient safety
• Impact on patient and carer
experience
Likely ease of
implementation
• Time taken to implement
• Healthcare sectors affected
• Likely stakeholder support
source for the evidence summaries are Cochrane reviews. Objectives:
To develop standard expressions in the English language for the
evidence and recommendations reflecting different grades of quality
and strength, and stating reasons for downgrading and upgrading.
Methods: Using the publications of the GRADE Group as a starting
point, the EBM Guidelines Editorial Team developed standard wording
in a consensus process. A web-based questionnaire was used in
2010 to examine the confidence of professionals in the evidence
supporting an intervention when the same evidence statement used
different verbs. The 132 respondents from several countries were
mainly physicians involved in developing guidelines and systematic
reviews. Results: On a 7-point Likert scale where 1 represented the
highest and 7 the lowest confidence in the evidence, different verbs
received mean scores as follows: is 2.12, seems 3.65, appears 3.67,
may 4.08, might 4.39, might possibly 5.02. Examples of evidence
statements and recommendations using standard expressions are
shown, and a website is presented that contains all the expressions
used in commenting on the evidence and the recommendations.
Conclusions: There is wide variation in professionals’ interpretation of
different wording, but wording that results in differences in confidence
can be found. We selected the following verbs to express the four
grades of the quality of evidence: high – is; moderate – appears;
low – may; very low – might possibly. Standard expressions can
make evidence and recommendations more transparent and easier to
interpret by professionals.
A1O4
Scientific evidence and practice: Bridging the
gap. A European Monitoring Center for Drugs
and Drug Addiction (EMCDDA) project to promote
Best Practice in drug addiction
Bo A1 , Allara E2 , Ferri M1
1 EMCDDA, Lisbon, Portugal; 2 University of Turin, Italy
Background: Best Practice is the best application of the available
evidence to current activities in the drug field. Evidence-based
knowledge and practice, that is the experience and lessons learned
from the implementation of evidence-based interventions, are two
key dimensions of the Best Practice concept. Objectives: To
develop a knowledge-base on the EMCDDA website providing reliable
information on the latest scientific evidence as well as on the practical
aspects of implementation throughout Europe. Methods: We search
for systematic reviews and summarise resultsin plain language modules
following the Patients-Interventions-Comparisons-Outcomes-Type of
studies logic. Interventions are scored for the level of evidence based
on the GRADE assessment system, and results are grouped by the
level of impact on patients (the upper level being ‘Beneficial’ and
the lower ‘Evidence of ineffectiveness’). Tables with summaries
of ongoing studies are also available. The EMCDDA’s network
Table 3 (A1O2). Summary of rationale for exclusion of reviews.
Rationale for exclusion
Number of reviews excluded
Not licensed/ available in the UK
6
Clinical question not relevant to
UK practice
5
Recommended change in practice
was in line with standard UK
practice therefore no ‘gains’
3
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Examples
Intravenous alpha-1 antitrypsin augmentaion therapy for treating
patients with alpha-1 antitrypsin deficiency and lung disease
Hosptialisation and bed-rest for multiple pregnancy
Hydralazine for essential hypertension Antibiotics to reduce
post-tonsillectomy morbidity
Heroin maintenance for chronic (treatment resistant)
Heroin-dependent individuals Cervico-thoracic or lumbar
sympathectomy for complex regional pain syndrome
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
6
Table 3 (A1O2). Continued.
Rationale for exclusion
Additional analysis (for example
non-RCT data or safety data)
would be required before a
disinvestment conclusion could
be confirmed.
Number of reviews excluded
10*
* For 4 of these, there were specific
patient safety concerns about the
consequences of disinvestment
The review provided evidence that
the intervention was potentially
beneficial and thus not a
potential candidate for
disinvestment
Complete or partial clash with
existing NICE guidance e.g. The
intervention may be
recommended by NICE in
particular circumstances which
were not directly considered in
the review
Complex or controversial topic
requiring more a) extensive
analysis b) review of evidence
or c) detailed explanation of
findings than afforded by this
process
NICE guideline in progress
specifically addressing topic
4
Tailored interventions to overcome identified barriers to change:
effects on professional practice and health care outcomes
Interventions for smoking cessation and reduction in individuals
with schizophernia
Drug therapy for the management of cancer related fatigue
5
Adenoidectomy for otitis media in children
Rehabilitation interventions for improving physical and
psychosocial functioning after hip fracture in older people
Carbamazepine for acute and chronic pain in adults
Glatiramer acetate for multiple sclerosis
1
Antibiotics for preterm rupture of membranes
A2 Knowledge translation and
communicating evidence
A2O1
Impact of reporting the number treated
needlesslyon perceived effectiveness and
decision to adopt an intervention
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Teicoplanin versus vancomycin for proven or suspected infection
Physical methods for preventing DVT in stroke
Minimally invasive surgery versus open surgery for the treatment
of solid abdominal and thoracic neoplasms in children
Orthodontics for treating temporomandibular joint (TMJ) disorders
Anticoagulant therapy for deep vein thrombosis (DVT) in pregnancy
Lay health workers in primary and community health care for maternal
and child health and the management of infectious diseases
Noninvasive positive pressure ventilation as a weaning strategy
for intubated adults with respiratory failure
6
of national focal points regularly submits examples of interventions
that are assessed according to the level of evidence. Results:
The Best Practice Portal (BPP) (www.emcdda.europa.eu/best-practice)
currently contains modules on treatment, and harm reduction and
prevention. The modules are biannually updated with new evidence. A
system for collecting relevant questions and commissioning Cochrane
systematic reviews is being studied. Experience from practice is
presented in a separate section, including a collection of European
standards and guidelines and a collection of real-life projects from 30
European countries (EDDRA database). Conclusions: Identification of
evidence and implementation experiences has important implications
in highlighting research gaps as well as defining research priorities.
The BPP is the result of our first effort to bridge evidence and practice,
to disseminate and promote evidence-based interventions throughout
Europe and foster a common approach.
Nabhan A1
1 Ain Shams University, Egypt
Examples
Background: The success of evidence-based practice depends on
effective communication of clinical trial results. It is suggested that
trials should not only provide figures that indicate the positive impact
of interventions, but also those that indicate how many would not
benefit, the ‘number treated needlessly’ (NTN). Objectives: To assess
the impact of explicit reporting of NTN on perceived effectiveness
and decision to adopt an intervention among professionals and
medical students. Methods: Cross-sectional survey with random
allocation to different formats for presenting effect measure (absolute
risk reduction (ARR), relative risk reduction (RRR), number needed to
treat (NNT), and NTN). The clinical scenario was ‘‘use of statins for
the primary prevention of cardiovascular disease’’. The outcomes were
persuasiveness (measured as a decision to adopt the intervention) and
perception (measured on an 11-point scale of perceived effectiveness).
The study included 1472 participants. Results: The percentage
of participants who decided to adopt the intervention was 92.4%,
87.8%, 67.7%, and 26.2% when RRR, NNT, ARR, and NTN were
used, respectively. Comparing ARR with NTN, perceived effectiveness
was smaller with NTN [Mean Difference (MD) −1.68 (95% Confidence
Interval (CI) −2.06 to −1.30)] and Odds Ratio (OR) for decision to
adopt the intervention was 0.17 (95% CI 0.12 to 0.23). Comparing
RRR with NTN, perceived effectiveness was smaller with NTN [MD
−3.88 (95% CI −4.27 to −3.49)] and OR for decision to adopt the
intervention was 0.03 (95% CI 0.02 to 0.05). Comparing NNT with
NTN, perceived effectiveness was smaller with NTN [MD −3.14 (95%
CI −3.58 to −2.70)] and OR for decision to adopt the intervention
was 0.05 (95% CI 0.03 to 0.07). Conclusion: Explicit reporting of
the number treated needlessly has a significant impact on perceived
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
7
effectiveness and decision to adopt an intervention. It is likely to help
people to make a more balanced, well-informed decision.
A2O2
The use of synthesis methods in support
of developing a new surveillance initiative
Rajic A1 , Tuševljak N2 , Dutil L1 , McEwen S2
1
Public Health Agency of Canada, Canada; 2 University of Guelph, Canada
Background: Research synthesis methods can be used to generate
evidence-based inputs for policy and decision-making. We describe
the use of three research synthesis methods: scoping study, systematic
review (SR) and meta-analysis (MA) in support of developing a potential
new surveillance program for zoonotic bacteria and antimicrobial
resistance on seafood in Canada. Methods: In consultation
with principal stakeholders, a scoping study was conducted to
characterize the global primary research investigating zoonotic bacteria,
antimicrobial use and antimicrobial resistance. Evidence-maps were
evaluated for 8 selected zoonotic bacteria and all finfish, crustacean
and mollusc species. SR-MA were prioritised to evaluate retail (point
of consumer purchase) prevalence of E. coli, Salmonella, Aeromonas
and Vibrio, which had the most underpinning research, on products
frequently consumed based on Canadian consumption and import data;
salmon, tilapia, shrimp/prawns, oysters/clams/mussels. Random-effects
meta-analyses were conducted on subsets of homogenous data by
aquatic species, bacteria, and sampling region when >2 lines of data
were available, otherwise summarized by median/range. Results:
Evidence maps included 1760 relevant papers and SR-MA was
prioritized for 56. Pooled prevalence data for Aeromonas, E. coli
and Salmonella for retail salmon were 13% (6 to 27%), 2% (0
to 11%) and 0% (0 to 5%) respectively. Many subsets were
heterogeneous. The highest proportion of studies reported Vibrio
prevalence on clams/oysters/mussels ranging from less than 10% to
greater than 90%. Overall, there were few studies, and data gaps
were identified for some large seafood exporting countries including
Canada. Conclusions: Through the scoping study we have rapidly
identified and evaluated the distribution, relevance and overall utility of
research pertaining to this broad topic and prioritized focused SR-MA
questions relevant to our stakeholders. The data gaps and limitations
identified may be used to prioritize future research and the knowledge
summaries from the scoping study and SR-MA may be used in the
design of targeted, risk-based surveillance, prioritising initial baseline
bacteriological surveys and as inputs for risk assessments.
A2O3
Grading evidence and recommendations:
Randomised experimental evaluation of four
grading systems
Perez G1 , Cuello C2 , Pacheco K3
1 Red Cochrane Iberoamericana, Mexico; 2 ITESM, Red Cochrane
Iberoamericana, Mexico; 3 ITESM, Mexico
Background: More than 60 systems for grading evidence and
recommendations have been described, reflecting discrepancies among
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
those creating and synthesizing evidence. Objectives: To evaluate
the effect of presenting a recommendation using different grading
systems to determine to what extent the system changes the clinician’s
eventual response to a particular clinical question. Methods:
Randomised experimental study in clinicians’ offices and academic
settings. Paediatricians and paediatric residents in private and
public practice in Mexico were handed case notes of a child with
diarrhoea and a question about clinician preference for using an
anti-diarrhoeal medication. The same evidence was provided in a
clinical recommendation but with different presentation according to
the following grading systems: NICE (National Institute for Health and
Clinical Excellence), SIGN (Scottish Intercollegiate Guideline Network),
GRADE (Grading of Recommendations Assessment, Development and
Evaluation) and CEBM (Centre for Evidence-Based Medicine, Oxford).
The main outcome measure was the mean change in direction from
baseline response (measured on a 10 cm visual scale and a Likert
scale) and among groups. Results: 216 subjects agreed to participate.
Most participants changed their decision after reading the clinical
recommendations (mean difference (MD) 0.7 cm (95% confidence
interval (CI) 0.29 to 1.0; P < 0.001)). By groups, mean change (95%
CI) from baseline was 0.04 (−0.68 to 0.77) for NICE, 0.31 (−0.41 to
1.05) for SIGN, 2.18 (1.48 to 2.88) for GRADE and 0.08 (−0.52 to
0.69) for CEBM (P = 0.007 between groups). In a final survey, a small
difference was noted regarding the clarity of the results presented with
the GRADE system. Conclusions: The clinician’s decision to use a
therapy was influenced most by the GRADE system. Trial registration
number NCT00940290.
A2O4
Understanding subjectivity in the meta-analytic
process: Meta-analysts’ variation in managing
variability
Chan LS1 , Macdonald ME1 , Carnevale FA1 , Steele RJ1 , Shrier I1
1
McGill University, Canada
Background: When meta-analysts arrive at discrepant conclusions
from different meta-analyses, methodological differences are assumed
to be the primary reason for the variation. Our research team
has previously shown that discordant interpretations can exist even
when meta-analysts are presented with identical data. Objective:
To better understand the analytic processes meta-analysts employ
when interpreting meta-analyses. Methods: For this mixed-methods
study, we recruited authors of meta-analyses/systematic reviews using
theoretical sampling and the following variables: gender, professional
training, number of meta-analyses conducted. Fourteen participants
have been recruited to date. After asking participants about their
general meta-analytical approach, they were provided with simulated
data sets (based on published meta-analyses) in relation to treatments
for four different conditions. They were then asked to provide treatment
effect estimates and confidence intervals after 3, 5, 10, and 20 studies
for each of the four conditions. Using grounded theory methodology,
we conducted semi-structured qualitative interviews to render explicit
the different analytical processes employed. Results: Preliminary
findings suggest that meta-analysts engage in the following analytical
processes: 1) scrutinizing the data for ‘poolability’; 2) managing
a ‘part-whole’ tension (i.e., shifting between a focus on individual
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
8
studies and the entire dataset); and 3) determining a treatment
decision. Participants demonstrated variation within these processes
as well as in the micro-processes they used to frame and formulate
interpretations. Finally, we describe moral dimensions that shape
meta-analysts’ analytic processes including: a desire to maintain the
integrity of the whole; refusing to recommend a treatment or assign a
point estimate because it would not be ‘right’; a concern about financial
resources required to further evaluate a treatment; and complying with
external standards. Conclusion: These results demonstrate that
meta-analysts engage in subjective analytical processes, including
important moral considerations, when interpreting meta-analyses.
These processes should be considered to improve the format of
meta-analysis presentation.
A3 Knowledge translation and
communicating evidence
A3O1
Cross-sectional study of abstract quality
Pawlak M1 , Bierling S1 , Dellavalle R1
1 University of Colorado, Denver, USA
Background: Abstracts are often the only lasting evidence of research
presented at conferences. Abstracts provide valuable information that
may not be published outside of conferences. Minimum standard
quality and data should be reported in abstracts so the readers
can assess the research performed. Objectives: We set out to
descriptively quantify the quality of abstracts presented orally at the
19th Joint Colloquium of the Cochrane and Campbell Collaborations.
Methods: A modified version of the CONSORT for Abstracts was
developed to measure the quality of abstracts for all study types. The
checklist consisted of 11 items evaluating title, methods, results, and
conclusion. Two independent reviewers used the checklist to review
126 abstracts presented orally at the 18th Joint Colloquium of the
Cochrane and Campbell Collaborations in 2010. Descriptive analysis
was used to characterize the quality of abstracts. Results: Of the 126
abstracts, 25 were not research in nature, and were excluded from the
analysis. The mean score was 8.1 (range 1 to 11) and the median score
was 8. The items most abstracts did not report were study design in the
title (76%) and participant eligibility (37%). Outcomes were not clearly
defined in 30% of the abstracts. Studies were incomplete in 22% of
the abstracts at the time of submission. Conclusions: We recommend
the use of a standardized methodology for reporting in abstracts to
ensure the quality of the report. The findings of abstracts that were
incomplete should be published in print or website versions as an
addendum. We suggest that conferences provide abstract submission
guidelines that are more concrete and descriptive.
A3O2
Reporting of effect direction and size in abstracts
of systematic reviews
Beller E1 , Glasziou P1 , Hopewell S2 , Altman D2
1 Bond University, Australia; 2 Centre for Statistics in Medicine, UK
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Access to the full text of journal articles is often limited
by cost and difficulty of internet access, whereas abstracts are freely
available. Abstracts need to be clear, and contain sufficient detail
to enable use of the results. The abstract of a systematic review
should clearly state the direction, size and statistical uncertainty of
estimated intervention effects, and should not require expert statistical
knowledge to interpret. Objective: To find out how many abstracts
of systematic reviews present the results clearly and unambiguously.
Methods: Two authors extracted details from abstracts on written
and numeric presentation of results of 182 systematic reviews (64
Cochrane and 118 non-Cochrane) published in 2009. Selected text
was extracted to provide examples of good and poor wording of
results. Results: The direction of the intervention effect was described
in words in 105 of 182 (58%) abstracts, with a further 34 (19%)
deducible from the numeric results, and 43 (24%) not deducible from
the abstract. The size of effect was given in words in only 22 (12%)
abstracts. Cochrane and non-Cochrane reviews had similar quality
of reporting results. 105 abstracts (58%) met a minimum standard
for reporting of results, but only 11 (6%) included both direction
and size of effect in words, size of effect numerically, and confidence
interval. Reviews that had statistically-significant results were reported
better than non-significant ones (76% vs 29% meeting the minimum
standard). Across all abstracts 45 (25%) did not present any numerical
summary of the results. Conclusions: The majority of abstracts of
systematic reviews fail to communicate their message with regards
to the size and direction of the intervention effect and its statistical
precision. Examples of better reporting demonstrate that results can
be expressed clearly and succinctly. Simple improvements to abstracts
of systematic reviews would enhance their value.
A3O3
Investigating and improving the understanding
of Cochrane diagnostic test accuracy reviews
(DTARs)
Zhelev Z1 , Garside R1 , Hyde C1
1 Peninsula College of Medicine and Dentistry, Universities of Exeter and
Plymouth, UK
Background: The understanding and application of test accuracy
information for clinical and policy decision making is difficult, and
further complicated when primary test evaluations are systematically
reviewed and meta-analysed. No research has yet explored health
professionals’ understanding and interpretation of such reviews. At
the 2010 Colloquium we presented the protocol for the current study,
which is now underway. Objectives: We aim to explore how clinicians
and policy makers, who vary in knowledge and experience with test
accuracy and systematic reviews, extract and interpret information
from Cochrane Diagnostic Test Accuracy Reviews (DTARs). We hope
that the findings will help the Cochrane Collaboration to improve
the presentation and accessibility of this type of review, and will
ultimately result in increased use of test accuracy information in
clinical and policy decision making. Methods: An online survey
gathered preliminary information about clinicians’ and policy makers’
experience with test accuracy information and systematic reviews.
Based on this, a purposive sample is being selected to take part in
cognitive interviews, using one of the three existing Cochrane DTARs
as material. The interviews are video-recorded and transcribed, and
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
9
qualitative analysis is carried out to identify patterns in participants’
‘reading’ and interpretation of the reviews with a particular focus
on the difficulties encountered. Study progress and preliminary
findings: Final results will be available by the end of August 2011.
Preliminary analysis of the survey and the number of interviews
completed so far suggests, however, that there are a number of common
difficulties experienced when reading a DTAR including statistical and
clinical terminology, the way in which information is structured and
presented, and navigating the document. This points to specific
ways of improving DTARs’ presentation and accessibility. In addition,
the study demonstrates the value of this innovative way of using
cognitive interviewing and the possibilities of its application in other
research.
A3O4
To what extent is the clinical context considered
in diagnostic test accuracy (DTA) reviews?: A
methodological review
Davenport C1 , Hyde C2
1 University of Birmingham, UK; 2 University of Exeter, UK
Background: Statistical and methodological issues have, until
relatively recently, dominated the test accuracy research landscape.
Consideration of clinical context (the intended setting application and
role of tests, the downstream consequences of test results and the use
of test accuracy measures to convey contextual information) has only
relatively recently received attention. Consideration of test accuracy
in the absence of contextual information may mislead when making
recommendations about test use. Objectives: To investigate the
extent to which a representative sample of diagnostic test accuracy
(DTA) reviews represent the clinical context in which an index test
is to be used when formulating a review question, deciding on
synthesis methods, reporting results and making recommendations,
and on the basis of results to provide recommendations for the
reporting of contextual aspects of DTA reviews. Methods: Published
and unpublished reviews were sought by interrogation of the DARE
database; the Cochrane Database of Systematic Reviews ; the database
of systematic reviews hosted by the Aggressive Research Intelligence
Facility, University of Birmingham; the UK NHS National Research
Register; and contact with the Cochrane DTA working group. A final
random sample of 100 reviews was included from 271 reviews eligible
on the basis of title and abstract. Results: 100 included reviews
represent 17 disease topic areas and include between 1 and 50 index
tests. Preliminary scrutiny of included studies suggests: lack of clarity
of question formulation, particularly with respect to index test role
and current testing practice; a lack of consistency between question
formulation, methods and reporting of results; and unsupported use
of test accuracy outcome measures. Conclusions: The findings of
this review will have implications for the development of standards
for reporting of DTA reviews and the development of the ummary of
results table for Cochrane DTA reviews.
A4 Knowledge translation and
education
A4O1
Epidemiology, quality and reporting
characteristics of systematic reviews of
acupuncture interventions published in Chinese
journals
Ma B1 , Qi G2 , Li X1 , Wang T1 , Chen Z1 , Zhang Z3
1 Evidence-Based Medicine Center, Institute of Traditional Chinese and
Western Medicine, School of Basic Medical Sciences, Lanzhou University,
Lanzhou,China; 2 Evidence-Based Medicine Center, Institute of Traditional
Chinese and Western Medicine, School of Basic Medical Sciences, Lanzhou
University, Lanzhou,China; The First Clinical School of Lanzhou University,
Lanzhou, China; 3 Evidence-Based Medicine Center, Institute of Traditional
Chinese and Western Medicine, School of Basic Medical Sciences, Lanzhou
University, Lanzhou,China; Ministry of Health of the People’s Republic of
China, Beijing, China
Background: Systematic reviews (SRs) of acupuncture have become
increasingly popular in China and have been published in large
numbers. This review provides the first examination of epidemiological
characteristics of these SRs as well as compliance with the PRISMA
and AMSTAR guidelines. Objectives: To examine epidemiological
and reporting characteristics as well as methodological quality of
SRs of acupuncture published in Chinese journals. Methods: Four
Chinese databases were searched (CBM, CSJD, CJFD and Wanfang
Database) for SRs of Traditional Chinese Medicine (TCM), from inception
through December 2010. Data were extracted into Excel spreadsheets.
The PRISMA and AMSTAR checklists were used to assess reporting
characteristics and methodological quality, respectively. Results: A
total of 88 SRs were identified, None of the reviews had been updated.
Less one-third (27.3%) were written by clinicians and one-third (33%)
were reported in specialty journals. The impact factors of 53.4%
of the journals published in were zero. Information retrieval was
not comprehensive in more than half (59.1%) of the reviews. Most
(62.8%) did not reported information about quality assessment, while
less than half (36.4%) reported assessing for publication bias. Statistical
mistakes appeared in one-third (30.3%) of reviews. Though 93.2%
reviews used the term ‘systematic review’ or ‘meta analysis’ in the
title, no reviews reported conflicts of interest and were updated two or
more years after they were published. Conclusions: While many SRs
of acupuncture interventions have been published in Chinese journals,
the quality of these reviews is troubling. As a potential key source of
information for clinicians and researchers, not only were many of these
reviews incomplete, some also contained mistakes or were misleading.
Focusing on improving the quality of SRs of acupuncture, rather than
continuing to publish them in great quantity, is urgently needed in
order to increase the value of these studies.
A4O2
Credence: creation of reliable evidence in
developing countries by continuous enabling
Mathew ME1 , Chandra S2 , Sriganesh V3 , Ejaz K4 , Sharma A5
1
Manipal University, India; 2 National Institute of Mental Health and Neuro
Sciences, India; 3 Quality Medical Knowledge Foundation, India; 4 Aga Khan
University Hospital, Pakistan; 5 Emory University, USA
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
10
Background: The Cochrane Collaboration has an efficient system
in place, such that almost anyone interested, can write a Cochrane
systematic review. The Cochrane Handbook of Systematic Reviews
is so detailed that an author can find the answer to any question
encountered. However undertaking a review is a complex and laborious
task, and requires constant support. In countries like India, the
few existing authors are dispersed geographically and do not have
sufficient networking to form this support system to mentor new
authors. Hence reviews with specific regional importance may be
lacking. Intervention: In India, a group of medical professionals
and students from around India, with support from the Cochrane
South Asian Center and Network, have formed a network of authors.
We call it ‘‘Credence: Creation of Reliable evidence in developing
Countries by continuous Enabling’’; a training and support system
for authors. The contact author is an interested novice student,
so that he/she will get the maximum experience from writing the
review. The main mode of communication is by email and Facebook.
By writing the review, the novice authors learns and graduates
to an expert by an activity-based learning approach. The new
experienced author will serve to mentor more students. Result: In
two months, we formed five teams with four to five members each,
one with a registered title, one in the process of title registration and
remaining three working towards a title. Between the authors we
have six reviews in various stages of publication. Conclusion: We
hope to kindle the mentor-student relationship, by bringing together
experts and novices to a common platform using Internet networking
technologies. This will generate the critical mass of experienced authors
who will be able to write reviews which are more relevant to their
setting.
A4O3
Turning knowledge into action: Translating
Cochrane reviews in the healthcare setting
Koster MA1
1 Kaiser Permanente Southern California, USA
Background: The Cochrane Library provides a rich source of
evidence-based systematic reviews that can be used by healthcare
organizations to improve the quality and delivery of patient care.
However, little is known about the processes by which healthcare
organizations translate Cochrane reviews into action within the
healthcare setting. Objectives: To describe and evaluate evidence
translation strategies used by a large US health maintenance organization to improve care quality for adults with chronic obstructive
pulmonary disease (COPD). Methods: Existing knowledge syntheses
from Cochrane reviews were used to develop and localize content
for a clinical practice guideline on diagnosis and management of
COPD. Interventions aimed at the practitioner, patient and systems
levels were then tailored to facilitate awareness and implementation of
the guideline recommendations. Practitioner interventions included a
combination of passive and active strategies, including basic knowledge
dissemination (distribution of guidelines via direct mail and online),
interactive online continuing medical education, development of
point-of-care job aids, and audit/feedback. Patient-level interventions
included point-of-care education booklets, online resources for COPD
self-management, and proactive outreach for spirometry testing.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Interventions at the systems level included development and provision of
patient outreach lists and computerized decision support, as well as site
visits to implement and evaluate workflows and outreach processes and
obtain feedback from local leaders and champions. Monthly reporting
and review of three measures (spirometry testing for assessment
and diagnosis of COPD, and management of COPD exacerbations
with systematic corticosteroids and bronchodilators) was conducted
to monitor performance. Results: Over a two-year period (2008
to 2010), continued increases in the rates of spirometry testing and
management of COPD exacerbations with systemic corticosteroid and
bronchodilator medications were observed. Conclusions: Evidence
translation strategies incorporating Cochrane systematic reviews have
the capacity to significantly improve the quality and delivery of
patient care.
A4O4
Understanding how meta-analysts interpret
meta-analyses
Shrier I1 , Steele RJ1 , Carnevale F1 , Chan LS1 , Macdonald ME1
1 McGill University, Canada
Background: Understanding how researchers/clinicians interpret
evidence may improve communication of meta-analysis results.
Objective: To explore discrepancies in treatment recommendations
based on identical meta-analyses. Methods: For the quantitative
component of our mixed-methods study, we simulated meta-analysis
data after 3, 5, 10 and 20 studies for 4 different scenarios. Based
on these data (included individual study results, fixed and random
effects point estimates, I2 ), published meta-analysts estimated the
treatment effect at each point, and whether the treatment was
shown beneficial, or would be shown beneficial in the future, and if
they would recommend treatment. For the qualitative component, we
conducted individual semi-structured interviews with the meta-analysts
to substantiate their quantitative responses. Results: These results
are based on the 14 meta-analysts analyzed thus far. In general,
participants did not believe a treatment was beneficial if there were
≤5 studies. An exploratory Classification and Regression Tree analysis
suggested recommending treatment was mostly dependent on the
meta-analyst believing the treatment would be proven beneficial in
the future, which itself was determined by the treatment being
shown beneficial at that time. The total number of events was
the most important factor determining if treatment was considered
beneficial, provided the confidence interval did not cross the null
(86% recommended and 14% unsure). When the confidence interval
included the null, 63% of meta-analysts did not recommend treatment.
When total events were low, those with higher risk-taking behaviour
were more likely to consider (Yes or Unsure) recommending treatment
(71% vs. 16%). Our qualitative analysis corroborated the quantitative
findings. Conclusions: Decisions to recommend treatment based
on meta-analyses appear to be generally related to the number of
total events in the meta-analysis, and whether the confidence interval
crosses the null.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
11
A5 Knowledge translation and
communicating evidence
A5O1
What do we know about interpretation and
application of test accuracy measures?
Davenport C1 , Hyde C2
1 Birmingham University, UK; 2 Exeter University, UK
Background: The widespread belief that decision makers have
difficulty understanding and applying test accuracy information has not
been based on a systematic interrogation of the evidence base to allow
quantification or characterisation of the extent of the problem. Aims
and objectives: To comprehensively ascertain literature pertinent
to the understanding and application of test accuracy measures in
order to identify facilitators and barriers to their use by decision
makers. Methods: Bibliographic searches were conducted in 2003,
2005, 2007 and 2010, across 11 databases representing medicine,
psychology and education. Searches were iterative, purposive and
supplemented by reference checking included studies and contact with
experts. A narrative synthesis of empirical and theoretical test accuracy
and risk communication literature was undertaken. Results: 64 test
accuracy and 21 risk communication papers were included. Research
is characterised by self selected samples, lacks external validity and
primary care is under-represented. Ability to define the most commonly
used metrics (sensitivity, specificity, predictive values) is poor. Predictive
values and test errors are promoted as most intuitive although there
is no empirical evidence supporting the superiority of a single test
accuracy metric for diagnostic decision making. Natural frequency
and multiple presentation formats facilitate understanding. Verbal
descriptions and negative test results may be less well understood.
Self-reported use of measures varies: 80% for predictive values, 4%
for sensitivity and specificity and 1% for ROC curves and likelihood
ratios. Pre-test probability and test accuracy estimation is inaccurate
and highly variable which has implications for probability revision.
Conclusions: The emphasis in the literature has been on identifying
the best single metric rather than identifying an optimal combination
and understanding of meta-analytic summary measures has not been
investigated. Investigation of contextual and motivational influences
on test and test-treat thresholds is required to identify test accuracy
magnitudes that will have most impact on diagnostic and therapeutic
yield.
A5O2
The challenge of assessing quality of evidence
generated from mixed treatment comparisons
and a potential solution
Soares M1 , O’Meara S2 , Cullum N2
1
Centre for Health Economics, University of York, UK; 2 Department of
Health Sciences, University of York, UK
Background: Mixed treatment comparisons (MTCs) allow the
synthesis of direct and indirect effectiveness evidence and can be
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
used, even when available data are limited, to help healthcare
decision-makers interpret all available evidence. However, ensuring
that estimates from these ‘sparse’ MTCs are interpreted correctly
requires the development of appropriate quality of evidence
assessments. We applied a MTC approach to the evidence for wound
dressings for diabetic foot ulcers. In order to produce a ‘Summary
of Findings’ table we attempted to adapted GRADE to allow for the
MTC. Objectives: i) to determine the relative effects of alternative
wound dressings on diabetic foot ulcer healing; ii) to assess how to
reflect the volume and quality of research evidence in a MTC systematic
review. Methods: Systematic literature searches and Bayesian mixed
treatment comparison with sensitivity analyses and assessment of
inconsistencies. The GRADE evidence rating approach was applied
to direct network links and a modified GRADE approach applied to
MTC evidence estimates based on: the limitations of the informing
studies; impact of sensitivity analyses; the volume of direct and indirect
data and data consistency; uncertainty in the resulting estimates and
publication bias related to informing studies. Results: Fifteen RCTs
(nine dressings) were included (Figure 1). Three dressings estimated as
having the highest probabilities of being the ‘best’ for ulcer healing had
the lowest quality direct evidence. Four dressings informed by higher
quality evidence were estimated to have no probability of being the best
dressing. A modified GRADE approach for MTCs helped systematically
assess and communicate the quality of this MTC-derived evidence to
aid interpretation. Conclusions: MTCs are an important evidence
synthesis tool likely to be increasingly incorporated into systematic
reviews and overviews of reviews. Quality assessment methods for
such outputs are required. We suggest that adpated GRADE evidence
quality rating methods should be considered for this purpose.
A5O3
Communicating Cochrane: An update on the
Collaboration’s communications initiative
Tarbett L1
1 Canadian
Cochrane Centre, Canada
Background: The Cochrane Collaboration conducted a Strategic
Review of the organization in 2009 in order to identify areas in
which the Collaboration needed to grow, improve, or change. One
of these areas was communications and marketing. To help meet
this need a Marketing and Communications Working Group (MCWG)
was established to assess the current communications situation and
identify potential strategies. One of the recommendations was to
hire a professional communications and marketing firm to aid in
this important endeavor. Objectives: A Request for Proposal was
circulated internationally to various professional firms and Delta Media
(Ottawa, Canada) was approved for hire by the Steering Group at the
mid-year meetings in 2011 in Split, Croatia. Delta Media will provide
the professional expertise the Collaboration currently lacks internally
in order to further increase the Collaboration’s profile globally and
have the efforts of its 28,000 outstanding contributors recognized.
Methods: Delta Media will use tools provided by the Collaboration
(Scoping Document created by the MCWG)and interview various key
players in order to understand the unique nature of the organization as
well as its unique communications needs and strengths. Delta Media
will draw on its nearly 20 years of experience, proven expertise and
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
12
Figure 1 (A5O2). A network summary of all dressing comparisons informed by direct trial data. Figures presented are the number of studies informing
the link (n=), odds ratios and intervals around each ratio. QOE = Quality of evidence.
current market research in order to advise the Collaboration on where
to invest communications/marketing funds to receive the ‘best bang
for our buck’. The firm will work closely with individual members and
departments, such as the Web Team, to help the Collaboration reach
its full communications potential. Results: Delta Media, will enhance
the communications impact of the Collaboration markedly and thrust
the organization into the mainstream when it comes to evidence-based
research. Initiatives implemented or advised by Delta Media will be
reported to Colloquium participants in order to apprise them of the
progress made in this important initiative as well as future plans.
A5O4
Approaches to estimate and present baseline
risks: Recommendations for Cochrane reviews
‘Summary of findings’ tables
Hayden JA1 , Tomlinson G2 , Ni A3 , Parker R4 , Cartwright J4 ,
van Tulder M5 , Altman D6
1 Dalhousie University, Canada; 2 University Health Network, Canada;
3
Canada; 4 Capital District Health Authority, Canada; 5 EMGO,
Netherlands; 6 Oxford, UK
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: The ‘Summary of findings’ (SoF) table is a key tool
to present the main findings of Cochrane reviews in a transparent
and simple format. The SoF table presents an absolute measure
of treatment effect, using a measure of baseline risk to convert
relative effect sizes. This is desirable as an absolute measure provides
an estimate of how likely an intervention is to work in a specific
population, or in an individual patient. Objectives: We explored the
relevant methodological literature, compared and identified strengths
and weaknesses of different approaches to estimate baseline risk
toward the goals of valid, feasible and understandable presentation
of absolute measures of effect. Methods: Baseline risk estimates
for health populations were extracted from randomized controlled
trials, prognostic systematic reviews, and individual prognostic studies.
The table presents data sources and estimate measures compared.
Simulation modeling was used to assess the potential impact of more
or less extreme estimates. Results: A summary of the methodological
literature, results of exploratory analyses and simulation studies have
been conducted. These analyses have demonstrated that the best
estimate of baseline risk depends on the review question, including
the size of the treatment effect, how well the trial control groups
represent the population of interest, and the heterogeneity in the
baseline risk values. It is appropriate to estimate the baseline risk
from the median RCT control group rate in many situations; in some
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
13
Table 1 (A5O4). Approaches to estimate baseline risk
explored.
Data Sources
– Selected RCT(s) (control group):
largest study; clinically selected
study
– MA of RCTs (control group):
studies from current review;
related reviews
– Selected prognosis study(ies):
general population; defined by
PFs or risk prediction models
– MA of prognosis studies
+/− clinical judgment
Measure(s) of Estimate Presented
– Measure of central tendency
– For Subgroups
– For Risk categories
+/− probability distributions
of estimates
RCT = randomized controlled trial; MA = meta-analysis; PF = prognostic factor.
cases estimation of baseline risk from prognostic evidence is more
appropriate. Conclusions: We have developed several resources for
reviewer authors and Cochrane Review Groups in making decisions
about baseline risk estimates including: i) an algorithm to guide
decision-making about appropriate choice for baseline risk estimate;
ii) recommendations for identifying baseline risk estimates from
prognosis studies; and iii) a tool to explore the impact of changes
in baseline risk estimates on important absolute treatment effect.
A6 Knowledge translation in patient
safety
A6O1
Implementation and evaluation of an
international standardized operating protocol
for the prevention of wrong site surgery
in Germany
Fishman L1 , Renner D1 , Kolbe M2 , Lessing C2 , Thomeczek C1
1 Agency for Quality in Medicine, Germany; 2 Coalition for Patient Safety
c/o Institute for Patient Safety, University of Bonn, Germany
Background: The objective of the WHO initiative ‘Action on Patient
Safety: High 5s’ is to implement and evaluate standardized operating
protocols (SOPs) in a multi-national learning community in order
to improve patient safety in hospitals. High 5s SOPs have been
developed for three topics, namely 1) correct site surgery (CSS), 2) safe
management of concentrated injectable medicines, and 3) medication
reconciliation. Methods: In Germany the SOPs are implemented
successively, beginning with the CSS SOP. In 2009 to 2010, 17 German
hospitals of various levels of care were recruited to participate in the CSS
module. The core element of this SOP is a surgical checklist used both
for SOP implementation and data collection. In hospital workshops the
SOP and checklist, which were developed in the USA, were adapted
nationally and locally to the preoperative processes in the participating
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
German hospitals while maintaining standardisation as according to the
minimum High 5s criteria. The surgical checklists are distributed to the
hospitals as carbon copies. After completion of the process, the copies
are transferred for data analysis. Five internationally standardised
process and outcome indicators are generated from the checklist data
on a monthly basis. These allow a continuous assessment of the
degree of realisation and the effects of the SOP, and are complemented
by qualitative evaluation components addressing barriers and success
factors for implementation. Results and Conclusion: Adaptations
at both a national and hospital level are necessary in order to ensure
acceptance and thus successful implementation of the SOPs. The
comprehensive evaluation concept within the High 5s Project, which is
applied in numerous hospitals in varying healthcare systems, provides
an unprecedented opportunity to analyse the feasibility and impact
of specific standardised patient safety solutions locally, nationally
and internationally. First evaluation results will be presented at the
conference.
A6O2
Warnings for serious adverse events in drug
labelling: Are same-class drugs treated the
same?
Panagiotou OA1 , Contopoulos-Ioannidis DG2 , Papanikolaou PN1 ,
Ntzani EE1 , Ioannidis JP3
1
Department of Hygiene and Epidemiology, University of Ioannina School
of Medicine, Ioannina, Greece; 2 Department of Pediatrics, University of
Ioannina School of Medicine, Ioannina, Greece; 3 Stanford Prevention
Research Center, Stanford University School of Medicine, Stanford, USA
Background: Adverse events carry major morbidity and are a
considerable cause of mortality. Black box warnings (BBWs) are
the strongest medication-related safety warnings in a drug’s labeling
information and are placed on a drug label to highlight major risks.
Absence of a BBW or asynchronous addition of a BBW among
same-class drugs could create confusion in prescription patterns, or
could differentially favor drugs in the same category and should,
therefore, be justified appropriately. Objectives: We aimed to
evaluate systematically the concordance in the BBW labeling between
drugs of the same class. Methods: We tagged the 20 top-selling
drugs for 2008 (10 with BBWs and 10 without BBWs on their label)
that belonged to different drug classes. Labeling information on all
drugs in these 20 classes was captured, and differences were recorded
regarding the presence and timing of acquisition of BBWs for same-class
drugs. Results: We assessed 176 different agents, belonging to the
20 evaluated drug classes. Seven of these agents had been withdrawn
for safety reasons. The reasons for withdrawal became BBWs in other
same-class agents in only 2 of the 7 cases. BBW-related differences
were identified in 9 of the 20 classes, corresponding to 15 BBWs that
were not present in all drugs of the same class. The information for 10
of the 15 different BBWs were included in the labels of same-class drugs
as simple warnings or text, while it was absent entirely in 5 BBWs.
The median interval from the time the BBW had appeared in another
drug of the same class was 66 months. Conclusions: Differences in
BBW labeling in same-class drugs are common and shape perceptions
of safety among similar agents. BBW labeling needs to become more
systematic, and requires clear justification for the differences observed
within drug classes.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
14
A6O3
An investigation into the assessment and
reporting of harms in clinical studies
Smyth R1 , Williamson P1
1 University of Liverpool, UK
Background: The recent ORBIT (Outcome Reporting Bias in Trials)
study has shown that outcome reporting bias is an under-recognised
problem that affects the conclusions of efficacy outcomes in a
substantial proportion of Cochrane reviews [1]. Selective reporting
of harms is expected, however there is a lack of research regarding
whether standards for assessment and reporting of harms are being
followed. Objectives: To examine the assessment and reporting
of harms in clinical studies, and to examine the impact of selective
reporting of harms in clinical trials on systematic reviews. Methods:
We refer to three studies that demonstrate the problem and potential
impact of poor reporting of harms: the ADRiC (Adverse Drug Reactions
in Children) systematic review; further results from the ORBIT study;
and a case study meta-analysis demonstrating the effect on the
benefit-harm ratio. Results: Results from the ADRiC systematic review
showed that the reporting of ADRs, in studies where detection of
ADRs was a primary aim, was poor, with only 3/101 studies including
data on causality, predictability and avoidability. The findings from
the interviews from the ORBIT study provides an explanation of why
selective reporting occurs in relation to harms. Poor reporting of
gastro-intestinal bleeds in trials of aspirin is shown to affect the
benefit-harm ratio. Conclusions: This presentation will provide a
forum for discussion of the procedures for assessing and reporting
harms in clinical trials. This will improve the ability of decision makers
to make informed decisions that consider both benefits and harms of
an intervention, ultimately improving patient safety.
Reference
1. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R,
Williamson PR. The impact of outcome reporting bias in randomised
controlled trials on a cohort of systematic reviews. BMJ (2010); 340:
c365. doi: 10.1136/bmj.c365.
A6O4
What is a rapid review?
Harker J1 , Kleijnen J1,2
1 Kleijnen Systematic Reviews Ltd, UK; 2 CAPHRI, University of Maastricht,
Netherlands
Background: Commissioners of Health Technology Assessments
(HTAs) require timely and cost-effective reviews to attain efficacious
decisions on health care and treatments. In recent years, there has
been an emergence of ‘Rapid Reviews’ within HTAs; however, there are
no known published guidelines or agreed methodology. Objectives:
In order to answer the research question: ‘What is a Rapid Review,
and is methodology consistent in HTA Rapid Reviews?’ a study was
undertaken in a sample of Rapid Review (RR) HTAs from the Cochrane
Database to investigate equalities and/or differences in RR methodology
used. Methods: The Cochrane Database of Systematic Reviews was
searched, to locate RR HTAs from 2000 onwards. Each RR was
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
examined to investigate the individual methodology used for searching,
inclusion screening, quality assessment, data extraction and synthesis.
Methods of each RR were compared to investigate whether differences
and similarities in methodologies used, in comparison with recognised
methods for Systematic Reviews (SRs). Results: 46 Rapid Reviews were
included. There was a wide diversity of methodology, with some reviews
utilising well-established SR methods, but many others diversifying in
one or more areas, i.e. searching, clarity of research questions,
inclusion screening, quality assessment, data extraction, synthesis
methods, report structure and number of reviewers. Conclusions:
Despite the number of Rapid Reviews published within HTAs over recent
years, there is no agreed and tested methodology and it is unclear how
RRs differ from SRs. In a sample of HTA RRs from 2000 to 2011, there
was a wide diversity of methodology utilised in all aspects of these
reviews. There is scope for wider research in this area to investigate
diversity of methods in more depth so that eventually, recommendations
could be made for clear and systematic methods for Rapid Reviews;
thus facilitating equity and credibility of this type of important review
methodology.
B1 Diagnostic test accuracy review
methods
B1O1
QUADAS-2: an updated quality assessment tool
for diagnostic accuracy studies
Whiting P1 , Rutjes A2 , Westwood M3 , Mallett S4 , Deeks J5 , Reitsma J6 ,
Leeflang M7 , Sterne J1 , Bossuyt P7
1 School of Social and Community Medicine, University of Bristol, England;
2 Division of Clinical Epidemiology and Biostatistics, Institute of Social and
Preventive Medicine, University of Bern, Switzerland; 3 Kleijnen Systematic
Reviews, York, England; 4 Centre for Statistics in Medicine and Department
of Primary Health Care, University of Oxford, England; 5 Unit of Public
Health, Epidemiology & Biostatistics, University of Birmingham, England;
6
Julius Center for Health Sciences and Primary Care, UMC, University of
Utrecht, Netherlands; 7 Department of Clinical Epidemiology, Biostatistics
and Bioinformatics, AMC, University of Amsterdam, Netherlands
Background: QUADAS was developed in 2003 to assess the quality
of diagnostic test accuracy (DTA) studies included in systematic
reviews. It includes 14 items covering risk of bias, variability and
reporting. Personal experience, anecdotal reports, and feedback via
the Cochrane Collaboration suggested that some elements required
improvement. Objectives: To update the QUADAS tool to produce
QUADAS-2. Methods: We used a four-phase approach to develop
QUADAS-2 (Figure 1): defining the scope, developing the evidence
base, face-to-face consensus meeting, and refinement through piloting.
We established a broad team including methodological experts and
review authors working on DTA reviews. Results: We defined
quality in terms of risk of bias and concerns regarding applicability.
Based on the evidence we identified items for removal, items to be
re-worked, and items to be considered for inclusion. Discussions at
the face-to-face meeting informed decisions regarding the structure
and content of QUADAS-2. After four rounds of piloting, agreement
was reached on the final tool. QUADAS-2 consists of four domains
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
15
Figure 1 (B1O1). Development process for QUADAS-2.
covering: patient selection; index test; reference standard; and flow
of participants through the study and timing of tests. All domains are
assessed in terms of the risk of bias. Patient selection, index test, and
reference standard are also assessed in terms of concerns regarding
applicability to the review question. QUADAS-2 is applied in four
phases: (1) summarise the review question; (2) tailor the tool to the
review including production of review specific guidance; (3) construct
a flow diagram for the primary study; and (4) assess risk of bias and
concerns regarding applicability. Conclusions: The development of
QUADAS-2 was evidence based and included detailed piloting and
refinement. We feel that QUADAS-2 is a considerable improvement on
the original QUADAS tool and will allow more objective rating of the
risk of bias and applicability of primary DTA studies.
B1O2
Updating QUADAS: User experience informs the
development of QUADAS-2
Westwood M1 , Whiting P2 , Rutjes A3 , Mallet S4 , Deeks J5 , Sterne J2 ,
Leeflang M6 , Reitsma J7 , Bossuyt P6
1 Kleijnen Systematic Reviews Ltd, UK; 2 University of Bristol, UK; 3 University
of Bern, Switzerland; 4 University of Oxford, UK; 5 University of Birmingham,
UK; 6 University of Amsterdam, Netherlands; 7 University Medical Centre
Utrecht, Netherlands
Background: QUADAS is a widely used, evidence-based quality
assessment tool, developed for use in systematic reviews of diagnostic
test accuracy (DTA) studies. The experience of the QUADAS steering
group, anecdotal reports, and feedback via Cochrane Collaboration
members suggested some problems with QUADAS. We therefore
decided to revise QUADAS using experience of its application, and
new evidence regarding sources of bias and variation to produce
QUADAS-2. This abstract describes evidence derived from user
experience. Objectives: To gather structured feedback from review
authors who have used QUADAS, and to summarise the findings
of studies that have evaluated QUADAS. Methods: We sought
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
information on users’ experiences of QUADAS using a web-based
questionnaire to elicit feedback from authors of Cochrane DTA
reviews and protocols, and authors of reviews on DARE which
had used QUADAS. We conducted a systematic review of studies
that had evaluated QUADAS. Results: 64 authors completed the
questionnaire. The majority (70% to 89%) rated QUADAS as good
or very good for coverage, ease of use, clarity and validity. Negative
features highlighted were the mixing of questions relating to bias and
applicability, and inability to use QUADAS to assess studies comparing
multiple index tests. Approximately 15% of review authors reported
omitting items on reporting quality from their QUADAS assessments.
Despite guidance, 20% of review authors reported using QUADAS
to calculate summary scores. Four studies evaluated QUADAS; key
recommendations were that QUADAS-2 should be expanded to cover
studies comparing multiple index tests and topics where follow-up is the
reference standard. Conclusions: The experience of review authors
using QUADAS informed development of QUADAS-2 and was largely
consistent with the experience and conceptual decisions of the steering
group. The experience of QUADAS users will continue to be important in
informing future developments and ensuring continued relevance and
wide up-take.
B1O3
Forecasting yesterday’s weather: The risk of
spectrum bias from the inclusion of already
diagnosed/treated patients in studies of
depression screening tools
Thombs B1 , Arthurs E2 , El-Baalbaki G1 , Meijer A3 , Ziegelstein R4 ,
Steele R1
1 Jewish General Hospital and McGill University, Canada; 2 Jewish General
Hospital, Canada; 3 University of Groningen, Netherlands; 4 Johns Hopkins
University School of Medicine, USA
Objective: Screening involves the use of a test or procedure to
detect the presence of a disease in individuals not already seeking
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
16
treatment for symptoms and not already diagnosed with the condition.
Studies evaluating the accuracy of screening instruments that include
already diagnosed or treated patients are known to produce inflated
estimates of screening sensitivity and case yield. The objectives
of this study were to investigate: (1) the proportion of original
studies included in systematic reviews and meta-analyses on the
diagnostic accuracy of depression screening tools that appropriately
exclude already diagnosed or treated patients; and (2) whether
systematic reviews and meta-analyses of the accuracy of depression
screening tools evaluate possible bias due to the inclusion of already
diagnosed or treated patients. Design: Systematic review. Data
sources: MEDLINE, PsycINFO, CINAHL, EMBASE, ISI, SCOPUS, and
Cochrane databases were searched January 1, 2005 to October 29,
2009. Eligibility criteria for selecting studies: Systematic reviews and
meta-analyses in any language that reported on the diagnostic accuracy
of depression screening tools. Results: Only 8 of 197 (4.1%) unique
publications from 17 systematic reviews and meta-analyses specifically
excluded already diagnosed or treated patients. No systematic reviews
or meta-analyses commented on possible bias from the inclusion
of already diagnosed or treated patients, even though 10 reviews
used quality assessment tools with items to rate risk of bias from
patient sample composition. Conclusions: Studies of the accuracy of
depression screening tools rarely exclude already diagnosed or treated
patients, a potential bias that is not evaluated in systematic reviews
and meta-analyses. This may result in inflated accuracy estimates on
which clinical practice and preventive care guidelines are often based,
a problem that takes on greater importance as the rate of diagnosed
and treated depression in the population increases.
B1O4
A novel approach to evaluate the diagnostic
accuracy of a sequence of tests
Novielli N1 , Cooper NJ2 , Abrams KR2 , Sutton AJ2
1
University of Birmingham, UK; 2 University of Leicester, UK
Background: One diagnostic test is rarely sufficient to complete a
diagnosis. Despite this, most diagnostic accuracy studies focus on
the evaluation of an individual test, and therefore approaches for the
meta-analysis of this diagnostic literature have also focused exclusively
on estimating the accuracy of individual tests. Where the accuracy of
sequences of tests has been modelled previously, the often dubious
assumption of test independence has been made (1). Hence, there
is a need to develop meta-analytic approaches to correctly estimate
the accuracy of test sequence strategies. Objective: To develop a
modelling approach to simultaneously synthesise studies of individual
and multiple diagnostic test data in order to estimate the accuracy of
test sequence strategies. Methods: The methodological framework
developed is broadly applicable to contexts where multiple tests are of
interest. Here we describe the approach using the motivating example
of the Ddimer and Wells tests for diagnosing deep vein thrombosis. This
random-effects modelling framework allows: 1) the inclusion of studies
evaluating either test singularly and in combination (complete or partial
reported data) and thus incorporates all available evidence; 2) for the
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
dependency between diagnostic tests; and 3) the incorporation of
tests with multiple thresholds (Wells score). Results: When results of
this synthesis are compared to simpler, but invalid, modelling which
assumes the performance of both tests is independent, considerably
different estimates for test strategies are observed. Conclusions:
Accurate estimation of the accuracy of test sequences is critical for
evidence-based decision making. For the first time a meta-analytic
approach to do this has been developed.
Reference
1. Novielli N, et al. How is evidence on test performance synthesised
in economic decision models of diagnostic tests? Value in Health 2010;
13(8): 952–957.
B2 Searching and information retrieval
B2O1
Study selection by means of a web-based Trial
Selection DataBase (webTSDB)
Hausner E1 , Ebrahim S1 , Herrmann-Frank A1 , Janzen T1 , Kerekes MF1 ,
Pischedda M1 , Waffenschmidt S1 , Kaiser T1
1 Institute for Quality and Efficiency in Health Care (IQWiG), Germany
The German Institute for Quality and Efficiency in Health Care (IQWiG)
is an independent scientific institute that investigates the benefit and
harm of medical interventions for patients. For this purpose IQWiG
prepares systematic reviews on the basis of a systematic literature
search. To minimize errors, the process of study selection is performed
by 2 reviewers independently of each other. In order to achieve
a comprehensible and efficient selection of studies, processes are
continuously optimized by the Institute’s Information Management
Section in cooperation with other departments. In addition to the
establishment of a 3-step screening process aimed at improving external
transparency, an MS ACCESS-based database for study selection was
programmed and implemented, also prompted by the fact that suitable
solutions were not available on the market. In 2009, due to positive
experiences with this database, a web-based application was developed
with an external service provider (web-based Trial Selection DataBase,
webTSDB). The advantages of webTSDB are, in particular, the efficiency
and transparency of the procedure, as well as the possibility of
simple documentation of the whole process. General functions and
possibilities of webTSDB are as follows: • simultaneous work on
the project by reviewers, independent of location, • project-specific
deposit of inclusion and exclusion criteria, • consensus function
for citations with inconsistent classification, • deposit of comments,
• preparation of reports and flow charts, • linking to full texts,
as well as • automatic data-storage. WebTSDB has been used
throughout the Institute since June 2010. The aim of this contribution
is to present the advantages, functions and possibilities of this
software.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
17
Figure 1 (B2O1).
Figure 2 (B2O1).
B2O2
Assessing the completeness of systematic
reviews via the ‘related articles’ function and/or
a simple structured Boolean search in
PubMed – a pilot study
Hausner E1 , Janzen T1 , Kaiser T1
1 Institute for Quality and Efficiency in Health Care (IQWiG), Germany
Background: From July 2011 the Institute for Quality and Efficiency
in Health Care (IQWiG) will assess the additional benefit of
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
newly-approved drugs in Germany on the basis of dossiers provided by
manufacturers. This early benefit assessment includes an evaluation
of the completeness of the evidence base provided. Objectives: To
assess whether the ‘related articles’ function and a simple structured
Boolean search (SSBS) in PubMed, alone or in combination, are
efficient search tools to assess the completeness of an evidence
base. Methods: Eligible citations were those included as relevant
in 6 randomly-selected systematic reviews (SRs) on drug interventions
published by IQWiG. Firstly, these citations were searched for in
PubMed and the corresponding ‘related articles’ function was applied;
3 sets of articles (the first 20, 50, and all related articles) were
exported. In a second step, an SSBS was performed for each SR and
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
18
the articles retrieved were exported. In a third step, the number of
citations screened and number of relevant citations identified were
determined for each search tool. In addition, sensitivity and precision
were calculated to assess efficiency. Results: A total of 121 relevant
citations were identified (4 to 55 per SR). The best sensitivity was
achieved by the ‘all related articles’ function, but precision was very
poor (100%; 0.44%). The combination of the ‘first 20 related
articles’ function and an SSBS achieved the best optimization between
sensitivity and precision (97% vs. 5%). The highest precision was
achieved by the ‘first 20 related articles’ function (13%); however, this
was accompanied by a decrease in sensitivity (80%). Conclusions:
Our findings indicate that the combination of the ‘first 20 related
articles’ function and an SSBS in PubMed is an efficient method to
assess the completeness of an evidence base. Our approach will be
applied and evaluated using the dossiers submitted.
B2O3
Primary outcomes reported in abstracts
and ClinicalTrials.gov – do they agree?
Huynh L1 , Scherer R2 , Ervin A2 , Dickersin K2
1 Johns Hopkins Bloomberg School of Public Health, USA; 2 Johns Hopkins
Bloomberg School of Public Health, US Cochrane Center, USA
Background: The 2011 Institute of Medicine (IOM) standards for
systematic reviews recommend handsearching conference abstracts to
identify otherwise unpublished research results and address potential
reporting biases. We hypothesized that the sparse description
of randomized controlled trial (RCT) primary outcomes presented
in conference abstracts could be supplemented using trial registry
information. Objectives: To compare primary outcomes described
in conference abstracts with those reported in ClinicalTrials.gov and
assess the relationship between primary outcome reporting and industry
sponsorship. Methods: Eligible conference abstracts described RCT
results, had a valid ClinicalTrials.gov registration number, and were
presented at the 2007 to 2009 Association for Research in Vision
and Ophthalmology meetings. Two reviewers independently extracted
information from the abstract and ClinicalTrials.gov. Discrepancies
were resolved by consensus. Results: We identified 152 eligible
abstracts; 40 explicitly reported at least one primary outcome in
the abstract and primary outcome field at ClinicalTrials.gov. Of 80
primary outcomes reported in abstracts, 18% (14/80) agreed with
one reported in ClinicalTrials.gov. Forty-nine percent (39/80) agreed
partially, in that additional information (eg. related to time point)
was present in ClinicalTrials.gov (17/39; 44%), the abstract (19/39;
49%), or both (3/39; 8%). Thirty-four percent (27/80) of primary
outcomes reported in the abstract were not reported as such in
ClinicalTrials.gov, although 6 were reported as secondary outcomes.
Industry sponsorship was reported in 6/14 abstracts where a primary
outcome was reported in the abstract but not in ClinicalTrials.gov,
and in 14/26 abstracts where all primary outcomes reported in the
abstract were also reported in ClinicalTrials.gov. Conclusions: Our
data suggest that investigators may not be updating ClinicalTrials.gov
trial information. Both conference abstracts and ClinicalTrials.gov
provide unique primary outcome information not available in the other
source. Systematic reviewers should obtain outcome information from
both conference abstracts and registries. Study protocols with primary
outcome information should be available from trial registries.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
B2O4
PRISMA in practice: keeping track of the
literature search documentation. What are
the problems?
Mann M1 , Sampson M2 , Cooper C3 , Stanfield C4 , Rader T5
1
Support Unit for Research Evidence, Cardiff University, UK; 2 Children’s
Hospital of Eastern Ontario, Ottawa, Canada; 3 Peninsula Technology
Assessment Group (PenTAG), Peninsula College of Medicine & Dentistry,
University of Exeter, UK; 4 EPPI-Centre, Social Science Research Unit,
Institute of Education, University of London, UK; 5 Cochrane Cochrane
Musculoskeletal Group, University of Ottawa, Canada
Background: The PRISMA Statement (Preferred Reporting Items for
Systematic Reviews and Meta-Analyses) calls for detailed reporting of
elements of the search process, and has been adopted by The Cochrane
Collaboration with the intent of advancing the quality and transparency
of Cochrane reviews. At a recent colloquium, Information Retrieval
Methods Group participants expressed concern at the feasibility of
implementing these reporting requirements. Objectives: This study
examines adoption of PRISMA search reporting, and challenges related
to PRISMA reporting standards in the context of new Cochrane reviews.
Methods: We examined 19 reviews flagged as New’ in The Cochrane
Library (Issue 4, 2011) for reporting aspects of PRISMA elements 7,
8 and 17; total retrieval size, number of unique records, the number
of unique studies identified, as well as a flow diagram and transcript
of the search. Separately, a survey of research communities, including
Trials Search Coordinators (TSCs), Information Specialists, and authors,
explored practice and perceived challenges in search documentation.
Results: In our search results all the new reviews incorporated
a transcript of the search strategies. Five reviews (26%) included
a flow diagram, and these varied in the level of detail provided.
Exclusion reasons were rarely provided for records found ineligible
based on title and abstract. Survey responses provided insights into
the difficulties faced by those responsible for search documentation.
These included: lack of time, lack of tools, lack of clear understanding
of the requirements, and a lack of clear assignment of responsibility
for recording and reporting these elements. In our presentation we will
fully discuss the findings. Conclusions: The Cochrane Collaboration
adopted PRISMA in 2009; however, support for PRISMA flow diagrams
was only available in RevMan version 5.1.1 and reporting compliance
remains incomplete. Training and support for TSCs and search authors
will help achieve full implementation of this standard.
B3 Searching and information retrieval
B3O1
Screen2Go: a pilot smartphone app for citation
screening
Huckvale K1 , van Velthoven M1 , Car J1
1 Imperial College London, UK
Background: Although Cochrane’s RevMan software supports
authors during the latter stages of a review, citation searching and
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
19
Figure 1 (B3O1).
computer. Applied to a real-world screening of some 12,000 articles,
there was no obvious deficit in the precision of screening decisions
made using this workflow. Conclusions: A smartphone app may be an
effective way to support the citation screening process by allowing users
to review smaller batches of citations at convenient times. A future
randomised crossover study will formally compare the performance of
screeners using the tool against usual practice.
B3O2
Could a search for a diagnostic test accuracy
review be restricted to MEDLINE?
van Enst WA1 , Scholten RJPM1 , Hooft L1
1 Dutch Cochrane Centre, Netherlands
abstract screening are not part of the current or planned feature set.
Screening up to several thousand citations can be a daunting task for
a researcher with multiple demands on their time. Objectives: To
develop a pilot smartphone and tablet app to explore the feasibility
and acceptability of a convenience approach to citation screening. This
would allow a user to screen batches of citations during otherwise
unproductive periods of the day, for example when commuting to
work. Methods: We used a rapid application development approach
to build a prototype using the Microsoft Visual Studio SilverLight
software development environment for Windows 7 Phone. Results:
The prototype app was developed to support a simple workflow.
Citations exported from bibliographic software, such as Endnote, can
be uploaded to a secure website and are automatically downloaded
to the phone for screening ‘offline’. When a mobile data connection
becomes available, screening choices are sent back to the website
where the user can then view and export batches of citations grouped
by decision. The app proved highly acceptable to a small group of
test users. It appeared most effective for rapidly excluding citations
of obvious irrelevance. Citations with missing information had to be
marked as ‘unsure’ and triaged in a separate step using a desktop
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Often, a search for a review of diagnostic test accuracy
studies (DTA review) results in a high number of references to screen.
However, if the search could be limited to only the freely accessible
database MEDLINE, the number of references needed to screen may be
reduced and costs may be spared. Objectives: To assess the number
of studies that will be missed when a search for diagnostic accuracy
studies is restricted to MEDLINE and whether this will introduce bias in
a meta-analysis. Methods: We collected diagnostic reviews published
in high impact journals between 2006 and February 2011, that included
a search in MEDLINE and in at least one other biomedical database,
stated the search date, and did not apply language restrictions. For
each meta-analysis, we identified the number of studies that could
not be found in MEDLINE. When the meta-analysis included ≥10
studies, we repeated the meta-analysis, but including only those
studies that were indentified in MEDLINE. We calculated the ratio of
the diagnostic odds ratios (RDOR) based on the MEDLINE only studies
versus all studies. Results: We identified 615 reviews of which 44
reviews with 76 meta-analyses were included. In 65 meta-analyses
(85.5%) the studies were all identified in MEDLINE. In the remaining
11 meta-analyses, 87.5 to 99.0% of the studies were identified in
MEDLINE. Of those, 10 meta-analyses included ≥10 studies. However,
due to the very low proportion of non-MEDLINE studies we did not find
repeating meta-analysis helpful. Therefore no RDOR was calculated.
Conclusions: Surprisingly, the extra yield of a search for primary
diagnostic accuracy studies in other databases than MEDLINE was very
low. Based on our relatively small sample, searching in databases other
than MEDLINE hardly seems to affect the outcome of a DTA review.
B3O3
Endorsement of the ICMJE’s clinical trial
registration policy by biomedical journals:
A survey of instructions for authors
Hooft L1 , Molenaar N2 , van der Haar S2 , Scholten R3
1 Dutch Cochrane Centre/Netherlands Trial Register, Netherlands; 2 Netherlands
Trial Register, Netherlands; 3 Dutch Cochrane Centre, Netherlands
Background: Publication and selective outcome reporting bias are
recognized concerns in the medical literature. Prospective registration
of trials may provide transparency in clinical trial conduct and reporting.
Journals play an important role in encouraging investigators to
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
20
B3O4
A systematic review of studies that develop
or evaluate search filters for the retrieval
of diagnostic studies in MEDLINE
adopt trial registration. In September 2005, the members of the
International Committee of Medical Journal Editors (ICMJE) issued a
joined registration policy that included that submitted manuscripts
would only be considered for publication if they had been registered in
a public register before the enrolment of the first patient. Objectives:
To determine to what extent biomedical journals have adopted the
ICMJE’s clinical trial registration policy and whether reported data are
cross-checked with the prospectively registered data. Methods: Two
researchers searched for items addressing prospective trial registration
in the Instructions for Authors on the websites of journals following the
ICMJE’s Uniform Requirements for Manuscripts (URM) and journals that
endorsed CONSORT. Only Instructions for Authors available in English
were analyzed. Results: We identified 866 biomedical journals.
Of those 143 included non-English instructions, were discontinued
journals, or journals without a website. Prospective trial registration
was required by 319 of the remaining 723 journals (44%) and 145 of
these (45%) referred to the WHO International Clinical Trials Registry
Platform (ICTRP). No detailed information was given on whether or
how submitted manuscripts will be compared with trial registry entries
as a component of peer review. Results of an additional survey of
journal editors about this topic will be available at the Colloquium.
Conclusions: Although the number of biomedical journals requiring
registration is encouraging, there is still room for improvement. The
registration policy suggests that submitted manuscripts are compared
with trial registry entries as a component of peer review; clear
instructions to authors were mostly not given. Reviewers’ measure
of registration quality should be more than verifying the registration
number at manuscript submission.
Beynon R1 , Leeflang M2 , Eisinga A3 , McDonald S4 , Mitchell R4 ,
Whiting P1 , Glanville J5
1
School of Social and Community Medicine, University of Bristol, UK;
2 Department of Clinical Epidemiology and Biostatistics, University of
Amsterdam, Netherlands; 3 UK Cochrane Centre, National Institute for
Health Research, UK; 4 Australasian Cochrane Centre, Monash Institute of
Health Services Research, Australia; 5 York Health Economics Consortium
Ltd, University of York, UK
Background: Search filters are developed to improve the efficiency
and effectiveness of searching and are created by identifying and
combining search terms to retrieve records with a common feature
(e.g. diagnostic accuracy). Objectives: To determine whether any
existing methodological search filters designed to retrieve diagnostic
test accuracy (DTA) studies perform well enough to be recommended for
use in Cochrane DTA reviews. Methods: We conducted a systematic
review of studies that developed and/or evaluated methodological
filters designed to retrieve DTA studies in MEDLINE. Eligible studies
had to report a measure of filter performance such as sensitivity
or precision. Studies were identified through searching electronic
databases and reference screening. Two review authors independently
screened search results. Inclusion assessment, data extraction and
methodological assessment were conducted by one author and checked
by a second. Methodological assessment was undertaken using a
Figure 1 (B304). Study selection process.
1683 hits
Excluded
64 included on title/abstract
35 did not develop or evaluate a
diagnostic filter
5 did not report on diagnostic
filters in general
15 studies reported in 16 papers
5 did not report performance
data
3 reported EMBASE filters
7 MEDLINE
development
studies
2 MEDLINE
development and
evaluation
studies
6 MEDLINE
evaluation
studies
56 Filters reported
27 filters
development
data only
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
12 filters
development
and evaluation
data
17 filters
evaluation
data only
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
21
modified checklist. Data to calculate sensitivity, specificity, precision
and the number of abstracts needed to read (NNR) to identify one
relevant study were extracted for each filter evaluation and were
summarized in a forest plot. Results were analysed separately for
studies that developed the original filters (development studies) and
studies that evaluated existing filters (evaluation studies). Results:
Fifteen studies (16 papers) reporting on 56 filters were included:
9 development studies, 6 evaluation studies, and 2 studies that both
developed and evaluated filters (Figure 1). None of the filters had
consistently high sensitivity and low NNR in the evaluation studies.
Estimates of sensitivity ranged from 29% to 100% and NNRs, where
available, ranged from 9 to 100. Estimates of sensitivity in the
development studies were higher than those reported in the evaluation
studies in 28 of 33 comparisons, and estimates of NNR were lower in
5 out of 10 comparisons. Conclusions: None of the filters evaluated
showed sufficient sensitivity and precision to be recommended for
identifying DTA studies in MEDLINE.
B4 Methods for preparing reviews
(non-statistical)
B4O1
The proposed Cochrane agenda setting and
priority setting methods group: establishing
a research program
Nasser M1 , Welch V2 , Ueffing E2 , Tugwell P2 , Crowe S3 , Lieberati A4
1 Peninsula Dental School, University of Plymouth, Plymouth, UK; 2 Campbell
and Cochrane Equity Methods Group, Institute of Population Health,
University of Ottawa, Canada; 3 James Lind Alliance, Oxford, UK; 4 Italian
Cochrane Centre, Italy
Background: There is a continuing interest in defining priority
topics for systematic reviews inside and outside of the Cochrane
Collaboration. However, there is no consensus on the best methods
for priority setting. We are establishing a new Methods Group in the
Cochrane Collaboration to address this gap. Objectives: To identify
methodological challenges that the Cochrane Collaboration is facing
that can be addressed by the future Methods Group. Methods: We
organized five events (workshops and meetings) at previous Cochrane
meetings: 1. A workshop at the Cochrane Colloquium in Freiburg,
2008. 2. A workshop at the Cochrane Colloquium in Singapore,
2009. 3. An exploratory meeting in Singapore, 2009. 4. A workshop
at the Cochrane Colloquium in Keystone, 2010. 5. A workshop at
the UK Cochrane Contributors Meeting, Belfast, 2011. During these
sessions, we provided the participants with the opportunity to discuss
challenges they are facing to implement a priority-setting approach
in their Cochrane entity. Results: We identified several challenges
that the participants faced in planning, conducting, and implementing
a priority-setting process in their Cochrane entity. We categorized
them into methods and process challenges and implementation
issues. The most important methods challenges included: • dealing
with inequalities in developing and conducting priority setting. •
uncertainties around whom to consult in the prioritisation process and
challenges in involving multiple stakeholders in the process • identifying
appropriate criteria for conducting a priority-setting strategy. •
difficulties in evaluating, implementing, and disseminating prioritisation
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
processes. • methods to prioritise updating of Cochrane reviews. •
how to maintain reviewers enthusiasm and making Cochrane reviews
relevant and up to date Conclusions: There are several methods
questions on how to prioritize topics for Cochrane reviews that the
proposed Methods Group could address. We are currently conducting
a number of methods studies to address these challenges.
B4O2
Evaluating priority-setting approaches: tools for
a Cochrane Methodology Review
Nasser M1 , Welch V2 , Ueffing E2 , Tugwell P2 , Chalkidou K3 , Sawicki PT4 ,
Crowe S5
1
Peninsula Dental School, University of plymouth, Plymouth, UK; 2 Campbell
and Cochrane Equity Methods Group, Institute of Population Health,
University of Ottawa, Canada; 3 National Institute for Health and Clinical
Excellence, London, UK; 4 Institute of Health Economics and Clinical
Epidemiology, University of Cologne, Cologne, Germany; 5 James Lind
Alliance Secretariat, James Lind Initiative, Oxford, UK
Background: There is a continuing interest in defining priority topics
and setting a research agenda for systematic reviews inside and outside
of the Cochrane Collaboration. However, there is no consensus on
the best methods of priority-setting. Therefore, we are conducting
a Cochrane methodology review to address this gap. There is no
agreement on the tools to critically appraise the quality (transparency,
effectiveness, rigor, etc.) of conducting and reporting of the priority
setting methods. Objectives: We aimed to develop critical appraisal
checklists to evaluate the quality of the priority setting methods in the
Cochrane methodology review. Methods: We did a search for studies
on the criteria for good or successful priority-setting methods and
studies on how equity can be incorporated in priority-setting methods.
The results of the literature search were discussed in two workshops
at previous Colloquia (Singapore and Keystone). The results of the
discussions along with the consultation of experts in the field were
used to develop two checklists for the Cochrane methodology review.
Results: We developed two checklists, one of the checklists with eight
questions to evaluate the transparency and rigour of the process of
conducting and reporting of the priority setting methods. We also
developed an equity lens with 17 questions across the different steps
of a prioritization process. Conclusions: The checklists could be a
potentially helpful tool to evaluate primary studies for our Cochrane
methodology review and might even help primary researchers in
planning and developing priority-setting methods. We will be working
further to incorporate other aspects of critically appraising priority
setting methods, especially differentiating priority setting methods
with better outcomes like efficiency/value of money, and impact. The
project also provides guidance for authors of methodology reviews on
methods in situations where there is no agreed critical appraisal tool.
B4O3
Modifications to the Risk of Bias tool: a case
study of 204 trials
Hartling L1 , Bialy L1 , Armijo-Olivo S1 , Ha C1 , Lacaze-Masmonteil T1 ,
Vandermeer B1 , Dryden DM1
1 Canada
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
22
Background: In 2008 The Cochrane Collaboration introduced the
Risk of Bias (RoB) tool to assess internal validity of randomized
controlled trials (RCTs). Modifications to the tool were released in
2011 after user testing and feedback. Objectives: To describe the
results of applying the modified RoB tool. Methods: Two researchers
independently applied the tool to 204 RCTs. Disagreements were
resolved through consensus. We included several modifications to the
original tool. First, we assessed blinding separately for participants,
investigators, and outcome assessors. Second, we assessed potential
influence of study sponsorship separately from ‘other sources of bias’.
We assessed agreement across blinding domains using kappa. We
assessed correlation between different blinding domains, and between
funding and ‘other’ domains using Kendall’s Tau. Results: Risk of
bias assessments varied among the three blinding domains. Of note
was the higher frequency of low risk of bias for blinding of outcome
assessors (51%) compared with blinding of participants (24%) and
investigators (31%). The agreement for the blinding domains was
fair (κ = 0.32). The pair-wise correlation for the blinding domains
was moderate (τ = 0.39, participant versus investigator; τ = 0.38,
participant versus outcome assessor; τ = 0.45, investigator versus
outcome assessor). For other sources of bias, risk of bias was low in
118 (58%), high in 33 (16%), and unclear in 53 (26%). For sources
of funding risk of bias was low in 77 (38%), high in 14 (7%), and
unclear in 113 (55%). The correlation between ‘other sources of bias’
and ‘source of funding’ was weak (τ = 0.04). Researchers applying
the RoB tool found it easier to assess blinding as three separate
domains rather than a single item. Conclusions: This study provides
evidence that risk of bias due to blinding varies depending on the
targeted individuals. Risk of bias due to ‘other’ sources is different
from inappropriate influence of study sponsor.
B4O4
Consensus-based recommendations for
investigating clinical heterogeneity in systematic
reviews
Gagnier JJ1 , Morgenstern H2 , Moher D3
1 Assistant Professor, Departments of Orthopaedic Surgery and Epidemiology, University of Michigan, Ann Arbor, Michigan, USA; 2 Professor,
Department of Epidemiology, University of Michigan, Ann Arbor, Michigan,
USA; 3 Senior Scientist, Clinical Epidemiology, Ottawa Health Research
Institute, Ottawa, Ontario, Canada
Background: It has been argued that systematic reviews (SRs) fail
to inform clinical decision-making due to their results being far too
general such that the findings do not apply to individual clinical cases.
Also, few SRs investigate clinical reasons for heterogeneity and when
they do often fail to use valid statistical or other methods. While there
is some consensus on methods in SRs for investigating statistical and
methodological heterogeneity, little attention has been paid to clinical
aspects of heterogeneity. Objective: To develop recommendations
for investigating clinical heterogeneity in SRs. Methods: We
identified and invited potential participants with expertise in SR
methodology, SR reporting, heterogeneity, statistical aspects of
meta-analyses, or those who published papers on clinical heterogeneity.
Three phases are being conducted: i) pre-meeting item generation,
ii) face to face consensus meeting in the form of a modified Delphi
process, and iii) post-meeting feedback. Results: A total of
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
19 participants have agreed to attend the meeting that is being
held 3–4 June 2011 at the University of Michigan, USA. Currently, we
are performing the pre-meeting item generation that consists of phone
calls with all participants, asking them to indicate important potential
recommendations that need to be discussed during the face-to-face
meeting. In addition, we have extracted recommendations from two
recent comprehensive methodologic reviews in the area. We are
collating and cross-referencing the phone call results and the extracted
data from the methodologic reviews to construct items to be discussed
during the face-to-face consensus meeting in June. We anticipate
that by July 2011, we will have completed a manuscript that outlines
the results of this consensus meeting. This manuscript will outline
the first set of consensus-based guidelines for investigating clinical
heterogeneity in SRs. Discussion: We expect that these guidelines for
investigating clinical heterogeneity in SRs will improve the validity of
such investigations resulting in SRs with more applicable results.
B5 Methods for preparing reviews
(statistical and non-statistical)
B5O1
Intensity of a public health intervention
in a systematic review: Assessing it
Baker P1 , Francis D2 , Soares J3
1 School of Public Health Queensland University of Technology and
Epidemiology Central Regional Services Queensland Health, Australia;
2 Epidemiology Central Regional Services Queensland Health, Australia;
3 Division of Nutrition, Physical Activity and Obesity, National Center for
Chronic Disease Prevention and Health Promotion, Centers for Disease
Control and Prevention, USA
Background: Intensity of an intervention is not well defined in the
public health literature and it may be one explanation for differences
that exist in the outcomes between public health studies. Objectives:
The aim of this study was to describe the approach used in a recent
review about community wide interventions for increasing physical
activity to categorise the intensity of interventions to assess whether
this could help explain differences that existed in the outcomes reported.
Methods: A subjective criterion, by which each study was rated by two
independent reviewers, was developed. The criteria were based on six
characteristics and attributes that we hypothesised would be important.
Each characteristic was assessed as ‘more intensive’, ‘less intensive’ or
‘unclear’. It included: 1) development of community partnerships or
coalition; 2) levels of intervention; 3) reach of strategies; 4) continuous
provision of intervention; 5) cost; and 6) statement of intensity by
the authors. The overall assessment of intensity for each study was
categorised as ‘high’, ‘medium, ‘low’ or ‘unclear’. Results: The tool
was simple to apply. Nine of the 25 studies included in the review
were classified as being ‘high intensity’, 10 of ‘medium intensity’ and
six of ‘low intensity’. Seven of the 25 studies reported some improved
physical activity outcomes and the majority of these (n = 5) were
classified as high intensity. Consistency between assessors will be
investigated and reported. Conclusions: Interpretation of the studies
in this review was challenging due to the overall poor quality of the
included studies, however assessment of intensity is potentially useful
for understanding heterogeneity of effectiveness between public health
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
23
studies. We recommend that intensity of a public health intervention
be assessed in reviews of public health interventions where there is
likely to be significant difference in the interventions and populations
at which they are targeted.
B5O2
The implications of different intervention
definitions on a network meta-analysis of
acupuncture and other physical therapies for the
relief of chronic pain due to osteoarthritis of the
knee
Rice SJC1 , Woolacott NF1 , Corbett MS1 , Slack R1
1 University of York, UK
Background: A systematic review and network meta-analysis of
acupuncture and other physical therapies for the relief of chronic
pain due to osteoarthritis of the knee was undertaken. There are
several different physical therapies, variations of the same type of
physical therapy, and combinations of therapies for osteoarthritis of
the knee pain. The definition of treatment groups need to enable useful
interpretation of the results for policy. Objectives: The objective of this
communication is to present the implications of different intervention
definitions on the network meta-analyses in this review. Methods:
Given the multiple possible combinations of physical therapies for
osteoarthritis of the knee, combinations of the main interventions were
excluded from the analysis to make the project manageable. For one
network analysis, 22 main interventions were defined and each one
had the possibility of five adjunct treatments resulting in 110 possible
treatment groups. For another network analysis, the five adjunct
treatments were grouped into one group, leaving 22 interventions
defined. For another, all physiotherapy interventions were combined
into one group to reflect options available to general practitioners in the
UK. Results: For the detailed network there were either no significant
differences between the treatment effects of the interventions with the
same principal physical therapy or the face validity was questionable.
With more interventions defined, there were fewer trials per treatment
comparison in the model and there was a low power to detect
treatment effect differences. Results for the grouped interventions
network were influenced by the new network structure and additional
interventions and trials being captured within the broader treatment
definition. Conclusions: In complex network meta-analyses decisions
made about the definition of interventions are important. A too great
an emphasis on differences between interventions results in a large
network comprised of few trials per treatment comparison in the model,
with low power to detect treatment effect differences.
B5O3
Latent class bivariate model for the
meta-analysis of diagnostic test accuracy studies
Eusebi P1 , Reitsma JB2 , Vermunt JK3
1 Department of Epidemiology, Regional Health Authority of Umbria, Italy;
2 Department of Clinical Epidemiology, Biostatistics and Bioinformatics,
Academic Medical Center Amsterdam, Netherlands; 3 Faculty of Social and
Behavioural Sciences, Department Methodology and Statistics, University
of Tilburg, Netherlands
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Several statistical methods for meta-analysis of data
from diagnostic test accuracy studies have been proposed. The
Bivariate Model is a rigorous method for this purpose by directly
analyzing estimates of sensitivity and specificity. Objectives: Our
research is motivated by a re-analysis of data of a published systematic
review (1). In this meta-analysis three imaging techniques were
compared for the diagnosis of lymph node metastasis in women with
cervical cancer. Observing the heterogeneity amount on the data we
aim at implementing new methods for helping the understanding on
the relationship between sensitivity and specificity. Methods: We
propose the Latent Class Bivariate Model, an extension of the Bivariate
Model by means of a discrete latent variable for finding clusters of
studies. Results: Several type of models were fitted with Latent
GOLD software (2). The best model is the Latent Class Bivariate Model
with the type of test as a nominal covariate. This model detected
two latent classes of studies. Studies belonging to the first latent
class show lower sensitivity but higher specificity and almost all the
studies are X-ray computed tomography (CT) and magnetic resonance
imaging (MRI); in that class sensitivity and specificity appear to be
negatively correlated. Studies belonging to the second latent class
show lower specificity but higher sensitivity and almost all the studies
are lymphangiography (LAG); in that class sensitivity and specificity
appear not to be correlated. Conclusions: What is added by the
latent approach is that it provides an explanatory and confirmatory tool
for investigating and testing different patterns of heterogeneity across
studies. We tested the performance equivalence of CT and MRI and
the different correlation between sensitivity and specificity in LAG and
CT/MRI studies. Additional insight and data-driven hypothesis can be
generated for future subgroup meta-analysis.
References
1. Scheidler J, Hricak H, Yu KK, Subak L, Segal MR. Radiological
evaluation of lymph node metastases in patients with cervical cancer.
A meta-analysis. JAMA. 1997;278(13):1096–101.
2. Vermunt JK, Magidson J. (2008). LG-Syntax User’s Guide:
Manual for Latent GOLD 4.5 Syntax Module, Belmont, MA: Statistical
Innovations Inc.
B5O4
A new strategy for meta-analysis of continuous
covariates in observational studies
Sauerbrei W1 , Royston P2
1
IMBI, University Medical Center Freiburg, Germany; 2 MRC Clinical Trials
Unit, London, UK
Background: Assessing the effect of a continuous covariate in a
single study requires the determination of a dose-response relationship
in a model adjusting for other covariates. Even with individual patient
data (IPD), summarizing the results of several dose-response functions
in a meta-analysis is not straightforward. Objectives: To describe
a new procedure which determines in single studies an adjusted
dose-response function for a continuous variable and which also
summarizes such functions across studies. Methods: In single studies
we determine functions using fractional polynomials, with or without
adjustment for confounders. Three different methods to select the FP
function are proposed. To average functions from several studies, a
fixed or random effect approach can be used. Results: Using data
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
24
from the US SEER database in which each individual registry is treated
as a single study, we derive adjusted overall estimates of the functional
forms for the association between time to breast cancer death and a
continuous covariate. Whereas the individual functions for nodes, a
factor with a large effect, are similar across studies, those for age, a
covariate with a weak effect, show considerable variability. Because
of their different weights, whether to use a fixed or a random effects
model affects the average function. Conclusions: For an ‘ideal’
situation with IPD and only minor variations between studies with
respect to measurement techniques and confounders, our approach
allows one to model dose-response relationships in single studies and
to summarize them in an average function. Modifications are available
if the data situation is less than ideal.
Figure 1 (B601). Impact of future study on meta-analysis of trials for
obstructive sleep apnoea. Oral appliance versus continuous positive
airways pressure; outcome difference in Epworth score.
B6 Statistical methods
B6O1
Graphical augmentations to the funnel plot for
assessing the impact of additional evidence on
a meta-analysis
Langan D1 , Higgins J2 , Gregory W3 , Crowther M4 , Sutton A4
1 Clinical Trials Research Unit, University of Leeds, UK; 2 MRC Biostatistics
Unit, Cambridge, UK; 3 Clinical Trials Research Unit, University of Leeds,
UK; 4 University of Leicester, UK
Background: Methods to quantify the potential impact of future
evidence on a systematic review are currently under-developed but
would greatly aid decisions on both review updating priorities and
recommendations for future research. Objectives: Develop novel
overlays to the funnel plot to provide a visual illustration of the impact
that a new study would have on a given meta-analysis. The additional
features help inform: i) the current robustness of a meta-analysis;
ii) sample size calculations for the design of future studies to be added
to the meta-analysis; and iii) the update prioritisation strategy for
a portfolio of meta-analyses (such as those managed by Cochrane
Review Groups). Description of developed methodology: Several
inter-related overlays to the funnel plot are described and illustrated
using data from Cochrane reviews. These include: i) statistical
significance contours, which define regions of the funnel plot in which
a new study would have to be located in order to change the statistical
significance of the meta-analysis (an example of these for a sleep
apnoea review is included in the figure); and ii) heterogeneity contours,
which show how a new study would affect the extent of heterogeneity
in a given meta-analysis. The use of the aforementioned overlays
simultaneously with further features including: pooled treatment
effects, lines of no effect, confidence and prediction intervals, simulated
new studies and sample size guidelines, are all discussed. Details of
free software for creating the plots in both R and STATA are provided.
Suggestions for how review editors, review authors and trial designers
could use the plots will be considered. Conclusions: The authors
believe the developed enhancements to the funnel plot provide a lot
of accessible information about the impact of future evidence on an
existing evidence base. It is hoped editors, systematic reviewer authors
and trialists will find such plots useful.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
B6O2
Bayesian approach to meta-analysis. What can
you gain?
Walczak J1 , Borowiack E2 , Nikodem M1 , Siedmiogrodzki K1 ,
Zapalska A2 , Khan K3 , Meads C3 , Mol B4 , Oude Rengerink K4 ,
Thangaratinam S3 , Zamora J5
1 CASPolska Association, Poland; 2 Arcana Institute, Poland; 3 Queen Mary,
University of London, Great Britain; 4 Academic Medical Center Amsterdam,
Netherlands; 5 Hospital Universitario Ramon y Cajal, Madrid, Spain
Background: The Bayesian approach gives new opportunities such
as aggregating different types of data, combining direct and indirect
comparison or assessing clinical significance. On the other hand,
the variety of Bayesian models can beconfusing, and implementation
difficulties can cause unwillingness to apply it. Objectives: The aim
is to systemize the knowledge of application of Bayesian statistics
in the area of meta-analysis, and to compare it with traditional
statistical methods. We want to identify the situations in which use
of the Bayesian approach is really worthwhile. Methods: Initially the
systematic reviews were conducted using existing statistical methods
for meta-analyses. Special attention was given to different Bayesian
models which were then implemented in the WinBUGS environment
and examined on different data sets. Results: In the case of
regular meta-analysis of dichotomous data, applying basic Bayesian
models leads us, in fact, to similar results of estimation as the
Mantel-Haenszel or DerSimonian-Laird method. The real advantage of
the Bayesian approach is noticed if we expect something more than
typical meta-analysis, especially if we have to deal with the following
problems: 1. Assessing the clinical significance – for instance, assessing
the chance that relative risk is greater than 1.25 (or any other level
of significance). 2. Combining data from different type of studies
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
25
-including extra information (e.g. results of non-randomized studies)
to meta-analysis, keeping a moderate ‘level of conviction’ to this extra
data. 3. Combining direct and indirect evidence (Mixed Treatment
Comparison). Conclusions: Bayesian statistics give us technical
opportunities to improve meta-analysis, especially in the area of
aggregating multi-type data. On the other hand, there is no significant
difference between the results obtained by the Bayesian and traditional
approach in the case of simple meta-analysis of regular data. Moreover,
if WinBUGS codes are prepared, conducting the calculations is not as
difficult as one may think.
B6O3
A Bayesian selection model for incorporating
prior information for publication bias in
meta-analysis
Mavridis D1 , Cipriani A2 , Sutton A3 , Salanti G1
1 Department of Hygiene and Epidemiology, University of Ioannina School of
Medicine, Ioannina, Greece; 2 Department of Public Health and Community
Medicine, Section of Psychiatry and Clinical Psychology, University of
Verona, Policlinico ‘G.B. Rossi’, Italy; 3 Department of Health Sciences
University of Leicester, UK
Background: The Copas parametric model is aimed at exploring the
potential impact of publication bias via sensitivity analysis, by making
assumptions regarding the probability of publication of individual
studies related to the standard error of their effect sizes (Copas
and Shi). Review authors often have prior knowledge about the
extent of selection in the set of studies included in a meta-analysis.
However, a Bayesian implementation of the Copas model has not been
studied yet. Objectives: To present a Bayesian selection model for
publication bias and to extend it to the case of multiple-treatments
meta-analysis. Methods: We explored various models corresponding
to the shape of the association between precision and probability of
publication. We used a dataset of antidepressant studies submitted to
the US regulatory agency (Food and Drug Administration), according
to Turner and colleagues (Turner et al.). We took advantage of the
greater flexibility offered in the Bayesian context to incorporate in
the model prior information on the extent and strength of selection.
To derive prior distributions, we used both external data and an
elicitation process of expert opinion. Results: The Copas model
assumes a likelihood approach which requires strong assumptions
for the model to be identified and a sensitivity analysis is employed.
Bayesian implementation using informative priors provides a solution
to this problem by explaining adequately the selection process and
giving clear-cut results. Extension of the model for multiple-treatments
meta-analysis allows incorporation of comparison-specific priors beliefs
and thus allowed a flexible class of models to be fit. Conclusions: A
Bayesian framework utilizing expert opinion provided a powerful and
easily understood approach.
References
Copas JB, Shi JQ. A sensitivity analysis for publication bias in systematic
reviews. Stat Methods Med Res. 2001;10(4):251–65.
Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective
publication of antidepressant trials and its influence on apparent
efficacy. N Engl J Med. 2008;358(3):252–60.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
B6O4
The implications for meta-analysis of different
methods of analysing count data
Herbison P1
1 University of Otago, New Zealand
Background: In randomised trials with count outcomes (such as
number of falls/exacerbations of asthma) it is possible to analyse the
results in many different ways. It is not known whether it is possible
to combine the results of these different methods of analysis in a
meta-analysis. Objectives: To see if different methods of analysis
provide results that are alike enough to combine in a meta-analysis.
Methods: Simulation study with count data with a range of means
and amount of overdispersion. Different analyses were rate ratios,
poisson regression, negative binomial regression, dichotomising the
data (with and without the event), three forms of survival analyses,
ratio of means and ratio of medians. Confirmation by the analysis of
individual patient data from at least 14 randomised trials. Results:
With a low mean, (about 20% have the event) most methods of
analysis produced similar answers, except ratio of medians, which was
not possible to calculate. As the mean increased, dichotomising the
data increasingly underestimated the treatment effect, as did time to
first event and this effect was noticeable even when 50% had the event.
Survival models allowing for multiple events helped, but not completely.
Negative binomial gave results very similar to poisson regression, even
for considerable amounts of overdispersion, where these models were
better fits to the data. These results were confirmed by the analyses
of the individual patient data. Conclusion: It should be possible to
combine more studies in a meta-analysis than was previously expected.
Relative risks from dichotomised data and hazard ratios from time to
first event analyses should only be included if the event rate is very
low. Means can be converted to ratios and an approximate standard
deviation calculated, but for the ratio of medians it is not possible to
work out the standard deviation from published data.
B7 Investigating bias
B7O1
A descriptive analysis of risk of bias in neonatal
trials and the association with effect estimates
Bialy L1 , Lacaze-Masmonteil T2 , Dryden DM1 , Ha C1 , Armijo-Olivo S1 ,
Vandermeer B1 , Hartling L1
1
Alberta Research Centre for Health Evidence, Canada; 2 Children’s Hospital
of Eastern Ontario, Canada
Background: Research that quantifies the impact of different biases on
effect sizes in neonatal randomized controlled trials (RCTs) has shown
conflicting results. A meta-epidemiological study to quantify bias in
neonatal RCTs will inform the design, conduct, and interpretation of
research in this vulnerable population. Objectives: To describe a
sample of neonatal RCTs in terms of methodology, risk of bias, and
associations with effect estimates. Methods: We included all neonatal
RCTs (n = 208) in the Cochrane Database of Systematic Reviews that
examined the following key treatments: surfactant, corticosteroids,
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
26
nitric oxide, indomethacin, ibuprofen, and head or total body cooling.
Risk of bias was assessed on nine domains by two independent
reviewers. Meta-epidemiological methods will be used to quantify
the association between pre-specified methodological characteristics
and effect estimates. Results: The assessed RCTs were published
between 1972 and 2009 with 52% conducted across multiple centres.
A single or composite primary outcome was reported in 49%. Trials
were supported by grants from academic institutions or governments
(37%) or pharmaceutical industry (26%); 37% did not report a funding
source. No RCTs had an overall ‘low’ risk of bias assessment; 42%
were assessed as high risk of bias and 58% were unclear. For allocation
concealment and blinding of participants and investigators most RCTs
received an unclear risk of bias (69%, 74%, and 64%, respectively).
Nineteen percent were assigned a high risk for selectively reporting
outcomes and 60% were unclear. Analyses of association between
risk of bias and effect estimates are ongoing. Conclusions: We
have described the risk of bias for RCTs in 6 key areas of neonatal
medicine. This body of literature shows methodological limitations in
terms of selective reporting and documentation of blinding methods.
It is critical to understand and quantify the potential impact of these
biases on effect estimates in order to better ensure the delivery of
optimal neonatal care.
B7O2
Single-center trials show larger treatment
effects than multicenter trials: evidence from
a meta-epidemiological study
Dechartres A1 , Boutron I2 , Trinquart L3 , Charles P4 , Ravaud P2
1 INSERM, Universite Paris Descartes, Hotel-Dieu Hospital (APHP), France;
2
French Cochrane Centre, INSERM, Universite Paris Descartes, Hotel-Dieu
Hospital (APHP), France; 3 French Cochrane Centre, France; 4 Cochin
Hospital (APHP), Universite Paris Descartes, France
Background: For many physicians, multicenter randomized controlled
trials (RCTs) are considered more valid than single-center RCTs but
there is no formal evidence about this fact. Objectives: We
performed a meta-epidemiological study to evaluate whether treatment
effect estimates differ between single-center and multicenter RCTs.
Methods: PubMed was searched for meta-analyses of RCTs with
binary outcomes published between August 2008 and January 2009
and in the first 6 months of 2010 in the 10 leading journals of each
medical specialty. One issue of the Cochrane Database of Systematic
Reviews was also searched. All individual RCTs included in the
meta-analyses were selected and searched. Data were extracted and
quality assessed by use of the Risk of Bias (RoB) tool of The Cochrane
Collaboration. The primary outcome was the ratio of odds ratio (ROR)
quantifying the difference in estimated intervention effect between
single-center and multicenter RCTs. A ROR < 1 would indicate larger
estimates of the intervention effect in single-center trials. We performed
sensitivity analyses with adjustment for sample size, risk of bias within
RCTs and variance of the log OR to take into account publication
bias. Results: We selected 48 meta-analyses including 421 RCTs:
223 were single-center and 198 multicenter RCTs. Single-center RCTs
showed larger intervention effect than multicenter RCTs (combined
ROR = 0.73, 95% CI: 0.64 to 0.83), with low heterogeneity across
individual meta-analyses (I2 = 12.0%, p = 0.24). Adjustment for
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
sample size yielded consistent results (ROR = 0.85, 95% CI: 0.74
to 0.97), as did adjustment for ROB within RCTs, such as allocation
concealment (ROR = 0.76, 95% CI: 0.67 to 0.86) and variance of log
OR (ROR = 0.83, 95% CI: 0.72 to 0.96). Conclusions: Single-center
RCTs showed larger treatment effects than multicenter RCTs, which
was consistent in all sensitivity analyses. Our results suggest taking
this item into consideration when interpreting the results of RCTs and
meta-analyses.
B703
PROSPERO: An international prospective register
of systematic review protocols
Booth A1 , Clarke M2 , Ghersi D3 , Moher D4 , Petticrew M5 , Stewart L1
1 Centre for Reviews and Dissemination, UK; 2 UK Cochrane Centre, UK;
3
International Clinical Trials Registry Platform, WHO, Switzerland; 4 Clinical
Epidemiology Program, Ottawa Hospital Research Institute, Canada;
5 Department of Scoial and Environmental Health Research, London School
of Hygiene and Tropical Medicine, UK
Background: Support has grown for the prospective registration
of protocols for systematic reviews to encourage transparency in
conduct and reporting. Historically, systematic review protocol
registers were limited to organisations such as The Cochrane and
Campbell Collaborations. Following publication of the PRISMA
statement, the UK Centre for Reviews and Dissemination (CRD)
took the initiative to develop an international prospective register
of systematic reviews with health-related outcomes. Methods: An
international consultation was carried out to identify the key items
for inclusion in the register. These were divided into required and
optional items. A two-round electronic modified Delphi survey design
was used. The panel was comprised of relevant experts, including
commissioners, clinical and academic researchers, methodologists,
statisticians, information specialists, publishers and users of systematic
review evidence. Results: In February 2011, CRD launched PROSPERO,
an international register of systematic reviews at the protocol stage.
The required dataset, established through the consultation, consists of
22 required items and 18 optional items. The dataset captures the
key attributes of review design as well as the administrative details
necessary for registration. PROSPERO offers free registration and free
public access. The dedicated web-based interface is electronically
searchable and open to all prospective registrants. Cochrane protocols
do not need to be registered individually as automatic uploading will be
developed. Initially inclusion is restricted to systematic reviews of the
effects of interventions and strategies to prevent, diagnose, treat, and
monitor health conditions, for which there is a health-related outcome.
The long term aim is to widen the scope to include all ongoing
systematic reviews that have a health related outcome. Conclusions:
PROSPERO should provide a way of assessing the risk of bias in
systematic reviews, support the efficient use of funding and timely
updating, and contribute to improving the quality of systematic reviews
and the decisions that rely upon them.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
27
B7O4
Characteristics of physicians who receive large
payments from industry and the accuracy of their
disclosures in concurrent publications
Norris SL1 , Holmer HK1 , Burda BU2 , Ogden LA1 , McAteer AM1
1
Department of Medical Informatics and Clinical Epidemiology, Oregon
Health & Science University, USA; 2 Oregon Evidence-Based Practice Center,
Kaiser Permanente Center for Health Research, USA
Background: Authors’ financial conflicts of interest (COI) may bias the
results of primary studies and therefore systematic reviews. There are
few data, however, on the accuracy of COI disclosures in publications
of primary research. Objective: To examine the accuracy of disclosures
by individuals known to have received money from drug companies.
Methods: ProPublica, an investigative journalism organization,
identified 384 individuals (mainly physicians) who received more than
$100,000 from one or more of seven major pharmaceutical companies
for consultancies and other peer educational activities over varying
periods between January 2009 and September 2010. We explored the
demographic characteristics and clinical specialties of these individuals
using several public databases, as well as disclosures in their concurrent
publications identified in MEDLINE. Results: Of the 384 individuals,
373 were physicians, including psychiatrists (32%), internists (31%),
and a variety of other clinical specialties. Seventeen physicians had
received state disciplinary actions. The mean age of the physicians
was 53.5 years and 11% were female. The median total payment
was $126,724 (range, $100,047 to $303,558). Only 159 individuals
had published in the relevant time period, authoring more than 700
publications. Of these, greater than half contained disclosures. Of
publications with disclosures, approximately 15% had discrepancies
between the disclosures and information provided by ProPublica.
We will present additional data on the nature of the discrepancies.
Conclusions: US medical providers who receive large sums of money
from the pharmaceutical industry for speaking and other non-research
engagements frequently do not report this information in concurrent
publications. The systematic reviewer thus has incomplete information
upon which to evaluate the validity of primary studies. Our conclusions
are limited by the small number of pharmaceutical companies providing
payment information to the public.
B8 Investigating bias
B8O1
Can trial quality be reliably assessed from a trial
publication?
Vale C1 , Tierney J1 , Burdett S1
1 Meta-analysis Group, MRC Clinical Trials Unit, London, UK
Background: Assessment of risk of bias (RoB) using information from
publications of randomised controlled trials (RCTs) is common practice
for systematic reviews. It has been proposed as a basis for adjusting
the results of meta-analyses to account for trials assessed at high
risk. Objectives: We aimed to assess whether RoB assessments from
trial reports are comparable to those using supplementary information
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
obtained from trialists. Methods: We sampled published RCTs
included in our meta-analyses of individual participant data (IPDMA)
for which trial protocols or forms detailing trial design were available.
RoB was assessed using criteria outlined in the Cochrane Handbook
and the Cochrane Risk of Bias tool (RevMan 5.1). Assessments were
conducted for each trial based on trial reports only, or trial reports
plus supplementary information from trialists, and were compared for
consistency within domains and overall. Results: Preliminary results
are based on 25 RCTs from 3 IPDMAs in cancer. Using trial reports,
the RoB was judged to be low (14 trials) or unclear (11 trials) for
sequence generation and low (10 trials) or unclear (15 trials) for
allocation concealment. However, supplementary information resulted
in low RoB assessments for all trials for sequence generation and
for 24 trials for allocation concealment. Selective reporting bias was
judged to be high (14 trials) or unclear (10 trials) from publications
alone, however supplementary data led to 24 trials being assessed
as low RoB. Conclusions: Supplementary information from trialists
can alter high or unclear judgements of bias, particularly in regard to
selective reporting of outcomes. Our results suggest review authors
should be cautious in adjusting the results of meta-analyses based
on RoB assessments from publications, particularly where assessments
are unclear. Results will be updated to include 7 additional IPDMAs
(76 RCTs) and the randomisation integrity will be further explored
using IPD.
B8O2
Inter-rater reliability of a new instrument for
assessing potential for bias in prognosis studies
Peterson K1 , Carson S1 , Carney N1
1
Oregon Evidence-based Practice Center, Oregon Health & Science
University, USA
Background: There is no gold standard for assessing the potential for
bias in prognosis studies. The best available recommendations come
from a 2006 ‘review of reviews’. However, very little formal testing
has been done on applying those recommendations. Objectives: To
assess inter-rater reliability of an instrument for assessing potential for
bias in prognosis studies. Methods: We developed an instrument
for assessing the potential for bias in prognosis studies based on
the recommendations of Hayden et al. Our instrument consists of
24 items across 6 domains (patient selection methods, prognostic
factor measurement, outcome measurement, follow-up, analysis and
reporting methods, and measurement of confounders). The potential
for bias was rated as low, medium, or high for each domain
and for each study overall. Two reviewers independently assessed
37 studies included in a systematic review conducted for the Brain
Trauma Foundation’s second edition of Early Indicators of Prognosis
for Moderate to Severe Traumatic Brain Injury. Inter-rater reliability
was estimated using Cohen’s kappa. Results: Inter-rater reliability
was moderate for the overall potential for bias (weighted Cohen’s
kappa: 0.78; 95% Confidence Interval (CI) 0.52 to 1.05). Although
observed agreement was generally high across the individual domains
(range 74% to 93%), kappa values ranged widely. For example,
kappa values indicated ‘less than chance’ reliability for the outcome
measurement domain (−0.03; 95% CI −0.31 to −0.26), but ‘almost
perfect’ reliability for the analysis and reporting methods domain (0.83;
95% CI 0.52 to 1.14). Kappa values were lowest for domains having
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
28
distributions that were skewed toward ratings of low to medium
potential for bias. Conclusions: Our instrument had moderate
inter-rater reliability for assessing overall potential for bias in prognosis
studies. Before our instrument can be used with confidence, however,
more reliability and validity testing is needed on studies representing
broader ranges of validity and clinical topic areas.
B8O3
Risk of Bias Assessment tool for Non-randomized
Studies (RoBANS): Development and validation
of a new instrument
Park J1 , Lee Y1 , Seo H2 , Jang B1 , Son H1 , Kim SY3 , Shin S4 , Hahn S5
1 Department of Health Technology Assessment Research, National
Evidence-based Healthcare Collaborating Agency, Seoul, Korea;
2
Department of Nursing, College of Medicine, Chosun University, Gwangju,
Korea; 3 Department of Family Medicine, College of Medicine, Hallym
University, Seoul, Korea; 4 Department of Pulmonary Medicine, Ajou
University, Suwon, Korea; 5 Department of Medicine, College of Medicine,
Seoul National University, Seoul, Korea
Background: Few validated checklist tools are available to assess the
risk of bias (RoB) of non-randomized studies (NRS), although systematic
reviews including NRS have increased in number, for many reasons.
Objectives: The aims of the study are to develop and validate a new
RoB tool for NRS. Methods: After deciding principles and reviewing
relevant tools, we developed the Risk of Bias Assessment tool for
Non-randomized Studies (RoBANS) tool. Subsequent testing involved
the assessment of inter-rater agreement between reviewers, time to
apply the RoBANS tool, degree of correlation for overall risk compared
with overall quality scores, user-convenience, and content validity.
Results: The RoBANS contains 6 domains including the selection
of participants, confounding variables, measurement of intervention
(exposure), blinding of outcome assessment, incomplete outcome data
and selective outcome reporting (table). Inter-rater agreement of the
RoBANS tool varied in individual domains (weighted κ = 0.38 to
0.74). The mean time to complete the RoBANS tool was less than
for the MINORS (Methodological Items for NOn-Randomized Studies)
(9.5 minutes (SD 3.39) per study vs. 10.45 (SD 3.54), P = 0.18).
There was a moderate correlation between overall RoB compared with
the MINORS (Kendall’s τ = 0.384). Reviewers evaluated RoBANS as
‘good’ in terms of user-convenience for 3 items (mean 5.7 of 7 as the
Likert scale; 0.81 %). External experts (n = 8) evaluated RoBANS
as ‘fair’ in terms of face validity for 7 items (mean 5.4 of 7 as the
Likert scale) and all recommended its use to assess the RoB for NRS.
Conclusions: RoBANS is a valid tool designed to assess the RoB of
NRS. Since RoBANS is harmonized with the Cochrane’s RoB tool and
GRADE (Grading of Recommendations Assessment, Development and
Evaluation), and can be incorporated into RevMan and GRADEpro, it
appears to be useful to people undertaking systematic reviews.
B8O4
Initial experience with a pilot Cochrane tool for
assessing risk of bias for non-randomized studies
applying a web-based survey of content experts
to derive criteria for imbalance
Palmer DA1 , MacLennan SJ2 , Imamura M2 , Reeves BC3 , Clubb A1 ,
Duboy AJ1 , Neuberger MM1 , Dahm P1
1 Department of Urology, College of Medicine, University of Florida,
Gainesville, Florida, USA; 2 Academic Urology Unit, University of Aberdeen,
Aberdeen, UK; 3 Clinical Trials and Evaluation Unit, University of Bristol,
Bristol, UK
Background: To date, few tools exist for assessing the risk of bias
(RoB) from confounding variables in non-randomized studies (NRS).
Objectives: We tested a pilot Cochrane tool for RoB to assess its
applicability and value in quantifying RoB in NRS. Methods: We
applied the Cochrane tool to a systematic review of NRS evaluating the
effectiveness of lymph node dissection in patients undergoing radical
Table 1 (B803). The developed and validated version of RoBANS.
Domain
Description
Risk of bias
Selection of participants
Selection bias caused by inadequate selection of participants
Confounding variables
Selection bias caused by inadequate confirmation and consideration of
confounding variable
Intervention (exposure) measurement
Performance bias caused by inadequate measurements of intervention (exposure)
Blinding of outcome assessment
Detection bias caused by inadequate blinding of outcome assessment
Incomplete outcome data
Attrition bias caused by inadequate handling of incomplete outcome data
Selective outcome reporting
Reporting bias caused by selective outcome reporting
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Low
High
Unclear
Low
High
Unclear
Low
High
Unclear
Low
High
Unclear
Low
High
Unclear
Low
High
Unclear
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
29
cystectomy. Three independent review authors abstracted data on
methodological quality using the Newcastle-Ottawa instrument and
the Cochrane tool for assessing RoB.[1] The Cochrane tool assesses the
precision, imbalance, and adjustment for each confounding variable,
and overall RoB from confounding, blinding, incomplete outcome
data and selective reporting, on 1 (low ROB) to 5 (high RoB) point
scales. We pilot-tested the data abstraction form and developed
detailed guidelines to optimize inter-observer agreement. Results:
Five NRS met the inclusion criteria. Confounders were identified in
advance by literature search and responses to a web-based survey of
eleven bladder cancer experts who were also asked to categorize what
difference in the proportion of a confounding variable represented ‘no’,
a ‘small’, ‘moderate’ or ‘large’ imbalance. To evaluate imbalance,
each confounder was evaluated dichotomously (e.g. organ-confined
versus non-organ-confined). Threshold values were derived using
the median responses; e.g. for non-organ-confined disease, these
were set at less than or equal to 2%, 2.1 to 5%, 5.1 to 7%,
greater than 7%, respectively, thereby providing explicit anchors
for imbalance criterion scoring. Conclusions: The novel RoB tool
represents a valuable instrument for quantifying the RoB of NRS
that deserves further formal investigation in a larger, more diverse
sample of studies. The described survey method appears an effective
and efficient method to define the necessary imbalance criterion
thresholds.
Results: We have co-authored one published SR, one protocol, one
peer reviewed manuscript and are currently in the process of updating
one SR and authoring two others. One poster was presented at
the 2010 Cochrane Colloquium and an editorial published on The
Cochrane library. Within the ambit of programme development, we
have adapted a SR module for amalgamation within the University
of the West Indies MSc Epidemiology and delivered a module to one
cohort of students. We made links with the Pan American Health
Organization, The International Initiative for Impact Evaluations and
several Cochrane entities. We aim to achieve a mass of six trained
individuals by conducting SR workshops and collaborating with INCLEN
partners to host fellowships and establishing networks to help sustain
activity. Conclusions: This experience responds to a growing and
unique need for evidence based healthcare in developing countries.
Capacity building activities such as fellowships and regional training
workshops are viable avenues for developing the required cadre for the
Caribbean region. The concept and execution could serve as a model
for the developing world.
Reference
Struthers C1 , Noel-Storr A1 , Marcus S1 , McCleery J1 , McShane R1
1 Cochrane Dementia and Cognitive Improvement Group, UK
1. Reeves BC, Shea B, Wells GA. Classifying non-randomised studies
(NRS) and the assessing the risk of bias for a systematic review.
Workshop, 18th Cochrane Colloquium, Keystone 2010.
C1 Education and training
C1O1
Expanding the Cochrane Collaboration in low
and middle income countries: A Caribbean
experience
Francis D1 , Pardo Pardo J2 , Benkhalti Jandu M2 , Tulloch-Reid M1 , Wilks
R1 , Tugwell P2
1 Epidemiology Research Unit, University of the West Indies Mona, Jamaica;
2
Center for Global Health, University of Ottawa, Canada
Background: The Caribbean region requires a cadre of trained health
scientists to review existing research and provide evidence based
recommendations to address regional needs. We report on efforts
to establish a Cochrane branch through our membership within the
International Clinical Epidemiology Network (INCLEN) to address this
deficiency. Methods: A fellowship training programme facilitated
through a scholarship grant from the Canadian Bureau of International
Education and an INCLEN partner was used to train a suitable candidate
for this process. Training was carried out at the Center for Global
Health, University of Ottawa over a 4 to 6 month period and focused
on systematic review (SR) author training, programme development
in evidence based medicine and capacity building and networking.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
C1O2
Feasibility of recruitment and training of carers
of people with dementia as a route to
Cochrane-NHS engagement
Background: Relatives of people with dementia often want to
participate in research, but their time is limited because of their
caregiving role. Objectives: The first, innovative part of the
proposal is to recruit carers and former carers of people with
dementia to a well-defined task: reading trial reports, extracting basic
information, and entering it into the Dementia Group’s online register
of studies – ALOIS. The volunteers are also invited to work through a
series of modules about EBM using examples, all directly related to
dementia. Methods: A range of publicity and outreach methods were
used to recruit volunteers (see leaflet attached). Also an EBM skills
workshop for carers was held in Oxford which was very well attended.
We will conduct a before-and-after study where the participants’ skill
at the coding task will be rated, and a questionnaire will be used to
assess changes in involvement with dementia-related NHS activities,
primary dementia research, and attitudes. Results: At the project
half-way point (see poster attached), we had recruited 35 volunteers
(20 carers or former carers). We have now reached 40, and this has
translated into 65 new study records being coded by new volunteers
and published on ALOIS, and a further 70 studies currently assigned to
volunteers for coding. A slightly unexpected but welcome development
is that schools are now interested in participating, and we are currently
looking at how adapt the model for this purpose. We are also hopeful
that the model might be further adapted as a Cochrane-wide ‘citizen
science’ initiative in conjunction with the development of the Cochrane
Register of Studies. Conclusion: Involving and training volunteer
‘expert carers’ in maintaining a trial register is feasible. We will
explore whether the model can be adapted to bring wider benefits
for public engagement with the NHS and participation in primary
research.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
30
Figure 1 (C102).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
31
Figure 2 (C102).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
32
C1O3
A pilot study of an evidence-based practice
training program for Filipino physiotherapists:
emerging evidence on outcomes and acceptability
Dizon JM1 , Grimmer-Somers K2 , Kumar S2
1 International Centre for Allied Health Evidence, University of
South Australia, Australia; and University of Santo Tomas, Philippines;
2 International Centre for Allied Health Evidence, University of South
Australia, Australia
Background: Evidence-based practice (EBP) has been challenging
for health professionals across the globe. Issues of implementing the
evidence become even more challenging among health professionals
in developing countries, due to cultural and contextual issues which
compound the existing barriers reported in the literature. An EBP
training (EBP for FilPTs) was developed for physiotherapists in a
developing country (Philippines) based on the literature and local
context exploratory studies. Objectives: This study reports the
findings of pilot testing the EBP for FilPTs (Table 1). The study
objectives are to: (1) Estimate the effect of the EBP training program
Table 1 (C103).
Outline of the training
Schedule
Topic
Strategy
Before the scheduled
training
08:30 – 09:00
09:00 – 09:45
09:45 – 10:00
10:00 – 10:30
Baseline knowledge and skills
assessment
Introduction to EBP
Hierarchy of evidence and study designs
Break
PICO Formulation
Adapted Fresno test
Questions on attitudes to EBP
Lecture
Lecture
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Participant’s Output
Lecture on drafting PICO Practical
session on:
a. Drafting PICO from a case
b. Drafting own PICO
Own PICO
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
33
Continued
Schedule
Topic
Strategy
Participant’s Output
10:30 – 11:15
11:15 – 12:00
12:00 – 01:30
01:30 – 03:00
Designing the search
Conducting the search
Lunch
Appraising the evidence
Lecture -demo
Practical session
Own search strategy
Acquire evidence
Lecture
Practical session
Appraised paper
03:00 – 03:15
03:15 – 04:30
Break
What does the evidence say and how
applicable is it? How can it be
applied?
EBP Checklist
Introduction to the online support
Post program knowledge and skills
assessment
Lecture
Practical session
Answer to own PICO and
completion of the EBP checklist
04:30 – 04:45
04:45 – 05:00
Demonstration
Adapted Fresno test, Questions on
attitudes to EBP and answer
evaluation questions
Table 2 (C103).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
34
in enhancing knowledge and skills in EBP; (2) Determine the sample
size for a subsequent RCT; and (3) Test the acceptability of the
EBP training program. Methods: Physiotherapists were randomly
allocated to the intervention (EBP for FilPTs) and the waitlist control
groups. An EBP Checklist (Table 2) specifically developed for the use of
Filipino physiotherapists to apply the findings, was introduced to the
intervention group. Pre-post intervention measures of knowledge and
skills in EBP were assessed using the Adapted Fresno test. Participants
and observers completed an evaluation summary and commented on
the program’s objectives and strategies, and its acceptability in the
context of practice. Results: The EBP for FilPTs resulted in significant
and large improvement (Effect Size = 0.8) in knowledge and skills. A
sample of at least 54 physiotherapists is required for the randomised
controlled trial (RCT). Minor changes were required based on the
evaluation. Overall, the EBP for FilPTs is a useful and acceptable
training for physiotherapists in the Philippines. Conclusions: The EBP
for FilPTs is considered to be an effective and acceptable medium to
teach EBP to physiotherapists in the Philippines. Currently, the RCT
is being conducted to assess the effectiveness of the EBP for FilPTs in
improving knowledge, skills, attitudes and behaviour in EBP.
C1O4
Evidence-based medicine training: Reaching the
front line clinician in an integrated medical care
system
Ching K1 , Regidor D2 , Okawa G1
1 Hawaii Permanente Medical Group, US;
Management Institute, US
2 Kaiser
Permanente Care
Background: A shared language surrounding assessment of medical
literature and translation into evidence-based practice (EBP) is central
to providing high-quality, coordinated, outcomes-based medical care.
There are few structured evidence-based medicine (EBM) training
programs targeting front line clinicians in a community setting. A
large regional medical group (more than 400 physicians) embraced a
program promoting a system-wide culture of EBM. EBM foundations
training was developed to provide reinforcement through hands-on
experience and system-wide dissemination through a regional grand
rounds program. Objective: To create a culture of evidence-based
practice in a single healthcare system through integration of an
evidence-based foundations training course into a regional grand
rounds program. Methods: A four-hour interactive training session
targets the principles of EBM and EBP during grand rounds content
development. Further EBM support is provided at follow-up sessions.
In the grand rounds program, presenters must provide a rationale
for key clinical recommendations by synthesizing the best available
evidence using standard formats for evidence and recommendation
grading. Physician clinics close for grand rounds, consistent language
for medical evidence is used system-wide, and intranet posting reaches
about 2000 healthcare professionals. Results: Since 2009, there have
been 51 course participants, including physicians and allied health
professionals. Grand rounds attendance averages 57% of the medical
group. To date, over 68 graded recommendations were made across 21
clinical topics. Course evaluations show high satisfaction and achieved
course objectives. Grand rounds participant evaluations demonstrate
clarity of the evidence base for clinical recommendations (96% positive
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
response). Improved outcome measures from specific recommendations
can be demonstrated as part of a broader, multi-faceted process, of
which EBP is an important component. Conclusion: A structured,
interactive EBP training program integrated into a regional
grand rounds program can successfully reach front line clinicians,
and advance evidence-based culture throughout a healthcare
system.
C2 Global health and equity
C2O1
Lack of equity reporting in Cochrane reviews
of effectiveness in chronic obstructive
pulmonary disease
Ospina M1 , Armijo-Olivo S2 , Bond K3 , Rowe B4
1 University of Alberta School of Public Health and Institute of Health
Economics, Canada; 2 Alberta Research Centre for Health Evidence and
Department of Pediatrics, University of Alberta, Canada; 3 Institute
of Health Economics, Canada; 4 Alberta Research Centre for Health
Evidence and Department of Emergency Medicine, University of Alberta,
Canada
Background: The importance of socioeconomic gradients in determining health status and treatment response is recognized worldwide.
Hence, it is crucial to understand how Cochrane reviews account
for potential variations in the distribution of harms and benefits,
especially for conditions in which large socioeconomic disparities
are documented, such as chronic obstructive pulmonary disease
(COPD). Objectives: To assess to what extent Cochrane reviews
for COPD describe individual study baseline data relevant to equity and
incorporate these factors into the effectiveness analysis. Methods:
Electronic searches of The Cochrane Library (Issue 4, 2011) were
conducted to identify reviews of effectiveness in COPD. Baseline and
study outcome data reported in the reviews were classified using
the PROGRESS-Plus framework, which describes factors relevant to
assessing health inequities. Results: Forty-three Cochrane COPD
reviews were identified (26 pharmacological; 17 non-pharmacological).
Eighty-four percent provided individual study baseline data by at
least one PROGRESS-Plus factor: age (81.3%), sex (72%), place of
residence, occupation, marital status and social capital (2.3% each;
all in non-pharmacological reviews). Pharmacological reviews were
not more likely to report equity-relevant baseline data compared to
non-pharmacological reviews (odds ratio 0.2 (95% confidence interval
0.034 to 1.18)). Other equity factors (education and social exclusion)
were described in the discussion of three non-pharmacological reviews
but not analyzed. Three pharmacological reviews planned effectiveness
subgroup analysis by age; however, none of the reviews incorporated
any of the PROGRESS-Plus factors in the analysis of effectiveness.
Conclusions: Equity is seldom addressed in Cochrane COPD reviews
of effectiveness. While the characteristics of age and sex are often used
to describe study participants, these and other potential equity factors
are consistently excluded from the analysis of therapeutic effectiveness.
There is an urgent need to better synthesize the evidence of how
the effectiveness of chronic disease treatments may vary according to
socioeconomic disparities, especially in COPD.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
35
C2O2
Synthesizing health systems evidence:
The contribution of a network based
on North-South collaboration
Koehlmoos TP1 , Pantoja T2 , Kiwanuka S3 , Meng Q4 , Oxman A5 , Oliver S6 ,
Garner P7 , Bennett S8 , Ransom MK9
1 ICDDR, B, Bangladesh; 2 Escuela de Medicina, Pontificia Universidad
Católica de Chile; 3 School of Public Health, Makerere University, Uganda;
4 Center for Health Management and Policy, Shandong University, China;
5 EPOC satellite for low and middle income countries, Norwegian Knowledge
Centre for the Health Services; 6 EPPI-Centre, Institute of Education, UK;
7 Liverpool School of Tropical Medicine, UK; 8 Bloomberg School of Public
Health, Johns Hopkins University, USA; 9 Alliance for Health Policy &
Systems Research, World Health Organization, Switzerland
Background: In 2007, a dearth of systematic reviews focused
on, and review teams who were capable of, synthesizing health
systems evidence to improve health and decision making in lowand middle-income countries (LMICs) led the Alliance for Health
Policy & Systems Research (HPSR) to create a network of Centres
for Systematic Review (CSRs) focusing on health finance, human
resources, the non-state sector, and the advancement of HPSR synthesis
methodology. Methods: After competitive bids, CSRs were established
in Bangladesh, China, Chile and Uganda. Support for capacity building
was provided by the Cochrane Effective Practice and Organization of
Care Group, the EPPI-Centre and the Cochrane Infectious Diseases
Group. The CSRs participated in two workshops, multiple individual
training activities, and long-distance mentoring. Results: In four years
the network produced more than twelve systematic reviews on priority
topics including Cochrane reviews on dual practice, social franchising,
and community health insurance, and more than ten presentations
at high-profile international fora. Teams identified methodological
challenges in the field and collected resources to address them. CSR
members contribute to the global review community by serving as
editors within the Cochrane and Campbell Collaborations and on The
Cochrane Library Oversight Committee. They facilitate capacity building
and extensive engagement with policy makers in their home countries
and regions. The Pan-American Health Organization (PAHO) recognized
one CSR-produced Cochrane protocol on emigration policy as the most
policy-relevant protocol for the Health Agenda for the Americas in 2010.
Conclusions: LMIC-based review teams mastered an array of review
skills but more importantly pioneered new methodology, expanded
capacity building networks, facilitated knowledge translation within
their settings and contributed actively to the global synthesis movement.
Maintaining this capacity requires sustainable funding for individuals
and organizations but also investment in information, communication
and technology to support partnerships and rapid access to synthesized
evidence.
C2O3
Do we need a PRISMA-Equity extension for
systematic reviews?
Welch V1 , Tugwell P1 , Petticrew M2 , Moher D1 , Ueffing E1 , Pardo J1
1 University of Ottawa, Canada; 2 London School of Hygiene & Tropical
Medicine, UK
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Equity-focused reviews are increasingly promoted and
published by groups such as the Cochrane Public Health Review Group,
the Cochrane Effective Practice and Organization of Care Review
Group, the HIV/AIDS Review Group and the Campbell International
Development Group. These reviews may have specific methods,
participants or settings that need to be reported transparently which
are not included in the existing, internationally-recognized reporting
guidelines for systematic reviews PRISMA(Preferred Reporting Items
for Systematic Reviews and Meta-analyses). Objectives: To assess
whether equity-focused reviews require additional or modified items in
the PRISMA reporting guidelines. Methods: We conducted a Cochrane
methodology review and a methodology study and we assessed three
equity-focused systematic reviews using PRISMA. Results: We found
that equity analyses and applicability judgments were reported in
insufficient detail to judge their credibility in our methodology study
and Cochrane methodology review. We found 11 areas where PRISMA
items needed to be modified or added to reflect special methods
and concerns in equity-focused systematic reviews. These items
related to the definition of health equity and rationale for assessing
it, choice of study designs, search strategies, process evaluation,
analysis of effects on equity in health status and judgments about
applicability and generalizability. Conclusions: There is a need
for modified PRISMA reporting guidelines for equity-focused reviews
to reflect specific concerns and methods used. The Campbell and
Cochrane Equity Methods Group is leading the development of
an Equity-Extension to PRISMA, and invites Cochrane members to
participate in a Delphi process and online survey to select and prioritize
items.
C2O4
What’s the story?: The Cochrane Library
publicity, global media coverage and author
experience
Pentesco-Gilbert D1 , Stewart G1 , Beal J1
1
Wiley-Blackwell, John Wiley & Sons, Chichester, UK
Background: Reviews from The Cochrane Library, published in
the Cochrane Database of Systematic Reviews are often reported by
newspaper and broadcast journalists, and bloggers. Press releases
highlighting selected new and updated reviews are an important part
of the publication process and marketing strategy for The Cochrane
Library. Increasingly reviews are used as referential sources by media
to help confirm what is known about a topic. Authors and editors
of Cochrane reviews are involved in the authoring of the releases
and media coverage. Objectives: The objective of Wiley–Blackwell’s
publicity strategy is to disseminate the findings of Cochrane reviews
and raise the international profile of The Cochrane Library through
professional and consumer media, and to increase usage. Methods:
A number of new and updated reviews are selected for publicity
activity from the monthly issue publication of the Cochrane Database
of Systematic Reviews. By analysing the resulting coverage, including
a regional breakdown of where different reviews are reported, we
can provide an interesting snapshot of the issues of interest to the
international community. We also look for a corresponding increase
in usage and citations for the articles that are most widely reported.
Author feedback on their media experiences is collected post publication
and will be summarized to identify opportunities to improve the quality
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
36
of media stories and author satisfaction. Results: An up-to- date
analysis of international media coverage for the period January 2010
to June 2011 will be presented. Conclusions: Media coverage of
new and updated Cochrane reviews raises the international profile
of The Cochrane Library in both professional and consumer media,
and increases usage of Cochrane reviews. A review of the impact of
monthly publication (introduced in January 2010) and increased use of
social medial tools together with author experiences will help to inform
future publicity approaches.
C3 Consumer issues and shared
decision making
C3O1
Consumer participation in The Cochrane
Collaboration – comparison of two models
Gyte G1 , Horey D2
1 Consumer Editor, Pregnancy & Childbirth Group, UK; 2 Editor, Consumers
and Communication Group, Australia
Background: Consumers have been integral to The Cochrane
Collaboration since its inception in 1993, making the organisation
a leader in consumer involvement across the world. However, we still
do not know how to assess the contribution of consumers, or know
how to get their input in the most feasible and acceptable way. It is
likely that different models will work better for different entities. With
the Collaboration looking to review its standards, it is important to
ensure that involving consumers is done effectively so that consumer
input is not just a token gesture. One Cochrane Review Group operated
a complex model of consumer participation for ten years (from 1998 to
2007), overseen by a Consumer Panel Coordinator (a funded part-time
post) and aided by two volunteer co-ordinators in USA and Australia.
This model was evaluated in 2003 using interviews with editors,
consumers and review authors. When funding to the Review Group
was reduced in 2007, the Consumer Panel model ceased, and the
Group embarked on an alternative model of involving consumers using
staff at the editorial base (2008 to present). Objectives: To look
at the advantages and disadvantages of the two models of consumer
involvement used by the Cochrane Review Group between 1998 and
2011. Methods: Interviews will be conducted with editors and
consumers with experience of both models of consumer input. Findings
will be compared with those from the earlier evaluation. Conclusions:
The lessons learned from both models of consumer participation will
be presented. Ideas for other consumer involvement models and
possible outcome measures will be proposed which may help the
Collaboration continue to lead the world in involving consumers in
health research.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
C3O2
Spreading Cochrane press releases to lay people
and Italian journalists
Colombo C1 , Pecoraro V2 , Banzi R1 , Moja L3 , Cima S4 , Mosconi P1 ,
Liberati A5 , Pistotti V2
1 Mario Negri Institute for Pharmacological Research, Milan Italy; 2 Italian
Cochrane Centre, Mario Negri Institute for Pharmacological Research, Milan
Italy; 3 University of Milan, Italy; 4 Zadig, Scientific Publishing Company,
Italy; 5 University of Modena and Reggio Emilia, Italy and Italian Cochrane
Centre, Mario Negri Institute for Pharmacological Research, Milan Italy
Background: Since 2003, the Mario Negri Institute, the Italian
Cochrane Centre (ICC) and the scientific publishing company Zadig,
have developed ‘PartecipaSalute’, a national program to foster
partnerships among patient organizations and healthcare professionals.
The Italian translation of several Cochrane Reviews’ abstracts and plain
language summaries are hosted in the PartecipaSalute website, with the
objective to reduce users’ distance from evidence-based information.
The Cochrane press releases are appealing for a general audience,
and therefore could represent a preferred tool for dissemination.
Objectives: To spread the Cochrane press releases to lay people and
journalists and explore their popularity in Italy. Methods: Since 2007,
the ICC receives Cochrane press releases from the Publisher Wiley a
few days before the release of a new issue of The Cochrane Library.
These are translated and adapted to the Italian setting, by researchers
working on methodology and communication aspects. The press
releases are then sent to a mailing list of 284 journalists and published
on the PartecipaSalute portal. The access to this website section was
monitored from 2007 to 2011 to explore the popularity among lay
people visiting the portal. Monitoring of press coverage in a sample of
journals is ongoing. Results: One hundred and ten press releases have
been published and mailed to the journalists from January 2007 to
April 2011. The median website visits per month were more than 28.6
(range 17.617 to 39.616). We will provide data on the Cochrane press
releases section and about press coverage. Conclusions: Preliminary
data show that the press coverage of Cochrane press releases is
sensitive to hot media topics (eg. vaccines, zinc for the common cold,
acupuncture). The impact of the press releases will be evaluated.
C3O3
Involving young people in all stages
of a systematic review
Liabo K1
Science Research Unit, Institute of Education, University of London,
UK
1 Social
Background: A group of young people with experience of being in
foster or residential care were invited to be involved in setting the
question for a systematic review on a health-related topic, and to
participate in all stages of the research. Objectives: To present the
review and describe how the young people were involved in all stages
of the research. To consider the negative and positive impact of their
involvement, on the review and on the researcher. To discuss how
consumer involvement may be optimised without compromising on the
review quality. Methods: The systematic review followed standard
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
37
systematic review methods, but most of the decisions were made in
collaboration with the young people. The young people were given
some research training, and decisions were made using closed and
open voting, following group discussions. The protocol was developed
from a template, containing open and closed questions. The researcher
recorded all meetings and analysed the interaction and decisions
made, using thematic analysis. Results: The young people’s decision
to address the effectiveness of educational support reflects a wider
view of what constitutes health. They were successfully involved in
developing the protocol, contributing to the searching and screening of
24% of all electronic hits. Their involvement in the critical appraisal of
included studies and the synthesis was limited. Conclusions: Young
people can be involved in most stages of a systematic review, but such
participation requires the researcher to think of creative methods for
presenting what a systematic review is, and explaining some of the key
principles of reviewing. Some young people preferred to provide advice
on the framework of the review, whereas others took part in research
activities.
C3O4
Do Cochrane systematic reviews report
patient-important outcomes?
Howell M1 , Tong A1 , Wong G2 , Howard K1 , Webster A1 , Craig J1
1
University of Sydney, Australia; 2 Westmead Hospital, Australia
Background: Despite the proliferation of trial evidence, uncertainty
remains about the effect of interventions on outcomes that are
important and relevant to patients. Many studies report endpoints
that have little meaning for patients. Objectives: To assess
the concordance between the prioritised outcomes associated with
immunosuppressant medications among kidney transplant recipients
with outcomes reported in Cochrane systematic reviews. Methods:
Identification and ranking of outcomes important to kidney transplant
recipients was undertaken using the nominal group technique. Each
group, which involved 6 to 10 participants, developed and ranked
in order of importance a list of outcomes considered relevant
to immunosuppressant medications. For the participant rankings,
a maximum priority score was calculated and data presented as
a percentage of the maximum possible priority score. The top
15 patient-important outcomes were compared with outcomes reported
in Cochrane systematic reviews on immunosuppressive regimens in
kidney transplantation. Results: Across the eight groups, we
identified 48 outcomes considered relevant to kidney transplant
recipients. The top ranked 15 were: graft loss, kidney function,
damage to other organs, death, all cancers, diabetes, skin cancer,
cardiovascular disease, susceptibility to infection, night tremors, weight
gain, bone disease, impact on family, depression, and gastrointestinal
problems. Across the five Cochrane systematic reviews, 5 (33%)
to 9 (60%) of the top 15 patient ranked outcomes were reported.
No reviews included damage to other organs, night tremors, impact
on family and depression outcomes. Only one review addressed
the following outcomes: skin cancer, cardiovascular disease, weight
gain, bone disease and gastrointestinal problems. Conclusions:
Patient-important outcomes following kidney transplantation include
serious and fatal consequences as well as emotional and physical side
effects. Strategies to integrate patient priorities for immunosuppressive
treatment outcomes in clinical trials and systematic reviews are needed.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
C4 Editorial processes and supporting
review authors
C4O1
National Institute of Health Research (NIHR)
Cochrane Collaboration Programme Grant
Scheme – Gynaecological Cancer Review Group
Williams C1 , Morrison J1 , Bryant A1 , Quinn G1 , Jess C1 , Hayes J1
1 Cochrane Gynaecological Cancer Review Group, UK
Background: The Cochrane Gynaecological Cancer Review Group
(CGCRG) was awarded an NIHR Cochrane Collaboration Programme
Grant in September 2007. The Group decided to use the funds to
help support authors preparing systematic reviews on topics of high
priority to NHS patients which would also inform the gynaecological
oncology community on an international level. The NHS in the UK has
made major improvements in gynaecological cancer care and it is one
of the most integrated and multidisciplinary areas in medicine with
an active Improving Outcome Guidance Programme. This is important
as there are seven tumour sites with multiple complex care pathways.
The CGCRG is a highly collaborative multidisciplinary group with an
excellent track record in developing teams to prepare reviews. The
applicants of the award scheme all have strong links to all major
professional and consumer organisations involved in gynaecological
cancer. Objectives: Improvement of patient care by preparing
systematic reviews, which inform national and international guidelines.
Methods: After a major prioritisation project, titles were allocated to
UK sub-speciality trainees who expressed an interest and demonstrated
ability in conducting systematic reviews. The authors worked with
experienced editors as well as with a dedicated methodologist who
was employed throughout the grant period. All review authors received
training and mentoring and were encouraged to meet regularly with
their appointed editor and methodologist to progress the review
throughout the entire process. The editorial base provided support,
maintaining communication between the editors and authors, and
managed the process of statistical, methodological and external peer
review. Results: Thirty seven reviews and updates were produced
in this programme and are being translated by British Gynaecological
Cancer Society and Network Site Specialist Groups into guidelines and
protocols to improve patient care. The Group has subsequently been
awarded a second grant based on the success of this scheme.
C4O2
Testing selective responses of Cochrane groups
to the request of conducting a Cochrane
systematic review: A crossover randomized
controlled trial
Ciapponi A1 , Glujovsky D1 , Rey Ares L1 , Garcı́a Martı́ S1 , Reveiz L2
1 IECS, Institute for Clinical Effectiveness and Health Policy Buenos Aires.
Argentina; 2 Andean Cochrane Branch of the Ibero-American Cochrane
Center
Background: Thousands of people from around the world contribute
to The Cochrane Collaboration. Although there are a number of
steps that need to be followed to register a review, titles are
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
38
Table 1 (C4O2). Not-native English speaker request model.
CRG X
Dear Xxx Xxx
My name is Yyy Yyy and I am an Physician from OOO.
I have just completed a non-Cochrane review and am now interested
in performing a Cochrane Review. Specifically, I am interested in your
group’s topics. Would you please inform me if there are any available
topics or issues that I may assist you with?
Thank you for your time and attention. I look forward to hearing from
you soon.
Respectfully
Yyy Yyy
accepted at the discretion of each Cochrane Review Group (CRG)
and criteria for accepting those titles are not explicit [1]. Previous
studies found that authors reported having problems in getting titles
registered [2,3]. Objectives: To assess if the first language or country
location of ‘simulated newcomers’ influence the response patterns to
register titles in CRGs. Methods: Two hypotheses were addressed:
1) native English speakers (NESs) receive more positive responses
(+R) than non-NESs (NNESs); 2) responses will not differ by country
income (low income countries (LMC) versus High Income Countries
(HIC)). After ethics committee approval, we emailed substantially
identical versions of a title request (Table 1) to 53 CRGs written by:
‘NES-LIC’, ‘NES-HIC’, ‘NNES-LIC’ and ‘NNES-HIC’. In this crossover
RCT (one month of washout period), CRGs were randomized centrally
by computer software. The answers were independently classified
as ‘positive’ (offer review titles/updates, specific topics or mentors);
‘negative’ (explicit -R or requirements without help offer) or ‘neutral-R’
(the others) by two assessors blinded to interventions and e-mail
senders/recipients. Results: No response was obtained in 23/106
(21%) of emails. The patterns of responses were: 24% positive,
40% neutral and 36% negative. NES received significant more +R
than NNES (P = 0.031) as well as HIC than LIC (P = 0.031),
being worst for NNES-LIC and best for NES-HIC (Table 2; Figure 1).
Table 2 (C4O2). CRG’s response pattern by first language or location of ‘simulated newcomers’.
NNES/NES: Not Native/Native English speaker; LIC/HIC: Low/High Income Country
NNES
N (%)
NES
N (%)
LIC
N (%)
HIC
N (%)
Negative [20]
15 (36.59)
5 (11.90)
12 (25.00)
8 (22.86)
Neutral [33]
14 (34.15)
19 (45.24)
24 (50.00)
9 (25.71)
Positive [30]
12 (29.27)
18 (42.86)
12 (25.00)
18 (51.43)
Total [83]
41 (100.00)
42 (100.00)
48 (100.00)
35 (100.00)
Responses [N]
Pearson chi 2 = 0.031
Postivie vs. Negative/No Reply
Pearson chi 2 = 0.031
LIC: RR 3 (95% CI 0.98–9.20)
HIC: RR 1.29 (95% CI 0.66–2.53)
Figure 1 (C402). CRG’s response pattern by first language or location of ‘simulated newcomers’.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
39
Besides no response, frequent barriers included requesting curriculum
vitae and newcomer’s publication list. Some CRGs had very positive,
warm and collaborative responses. Conclusions: Although CRGs
should focus on potential authors that will produce high quality and
timely protocols and reviews, criteria for registering titles should be
explicit and standardized. The Cochrane Collaboration should consider
consistent ways of supporting authors from NNES-LIC.
References
1. The Cochrane Collaboration. Registering Titles. Available at: http://
www.cochrane.org/cochrane-reviews/registering-titles. Access in April
2011.
2. Ciapponi A RL, Martı́-Carvajal A, Ortiz Z, Sanchez-Gomez LM,
Delgado-Ramirez MB, Manterola-Delgado CG, Martı́nez-Pecino F. How
difficult is to register a Cochrane title? An lberoamerican authors
sample. XIV Cochrane Colloquium Dublin, Ireland 2006: [P003].
3. D Gillies HM, K New, V Pennick, Z Fedorowicz, J Van Der Wouden,
J Oliver, R. Scholten, A Ciapponi, J Verbeek. A collaboration-wide
survey for Cochrane authors. XVI Cochrane Colloquium. Freiburg,
Germany 2008: [P04-33].
C4O3
Impact, accountability, and sustainability of the
Cochrane prioritization project – Eyes and Vision
Group experience
Li T1 , Dickersin K2
1 Cochrane Eyes and Vision Group US Project, Johns Hopkins Bloomberg
School of Public Health, US; 2 Cochrane Eyes and Vision Group US Project,
US Cochrane Center, Johns Hopkins Bloomberg School of Public Health,
US
Background: The Cochrane Eyes and Vision Group (CEVG) US Project
was funded by the Cochrane Collaboration Prioritization Fund from
2007 to 2009 to test a framework for prioritizing systematic reviews.
Objectives: To describe the impact, accountability, and sustainability
of the project. Methods: A summary of research activities, professional
collaborations, funding, and research opportunities. Results: We
identified priority review topics related to open angle glaucoma using
practice guidelines and a Delphi survey of clinicians. We searched for
and appraised the methodological quality of existing systematic reviews
for priority topics (Cochrane and non-Cochrane). For topics without
any high quality review, we sought approval from the CEVG editor
and posted the topics on the CEVG website. For priority topics with
non-Cochrane reviews, we encouraged authors to convert them into
Cochrane reviews as appropriate. We presented project findings at each
stage, at the Cochrane Colloquia (2008 to 2010) and other international
meetings. This project comprised one part of TL’s PhD thesis, and has
resulted in two publications. An additional manuscript is under revision.
We involved various stakeholders in the project, including CEVG and
its Advisory Board, the US Cochrane Center, local biostatisticians and
epidemiologists, guideline developers at the American Academy of
Ophthalmology, and clinicians and vision research scientists from the
American Glaucoma Society and Asian-Pacific region. Findings from this
project also provided the groundwork for an additional grant award to
our team from the National Eye Institute, to develop research priorities
related to diabetic retinopathy and primary angle-closure glaucoma.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Two Master’s students are pursing the newly-funded projects as their
thesis research. The American Optometric Association is considering
collaboration with us to update their guidelines on diabetic eye disease.
Conclusions: The impact of the Collaboration’s Prioritization initiative
has been profound for our group and our research area. We have
demonstrated accountability to the Collaboration and sustainability of
the framework.
C4O4
Capacity building in systematic reviews in South
Asia: a South-North collaboration to expand the
production and translation of synthesized
evidence
Koehlmoos TP1 , PIgott T2 , Thomson H3 , Ueffing E4 , McGowan J5 , Gazi R6 ,
Hossain S6 , Snilstveit B7 , Howard G8
1 Health & Family Planning Systems Programme, ICDDR,B, Bangladesh;
2
School of Education. Loyola University Chicago, USA; 3 Medical Research
Council, Social & Public Health Sciences Unit, UK; 4 Campbell and Cochrane
Equity Methods Group, Institute of Population Health, University of
Ottawa, Canada; 5 Centre for Global Health, Departments of Medicine
and Family Medicine, University of Ottawa, Canada; 6 Health Systems &
Infectious Diseases Division, ICDDR,B, Bangladesh; 7 International Initiative
for Impact Evaluation (3ie), UK; 8 South Asia Research Hub, Department for
International Development, India
Background: A substantial body of evidence exists to answer many
of the questions asked by policymakers and development partners
in low- and middle-income countries (LMICs). However, evidence
is often scattered, inaccessible and rarely presented in a form that
provides an indication of the quality of evidence. Systematic reviews
in all sectors have the potential to contribute to improving the
evidence base from which policies and interventions can be developed
and implemented. The principles that underpin the international
effort to prepare and use systematic reviews of controlled trials for
evaluating clinical practice, particularly in high income countries,
can be similarly applied to the challenges of poverty reduction and
development in LMICs. Objectives: To support researchers in
South Asia through the provision of capacity building in systematic
review techniques using training, workshops, online training and
support, and quality assurance. Methods: Health & Family Planning
Systems Programme (ICDDR,B) in collaboration with the Campbell
Collaboration, the Social and Public Health Sciences Unit of the Medical
Research Council UK (SPHSU MRC), the Campbell & Cochrane Equity
Methods Group and the International Initiative for Impact Evaluation
(3ie) using funding from the Department for International Development
(UK) will develop and deliver i) a programme of systematic review
training in South Asia, ii) guidelines for development partners on
formulating review questions, and iii) systems of remote partnership,
networking and support to regional review teams. Results: The
courses are being developed with a first run in June 2011. This section
will be updated with details of participants, their feedback and any
successful assistance/review protocols in October 2011. Success of
the project will be measured by the number of regional review teams
who receive support and bring in funding and/or complete reviews.
Conclusions: LMIC-based researchers will be equipped to undertake
systematic reviews to help inform decision making in different sectors of
development.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
41
Posters sessions
Searching and information retrieval
Day: 20 Oct (P1A1–P1A55)
P1A1
Is the retraction of journal articles in electronic journals
and databases consistent and timely? A case study
Wright K1 , McDaid C1
1 Centre for Reviews and Dissemination, UK
Background: It is essential that the processes used for literature
searching and selecting studies for inclusion reliably identify retracted
papers so they are not unwittingly included in systematic reviews. There
is, however, evidence that retracted papers continue to be cited without
reference to the retraction. In 2009 multiple papers authored by Dr Scott
S Reuben were retracted due to falsification of data. We used these
citations to investigate whether databases and journals recorded the
information on retraction appropriately. Objectives: To investigate
whether three bibliographic databases recorded notices of retraction
in a clear and timely way and whether electronic journals displayed
information about retractions in a way that prevented inadvertent use of
fraudulent research data. Methods: We searched MEDLINE, EMBASE
and CENTRAL databases for 18 papers by Reuben that were known to
have been formally retracted, based on retraction notices. We noted
whether the database recorded that each of the articles was retracted.
The 18 papers were published across nine journals. We retrieved one
paper from each of the nine electronic journals and assessed whether
it was possible to identify the study as being retracted. Results: In our
case study all of the retracted papers identified on MEDLINE had been
annotated appropriately as compared to 6% of EMBASE records and
80% of CENTRAL records. Similarly, there was a high level of variability
across journals in how clearly they annotated that a paper had been
retracted. Conclusions: Lack of consistency in how databases and
electronic journals record retractions and the time taken to record the
retractions make it difficult for researchers and information specialists
to identify retracted studies and extra care is needed to ensure that
they are not included in systematic reviews.
P1A2
Comprehensive searching for systematic reviews:
A comparison of database performance
Beyer F1 , Wright K1
1 Centre for Reviews and Dissemination, UK
Background: It is generally acknowledged that searching for studies to
populate a systematic review should be as comprehensive as resources
allow. Where choices have to be made between available databases,
little evidence-based guidance exists about how to prioritize resources
to search or when to stop searching. Objectives: This study aimed:
(i) to compare the individual contribution of databases searched to
identify studies meeting the inclusion criteria for a recently completed
systematic review; (ii) to identify the best combination of databases
to retrieve all relevant studies. Methods: A systematic review of
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
the non-pharmacological management of frozen shoulder was used
as a case study. The total number of studies included in the review
provided the reference standard. A spreadsheet was used to record
for each included study a) whether it was retrieved by the appropriate
search strategy and b) whether it was indexed on each database.
Yields and number needed to read (NNR) from different combinations
of databases were investigated. Results: The review included 27
randomized controlled trials (RCTs), one controlled clinical trial (CCT)
and two case series. Of the 28 controlled trials, 27 were indexed
in at least one of sixteen analysed databases. Recall for individual
databases ranged from 0% to 90% (median 40%), and precision
from 0.0% to 4.3% (median 1.2%). For the strategies used, the best
combination of resources to retrieve all studies was either CENTRAL,
EMBASE, Science Citation Index, and reference checking; or CENTRAL,
EMBASE, MEDLINE (including PREMEDLINE), and reference checking.
Conclusions: Of the databases searched, CENTRAL displayed the
best combination of recall and precision. In terms of combinations of
databases a minimum of three databases, including CENTRAL, plus
reference checking were required to retrieve all included studies.
P1A3
Threading the needle we found in the haystack: Identifying
ongoing trials with the Pan African Clinical Trials Registry
Abrams A1 , Pienaar E1
1 South African Cochrane Centre
Background: Cochrane review authors face difficulties identifying
ongoing trials in regions where trials registries do not exist. The Pan
African Clinical Trials Registry (www.pactr.org) provides a platform to
prospectively register all clinical trials conducted in Africa as the first
regional World Health Organization (WHO)-endorsed primary register in
the region. Objectives: 1) To categorise trial interventions registered
on www.pactr.org and map these to published Cochrane reviews or
protocols; 2) To demonstrate the value of searching www.pactr.org for
review authors. Methods: Registered trial details were downloaded
from www.pactr.org on 11 April 2011. Two independent investigators
extracted trial data, including disease and intervention details. We
searched the Cochrane Database of Systematic Reviews (CDSR) 2011,
Issue 3 for published reviews or protocols evaluating the interventions
reported in these trials. Results: Forty-six trials are registered on
www.pactr.org. Thirty are randomized controlled trials of efficacy. We
identified 20 trials evaluating treatments, eight evaluating prevention
interventions and two diagnostic trials. The 20 treatment trials
evaluated drugs for malaria (6), HIV (3), tuberculosis (2), anaesthesia
(1), depression (1), fertility (1), hyperkalaemia (1), fluids for surgical
irrigation (1), postpartum haemorrhage (1), an electronic decision
making tool (1), provision of economic incentives (1) and effects
of patient advocates (1) on adherence. The eight prevention trials
evaluated drugs for malaria (5), HIV (1), tuberculosis (1) and
a behavioural intervention for sexual risk-reduction in HIV-positive
adults (1). One trial assessed diagnostic algorithms and another, a TB
diagnostic test. Our search of CDSR identified thirteen reviews and two
protocols where data from twenty of the above ongoing trials would
be eligible for inclusion. No review reported these. Conclusions:
www.pactr.org is a free, useful tool for identifying ongoing African
trials to be included in updates of reviews, and for assessing topics
where reviews should be undertaken.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
42
P1A4
Filled Gaps? Data after a year of the Pan African
Clinical Trials Registry efforts to increase understanding
and awareness of the child-focused clinical trial landscape
Abrams A1
1
South African Cochrane Centre, South Africa
Background: Cochrane reviews of child-focused interventions often
find that trial data in children is insufficient to draw conclusions.
In 2009, the Pan African Clinical Trials Registry (www.pactr.org) in
partnership with the World Health Organization developed a Child
Strategy as part of a multi-regional initiative to increase child-focused
clinical trial activity and registration worldwide. www.pactr.org, based
at the South African Cochrane Centre, provides prospective registration
for all clinical trials conducted in Africa. Objectives: To describe trials
affecting child-health registered on www.pactr.org. Methods: We
downloaded details of trials registered on www.pactr.org on 11 April
2011. A single investigator extracted trial data including details of
disease, trial intention (prevention, treatment, etc.) and participant
ages. Results: 23 of 46 (50%) registered trials investigated topics that
affect child health. Of nine trials in pregnant women, five researched
malaria (three evaluated drug prevention; two treatment), one assessed
patient advocacy for preventing mother-to-child HIV transmission, and
three researched diseases other than HIV or malaria (one evaluated
anaesthesia for Caesarian Section, one post-partum hemorrhage
Figure 1 (P1A4). Disease-type Researched by Participant Age.
10
8
6
Other
4
TB
2
HIV/AIDS
0
Malaria
Pregnant
Women
Infants
Older
Children
Large age
range
Figure 2 (P1A4). Country location of trials including children.
Egypt
Mali
The Gambia
Guinea-Bissau
Nigeria
Ethiopia
Cameroon
Burkina Faso Benin
Uganda
Kenya
Gabon
Tanzania
Ghana
1 trial
2 trials
Zambia
Rwanda
Mozambique
3 trials
4 trials
5 trials
treatment, and one, progesterone for unexplained infertility). In
infants, two trials evaluated HIV vaccines. Of 9 trials in older
children, three assessed antiretrovirals in HIV-infected children, four
researched malaria (two evaluated drug treatment, one vaccine and
one, post malaria anemia treatment), one assessed Tuberculosis
treatment efficacy and another, the use of electronic decision making
tools. A trial promoting adherence via economic incentives for TB, one
analyzing malaria prevention, and one evaluating a meningitis vaccine
included children and adults. Compared to last year’s baseline data
there is a slight increase (from 48% to 50%) in the overall profile of
child-focused trials registered. Child-focused trial activity is present
in 19 African countries (an increase from last year’s 14; see map).
Conclusions: Continued efforts of the Child Strategy, to encourage
child-focused trial research and registration in the region, will ensure
www.pactr.org’s value as a research tool for those affiliated with the
Cochrane Child Health Field.
P1A5
From final search to publication of HTA reviews: Does
the time gap have an influence on the usefulness of their
results?
Parekh-Bhurke S1 , Cook A1 , Chatting D1 , Fairbank E1 , Clarke A2
1 NETSCC, University of Southampton; 2 University of Warwick
Background: Systematic reviews (SRs) are important building blocks
for clinical practice guidelines and health technology assessment (HTA)
reports. With a rapid growth in evidence, it becomes necessary
to update SRs regularly to maintain their importance in informing
healthcare policy and practice. There is very little guidance available
about when and how to update SRs. Moreover, the updating practices
of some organizations that commission or produce SRs are unclear.
Objectives: We examined the effect of the time gap between final
search and publication of HTA systematic reviews on the usefulness
of results and how these can be mitigated for HTA reviews to
ensure their up-to-date status. We also examined the consistency
in reporting of these reviews using the Preferred Reporting Items for
Systematic Reviews and Meta-Analyses (PRISMA) checklist. Methods:
We identified and included 14 HTA reviews published from 2005 to
2007. Reviews including at least one randomized trial and at least
one meta-analysis were considered. The date of publication, last
search date for each review and the date of publication of all the
included trials in the review was extracted by dual data extraction.
We estimated the probability of trials being missed during the period
between the date of last search performed by authors and the date
of publication of the review. Each review was also checked against
the PRISMA checklist. For reviews where significant numbers of trials
had been missed, we repeated the original review searches to identify
missed trials, and assessed their impact on results using cumulative
meta-analysis. Results: The study identified that there were several
reviews or meta-analyses included in a single report making it difficult
to report if updates were required. We will make recommendations for
conduct of systematic reviews by the HTA programme and others. Our
study will also contribute to understanding best practice for updating
systematic reviews.
Malawi
South Africa
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
43
Figure 1 (P1A6). Countries of first author.
P1A6
Bibliometric analysis of 100 most cited Cochrane
Systematic Reviews
Uthman O1 , Okwundu C2
1
University of Birmingham, UK; 2 Stellenbosch University, South Africa
Background: Systematic reviews can be very useful decision making
tools because they objectively summarize large amounts of information,
identify gaps in medical research and identify beneficial or harmful
interventions. The number of citations an article receives after its
publication reflects its impact on the scientific community. There have
been a few recent attempts to identify and analyze the ‘most cited
articles’ in various specialties. This is an update of a previous analysis of
the top cited Cochrane systematic reviews which we reported two years
ago. Objectives: To identify the 100 most cited Cochrane systematic
reviews (CSR). To assess whether or not citation classics exist in CSR
and to examine characteristics of frequently cited CSR. Methods: 100
most frequently cited CSR were identified using the database of the
Science Citation Index of the Institute for Scientific Information in April
2011. Results: The most cited CSR received 174 citations, and the
least-cited CSR received 33 citations. Six CSR were cited more than
100 times. These CSR were published between 2005 and 2009. The
100 originated from 19 countries (Figure 1), with the United Kingdom
contributing 31 articles, followed by Canada (n = 13), USA (n = 11)
and Australia (n = 10). Obstetrics and gynaecology, cancer, diabetes
mellitus, neuromuscular disease and smoking cessation were the main
topics covered. Considerable attention was also given to asthma,
dementia, chronic inflammatory disease, obesity and hypertension.
Conclusions: At present, only six CSR can be classified as a ‘citation
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
classic’ (with 100 or more citations). The United Kingdom, Canada,
and United States have produced the greatest number of highly
cited CSR.
P1A7
User-friendly search strategies for geriatric evidence
in Medline
van de Glind EMM1 , van Munster BC2 , Spijker R1 , Scholten RJ1 , Hooft L1
1 Dutch Cochrane Centre, Netherlands; 2 Academic Medical Center, Department
of Internal Medicine, Netherlands
Introduction: Geriatric patients need specific treatments provided
by doctors who choose these treatments based on existing evidence.
However, evidence on many geriatric topics is hard to find. We
aimed to create user-friendly search strategies with excellent operating
characteristics to find geriatric evidence in Medline quickly and easily.
Methods: We used a diagnostic test framework to create the search
strategies. We compared the search strategies’ retrieved articles with
a reference set as index test. This reference set consisted of 2255
articles from 23 biomedical journals in Medline. Each of these articles
was labeled as relevant, not relevant or possibly relevant for geriatric
medicine. From the relevant articles, we identified search terms to
compose different search strategies. We compared the retrieved result
of the different search strategies with the reference set to calculate the
sensitivity, specificity, precision, accuracy and number-needed-to read
(NNR). Results: The most sensitive search strategy had a sensitivity
of 92.0%, a specificity of 86.9%, a precision of 71.1%, an accuracy
of 88.2% and a NNR of 1.40. The most specific search strategy had
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
44
a specificity of 96.0%, a sensitivity of 69.6%, a precision of 86.1%,
an accuracy of 89.2% and a NNR of 1.16. Conclusion: Our geriatric
search strategies simplify searching for relevant literature and therefore
contribute to a better evidence-based practice. The search strategies
are useful to both the clinician who wishes a quick answer to a clinical
question and to the researcher who wants to find as many articles as
possible without missing too much relevant information.
P1A8
Testing search strategies for geriatric medicine in the
Cochrane Database of Systematic Reviews
van de Glind EMM1 , Young C2 , Spijker R1 , Scholten RJ1 , van Munster BC3 ,
Stott DJ2 , Hooft L1
1 Dutch Cochrane Centre, Netherlands; 2 University of Glasgow, UK;
3 Academic Medical Center, Netherlands
Background: Finding relevant information in databases is the
cornerstone of evidence-based medicine.
However, searching
databases can be challenging. Recently, sensitive and specific search
strategies were developed to find information in MEDLINE for geriatric
medicine quickly and easily. These search strategies proved to have
excellent sensitivity and specificity. Objective: To test the operating
characteristics of geriatric search strategies in the Cochrane Database
of Systematic Reviews (CDSR). Methods: A reference set was obtained
from the CDSR. One assessor labeled all 6437 reviews and protocols
published until March 2011 as either relevant to geriatric medicine or
not. We translated the search terms from MEDLINE into Cochrane.
Then we compared the reviews that were found with the search strategy
with the labeled reference set obtained from the CDSR. Results: Only
373 (6%) of the articles in the CDSR were considered relevant to
geriatric medicine. The most sensitive search strategy had a sensitivity
of 65%, a specificity of 76% and a number-needed-to-read (NNR)
of 7. The most specific search strategy had a sensitivity of 45%, a
specificity of 98% and a NNR of 1.4. Conclusion: A small percentage
of reviews in the CDSR contain information on the elderly. Compared to
the performance in MEDLINE, the sensitive search strategy performed
slightly worse in the CDSR. Although it found two thirds of the relevant
reviews, there were a lot of false positive findings resulting in a higher
NNR. The specific search strategy retrieved very few false positives.
However, half of the relevant reviews were missed. Differences in the
performance of the search strategies could be due to a lower prevalence
of relevant articles in the CDSR as well as the use of different criteria
for considering articles as relevant to geriatric medicine. By using the
CDSR as another reference set we have proven that the geriatric search
strategies are robust.
P1A9
Reconceptualising searching and screening: How new
technologies might change the way in which we identify
studies
Thomas J1 , O’Mara A1
1
EPPI-Centre, Institute of Education, UK
Background: Typical reviews deal with the ‘information explosion’
by narrowing their search for studies (e.g., applying search filters).
Relevant evidence can be missed through this approach. Current
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
methods to minimise the risk of missing relevant studies involve
searching broadly and screening potentially tens of thousands of
records, which is not always practical. Resource-efficient approaches
that maximise sensitivity are needed. Objective: To evaluate whether
new technologies allow us to search broadly without increasing the
screening workload through semi-automated screening approaches.
Specifically, we evaluate two types of text mining: a support vector
machine using active learning (Wallace et al., 2010) and TerMine term
clustering. Methods: Text mining techniques were employed in an
ongoing review to prioritise records for screening and to classify the
records automatically as includes or excludes. Screening prioritisation
was assessed by comparison with a ‘baseline inclusion rate’ and
through the novel application of power calculations. Classification
was assessed through the stability of the classifier and the calculation
of performance metrics (precision, recall, F-values). The classification
procedure was also evaluated using simulations of completed reviews.
Results: Screening prioritisation worked when sufficient information
was provided to the text mining tool; in the ongoing review, only
25% of all records were screened manually to identify the expected
total number of included studies. Classification reduced the manual
screening required in all reviews evaluated, although it worked better
for some datasets than others. Conclusions: Systematic reviews need
to develop ways of handling the growing amount of evidence available.
Text mining is a promising approach that shifts the emphasis of
identification from the searching stage to screening. Reconceptualising
searching permits broad searches to be conducted and allows reviewers
to be more precise in estimating the number of potentially missing
relevant studies than can be achieved by narrowing the search process.
Areas for further development are suggested.
P1A10
Sorting the wheat from the chaff: How does the Cochrane
HIV/AIDS Specialized Trials Register compare to searching
standard electronic databases?
Zani B1 , Oliver J1 , Siegfried N1
1 South African Cochrane Centre
Background: The Cochrane HIV/AIDS Specialized Register has been
housed at the South African Cochrane Centre since 2004. Records
from quarterly searches are coded manually and include a term for
randomized controlled trials (RCTs). In this study, we test the accuracy
of our Register in identifying RCTs for Cochrane reviews. Objectives:
1. Determine the precision and sensitivity of the Register compared to
electronic searches for identifying RCTs for Cochrane reviews; 2. Identify
limitations of the Register. Methods: We searched Archie to identify
Cochrane HIV/AIDS reviews published in 2010. We extracted relevant
keywords and used them to search our Register for RCTs relevant in
each review. We calculated precision by comparing the number of
RCTs identified with the number of records retrieved from the searches,
and sensitivity by comparing the number of relevant RCTs identified
with the number of RCTs included in each review. Results: We
identified 18 reviews, and have analyzed three to date. For one review,
four RCTs were identified from 103 records retrieved from standard
electronic databases (precision: 3.9%). Searching the Register found
all four RCTs from 10 records, and precision improved to 40%, with no
loss of sensitivity. However, for the other two reviews, with precision
increasing from 0.6% (13/2317) to 9.5% (8/84) when searching the
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
45
Table 1 (P1A10). The precision and the sensitivity of the Cochrane HIV/AIDS Specialized register for identifying randomized
controlled trials for inclusion in Cochrane Reviews.
Review
Database
Records retrieved from searches
Massage therapy review 2010
E-databases
The Register
Home-based care review 2010
E-databases
The Register
Oropharyngeal candidiasis review 2010
E-databases
The Register
103
10
2317
84
640
132
RCTs included in the reivew
4
4
13
13
36
36
Included RCTs identified from searches
4
4
13
8
34
32
Precision (%)
3.9
Sensitivity (%)
100
40
100
Register and from 5.6% (34/640) to 24.2% (32/132), with 38% (5/13)
and 6% (2/34) loss in sensitivity, respectively. Reasons for loss of
sensitivity included keywords not appearing in the title or abstract of
the Register record (n = 1); records coded as awaiting assessment
(n = 3); records not identified in the Register (n = 3). Conclusions:
Searching the Register for HIV/AIDS trials greatly increased precision
but some sensitivity was lost with reasons identified. The Register
is not yet sufficiently robust to be the sole database searched
and additional database searches are required to maintain review
quality.
P1A11
Finding public health evidence: How many stones need to
be turned?
0.6
100
9.5
5.6
24.2
61.5
94.4
88.9
authors retrieved the included studies. Results: We will provide data
on which resource retrieved each included study, if a study is unique
to a particular database and the proportion that were identified from
either Medline, Embase or the Cochrane Library. Conclusions: The
results will be useful for trial search coordinators in providing guidance
to authors and to authors themselves in planning their searches, and
will include guidance on a ‘core set’ of databases to search for public
health topics.
P1A12
Are the searches of network meta-analysis comprehensive
or well-reported?
Morgan H1 , Bauschmann M2 , Weightman A1
1 Cochrane Public Health Group, UK; 2 Cardiff University, UK
Tian J1 , Yang K1 , Liu A2 , Li L1
1 Evidence Based Medicine Center of Lanzhou University, School of Basic
Medical Science of Lanzhou University, Lanzhou 730000, China; 2 The
Health School of Gansu Province, Gansu, China
Background: Identifying studies that are relevant to the public
health field can be challenging. The public health field is broad,
and interventions can be complex involving many varied settings,
communities and populations. Also there is a lack of uniform
terminology to describe and organise the field. Controlled trials are often
not suitable for evaluating public health interventions and therefore
controlled before and after and interrupted time series designs are
frequently utilised. The combination of these elements can require the
use of more complex search methods such as searching a broad number
of databases that are relevant to the particular public health topic.
Objectives: This current project follows on from a well received poster
that was presented at the 18th Cochrane Colloquium (Mann M et al.
Searching for success for a public health intervention review). Mann
et al. investigated the need to search multiple databases to identify
evidence for their review of collaboration between local health and
government agencies, concluding that it was an important component
of the review process. We will expand the investigation to 12 Cochrane
reviews relevant to public health to further analyse if searching multiple
databases is essential in conducting a public health review. Methods:
From the 12 reviews we will identify which databases searched by the
Background: Network meta-analyses (such as mixed treatment
comparisons meta-analyses, multiple-treatments meta-analyses) could
combine information from all randomized controlled trials (RCTs) when
many treatment regimens already exist, to rank their benefits (and
harms) to choose the best option for patients. Therefore finding
all available RCTs is very important. Objectives: To evaluate
whether comprehensive searches were well conducted or reported
in network meta-analyses. Methods: We searched Pubmed, The
Cochrane Library, Embase, ISI Web of Knowledge, and Google Scholar
using ‘Network meta analysis’ OR ‘mixed treatment comparisons
meta analyses’ OR ‘multiple treatments meta analysis’ in title/abstract
without language, time, publication type restrictions. The first search
was conducted on 2010 October 21, and updated on 2011 February 21.
We included any network meta-analysis, regardless of the conditions or
interventions. Letters or abstracts were excluded. Results: We found
37 network meta-analyses, of which six included RCTs from relevant
systematic reviews. For the remaining 31 network meta-analyses,
the median number of databases searched was 3 (IQR 2-4, range
1–12) and 58.1% searched three databases or more. 37.8% did
nothing beyond searching databases. Only 12.9% included the full
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
46
electronic search strategy for at least one database. 41.9% did not
present search terms, 12.9% did not report when they conducted
their searches. Conclusions: Our study results demonstrate that
the searches in network meta-analyses are not well conducted, which
may lead to biased results, and this needs to be improved in the
future.
P1A13
What is the role of public trials registers in Cochrane
Systematic Reviews?
Blackhall K1 , Glanville J2 , Ker K1
1 London School of Hygiene & Tropical Medicine, UK; 2 Universtiy of York,
UK
Background: All member journals of the International Committee of
Medical Journal Editors (ICMJE) now require that clinical trials must
be registered in a public trials register before they are considered for
publication. Such registration ensures that trial details are available
to all researchers, including systematic reviewers, irrespective of the
results, which may encourage improvements in trial methods and
minimise publication bias. The introduction of the ICMJE requirement
in July 2005 is likely to have led to an increased number of trials being
registered and there is interest in the extent to which such registers
can now be relied upon as a source of trials for Cochrane systematic
reviews. Objectives: To explore the current state of trial registration
and to assess whether it is adequate to search for trials, for inclusion
in systematic reviews, using only public trials registers. To assess the
number and type of trials which would be excluded if unregistered trials
were not eligible for inclusion in Cochrane Reviews. Methods: We
identified three recently updated Cochrane Injuries Group reviews with
a large number of included trials published since 2005. We examined
the proportion of the included trials which had been registered and
the trials register in which they appeared. We explored the yield
of searches in individual databases compared to searching via the
ICTRP search portal. Results: Results will be presented at the
Colloquium. Conclusions: Using this sample we have examined
whether extensive database searches were necessary, or whether
we might have found the included trials by searching trials registers
alone. In addition we will be able to report on search approaches
for trials registers and the number and type of trials which are
unregistered. It is recognised that results may not be extrapolated to all
reviews.
P1A14
A survey of searching CBM and LILACS databases
in Cochrane Systematic Reviews
Chen Y1 , Yao L2 , Wang Z2 , Liang H2 , Liang F2
1 Evidence-Based Medicine Center of Lanzhou University, China; 2 School
of Basic Medical Sciences of Lanzhou University, China
Background: Ideally, unbiased systematic reviews should always
search and include all relevant trials, independent of the language of
publication. The inclusion of studies published in languages other than
English could help avoid bias. The Cochrane Handbook recommends
systematic reviewers to search CBM (Chinese Biomedical Literature
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Database) and LILACS (Latin American and Caribbean Health Science)
besides MEDLINE, EMBASE and Cochrane Central Register of controlled
Trials (CENTRAL). Objectives: To investigate how many Cochrane
Systematic Reviews searched CBM and LILACS Database. Methods:
We searched the full text in the Cochrane Database of Systematic
Reviews (4594 reviews and 2008 protocols in Issue 4, 2011) using the
terms ‘CBM’, ‘Chinese Biomedical Literature Database’, ‘LILACS’, ‘Latin
American and Caribbean Health Science’. We then screened all included
studies. Results: 222 (3%) systematic reviews or protocols searched
CBM and 1353 (20%) systematic reviews or protocols searched LILACS.
Only 41 (0.6%) systematic reviews or protocols searched both CBM
and LILACS from the year 2002. Among these 41 systematic reviews,
28 (68%) have Chinese authors and 13 (32%) are about Traditional
Chinese Medicine. Conclusions: Although searching for trials in
CBM and LILACS has been demonstrated to improve the quality of
systematic reviews, less than 1% of published Cochrane systematic
reviews searched both of them. The number of Cochrane systematic
reviews that searched LILACS is 6-fold the number of Cochrane reviews
that searched CBM. We suggest more Cochrane systematic review
authors search non-English databases, such as CBM and LILACS.
P1A15
The sensitivity and precision of adverse effects search
filters in MEDLINE, EMBASE and Science Citation Index
(SCI)
Golder S1 , Loke Y1
1 Adverse Effects Methods Group, UK
Background: Systematic reviews should use search strategies which
aim to identify as many relevant papers as possible. However, searching
for information on adverse effects is challenging, not least because
adverse effects terms (either specific such as ‘venous thrombolism’or
‘weight gain’ or generic such as ‘adverse event’ or ‘side effect’) may
not be included in the title, abstract or indexing of bibliographic
records in databases such as MEDLINE and EMBASE. Objectives: To
assess the feasibility of using adverse effects terms when searching
electronic databases to retrieve papers that report adverse effects data.
Methods: 242 included studies from 26 systematic reviews on adverse
effects were analysed to ascertain whether the corresponding records
in MEDLINE, EMBASE and Science Citation Index (SCI) included adverse
effects terms in the title, abstract or indexing. In addition published
adverse effects search filters devised for MEDLINE and EMBASE were
tested to assess how many studies would have been missed had these
filters been applied. Results: Records in EMBASE (89%) were more
likely to contain adverse effects terms in the title, abstract or indexing,
than MEDLINE (80%) or Science Citation Index (SCI) (70%). The
percentage of papers which would be missed with a combined search
using adverse effects terms in MEDLINE and EMBASE was 8%. This
figure is much lower than the 23% identified by a previous study
in 2001. The sensitivity of published adverse effects search filters
varied in MEDLINE from 1% to 93% and in EMBASE from 58% to
97%. Higher sensitivity was achieved when named adverse effects
were included in the search strategies. Conclusions: The proportion
of records that include adverse effects terms in the title, abstract or
indexing appears higher than in 2001. Although no adverse effects
search filters captured all the relevant records, high sensitivity could be
achieved, particularly in EMBASE.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
47
P1A16
The usefulness of different information sources for
retrieving adverse effects data for a systematic review
Golder S1 , Loke Y1
1 Adverse Effects Methods Group, UK
Background: Authors of systematic reviews tend to focus on searching
MEDLINE and reference checking for identifying information on adverse
effects. However, MEDLINE may not yield the most information on
adverse effects. Objectives: To determine the relative value and
contribution of searching different sources to identify adverse effects
data. Methods: A systematic review of thiazolidinedione-related
fractures in patients with type 2 diabetes mellitus was used as a case
study. Searches were conducted in over 20 different sources including
MEDLINE, EMBASE, Derwent Drug File, selected Internet sources and
reference texts. A record was made for each relevant reference noting
whether it was retrieved with a search for thiazolidinediones and
fractures and whether it was indexed/available in each source at the
time of searching. A record was made of the sensitivity, precision, and
number needed to read from searching each source and from different
combinations of sources. Results: The review included 58 relevant
references. The highest number of relevant references (35) were
retrieved from Science Citation Index (SCI), followed by BIOSIS Previews
(27) and EMBASE (23). The precision of the searches varied from 0.88%
(Scirus) to 41.67% (CENTRAL). With the search strategies used, the
minimum combination of sources required to retrieve all references
was: the GSK website, Science Citation Index (SCI), EMBASE, BIOSIS
Previews, British Library Direct, Medscape DrugInfo, handsearching
and reference checking, AHFS First, and Thomson Reuters Integrity
or Conference Papers Index (CPI). However, high numbers needed to
read were required for BIOSIS Previews and Medscape DrugInfo even
when searched after other sources with duplicate records removed.
Conclusions: In order to identify all the relevant references for this
case study a number of different sources needed to be searched. The
minimum combination of sources required to identify all the relevant
references did not include MEDLINE.
P1A17
A comparison of Embase systematic review filters
Martin S1 , McHugh J1
1 BMJ Publishing Group, UK
Background: The BMJ Evidence Centre has a transparent and
rigorous methodology. Our products are based on systematic searches
of Medline, Embase and the Cochrane Library for systematic reviews
and randomised controlled trials. Our challenge was to develop new
filters for Embase searches to increase their specificity, without losing a
great deal of sensitivity, in order to increase our productivity for a variety
of BMJ Evidence Centre products. Objectives: To compare the new
Embase systematic review filter used for BMJ Evidence Centre searches
against other published Embase systematic review filters. Methods:
We have done an extensive literature search for all other available filters
to use as comparisons and have based our methods on the relative
recall method as reported by Sampson M. et al. (2006) Results:
We will report specificity and sensitivity; highlight the strengths and
weaknesses of our research methods; discuss considerations and
outcomes and present our conclusions.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Reference
1. Sampson M et al. An alternative to the hand searching gold
standard: validating methodological search filters using relative recall.
BMC Research Methodology 2006 6:33 [http://www.biomedcentral.
com/1471-2288/6/33]
P1A18
A comparison of Embase randomised controlled trial filters
Martin S1 , McHugh J1
1 BMJ Publishing Group, UK
Background: The BMJ Evidence Centre has a transparent and
rigorous methodology. Our products are based on systematic searches
of Medline, Embase and the Cochrane Library for systematic reviews
and randomised controlled trials. Our challenge was to develop new
filters for Embase searches to increase their specificity, without losing
a great deal of sensitivity, in order to increase our productivity for a
variety of BMJ Evidence Centre products. Objectives: To compare
the new Embase randomised controlled trial filter used for BMJ
Evidence Centre searches against other published Embase randomised
controlled trial filters. Methods: We have done an extensive literature
search for all other available filters to use as comparisons and have
based our methods on the relative recall method as reported by
Sampson M. et al. (2006). Results: We will report specificity and
sensitivity; highlight the strengths and weaknesses of our research
methods; discuss considerations and outcomes and present our
conclusions.
Reference
1. Sampson M et al. An alternative to the hand searching gold
standard: validating methodological search filters using relative recall.
BMC Research Methodology 2006 6:33 [http://www.biomedcentral.
com/1471-2288/6/33]
P1A19
A comparison of two search methods for determining the
scope of systematic reviews
Forsetlund L1 , Kirkehei I1 , Harboe I1 , Odgaard-Jensen J1
1 Norwegian Knowledge Centre for the Health Services, Norway
Background: One of the core activities of the Norwegian Knowledge
Centre for the Health Services is the summary of research literature
into systematic reviews on request. A pilot search to assess the scope
of a new review is conducted before starting the reviewing process.
These searches are conducted in pre-specified databases. However,
new search facilities are sometimes launched, which challenge previous
search methods. Objectives: To develop a pragmatic approach for
systematically comparing two search methods for determining the
scope of a systematic review. The first method (Direct Search Method)
included performing direct searches in the Cochrane Database of
Systematic Reviews (CDSR), Database of Abstracts of Reviews of
Effects (DARE) and the Health Technology Assessments (HTA). Using
the comparison method (NHS Search Engine) we performed searches
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
48
via the search engine of the British National Health System, NHS
Evidence. Methods: We used an adapted cross-over design with a
random allocation of the 55 requests for systematic reviews. The main
analyses were based on repeated measurements adjusted for the order
in which the searches were conducted. Results: The Direct Search
Method generated on average fewer hits, 48% (95% CI 6% to 72%),
had a higher precision, 0.22 (95% CI 0.13 to 0.30), and more unique
hits than when searching by means of the NHS Search Engine, 50%
(95% CI 7% to 110%). On the other hand, the Direct Search Method
took longer, 14.58 minutes (95% CI 7.20 to 21.97), and was perceived
as somewhat less user-friendly than the NHS Search Engine, −0.60
(95% CI −1.11 to −0.09). Conclusions: Although the Direct Search
Method had some drawbacks such as being more time-consuming and
less user-friendly, it generated more unique hits than the NHS Search
Engine, retrieved on average fewer references and fewer irrelevant
results.
P1A20
Using a collaborative online platform to identify
systematic reviews in LILACS
Rada G1 , Ciapponi A2 , Glujovsky D2 , Rivera S1 , Roa M1 , Abdala V3
1 Evidence Based Health Care Program. Faculty of Medicine, Pontificia
Universidad Católica de Chile, Chile; 2 Argentine Cochrane Centre IECS,
Institute for Clinical Effectiveness and Health Policy, Argentina; 3 Servicios
Cooperativos de Información – BIREME/OPS/OMS, Brasil
Background: It is desirable that users have easy access to systematic
reviews (SR) instead of potentially biased narrative reviews (NR).
Methodological filters for databases are efficient tools for identifying
SR but NR and other articles are also retrieved. The user demands some
ability to discriminate between these, which is frustrating and prone to
mistakes. Considering the limitations of database-specific strategies to
perform this differentiation, we organized a collaborative web-based
model to identify SR in LILACS. Objectives: To identify SR in LILACS,
in order to facilitate their retrieval by users, including organisations
appraising and disseminating SR (e.g. DARE). Methods: A search
strategy developed for this project (based on Montori 2005 and the
Ecuador Cochrane Center strategy) was deployed from January 2008 to
February 2011. Study selection will be performed through EROS (Early
Review Organizing Software), a web-based software for the initial
phases of the SR process. Pairs of researchers will independently assess
by title/abstract considering for full text assessment all potentially
eligible SR (e.g. those mentioning at least a search in the methods
section). They will assess SR as detailed in Table 1 and will also
register if a meta-analysis is presented. Discrepancies will be solved by
consensus. The search strategy will be run quarterly followed by the
described process of SR identification. A filter and label of confirmed SR
will be incorporated into LILACS. Results: The search strategy retrieved
1819 references. Results about SR identification and classification will
be presented at the Colloquium. Once this project is in full operation,
the users of LILACS will easily access confirmed SR. This collaborative
model may be extended to randomized clinical trials and other study
designs.
P1A21
Searching the Chinese literature for public health
interventions: Can it add value to a systematic review?
Baker P1 , Sun J1 , Weightman A2 , Francis D3
1 School of Public Health Queensland University of Technology and
Epidemiology Central Regional Services Queensland Health, Australia;
2 Support Unit for Research Evidence (SURE), Information Services, Cardiff
University, UK; 3 Epidemiology Central Regional Services, Queensland
Health, Australia
Background: Identifying studies for inclusion in systematic reviews of
public health interventions requires a comprehensive search strategy,
usually including multiple databases. Increasingly, consideration has
also been given to identifying studies only published in the Chinese
Table 1 (P1A20). Definition of systematic reviews.
*DARE: ≥4 criteria out of the first 5 (1–3 are mandatory)
#
Oxman and Guyatt 1991: 1–4 & 6–7
# Cochrane Collaboration, CRD, MOOSE, Potsdam Consultation, QUOROM, AHRQ: 1–4 & 6–8
1.
2.
3.
4.
5.
Were inclusion/exclusion criteria reported?
Was the search adequate?
Were the included studies synthesized?
Was the validity of the included studies assessed?
Are there sufficient details about the individual included studies presented?
6. Was the data extraction process adequate?
7. Was the study selection process adequate?
8. Was ‘PICO’ used to focus the question(s)?
*DARE. (Accessed 04/08/2011, 2011, at http://www.crd.york.ac.uk/cms2web/AboutDare.asp.)
# Sander L, Kitcher H. Systematic and Other Reviews: Terms and Definitions Used by UK Organizations and Selected Databases. Systematic Review and
Del-phi Survey. In: National Institute for Health and Clinical Excellence. London; 2006.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
49
language and only cited in Chinese databases. Searching such
databases can be challenging for non-Chinese speaking authors as
it requires the use of Chinese characters. Objectives: To describe
the challenges, and assess the value of searching for studies of
interventions in Chinese databases based on the recent Cochrane
review ‘Community wide interventions for increasing physical activity’.
Methods: In the published protocol 24 databases were listed but these
did not include a specialist Chinese database. One expert advised that
relevant studies were undertaken in China. Subsequently five Chinese
databases were searched: CAJD, CCND, CPCD, CJSS, and CDFD through
http://www.global.cnki.net/. Searching, screening, translation and
data extraction were assisted by a Chinese epidemiologist. Results:
The search (January 1995 to December 2009) yielded 124 hits and
resulted in 4 included studies that were unique to these databases.
These represented 16% (4 of 25) of the review’s included studies.
Furthermore, the Chinese studies represented 33% (3 of 9) of the
most intensive studies included and 67% of studies from low-income
countries (4 of 6). The papers tended to be brief and required contact
with authors for clarification and confirmation of important information.
Conclusions: The additional searches of the six Chinese databases
provided further complexity to an already complex and comprehensive
search strategy. The search strategy was less sophisticated and was
limited by challenges in translation of search terms from English to
Chinese. Despite these challenges, the strategy added further studies
of value to the review which would not have been included otherwise.
We recommend that, where indicated, searching of Chinese databases
be undertaken.
P1A22
Hard-to-reach and difficult-to-define: searching for
‘hidden’ populations. An example from public health
Cooper C1 , Levay P2 , Lorenc T3 , Craig G4 , Marrero-Guillamón I5
1 Peninsula Technology Assessment Group (PenTAG), Peninsula College of
Medicine & Dentistry, University of Exeter, UK; 2 Information Specialist,
National Institute for Health and Clinical Excellence, UK; 3 Department of
Social and Environmental Health Research, London School of Hygiene and
Tropical Medicine, UK; 4 City University, London, UK; 5 Associate Research
Fellow, Department of Iberian and Latin American Studies, Centre for
Iberian and Latin American Visual Studies, Birkbeck, University of London,
UK
Background: Writing efficient search strategies for systematic reviews
requires defining, a priori, the topics of a review within a search. A
difficulty arises when a population or set of interventions is unclear or
unknown: how best to operationalise these points in the context of
the search? Objectives: This paper presents a search designed for
qualitative evidence on ‘hard-to-reach’ groups at risk of tuberculosis.
It explores the methods we used to construct a search filter for this
ill-defined population and whether the collaborative approach we
used was an effective method for defining difficult-to-define concepts.
Methods: Information specialists extensively tested potential terms
and themes in a variety of resources and drafted the filter through
database searching, citation chasing, expert/advisory contact, grey
literature searching and working collaboratively with the review team.
Hard-to-define populations were then analysed as a combination
of specific risk groups (e.g. prisoners) coupled with risk factors
(e.g. drug use) to ensure a highly sensitive population filter. The
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
original search combined three themes (TB + Population + Qualitative
terms). To test the efficiency of the filter, we re-ran our original
search in Medline without the population filter. The results of
the new search (TB + Qualitative terms) were screened to see if
any populations were retrieved which had not been defined in the
filter. Results: The new search yielded 5454 studies not located
in the initial search. No additional populations were identified after
screening the extra studies. The sensitive population filter appeared
to capture all relevant populations, suggesting that the time invested
in creating the search strengthened the conceptual understanding
of the population under review. Conclusions: Dialogue between
information specialists, review teams and topic experts is an effective
method for operationalising poorly defined concepts. A well defined
search filter benefits both the retrieval of evidence and the scope of the
review.
P1A23
Word frequency analysis of over a million words to support
the development of search strategies on ‘health-related
values’
Petrorva M1 , Sutcliffe P2 , Fulford B3 , Dale J2
1 Egenis (ESRC Centre for Genomics in Society), University of Exeter, Exeter,
UK and Health Sciences Research Institute, Warwick Medical School,
University of Warwick, Coventry, UK; 2 Health Sciences Research Institute,
Warwick Medical School, University of Warwick, Coventry, UK; 3 Institute
of Clinical Education, Warwick Medical School, University of Warwick,
Coventry, UK
Background: Systematic reviews increasingly incorporate or are
complemented with findings from research on ‘values’, understood
broadly to include ethical values, beliefs, preferences, experiences,
satisfaction, quality of life, etc. Research on such issues is scattered.
The vocabulary for and the boundaries of such topics are also notoriously
contentious. Objectives: This study used word frequency analysis
to 1) generate a broad pool of search terms, along with data on
their precision and sensitivity, to support systematic review searches;
and 2) develop a ‘brief values filter’ for scoping searches. Methods:
Datasets of MEDLINE records on Diabetes, Obesity, Dementia and
Schizophrenia were used (2004–2006; 4,440 citations; 1,110,291
words). Word frequency analysis was performed using Concordance
and SPSS. Text words and MeSH terms of high frequency and high
precision were compiled into a search filter. It was validated on
Dentistry and Food Hypersensitivity. Results: 144 unique text words
and 124 unique MeSH terms of moderate and high frequency (≥20)
and very high precision (≥90%) were identified. 19 text words and
7 MeSH terms had such excellent performance parameters across at
least three topics. These were compiled into a brief values filter.
In the derivation dataset, it had sensitivity of 76.8% and precision
of 86.8%. In the validation datasets, its sensitivity and precision
were, respectively, 70.1% and 63.6% (Food Hypersensitivity) and
47.1% and 82.6% (Dentistry). Conclusions: Both ‘health-related
values’ and word frequency analysis-based approaches to search filter
development are areas of substantial potential. A conceptualisation in
terms of health-related values may underpin a systematic and coherent
picture of the psychological, social, cultural, ethical and political factors
associated with a particular healthcare concern. Word frequency
analysis need not only precede search strategy development: the ideal
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
50
sample for it would comprise publications from completed reviews,
with findings supporting search strategy improvements for subsequent
reviews.
P1A24
Unique trial identification numbers are underreported
in biomedical publications
van de Wetering F1 , Haring T1 , Scholten R1 , Hooft L1
1 Dutch Cochrane Centre, Amsterdam, Netherlands
Background: To control for selective reporting, registering each study
at inception in a public register is advocated. After registration an
unique registration number is assigned to each study which enables
identification of future publications and increases transparency. The
International Committee of Medical Journal Editors (ICMJE) therefore
recommends reporting of the registration number at the end of the
abstract. Objective: To asses the reporting of unique trial identification
numbers in biomedical publications. Methods: We searched the
Netherlands Trial Register (NTR) for all randomized clinical trials (RCTs)
registered in 2005 that notified study completion by August 31, 2008.
We used the unique identification number to search MEDLINE for
publications. If no publication was found, the author’s name(s) or
elements of the title were used to identify possible publications or we
contacted the primary author. We also evaluated whether the journals
in which the studies were published were following the ICMJE’s
Uniform Requirements for Manuscripts (URM). Results: We included
542 RCTs. Of those, 282 (52%) were published, of which 164 (58%)
had reported the trial identification number (yet not necessarily at the
end of the abstract). Of the remaining 118 trials, 55 (47%) were
published in ICMJE journals. Conclusions: A substantial percentage
of registered trials did not report the trial registration number in the
abstract of their publication. Almost half of those were published in
ICMJE journals that ideally should follow their own recommendations.
Hence, there seems to be a need to promote accurate reporting of
trial identification numbers to enable easy identification of published
reports of registered studies. This could possibly be done by adding the
unique trial identification number as an item to the CONSORT checklist.
P1A25
Access to ongoing paediatric drug trials in the Netherlands
Hooft L1 , Weber E2 , Hamelinck V1 , Offringa M2
1 Dutch Cochrane Centre, Netherlands; 2 Academic Medical Center,
Netherlands
Background: As the number of paediatric clinical trials increases
it is essential that their existence is publicly documented and that
eventually their results will be made publicly available. Registering
clinical trials can bring us one step closer to an unbiased and efficient
clinical world. Objectives: To determine a) the accessibility of
ongoing paediatric trial data, b) how many retrospective clinical trial
registers include paediatric drug trials, and c) the ease of searching
for ongoing paediatric trials. Methods: We asked paediatricians and
Medical-Ethical Review Boards of all eight paediatric University Medical
Centres in the Netherlands to provide a list of all known paediatric drug
trials, which were planned or ongoing between September 2005 and
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
December 2008. Simultaneously, we identified Dutch paediatric drug
trials through the (inter)national prospective trial registers using the
WHO Search Portal of the International Clinical Trial Registry Platform
(ICTRP). Results: The field identified 172 paediatric drug trials, of
which 97 (56%) had been registered prospectively. A total of 299
Dutch paediatric drug trials were found searching the WHO Search
Portal. Most of these (70%) were registered in ClinicalTrials.gov, 25%
in the Netherlands Trial Register (NTR), and 5% in ISRCTN.org. The
WHO Search Portal facilitated searching for paediatric trials, resulting
in a very low number needed to screen (1.2). However, not all trials
(17%) registered in the NTR were found by using the WHO Search
Portal. Conclusions: There are disparities between the number of
ongoing trials, the completeness of registration and the sensitivity
of the current search tools. It seems that ongoing paediatric drug
trials are still not in the public domain because investigators and
industry declare that they don’t want the information to be publicly
available. Searching the WHO Portal for paediatric trials is easy,
although there is reason to believe that not all registered trials can be
found.
P1A26
A central repository for data extracted in systematic
reviews
Hadar N1 , Keefe S1 , Parkin C1 , Iovin R1 , Lau J1
1
Tufts Medical Center, USA
Background: Systematic reviews are labor-intensive and many overlap
in the literature reviewed. Objectives: To describe the challenges
involved in creating a repository for reusable extracted data. Methods:
Web-based electronic data extraction tool and repository. Results: The
availability of a freely accessible Web-based repository of systematic
review data will minimize the unnecessary replication of effort where
the details of the studies have already been previously extracted.
Additional benefits include: a promotion of transparency and reliability
in the systematic review process, enhanced cooperation and utility
across related resources, a more efficient means of producing and
updating systematic reviews, quick and simultaneous access to related
reviews, and improved access to detailed information on review
evidence by various stakeholders. The following requirements guided
our development of this system: 1) ease of use with a low burden
of contribution; 2) flexible and adaptable to all manners of research
questions, and 3) interoperable with other existing systematic review
databases. The repository’s Web interface would serve both as an
archive and data extraction tool. Technical challenges included
designing a Web-based system that balances ease of use with the
availability of additional features, like importing and exporting data
in different formats. Non-technical challenges primarily concerned
development of governance and data quality assurance processes. A
panel of technical experts knowledgeable in trial registries, databases,
systematic review methodologies, and/or bioinformatics was convened
to help address these issues. Pilot tests are underway to refine this
system. Conclusions: A number of technical and non-technical issues
need to be solved in implementing a data repository. The success
of such an archive requires close collaboration with stakeholders to
create a resource that researchers want to use and that encourages
participation.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
51
Figure 1 (P1A27). Cochrane Database utilization analysis.
P1A27
Cochrane Database utilization analysis:
A 10 year trend of one medical center in Taiwan
Chen KH1 , Chen C1 , Kuo KN2 , Tzeng PC1 , Chien LT1 , Liu HE3
1 Wan Fang Hospital, Taipei Medical University, Taiwan; 2 Taipei Medical
University, Taiwan; 3 Chang Gung University, Taiwan
Background: The Cochrane Database is one of the most valuable
medical library resources. Since 2001, Wan Fang Hospital Medical
Center in Taiwan started using the Cochrane Database. Objectives:
To analyze the utilization of the Cochrane Database in a medical
center in Taiwan and its correlation to EBM events. Methods:
1) We obtained the monthly log-in information for the Cochrane
Database from January 2001 to December 2010; 2) We collected all
major evidence-based medicine (EBM) events in Taiwan over the past
decade. These events were categorized as: (1) Nationwide events, (2)
Organizational regulation, and (3) EBM educational activities; 3. We
identified the top 10 utilization months and the EBM events held at the
time. Results: December is the annual top month in utilization of the
Cochrane Database. It may be related to the factor of organizational
regulation, as the annual evaluation in nurse promotion requires an
EBM related report (Figure 1). The top five months in utilization of
the Cochrane Database were all in 2010 when there were major EBM
activities, including: (1) Nationwide events including online publishing
of the traditional Chinese translation of all Cochrane abstracts by
National Health Research Institutes and EBM contests held by Taiwan
Joint Commission on Hospital Accreditation; (2) EBM educational
activities, such as Cochrane systematic review workshop. The regular
Cochrane Database training lectures and hospital accreditation did not
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
significantly correlate with the utilization of the Cochrane Database.
Conclusions: The key factors for increasing the utilization of the
Cochrane Database in a Taiwan medical center between 2001 and
2010 were: the annual nurse promotion evaluation, the publishing
of the traditional Chinese translation of Cochrane abstracts, EBM
contests, and Cochrane systematic review workshop.
P1A28
Locating evidence for developing countries: A case study
of three public health reviews
Stansfield C1
1 EPPI-Centre, Social Science Research Unit, Institute of Education,
University of London
Background: Identifying relevant research evidence to inform
decision-making in developing countries is not always straightforward.
The recent programme of reviews commissioned by the UK Department
for International Development offered an opportunity to evaluate
search strategies for such a purpose. Analysing where the research
studies they included were found is useful to inform search strategies
in future reviews. Objectives: To assess the utility of different search
sources in identifying research for three rapid systematic reviews that
concern public health interventions for women in developing countries.
The studies in these reviews used controlled trials and other study
designs. Methods: A retrospective analysis was undertaken of the
search source locations of the studies that were included in the three
reviews. Review management software was used to determine how
each study had been identified from the research literature, including
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
52
those found in multiple locations. Comparisons were made for the
search sources across the three reviews. Results: A total of 34 studies
were located across a range of international medical and social science
bibliographic databases, specialised registers, and other sources. Over
half, N = 19 (56%), were from one source only. For all reviews,
handsearching websites and contact with authors contributed over a
third of studies. Data is presented for each type of search source and
for controlled trials and those of other study designs, for each review
and collectively. Conclusions: Information resources in this area are
evolving. This analysis highlights challenges in identifying public health
research applicable to developing countries. For the three reviews it
was important to seek out regional and topic-specific sources, contact
other researchers and handsearch websites, in addition to searching
large international databases in the fields of medicine and social
science.
P1A29
‘Irrelevant’ or ‘neglected’? An analysis of RCTs in the
ARI Group
Thorning S1 , Van Driel M1 , Del Mar C1
1
Acute Respiratory Infections Group, Australia
Background: Acute respiratory infections (ARI) are commonly
managed problems in general practice. The Cochrane ARI group
supports 120 reviews covering a wide range of topics relevant to
primary care. Topics of Cochrane reviews are based on questions
relevant to clinical practice. The ARI reviews include RCTs that have
been indexed in the group’s specialized register in CENTRAL. It is
unclear if RCTs that have not been included in reviews are ‘irrelevant’
or simply ‘neglected’. Objectives: 1. Identify the proportion of RCTs
indexed in CENTRAL included in ARI systematic reviews. 2. Assess the
number of available RCTs that are ‘irrelevant’ (not useful for clinical
practice) or ‘neglected’ (relevant and not yet reviewed). Methods:
We selected 3 topics relevant to primary care within the scope of the
ARI group. We identified the relevant Cochrane reviews. Records were
retrieved from CENTRAL on each of the selected ARI topics. Each
record was checked for RCT characteristics and coded. A bibliometric
analysis of included RCTs and number of potentially useful RCTs will
be completed. Analysis by a panel of primary care clinicians of the
non-included RCTs will classify questions covered by the available RCTs
as ‘relevant’ or ‘irrelevant’. Results: Bibliographic analysis is currently
being conducted and results will be provided at the Colloquium in
October. Conclusions: Cochrane reviews cover clinically relevant
topics, but include only a small proportion of available RCTs. Cochrane
reviews could play a more proactive role in identifying topics to cover
in future clinical research and ensuring their clinical relevance.
produce recommendations on the use of new and existing medicines,
products and treatments in the NHS. The STA process is open
and transparent, and involves the submission of evidence by the
manufacturer or sponsor of the technology. NICE contracts independent
Evidence Review Groups (ERG) to appraise and assess the evidence
submission. As part of the STA process, the ERG must appraise and
critique the search methods which underpin the systematic review
and economic model components of the evidence submission. This
project describes how an ERG Information Team tested a standardised,
structured and reproducible approach to reviewing search methods
in manufacturer’s submissions (MS). Although an evidence based
checklist [1] exists for peer review of individual search strategies,
the ERG must appraise search strategies and the clarity of search
method reporting. Based on the methods described by McGowan [1],
Sampson [2,3] and NICE [4], we developed a tool for the assessment
of STA search methods. Objectives: To create and pilot a search
appraisal checklist designed specifically for evaluating industry evidence
submissions. Methods: Data were collected on types of errors
made in the study identification sections of two STA manufacturers’
submissions. This evidence was incorporated to create the ASSIST
checklist, which will be piloted on further STA’s for fine-tuning and
evaluation with an expected project conclusion in mid-July 2011.
The pilot checklist included 33 items in 10 domains. Results and
Conclusions: Preliminary findings during the pilot stage have shown
that implementation of the ASSIST checklist enabled standardised
appraisal of search methods. The ERG identified several shortcomings
in MS searches, including typographical errors, incorrectly combined
line numbers, inappropriate subject headings explosion and errors in
use of study design filters. The finalised checklist and evaluation will
be presented.
References
1. McGowan J, Sampson M, Lefebvre C. An evidence based checklist
for the peer review of electronic search strategies (PRESS EBC). Evidence
Based Library and information practice 2010;5(1):1–6.
2. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C.
An evidence-based practice guideline for the peer review of electronic
search strategies. J Clin Epidemiol 2009;62(9):944–52.
3. Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J. PRESS :
peer review of electronic search strategies. Ottawa: Canadian Agency
for Drugs and Technologies in Health (CADTH), 2008 Available from:
http://www/cadth.ca/index.php/en/hta/reports-publications/search/
publication/781
4. National Institute for Health and Clinical Excellence. Single Technology Appraisal: specification for manufacturer/sponsor submission of
evidence. London: NICE, October 2009, 2009. 76p.
P1A30
Appraisal of search strategies in industry submissions for
technology appraisal (ASSIST): Reviewing search methods
of industry submissions to NICE using a structured checklist
P1A31
Syncope and other adverse events associated with
cholinesterase inhibitors: Systematic review of the
literature versus analysis of the FDA Adverse Events
Database
Allen A1 , Misso K1 , Riemsma R1 , Kleijnen J1
1 Kleijnen Systematic Reviews Ltd, UK
López-Arrieta JM1
1 Cochrane Dementia & Cognitive Improvement Group, Spain
Background: The UK National Institute for Health and Clinical
Excellence (NICE) undertakes Single Technology Assessments (STA), to
Background: Cholinesterase inhibitors (CI) are commonly prescribed
to treat the symptoms of Alzheimer’s disease (AD). An aging population
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
53
will increase the number of CI users, who are more prone to the
adverse effects (AE) of these drugs. Many of the AE are potentially
severe, such as syncope and its consequences. Until recently their
AE profile has received little attention. Objectives: 1) To compare
the efficiency of methods of identifying and confirming the specific
association of bradycardia and related serious consequences with
CI (BRSCI) in a systematic review (SR) of the biomedical literature
with that of the United States Food and Drug Administration (FDA)
spontaneous Adverse Event Reporting System (AERS) database; 2) To
determine which method could have detected the AE earlier; 3) To
determine whether differences exist in these AE between the various
CI. Methods: We performed a SR and descriptive analysis of primary,
secondary (reviews) and tertiary literature (critical summaries) of the
composite event of: bradycardia, bradyarrythmia, pacemaker insertion,
complete atrio-ventricular block and hip fracture, associated with
CI. We performed an analysis of the AERS database searching for
the composite event (AERSDACE), using the Bayesian ‘Multi Gamma
Poisson Shrinker’ method. Results: None of the clinical trials detected
a single BRSCI adverse event. 13 patients with BRSCI were identified
through case reports. Three large cohort studies yielded 831 individuals
with BRSCI. A total of 246 patients were identified that possessed
one or more of the defined composite AE through AERSDACE. A
statistically strong signal of disproportionate reporting was observed
by the Empirical Bayesian Geometric Mean, 7.07 (95%CI: 6.23–8.03),
detected at one year after approval. Conclusions: Publication of
case reports of CI’s AE are a less efficient form of detection and
epidemiological studies take many years to test a hypothesis. Analysis
of large spontaneous adverse event databases is the most efficient
primary source to identify non-common AE.
Table 1 (P1A31). Number of events constituting the
bradycardia-related composite event, with MedDRA (PT) codes.
MedDRA PT code
Event
Number of cases
10003673
Complete atrio-ventricular block
27
10049765
Bradyarrhythmia
10006093
Bradycardia
10040741
Sinus bradycardia
10016450
Femoral neck fracture
2
10016454
Femur fracture
3
10020100
Hip fracture
10007598
Cardiac pacemaker insertion
7
168
35
12
9
P1A32
Developing a search filter for identifying primary care
studies in general medical journals
Roberts NW1 , Gill PJ2 , Wang KY2 , Heneghan C2
1 Bodleian Health Care Libraries, University of Oxford, UK; 2 Department of
Primary Health Care, University of Oxford, UK
Figure 1 (P1A32). Flow chart of gold standard selection for primary care search filter.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
54
P1A33
The rate of publication of randomized controlled trials
is near-exponential (2.5 times increase/decade with >1.5
million computer listed trials). A new strategy is required
to fufil the Cochrane Challenge
Chow TKF1
Hill Hospital, Eastern Health, Australian & New Zealand College of
Anaesthetists, Monash University, Australia
1 Box
Background: The number of randomized controlled trials has
increased exponentially over the past 30 years. Although the Cochrane
Collaboration’s achievements have been exemplary, the Cochrane
Challenge cannot be fulfilled because we are bogged down by
unscientific methods of identifying quality randomized trials. This
prohibits the periodic update of systematic reviews. Objectives:
To seek consensus on a search strategy to fulfil the Cochrane
Challenge. Methods: The search strategies recommended in the
Cochrane Handbook are a combination of handsearching and electronic
search. The former is time consuming, expensive and impossible to
duplicate and validate the authors’ findings. The electronic search, the
Optimally Sensitive Search Strategy, is a 29-line algorithm shown
to be highly sensitive in identifying randomized trials but lacks
specificity. (Chow TKF et al. 2004) We present a simpler, single-line
algorithm: (double blind$ or random$).af. We applied this strategy
to a systematic review pertaining to post-thoracotomy pain (Joshi
GP et al. 2008) to demonstrate its ease of use, high sensitivity
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Figure 1 (P1A33). Number of RCTs 1980–2009.
120000
100000
80000
Number of RCTs
Background: Searching and identifying primary care relevant studies
in the literature is challenging. Although specialist journals exist,
relevant research is often published in general medical journals. To our
knowledge, search strategies for finding studies relevant to primary
care have yet to be developed and validated. Objectives: To develop
and validate a search filter for identifying studies of relevance to
primary care in Medline. Methods: We conducted a Medline search
of articles published in five core medical journals at five yearly intervals
(1998, 2003 and 2008). We excluded comments, editorials, letters
and news publication types. All articles were randomly allocated to
one of two sets: Reviewer A screened the titles and abstracts of
the first set, while Reviewer B screened the second set. In cases of
uncertainty the full-text was evaluated. The gold standard dataset
was randomly divided into a development set and validation set. A
textual analysis was conducted on the titles, abstracts and MeSH terms
of the development set to extract frequently occurring words. Test
search strategies were then developed and sensitivity/specificity data
were derived using the validation set. The search strategies were
further tested in Medline. Results: A total of 12,045 articles were
retrieved in the search. After exclusion of specified publication types the
title/abstract of 9,028 records were screened and a further 495 records
required full-text evaluation. The gold standard dataset included 371
articles (Figure 1). The sensitivity, specificity and precision of different
search strategies will be reported. Conclusions: Using textual analysis
we have developed a range of Medline search strategies to identify
articles relevant to primary care. However, inadequate description of
the clinical research setting, inconsistent use of terminology to describe
‘primary care’ and the overlap between primary care and the public
health or outpatient setting make the construction of a highly efficient
search filter highly challenging.
60000
40000
20000
0
1975
1980
1985
1990
1995
2000
2005
2010
2015
Year of Publication
Table 1 (P1A33). Applying Strategy (double blind$ or
random$).af to identify postthoracotomy pain relief
randomized trials compared to Joshi GP et al. in November
2008 and May 2011.
Searches
Results 2008
1 (double blind$ or random$).af
2 exp pain/ or exp analgesia/ or exp
analgesics/
3 (pain or analge$).af
4 1 and (2 or 3)
5 (thoracotomy or thoracic surgery).af
6 4 and 5
7 Remove duplicates from 6
Results 2011
1,050,252
1,250,315
1,469,054
1,640,096
783,298
152,708
107,370
1,176
866
1,096,662
207,555
146,020
1,593
1,096
Search 1 conducted on the 25th of November 2008: EMBASE 1980 to
2008 week 47 and Ovid MEDLINE 1950 to November Week 2 2008.
Search conducted 2 months after publication of Joshi GP et al. 2008
Search 2 conducted on the 6th of May 2011: EMBASE 1980 to 2011
Week 17 and Ovid MEDLINE 1948 to April Week 4 2011.
in identifying quality randomized trials, high specificity in excluding
non-randomised trials and its advantages to updating systematic
reviews in a timely fashion. Results: Applying (double blind$ or
random$).af to Ovid EMBASE/MEDLINE, over the past 30 years (1980
to 2009) demonstrated that the number of randomized trials increased
at a near-exponential rate (2.5 times per decade). At the presentation
of this paper, there will be over 1.5 million randomized trials listed.
Adding the strategy to a post-thoracotomy pain relief algorithm, we
demonstrated that comparable conclusions to Joshi’s results could have
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
55
been achieved by reviewing 866 articles. Furthermore, this review can
be updated currently (6 May 2011) by reviewing an additional 230
articles. Conclusions: (double blind$ or random$).af is reproducible,
highly sensitive and specific. If adopted universally, it can manage the
impending information overload and fulfil the Cochrane Challenge.
P1A34
Where and how to find data on safety: What do systematic
reviews of complementary therapies tell us?
Pilkington K1 , Boshnakova A2
1 University of Westminster, UK; 2 Bazian, UK
Background: Successfully identifying relevant data for systematic
reviews with a focus on safety may require retrieving information from
a wider range of sources than for ‘effectiveness’ systematic reviews.
Searching for safety data continues to prove a major challenge.
Objectives: To examine search methods used in systematic reviews
of safety and to investigate indexing. Methods: Systematic reviews
focusing on safety of complementary therapies and related interventions
were retrieved from comprehensive searches of major databases. Data
was extracted on search strategies, sources used and indexing in
major databases. Safety related search terms were compared against
index terms available on major databases. Data extraction by one
researcher using a pre-prepared template was checked for accuracy
by a second researcher. Results: Screening of 2563 records resulted
in 88 systematic reviews being identified. Information sources used
varied with the type of intervention being addressed. Comparison of
search terms with available index terms revealed additional potentially
relevant terms that could be used in constructing search strategies.
Seventy-nine reviews were indexed on PubMed, 84 on EMBASE, 21 on
CINAHL, 15 on AMED, 6 on PsycINFO, 2 on BNI and HMIC. The mean
number of generic safety-related indexing terms on PubMed records
was 2.6. For EMBASE the mean number was 4.8 with at least 61
unique terms being employed. Most frequently used indexing terms
and subheadings were adverse effects, side effects, drug interactions
and herb-drug interactions. Use of terms specifically referring to
safety varied across databases. Conclusions: Investigation of search
methods revealed the range of information sources used, a list of which
may prove a valuable resource for those planning to conduct systematic
reviews of safety. The findings also indicated that there is potential to
improve safety-related search strategies. Finally, an insight is provided
into indexing of and most effective terms for finding safety studies on
major databases.
P1A35
Coding clinical trials and systematic reviews in
The Cochrane Library using standard international schemas
Hawkins B1 , Ervin A1 , Dickersin K2
1 Cochrane Eyes and Vision Group, USA; 2 US Cochrane Center/Cochrane
Eyes and Vision Group, USA
Background: Registers and databases of reports of clinical trials
are valuable resources. However, their utility for retrieval is limited
by variations in terminology used by different medical specialties and
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
researchers. Coding them using standard international schemas may
improve access. Objectives: Our goal was to evaluate in a pilot
study coding of health conditions using the International Classification
of Disease (ICD-10) and interventions using the beta-test version
of the International Classification of Health Interventions (ICHI) of
systematic reviews and clinical trials in the Cochrane Eyes and Vision
(CEVG) register of clinical trials within the Cochrane Central Register
of Controlled Trials (CENTRAL). Methods: Two raters coded all 43
CEVG systematic reviews published in The Cochrane Library by March
1, 2007. After discussing disagreements and agreeing on coding
conventions, one rater coded a 10% random sample of approximately
10,000 entries in the CEVG register as of January 2007. Results:
Inter-rater agreement using the ICD-10 was 81% for the systematic
reviews. Of 981 trial reports in the random sample, 716 (73%) could be
coded using titles, and abstracts when available. ICHI was inadequate
for coding interventions and was supplemented by ad hoc codes for
outpatient procedures and drug classes. Conclusions: We found
ICD-10 to be a satisfactory schema for coding disorders of eyes and
vision. However, we are seeking an alternative to the ICHI, designed
primarily for hospital procedures; the majority of participants in trials in
eyes and vision are outpatients. Also, we are conducting a pilot study
to evaluate automated conversion of MeSH terms to ICD-10 codes.
P1A36
Improving the efficiency of searches in Cochrane
Systematic Reviews
Thorning S1 , Del Mar C1 , Van Driel M1
1 Acute Respiratory Infections Group, Australia
Background: Search strategies have become increasingly complicated
with time. This increases effort for both searching and parsing effort. We
wondered whether simpler searches would reduce unnecessary ‘hits’
while preserving the core ones. Objectives: To test whether simple
searches will improve specificity without compromising sensitivity.
Methods: We have searched past Cochrane reviews in the acute
respiratory infections (ARI) Group with two or more search strategies to
test whether included studies identified in the reviews were retrieved.
Results: These are presented with recommendations about expanding
or contracting either the aetiological agents; the symptoms; or the
disease search terms. Conclusions: These recommendations should
improve the efficiency of searching and parsing titles.
P1A37
Accuracy of the MeSH term ‘Breast Neoplasms’:
Ten years on
Tai FM1 , Willson ML1 , Ghersi D1
1 Cochrane Breast Cancer Group, Australia
Background: MeSH terms are designed to categorise articles into
topic areas which allows us to search with some degree of specificity.
In 1999, we analysed whether the MeSH term ‘Breast Neoplasms’ was
correctly assigned to records in MEDLINE. We found that the MeSH
term was accurately assigned to over 85% of the retrieved records.
As our group’s search strategy has since been revised, the accuracy
of the MeSH term ‘Breast Neoplasms’ needed to be reassessed.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
56
Objectives: To evaluate the accuracy of the MeSH term ‘Breast
Neoplasms’. Methods: We applied our search strategy to MEDLINE
for trials published in 2009/10. The retrieved records were imported
into Reference Manager 11 and coded by two authors. Records were
categorised as whether they were related to breast cancer, indexed with
the MeSH term ‘Breast Neoplasms’ and if the trial was randomised.
Full manuscripts were retrieved, if necessary. Any uncertainty while
coding was resolved through discussion. Results: 2000 records were
retrieved (87 were duplicates). 1473 records were tagged with the
MeSH term ‘Breast Neoplasms’ of which 164 records were not relevant
to breast cancer but on cancer in general (121) or incorrectly tagged
(43). Of the 440 records without the MeSH term ‘Breast Neoplasms’,
36 were relevant to breast cancer and on reconstructive surgery,
risk factors, diagnostics and endocrine therapies. There were 501
randomised trials on breast cancer, 21 of which were not assigned the
‘Breast Neoplasms’ MeSH term. These included trials on reconstructive
surgery, mammographic density and drug therapies. This indicated
that the MeSH term was relatively sensitive with minor (<5%) errors.
Conclusions: MeSH terms are used to identify relevant publications
when searching databases. Our analysis shows that the term ‘Breast
Neoplasms’ is not always accurately assigned. Systematic reviewers
should be aware of the limited accuracy of MeSH terms when searching
for studies.
P1A38
Inconsistent quality of reporting of searching clinical trials
registries in Cochrane systematic reviews and protocols
Ko H1 , Tai FM1 , Ghersi D2 , Askie LM1
1 Australian New Zealand Clinical Trials Registry, Australia; 2 Research Policy
and Cooperation, World Health Organisation, Switzerland
Background: Searching of clinical trials registries (CTRs) for registered
clinical trials has not been mandatory for systematic reviews (SRs) and
protocols published in the Cochrane Database of Systematic Reviews
(CDSR), however some Review Groups are encouraging this practice.
In the 2010 Cochrane Colloquium, Ghersi et al and Hooft et al
reported there was low searching of CTRs. Objectives: To assess
the quality and consistency of reporting of searching International
Clinical Trials Registry Platform (ICTRP) and/or primary CTRs, as well as
other methods to identify ongoing clinical trials, in SRs and protocols.
Methods: SRs and protocols that used the keywords ‘trial registry’,
‘trial registries’, ‘trial register’, or ‘trial registers’ were retrieved on 29
March 2011. Two authors independently evaluated methods sections’
reporting of searching (1) the ICTRP for ongoing clinical trials, (2)
primary CTRs listed in the WHO ICTRP, and/or clinicaltrials.gov, and
whether (3) other sources were used to identify ongoing clinical trials.
Disagreements were resolved by consensus. Results: 414 SRs and
228 protocols were retrieved. Overall, 247 (60%) SRs and 167 (71%)
protocols reported searching ICTRP and/or CTRs. Table 1 shows the
breakdown of CTR searching methods reported. SRs with their latest
search performed from 2008 show >70% of SRs reporting searching
CTRs, with most preferring to search a variety of CTRs or searching
both ICTRP and CTRs. The amount and names of CTRs used were
variable. SRs that have not updated their search since 2007 have
minimal or no reporting of searching CTRs. Searching other sources
such as experts or pharmaceutical companies about ongoing trials was
very common. Conclusions: The quality and consistency of reporting
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Table 1 (P1A38). Reporting of CTR searching.
Reporting of CTR Searching
SRs
Protocols
ICTRP only
9
21
CTRs only
172
80
Both ICTRP and CTRs
66
61
Unclear for both ICTRP and CTRs
16
17
Searched ICTRP and unclear for CTRs
0
0
Searched CTRs and unclear for ICTRP
2
0
Unclear ICTRP and no CTR
0
0
Unclear CTR and no ICTRP
0
1
of searching CTRs for ongoing clinical trials is highly variable. More
explicit and consistent search protocols across all SRs and protocols
are needed, as well as updating of search strategies of older SRs.
P1A39
The current status of clinical research in traditional korean
medicine, South Korea
Jang B1 , Go H2 , Lee Y1 , Park S2 , Kim J3 , Cho KH4
1 National evidence-based healthcare collaborating agency, Seoul, South
Korea; 2 Dept. of Internal medicine, Hospital of Oriental Medicine, Semyung
University, Chungcheongbuk-do, South Korea; 3 Dept. of Gastroenterology,
Hospital of Oriental Medicine, Kyung Hee University, Seoul, South Korea;
4 Dept. of Cardiovascular and Neurologic Diseases (Stroke Center), College
of Oriental Medicine, Kyung Hee University, Seoul, South Korea
Background: The healthcare system in South Korea comprises of two
systems: one is Western medicine and the other Traditional Korean
Medicine (TKM). TKM has been criticized for uncertain evidence.
Objectives: To examine the current status of clinical research in TKM
and to assess ‘risk of bias’ (ROB) in randomized clinical trials (RCTs) in
TKM. Methods: Korean Oriental Medicine Society (KOMS), a special
committee for EBM, reviewed 17 journals related to TKM in South
Korea (from the first issue to May 2010), the Cochrane Central Register
of Controlled Trials (The Cochrane Library Issue 2, 2010), and PubMed
(1966 to May 2010). TKM clinical trials reported by traditional medical
doctors were included. Data were extracted from included articles on
objectives, design, setting, participants, interventions, main outcome
measures, main results and conclusions, using a predetermined form
of structured abstract. We selected eligible TKM RCTs and assessed
‘risk of bias’. Results: We reviewed 12,653 articles from the 17
journals and 41 articles from CENTRAL and PubMed. 1,004 articles
were left after non-clinical articles were excluded. Of these, 306 were
TKM eligible studies. Of these studies, 57 were on musculoskeletal
disorders and 55 on the circulatory system. 133 of the 306 TKM
eligible studies were RCTs, of which 69 were on acupuncture and 25
on herbal medicine. We found a large proportion of ‘unclear’ risk of
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
57
Figure 1 (P1A39).
bias in the criteria of ‘Allocation concealment’, ‘Blinding of participants
and personnel’, ‘Blinding of outcome assessment’ and ‘Other bias’.
On the other hand, we found a large proportion of ‘low’ risk of bias
in the criteria of ‘Incomplete outcome data’ and ‘Selective reporting’.
Conclusions: Clinical research in TKM is insufficient and many studies
focused on musculoskeletal disorders and the circulatory system. There
may be risk of bias in terms of ‘allocation concealment’ and ‘blinding’.
For high-quality research in TKM, further research is needed with
randomized and blind RCTs.
the medical literature and for specialty areas and diseases the scatter
follows the pattern predicted by Bradford’s law. The scatter has
implications for both clinicians trying to keep up to date and for those
performing systematic reviews.
P1A40
How does the number and scatter of trials vary
by discipline?
Chen C1 , Chen KH1 , Kuo KN2 , Tzeng PC1 , Chien LT1 , Lo HL2 , Tam KW3
1 Wan Fang Hospital, Taipei Medical University, Taiwan; 2 Taipei Medical
University, Taiwan; 3 Taipei Medical University Hospital, Taiwan
Glasziou P1 , Thorning S1 , Erueti C1 , Hoffmann T1
1
CREBP, Bond University, Qld, Australia
Background: The National Health Research Institutes in Taiwan has
translated abstracts of the Cochrane Database of Systematic Reviews
(CDSR) into traditional Chinese since 2008. This is the only contracted
translation activity of CDSR abstracts in East Asia. During the past two
years, more than 800 participants have joined the translation task.
The translated Chinese Abstracts of CDSR (CACDSR) were open to the
public online since 2010. (http://clc.nhri.org.tw/admin/clcmain1.aspx).
Objectives: To analyze the utilization of CACDSR in three university
hospitals in the past eight months. Methods: 1. Obtain the monthly
users’ login information of CACDSR in three university hospitals from
September 2010 to April 2011. 2. Analyze the trend of utilization
among physicians, nurses, and other healthcare professionals. 3.
Collect information on evidence-based medicine (EBM) events in these
hospitals over the data collecting period. Results: 1. The utilization of
CACDSR among nurses has increased significantly in the study period.
The overall trend showed that the hit number in nurses is significantly
higher than that of the other healthcare professionals. 2. The total hits
of CACDSR from nurses are 9 times higher than that from doctors. When
promoting a database, nurses are unnegligible. 3. The peak months of
utilization had EBM activities at the time, such as an EBM workshop,
an EBM contest and evidence-based quality improvement activities in
the hospitals. These events may have stimulated the utilization in
CACDSR. 4. The lowest month of utilization, i.e. Jan. 2011, can
be explained by the influence of the long vacation around Chinese
New Year. Conclusions: The study indicates the strong demands for
Chinese translated databases while promoting evidence-based practice
and continuous education for nursing professionals in Taiwan. EBM
Background: Bradford’s law of scatter suggests that if the journals
in a field are sorted by number of articles and divided into 3 equal
sized groups the number of journals will be in the proportions 1:n:n2 .
Bradford’s Law has been shown to apply to trials overall but little
work has been done for specialties of diseases. Objectives: To assess
whether and how Bradford’s law applies to trials and reviews overall and
in different specialty areas. Methods: We downloaded all trials (using
publication type) in MEDLINE for 2009, and classified trials in each of
14 clinical areas and specific disorders that are major contributors to
the global burden of disease: mental disorders, depression, alcohol
related disorders, heart disease, myocardial ischemia, nervous system
diseases, cerebrovascular, dementia, otorhinolaryngologic, hearing
loss, endocrine disorders, diabetes, cancer and lung cancer. Results:
The number of journals covering all trials in an area was highly
correlated with the number of trials (r = 0.97), but varied by clinical
area. Depression had most with 318 trials in 214 journals – 1.5 trials
per journal. Cardiology had least scatter with 1,459 trials in 374
journals – 3.9 trials per journal. The number of journals to identify
half the trials in a condition ranged from 9 for lung cancer trials to
79 for stroke trials. The journal with the highest number of trials in
each area was generally a journal within that speciality, e.g. Stroke
for cerebrovascular disease and Diabetes Care for diabetes. However,
general medical journals and journals in related subspecialties often
occurred in the top 10. Conclusions: Trials are widely scattered in
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P1A41
Utilization analysis of Chinese abstract of the Cochrane
Database of Systematic Reviews in three university
hospitals
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
58
Figure 1 (P1A41). CACDSR monthly utilization and trend among health professionals.
workshops, EBM contests, and evidence-based quality improvement
activities may accelerate the utilization of evidence-based databases.
P1A42
Development of search strategies to identify reports
of off- label drug use in MEDLINE
Mesgarpour B1 , Müller M1 , Herkner H2
1 Department of Clinical Pharmacology, Medical University of Vienna,
Vienna, Austria; 2 Department of Emergency Medicine, Medical University
of Vienna, Vienna, Austria
Background: It has been reported that a large number of medications
are used routinely for unapproved indications or in incorrect dosages,
routes of administration or age groups. These are so-called ‘off-label’
uses. However, the efficacy and safety of off-label drug usage have
not been sufficiently proven. Objectives: Because of variation in the
description of using medications beyond their approval and diversity
in study type, the off- label drug use studies are not easy to detect
in electronic literature databases. This research aimed to develop a
search strategy that facilitates finding the off- label drug use studies
in a bibliographic database, Ovid MEDLINE. Methods: MEDLINE
via OvidSP was searched from 1948 to 20 January 2011. Search
queries, including search words and strings, were conceived based
on definitions of off-label use of medications. The sensitivity and
specificity of search queries and their combinations as well as the
subject heading ‘Off-Label Use.sh.’ were determined by comparison
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
with a gold standard reference set. The gold standard was compiled
from reports of off- label drug use identified by a sensitive search of
MEDLINE and EMBASE. We attempted to achieve the highest possible
sensitive search strategy and an optimal balance of sensitivity and
precision. Results: Out of 4334 retrieved documents, 2177 (50.2%)
off- label drug use studies were found. The most sensitive single term
was ‘off label*.af.’ (sensitivity 40.9%, precision 84.4%). A top sensitive
strategy was revealed by combining 31 search queries with sensitivity
of 53.3% and precision of 60.3%. Maximizing precision, a top search
strategy achieved a value of 84.0%, but with compromised sensitivity
at 49.0%. Conclusions: We empirically developed two versions of
highly sensitive search strategies, which can achieve high performance
for retrieving off-label drug use documents in Ovid MEDLINE.
P1A43
Developing software for combining search queries to
design efficient search strategies
Mesgarpour B1 , Mesgarpour M2 , Müller M1 , Herkner H3
1 Department of Clinical Pharmacology, Medical University of Vienna,
Vienna, Austria; 2 School of Mathematics, University of Southampton,
Southampton, United Kingdom; 3 Department of Emergency Medicine,
Medical University of Vienna, Vienna, Austria
Background: Diagnostic test methods usually have been applied in
studies with search strategies development purpose to evaluate the
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
59
search terms and search strategies. These methods compile a set of
relevant studies as the ‘‘gold standard’’ and determine the sensitivity,
specificity, precision and accuracy of searches. If the gold standard
has been produced by following the sensitive search in bibliographic
databases, the sensitivity and precision of search queries and strategies
have been evaluated and compared. Objectives: This study aimed
to build and develop a computational program for combining search
queries to design a highly sensitive search strategy, as well as an
optimal sensitive and precise one. Methods: An algorithm has been
introduced to measure the sensitivity and precision of search terms
and their combinations with purpose of finding the most sensitive
combination of search terms. Then, a C++ code is developed
to capture user-defined search terms and their dependencies and
constraints to derive optimized combinations. In order to test and
improve the program, our former data on developing search strategies
have been assessed continuously to identify reports of off- label drug
use in MEDLINE and EMBASE via OvidSP. Results: Two versions of
the software, SenPrecOptim, have been developed. They automatically
perform calculations regarding the sensitivity and precision of search
queries and all their possible combinations. The software outputs the
optimized ranked list of search queries combinations based on the
sensitivity in a .csv file. The best balance of sensitivity and precision
strategy of optimized outputs can be obtained by finding the best-fit
line on the scatter plot of sensitivity versus precision. Conclusions:
The evaluation of search queries combinations can be enhanced by
utilization of this software. The output of this yields the best-balanced
sensitive and precise search strategies.
P1A44
Identification of biomedical journals in Spain and Latin
America
Bonfill X1 , Bonfill A1 , José E1 , Garcia M2 , Raiher S2 , Aranda E3 , Rada G4 ,
Rincón D5 , Tristán M6 , Torres A7 , Hidalgo R8 , Simancas D8 , López L9 ,
Jiménez C10 , Piña M10 , Correa R11 , Rojas A12 , Loza C13 , Féliz E14 ,
Gianneo O15 , Patrón C15 , Martı́ A16 , Posso M1 , Osorio D1
Iberoamerican Cochane Centre and Network (1 Spain; 2 Argentina; 3 Bolivia;
4 Chile; 5 Colombia; 6 Costa Rica; 7 Cuba; 8 Ecuador; 9 Guatemala; 10 México;
11 Panamá; 12 Paraguay; 13 Perú; 14 R. Dominicana; 15 Uruguay; 16 Venezuela)
Background: It is difficult to identify the existing journals belonging to
a particular area. In Latin America, there are some specific databases
(LILACS, SciELO, Latindex) which are very useful but many journals
are not indexed and therefore not easy to detect by other means.
Objectives: To identify all biomedical journals published in Spanish,
both in Spain and Latin America in order to analyse and describe their
main features. Methods: Biomedical journals that may publish original
clinical research papers were selected. One or more people affiliated
with the Iberoamerican Cochrane Network assumed the responsibility of
identifying the respective national journals included into the specialised
databases (Medline, LILACS, SciELO, Latindex), publishers’ catalogues
or available at libraries. For each journal identified, the following
characteristics were collected: name of the journal; ISSN; activity in
2009 (yes / no); period of activity; publisher; databases in which it
is indexed; medical specialty. Further criteria were applied at the
coordinating centre, in consultation with reviewers, for increasing
consistency. Results: 2145 biomedical journals published in Spanish
that met the inclusion criteria in 2009 were identified. After keeping
apart Mental Health and Nursing and refining data, the distribution of
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
included journals (948) by country is: Argentina 155, Bolivia 85, Chile
90, Colombia 57, Costa Rica 16, Cuba 43, Ecuador 32, El Salvador
6, Spain 253, Guatemala 2, Honduras 3, Mexico 100, Nicaragua
0, Panama 4, Paraguay 5, Peru 32, R. Dominicana 7, Uruguay
18, Venezuela 40. Results will be presented by country, specialty,
and database and activity period. Conclusions: Many biomedical
journals are publishing clinical studies in Spanish although only a small
proportion is indexed in bibliographic databases. A detailed analysis
of collected data will permit making useful recommendations. Also,
electronic and hand searches will be more efficient and comprehensive
for identifying published randomised clinical trials and systematic
reviews.
P1A45
Systematic Review Service (SRS). China
Xia J1
1 Cochrane
Schizophrenia Group, UK
Background: Cochrane groups have a problem with Chinese
literature. China has modernized scientific research and the number
of trials has hugely increased in the last decade (McDonald, 2006).
There are problems with access and quality. Access is limited. For
example, only 1% of trials in one area of health care are found on
PubMed (Chakrabarti, 2007). China has five overlapping but distinct
major biomedical databases. Collectively, they index thousands of
Chinese biomedical journals (Xia, 2008) and these are largely ignored
by Cochrane reviewers. While the quantity of Chinese papers has
increased dramatically, quality is questionable (Lancet 2010). Quality
assurance and governance must and will be improved, but until that
researchers would have to manually select good Chinese studies from
other less high quality work. If work from the whole of China is not to
be disregarded – a systematic but clearly parochial approach contrary to
the Collaborations’ founding principles – Cochrane reviewers must be
helped identify, sift, quality assure and data extract work from mainland
China. Conclusion: This work can only be undertaken efficiently by
trained researchers fluent in Mandarin, scientific methods and Cochrane
methodology. SRS (http://www.srs-org.com) is established by a team
with a track record in production and maintenance of complex Cochrane
reviews. SRS can i. help identify trials relevant to a specific review or
area of interest; ii. select relevant trials for inclusion; iii. data extract; iv.
work within RevMan to allow data to be used (or not used); v. provide
full translation service (often unnecessary); vi. contact authors for
additional information. Each service can stand alone or be integrated
into a package. SRS will allow consideration of previously inaccessible
trial data from China within Cochrane reviews and, by that, facilitate
improvement of both reviews and trials.
References
Chakrabarti, A., Adams, C.E., Rathbone, J., Wright, J., Xia. J.,
Wong, W., Von Reibnitz, P., Koening, C., Baier, S., Pfeiffer, C.,
Blatter, J., Mantz, M. & Kloeckner, K. (2007). Schizophrenia trials in
China: a survey. Acta Psychiatrica Scandinavica. Vol 116(1), pp. 6–9.
The Lancet. (2010). Scientific fraud: action needed in China. Lancet.
Vol 375(9709) pp. 94.
McDonald, J. (2006). OECD: China to spend $136 billion on R&D.
Business Week. Available from: http://www.businessweek.com/ap/
financialnews/D8LQ0OI00.htm (accessed 6 December 2006).
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
60
Xia, J., Wright, J. & Adams, C.E. (2008). Five large Chinese biomedical bibliographic databases: accessibility and coverage. Health
Informationa and Library Journal. Vol 25(1), pp. 55–61.
P1A46
Handsearching respiratory conference abstracts: a
comparison with abstracts identified by an EMBASE search
Stovold E1 , Hansen S2
1 Cochrane Airways Group, St George’s, University of London, UK;
2
Cochrane Airways Group, New Zealand
Background: Conference abstracts make an important contribution
to the Airways Group Register (AGR). The abstracts of key national
and international respiratory conferences are manually handsearched
for reports of trials. Identifying these abstracts manually is time
consuming. The EMBASE database now includes conference abstracts
from relevant respiratory meetings and these are being identified by the
weekly EMBASE searches run for the AGR. Objectives: To compare
abstracts found using the Airways Group EMBASE search strategy with
the abstracts found by handsearching and determine if an EMBASE
search would be sufficient in the future. The Chest 2010 meeting
abstracts were used for the comparison. Methods: The Chest 2010
conference abstracts were handsearched via the Chest website. Each
individual abstract was read and assessed for study design and the
citation was downloaded for submission to the CENTRAL database.
Those abstracts relating to the scope of the Airways Group were
tagged and added to the Airways Group Register of Trials. When
the weekly search alerts from EMBASE were downloaded, the Chest
2010 abstracts that had been identified by the Airways Group search
strategy were extracted. They were manually checked to see if all the
abstracts relating to the scope of the Airways Group already identified
through handsearching were present. Results: The EMBASE search
identified 151 abstracts from Chest 2010. Handsearching identified
47 RCT/CCTs, 34 of which were suitable for the AGR. A comparison
of the two sets found 7/34 missing from the EMBASE set. Five were
trials in pulmonary hypertension, and 2 of the missing abstracts were
trials in asthma. Conclusions: The addition of conference abstracts
to EMBASE is a welcome development, but handsearching the major
respiratory conference abstracts is still worthwhile in view of the missing
references.
P1A47
Ankle brachial index to predict cardiovascular events
in primary health care, a systematic review
Galvão ALC1 , Stein AT2 , Pellanda LC3 , Marcadenti A4
1 Grupo Hospitalar Conceição; 2 Grupo Hospitalar Conceição / UFCSPA /
ULBRA; 3 Instituto de Cardiologia – Fundação Universitária de Cardiologia /
UFCSPA; 4 Grupo Hospitalar Conceição / Cardiology postgraduation course
UFRGS
Background: Cardiovascular diseases still remain the most frequent
cause of death in many countries, despite all measures and efforts that
have been made to control risk factors and therapeutic approaches.
Approximately half of all coronary deaths are not preceded by cardiac
symptoms. In addition near 50% of all patients who have acute
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
coronary syndrome are stratified, by risk score models, as just being
an intermediate risk patient. The prevention of such events remains
a serious public health challenge. Many previous studies have shown
that the measurement of the ankle brachial index (ABI) may improve
the accuracy of cardiovascular risk prediction. Objectives: The aim
of this review is to provide an evidence-based approach to show how
the measurement of the ABI, as an indicator of atherosclerosis, can
improve risk assessment and predict cardiovascular events beyond the
traditional used risk scores. Methods: The ABI is the ratio of systolic
pressure measured by doppler at the ankle to that in the arm. Risk
prediction using the Framingham risk score (FRS) was also reviewed. A
search of MEDLINE, Scielo, Cochrane Library, LILACS and EMBASE was
conducted using common text words for the term ankle brachial index
combined with primary health care and cardiovascular risk. A review
of reference lists and conference proceedings wasperformed. We have
identified relevant studies published about this topic and have been
doing a critical appraisal of the articles selected. Studies for inclusion
were independently assessed by 2 collaborators and disagreement was
resolved by a third reviser. Results: Preliminary results have shown
that this question has not been completely replied in the literature and it
is quite important for public health policies on this issue. Conclusions:
Measurement of the ABI in addition to other cardiovascular risk score
models can stratify patients more accurately and also may improve the
accuracy of cardiovascular risk prediction.
P1A48
Geographic and language distribution of trials in Cochrane
Systematic Reviews related to acupuncture
Tung Y1 , Hou M1 , Chen Y1
1 Changhua Christian Hospital, Taiwan
Background: In the Cochrane Library, the systemic reviews of
trials should be as global as possible and with no limitation of
language or geography. When the topic is related to acupuncture,
some basic diagnostic criteria of acupuncture must be included.
Objectives: To study whether the Cochrane Database Systematic
Reviews (CDSR) places language and geographical restrictions on
searching for acupuncture, and whether the trials used the diagnostic
criteria of ‘pattern identification’ in their researches. Methods: We
searched ‘acupuncture’ published in the CDSR, years from 2009 to
2011 were all included. We first examined the published country
and whether any language restrictions existed. Then we examined
the reviews descriptions and whether the pattern identification was
mentioned on their researches. We excluded the studies without direct
relevance to acupuncture. Moxibustion, transcutaneous electrical nerve
stimulation and Chinese herb were excluded. Results: There are 56
results out of 6,602 records for: ‘acupuncture’ in Record Title. One
of the reviews in CDSR had imposed language restrictions in Chinese.
The other review had language restrictions in English and Chinese.
There are 25 reviews had no language restrictions. And the remaining
reviews had unclear restriction on language. We also noticed all
56 studies are from two continents and 7 countries. There is one
review from Germany, one from Norway, three from Canada, six from
USA, and eight from Australia. With literal inspection on the type of
participants, all of the included trials in 56 studies processed without
basic diagnostic criteria of pattern identification. Conclusions: There
appears to be most of the reviews do not have any language restrictions
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
61
on searches. And all the reviews are from very few countries. It reminds
us there might not have enough diversity to show the real world.
Besides pattern identification is not addressed enough, which would
reduce the accuracy of the reviews for acupuncture.
P1A49
Drugs costs before and after bariatric surgery:
A systematic review
Stein AT1 , Lopes EC2 , Heineck I3 , Athaydes G4 , Souto KP5 , Meinhardt NG5
1 Grupo Hospitalar Conceição/Ufcspa/Ulbra, Brazil; 2 Epidemiology Postgraduation Course, Ufrgs-Brazil; 3 Pharmaceutical Science Postgraduation
Course, Ufrgs-Brazil; 4 Pharmaceutical Course, Ufrgs-Brazil; 5 Obesity Unit
Grupo Hospitalar Conceição
Background: Surgical treatment has shown to be the only effective
method for keeping the weight loss for a long period in morbidly obese
people. The surgical procedure requires long-term monitoring, both
nutritional and pharmacologic. Objectives: To review the impact of
bariatric surgery on drug costs. Methods: We have used the following
MESh terms: obesity, bariatric surgery and drug costs, and have
searched in nine databases (including Medline, Embase and Cochrane
Library), until January 2011. In addition, reference lists were reviewed
and contacted experts to identify additional published and unpublished
references. Inclusion criteria: adults, BMI >35 kg/m2, drug use and
comparision of costs before and after surgery. Articles were reviewed by
two independent reviewers and differences were resolved by consensus.
Results: The search identified 501 citations, and ten articles were
identified as potentially relevant. Three articles were excluded, one
that compared cost between obese and nonobese patients, another
one in which compared total costs in health, although, cost had
not been considered and the last one compared drug costs between
surgical therapy and conventional therapy. The studies included in
this review were cohorts, ranging from 50 to 965 patients. The
systematic review has shown a decrease on drug expenditure between
pre and post-operative, which was statistically significant (p < 0.05).
Although, one of the studies had found an increase of 32.1% in
expenditure after a 6-year monitoring. Conclusions: Even though,
the studies that were reviewed have demonstrated cost reduction with
regards to the use of drugs, the time framework in the postoperative
period is short. The study that has followed patients for a longer period
of time showed a different outcome in relation to others, ie an increase
in drug spending after surgery.
P1A50
The updating of clinical practice guidelines: A systematic
review of the research evidence
Martı́nez Garcı́a L1 , Arévalo-Rodrı́guez I2 , Solà I1 , Hemens B3 , Vandvik
PO4 , Haynes B3 , Alonso-Coello P1
1 Iberoamerican Cochrane Centre, Biomedical Research Institute Sant
Pau (IIB Sant Pau), Barcelona, Spain; 2 Clinical Research Institute,
GETS, Universidad Nacional de Colombia, Bogotá, Colombia; 3 McMaster
University, Hamilton, Canada; 4 Norwegian Knowledge Centre for the
Health Services, Oslo, Norway
Background: Scientific knowledge is in constant change. The flow
of new information requires a frequent revaluation of the available
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
research results. Clinical practice guidelines (CPGs) are not exempted
from this phenomenon and need to be kept updated to maintain the
validity of their recommendations. Objectives: To identify, describe
and compare methods for updating CPGs. Methods: Selection criteria:
We included methodological research studies evaluating: 1) Strategies
to evaluate if CPGs are out of date; 2) Strategies to update CPGs; and
3) Living CPGs strategies. Search strategy: The search strategy included
MEDLINE (March 2011) and The Cochrane Methodology Register (the
Cochrane Library Issue 3 2011). There were no restrictions concerning
language or type of publication. Data collection and analysis: Authors
independently selected studies and extracted data. The results are
presented narratively due to differences in methodology between the
studies. Results: The review included eight studies. Most of them
described a method for updating one or more CPGs, except one study
that applied it on 6 topics and another on 15 recommendations. Four
studies described strategies to evaluate if CPGs were out of date, three
studies described strategies to update CPGs and one study described a
living CPG strategy. Five studies described a single method and three
studies compared two methods. Regarding the literature searches for
new evidence, those evaluating whether guidelines were out of date
or those designed for the use in living guidelines applied more limited
approaches. On the other hand strategies addressed to update CPGs
included more exhaustive literature searches. Conclusions: There is
scarce and heterogeneous research about the updating of CPG. The
development of efficient strategies to identify new relevant evidence is
needed to improve the timeliness and reduce the burden of updating
CPGs.
P1A51
Uncommon language: The challenges of inconsistent
terminology use for evidence synthesis
Parker RM1 , Hayden JA2
1 Capital District Health Authority, Canada; 2 Dalhousie University, Canada
Background: Varied use of terminology around a common theme
creates barriers to communication and presents obstacles to systematic
literature retrieval. Concepts such as baseline risk and prognosis may be
expressed in a variety of ways depending on the context including time,
geographic region, and research discipline. Bias may be introduced
into systematic reviews on methods and epidemiology topics if search
terms used do not include all the possible ways of conceptualizing the
topic. Conversely, inclusion of terms whose meanings vary depending
on the context will impact the specificity of a search. Objective:
Explore the context distribution and potential impact of different health
concept terminology in peer-reviewed literature using the examples of
baseline risk and prognosis concepts. Methods: For each concept,
MEDLINE citations and the full-text of general medical and specialty
journals will be searched using a set of terms identified from the
literature and by consulting experts. Frequency of use for each
term will be assessed in different contexts. Additionally, MEDLINE
search results will be semantically analyzed to plot the frequency of
citations containing the terms of interest across time, region, and
discipline. Context analysis of term usage within full text and citations
will be conducted using text mining of a corpus identified by the
searches. Results: An initial search related to the concept of baseline
risk (including absolute risk, overall prognosis, natural history, etc.)
showed that the relative frequency of terms vary depending on the
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
62
journal. Semantic analysis of MEDLINE citation results also found
discipline-specific term preference. Prognosis-related terms will be
searched to assess terminology variability around another complex
concept. Conclusions: The use of terms varies between disciplines
and regions of practice. Complete results including context analysis
and impact will be presented.
P1A52
A snowballing technique to ensure comprehensiveness
of search for systematic reviews: A case study
Vedula S1 , Mahendraratnam N1 , Rutkow L1 , Kaufmann C1 , Rosman L2 ,
Twose C2 , Dickersin K1
1 Johns Hopkins Bloomberg School of Public Health, USA; 2 Welch Medical
Library, Johns Hopkins University, USA
Background: Increasingly, trial data are becoming available through
internal company documents made accessible through litigation. We
performed a systematic review of how such documents had been used
in the past and how they were identified. Systematic reviews, such
as ours, that address complex questions pose a challenge in terms of
searching for potentially eligible studies. Objective: The objective
of this study was to evaluate a ‘snowballing’ technique to iteratively
revise a search strategy used for a systematic review of studies using
data from internal industry documents. Methods: We identified
a set of 19 ‘index’ articles that were eligible for inclusion in our
systematic review. From this set of ‘index’ articles, we developed an
initial search strategy for PubMed and Embase using a combination of
indexing terms and keywords. After de-duplicating abstracts retrieved
from the two databases, two reviewers independently screened the
titles and abstracts, and subsequently the potentially eligible full text
articles, against pre-specified eligibility criteria. We will examine the
indexing terms and text words for all studies categorized as eligible
and revise our initial search strategy using additional index terms and
text words. We will compare the results of our ‘snowball’ approach
to a ‘reference standard’ to estimate the recall and precision of our
approach. Results: The initial strategy retrieved 7061 results in
PubMed and 2244 results in Embase. Independent screening of titles
and abstracts and full-text articles is in progress. Findings related to our
analyses of recall and precision will be described in our presentation
at the Cochrane Colloquium. Conclusions: Searches for systematic
reviews addressing complex questions using a ‘snowballing’ approach
could be tested for systematic reviews addressing other types of
questions.
P1A53
Searching far and wide for occupational prevention
literature: Where do we go to find relevant studies?
Van Eerd D1 , Mahood Q1 , Slack T1
1
Institute for Work & Health, Canada
Background: The Institute for Work & Health (IWH) initiated a
four-year pilot project to conduct reviews on the effectiveness of
interventions to prevent workplace injury, illness or disability. As part
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
of the pilot the IWH has undertaken an examination of the methods
of searching for and reviewing the prevention literature. One method
of evaluating our searches is to analyze where studies come from
to improve future search strategies and reviews. Objectives: To
determine which bibliographic databases yielded the highest number
of included studies in the area of occupational health and safety reviews
and add to the research on health care review methodology. Methods:
A retrospective analysis was done across 14 reviews, spanning a six year
time interval, to identify which bibliographic resources yielded relevant
studies for each review. In addition we will conduct a sensitivity and
specificity analysis of our searches. Results: Preliminary results, based
upon 9 reviews, indicate that those databases that typically yield the
higher numbers of included studies in clinical reviews i.e. Medline and
Embase, also yielded similar proportions in the occupational health
and safety literature. Not surprisingly the results across 12 somewhat
diverse bibliographic databases indicate the importance of searching
multiple databases. No single database yielded all of the relevant
studies. Data will be presented for each source and for individual
reviews in order to determine the sensitivity and specificity of each.
This analysis will represent over 100,000 studies from 12 different
bibliographic databases. Conclusions: Our analyses will lead to more
sensitive and specific search strategies in the area of occupational
health and safety. To conduct a comprehensive systematic review on
the effectiveness of interventions to prevent workplace injury, illness
or disability it is important to include multiple databases. However,
to capture relevant prevention studies it is necessary to screen many
irrelevant studies.
P1A54
Identification and analysis of search strategies and filters
carried out in systematic reviews indexed in Lilacs.
Overview
Puga MEDS1 , Pereira RA1 , Figueiró MF2 , Melnik T1 , Carvalho APV1 ,
Riera R1
1 Brazilian Cochrane Centre, BR; 2 Institute of Education and Research
Hcor-BR
Background: The search methodology group from Brazilian Cochrane
Centre has worked with the resources and tools available in the
new LILACS (Latin-American and Caribbean Health Sciences Database)
website (IAX format). This format displays the search results organized
according to the study design and there are 1,218 systematic reviews
(SR) – including Cochrane and non-Cochrane – currently available.
Objectives: To identify and assess the search strategies and filters
carried out in all systematic reviews indexed in the new Lilacs’ Database
website. Methods: Through an independently way three researchers
are assessing the methods section from each one SR aiming to identify:
a) the search strategies and filters used for clinical trials, b) the grade
of sensitivity and specificity of these filters, and c) the database and
search terms (MeSH, DeCS or synonymous) used by the SR authors.
Results: The results will be available in June 2011, but as potential
findings, the researchers speculate that most of the reviews do not use
LILACS database. Also, it is expected that synonymous are used instead
of MeSH terms in non-Cochrane SR. Conclusions: The conclusions
will be available in June 2011.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
63
P1A55
Title-abstract versus title-only citation screening
strategies for systematic reviews and meta-analyses
Mateen F1 , Oh J1 , Tergas A1 , Bhayani N1 , Kamdar B1
1 The Johns Hopkins Bloomberg School of Public Health, Baltimore,
Maryland, USA
Background: Systematic reviews are foundational for evidence-based
care, but require a substantial time investment by the authorship team.
There are currently no data or guidelines regarding the most effective
strategy for screening citations before full-text review. Objective: To
compare title-abstract and title-only strategies for screening citations
for a systematic review. Methods: A five-member group of physicians
performed a systematic review and meta-analysis on light at night and
breast cancer risk using Medline and Embase (January to March 2011).
Group members were assigned in a round-robin fashion to two reviewer
pairs. Each reviewer assessed whether citations included 1) primary
data, 2) the exposure of interest, and 3) the outcome of interest.
A titles-only screening strategy was undertaken first, followed by a
combined titles-abstract strategy on the same citation list. Outcome
measures included inter-rater agreement and reliability and articles
ultimately included for review following both screening strategies.
Results: 2965 records were screened following exclusion of duplicate
citations. Agreement/reliability between the five reviewer pairs for a
title-only screening strategy was 89–94% with kappa = 0.54, with
575 records remaining for abstract review. Agreement/reliability for
a title-abstract screening strategy was 96–97% with kappa = 0.56,
with 257 records remaining for full text review. The final systematic
review and meta-analysis included 13 articles, all of which were
identified by both screening strategies. Conclusions: Compared
to titles-abstracts strategy, screening titles-only resulted in a higher
number of disagreements but had a similar inter-rater reliability and
did not lead to exclusion of any articles used in the final analysis. Initial
screening via a titles-only approach may save considerable time in the
systematic review process.
Editorial processes and supporting
review authors
Day 20 Oct (P1A56–P1B80)
P1A56
International activity within Cochrane Review Groups
Allen C1 , Clarke M2 , Richmond K1
1
The Cochrane Collaboration Secretariat, Oxford, United Kingdom; 2 Centre
for Public Health, Queen’s University Belfast, United Kingdom
Background: Since 2000, a periodic count has been done of members
of Cochrane Review Groups (CRGs), subdivided by country of residence.
Objectives: To assess international activity within CRGs. Methods:
Data were retrieved on 12 April 2011 for all active contributors to
CRGs whose contact details were in The Cochrane Collaboration’s
contact database (Archie). One author (CA) analyzed the data, by role
within the CRG and country of residence. Unlike previous years when
individuals were counted repeatedly each time they appeared in the
audit, people listed more than once are now counted once only. This
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
made comparisons to absolute numbers in earlier years difficult, hence
the focus on percentages in the following results. Results: Among 53
CRGs, there were 24 228 individual contributors from 118 countries.
Of these, 20 246 (84%) contributors were listed as authors, which is
a higher proportion than that estimated for recent years (when it was
usually around 75%), and much higher than in earlier years (e.g. 52%
in 2000). There are now 4618 contributors in countries with low- and
middle-income economies (19% of all contributors, compared to 6%
in 2000). Of these 4618 people, 4346 people were listed as authors
(94%). Conclusions: The international diversity of people involved
in the work of The Cochrane Collaboration continues to grow, and
more than 24 000 individuals are contributing to its work. For many of
these people, this involves direct involvement in the preparation and
maintenance of systematic reviews, ensuring the continuing success of
the Collaboration as the world’s largest producer of systematic reviews
in health care. The growth in the contribution of people from low- and
middle-income countries has been sustained, with more than 4000
authors in these countries. Other data analyses will be shown in the
poster. The data will be available at http://www.cochrane.org.
P1A57
Experiences of undertaking a diagnostic test accuracy
review with the Cochrane Oral Health Group
Walsh T1 , Littlewood A1 , Liu J2
1 University of Manchester; 2 University of Dundee
Background: Oral cancer has been identified as a high priority
area of research for the Cochrane Oral Health Group (COHG). Oral
cancer is the sixth most common cancer globally, with incidence
increasing worldwide. External funding from the National Institutes of
Health (USA) to members of editorial staff of the COHG enabled four
intervention reviews in the treatment of oral cancers and screening
programmes for the early detection and prevention of oral cancer to
be carried out. A diagnostic test accuracy (DTA) review for screening
and diagnosis of oral cancer was identified as a priority review for
the group. Objectives: To document the issues encountered by
the review team and COHG editorial staff in undertaking their first
DTA review. Methods: Editorial staff received training on the
methods and processes involved for the production of DTA reviews.
A multidisciplinary review team comprising clinicians, a statistician
and a methodologist was formed to write the review, with input
from an expert panel of oncologists. At least one review author
was required to attend four DTA workshops. Results: Formulating
the research question was difficult, as substantial prior clinical and
methodological knowledge was required at the outset. Standard
diagnostic practice was not always ‘standard’; index tests could be
quite esoteric but still needed to be specified a priori; reference
standard differed between comparisons. A scoping literature search
and guidance from the Trials Search Co-ordinator were essential. From
scoping two separate but related research questions emerged. Moving
from basic analysis required detailed statistical knowledge and input.
Detailed knowledge of study design (usually not randomised controlled
trials) and clinical experience of target disease were needed to properly
evaluate methodological quality. Conclusions: In our experience, DTA
reviews are more difficult than intervention reviews, but not impossible.
They require a very clear understanding of the nature of the problem and
current clinical practice, and commitment from a multidisciplinary team.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
64
P1A58
Epidemiology, quality and reporting characteristics
of randomized controlled trials on traditional Chinese
medicine for gastric cancer published in Chinese journals
Ma B1 , Ke Y2 , Su X2 , Xian L3 , Li Y2 , Yang K2
1 Evidence-Based Medicine Center, Institute of Traditional Chinese and
Western Medicine, School of Basic Medical Sciences, Lanzhou University,
Lanzhou 730000, China; 2 Evidence-Based Medicine Center, Institute
of Traditional Chinese and Western Medicine, School of Basic Medical
Sciences, Lanzhou University, Lanzhou 730000, China 2 The First Clinical
School of Lanzhou University, Lanzhou 730000, China; 3 Evidence-Based
Medicine Center, Institute of Traditional Chinese and Western Medicine,
School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000,
China 3 The School of Public Health, Lanzhou University, Lanzhou 730000,
China
Background: Randomized Controlled Trials (RCTs) of traditional
Chinese medicine (TCM) have been produced in large numbers. But
the epidemiological characteristics, reporting detail, scientific validity is
unknown. Objectives: To provide the comprehensive information and
guide the development of RCTs on TCM in China. Methods: Chinese
Biomedicine Literature Database, Chinese Scientific Journal Full-text
Database and Chinese Journal Full-text Database were searched
to December 2010. Search terms included ‘stomach neoplasms’,
‘Traditional Chinese Medicine’ and ‘RCTs’ etc (MESH or EMTREE).
Data were collected using an Excel form designed to examine the
epidemiological and reporting detail. The CONSORT checklist was used
to appraise the reporting quality. Results: A total of 102 randomized
controlled trials (RCTs) published in 72 periodicals were included. Only
41.67% were published in specialty journals, 21.57% were included
in Chinese Science Citation Database as cited by journals. Two studies
reported informed consent. Two studies stated ‘randomization’ in
title. 52.94% and 11.76% of trials reported the specific inclusion
and exclusion criteria respectively. None of the studies stated the
method of sample size calculation. Only 17 trials reported how the
random allocation sequence was generated. A participant flow chart
was not described in any of the trial reports. Only two articles stated
the limitations of the study. None of the studies reported registration
and protocol. Fund sources were not reported in 82.35% trials.
Conclusions: A large number of RCTs were published in China, but
the reporting information was insufficient and the scientific validity
worrying. We suggest three measures to improve the quality of clinical
trials in China. Firstly, enhancing the clinical trials registration policy.
Secondly, the Chinese Medical Journal should endorse the CONSORT
statement in their ‘Instructions to authors’. Finally, medical schools
should introduce a registration policy and reporting guidelines as soon
as possible.
of Traditional Chinese and Western Medicine, School of Basic Medical
Sciences, Lanzhou University, Lanzhou 730000, China; The First Clinical
School of Lanzhou University, Lanzhou 730000, China; 3 Evidence-Based
Medicine Center, Institute of Traditional Chinese and Western Medicine,
School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000,
China; The Second Clinical School of Lanzhou University, Lanzhou 730000,
China; 4 Evidence-Based Medicine Center, Institute of Traditional Chinese
and Western Medicine, School of Basic Medical Sciences, Lanzhou
University, Lanzhou 730000, China; Ministry of Health of the People’s
Republic of China, Beijing 10044, China
Background: A large number of randomized controlled trials (RCTs)
in pediatrics have been published in China. However, there have been
no systematic assessments whether the reporting quality has improved
since the CONSORT checklists were introduced in China in 1997.
Objectives: This study was undertaken to evaluate the reporting
quality of RCTs published in pediatric journals in China. Methods: Six
core pediatric journals were searched from inception through December
2010 which included Journal of Clinical Pediatrics, Chinese Journal
of Contemporary Pediatrics, Chinese Journal of Practical Pediatrics,
Chinese Journal of Evidence-based Pediatrics, Chinese Journal of
Pediatrics, and Chinese Journal of Pediatric Surgery. The CONSORT
checklists were used to assess the trial reporting quality. Data was
collected using a standardized form. Analyses were performed using
Excel and SPSS. Results: A total of 619 RCTs were identified, in
which 157 RCTs were published before 1997. Compared to literature
published before 1997, there was significant improvement of quality
in some aspects of the trial reporting such as: title and abstract,
introduction, trial design, statistical methods, recruitment, baseline
data and ancillary analyses (P < 0.05). However, some important
methodological components of RCTs were still incompletely reported
such as: sample-size (0.63% vs.1.08%), sequence generation (3.18%
vs. 7.58%), allocation concealment (0% vs. 1.08%), and blinding
(0% vs. 0.87%), same as in funding (2.55% vs. 4.55%) and
protocol (0% vs. 2.16%). Besides, no trials reported the registry.
Conclusions: Overall, the reporting quality of RCTs in pediatrics has
changed significantly in most of the aspects we analyzed since the
introduction of CONSORT checklists in China; however, there is room
for improvement in some aspects such as methodology components,
registry, protocol and funding. Therefore, despite the advancements
observed, there are still several areas that can be improved in pediatric
RCTs.
P1A60
Reporting standard for qualitative research: A good
or a bad idea? Preliminary results from a Delphi study
P1A59
Changes in randomized controlled trial reporting quality
over time: A systematic review of pediatric journals from
China
Hannes K1 , Heyvaert M1 , Slegers K2 , Van den Brande S3 , Vandewiele W4 ,
Van Houdt S5 , Van Nuland M5
1 Centre for Methodology of Educational Research, K.U.Leuven, Belgium;
2 Centre for User Experience Research, K.U.Leuven, Belgium; 3 Interdisciplinary Centre for Religious Studies, K.U.Leuven, Belgium; 4 Criminal law and
Criminology, K.U.Leuven, Belgium; 5 Academic Centre for General Practice,
K.U.Leuven, Belgium
Ma B1 , Li H2 , Liu W1 , Zhang Y3 , Hu Q1 , Zhang Z4
1
Evidence-Based Medicine Center, Institute of Traditional Chinese and
Western Medicine, School of Basic Medical Sciences, Lanzhou University,
Lanzhou 730000, China; 2 Evidence-Based Medicine Center, Institute
Background: For experimental, diagnostic and longitudinal research,
standards have been developed with criteria guiding a researcher
in how to report the research procedure and findings, in order to
increase the quality of reporting and to facilitate the critical appraisal
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
65
of methodological quality. There have been attempts to develop such
a standard for qualitative research as a result of individual, academic
endeavour. However, the criteria identified have not been considerably
taken up by researchers conducting basic qualitative research. This
might be related to the variety of different paradigms and the broad
range of designs, data-collection and analysis techniques that one
could opt for when conducting qualitative research. Objectives:
To explore the possibility of developing a consolidated standard for
qualitative research. In the potential absence of a consensus on
a general standard, to identify particular qualitative methodologies
and/or approaches for which reporting guidance should specifically be
considered or might be counter-productive. In case of agreement on the
need for reporting criteria for particular designs, to reach consensus on
specific reporting criteria. Methods: A Delphi study will be conducted
to try and reach consensus between an international panel of experts
selected from a broad pallet of approaches or designs. Approximately
25 experts will contribute to this study. A multidisciplinary team
of researchers will assist in initiating discussion and analyzing data.
Results: We will report the findings from the first Delphi-round.
This round includes expert opinions related to the evaluation of the
potential benefits, feasibility, appropriateness, potential effectiveness
and meaningfulness of general standards of reporting for qualitative
research. Depending on the progress of the research, we will also
report on the selection of methodologies that are in need of specific
guidance in reporting. Both issues will feed into the potential final
round focusing on reporting criteria for particular designs.
P1A61
Obtaining funding for IPD meta-analyses: Top tips
for a successful application
Rydzewska L1 , Tierney J1 , Vale C1
1
IPD Meta-analysis Methods Group, UK
Background: Individual participant data (IPD) meta-analyses are
usually more time and resource intensive than conventional summary
data reviews; but benefit from improved data quality and analysis
and are considered to be the ‘gold standard’ of systematic reviews.
However, members of the Individual Participant Data Meta-analysis
Methods Group (IPDMAMG) have highlighted difficulties in obtaining
funding for these IPD projects. Objectives: To collate positive
and negative feedback received from funders and provide tips for
submitting new funding applications. Methods: We surveyed all 70
members of the IPDMAMG, asking them to provide details of negative
and positive feedback received from both successful and unsuccessful
funding applications. Feedback was collated and appraised to identify
key themes according to the frequency of particular responses. Based
on these identified themes, a list of ‘top tips’ was compiled. Results:
We received responses relating to 18 IPD funding applications: nine
successful, eight failed, and one ongoing at the time of the survey;
and comments from funders were wide-ranging. As for any project;
the originality, relevance and potential impact are important; the aims
and methods need to be clearly stated and the group applying should
either have appropriate experience or have formed a collaboration to
achieve this. However, perhaps less obvious, is the need to clearly
explain clearly why IPD are required, for example if summary data are
insufficient to address the question, together with an indication of how
likely it is that the IPD will be obtainable. Moreover, as IPD reviews do
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
cost more than standard reviews, ample justification for the resource
requested is also vital. Conclusions: Surveying the members of the
IPDMAMG provided a valuable resource for those seeking funding for
IPD projects, which we hope will improve the chances of a successful
application. This resource is available by contacting [email protected].
P1A62
Helping readers understand review update status: Pilot of
a simple system
Sinclair D1 , Garner P1
1
Cochrane Infectious Diseases Group, UK
Background: The Cochrane Library’s unique selling point is that it
contains up-to-date reviews. However, this is not always the case:
some reviews address historical questions; in some reviews the weight
of evidence is such that no further trials are justified; and in others, the
authors are in the process of updating the review. It is not immediately
explicit to The Cochrane Library readership what the update status
of a review is. Objectives: To develop a clear, simple, transparent
classification system for reviews to help readers understand whether
our reviews are up-to-date or relevant. Methods: We established
a classification system based on two parameters: relevance and
status. For relevance we identified two options: historical question,
where an intervention or policy had been superseded by new medical
developments (such as a new drug), and a current question, which
was still relevant to current policy or practice. Thus a review that was
no longer relevant to current practice was labelled as ‘Historical: no
update intended’. For reviews with a current question, we then used
three status categories: ‘up to date’, ‘update pending’, or ‘no update
intended’. There then follows a slot, which allowed the author or
editor to provide a little more detail. For ‘update pending’ we might
have given the number of trials waiting assessment and refer people
to the ‘trials waiting assessment section’. For ‘no update intended’ we
had two categories: ‘review findings conclusive’, and ‘topic covered
in another review (see xxxx review)’. Results: We have applied
this to over 80 of our reviews and are currently inserting this in our
published reviews. We will provide examples to illustrate how this
works. Conclusions: It is worth other groups considering adopting
this approach.
P1A64
A tool to improve efficiency and quality in the production
of protocols for Cochrane Reviews of Diagnostic Test
Accuracy
Pennant M1 , Wisniewski S1 , Hyde C2 , Davenport C1 , Deeks J1 , on behalf
of the Cochrane DTA ET3
1 The University of Birmingham, UK; 2 The University of Exeter, UK;
3 Cochrane Diagnostic Test Accuracy Review Unit, UK
Background: The Cochrane Diagnostic Test Accuracy Editorial Team
(DTA ET) organises methodological peer review of DTA protocols
and reviews. In the first part of this process, submitted protocols
are sent for peer review by DTA statisticians, search specialists and
clinical epidemiologists, and discussed by the DTA ET. Feedback to
authors highlights methodological issues that need to be addressed
before the review can proceed. Over 50 protocols have passed
through this system, and patterns of common problems have emerged.
Objective: To review common errors in protocols for DTA reviews in
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
66
order to develop a structured tool to assist: authors, Cochrane Review
Groups, and reviewers in preparing and reviewing submitted reviews.
Methods: Areas of methodological weakness, considered important
to the integrity of reviews of DTA, were identified by reviewing reports
of peer reviewers and the DTA ET from the first submissions for a
sample of 45 protocols. We reported on the nature and frequency
of these problems, and produced a structured tool to help identify
these problems, that links to the relevant DTA Handbook sections.
Results: Important areas of DTA review methodology were identified
as: definition of the research question (particularly describing the
alternative diagnostic pathways), choice of reference standard, design
of the search strategy, quality assessment of the included studies and
the statistical methods for meta-analysis. Work continues to further
evaluate the comprehensiveness of protocol submissions against these
criteria in the sample of 45 first protocol submissions. Conclusions:
This tool is likely to be helpful for both authors and DTA ET protocol
assessors, making the DTA protocol development and assessment
process more transparent, efficient and robust.
P1A65
UK prioritisation methods for Cochrane Reviews
Bailey SE1 , Stamp RJ1 , Clarke M2
1
NIHR Evaluations, Trials and Studies Coordinating Centre, Southampton,
England; 2 Centre for Public Health, Queen’s University Belfast, Northern
Ireland
Background: The National Institute for Health Research (NIHR) is
a strong supporter of The Cochrane Collaboration in the UK. With a
planned investment of £21 million over five years up to March 2015, it
is the largest single contributor to the infrastructure costs of Cochrane
entities. However, despite this financial support, the potential review
workload is always likely to be greater than the resource available to
support it, creating the need to prioritise review topics. Objectives:
To share methods for prioritising review topics as reported to their
funders by UK-based Cochrane review groups. Methods: UK-based
Cochrane review groups submit an annual report of expenditure and
review production during each year. As part of this report, groups
are asked to provide details about their prioritisation methods. Their
answers have been grouped in to broadly consultative and analytical
approaches. Results: The most frequently cited approaches to
review topic prioritisation will be presented. This information should
complement the results of other work in The Cochrane Collaboration,
such as the projects commissioned by the Steering Group to develop
prioritisation methods, and may be valuable to Cochrane Review
Groups considering how best to utilise limited resource and capacity.
Conclusions: Prioritisation of review topics at title stage and for
updating is essential to ensure that limited resource and capacity
are used in ways that best address key questions faced by patients,
front-line professionals and by national decision-making bodies and
other policy customers in health care.
P1A66
Improving the participation of women from developing
countries in the Cochrane Collaboration: A pilot study
Koehlmoos TP1 , Nasser M2 , Kiwanuka S3
1 ICDDR, B, Bangladesh; 2 Peninsula Dental School, United Kingdom;
3 Makerere University School of Public Health, Uganda
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: There is increased interest around women’s involvement
in the Cochrane Collaboration (CC). Broader participation is one of the
principles of the Collaboration and increasing women’s involvement is
one way to achieve this goal. Objectives: We aimed to determine
the facilitators who are based in or hail from developing countries
(DC), who facilitate women getting involved with the Collaboration.
Methods: We developed a questionnaire and used snowball sampling
to survey women from developing countries involved in the Cochrane
Collaboration. Results: Nine women responded to our survey. They
were from Iran, Bangladesh, USA, Argentina and Uganda and are
based in the UK, USA, Bangladesh, Uganda and Iran. All of them were
authors of Cochrane reviews. Other roles that they have filled were:
centre staff, handsearcher, and referees. They have been involved with
the CC two to six years. Only one of them had an influential role in the
CC; however, the rest would be interested in taking a more influential
role in the CC. They became involved either through a local initiative
or another Cochrane contributor. The factors they saw as facilitating
greater women involvement included: (a) Increasing opportunities for
women to participate and network (e.g. engaging them in review
teams with more equal treatment) (b) Scholarship for women to
attend the Colloquium and other relevant events (c) Increasing training
opportunities (d) Providing opportunities at a local level and building
toward wider participation. (e) More long term and sustainable funding
opportunities in developing countries. Conclusions: This pilot study
provides a picture of the view of women from developing countries who
are widely involved in the Cochrane Collaboration. We recommend
that the Collaboration would more strategically involve these women
in planning wider participation in the Cochrane collaboration. We will
report the results from other women who will respond in October 2011.
P1A67
From Barcelona to Madrid: History and quality of update
reporting of Cochrane Reviews flagged as updates in 2003
and analysed for the Barcelona Colloquium
Bastian H1 , Clarke M2 , Doust J3 , Glasziou P4
1 National Center for Biotechnology Information, National Library of
Medicine, NIH, USA; 2 Professor/Director of all-Ireland Hub for Trials
Methodology Research, Centre for Public Health, Queen’s University
Belfast, Northern Ireland; 3 Professor of Clinical Epidemiology, Centre for
Research in Evidence Based Practice, Bond University, Australia; 4 Professor
of Evidence-Based Medicine, Director, Centre for Research in Evidence
Based Practice, Bond University, Australia
Background: Cochrane reviews are intended to be updated periodically, generally every two years. The Cochrane Handbook requires
that reports of updates help readers to quickly and clearly identify the
changes. Objectives: To assess the quality of update reporting and
map the updating history of Cochrane reviews flagged as updated
in 2003. Methods: Publication and updating histories of a cohort
of reviews with update flags in 2003 (reported on at the Barcelona
Colloquium) were compiled from The Cochrane Library (Issue 4, 2011)
and Pub Med. Analyses were done of the time to each update.
Results: Of 177 Cochrane reviews with update flags in 2003, 40 have
been updated only once (23%). Seven (4%) have been determined to
have stable evidence. If a review is assumed to be out-of-date after two
years, only three (1.7%) of the reviews have been up-to-date ever since
their original publication, and one of these was withdrawn in 2005.
Based on an updating interval of two years, the 177 reviews were out
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
67
of date for 45% of their publication lives. If the interval was three
years, the reviews were out of date for 28% of their lives; or 15% if the
interval was four years. The median time to an update was between
two and three years. A minority of the reviews reported all searches
(33%) or whether additional studies were included (39%) in each
update in their ‘What’s new’, history or notes section. Conclusions:
Even among Cochrane reviews that had been updated at least once,
most are not updated within two years or provide information in
accordance with the recommendations in the Cochrane Handbook. The
quality of reporting of the updating is poor.
P1A68
Report on The Cochrane Empty Review Project Consensus
Meeting: A proposal of empty review reporting guidelines
and an invitation for feedback
Montgomery P1 , Yaffe J2 , Shepard L2 , Hopewell S3
1
University of Oxford, UK; 2 University of Utah, USA; 3 University of Oxford
and UK Cochrane Centre, UK
Background: There is currently limited guidance in the Cochrane
Handbook for the reporting of systematic reviews with no studies
eligible for inclusion. This project has previously reported considerable
inconsistency in the reporting of these so-called empty reviews
across reviews and review groups. The Cochrane Empty Review
Project convened an interdisciplinary consensus meeting of Cochrane
Collaboration contributors in June 2011. The purpose of the meeting
was to formally discuss the issues of empty reviews and to consider
guidelines for the reporting of these reviews. Objectives: The aim
of this presentation is to report the outcomes of The Cochrane
Empty Review Project Consensus Meeting, and to invite further
correspondence and feedback. Methods: Following the suggestions
of Moher et al 2010 for the development of reporting guidelines,
The Cochrane Empty Review Project identified the need for reporting
guidelines for reviews without included studies by: reviewing the
literature, identifying all empty reviews within the Cochrane Database
of Systematic Reviews (CDSR), describing these reviews, examining
how these reviews report implications for practice, and consulting with
other systematic review groups to find existing guidelines for reporting
empty reviews. This information was then presented for discussion at
a formal consensus meeting. Results: The key proceedings from the
June 2011 consensus meeting are presented in this oral presentation.
Topics include: possible recommendations for the incorporation of the
guidelines, the updating of empty reviews, the referencing of excluded
studies, and the reporting of the implications for practice and research
sections. Conclusions: The open discussion of Cochrane contributors
regarding these proposed recommendations is invited; preparatory to
the submission of the initial report and guidelines to the Cochrane
Collaboration and the Cochrane Handbook Committee.
P1B69
The Cochrane Empty Review Project: Prevalence and
characteristics of systematic reviews without included
studies in The Cochrane Database of Systematic Reviews
Shepard L1 , Yaffe J1 , Montgomery P2 , Hopewell S3
1 University of Utah, USA; 2 University of Oxford, UK; 3 UK Cochrane Centre
and University of Oxford, UK
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Little is known about systematic reviews that report no
included studies. These so-called ‘empty reviews’ have been primarily
discussed by clinicians and policy-makers expressing frustration with the
lack of implications for decision-making in these reviews. Objectives:
The Cochrane Empty Review Project sought to describe the incidence,
prevalence, and characteristics of systematic reviews without included
studies in the Cochrane Database of Systematic Reviews (CDSR).
Methods: Empty reviews within the CDSR as of 15 August 2010 were
identified, extracted, and coded for analysis. We examined the numbers
and proportions of empty reviews, date of first publication, and recency
of the last update by Cochrane Review Group (CRG). Number of
excluded studies, studies awaiting assessment, and on-going studies
listed within empty reviews were also examined. Results: The
Cochrane Empty Review Project identified 376 empty reviews in The
CDSR, representing 8.7% of all active reviews. Forty-five (86.5%) of
the 52 CRGs sustain at least one empty review, ranging from 0 to
35 (0 to 26.9%) per CRG. The average time since first publication
is 4.2 years (SD 3.4). These empty reviews were updated within an
average of 2.8 years (SD 2.2), with a range of 0 to 11 years since
update. The identified reviews listed an average of 9.6 excluded studies
(SD 14.5), but varied from 0 to 124 excluded studies listed. Empty
reviews typically contained few, if any, studies awaiting assessment or
on going studies, with respective averages of 0.1 (SD 0.68) and 0.14
(SD 0.49) per review. Conclusions: Reporting and updating of empty
reviews appears to vary widely. This may or may not be explained
by differences in editorial policy or practice across CRGs. Further
analysis is warranted and development of more explicit guidance for
the reporting of systematic reviews without included studies in The
CDSR is recommended.
P1B70
A comparative analysis of systematic reviews published
in Chinese academic journals and the Cochrane Library
Chen Y1 , Yao L2 , Liang F2 , Wang Q2 , Li J2 , Wei D2 , Xu H2
1 Evidence-Based Medicine Center of Lanzhou University, China; 2 School
of Basic Medical Sciences of Lanzhou University, China
Background: Should we conduct more than one systematic review for
the same topic or question? How many systematic reviews are similar
between Chinese systematic reviews and Cochrane systematic reviews?
Does it make sense for identified Cochrane systematic reviews that
some authors still conducted the same systematic review? Objectives:
To detect the degree of similarity of systematic reviews between
Chinese academic journals and the Cochrane Library. Methods: We
searched four of the main Chinese medical databases WANFANG,
China National Knowledge Infrastructure/Chinese Academic Journals
full text Database (CNKI), VIP (a full text database of China) and
China Biomedicine Database (CBM); using the term ‘systematic review’
or ‘meta analysis’ in the title from 1990 to 2010. Results: We
identified 2648 Chinese systematic reviews about healthcare. We
conducted a comparative analysis of these systematic reviews with
all Cochrane systematic reviews using the PICO method. We found
that about 20% of Chinese systematic reviews were very similar to
Cochrane systematic reviews. Most of them were published after the
corresponding Cochrane systematic review. Further investigation is
ongoing. Conclusions: We will draw a distribution map between
Chinese systematic reviews and Cochrane systematic reviews and
identify blank and duplicate systematic reviews.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
68
P1B71
Review or original article? The manuscript category
of systematic review and meta-analysis in high-impact
biomedical journals
Tam K1 , Tsai L2 , Kuo KN3 , Chen C3 , Lo H3
1 Department of Surgery and Evidence-based Medicine Center, Taipei
Medical University Hospital, Taiwan. Center of Evidence-based Medicine,
College of Medicine, Taipei Medical University, Taiwan; 2 Evidence-based
Medicine Center, Taipei Medical University Hospital, Taiwan; 3 Center of
Evidence-based Medicine, College of Medicine, Taipei Medical University,
Taiwan
Background: Unlike a narrative review, a systematic review involves
the application of scientific strategies, in ways that limit bias, to the
assembly and critical appraisal of all relevant studies that address a
specific clinical question. A meta-analysis is a type of systematic review
that uses a statistical strategy for assembling the results of several
studies into a single estimate. However, when an author submits
a systematic review and meta-analysis to journals, the manuscript
category between a review and original article is indistinct. Objectives:
To investigate the manuscript category of systematic reviews and
meta-analysis in biomedical journals. Methods: Biomedical journals
(impact factor >6) that consider systematic reviews and meta-analyses
in the field of clinical sciences for publication were included. The
Instructions to Authors of biomedical journals and the article category
printed on the front page of the literature were reviewed for evidence
of an editorial policy on the manuscript category. Results: 63 of
311 biomedical journals publish systematic reviews and meta-analyses
of clinical issues. In the Instructions to Authors, 4.76% classified a
systematic review and meta-analysis as an original article, 15.9% as
a review, 20.6% as an independent type of manuscript, and 58.7%
did not mention any policy on the article type for systematic review
and meta-analysis. For the article category posted at the front page of
the literature, 31.7% printed systematic reviews and meta-analyses as
an original article, 9.52% as a review, 4.76% as a meta-analysis, and
39.7% did not reveal the article type on the front page. Conclusions:
Most of the high-impact clinical biomedical journals did not mention
their policy on classification of systematic reviews and meta-analyses
in the Instructions to Authors. However, a relatively large proportion of
journals recognize a systematic review and meta-analysis as an original
article.
P1B72
Evaluation of data-synthesis in systematic reviews
of studies on outcome prediction models
van den Berg T1 , Heymans MW1 , Leone SS2 , Vergouw D3 , Hayden JA4 ,
Verhagen AP5 , de Vet HC1
1 Department of Epidemiology and Biostatistics and the EMGO Institute
for Health and Care Research, VU University Medical Centre, Netherlands;
2 Department of General Practice and the EMGO Institute for Health and
Care Research, VU University Medical Centre, Netherlands; 3 Department
of General Practice and the EMGO Institute for Health and Care Research,
VU University Medical Centre, Netherlands; 4 Department of Community
Health and Epidemiology, Dalhousie University, Canada; 5 Erasmus Medical
Centre, Department of General Practice, Netherlands
Background: Many prognostic models have been developed. Different
types of models, i.e. prognostic factor and outcome prediction models,
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
serve different purposes, and this should be reflected in how the results
of systematic prediction reviews are summarized. Objective: To
investigate how authors of reviews synthesize and report the results of
primary outcome prediction studies. Study selection: 41 systematic
reviews of outcome prediction studies in English listed in MEDLINE
between October 2005 and October 2009. Two review authors,
using standardized forms for all reviews, independently assessed
characteristics of the reviews and the primary studies that were
included. Results: The type of primary studies included (prognostic
factor or outcome prediction) was unclear in two-thirds of the reviews.
A minority of the reviews reported uni-variable or multi-variable
point estimates and measures of dispersion from the primary studies.
Moreover, the variables considered for outcome prediction model
development were often not reported, or were unclear. In most reviews
there was no information about model performance. Quantitative
analysis was performed in nine reviews; and 40 reviews assessed the
primary studies qualitatively. In both analyses a range of different
methods was used to present the results of outcome prediction studies.
Conclusion: Different methods are applied to synthesize primary study
results. Quantitative analysis is rarely performed, the nomenclature and
reporting of primary studies is poor, and performance parameters of the
outcome prediction models are rarely mentioned. Therefore, there is
much room for improvement in reviews of outcome prediction studies.
P1B73
Editorial policies of hematology and oncology journals: an
analysis of author instructions
Meerpohl JJ1 , Wolff RF2 , Grambauer N3 , Antes G4 , von Elm E5
1
German Cochrane Center, Institute of Medical Biometry and Medical
Statistics & Pediatric Hematology & Oncology, Center for Pediatrics
and Adolescent Medicine, University Medical Center, Freiburg, Germany;
2 Kleijnen Systematic Reviews, York, UK; 3 Institute of Medical Biometry and
Medical Statistics, University Medical Center Freiburg, Germany; 4 German
Cochrane Center, Institute of Medical Biometry and Medical Statistics,
University Medical Center, Freiburg, Germany; 5 Cochrane Switzerland,
CHUV and University of Lausanne, Switzerland & German Cochrane
Center, Institute of Medical Biometry and Medical Statistics, University
Medical Center, Freiburg, Germany
Background: Complete and accurate reporting is a prerequisite for
inclusion of original studies in systematic reviews. The continued
debate about ethics and quality of biomedical publishing has led
to recommendations aiming to enhance reporting quality and good
publication practice. However, it is unclear to what extent these
recommendations have been implemented by specialty journals.
Objectives: We analyzed whether specific recommendations were
included in author instructions of hematology and oncology journals and
whether their endorsement was associated with journal characteristics.
Methods: We identified 167 unique journals in the subject categories
‘Hematology’ and ‘Oncology’ of the 2008 Journal Citation Report
publishing original research articles. From March to May 2010
we extracted information regarding endorsement of the Uniform
Requirements for Manuscripts (URM) of the ICMJE and of five major
reporting guidelines such as the CONSORT statement, disclosure of
conflicts of interest and trial registration from the journals’ online
author instructions. Two investigators collected data independently.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
69
We calculated a global mixed model for endorsement of editorial
policies accounting for heterogeneity by publisher, followed by separate
multivariate logistic regression models. Results: The URM were
mentioned in the author instructions of 89 journals (53.3%). Endorsement of reporting guidelines was low: CONSORT was referred to
most frequently (42 journals; 25.1%); each of the other four reporting
guidelines were mentioned in less than 10% of author instructions.
132 journals (79%) explicitly required authors to disclose conflicts of
interest, and 55 (32.9%) recommended or required trial registration.
After model selection, only journal impact factor had a highly significant
positive association with endorsement of these policies in the individual
models (see Table 1). Conclusion: Author instructions of specialty
journals serving the hematology and oncology research community
do not yet reflect specific recommendations aiming to enhance the
quality of biomedical publications. Journal impact factor is positively
associated with endorsement of these policies.
since publication of the original review, 41 new eligible studies were
identified in addition to the eight included in the original review.
When comparing endorsing and non-endorsing journals, items such as
sequence generation, allocation concealment and participant flow were
reported better in those endorsing CONSORT. Further details of the
comparison between endorsers and non-endorsers as well as between
trials published before and after CONSORT publication (both 1996 and
2001) will be presented. Conclusions: This review will provide further
evidence on whether CONSORT is effective at improving the reporting
of RCTs. This information will be helpful to authors, peer-reviewers
and journal editors in helping to decide whether to endorse CONSORT.
P1B75
A case report on updating Cochrane Reviews
Nicola Maayan
Enhance Reviews Ltd, UK
P1B74
The influence of CONSORT on the quality of reporting
of RCTs: An updated systematic review
Turner L1 , Moher D1 , Shamseer L1 , Weeks L1 , Peters J1 , Plint A2 , Altman D3 ,
Schulz K4
1 Ottawa Hospital Research Institute, Ottawa, Canada; 2 Children’s Hospital
of Eastern Ontario, Canada; 3 University of Oxford, UK; 4 FHI, USA
Background: The Consolidated Standards of Reporting Trials
(CONSORT) statement was developed in response to concerns about
the quality of reporting of randomized controlled trials (RCTs). It is
an evidence-based minimum set of recommendations for reporting
RCTs, intended to facilitate complete and transparent reporting and
aid in critical appraisal and interpretation. A 2006 systematic review
examining the effectiveness of CONSORT for improving the reporting of
RCTs in endorsing journals (i.e. those which, at minimum, recommend
that authors use CONSORT), found CONSORT endorsement to be
associated with better quality of reporting, despite poor methodology of
some included studies. Five years on from the publication of that review,
an update is needed. Objective: To update the systematic review
of CONSORT effectiveness by Plint et al. Methods: Conventional
systematic review methods employed in the original review were
followed. The search for new comparative studies evaluating the
quality of reporting of RCTs spanned August 2005 to March 2010. Two
reviewers independently screened studies for eligibility; data extraction
and study validity assessments were conducted by a single reviewer
and verified by a second reviewer. Results: In the five year period
Background: Updating reviews is a challenge faced by all review
groups within The Collaboration. Many groups have tried to create
mechanisms to help authors update their review, but the challenge
of updating increases as more Cochrane reviews are published every
year. Objectives: We report on a partnership between the Cochrane
Schizophrenia Group (CSzG) and a small company specialising in
systematic reviews (Enhance Reviews Ltd) on a project to update
various reviews. Methods: The CSzG set up a list of priority reviews
to be updated. Our team worked on tasks involved in updating the
review, in consultation with the authors. Results: Six reviews were
updated over seven months; and the updates differed on their level of
complexity, with some more challenging than others. One challenge
involved updating changes in the methods of systematic reviews, in
particular new areas such as: Risk of Bias, Summary of Findings and
GRADE. A further challenge was that it was difficult in some reviews
to measure the amount of work involved; sometimes it was necessary
to return to the original studies to complete tasks, such as updating
the Risk of Bias Tables to the new format. The project allowed us
to calculate the amount of work required according to the number
of included studies and tasks involved. Conclusions: Given these
challenges, and based on our experiences, we suggest that there should
be an evaluation process before embarking on an update to estimate
the scale of the tasks involved. Updates should involve partnership
with authors so they can indicate which tasks would be most helpful to
them in supporting the update of their review, and allow them to deal
with specific issues in their field, such as the discussion. We encourage
editorial teams to act as an intermediary between authors and those
performing the update.
Table 1 (P1A73). Association of journal impact factor with editorial policies.
Editorial policy
URM
Any reporting guideline
Disclosure of COI
Trial registration
Odds ratio per change in log IF
2.01
7.17
3.25
4.51
95% CI
[1.28; 3.32]
[3.32; 15.52]
[1.80; 6.41]
[1.87; 11.79]
Odds ratio for change of IF from 1 to 2
1.62
3.92
2.27
2.84
95% CI
[1.18; 2.30]
[2.30; 6.69]
[1.50; 3.63]
[1.54; 5.53]
p-value
<0.01
<0.001
<0.001
<0.01
COI – Conflicts of Interest; JIF – Journal Impact Factor; URM – Uniform Requirements for Manuscripts of ICMJE.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
70
P1B76
When should systematic reviews of effects restrict studies
to those conducted in low- and middle-income countries:
A proposal?
Munabi-Babigumira S1 , Lewin S1 , Glenton C1 , Oxman A1
1
Norwegian Knowledge Centre for the Health Service, Norway
Background: The Norwegian satellite of the Cochrane Effective
Practice and Organisation of Care (EPOC) group produces reviews that
address health system questions relevant to low- and middle- income
countries (LMICs). It is not always clear if review authors should restrict
searches to LMICs or focus on the global evidence. Objectives: To
suggest issues to consider when deciding whether or not to restrict
systematic reviews to LMICs. Methods: Reasons to restrict the focus
of a review to LMICs were developed through discussions among the
EPOC editorial team. Results: Suggested reasons to restrict a review
to LMIC are: i) The intervention(s) and/or problem that the review
addresses is highly relevant in LMIC and of little or no relevance in
high-income countries (HICs), and; ii) There are compelling reasons
to believe that the problem or the intervention(s) or the outcomes
of interest are different in LMICs and HICs, and the intervention(s)
would be expected to function differently in LMICs and HICs, so that the
evidence would be unlikely to be transferable between LMICs and HICs.
Reasons that do not adequately justify restricting the focus of an EPOC
review to studies from LMICs include: i) The interventions are common
in LMICs and uncommon in HICs, but there are not compelling reasons
to believe that the problem or the interventions or the outcomes of
interest are different in LMICs and HICs, and the intervention(s) would
not be expected to function differently in LMICs and HICs; and ii) The
researchers or those who commissioned the review are particularly
interested in evidence from LMICs. Conclusions: When proposing
a review that is restricted to evidence from LMIC, authors need to
justify why. Further work is needed to examine whether there are
differences in effects for similar interventions implemented in LMICs
and HICs.
P1B77
Using program theory to understand heterogeneity
in systematic effectiveness reviews
Cargo M1 , Stankov I1 , Thomas J2 , Hannes K3 , Saini M4 , Mayo-Wilson E5 ,
Rogers P6
1 University of South Australia; 2 Evidence-based Policy and Practice
Information Centre; 3 K.U. Leuven; 4 University of Toronto; 5 Oxford
University; 6 Royal Melbourne Institute of Technology
Background: Program theory can provide reviewers with a structured
approach for outlining the action model(s) (i.e., characteristics of
implementer, implementing system, ecological context, partnering
organisations, participants, intervention), change model(s) describing
the intended sequence of proximal to distal outcomes in participants,
and identifying relevant process evaluation variables (i.e., fidelity, dose).
Integrating this information can strengthen attributions of effectiveness
and inform knowledge translation. Objectives: To assess the extent
to which and how reviewers account for aspects of the action model,
process evaluation measures and change model(s) underpinning their
reviews. Methods: Twenty-seven systematic effectiveness reviews
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
on social welfare, criminal justice and education published by the
Campbell Collaboration (some co-registered with Cochrane) with
reported outcomes on children or youth were reviewed. An assessment
tool, informed by a program theory approach, was developed to
assess elements of the action model, change model and process
evaluation measures within reviews. Two raters completed extractions
using a 7-point rating scale and noting comments in EPPI-Reviewer
Version 4.0. Results: Within the action model, the majority of
reviews accounted for one or more aspects of the intervention
(i.e., format) and participant characteristics with aspects of the
implementing organisation, implementer and partnerships considered
less often. Overall, action models could have been more clearly
defined. Intervention dose was the most common process evaluation
measure. For the change model, most reviews were ‘black box’ with
change processes remaining undifferentiated. Reviewers expressed
interest in identifying active ingredients of interventions and integrating
qualitative evidence. This presentation will feature exemplars to
illustrate how program theory can be integrated into reviews to
understand heterogeneity. Conclusions: Program theory can add
value to reviews by informing the review objectives, inclusion/exclusion
criteria, search strategies, and the operationalisation/analysis of
variables related to the action model(s), change model(s) and process
evaluation. Resources, training, and guidance are needed to support
reviewers.
P1B78
Analysing the growth in impact of the Cochrane Database
of Systematic Reviews (CDSR)
Pentesco-Gilbert D1 , Stewart G1
1
Wiley-Blackwell, John Wiley & Sons, Chichester, UK
Background: The Impact Factor is calculated and published as part
of the Journal Citation Report by a commercial company, Thompson
ISI, each year in June. The Cochrane Database of Systematic Reviews
(CDSR) received its first Impact Factor in 2007. In a recent article in
The Lancet Richard Horton noted, ‘In 2007, CDSR had an impact factor
of 4.65 (14th place in the general medicine category, the same group
as The Lancet ). In 2008, it rose to 5.18 (12th ). In 2009, it rose again
to 5.65 (11th ). The CDSR is improving its quality year-on-year.’(1)
Methods: Citation data taken from the ISI Web of Science covering
the years 2007 to 2010 (data for 2010 will not be available until June
2011) will be analyzed in order to identify which individual articles,
subject areas and review groups have contributed most to the rise
in the Impact Factor of the CDSR. Results: Can the rise in impact
factor be attributed to a small number of review groups or has each
individual review group contributed? Has the CDSR impact factor
relied on a small number of highly cited articles? Should highly cited
articles influence the topic areas where future review articles should
be commissioned? Conclusions: Conclusions will be drawn from
the analysis once the data is made available from Thompson ISI in
June 2011.
Reference
1. Horton R. (2010) Offline: what about the football? The Lancet, 376
(9735), 80.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
71
P1B79
Does one click access matter? Global use of The Cochrane
Review: 2010 analysis
Global health and equity
Day: 20 Oct (P1B81–P1B102)
Wilkins S1 , Stewart G2 , Pentesco-Gilbert D2
1 Wiley-Blackwell, Oxford, UK; 2 Wiley-Blackwell, Chichester, UK
P1B81
Ivermectin and benzimidazoles for treating strongyloides
infection
Background: Evidence Aid, The Cochrane Collaboration and WileyBlackwell first introduced the free, one-click access policy for Cochrane
Reviews in 2007. This was for countries in the World Bank list of
low-income economies. The initiative was extended in September
2010, allowing direct no-cost access to people in a greater number
of countries all over the world. People in more than 100 countries,
in some of the most impoverished parts of the world, now have
free access to The Cochrane Library through the Internet. Methods:
The number of unique visits to www.thecochranelibrary.com, number
of visits by country, and most accessed Cochrane Reviews on Wiley
InterScience/Wiley Online Library (from August 2010) for 2010 were
taken from the Wiley Usage Data Warehouse, an analysis tool that
stores and tracks usage data. Another set of usage data was obtained
from all other licensed providers of The Cochrane Library, showing the
number of abstracts and full text articles that were accessed during
2010. A subset of data will be collected and interrogated for low- and
middle-income countries with one-click access covering access denied
data and the comparative access periods (From switch-on September
2010). Results: Data taken from all providers of The Cochrane Library
show that during 2010 use increased. Full text use on the Wiley
platform increased by 17%. Use by the World Bank list of low-income
economies increased by 74% from 2009 data. Countries in this
programme with the most downloads included: Vietnam, Pakistan
(free access through Evidence Aid after flood disaster), Ecuador,
Nigeria, and Kenya. Conclusions: Global use of The Cochrane Library
continues to grow. The initiative for access for low- and middle-income
countries is a major development, above and beyond the provision
of knowledge for natural disasters. This positions The Cochrane
Collaboration, Wiley-Blackwell and Evidence Aid at the forefront of
information provision.
Echevarria J1 , Gotuzzo E1 , Samalvides F1 , Terashima A1
1 Instituto de Medicina Tropical Alexander von Humboldt-Lima, Peru
P1B80
How to involve consumers: A guide for Cochrane Editors
McIlwain C1
Consumer Network
1 Cochrane
Objectives: A new guide is available to help editors use consumer
input in systematic reviews. You will learn techniques for: • locating
consumers with review-specific interests, • using available resources
to facilitate involvement, • encouraging repeat participation by
meeting consumer needs, and • reporting on consumer involvement.
Description: Funding organizations have been making a major
commitment to consumer involvement in research; yet the best methods
to find and include consumer input are still unclear. Some materials
are available for training and involving consumers, but none pertain
specifically to the Cochrane review process. This presentation will
use a new Cochrane guide ‘How to Involve Consumers in Cochrane
Systematic Reviews’ to chart your way through consumer participation
in research.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Strongyloidiasis is a common and globally distributed
infection caused by strongyloides stercoralis. This infection is capable
of transforming into a serious and sometimes fatal illness in people
with comorbilities (malnutrition, immunocompromised, etc). The
benzimidazoles and ivermectin are the only two types of drugs available
for strongyloidiasis. Objectives: To evaluate antistrongyloidal drugs
for treating strongyloides infection. Methods: We searched the
Cochrane Infectious Diseases Group Specialized Register (July 2010);
the Cochrane Central Register of Controlled Trials (CENTRAL), published
in The Cochrane Library; MEDLINE (1966 to July 2010); EMBASE
(1980 to July 2010); LILACS (July 2010). We also searched the
metaRegister of Controlled Trials (mRCT) using ‘stronglyloid∗ ’ as a
search term. Randomized controlled trials of benzimidazoles (including
thiabendazole and albendazole) versus ivermectin for treating strongyloides stercoralis were included. Results: Ten trials (992 participants)
met the inclusion criteria. Seven trials compared benzimidazoles
vs Ivermectin, showing an effect in favour of Ivermectin 87%
vs 60% (RR 1.40, 95% CI 1.11 to 1.78; I2 = 87%, Chi2
p-value =< 0.00001. There was no association of failure of therapy
with type of benzimidazole. There were several adverse clinical
events reported, but no deaths, after administration of benzimidazoles
and ivermectin. The most serious adverse event reported in the
thiabendazole group was nausea and disorientation. There were
more adverse events in the benzimidazole group (overall RR: 1.44,
95% CI 1.23 to 1.70, I2 = 84%, Chi2 = 48.98). Conclusions: Both
ivermectin and benzimidazoles are efficacious for treating strongyloidiasis,
but ivermectin was superior in terms of efficacy and adverse events.
Using ivermectin as a single doses improves adherence. It is not
possible to conclude whether ivermectin or benzimidazoles has an
effect on morbidity caused by infection. Thiabendazole might be less
tolerated than ivermectin or other benzimidazoles based on serious
adverse events (disorientation and nausea).
P1B82
Sex- and gender-based analyses in Cochrane Reviews:
Methods and lessons learned
Ueffing E1 , Pardo Pardo J1 , Welch V1 , Borkhoff CM1 , Kristjansson E1 ,
Petticrew M2 , Tugwell P3
1 Campbell and Cochrane Equity Methods Group, Canada; 2 Campbell
and Cochrane Equity Methods Group and London School of Hygiene and
Tropical Medicine, UK; 3 Campbell and Cochrane Equity Methods Group,
Dept of Medicine, Dept of Epidemiology and Community Medicine, Faculty
of Medicine, University of Ottawa, Canada
Background: The Canadian Institutes of Health Research (CIHR)
notes that ‘Accounting for sex and gender in health research has
the potential to make health research more just, more rigorous and
more useful’. As awareness around equity and justice issues grows,
formal sex and gender policies have been established. For example,
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
72
CIHR-funded researchers must address sex and gender, as appropriate,
when designing their research while the American National Institutes
of Health (NIH) guidelines require funded clinical research to examine
differential effects of interventions by sex/gender and race/ethnicity.
Objectives: To report on findings from studies evaluating the degree
to which systematic reviews have addressed sex and gender. To
describe how methods developed by international equity researchers
can be used for sex- and gender-based reporting and analyses.
Methods: Recently, teams have published guidance on how to
address equity and the social determinants of health in systematic
reviews. These methods include logic models, subgroup analyses,
and process evaluations; sex and gender considerations are crucial
for each of these methods. Results: Specific methods for including
sex and gender considerations in systematic reviews will be presented,
with examples from international research reports, national guideline
panels, and published reviews. Conclusions: Sex- and gender-based
analyses can help researchers to ensure that: 1) systematic reviews are
suitably designed; 2) results from systematic reviews are available for
men and women, males and females where appropriate; and 3) that
policies and decisions based on systematic review evidence are just
and relevant.
P1B83
Methodological quality of randomized controlled trials on
gastric cancer treated with traditional Chinese medicine
published in Chinese journals
LI H1 , HU Q1 , LIU W1 , MA B1
1 Evidence-Based Medicine Center, Institute of Traditional Chinese and
Western Medicine, School of Basic Medical Sciences, Lanzhou University,
Lanzhou 730000, China
Background: Randomized controlled trial (RCTs) on gastric cancer
treated with Traditional Chinese Medicine (TCM) have been published
in large numbers. However, methodological quality of these RCTs were
unknown. Objectives: The purpose was to assess methodological
quality of RCTs on gastric cancer treated with TCM. Methods:
Chinese Biomedicine Literature Database (CBM), Chinese Scientific
Journal Full-text Database (CSJD), Chinese Journal Full-text Database
(CJFD), and Wangfa Database were performed in the end of the Dec.
2010. The search terms included ‘Traditional Chinese Medicine’ and
‘Chinese herbs’ (Mesh or EMTREE), ‘Gastric Cancer’, ‘Stomach Cancer’
and ‘Stomach Neoplasms’, (Mesh or EMTREE). Besides, Google were
searched and the range of snowballing method was used. The Cochrane
Handbook 5.0.1 was used to evaluate the methodological quality of
included studies. Results: Totally 102 RCTs were identified and
published in 72 different Chinese Journals. Less half (41.67%) were
published in specialty journals. Less one-third (21.57%) were published
in journals cited by Chinese Science Citation Database (CSCD). The
sample of mostly trials (91.18%) less than 100 cases. Only 52.95% and
11.76% trials reported the specific inclusion and diagnostic criteria,
88.24% did not reported exclusion criteria. All of them did not register
in any clinical trials registry platform. Only 17 studies reported the
correct randomization method. All trials did not report allocation
concealment. Only 3 studies reported blinding, but no one reported
whose were blinded. Statistic mistakes appeared in one-third (29.3%)
studies. Conclusions: RCTs on TCM were published in large number
in Chinese Journals in recently years, but the quality of these review
were worrying. The information reported from these RCTs were not
comprehensive and even some were mistake. It did not only provide
evidence for clinicians but also misled them. Hence, the most urgent
and important thing is to improve the quality of RCTs on TCM, not
accelerate the quantity in China.
P1B84
Financial health equity. Intervention for balance and
financial stability of institutions providing national health,
health promoters and insurers
Kadar I1
1 Professor
of epidemiology and health care systems, Antonio Narino
University, Colombia
Background: The quality of health systems is measured and influenced
by limiting factors such as access, timeliness, coverage and equity in
service to the population. France is known for its good health system,
as the country spent $ 4056 (USD) per year per person on health,
which is equivalent to 11.1% of gross domestic product (GDP); this
allocation of fundsmay have contributed to France having the highest
life expectancy with age 81. In contrast, the Republic of Congo
invested just $ 42 (USD) per year per person, corresponding to 2.1%
of GDP, and the population has a life expectancy of only 54 years
(WHO 2006, Table 1). These investments constitute a limit for health
management. Developing countries have a very limited amount of
resources to establish efficient and effective health care systems.
Objectives: • achieve efficient cost-effective management. • reward
healthcare companies who (a) work toward prevention and reduction
of prevalent diseases that generate high costs, and (b) reduce spending
generated by the advanced stages of preventable diseases. Methods:
• develop standardized guidelines for the management of diseases with
evidence-based scientific rigor and at an acceptable cost. • develop
a monitoring system for health care promotion. • develop a method
of monitoring insurance companies’ and health service institutions’
(e.g. hospitals) use of disease prevention and control of risk factors
Analyze the results including financial implications to generate internal
policies. Results: Building financial stability for all companies that
comply with health prevention policies, delivering a budget increase
for those healthcare promoting companies whose load of high-cost
patients exceeds the standard deviation and decrease the budget for
Table 1 (P1B84).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
73
those who have reduced load. This will reward only those companies
who implement an effective intervention and prevention procedure that
has reduced the cost of these diseases over time.
P1B85
Agreement and alignment: Current status
of clinical practice guidelines within the Southern
African Development Community
Kredo T1 , Gerritsen A2 , van Heerden J3 , Conway S4 , Siegfried N1
1 South African Cochrane Centre, South Africa; 2 Epi Result, South africa;
3
Benguela health, South Africa; 4 Southern African Regional Programme
for Access to Medicines, South Africa
Background: Reducing the burden of disease relies on availability of
evidence-based clinical practice guidelines (CPGs). There is limited data
on availability, quality and content of guidelines within the Southern
African Development Community (SADC). This evaluation aims to
address this gap in knowledge and to provide recommendations
for regional guideline development. Methods: We prioritised five
diseases: HIV in adults, malaria in children and adults, pre-eclampsia,
diarrhoea in children and hypertension in primary care. A comprehensive
electronic search, supported by email contact with SADC Ministries
of Health was used to locate guidelines. The AGREE II tool was
applied by independent reviewers to evaluate 6 quality domains
reporting the guideline development process. Individual domains were
scored and percentages calculated. Alignment of the evidence-base
of the guidelines was evaluated by comparing content with key
recommendations from accepted reference guidelines, identified with
a content expert, and percentage scores were calculated. Results:
We identified 30 guidelines from 13 countries, between June and
October 2010. Publication dates ranged from 2003 to 2010. Overall
the ‘scope and purpose’ and ‘clarity and presentation’ domains of
the AGREE II instrument scored highest, median 58% (range 19–92)
and 83% (range 17–100). ‘Stakeholder involvement’ followed with
median 39% (range 6–75). ‘Applicability’, ‘rigour of development’
and ‘editorial independence’ scored poorly, all below 25%. Alignment
with evidence was variable across member states, the lowest scores
occurring in older guidelines or where the guideline being evaluated was
part of broader primary healthcare CPG rather than a disease-specific
guideline. Conclusion: This review identified quality gaps and variable
alignment with best evidence in available guidelines within SADC for
five priority diseases. Future guideline development processes within
SADC should better adhere to global reporting norms requiring broader
consultation of stakeholders and transparency of process. A regional
guideline support committee could harness local capacity to support
context appropriate guideline development.
P1B86
Neglected tropical diseases: Are global policies based
on reliable evidence?
Nagpal S1 , Garner P1
1 Cochrane Infectious Diseases Group, Liverpool School of Tropical
Medicine, UK
Background: Neglected Tropical Diseases is a term that has appeared
in the last 20 years, and is attracting global donor funds to implement
big programmes in tropical countries. With expanding investment,
it is particularly important that policies are informed by reliable
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
research. Objectives: To analyse the evolution of the term ‘neglected
tropical diseases’ through bibliographic analysis; and to assess whether
policies promoted for investment are underpinned by reliable syntheses
of research. Methods: We retrieved all articles on MEDLINE to
April 2011 containing the words ‘neglected tropical disease(s)’ in the
title or abstract. We used content analysis to map the concept,
the organisations and personalities involved in its evolution, and the
diseases it includes. We examined the percentage of articles that
contained research data. Finally, we took the top ten cited articles
in this field, summarised the policies and strategies that were being
advocated, and if reliable research evidence (systematic reviews or
trials) is available for each, and if it is cited. Results: ‘Neglected
tropical diseases’ is a term that was coined in 2005, and refers to
tropical infections once the ‘big three-malaria, tuberculosis and HIV’
have been taken out. The first publications with the mention of NTDs
appeared in 2005 and 2006, since then there has been a steady
increase in the number of articles up to 83 in 2010. We are currently
analysing the top 10 cited policy papers against existing reliable
research evidence. Conclusions: We will present an assessment of
the evidence-base behind current intervention strategies for neglected
tropical diseases, and provide an agenda for further synthesises and
primary research.
P1B87
High 5s project. Hospital survey on patient safety culture
results from Germany
Kolbe M1 , Lessing C1 , Fishman L2 , Renner D2 , Thomeczek C2
1 Institut for Patient Safety, University of Bonn; 2 German Coalition for
Patient Safety
Background: In the World Health Organisation initiated High 5s
project a multi-national learning community has set itself the goal
of reducing three serious patient safety problems by implementing
standardized operating protocols (SOP). One component within the
scope of the project is to conduct a survey on patient safety culture
in participating hospitals on a voluntary basis. Method: The present
study is based upon the Hospital Survey on Patient Safety Culture
(HSOPS) of the Agency for Healthcare Research and Quality. The
survey was validated with more than 1,400 hospital employees from
21 hospitals across the US. For the realisation of the survey in German
hospitals within the High 5s Project, the HSOPS was translated into
German and adapted according to national data protection regulations.
The statistical evaluation is carried out by the research group Westat.
In the scope of the High 5s Project, the survey is to be conducted
at two points in time. Results: In the first inquiry period, the
HSOPS was conducted at some participating hospitals. The results
of the HSOPS grouped according to the 12 safety culture dimensions
show that Communication Openness has the highest rate of positive
response and Organizational Learning - Continuous Improvement is
rated lowest. Conclusions: About a third of the participating hospitals
conducted the survey in spite of the higher workload for hospital staff.
Based on the safety culture dimensions, certain organisational factors
concerning patient safety can be described in addition to the High 5s
SOP indicators. First trends are apparent as measured by the rate
of positive response. The second survey period will demonstrate if
conclusions can be reached regarding a change in the perspectives of
patient safety by employees as a result of implementing the SOP.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
74
P1B88
Cutaneous Leishmaniasis International Observatory:
An initiative for Evidence-based Global Health
González U1 , Ruiz-Baqués A2
1 Research Unit for Evidence-based Dermatology, Hospital Plató, Barcelona,
Spain; 2 Topping Medical Research, UAB Research Park, Barcelona, Spain
Background: Cutaneous Leishmaniasis (CL) is a neglected mosquitoborne protozoan disfiguring and stigmatizing infection that causes a
great suffer of the patients and their families. It is a global increasing
public health problem but still does not receive the necessary attention.
It is a disease that is present in 82 countries and its incidence is
estimated at 10 million of affected people. An international team has
developed with the support of the WHO, three Cochrane systematic
reviews focused on the role of control strategies for CL. Objective: To
develop a Cutaneous Leishmaniasis International Observatory (CLIO),
as a non-for-profit model of knowledge management, based on the
scientific evidence and for improving the lives of people with CL.
Methods: Our main need is to seek contacts, ideas and interest
of institutions, organizations, companies and individuals to lead and
develop this action and to help complete the project in the best
conditions. Of particular interest is to periodically maintain and actively
disseminate a comprehensive and rigorous analysis and synthesis of
the information derived from updated Cochrane reviews. Results:
We have collected the opinion and support of different people and
organizations. We have developed a strategic plan for the commitment
of CLIO to reach real and significant impact in guiding and promoting
practice, policy decisions and in helping to identify research priorities
around the world. We want healthcare decisions in CL to be informed
by reliable, up-to-date, scientifically defensible and relevant evidence.
Conclusions: The major interests of CLIO are the update of the
information and to develop networking activities and communication
actions for actively disseminate what is best known about the effects
of control interventions in CL.
P1B89
The role of the medical information database in managing
the East Japan disaster
Nango E1 , Naito T2 , Toyoshima Y3 , Yoshimoto H4 , Nakayama T5
1 Tokyo-kita Social Insurance Hospital, Japan; 2 Fukuoka Dental College,
Japan; 3 The Dai-ichi Life Insurance Company, Japan; 4 Mie University
Graduate School of Medicine, Japan; 5 Kyoto University Graduate School of
Medicine, Japan
Background: On 11 March 2011, a massive earthquake measuring 9.0
on the Richter scale hit northeastern Japan, triggering a 10-meter-high
tsunami that engulfed the northeast coast of Japan. The number
of deaths has reached 14,435 and the number of missing persons
has been estimated at 11,601 (as of 27 April 2011). The total
number of evacuees approximately 335,000 (as of 20 March 2011).
There is concern about health conditions of the evacuees due to
the insufficient infrastructure, medical services, lack of privacy and
anxiety about the future. Medical support teams are also facing
unusual medical demands such as infectious diseases, cardiovascular
diseases, trauma, post-traumatic stress disorder and radiation-related
injuries. Objectives: To define what types of medical information are
needed for the workforces involved in this disaster and to improve the
dissemination of information. Methods: We searched the medical
information services that had been provided via the internet for this
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
disaster and counted the number of people who were accessing the
databases. Results: Cochrane Evidence Aid for earthquakes started
to provide service on 12 March 2011. A Japanese translation of their
information was prepared by Kyoto University School of Public Health
and publicized on 13 March. The number of pages accessed increased
sharply after the provision of the Japanese translation and the notice
by an internet discussion group for medical personnel. UpToDate and
DynaMed started to provide free access to Japanese medical personnel
on 14 March. The most frequently accessed content was ‘Treatment
of radiation injury in the adult’ provided by UpToDate. Conclusions:
Although there are many useful evidence-based databases, almost all
information is written in English, and is hard to use for non-native
users of English. Translation and localization of databases and their
dissemination are therefore also considered to be important factors for
the timely provision of information.
P1B90
A systematic review identifies substantial heterogeneity
of studies evaluating effects of rural-to-urban migration
on cardiovascular risk factors in low and middle income
countries
Hernandez AV1 , Pasupuleti V1 , Deshpande A1 , Bernabe-Ortiz A2 ,
Miranda JJ2
1
Cleveland Clinic, Ohio, USA; 2 Universidad Peruana Cayetano Heredia,
Lima, Peru
Background: Limited information is available about the effects of
rural-to-urban within-country migration on cardiovascular (CV) risk
factors in low and middle income countries (LMIC). Objectives:
We performed a systematic review of studies evaluating these
effects and having rural and/or urban control groups. Methods:
Two teams of investigators independently searched observational
studies published between 1960 and June 2010. Studies evaluating
international migration or those from developed countries were
excluded. Three investigators extracted the information stratified by
gender. We captured information on 17 known CV risk factors. Results:
Eighteen studies (n = 58,536) were included. Studies were highly
heterogeneous with respect to age, study design, sampling frame
for migrants, years of urban exposure for migrants, and reported CV
risk factors. Fourteen studies were cross-sectional; only six studies
adjusted for confounders. In migrants, commonly reported CV risk
factors – systolic and diastolic blood pressure, body mass index, obesity,
total cholesterol, and LDL – were usually higher or more frequent when
compared to the rural group and usually lower or less frequent when
compared to the urban group. This gradient was usually present
in both males and females separately. Anthropometric (waist-to-hip
ratio, hip/waist circumference, triceps skinfolds) and metabolic (fasting
glucose and insulin, insulin resistance) risk factors usually followed
the same gradient, but conclusions are weak due to the paucity of
information. Hypertension, HDL, and other uncommonly reported risk
factors such as fibrinogen and C-reactive protein did not follow any
pattern among groups. Conclusion: In LMIC, studies investigating the
effect of rural-to-urban within-country migration on CV risk factors are
heterogeneous. A gradient, higher or more frequent in migrants than in
the rural groups but lower or less frequent than the urban groups, was
observed in most but not all CV risk factors. Such gradients may or may
not be associated to differential CV events and long-term evaluations
remain necessary.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
75
P1B91
Integrating prevention of mother-to-child HIV
transmission programmes to improve uptake:
A systematic review
Tudor Car L1 , Van Velthoven MH1 , Brusamento S1 , Elmoniry H1 , Car J1 ,
Majeed A1 , Tugwell P2 , Welch V2 , Marusic A3 , Atun R4
1 Department of Primary Care and Public Health, School of Public Health,
Imperial College London, UK; 2 Centre for Global Health, University
of Ottawa, Ottawa, Canada; 3 University of Split School of Medicine,
Department of in Research in Biomedicine and Health, Split, Croatia;
4 Imperial College Business School, Imperial College London, UK
Background: Prevention of mother-to-child transmission (PMTCT)
of HIV interventions can significantly reduce transmission rates,
responsible for more than 90% of HIV infections in children. The World
Health Organisation recommends integration of PMTCT programmes
with other healthcare services to increase access and improve uptake
of these interventions. Objectives: To assess the effect of integrated
PMTCT interventions compared to stand-alone or partially integrated
services on PMTCT uptake in low- and middle-income countries.
Methods: We searched for experimental, quasi-experimental and
controlled observational studies in any language in 21 databases and
grey literature sources. We also scanned reference lists of included
articles and relevant systematic reviews. Results: Out of 28 654
citations retrieved, five studies met inclusion criteria (Figure 1; Table 1).
A cluster randomised controlled trial reported higher probability of
nevirapine uptake at the labour wards implementing HIV testing and
structured nevirapine adherence assessment. A stepped wedge study
showed marked improvement in antiretroviral therapy (ART) enrolment
and initiation in integrated care, but the median gestational age of
ART initiation, ART duration or 90 days ART retention did not differ
significantly. A retrospective cohort study reported no significant
difference either in the ART coverage or eight weeks of ART duration
before the delivery between integrated, proximal and distal partially
integrated care. Two before-and-after studies assessed the impact
of integration on HIV testing uptake in antenatal care. The first
study reported that significantly more women received information on
Figure 1 (P1B91). Study selection flow-diagram.
Citations identified by
search strategy
(n=28 654)
Duplicates removed
(n=4032)
Titles and abstracts
screened
(n=24 622)
Articles excluded
becuse not relevant
(n=24 596)
Full-text articles screened
(n=26)
Full-text articles
assessed for eligibility
(n=15)
Conference proceedings excluded
because of lack of data
(n=11)
Articles excluded
becuse not relevant or
second report
(n=10)
Articles included (one
second report) in the
systematic review
(n=5)
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
Before and
after
Stepped
wedged
Killam, 2010
Zambia
Jul-07–Jul-08
Study design
Kasenga 2009
Malawi
Jan-05–Dec-07
Author Year of
publication, Country
Study duration
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Women in ANC
Intervention:
17,619 Control:
13,917
Women in ANC
Intervention 01,063
Control 196
Participants and
sample size
Table 1 (P1B91). Characteristics of included studies.
• Women received ART in
ANC, twice per week.
at ANC
• HIV testing of women
• Women received ARV
prophylaxis in LW
• Safe delivery
• Women received infant
feeding counselling
• HIV testing and
counselling & services
integrated within ANC
Intervention
2. Percentage of HIV
positive women who
underwent to CD4 control
• Women referred for ART
from ANC to ART clinic
5. Percentage of women on
care at 90 days of follow
up
4. Percentage of women
receiving HAART as part
of PMTCT
3. Percentage of women
enrolled in ART clinic
1. Percentage of women
who tested in ANC
ANC
• HIV testing of women at
ANC
Partially integrated
• Women received ARV
prophylaxis in LW
• Safe delivery
• Women received infant
feeding counselling
1. Percentage of women
tested in ANC
• Non-integrated HIV
testing of women not
provided in ANC but in
separate VCT unit
2. Percentage of women
receiving ARV prophylaxis
in LW
ANC
Primary outcomes reported
Non-integrated
Comparison
5. 91% vs. 88%, p = 0.3
4. 14% vs. 33%, p < 0.001
3. 25% vs. 44%, p < 0.001
2. 85% vs. 85.1%, p = 0.98
1. 98.4 vs. 97.9% p = 0.51
Control vs. intervention:
2. 92% vs. 97%, p = 0.404
1. 53% vs. 79%, p < 0.001
Control vs. intervention:
Results
76
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Stinson, 2010
RSA
Jan–Dec 2005
Megazzini, 2010
Zambia
Oct-05–Jan-06
Author Year of
publication, Country
Study duration
Cohort
cRCT
Study design
Table 1 (P1B91). Continued.
Fully integrated
one point: 4,823
Partially integrated
proximal: 4,783
Partially integrated
distal: 5,381
Women in ANC
Intervention: 2,435
Control: 2,106
Women at LW
Participants and
sample size
• Women received ART
at the ANC, twice per
week.
• HIV counselling, testing
and ARV prophylaxis.
• Formal nevirapine
adherence assessment
• Nevirapine administration
as indicated
• HIV testing of women of
unknown serostatus in LW
• Women received ARV
prophylaxis at ANC
• HIV testing of women
at ANC
Intervention
• Women were referred to
receive ART to the HIV
clinic in another health
facility within 5 Km.
• HIV counselling, testing
and ARV prophylaxis.
Partially integrated distal:
2. Percentage of HIV positive
women who underwent to
CD4 control
• Women were referred to
receive ART to the HIV
clinic in the same health
facility
4. Percentage of HAART eligible
women receiving ARV
prophylaxis
3. Percentage of women
starting HAART as part of
PMTCT
1. Total: 88.1%
1. Percentage of women who
tested in ANC
• HIV counselling, testing
and ARV prophylaxis.
5. 26% vs. 28% vs. 29%
4. 48% vs. 47%
3. 55% vs.48% vs. 47%
2. Total: 97%
Integrated vs. proximal vs. distal:
• Intervention: from 42% to
54% (difference ranged
−10-+33%)
Partially integrated proximal: ANC
• No formal nevirapine
adherence assessment
• Women received ARV
prophylaxis in ANC
• Percentage of women/infants
receiving ARV prophylaxis
• Control: from 53% to 43%
(difference ranged −13-0%)
• HIV testing of women
at ANC
Baseline period/intervention
period:
Results
LW
Primary outcomes reported
Partially integrated
Comparison
77
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Before and
after
Study design
Intervention: 4,089
Control: 4,142
Women in ANC
Participants and
sample size
2. Percentage of women tested
• Counselling not provided in
ANC but in a separate
location by a counsellor
• Infants ARV prophylaxis in LW
• Women received ARV
prophylaxis in LW
• Infants ARV prophylaxis
in LW
• Women received ARV
prophylaxis in LW
• Safe delivery in LW
• Women received infant
feeding counselling from the
counsellor
• ARV prophylaxis for women
and infant provided from the
counsellor at 34 weeks
1. Percentage of women who
received information on
PMTCT
• HIV testing of women not
provided in ANC but in
off-site laboratory
4. Percentage of women receiving
ARV prophylaxis
LW
3. Percentage of women who
received their test result
ANC:
Primary outcomes reported
Partially integrated
Comparison
• Safe delivery in LW
• Women received infant
feeding counselling at
ANC
• ARV prophylaxis
dispensed any time from
the 2nd trimester
• Infant ARV prophylaxis
dispensed at ANC
• Women received ARV
prophylaxis at ANC
• HIV testing of women
provided in ANC
• Nurse counsellor
provided all the PMTCT
interventions in ANC
Intervention
ANC = antenatal care; LW = labour ward; ARV = antiretroviral; VCT = voluntary counselling and testing ART = antiretroviral therapy;
PMTCT = prevention of mother to child transmission; RSA = Republic of South Africa.
Van’t Hoog, 2005
Kenya
Nov-01–Aug-03
Author Year of
publication, Country
Study duration
Table 1 (P1B91). Continued.
4. 57% vs 70%, p < 0.001
3. 89% vs 90%, p = 0.233
2. 62% vs 76%, p < 0.001
1. 77% vs 92%, p < 0.001
Control vs intervention:
Results
78
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
79
PMTCT, were tested and learned their HIV status. The second study also
reported significant increase in HIV testing uptake after integration.
Conclusions: Limited evidence supports the effectiveness of PMTCT
programmes compared to partially integrated care. More research
measuring coverage and other relevant outcomes is urgently needed
to inform the design of services delivering PMTCT programmes.
P1B92
Evidence-based African first aid guidelines and training
materials
Van de Velde S1 , De Buck E1 , Vandekerckhove P2 , Volmink J3
1 Expertise Centre, Belgian Red Cross-Flanders; 2 Belgian Red
Cross-Flanders; 3 South African Cochrane Centre; Faculty of Health Sciences,
Stellenbosch University
Background: In Sub-Saharan Africa, 40% of the burden of disease
and injury can potentially be addressed by emergency care. Adequate
pre-hospital care is vital, but often unavailable in Sub-Saharan Africa.
Initial help and emergency transport often depends on basic first
responders. The World Bank identified first aid training as a
cost-effective way to save lives in Sub-Saharan Africa. Although
well-intentioned, first aid training in Sub-Saharan Africa often depends
on didactical materials from non-African settings. This may lead
to misdirected, inadequate or even harmful training instructions.
Objectives: To produce and implement evidence-based guidelines on
how to train basic first responders to manage emergency situations
in an African context. Methods: Two authors searched 8 databases
(including MEDLINE, EMBASE, The Cochrane Library, AFRICAN INDEX
MEDICUS) to identify relevant evidence on the effectiveness, safety,
and feasibility of various first aid procedures. A separate search
was done for studies on African cultural remedies and preferences.
A multidisciplinary panel of eleven African experts discussed each
recommendation until they reached agreement. The quality of evidence
and strength of recommendations were determined according to
GRADE. Four peer reviewers revised the guidelines. To implement
the guidelines we developed a flexible didactic materials kit and
an implementation guide. Between June and December 2010 we
piloted the training materials and implementation guide in Uganda
and Swaziland. Results: Overall we screened 24,000 references
and selected 143 publications for the guideline. It is planned to
implement AFAM in the Red Cross national societies from Namibia,
Mozambique, South Africa, Malawi, Uganda, Swaziland, the Comoros
and Kenya. At the colloquium we will present results of the guideline
and discuss experiences with its implementation. Conclusions: This
project allowed us to learn about successful first aid practices that
made the best use of the limited resources available.
P1B93
Criteria for judging the effectiveness of public health
interventions
Denison E1 , Vist GE1
1 The Norwegian Knowledge Centre for the Health Services, Norway
Background: Judging the effectiveness of public health interventions
(i.e., health promotion and prevention) includes assessment of both
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
the methodological quality of evaluations and the relevance of
interventions. Many interventions are complex and may involve
different sectors and agencies, and it is sometimes argued that
methodologically rigorous evaluations may miss anticipated effects.
However, to our knowledge, it has not been described which criteria
for study design and methodological quality are actually used by
researchers who systematically review the effects of public health
interventions. Objectives: To describe criteria used to evaluate
methodological quality used in systematic reviews (SRs) of effects
of public health interventions. Methods: A systematic literature
search was carried out in 13 electronic data bases from January
2005 to January 2011 to identify SRs addressing effects of public
health interventions. We included non-medical interventions targeting
children, adults and older persons where health related outcomes were
reported. We extracted data on 1) inclusion criteria for study design,
2) quality assessment of included primary studies, 3) meta-analysis,
4) grading of results and 5) first author affiliation. The most recent
review from research milieus that had published two or more SRs was
included. Results: The literature search identified 7,850 references,
801 were read in full text, 644 publications were assessed as SRs.
Data from the 125 selected SRs show that 1) inclusion criteria for
study design involved randomised controlled trials (24%), studies
with a control group (36%), no design limitations (29%) and no
specification reported (11%), 2) quality assessment of primary studies
was performed in 69% of the SRs, 3) meta-analysis was performed in
34% and 4) grading of the results was done in 7%. Conclusions:
Criteria regarding study design and assessments of methodological
rigour were to a large extent applied in SRs evaluating the effect of
public health interventions.
P1B94
Using ‘Implications for practice’ from Cochrane reviews to
promote worldwide improvements in healthcare
Newbatt E1 , Gholitabar M1 , McGuire H1 , Ullman R1
1 National Collaborating Centre for Women’s and Children’s Health, UK
Background: The National Collaborating Centre for Women’s
and Children’s Health (NCC-WCH) is contracted to produce clinical
guidelines for use in the UK National Health Service (NHS). These
guidelines provide specific granular evidence and/or consensus-based
recommendations for the care of women and children using the
NHS. Because of this, our recommendations may not be translated
easily into clinical practice in other countries.Cochrane reviews adopt
a more ‘holistic’ approach by including studies from all countries
and writing ‘Implications for practice’ based on the findings of the
review. However these ‘Implications for practice’ are not always
directly relevant to both developed and developing countries, and an
alternative approach might be to use sub-group analyses to inform
specific ‘Implications for practice’ for different countries. This should
help promote global improvements in the quality of healthcare through
the increased use of more context-specific ‘Implications for practice’
in Cochrane reviews. Objectives: To evaluate how the results
of systematic reviews, and the ensuing ‘Implications for practice’ and
recommendations, are affected by the inclusion or exclusion of evidence
from different countries. Methods: We identified systematic reviews,
in NCC-WCH guidelines, that included evidence from all countries,
and then performed sub-group analyses to examine the effect of
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
80
including only evidence from developed or developing countries.
We then examined the potential implications for creation of more
context-specific practice recommendations. Results: Results will be
presented at the Colloquium. Conclusions: Conclusions will be
presented at the Colloquium.
P1B95
Evidence-based medicine as a safety parameter for the
judicialization of health policies
Marin dos Santos D1 , Puga ME1 , Bartolomei C1 , Marchewka T1
1 Brazilian Cochrane Centre and Federal University of São Paulo, Brazil
Background: In Brazil and other countries, the judiciary branch plays
an active role in health policies, due to the phenomenon called judicial
activism or judicialization of health. Objectives: To establish useful
parameters to support the judicial power in cases involving health
policies and drugs. Discussion: It is known that scientific evidence
is associated with a lower rate of errors. Though, it is expected that
legal practitioners recognize evidence as a basis for decision-making.
However, one of the greatest challenges on implementing this premise
is the broad discretion held by the judges, as there is no hierarchy of
legal evidence (Principle of free persuasion of the judge). Conclusions:
The Brazilian Cochrane Center, from the experience gained in tackling
the issue, believes it is possible to establish three precepts, which
would ensure that judicialization of health policies could be conducted
safely. First of all, when the object of the lawsuit reveals a high level
of scientific evidence, the judicial intervention will be legitimate. The
inertia of the government, in such situations, can be characterized
as faut du service, legitimizing judicial mediation. In the other hand,
it there is no available evidence, the judiciary branch may, as a
rule, dismiss the case. However, in such situations, the judge can
conduct an extensive analysis of the peculiarities of the case, trough
the performance of forensic expertise. Finally, if there is scientific
evidence that points against the intervention, the judicialization of
health policies may cause damage to the collective and the individual’s
health. By ignoring evidence, the Judiciary Branch will be denying the
constitutional right to health; though, for reasons unrelated to science,
believes to be ensuring it.
P1B96
A novel approach to building evidence for improving
communication about childhood vaccinations in low- and
middle-income countries: the ‘communicate to vaccinate’
(COMMVAC) project
Lewin S1 , Hill S2 , Kaufman J2 , Galak N2 , Kis-Rigo J2 , Anderson L3 ,
Bosch-Capblanch X4 , Hussein L5 , Lin V3 , Mahomed H5 , Rhoda L5 ,
Robinson P3 , ben Saude de Castro Freire S6 , Waggie Z5 , Wiysonge C5
1 Norwegian Knowledge Centre for the Health Services and Medical
Research Council of South Africa, Norway; 2 Centre for Health Communication and Participation, Australian Institute for Primary Care and Ageing,
La Trobe University, Australia; 3 La Trobe University, Australia; 4 Swiss
Tropical and Public Health Institute, Switzerland; 5 Institute of Infectious
Disease and Molecular Medicine, University of Cape Town, South Africa;
6 International Union for Health Promotion and Education, France
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Effective provider-parent communication can improve
childhood vaccination uptake and strengthen immunization services
in low- and middle-income countries (LMICs). However, demand-side
interventions to improve vaccination uptake have been neglected and
existing rigorous research is often not readily found or easily applicable
to LMICs. This makes it difficult for policy makers to use evidence to
inform policies and practice. Objectives: To describe the approaches
used by the COMMVAC project to explore, evaluate and disseminate
evidence on strategies for improving communication about childhood
vaccinations with parents and communities in LMICs. Methods:
COMMVAC uses an innovative combination of methods. First, we
are producing a systematic map of communication interventions.
Systematic maps use the same rigorous methods as systematic reviews
of effects but focus on describing the range of interventions. For
each intervention identified, we are extracting information on the
populations targeted, settings, intervention purpose and delivery (and
evaluation design and outcomes in trials). Second, we are developing a
taxonomy of interventions to improve communication around childhood
vaccination so as to: (1) understand the relations between different
types of interventions; (2) facilitate conceptual mapping of these
interventions; and (3) clarify the key purposes of interventions. Third,
we will hold deliberative fora with key stakeholders to discuss priorities
for systematic reviews of effects, informed by the systematic map
and taxonomy. Fourth, we will conduct systematic reviews on high
priority topics. Finally, we will produce web-based evidence summaries
that translate the review findings into accessible messages for LMICs
and allow users to add implementation commentary. Conclusions:
COMMVAC takes a novel approach to building knowledge resources
and making more effective use of existing research and practice
descriptions. Key outputs will include high quality evidence on the scope
and effects of interventions to improve provider-parent communication
around vaccination and knowledge resources tailored for LMICs.
P1B97
VIP (Very Integrated Program) Project on alcohol and drug
abusers in Southern Sweden – RCT of efficacy of the adding
the health promotion and rehabilitation for treatment of
alcohol and drug abusers
Hovhannisyan K1 , Ehrnström M1 , Kovacs J1 , Skagert E1 , Hau Skibelund D1 ,
Thornqvist K1 , Tønnesen H1
1 Lund University, Sweden
Background: Alcohol and drug abuse are followed by tremendous
physical, psychological and social problems as well as early death.
Heavy smoking, poor nutrition, physical inactivity and chronic diseases
(co-morbidity) are often part of these problems and illnesses.There
seems to be a large potential for a better outcome by including
smoking cessation, physical training, diet and nutrition as well as
co-morbidity treatment in a multi-disciplinary setting – a potential not
used yet. This Very Integrated Program (VIP) is inspired from the
rehabilitation offered to patients with chronic diseases and surgical
patients having a likewise unhealthy lifestyle and similar co-morbidities.
Aim: To evaluate the effect of adding the VIP program to the usual
alcohol and drug intervention on the outcome for alcohol and drug
abusers compared to the usual intervention alone. The VIP project
consists of 3 steps: 1. To map the health status and estimate the
potential improvement for 400 alcohol and drug abusers 2. To pilot
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
81
test the VIP program 3. To evaluate it compared to the daily routines
for 260 alcohol and drug abusers with a poor health status in a
randomised controlled trial. Main outcome: Change from abuser to
non-abuser. Secondary outcomes are health status; quality of life, use
of health services, time return to work (or similar activity level), harm
reduction, and cost-effectiveness. The results will be published in
scientific journals, reports and in the public media. The VIP project
takes place at the Addiction Centre Malmö, Psychiatry Skåne, Sweden
in collaboration with a multidisciplinary staff group. It is organised
by Lund University Clinical Alcohol Research Center at the University
Hospital Malmö, Sweden in collaboration with WHO Collaborating
Centre for Evidence-Based Health Promotion in Hospitals & Health
Services, Bispebjerg University Hospital, Copenhagen, Denmark.
P1B98
The international distribution of RCTs in Cochrane
dementia intervention reviews
Marcus S1 , Noel-Storr A1 , Struthers C1
1 Cochrane Dementia and Cognitive Improvement Group, United
Kingdom
Background: There were an estimated 35.6 million people with
dementia in 2010, the numbers nearly doubling every 20 years, with
an expected 65.7 million in 2030 and 115.4 million in 2050. 58%
of all people with dementia worldwide live in low and middle income
countries, rising to 71% by 2050 (Alzheimer’s World Report, 2009).
Objectives: This study sets out to assess the international distribution
of both included and excluded studies within published Cochrane
Dementia and Cognitive Improvement Group (CDCIG) intervention
reviews. Methods: Over 100 reviews are to be assessed for
this study. The following information will be extracted from each:
• the country of origin of both included and excluded studies •
the country or countries of recruitment of participants from included
and excluded studies • the absolute number of participants from each
included and excluded study • the main diagnostic criteria used for each
included study. Results: The final data collection for this study will be
in July 2011. It is expected that less than 10% of included Cochrane
dementia studies will be from low and middle income countries yet
this is where proportionate increases in the number of people with
dementia will be much steeper compared with high income countries.
P1B99
Assessing applicability, scaling up and equity issues in
Cochrane reviews: A worked example from a review of
lay health worker interventions in primary and community
health care for chronic conditions
Lewin S1 , Munabi-Babigumira S2 , Glenton C2 , Scheel I2 ,
Bosch-Capblanch X3 , Daniels K4 , van Wyk B5 , Aja G6 , Zwarenstein M7
1 Norwegian Knowledge Centre for the Health Services, Norway and
Medical Research Council, South Africa; 2 Norwegian Knowledge Centre
for the Health Services, Norway; 3 Swiss Centre for International Health,
Swiss Tropical and Public Health Institute, Switzerland; 4 Health Systems
Research Unit, Medical Research Council, South Africa; 5 School of Public
Health, University of the Western Cape, South Africa; 6 Department of
Health Sciences, Babcock University, Nigeria; 7 Combined Health Services
Sciences, Sunnybrook Health Sciences Centre, Canada
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: In addition to evidence of effectiveness, review users
often require information on the equity impacts of interventions; the
extent to which interventions can be applied in other settings and
whether similar effects might be expected (applicability); and factors
likely to influence scaling up. However, these issues are often not
discussed by review authors. Objectives: To assess the extent to
which evidence is available to facilitate assessment of applicability,
scaling up and equity issues from RCTs included in a review of the
effects of lay health workers (LHWs) for chronic conditions. Methods:
For each included RCT, data was extracted on intervention context and
on the groups targeted by the intervention, including their geographic,
demographic and socio-economic details. We used these data to
consider issues of applicability, scaling up and equity. Results: Overall,
the included studies provided little data regarding the differential effects
of the interventions across different populations making equity effects
difficult to assess. Most studies were undertaken in high income
countries and it is not clear whether the effects could be transferred
to other settings with different health care systems. However, most of
these studies were directed at low income or ethnic minority groups in
these settings. To the extent that these groups share similar constraints
in accessing health care with service users in low- and middle-income
countries (LMICs), the interventions may be applicable in LMICs.
Consideration of factors affecting the scaling up of the interventions
was limited by poor intervention descriptions. Conclusions: Assessing
applicability, scaling up and equity issues for effective interventions
continues to be hampered by limited data in trial reports. Guidance for
trialists on the reporting of such data needs to be applied.
P1B100
Clinical practice guideline: Dental care for patients with
epidermolysis bullosa
Krämer SM1 , Villanueva J1 , Serrano C2 , Galvez P3 , Zillmann G3 , Araya I1 ,
Oliva P4 , Carrasco A1 , Brignardello R1 , Yanine N1 , Cornejo M1
1 Centro adherido Cochrane de la Facultad de Odontologia, Universidad de
Chile, Chile; 2 Universidad de Valencia, Spain; 3 Universidad de Chile, Chile;
4
Universidad del Desarrollo de Concepción, Chile
Background: Epidermolysis Bullosa (EB) is a rare disease with
multiple oral manifestations requiring a special approach from the
dentist. The scientific literature is scarce, making it difficult for dentists
with no experience on EB to know how to approach patients in a
safe manner. The present guideline has been developed using a
standard methodology. Objective: Provide the users with information
on the special care people with EB require before, during and after
dental treatment. Methods: A systematic review of the literature
was performed. The information gathered was discussed at a two
day consensus meeting including dentists from different countries
and patient representatives. To formulate the recommendations
the SIGN system was used. The draft document was sent for
external review to dental professionals and patient groups. Results:
Key recommendations include: A preventative protocol is the dental
management approach of choice. Patients with EB should be referred
to the dentist for the first consultation at the age of 3 to 6 months.
A dietary caries-prevention programme should be instigated at early
age. Patients with the severe generalized RDEB subtype of EB require
the most treatment modifications. Extreme care of fragile tissues is
important: little pressure (compressive forces) can be applied, but no
sliding movements should be used. Lips should be lubricated before any
procedure is performed. Bullae occurring during treatment need to be
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
82
drained to avoid spreading. Sutures can be used safely in all patients.
Conclusions: This guideline will help dentists treating patients with
EB to provide safe and evidence based treatments.
P1B101
Examining ‘transportability’ issues in systematic reviews:
The example of parenting interventions in developing
countries
Gardner F1 , Knerr W1 , Lucie C2
1 Oxford University, UK, Dept of Social Policy & Intervention; 2 Oxford
University, UK; University of Cape Town, South Africa
Background: Parenting interventions have been shown to improve
parent/child outcomes in high-income countries. Many are potentially
applicable across cultures and countries, but little is known about their
effectiveness in low-and middle-income countries (LMICs). Prior reviews
identify few trials in LMICs and even fewer trials of interventions with
prior established efficacy. Global interest in parenting interventions
has not been met by empirical guidance on whether/how interventions
translate between countries. Objectives: To investigate effectiveness
of parenting interventions for reducing harsh/abusive parenting and
increasing positive parenting in LMICs; to explore methods for
sysnthesising characteristics of ‘transported’ interventions, as a step
towards further understanding transportability mechanisms. Methods:
A range of databases were searched; unpublished reports were hand
and online-searched, and experts contacted. Included were randomised
trials with no-intervention, alternative-intervention, or services-as-usual
comparisons. High heterogeneity precluded meta-analysis, but characteristics of included studies were discussed narratively according to
type of delivery mode, outcome, level of evidence in country of
origin, and extent of adaptation for current setting. Results: Twelve
studies (N = 1580) in nine countries reported results favouring
intervention on a range of measures, but overall validity is unclear
due to significant/unclear risks of bias. Results of the two largest,
highest-quality trials suggest interventions may be feasible and effective
for
improving
parent–child
interaction
and
parental
knowledge/ attitudes in LMICs. Results provide initial evidence on
feasibility of coding factors related to transportability across countries,
although this was sometimes limited by poor reporting, and lack
of efficacy evidence from countries of origin. Conclusions: While
limited conclusions can be drawn, there is promising evidence that
parenting interventions can be implemented and may be effective
in improving positive parenting in LMICs. The results raise key
methodological and implementation issues for wider investigation,
e.g., how best to synthesise evidence about transportability; relative
benefits of ‘transported’ versus indigenous interventions; adaptation
to new service contexts.
P1B102
Re-analysing equity in a Cochrane review: Smoking
cessation in pregnancy
O’Mara A1 , Oliver S1 , Chamberlain C2 , Caird J1 , Barnett-Page E1 , Thomas J1
Institute of Education, London, UK; 2 Centres Collaboration,
Women and Children’s Program, Southern Health, Clayton South, Australia
1 EPPI-Centre,
Background: A Cochrane review about smoking cessation in
pregnancy concluded that interventions can reduce the proportion
of smokers and the incidence of low birthweight and preterm births
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
(Lumley et al. 2009). Another systematic review (Arblaster and
colleagues 1996) suggested a number of intervention features that
are promising for reducing inequalities, such as the involvement
of peers. Objectives: To assess the impact of interventions on
smoking cessation in pregnancy for disadvantaged groups, and to
evaluate the relative effectiveness of intervention features suggested
by Arblaster and colleagues for reducing health inequalities. Methods:
Various meta-analytic models (meta-regression, multivariate multilevel
modeling) assessed the relative impact of interventions on women
differing in their socioeconomic status (SES) and ethnicity. The
effectiveness of intervention features at reducing inequalities was
also explored: using incentives; addressing the expressed concerns of
pregnant smokers; involving participants’ peers; and consulting women
to design interventions. Some variables were addressed descriptively
(e.g., young mothers as a disadvantaged group; structural measures as
an intervention feature). Results: Interventions reduced the number
of women smoking in late pregnancy regardless of SES or ethnicity.
Interventions with incentives were more effective than those without;
no other intervention features were statistically significant predictors of
effectiveness. Interventions were relatively more effective at increasing
birthweight than stopping smoking altogether. Conclusions: Smoking
cessation programmes are effective for all women and should be widely
implemented. Analysis is ongoing to identify promising elements for
smoking cessation interventions to reduce inequalities.
References
1. Lumley J, Chamberlain C, Dowswell T, Oliver S, Oakley L, Watson L.
(2009) Interventions for promoting smoking cessation during
pregnancy. In: the Cochrane Database of Systematic Reviews, 2009:
Issue 3. Art. No.: CD001055. DOI: 10.1002/14651858.CD001055.
pub2.
2. Arblaster L. (1996) A systematic review of the effectiveness of
health service interventions aimed at reducing inequalities in health.
Journal of Health Services Research and Policy 1: 93–103.
Knowledge translation in patient safety
Day: 20 Oct (P1B103–P1B117)
P1B103
The building blocks of safety: Creating multi-disciplinary
micro-systems and departmental PSOs as the foundation
for a successful patient safety program
Shaw K1 , Rickard S2 , Reay T3 , Thomson D4
1 Department of Pediatrics, Children’s Hospital of Philadelphia, Philadelphia,
Pennsylvania, USA; 2 Center for Complexity and Systems Thinking, Lund
University, Lund, Sweden; 3 Department of Strategic Management and
Organization, School of Business, University of Alberta, Edmonton, Canada;
4 Cochrane Child Health Field, Department of Pediatrics, University of
Alberta, Edmonton, Canada
Background: The Children’s Hospital of Philadelphia (CHOP) has
created an infrastructure to foster patient safety (PS) and communication
across disciplines, across units, and between unit-based and executive
leadership. The patient safety program at CHOP: Patient safety
programs must be designed to consider the following organizational
factors: Structures (formal reporting relationships and grouping of
individuals). The Chairs of the Academic Departments and hospital
Chief Operating Officer (COO) supported the formation of Departmental
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
83
Patient Safety Officers (d-PSOs) from the key clinical areas who
work with both hospital leaders and unit-based clinical leadership.
Systems (ways tasks are organized and completed). To support
collaborative work and learning, multi-disciplinary micro-systems were
created at the unit level with nurse/physician partnerships, Quality
and PS Committees, quality coordinators, and systems which allow
staff of all levels time to participate in PS meetings and walk-rounds.
Leadership (support from multiple champions from different areas of
the organization). Physicians, key members of PS teams, were provided
with protected time and financial support to partner with nursing staff
at the unit level and to create d-PSOs to promote communication
between the units and the hospital administration. Culture (beliefs and
assumptions that guide activities). A ‘just culture,’ where individuals
feel free to report and learn from safety events (without feeling
threatened by the consequences), has been introduced. The Unit-based
PS walk-rounds and unit-based Quality and Safety Committees allow
all staff, clinical and non-clinical, to identify safety issues and practices,
foster communication, and promote analysis of problems. Politics
(use of power to accomplish particular goals). The PS program
was facilitated by recognizing and working with varying power levels
within units (micro-system) and the hospital leadership (macro-system).
Conclusions: At CHOP, the alignment of organizational structures,
systems, leadership, culture and power have established the foundation
for becoming a high reliability organization in regards to patient safety.
P1B104
Assessment of hospital capacities in patient safety in the
Philippines
Acuin J1
De La Salle University, Philippines
1
Background: Despite growing public concerns and more stringent
government regulation, Philippine public hospital programs to ensure
patient safety are perceived to be hampered by severe infrastructural
limitations. Little attention, however, is paid to human resource
and programmatic constraints in aligning hospital cultures with
organizational goals. Objectives: To assess human resources,
programs and patient safety climates in government hospitals.
Methods: Four 200- to 400-bed public hospitals (one children’s
hospital (A), one transplant center (B), one general hospital in an
island province (C) and another in Manila (D)) that had begun patient
safety programs were included. Human resource and programmatic
markers, based on Department of Health hospital accreditation
standards implemented since 2010, were determined from key
informant interviews and policy document reviews using the Institute
for Healthcare Improvement (IHI) framework for achieving patient
safety and high reliability. The 19-item IHI Safety Climate Survey was
administered to managers and staff in high-risk patient care areas.
Summary estimates were calculated for each hospital. Results: Except
for hospital C, the hospitals generally had designated officers and
working committees. Hospital C had the lowest nurse to patient
bed ratio (1:38). Programmatic markers were generally absent in all
four hospitals. Thus, even when working committees are present,
the scopes of responsibilities in ensuring patient safety appear limited
and underdeveloped. Except for hospital A, the hospitals had no
patient disclosure processes. About 60% of the 131 respondents
(range, 50%–82%) viewed the safety climate as positive. Overall
and patient safety means were highest in hospital B. The lowest
scored items involved management action on staff’s safety suggestions
and management’s willingness to compromise safety concerns for
productivity. Conclusion: The number of hospital staff engaged in
patient safety appears to be adequate. However, they have limited
formal programmatic roles and have fair regard for their hospitals’
safety climates.
P1B105
Safe care for children: A scoping review of what we know,
and what we need to find out
Thomson D1 , Seto I1 , Scanlon M2
1 Cochrane Child Health Field, Department of Pediatrics, University of
Alberta, Edmonton, Canada; 2 Department of Pediatrics, Critical Care,
Medical College of Wisconsin, Milwaukee, WI, USA
Table 1 (P1B104). Patient Safety Markers and Safety Climate Scores in 4 Philippine hospitals.
Hospitals
Patient safety markers and safety culture scores
A
B
C
D
Average
Designated QI / PS Officer
Working Patient Safety Committee
Working Therapeutics Committee
Working Infection Control Committee
Infection control nurse to bed ratio
Adverse event tracking systems
Healthcare associated infection surveillance
Patient falls prevention
Standardized patient disclosure processes
Overall Mean—All Questions & Respondents∗
Safety Climate Mean∗
Percent Respondents Viewing Safety Climate as Positive:
Number of respondents
Yes
Yes
No
Yes
1:250
Yes
No
No
Yes
3.97
3.91
0.53
40
Yes
Yes
Yes
Yes
2:400
Yes
No
No
No
4.35
4.24
0.82
38
No
No
No
No
4:150
No
No
No
No
4.07
3.98
0.50
16
Yes
Yes
Yes
Yes
2:350
No
No
No
No
4.10
3.93
0.51
37
–
–
–
–
–
–
–
–
–
4.13
4.02
0.60
131
∗ based
on this scale: 1—disagree strongly; 2—disagree slightly; 3—neutral; 4—agree slightly; 5—agree strongly
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
84
Background: Children differ from adults in many profound ways.
These unique aspects to childhood affect all aspects of medical
care, including patient safety. Therefore, any study of pediatric
patient safety must be carried out on the basis of research which
explicitly incorporates an understanding of the physiological and
psychosocial stages of childhood. Objectives: To map the research
published in pediatric patient safety in the period 1999–2010. To
establish what this research demonstrates about the characteristics
of childhood that affect patient safety, as well as the gaps that
remain. Methods: We carried out a scoping review of literature
in pediatric patient safety from the period 1999–2010. Scoping
reviews illustrate the volume, nature and characteristics of a body
of research. They identify and map all relevant literature, regardless
of study design, and identify research gaps (Arksey and O’Malley
2005). We searched for all literature, and then identified a subset
concerned specifically with pediatrics. We analysed this subset to
identify what is known about pediatric characteristics of patient safety.
Results: Preliminary results: pediatric trials are between 1.47%
and 7.21% of total published studies each year, whereas children
in the 0–19 age range make up 34.9% of the world’s population
(US Census Bureau). Final results will be discussed at the Cochrane
Colloquium. We found two broad categories to pediatric aspects of
patient safety: 1) Nature of childhood (varies across age groups),
e.g. physiological, developmental, social/family context; and 2)
Aspects of the system of care (varies across settings), e.g. medication
prescribing and administration, organizational approaches (including
staffing, education and training), and technology (e.g. computerized
physician order entry systems). Conclusions: Addressing pediatric
patient safety needs to be based on evidence that considers the
particular physiological and psychosocial aspects of childhood and
adolescence, as well as the settings in which pediatric care is
administered.
P1B106
Six Sigma applications for patient safety in a university
hospital
Mejia HM1
Management Chief University Hospital Fundacion Santa Fe de
Bogota Colombia
1 Clinical
Background: The Fundación Santa Fe de Bogota University Hospital
is a private high-complexity institution with 205 beds. Since its
foundation in Bogotá in 1983, the hospital has been recognized for
its commitment to safety, innovation and the search for excellence
in all its processes, and it was awarded accreditation by the Joint
Commission International. Objectives: The Six Sigma improvement
tool was implemented in 2003. With its structured problem-solving
approach and the zero-error goal, it has had a positive impact on
patient safety through different project implementations. Methods:
At present, we run our own independent training process and there
are already 488 green belts, 41 black belts and 4 master black
belts. Our portfolio of projects consists of 88 projects, 57% of which
focus on patient safety. Results: Examples of projects on patient
safety: 1. Prevention of ventilator-associated pneumonia (VAP) in
critical care units, 2007–2010: preventive measures were standardized
and effective controls were put in place, improving from Sigma
levels 1.7 to 2.5. 2. Optimization of the blood transfusion process,
2007–2010: improvement by means of process standardization and the
implementation of controls translated into a Sigma level improvement
from 4.0 to 4.7. Conclusions: Because of their extreme variability,
healthcare processes are a great opportunity for improvement. The Six
Sigma training and project development methodology support patient
safety culture and help create open minds. Six Sigma has taught
us that well-chosen projects, well-integrated teams and a committed
leader are the keys to success. Six Sigma strengthens teamwork and
the search for excellence in patient safety.
Figure 1 (P1B106).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
85
Figure 2 (P1B106).
Figure 3 (P1B106).
P1B107
A survey analysis of patient for patient safety in China
Zhang Q1 , Li Y1 , Zhang L1 , Yin Q1 , Shang L1 , Wei X1 , Zhang M2
1 West China School of Medicine, Sichuan University, Chengdu, China;
2
Chinese Cochrane Centre, West China Hospital, Sichuan University,
Chengdu, China
Background: Patient safety involves not only medical institutions and
medical workers but also patients. Much research has been done
to promote patient involvement in patient safety recently. But very
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
little is known of this area in China. Objectives: To investigate the
knowledge, attitudes and actions of patients toward patient safety
and to provide evidence on how to promote patient safety in China.
Methods: A questionnaire was developed on patient safety and
distributed to patients by a group of volunteer medical students. SPSS
and MS Excel were used to analyze the data. Results: The volunteer
medical students distributed 350 questionnaires at random to patients,
of which 326 were completed with some items left blank. The sample
of patients had a mean age of 38.8, 42.2% were male and 57.8%
were female, and over 88% had completed less than undergraduate
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
86
Figure 4 (P1B106).
Figure 5 (P1B106).
Figure 6 (P1B106).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
87
Table 1 (P1B107). Representation of patients’ knowledge,
attitudes and actions of patient safety in China. (Zhang et al.).
Characteristic
All patients
If you have insurance about your health?
Yes
No
What is your biggest worry in your medical process?
High cost
If the treatment is effective
The adverse effect of the treatment
The quality of the hospital
The quality of the doctors
Do you know what is a medical error?
Yes
No
Do you know the your drug’s adverse reaction?
Yes
No
Do you know that it is possible that you could be
infected in hospital?
Yes
No
Do you think that the doctors/ nurses should wash
their hands before doing examinations on you?
Yes
No
Do you think that the medical device the
doctors/nurses use is disinfection?
Yes
No
I don’t know
Will you remind the medical staff if you find something
unhygienic?
Yes
No
Will you take legal action if you or your family member
is the victim of the medical error?
Yes
No
Patients
No. (%)
326 (100)
326 (100)
261 (80)
65 (20)
326 (100)
62 (19)
134 (41)
52 (16)
7 (2)
22 (7)
326 (100)
110 (34)
216 (66)
324 (100)
103 (32)
224 (68)
326 (100)
104 (32)
222 (68)
325 (100)
287 (88)
38 (12)
322 (100)
241 (75)
9 (3)
75 (22)
324 (100)
291 (90)
33 (10)
326 (100)
238 (73)
88 (27)
P1B108
Patients for patient safety research in The Cochrane
Library
Zhang Q1 , Zhang M2
1 West China School of Medicine, Sichuan University, Chengdu, 610041
China; 2 Chinese Cochrane Centre, West China Hospital, Sichuan University,
Chengdu, 610041 China
Background: Patients for patient safety has been increasingly
important and much research has been done in the last decade.
Research has focused on assessing the patients’ attitudes toward
patients’ engagement and to evaluate interventions to encourage
patients in safety-related actions. Objectives: Our study aims to
investigate the current research status of patients for patient safety in
The Cochrane Library. Methods: We developed a search strategy with
MeSH terms including iatrogenic disease or medical errors/ prevention
and control, medical errors/ adverse effects, safety management or
cross infection/ prevention and control. The Cochrane Library Issue 4,
2011 was searched. We selectively reviewed the evidence that was
likely to assess patients’ attitudes or evaluate interventions to promote
patient participation in patient safety by first screening identified titles
and then re-reading the abstracts. Results: Searches of The Cochrane
Library yielded 236 candidate articles, of which only three studies
met our needs. There were no studies designed to assess patients’
attitudes to patient safety. Three interventions from three studies
published from 2008 to 2010 were evaluated. They were treatment
education programs, secure electronic messages and a decision aid
about risk of acute coronary syndrome. All of these interventions aimed
at increasing the knowledge and information of patients, which could
promote patient involvement in patient safety. Two of them were in
specialist areas: one on occurrence of hemorrhagic or thromboembolic
events and another on acute coronary syndrome. Except for one study
that is still in progress, other two studies showed improvement in
patient safety. Conclusions: Patient safety research, which is very
important, is very limited. Greater attention should be paid by the
Collaboration to this field to ensure that safer health care will be
provided for patients.
P1B109
Patients for patient safety: A systematic review
Li Y1 , Huang Y1 , Nie Y1 , Li J1 , Zhang M2
1 Dept. of evidence-based medicine and clinical epidemiology; 2 Chinese
Cochrane Center
education. About 80% of patients had health insurance. Most patients
thought that effectiveness of the treatment was their biggest worry
during the medical process, while only 16% of patients worried about
the adverse effect of the treatment. Of the patient respondents, 66%
do not understand the concept of a medical error, 68% do not know
the adverse effect of their drugs, and 68% do not know the possibility
of infection in hospital. However, 88% of patients know that medical
staff should wash their hands before doing examinations and 90%
of patients would remind the medical staff if they find something
unhygienic. If they or their family members were victims of a medical
error, 73% of patients would take legal action. Conclusions: Patient
safety information needs to be disseminated among patients. Most
patients trust doctors and do not care about their safety on their
own initiative. However, most patients show enthusiasm to protect
themselves when medical error happens.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Patients play an important role in improving patient
safety. However, little is known about how and when to involve
patients, and the effects after involvement in patient safety. Objectives:
To systematically review the literature, summarize and compare the
information on patients for patient safety at home and abroad, including
patients’ knowledge, attitude, influencing factors and the effects
after participation in order to provide evidence-based suggestions
for education, training and research of patients for patient safety in
China. Methods: We systematically searched the Chinese Biomedical
Literature Database, EMBASE and MEDLINE to identify primary studies
about patients for patient safety. Two researchers independently
identified the eligible studies, assessed the risk of bias using the
Cochrane Handbook 5.0 and Critical Appraisal Skills Programme,
and extracted data. A meta-ethnographic approach was used to
summarize the results. Results: We identified 28 eligible studies with
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
88
Table 1 (P1B108). Details of studies that evaluated interventions to engage patients in safety-related action. (Zhang et al.)
(n=3).
Reference
Pernod, G., J.
Labarère, et al.
(2008)
Weingart, S. N.,
H. E. Hamrick,
et al. (2008)
Pierce, M. A.,
E. P. Hess, et
al. (2010)
Patients
Diagnosed as having
deep vein
thrombosis or
pulmonary
embolism requiring
therapy with a
vitamin K
antagonist for 3
months or more.
267 subjects who
received and
opened a
MedCheck
message from April
2001 to June 2002.
Adults presenting to
Emergency
department (ED)
with a complaint of
chest pain who are
being considered
for admission for
prolonged ED
observation in a
specialized unit
and urgent cardiac
stress testing.
Intervention
Outcome
Main findings
Interventional group
received a specific
oral
anticoagulation
treatment
educational
program.
The occurrence of
hemorrhagic or
thromboembolic
events.
Main outcome criteria
were observed 5
times in the
experimental and
15 times in the
control group.
(p < 0.01).
Patient education
using an
educational
program reduced
VKA-related
adverse event
rates.
MedCheck, a
medication safety
application, sent
patients a secure
electronic message
10 days after they
received a new or
changed
prescription.
We analyzed patient
and clinician
response rates and
times, examined
patient-clinician
communication
about medications,
and identified
adverse drug
events (ADEs).
Patients and
physicians
responded
promptly to
patient-directed
electronic
medication
messages,
identifying and
addressing
medication-related
problems including
ADEs.
Receiving CHEST
PAIN CHOICE (a
decision aid that
communicates the
short-term risk of
acute coronary
syndrome).
Six outcomes,
including the rate
of delayed or
missed ACS.
Patients often
identified problems
filling their
prescriptions
(48%), problems
with drug
effectiveness
(12%), and
medication
symptoms (10%).
Patients
experienced 21
total ADEs; they
reported 17
electronically.
In progress.
five conducted in China. Except for one randomized controlled trial
(RCT), the studies had cross-sectional designs. 1. Patients’ perceptions
and attitudes: patients’ knowledge of patient safety was generally
poor, but patients were eager to participate. 2. Factors affecting
patients’ involvement: patient’s characteristics (gender, age, race and
education), disease-related factors (severity of disease, symptoms,
treatment and prognosis), medical staff-related factors (attitude of
medical staff), and emotional factors (feeling uncomfortable). 3. How
to participate: improving patients’ knowledge, acquiring therapeutic
information, cooperating, reminding and communicating with medical
staff, etc. 4. Outcomes after involvement: patients’ participation can
effectively standardize the performance of physicians, prevent unsafe
incidents, reduce harms, and promote patient safety. Conclusions:
Current studies cover a wide range of topics related to patients
engaging in patient safety, but research in China is very limited and
covers only the patients’ knowledge and willingness. Future studies
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Evaluative method
This pilot randomized
trial will develop
and evaluate a
decision aid for use
in ED chest pain
patients at low risk
for ACS, which
could reduce the
rate of delayed or
missed ACS.
should emphasize the ways to promote patients’ involvement in patient
safety and the outcomes after involvement.
* This study is funded by NSFC 70973083 and CMB.
P1B110
Knowledge translation in patient safety: What barriers are
present to assess the evidence in practitioners?
Perez S1 , Rodriguez V1 , Cattivera C1 , Pardo J2 , Garcı́a Elorrio E1
1
IECS, Argentina; 2 Centre for Global Health, Institute of Population Health,
University of Ottawa, Canada
Background: The gap between evidence and practice is important
for patient safety. Besides resistance to change, scarce leadership
support, resource availability, access and appraisal of evidence are
also contributing factors. To reduce this gap, a collaborative forum in
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
89
Table 1 (P1B110).
Barriers identified
Lack of economical support
Language (english and others)
Deficiency of training in meta analysis
interpretation
Absence of protected time during daywork
Lack of formal training in EBM
Lack of use of the Cochrane Library
Absence of support from the Board of
director
Absence of support from the colleagues
% (n/40 surverys)
92,50% (37/40)
45% (18/40)
32,5% (13/40)
37,5% (15/40)
35% (14/40)
27,5% (11/40)
25% (10/40)
22,5% (9/40)
hospital patient safety was recently launched in Argentina, integrating
patient safety practitioners and patient representatives. Experiences
are shared and evidence is disseminated at the forum. Objective:
To provide an exploratory description of the barriers to accessing
and assessing the evidence in a professional community of interest in
patient safety. Methods: After a literature review on instruments to
reveal the presence of barriers, we developed a preliminary 25-item
survey that was tested to confirm understanding. No qualitative
techniques were conducted at this time, given the exploratory purpose
of this report. The survey was distributed among the participants of
the forum, who completed it online, to ensure confidentiality, after
giving consent. Results: 40 surveys were collected (response rate
41%). The distribution of responders was 55.81% physicians, 27.91%
nurses, 11.63% pharmaceutics and others 4.65%. 73.71% have more
than 15 years working at the institute. 36.59% belonged to hospitals
larger than 200 beds. We will describe the frequency of responses on
the presence of certain barriers. Knowledge of The Cochrane Library
was 72.5% but only 52.5% knew how to interpret a meta-analysis.
Internet connection was not an issue for any responder. Conclusions:
Many barriers were present among patient safety practitioners. Several
findings will be confirmed in a larger study, which will also promote
the use of knowledge translation techniques that will surely reduce the
gap. This line of work may help to prioritize systematic reviews in this
field to reduce the impact of such barriers.
P1B111
Pediatric patient safety: the distinctive nature of childhood
must be part of the equation
Thomson D1 , Matlow A2 , Rickard S3 , Scanlon M4 , Williams K5
1 Cochrane Child Health Field, Department of Pediatrics, University of
Alberta, Edmonton, Canada; 2 Medical Infection Prevention & Control
Programme, The Hospital for Sick Children; Centre for Patient Safety,
University of Toronto, Toronto, Canada; 3 Center for Complexity and
Systems Thinking, Lund University, Lund, Sweden; 4 Department of
Pediatrics, Critical Care, Medical College of Wisconsin, Milwaukee, WI,
USA; 5 Cochrane Child Health Field; APEX Australia Chair of Developmental
Medicine, University of Melbourne; Department of Developmental
Medicine, Royal Children’s Hospital, Melbourne, Australia
Background: Children’s health is affected by several characteristics:
developmental change over time; dependency on adults; and a distinct
epidemiological profile of disease. These items, when combined
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
with healthcare provider and system factors, create vulnerabilities for
pediatric care that must be taken into account in pediatric patient
safety (PPS) programs. The above-noted issues affect the risk of
iatrogenic harm in many ways, of which the following are a few
examples. 1) Children’s developmental processes affect their responses
to medications, and medication dosages. However, there is a paucity
of evidence that is directly relevant to children, and children of different
ages, on the effectiveness and efficacy of drug interventions. The
epidemiology of adverse events differs between adults and children.
2) Children’s dependency on adults can affect the risk of harm. For
example, preverbal children cannot advocate for themselves. Moreover,
family dynamics (e.g. multiple caregivers) can make it difficult to get
a complete or accurate medical history. 3) In many countries, some
health care is delivered in school and community settings, yet we
know little about the potential for harm in these environments.
More research is also needed on how delivery of care to children in
lower-and-middle-income countries affects PPS. 4) Much health care for
children is given by providers without specialized training in pediatrics
(e.g. family physicians, staff in general emergency departments), who
may not recognize differing trajectories of disease in children, which
creates conditions for errors in diagnosis and treatment. Conclusions:
The dynamic nature of childhood influences the accuracy of diagnoses,
effectiveness of treatments and risk of harm to children. Including
more specific information that is relevant to these issues in systematic
reviews will advance PPS endeavours. More research is needed on PPS
in various settings of care and on global health issues in PPS.
P1B112
Evidence evaluation in ranking target areas of interest for
a national patient safety campaign
Lauvrak V1 , Norderhaug IN1 , Saunes IS2 , Krogstad U2
1 Norwegian Knowledge Centre for the Health Services (NOKC), Department
of Evidence-Based Practice, Specialist Health Care Unit, Norway; 2 NOKC,
Department of Quality Measurements and Patient Safety, Secretariat for
the Norwegian Patient Safety Campaign, Norway
Background: In 2009, the Norwegian Knowledge Centre for the
Health Services (NOKC) was commissioned by the Ministry of Health
and Care Services to prepare a national patient safety campaign.
A broad group of health care experts was established to use a
formalized consensus process to rank eleven possible target areas
according the following set of criteria: 1) Potential for improvement
in Norway; 2) Interventions documented as effective; 3) Access to
data and methods to evaluate the effect of interventions; and 4)
Support among health care professionals in Norway. Target areas
were: safe surgery, postoperative infections, urinary tract infections,
CVK-infections, medication errors and harm, stroke, waiting time cancer
patients, patient falls, decubitus ulcer, early rehabilitation hip fracture,
and rapid response team heart failure. Objectives: Provide information
on the effect of specified interventions. Methods: For one area, a
systematic review of primary studies was prepared. For eight areas,
short reports, based on systematic reviews identified in The Cochrane
Library, the CRD – databases and by hand searches, were prepared.
The most updated (2005–2010) high quality systematic review for
each predefined intervention was selected. The level of evidence for
predefined patient safety relevant outcomes was graded according
to the GRADE approach. Methods and results were communicated
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
90
written and orally to the expert group who scored each area against all
four predefined criteria. Results: Most interventions were supported
by low or very low quality of evidence. No intervention was rated
high quality, and only a few were rated moderate quality. For some
interventions, relevant scientific evidence was not found. Nevertheless,
the consensus process resulted in ranking four areas (safe surgery,
postoperative infections, medication errors and harm, and urinary tract
infections) as appropriate for the campaign. Conclusion: Uncertainty
of effectiveness did not in itself disqualify an area from being ranked as
appropriate for the campaign, but it was transparently acknowledged
together with the other three assessment criteria.
P1B113
Opioids for chronic pain: evidence of effectiveness,
consistency of use, and public health issues
Harris J1
1 The Permanente Medical Group, USA
Background: There is increasing concern about the personal and
public safety of prescription opioids for chronic pain. In the United
States (US), opioid prescription rates, emergency room visits, and
mortality rates have risen sharply in the last 10 years. However, it is
not clear that opioids are effective for chronic pain. There are high
levels of adverse effects and psychiatric comorbidities among patients
using chronic opioids. Consistent adherence to evidence-based best
practices would significantly improve patient safety. Objectives: 1. To
review and assess the literature on the relative effectiveness, risks and
benefits of opioid medications for chronic pain. 2. To review and
assess the literature on management of opioid use for chronic pain.
3. To suggest a research agenda to support more effective chronic
pain management. Methods: We searched The Cochrane Library,
other databases, and evidence generated to support guidelines for
systematic reviews on opioid use in chronic pain. We then classified
and critiqued the evidence. We also reviewed the evidence and
behavior change techniques used to support safe prescribing and pain
management at Kaiser Permanente Medical Centers. Results: Most
literature in this area consists of observational studies that are low
quality according to the GRADE system. Virtually all studies lasted less
than 3 months, which is not considered chronic. Retrospective cohort
studies described high levels of psychiatric comorbidities and adverse
effects of opioids. Pilot studies have shown wide variance in opioid
prescribing patterns, which seem stable over time despite educational
interventions. Conclusion: The effectiveness, efficacy and safety
of opioids for chronic pain are not clear based on current evidence.
Increasing use of chronic opioids is associated with public health
problems. More consistent, evidence-based prescribing patterns would
improve effectiveness and safety. Research designs could be improved.
P1B114
Evidence to support patient safety improvement
Harris J1
1
The Permanente Medical Group, USA
Background: The U.S. Institute of Medicine estimated that over
100,000 Americans die each year because of medical errors, and
many more sustain illnesses or injuries. Adherence to evidence-based
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
best practices and reduction in unwanted variation in care would
significantly improve patient safety. Objectives: 1. To review and
assess the literature on effective practices to prevent events that should
never happen, such as objects left in the body after surgery, medication
errors, and central line and other hospital-acquired infections. 2. To
review and assess the literature on effective implementation of safety
improvement projects. 3. To suggest a research agenda to support
safety improvement. Methods: We searched The Cochrane Library,
major data bases, and evidence generated to support guidelines
for systematic reviews on patient safety improvement. We then
reviewed and critically appraised the literature we found. We also
reviewed the evidence and behavior change techniques used to support
effective safety improvements at Kaiser Permanente Medical Centers
in Northern California. Results: Most literature in this area consists
of observational studies, including cohorts and case series that would
be classified as low quality in the GRADE system. Some studies are
‘action research,’ applying checklists and protocols and tracking results.
Nevertheless, use of protocols and guidelines has resulted in ongoing,
measurable improvements in several areas. Conclusion: The lower
quality studies, checklists and organizational interventions reduced
adverse event rates substantially. Because they might deny one cohort
protection from adverse events, randomized controlled trials (RCTs)
may not be ethical in safety sensitive areas. Research should use other
designs to prove or disprove the effectiveness of processes to protect
patient safety, and focus on methods to ensure consistent application
of proven procedures.
P1B115
Extracting clinically relevant messages from drug class
review
Melvin C1 , Carey T1 , Ranney L1
1 University of North Carolina at Chapel Hill, USA
Background: Evidence from drug class reviews is often not accessible
to practicing clinicians nor is it presented in a way that allows clinicians
to use the information to guide treatment and prescribing decisions.
Information on drug safety and efficacy is often difficult to interpret
clinically, especially in the absence of head-to-head trials. Nevertheless,
information from such reviews can be very helpful to clinicians as they
evaluate evidence of drug efficacy for a particular medical condition.
We describe a process for extracting key concepts and clinically relevant
messages from a drug evidence review. Our outcomes of interest were
remission, recurrence, and safety and adverse events associated with
the use of anti-epileptic drugs for bipolar disorder. Objectives: Our
objectives were to update and review evidence from a drug effectiveness
report on the efficacy of anti-epileptic drugs for the treatment of bipolar
disorder and to distil this evidence into clinically relevant messages for
communication to prescribers through a national dissemination effort.
Our overall goal was to improve treatment of patients with bipolar
disorder. Methods: We began with a published drug effectiveness
report and used standard systematic review techniques: an expert,
scientific panel with no ties to pharmaceutical companies to update
the drug effectiveness report, grade and synthesize evidence found
in the updated review and published report, and determine clinical
implications; and audience research with prescribers to inform the
development of key concepts and clinically relevant messages. Results:
Our approach reduced a 735-page review to ten key concepts and four
clinically relevant messages while maintaining scientific integrity and
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
91
credibility. These messages were used to inform national dissemination
strategies. Conclusions: It is possible to use established approaches
to literature review and synthesis in combination with scientific expert
review and targeted audience research to distil volumes of evidence
into clinically relevant, short messages that busy clinicians can use with
confidence.
P1B116
Instruments and methodologies used for detecting and
characterizing adverse events in ambulatory care: A
systematic review
Gaitán Duarte H1 , Vallejo Ortega MT1 , Garcı́a Dieguez M2 , Esandi ME2 ,
de Luca M2 , Milberg M2 , Cho M3 , Monserrat-Capella D3 , Reveiz L3
1
Universidad Nacional de Colombia, Colombia; 2 Academia Nacional de
Medicina, Argentina; 3 Organización Panamericana de la Salud, Estados
Unidos
Background: Unintentional damage to a patient caused by health
care is known as an adverse event (AE).The World Health Organization
(WHO) published a review about methods and measures used in
research to assess patient safety in primary care. The search was
limited to studies published in English up to 2007; the review included
only studies that evaluate one type of adverse event. A systematic
review updating and complementing previous evidence is proposed.
Objectives: 1. To describe methods, sources of information and
tools used to estimate the frequency of adverse events. 2. To
describe the adverse events reported in studies, according to their
type, frequency, origin, preventability, and seriousness. 3. To compare
instruments focusing on their constitutive domains, resources and
applicability. Methods: An advanced search was performed in
MEDLINE (PubMed), The Cochrane Library, LILACS, SciELO and EMBASE
to look for randomized controlled trials; cohorts; case control studies;
before and after studies; and cross-sectional studies. Studies had to
include at least two different types of AE on ambulatory care regardless
age or underlying condition. Tools to detect, qualify and classify AE
were identified and compared. Two authors independently will review
titles and abstracts to identify the eligibility criteria. Full text will be
assessed to check the inclusion criteria and to extract data. Results:
The search in MEDLINE identified 1836 articles, EMBASE 3846, LILACS
24, the Cochrane Database of Systematic Reviews 12, and DARE 37.
Full results of the assessment of the identified Cochrane Reviews will be
completed by October 2011. Conclusions: Findings will be relevant
for further investigations on AE. Results of this systematic review will
be used as a basis for the elaboration of the Investigation Protocol on
Adverse Events in Outpatient Care for Latin America and Caribbean
countries supported by PAHO.
P1B117
Update of Cochrane Systematic Review of audit and
feedback demonstrates stability in findings. Will future
trials be helpful?
Ivers N1 , Jamtvedt G2 , Flottorp S2 , Young J3 , French S4 , O’Brien MA5 ,
Johansen M2 , Odgaard-Jensen J2 , Oxman A2 , Grimshaw J6
1 Department of Family Medicine, Women’s College Hospital, Canada;
2 Norwegian Knowledge Centre for the Health Services, Oslo, Norway;
3
Surgical Outcomes Research Centre (SOuRCe), Central Sydney Area
Health Service, Camperdown, Australia; 4 Primary Care Research Unit,
University of Melbourne, Carlton, Australia; 5 School of Rehabilitation
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Science, McMaster University, Hamilton, Canada; 6 Clinical Epidemiology
Program, Ottawa Health Research Institute, Ottawa, Canada
Background/Objective: This review updates a previous Cochrane
review aiming to assess the effects of audit and feedback (AF) on
healthcare provider professional practice, which found a median 5%
improvement in compliance with desired practice across a range of
dichotomous outcomes. Methods: We searched MEDLINE, EMBASE,
CINAHL and the Cochrane Effective Practice and Organization of Care
(EPOC) database up to December 10, 2010 for randomized trials of AF.
We defined AF as a summary of clinical performance given to health
care providers. For this update, trials were included only when AF was
judged to be the core, essential feature of the quality improvement
intervention; thus, some previously included trials were excluded.
Data were double-abstracted independently. For each comparison, we
calculated the risk difference (RD) of compliance with desired practice,
adjusted for baseline compliance. Results: 144 trials were included
in the review; 91 were from the previous review. The interventions
were heterogeneous in their design and addressed a variety of clinical
problems. Considering only trials with dichotomous outcomes, when
AF alone was compared to usual care (41 comparisons), there was
a 4.1% increase in compliance with desired practice. When AF was
considered a core, essential aspect of a multifaceted intervention
(56 comparisons), the median increase in compliance was 5.3%. In
trials comparing combined educational outreach plus AF to AF alone
(16 comparisons), the median increase in compliance was 4.4%.
Conclusions: AF has a small but important positive effect on clinician
behaviour. This effect may be increased by concurrent interventions,
especially educational outreach. This update led to minimal changes
in the estimate of effect despite a substantial increase in the number
of studies. The stability in effect estimate suggests that future trials
comparing AF to usual care may not provide valuable new knowledge,
unless carefully crafted to assess specific aspects of intervention
design.
Education and training
Day 20 Oct
P1B118
A survey of the awareness rate of GRADE in China
Chen Y1 , Yang K1 , Wang Z2 , Yao L2 , Wang X2
1 Evidence-Based Medicine Center of Lanzhou University, China; 2 School
of Basic Medical Sciences of Lanzhou University, China
Background: Evidence based medicine and systematic reviews
are becoming better known among clinicians, researchers and
policy-makers; it is imperative to promote the development of GRADE
in China. Although it has been ten years since the GRADE system first
emerged, it is still a relatively new concept in China. Objectives: To
investigate the awareness of the GRADE system in China. Methods:
We conducted a two-part survey of health professionals. The first
part using a survey questionnaire given to attendees on the 6th
Asia-Pacific Evidence Based Medicine Conference on September 25;
the second using a web-based survey on http://www.dxy.cn (The
biggest medical community site in China) from October 13 to 28.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
92
Results: Of the 245 respondents who completed a questionnaire,
118 (48 %) had heard of the GRADE system. Among those who
had heard of GRADE, 53 (45 %) had accessed the web site of
GRADE; 20 (17 %) had used the GRADEpro program; 91 (77 %)
didn’t knew how many levels existed for grading the quality of
evidence and 102 (86 %) didn’t how many levels existed regarding
the strength of recommendations in the GRADE system; 6 (5 %)
knew of all the upgrading factors and 39 (33 %) knew of all of
the degrading factors of ‘quality of evidence’; 8 (7 %) correctly
answered factors affecting the strength of recommendations; 114
(96 %) thought GRADE was important or very important to clinical
practice in China and 115 (97%) thought it was necessary to
introduce and apply the GRADE system in China. Conclusions:
The survey yielded useful information about respondent’s knowledge
and perceptions of the GRADE system. Researchers and doctors
have limited familiarity with the concept of GRADE but most of them
thought the GRADE approach was important to clinical practice, and
it that it is necessary to introduce and apply the GRADE system in
China.
P1B119
Building capacity in EBHC: What is happening in the African
region?
Rohwer A1 , Young T2
1 Stellenbosch University, South Africa; 2 Stellenbosch University, South
African Cochrane Centre, South Africa
Background: Evidence based Health Care (EBHC) is recognized
internationally as a key competency for health care professionals. The
practice of EBHC involves converting information needs into answerable
questions; finding the best evidence; critically appraising the evidence
for validity and usefulness; applying the results of the appraisal into
practice; and evaluating performance. To inform EBHC capacity
development initiatives, we assessed current training initiatives in the
region. Objectives: To identify current training activities in the field
of EBHC. Methods: We contacted medical schools, organizations
and institutions as well as known experts in the field and searched
the Internet. Information obtained was summarized in a table format
and circulated to key contacts to validate. Results: Of 64 medical
schools in the region, only 40 had available e-mail addresses. Of
the 40 e-mails sent, mail delivery failed in six cases (15%) and
four responded (10%). Nineteen institutions and organizations were
contacted by sending 38 e-mails to various contact persons. Mail
delivery failed in two cases (5%) and 14 responded (37%). Additional
information on two training initiatives came from the Internet. We
identified 37 training programs. Seven universities and 10 other
organizations present these programs; 16 (43%) of the courses are
presented in South Africa. Three are postgraduate degrees (Clinical
Epidemiology (two), Applied Clinical Research and EBM,); one is a
postgraduate diploma (Research Methodology); six are modules within
university degrees (two undergraduate and four postgraduate); 19
short courses (≤1 week) and eight intensive courses (≥2 weeks) on
EBHC. Two of the courses are e-learning courses, while three are
blended. Conclusions: Despite low response, the findings indicate a
paucity of EBHC capacity development programs in the African region.
There is a need to use innovative strategies to increase the reach of
EBHC capacity development programs.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P1B120
Reporting quality of randomized controlled trials on
gastric cancer treated with traditional Chinese medicine
published in Chinese journals
Qi G1 , Lin X1 , Chen Z1 , Wang T1 , Ma B1
1 Evidence-Based Medicine Center, Institute of Traditional Chinese and
Western Medicine, School of Basic Medical Sciences, Lanzhou University,
Lanzhou 730000, China
Background: Randomized controlled trials (RCTs) of gastric cancer
treated with traditional Chinese medicine (TCM) have become
increasingly popular in China and have been published in large
numbers. However, the quality of reporting in these RCTs was
unknown. Objectives: To assess the reporting quality of RCTs
of gastric cancer treated with TCM. Methods: We searched the
Chinese Biomedicine Literature Database (CBM), Chinese Scientific
Journal Full-text Database (CSJD), Chinese Journal Full-text Database
(CJFD), and Wangfa database to the end of June 2010. The search
terms included ‘Traditional Chinese Medicine’ and ‘Chinese herbs’
(MESH or EMTREE), ‘Gastric Cancer’, ‘Stomach Cancer’ and ‘Stomach
Neoplasms’ (MESH or EMTREE). We also searched using the Google
search engine using the snowballing method. The reporting quality
of included studies was evaluated based on the CONSORT 2010
statement. Results: A total of 102 RCTs were included. Only 12
(11.8 %) RCTs included studies that reported specific inclusion and
exclusion criteria. Some important methodological components of
RCTs, such as: sample size calculation (0 %), randomization sequence
(16.7 %), allocation concealment (0.0 %) and blinding (2.0 %), were
incompletely reported. Only half of the trials (54.9 %) defined the
criteria of subjective measures, and none of them registered in any
clinical trials registry platform or agency. Conclusions: Although many
such RCTs have been published, the reporting quality is troubling. As a
potential key source of information for clinicians and researchers, not
only were many of these RCTs incomplete, some contained mistakes or
were misleading. Focusing on improving the quality of RCTs of TCM,
rather than continuing to publish them in great quantity, is urgently
needed in order to increase the value of these studies.
P1B121
An effective model for training social workers
in evidence-based practice in Korea
Hwang S1
1 Kyungpook National University, South Korea
Background: There has been a growing interest in evidence-based
practice (EBP) among social workers in Korea. Objectives: This study
was designed to evaluate the impact of various continuing education
training models on the EBP process with community practitioners.
Methods: A replicated one-group pre-test-post-test design was
used on an outcome measure that assessed the level of workshop
participants’ attitudes toward the EBP process. Results: There
were significant changes between pre- and post-measurements and
moderate to strong effect sizes (0.48 to 0.95) for the dependent
variable in the desired direction. Conclusions: Overall, the results
support the effectiveness of all three types of EBP training models for
Korean social work professionals.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
93
P1B122
The evaluation of the effectiveness of a postgraduate
training program of systematic review development
Weng YH1 , Shih YH2 , Lo HL3 , Kuo KN3 , Chen CF4 , Chiu YW2
1 Chang Gung Memorial Hospital, Chang Gung University College of
Medicine, Taiwan; 2 Institute of Population Health Sciences, National
Health Research Institutes, Taiwan; 3 College of Medicine, Taipei Medical
University, Taiwan; 4 Department of Surgery, Taipei Medical
University-Wan Fang Hospital, Taiwan
Background: Systematic reviews provide critical exploration,
evaluation, and synthesis of the unmanageable amount of information;
and separate the insignificant, unsound, or redundant deadwood in the
medical literature from the salient and critical studies that are worthy of
reflection. They have been regarded as the highest quality of evidence
for clinical practice. Objectives: The National Health Research
Institutes of Taiwan provided a series of education for postgraduate
health professionals to learn how to conduct systematic reviews. The
curricula of the two-day workshop included training on how to perform
meta-analysis using RevMan 5. The aim of this study was to investigate
the effectiveness of this training program. Methods: A pre- and
post-survey of participants was carried out to examine views related to
systematic reviews including changes in beliefs, attitudes, knowledge,
skills, and perceived barriers. Results: The participants’ professions
included: physicians, nurses, pharmacists, nutritionists, and public
health experts. After educational training, participants’ knowledge
and skill in the production of systematic reviews significantly increased
(P < 0.001). They were more likely to believe that systematic reviews
can improve the quality of patient care (P < 0.001), and thus to
support the implementation of systematic reviews (P < 0.001). The
prevalence of perceived barriers to produce systematic reviews also
declined (P < 0.001). In addition, the most common barrier to
meta-analysis was lacking English language skills. Conclusions: The
belief, attitude, knowledge, and skill in relation to systematic reviews
were improved after the training program. This suggests that education
is useful in disseminating systematic reviews.
P1B123
Complex interventions required to comprehensively
educate allied health practitioners on evidence-based
practice
Dizon JM1 , Grimmer-Somers K2
1
International Centre for Allied Health Evidence, University of South
Australia, Australia and University of Santo Tomas, Philippines;
2 International Centre for Allied Health Evidence, University of South
Australia, Australia
Background: There is currently no strong evidence regarding the
most effective training approach for allied health professionals that will
support them to consistently apply the best research evidence in daily
practice. Current evidence-based practice training tends to be ‘one size
fits all’ and is unlikely to be appropriate for all allied health disciplines,
because of the variability in their tasks and scope of practice. The
scant body of evidence regarding the effectiveness of evidence-based
practice training for allied health practitioners provides some support
for improving knowledge and skills, but equivocal evidence about
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
influencing behaviours and attitudes. Findings: This paper is a
reflection of the findings of our systematic review in relation to issues
associated with training allied health professionals in evidence-based
practice (EBP), particularly the training required to change their attitudes
and behaviours towards research evidence implementation. There are
key issues to consider when designing effective EBP training programs
for allied health practitioners, such as; disciplines differ in terms of
their contexts and scopes of practice, their clinical training and the
research that matter to their day-to-day practice. We propose a
new model of evidence-based practice training, based on the Medical
Research Council’s concept of complex interventions and Hawe and
colleagues’ work regarding standardizing complex interventions by
considering the interventions as fixed (core, constant) components and
variable components. We suggest that EBP training should be similarly
considered, as a complex intervention, with fixed and variable aspects.
Conclusions: We believe that by offering training in evidence-based
practice based on complex interventions relevant to the needs of the
attendees, using fixed and variable components, there may be greater
success in significantly influencing knowledge skills, attitudes and
behaviours.
P1B124
Evidence based practice workshop for senior managers
in Ethiopia
Abdulwadud O1 , Young T2 , Hailu K3 , Rangwala A4 , Midekssa M3 ,
Volmink J2 , Garner P5
1 Ministry of Health, Ethiopia; 2 SA Cochrane Centre, SA; 3 AIHA-TC,
Ethiopia; 4 AIHA, Washington USA; 5 Liverpool STM, UK
Background: Ethiopia is a low-income country and has free access to
The Cochrane Library (CL). There is only one Cochrane review author
in Ethiopia and awareness about the Cochrane Collaboration (CC), the
South African Cochrane Centre (SACC), the CL and evidence based
practice (EBP) is poor. Objectives: To raise senior managers’ and
partners’ awareness about EBP, the CC, SACC and lay a foundation to
strengthen EBP. Methods: After four months of planning, a one-day
national awareness-raising workshop was conducted on December 1,
2010 in Addis Ababa by the American International Health Alliance
(AIHA), the SACC and Liverpool School of Tropical Medicine (LSTM).
Results: Sixty-five senior managers from Ethiopia, South Africa,
Nigeria, Namibia, Zambia, and Tanzania attended the workshop.
The interactive workshop consisted of five sessions. It started with
international and African initiatives in EBP including the work of the
CC and the SACC in promoting EBP in Africa. Participants then
presented on how EBP is promoted or used in their institutions and
during the 4th session, a demonstration of the CL and how to search
for evidence was given. Lastly, participants discussed how to advance
EBP in their countries. Conclusions: The national workshop was very
successful and participants learnt from it. However EBP faces many
challenges in Ethiopia: there is only one Cochrane review author in
Ethiopia, awareness is poor, and no institution is actively leading and
supporting EBP activities. To address the challenges, a committee of
key stakeholders was established to take over the leadership role to
advocate for EBP and coordinate all the efforts to promote EBP in
Ethiopia.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
94
P1B125
Measuring the quality of end-of-life care: Palliative care
of cancer patients
KO W1
1 National Institute for Cancer Research, Italy
Background: Among various chronic diseases, cancer is one of which
could cause enormous pain to a patient during the final stages of one’s
life. The recent Hospice Movement since the 1960s marked a revival of
palliative care as an alternative option for terminally ill patients. While a
lot of palliative care emphasises the role of pain soothing, there are also
substantial publications that try to measure the quality of the end-of-life
care1,2 . Objectives: The quality of the end-of-life care is important as
it enhances the capability of cancer patients to die in a dignified and
preferred manner. However, existing indicators have different focuses
and hinder cross-country comparisons. Methods: This paper attempts
to do a systemic literature review of the European publications on
the measurement of the concept of end-of-life care, with a particular
attention given to cancer patients. Despite the differentiations on the
symptoms and severity of pain suffered by cancer patients at the final
stage of their lives, the paper aims at surveying the existing criteria
employed in measuring the quality of the care offered. This is a precursor
for attempting to find how a standardization of the measurement is
viable for comparisons. Conclusions: The paper concludes with a
discussion on the possibilities of deriving a comparable measurement
on the quality of end-of-life care of cancer patients.
References
1. Engelberg RA, Downey L, Wenrich MD, Carline JD, Silvestri GA,
Dotolo D, et al. (2010) Measuring the quality of end-of-life care.
Journal of Pain and Symptom Management, 39(6):951–71.
2. Singer PA, Bowman KW (2002) Quality end-of-life care: A gloabal
perspective. BMC Palliative Care, 1(1):4.
P1B126
Development of a systematic review unit of the Botucatu
Medical School (FMB), São Paulo State University in Brazil
El Dib R1 , Rudge MV1 , Schellini SA1 , Barretti P1 , Cataneo A1 , Nogueira
CR1 , Fecchi D1 , Betini M1
1 São Paulo State University, Brazil
Background: One of the aims of the Botucatu Medical School (FMB)
is to introduce the principles of Evidence-Based Medicine (EBM) to
clinicians, physician residents, graduate and postgraduate medical
students, and health professionals. The new model of self-education
allows the production of systematic reviews and meta-analyses and
to help in the design and conduction of randomized clinical trials at
the FMB Clinical Hospital. Objective: We present the development
of a systematic review unit (SRU). Methods: A systematic review unit
was established consisting of a scientist in EBM (RED) trained at
McMaster University and worked in the Brazilian Cochrane Centre,
two surgeons (SAS, AC), an information retrieval specialist (MB), three
clinical physicians (MVR, PB, CRN), and one researcher (DF). Weekly
meeting discussion of all members of the SRU is performed to access
relevance prior to conducting a systematic review. Systematic reviews
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
and meta-analyses are conducted according to the principles of the
Cochrane Collaboration and written based on the PRISMA statement.
Furthermore, methodological studies are performed dealing with the
problem of lack of clinical trials. All teachers, health professionals
and students are welcome to perform systematic reviews. They are
trained in EBM by workshops coordinated by Prof. Dr. Regina El Dib.
Results: 22 projects for systematic reviews are currently underway
(six are registered in the Cochrane groups). Other three completed
systematic reviews were done and one of them paved the way for the
successful design and conduct of a pragmatic trial. This trial and review
were funded by the National Council for Scientific and Technological
Development (CNPq). The other two reviews were also funded by CNPq
and by São Paulo Research Foundation (FAPESP). Three clinical trials
have been designed according to the reviews protocol. Conclusions:
A SRU increased the production of systematic reviews and clinical trials
to help patients make healthcare decisions.
P1B127
Introduction to evidence based practice in the
Humanitarian Services of the Belgian Red Cross-Flanders:
Experiences from a newly developed workshop
Dieltjens T1 , Adriaenssens L1 , Pauwels N1 , Van de Velde S1
1
Belgian Red Cross-Flanders, Belgium
Background: Evidence-based practice (EBP) has become an integral
part of the activities and programmes developed by the Belgian
Red Cross-Flanders. The Humanitarian Services of the Belgian Red
Cross-Flanders work with volunteers and staff that are, for the most
part, unfamiliar with EBP. Effective communication and training is
essential to avoid misunderstandings about the concept of EBP and
to engage them towards EBP. Objectives: To evaluate a workshop
designed to introduce EBP to a public with very different backgrounds.
Methods: The main learning objectives were that the participants
understand the importance of EBP and have a critical attitude towards
the validity and usefulness of various sources of information. We will
use brief presentations combined with an interactive approach and
group work to facilitate the learning of the critical components of EBP.
The workshop is designed in collaboration with didactical experts. A
questionnaire was developed to measure the attitude of the participants
towards EBP before and after the course. Results/Conclusions: The
workshop will be piloted in June 2011. We will present the format of
the workshop and the results of the pilot study during the Cochrane
Colloquium.
P1B128
Educational models and their application in medical
research training: A meta-analysis
Ejaz K1 , Zia N1 , Shamim MS2
1
Aga Khan University Hospital, Pakistan;
Pakistan
2
Memon Medical Institute,
Background: Different educational models by theorists take into
account several elements of adult learning. In medical research training
such models are seldom reported. Models by Kolb and Dunn & Dunn
have proven to be successful in adult education. With reference
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
95
to medical research training the application of these models may
prove to be helpful. Objectives: To assess the impact of various
educational models on medical research training. Methods: This
is an ongoing meta-analysis of observational studies. Our inclusion
criterion is original research from all years and languages. Only those
articles which have full-text assess are included. A search strategy
based on our inclusion criteria was developed. To-date, ten databases
have been searched. Data is being extracted using MeSH and key
words, ‘medical’, ‘research’, ‘education’, ‘teaching’ and ‘educational
models’. The retrieved articles are being reviewed and tabulated using
a standardized form. The form includes: study description, rationale for
selection, coding details, assessment of confounders, study quality and
heterogeneity. The description of statistical methods, tables and figures
are also being assessed. Results: We are currently in the process of
identifying and screening the articles according to our inclusion criteria.
So far 50,000 titles, from ten major databases, have been reviewed.
Five met our inclusion criteria. We are searching further. We are
removing the duplicate records and abstract-only articles. The full-text
articles not fulfilling our inclusion criteria will be excluded and the
reasons for their exclusion will be noted. Conclusions: As this is still
an ongoing project we cannot make a final conclusion. However, the
team’s current observation is that a minimal amount of original work is
reported in online databases. Original research projects which involve
medical research teaching use ad-hoc methods instead of standardized
models. This may result in a future development of educational model
for medical research training.
P1B129
Mentoring clinicians in conducting systematic reviews
through a clinical fellowship program
Chau M1 , Pitt V1 , Green S2 , Gruen R1 , Bragge P1
1
National Trauma Research Institute & Monash University, Australia;
2 Australasian Cochrane Centre & Monash University, Australia
Background: Knowledge of current clinical evidence combined with
clinical experience is pivotal to evidence-based practice. However,
successfully engaging clinicians in the process of: searching for,
appraising, and interpreting research evidence is challenging due to
competing clinical commitments. Objectives: The Neurotrauma
Evidence Translation (NET) Clinical Fellowship Program aims to build
capacity for evidence-based practice and clinical leadership through
mentoring clinicians to conduct systematic reviews to inform clinical
practice. Methods: Planning the design and implementation of the
NET Clinical Fellowship Program involved a review of similar programs
used to engage clinicians in systematic reviews. The program employs
a research fellow and research assistant with extensive systematic
review experience who are responsible for recruiting appropriate
candidates and supporting them in identifying a clinical question, title
registration, one-to-one training and mentorship through the entire
systematic review process. Topic selection is supported through a
related neurotrauma evidence project, The Global Evidence Mapping
(GEM) Initiative, which has identified high priority neurotrauma topics
where systematic reviews are lacking. The Cochrane Injuries Group has
also been consulted to identify potential review topics (including reviews
in need of updating). Results: The program commenced in 2010
and has recruited four clinical fellows to date. Potential challenges,
and strategies for addressing these, are presented based on similar
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
reported experiences in the literature. Conclusions: Programs that
engage clinicians in systematic reviews can support implementation of
evidence into practice and promote clinical leadership. Consideration
should be given to: how clinicians are recruited and engaged within
programs, what topics should be addressed, how adequate training
and support will be provided to meet candidates’ needs, and how the
research findings can impact on local and international practice.
P1B130
Current practice of Filipino physiotherapists in Metro
Manila, Philippines
Dizon JM1 , Grimmer-Somers K2 , Kumar S2
1 International Centre for Allied Health Evidence, University of
South Australia, Australia and University of Santo Tomas, Philippines;
2 International Centre for Allied Health Evidence, University of South
Australia, Australia
Background and Objectives: Country-by-country differences exist
in which physical therapists make decisions regarding treatment. In the
Philippines, there is no published information about physical therapists
and what underpins their decisions in patient treatment. Thus, we
conducted this study aimed at creating a profile of physical therapists
in the Philippines and their bases for selecting of treatment. Methods:
We used the Physical Therapy Profile Questionnaire (PTPQ) from our
previous study. The PTPQ was administered in two ways. Paper copies
were distributed in the national capital region of the Philippines, where
71 % of physical therapists practice and, a link to an electronic copy
of the questionnaire via the Survey monkey software. Results: A
total of 133 (response rate of 61.5 %) physical therapists took part in
this survey. Seventy-three (55 %) were females and 60 (45 %) were
males. Only 12 (9 %) completed a master’s degree related to physical
therapy. The largest percentage of respondents had been practicing for
five years or less, N = 89 (67 %). The majority of respondents were
in general areas of practice. Most physical therapists were based in
hospitals and majority performed clinical work with/without other roles
in practice. The most common bases for treatment interventions in
daily practice were: (1) medical prescription, (2) clinical experience and
(3) undergraduate education. When faced with a new or difficult case,
the most common bases for treatment decisions were: (1) medical
prescription, (2) clinical experience and hospital protocol (equally
rated) and (3) journal articles. Conclusions: Physical therapists in
the Philippines are composed of a young population and are mostly
based in hospitals. Treatment is based on the doctors’ referral,
which includes a medical prescription of treatment. This challenges
their capacity to deliver treatment with the best evidence. Therefore,
training on evidence-based practice may be useful for them to make
recommendations for treatment with the best evidence.
P1B131
Access alone does not guarantee use: The need for training
to use evidence based resources
Sriganesh V1
1 QMed Knowledge Foundation, India
Background: In the last few years, access to evidence based resources
has increased in developing countries. However there is not a
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
96
proportional increase in the qualitative and quantitative use of such
resources. Observation: As information professionals helping health
professionals in a developing country, we noticed that many of them
were not aware of important resources and their access rights. If
they knew about them, they did not know about their key features.
For instance they were not aware of MeSH and the Single Citation
Matcher function in PubMed, or about the importance of forest plots
in Cochrane reviews. Our intervention: In the last three years we:
delivered lectures about evidence based resources, informed about free
access to the Cochrane Library and conducted training programs in the
use of PubMed. This sensitization has resulted in requests for more
such programs and also better utilization. Recently the Medical Council
of India has also stressed the need for such programs in medical school
programs. Steps ahead: We are planning to work in partnership
with medical schools to introduce ‘Train the trainer’ programs for the
effective use of evidence based resources. Through these programs, we
wish to create pools of trainers in every medical school. We would like to
measure the impact of our programs and see if the awareness promotion
and training makes a difference in the quantitative and qualitative use
of EBM resources; especially the Cochrane Library, for which we have
a country wide license. Conclusions: We believe that if students learn
to use evidence based resources correctly, it would become routine
for them to use them when they become residents, medical teachers
or practitioners. It is very important that if country-wide access to
resources like the Cochrane Library is provided, it must be supplemented
with awareness and proper training to translate to evidence based
healthcare!
P1B132
Igniting minds: Use of Cochrane resources as educational
tools for training medical students in developing countries
Chandra S1
1 INFORMER/South Asian Cochrane Network, India
Background: Academic centers of excellence which focus equally on
research, education and patient-care are needed to cater to current
healthcare demands. Developed countries have been successful in
integrating research into their medical curriculum, transitioning from
a textbook-based system to an evidence-based one. Developing
countries like India lack the necessary infrastructure, training and
policies to implement such a system. Intervention: Based on
this need to groom the next generation of healthcare professionals, a
national platform to advocate and encourage the work of young medical
researchers was formed under the banner of ‘The Forum for Medical
Students’ Research, India’ (INFORMER). In 2009, at the Forum’s annual
congress, Dr Prathap Tharyan, director of the South Asian Cochrane
Network delivered a lecture on Evidence-based medicine (EBM). As
a continuation, collaboration was forged between INFORMER and
the Cochrane Centre based in Vellore. This collaboration was
furthered during the Third South Asian Regional Symposium on
Evidence-Informed Healthcare, at Christian Medical College, Vellore.
Here representatives from both organizations deliberated and, with
input from the Symposium participants, enlisted measures which
they could adopt to increase student participation in the Cochrane
Collaboration’s activities. Results: The measures included the
setting up of a formal students’ group linked with the regional
Cochrane Centre, journal club discussions, introductory workshops on
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
evidence-informed healthcare and Cochrane reviews, training modules
targeted at teaching the basics of EBM to students, involvement of
students in writing and various other aspects of Cochrane reviews,
and establishing a database to link novices and mentors with similar
research interests. Since the implementation of the aforementioned
methods, there has been a surge in Cochrane activity from the Indian
subcontinent. Conclusion: The Cochrane Collaboration can serve
as a good quality, accessible and affordable educational resource
for medical students, especially in developing countries which have
inadequate academic centers of excellence.
Information and communication technology
Day 20 Oct (PB133–PB134)
P1B133
Do e-interventions for medication management
in ambulatory care pose a threat to patient safety?
A systematic review
Carling C1
1 Norwegian Knowledge Centre for the Health Services
Background: Enthusiasm for the implementation of e-interventions
for medication management to decrease adverse events is richly
supported by empirical studies. However, there is concern that they
can introduce new problems and facilitate or cause events harmful to
patients. Objectives: To identify and summarize empirical evidence
of e-intervention outcomes in ambulatory settings that represent a risk
for patient safety. Methods: We conducted a systematic search of the
literature for studies in which e-interventions, i.e., electronic patient
record, clinical decision support, electronic transmission of prescriptions
to pharmacies, or electronically-generated paper prescriptions, were
employed in the process of medication management in the ambulatory
setting. We included studies with quantifiable outcomes and a
wide range of designs to increase the probability of finding seldom
or previously unknown adverse outcomes in studies not necessarily
designed or powered to identify them. Results: We identified
87 studies that met inclusion criteria whereof 55 were randomized
controlled trials (RCTs) and two quasi-randomised. None set out
to reveal threats to patient safety. One observational study found
overall worse outcomes, i.e. more new e-prescriptions needed
pharmacist intervention than new paper prescriptions. Two other
observational studies reported that some out-patients did not retrieve
their e-prescribed drugs from pharmacies. One RCT reported more
major bleeds in computer-generated anti-coagulant dosing, although
computer dosing resulted in better clinical events overall. Otherwise,
e-interventions did not result in overall inferior patient or process
outcomes. Conclusions: Evidence of threats to patient safety in
the identified studies was scant and mainly anecdotal. Observational
studies may identify previously unknown or seldom outcomes that
are threats to patient safety. Studies using robust designs such as
the randomized controlled trial powered to reveal seldom outcomes
are needed to test hypotheses generated based on findings from
observational studies.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
97
P1B134
Are computerized clinical decision support systems
effective at improving the process of care and patient
outcomes, and what makes an effective system?
A decision-maker-researcher partnership systematic
review
Roshanov P1 , Hemens BJ1 , Nieuwlaat R1 , Souza N1 , Sahota N2 ,
Wilczynski J1 , Fernandes N1 , Haynes RB1
1 McMaster University, Canada; 2 University of Saskatchewan, Canada
Background: Despite decades of research on computerized clinical
decision support systems (CCDSSs), results from rigorous evaluations
remain mixed and little is known about what makes an effective
system. Previous systematic reviews seeking characteristics important
for success were limited by small sample sizes and poor quality of
primary studies. Objective: To determine if CCDSSs are effective
at improving care processes or patient outcomes, and to identify
characteristics associated with effectiveness. Methods: A systematic
review of randomized trials comparing use of CCDSSs to usual practice
or non-CCDSS controls in clinical care settings. We conducted
literature searches to January 2010 in bibliographic databases and
scanned reference lists. Guided by partnerships with clinicians
and senior hospital administrators, we considered over 50 trial and
system characteristics. Authors of all included primary studies were
contacted to provide additional information and to help select features
potentially associated with effectiveness. Two outcome categories
were analyzed independently: ‘process of care outcomes’ (such as
appropriate monitoring of disease) and ‘patient outcomes’ (such as
values of test results). Results: In the 166 included trials, success rates
for process of care and patient outcomes averaged 60% and 24%,
respectively, across 6 categories of application: toxic drug monitoring
and dosing, acute care, primary preventive care, chronic disease
management, diagnostic test ordering, and drug prescribing. We will
use univariate and multivariate analyses to identify features significantly
associated with effectiveness. Conclusions: CCDSSs affect the
process of care but have limited benefit for patients. Our analysis
will provide empirical guidance on effectiveness, optimal design and
implementation.
Methods for preparing reviews (non-statistical)
Day: 20 Oct (P2A135–P2A196)
P2A135
The COMET (Core Outcome Measures in Effectiveness
Trials) initiative
Williamson P1 , Altman D2 , Blazeby J3 , Clarke M4 , Gargon E1
1 University of Liverpool, UK; 2 University of Oxford, UK; 3 University of
Bristol, UK; 4 Queens University Belfast, UK
Background: Systematic reviews are hampered by inconsistencies in
outcomes assessed and reported in otherwise eligible studies. Many
meta-analyses have to exclude key studies because relevant outcomes
were not reported. Much could be gained if an agreed minimum set
of appropriate and important outcomes was measured and reported in
all clinical trials in a particular area. Why standardise outcomes?
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
The design of new trials would be simplified, their risk of measuring
inappropriate outcomes would be reduced, and selective reporting of
outcomes would be less likely. It would be easier to compare, contrast
and combine studies in systematic reviews, and the choice of primary
and secondary outcomes for reviews would be more straightforward.
Core outcome sets would help review authors to present their findings
clearly and succinctly, for example within Summary of Findings tables.
Aims: COMET initiative brings together researchers interested in
standardised outcome sets, with well-attended international meetings
in 2010 and 2011. COMET aims to foster and facilitate research
by providing guidance on developing a core outcome set, methods
to include user involvement in this process, and preparing reporting
standards for such projects. Work is ongoing to identify, collate and
maintain relevant resources in a searchable database on the internet.
More than 50 completed projects in various areas of health/health
care have been identified. Several examples of planned and ongoing
work have also been recorded. The database will be demonstrated
and progress to date will be presented. Implications for The
Cochrane Collaboration and systematic reviews: If successful,
COMET will help review authors choose outcomes, and will increase
the likelihood that these outcomes have been measured in eligible
studies, thereby decreasing the likelihood of important studies being
excluded. By improving the evidence base for reviews, COMET
will make it easier for people to make well-informed decisions in
health care.
P2A136
Purity or pragmatism: how does systematic review
methodology from health care translate to the field
of development?
Stewart R1 , van Rooyen C2 , de Wet T2
1 EPPI-Centre and Consumers and Communication Review Group, UK;
Centre for Language and Culture, University of Johannesburg, South
Africa; 2 Dept of Anthropology and Development Studies & Centre for
Language and Culture, University of Johannesburg, South Africa
Background: Systematic review methodology pioneered in health
care has been applied increasingly to questions of importance to
lower- and middle-income countries. The development industry itself
is also engaging with the evidence-based approach with initiatives
such as the Abdul Latif Jameel Poverty Action Lab (J-PAL) and the
International Initiative for Impact Evaluations (3ie). However, the
evidence-based approach does not necessarily translate smoothly
to the field of development. Systematic review methodology is
based on the gold standard of randomised controlled trials. This
gold standard is now widely accepted in health care, but such a
narrow approach to evidence is not so easily accepted in a field
where trials are often lacking, solutions urgently required, and
regional and local variations common. Methods: In the last
twelve months, the UK Department for International Development
(DFID) has undertaken a pilot, commissioning academics to undertake
systematic reviews in development. We undertook one such review
and in doing so made a number of pragmatic methodological
compromises. Having completed and disseminated our review, we
have now reflected on these compromises to assess what might be
learnt for future reviews. Results: We have identified lessons for
future commissioners and researchers employing systematic review
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
98
methodology in development. These relate to multi-disciplinary
teamwork, application of regional filters, drawing on evidence from a
range of study types, and exploring mechanisms for change through the
development and testing of causal pathways. Conclusion: There are
still unanswered questions for reviewers in the field of development,
particularly surrounding questions of study design, generalizability
and applicability. Whilst adhering to the highest standards of
systematic review methodology may not be possible (or supported
by all), through transparent and complete reporting, we believe these
discussions can advance and a balance between purity and pragmatism
achieved.
P2A137
Reviewing psychometric properties of HRQL measures
in mental health: Practical experience
Papaioannou D1 , Brazier J1
1
ScHARR, University of Sheffield, UK
Background: Generic health-related quality of life measures (HRQL)
are claimed to be applicable across disease areas. It is important to have
robust methods of reviewing evidence that HRQL measures are valid
and responsive in individual conditions. Objectives: To review the
methods of systematically reviewing the literature on the validity and
responsiveness of the SF-36, SF-12, SF-6D and EQ-5D in five mental
health conditions. Methods: Systematic reviews were undertaken
in schizophrenia, depression, anxiety, bipolar disorder and personality
disorders. Ten databases were searched up to August 2009. Studies
were appraised and data extracted. A narrative synthesis was performed
on construct validity including known groups validity (KGV) (ability to
detect differences in HRQL scores between groups), convergent validity
(CV) (strength of association between generic HRQL and other related
measures, e.g. symptoms or function) and responsiveness (R) (i.e.
changes in scores in responders/non-responders to treatment and
correlation with changes in related measures). Results: There were
several problems in the review process including determining eligibility
at the abstract level and lack of relevant quality assessment criteria.
The greatest problem was determining how to synthesize the data
from included studies, particularly in relation to the robustness of
methods used to assess KGV, CV and R. For example, schizophrenia
studies demonstrated crude KGV by detecting a difference between
scores of the general population and individuals with schizophrenia.
The methods used to assess CV by measuring the correlation between
HRQL and clinical measures might not truly indicate its validity or
otherwise. Conclusion: There are no guidelines on how to perform
such reviews. However, such reviews are essential to determine
if generic HRQL measures are valid for use in cost-effectiveness
analyses for all diseases/conditions. Determining the methods that are
appropriate to demonstrate KGV, CV and R is essential to developing
the methods of systematic reviewing psychometric properties of HRQL
measures.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P2A138
External validity and generalizability of systematic reviews
in primary health care: Adapting the methodology to
conduct the review
Nasser M1 , van Binsbergen JJ2 , van de Laar FA2 , van Weel C2 , Buechter R3
1
Peninsula Dental College, University of Plymouth, Plymouth, UK;
2 Department of primary care and community care, Radboud University
Medical Centre, Nijmegen, Netherlands; 3 Department of Health Information,
German Institute for Quality and Efficiency in Health care (IQWiG), Cologne,
Germany
Background: Systematic reviews of health care interventions can be
helpful sources to inform clinical decision-making in primary care but
need to provide adequate information for decision makers to make
judgments about the external validity of the included studies and about
the applicability and transferability of the results to their individual
patients. Objectives: We aim to introduce a guide on how systematic
reviewers can adapt the methodology of their systematic reviews
to make them more relevant to primary health care. Methods: We
developed a conceptual framework based on a literature search, guided
by an extrapolation checklist for systematic reviews, the experience of
the author team, and the Wonca definition of primary care for systematic
reviews. The framework was discussed at the Heelsum workshops that
aim to raise discussion on the alliance between primary care and public
health. We piloted the checklist using 10 systematic reviews. Results:
Based on the literature search, we identified sixteen questions across
different steps of conducting a systematic review from identifying
questions, searching literature towards quantitative synthesis that
could guide the systematic reviews. The questions also cover the
different aspects of primary care that we identified in the definition
provided by Wonca for family medicine: diversity of patient groups,
contextual factors, and person-centred approach. Conclusions: The
16 identified questions can help systematic reviewers in improving the
relevance of their review to primary care. We are currently evaluating
the external validity of 100 systematic reviews including Cochrane
reviews using the conceptual framework and will present those results
in the Cochrane Colloquium, October 2011.
P2A139
Prioritising whether and when to update Cochrane reviews
Hopewell S1 , Takwoingi Y2 , Sutton A3
1
UK Cochrane Centre, UK; 2 Public Health, Epidemiology and Biostatistics,
University of Birmingham, UK; 3 Department of Health Sciences, University
of Leicester, UK
Background: Given the increasing workload of Cochrane Review
Groups and review authors, there is a need to develop an
evidence-based approach for updating Cochrane reviews based on
priority rather than the current arbitrary and often unmet two years.
Objectives: To develop and validate a decision tool to determine
when a Cochrane review should be prioritized for updating. Methods:
We have refined and amalgamated two complementary methodologies
proposed for prioritizing systematic review updates namely: i) a
qualitative tool based on a broad range of updating signals including:
publication of new study, information from existing studies, changes
in methodology, user feedback, and clinical question; and ii) formal
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
99
statistical methods which assess the inclusion of new studies and the
likelihood that a review’s conclusions will change, if updated, based
on the primary outcome meta-analysis within the review. Formal
piloting was used to refine and test the validity and reliability of
the enhanced tool based on a retrospective sample of reviews. All
Cochrane reviews flagged as updated during 2009 were identified. For
each of the reviews, the following data were extracted: i) whether
the conclusions of the update had changed; ii) the meta-analysis of
the primary outcome; and iii) the corresponding meta-analysis in the
previous version of the review. In doing this, new trials added to
the review in the most recent update were identified. The statistical
methods (programmed using STATA macro metarank) to evaluate the
quantitative signals for the need to update a Cochrane review for each
meta-analysis dataset was then run on all identified meta-analyses.
We then compared the results of the statistical analyses with results of
the updated reviews to determine priorities for updating. Results: The
results of this formal pilot and a demonstration of the decision tool will
be presented at the Colloquium.
P2A140
Uncertainty in control group risk can, in systematic
reviews, undermine confidence in estimates of effect
Guyatt G1 , Vandvik P2 , Spencer F1 , Akl E3 , Falck-Ytter Y4 , Schunemann H1
1 McMaster University, Canada; 2 Norwegian Knowledge Centre for the
Health Services, Norway; 3 State University of New York at Buffalo, USA;
4 Case Western Reserve University, USA
Background: In general, systematic reviews focus on estimates of
relative effect when presenting binary outcomes. Ratings of confidence
in estimates (quality of evidence) therefore reflect confidence in relative
effect estimates. Treatment decisions, however, must be based on
trade-off in absolute effects, often calculated by applying relative risk
to estimates of control group risk. Objectives: To explore how
uncertainty in control group risk influences confidence in estimates
of effect in the context of evidence summaries prepared for clinical
practice guidelines. Methods: We surveyed the evidence summaries
prepared for the ninth iteration of the American College of Chest
Physicians Antithrombotic Guidelines looking specifically at sources of
estimates of control group risk and possible sources of uncertainty
in control group risk. Results: We found numerous instances
when uncertainties in estimates of control group risk potentially
compromised confidence in estimates of absolute treatment effect.
Examples included the following. Risk of bias in estimates of
symptomatic thrombosis risk arose in randomized trials that used
screening venography to diagnose asymptomatic thrombosis, and in
observational studies that failed to report antithrombotic prophylaxis.
Risk of bias also increased in observational studies that used billing
codes to determine thrombotic outcome diagnoses. Small sample size
in observational studies resulted in imprecision of estimates of control
group risk. Reliance on estimates from randomized trials raised issues
of directness or applicability to clinical populations. Inconsistency in
results from observational studies sometimes undermined confidence
in estimates of control group risk. Conclusions: Limitations in
the quality of evidence regarding estimates of control group risk is
an under-appreciated source of decreased confidence in estimates
of absolute treatment effects in evidence summaries from systematic
reviews.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P2A141
Estimating risk differences in patient-important events
when studies have focused on surrogate outcomes:
Application to systematic reviews and practice guidelines
Guyatt G1 , Akl E2 , Murad MH3 , Falck-Ytter Y4 , Gould M5 , Eikelboom J1
1 McMaster University, Canada; 2 State University of New York at Buffalo,
USA; 3 Mayo Clinic, Rochester, USA; 4 Case Western Reserve University,
USA; 5 Keck School of Medicine of the University of Southern California,
USA
Background: Four chapters in the ninth iteration of the American
College of Chest Physicians antithrombotic guidelines addressed
thrombosis prevention. Assessing risk differences (RDs) in symptomatic
thrombosis proved problematic because most studies focused on
a surrogate outcome, asymptomatic thrombosis. Objectives: To
implement optimal strategies assessing RD. Methods: Panels
considered three approaches: i) using estimates of symptomatic
thrombosis from randomized controlled trials (RCTs) for both control
group risk (CGR) and relative risk (RR); ii) using estimates of
symptomatic thrombosis from RCTs for CGR and estimates of a
composite of asymptomatic and symptomatic events from RCTs for RR;
and iii) using estimates of symptomatic thrombosis from observational
studies for CGR and estimates of a composite of asymptomatic and
symptomatic events from RCTs for RR. Results: All three approaches
proved feasible and applicable; examples follow. The chapter dealing
with prophylaxis in hospitalized medical patients used approach A
and estimated RD −34 symptomatic thrombotic events per 1000
(95% CI −51 to 0), limited by imprecision. The chapter addressing
orthopedic surgical patients, using approach B, chose an untreated
CGR from the low-molecular-weight heparin (LMWH) arm of recent
RCTs after accounting for the effect of LMWH against placebo in
earlier studies. Using the RR from RCTs for the combined endpoint
of symptomatic and asymptomatic deep venous thromboses (DVTs),
they estimated the effect of unfractionated heparin as RD of −10
(95% CI −9 to −12). Using asymptomatic events required rating
down for indirectness. The chapter addressing non-orthopedic surgery
thromboprophylaxis used approach C for the intermittent pneumatic
compression recommendation. An observational study provided an
estimate of 26 events per 1,000. Use of RR from RCTs resulted
in an RD estimate of −13 per 1000 (95% CI −7 to −20) rated
down for indirectness because asymptomatic events were used for
RR. Conclusions: A variety of approaches estimating risk differences
when RCTs have focused on surrogate endpoints proved useful.
P2A142
The evidence base for interventions delivered to children
in primary care: An overview of Cochrane systematic
reviews
Gill P1 , Wang KY1 , Mant D1 , Hartling L2 , Heneghan C1 , Perera R1 ,
Klassen T3 , Harnden A1
1 Department of Primary Health Care, University of Oxford, Oxford, UK;
2 Alberta Research Centre for Health Evidence, Department of Pediatrics,
University of Alberta, Edmonton, Alberta, Canada and Cochrane Child
Health Field, Department of Pediatrics, University of Alberta, Edmonton,
Alberta, Canada; 3 Manitoba Institute of Child Health, Department of
Pediatrics and Child Health, University of Manitoba, Winnipeg, Manitoba,
Canada and Cochrane Child Health Field, Department of Pediatrics,
University of Alberta, Edmonton, Alberta, Canada
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
100
Background: As a first step in developing a framework to evaluate
and improve the quality of care of children in primary care, there is a
need to identify the evidence base underpinning interventions relevant
to child health. Objective: To identify all Cochrane systematic reviews
relevant to the management of childhood conditions in primary care
and to assess the extent to which Cochrane reviews reflect the burden
of childhood illness presenting in primary care. Methods: We used
the Cochrane Child Health Field register of child-relevant systematic
reviews to complete an overview of Cochrane reviews related to
the management of children in primary care. We compared the
proportion of systematic reviews with the proportion of consultations
in Australia, US, Dutch and UK general practice in children. Results:
We identified 396 relevant systematic reviews; 358 included primary
studies on children while 251 undertook a meta-analysis. Most reviews
(n = 218, 55%) focused on chronic conditions and over half (n = 216,
57%) evaluated drug interventions. Since 2000, the percentage of
pediatric primary care relevant reviews only increased by 2% (7% to
9%) compared to 18% (10% to 28%) in all child-relevant reviews.
Almost a quarter of reviews (n = 78, 23%) were published on
asthma treatments which only account for 3–5% of consultations.
Conversely, 15–23% of consultations are due to skin conditions
yet they represent only 7% (n = 23) of reviews. Conclusions:
Although Cochrane systematic reviews focus on clinical trials and do
not provide a comprehensive picture of the evidence base underpinning
the management of children in primary care, the mismatch between
the focus of the published research and the focus of clinical activity
is striking. Clinical trials are an important component of the evidence
base and the lack of trial evidence to demonstrate intervention
effectiveness in substantial areas of primary care for children should be
addressed.
P2A143
Variability of outcome measures among trials of topical
Chinese herbal medicines for musculoskeletal injuries
Wang L1 , Yuan Q1 , Li X1 , Li Y1
1
Chinese Cochrane Centre, China
Background: Traditional Chinese herbal medicines showed promising
effects for musculoskeletal injuries. However, most trials addressing
the effects of topical Chinese herbal medicines are of low quality, often
with variable and subjective outcome measures. Objectives: To assess
the variability of outcome measures among trials of topical Chinese
herbal medicines for musculoskeletal injuries. Methods: We searched
PubMed, EMBASE, CENTRAL, Chinese Biomedical Database, CNKI, and
VIP from the inception to December 2010. Randomized trials of topical
Chinese herbal medicines for musculoskeletal injuries were included.
We analyzed the outcome measures of the trials. Results: Forty-five
trials were included. Seventeen trials reported the overall efficacy with
four categories of ‘‘cured’’, ‘‘excellent’’, ‘‘good’’, and ‘‘no effect’’, and
nineteen trials with three categories of ‘‘excellent’’, ‘‘good’’, and ‘‘no
effect’’. However, criteria to determine the efficacy varied among trials.
Among seventeen trials, ten reported curing using ‘‘symptoms and
physical signs of patients disappeared, and physical function recovered
to normal level’’; three adding ‘‘overall improvement >95%’’; two
using ‘‘>90%’’. Two other trials did not report any definition of curing.
The thresholds of ‘‘excellent’’ efficacy of overall improvement ranged
from ‘‘>75%’’ (1 trial), ‘‘>70%’’ (6 trials), ‘‘>2/3’’ (11 trials), to
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
‘‘>50%’’ (2 trials). Twelve trials described excellent as ‘‘symptoms
disappeared and physical function recovered’’, coinciding with the
definition of ‘‘curing’’. Four trials described ‘‘curing’’ as ‘‘symptoms
and physical signs significantly improved’’. Eight trials reported pain
score reduction. Among them, five trials used visual analogue scales,
three used a self-categorized numerical rating scale, and one used
the WOMAC index. Conclusions: The outcome measures of topical
Chinese herbal medicines for musculoskeletal injuries greatly varied
among trials, which may greatly influence the data extraction and
synthesis, even mistakenly combining data for specific outcomes.
Criteria to define the outcome measures and validated pain scores for
Chinese herbal medicines are urgently required.
P2A144
Still too little qualitative research to shed light on results
from reviews of effectiveness trials: A case study of a
Cochrane review on the use of lay health workers
Glenton C1 , Lewin S2 , Scheel I3
1
Nordic Cochrane Centre, Norwegian branch, Norway; 2 Cochrane Effective
Practice and Organisation of Care Review Group, Norway; 3 Norwegian
Knowledge Centre for the Health Services, Norway
Background: Qualitative research alongside trials of complex
interventions can be used to explore processes, contextual factors
or intervention characteristics that may have influenced trial outcomes.
This research may also shed light on the results of systematic reviews
of effectiveness by identifying factors that help explain heterogeneous
results across trials. In a Cochrane review on the effects of lay health
worker programmes, we identified 82 trials. These trials showed
promising benefits but results were heterogeneous. Objectives:
To use qualitative studies conducted alongside the included trials to
explore factors and processes that might have influenced intervention
outcomes. Methods: Qualitative research carried out alongside the
trials was identified by contacting trial authors, checking references,
searching PubMed for related studies, and through citation searches.
For included qualitative studies, we extracted information regarding
objectives, methods, and results. Results: For 52 (63%) of the trials,
no qualitative research had been conducted. For 16 (20%) trials,
qualitative data collection had been done but was unavailable or had
been done before the trial. For 14 (17%) trials, qualitative research
had been done during or shortly after the trial. Most of this qualitative
research aimed to elicit trial participants’ perspectives and experiences
of the intervention. A common theme was participants’ appreciation
of the lay health workers’ shared circumstances, for instance, with
regard to social background or experience of the health condition.
In six studies, researchers explored the experiences of the lay health
workers themselves. Issues included the importance of supervision and
health professionals’ support or lack of support. Overall, descriptions
of methods and results were often sparse. Conclusions: Qualitative
studies carried out alongside trials of complex interventions could help
us understand heterogeneity in systematic reviews of effectiveness.
However, for interventions of lay health worker programmes at
least, too few such studies exist for these opportunities to be
realised.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
101
P2A145
Guidelines for conducting and reporting reviews of
reviews: Dealing with topic relevance and double-counting
O’Mara AJ1 , Jamal F2 , Parry W3 , Lorenc T4 , Cooper C5
1 EPPI-Centre, Institute of Education, UK; 2 Institute for Health and Human
Development, University of East London, UK; 3 Centre for Longitudinal
Studies, Institute of Education, UK; 4 Department of Social and
Environmental Health Research, London School of Hygiene and Tropical
Medicine, UK; 5 Peninsula Technology Assessment Group (PenTAG),
Peninsula College of Medicine & Dentistry, University of Exeter, UK
Background: Reviews of review-level evidence (tertiary reviews) are
desirable when a research question is time-sensitive and/or the scope
is broad. However, reviews included in a tertiary review often have
only partial overlap with the tertiary review’s research question and,
consequently, not all included studies are relevant to the tertiary
review. Additionally, the reviews can include some of the same primary
studies (known as double-counting). These concerns might lead to
biases in the evidence base. Objectives: To explore the issues of
relevance and double-counting in a tertiary review and to present
guidelines for identifying and addressing potential related problems.
Methods: We examined data from a completed systematic tertiary
review on a public health effectiveness topic. We established the
relevance of the included reviews by determining how many of the
primary studies included in each review met our inclusion criteria
and by analysing the included reviews’ synthesis sections using the
PICO elements (population, intervention, comparison, and outcome)
in relation to the tertiary review research question. We demonstrate
graphical (plot-based) approaches and a matrix-based approach to
establishing the extent of double-counting. Results: Of 20 reviews
that met our inclusion criteria, 10 reviews had less than 50% of the
primary studies included that were relevant to our research question.
Exploring the synthesis sections of included reviews using a PICO
framework was useful in establishing the degree of relevance of the
findings to the tertiary review research question and yielded a review
‘utility’ rating. Our graphical and matrix-based approaches allowed us
to evaluate the extent of double-counting across reviews; 14 primary
studies were included in more than one review, with some studies
appearing in four reviews. Conclusions: Issues of relevance and
double-counting need to be assessed in tertiary reviews, but are often
overlooked. The guidelines proposed can help identify potential biases
and attempt to address them.
P2A146
Using EpiData software to extract data for Cochrane
systematic reviews
Liu Z1 , Li X1 , Visconte D2 , Lewith G2 , Liu J1
1 Center for Evidence-Based Chinese Medicine, Beijing University of Chinese
Medicine, China; 2 Complementary and Integrated Medicine Research Unit,
Primary Medical Care, Southampton, UK
Figure 1 (P2A146).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
102
Background: Data collection is very important for systematic reviews.
Data collection forms should be designed carefully to target the
objectives of the review. Instruments for data collection and procedures
used to collect data affect the accessibility and accuracy of extracted
data. Objectives: We explored the feasibility of using EpiData software
for data collection in a Cochrane systematic review. Methods: We
used EpiDatatocollect data forthe review ‘Chinese herbal medicines
for hypercholesterolemia’. We designed the data extraction form in
MS Word based on the suggested model inthe Cochrane Handbook
to include factors that should be considered in research methodology
and study outcomes. We converted the Word document into EpiData
questionnaires and reviewed and checked the information before
creating a data file. The EpiData file was used for data extraction to
ensure that all required data were obtained from the included studies.
We also designed a Word document to extract text data longer than
80 characters. Results: Examples of items we extracted are illustrated
in Figure 1. The extracted EpiData database was converted to a
SPSS dataset and used for statistical analysis. It is noteworthy that the
number of text characters should not exceed 80 inthe EpiData database;
which is why we used aWord document to extract text data such as
aims of included studies for tabular presentation in elements such
as ‘Characteristics of included studies’. Conclusions: The EpiData
application is suitable for review data extraction for quantitative data
or text data less than 80 characters. It has many advantages over
other tools used for data extraction such as Word or Excel formats.
An intuitive user interface madeaccurate data extraction easy. Users
can double enter data using the ‘match records by fields’ option to
avoid data entry errors. We found that data extracted usingthe EpiData
application could be easily converted to another dataset for statistical
analysis.
P2A147
Indirect meta-analysis of uncontrolled Studies about two
independent regimes: A case report
Sun T1 , Wang J1 , Fang J1
1
Shanghai Institute of Digestive Disease, Shanghai Jiao-Tong University
School of Medicine Ren-ji Hospital, Shanghai 200001, China
Background: Conducting randomized controlled studies inpopulations with rare diseases is challengingbecause of difficulties in
participant recruitment. There are far fewer challenges in conducting
uncontrolled studies in these populations, even if participant numbers
are relatively small. Objectives: To investigate if indirect meta-analysis
of uncontrolled studies in two independent regimeswas possible and
whether the results werecredible. Methods: Systematic searches
were performed in PubMed, The Cochrane Library, EMBASE and ISI
Web of Knowledge to identify randomized controlled trials (RCTs) in
gemcitabine-cisplatin compared with gemcitabine alone for advanced
biliary tract cancer (BTC), or phase II trials reporting on the efficacy
detected in gemcitabine alone or in combination with cisplatin for
advanced BTC. The pooled results for each outcome of each regime were
calculated, and compared using Chi2 analysis. Direct comparisons were
also conducted to compare gemcitabine-cisplatin with gemcitabine.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Results: The pooled overall response rate (ORR), T-cell receptor
(TCR), mean progression-free survival (PFS) and overall survival (OS)
of 10 phase II studies (330 patients) in gemcitabine-cisplatin for
BTC was 25.45% (95% CI 20.75% to 30.15%), 52.12% (95%
CI 46.73% to 57.51%); 9.02 months, 13.65 months, respectively.
The pooled ORR, TCR, mean PFS and OS of eight phase II studies
(208 patients) in gemcitabine for BTC was 20.19% (95% CI 14.74%
to 25.65%), 55.77% (95% CI 49.02% to 62.52%); 5.94 months,
4.98 months, respectively. Results of indirect comparison from 18
uncontrolled studies (538 patients) showed that there were no
significant differences in efficacy (ORR, TCR, PFS and OS) between
the groups. Results of direct comparison from three controlled studies
(579 patients) showed gemcitabine-cisplatin combination prolonged
the median time to progression, substantially increased the OS and
produced a statistically significant increase in TCR compared with
gemcitabine alone. Conclusions: From this case report, discrepancies
between indirect and direct meta-analysis were found. Although
indirect meta-analysis might be possible, we must be cautious about
the results produced.
P2A148
A rapid systematic review versus a Cochrane systematic
review: An empirical comparison
Rizzo M1 , Llewellyn A1 , Martin A1
1 Matrix Evidence, UK
Background: There is increased pressure to conduct systematic
reviews as comprehensively as possible within the constraints of a
timescale that meets the needs of policy makers. Objectives: We
aimed to evaluate how the findings and conclusions of a full Cochrane
systematic review differ when certain aspects of the methodology are
limited in order to conduct a more rapid systematic synthesis of the
evidence. Methods: Blinded to the results of a Cochrane systematic
review on the effectiveness of directly observed therapy for treating
tuberculosis, we limited three key features of the Cochrane review
protocol. 1) The literature search: using the same key words as
those used in the full Cochrane review, we reduced the number of
databases searched; 2) study selection: using the same inclusion
criteria as the Cochrane review, we used only one researcher to select
studies; and 3) data collection and synthesis: using the same data
collection and synthesis procedures to meta-analyse the data, we used
only one researcher to extract the data. Any queries regarding study
selection and data collection were resolved with a second researcher.
Results: The results compare how limiting certain aspects of the
methodology of a full systematic review impact on: 1) the studies
identified from the search; 2) the studies included into the review after
screening; and 3) the results of the meta-analysis after data collection.
Conclusions: The results of this more rapid systematic synthesis of
data are compared with those of the Cochrane review to understand
how limiting methodological aspects impact on the conclusions which
can be drawn for policy makers. Based on these findings, the benefits
and limitations of each approach are discussed, as are the limitations
of the study.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
103
P2A149
Development and Evaluation of a Study Designs Typology
for Medical Literature of Intervention
Seo HJ1 , Kim SY2 , Park JE3 , Lee YJ3 , Shin SS4 , Hahn SK5
1 Department of Nursing, College of Medicine, Chosun University, Republic
of Korea; 2 Department of Family Medicine, Hallym University College
of Medicine, Republic of Korea; 3 Department of Health Technology
Assessment Research, National Evidence-Based Healthcare Collaborating
Agency, Republic of Korea; 4 Department of Pulmonology, Ajou University
Hospital, Republic of Korea; 5 College of Medicine, Seoul National
University, Republic of Korea
Background: It is difficult to adequately classify the study designs
of included studies in the process of performing systematic review.
Objectives: To develop study designs typology in systematic reviews
of intervention and test the inter-rater reliability, feasibility and
comparison with an existing typology tool (Figure 1). Methods: To
develop the typology, our study was conducted as follows: 1) establish
a principle of development, 2) investigation of the existing study
design typology, 3) develop new typology making up for the weak
points in existing typology, 4) develop instructions for application of
the tool to intervention studies, and 5) consult with experts about
systematic reviews and clinical practice guidelines. To test the validity
of the new typology, we used a sample of 70 studies. To assess the
reliability of agreement among the three raters for all study design
classification ratings and the agreement between the new typology and
the National Institute for Health and Clinical Excellence (NICE) typology,
we calculated the weighted kappa. We measured time and feasibility to
apply the new typology tool compared with NICE. Results: Inter-rater
reliability was almost perfect among the testers (κ = 0.82) and the
mean time to complete the new typology was significantly shorter
than for the NICE typology (113.4 seconds (95% CI 99.4 to 127.4)
vs. 132.6 seconds (95% CI 116.1 to 149.1), P = 0.046 for paired
t-test). To evaluate the clarity and simplicity using 7 point Likert scale,
completeness, and user-friendliness, the new typology was rated with
mean scores 6, 6.6, and 6.3, respectively. Conclusions: We suggest
the use of the new typology to classify study designs of systematic
reviews and guideline development for interventions. Application of the
new tool to classify study designs should be accompanied by adequate
training and decision-making based on explicit instruction.
P2A150
Validation and application of a systematic review and
narrative synthesis within mental health services research
Bird V1 , Leamy M1 , Williams J1 , Le Boutillier C1 , Slade M1
1
Institute of Psychiatry, King’s College London, UK
Background: The importance of combining qualitative and
quantitative data when designing complex interventions for patient
benefit is growing in recognition. Methods for systematically
synthesising qualitative and quantitative evidence have been developed
Figure 1 (P2A149).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
104
and are increasingly utilised in health services research. Objectives:
To demonstrate both the validity of a systematic review and narrative
synthesis and the application of the results in designing and testing
a complex intervention for mental health service users. Methods:
We combined qualitative and quantitative data by conducting a
systematic review and narrative synthesis using a multistage literature
search aimed at identifying mainstream and grey literature relating
to patients’ experience of recovery. Three stages were included
in the narrative synthesis with tabulation, cross-tabulation, in-depth
thematic analysis and vote counting used to develop the preliminary
synthesis. Relationships between studies of 1) differing quality
and 2) different populations were explored using conceptual and
methodological triangulation. We tested the validity of this approach
through critical reflection, expert consultation with different stakeholder
groups (patients, carers, expert researchers, health care professionals
and individuals from black and minority ethnic backgrounds) and
patient focus groups. We applied the results of the synthesis to develop
a complex intervention. Results: 97 papers including 87 models of
recovery were included in the synthesis. The validation phase suggested
that the synthesis was a robust model which captured the experiences
of patients. The synthesis has been used to inform development of
a standardised measure (INSPIRE) and an intervention and training
package for frontline mental health staff (REFOCUS). Conclusions:
Narrative synthesis offers a robust and valid way of synthesising
evidence, particularly where qualitative evidence predominates and for
topics where patient experience is key. The narrative synthesis has been
successfully applied to the design and development of tools currently
being evaluated in a multisite cluster randomised controlled trial.
P2A151
Systematic review and synthesis of quantitative and
qualitative research: A worked example to assess
community preferences
Tong A1 , Howard K1 , Jan S2 , Cass A2 , Rose J1 , Chadban S1 , Allen R1 ,
Craig J1
1 University of Sydney, Australia; 2 The George Institute for Global Health,
Australia
Background: The synthesis of qualitative research and mixed method
synthesis are important and emerging methodologies in health research.
It is increasingly recognised as a useful approach for understanding
consumer and community preferences for health services and policy.
Objectives: To describe our methodology for appraising and analysing
the findings of qualitative and quantitative studies on community
preferences for the allocation of solid organs for transplantation,
and to describe the utility of this approach. Methods: For the
quantitative survey studies, we appraised study reporting according to
a comprehensive list of criteria we developed based on items reported
in the included studies and additional items in published guides
for designing survey studies. We synthesised the survey results by
extracting items assessed by the survey and the respective participant
responses. For the qualitative studies, we used the consolidated criteria
for reporting qualitative studies (COREQ) framework which included
criteria specific to the research team, study methods, context of the
study, analysis and interpretations. We performed a textual synthesis
of the results and conclusions reported by the primary author of each
study. Results: Fifteen studies were included (n = 5563 respondents).
We identified 15 attributes relating to community preferences for
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
organ allocation and identified consistencies across survey studies
in community preferences for prioritising for ‘younger age’, ‘time on
waiting list’, and ‘better prognosis’ after transplant. Seven main themes
underlying their preferences were identified from the textual synthesis:
maximum benefit, social valuation, moral deservingness, prejudice, fair
innings, first come, first served, and medical urgency. Conclusions:
Using this approach, we found that community preferences for organ
allocation hinge on a complex balance of efficiency, social valuation,
morality, fairness, and equity principles. Pairing qualitative and
quantitative studies within a review can generate more complete data,
corroborate findings across studies, and enhance or explain insights
attained with the complementary method.
P2A152
Causal mapping for systematic reviews of complex social
interventions
Lorenc T1 , Clayton S2 , Neary D2 , Petticrew M1 , Whitehead M2
1
London School of Hygiene & Tropical Medicine, UK; 2 University of
Liverpool, UK
Background: There is increasing interest in the use of theory and
logic models to inform systematic reviews of complex interventions.
However, there is limited methodological guidance as to how theory
can be integrated into systematic reviews. Objectives: To describe the
use of causal mapping as a theory-based framework for a systematic
review relating to crime, fear of crime, health and the built environment.
Methods: Our causal map was intended to provide a framework for an
evidence synthesis project including systematic reviews of intervention
effectiveness and of qualitative evidence. We utilised a non-systematic
iterative method to locate relevant theoretical and empirical literature,
and synthesized findings and theories narratively using the emergent
conceptual framework, in consultation with the review Advisory Group.
The map was used to inform the development of methods for the
systematic reviews. Results: The causal map included a wide range
of factors and drew on theory and data in several different areas. It
provided an outline structure of both the conceptual relationships of
key review terms and their potential causal interactions. It also helped
to identify potential pathways of intervention effectiveness; to clarify
methodological issues in the primary evidence; and to situate the
systematic review findings within a broader context. Purely theoretical
literature was of less value than empirical observational and qualitative
evidence, although the mapping methodology may not be well suited
to the integration of insights from qualitative research. The validity
of the mapping process is limited by the non-systematic methods;
however, there are several reasons why full systematic review methods
may not be appropriate for causal mapping. Conclusions: Causal
mapping may be a valuable tool in systematic reviews of complex
interventions. However, there remain a number of questions about
appropriate methodologies for collecting and synthesizing theoretical
and empirical literature for causal maps.
P2A153
A worked example of ‘bestfit’ framework synthesis: A
potentially more consistent or reliable form of qualitative
data synthesis?
Booth A1 , Cooper K1
1 University of Sheffield, UK
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
105
Background: A variety of different approaches to the synthesis of
qualitative data are advocated in the literature, but none is recognised
as the preferred method. Objectives: To describe the application
of a pragmatic ‘best fit’ framework synthesis approach to qualitative
evidence synthesis. Methods: An evaluation of a novel version of
framework synthesis as an approach to qualitative systematic review.
The case study was a systematic review exploring adults’ views about
taking various potential agents for the primary prevention of colorectal
cancer. An existing model was identified from the literature that
conceptualised attitudes of adult women to the taking of several of
these agents. The model identified did not entirely match the topic
under study, but was a ‘best-fit’ and provided a relevant pre-existing
framework and themes against which to map and code data. Results
and Discussion: Twenty papers from North America, Australia, the
UK and Europe met the criteria for inclusion. Fourteen themes were
identified a priori from the previously identified model, which were then
used to code the extracted data. The production of clear definitions
for these themes reduced the scope for subjectivity of interpretation
and thus enabled rapid and consistent coding of the studies’ data
by more than one reviewer. The final synthesis also required some
secondary thematic analysis to develop themes for data not covered
by the a priori framework. The result was a more sophisticated
model with additional themes. Conclusion: This novel and pragmatic
‘best fit’ approach to framework synthesis was found to be fit for
purpose. It offered a means to reinforce, critique, and develop
an existing published model, conceived for a different but relevant
population, and produced a relatively consistent or reliable process
when compared to more interpretative forms of synthesis. Future
research should seek to test further this approach to qualitative data
synthesis.
P2A154
Mixed-methods synthesis for community-level public
health interventions: Some considerations from a review
of skin cancer prevention
Lorenc T1 , Jamal F2 , O’Mara A3 , Cooper C4
1 London School of Hygiene & Tropical Medicine, UK; 2 University of
East London, UK; 3 EPPI-Centre, Institute of Education, UK; 4 Peninsula
Technology Assessment Group (PenTAG), Peninsula College of Medicine &
Dentistry, University of Exeter, UK
Background: The potential value of using systematic reviews of
qualitative research and mixed-methods synthesis in conjunction with
reviews of the effectiveness and cost-effectiveness of interventions is
generally recognised. However, few studies have directly compared
different synthesis methodologies. Objectives: To explore methods
for mixed-methods synthesis in the context of community-level
interventions, and identify potential challenges and directions for
future research. Methods: A systematic review of the effectiveness of
community-level skin cancer prevention interventions was conducted by
CRD1. Our systematic review of qualitative evidence, funded by NICE,
was conducted in parallel and aimed to identify barriers and facilitators
to such interventions. We used thematic synthesis to identify themes in
the qualitative evidence. After the completion of the NICE review, we
carried out further analysis comparing two methods for cross-review
synthesis: constructing a matrix of themes from the qualitative research
and identifying themes addressed (or not addressed) in the intervention
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
research; and constructing causal pathways for selected interventions
and using these to identify relevant qualitative themes. Results: Both
models displayed the potential for qualitative synthesis to enhance
the evidence base. However, analysis using a cross-study matrix
for community-level interventions encountered some limitations, due
to the lack of explicit cognitive content in the interventions, and
to the limited extent of the effectiveness data. Causal pathway
analysis provided a richer account of the relationships between
the two forms of data, but does not enable rigorous testing of
hypotheses, and requires potentially questionable interpretation of
the effectiveness findings. Conclusions: Different approaches to
mixed-methods synthesis have different strengths; there may be a
trade-off between rigour and informativeness. The findings suggest
broader questions about the role and purpose of mixed-methods
synthesis.
Reference
1. McDaid C et al. (2010). Sun protection resources and environmental
changes to prevent skin cancer: A systematic review. York: Centre for
Reviews and Dissemination.www.nice.org.uk/nicemedia/live/11871/
49660/49660.pdf
P2A155
Overviews of systematic reviews often do not assess
methodological quality of included reviews
Büchter R1 , Pieper D2 , Jerinic P3
1 Department of Health Information, Institute for Quality and Efficiency
in Health Care (IQWiG), Germany; 2 Section EbM/HTA, Institute for
Research in Operative Medicine (IFOM), Faculty of Health, Department of
Medicine, Witten/Herdecke University, Germany; 3 University of Cologne,
Germany
Background: There is now often more than one systematic review
on the same or a similar question, which has led to the development
of systematic reviews of systematic reviews (overviews). Overviews
offer the opportunity of providing decision makers with a broader
summary of evidence and contrasting the results of different reviews
on the same question. Little is known about their characteristics so
far. Objectives: We set out to examine a sample of overviews
in terms of different descriptive and methodological characteristics,
including which databases were searched and how the quality of the
included systematic reviews was assessed. Methods: We searched
MEDLINE, DARE, the CDSR and different HTA databases for overviews
of systematic reviews. Two reviewers independently screened titles
and abstracts of the retrieved citations for eligible publications. Data
extraction was conducted by one reviewer and checked by a second.
Results: A total of 99 overviews were eligible. The median publication
date was 2008 (interquartile range [IQR]: 2005–2010). The overviews
included a median of 15 reviews (IQR: 7 to 35). Typically, four electronic
databases were searched for systematic reviews (IQR: 2 to 6). The
quality of the included reviews was systematically appraised in a good
half of the overviews (56/99 [57%]). To assess methodological quality,
most overviews used the Oxman&Guyatt Index (24/99 [24%]), followed
by AMSTAR (8/99 [8%]) and PRISMA/QUORUM (5/99 [5%]). Primary
studies published after the search date of the most recent included
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
106
review were included in 5 overviews [5%]. Conclusions: Overviews
should use rigorous methods, if they ought to be a useful tool in health
care decision making. The use of a quality assessment tool can help
to determine if eligible reviews meet minimum quality requirements.
In our sample only a small proportion of overviews used a validated
quality assessment tool.
P2A156
The data was unobtainable. A survey of the reporting
of missing data in a sample of Cochrane reviews
Struthers C1 , Noel-Storr A1
1
Cochrane Dementia and Cognitive Improvement Group, UK
Background: It is widely acknowledged that publication bias is
one of the most difficult to avoid in the preparation of systematic
reviews. There are many factors that lead to data from clinical trials
being unavailable for a meta-analysis, and this single source of bias
significantly undermines the power of a Cochrane review to reflect the
true picture of an intervention’s effectiveness. Objectives: We decided
to investigate the occurrence and extent of unobtainable results data
in studies included in Cochrane reviews. We hope that our findings
will lead to a more robust, proactive and standardized reporting of
unavailable data that will reduce its occurrence, and thereby improve
the quality of Cochrane reviews. Methods: We developed a search
strategy using terms designed to find all Cochrane reviews where data
from trials eligible for inclusion were incomplete or unobtainable. We
selected a random sample of 100 full reviews, and updated reviews.
We also randomly selected Cochrane reviews with studies awaiting
classification, or ‘ongoing’ to see if that masked underlying publication
bias. Results: We analysed the reviews in both sample sets using a
number of measures, including the proportion of eligible trials where
the data were missing for the reasons outlined above. We also
looked at the efforts made to obtain data, and how their absence was
reported in the reviews. Discussion: As part of the development of
a Cochrane-wide citizen science initiative, we are considering how to
include the tracking of results data in a portfolio of volunteer tasks.
We are looking forward to a future where public involvement can bring
pressure to bear on research authorities to make sure that all clinical
trials are registered, protocols adhered to, and results data are made
freely available for analysis by review authors and others in a timely
manner.
P2A157
Mixing with the ‘unclean’: Including non-Cochrane reviews
alongside Cochrane reviews in overviews of reviews
Foisy M1 , Becker LA2 , Chalmers JR3 , Boyle RJ4 , Simpson EL5 , Williams HC6
1
Cochrane Child Health Field, Canada; 2 Cochrane Comparing Multiple
Interventions Methods Group, USA; 3 University of Nottingham, UK;
4 Imperial College London, UK; 5 Oregon Health and Science University,
USA; 6 Nottingham University Hospitals NHS Trust, UK
Background: Overviews of reviews are ‘systematic reviews of
systematic reviews’ that examine all possible interventions for a certain
condition. To date, no overviews published in The Cochrane Library
or by the Child Health Field have included non-Cochrane reviews. Our
most recent overview on eczema prevention is the first overview to
include both Cochrane and non-Cochrane reviews. Objectives: To
discuss the methodological challenges we faced, how we overcame
these challenges, and what we learned from the process of including
non-Cochrane reviews in overviews. Challenges: 1) Two of our three
potentially included non-Cochrane reviews did not contain data that
enabled calculation of risk ratios and were therefore excluded; 2) many
non-Cochrane reviews analyzed data from Cochrane reviews we had
already included in our overview; and 3) we debated using scores
from the AMSTAR tool (Assessment of Multiple Systematic Reviews)
as an inclusion criterion. Our main challenge, however, centred on
the fact that Cochrane and non-Cochrane reviews often examined the
same interventions and contained many of the same trials. Deciding
which reviews to include in our overview involved a trade-off between
inclusivity and ensuring the project did not become too unwieldy or
large for the end user. Solution: We used a consensus-based process
to concurrently develop two sets of inclusion criteria: one assessed
the eligibility of each review individually, and the other ensured that
only one eligible review per topic area was included in our overview
(Table 1). Conclusions: Six Cochrane reviews and one non-Cochrane
review were included in our overview. Data extraction proceeded
largely without difficulty, mostly due to our development of logical
a priori inclusion criteria which other groups undertaking overviews
might find helpful. Including non-Cochrane reviews allowed us to
consider a broader range of interventions, although this was limited by
our inability to extract usable data from two potential non-Cochrane
reviews.
Table 1 (P2A157). Criteria for considering reviews for inclusion.
General inclusion criteria for individual reviews:
Participants must have been 0-18 years of age.
Search for trials must have been conducted no
more than five years ago.
Trials in reviews must have been randomized
controlled trials (RCTs). Trials examining
breastfeeding and pet avoidance were not
subject to this restriction because RCTs in these
topic areas are ethically difficult to conduct.
Reviews must have contained at least one piece
of quantitative data to enable calculation of a
risk ratio and confidence interval.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Inclusion criteria to select one eligible review per topic area:
All Cochrane reviews meeting our general inclusion
criteria were included.
A maximum of one non-Cochrane review was included in
each topic area that did not have an included Cochrane
review.
If two or more non-Cochrane reviews covering the same
topic area satisfied the general inclusion criteria, we
included the most up-to-date review.
Non-Cochrane reviews were excluded if they analyzed
data from one or more Cochrane reviews that we had
already included in our overview.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
107
P2A158
Risk difference estimation and reporting in systematic
reviews
Alonso-Coello P1 , Neumann I2 , Carrasco-Labra A3 , Brignardello-Petersen
R4 , Irfan A5 , Solà I1 , Dahm P6 , Glujovsky D7 , Johnston B8 , Martinez L1 ,
Ramirez-Morera A9 , Sun X10 , Vandvik PO11 , Akl E12 , Tikkinen KA13 , Iorio
A14 , Santesso N8 , Brozek J8 , Schünemann H14 , Guyatt G14
1 Iberoamerican Cochrane Centre, Institute of Biomedical Research (IIB Sant
Pau) Barcelona, Spain; 2 Department of Medicine, Pontificia Universidad
Catolica de Chile, Santiago, Chile; Department of Clinical Epidemiology
& Biostatistics, McMaster University, Hamilton, Canada; 3 Evidence Based
Dentistry Unit, Faculty of Dentistry, Universidad de Chile, Chile; Department
of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton,
Canada; 4 Evidence Based Dentistry Unit, Faculty of Dentistry, Universidad
de Chile, Chile; Department of Health Policy, Management & Evaluation,
University of Toronto, Canada; 5 Internal Medicine Residency Program,
University of Illinois at Urbana-Champaign, USA; 6 Department of Urology,
College of Medicine, University of Florida, Gainesville, Florida, USA;
7
Institute for Clinical Effectiveness and Health Policy, Buenos Aires,
Argentina; 8 Department of Clinical Epidemiology & Biostatistics, McMaster
University, Hamilton, Canada; 9 IHCAI Foundation, Costa Rica; 10 Center
for Health Research, Kaiser Permanente Northwest, Portland, USA;
11 Norwegian Knowledge Centre for the Health Services, Oslo, Norway;
12 Department of Medicine, State Univeristy of New York at Buffalo,
NY, USA; Department of Clinical Epidemiology & Biostatistics, McMaster
University, Hamilton, Canada; 13 Department of Urology, Helsinki University
Central Hospital, Helsinki, Finland; Department of Clinical Epidemiology
& Biostatistics, McMaster University, Hamilton, Canada; 14 Department
of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton,
Canada
Background: Trading off desirable and undesirable consequences
of alternative patient management strategies requires knowledge of
estimates of both relative and absolute measures of effect (AME). It is
unclear to what extent and how authors of systematic reviews (SRs)
determine and report AME. Objectives: To determine the proportion
of SRs reporting AME and to evaluate how these are derived and
interpreted. In particular, we will evaluate the proportion of SR that
use relative measures for benefit outcomes and for harm. Additionally,
we will test if AM reporting is associated with prespecified study
characteristics. Methods: We will include SRs of interventions that
report dichotomous outcomes. We will search MEDLINE for a random
stratified sample of Cochrane and non-Cochrane SRs. From a pilot study
of 50 SRs, we will calculate the final sample size to test our hypothesis,
which is that a considerable proportion of SRs do not report AME
and that the methods used are often inadequate. We will determine
the proportion of SRs reporting AME and how they are obtained
(e.g. meta-analytically combined across studies or computed based on
relative effect measures and assumed baseline risk) and reported (e.g.
risk difference or number-needed-to-treat). We will also assess how
baseline risk was obtained (e.g. as mean or median from included trials
or from representative observational studies), and whether authors
provided AME for more than one risk group. Finally, we will conduct
multivariable logistic regression analyses to examine the relationship
of reporting AME with a set of prespecified study characteristics (e.g.
type of SR [Cochrane vs not Cochrane], statistical significance or source
of funding). Discussion: The project results will be presented at
the Colloquium. Our results will inform the extent of limitations of
determining and reporting AME in SRs and lay the foundation for
further studies about the implications for decision-making.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P2A159
Using qualitative evidence to inform a review
of effectiveness: A case study of integrating qualitative
data in a systematic review of effectiveness of ECT for
adult depression
Berg R1 , Leiknes KA1 , Hoie B1
1 Norwegian Knowledge Centre for the Health Services, Norway
Background: The issue of combining data from different traditions is
of increasing importance as qualitative research is now being integrated
within Cochrane reviews. Objectives: To describe the process of and
results derived from conducting a systematic review in which we, as
one of the first Cochrane registered reviews, integrated qualitative
data. Methods: The Cochrane guidance on including qualitative
evidence in effectiveness reviews was used to design an integrative
methodological approach, which aimed to inform and extend the
review, as defined in Chapter 20 of The Cochrane Handbook. We
present the first phases of the review, in which the integrative approach
was used with the aim of informing the effectiveness review question
and Patient Intervention Comparison Outcome(PICO) elements. We
conducted a systematic literature search and used content analysis to
identify key concepts in included studies. A data recording form was
used to identify text units pertaining to review elements and determine
the frequency of categories. Dominant categories were used to form
suggestions for refinement of the protocol. Results: The literature
search for qualitative studies to inform the review yielded 66 studies of
which 54 were examined in fulltext. Five studies met our pre-defined
inclusion criteria. Based on these studies several dominant categories
emerged within the PICO element areas population and outcome.
Thus, we refined the protocol with respect to these elements by adding
subgroup analyses and outcomes not already included. Challenges
encountered during the process were skills required for synthesis of
qualitative evidence, and practical integration of modifications within
the Archie system. The merits of our approach included upholding
expected standards of rigour and transparency, applying systematic
methods, and heeding the importance of patient perspectives in health
care. Conclusions: This review illustrates how qualitative evidence
can add value to a review of effect and shows how incorporating
qualitative material need not compromise the trusted methods of the
Cochrane Collaboration.
P2A160
Methodological perspectives on a realist synthesis: An
explanatory analysis of the effectiveness of female genital
mutilation-cutting abandonment efforts
Berg R1 , Denison E1
1
Norwegian Knowledge Centre for the Health Services, Norway
Background: Realist synthesis is an emerging approach to
systematically reviewing research evidence on complex social
interventions, which provides an explanatory analysis of how and
why they are (or not) effective in particular contexts or settings. While
there is considerable interest in conducting realist syntheses there are
few exemplars of how to do this. Objectives: We present results
of and methodological perspectives on a realist synthesis of FGM/C
abandonment efforts. Methods: The Cochrane Handbook guidance
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
108
on including qualitative evidence in reviews of effects was used to inform
the development of a realist synthesis approach and conduct of our
review. Exhaustive searches sought two types of studies: 1) controlled
studies to examine the effectiveness of FGM/C abandonment programs;
2) studies using qualitative and other methods to describe the factors
related to the continuance and discontinuance of FGM/C (‘views’
studies). We combined quantitative data with data from views studies
and the results of the synthesis were described in a model explaining
how efficacy (outcome patterns) of an intervention varied depending on
the particular configuration of its constituent mechanisms and contexts.
Results: The searches yielded 7517 records. For the effectiveness
synthesis, we included eight controlled before-and-after studies of low
methodological quality (7042 participants). There were four types of
interventions, covering seven intervention areas in Africa. Few studied
the field’s reality before rolling out the intervention. Four meta-analyses
showed limited effectiveness and considerable heterogeneity affected
the interpretation. For the views synthesis, we located 26 studies,
covering all but one intervention area. The synthesis of these studies
was used to illuminate the effectiveness findings. In general, sexual
morals and religion were poorly addressed and the degree to which the
interventions were appropriate responses to the populations’ situation
regarding FGM/C varied. Conclusions: Realist synthesis is a valuable
but time-consuming approach requiring advanced reviewer skills in
several assessment and analysis areas.
P2A161
New and emerging methods of research synthesis: Are
they given sufficient attention?
Petrova M1
1 Egenis (ESRC Centre for Genomics in Society), University of Exeter, Exeter,
UK
Background: In spite of meta-analysis remaining the cornerstone of
evidence synthesis, numerous alternative methods of statistical and
non-statistical synthesis have recently emerged. Their proliferation has
been driven by needs that the ‘exemplar systematic review’ meets
incompletely, such as the need to address broad research questions
and heterogeneous study types or to incorporate multiple perspectives
and levels of analysis. Healthcare quality and patient safety are prime
examples of areas where such demands arise. The field of emerging
research synthesis methods is poorly mapped but lively. Four review
publications identify over 40 such methods among themselves. These
methods have developed in relative isolation. Their terminology is
highly varied. Some of their constitutive practices are at odds with
traditional systematic review ones. Many of them simply extend single
study designs and lack the level of abstraction, articulation and detail
required for wide usage. There are thus substantial barriers to their
uptake. Yet those methods try to solve important research synthesis
concerns. Are we giving them sufficient attention? Objectives:
This presentation will begin with an overview of the field of emerging
health research synthesis methods. It will then address the higher-order
dimensions along which they tend to be compared and the lines along
which they translate desiderata for systematicity, transparency and
robustness into specific procedures. Finally, it will propose further
criteria for evaluating the capacity of a synthesis method to instantiate
such desiderata. Methods and data sources: A case study of the
contents and structure of a heterogeneous pool of research papers
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
on cancer. Ideas from the philosophy and social studies of science.
Conclusions: The current edition of the Cochrane Handbook offers
limited guidance to non-standard synthesis methods. A debate is
needed as to whether extending the Handbook in this direction is
required or whether there are positive reasons for taking aconservative
attitude.
P2A162
Feasibility of fast track systematic review for policy
makers: Interchangeability between pneumococcal
conjugate vaccines
Ciapponi A1 , Garcı́a Martı́ S1 , Rey-Ares L1 , Glujovsky D1 , Bardach A1 ,
Valanzasca P1 , Lee A1 , Cafferata ML1
1
IECS – Institute for Clinical Effectiveness and Health Policy, Argentina
Background: Streptococcus pneumoniae is a leading cause of serious
illness among children worldwide. Pneumococcal conjugate vaccines
(PCV) including 7 to 15 serotypes have been developed but there is
uncertainty about the effects of interchange schemes among them. The
Pan American Health Organization (PAHO) committed us a systematic
review (SR) for an evidence based decision that would be taken in the
next month at an expert meeting. Objectives: To assess the efficacy,
cost-effectiveness, immunogenicity and safety of interchangeability
among PCV schemes in one month. Methods: A systematic search
was conducted in December 2010 on MEDLINE, EMBASE, LILACS and
CCTR. Databases containing regional proceedings were also searched.
No language or temporal restriction was imposed. We included
randomized controlled trials (RCTs), economic evaluations, SR and
meta-analysis evaluating antibody response, cost-effectiveness and
effectiveness of the interchangeability among PCV. Study selection
was performed using EROS (Early Review Organizing Software), a
web-based software to serve in the initial phases of a SR. Four pairs of
reviewers independently selected, assessed the quality and extracted
the data of the studies. Discrepancies were solved by consensus of the
team. A librarian uploaded fulltext during the project as soon each
agreement was obtained. Results: 21 of 159 studies were included.
Direct information on interchangeability among PCV for primary series
was not available. Some studies showed similar immunogenicity
and safety between PHiD-CV and PCV7. PhiD-CV and PCV13 were
consistently more cost-effective than PCV7 at a constant price. PHiD-CV
vs. PCV13 comparitive results varied according to price, indirect effects
and indirect costs. PHiD-CV gains more quality-adjusted life years due
to the prevention of more frequent yet less severe events such as otitis
media; and PCV13 prevents less frequent events but more costly as
invasive diseases. Conclusions: Considering the absence of direct
evidence, the PAHO meeting recommended not interchanging PHiD-CV
with PCV7 unless necessary. The web-based software facilitated this
intense process.
P2A163
Use of WOMAC for the Assessment of Treatment Benefit
for the Pain of Osteoarthritis of the Knee
Woolacott N1 , Corbett M1 , Slack R1 , Rice S1
1 CRD, University of York, UK
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
109
Background: For the purposes of meta-analysis and network
meta-analysis the use of standard outcome measures is ideal. In the field
of osteoarthritis research The Western Ontario and McMaster University
Osteoarthris Index (WOMAC) was developed as an osteoarthritisspecific measure of disability. It comprises three components: pain,
stiffness, physical function, which can be reported separately or as
an overall index. In 1994 a consensus meeting recommended the
use of WOMAC as a primary measure of efficacy in osteoarthritis.
Objectives: Within the context of investigating the efficacy of physical
interventions for the relief of the pain of osteoarthritis of the knee
(OAK), we investigated to what extent WOMAC had been adopted.
Methods: We conducted a systematic review of physical therapies for
pain relief in OAK. A range of sources were systematically searched in
December 2009–January 2010. Results: A total of 138 original trials
formed the basis of the review. Pain was measured using a variety of
scales, with WOMAC pain scores making up 41% of the studies: pain
was recorded using a WOMAC Likert scale in 29 studies and using a
WOMAC visual analog scale in 28 studies. However, in many cases
unexplained modifications and transformations of the WOMAC scores
had been used such that comparability across trials, purportedly using
the same measure, could not be assumed. Only 26 studies reported an
overall WOMAC index (19%); 25 reported (some if not all) individual
WOMAC subscores in addition to pain. Conclusions: The limited
use of WOMAC, coupled with the wide range of other tools used in
studies in our review, was a source of heterogeneity between studies.
Further standardisation of the use of the WOMAC scoring system, with
improved reporting of it, in trials of OAK is required. Caution in the
pooling of the different pain scales and other similar score data is
needed.
P2A165
Defining study designs in The Cochrane Collaboration: Do
all Cochrane Review Groups talk the same language?
López-Alcalde J1 , Calcerrada N1 , Callejo D1 , Reza M1 , Gracia FJ1 , Maeso
S1 , Escalona S1 , Nieto B1 , Fernández de Larrea N1 , Valentı́n B1 , Blasco JA1
1 Laı́nEntralgo Agency, Spain
Background: Cochrane systematic reviews (SR) focus on randomized
clinical trials (RCT) but some Cochrane Review Groups (CRG) also
consider non-randomized studies (NRS) as eligible. However, NRS cover
designs defined with different labels not consistently applied among
researchers. Objectives: To describe the study designs considered as
eligible by the CRG and the definition provided for NRS. Methods:
We consulted CRG websites and their corresponding modules of The
Cochrane Library (March 2011). At least two researchers assessed the
information provided by each CRG. Any disagreement was resolved
through discussion and, if necessary, with the involvement of a third
researcher. The inter-rater reliability in the assessments of key items
was rated by using the kappa statistic. We registered the following
information for each CRG: 1) Are NRS eligible for inclusion? 2) If so, is
this justified with any reason? 3) What ‘study design label’ is used (if
any) for each study design? 4) What definition (if any) is provided for
each study design? 5) How are key ‘study design features’ described?
We considered as key study design features those detailed in Tables
13.2a and 13.2b of The Cochrane Handbook: Which was the unit of
allocation? Was there a comparison? How were participants allocated
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
to groups? Which parts of the study were prospective? On which
variables was comparability between groups assessed? Results: We
identified 53 CRGs. The preliminary results show that around 60% of
the CRGs analysed considered NRS as eligible, although this decision
was justified with explicit reasons by fewerthan 50% of the CRGs.
The CRGs analysed did not use common study design labels and did
not explicitly describe allstudy design features suggested inChapter
13 of The Cochrane Handbook. Conclusions: Consistent definitions
of key study design features for NRS are needed in The Cochrane
Collaboration.
P2A166
Why and when children are different: Guidance on
appropriate inclusion of child health data in systematic
reviews
Thomson D1 , Williams K2 , Hartling L3
1 Cochrane Child Health Field, Department of Pediatrics, University of
Alberta, Edmonton, Canada; 2 Cochrane Child Health Field; APEX Australia
Chair of Developmental Medicine, University of Melbourne; Department of
Developmental Medicine, Royal Children’s Hospital, Melbourne, Australia;
3 Cochrane Child Health Field and Alberta Research Centre for Health
Evidence, Department of Pediatrics, University of Alberta, Edmonton,
Canada
Background: Children’s responses to health care interventions often
differ significantly from those of adults. Consequently, systematic
review (SR) authors must consider the physiological, developmental
and psychosocial aspects of childhood. Guidance for SR authors in this
area remains limited. Key considerations: We identified resources
to assist decision-making in the following areas: Population. A key
decision at the outset of the review is whether to focus on child data
only, or to include both adult and child data. This decision is often
based on clinical expertise about the condition and its trajectory at
different ages; care guidelines can also be used. Outcomes: The
selection of a priori outcomes should be based on sound knowledge of
the disease trajectory and children’s development. Relevant short-term
outcomes may differ for children and adults, and similar outcomes
may be assessed using different measurement tools. Recognition that
the benefits and harms of treatment for children may unfold across
subsequent decades of life may influence the choice of outcomes to be
assessed and study designs to be included. Subgroups: Within the age
range of 0–18 years, one can expect differing reactions to interventions.
Authors should analyse age groups where there may be important
similarities or differences in terms of physiological and psychosocial
development. Searching: Databases have varying definitions of ‘child’
and other age-based terms, and may be inconsistent in indexing
pediatric material. The assistance of a librarian experienced with
using pediatric-search filters will help to identify relevant material.
Conclusions: Key stages of carrying out SRs in children are affected
by the dynamic nature of childhood. Authors are urged to carefully
consider how children’s psychosocial and physiological development
along the age span will influence their response to interventions. Work
is under way to define the key developmental stages of childhood and
how they intersect with different disease conditions.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
110
P2A167
Systematic reviews of complex interventions: From bare
bones to greater policy relevance
Snilstveit B1 , Waddington H1
1 International Initiative for Impact Evaluation (3ie), UK
Background: Systematic reviews have traditionally focused on
synthesising quantitative evidence, typically from randomised controlled
trials. For questions regarding the effectiveness of simple interventions,
such as effectiveness of a drug, ‘bare bones reviews’ are invaluable.
However, when applied to systematic reviews of complex interventions
the bare bones approach to systematic reviewing has a number of
limitations, including its usefulness for informing policy and practice.
This has been particularly highlighted by the recent rise in demand
for systematic reviews in the field of international development.
Objectives: Outline an approach to systematic reviewing that is
appropriate for reviews of complex interventions, with the aim of
enhancing the usefulness of such reviews for policy and practice.
Methods: Based on a review of the methodological literature in this
field and the experience of 3ie’s systematic reviews program, this
paper provides an outline 3ie’s methodological approach to systematic
reviews. Results: Theory-based systematic reviews, which summarise
evidence on what works, when and why, can improve the policy
relevance of systematic reviews of complex interventions. This approach
situates reviews in the broader context of the underlying programme
theory, reporting and synthesising evidence on all assumptions and
links in the causal chain. Reviews that answer these questions draw
on a range of study types. Answering the what works and what
doesn’t questions means drawing on effectiveness studies, conducted
to standards of high quality impact evaluation. But formulating
answers to the when and why questions, may require a broader
range of evidence from both quantitative and qualitative research.
Conclusions: Systematic reviews of complex interventions should
draw on a program theory, and when appropriate and feasible, use a
mixed methods approach to collect and synthesise evidence along the
causal chain.
P2A169
Efficacy and safety of atypical antipsychotic drugs
(quetiapine, risperidone, aripiprazole and paliperidone)
compared with placebo or typical antipsychotic drugs for
treating refractory schizophrenia: Overview of systematic
reviews
Melnik T1 , Soares B1 , Puga E1 , Atallah A1 , Rufino C1
1 Brazilian Cochrane Center, São Paulo, Brazil
Background: According to some cohort studies, the prevalence of
refractory schizophrenia (RS) is 20% to 40%. Objectives: Our
aim was to evaluate the effectiveness and safety of aripiprazole,
paliperidone, quetiapine and risperidone for treating RS. Methods:
This was a critical appraisal of Cochrane reviews published in
The Cochrane Library, supplemented with reference to more recent
randomized controlled trials (RCTs) on RS. The following databases
were searched: MEDLINE (1966–2009), CENTRAL: Controlled Trials of
the Cochrane Collaboration (2009, Issue 2), EMBASE (1980–2009),
LILACS (1982–2009). There was no language restriction. Randomized
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
controlled trials, systematic reviews and meta-analyses evaluating
atypical antipsychotics for treating RS were included. Results:
Seven Cochrane systematic reviews and 10 additional RCTs were
included in this review. The data generally showed minor differences
among the atypical antipsychotics evaluated and typical antipsychotics,
regarding improvement in disease symptoms, despite better adherence
to treatment with atypical antipsychotics. Risperidone was specifically
evaluated in patients with RS in one of the systematic reviews included,
with favorable outcomes, but without definitive superiority compared
with other drugs of proven efficacy, such asamisulpride, clozapine and
olanzapine. Conclusions: The findings underscore the difficulty in
treating these patients, with high dropout rates and treatment patterns
of modest improvement in assessments of effectiveness. Atypical
antipsychotics have advantages over typical antipsychotics mainly
through their better safety profile, which leads to better adherence to
treatment. A combination of antipsychotics may also be an option for
some refractory patients.
P2A170
Non-randomised studies in reviews of effective
professional practice and the organisation of health care:
Do authors consider them and do they find them?
Glenton C1 , Lewin S2 , Mayhew A2 , Scheel IB3
1 Norwegian Branch, Nordic Cochrane Centre, Norway; 2 Cochrane Effective
Practice and Organisation of Care Group, Canada; 3 Norwegian Knowledge
Centre for the Health Services, Norway
Background: The Cochrane Effective Practice and Organisation of
Care Group (EPOC) accepts inclusion of several study designs in
addition to randomised trials, including quasi-randomised controlled
trials; interrupted time series; and controlled before-after designs.
But how often are these study designs included and is it worth
the effort? Objectives: To explore: 1) How many EPOC reviews
include non-randomised studies; 2) How many and what type of
non-randomised studies they find; 3) Whether there is a connection
between the proportion of non-randomised trials identified and the
review topic. Methods: We examined the 65 EPOC reviews published
before March 2010, and recorded the types of study designs considered.
For reviews that included nonrandomised studies, we calculated
the proportion of included studies that were nonrandomised. We
categorised the reviews according to topic, using the categories
adopted by the Health Systems Evidence Database, and explored
whether there were differences across categories in the proportion of
nonrandomised studies. Results: We found that: 75% of all EPOC
reviews include nonrandomised studies in their inclusion criteria; the
proportion of non-randomised studies in each review varied widely
(median: 33%, range: 0 to 100%). The proportion of non-randomised
studies varied according to intervention type: Healthcare delivery
interventions, median 18.5% (range 0 to 100%); implementation
strategies, median 33% (range 0 to 100%); financial interventions,
median 50% (range 33 to 100%); and governance interventions,
median 100% (range 66 to 100%). Conclusions: The majority of
EPOC reviews do search for non-randomised studies. However, the
degree to which they find such studies varies greatly, and may be
influenced by intervention type. Given that non-randomised studies
are often at higher risk of bias, and that including them entails extra
effort, review authors should consider whether the benefits justify this
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
111
decision. Research should explore whether it is more useful to include
non-randomised studies in reviews of some intervention types than
others.
P2A171
Determining appropriate inclusion criteria for reviews of
public health interventions: Skin cancer prevention as a
case study
McDaid C1 , Sowden A1 , Paton F1
1 Centre for Reviews and Dissemination, UK
Background: We undertook a systematic review to evaluate the
effects of provision of sun protection resources (e.g. shade, sunscreen)
combined with information provision in the prevention of skin cancer,
commissioned by Centre for Public Health Excellence at NICE. Broad
inclusion criteria were applied; studies were included regardless of
design or quality. Due to limitations in the included studies it
was difficult to determine the contribution of various intervention
components to outcome. This raised the question of whether it is
appropriate to adopt broad inclusion criteria for answering a policy
relevant question. Objectives: To assess the impact on review
conclusions of varying the inclusion criteria where the intervention
includes multiple components. Methods: We used a completed
systematic review as a case study. We investigated the inclusion of
studies according to three sets of criteria: i) studies with appropriate
assessment of the sun protection resource and information components;
ii) studies where the main intervention components were of at least
equal importance and iii) the original criteria. Narrative synthesis was
undertaken. We assessed whether choice of criteria influenced the
number of studies included and the overall conclusions. Results:
Applying the narrowest inclusion criteria resulted in a single included
study; the second set resulted in five included studies; 27 studies
were included in the original synthesis, using the broadest criteria.
The overall review conclusions remained the same, irrespective of
criteria applied. The review would have been less resource intensive
with tighter inclusion criteria, although the commissioner would not
have had access to the same breadth of evidence, regardless of its
limitations. Conclusions: Careful consideration should be given to
the inclusion criteria selected for reviews in public health as decisions
will affect the number of studies included, the amount of data to
be extracted, quality assessed and synthesised, without necessarily
impacting on the conclusions reached.
P2A172
Piloting realist review methods to assess
cost-effectiveness within he context of a systematic review
Anderson R1 , Pearson M1 , Shepperd S2 , Pawson R3 , Hunt H1 , Cooper C1 ,
Shemilt I4
1 Peninsula Medical School, University of Exeter, UK; 2 University of Oxford,
UK; 3 University of Leeds, UK; 4 University of Cambridge, UK
Background: Realist review has been used to explain how and
why complex health programmes are effective or not. However, this
approach has not been extended to explaining the resource use, cost
and cost-effectiveness of programmes or changes to services within
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
the context of evidence synthesis. Some intervention mechanisms,
their contexts and outcomes may be inherently resource-related, while
others may have a behavioural focus and be harder to define in terms
of resources—and therefore be harder to cost. The development
and testing of more explanatory systematic review methods has been
strongly advocated by the Campbell & Cochrane Economic Methods
Group. We have been funded by the NIHR Service Delivery and
Organisation programme (UK) to conduct a systematic review that
summarises the evidence of the effectiveness and cost-effectiveness of
‘step-down’, ‘hospital at home’ and other forms of intermediate care
that aim to provide an alternative to acute inpatient care. Objectives:
To present the methods (protocol) and early findings of this realist
review, with a particular focus on the adaptation of methods for
assessing and explaining the cost-effectiveness of intermediate care.
Methods: We will present the protocol for this review, describe any
amendments to the planned methods, and reflect on the emerging
challenges and insights from half-way through the 10-month project.
Our intention is, as a first stage, to identify key intervention mechanisms
and their related contexts which explain effectiveness, and then assess
their resource requirements and opportunity costs as the second and
third stages of analysis. Results: The review protocol is in development
and will be finalised by June 2011, preliminary findings will be available
by October 2011. Conclusions: On the basis of this review we will
comment on how realist review methods may complement established
Cochrane methods for systematic reviews of effectiveness.
P2A173
Amalgamating individual patient data for meta-analysis:
An IT perspective
Hilken N1 , Middleton L1 , Champaneria R1 , Daniels J1
1 University of Birmingham, UK
Background: Individual patient data (IPD) meta-analysis requires raw
data from authors of primary studies, which can be provided in various
formats. An IPD meta-analysis to evaluate the relative effectiveness of
hysterectomy, endometrial ablative techniques and Mirena for heavy
menstrual bleeding illustrates potential problems in collating disparate
data sets. Objectives: To describe the data management and
synthesis aspects of an individual patient data meta-analysis from an
IT perspective. Methods: We requested IPD from 30 RCT authors and
received responses from 17. Data were transformed into Third Normal
Form (3NF) for each study, and translated to use homogeneous naming
and coding conventions. A data import specification was created using
Microsoft Excel that used VBA macros to create the necessary SQL
scripts. The method provided a single authoritative source recording
all data transformations. All import code was directly generated in a
swift and repeatable manner from the specification itself. Results: IPD
from 17 studies was received over a 16 month period, as spreadsheets,
databases, SAS, SPSS files or on paper, mostly lacking descriptive
meta-data. The specification spreadsheet defined all coded lists for
discrete variables, and mappings for each data set to a common coding
convention. Automated code generation allowed collaborative effort
from staff without programming skills on the import specification.
VBA macros create SQL scripts for creating tables, relationships, views
and data inserts, checks, and SAS files necessary to consume the
data. Recommendations: An IPD meta-analysis is a substantial
undertaking, particularly where non-standard ordinal data variables
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
112
such as ‘satisfaction’ are included. We recommend the creation of a
master relational database, with automated data import to maintain
data integrity and quality. Trialists should also consider further data
use, apply accepted data standards, use relational databases and
thoroughly document datasets, and ensure meta-data are stored along
with data in the long term.
P2A175
A descriptive analysis of overviews of reviews published
between 2000 and 2010
Hartling L1 , Chisholm A1 , Thomson D1 , Dryden DM1
1
Canada
Background: Overviews of systematic reviews (overviews) compile
data from multiple systematic reviews (SRs) and are a new method
of evidence synthesis. Objectives: To describe the methodological
approaches in overviews of healthcare interventions. Methods: We
searched four databases from 2000 to May 2010. We included studies
if they: stated a clear objective; examined an intervention; used
explicit methods to identify SRs; collected and analysed outcome data
from the SRs; and intended to include only SRs. Two researchers
independently screened studies and applied eligibility criteria. One
researcher extracted data with verification by a second. We conducted
a descriptive analysis. Results: The search yielded 1661 citations; 70
overviews were included. The number of overviews increased from 1 in
2000 to 10 in 2009. The interventions were pharmacological (n = 17),
non-pharmacological (n = 42), or both (n = 11). Inclusion criteria
were clearly stated in most overviews (n = 63). The majority searched
at least two databases (n = 47), reported the years and databases
searched (n = 48), and provided key words (n = 48). Eighteen
overviews included Cochrane SRs only. Two reviewers independently
screened and completed full text review in 27 overviews. Methods of
data extraction were reported in 32. Quality of individual studies was
extracted from the original SRs in 11 overviews. Quality assessment
of the SRs was performed in 32 overviews; at least 13 different tools
were used with the Oxman and Guyatt tool most frequent (n = 12).
The quality of evidence was assessed in 11 overviews. Most overviews
provided a narrative or descriptive analysis of the included SRs; a
quantitative analysis was conducted in only two cases. One overview
conducted indirect analyses. Publication bias was discussed in 19
overviews. Conclusions: This study shows considerable variation in
the methods used for overviews. There is a need for methodological
rigour and consistency in overviews, as well as empirical evidence to
support the methods employed.
P2A176
Methodological challenges associated with network
meta-analyses: The experience of the Musculoskeletal
Group
Tanjong Ghogomu E1 , Maxwell L1 , Singh J2 , Christensen R3 , Wells G1 ,
Buchbinder R4 , Tugwell P1
1 Cochrane Musculoskeletal Group, Canada; 2 Cochrane Musculoskeletal
Group, USA; 3 Cochrane Musculoskeletal Group, Denmark; 4 Cochrane
Musculoskeletal Group, Australia
Background: Many traditional meta-analyses summarize the results
of studies that compare an intervention with placebo. Often there is
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
limited or insufficient evidence from direct head-to-head comparisons
of competing interventions. A network meta-analysis enables the
comparison among different interventions in the absence of direct
head-to-head comparisons. Evidence from a network meta-analysis is
reliable if studies with very similar study characteristics and eligibility
criteria (study designs, populations, interventions, comparisons and
outcomes) are combined. Objectives: To discuss the methodological
and process challenges involved in network meta-analyses and how to
overcome them. Methods: We used the example of two Cochrane
overviews of reviews on biologics involving network meta-analyses
to illustrate the methodological challenges encountered and how we
overcame them. We compared the efficacy and safety of six different
biologics in patients with rheumatoid arthritis in the first overview. We
found limited evidence on the adverse effects such as tuberculosis, life
threatening fungal infections and lymphomas that may be common
with biologics, so we undertook a second overview to assess the
adverse effects of nine biologics across different indications of use. The
unit of analysis was different in the two overviews: Cochrane systematic
reviews in the first, and clinical trials in the second. Results: We faced
different methodological considerations, including eligibility criteria and
methods of assessment of risk of bias and analysis depending on the
type of overview most appropriate to answer our question. Challenges
included managing a large volume of data for screening and extraction;
achieving consensus with a large team of extractors; preparing data for
analysis; and presenting results using the Review Manager overview
template. Conclusions: Our experience showed that where limited
evidence is available for direct comparisons of different interventions,
well-conducted indirect comparisons based on network meta-analyses
are challenging to undertake, but necessary to guide health decision
making.
P2A177
Are all Cochrane reviews born equal? Methodological
quality across different Review Groups
Pantoja T1 , Moreno G2 , Mayhew A3 , Rivera S2 , Claro JC4
1
Departamento Medicina Familiar, Pontificia Universidad Católica de Chile/
Cochrane EPOC group; 2 Departamento Medicina Familiar, Pontificia
Universidad Católica de Chile; 3 Cochrane EPOC group; 4 Departamento
Medicina Interna, Pontificia Universidad Católica de Chile
Background: High quality systematic reviews are increasingly
recognized as providing the best evidence to inform healthcare decisions
at different levels. The quality of a review—and then the confidence
that decision-makers put on its findings—depends on the extension to
which its design will generate unbiased results. Published reviews vary
considerably in their quality, with Cochrane reviews rating consistently
better than non-Cochrane reviews. However, little is known about the
quality of reviews across different Review Groups in the Collaboration.
Considering that the review production process in the Collaboration
is highly decentralized some variation is expected, but its magnitude
is not known. Objective: To compare the methodological quality
of systematic reviews produced by different Cochrane Review Groups.
Methods: All new reviews published in issues 3 and 4, 2011 of The
Cochrane Library were selected for appraisal. Each selected review
was independently assessed by two evaluators using the AMSTAR tool.
The percentage of reviews from each Review Group that met each of
the 11 AMSTAR items and the proportion that met a ‘high-quality’
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
113
score were calculated. Data were analyzed using non-parametric
approaches. Results: Fifty reviews were selected for appraisal
(31 from issue 3;19 from issue 4). In our initial assessment of a
subsamplesome differences in ‘quality’ were found. However, we have
not completed the assessment of the full sample in order to attribute
those differences to the Review Groups where they were produced.
Conclusions: It might be differences in the ‘methodological quality’
of reviews produced by different Review Groups. If this is verified in
the analysis of the whole sample of reviews more work should be done
within the Collaboration to assure appropriate quality standards of
its main product through changes in the editorial processes of review
groups.
P2A178
Systematic reviews of the effectiveness of quality
improvement strategies and programmes: Some
methodological issues
Pantoja T1 , Mayhew A2 , Grimshaw J2
1 Departamento Medicina Familiar, Pontificia Universidad Católica de Chile,
Chile; Cochrane Effective Practice and Organisation of Care Group, Canada;
2 Cochrane Effective Practice and Organisation of Care Group, Canada
Background: Well conducted systematic reviews are increasingly
seen as providing the best evidence to guide choice of quality
improvement strategies in healthcare. But in contrast with reviews of
pharmacological interventions, a number of methodological challenges
should be addressed by reviewers undertaking reviews in this area.
Objective: To discuss methodological issues relevant to the conduct of
reviews of quality improvement strategies and programmes. Methods:
We built on our experiences in the Cochrane Effective Practice and
Organisation of Care (EPOC) Group and on a focused review of
the methodological literature in the area. Results: We have
selected the following issues for a further analysis: i) Formulating
a relevant question and inclusion criteria: study designs, populations
and outcomes; the use of non-randomised evidence for effectiveness
questions; the lack of an accepted taxonomy for quality improvement
strategies; and the broadness/narrowness of the review question (the
‘lumping versus splitting’ debate). ii) Identifying and screening evidence
sources: published versus nonpublished literature; search strategy
considerations. iii) Methods of synthesis/analysis: appropriateness
of meta-analysis; other synthesis approaches (range of effect sizes);
dealing with expected heterogeneity; identifying effect modifiers.
Conclusions: We have discussed issues that authors will face
when conducting reviews of the effectiveness of quality improvement
strategies. Additionally we have presented some of the current
methodological developments in order to contribute to more sound
and informative reviews in this field.
P2A179
Minimal clinically important differences in acute pain
Tendal B1 , Ortner N1 , Gøtzsche PC1 , Hróbjartsson A1
1 The Nordic Cochrane Centre, Norway
Background: Pain is an important outcome in a large number of
clinical trials and systematic reviews. Trials investigating interventions
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
for painful conditions frequently present their findings as the difference
between the mean pain score. However, it is unclear when such a
difference is large enough to be clinically relevant, making interpretation
of such measures challenging. An attempt to solve this problem is to
empirically determine a minimal clinically important difference (MCID)
i.e. the smallest difference in a score which patients perceive to be
beneficial or harmful and which would lead the clinician to consider
a change in treatment. The MCID is also important in order to
estimate adequate sample sizes in clinical trials. Empirical studies
aimed at determining MCIDs report differing results. Objectives:
To assess the typical minimal clinically important difference in acute
clinical pain and to examine reasons for its variability. Methods:
We searched PubMed, EMBASE and The Cochrane Library with no
language restrictions. We included empirical studies examining MCID
in acute clinical pain on one-dimensional pain scales (e.g. visual
analogue scale or numeric rating scales). Studies using retrospective
pain assessment were excluded. Two independent observers extracted
data. We plan to estimate the average MCID with random effects
meta-analyses, and to perform subgroup analyses to examine the
effects of co-factors (baseline pain, gender and age). If studies are
considered too heterogeneous for a meaningful quantitative summary,
we will summarise results qualitatively. Results: We have preliminary
included 23 studies and analyses are ongoing, to be presented at
Cochrane Colloquium.
P2A180
Does patient expectation affect treatment outcomes? A
preliminary investigation on expectations and outcomes
Lee H1 , Zhang X1 , Cho S2 , Lee H1
1
Acupuncture & Meridian Science Research Centre, Kyung Hee University,
Korea; 2 Hospital of Korean Medicine, Kyung Hee University Medical Centre,
Korea
Background: Non-significant findings from sham/placebo-controlled
trials are usually interpreted as the intervention being merely a placebo
in spite of a range of influencing factors, including expectation.
Although patients’ belief or expectation regarding the benefit of
treatment is regarded as a potentially important factor for yielding
positive outcomes, data from clinical trials have shown inconsistent
findings. Objectives: To investigate the impact of patients’
expectation regarding treatment on outcomes in randomised sham/
placebo-controlled trials reporting pain-related outcomes. Methods:
Electronic searching was conducted in Ovid MEDLINE in April 2011 to
include: (1) randomized sham/placebo-controlled trials or secondary
analysis of relevant RCT data; and (2) studies measuring general or
treatment-specific expectations and the association with analgesic
outcomes. We excluded laboratory-based experiments involving
healthy volunteers or studies that did not report analgesic outcomes. As
substantial heterogeneity between studies was expected, a qualitative
review was conducted. Methodological quality assessment was
performed using the modified Oxford quality scale. Results: Of
174 studies, 5 reports including 6 RCTs and 4 secondary analyses
of 7 relevant RCTs involving 2645 patients met our inclusion criteria.
Interventions included acupuncture (n = 6), manipulation (n = 1),
and drugs (n = 2) in mainly chronic pain conditions. All included
studies were assessed as of highquality (≥4 points). At different
time points, 7 studies measured patients’ expectations or belief in the
intervention, while the other 2 studies additionally sought physicians’
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
114
expectations. Trials where no difference was found between tested
intervention and placebo control, reported heterogeneous findings,
i.e. high expectation does not always lead to better analgesic
effect. Blinding success, how expectation is measured, and expectation
measurement time-points varied across the studies, which may have
contributed to conflicting results. Conclusions: The current evidence
doesnot fully support the notion that expectation produces less pain,
calling for further investigation of this issue. A careful approach to
interpreting non-significant findings of sham-controlled trials is needed
in systematic reviews.
P2A181
Consulting with young people to inform systematic
reviews: An example from a review on the effects
of schools on health
Jamal F1 , Harden A1 , Wells H2 , Bonell C2
1 University of East London, Institute for Health and Human Development,
UK; 2 London School of Hygiene and Tropical Medicine, Department of
Social and Environmental Health Research, UK
Background: There has been increasing interest in involving the
public in systematic reviews to help ensure the review is focused on
issues that are important and relevant to them. Case examples of
this involvement are rare, especially for reviews focused on children
and young people. Objectives: This study aims to describe the
process and outcome of consulting with young people in a review of
the effects of school environment interventions on children and young
people’s health. Young people’s perspectives were sought alongside
the perspectives of teachers, policy-makers and academics to determine
the scope and priorities for review. Methods: Consultations were
conducted with a pre-existing group of young people brought together
to advise on public health research. Their views were sought at three
stages: 1) at the beginning when general views relating to the policy
problem under study were elicited; 2) half-way through when views
on the map of identified evidence were sought; and 3) near the end
when the review results were presented and discussed. Face-to-face
to consultation methods were used, supplemented with online social
networking for young people to continue discussion. Results: Each
consultation stage helped shape the review process. For example, the
first and second consultations helped researchers to understand the
policy problem in its current context and informed the development of
inclusion criteria for priority review areas. Face-to-face consultation
was better at engaging young people but social networking gave the
review team greater flexibility to elicit further views when unanticipated
issues arose that needed immediate input. Conclusions: Consulting
with young people alongside other stakeholders added considerable
value to the review, especially in terms of helping the review team to
prioritise in a relevant and meaningful way. The use of a pre-existing
group facilitated engagement, but the use of social networking requires
further evaluation.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P2A182
Using systematic reviews to study adverse drug reactions:
The example of heparin-induced thrombocytopenia
Junqueira DRG1 , Carvalho MDG2 , Perini E1
1 Centre of Drug Studies, School of Pharmacy, Federal University of Minas
Gerais, Brazil; 2 Department of Clinical and Toxicological Analysis, School
of Pharmacy, Federal University of Minas Gerais, Brazil
Background: Methodology to systematically review drug efficacy
based on randomized controlled trials (RCT) is well-established.
Nevertheless, evidence-based healthcare may be supported by balanced
evidence on benefits and harms of healthcare interventions.
Development of standard recommendations to systematically review
harm effect with the same emphasis on treatment benefit is of
remarkable importance. Objectives: To describe the experience of
conducting a systematic review for an adverse drug reaction (ADR).
Methods: We conducted a systematic review based on standard
methodology drawn on the Cochrane Handbook for Systematic Reviews
of Interventions. The review was adapted to evaluate evidence about
the risk of developing an ADR, heparin-induced thrombocytopenia
(HIT). HIT is a paradoxical and unpredictable reaction diagnosed by
clinical and laboratory assessment. Results: The review question
focussed on a specific ADR in a selected, susceptible population. This
was a pragmatic decision mostly concernedwith the quality evaluation
of the ADR and impact of the clinical decision. We considered both
blinded and unblinded RCTs for inclusion, and we did not limit our
search strategy to ‘adverse effects’.. Search strategies considered
the intervention (anticoagulant) and the outcome (HIT). The quality
assessment applied the Handbook directions, but special attention was
paid to the appropriateness of HIT diagnosis according to its clinical
and serological profile. We included assessment of adequacy of ADR
monitoring as an additional bias criterion. Only one study was included.
Most studies were excluded because the definition of HIT or diagnosis
process was inappropriate. Conclusions: The few studies included
in the review illustratethe limitations of RCT to evaluate unpredictable
adverse events induced by drug use, such as HIT. Characteristics
of some ADRs, together with poor quality reporting, analysis and
discussion, may justify a systematic review focused on specific adverse
outcomes and inclusion of observational studies.
P2A183
A network meta-analysis of randomized controlled trials
of target therapy for mRCC
Chan A1 , Leung H2
1 Chi Mei Medical Centre, Tainan, China; 2 Wan Fang Hospital, China
Background: there are six new target therapeutic drugs approved
for the treatment of metastatic renal cell carcinoma (mRCC) since
2006. At present, no direct head to head trial to compare their
efficacy is available. Methods: A systematic literature search of
the MEDLINE, EMBASE, CANCERLIT and CENTRAL databases was
performed. All randomized clinical trials of sorafenib, sunitinib,
everolimus and pazopanib used to treat mRCC were included. The
study selection, data extraction, and quality assessment were performed
independently by two reviewers, with all disagreements resolved by
consensus. The main outcome was benefit (defined as an improvement
in progression median disease-free survival or overall survival). We
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
115
did the network meta-analysis using mixed-effects logistic regression
to perform an indirect comparison of the treatment effects between
target therapeutic drugs. The quality appraisal was analyzed using the
Jadad scale. Results: Compared with placebo or interferon-α, target
therapeutics may improve survival benefit (Hazard ratio (HR): 0.51,
95% CI 0.05 to 0.98). The proportion of patients who discontinued
the study drug due to adverse events was similar between sorafenib
and pazopanib; but everolimus was higher than sunitinib (10% in
everolimus and 6% in the sunitinib). Sorafenib is likely less effective
than other target therapeutics, although this difference was statistically
significant for the comparison with sunitinib (HR: 0.47, 95% CI 0.316
to 0.713) and everolimus (HR: 0.159 95% CI 0.091 to 0.276).
However, the effectiveness of sorafenib was not statistically different
from pazopanib by using placebo as a comparator (HR: 0.957, 95%
CI: 0.657 to 1.393). The results of quality assessment showed that
60% of studies were scored 5 and 40% were scored 3. Conclusions:
Our findings suggest that most of the targeted therapeutic drugs
included in this study have reportedly favorable PFS or overall survival
for patients with mRCC compared to IFN-α or placebo. Given the
limitations of indirect comparisons, sorafenib is likely less effective than
sunitinib and everolimus, but have similar effectiveness to pazopanib.
Sunitinib was safer than everolimus, sorafenib and pazopanib. More
RCTs are required to confirm these findings and investigate the clinical
effectiveness of targeted therapeutics in the treatment of mRCC.
P2A184
How useful is GRADEpro for assessing evidence from
epidemiological or diagnostic accuracy studies
Gholitabar M1 , Ullman R1 , Newbatt E1 , Fields E1 , Kenny J1
1 National Collaborating Centre for Women’s and Children’s Health
(NCC-WCH), UK
Background: GRADEpro software was developed to provide a
transparent and logical approach to grading evidence and strength of
recommendations or implications for practice for interventional studies.
GRADEpro uses a default for rating the quality of evidence based on
the study design (RCT or observational) and this is downgraded or
upgraded based on other factors including limitations, inconsistency,
indirectness and imprecision. Objectives: To examine the challenges
of using GRADEpro to different types of reviews and present some
possible solutions to the problems. We used GRADEpro to assess
the quality of evidence for two NICE clinical guidelines. Methods:
Three worked examples illustrating the problems of using GRADEpro
in different types of reviews will be presented. These are: i) assessment
of non-comparative data using GRADEpro, ii) rating imprecision where
the clinical cut-offs for potential benefits and harms are unclear, and
iii) presentation of findings where denominators differ between studies.
Results: Some practical solutions to commonly encountered reviewing
problems will be presented and potential for application of the GRADE
approach across a range of study types discussed. Conclusions:
Although GRADE is a methodologically rigorous method that has been
well developed for use in intervention studies questions remain about
its consistency and objectivity, especially when applied to different
study types. Further development will help to improve the GRADE
approach further but this needs to be undertaken with extensive field
testing as this is the only way to ensure its methods stand up across a
wide range of situations.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P2A185
Systematic reviews in dentistry have problems including
split-mouth and cross-over trials in meta-analysis
Carrasco-Labra A1 , Brignardello-Petersen R2 , Araya I3 , Yanine N3
1 Evidence Based Dentistry Unit, Faculty of Dentistry, Universidad de Chile,
Chile, Department of Clinical Epidemiology & Biostatistics, McMaster
University, Canada; 2 Evidence Based Dentistry Unit, Faculty of Dentistry,
Universidad de Chile, Chile, Department of Health Policy, Management
& Evaluation, University of Toronto, Canada; 3 Department of Oral &
Maxillofacial Surgery, Evidence Based Dentistry Unit, Faculty of Dentistry,
Universidad de Chile, Chile
Background: Split-mouth and cross-over trials are frequently used in
dentistry. There is some evidence about systematic reviews’ authors
having problems trying to incorporate these types of studies in
meta-analysis. Objectives: To determine to what extent authors
are including the evidence from split-mouth and cross-over trials in
systematic reviews and meta-analysis in dentistry. Methods: All the
systematic reviews published by the Oral Health Group in the Cochrane
Database of Systematic Reviews up to the issue 3, March 2011 were
included. Two researchers examined independently and in duplicated
the references list of included and excluded studies. Each reference
was searched in MEDLINE to determine whether the trial corresponded
to a split-mouth or cross-over trial. When one of these studies was
detected, the reasons for exclusion from the review were registered.
When these studies were included in the systematic review, it was
evaluated whether they were included in the meta-analysis. Also the
methods section of each review was assessed and the approaches
for including split-mouth and cross-over with the rest of the evidence
were recorded. Results: A total of 114 systematic reviews from the
Oral Health Group were retrieved. From these, 51 (44.7%) reviews
had split-mouth or cross-over trials in the reference list of included or
excluded studies. Of the reviews that included these types of design
19 (57.6%) were not able to include them in the meta-analysis. The
most frequent reasons for exclusion due to methodological flaws were
inadequate data analysis or inappropriate design (n = 40) and the
condition of non-randomized allocation of the treatment to time/side
(n = 20). Conclusions: Authors face many challenges when trying
to incorporate the evidence from split-mouth or cross-over trials in
systematic reviews. Even when these studies are included, most of
the times they do not contribute to the estimation of the treatment
effect.
P2A186
Designs and analysis of N-of-1 trials: A systematic review
Perdices M1 , Barrowman N2 , Sampson M3 , Shamseer L4 , Bukutu C5 ,
Vohra S6
1
Royal North Shore Hospital and University of Sydney, Australia; 2 Children’s
Hospital of Eastern Ontario Research Institute, Canada; 3 Children’s Hospital
of Eastern Ontario, Canada; 4 Ottawa Hospital Research Institute; 5 Alberta
Centre for Child, Family & Community Research, Canada; 6 University of
Alberta, Canada
Background: Well designed N-of-1 trials provide a valuable alternative
methodology for evaluating the therapeutic efficacy of pharmacological
or behavioural interventions when randomized controlled trials (RCTs)
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
116
are not feasible. Objectives: To evaluate methodological design,
analysis and meta-analysis used in N-of-1 trials. Search strategy:
We searched the following databases from their inception to August
2007: MEDLINE, PsycINFO, ERIC, Sociology Abstracts, PsycBITE, AMED,
EMBASE, CINAHL, and the Cochrane Database of Systematic Reviews.
Reference lists of included studies were checked. Selection criteria:
Original studies specifically dealing with methodological design and
statistical analysis of N-of-1 trials. Reports of N-of-1 trials themselves
were not eligible for this review. Data collection and analysis:
Two reviewers independently assessed the suitability of studies for
inclusion. For each paper, two authors independently extracted
content information for 64 predefined items specifically related to
design, methodology, statistical analysis and meta-analysis. Results:
We identified 95 papers that satisfied the inclusion criteria. Of these,
51 were general reviews discussing strengths, limitations and sources
of bias of basic N-of-1 designs (e.g., A-B-A-B, multiple baseline).
Nine of these specifically reviewed randomisation test procedures as
an adjunct for increasing methodological robustness. An additional
44 papers discussed specific data analysis techniques. Of these, 22
dealt with statistical analysis techniques exclusively, 3 with visual
analysis exclusively, and 11 with comparison of statistical and visual
analysis, highlighting the strengths and weaknesses of each approach.
Eight papers discussed methods for quantitative synthesis of data.
Conclusions: Although consensus exists about aspects of basic design
and analysis of data for N-of-1 trials to ensure internal and external
validity, issues related to generalisability of results and meta-analysis
of data remain contentious.
P2A187
A coherent framework for choosing ‘fit for purpose’ review
methods
Oliver S1 , Thomas J1 , Gough D1
1 EPPI-Centre, Institute of Education, UK
Background: Systematic review methods span qualitative syntheses
(e.g. meta-ethnography, thematic synthesis), mixed methods syntheses
(e.g. realist synthesis, framework synthesis) and quantitative synthesis
(e.g. statistical meta-analysis and meta-regression). They differ in
terms of how they identify studies, appraise their quality and synthesise
findings. Review methods have been applied to different types of
questions and different types of studies, often developed by different
teams working independently without a clear methodological overview.
Objectives: To make sense of the diversity of review methods available
with a coherent framework for choosing ‘fit for purpose’ review
methods. Methods: Systematic reviews from a purposive sample that
either configured or aggregated findings (1) were analysed in terms
of the source and type of their questions, their purposes (generating,
exploring or testing theory), data (qualitative, quantitative, or mixed),
methods (linear, iterative; qualitative, quantitative, or mixed), and
products (syntheses of concepts or measures). Results: A series of
visual representations of the diversity of review questions, purposes,
data and methods evolved through discussions to provide a coherent
framework for choosing between review methods. Conclusions: Three
major pathways for reviews begin with: clear conceptual frameworks
(for testing hypotheses), partially developed frameworks (for exploring
theory); or tentative concepts (for generating theory). These pathways
differ in terms of whether: they aim to find all relevant studies or
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
all relevant concepts; the search strategy is linear or iterative; they
appraise the study methods, findings or both; they aggregate or
configure the findings of studies; they comment on the risk of bias or
coherence of findings; and how well they are suited to instrumental
models of knowledge transfer, or interactive or enlightenment models
of knowledge exchange.
Reference
Sandelowski M. Reading, writing and systematic review. J Adv Nurs.
2008;64(1):104–10.
P2A188
Incorporating the GRADE approach in overviews
of systematic reviews: an example from an overview
in neonatal respiratory care
Rojas M1 , Lozano J2 , Solà I3 , Bonfill X3
1
Pontificia Universidad Javeriana, Colombia; 2 Florida International
University College of Medicine, USA; 3 Iberoamerican Cochrane Centre,
Spain
Background: Evidence from systematic reviews (SR) addressing
different comparisons for the same treatment could provide effect
estimates of different quality. Overviews of SR incorporating GRADE
to rate the quality of evidence would provide readers with summaries
of evidence and classification of its quality. The accuracy and
reproducibility of the quality evaluation process entails some challenges
for overview authors. Objectives: To describe the reliability
aspects in the evidence synthesis process applying the GRADE
framework, in the development of an overview of SR in the field
of respiratory care in neonates. Methods: The selection process was
completed independently by two authors. A third author assessed
the methodological quality of eligible SR using the AMSTAR. Two
authors independently summarized the evidence for each comparisons/
outcomes in GRADE profiles. Consensus about the importance of
outcomes for patients was reached ad hoc. Results from trials
published after the SRs’ search dates were added to the profiles.
A third author compared the quality ratings provided by authors
for each outcome and checked the accuracy of the overall process.
Measurement of inter-rater agreement was done using the Cohen’s
kappa coefficient. Reasons given by authors to rating down the quality
of evidence for each outcome were also compared and observed
agreement presented. Results: The presentation will describe the
above mentioned methods implemented in an overview assessing the
effect of different ventilation strategies in preventing bronchopulmonary
dysplasia and other neonatal outcomes in preterm infants with
respiratory distress syndrome. A total of six SR addressing eight different
comparisons between ventilation modes and strategies were included,
and seven outcomes assessed. Only one additional trial was found and
incorporated in the meta-analyses for each outcome. Conclusions:
The example illustrates some challenges in rating the quality of evidence
from the GRADE framework in the context of developing overviews of
SR, and the need of integrating reliable consensus methods among
reviewers.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
117
P2A189
A pilot Mixed Methods Appraisal Tool (MMAT) for
systematic mixed studies reviews
Pluye P1 , Cargo M2 , Robert E3 , Bartlett G1 , O’Cathain A4 , Griffiths F5 ,
Boardman F5 , Gagnon M6 , Rousseau M3
1
McGill University, Canada; 2 University of South Australia, Australia;
3 Université de Montréal, Canada; 4 University of Leeds, UK; 5 University of
Warwick, UK; 6 Université Laval, Canada
Background: A new form of literature review has emerged, mixed
studies review. These reviews include qualitative, quantitative and
mixed methods studies. The Mixed Methods Appraisal Tool (MMAT)
is intended to be used as a critical appraisal tool for systematic
mixed studies reviews (SMSRs). In SMSRs, reviewers appraise studies
that use different methods allowing them to obtain in-depth answers
to complex research questions. However, appraising the quality of
studies with different methods remains challenging. To facilitate
this, the MMAT has been developed. The MMAT has been pilot
tested using 32 evaluation studies for efficiency and reliability.
On average, it took 14 minutes to appraise a study. Agreement
between reviewers was moderate to perfect with regards to MMAT
criteria, and substantial with respect to the overall quality score of
appraised studies. Content validation: The current 2011 version
of the MMAT is based on a literature review of MSRs, a mixed
methods framework, and has been revised using feedback from four
90-minute workshops. The workshops involved diverse audiences
such as graduate students enrolled in a mixed methods research
course, researchers and research professionals with experience in
qualitative, quantitative, and mixed methods research, and members
of the Cochrane collaboration with experience in SMSRs or in
systematic reviews of qualitative studies, e.g., meta-ethnography, or of
quantitative studies, e.g., systematic reviews of randomized controlled
trials. Results: The current 2011 version of the MMAT (checklist
and tutorial) contains two screening questions, and 19 questions
corresponding to four types of studies (methodological domains):
qualitative research, randomized controlled trials, non-randomized
studies, quantitative descriptive, and mixed methods studies (http://
mixedmethodsappraisaltoolpublic.pbworks.com). Conclusion: The
MMAT is validated and unique for appraising the most common types
of study designs, including mixed methods studies. The development
of the MMAT is supported by an international research team with
expertise on qualitative, quantitative and mixed methods research.
P2A190
Systematic reviews of adverse events. Is screening by title
and abstract enough?
Zorzela L1 , Punja S1 , Joffe A1 , Hartling L2 , Loke Y3 , Vohra S4
1 University of Alberta, Canada; 2 Univerity of Alberta, Canada; 3 Univeristy
of east Anglia, UK; 4 University of Alberta, Canada
Background: Papers selected for systematic reviews are generally
screened by title and abstract. Since harms are reported poorly,
screening using title and abstract may miss relevant papers when
compared to screening full text. Objectives: To identify if systematic
reviews of harms using the traditional method of screening papers
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
based on title and abstract will omit relevant studies, in comparison
to screening based on full text. Methods: We are proceeding with
two systematic reviews of adverse events associated with propofol
infusion in pediatric patients. Only clinical trials will be selected
(randomised or quasi-randomised). In the first review, papers are
screened by title and abstract, looking for reports of presence or
absence of adverse events. In the second review, the papers are
screened using full text for reports of adverse events. Results:
Both reviews are underway and full results will be shared at the
Cochrane Colloquium. A total of 109 papers fulfilled the inclusion
criteria and were selected, of those, 59 publications reported adverse
events in title or abstract and 14 only reported at the results
section. Thirteen publications had not reported the presence or
absence of adverse events. A total of 23 non-English publication
are in the process of being translated. Conclusions: Although at
a preliminary state, the partial data suggest that clinical trials do
not consistently report the presence or absence of adverse events.
Systematic reviews of harms may need to screen studies based
on full text in order to capture all applicable studies and reduce
type II error.
P2A194
Challenges facing reviewers preparing overviews
of reviews
Flodgren G1 , Shepperd S1 , Eccles M2
1 University of Oxford, UK; 2 Newcastle University, UK
Background: An overview of reviews compiles evidence from multiple
reviews into one accessible document, thus helping decision makers to
choose effective healthcare interventions. Overviews are challenging.
For instance, results from individual studies are not always adequately
presented in reviews to allow optimal compilation of data. Objectives:
To describe the difficulties of preparing an overview of reviews.
Methods and Results: We summarised evidence from systematic
reviews on the effect of financial incentives on professional practice and
patient outcomes, including both Cochrane (CR) and non-Cochrane
reviews (NCR). From a large number of seemingly eligible reviews,
only 2 CRs and 2 NCRs, presenting quantitative data and fulfilling
the DARE criteria for an adequate search, were included. Excluded
reviews typically did not present numerical data or failed to present
data at the level of individual studies. Comparison of the quality
of the evidence reported in included reviews was difficult because
they used different assessment tools. We used the AMSTAR tool to
assess the methodological quality of reviews, and found the quality
to be high even though the presentation of numerical data was
incomplete for many outcomes. We attempted to retrieve missing data
by consulting individual studies or their authors. Still, we could not
retrieve the information needed for meta-analysis. Instead, results were
summarised using vote counting within financial incentives across type
of outcome, and within outcomes across types of financial incentives.
Conclusions: Inadequately reported data in reviews is a major
problem in the preparation of overviews over reviews. Differences in
the tools used by reviewers to assess study limitations, is another.
This makes it very difficult to summarise and evaluate large amounts
of heterogeneous data, in order to present it in an accessible format.
Researchers and reviewers alike must improve the reporting of data.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
118
P2A195
GRADE in Cochrane systematic reviews: An investigation
Xiao J1 , Zhen H1 , Chen Y1 , Yuan J1 , Ke W1 , Li Y1 , Li M1 , Zhao X1 , Li X1 ,
Liu Y1
1 Evidence-Based Medicine Centre, Lanzhou University, China
Background: GRADE system aims to evaluate the quality of evidence.
Objectives: To assess the use of GRADE in Cochrane systematic
reviews (CSRs). Methods: We searched The Cochrane Library (2011,
3 issue) for CSRs. Two reviewers independently selected CSRs which
used GRADE. Then the data was extracted and analysised. Results: In
total 2,169 CSRs were searched. 307 CSRs have been identified using
GRADE to evaluate the quality of evidence. Most (81.11%; 249/307)
reviews were published after 2008, published by the Cochrane
Musculoskeletal Group, Cochrane Neuromuscular Disease Group and
Cochrane Schizophrenia Group. There were a total 1,213 outcomes
which were assessed by GRADE. Up to 14.59% (177/1,213) outcomes
were rated as high quality; 33.8% (410/1,213) outcomes were rated
as moderta quality; 28.20% (342/1,213) outcomes were rated as
low quality; 12.12% (147/1,213) were rated as very low quality.The
outcomes were downgraded mainly because study limitations and
imprecision. Conclusions: Nowadays, few CSR uses GRADE system.
Study limitations was the main factor which might decrease the quality
of evidence.
P2A196
Drug reviews should state industry-related bias
in abstracts and plain language summaries
Napoli M1 , Simi S2
1 Center for Medical Consumers, New York, USA; 2 Italian National Research
Council (CNR), Institute of Clinical Physiology, Pisa, Italy
Background: The abstract and plain language summaries (PLS) are the
sections from Cochrane reviews most read, including many clinicians.
Many, if not all, trials included in Cochrane drug reviews are conducted
by drug makers. The withholding of serious adverse events data from
industry-funded trials is not unusual. Such industry-related biases may
be noted within the text of a Cochrane drug review; yet they rarely
appear in the abstract or the PLS. Objective: This concern was initially
aired on the Cochrane Consumer Network e-discussion list. We now
bring this discussion to the larger community of Cochrane contributors,
many of whom were in the forefront of exposing bias in industry-funded
research. Review authors should address industry-related bias in the
abstract and PLS when relevant. This would be consistent with the
Cochrane handbook which suggests the abstract and the PLS be written
as stand-alone documents. Interventions: Our poster will come with
handouts that show how two Cochrane review groups have already
described industry-related biases in the abstract and PLS. During ‘poster
prime time’ we will be present for discussions with poster viewers and
we will provide a feedback sheet to record their interest in this topic.
Conclusion: If Cochrane drug reviews fail to address industry-related
bias in the abstract and PLS, consumers and physicians are denied
information needed to make informed decisions. This is particularly
critical now that millions of healthy people worldwide are on long-term
drug therapy merely because they have a risk factor like bone loss or
high cholesterol.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Investigating bias
Day 21 Oct (P2B197–P2B260)
P2A197
Association between risk of bias and the effects of
interventions on mortality in randomized controlled trials
in critical care medicine
Unverzagt S1 , Prondzinsky R1
1 University Halle, Germany
Background: Methodological deficiencies are known to affect the
results of randomized controlled trials (RCTs). Differences in the risk
of bias may indicate that the effect estimates of some trials are more
biased than others. Objectives: This work concentrates on reviews
within emergency and critical care medicine and estimates the extent of
bias on mortality. RCTs in patients with acute and partially rare disease
are typically small and heterogeneous because of organizational, ethical
and judicial problems. High risk of bias can result from long periods of
recruitment with changing standard treatment or problems in funding.
Risk of bias was assessed using the Cochrane Collaboration’s tool
and six additional indication-specific quality measures: early stopping,
pre-intervention, conflict of interest, baseline imbalances, cross-over
trials, and insufficient length of follow-up. Methods: Subgroup
analyses, logistic regression models, and fixed and random effect
models were used to investigate whether quality measures explain any
of the heterogeneity of odds ratios in 73 randomized trials from eight
meta-analyses of Cochrane reviews of patients with sepsis, or septic
or cardiogenic shock. Results: The percentage of trials with low risk
of bias varied between 23% in early stopping and 81% in addressing
incomplete outcome data. Overall, odds ratios were underestimated in
trials with inadequate allocation concealment, selective reporting, or
if a contra-active or similar supporting pre-randomization intervention
could have influenced the effect of the randomized intervention or
cross-over could not be ruled out. Odds ratios were overestimated in
RCTs with incomplete reporting of mortality. Full data will be presented
at the colloquium. Conclusions: Investigated sources of bias can
lead to exaggerated or reduced estimates of the intervention effect in
this medical area. These associations will support the assessment of
risk of bias in Cochrane reviews on interventions in intensive care and
emergency medicine.
P2A198
Extraction error or interpretation: A case study of event
data extraction in systematic reviews from three
high-impact, high-quality peer-reviewed sources
Carroll C1 , Scope A1 , Kaltenthaler E1
1
University of Sheffield, UK
Background: Data extraction is a crucial but under-researched area
of systematic review process and methods. Data extraction errors may
occur in all variables extracted for a review, but outcomes appear
to generate the most errors. Objectives: To assess differences
in the event data extracted and analysed for three key outcomes
in three systematic reviews on the same topic published within 18
months of one another in the following sources: The Cochrane Library
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
119
(CL); the British Medical Journal BMJ); and the authors’ own Health
Technology Assessment review. Methods: We compared event data
from reviews assessing the effectiveness of total hip replacement
versus hemiarthroplasty for the treatment of displaced intracapsular
hip fracture. We analysed the event data extracted and analysed
for the outcomes of dislocation rates, one-year mortality rates, and
revision rates, that were common across all three reviews, from those
trials common to all three reviews. Differences and possible bias were
investigated. Results: Across the three outcomes, extraction errors
accounted for between 8% and 25% of the event data extracted and
analysed by the BMJ review and between 0% and 17% in the CL
review, respectively. A further 8–33% differences in the BMJ review and
8–17% in the CL review might be explained by issues of interpretation,
eg. applying apparent ‘best case’ scenario analysis, but not justifying
or explaining such choices. These differences did lead to small
differences in meta-analysed relative risks, but none was significant.
Conclusions: Systematic reviews require full implementation of its
required processes to offer a robust method of evidence review and
synthesis. The presence of errors in key outcome data suggests
that full and proper implementation is not always achieved, even
with apparent ‘double data extraction’ within high-quality sources of
systematic reviews. Secondly, reviewers should make every effort to
clarify or explain their choice of data.
P2A199
Heterogeneity in systematic reviews on spinal surgery: A
meta-epidemiological study
Jacobs W1 , Kruyt M2 , Verbout A2 , Oner C2
1 Leiden University Medical Center, The Nethrlands; 2 University Medical
Center Utrecht, Netherlands
Background: Methodological design characteristics of trials in
meta-analyses have been shown to influence pooled effect sizes
and are therefore recognised as potential sources of heterogeneity
in several medical fields. However, for spinal surgery, the presence
and the strength of the effect of effect modifiers have not been
assessed. Objectives: The goal of this study was to identify
the influence of design characteristics on effect size in studies on
effectiveness of spinal surgery. Methods: Searches were performed in
MEDLINE, CDSR and EMBASE. Methodological quality of the included
meta-analyses was assessed with the Amstar tool. The effect sizes of
trials included in the meta-analyses were assessed. The differences in
effect sizes were calculated as risk differences (RD). Relation of RDs to
sponsoring, randomization, allocation concealment, blinding and study
size was assessed with meta-regression adjusted for multiple testing.
Results: Seven reviews consisting of 76 studies and 8191 patients
were included. Data provided by the systematic reviews alone was
insufficient to analyse heterogeneity. Meta-regression analysis of the
individual trials though, showed that sample size, strict randomization
and outcome blinding were significant factors influencing study effect.
Validly randomised studies showed an increase in RD of 5.4% (95%
CI 1.2 to 9.6; p = 0.044) compared to not validly randomised and
observational trials. Studies with adequate observer blinding showed
a 7.2% decrease in RD (95% CI 0.8 to 13.7; p = 0.049). For each
increase of 100 patients, the RD decreased 3.6% (95% CI 0.4 to
6.8; p = 0.098). Conclusions: Contrary to basic methodological
assertions, formal and strict randomization appeared to increase the
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
risk difference significantly in spinal surgery research. Sufficient sample
size and observer blinding, on the other hand, can decrease this
risk difference. This may imply that by trying to limit bias, strict
randomization may actually cause bias.
P2A200
Attributes of non-randomized studies associated with bias
Sandhu L1 , Tomlinson G1 , Kennedy E1 , Wei A1 , Baxter N1 , Urbach D1
1 University of Toronto, Canada
Background: Randomized controlled trials with inadequate allocation
concealment or a lack of double-blinding are associated with biased
estimates of treatment effects. However, empirical evidence of bias
associated with the methodologic attributes of non-randomized studies
is lacking. Objectives: To identify methodologic attributes of
non-randomized studies that are associated with biased estimates
of treatment effect. Methods: We identified English language
randomized and non-randomized studies comparing laparoscopicassisted and conventional surgery for colon cancer. We estimated odds
ratios of the risk of post-operative complications as the treatment effect
of interest. Meta-regression was used to determine the association
between systematic outcome assessment—an explicit statement of
a standardized protocol for the collection of outcome data—and
estimates for treatment effect. Results: We identified 155 comparative
studies from 6,261 abstracts. Of which, 62 studies (41 NRS and
21 RCTs) involving 857,132 patients provided estimates of the risk
of post-operative complications. Studies where outcomes were not
assessed in a systematic manner estimated larger treatment effects;
odds ratios were exaggerated by 26% (95% CI, 4%–44%) as compared
with studies with systematic outcome assessment. Conclusions:
Non-randomized studies with systematic outcome assessment are less
prone to bias.
P2A201
Reporting of participant flow diagrams in published
reports of randomized trials
Hopewell S1 , Hirst A2 , Collins G2 , Mallett S2 , Yu L2 , Altman D2
1 Centre for Statistics in Medicine, University of Oxford, UK; UK Cochrane
Centre, UK; 2 Centre for Statistics in Medicine, University of Oxford, UK
Background: Reporting of the flow of participants through each
stage of a randomized trial is essential to assess the generalisability
and validity of its results. Objectives: To provide a comprehensive
assessment of the type and completeness of information reported in
CONSORT flow diagrams published in current reports of randomized
trials. Methods: A cross-sectional review of all primary reports of
randomized trials which included a CONSORT flow diagram indexed in
PubMed core clinical journals (2009). We assessed the proportion of
parallel group trial publications reporting specific items recommended
by CONSORT for inclusion in a flow diagram. Results: Of 469
primary reports of randomized trials, 263 (56%) included a CONSORT
flow diagram of which 89% (237/263) were published in a CONSORT
endorsing journal. Reports published in CONSORT endorsing journals
were more likely to include a flow diagram (62%; 237/380 versus
29%; 26/89). Ninety percent (236/263) of reports which included a
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
120
flow diagram had a parallel group design, of which 49% (116/236)
evaluated drug interventions. Eighty-one percent (191/236) reported
the overall number of participants assessed for eligibility, 71%
(168/236) the number excluded prior to randomization and 98%
(231/236) the overall number randomized. Reasons for exclusion prior
to randomization were more poorly reported. Ninety-four percent
(223/236) reported the number of participants allocated to each arm
of the trial. However, only 40% (95/236) reported the number who
actually received the allocated intervention, 67% (158/236) the number
lost to follow up in each arm of the trial, 61% (145/236) whether
participants discontinued the intervention during the trial and 54%
(128/236) the number included in the main analysis. Conclusions:
Over half of the published reports of randomized trials included a
diagram showing the flow of participants through the trial. However,
information was often missing from published flow diagrams, even in
articles published in CONSORT endorsing journals.
P2A202
Quality of studies reporting patient-reported outcomes
identified within a systematic review on low-dose rate
brachytherapy for localized prostate cancer
Peinemann F1 , Grouven U1 , Sauerland S1
1 IQWiG
Background: Permanent interstitial low-dose rate brachytherapy
(LDR-BT) is a treatment alternative for patients with localized prostate
cancer. As evidence from research on effectiveness or safety may not
be decisive, patient-reported outcomes (PRO), such as function, bother,
and generic health-related quality of life (HRQL), could have an impact
on medical decision making. Methods: Bibliographic databases
(MEDLINE, EMBASE, and The Cochrane Library) were searched until
June 2010 for randomized as well as non-randomized parallel group
studies. Validated questionnaires, 70% or higher response rate, and
80% or more data with relevant diagnosis and therapy were required
for inclusion of studies. For non-randomized controlled studies, baseline
data were required to be comparable between treatment groups or
results were required to be adjusted for major confounders. Results:
We identified only 1 randomized study among 61 studies potentially
relevant for inclusion. PRO were reported in 32 of 61 and 7 of these 32
studies were finally included. Thus, 25 studies were excluded because
baseline data were not comparable and not adjusted or stratified
(16 studies), response was below 70% (4 studies), or data were not
reported separately for the accordant groups or outcomes (5 studies).
Of the 7 studies, 7/5/5 studies evaluated function/bother/generic HRQL
scores using 7/2/3 different instruments, respectively. Conclusions:
We found that the majority of identified non-randomized controlled
PRO studies showed selection bias too high for inclusion. Included PRO
studies lacked statistics about differences across treatment groups and
various scales complicated the synthesis of results.
P2B203
Systematic reviews of paediatric surgical interventions:
quantity, coverage and quality
McGee R1 , Craig J2 , Webster A2
1 Sydney School of Public Health, University of Sydney, Australia; 2 Cochrane
Renal Group, Sydney, Australia
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: The practice of evidence-based surgery depends on high
quality systematic reviews being readily available. Previous research
has highlighted the lack of high quality research in paediatric surgery
although systematic reviews have not been assessed. Objectives: To
determine the quantity, coverage and reporting quality of systematic
reviews of paediatric surgical interventions. Methods: MEDLINE,
Embase, DARE, PROSPERO and the Cochrane Database of Systematic
Reviews were searched for any systematic reviews of paediatric surgical
interventions published before 2011. The PRISMA checklist was used
to assess reporting quality. Results: Fifteen reviews were identified
of which 13 were Cochrane reviews. The median year of publication
was 2010 and the earliest was 2001. The most commonly assessed
area was the use of surgical treatments for the management of otitis
media (four reviews). Reporting quality was mixed (see Figure 1). In
particular, one-third of included reviews did not declare their funding
source or the role that funders played in the conduct of their review.
No review utilised the PRISMA flow diagram to illustrate the results
of their literature search. Conclusions: Systematic reviews are
unavailable for many areas in paediatric surgery. Surgeons trying
to assess the safety and efficacy of an intervention will therefore be
required to conduct their own assessments of the primary literature or
extrapolate from adult systematic reviews. To reduce the discrepancy
between best reporting quality as defined by PRISMA and the Cochrane
Collaboration, the Cochrane Handbook for Systematic Reviews of
Interventions and the PRISMA statement should be more closely
aligned. Already some progress has been made in this regard with
the recent upgrade to the Cochrane collaboration’s Review Manager
Software, which now includes the ability to create a PRISMA flow
diagram.
P2B204
Core outcomes in systematic reviews of paediatric surgical
interventions
McGee R1 , Craig J2 , Webster A2
1 Sydney School of Public Health, University of Sydney, Australia; 2 Cochrane
Renal Group, Sydney, Australia
Background: Recent initiatives have highlighted the importance of
standardised outcome measures in clinical trials to reduce reporting
bias, misinformation and wasted resources. Although it has received
less attention, the principles behind these efforts remain the same
for systematic reviews. Objectives: To determine which outcomes
are assessed and how they are defined in systematic reviews of
paediatric surgical interventions. Methods: MEDLINE, EMBASE, DARE,
PROSPERO and the Cochrane Database of Systematic Reviews were
searched for any systematic reviews of paediatric surgical interventions
published before 2011. All specified outcomes and related definitions
were recorded. Results: Fifteen reviews were identified of which 13
were Cochrane reviews. The mean number of outcomes per review was
six, with a range of two to 14 and a mean of two primary outcomes.
Outcome definitions were provided in three reviews. Time-points for
outcome assessment were only pre-specified in one review. Mortality,
adverse events (only one review was specific about which adverse
events were to be recorded) and quality of life (only one specified for
whom quality of life was being assessed) were the most commonly
reported outcomes (five reviews each). Conclusions: Outcomes
assessed in systematic reviews of paediatric surgical interventions are
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
121
Figure 1 (P2B203). Star plot of the percentage of paediatric surgery reviews adequately reporting PRISMA items.
inconsistent, poorly defined and reported, and are therefore susceptible
to a high risk of outcome reporting bias. While we have assessed reviews
covering a range of surgical treatments, there is a core set of outcomes
that should be assessed in all reviews of paediatric surgery, e.g.
mortality, adverse events and quality of life for both the child and carer.
As the Cochrane Collaboration publishes the most systematic reviews
of paediatric surgical interventions, we suggest that they develop and
promote a set of core outcomes for inclusion in systematic reviews
of paediatric surgical interventions. In addition, a set of standardised
outcome definitions may improve the reporting quality of systematic
reviews.
P2B205
‘Fatal flaws’ in randomised controlled trials
of interventions for low-back pain
Henschke N1 , Ostelo R2 , van Tulder M3
1 Musculoskeletal Division, The George Institute for Global Health, Australia;
2 EMGO Institute for Health and Care Research (EMGO+), Netherlands;
3 Department of Health Sciences, VU University, Netherlands
Background: When performing a systematic review, it is possible that
a study could be included even if it has a serious methodological flaw
which can render the results implausible. These ‘fatal flaws’can include
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
excessive attrition or a large proportion of drop-outs and withdrawals.
However, there is no consensus regarding which methodological flaws
can be considered ‘fatal’, and what we should do with trials which
have them. Objectives: The aim of this study is to identify, via
consensus among a panel of experts, which features of randomised
controlled trials (RCTs) should be considered ‘fatal flaws’ and to
develop recommendations for dealing with them. The focus for the
study will be RCTs on interventions for low-back pain. Methods:
A modified-Delphi technique using three survey rounds was used to
select and reduce a list of methodological characteristics. Experts in
low-back pain, systematic reviews, and methodology were invited to
participate in the study. These experts were asked to individually rate
the list of characteristics according towhether they consider the item
to be a serious methodological flaw that could render the results of a
study implausible. During three surveys, participants were given the
opportunity to provide reasons for their decision, to add items, or to
modify the wording of items. Consensus was considered to be reached
when the rating of at least two-thirds (67%) of the panel members
indicated agreement with an item on a five-point scale. Results: The
respondent panel consists of 68 experts. At present, the first two
rounds of the Delphi technique have been completed and the third
round is underway. Final results will be presented at the colloquium.
Conclusions: The identification of serious methodological flaws and
how to deal with trials which have them may lead to improved validity
of the results in systematic reviews.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
122
P2B206
Trends over time in the size and quality of randomised
controlled trials of interventions for chronic low-back pain
Henschke N1 , Kuijpers T2 , Rubinstein S3 , van Middelkoop M4 , Ostelo R3 ,
Verhagen A4 , Koes B4 , van Tulder M5
1
Musculoskeletal Division, The George Institute for Global Health, Australia;
2 Dutch Institute for Healthcare Improvement CBO, Netherlands; 3 EMGO
Institute for Health and Care Research, Netherlands; 4 Department of
General Practice, ErasmusMC, Netherlands; 5 Department of Health Sciences,
VU University, Netherlands
Background: Despite the high prevalence and enormous burden of
chronic low-back pain (CLBP), there remains uncertainty regarding the
most effective form of therapy. This continuing uncertainty is often
considered to be due to methodological shortcomings or a lack of
adequate reporting. In light of the increasing number of randomised
controlled trials (RCTs) and continuing initiatives to improve the quality
of clinical research, it could be expected that the methodological
quality of RCTs in the field of CLBP may have improved during
recent years. Objectives: This study aimed to identify and describe
trends over time in the study design characteristics and risk of bias
in CLBP trials performed over the past 30 years. Methods: We
extracted 157 randomised trials of interventions for CLBP from four
recently published systematic reviews. The reviews included RCTs
on physical and rehabilitation interventions, injection therapy and
denervation procedures, complementary and alternative therapies, and
pharmacological interventions for chronic LBP. Study level data were
extracted and analysed for trends associated with year of publication.
Results: Overall, the mean sample size in the RCTs was 141
(median 70; range 17 to 3093). There was a slight increase in the
median number of risk of bias criteria fulfilled from trials published
prior to 1995 to those published after 1996. The analysis showed that
in more recent years RCTs of medical interventions were more likely
to be successfully blinded than RCTs of non-medical interventions.
Conclusions: Over the past 30 years, the number of RCTs for CLBP
has increased exponentially. However, there does not appear to be
a corresponding increase in sample size or a decrease in risk of bias.
Further research is needed into specific risks of bias within RCTs for
CLBP and the effect they have on the plausibility of the results.
P2B207
The risk-of-bias and standards of reporting of published
randomized controlled trials of medical education (Med
Ed) research
Horsley T1 , Rabb D1 , Campbell C1 , Hamstra S2 , Cook D3
1 Royal College of Physicians and Surgeons of Canada; 2 AIME University of
Ottawa; 3 MAYO College of Medicine
Background: Controlling bias within randomized trials (RCTs) has
been empirically shown to reduce errors in estimates of treatment
effects. In theory, the RCT is less susceptible to bias than any other
research study design. It is this ‘susceptibility to bias’ that is the focus
of this work. Objectives: We aimed to establish the risk of bias and
standard of reporting of RCTs of medical education (Med Ed) research.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Methods: Records were independently assessed until a representative
sample (N = 150) was achieved. Publications were evaluated in
duplicate using the Cochrane risk of bias (ROB) assessment tool and
the CONSORT reporting guideline for RCTs. Results: Reporting of
components of the CONSORT guidelines (including, but not limited
to, sample size calculation, blinding, baseline characteristics, and
sequence generation) are woefully inadequate. Med Ed RCTs were
predominately rated to be of ‘unclear’ risk of bias. Conclusions:
The Med Ed discipline should strongly consider using CONSORT for
improving reporting of RCTs. A proposal for developing an extension
for RCTs of education for consideration by the CONSORT consortium
is underway. Considerations for risks of bias should be made when
interpreting Med Ed RCTs.
P2B208
Are overviews of (systematic) reviews really of good
quality and transparent information?
Li L1 , Yang K2 , Han D3 , Tian J2 , Sun T4
1 Evidence Based Medicine Center of Lanzhou University, School of Basic
Medical Science of Lanzhou University, Lanzhou 730000, China; The first
Clinical Medical College of Lanzhou University, Lanzhou 730000, China;
2 Evidence Based Medicine Center of Lanzhou University, School of Basic
Medical Science of Lanzhou University, Lanzhou 730000, China; 3 The
second Clinical Medical College of Lanzhou University, Lanzhou 730000,
China; 4 Shanghai Institute of Digestive Disease, Shanghai Jiao-Tong
University School of Medicine Ren-ji Hospital, Shanghai 200001, China
Background: Overviews of (systematic) reviews, or umbrella reviews,
are designed to compile evidence from multiple systematic reviews of
interventions into one document and provide the reader with a quick
overview relevant to a specific decision. Objectives: To evaluate
reporting and methodological qualities of overviews of reviews.
Methods: Searches were conducted using (‘overview’ AND (‘meta
analys*’ OR ‘systematic review*’)) OR ‘umbrella review’ in title/abstract
in PubMed, the Cochrane library, EMBASE, ISI Web of Knowledge. All
searches were conducted in February 2010, and updated in September
2010. We included those overviews that only included systematic
reviews or meta-analyses. We developed an 18-item assessment tool
to assess the methodological and reporting qualities of overviews of
systematic reviews. Two independent reviewers assessed the qualities
with this checklist, and resolved differences with a third reviewer.
Results: We found 41 overviews of systematic reviews, whose mean
total reporting and methodological score was 10.78 (SD 3.84) of
18 items and 3.05 (SD 2.09) of 8 items. Some necessary items
for the reporting and methodological qualities in the overview were
not reported, for example 69% mentioned inclusion and exclusion
criteria, 76% mentioned information sources, 49% mentioned review
selection, 44% mentioned data collection, 7% mentioned reporting
quality assessment, 46% mentioned methodological quality
assessment, and 20% mentioned quality of evidence assessment.
Conclusions: The reporting and methodological qualities of overviews
of systematic reviews were very poor, and there is still much room for
improvement.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
123
P2B209
What are the reporting and methodological qualities
of network meta-analysis?
Li L1 , Sun T2 , Han D3 , Yang K4 , Tian J4
1 Evidence Based Medicine Center of Lanzhou University, School of Basic
Medical Science of Lanzhou University, Lanzhou 730000, China; The first
Clinical Medical College of Lanzhou University, Lanzhou 730000, China;
2 Shanghai Institute of Digestive Disease, Shanghai Jiao-Tong University
School of Medicine Ren-ji Hospital, Shanghai 200001, China; 3 The second
Clinical Medical College of Lanzhou University, Lanzhou 730000, China;
4 Evidence Based Medicine Center of Lanzhou University, School of Basic
Medical Science of Lanzhou University, Lanzhou 730000, China
Background: Network meta-analysis can combine information from
all randomized controlled trials when many treatment regimens already
exist, in order to rank the regimens to determine the best option for
patients. But is is not clear what are the reporting and methodological
qualities of network meta-analysis. Objectives: To evaluate the
reporting and methodological qualities of network meta-analyses.
Methods: We searched Pubmed, the Cochrane library, Embase,
ISI Web of Knowledge, and Google Scholar using ‘Network meta
analysis’ OR ‘mixed treatment comparisons meta analyses’ OR ‘multiple
treatments meta analysis’ in title/abstract without language, time, or
publication type restrictions up to 2010 October 21 and updated up to
2011 February 21. We included any network meta-analysis, regardless
of the conditions or interventions. We evaluated the reporting
qualities of network meta-analysis using the PRISMA statement and
methodological quality was assessed using OQAQ as a network
meta-analysis was considered as a meta-analysis. Two trained authors
independently selected studies, abstracted data, and assessed the
qualities, and disagreement was resolved by a third author. Results:
We found 37 network-meta analyses. Thirteen items were reported in
less than 75% articles, in which seven items were reported in less than
50%. Fifty-one percent tried to avoid bias in the selection, and 54%
reported criteria used for assessing validity, but 24% were assessed
appropriately. Conclusions: Some necessary methodological and
reporting quality items were not performed or reported adequately, so
there is much room to improve.
P2B210
Epidemiology and quality of reporting of conference
abstracts related to randomized controlled trials in renal
transplantation
Wang X1 , Jiang J2 , Li L3 , Tian J3 , Tian H3 , Ying X4
1 Evidence Based Medicine Center of Lanzhou University, School of Basic
Medical Science of Lanzhou University; The second Clinical Medical College
of Lanzhou University, Lanzhou 730000, China; 2 Evidence Based Medicine
Center of Lanzhou University, School of Basic Medical Science of Lanzhou
University, The second Clinical Medical College of Lanzhou University,
Lanzhou 730000, China; 3 Evidence Based Medicine Center of Lanzhou
University, School of Basic Medical Science of Lanzhou University, Lanzhou
730000, China; 4 The second Clinical Medical College of Lanzhou University,
Lanzhou 730000, China
Background: Transparent and accurate conference abstracts of
randomized controlled trials (RCTs) are important, especially when
they are used to make health-care decisions. However, the reporting
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
quality of conference abstracts related to RCTs in renal transplantation
is unknown. Objectives: To assess the reporting quality of conference
abstracts of RCTs in renal transplantation and to identify factors
associated with high quality. Methods: Web of science was searched
to identify RCTs of renal transplantation and limited to conference
proceedings. Their qualities were evaluated with a modified version
of CONSORT for reporting randomized trials in Journal and conference
abstracts. The percentage of trials that scored ‘yes’ on each quality item
and the 95% confidence interval (CI) was calculated. Differences of
the reporting quality of different items (region, center, published year)
were tested using Chi-square analysis expressed as odds ratio (OR) with
95% CI with Mantel-Haenszel method. Results: We included 112
RCTs. More than half of conference abstracts reported specific objective
(86%, 95% CI 91% to 77%), recruitment (55%, 95% CI 64% to 46%),
number of participants randomized (63%, 95% CI 71% 53%), primary
outcome (51%, 95% CI 60% to 42%), and general interpretation
of the results (90%, 95% CI 94% to 83%). All of the abstracts
reported the contact details for the corresponding author. Only 2%
of the abstracts provided details of blinding. No abstract explained
the methods of random sequence generation, allocation concealment,
source of funding or trial registration. Compared to abstracts not
reporting center, the quality of single-center (OR 1.93, 95% CI 1.25 to
2.97) and multi-center (OR 1.84, 95% CI 1.44 2.35) was significantly
higher. The reporting quality of European is significant lower than North
American (OR 0.68, 95% CI 0.50 to 0.93). But statistically significant
differences were not found between publication before and after 2008.
Conclusions: The reporting quality of conference abstracts of RCTs
in renal transplantation was suboptimal, especially for methodological
items where improvements are needed.
P2B211
The relationship between risk of bias and effect size:
randomized controlled trials
Dolores F1 , Héctor M1
1 University of Valencia Spain
Background: Randomized controlled trials (RCTs) are the gold
standard for trials assessing the effects of therapeutic interventions and
provide the best evidence to inform and guide clinical decision-making.
Therefore, it is important to understand how they are conducted and
how to evaluate the strength of evidence. Bias is a systematic tendency
to produce an outcome that differs from the underlying truth. Bias
in clinical trials falls into four categories: selection bias, performance
bias, detection bias and attrition bias. There are numerous tools to
assess methodological quality of primary studies. Understanding the
relationship between the risk of bias and the effect size can help plan
research to monitor the presence of such bias. Objectives: Our work
has three objectives. First, provide an analysis of the relationship
between bias and effect size. Second, analyze the bias and reporting
checklists (example CONSORT). Third, we review and analyze the
Cochrane Collaboration’s tool for assessing risk of bias. Methods:
Systematic review. Results: The methodological quality of studies can
have a substantial impact on estimates of treatment effect, which may
affect the validity of the conclusions. Conclusions: Our work helps to
identify areas of strength and weakness in the existing evidence and
to formulate recommendations to improve the conduct and value of
future research.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
124
P2B212
Definitions of intention-to-treat in the face of missing
outcome data: A systematic survey of methodology papers
Alshurafa M1 , Guyatt GH1 , Akl EA2 , Haines T1 , Gentles S1 , Rios L1 , Tran
C3 , Lamontagne F4 , Moayyedi P5 , Walter SD1 , Briel M6
1
Department of Clinical Epidemiology and Biostatistics, McMaster
University, Hamilton, Canada; 2 Departments of Medicine and Family
Medicine, State University of New York at Buffalo, Buffalo, USA; 3 Chau Tran
Consulting, Toronto, Canada; 4 Centre de recherche clinique Étienne-Le
Bel, Université de Sherbrooke, Sherbrooke, Canada; 5 Department of
Medicine, McMaster University, Hamilton, Canada; 6 Basel Institute for
Clinical Epidemiology and Biostatistics, University Hospital Basel, Basel,
Switzerland
Background: Intention-to-treat (ITT) has been established as a quality
label for randomized clinical trials. However, surveys of randomized
trials showed that trialists hold a variety of views on the relationship
between missing outcome data (MOD) and ITT and how MOD should
be handled under ITT. Objectives: We surveyed articles addressing the
methodology of ITT in relation to MOD. Methods: We systematically
searched MEDLINE up to December 2008 for methodological articles
written in English that devoted at least one paragraph to ITT and two
other paragraphs to either ITT or MOD/loss to follow-up. We excluded
original trial reports, observational studies, or clinical systematic
reviews. Working in teams of two, we independently extracted relevant
information from each eligible article. Discrepancies between data
extractors were resolved by consensus or by third party arbitration if
necessary. Results: Of 1007 titles and abstracts reviewed, 66 articles
met our eligibility criteria. Five (8%) did not provide a definition of
ITT; 25 (41%) mentioned MOD but did not discuss its relationship
with ITT; and 36 (59%) discussed the relationship of MOD with ITT.
These 36 articles described one or more of three distinct relationships:
full-follow-up is required for ITT (58%); ITT and MOD are separate
issues (17%); and ITT requires a specific strategy for handling MOD
(78%); 17 (47%) described more than one relationship. The most
frequently mentioned strategies for handling MOD were using the last
outcome carried forward (50%), sensitivity analysis (50%), and use
of available data to impute missing data (46%). Most articles (81%)
specifically excluded complete case analysis under ITT. Conclusions:
We found that there is no consensus on the definition of ITT in relation
to MOD. For conceptual clarity, we suggest considering deviations
from randomization separately from issues of MOD when reporting
randomized trials.
P2B213
Methodological and reporting quality of systematic
reviews on influenza
Jiang J1 , Wang X1 , Tian J2 , Lin Q1 , Li L1
1 Evidence Based Medicine Center of Lanzhou University, School of Basic
Medical Science of Lanzhou University; The second Clinical Medical College
of Lanzhou University, Lanzhou 730000, China; 2 Evidence Based Medicine
Center of Lanzhou University, School of Basic Medical Science of Lanzhou
University, Lanzhou 730000, China
Background: Influenza is becoming more and more threatening to
the world, and many systematic reviews about preventing and treating
influenza are conducted. But the methodological and reporting quality
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
of these systematic reviews (SRs) is unknown. Objectives: To
assess the methodological and reporting qualities of SRs on influenza
and compare these qualities of Cochrane systematic reviews (CSRs)
with non-cochrane systematic reviews (NCSRs). Methods: PubMed,
EMBASE, Web of science and the Cochrane Library were searched for
SRs on influenza using ‘systematic reviews’, ‘meta-analysis’, ‘influenza’,
‘flu’ in 2011 February. We assessed the methodology quality with
OQAQ and AMSTAR, and the reporting quality using PRISMA. The
result of ‘yes’ for AMSTAR or PRISMA was scored ‘1’, and other
conditions was marked ‘0’. The total scores of AMSTAR and PRISMA
for each SR were calculated. For OQAQ, 1, 3, 5 or 7 was marked for
each SR if there were extensive, major, minor or minimal flaws. The
data was expressed with mean ± SD and T-test was used to test the
differences of the qualities between CSRs and NCSRs by SPSS 17.0.
Results: In total, 28 SRs (15 CSRs and 13 NCRs) were identified. The
methodological quality score of all SRs using OQAQ and AMSTAR were
4.21 ± 0.99 and 7.86 ± 1.67, respectively. The reporting quality score
of all SRs using PRISMA was 20.18 ± 4.09. Compared with NCSRs,
CSRs showed better qualities in reporting by PRISMA (MD 5.67, 95%
CI 3.14 to 8.20) and in methodology by OQAQ (MD 1.25, 95% CI 0.67
to 1.83) and AMSTAR (MD 1.23, 95% CI 0.01 to 2.45). Conclusions:
The reporting and methodological qualities of SRs on influenza were
not very low, but they need to be improved. The methodological and
reporting qualities of CSRs were better than those of NCSRs; therefore,
the PRISMA and AMSTAR/OQAQ should be recommended for NCSRs
especially.
P2B214
Characteristics of randomized controlled trials (RCT)
published in Latin America and the Caribbean’s (LAC)
in 2010
Reveiz L1 , Glujovsky D2 , Pinzon CE3 , Asenjo-Lobos C4 , Cortes M5 ,
Canon M6 , Bardach A2 , Comandé D2 , Cardona AF7
1 Policy and Research, Health Systems Based on Primary Health Care.
Pan American Health Organization, WDC, USA; 2 Argentine Cochrane
Centre IECS, Institute for Clinical Effectiveness and Health Policy Buenos
Aires, Argentina; 3 Instituto de Investigaciones, Fundación Universitaria
Sanitas, Bogotá D.C., Colombia; 4 Centro Rehabilitación Oral Avanzada e
Implantologı́a (CRAI) Universidad de Concepción (Centro Adherido Chileno
de la Red Cochrane Iberoamericana) – Chile; 5 Chilean branch of the
Iberoamerican Cochrane Network, Universidad Católica de la Santı́sima
Concepción, Chile; 6 Facultad de Medicina, Fundación Universitaria Sanitas,
Bogotá D.C., Colombia; 7 Grupo Oncologı́a Clı́nica y Traslacional, Instituto
de Oncologı́a, Fundación Santa Fe de Bogotá, Bogotá D.C., Colombia
Introduction: Few studies have evaluated the characteristics of RCTs
conducted in LAC countries. Methods: A review of RCTs published in
2010 in which the author’s affiliation was from LAC was performed in
PUBMED-LILACS; the RCT was included if it explicitly used the word
‘random’ or variations thereof. Two reviewers independently extracted
key data and assessed the risk of bias (RoB). Results: We identified
1695 references including 526 RCTs (N = 73.513) recruiting 139.8
(SD = 284.53) participants on average. English was the dominant
publication language (93%); with Spanish (3.4%) and Portuguese
(2.9%) following. We found RCTs with affiliations representing 19
LAC countries; 70% were from Brazil. Maternal health represented
6.7% and neglected diseases-HIV-tuberculosis-malaria 3.8%. We
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
125
assessed the following characteristics in a sample of 326/526 RCTs.
Settings/Authorship: 7.7% were multinational and 12.0% multicenter.
10.6% included a co-author from a different country (the USA being
the most frequent). Participant’s sex: 75.3% of RCTs recruited men
and women; 19.6% exclusively women. Overall, 57.4% of participants
were women. Gender analysis was performed in 6.8% of studies.
19% were <18 years. Type of intervention: Drugs (47.3%) and
procedures (24.5%) were the most frequent. Follow-up: Short term
(<1 month) (38%) and 1-6 months 39% were the most frequent.
Risk of bias: Scores were low (25.2%), unclear (60.1%), and high
(14.7%). Ethic/Conflict of interest: 95.1% reported approval by ethics
review committee and 93.1% presented an informed consent process.
Conflicts of interest were declared by at least one author in 11.5%
of RCTs; 42.5% failed to provide a declaration. Funding: Public
33.9%; private (i.e. pharmaceutical) 14.4%; not reported 36.6%;
others (i.e. mixed-NGO) 15.1%. More frequent funders were FAPESP,
CAPES and CNPq from Brazil. Conclusion: There is plenty of room
for improvement in South–South collaboration and RCTs addressing
regional priorities. Methods and result reporting of RCTs should be
improved to meet international standards.
P2B215
Impact of the trial registration initiative in Latin America
and the Caribbean: a study of randomized controlled trials
published in 2010
Reveiz L1 , Bonfill X2 , Glujovsky D3 , Pinzon CE4 , Asenjo-Lobos C5 , Cortes
M6 , Canon M7 , Bardach A3 , Comandé D3 , Cardona AF8
1 Policy and Research, Health Systems Based on Primary Health Care, Pan
American Health Organization. WDC, USA; 2 Servicio de Epidemiologı́a
Clı́nica y Salud Pública, Centro Cochrane Iberoamericano, Hospital de la
Santa Creu i Sant Pau (UAB), Barcelona, España; 3 Argentine Cochrane
Centre IECS, Institute for Clinical Effectiveness and Health Policy Buenos
Aires. Argentina; 4 Instituto de Investigaciones, Fundación Universitaria
Sanitas. Centro Colaborador de la Colaboración Cochrane, Bogotá D.C.,
Colombia; 5 Centro Rehabilitación Oral Avanzada e Implantologı́a (CRAI)
Universidad de Concepción (Centro Adherido Chileno de la Red Cochrane
Iberoamericana) – Chile; 6 Facultad de Medicina, Centro Colaborador de la
Colaboración Cochrane, Universidad Católica de la Santı́sima Concepción.
Chile; 7 Facultad de Medicina, Fundación Universitaria Sanitas, Bogotá
D.C., Colombia; 8 Grupo Oncologı́a Clı́nica y Traslacional, Instituto de
Oncologı́a, Fundación Santa Fe de Bogotá, Bogotá D.C., Colombia
Introduction: Adherence to clinical trial registration in Latin American
and the Caribbean (LAC) countries is yet to be evaluated. Objective:
Determine the prevalence of registration of randomized controlled
trials (RCT) published in PUBMED-LILACS in 2010 from LAC in
the International Clinical Trial Registry Platform (ICTRP) compliant
databases; and to compare methodological characteristics of registered
vs. non-registered RCT. Methods: A search for detecting RCTs in
which at least the first author had a LAC’s affiliation was made. The
RCT was included if it explicitly used the word ‘random’ or variations
thereof. Trials not including human subjects were excluded. No
language restrictions were applied. We assessed the full text of the
reports to determine if the RCT had been registered in a register
contributing to the ICTRP. Data from studies were independently
extracted by two authors. We assessed the risk of bias (RoB) in all
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
registered RCTs (n = 89) and in a sample of non-registered RCTs
(n = 237). Results: The search identified 1695 references; we
included 526 RCTs from 19 countries of which 16.9% (89/526) were
registered in the ICTRP. Prospective registration was done in 21 (4.0%)
RCTs. We found a significant difference in the overall RoB assessment
between registered (5.1% high; 60.3% unclear; and 34.6% low risk)
and non-registered RCTs (high 18.5%; unclear 59.3%; and 22.2%
low risk); significant differences were also found in the proportion of
‘allocation concealment’ and ‘other types of bias’ domains of the RoB.
The number of RCTs reporting conflict of interests, approval by an ethics
committee and the provision of informed consent were significantly
higher in registered studies. No significant differences were found
when analyzing only prospectively registered vs. non-registered RCTs.
Conclusion: There is a significant association between clinical trial
registration with lower RoB and with a better quality assessment. There
is room for improvement in trial registration adherence.
P2B216
Does the sensitivity of tests for Down Syndrome really
decrease with maternal age? A case-study illustrating the
potential impact of loss to follow up on review conclusions
Pennant M1 , Guo B1 , Alldred K2 , Neilson J2 , Alfreivic Z2 , Deeks J1
1 The University of Birmingham, UK; 2 The University of Liverpool, UK
Background: In a Cochrane Diagnosis Test Accuracy review of
maternal screening for Down’s Syndrome, we observed that the
estimated sensitivity of the 2nd trimester serology based triple test
appeared to be 20% lower in studies of women aged >35 compared
to studies unrestricted by age. Objectives: To investigate whether the
study design, and associated risk of loss to follow up, could explain
the observed relationship. Methods: Methodological characteristics
of the included studies were evaluated and considered. Results
were plotted in summary ROC space identifying studies according to
their methodological characteristics as well as maternal age. Bivariate
random effects meta-regression models were used to estimate the effect
of study characteristics on test performance. Sensitivity analyses were
used to investigate whether differential loss to follow-up could explain
the observed relationship with age. Results: Two distinct groups of
studies were evident: those in which the reference standard diagnosis
involved follow-up to birth (risking loss to follow-up) and those where
all patients received invasive confirmatory testing (minimising loss to
follow-up). Only studies of older mothers used the latter design.
Comparison of test accuracy between these groups for the largest
dataset showed, for a false positive rate of 5%, significantly higher
sensitivity (68.6%; 95% CI 62.3–74.3%, n = 11 versus 48.4%, 95%
CI 40.7–56.2%, n = 13, p < 0.0001) for follow up compared to
confirmation studies. Sensitivity analyses showed that the observed
relationship with maternal age could partially (but not totally) be
explained by differential loss-to follow-up in studies undertaken using
follow-up to birth. Conclusions: The apparent increase in sensitivity
in follow up compared to confirmation studies may partially be an
artefact of differential loss to follow-up, where test negative women
who had Down’s syndrome fetuses (false negatives) are more likely to
miscarry. In confirmation studies, all false negatives were likely to be
identified, giving more reliable estimates of test accuracy.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
126
P2B217
Split-Mouth and crossover trials in dentistry are neither
well differentiated, nor well analyzed or reported
Brignardello-Petersen R1 , Carrasco-Labra A2
1 Evidence Based Dentistry Unit, Faculty of Dentistry, Universidad de
Chile, Chile, Department of Health Policy, Management & Evaluation,
University of Toronto, Canada; 2 Evidence Based Dentistry Unit, Faculty of
Dentistry, Universidad de Chile, Chile, Department of Clinical Epidemiology
& Biostatistics, McMaster University, Canada
Background: Crossover and split-mouth randomized controlled trials
(RCTs) are designs commonly used in dentistry. Although their
statistical analysis should be similar, they have different fundaments
and are indicated when the population and interventions under study
have specific characteristics. However, it seems that many researchers
are not aware of the methodological differences between them and
tend to describe split-mouth trials as crossover, failing to accomplish
crucial methodological requirements of the former design. Objectives:
To determine if the requirements of the design and analysis of
crossover and split-mouth RCTs are being considered in the field of
oral and maxillofacial surgery (OMFS). Methods: Systematic Review.
A comprehensive searching in MEDLINE was used to retrieve all the
split-mouth and crossover trials published in the four journals of OMFS
of highest impact factor. RCTs classified by the authors as split-mouth
or crossover, or if the methods section described the methodology of
one of these designs were selected. Two evaluators extracted the data
independently. Results: A total of 152 articles were retrieved from the
electronic searching. The final sample was composed of 41 articles,
from which 82.93% were classified as crossover by the authors. Of these
trials, only 6% had a crossover design. None of the trials considered the
possible carry-over or carry-across effect that could bias the results. Of
the whole sample, 43.9% analyzed the results without considering the
paired nature of the data. The presence of a statistician, the journal,
and the year of publication showed no statistical association with the
reporting of a proper analysis (p > 0.05). Conclusions: In general,
authors of RCTs in OMFS do not recognize the differences between
crossover and split-mouth trials. There is room for improvement in the
particular aspects of these designs.
P2B218
Are there discipline-specific quality measures?
A systematic review of meta-epidemiological studies
Jacobs W1 , Kruyt M2 , Moojen W1 , Verbout A2 , Oner C2
1 Department of Neurosurgery, Leiden University Medical Centre (LUMC),
Leiden, Netherlands; 2 Department of Orthopedic Surgery, University
Medical Center Utrecht (UMCU), Utrecht, Netherlands
Background: Several studies with a meta-epidemiological approach
have been published on effect of quality measures in clinical research.
The results from these studies show some variations. Objectives:
The goal of this systematic review of meta-epidemiological studies
was to systematically search and evaluate the evidence for effect
of quality measures in clinical research and to determine whether
differences in effect of quality measures could be attributed to type
of intervention. Methods: Systematic search of Medline, EMBASE
Web of Science, Cochrane methodology register, and reference lists
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
for meta-epidemiological studies with estimates of effect of quality
measures. References were selected by predefined criteria and
appraised with the Amstar tool adjusted for the purpose. Selection and
quality evaluation were performed in duplicate. We anticipated, but
were not limited to, effects of study size, study type, randomization,
concealment, blinding (care-provider, patient and observer), and
sponsoring. We made a distinction between surgical, pharmaceutical,
and treatment programs (psycho-therapeutical, physio-therapeutical)
as separate types of interventions. Heterogeneity was calculated
to identify differences between types of intervention. Results: We
found 23 meta-epidemiological studies from which we were able to
extract 50 effect estimates of 11 different quality measures. Eleven of
these estimates were statistically significant. All of these confirmed the
common assumptions of smaller effect estimates in studies with ‘better’
quality, defined as larger studies, use of randomised allocation, proper
allocation concealment, and double blinding. The only exception is a
surgical study that found an increased effect estimate in randomised
studies. Conclusions: Study size and allocation concealment were
more consistently found to have an influence on effect size than other
quality measures. For randomization and allocation concealment there
are indications that the effect size might be different between different
types of interventions.
P2B219
Epidemiology and publication history of randomized
controlled trials that were discontinued
Blümle A1 , Kasenda B2 , Saccilotto R2 , Briel M2 , von Elm E3
1 German Cochrane Centre, Department of Medical Biometry and Statistics,
University Medical Center Freiburg, Freiburg, Germany; 2 Basel Institute
for Clinical Epidemiology and Biostatistics, University Hospital Basel, Basel
Switzerland; 3 Cochrane Switzerland, Institute of Social and Preventive
Medicine, CHUV and University of Lausanne, Switzerland
Background: Little is known about the epidemiology and publication
history of randomized controlled trials (RCTs) discontinued due to
futility, harm, or insufficient recruitment. We hypothesized that the
data already collected are often not made available to the scientific
community. The lack of reporting potentially enhances the problem
of wasted research efforts and raises ethical concerns, because trial
participants consent on the premise of contributing to the advancement
Table 1 (P2B219). Reasons for discontinuation in 64 RCTs.
Reasons for discontinuation
N (%)
Slow recruitment
Futility
Harm
Funding problems
Others∗
Total
26 (41)
8 (12)
5 (8)
9 (14)
16 (25)
64 (100)
∗ This
category includes the following: parallel study in USA showed
no effect (N = 1), unknown (N = 2), not clearly specified (N = 5),
protocol violation (N = 1), loss of study data due to technical
defect (N = 1), closing of liver transplant program (N = 1), change
in researchers’ career (N = 5).
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
127
of medical knowledge. It may also lead to bias in meta-analyses if
relevant data from discontinued trials are not considered. Objectives:
To describe the epidemiology and publication history of discontinued
RCTs. Methods: We included protocols of RCTs approved by the
research ethics committee of the University of Freiburg (Germany)
between 2000 and 2002. We searched electronic databases and
surveyed trialists for information about project/publication status. In
case of discontinuation, further information about the reasons was
collected. Results: Of 418 approved RCTs, 194 (46%) were published
in full, and 224 (54%) were not. Overall, 300 (72%) RCTs were either
completed according to protocol, ongoing, or not started at all; for 54
(13%) we received no survey response, and 64 (15%) were definitely
discontinued. Of the latter, 8 (13%) were published and 56 (87%) were
not. The reasons for discontinuation were slow recruitment (41%),
futility (12%), harm (8%), funding problems (14%), and others (25%)
(Table 1). Conclusions: About half of all approved RCTs were not
published in full and at least 15% were discontinued. Most of the data
collected until discontinuation were not published and thus not easily
available for meta-analyses. Slow recruitment was the most frequent
reason for discontinuation. Our findings emphasize the necessity for a
screening tool to identify trials at risk of being discontinued because
of recruitment problems. This could help reduce the waste of scarce
resources in clinical research.
P2B220
Sponsors’ influence on reporting of results
in industry-sponsored trials: a comparison of protocols
and published papers
Lundh A1 , Krogsbøll LT1 , Gøtzsche PC1
1 The Nordic Cochrane Centre, Denmark
Background: Bias in industry-sponsored trials is common and
interpretation of results may be distorted in favour of the sponsors’
products. Therefore many industry-sponsored trials are coordinated by
seemingly independent steering committees. However, this may not
prevent the sponsors’ influence on reporting of results. Objectives:
To investigate the sponsors’ influence on reporting of results in
industry-sponsored trials. Methods: We included a sample of
industry-sponsored trials published in The Lancet in 2008–2009 and
corresponding trial protocols provided by the editors. For each protocol
and published paper, we extracted information on coordination of
publication, sponsors’ publication rights and involvement of medical
writing assistance. We compared the information from protocols with
the published papers. Results: We identified 169 papers of randomised
trials and included 69 (41%) industry-sponsored trials. An additional
12 (7%) fully industry-funded trials with trial conduct apparently
independent of the sponsor were analysed separately. In 24 out of the
69 trials (35%) the sponsor or a hired contract research organization
were involved in coordinating writing of the manuscript, in 10 (14%)
the sponsor was not involved and in 35 (51%) it was not described.
In 64 trials (93%), the sponsor had influence over publication of the
results through co-authorship or an explicitly stated right to approve or
comment on the paper, and 37 trials (54%) had evidence of medical
writing assistance supported by the sponsor. For the 12 apparently
independently conducted trials 4 described the sponsors’ involvement
in reporting of results, without this being stated in the published
papers. Conclusions: The sponsors are usually involved in reporting
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
of results in industry-sponsored trials. Even for fully industry-funded
trials that appear to be conducted independently, the sponsors are also
sometimes explicitly involved in the reporting of results.
P2B221
Critical appraisal checklist for systematic reviews
of clinical prediction models
Moons KG1 , Bouwmeester W1 , Collins G2 , Mallet S2 , Altman D2 ,
Reitsma J1
1
Julius Center, UMC Utrecht, Netherlands; 2 Centre for Statistics in
Medicine, Oxford, UK
Background: The introduction of evidence based medicine resulted
in a clear shift from implicit to explicit reasoning in medicine, including
the appreciation of multivariable diagnostic and prognostic prediction
models. This is reflected by a sharp increase in published clinical
prediction models. Systematic reviews try to assess and summarize the
evidence. There is yet no tool to appraise clinical prediction studies.
Objective: To provide a comprehensive list of items that are relevant
for systematically reviewing and critically appraising publications on
clinical prediction models. Methods: To indentify the relevant
items for critically appraising clinical prediction research, we studied
existing reporting guidelines (including in other medical research areas),
various quality assessment tools, systematic reviews of prediction
research, and methodologic key publications. Subsequently, experts
were consulted for additional items. Results: Items that are most
important for systematically reviewing publications on clinical prediction
research include study design, subject selection methods, assessment,
definition, and coding of outcomes and candidate predictors, statistical
power, statistical techniques used, the reporting and handling of
missing values, predictor selection approaches, assessment of predictive
performance and validation of the final model, and model presentation.
Conclusions: This overview of items can assist systematic reviewers
in the appraisal of clinical prediction studies.
P2B222
Comparison of adverse effects data derived from different
study designs
Golder S1 , Loke Y1 , Bland M2
1
Adverse Effects Methods Group, UK; 2 Department of Health Sciences,
University of York, UK
Background: There is considerable debate as to the relative merits
of using randomised controlled trial (RCT) data compared to data
from observational studies in systematic reviews of adverse effects.
Objectives: This meta-analysis of meta-analyses aimed to assess the
level of agreement or disagreement in the estimates of harm derived
from meta-analysis of different study designs. Methods: Searches
were carried out in 10 databases in addition to reference checking;
contacting experts; citation searching; and handsearching key journals,
conference proceedings and websites. Studies were included where
a pooled relative measure of an adverse effect (odds ratio or risk
ratio) from one study design could be directly compared, using the
ratio of odds ratios (RORs), with the pooled estimate for the same
adverse effect arising from another study design. Results: Thirty-nine
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
128
studies, yielding 111 meta-analyses were identified for inclusion. The
pooled ratio of odds ratios of RCTs compared to observational studies
was estimated to be 1.03 (95% CI 0.93–1.15) and there was less
discrepancy with larger studies. Other meta-analysis of meta-analyses
of different types of observational studies (such as cohort studies
and case-control studies) also indicated no significant difference in
estimates of adverse effects derived from different study designs.
In almost all instances, the estimates of harm from meta-analyses
of the different study designs had 95% confidence intervals that
overlapped. In terms of statistical significance, in nearly two-thirds
of the meta-analyses the results were in agreement (both studies
showing a significant increase or significant decrease or no significant
difference). In only two meta-analyses was there opposing statistical
significance. Conclusions: Empirical evidence from this overview
indicates that, on average, there is no difference in the risk estimate of
adverse effects derived from meta-analyses of different study designs.
This suggests that systematic reviews of adverse effects need not be
restricted to specific study designs.
P2B223
How do systematic reviews of insulin and drug therapies
for diabetes deal with heterogeneity of results?
Sun X1 , Troung J2 , Dattani N3 , Wong E2 , Kapend P2 , Bianca H4 ,
Sherifali D2 , Wang L5 , Nereberg K2 , Guyatt G6
1
Center for Health Research Kaiser Permanente Northwest, USA;
2 McMaster University, Canada; 3 University of Toronto, Canada;
4 Copenhagen University Hospital, Denmark; 5 Chinese Evidence-based
Medicine Center, Sichuan University, China; 6 Department of Clinical
Epidemiology and Biostatistics, McMaster University, Canada
Background: The problem of how to deal with heterogeneity of
results is ubiquitous in systematic reviews. Appropriate approaches
include the use of a small number of pre-specified hypotheses, and
the interaction test. Objective: To determine the rigor with which
recent systematic reviews of therapy for diabetes explore heterogeneity.
Methods: We searched Medline, EMBASE, and Cochrane Database of
Systematic Reviews for systematic reviews, published in any language
from January 2008 to May 2010, that, first, assessed effects of insulin
and anti-diabetes drug therapies, both in single use or combination, for
any type of diabetes mellitus; and, second, reported a meta-analysis of
two or more randomized controlled trials (RCTs). Two methods trained
reviewers, independently and in duplicate, conducted study screening
and data extraction. Results: Of 46 reviews that proved eligible, 29
(59%) explored heterogeneity by subgroup analyses (n = 27, 59%)
and/or meta-regression (n = 5, 11%), and 21 out of 29 reviews (72%)
reported heterogeneity p < 0.1 or I-square ≥ 50%. Of the 29 reviews
exploring heterogeneity, 6 (20%) clearly pre-specified heterogeneity
variables, and 9 (31%) clearly used the interaction test. Explorations
of heterogeneity tests a median of 5 (Inter-quartile: 2-6) variables.
Variables explored included patient characteristics (17 of 29, 59%),
intervention characteristics (24 of 29, 83%), and applied risk of bias
and study characteristics (e.g. duration of follow up) (11 of 29, 38%).
Many of these characteristics were also used as factors for sensitivity
analyses (data not shown). Two of 5 studies including meta-regression,
each having 14 and 22 RCTs, also undertook multivariable analyses of 4
independent variables. Conclusions: Exploration of heterogeneity was
poorly performed in systematic reviews of insulin and drug therapies for
diabetes. In particular, systematic reviewers often used patient baseline
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
characteristics for exploring heterogeneity, which may be problematic.
Systematic reviewers also did not well differentiate subgroup analyses
and sensitivity analyses.
P2B224
Sponsorship of drug and device studies and research
outcome
Lundh A1 , Lexchin J2 , Sismondo S3 , Busuioc OA3 , Bero L4
1 The Nordic Cochrane Centre, Denmark; 2 School of Health Policy and
Management, York University, Canada; 3 Department of Philosophy,
Queen’s University, Canada; 4 Institute for Health Policy Studies &
Department of Clinical Pharmacy, University of California-San Francisco,
USA
Background: Previous systematic reviews have documented that
funding of drug studies by drug companies is associated with findings
favourable towards the sponsor’s product. However, meta-analyses
of drug studies frequently fail to disclose the underlying funding of
included studies or to investigate whether the source of funding
influences results. To determine if funding bias should be taken into
account in meta-analyses, it is important to know its extent. This
study updates previous reviews showing the association of funding
and research outcome to determine if the association persists and
additionally investigates device studies. Objectives: To investigate
whether funding of drug or device studies by drug or device companies
is associated with results and conclusions favourable to the sponsor.
To investigate whether studies funded by drug or device companies
differ in their risk of bias compared with studies with other sources
of funding. Methods: We searched major bibliographic databases,
reference lists, previous systematic reviews and author files. Two
reviewers included studies and final inclusion was by consensus of
all authors. Two reviewers extracted data from included papers and
assessed studies for risk of bias. Authors were contacted in order
to obtain missing data. Data was analysed in RevMan. Results:
We identified 2,579 studies in the search, 72 of which appeared to
meet our inclusion criteria and were retained for final assessment by
all authors. Sixty-three were published after the search period of our
previous systematic review. Additional results will be presented at the
Colloquium. Conclusions: It appears that the evidence of funding
bias has grown, but our conclusion awaits results from data analysis.
P2B225
Adding unpublished Food and Drug Administration (FDA)
data changes the results of meta-analyses
Hart BL1 , Lundh A2 , Bero LA1
1
University of California, San Francisco; US Cochrane Center, San Francisco;
2 Nordic Cochrane Center, Denmark
Background: Studies examining selective reporting of drug trial data
show that positive results are more likely to be published than null or
negative results. Consequently, meta-analyses based only on published
data may over-estimate drug efficacy and under-estimate drug harms.
Food and Drug Administration (FDA) reviews of drug trials are publicly
available, making them a good source of unpublished trial data for
inclusion into meta-analyses. Objective: This study examines the effect
of including unpublished FDA trial data on the results of published
meta-analyses across a variety of drug classes. Methods: We studied
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
129
24 drugs approved by the FDA between 2001–2002 with previously
identified unpublished outcome data. We performed a systematic
search of PubMed, Embase, and the Cochrane library in November
2010 to identify relevant meta-analyses for each of the study drugs.
Two authors independently screened meta-analyses and selected one
for analysis when multiple meta-analyses for a drug were included.
Two authors independently extracted data from meta-analyses and
the FDA reviews. We calculated summary statistics both with and
without unpublished trial data. Results: After excluding 11 drugs
in which no relevant meta-analyses were identified, 13 drugs and
42 meta-analyses met our inclusion criteria. Each drug had 1 to
8 corresponding meta-analyses (median = 2). To date, we have
completed analyses for 4 drugs and recalculated summary statistics
for 15 outcomes. For 11 outcomes, the change in summary statistics
after inclusion of unpublished data demonstrated less benefit or more
harm of the drug (range in percent change of summary statistics:
4% – 41%). Four outcomes showed more benefit of the drug after
inclusion of unpublished data (range in percent change of summary
statistics: 4% to 9%). Discussion: This preliminary analysis suggests
that the inclusion of unpublished trial data in meta-analyses affects their
results. The magnitude and direction of the effect varies between drugs.
P2B226
Overestimation of treatment benefits in single-centre
osteoarthritis trials may be related to sample size:
Meta-epidemiological study
Nüesch E1 , Trelle S1 , Reichenbach S1 , Jüni P1
1 University of Bern, Switzerland
Background: Single-centre trials might differ from multicentre trials
in characteristics of included patients, implemented interventions,
methodological quality and sample size, which may result in different
estimated treatment benefits. Objectives: We examined whether
estimated treatment benefits differ between single-centre and
multicentre osteoarthritis trials and whether this can be explained
by components of methodological quality and sample size. Methods:
We performed a meta-epidemiological study of 13 meta-analyses with
154 trials that compared therapeutic interventions with placebo or
non-intervention control in patients with hip or knee osteoarthritis. We
calculated standardised mean differences (SMDs) from the differences
in means of pain intensity between groups at the end of follow-up
divided by the pooled SD and compared SMDs between trials with
and without multiple centres. We used stratification by components of
methodological quality and sample size to derive differences between
single and multicenter trials adjusted for these characteristics. Results:
The 80 trials with multiple centres did not differ clearly from the 74
trials without multiple centres in their methodological quality, but
randomised more patients (P = 0.027). On average, multicentre
trials showed less beneficial effects compared to single-centre trials
(difference in SMDs −0.20, 95% CI −0.38 to −0.02, P = 0.028) with
small to moderate heterogeneity between meta-analyses (tau2 = 0.05,
P for heterogeneity 0.002). Differences in treatment benefits were
robust when analyses were adjusted for adequate generation of
random sequence (−0.17, 95% CI −0.36 to 0.02, P = 0.07), adequate
allocation concealment (−0.17, −0.33 to −0.01, P = 0.034) and
blinding of patients (−0.19, −0.34 to −0.02, P = 0.029); were
attenuated when adjusted for intention-to-treat analyses (−0.13,
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
−0.27 to 0.01, P = 0.08); and disappeared when adjusted for sample
size (−0.07, 95% CI −0.27 to 0.14, P = 0.51). Conclusions: There is
an overestimation of treatment benefits in single-centre osteoarthritis
trials. This overestimation is not evident after adjustment for sample
size.
P2B227
Assessing the quality of non-randomised comparative
studies: Our experience of using the Cochrane
Collaboration’s risk of bias tool
Robertson C1 , Ramsay C1 , Gurung T1 , Mowatt G1 , Pickard R2 , Sharma P1
1
Health Services Research Unit, University of Aberdeen, UK; 2 Institutue of
Cellular Medicine, Newcastle University, UK
Background: We describe our experience of using a modified version
of the risk of bias (RoB) tool for non-randomised comparative
studies. Objectives: – To assess inter-rater agreement for RoB
assessment; – To assess time to complete RoB assessment; – To explore
the association between RoB and treatment effect size. Methods:
Our sample comprised full text primary reports included in a systematic
review comparing laparoscopic versus robotic prostatectomy for
localised prostate cancer (n = 49). We excluded non-English language
publications and conference abstracts. Three teams of two reviewers
assessed individual categories as high, low or unclear RoB. Joint
agreement was then reached for overall RoB. We explored differences
in treatment effect size according to RoB for the outcome positive
surgical margin (n = 9). Inter-rater agreement was assessed using
the Kappa statistic. We weighted the analysis to reflect higher
disagreement between high and low risk compared to high/low and
unclear judgements. We timed individual RoB assessment and time
to reach joint agreement. Results: Twenty five studies were judged
as having high overall RoB, 13 were judged as low RoB and 11 were
judged unclear. Standard and weighted Kappa values for inter-rater
agreement were 0.34 and 0.35 respectively. The median (range)
time for individual assessment was 30 minutes (10 to 49 minutes).
The median time for reaching agreement between reviewers was 10
minutes (2 to 38 minutes). The effect estimate for all studies was 0.61
(95% CI 0.46 to 0.83). The effect estimate for low risk studies was 0.73
(95% CI 0.29 to 1.75). Conclusions: We achieved fair agreement
between reviewers for RoB assessment of non-randomised studies.
Although the process was time consuming, using a modified version of
the Cochrane (RoB) tool proved useful for demonstrating conservative
effect estimates in our systematic review. We suggest including more
RoB levels and further validation could improve inter-rater agreement.
P2B228
A meta-epidemiological approach for evaluating bias and
small study effects in networks of interventions
Chaimani A1 , Schmid C2 , Vasiliadis H3 , Salanti G1
1 Department of Hygiene and Epidemiology, School of Medicine, University
of Ioannina, Greece; 2 Sackler School of Graduate Biomedical Sciences,
Tufts University School of Medicine, Boston, MA, US; 3 Molecular Cell
Biology and Regenerative Medicine, Sahlgrenska Academy, University of
Gothenburg, Sweden
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
130
Background: Investigation of the impact of study-specific biases
typically requires a large number of studies. Meta-epidemiological
approaches consider collections of independent pairwise
meta-analyses, where the impact of bias in each one of them might
be different, depending on the clinical area. Network meta-analyses
include pairwise meta-analyses assumed ‘exchangeable’ in everything
but the comparison being made. Hence biases can be studied
simultaneously by borrowing strength across comparisons even when
each pairwise meta-analysis comprises a small number of trials.
Objectives: To suggest a meta-epidemiological technique for
exploring study-specific bias sources and small-study effects using
a collection of published networks of interventions. Methods: We
searched for and collected all star-shaped networks of interventions
published until March 2011. We applied various multiple-treatment
meta-regression models that used as covariate a) the probability that
a trial is at risk of bias in terms of adequacy of sequence generation,
allocation concealment and blinding b) a measure of precision or
variance to explore small study effects. We assume exchangeable
bias parameters across comparisons within each network and across
networks for similar outcomes. The fit of the different models and
the amount of heterogeneity explained were compared. An average
effect per bias source was estimated. Results: The adjustment
for small study effects or bias items showed in some networks a
slight improvement in the fit of the model and a respective reduction
in heterogeneity. Borrowing strength across networks showed that
a) small study effects exhibit an important amount of heterogeneity
across networks b) bias sources overestimate the effectiveness of the
interventions compared to common comparator, in agreement with
previous findings. Conclusions: This meta-epidemiological approach
enables investigation of bias and heterogeneity while adjusting for
differences in effectiveness between treatments. As networks of
interventions become popular, large-scale investigation of bias should
consider them as source of evidence.
P2B229
First results of a Systematic Review of Meta-analyses on
the Efficacy of SSRI and SNRI in the Treatment of Major
Depression
Koesters M1 , Zhang Y1 , Weinmann S2 , Cipriani A3 , Barbui C3 , Becker T1
1 Department of Psychiatry II, Ulm University, Germany; 2 Deutsche
Gesellschaft fuer Internationale Zusammenarbeit, Germany; 3 Department
of Public Health and Community Medicine, Section of Psychiatry and
Clinical Psychology, University of Verona
Background: Systematic reviews and meta-analyses are the gold
standard to assess efficacy and acceptability of treatments. In the
field of antidepressant trials, hundreds of studies are available and
many systematic reviews have been carried out so far, sometimes
with conflicting results. It is known that the quality of the reviews
may play an important role. However, this issue has not been yet
systematically investigated. Objectives: To examine the quality of
meta-analyses of serotonin reuptake inhibitor treatment of depression
and to shed light on the influence of methodological aspects on the
results of these meta-analyses. Methods: A comprehensive literature
search for systematic reviews including meta-analysis of RCTs on SSRI
and SNRI treatment in major depression was conducted. Studies
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
comparing SSRI and SNRI to placebo or all kinds of pharmacological
antidepressant treatments were included. Results were screened for
inclusion or exclusion by two independent investigators. Relevant data
was extracted from the included studies, and quality assessment of
systematic reviews was performed. Influence of a variety of factors
on the results of reviews was analysed by regression analyses and
t-tests. Results: The analyses included 134 reviews. Depending on
the rating scale, the average quality of included studies was moderate
or good, with a significant increase of study quality over the past
20 years. Blinding of reviews regarding authors and results had
no impact on quality ratings. The results pointed out that there is
room for improvement of current quality measurements for systematic
reviews. Only involvement of drug companies in the publication of
meta-analyses showed an impact on the effects of the review. All other
factors tested failed to reveal statistically significant relationships with
the results of the review.
P2B230
What can we learn from Chinese randomized controlled
trials? A systematic review of Chinese venlafaxine studies
Koesters M1 , Zhang Y1 , Ma YC2 , Weinmann S3 , Becker T1 , Jin WD2
1
Department of Psychiatry II, Ulm University, Germany; 2 Tongde Hospital,
Zhejiang Province, China; 3 Deutsche Gesellschaft fuer Internationale
Zusammenarbeit, Germany
Background: Despite an impressive increase of Chinese studies the
interest in these studies is still relatively low in Western countries,
and Chinese randomised controlled trials (RCTs) are rarely included in
Western systematic reviews. Objectives: We systematically reviewed
blinded Chinese venlafaxine RCTs, based on a search in Chinese
and western databases. This study aims at enhancing insight into
the quality of Chinese RCTs and to investigate if venlafaxine is an
effective treatment option in Chinese populations. Methods: Chinese
databases (CNKI/VIP) and western databases were searched for blinded
randomized controlled trials. Trials comparing venlafaxine to other
antidepressants or placebo were included if the patients had a diagnosis
of depression according to CCMD, DSM or ICD. Effect sizes were
calculated as Hedges’ g for rating scale scores and Mantel-Haenszel
risk ratios (MH RR) for response and remission data. Effect sizes were
combined in a fixed-effects model. Results: Twenty-five studies were
included. Nine trials compared venlafaxine to SSRIs; placebo-controlled
trials were lacking. Quality was at best modest. All trials were
underpowered. There were more responders (MH RR, 1.08; 95%
confidence interval [CI], 1.02–1.15) and remitters (MH RR, 1.12; 95%
CI, 1.02–1.24) in venlafaxine groups compared with those in TCA
groups. HAMD end point scores in the venlafaxine groups were lower
(Hedges’ g = 0.16; 95% CI, 0.04–0.27), and venlafaxine was better
tolerated than tricyclic antidepressant (Hedges’ g = 0.56; 95% CI,
0.37–0.74). There were no significant differences between venlafaxine
and selective serotonin reuptake inhibitor on any of these parameters.
Analyses of publication bias were inconclusive. Conclusions: Chinese
researchers have published a number of randomized controlled trials
comparing venlafaxine to active comparators, but study quality was
found to be low. To make optimal use of their research potential
Chinese researchers will have to improve trial reporting and the
peer-review process.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
131
P2B231
Industrial funding of research is associated with practice
of survival analysis
Vlassov V1 , Gemjian E2
1 First Moscow State Medical University, Russia; 2 National Haemathology
Center, Moscow, Russia
Background: Industrial funding of the biomedical studies is associated
with practices which increase the possibility of bias, including subgroup
and per protocol analyses, and testing of multiple hypotheses or multiple
variables. Objective: To find out if industrial funding associated with
the use of survival analysis (SA) to estimate the cause-specific survival
(CSS). Methods: We searched MEDLINE by ‘survival analysis’. CSS
studies were found by ‘specific survival’. Only human-related studies
were selected. To evaluate the source of financial support, the full texts
of the articles were checked. The primary analysis was limited to year
2009. Results: In 2009,12787 reports were classified as using SA, and
only 423 (3%) employed analysis of CSS. All 104 articles using CSS and
available in full text were evaluated, as well as 104 of the first articles
of 3813 using SA but not CSS and available in full text. Evaluation
of the sources of funding were done by two authors independently
with 30% overlap to estimate the agreement (K = 0.67). Of 104
studies using CSS, 16 (15%) were funded by industry, while of 104
studies using SA without estimates of CSS, 7 (6.7%) reported industrial
funding. The 95% confidence interval for the difference is 0.001 to
0.175. Conclusions: The possibility of finding significant differences
in specific-survival when total survival is not different between groups
may be an attractive perspective. This possibility is used by scientists
‘torturing’ the data in search of statistical significance. Authors of
systematic reviews need to know about this possible source of bias in
estimates of survival. The full presentation of this study will include the
data on use of specfic-survival in clinical trials.
P2B232
True methodological quality of trials are not reflected
in their reporting
Mhaskar R1 , Magazin A2 , Soares H2 , Kumar A1 , Djulbegovic B1
1 University of South Florida, Center for Evidence based Medicine, USA;
2 Moffitt Cancer Center, USA
Background: Biased results from poorly designed and reported
RCTs can mislead decision making. Whether publications accurately
reflect the actual methodological quality of randomized controlled
trials (RCTs) has not been comprehensively evaluated. Additionally,
it has been hypothesized that trial sample size is associated with
better methodological quality. Objective: To compare the reported
methodological quality of a RCT as reflected in publications with
actual methodological quality as depicted in the protocols, and
assess association of RCT sample size with published versus actual
methodological quality. Methods: All consecutive published phase III
RCTs conducted by 8 National Cancer Institute sponsored Cooperative
Groups (NCICOG) until year 2006 were eligible for inclusion. Data
on methodological quality domains relevant to minimizing bias and
Figure 1 (P2B232).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
132
random error were extracted from protocols and publications for each
study. Results: A total of 429 RCTs met the inclusion criteria. Overall
reporting of elements of methodological quality domains relevant
to minimizing bias (Figure 1A) and random error (Figure 1B) was
poor and did not reflect the actual high methodological quality of
RCTs. Similarly, the results showed a positive correlation between RCT
sample size and reported methodological quality domains of adequacy
of blinding procedures, ITT analysis and the choice of the comparator
(p value < 0.001). However, this correlation disappeared when actual
methodological quality was considered. Conclusion: This largest study
to-date comparing published versus actual methodological quality of
RCTs shows that poor quality of reporting by NCICOGs does not
reflect the actual high methodological quality. Similarly, we found no
association between RCT sample size and its actual methodological
quality. The positive correlation between methodological quality
of reporting and RCT sample size is misleading. Our findings
underline the need for adherence to the CONSORT statement by
authors and journal editors for transparent evaluation of RCTs and
highlight the importance of publication of RCT protocols in the public
domain.
P2B233
Impact of reported vs. actual methodological quality
assessment on treatment effect size
Mhaskar R1 , Magazin A2 , Kumar A1 , Djulbegovic B1
1
University of South Florida, Center for Evidence based Medicine, USA;
2 Moffitt Cancer Center, USA
Background: Discrepancy between reported methodological quality
and the actual methodological quality of randomized controlled trials
(RCTs) has been documented. However, the relative impact of reported
versus actual methodological quality on the treatment effect size (ES)
has not been assessed. Objective: To assess the impact of reported
methodological quality versus actual methodological quality on ES.
Methods: All consecutive terminated phase III RCTs published by
8 National Cancer Institute sponsored Cooperative Groups until year
2006 were eligible for inclusion. Data on methodological quality
domains relevant to minimizing bias and random error were extracted
from protocols and publications for each study. ‘Actual quality’ was
assessed based on data from either protocols or the publications.
The hazard ratio (HR) for overall survival was used as the ES.
Association between methodological quality (reported vs. actual)
with ES was conducted using standard meta-epidemiologic methods.
Results: A total of 429 RCTs met the inclusion criteria. There
was no statistically significant difference between associations of ES
and actual vs. methodological quality for the following domains:
adequacy of randomization sequence generation, description of drop
outs, intention to treat analysis, pre specification of alpha and beta
errors (Figure 1). However, on average, poorly reported allocation
concealment exaggerated the ES by 6% (Ratio of HRs: 0.94, 95% CI:
0.88–0.99). Also, poorly reported blinding inflated the ES by 24%
(Ratio of HRs: 1.24, 95% CI: 1.05–1.43). Nonetheless, in the ‘actual
quality’ assessment, no significant association between ES and any of
the methodological quality domains was detected. Conclusion: Our
study results show that the assessment of the impact of quality on
the ES based on the quality of reporting only can produce misleading
results. These findings are important for users of research evidence,
Figure 1 (P2B233).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
133
systematic reviewers, meta-epidemiologic research, as well as further
development of the Cochrane ‘risk of bias’ table which relies on
reported methodological quality.
P2B234
Observer bias in randomised clinical trials with continuous
outcomes: An analysis of trials with both blind and
non-blind outcome assessors
A1 ,
ASS1 ,
F2 ,
B2 ,
J3 ,
Hróbjartsson
Thomsen
Emanuelsson
Thomsen
Hilden
Ravaud P4 , Boutron I4 , Brorson S5
1 The Nordic Cpochrane Centre, Denmark; 2 The Nordic Cochrane Centre,
Denmark; 3 University of Copenhagen, Denmark; 4 The French Cochrane
Centre, France; 5 Herlev University Hosptal, Denmark
Background: The typical degree of observer bias in randomised clinical
trials with non-blind outcome assessors of continuous outcomes is not
known. Previous analyses which normally involve comparisons between
trials with blind assessors and ‘similar’ trials with non-blind assessors,
are susceptible to confounding. In an earlier work, based on trials that
use both blind and non-blind outcome assessors of the same binary
outcome, we found that substantial observer bias could result from
misclassification of few patients. Trials with continuous outcomes may
have a different susceptibility to observer bias, and an estimate of the
typical degree of bias in such trials is important for the assessment of
risk of bias. Objectives: To estimate the degree of observer bias in
randomised clinical trials with continuous outcomes, and to identify
factors associated with large observer bias. Methods: A systematic
review of randomised clinical trials with both blind and non-blind
assessment of the same continuous outcome. We searched PubMed,
EMBASE, PsycINFO, CINAHL, The Cochrane Trials Register; HighWire
Press and Google Scholar. We plan to extract data from one outcome
per trial, and to calculate the difference between the standardized mean
difference (SMD) based on blind assessments and the SMD based on
non-blind assessments: dSMD. We plan to pool the dSMDs with inverse
variance meta-analysis using random effects models. Furthermore, we
will study whether the size of dSMD is associated to a) high degree
of outcome subjectivity, e.g. global clinical improvement; b) close trial
involvement of the non-blind assessors (e.g. they also treat patients);
and c) high degree of sensitivity to reporting and behavior of non-blind
patients (e.g. symptom interviews). Results: At the time of abstract
submission we had identified 23 eligible trials. We expect full outcome
data from 17 trials. Conclusions: Pending.
P2B235
Pooled standardised mean differences and estimated
heterogeneity between trials depend on the type
of assessment tool analysed: Meta-epidemiological study
da Costa BR1 , Rutjes AWS1 , Johnston BC2 , Nüesch E1 , Reichenbach S1 ,
Guyatt G2 , Jüni P1
1
Division of Clinical Epidemiology & Biostatistics, ISPM, University of
Bern, Switzerland; 2 Department of Clinical Epidemiology and Biostatistics,
McMaster University, Canada
Background: Investigators performing systematic reviews and
meta-analyses are typically faced with data derived from different
assessment tools to measure a common concept such as pain. A
common method is to combine data regardless of instrument using
standardised mean differences (SMD). However, it is unclear whether
the estimated SMD and statistical heterogeneity will depend on the
choice of instrument used. Objectives: To determine whether the
two most frequently used instruments to assess knee or hip pain
in osteoarthritis trials, the WOMAC Pain subscale (WOMAC) and
Pain Overall measured on a visual analogue scale (VAS), result
in different pooled estimates of treatment effects and statistical
heterogeneity between trials. Methods: Meta-epidemiological study
of meta-analyses of large-scale osteoarthritis trials comparing active
treatment with placebo, sham, or non-intervention, reporting at
least one follow-up for both, WOMAC and VAS, with at least 100
patients per group. We pooled SMDs for each instrument using
an inverse-variance random-effects model within each meta-analysis
and subsequently combined pooled estimates of treatment effect and
statistical heterogeneity across meta-analyses using a random-effects
model. Estimates were compared between instruments using a paired
sign-rank test. Negative SMDs indicated a benefit of experimental
interventions as compared with control. Results: We identified 5
meta-analyses including 24 large-scale trials with 13400 patients that
reported on both WOMAC and VAS. The average SMD was −0.22
(95% CI −0.34 to −0.09) according to WOMAC and −0.32 (95%
CI −0.46 to −0.18) according to VAS (p for difference = 0.038,
see Table 1). Corresponding I2 estimates were 53% (95% CI 0% to
Table 1 (P2B235). Standardized mean differences and I2 values presented per meta-analysis and pooled across meta-analyses
for each outcome.
WOMAC
Intervention
NSAIDS
Food Supplements
Opioids
Viscosupplementation
Acupuncture
Overall
VAS
SMD
I2
SMD
I2
Total number of patients
−0.31
−0.26
−0.35
−0.07
−0.01
−0.22
17%
89%
44%
34%
0%
53%
−0.42
−0.50
−0.34
−0.03
−0.12
−0.32
80%
96%
71%
78%
83%
85%
9453
1271
1495
828
353
13400
WOMAC: WOMAC Pain subscale; VAS: Pain Overall measured on a visual analogue scale; SMD: standardized mean difference; NSAIDS:
non-steroid anti-inflammatory drugs.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
134
80.7%) and 85% (95% CI 66.6% to 93.4%), respectively (p = 0.043).
Conclusions: Differences in estimated SMDs between WOMAC and
VAS were small on average, but WOMAC resulted in considerably
lower heterogeneity between trials. Therefore, WOMAC should be
given precedence over VAS when extracting data for meta-analysis of
osteoarthritis trials using SMDs.
P2B236
Methodological adequacy of articles published in two
open-access Brazilian cardiology periodicals
Macedo CR1 , Silva DL1 , Melnik T1 , Rachel R1 , Pedrosa A1 , Torloni R1 , Silva
EMK1 , Atallah AN1 , Puga ME1
1 Brazil Cochrane Center, Br
Background: The use of rigorous scientific methods has contributed
towards developing scientific articles of excellent methodological
quality. This has made it possible to promote their citation and
increase the impact factor. Brazilian periodicals have had to adapt to
certain quality standards demanded by these indexing organizations,
such as the content and the number of original articles published in each
issue. Objectives: This study aimed to evaluate the methodological
adequacy of two Brazilian periodicals within the field of cardiology
that are indexed in several databases and freely accessible through
the Scientific Electronic Library Online (SciELO), and which are now
indexed by the Web of Science (Institute for Scientific Information, ISI).
Methods: All the published articles were evaluated according to merit
assessment (content) and form assessment (performance). Results:
Ninety-six percent of the articles analyzed presented study designs
that were adequate for answering the objectives. Conclusions:
These two Brazilian periodicals within the field of cardiology published
methodologically adequate articles, since they followed the quality
standards. Thus, these periodicals can be considered both for
consultation and as vehicles for publishing future articles. For further
analyses, it is essential to apply other indicators of scientific activity such
as bibliometrics, which evaluate quantitative aspects of the production,
dissemination and use of information, and scientometrics, which is also
concerned with the development of science policies, within which it is
often superimposed on bibliometrics.
P2B237
Are we evaluating methodological quality in the same
way? Comparison of tools used in general health research
and physical therapy with the ‘Risk of bias’ tool
Armijo-Olivo S1 , Ospina M2 , Rowe B3 , Hartling L1
1 Alberta Research Center for Health Evidence, Department of Pediatrics,
University of Alberta, Canada; 2 School of Public Health, University of
Alberta, Canada; 3 Faculty of Medicine and Dentistry, University of Alberta,
Canada
Background: Assessing the methodological quality of randomized
controlled trials (RCTs) is crucial to understand how bias may affect
study results and to accurately identify treatment effects. A wide
variety of tools are available to evaluate RCT quality; however, it is
unknown how these tools compare in terms of the items included.
Objectives: To describe the frequency of individual items across
quality assessment tools and to compare tools used in physical therapy
(PT) to those used in general health research. Methods: We identified
tools used to evaluate the methodological quality of RCTs in general
health research and PT following a comprehensive search and using
a systematic approach. We extracted the items included in each
tool. We calculated the frequency of each item used across tools and
compared these among tools used in general health research and PT.
Comparisons were also made with the Cochrane Risk of Bias (RofB) tool.
Results: Twenty-one tools were identified: 15 used in general health
research and seven in PT. Ninety-five different items were used across
tools. The most frequent items appearing in 15 of the general tools
(67%) were: description of randomization, withdraws and dropouts,
inclusion and exclusion criteria, and appropriate statistical analysis.
In contrast, the most frequent items appearing in six out of seven
tools used for PT (85.7%) were: baseline comparability, blinding of
investigator/assessor, and use of intention-to-treat analysis. Prominent
features of the RoB tool such as sequence generation and allocation
concealment were included in four (57%) and five (71%) of the PT tools,
respectively. Ten (63%) and six (40%) of the general health research
tools included randomization and allocation concealment, respectively.
Conclusions: There is extensive variation in the items included across
quality assessment tools. It is critical that methodological features
associated with bias are captured in tools used for PT trials.
P2B238
Methodological quality of malaria RCTs conducted in
Africa
Lutje V1 , Gerritsen A2 , Siegfried N3
1 Cochrane Infectious Diseases Group, Liverpool, UK; 2 Department of Public
Health, University of Venda, Thohoyandou, South Africa; 3 South African
Cochrane Centre, Medical Research Council, Cape Town, South Africa
Background: Good methodological quality is necessary to reduce risk
of bias in randomized controlled trials (RCTs) and in meta-analyses.
As part of a review of clinical and methodological characteristics of
malaria RCTs conducted in Africa, we assessed the methodological
quality of trials conducted after the publication of the original CONSORT
statement in 1996. This is a novel analysis that can highlight training
needs for clinicians conducting trials in potentially resource-limited
settings. Objectives: To analyse the methodological quality of malaria
RCTs conducted in Africa between 1997 and 2007. Methods: We run
systematic searches for malaria RCTs in electronic databases (Medline,
Table 1 (P2B238). Assessment of trial quality for malaria RCTs published 1997–2007. N = 60 trials.
Sequence generation
Allocation concealment
Blinding
Loss to follow up
Adequate
Not adequate
Unclear
Not done
Not possible
35
14
23
49
3
21
36
27
11
1
10
8
2
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
135
Embase, the Cochrane Library), and applied an African geographic
search filter to identify trials conducted in Africa. Results were exported
to the statistical package STATA 8 to obtain a random sample from
the overall data set. We evaluated 60 trial reports published between
1997 and 2007 for risk of bias according to 4 domains (randomized
sequence generation, allocation concealment, blinding, and loss to
follow-up). Results: Sequence generation was considered adequate
(as done by using a random numbers table or electronically generated)
in 35 reports, but was not clearly reported in 21 trials (Table 1). Many
RCTs did not mention methods of allocation concealment or blinding
of participants or intervention providers. In contrast, loss to follow
up was accounted for in most RCTs (49 out of 60). Conclusions:
The quality of malaria trials’ reports was not consistent among the
4 domains analysed: a large proportion of RCTs had a high risk of bias
for blinding and allocation concealment, whereas loss to follow-up
was mostly well reported. Similar suboptimal reporting has been
widely reported for RCTs in different healthcare areas, potentially
affecting the validity of trial results and the estimates of treatment
effects, and is not associated with trials conducted in resource-poor
settings.
P2B239
A systematic review on the efficacy of statins in animal
model
Pecoraro V1 , Moja L2 , Dall’Olmo L3 , Cappellini G4 , Liberati A1 ,
Garattini S4
1
Italian Cochrane Centre, Italy; 2 University of Milan, Italy; 3 I.O.V.-Istituto
Oncologico Veneto- I.R.C.C.S, Italy; 4 Mario Negri Institute for Pharmacological Research, Italy
Background: Statins are recognised as an effective treatment for
lowering cholesterol levels and reducing the risk of cardiovascular
diseases. They have been widely studied in basic and human research
and there are no systematic reviews of statins across animal models.
Objectives: We explored differences in the efficacy of statins in three
popular animal models: mice, rats and rabbits, using the technique
of systematic review (i.e. Cochrane methods). We also explored
whether the reporting was accurate, and the risk of bias. Methods:
We searched MEDLINE and EMBASE. Two independent reviewers
assessed the eligibility, extracted study details and biochemical
blood parameters, blood pressure, myocardial infarction and survival.
Weighted and standard mean difference random effects meta-analysis
was used to measure overall efficacy in specified species, strains and
subgroups. Results: We analyzed 161 studies. Most papers did not
use randomization (55%) or blinding (88%) and their reporting
presented several shortcomings. Statins lowered total cholesterol
in all species though with large differences in the effect size: −30% in
rabbits, −20% in mice and −10% in rats. Many meta-analyses suffered
substantial heterogeneity despite the postulated limited biological
variability in single species and strains. Few studies considered
strains at high risk of cardiovascular diseases even though mimic
human cardiovascular diseases better or examined hard outcomes.
A risk of pubblication bias our data also suggested. About 45% of
publications identified report no significant findings on total cholesterol.
Conclusions: Although statins showed substantial efficacy in animal
models, the preclinical data on conditions mimicking the human
pathologies for which the drugs are clinically indicated and utilized
are often scarce, incomplete and at risk of bias. This study raises a
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
number of concerns, particularly about the reliability and quality of
animal experiments that may drive future clinical trials in humans.
P2B240
Shifting effects in three randomised controlled trials on
music therapy in mental health
Gold C1 , Erkkilä J2 , Crawford M3
1 Uni Research, Bergen, Norway; 2 University of Jyväskylä, Finland; 3 Imperial
College London, UK
Background: Randomised controlled trials (RCTs) are based on the
assumption that the effects of interventions are constant across all
participants. This may not always be true in complex interventions
which are known to depend on provider’s expertise and experience and
may also be vulnerable to context variables (such as whether therapy
is delivered within a trial). Music therapy, with music experiences
and a therapeutic relationship as the elusive key components, is a
complex intervention with an emerging evidence base. Objectives:
To investigate time trends in effect over the recruitment period in
music therapy RCTs, which might indicate an impact of delivering
therapy within a trial. Methods: We included recent (2006-present),
methodologically rigorous RCTs on music therapy versus standard care
for people with mental disorders (schizophrenia or depression) that
used a parallel single-blind design to assess changes in a continuous
scale of mental health (symptoms of schizophrenia or depression).
Music therapy had to be delivered by experienced therapists. Individual
patient data were examined graphically and statistically. Results:
Three trials were included. In all trials, the comparative effects of the
interventions (music therapy versus standard care) were not constant,
but shifted over time. One trial suggested a? reverse effect in
the first ten participants than in later participants. The other trials
suggested a smaller effect in those first participants than in later ones.
Whereas the trials overall suggested superiority of music therapy, this
was not the case for the first participants recruited. Conclusions:
For some complex interventions, it cannot be assumed that effects
remain constant over the recruitment period of a trial. Delivering
therapy within a trial may be an unsettling experience for experienced
therapists, especially at the beginning of a trial. If replicated, our
findings will have implications for the analysis and reporting of RCTs
on complex interventions.
P2B241
The quality of randomized clinical trials in surgery can be
improved. A systematic review
Verhagen AP1 , v Middelkoop M1 , Rubinstein SM2 , Jacobs WC3 , Ostelo
RW2 , Peul W3 , Koes BW1 , v Tulder MW2
1 ErasmusMC, Netherlands; 2 EMGO-institute, Netherlands;
3 LUMC, Netherlands
Background: The field of spine surgery undergoes rapid introduction
of new surgical techniques. The effectiveness or efficacy of these
new techniques needs to be evaluated in rigorous randomized
clinical trials to support their use. Objectives: To evaluate the
methodological quality in surgery trials of the cervical and lumbar
spine to enhance awareness and endorsement of bias avoidance and
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
136
reporting recommendations of surgery trials. Methods: A risk of
bias assessment was conducted on all included studies in systematic
reviews of surgical interventions in patients with neck and low back
pain by two reviewers independently using the assessment form of
the Cochrane Back Review Group. A study was defined as having a
low risk of bias when fulfilling six or more of the items. A descriptive
analysis was performed. Results: In total, 90 RCTs were identified in
three systematic reviews comparing different surgical interventions with
conventional treatment or other surgical interventions. Of these studies
only a few trials had a low risk of bias, and most of them were published
between 2007–2009. The items that scored positive in most studies
were adequate randomisation, acceptable drop-out rate and timing of
outcome assessment. The most common items that were unclear or
scored negatively included the concealment of allocation, blinding of
patients, care provider or outcome assessor and the implementation
of intention-to-treat analysis. Conclusions: The quality of RCTs
on surgical interventions in the cervical and lumbar spine file has
increased over time, but is still poor and shows room for improvement.
Improvement can be reached by paying more attention to the
randomisation procedure (concealment), co-interventions, preventing
withdrawals and the analysis (intention to treat).
P2B242
Assessing risk of bias in non-randomised studies and
incorporating GRADE: Initial experience with a new
Cochrane ‘Risk of bias’ tool under development
MacLennan S1 , Imamura M1 , Dahm P2 , Neuberger M2 , Reeves B3 ,
MacLennan G4 , Omar MI1 , McClinton S5 , Griffiths L6 , N’Dow J7
1
Academic Urology Unit, University of Aberdeen, Aberdeen, UK;
2 Department of Urology, College of Medicine, University of Florida,
Gainesville, Florida, USA; 3 Faculty of Medicine and Dentistry, University
of Bristol, UK; 4 Health Services Research Unit, University of Aberdeen,
UK; 5 Urology Department, NHS Grampian, Aberdeen Royal Infirmary,
Aberdeen, UK; 6 Department of Cancer Studies and Molecular Medicine,
University of Leicester, Clinical Sciences Unit, Leicester General Hospital,
Leicester, UK; 7 Urology Department, NHS Grampian, Aberdeen Royal
Infirmary, and Academic Urology Unit, University of Aberdeen, Aberdeen,
UK
Background: In instances where randomised controlled trials (RCT) are
impossible or have not been conducted, clinical recommendations and
decision-making must rely on other evidence. If systematic reviewers
decide to include non-randomised studies (NRS), it is imperative to
use a standard method to assess and communicate the risk of bias
(RoB) in NRS. Objectives: To pilot a RoB tool for NRS and make
it commensurate with GRADE. Methods: An extended version of
the Cochrane RCT RoB tool was applied to NRS. This included an
additional item on the risk of findings of an NRS being explained
by confounding. Each pre-specified confounding factor was assessed
on the precision of measurement, baseline imbalance, and quality
of case-mix adjustment, on 5-point scales. Imbalance was judged by
clinical consensus, while other items were assessed by two independent
reviewers. Mean ‘adjustment’ scores per outcome across studies were
used to determine the quality of evidence according to GRADE. The
tool was applied to 33 NRS retrieved for a systematic review of surgical
interventions for localised renal cancer. Results: The initial 5-point
scale was unwieldy and lead to disagreement among reviewers. We
created scoring guidelines and re-piloted. RoB scores were tabulated
rather than aggregated to indicate where likely biases were located.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
All NRS were rated as either ‘low’ or ‘very low’ on GRADE; however,
determining an appropriate cut-off required considerable judgement.
Conclusions: Compared with RoB assessment in RCT, assessment
of NRS was more difficult and increased required time and expertise
resources. In areas where the quality of studies is known to be
very low, the added time and complexity may make the assessment
not worthwhile. Presentation of the large amount of information
generated by this tool is challenging. Further research needs to strike a
balance of making a ‘brief’ and ‘easy’ version while addressing complex
methodological issues inherent in NRS.
P2B243
Conflicting guidelines for screening mammography:
Influence of author’s specialty and conflicts of interest
Burda B1 , Holmer H2 , Ogden L2 , Fu R2 , Norris S2
1
Oregon Evidence-based Practice Center, Kaiser Permanente Center for
Health Research, USA; 2 Department of Medical Informatics and Clinical
Epidemiology, Oregon Health and Science University, USA
Background: Financial and intellectual conflict of interest (COI)
may explain conflicting recommendations in clinical practice guidelines
(CPGs) for screening mammography. Objectives: To assess the
financial and intellectual COI of the authors of CPGs and to examine
the relationship of COI to recommendations for routine screening
mammography in asymptomatic women 40 to 49 years of age who
are at average risk of breast cancer. Methods: We searched the
National Guideline Clearinghouse and MEDLINE for relevant CPGs
published from 2005 to 2010. We documented the disclosures and
specialties of the lead and secondary authors of these CPGs, and the
publications of the lead authors, and examined their relationship to
CPG recommendations. Results: Twelve CPGs were identified with
13 lead authors and 178 physician authors of various specialties.
None of the four CPGs recommending non-routine screening had
a radiologist member, while five of the eight CPGs recommending
routine screening had at least one (P = 0.05). There was a trend
that a CPG with a radiologist member was more likely to recommend
routine screening, although the association was not significant (odds
ratio [OR] 6.05, P = 0.14). The proportion of primary care physicians
on CPGs recommending routine versus non-routine screening was
significantly different (P = 0.01). The odds of a recommendation for
routine screening were related to the number of recent publications on
breast disease by the CPG lead author (OR 2.32 for each additional
publication, P = 0.02). Conclusions: Recommendations for or
against routine mammography screening in the target population may
reflect the specialty and intellectual interests of the CPG authors. Our
conclusions are limited by the observational nature of the data and by
the small number of included CPGs.
P2B244
Assessment of trial risk of bias among Cochrane reviews:
A cross-sectional analysis
Abou-Setta A1 , Dryden D1 , Hamm M1 , Moher D2 , Klassen T3 , Hartling L1
1 Alberta Research Centre for Health Evidence, Canada; 2 Ottawa Hospital
Research Institute, Canada; 3 Manitoba Institute of Child Health, Canada
Background: Risk of bias (RoB) of trials may affect the conclusions
of systematic reviews. Since 2008, Cochrane reviews are expected to
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
137
include a detailed assessment of bias for included trials using the RoB
tool. It is not clear how stringently this is followed, or how much
potential bias is present across Cochrane reviews. Objectives: To
assess the RoB among included trials in Cochrane reviews. Methods:
We searched the Cochrane Database of Systematic Reviews (CDSR)
for all active, completed reviews. Using a customized automated
algorithm (Visual Basic for Applications) we extracted RoB assessments
of included trials. An ‘unclear’ risk was imputed for missing assessments
except where RoB was not assessed at all. Results: In April 2011,
the CDSR contained 4,594 reviews. We excluded withdrawn reviews
(n = 204) and overviews of reviews (n = 3). Further, 3,013 (66%)
reviews were excluded for lack of data [RoB assessment not performed
(n = 2,599); no trials included (n = 411); other quality assessment
tool used (n = 3)]. In total, 1,374 reviews encompassing 17,220
trials were included in our analyses. Of these, 39% of trials had RoB
judgments reported for all domains; remainder had variable missing
assessments. Overall 3%, 53%, and 44% of trials were considered
to be ‘low’, ‘unclear’, and ‘high’ risk, respectively. The rationale
for the judgments was provided for 88% of the assessments. The
number of trials classified as ‘high’ has significantly decreased over
time, while trials classified as ‘low’ or ‘unclear’ risk have gradually
increased (p < 0.0001). The domain most often assessed as ‘low’ risk
was ‘incomplete outcome data’; ‘blinding’ was most often assessed as
‘high’ risk. Conclusions: RoB assessments for most trials included in
Cochrane reviews are ‘unclear’, likely due to poor reporting at the trial
level. Further, 97% of trials were assessed as ‘unclear’ or ‘high’ risk
which raises questions regarding the sensitivity/ specificity of the RoB
tool.
P2B245
Does stratification analysis indicate the confounding
factors in meta-analysis of observational studies?
Naing C1 , Mak JW1
1
International Medical University, Kuala Lumpur, Malaysia
Background: It has been suggested that any systematic review
including meta-analysis of observational studies must consider the risk
of bias in individual primary studies. To date, checklists for the reporting
of observational epidemiological studies have been developed, but their
usefulness is limited. Moreover, the design features of primary studies
rather than a design label is important because the risk of bias is affected
by the specific features of a study, not by a broad categorization of
the approach taken. As such, listing potential confounding factors in
the primary studies and stratification of participants into subgroups
with respect to potential confounding factors is straightforward and
valuable. Objectives: To analyse the effect estimates by stratification
of the participants according to the potential confounding factors.
Methods: As an illustration, we performed a meta-analysis of
observational studies identifying the association between hepatitis C
virus infection (HCV) and type 2 diabetes (T2D), following the Cochrane
guideline for the non-randomised studies. We stratified the participants
according to known risk factors age group, gender, and family history
of diabetes, and performed subgroup analysis. Results: We identified
observational studies addressing the association between HCV and T2D
(k = 21, n = 294437). A positive and significant association between
HCV and T2D was observed (odds ratio 1.71; 95% CI = 1.17–2.49).
Patients of age >40 years (7.47; 95% CI = 5.93–9.42), obesity (0.32:
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
95% CI = 5.93–9.42) and female sex (0.8: 95% CI = 0.66–0.97) were
also significant factors. Conclusions: Observational epidemiological
studies are generally poorly reported. The evaluation of methodological
quality and risk of bias consistently across primary studies is difficult.
The stratification by potential confounding factors in the interpretation
of the findings of meta-analysis of observational studies could be
valuable.
P2B246
Unplanned crossover in randomised controlled trials:
Consequences for efficacy and safety outcomes
D’Amico R1 , Bonafede E1 , Balduzzi S1 , Longo G1 , Guarneri V1 ,
Piacentini F1 , Moja L2 , Liberati A1
1 University of Modena and Reggio-Emilia, Italy; 2 University of Milan, Italy
Background: If a randomized controlled trial (RCT) shows results
in favour of an intervention, investigators might surmise that the
equipoise principle is no longer present. As a consequence, control
patients may be offered to cross over to the experimental arm, even
if it was unplanned. Objectives: We assessed: a) the prevalence of
the unplanned crossover in oncology trials; b) how it is dealt when
analysing data; and c) the potential magnitude and direction of bias.
Methods: RCTs published between 2000 and 2011 in the NEJM,
JNCI, Lancet, Lancet Oncology, JCO, Annals of Oncology, assessing the
efficacy and safety of treatments in oncologic patients were searched.
We explored the relationship between the effect and: a) the time when
the crossover occurred; b) the fraction of patients who decided to cross
over; and c) their clinical characteristics. The magnitude and direction
of bias were investigated through simulations. Results: Out of 164
papers analyzed so far, in 11 RCTs investigators gave patients the
opportunity to crossover to the experimental arm. The phenomenon is
more frequent in the second half of the decade. All studies compared
outcomes by using ITT and 20% used an additional censored analyses.
Simulations suggest that ITT analyses dilute the treatment effect size
for efficacy and safety, whereas the magnitude and direction of the
bias caused by the censored analysis seem to be less predictable. The
inverse probability of censoring weighted model was also used, but it is
highly speculative since it is dependant on the characteristics of those
patients who decided to cross over. Conclusions: While investigators
justify the unplanned crossover of patients on ethical grounds, it can
leave the scientific community with increased uncertainty of study
results, and may be a condition to inflate the net benefit and risk of an
intervention of clinical relevance.
P2B247
Applicability of R-AMSTAR instrument to appraise
systematic reviews
Martı́nez-Zapata MJ1,2,3,4 , Rigau D1,2 , Selva A2 , Gich I2,3,4 , Bonfill X1,2,3,4
1 Iberoamerican Cochrane Centre, Barcelona, Spain; 2 Biomedical Research
Institute Sant Pau (IIB Sant Pau), Barcelona, Spain; 3 CIBER of Epidemiology
and Public Health (CIBERESP), Barcelona, Spain; 4 Universitat Autònoma
de Barcelona, Spain
Background: The science of research synthesis aims to systematically
gather, examine and evaluate primary study reports that answer
a clearly formulated question. However not all systematic reviews
(SRs) reach the goal to minimize bias and some are furnished of
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
138
methodological flaws. The revised and validated AMSTAR (R-AMSTAR)
instrument has been generated to quantify the SRs quality. Objectives:
To describe the applicability and drawbacks of R-AMSTAR instrument.
Methods: We did and overview on the effectiveness of acupuncture
in obstetric-gynaecological disorders. A comprehensive literature
search was performed in CENTRAL, CDSR, DARE, MEDLINE databases
up to July 2010 and with no language restriction. Two reviewers
independently performed the literature selection and appraised the
retrieved SRs using the R-AMSTAR instrument. This instrument contains
11 questions (each with 3 or 5 items) rated from 1 to 4 (total score range
from 11 to 44). Although there were not a previous consensus between
reviewers, they were experienced in the critical appraisal of SRs. Each
item and overall agreement was tested using the kappa coefficient. The
intraclass correlation coefficient (ICC) was used to test the agreement
of R-AMSTAR sum scores. Results: We identified 728 references and
included 8 SRs. The median score obtained by R-AMSTAR tool was 32
(range 21–41). Kappa coefficient value of each item differed widely
and overall agreement was low (K = 0.338). The agreement in the sum
scores was ICC 0.506 (moderate agreement). A greater agreement
existed in the questions about ‘a priori design provided’, ‘a duplicate
study selection and data extraction’ and ‘the list of studies provided’.
Conclusions: A rather low agreement was obtained with the use of
the R-AMSTAR instrument. Some items are ambiguous and may not
be a direct attribute of SR methodological quality.
P2B248
How often do decision analyses of interventions agree
with matching systematic reviews?
Mhaskar R1 , Georgiev H1 , Wao H1 , Kumar A1 , Djulbegovic B1
1 University of South Florida, Center for Evidence based Medicine and
Health Outcomes Research, USA
Background: Systematic review (SR) of randomized controlled trials
(RCT) is reckoned the gold standard for informing treatment choice.
Decision analyses (DA) inform health care policy decisions in absence
of RCTs or SR of RCTs. However, oversimplification of real world
scenarios in DA can be problematic. We have previously shown
that DA disagrees with RCTs in 50% of cases. However, it is not
known how often the results of DA and SR of RCTs either agree
or disagree. Objective: To compare the conclusions of DA and
matching SR of RCTs. Methods: We searched PubMed up to 2008
for DAs comparing at least two interventions followed by SRs that
matched the DAs based on patient population, intervention, control,
and outcome criteria (PICO). From each DA and SR, we extracted
data on PICO, conclusion, and impact of sensitivity analyses on the
conclusion. Agreement between DA and SR was based on matching of
respective conclusions. Results: From 42,704 retrieved DA citations,
we found matching SR for 38 comparisons (Figure 1). We found 74%
(28/38) agreement between the conclusions of the DAs and the SRs.
The sensitivity analyses conducted in either DA or SR did not impact the
agreement. Two DA design characteristics were significantly associated
Figure 1 (P2B248).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
139
with agreement: use of single versus multiple data source (p = 0.048)
and use of meta-analysis data (p = 0.040). Conclusions: This first
study quantifying the correlation between the results of DA and SR of
RCTs suggests a high level of agreement. Use of meta-analysis data
and use of multiple sources of data appear to impact the agreement of
conclusions between DA and SR of RCTs.
P2B249
Epidemiology of the type of intention-to-treat reporting
in meta-analyses
Montedori A1 , Bonacini MI2 , Luchetta ML3 , Papini D4 , Rocchi RE1 ,
Orso M1 , Cozzolino F1 , Abraha I1
1 Regional Health Authority of Umbria, Italy; 2 Derriford Hospital, Plymouth,
UK; 3 Azienda Sanitaria Locale 3, Foligno, Italy; 4 Regional Agency for
Health of Emilia Romagna, Italy
Background: Despite the intention-to-treat (ITT) principle is the
recommended method to perform unbiased analyses in randomised
trials, it is inadequately applied and participants are often improperly
excluded. Recently, a modified intention-to-treat (mITT) analysis with
inconsistent descriptions is increasingly being used in randomised
trials. The reporting of mITT in trials is significantly associated with
post-randomisation exclusions. However, little is known about the
impact of mITT reporting trials in trials included in meta-analyses.
Objectives: To estimate the prevalence of the type of ITT in
meta-analyses and to assess the description of mITT reporting;
and to determine differences in terms of reporting, methodological
quality and effect size among the types of ITT. Methods: We
searched MEDLINE for systematic reviews indexed from 2005 to
2009. Montori’s search strategy was used (MEDLINE[Title/Abstract])
OR (systematic[Title/Abstract] AND review[Title/Abstract] OR metaanalysis[Publication Type]). From each year we randomly selected
10% of the records. Only therapeutic meta-analyses that included at
least one randomised trial were considered. Diagnostic, prognostic,
or epidemiological studies were excluded. Within each meta-analysis
trials were classified according to the type of ITT analyses used as
follows: ITT, trials reporting the use of standard ITT analyses; mITT,
trials reporting the use of ‘modified intention-to-treat’ analyses; or ‘no
ITT’ trials not reporting the use of any ITT analyses. Trials reporting the
use of ITT with descriptions or conditions different from the standard
ITT definition were classified as mITT. Items for quality measures
such as allocation concealment, blinding, and early stopping were
considered. Results: We identified 200 meta-analyses out of 569
records published in 2005, 210 of 657 published in 2006, 190 of 804
published in 2007, 197 of 863 published in 2008, and 186 of 988
published in 2009. Conclusions: Definitive results will be presented
at the Colloquium.
P2B250
Using AMSTAR to assess the methodological quality of
systematic reviews: An external validation study
Parmelli E1 , Banzi R2 , Fernandez Del Rio MDP3 , Minozzi S3 , Moja L4 ,
Pecoraro V3 , Liberati A1
1 Italian Cochrane Centre and University of Modena and Reggio Emilia, Italy;
2
Mario Negri Institute for Pharmacological Research and Italian Cochrane
Centre, Milan, Italy; 3 Italian Cochrane Centre, Mario Negri Institute for
Pharmacological Research, Milan, Italy; 4 University of Milan, Italy
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: AMSTAR is an instrument for the evaluation of
systematic reviews’ (SRs) methodological quality of conduct, a
pre-requisite for the interpretation and application of SRs findings.
Previous studies suggested that AMSTAR is reliable, valid and easy
to use. A broader range of assessors on different types of SRs is
needed. Objectives: To measure the reliability, construct validity and
feasibility of AMSTAR on a sample of SRs in different medical fields.
Methods: A convenient sample of SRs in different medical fields
was independently assessed by two reviewers using: AMSTAR and
the Overview of Quality Assessment Questionnaire (OQAQ). Reliability
was assessed using the weighted Cohen’s kappa. Construct validity
was investigated comparing the two instruments using Pearson’s Rank
correlation coefficient. Time to complete scoring was also recorded.
Results: Fifty-four SRs focusing on primary prevention of hypertension
and colorectal cancer screening and treatment were assessed using
both AMSTAR and OQAQ. Eleven out of 54 were Cochrane SRs. 78%
of the SRs analysed included a meta-analysis. Preliminary data on half
sample showed an overall score ranging from 0 to 11 (median: 6) for
AMSTAR and from 0 to 9 (median: 6) for the OQAQ. The inter-observer
agreement of the individual items on AMSTAR ranged from fair to very
good with a kappa going from 0.20 to 0.92. Four items scored a kappa
>0.75. Fair agreements related to items about methods to combine
results and conflict of interest. The kappa for each individual item for
OQAQ ranged from 0.25 to 0.81. The reliability of the total score was
fair for both checklists: kappa scored 0.35 for AMSTAR and 0.32 for
OQAQ. Final results will be presented at the Colloquium. Conclusions:
This study represents a further validation of the AMSTAR involving a
broader range of assessors and SRs, and could help to refine the
instrument and disseminate its use.
P2B251
Systematic evaluation of the methodology of randomized
controlled trials of anticoagulation in patients with cancer
Rada G1 , Schünemann H2 , Labedi N3 , Akl E3
1 Department of Internal Medicine, Evidence Based Health Care Program,
Faculty of Medicine, Pontificia Universidad Católica de Chile, Santiago,
Chile; 2 Department of Clinical Epidemiology and Biostatistics, McMaster
University, Hamilton, Canada; 3 Department of Medicine, State University
of New York at Buffalo, Buffalo, NY, USA
Background: Biased randomized controlled trials (RCTs) mislead
clinical practice and adversely affect patients’ outcomes. A number
of methodological features can limit bias in RCTs. Objectives: To
systematically assess the descriptive characteristics, the methodological
quality and their time trends in RCTs included in six systematic
reviews of anticoagulation in cancer patients. Methods: We conducted a comprehensive search, including several electronic databases
(MEDLINE, EMBASE, ISI the Web of Science, and CENTRAL) until
February 2010. We included RCTs in which the intervention and/or
comparison consisted of: vitamin K antagonists, unfractionated
heparin (UFH), low molecular weight heparin (LMWH), ximelagatran
or fondaparinux. We used a standardized form to abstract study
descriptive characteristics and rate methodological quality (blinding of
patients, providers, outcome assessors and data analysts; intention to
treat analysis; early stopping for benefit; and percentage follow-up). We
performed descriptive analyses and assessed the association between
the variables of interest and the year of publication categorized in
5-year intervals using chi-square test. Results: The title and abstract
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
140
screening of 3986 citations identified 69 potentially eligible RCTs, of
which 2 were additional reports of RCTs and were excluded. 67 unique
RCTs are represented in the sample, including 24071 participants.
Adequate allocation concealment was 67%, patients’ blinding 40%,
data collector’s blinding 40%, providers’ blinding 43%, outcome
assessors’ blinding 71%, data analysts’ blinding 16%, intention to
treat analysis 64%, and stopping early for benefit 3%. The mean
follow-up rate was 96%. Results of association between the variables
of interest and the year of publication will be presented at the
Colloquium. Conclusions: RCTs of anticoagulation in cancer patients
appear to use less rigorous outcome assessment methods and to have
deficiencies in key methodological features. It is not clear whether this
reflects a problem in the conduct or the reporting of these trials, or both.
Trialists of anticoagulation in patients with cancer should design more
rigorous trials and transparently report their methods and findings.
P2B252
An online tool to assess the potential impact of missing
outcome data on the estimates of treatment effect of
trials
Akl E1 , Agarwal N2 , Guyatt G3
1 Department of Medicine, State University of New York at Buffalo, NY,
USA; 2 State University of New York at Buffalo, NY, USA; 3 Department of
Clinical Epidemiology and Biostatistics, Hamilton, ON, Canada
Background: Systematic reviewers commonly deal with RCTs for
which relevant outcome data are missing. The Cochrane Handbook
recommends reviewers to assess ‘how sensitive results are to reasonable
changes in the assumptions that are made’. Objective: To develop an
online tool to assess the potential impact of reasonable assumptions
about missing data for binary outcomes on the estimates of treatment
effect of RCTs. Methods: We developed the tool as an online
interactive Java application. It conducts the sensitivity analysis at the
RCT level. For each trial arm (i.e., the intervention and control), the
user enters the number of participants randomized, the number of
observed events, and the number of participants for whom outcome
data is missing. For the RCT under consideration, the tool provides the
effect estimate as relative risk with its 95% confidence interval based
on each set of assumptions being tested. Results: The tool is available
freely online at: http://apps.medinnovations.us/. It allows the testing
of the following assumptions about the outcomes of participants for
whom data is missing: (1) none had an event; (2) all had an event;
(3) all in the treatment group had an event while none in the control
group had one (worst case scenario); (4) none in the treatment group
had an event while all in the control group had one (best case scenario);
(5) the event incidence among participants with missing data (relative
to observed participants) varies by ratios specified by the user for the
intervention and control groups separately. It also provides the results
for a complete case analysis (e.g., excluding participants for whom data
I missing from the analysis). Conclusions: The tool allows systematic
reviewers to conduct sensitivity analyses using the assumptions that
are most plausible for the specific question of their review.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P2B253
Conflict of interest policies for clinical practice guideline
organizations do not measure up to IOM standards
Norris SL1 , Holmer HK1 , Ogden LA1
1 Department of Medical Informatics and Clinical Epidemiology, Oregon
Health & Science University, USA
Background: Conflict of interest (COI) is an important potential
source of bias in the development of clinical practice guidelines (CPGs).
A 2011 Institute of Medicine (IOM) report delineates standards for
CPGs, including COI. Objectives: To describe the COI policies for
organizations producing a large number of CPGs and to compare
those policies to recommendations in the IOM report. Methods: We
performed a cross-sectional study of organizations listing five or more
CPGs within the National Guideline Clearinghouse between 1/1/09
and 11/10. We searched for COI policies in organizational websites
and in recent CPGs. We abstracted the domains of each policy
into a standardized template and compared them to recommendations
2.1–2.4 in the IOM report. Results: We identified 37 organizations that
fulfilled inclusion criteria. 33% of organizations had a COI policy specific
for CPGs; an additional 15% had a general COI policy encompassing
research and professional behavior, and the remainder had no policy
that we could identify. No organizational policy recommended that CPG
funders should be disclosed and only 6% of organizations precluded
industry funding. 17% of organizations considered COI in the selection
of CPG panel members, and two organizations (6%) had a policy that
prohibited the Chair from having a relevant COI. Only one organization
indicated that a minority of members of a CPG panel could have a COI.
Conclusions: Organizations producing large numbers of CPGs do not
meet the standards of the 2011 IOM report and half do not have a
specific policy. These organizations need to address this issue in order
to provide trustworthy CPGs.
P2B254
Reporting of conflicts of interest in Cochrane reviews of
trials of pharmacological treatments
Roseman M1 , Bero LA2 , Coyne JC3 , Lexchin J4 , Turner EH5 , Thombs BD1
1
McGill University and Lady Davis Institute for Medical Research, Jewish
General Hospital, Canada; 2 University of California, San Francisco, USA;
3 University of Pennsylvania School of Medicine, USA, and University of
Groningen, Netherlands; 4 University Health Network and University of
Toronto, Canada; 5 Oregon Health and Science University and Portland
Veterans Affairs Medical Center, USA
Background: Reporting guidelines recommend that reports of
trials of pharmacological treatments include disclosure of conflicts
of interest (COIs) resulting from pharmaceutical industry funding and
author-industry financial ties. A recent study (Roseman et al., JAMA,
2011) found that meta-analyses published in high-impact biomedical
journals rarely include information on study funding and other COIs
from included trials. The Cochrane Handbook specifies that trial funding
sources be collected in all reviews, and it suggests that potential bias
related to study sponsorship may be incorporated into risk of bias
assessments. Objective: To investigate whether systematic reviews
of pharmacological treatments published in the Cochrane Database of
Systematic Reviews report COIs from included trials. Methods: We
searched the Cochrane Database of Systematic Reviews using the MeSH
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
141
term ‘drug therapy’ to identify all systematic reviews of pharmacological
interventions published in 2010, with content assessed as up-to-date in
2008 or later and including ≥1 RCT. Two investigators independently
extracted data on whether Cochrane reviews reported study funding
sources and other COIs from included trials and, if so, where in the
review such information was reported. Results: Of 151 included
Cochrane reviews, 30 (20%) reported trial funding information for all
included studies; 16 (11%) reported for some, but not all, studies; and
105 (69%) did not report any information. Only 16 reviews (11%)
provided any information on trial author financial ties or employment.
Of 46 reviews that reported trial funding sources for at least some of the
trials reviewed, 29 (63%) noted this in the context of risk of bias, either
in the text and in ‘Risk of bias’ tables. Overall, however, the method
of reporting COIs was inconsistent across reviews. Conclusion: Less
than a third (31%) of Cochrane reviews of pharmacological treatments
report information on study funding from included trials, and only 11%
report on other potential sources of COIs.
P2B255
The placebo response in neuropathic pain trials: It is not
what you think
Selph S1 , Carson S1 , McDonagh M1
1
Oregon Evidence-based Practice Center; Oregon Health & Science
University; USA
Background: The placebo response varies across neuropathic pain
trials making it challenging to demonstrate a treatment effect and
difficult to compare trials of different medications. Objectives: The
primary objective was to determine the study-level characteristics which
predict the placebo response in trials of painful diabetic neuropathy.
A second objective was to identify characteristics that are associated
with changes in relative effect, independent of the placebo response.
Methods: A systematic review of the English-language literature
was performed and fair-good quality, randomized, placebo-controlled
trials measuring ≥50% pain relief from baseline as an outcome,
were identified. Medications included the tricyclic antidepressants, the
serotonin-norepinephrine reuptake inhibitors, gabapentin, pregabalin,
and other anticonvulsants. Meta-regression techniques were employed
to identify predictors of placebo response and predictors of relative
effect while controlling for the placebo response. Covariates included:
drug studied, publication year, trial duration, size and design, gender
distribution, age, duration of diabetes and neuropathy, number of
treatment groups, presence of a washout period, use of additional pain
medications, and rate of patient recruitment. Results: Twenty-five
trials of diabetic neuropathy were included. The best predictor of
placebo rate was whether the study was conducted in the United
States or Europe versus another country or region (p = 0.053). After
adjusting for placebo response, predictors of relative risk were year
of publication and length of trial. For example, assuming a 20%
placebo rate, a 6-week trial published in 1997 is 1.7 times more
likely to demonstrate pain relief versus placebo than a 12-week trial
published in 2005. In contrast to previous research, neither baseline
pain levels nor recruitment rate were found to predict placebo response.
Conclusions: Although the placebo response in neuropathic pain trials
is an important consideration for systematic reviewers, trial duration
and publication year may be equally important.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P2B256
Concordance of reporting of primary outcomes in abstracts
and full text of randomized clinical trials: Is there an
outcome reporting bias?
Ribic C1 , Lamontagne F2 , Johnston B3 , Briel M4 , You J1 , Sun X5 , Busse J6 ,
Bassler D7 , Vera C8 , Al-Shurafa M1 , Mills E9 , Guyatt G1 , Cook D1 , Akl E10
1 McMaster University, Canada; 2 University of Sherbrooke, Canada;
3
University of Alberta, Canada; 4 University Hospital Basel, Basel; 5 Sichuan
University, China; 6 The Institute for Work and Health, Canada; 7 University
Children’s Hospital, Germany; 8 Pontificia Universidad Catolica de Chile,
Chile; 9 Centre for International Health and Human Rights, Canada; 10 State
University of New York at Buffalo, USA
Background: Reporting the results of pre-specified primary outcomes
in the full text of randomized clinical trials (RCTs) but not in the
associated abstract, or reporting primary outcomes in the abstract
that are not reported in the full text, represents a potential source of
publication bias. Objectives: To assess the concordance in reporting
primary outcomes between abstracts and full text of RCTs published
in top medical journals. Methods: We identified all intervention
RCTs published from 2005 to 2007 in the NEJM, JAMA, Lancet,
BMJ, or Annals of Internal Medicine. Ineligible trials were cross over
and cluster designs. Teams of two reviewers appraised the abstract
then, separately, the full text of each selected article using pretested,
standardized forms. Abstractors were blinded to outcomes reported in
full text when screening abstracts. We calculated the concordance of
primary outcome reporting between abstracts and full texts. Results:
Of 709 eligible RCTs, abstracts reported one primary outcome (55.1%,
n = 391), multiple primary outcomes (25.6%, n = 182) or no primary
outcome (19.2%, n = 136). Of abstracts reporting one primary
outcome, 97.4% (n = 381) had concordant reporting in the full
text. We found 2.6% (n = 10) of RCTs had discordant reporting
of single primary outcomes in abstracts compared to full text. Of
these, 6 trials reported a single primary outcome in the abstract and
pre-specified multiple primary outcomes in the full text, while 4 trials
reported a single primary outcome in the abstract and did not specify a
primary outcome in the full text. Conclusions: In this representative
heterogeneous sample of RCTs published in high-impact journals, there
is a high concordance between single primary outcomes reported in
abstracts and those specified in the full text. However, approximately
one-fifth of the abstracts did not specify a primary outcome, potentially
introducing bias in primary outcome reporting.
P2B257
Conflict of interest among clinical practice guidelines
authors
Selph S1 , Holmer H2 , Ogden L2 , Norris S2
1 Oregon Evidence-based Practice Center; Oregon Health & Science
University; USA; 2 Department of Medical Informatics and Clinical
Epidemiology; Oregon Health & Science University; USA
Background: Conflict of interest (COI) among clinical practice
guideline (CPG) authors is an important potential source of bias
for recommendations. A recent report of the Institute of Medicine
(2009) suggested that updated descriptive information is needed on
these conflicts. Objectives: To describe the extent and predictors of
COI among CPG authors in the National Guideline Clearinghouse.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
142
Methods: A random sample of 250 CPGs was obtained on
11/22/2010. For each CPG, data were abstracted on characteristics
of the CPG and author disclosures. Regression techniques were used
to predict the presence of COI among authors. Results: Of 114
CPGs analyzed to date, over a third were developed and funded by
medical specialty societies, 3% were funded by industry and 34% by
government agencies. The median number of authors per CPG was 14;
21% of guidelines had ≥25 authors. The most common focus for CPGs
was disease management, followed by diagnosis and therapy. Author
disclosures were reported in 76% of CPGs. Of these, 30% reported
one or more COIs. An average of two authors (range 0–17) disclosed
any COI for each CPG, the most frequently reported disclosure was
paid consultancy for industry. Although 87% of organizations provided
a COI policy, 24% of guidelines reported no information on COIs.
Journal publication was a significant predictor of a guideline reporting
any authors with a COI (p < 0.0001). The year the CPG was released,
the focus of the CPG, and the country originating the CPG were not
significant predictors of COI reporting. Conclusions: Funding of
CPGs by private specialty organizations is common, conflicts of interest
among CPG authors is frequent, and a significant percentage of CPGs
do not provide disclosures. To be trustworthy, efforts are needed to
achieve accurate, relevant, and readily accessible author disclosures of
conflicts of interest in all clinical practice guidelines.
P2B258
Questionable early stopping: Case study of supplemental
perioperative oxygen and the incidence of surgical site
infection
Tikkinen KAO1 , Garcia-Alamino JM2 , Devereaux PJ3 , Guyatt GH3 ,
Alonso-Coello P4
1 Dept. of Urology and Dept. of Clinical Epidemiology & Biostatistics,
Helsinki University Central Hospital and McMaster University, Finland;
2
Dept of Primary Health Care, University of Oxford, UK; 3 Dept. of
Clinical Epidemiology & Biostatistics, McMaster University, Canada;
4 Iberoamerican Cochrane Centre, Clinical Epidemiology and Public Health
Department, Institute of Biomedical Research IIB Sant Pau, Spain
Background: The potential impact of stopping studies early on
estimates of effect in meta-analysis has been an area of controversy.
We encountered this issue in a systematic review of supplemental
oxygen in the prevention of surgical site infections (SSIs) site infections
which are a common, potentially serious, and costly complication
of surgery. Methods: We conducted a comprehensive search to
identify all published and unpublished randomized controlled trials
(RCTs) evaluating the effect of supplemental perioperative oxygen
on SSIs. Two reviewers independently determined study eligibility
and extracted data. Results: Seven RCTs that included a total of
4,544 patients met eligibility criteria. Studies suffered from moderate
risk of bias. Three out of seven (43%) trials were stopped early:
one for benefit (apparent 54% reduction in relative risk), one for
Figure 1 (P2B258).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
143
harm (apparent doubling of risk), and one for futility. The mean
number of SSIs was 129 (range 5–272) in the non-stopped and 32
(range 27–41) in the stopped trials. Confidence intervals around the
pooled estimate from all trials included a 35% reduction, and a 23%
increase, in relative risk (614 events; RR 0.89 95% CI 0.65 to 1.23),
with substantial heterogeneity (I2 = 68%) (Figure 1). Differences in
patients, interventions, outcome measurement, or risk of bias could
not explain differences in study results. Conclusions: Our systematic
review demonstrates how trials stopped early for benefit or harm tend
to provide misleading results. Stopping early for any reason reduces
the likelihood of accumulating sufficient evidence to provide definitive
results.
P2B259
The quality of trials in quality improvement: Systematic
review of risk of bias in diabetes studies
Ivers N1 , Tricco A2 , Turner L3 , Moher D4 , Grimshaw J5
1 Department of Family Medicine, Women’s College Hospital, University of
Toronto, Canada; 2 Li Ka Shing Knowledge Institute of St Michael’s Hospital,
Toronto, ON, Canada; 3 Ottawa Methods Centre, Ottawa Hospital Research
Institute, Canada; 4 Department of Epidemiology and Community Medicine,
Faculty of Medicine, University of Ottawa, Canada; 5 Clinical Epidemiology
Program, Ottawa Hospital Research Institute, Canada
Background: Of the 356 studies included in seven recent Cochrane
reviews evaluating interventions to improve health care professional
practices, 14.3% were deemed as having a high risk of bias. Despite
repeated calls for reducing the risk of bias in trials, no previous reviews
have assessed whether risk of bias in quality improvement (QI) trials
has changed over time. Objectives: To assess for changes in risk of
bias of QI trials over time. Methods: This was a secondary analysis
of data from a systematic review of 142 trials of QI interventions for
diabetes. Risk of Bias was abstracted using the Cochrane ‘Risk of
bias’ tool. We also grouped studies such that those with high risk of
bias in one of allocation sequence, concealment, blinding, or outcome
reporting were deemed high risk of bias overall. Results: In the
diabetes QI review, 25 trials were published before 2001, 50 from
2001–2005, and 68 from 2006–2010. There was no improvement
over time for 8 of 9 Risk of Bias domains; only ‘similarity in baseline
covariates’ showed a trend for improvement, with the proportion of
trials at high risk of bias decreasing from 47% prior to 2001 to 13% in
those published 2006–2010. Loss to follow up was a common source
of bias in recent trials, with 23% of trials published 2006–2010 judged
high risk of bias. Allocation sequence (58% unclear) and allocation
concealment (56% unclear) were poorly reported. Overall, 14% of the
trials were deemed high risk of bias; this has not improved over time.
Conclusions: Although other methodological reviews of RCTs have
shown improvement over time, this is not reflected in the QI literature.
There is a need to reduce the risk of bias and improve the quality of
reporting of QI trials so that stakeholders have adequate evidence for
action.
P2B260
Development of the McHarm: A tool evaluating validity
of the collection and reporting of harms
Santaguida P1 , Keshavarz H1 , MacQueen G2 , Levine M3 , Beyene J4 ,
Raina P1
1
Evidence-based Practice Centre, Department of Clinical Epidemiology and
Biostatistics, McMaster University, Hamilton, Canada; 2 Dept of Psychiatry,
University of Calgary Regional Clinical Dept Head, Alberta Health Services,
Calgary Health Zone Foothills Medical Centre, Calgary, Canada; 3 Dept.
of Clinical Epidemiology & Biostatistics and Dept. of Medicine, McMaster
University St. Joseph’s Healthcare Hamilton; 4 Department of Clinical
Epidemiology and Biostatistics, McMaster University, Hamilton, Canada
Background: Although the assessment of harms is almost always
included as an outcome in intervention and medical test studies, the
manner of capturing, and reporting harms is significantly different
than the outcomes of benefit. There is a need to consider criteria for
separate assessment of validity specific to harms. Objectives: To
develop criteria for assessing internal validity of collecting and reporting
harms and assess the reliability and validity of the items. Methods: A
systematic review of the literature to identify biases and items within
existing quality assessment tools was undertaken; criteria related to
validity were extracted. A Delphi Consensus exercise with 6 experts was
undertaken to reduce the items. Following consensus, the criteria within
the tool were pilot testing for phrasing and a help file was developed.
Test-retest reliability was undertaken. Construct, and discriminant
validity testing was undertaken. Results: From 5549 citations, 101
full text articles were screened and from this 70 studies had at least
one item. A total of 415 items were generated from the review; after
elimination of redundant items, 126 criteria were included in the Delphi
consensus. Following two rounds of consensus and piloting, 15 criteria
were included in the McHarm. Reliability testing showed acceptable
levels (internal consistency all studies = 0.82; 95% CI 0.79 to 0.85,
and intra-class correlation = 0.95, 95% CI 0.91 to 0.98). Construct
validity with the Chou and Helfand scale was 0.66; discriminant validity
with the Jadad scale was 0.22. Reliability and validity varied between
surgical and pharmacological studies suggesting a potential influence
by drug type. Conclusions: The McHarm contains 15 items related
to assessing the internal validity of capturing and reporting harms in
intervention studies and was shown to be reliable and valid.
Diagnostic test accuracy review methods
Day: 21 Oct (P2B261–P2B271)
P2B261
Reporting of meta-analyses and systematic reviews
of screening tests in China
Tian HL1 , Tian JH1 , Yi K1 , Gou YN1 , Yang KH1
1
Evidence-Based Medicine Centre, Lanzhou University, China
Background: As a developing country with a relative lack of medical
resources, for patients in China it is very important that early screening
and diagnosis is being carried out. Objective: To describe the
current status and production to date of meta-analyses and systematic
reviews of screening tests in China. Methods: We searched three
main medical databases from 1978 to 2011: CNKI (China National
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
144
Knowledge Infrastructure), VIP (a Chinese full text database) and
CBM (China Biomedicine Database). We used the following search
strategy: (diagnostic accuracy OR screen OR sensitivity OR specificity)
AND (systematic review OR meta-analysis). Results: We searched
1496 potentially relevant papers, nine studies met the inclusion
criteria. The first meta-analysis of a screening test was reported
in 2002. Five studies reported reference standard. For sensitivity
and specificity analysis, the summary receiver operating characteristic
(SROC) was the preferred method for analysis in eight studies, only
two studies used QUADAS tool. Five studies reported the sensitivity
analysis and heterogeneity analysis. Meta-disc was the most used
statistical analysis software (44.4% of all studies). Cancer was the
most common target disorder examined (44.4%). Only one study
clearly reported financial conflicts of interest. Two studies were
published in profesional evidence based medicine journals (Chinese
journal of evidence based medicine, Chinese journal of evidence-based
pediatric). Conclusion: The dissemination and production of the
meta-analyses and systematic reviews of screening test research in
China needs to be improved, especially in the methodological quality
and conflict of interest reported. We would like to suggest that
firstly: researchers should conduct systematic reviews of diagnostic test
accuracy according to the Cochrane Screening and Diagnostic Tests
Methods Group’s requirements regarding format as much as possible;
secondly, as a non-native English-speaking country, more monographs
and research about screening tests translated into Chinese is needed.
We used the search terms ‘‘systematic reviews’’ or ‘‘meta-analysis’’
and ‘‘sensitivity’’ or ‘‘specificity’’ or ‘‘accuracy’’ or ‘‘diagnostic’’ in title
and keywords to search the Chinese Biomedical Medicine database
(CBM) (1979–2010), the Wanfang database (2001–2005), the Chinese
National Knowledge Infrastructure (CNKI) (1994–2010) and the full
database of China (VIP) (1989–2010). We excluded duplicate papers,
papers not published in Chinese, conference abstracts, simple literature
evaluation, and protocols. Results: There were 153 studies included.
The number of DTA systematic reviews has increased over the last 10
years in China (Figure 1), but the uptake of new methods and statistical
software lags behind. Pooling sensitivity and specificity or using the
summary receiver operating characteristic (SROC) was the prior method
in 75% of studies. The fixed or random effects models were used in
reviews for 19% of studies. However, the bivariate random effects
model and the hierarchical summary receiver operating characteristic
(HSROC) model were not used in any of the included reviews. At the
same time statistical software developments were limited. Meta-DiSc
and Review Manager were used in 50% and 31% of the reviews
respectively. STATA was only applied in 7% and SAS in 5% of the
reviews. Other statistical software (SPSS, EXCEL, etc.) was used in
31% of the reviews. Conclusions: Even though the quantity of DTA
systematic reviews has increased in the last decade in China, uptake
of up to date statistical methods and software is still lagging behind. It
is necessary to increase the use of up to date statistical methods and
software in diagnostic systematic reviews in China.
P2B262
Methodological developments and statistic software used
in diagnostic systematic reviews in China
P2B263
Epidemic characteristics and quality of meta-analysis
or systematic reviews in screening tests
Chong L1 , Sun R1
1 Evidence-Based Medicine Centre, Lanzhou University, China
Background: With the introduction of Quality Assessment of
Diagnostic Accuracy Studies (QUADAS), the production of diagnostic
test accuracy (DTA) systematic reviews of diagnostics have further
progressed in China. The quantity of reviews has been growing from
2001 to 2010. But the developments of use of methods and statistical
software have only been assessed in small survey reviews so far.
Objectives: To evaluate the status of methodological developments
and statistical software in DTA reviews conducted in China. Methods:
Figure 1 (P2B262). The number of diagnostic systematic reviews in last
decade of China.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Tian HL1 , Gou YN1 , Yi K1 , Tian JH1 , Wang XJ1 , Yang KH1
1
Evidence-Based Medicine Centre, Lanzhou University, China
Background: These years there were a number of developments in the
methodology used in systematic reviews or meta-analysis of screening
tests. But the epidemic characteristics and quality of meta-analysis
or systematic reviews in screening tests has not been evaluated.
Objectives: To explore the epidemic characteristics and quality of
meta-analysis and systematic reviews in screening tests. Methods:
Eight databases (including MEDLINE and EMBASE) were searched.
Outcome analysis methods, methodological quality, statistical analysis
software and conflicts of interest reported were evaluated. Results:
The searches yielded 4,726 relevant articles. After application of
inclusion criteria, 98 studies were included. Cancer was the most
common category of target disorder (33% or studies), followed by
depression (6%) and alcohol abuse disorders (6%). The main methods
used for outcome analysis were sensitivity (96%; 94/98 of included
studies), specificity (96%; 94/98), positive predictive value (57.1%;
56/98), negative predictive value (52%; 51/98), positive likelihood ratio
(30.6%; 30/98), negative likelihood ratio (29.6%; 29/98), diagnostic
odds ratio (32.7%; 32/98), pre-test odds (8.2%; 8/98), post-test
odds (6.1%; 6/98), receiver operating characteristic curves (SROC:
8.2%; 8/98). Statistical analysis reported were: fixed-effects model
(8.2%; 9/98 of included studies), random-effects model (15.7%;
19/98), publication bias (5.6%; 9/98), bayesian bivariate hierarchical
mode (6.1%; 6/98), sensitivity analysis (22.4%; 22/98), heterogeneity
analysis (35.7%; 35/98). The included reviews assessed methodological
quality through QUADAS (30.6%; 30/98 of included studies), not
reported (56.1%; 55/98), and others (13.3%; 13/98) (Figure 1), and
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
145
Figure 1 (P2B263).
used the following statistical softwares: Meta-DiSc (19.4%; 19/98 of
included studies), STATA (14.3%; 14/98), SAS (4.1%; 4/98), RevMan
(6.1%, 6/98), others (13.3%; 13/98). Conclusions: The number of
meta-analyses and systematic reviews on screening tests has increased
over time (Figure 2). Sensitivity, specificity, positive predictive value and
negative predictive value remain the preferred methods for analysis.
Although quality assessment in meta-analyses has improved with the
introduction of QUADAS in 2004, most screening test reviews still do
not use it.
P2B264
Addressing methodological challenges in evaluating
diagnostic tests: Combining the GRADE approach and the
RAND/UCLA Appropriateness Method to produce clinical
recommendations
Ballini L1 , Vignatelli L1 , Negro A1 , Maltoni S1 , Grilli R1
1 Agenzia Sanitaria e Sociale Regionale Emilia-Romagna – Italy
Background: Developing recommendations for appropriate use
of diagnostic tests presents serious challenges from defining the
clinical rationale for the new test and its position in the current
diagnostic-therapeutic pathways to dealing with the lack of evidence on
impact of test on patient-important outcomes. Recent methodological
breakthroughs (1,2) provide valuable insights to address these
challenges. Objectives: develop a new methodology (transparent and
encompassing clinical reasoning) which combines the GRADE approach
for grading quality of evidence and strength of recommendations with
the RAND method of consensus; test the methodology to produce
recommendations on the use of Positron Emission Tomography (PET)
in oncology. Methods: two multidisciplinary panels of experts
were convened to produce recommendations on PET’s use in four
types of cancer (breast, esophageal, lung, and colo-rectal cancer).
Answerable research questions were developed by positioning and
comparing PET against existing diagnostic tests (replacement, add-on,
triage). Clinical rationale was based on capacity of new test to
modify initial diagnosis, subsequent change in therapeutic approach
and resulting clinical benefits for patients. Results from the systematic
review of literature on PET’s diagnostic accuracy were analysed and
presented using GRADE’s levels of evidence. Lack of direct evidence
on patient-important outcomes was bypassed using GRADE’s logic
for eliciting experts’ judgement on clinical consequences of testing
positive and testing negative. Results of two rounds of votes
on level of appropriateness were analysed using the RAND/UCLA
Appropriateness Method. Results: a total of 34 a voting forms for each
clinical question was generated via the above described methodology;
recommendations for each cancer were produced by the panels in just
two meetings; levels of agreement and disagreement were registered
and reported. Conclusions: 37 experts engaged in producing clinical
recommendations managed to work through the GRADE’s approach
and delivered 34 recommendations in a very short period of time.
References
1. Bossuyt PM, Irwig L, Craig J, Glasziou P. Comparative accuracy:
assessing new tests against existing diagnostic pathways. BMJ. 2006;
332(7549):1089–92.
2. Schünemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R,
Vist GE, Williams JW Jr, Kunz R, Craig J, Montori VM, Bossuyt P,
Guyatt GH; GRADE Working Group. Grading quality of evidence and
strength of recommendations for diagnostic tests and strategies. BMJ.
2008;336(7653):1106–10.
P2B265
Multivariate meta-analysis of diagnostic test accuracy
studies with multiple thresholds per study
Riley R1 , Guha A1 , Biswas A2 , Smith-Bindman R3 , Deeks J1
1 University of Birmingham, UK; 2 Indian Statistical Institute, India;
3
University of California, USA
Background: A typical meta-analysis of diagnostic test accuracy
studies uses a single two by two table per study. A bivariate
Figure 2 (P2B263).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
146
random-effects meta-analysis can synthesise these tables, utilising
their binomial distribution within-studies. For tests on a continuous
scale, each study’s two by two table corresponds to a threshold value
dichotomising the test into ‘positive’ and ‘negative’ groups. Sometimes
studies report multiple two by two tables, relating to multiple threshold
values, that collectively follow a multinomial within-study distribution.
However researchers often ignore this multinomial structure, and
meta-analyse thresholds separately assuming a binomial dsitribution.
Aims & Methods: Hamza 2009 meta-analysis studies with multiple
thresholds results and account for their multinomial structure; they focus
on complete data (all studies provide all thresholds) and estimating
ROC curves. In this poster we evaluate their approach given missing
data (some studies provide an incomplete set of thresholds) and focus
on sensitivity and specificity estimates. We apply the method to 30
studies of endovaginal ultrasound for detecting endometrial disease,
with up to 7 different thresholds per study and missing data. Results:
We show that, compared to a separate meta-analysis of each threshold
assuming a binomial distribution, the multinomial approach produces
different summary estimates for each threshold (with sensitivity or
specificity changed by up to 10%) and smaller standard errors (reduced
by up to 45%). For example, for a threshold of 8 mm the summary
sensitivity was 85% (binomial) and 79% (multinomial), with standard
errors of logit-sensitivity of 0.17 (binomial) and 0.09 (multinomial).
Conclusions: When studies provide test accuracy results for multiple
thresholds, these thresholds should be jointly meta-analysed accounting
for their multinomial structure; otherwise statistical precision is lost
and clinical conclusions may be misleading.
Reference
1. Hamza TH, Arends LR, van Houwelingen HC, Stijnen T. Multivariate
random effects meta-analysis of diagnostic tests with multiple
thresholds. BMC Medical Research Methodology 2009;9:73
P2B266
Overinterpretation of results in diagnostic test accuracy
studies: evidence of ‘‘spin’’
Ochodo E1 , de Haan M1 , Reitsma J1 , Lotty H2 , Bossuyt P1 , Leeflang M1
1 Academic Medical Centre, Amsterdam, Netherlands; 2 Dutch Cochrane
Centre, Netherlands
Background: Overinterpretation of the results by authors in order to
make the interventions look favourable is referred to as ‘spin’. This
has been shown to be common in randomized trials but may also play
a role in diagnostic accuracy studies. The clinical use of tests based
on inflated conclusions could lead to unnecessary testing and may
trigger physicians to make incorrect clinical decisions, compromising
patient safety. Objectives: To explore the mechanisms behind ‘spin’
in diagnostic test accuracy studies and to estimate their prevalence in
journals with a high impact factor. Methods: We searched MEDLINE
for diagnostic test accuracy studies published between January and
June 2010 in journals with an impact factor of 4 or higher. A random
selection was evaluated with a structured data extraction form by two
authors independently. Disagreement was resolved by consensus or
by a third party. Results: Our search yielded 420 eligible articles
of which we selected 85. The interrater agreements were 81% and
100% before and after discussion. Of the 74 articles included in
the analysis, 66 (89%) contained a form of spin. The mechanisms
behind spin included: not stating a prior study hypothesis (85%),
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
reporting positive test recommendations not in line with the test
accuracy measures (12%), basing test recommendations on results
from sub-group analysis only (12%), conclusions that differed from
the aim of the study (12%), an overoptimistic abstract, with selective
reporting of favourable results only, or stronger test recommendations
compared to the main text (45%). Conclusions: Overinterpretation
of results of diagnostic accuracy studies is common. We hope that
highlighting the mechanisms behind spin will enable peer reviewers
correctly sieve overoptimistic reports of diagnostic accuracy studies and
encourage investigators to be more clear in designing, more transparent
in reporting, and more stringent in interpreting test accuracy studies.
P2B267
What measures of accuracy should we present in DTA
reviews?
Jameson CM1 , Whiting PF1 , Burke M1 , Ben-Shlomo Y1 , Sterne JAC1
1 School of Social and Community Medicine, University of Bristol, UK
Background: Diagnostic test accuracy (DTA) systematic reviews are
standard ways of summarising research evidence. DTA reviews may
change clinical practice on an individual level, or their findings may
be incorporated into guidelines to influence practice at the health
service level. One factor that may affect a review’s potential to
influence clinical practice, and the way clinicians subsequently use
tests, is how clearly the results are presented and how easy they
are to interpret. Objectives: To review available evidence regarding
health professionals’ understanding of diagnostic information to inform
decisions on how best to present the results of DTA reviews. Methods:
MEDLINE, EMBASE and PsycINFO were searched until March 2011,
experts were contacted, citation searches of key papers were conducted,
and reference lists were screened. Primary studies were included
that reported on any outcomes relating to health professionals’
understanding of diagnostic test accuracy which was summarised
as sensitivity, specificity, likelihood ratios, predictive values, accuracy
or ROC curves. Due to the heterogeneity between studies in terms of
design, health professionals, and measures of accuracy investigated,
a narrative synthesis was used to combine results. Results: To
date 12 studies have been included; the review is ongoing. Initial
results suggest that sensitivity and specificity are generally poorly
understood by clinicians and are often confused with predictive values.
Clinicians tend to overestimate the impact of a positive test result on the
probability of disease and this overestimation increases with decreasing
pre-test probabilities of disease. The most informative measures for
clinicians may be estimates of the post-test probability of disease,
which can be presented as a range corresponding to different pre-test
probabilities. Conclusions: Conclusions will inform recommendations
for presenting estimates of test accuracy in DTA reviews.
P2B268
Discrepancy in relative test performance due to modelling
strategy in comparative diagnostic meta-analysis: A case
study
Takwoingi Y1 , Abba K2 , Garner P2 , Deeks J1
1 University of Birmingham, UK; 2 University of Liverpool, UK
Background: Hierarchical models for diagnostic test accuracy
meta-analysis include study-specific random effects to account for
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
147
between-study heterogeneity. In a comparative meta-analysis, equality
of variance parameters can be assumed for different tests whilst
allowing other model parameters to depend on each test. Objectives:
To demonstrate discrepancy in summary accuracy measures when
differences in heterogeneity exist between studies of different tests
and equal variances are assumed across tests. Methods: A Cochrane
review on rapid diagnostic tests (RDTs) for Plasmodium falciparum
malaria included an assessment of the relative accuracy of different
types of tests. Type 1 and Type 4 tests were compared by including
parameters in the hierarchical summary ROC (HSROC) model to allow
each type to have a different threshold, accuracy and summary ROC
shape. The impact on the variability of random effects of accuracy
and threshold was also investigated, and separate variance parameters
included if required based on likelihood ratio (LR) tests. Summary
sensitivities and specificities were derived from the models. The
analyses were done using SAS Proc NLMIXED. Results: Sixty-five
study cohorts evaluated Type 1 tests and were more heterogeneous
than the 16 that assessed Type 4 tests. Between models, little
or no difference in specificities for either test type was observed
(table 1). The sensitivity of Type 4 tests derived from the model with
equal variances (model 1) did not reflect the data, and differed from
that of the model with separate variances (model 2) by 5.5%. The
difference in sensitivity between Type 1 and Type 4 tests was statistically
significant (p < 0.001) in model 1 but not in model 2 (p = 0.20).
Conclusions: Modelling assumptions can affect conclusions of a
comparative meta-analysis. In this case, the more complex and
appropriate assumptions led to more conservative differences. The
effect of each test on model parameters should be investigated when
feasible.
P2B269
Can generalizability be considered in systematic reviews
of diagnostic test accuracy?
Scheibler F1 , Janssen I1 , Schröer-Günther M1 , Sauerland S1
1 Institute for Quality and Efficiency in Health Care, Germany
Background: Most diagnostic tests consist of variable components,
and very often the technical quality of the diagnostic device or the
experience of medical staff play an important role for test accuracy.
Therefore these factors should be considered carefully when results of
studies of diagnostic test accuracy (DTA) are synthetized in systematic
reviews (SRs). To our knowledge no systematic and validated method
for extracting and assessing variations in the generalizability of
diagnostic tests has been published so far (compare section 9.2.2
in the Cochrane Handbook on DTA). Objectives: To develop a
multi-item checklist on the generalizability of DTA studies. Methods:
Generalizability aspects were retrieved from the Cochrane Handbook,
methodological papers on quality assessment of diagnostic studies and
method sections of identified SRs in an overview of SRs on positron
emission tomography. An extraction sheet was developed consisting
of 7 items and a summary assessment item and tested in 7 SRs with
160 included primary DTA studies. Results: Due to the comprised
format and the clear-cut items the instrument proved to be feasible for
application within a systematic review. Limitations to be considered
are: i) generalizability differs with different health care settings; ii)
generalizability cannot be proven, but it is possible to find evidence for
non-generalizability; iii) in several aspects there seems to be a trade-off
between generalizability and internal validity (e.g. blinding); and iv) as
only few studies were categorized as non-generalizable in our feasibility
study, sensitivity analyzes did not show differences in diagnostic
accuracy depending on generalizability. Conclusions: Aspects of
generalizability should be assessed in a systematic and transparent way,
separately from aspects of bias. We demonstrate the current state of our
instrument and very much appreciate comments and suggestions for its
improvement.
P2B270
Methodological filters for the identification of delayed
cross-sectional studies
Noel-Storr A1 , Beecher D2
1
(CDCIG) University of Oxford, United Kingdom;
Sclerosis Group, Italy
2
Cochrane Multiple
Background: Up to now no published methodological filter designed
for the retrieval of diagnostic test accuracy (DTA) studies has proved
sensitive enough for use within a Cochrane DTA systematic review. A
recent study tested the hypothesis that normal cross-sectional studies
should be treated differently from delayed cross-sectional studies
Table 1 (P2B268). Variance parameters, summary sensitivities and specificities for Type 1 and Type 4 RDTs from models with and
without separate variances for the random effects.
Variance parameters
Type 1 RDTs
Type 4 RDTs
Summary estimates
Type 1 RDTs
Type 4 RDTs
Ratio
Model 1
Model 2
Common shape and equal variances
Common shape and separate variances
Accuracy
Threshold
Accuracy
Threshold
2.73
2.73
0.88
0.88
2.53
1.69
0.84
0.67
Sensitivity
Specificity
Sensitivity
Specificity
95.5 (93.9, 96.7)
86.0 (81.3 89.7)
0.90 (0.87, 0.94); p < 0.001
95.0 (93.1, 96.1)
98.8 (98.3, 99.2)
1.04 (1.03, 1.05); p < 0.001
94.8 (93.1, 96.1)
91.5 (84.7, 95.3)
0.96 (0.91, 1.02); p = 0.20
95.2 (93.2, 96.7)
98.7 (96.9, 99.5)
1.04 (1.02, 1.06); p < 0.001
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
148
(longitudinal analyses). Objectives: i) further test the hypothesis
that normal cross-sectional studies and delayed cross-sectional studies
are generally described differently in the published literature, ii) refine
an unpublished methodological filter designed for identification of
delayed cross-sectional studies (this will be done in MEDLINE (Ovid)),
and iii) develop an equivalent filter for use in PubMed. Methods:
An unpublished filter will be further validated by expansion of the
data set. Key elements of reports of potential studies for inclusion
within Cochrane DTA reviews that focus on longitudinal diagnosis and
prediction will be entered into textual analysis software so as to refine
the existing unpublished filter. The filter will then be further tested by
a large dataset of reports of potentially relevant studies from existing
literature on methodological filters. Finally, a PubMed equivalent filter
will be developed. Results: The results will show the new filter’s
sensitivity, specificity, precision and accuracy and will be presented at
the Cochrane Colloquium in Madrid, October 2011. Conclusions:
DTA studies generally fall into two camps – normal cross-sectional
studies (e.g. looking at the accuracy of a new test to see if someone is
pregnant or not) and delayed cross-sectional studies (e.g. a new test
to see whether someone will develop symptomatic dementia from a
‘pre-dementia’ state). The filters published so far treat these studies
as one. The literature therefore condemns these filters based on
potentially invalid criteria.
found. Study characteristics that may be influential in one comparison,
may have no influence at all in another comparisons.
Knowledge translation and communicating
evidence
Day 22 Oct (P3A272–P3B345)
P3A272
What kind of evidence can be found in incident reporting
systems? Systematic evaluation of incident reports to
identify risks using syringe pumps
Rohe J1 , Hahnenkamp C1 , Dichtjar T2 , Sangiuno H. A1 , Thomeczek C1 ,
Schleppers A2 , Fishman L1
1 German Agency for Quality in Medicine (AQuMed/AEZQ); 2 Professional
Association of German Anaesthesiologists, Germany
Background: Incident reporting systems in medicine enable health
care professionals to anonymously report patient safety incidents so
that others can learn and patient safety can be strengthened. The
German system, www.CIRS-AINS.de, is widely established in anesthesia
and encompasses 1606 anesthesia-specific reports. Objectives:
Learning opportunities of individual reports can be published in
case reports or alerts, but also in systematic evaluation. The
P2B271
method presented demonstrates the possibility to systematically analyze
Adjusting for indirectness in comparative test accuracy incident reports. Using the example of syringe pumps, reports were
meta-analyses
evaluated to show the risks of their use. Methods: Titles of
1606 incident reports were searched by the term ‘syringe pump’
1
2
2
1
Leeflang MMG , Di Nisio M , Rutjes AWS , Zwinderman AH , Bossuyt
1
(German ‘Perfusor’), 54 reports were identified. The reports were
PMM
1 University of Amsterdam, Netherlands; 2 University G. d’Annunzio Foundation, classified according to the phase (A-D) of the medication process in
which the incident originated. Within these phases 13 areas of risk
Italy
were identified. Results: Phase and area of risk. A. Prescribing:
Background: The accuracy of a diagnostic test should be compared to Discontinue medication when indicated; B. Transcribing/Documenting:
Communication of prescription; C. Dispensing: Preparation of syringe
the accuracy of its alternatives. Direct comparisons of tests, in the same
for pump; D. Administering: Programming/labelling of syringe pump,
patients and against the same reference standard, offer the most valid
Connection of lines, Lines intact, Power supply of syringe pump, Alarms
study design, but are not always available. Comparative systematic
reviews are therefore bound to rely on indirect comparisons. As the of syringe pump, Change of syringe, Syringe pump works correctly,
Syringe pumps and magnetic resonance imaging, Mounting of syringe
results from these comparisons can be biased, we investigated ways
pump; E. Monitoring: Monitoring of vital signs. The 13 areas of
to correct for indirectness. Methods: From a large systematic review
risk encompassed one to eight reports and exhibited different aspects
about the accuracy of D-Dimer testing for venous thromboembolism, we
within the area of risk. Conclusions: Reports from incident reporting
selected those comparisons between two assays that contained three
systems are one possibility to make experiences of individuals available
or more direct comparisons and four or more indirect comparisons
for systematic learning. Because not all safety incidents are reported,
or single assay studies. Each comparison was analyzed using the
data is not representative. The method presented can help–through
bivariate random effects meta-regression model with assay-type and
summation of individual evidence—to systematically identify areas of
directness as covariates in the model. In comparisons with a significant
risk during the medication process or within other areas of health
effect of directness on sensitivity or specificity, we included the
following study features to correct for these differences: referral filter, care. The identified areas of risk can be used to develop strategies to
diminish risk and strengthen patient safety.
consecutive enrolment, time-interval, type of reference standard and
verification. Results: Of the seventeen eligible comparisons, five
showed a significant effect of direct comparisons on logit-sensitivity or
P3A273
logit-specificity. A first analysis showed that in three of these analyses,
How to obtain NNT from Cohen’s d: Comparison of two
the addition of at least one of the study features decreased the impact
methods
of directness as a covariate. For example, in one analysis directness
Furukawa TA1 , Leucht S2
had a significant effect on specificity, but addition of time-interval or
1 Kyoto University, Japan; 2 Technische Universität München, Germany
reference standard removed this effect (from P = 0.048 to P = 0.22
resp. P = 0.12). Further analyses will be presented at the Colloquium.
Conclusions: Adjusting the effect of directness for study features
Background: Many indices of size of treatment effect (effect size:
seems to be possible in some instances, but no systematic effects were
ES) are found in the literature. The preferred one in evidence-based
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
149
medicine is the number needed to treat (NNT) while the most common
is Cohen’s d, when the outcome is continuous. There is confusion
about how to convert Cohen’s d into NNT. Methods: We conducted
meta-analyses of individual patient data from 10 randomized controlled
trials of second generation antipsychotics for schizophrenia (n = 4278)
to produce Cohen’s d and NNTs for various definitions of response,
using cutoffs of 10% through 90% reduction on the symptom severity
scale. These actual NNTs were compared with NNTs calculated from
Cohen’s d according to Kraemer’s method (Biological Psychiatry, 2006;
59:990–996) and Furukawa’s method (The Lancet, 1999; 353:1680).
Results: NNTs from Kraemer’s method overlapped with the actual
NNTs in 56%, while those based on Furukawa’s method fell within
the observed ranges of NNTs in 97% of the examined instances.
For various definitions of response corresponding with 10% through
70% symptom reduction where we observed a non-small number of
responders, the degree of agreement for the former method was at a
chance level (ANOVA ICC of 0.12, p = 0.22) but that for the latter
method was ANOVA ICC of 0.86 (95%CI: 0.55 to 0.95, p < 0.01).
Conclusions: Furukawa’s method allows more accurate prediction of
NNTs from Cohen’s d. Kraemer’s method gives a wrong impression
that NNT is constant for a given d even when the event rate differs.
P3A274
Development of a methodological framework for
systematic review-based evidence briefings
Chambers D1 , Wilson P1
1
CRD, University of York, UK
Background: As part of a larger research project, we are providing
a knowledge translation service to National Health Service (NHS)
decision-makers in the Leeds–Bradford area of England, translating
existing sources of synthesised and quality-assessed evidence, primarily
systematic reviews, into actionable messages. The service is particularly
aimed at commissioners of healthcare services but in principle could
cover a wide range of decisions in clinical effectiveness and service
delivery and organisation. Methods: We developed a framework
for production of evidence briefings based on a scoping review of
existing resources (recently published in the Milbank Quarterly ) and
our experience of evaluating the evidence base on models of service
provision for young people with eating disorders. Where appropriate,
we drew on the SUPPORT tools for evidence-informed policymaking.
Results: The framework assumes that briefings will be produced
in response to decision-makers’ requests and that the process will
involve active collaboration between decision-makers and researchers.
It covers development of the research question (including a checklist to
be completed at an initial meeting); identification of relevant systematic
reviews and economic evidence; relating the evidence to the local
setting; and consideration of issues of health equity and implementation
of any changes to practice or service delivery. Evaluation of use,
usefulness, and impact is an important part of the process. Results and
implications for further development and research will be presented
at the Colloquium. Conclusions: Production of evidence briefings is
labour-intensive but appears promising for increasing the impact of
systematic reviews and promoting evidence-informed decision-making.
Development of a standard approach, together with a more centralised
provision of this type of service, could offer economies of scale and
make use of national expertise to support the needs of decision-makers
at the local level.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P3A275
Cochrane reviews: Beyond implications for research
González U1
1 Research Unit for Evidence-based Dermatology, Hospital Plató, Barcelona,
Spain
Background: Cochrane systematic reviews are an ample source
of suggestions and information for further health-care research. At
present the section on ‘Implications for research’ specifically comments
on the need and nature of further research that would be most
desirable. Objective: To explore how Cochrane reviews can further
help to make suggestions to assist researchers to plan and conduct
useful clinical trials. Methods: Two papers of Cochrane reviews
are analyzed (González U, Pinart M, Reveiz L, et al. Designing and
reporting clinical trials on treatments for cutaneous leishmaniasis. Clin
Infect Dis 2010;51:409–19; and Gonzalez U, Whitton M, Eleftheriadou
V et al. Guidelines for designing and reporting clinical trials in
vitiligo. Arch Dermatol – in press-). This is a new initiative for helping
authors conduct high quality trials by elevating the importance of
research gaps and providing specific research guidelines for new trials.
Results: The structure of this new type of article is described and
recommendations on optimum designs for RCT in every specific area
based on the evidence synthesis are made. The first part includes
the known sources of bias from existing randomised controlled trials
such as selection bias, blinding assessment, attrition bias, and the
PICO framework. The second part provides specific recommendations
for designing and reporting the thematic trials and it is structured
according to the study question, the study design and reporting details.
Conclusions: Development of specific clinical research guidelines and
recommendations based in Cochrane reviews could improve and inform
future efforts for new thematic clinical trials and other initiatives as
outcome research. This could be a major need in the production of
the best evidence in all health areas as well as a unique type of work
within the medical literature and the Cochrane Library.
P3A276
Building a database of validated pediatric outcomes
Adams D1 , Sivakumar L1 , Nasser H1 , Surrette S1 , Hartling L1 , Vohra S1
1 University of Alberta, Canada
Background: Pediatric populations have increasingly been included
in clinical research that relies on the availability and use of appropriate
outcome measurement tools. Objectives: i) To develop an inventory
of valid and reliable pediatric outcome measurement tools and ii)
to identify gaps in outcomes reporting in publications of pediatric
randomized controlled trials (RCTs). Methods: The top 6 general
medicine journals and top 4 pediatric journals were searched for
pediatric RCTs published since 2000. Two independent reviewers
conducted screening and data extraction. Variables to be extracted
included: journal type (general versus pediatric), population age,
sample size, condition of interest, intervention, control, primary
outcome, outcome measurement tools, and information on
psychometric testing. Results: Searches identified 2229 unique
references. Most (70%) were identified from pediatric journals, with
ages ranging from 33 weeks gestation to 21 years. The most
common condition studied was psychological disorders (n = 12). A
single intervention was tested in 57% of trials and more than one
intervention was tested in 22 trials (36%). Half (48%) reported one
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
150
primary outcome, while 31% did not identify a primary outcome, and
13% identified more than one. We identified 79 different scale or
questionnaire-type measurement tools, from 31 studies, that measured
outcomes in 21 different conditions. For 41% of these 79 tools, authors
provided information on psychometric properties and included relevant
citations in 28% for these properties. The most commonly reported
properties were reliability (59%) and validity (41%). Conclusions:
A wide variety of pediatric outcome measurement tools are in use
by researchers. Psychometric properties of measurement tools are
inconsistently reported in pediatric RCTs, thus it is unclear to readers if
the tools are of high quality. Developing a comprehensive database of
validated pediatric outcome measures may facilitate use of high quality
pediatric research.
P3A277
Patient values and preferences for decision making
in antithrombotic therapy: A systematic review
MacLean S1 , Mulla S1 , Jankowski M2 , Akl E3 , Vandvik P4 , Ebrahim S5 ,
McLeod S6 , Bhatnagar N7 , Guyatt G1
1 Department of Clinical Epidemiology and Biostatistics, McMaster
University, Canada; 2 Department of Internal Medicine, Faculty of Medicine,
Jagiellonian University Medical College, Krakow, Poland; 3 Department of
Clinical Epidemiology and Biostatistics, McMaster University, Canada;
Department of Medicine, State University of New York at Buffalo, NY, US;
4 Norwegian Knowledge Centre for the Health Services, and Department
of Medicine Gjøvik, Innlandet Hospital Trust, Norway; 5 Department of
Biostatistics and Clinical Epidemiology, McMaster University, Canada;
6 The Division of Emergency Medicine, Department of Medicine, Schulich
School of Medicine & Dentistry, The University of Western Ontario; 7 Health
Sciences Library, McMaster University
Background: Development of clinical practice guidelines involves
making trade-offs between desirable and undesirable consequences
of alternative management strategies. Although the relative value
of health states to patients should provide the basis for these
trade-offs, few guidelines have systematically summarized the relevant
evidence. Objectives: In preparation for the 9th iteration of the
American College of Chest Physicians anti-thrombotic guidelines, we
conducted a systematic review relating to values and preferences of
patients considering anti-thrombotic therapy. Methods: We included
studies examining patient preferences for alternative approaches
to anti-thrombotic prophylaxis and studies that, in the context of
anti-thrombotic prophylaxis or treatment, examined how patients
value alternative health states and experiences with treatment. We
conducted a systematic search and compiled structured summaries of
the results. Steps in the process that involved judgment were conducted
in duplicate. Results: We identified 48 eligible studies. Sixteen dealt
with atrial fibrillation, 5 with venous thromboembolism, 4 with stroke or
myocardial infarction prophylaxis, 6 with thrombolysis in acute stroke or
myocardial infarction, and 17 with burden of anti-thrombotic treatment.
Findings included: 1) Values and preferences for treatments and for
health states consistently varied appreciably between individuals. 2)
Factors that influence apparent preferences for health states include
previous exposure to treatment, previous exposure to health states,
and the occurrence of a health state as an adverse effect of treatment.
3) Varying methods of preference elicitation lead to different choices.
4) A reasonable trade-off to assume between stroke and bleeds would
be a ratio of disutility of stroke to bleeds ranging from 2:1 to 3:1
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
(although there are troubling inconsistencies across studies). 5) For
most patients, warfarin therapy does not have important negative
impacts on quality of life, although there exists a minority for whom it
does. Conclusions: Systematic reviews such as ours should become a
standard for clinical practice guidelines.
P3A278
Publication of IPD meta-analyses in The Cochrane
Database of Systematic Reviews: Potential barriers
and opportunities
Rydzewska L1 , Tierney J1 , Clarke M1 , Stewart L1 , Rovers M2
1 IPD Meta-analysis Methods Group, UK; 2 IPD Meta-analysis Methods
Group, Netherlands
Background: Individual participant data (IPD) systematic reviews
are usually non-Cochrane reviews based on ‘raw data’ and are
produced by collaboration of a group of original researchers. IPC
are resource-intensive, but they can improve the quality of the data
and analyses, providing more definitive findings than are possible with
summary data, and are described as the ‘gold standard’ for systematic
reviews. Therefore, conversion to Cochrane reviews is encouraged.
Their collaborative nature with multiple authors prior to publication,
however, may difficult compliance with certain procedures, style, and
format recommendations for Cochrane reviews. Objectives: To
assess whether members of the IPD Meta-analysis Methods Group
(IPDMAMG) convert their IPD reviews to Cochrane reviews, and if
not, to describe the reasons why. Methods: Experiences of a small
group responsible for IPD reviews in cancer, who have published in
CDSR following publication in other medical journals, were reviewed
to collect members’ views. The survey across the IPDMAMG, was
then extended. Results: The initial survey highlighted the following
challenges: time taken to re-format reviews; obtaining permission
from the original journal; obtaining approval from all authors for
changes; providing new author declarations; and completing the risk
of bias tool retrospectively. Experiences varied depending on each
Cochrane Review Group’s (CRG’s) awareness of what IPD reviews
involve and their individual procedures. Results from the full survey
will be presented, with recommendations for facilitating the conversion
process, relevant to both CRGs and IPD review authors. Conclusions:
If more IPD reviews were converted to Cochrane reviews, this would
increase the amount of high-quality evidence in CDSR and the impact
of The Cochrane Collaboration’s output. It would reduce unnecessary
duplication of effort by making use of work already done for the IPD
review, with IPD authors benefitting from increased accessibility to
their findings and more opportunities to update reviews as new data
become available.
P3A279
Strategies and measures to improve the quality of clinical
trials in China
Wu T1 , Li Y1
1 Chinese Cochrane Centre, Chinese Clinical Trial Registry, West China
Hospital, Sihuan University
Background: The number of published ‘randomized controlled trials
(RCT)’, as reported by their authors, increased quickly in past years in
China. Authors of this paper, found that only about 7% of those trials
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
151
were authentic RCTs and that only about 0.2% were submitted to an
ethics review committee. Objective: Aiming to improve the quality of
clinical trials in China, a series of strategies and measures have been
developed and are being implemented by the authors. Methods:
1. Promoting transparency of clinical trials in three ways: prospective
registration of the trial before recruting participants, making the trial
process transparent and promoting reporting on the quality of the trial.
2. Improvement ethics review committees and increase submission
to these bodies. Results: A clinical trials quality control system has
been established and is functioning in China: 1. The Chinese Clinical
Trial Registry (ChiCTR) was established in October, 2004 becoming a
Primary Registry for WHO’s ICTRP in July, 2007. Registration in ChiCTR
is free. Up to 30 March, 2011, more than 1200 trials have been
registered. 2. The ‘Chinese Clinical Trial Registration and Publication
Collaboration (ChiCTRPC)’ was initiated in June 2006 amongst ChiCTR
and 48 key Chinese medical journals aiming to promote a policy that
would make trial registration mandatory in China. 3. The Clinical
Trial eCRF System was established to offer management of raw data
of clinical trial on a voluntary basis. In this system the case records
and the real-time results of the trials are sent to a central deposit, the
raw data and calculated data, except the private information of the
participants, can be publicly accessed. 4. The Chinese Ethics Committee
for Registering Clinical Trials (ChiECRCT) was established in 2008 to
review trial registration. 5. Cooperation with the CONSORT group and
EQUATOR has been established to promote the use of the CONSORT
guidelines in China. Conclusions: A comprehensive management
system of clinical trials in China has been established by the authors.
P3A280
Investigating determinants of experts’ judgement on
appropriateness for diagnostic tests when using the
‘Grading of Recommendations Assessment, Development
and Evaluation (GRADE)’ method for presenting evidence
Background: Within a guideline production programme on
appropriate use of Positron Emission Tomography (PET), a new
methodology—combining the GRADE approach for grading quality
of evidence and strength of recommendations with the RAND/UCLA
Appropriateness Method—was developed and tested. Objectives:
Evaluate whether and to what extent information provided to experts of
multidisciplinary panels determined their decision on appropriateness
of the diagnostic test. Methods: 37 experts, convened in two
multidisciplinary panels with the mandate to define appropriate use of
PET for 34 clinical indications in four types of cancer, were provided
with a voting form for each clinical question containing a range of
information (Table 1). Panellists individually rated the importance of
outcomes and level of appropriateness. Statistical analyses included:
correlation of each variable (sensitivity and specificity—favouring PET or
comparator—level of evidence (LoE), pre-test probability, importance
score for the four patient-important clinical outcomes) with rating
of appropriateness of all panellists for all clinical questions; multiple
linear regression modelling the relationship between the variables and
appropriateness rating for all panellists; multiple logistic regression
modelling the relationship between the variables and appropriateness
rating for each panel. Preliminary Results: correlation coefficients
of all variables with rating for appropriateness are reported in Table 2.
Linear regression analysis showed that the ten variables accounted
for 63.4% of variability of appropriateness rating; among them level
of evidence explained 36.5% of variability. The remaining analyses
will be completed in due time and final results will be presented.
Conclusions: LoE seems to play a major role in explaining the rating
of appropriateness. When the analysis is completed we expect to:
better quantify the relationship of all chosen explanatory variables with
the rating of appropriateness (dependent variable), establish to what
extent panellists used the information provided to make judgements,
and suggest explanation for would-be unpredicted correlations.
Ballini L1 , Negro A1 , Vignatelli L1 , Maltoni S1 , Grilli R1
1 Agenzia Sanitaria e Sociale Regionale Emilia-Romagna, Italy
Table 1 (P3A280). Information provided in voting form for each clinical question.
clinical rationale in support of FDG-PET
clinical effectiveness of therapeutic approach resulting from test results
suggested role of FDG-PET in diagnostic pathway (replacement; triage; add-on)
evidence of change in management – when available
pre-test probability
estimates of diagnostic accuracy for FDG-PET and comparator
level of evidence
list of patient-important clinical outcomes for patients testing true and false positive or true and false negative
matrix of ‘‘natural frequencies’’ reporting absolute numbers for true and false positive and negative results per 100 patients
estimates of impact on clinical outcomes – when available – and level of evidence
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
appropriateness
Panel 2
appropriateness
Panel 1
appropriateness
All panellists
0.22227
p < 0.0059
N 152
0.56239
p < .0001
N 107
0.35925
p < .0001
N 259
True positive
0.32992
p < .0001
N 152
0.53341
p < .0001
N 107
0.39336
p < .0001
N 259
True negative
0.19891
p < 0.0137
N 153
0.62271
p < .0001
N 107
0.36084
p < .0001
N 260
False negative
0.23894
p < 0.0031
N 151
0.33718
p < 0.0004
N 107
0.26006
p < .0001
N 258
False positive
0.56209
p < .0001
N 171
0.57848
p < .0001
N 122
0.60577
p < .0001
N 293
Level of Evidence
0.59193
p < .0001
N 176
0.58954
p < .0001
N 135
0.58621
p < .0001
N 311
Favours
PET
−0.37956
p < .0001
N 176
−0.35319
p < .0001
N 135
−0.34348
p < .0001
N 311
Sensibility
Favours
comparator
Table 2 (P3A280). Correlation coefficient of each variable with judgement of appropriateness (global and by panel).
0.00911
p < 0.9059
N 176
0.55358
p < .0001
N 135
0.25646
p < .0001
N 311
Favours
PET
0.3093
p < .0001
N 176
−0.30828
p < 0.0006
N 135
0.04966
p < 0.397
N 311
Specificity
Favours
comparator
0.06368
p < 0.4342
N 153
0.39895
p < .0001
N 110
0.24723
p < .0001
N 263
Pre-test
probabiliy
152
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
153
P3A281
Evidence-based guidelines in a teaching primary care
health service: The beginning of evidence-based medicine
in the Amazonian region
Andriolo RB1 , Silva BNG2 , Atallah N2
1 Universidade do Estado do Pará – Brazil; 2 Universidade Federal de São
Paulo, Brazil
Background: Evidence-based guidelines (E-BGs) are an important tool
to conduct translational research and for improving health services’
quality and outcomes. Objectives: To share information about the
first implementation of E-BGs in one teaching primary care health
service located in the Amazonian region. Methods: The following
actions were planned to achieve the objectives: i) mapping the health
care and administrative teams; ii) evaluating the teams’ knowledge and
perceptions of evidence-based medicine (EBM); iii) deliver an annual
EBM course with topics on the Cochrane Collaboration; iv) mapping the
diagnostic, preventive and therapeutic technologies most commonly
used by each health specialty; v) hold consensus meetings involving
internal and external health professionals, and the administrative
body to prioritise themes, taking into account existing uncertainty,
epidemiological relevance and costs; vi) preparing evidence-based
medicine recommendations (EBMR); vii) holding a new consensus
meeting to prepare E-BGs with multidisciplinary contributions to the
EBMR documents, according to the local context; viii) comparing
outcomes among patients when medical decisions were or were not
based on the guidelines (Figure 1). Results: The first phase, already
accomplished, shows that our health service is comprised of 50
administrative employees; 70 non-graduated health auxiliaries; and
112 graduate or post-graduate health professionals of different health
areas and specialties including pharmacists, cardiologists, clinicians,
otorhinolaryngologists, nurses, nutritionists, pediatricians and dentists
Figure 1 (P3A281).
and the service is responsible for more than 160,000 visits per year.
Conclusions: It is expected that successful implementation of the
E-BGs will improve health outcomes and be more economical in this
public health service. Moreover, this pilot experience with translational
research is intended to be reproduced in the Health Department of Pará
State, which is responsible for 143 cities located in a region with more
than 7 million people characterized by an extreme paucity of human
and economic resources for health care.
P3A282
Characteristics of studies included for HTA reports
Wolff R1 , Ravaud P2 , Liberati A3 , Bonfill X4 , Antes G5
1 Kleijnen Systematic Reviews Ltd, UK/German Cochrane Centre, Germany;
2 French Cochrane Centre, France; 3 Italian Cochrane Centre, Italy;
4 Iberoamerican Cochrane Centre, Spain; 5 German Cochrane Centre,
Germany
Background: Clinical trials are the backbone of evidence syntheses,
such as systematic reviews, guidelines, and health technology
assessments (HTA). Accordingly, usage of a clinical trial in evidence
syntheses can be used as proxy for quality and the practical value of
the trial and hence for the transfer of knowledge. Characteristics of
published studies were reported for various fields [1–6], specific journals
[7, 8], and the ‘Clinical Trials’ database, formerly known as ‘Cochrane
Central Register on Controlled Trials’ (CENTRAL) [9]. A recent paper
assessed the countries of origin of studies used for Cochrane reviews
[10]. However, only little is known on studies used for HTA reports.
These reports have crucial impact on health spending of countries
as they are used to inform decisions on reimbursement of drugs,
medical devices, and health technologies. Objectives: To examine
the characteristics of studies included for HTA reports. Methods:
HTA reports by members of the International Network of Agencies
for Health Technology Assessment (INAHTA) of four countries (France,
Germany, Italy, Spain) will be included. Characteristics of studies (e.g.
country of origin, year of publication, number of participants, type of
HTA report, type of study, and information on funding, if available) will
be extracted independently by two reviewers. Results will be presented
quantitatively and adjusted for economic power and size of countries.
Results/Conclusions: Preliminary results for Germany (based on
59 reports including 1099 studies) showed the USA as the leading
contributor (27%), followed by the UK (7%) and Germany (5%). All
countries are dependent on evidence from clinical trials generated
in other countries. This is particularly true for continental European
countries who contribute fewer trials to the global knowledge pool
as the UK and USA do. We will present detailed results for all four
countries and discuss possible implications of our findings.
References
1. Soteriades ES, Rosmarakis ES, Paraschakis K, Falagas ME (2005)
Research contribution of different world regions in the top 50 biomedical
journals (1995–2002). Fed Am Soc Exp Biol 20:29–34.
2. de Jong JW, Schaper W (1996) The international rank order of
clinical cardiology. Eur Heart J 17:35–42.
3. Rahman M, Haque TL, Fukui T (2005) Research articles published
in clinical radiology journals: trend of contribution from different
countries. Acad Radiol 12:825–829.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
154
4. Grossi F, Belvedere O, Rosso R (2003) Geography of clinical cancer
research publications from 1995 to 1999. Eur J Cancer 39:106–111.
5. Lambers Heerspink HJ, Knol MJ, Tijssen RJW, van Leeuwen TN,
Grobbee DE, et al. (2008) Is the randomized controlled drug trial in
Europe lagging behind the USA? Br J Clin Pharmacol 66:774–780.
6. Rahman M, Fukui T (2003) Biomedical publication – global profile
and trend. Pub Health 117:274–280.
7. Johnson CD, Green BN (2008) Trends in articles published over the
past 20 years in the journal of chiropractic education: country of origin,
academic affiliation, and data versus nondata studies. J Chiropr Educ
22:4–11.
8. Ozsunar Y, Unsal A, Akdilli A, Karaman C, Huisman TA, et al. (2001)
Technology and archives in radiology research: a sampling analysis
of articles published in the AJR and Radiology. American Journal of
Roentgenology. Am J Roentgenol 177:1281–1284.
9. Gluud C, Nikolova D (2007) Likely country of origin in publications
on randomised controlled trials and controlled clinical trials during the
last 60 years. Trials 8.
10. Wolff RF, Reinders S, Barth M, Antes G (2011) Distribution of
Country of Origin in Studies Used in Cochrane Reviews. PLoS ONE
6(4): e18798. doi:10.1371/journal.pone.0018798
P3A283
How many evidence based guidelines in China
interventions for use within NHS mental health services research and
2. to present the results of a pilot study where that measure is used to
rate ten different psychosocial interventions for psychosis. Methods:
A literature search to identify papers assessing blocks and enablers
of implementing interventions within NHS mental health services was
conducted. Papers were thematically analysed to identify common
themes which specifically related to features of the intervention. Vote
counting was used to assess the frequency of each theme across the
literature. Themes which were present in three or more papers were
retained and turned into an item on the feasibility measure. Scoring for
the measure was developed through a consultation process involving
researchers and clinicians. The feasibility measure was piloted by two
researchers assessing 10 different interventions for psychosis. Results:
95 barriers and facilitators were identified from 11 papers of which
39 themes were specifically related to the intervention and used in
the vote counting stage. The final tool included 16 items organised
into blocks and enablers. Each item was rated on a likert-type
scale. Results of the pilot will be discussed including Inter-rater
reliability. Conclusions: The feasibility measure is a novel approach
to evaluating a particular intervention and may be used alongside
efficacy and health economic evidence from systematic reviews. Rating
the feasibility of an intervention during the evidence review stage may
assist commissioners, policy makers, and guideline developers with
their decision-making process.
Yang K1 , Chen Y1 , Yao L2 , Liang H2 , Wang Z2 , Xiao X2 , Wang Q2
1
Evidence-Based Medicine Center of Lanzhou University, China; 2 School
of Basic Medical Sciences of Lanzhou University, China
P3A285
Logic model as a knowledge translation tool for complex
interventions in systematic reviews
Background: Little is known about quality and quantity of Chinese
clinical guidelines. Objectives: To systematically review all of Chinese
clinical guidelines. Methods: We searched for the term guideline
in CNKI (China National Knowledge Infrastructure/Chinese Academic
Journals full text Database), VIP and WANFANG (two Chinese fulltext
databases), and CBM (China Biomedicine Database Disc). Two groups
of review authors independently applied inclusion criteria, assessed trial
quality, and extracted data. Results: From 1978 to 2010, 397 clinical
guidelines were identified of which only 37 (9.3%) claimed that an
evidence based approached was used in the process of development;
3(0.8%) provided search strategies and 20 (5%) provided the levels
of evidence and the recommendation. Conclusions: There were very
few evidence based clinical guidelines in China. Further assessment of
all guidelines will be made using AGREE II tool.
Augustincic Polec L1 , Ueffing E1 , Welch V1 , Tanjong Ghogomu E1 , Pardo
Pardo J1 , Grabowsky M2 , Attaran A1 , Tugwell P3
1 Centre for Global Health, Institute for Population Health, University of
Ottawa, Canada; 2 National Vaccine Program Office, Washington, USA;
3 Centre for Global Health, Institute for Population Health, Faculty of
Medicine, University of Ottawa, Canada
P3A284
Evaluating the feasibility of interventions for use within
NHS mental health services
Bird V1 , Leamy M1 , Le Boutillier C1 , Williams J1 , Slade M1
1
Institute of Psychiatry, King’s College London, UK
Background: Implementation research is an emerging discipline
within the field of mental health research. The blocks and enablers
of successful implementation within the NHS are currently being
researched. The feasibility of an intervention has been identified as
one factor linked to its successful implementation. Objectives: 1. To
describe the development of a feasibility measure to rate potential
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Logic models are useful tools that are recommended
for equity-oriented systematic reviews. They aim to explain how
interventions work and how different factors interact. For this
systematic review, a logic model was developed to facilitate the
understanding of the complex interventions aiming at increasing the
use of insecticide-treated bednets. Objectives: To develop a logic
model to frame the research question and guide the systematic
review process. Methods: A multi-stage revision process was
conducted in which topic experts, stakeholders, and researchers with
expertise in systematic reviews, knowledge translation, malaria, and
health equity were involved and their feedback incorporated. This
multidisciplinary approach ensured that the logic model was examined
from different perspectives and that all essential components were
included. Results: Initially, a research question was specified based
on literature review and consulting stakeholders and malaria experts.
Populations, interventions, and outcomes were identified. Classifying
complex interventions was the most challenging part of developing the
logic model. Building on existing research, interventions were classified
in three major groups: interventions focusing on delivery mechanisms,
interventions focusing on the proper use of the bednets such as
education and peer monitoring, and interventions that combined the
first two groups. Intervention categories were further subclassified
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
155
accounting for the channel through which the intervention was
delivered, duration of the intervention, cost to the end user, choice
of the intervention and accessibility, and sector of delivery. The logic
model that was developed guided the systematic review process, and
it enabled us to present our research in an understandable manner
(Figure 1 and Figure 2). Conclusions: Logic models are useful
tools for conceptualizing complex interventions. They can facilitate a
systematic review process, and may be used in policy planning and field
research.
Figure 1 (P3A285). LOGIC MODEL: Strategies to increase the use of insecticide-treated bednets in households and vulnerable populations to reduce
morbidity and mortality from malaria in endemic settings.
Figure 2 (P3A285).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
156
P3A286
Systematic review of clinical practice guidelines and
consensus statements about living kidney donation
Tong A1 , de Bruyn J2 , Wong G3 , Chapman J3 , Craig J1
1 University of Sydney, Australia; 2 University of Amsterdam, Netherlands;
3 Westmead Hospital, Australia
Background: Clinical practice guidelines (CPGs) and consensus
statements provide a basis for implementing research evidence, clinical
expertise and patient preferences. In living kidney donation, donors
face the risk of acute surgical complications and psychosocial harm;
and in practice, there is variability in donor screening and follow up.
Objectives: To compare the quality, scope, content and consistency
of international CPGs and consensus statements on the assessment
and care of living kidney donors. Methods: We searched for CPGs and
consensus statements for living kidney donation in electronic databases
and guideline registries up to 21 February 2011. The methodological
quality of guidelines was assessed using the Appraisal of Guidelines
for Research and Education (AGREE) instrument. A descriptive textual
synthesis was used to analyse and compare guideline recommendations
and consensus statements. Results: Seven CPGs and one consensus
statement were included. Three indicated the levels of evidence
underpinning the evidence, only one graded their recommendations.
Lack of explicitness about the methodology and development of
CPGs/consensus statements was apparent. While a comprehensive
range of relevant topics were covered, the scope varied considerably
between each CPG/consensus statement. Across guidelines, there
was slight variability in thresholds for clinical contraindications for live
kidney donation. A major discrepancy concerned the acceptance of
minor (under 18 years) kidney donors. The importance of informed
voluntary consent, genuine altruistic motivation, adequate support,
and psychological health and resilience were recognised but specific
guidance or tools for conducting psychosocial assessments were
lacking. Conclusions: Comprehensive and rigorously developed
guidelines which include explicit links to high-quality evidence, graded
recommendations, and specific tools for evaluating the medical and
psychosocial well-being of living kidney donors may promote the uptake
and implementation of evidence and CPGs in clinical practice. This can
lead to better screening and care of living kidney donors to minimise
acute surgical complications, mortality and psychosocial harms.
P3A287
Chinese Cochrane Database of Systematic Reviews of
Cochrane Library: A nationwide survey of utilization in
Taiwan
Chiu YW1 , Wen YH2 , Kuo KN3
1 Institute of Population Health Sciences, National Health Research
Institutes, Taiwan; 2 Chang Gung Memorial Hospital, Chang Gung
University College of Medicine, Taiwan; 3 College of Medicine, Taipei
Medical University, Taiwan
Background: The Cochrane Library is the preeminent online database
of systematic reviews. The National Health Research Institutes (NHRI)
started to translate the abstracts of the Cochrane Database of
Systematic Reviews into Chinese (CDSR) in 2007. The abstracts
are being summarized in the Chinese Cochrane Database of Systematic
Reviews (CCDSR), available at http://clc.nhri.org.tw/admin/clcmain1.
aspx since 2008. Objectives: To investigate the behavior in the
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
usage of Chinese abstracts in CDSR. Methods: Online logs were
obtained for August 2008 through December 2010. Results: A total
of 3,549 abstracts, classified into 53 categories, are available. There
was monthly increase in the usage of the CCDSR during 2008 ad 2009.
Usage of the CCDSR increased also every year. The average usage
rates per month are 325 hits in 2008, 636 hits in 2009, and 6290
hits in 2010. Overall, the most frequent accessed term was kidney,
followed by pregnancy and delivery, newborn, and respiratory tract.
The most frequently accessed month was April, followed by October.
Conclusions: The Chinese abstracts in CCDSR provided by NHRI are
helpful for users who prefer to read Chinese and even more so for users
who can only read Chinese, such as patients and the public.
P3A288
Uptake of Cochrane Acute Respiratory Infections reviews
in primary care guidelines
van Driel M1 , Thorning S2 , Dooley L2 , van der Wouden H3
1 Bond University, Australia; 2 Cochrane Acute Respiratory Infections Group,
Australia; 3 Erasmus University Rotterdam, Netherlands
Background: The Cochrane ARI group supports 120 reviews of which
many are relevant for primary care. Evidence-based clinical practice
guidelines draw upon systematic reviews to formulate recommendations for clinicians. It is unclear how available Cochrane ARI reviews
have been cited and used in guidelines. Better knowledge of their
uptake and the nature of recommendations derived from the reviews
will provide better insight in gaps in the evidence and issues related
to translating Cochrane reviews into clinical guidelines. Objectives:
To assess how clinical practice guidelines refer to Cochrane ARI
Systematic Reviews. Methods: Scopus is the largest abstract and
citation database containing both peer-reviewed research literature
and quality web sources. We used the citation tracker features of
Scopus to identify guidelines that cite systematic reviews published
by the Cochrane ARI group in topics that are relevant to primary
care. In addition, we searched for nationally endorsed primary care
guidelines through professional organisations’ websites in Europe,
North America, Australia and New Zealand. We selected citations in
clinical guidelines and analysed how the recommendations correspond
with the conclusions of the review authors. Results: The project
is currently underway and results will be available at the Cochrane
Colloquium. Conclusions: Cochrane reviews in the area of acute
respiratory infections in primary care are cited in clinical practice
guidelines. Tracking and analysing their uptake can identify gaps in the
evidence and inform reviewers how to improve transparency of reviews
and relevance to primary care practice.
P3A289
Scoring trials on the efficacy-effectiveness continuum: A
systematic analysis
Witt C1 , Manheimer E2 , Lüdtke R3 , Hammerschlag R4 , Lao L5 , Berman B2
1 Institute for Social Medicine and Epidemiology, Charité University Medical
Center Berlin; 2 Cochrane Collaboration Complementary Medicine Field,
University of Maryland School of Medicine; 3 Carstens Foundation, Essen;
4 Oregon College of Oriental Medicine; 5 University of Maryland School of
Medicine
Background: ‘Efficacy’ refers to the extent to which a specific
intervention is beneficial under ideal conditions. Efficacy produces
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
157
results for an intervention under carefully controlled conditions chosen
to maximize the likelihood of observing an effect if it exists. In
contrast, ‘effectiveness’ is a measure of the extent to which an
intervention, when deployed in the field in routine circumstances,
does what it is intended to do for a specific population. To this
end, effectiveness trials use eligibility criteria, treatment protocols,
and outcomes that are close to usual care. For valid decision
making in usual care, there is an urgent need for more evidence
from Comparative Effectiveness Research (CER). PRECIS (Thorpe et
al. CMAJ. 2009;180(10):E47-E57.) was mainly developed to guide
the design of RCTs in 10 dimensions along the efficacy-effectiveness
continuum. It is of major interest whether these dimensions can be
applied to existing trials as a means of strengthening the evidence base
for CER. Objectives: To assess the efficacy-effectiveness continuum of
randomized studies using acupuncture for low back pain as an example.
Methods: All English language RCTs that compared acupuncture with
a conventional treatment control and had >30 acupuncture patients
were analyzed by 5 raters using a PRECIS-derived-scale, before and
after a consensus process. Results: 10 studies were evaluated with
PRECIS (119 abstracts, 44 publications screened). The first rating
showed large variance between raters and items (intraclass correlation
0.02–0.60). This was mainly due to missing information in the
publications and to difficulties in operationalizing the scoring items.
After the consensus discussions, the intraclass correlation improved to
0.20–1.00. Conclusions: To appraise the value of RCTs for CER,
clearer operational criteria are needed, raters have to be trained in
applying the criteria, and more detailed information is needed when
reporting RCTs.
P3A290
Interactive tools to teach the public to be critical
consumers of information about health care: What’s out
there?
Krause LK1 , Schwartz L2 , Woloshin S2 , Oxman A3
1 Colorado School of Public Health, USA; 2 The Dartmouth Institute for
Health Policy and Clinical Practice, USA; 3 Norwegian Knowledge Center
for the Health Services, Norway
Background: People are frequently exposed to claims about the effects
of health care. To make sense of this information, the public—including
consumers, journalists, and decision-makers—need to know how to
question and interpret the information presented, and when to seek
additional information. Objectives: To create an inventory of
interactive tools that teach the public the necessary skills to become
critical consumers of health information. Methods: We systematically
performed web (Google) and database (Medline) searches, and solicited
input from key informants to identify tools which teach the public how
to interpret information about the effects of health care. We included
tools that are interactive (i.e. allow for some kind of two-way transfer
of information), open-access, free, and web-based or downloadable.
Potentially eligible tools were screened and coded independently by 2 of
the authors. Results: Of the 3,148 potential tools initially identified,
63 were interactive and web-based or downloadable, including 20
which were free and open access (1 from the Medline database search,
9 from key informants, 10 from the web search). Among these, 15
targeted medical professionals/students. Only 5 targeted the public (4
in English, 1 in Italian). Interactive elements of these tools included:
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
multimedia presentations (e.g. narrated PowerPoint presentation,
cartoon animation), quizzes (users enter answers and receive feedback),
and an annotated newspaper story. Skills covered included: study
design, bias, risk, where and how to search, and how to appraise
research evidence. Only 1 of the tools underwent formal evaluation
for effectiveness (a non-randomized, controlled study). Conclusions:
Few free open-access interactive tools are available to help people
become critical consumers of information about the effects of health
care.
P3A291
Partnership to improve health research reporting
in Spanish-speaking countries-especially Latin America
and the Caribbean
Villanueva E1 , Simera I2 , Cuervo LG1 , Altman D3 , Moher D4 ,
Reveiz L1
1 Policy and Research, Health Systems Based on Primary Health Care,
Pan American Health Organization, WDC, USA. 2 Head of Programme
Development EQUATOR, Centre for Statistics in Medicine, Oxford, UK;
3 Director, Centre for Statistics in Medicine, University of Oxford, UK;
4
Senior Scientist, Clinical Epidemiology Program, Ottawa Hospital Research
Institute, Ottawa, Canada
Background: A large number of studies have critical deficiencies in
health research reporting—this seriously compromises further use of
new research findings. Effective dissemination and implementation
of research findings are compromised when research is reported
badly. Studies have shown that research publications frequently
lack or distort crucial information, making the assessment of study
validity and reliability difficult1 . This is a global problem, including
Latin America2 . Objective: To raise reporting standards in Latin
America and the Caribbean (LAC) by increasing awareness and
implementation of a key collection of reporting guidelines available
in Spanish and hosted in the EQUATOR website recently translated
into Spanish. Methods: Using reporting guidelines, checklists that
specify minimum information necessary for a complete and clear
account of research methodology and findings, can substantially
reduce deficiencies in research literature. The EQUATOR Network,
recently translated into Spanish, is an international initiative that aims
to increase the quality of health research reporting by promoting the
use of reporting guidelines. EQUATOR provides online resources for
authors, editors, peer reviewers, and guideline developers. Results:
The EQUATOR portal hosts over 100 reporting guidelines, listed by
study type, and other resources supporting responsible reporting. A
growing number of journals refer their authors to this resource. The
website use has been increasing every year but predominantly by
the English speaking countries (United States, United Kingdom, and
Canada). Making resources available in Spanish should increase
understanding of the principles of good research reporting and
improve the quality of Spanish scientific publications. EQUATOR
supports the use of guidelines through training courses for editors,
peer reviewers and researchers. The new partnership will allow
development of education activities that address language specific
needs. This collaboration should significantly improve dissemination
and implementation of reporting guidelines in LAC and positively
influence standards of research reporting and more efficient use of
published research.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
158
P3A292
Evidence-based practice in first aid and prevention
guidelines of Belgian Red Cross-Flanders
Pauwels NS1 , Dieltjens T1 , Van de Velde S1 , De Buck E1
1 Belgian Red Cross-Flanders, Mechelen, Belgium
Background: In the past, first aid handbooks for lay public were
primarily based on experience and expert knowledge. The Belgian
Red Cross-Flanders decided to use an evidence-based approach when
revising its first aid programme. The curriculum consists of diverse topics
related to first aid and prevention of sudden illness and trauma, issued in
didactical handbooks for diverse target groups. Objectives: To provide
evidence-based recommendations in a limited time span, 12 months, for
151 diverse topics. As a reference, the Scottish Intercollegiate Guideline
Network (SIGN) defines 40 questions as highly elaborate and two year
projects are common. Methods: To address these many topics in a
timely and cost-conscious manner, a pragmatic approach to develop
the guideline was used. To identify the best current evidence, reviewers
performed literature reviews for each topic (one reviewer/topic). If
an eligible guideline or systematic review was found, we formulated
draft recommendations based on this evidence. If not, we searched
controlled intervention studies. When no study was retained, the search
continued for cohort and case-control studies. Databases searched
included Guideline International Network Library, National Guideline
Clearinghouse, Cochrane Library, BestBETs, Medline, and Embase.
The quality of evidence was graded using GRADE. A multidisciplinary
panel of 11 experts discussed draft recommendations while taking
into account the evidence. Results: During a one year period, the
reviewers devoted 2 full-time equivalents (FTE) to prepare the evidence
summaries. No relevant evidence was found for 49 of the 151 topics
and for the other topics, the available evidence was limited. Eight
panel meetings (3-4h) were needed to finalise the recommendations.
Conclusions: Despite the massive number of topics, the first aid
curriculum was reviewed within a limited time frame. For the first
time, the integral first aid curriculum of the Belgian Red Cross-Flanders
systematically incorporates evidence and expert judgment.
P3A293
How evidence-based are the pharmaceutical industry’s
printed promotional material presented to physicians in
three Latin American countries
Ciapponi A1 , Reveiz L2 , Glujovsky D1 , Garcı́a Martı́ S1 , Rey Ares L1 , Bardach
A1 , Ramı́rez Moreno L3 , Ferreira AM3 , Rubinstein F1 , Rada G4
1 IECS – Institute for Clinical Effectiveness and Health Policy. Argentina;
2
Andean Cochrane Branch of the Ibero-American Cochrane Center; 3 Clı́nica
Reina Sofı́a, Bogotá Colombia; 4 Department of Internal Medicine. Faculty
of Medicine, Pontificia Universidad Católica de Chile
Background: A major marketing technique used by pharmaceutical
companies is direct-to-physician marketing. Of these, one of the
most used is the printed promotional material (PPM). Objectives: To
determine if the references provided by printed promotional material
(PPM) presented to physicians by pharmaceutical representatives
support its statements and to assess the validity of the PPM against
the best available evidence. Methods: A consecutive sample of all
the PPM distributed by pharmaceutical representatives was collected
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
from January 2010 to March 2011, in a family medicine practice in
Buenos Aires, Argentina; in a general medicine practice in Bogotá,
Colombia; and in an internal medicine ambulatory practice in a public
hospital in Santiago de Chile, Chile. Reprints and monographs were
excluded. The number and accuracy of PPM’s statements supported
by cited references; and the correctness of references cited in PPM
according to the International Committee of Medical Journal Editors
guidelines was evaluated. The retrievability of the references was
also assessed. Adverse reactions/warnings about drug interactions
and contraindications were collected. Relevant clinical information of
the material was translated to PICO format questions to systematically
search the best related evidence (systematic reviews [SR] or RCTs) in: 1)
The Cochrane library 2) MEDLINE. We assessed the risk of bias (ROB)
of identified SRs and RCTs using the AMSTAR instrument and the
Cochrane Handbook respectively. PPM agreement with citation/best
evidence was classified as ‘concordant’, ‘partially concordant’ or
‘discordant’. Pairs of independent reviewers graded the citations’ ROB
and the quality of the evidence using the GRADE system. Results:
See Table 1. Conclusions: -2/3 PPMs citations have serious flaws
(High ROB, discordant or reference unavailable); -50% of statements
are supported by the citation, and 30% are concordant with the best
available evidence; Cochrane SR are rarely cited by PPMs. -Physicians
should not trust in PPM and should demand they be regulated.
P3A294
Strategies to reduce attrition in randomised trials: a
methodology review
Brueton V1 , Tierney J2 , Stenning S2 , Nazareth I1 , Meredith S2 , Harding S3 ,
Rait G1
1 MRC GPRF, UK; 2 MRC CTU, UK; 3 MRC SPHSU, UK
Background: Attrition from randomised trials can introduce bias and
reduce statistical power affecting the generalizability, validity, and
reliability of results. Many strategies are used by researchers to reduce
attrition such as motivating and engaging participants and adding sites
to generate optimal data return or compliance to follow-up procedures.
Objectives: To quantify the effect of strategies to reduce attrition
from randomised trials in any health care setting. Methods: Included
studies were randomised evaluations of strategies to reduce attrition
embedded within randomised trials from all diseases and clinical
settings. The following sources were searched for eligible studies:
MEDLINE (1950 to present), EMBASE (1980 to present), PsycINFO
(1806 to present), DARE (most recent issue), CENTRAL (most recent
issue), CINAHL (1981 to present), C2-SPECTR (most recent date),
and ERIC (1966-present), Cochrane Methodology Register, Current
Controlled Trials metaRegister, WHO trials platform, Society for Clinical
Trials (SCT) conference proceedings (1980–2010), and publication
reference lists. A survey of all UK clinical trials units was conducted to
identify ongoing or completed but unreported studies. Data from trials
were pooled using the fixed effect model and heterogeneity explored
to determine the level of effectiveness of the type of strategies used
and the study design to reduce attribution. Results: From 19,281
abstracts 32 unique RCTs were identified from the following sources:
MEDLINE, CENTRAL, CINAHL n = 9; SCT abstracts 1980–2010 n = 4;
reference lists of relevant reviews n = 7; and of included trials n = 8
(7 duplicates); reference lists of excluded trials n = 1; word of
mouth n = 4; survey of CTUs n = 6. Five databases returned no
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
159
Table 1 (P3A293). Citations and best evidence supporting Printed Promotional Material (PPM) statements.
RCTs. Included RCTs evaluated 7 types of strategies to reduce attrition
including: communication, questionnaire length, incentives, participant
case management, visit frequency, behavioural, and methodological
interventions.
P3A295
Do the European clinical guidelines recommend evidence
based interventions for drug addiction? A benchmark
exercise
Ferri M1 , Bo A1 , Allara E2
1 EMCDDA Lisbon; 2 University of Turin, Italy
Background: An increasing body of evidence for interventions in
the Drug Addiction field is becoming available due to the decennial
activity of the Cochrane Group on Drugs and Alcohol. The European
Monitoring Center for Drugs and Drug Addiction (EMCDDA) created
an inventory of National Guidelines for the Treatment of Drug
Addiction by collecting information on 30 European Countries. A
preliminary analysis of the guidelines provided an overview of of the
recommendations for treatment. It also offers the basis for further
efforts aimed at promoting an evidence-based approach across Europe
in this area of work. Objectives: To describe the existing guidelines
for drug addiction treatment in 30 European countries in terms
of target clients, interventions and to perform a benchmark exercise
focused on 15 recommendations. Methods: Guidelines were collected
through a two-round survey and an international experts meeting. The
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
EMCDDA’s network of national focal points performed a benchmark
exercise comparing 15 recommendations from an evidence-based
guideline (WHO 2009) with similar recommendations in their National
Treatment Guidelines. Results: 141 national treatment guidelines
from 27 out of 30 countries were identified and obtained in full text
(availableat http://www.emcdda.europa.eu/best-practice). Almost half
of them (66/141) are about opioid dependence and 55 have Methadone
Maintenance as the main treatment. Overall recommendations are
similar across guidelines such as the recommendation for an integrated
package of long term substitution treatment combined with
psychological support. Difference exists in the process for the
development of the guidelines as some of them are evidence-based
while others use experts consensus. Divergence exists with regard to the
choice between Buprenorphine or Methadone for first-line treatment
and the drugs dosages. Conclusions: The study is a collection and
inventory of practice guidelines for the treatment of drug addiction.
Further studies are needed to assess the methodological validity
of the guidelines by the mean of standardized tools such as
AGREEII.
P3A296
Progress in time to update of Cochrane Systematic Reviews
Ngamjarus C1 , Jaidee W2 , Laopaiboon M1
1 Faculty of Public Health, Khon Kaen University, Thailand; 2 Faculty of
Public Health, Burapha University, Thailand
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
160
Background: Up-to–date evidence relevant to prevention and
treatment of any health-related events are important information for
healthcare decision-making and services. The Cochrane Collaboration
policy is that Cochrane systematic reviews should be updated within
two years. However, development of preventive and therapeutic
interventions can changed and differed over timeSystematic reviews
of the 52 Cochrane review groups may be updated at different time
periods. Objectives: This paper will present time to updating
systematic reviews of Cochrane review groups. Methods: The
original cohort of Cochrane systematic reviews firstly published in the
Cochrane Library between January 2000 and December 2005 were
studied. Update versions of the cohort reviews until 2011 issue 4 were
independently searched by two authors in the Cochrane Archie server
and Cochrane Library. We defined ‘update’ following the suggestions
in the Cochrane handbook. Time to update was measured from the
date of current update version and the date of previous consecutive
version. Disagreement between the two authors throughout the review
was solved by consensus and discussed with the third author. The
median survival time of updating and its 95% confidence interval (CI)
was analyzed by using STATA software. Results and Conclusions:
There were 623 Cochrane reviews from 47 review groups. Median
survival time to update of the reviews was 1.8 years (95% CI 1.6,
1.9 years). Fifty five percent of them were lastly updated before 2
years. The top three groups that had large number of update reviews
were Airways Group (56 reviews), Pregnancy and Childbirth Group (46
reviews), and Neonatal Group (40 reviews) with median survival time
to update equal to 2.0 years (95% CI 1.4, 2.6 years), 1.6 years (95%
CI 0.6, 2.5 years) and 1.7 years (95% CI 1.0, 2.5 years), respectively.
P3A297
Bringing evidence into practice: Use of a rapid review
methodology to formulate medical recommendations for
the Red Cross Blood Service
De Buck E1 , Van de Velde S1 , Dieltjens T1 , Vandekerckhove P1
1
Belgian Red Cross-Flanders
Background: The Belgian Red Cross-Flanders is active both
domestically and internationally in several fields: from blood supply
to emergency aid. The central thread in our strategic plan is to
have evidence-based program. In our field of work, no systematic
reviews or evidence-based guidelines are available yet. A rapid review
methodology was developed to promote the use of evidencebased
in our field of work. Objectives: To inform about the application
of evidence-based practice in an action-oriented organization. We
illustrate our methodology with a specific question from the Blood
Service, i.e. is it safe to use blood from hemochromatosis patients
for blood donations. These patients frequently need bloodlettings for
medical reasons. Methods: We used a rapid review methodology
and not a full systematic review. The methods were based on
published literature about this type of reviews. One reviewer reviewed
a restricted number of using specific search strategy and criteria.
Grey literature was excluded. GRADE was used to determine the
quality of evidence. An expert panel considered the evidence and
formulated recommendations for practice. Results: Six observational
studies were relevant to our question. The strength of the body
of evidence was low to very low. The available evidence indicates
that blood from hemochromatosis patients, taken during maintenance
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
therapy, comply with quality requirements and is safe to be used for
blood transfusion. Conclusions: To shift from an implementation
and execution organization to one which actively sets the trend, the
Belgian Red Cross introduced evidence-based practice in its activities.
We gave the example of hemochromatosis, because it changed the
blood donation procedures. A limitation of the rapid review is that
the conclusions are tentative and may be subject to change once a
systematic review is available.
P3A298
Deficient reporting and interpretation of non-inferiority
clinical trials in antiretroviral-naı̈ve HIV patients:
A systematic review
Hernandez AV1 , Pasupuleti V1 , Deshpande A1 , Collins JA2 , Vidal JE3
1 Cleveland Clinic, Ohio, USA; 2 Hospital Guillermo Almenara-EsSalud, Lima,
Peru; 3 Sao Paulo University School of Medicine, Sao Paulo, Brazil
Background: Non-inferiority (NI) randomized clinical trials (RCTs)
commonly evaluate efficacy of new antiretroviral (ARV) drugs in
ARV-naı̈ve HIV patients. Their reporting and interpretation have
not been systematically evaluated. Objectives: To evaluate the
reporting of NI RCTs of new ARV drugs in ARV-naı̈ve HIV patients
according to the CONSORT statement; to evaluate the degree of
misinterpretation of RCTs when NI was inconclusive or not established.
Methods: Systematic review of NI RCTs evaluating drugs in ARV-naı̈ve
HIV patients. Pubmed, The Web of Science, and Scopus were reviewed
until March 2011. Selection and extraction was independently done by
two reviewers. Key reporting information included: similarity to prior
RCTs of the active comparator, description of method to determine
the NI margin, use of confidence interval (CI) method to interpret the
primary outcome, use of blinding, and use of intention-to-treat (ITT)
vs. per protocol (PP) statistical analysis. When NI was inconclusive
or not established, we evaluated whether authors highlighted NI and
distracted readers with secondary results. Results: Fourteen RCTs were
selected (range 71–3316 patients). None of the RCTs gave information
about prior RCTs of the active comparator, and all used 2-sided CIs.
All studies described the NI margin between 10% and 15%, but only
5 explained the method to determine it. Blinding was used in 4
studies, and the appropriate PP was the primary analysis in 4 studies.
Five studies with NI inconclusive or not established highlighted NI or
equivalence, and they distracted readers with positive secondary results
in the abstract, results and conclusions. Conclusions: There is poor
reporting and interpretation of NI RCTs done in ARV-naı̈ve HIV patients.
Maximizing the reporting of the method of NI margin determination,
the use of blinding and per-protocol analyses, and interpreting negative
NI according to actual primary findings will improve the understanding
and translation of results into clinical practice.
P3A299
Setting priorities for comparative effectiveness research
on the management of primary angle closure (PAC): A
survey of Asia-Pacific clinicians
Yu T1 , Li T1 , Puhan M1 , Dickersin K1
1
Johns Hopkins Bloomberg School of Public Health, Epidemiology
Objective: To set priorities for new systematic reviews on the
management of primary angle closure (PAC), using clinical practice
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
161
all) to 10 (highly important). To assess understanding, we first asked
two knowledge assessment questions about the survey objective,
and allowed retries if the response was not correct. Respondents
were randomized to one of two questionnaires, which had different
orders of the same questions. The survey and instructions were in
English. Results: 175 agreed to participate in the survey, 132
responded (75.4% response rate), and 96 completed the questionnaire
guidelines and a survey of clinicians from the Asia-Pacific region.
Methods: We restated the American Academy of Ophthalmology’s
Preferred Practice Patterns (AAO PPPs) recommendations for
management of PAC into 42 answerable clinical questions. We
asked registrants at the Asia-Pacific Joint Glaucoma Congress 2010 in
Taipei to rate the importance of having an answer to each question for
good patient care, using a Likert-type scale from 0 (not important at
9
Figure 1 (P3A299). Scatter plots of rating results for 42 clinical questions using questionnaires (I and II) with different orders of the
same questions (a) Scatter plot of the mean ratings assigned in questionnaire I and questionnaire II for 42 clinical questions (with 45-degree line); the
mean ratings are generally lower in questionnaire II compared to questionnaire I. (b) Satter plot of the rankings (ranked by mean ratings) in questionnaire
I and questionnarire II for 42 clinical questions. The question with the highest mean rating is ranked 1. Numbers plotted on the figure refer to question
numbers in questionnaire I.
8
Pearson correlation coefficient = 0.77
9
37
38
5
10
23
4
3
24
21
7
14
13
26 1642
34
33
35
36
12
32
6
15
41 19
11
1
6
7
39
1718
20
8
22 2
31
30
29
25
28
5
27
40
5
6
7
8
Mean of the ratings assigned in questionnaire I
9
9
1
(a)
Spearman rank correlation
coefficient = 0.73
38
5
10
6
23
4
11
24
21
3
14
16
16
21
26
26
31
12
36
17
25
27
40
41
1
22
2
42
34
33
15
35
11
8
13
41
37
41
6
19
7
36
32
39
18
20
31
30
29
28
36
(b)
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
31
26
21
16
11
6
1
Ranking in questionnaire I
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
162
(54.9% usable response rate). For those who responded but did not
complete, half stopped after the two knowledge assessment questions.
Of 96 who completed, only 29 (30.2%) correctly answered the
knowledge assessment questions on the first attempt. The distribution
of importance ratings pooled from all questions was skewed to high
importance (mean = 6.92; median = 8). The order of questions
affected the importance ratings (Figure 1). We considered clinical
questions to be of priority if at least 75% of respondents gave
ratings above 5. Conclusions: Understanding the survey purpose
and questions affected our response rate, and question order affected
question importance rating. We identified a list of clinical questions
on the management of PAC for which research should be prioritized.
Where evidence does not exist, primary research and systematic reviews
should be initiated to address these questions.
P3A300
Course ‘Hot topics in pharmacotherapy’: A way to
disseminate evidence
Garjón Parra J1 , Erviti López J1 , Azparren Andı́a A1 , Gorricho Mendı́vil J1 ,
López Andrés A1
1 Servicio Navarro de Salud, Servicio Prestaciones Farmacéuticas, Spain
Background: There is in dire need for up-to-date clinical information
for doctors. However, medical education is provided to a great extent
by pharmaceutical companies and hence there is a high risk of biased
information. A new way to provide evidence-based education on
pharmacotherapy to a wide audience of health professionals is needed.
Objectives: To present controversial topics in drug therapy. To
provide useful evidence-based education for patient care providers. To
demonstrate the need for evidence-based pharmacotherapy and to get
practitioners involved in evidence-based decision making. Methods:
An annual two-day Drug Therapy Course organised by the Drug
Information Unit of the Navarre Health Service in Spain took place
in 2009. Five topics were addressed in each of the two scheduled
sessions (10 topics total). Criteria for selection of themes: sharply
defined, current controversy or new evidence, wide interest, variety.
Features of the course:-Accessibility: Free, scheduled during off-duty
days. Accredited for continuing medical education. -Peer teaching:
Lecturers were professionals involved in day-to-day patient care or
knowledgable about pharmacotherapy. Lecturers who were solely
academics or researchers were not considered. Group interaction
was promoted. Length or time: Short-duration conferences consisting
of a 20-minute presentation followed by a 10-minute discussion.
-Evaluation of each conference. Cost-consciousness: Lecturers were
paid ¤90 for each presentation. Conference room was provided
for free by the Government of Navarre. All educational materials
were available in the web-site, www.bit.navarra.es. Independent:
Free of sponsorship. Results: Participants: Year 2009: 140; Year
2010: 127; Year 2011: 170. Profession of attendees (Figure 1).
Conferences global evaluation (122 responses): Bad: 1%; Fair: 14%;
Good: 54%; Very good: 31%. Conclusions: It is feasible to
provide education that is independent of pharmaceutical company
sponsorship on pharmacotherapy to a wide audience of health
professionals in a constrained resources setting. There is good
acceptability of short conferences about new or controversial topics
given by non-academic clinicians in an enviroment that facilitates
participation.
Figure 1 (P3A300). Profession of registered attendees.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
163
P3A301
Do systematic reviews within EPOC contain sufficient
information about effect and enough information about
the intervention to facilitate informed policy
decision-making?
Holte HH1
1 Norwegian Knowledge Centre for the Health Services, Oslo, Norway
Background: Cochrane systematic reviews are intended to be part of
the documentation that is evaluated prior to making a policy decision
on how to improve local health services. Policy makers need to know if
the intervention is effective relative to alternatives. Descriptions of the
intervention need to be sufficiently detailed to evaluate if it is relevant
and feasible to implement in a local setting. When reading an Effective
Practice and Organisation of Care (EPOC) review, is there sufficient
information to decide if it is relevant and feasible to implement the
intervention that was reviewed in Norway’s local setting? Methods:
All the new reviews and updated reviews published in the Cochrane
Database of Systematic Reviews by the EPOC-group from January 2010
to January 2011 were examined. The information on the effect, the
relevance, and the feasibility of the intervention were first considered
by examining the title of the review, then the abstract. If it was still
deemed to be potentially relevant and feasible, then full review was
read. Results: Of the 28 reviews that were indentified, two reviews
were specific to low and middle income countries and two reviews
had no included studies. Sixteen reviews did not draw a conclusion
for various reasons. Of the eight reviews reporting an effect of the
intervention, none gave sufficient information that permitted me to
think that the intervention could be implemented in a local setting in
Norway. Discussion: The potential global applicability of Cochrane
reviews is one of the positive aspects of Cochrane systematic reviews.
EPOC reviews have special challenges because of the large variation in
national organizations of health care, and the relatively larger impact
of setting. Systematic reviews must include more information about
the interventions and the settings to make it possible to consider
implementation at local levels.
P3A302
How and when to pool data in meta-analyses to make the
results clinically meaningful
Ringerike T1 , Klemp M1
1 Norwegian Knowledge Centre for the Health Services, Norway
Background: Meta-analyses can be useful in deciding whether or
not an intervention is feasible. However, it is subject to discussion
which data should be included. An inclination to pool as much data as
possible to get a sturdy estimate must be balanced against a possible
increase in heterogeneity and difficulties in interpretation of the results
within the clinical context. We highlight different choices which have to
be made when comparing rivaroxaban to enoxaparin, drugs which are
used as prevention of deep venous thromboembolism (DVT) in patients
undergoing elective total hip or knee replacement surgery. Objective:
The objective is to raise awareness regarding which choices have to be
considered when pooling data and assessing what are their possible
impact on the results and interpretation. Methods: A systematic
literature search was performed. Data were extracted and pooled in
different ways to give risk ratios (RR) and 95% confidence intervals (CI).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Results: An overview of the data for prevention of DVT is presented
in Figure 1. Pooled data for hip replacement surgery, regardless of
dose and treatment duration gained RR 0.46 (0.26–0.84). However,
the approved dose of rivaroxaban is 10 mg and limiting the analysis
to this gave RR 0.48 (0.23–0.98). Finally, the pooled effect estimate
based on the doses and treatment durations representative of clinical
use within EU countries is limited to one study with an RR 0.28
(0.16–0.49). On the other hand, instead of narrowing the focus
one might want to expand to pool data regardless of hip or knee
replacement therapy. Conclusions: Always adjust the meta-analysis
according to the research question. Use of subgroups may facilitate
getting an overview of the data, interpretation and extrapolation to
similar research questions.
P3A303
Long-term harm and memory impairment in
electroconvulsive therapy treatment for depression:
preliminary results from a systematic review using
an integrative methodological approach
Leiknes KA1 , Høie B1 , Jarosch-von Schweder L2 , Berg R1 , Smedslund G1
1 Norwegian Knowledge Centre for the Health Services, Norway;
2
Department of Research and Development, St. Olavs University Hospital,
NTNU-Faculty of Medicine, Department of Neuroscience, Trondheim,
Norway
Background: A major concern in the use of electroconvulsive
therapy (ECT) for depression is the potential long-term harm on
cognitive (memory) function. Although evidence from controlled
clinical trials suggest that memory impairment may not last over six
months subjective reports of long-lasting memory loss have been
discovered, with long-lasting memory loss ranging from 29% to
79%. Results from systematic reviews of trials with long-term within
subjects, pre-ECT and post-ECT neuropsychological test data have
thus been called upon. Objectives: To present preliminary results
from included non-randomized (non-RCT) trials, concerning long-term
(>6 months) cognitive functioning after ECT for depression, registered
Cochrane review, using a novel, integrative methodological approach.
Methods: A broad systematic literature search on ECT for depression
was conducted. Medline, PsycINFO, Cinahl, SveMed+, Cochrane
Central, CCDAN-CTR, Embase, British Nursing Index and Ovid Nursing
databases were searched. All identified non-RCTs concerning ECT
administered to adults (age over 15 and below 65 years, at baseline)
for major depressive disorder (MDD) with a within subjects pre-ECT
and post-ECT neuropsychological testing of cognitive (memory) function
were full-texted screened. Mixed diagnostic populations (e.g. bipolar
disorder, schizophrenia) and other disorders where MDD data were
not separable were excluded. Results: Overall 132 non-RCT studies
concerning ECT and cognitive functioning were screened and 68 were
identified as containing neuropsychological tests to be examined. This
yielded only 22 full text to be included. Main reasons for exclusions were
mixed diagnostic study sample and old age (over 65 years), but also use
of non-standardized cognitive instruments and insufficient data report.
Diversity of neuropsychological test batteries for verbal, non-verbal,
autobiographical and subjective memory makes meta-analyses and
aggregation of the study data almost impossible. Conclusion: In spite
of the seemingly abundance of non-RCT trials concerned with ECT for
depression and cognitive functioning, long-term neuropsychological
data for specifically ECT and MDD are still very sparse.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
164
Figure 1 (P3A302).
P3A304
Use of indirect comparison methods in systematic reviews:
A survey of Cochrane review authors’ views
Abdelhamid A1 , Loke YK1 , Parekh S2 , Chen Y3 , Sutton A4 , Eastwood A5 ,
Holland R1 , Song F1
1 University of East Anglia, UK; 2 NETSCC- University of Southhampton, UK;
3 University of Birmingham, UK; 4 University of Leicester, UK; 5 University of
York, UK
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Many systematic reviews have evaluated multiple
interventions for a particular condition, but such reviews often do not
formally compare the relative effectiveness of interventions through
adjusted indirect comparison (AIC) or network meta-analysis. The use of
AIC has recently attracted growing interest. Objectives: To investigate
the views and knowledge of Cochrane systematic review (CSR) authors
regarding the use of AIC. Methods: An online survey was sent to 84
CSR corresponding authors of reviews we had identified as reviews in
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
165
which direct comparisons (DC) as well as AIC of interventions would
have been possible. Authors were also sent results of DC and AIC
compiled using data from their particular review. Results: Response
rate was 57%. Respondents were academics (54%), clinicians (27%)
or with equally shared roles (19%). When conducting the review in
question, only one quarter of authors had known about AIC methods
and have used them while over a third (35%) had knowledge of the
methods but had not used them. Forty percent (40%) either did not
know about AIC or had only heard about it. The majority of authors
(76%) would, at least sometimes, consider evidence from AIC. Some
authors expressed suspicion of the methods. Most authors (89%) felt
they needed more training, especially in assessing the validity of AIC.
Almost all authors felt that AIC could potentially be influenced by a
huge number of effect modifiers. Some respondents accepted that use
of AIC is needed as it may be the only source of information for relative
effectiveness of competing interventions, provided that authors and
readers are conscious of its limitations. Time commitment and resources
needed were identified as important concerns for Cochrane reviewers.
Conclusions: There is mostly an acceptance of the increasing demand
for AIC as well as an urgent need to develop structured guidance and
training for its use and interpretation.
P3A305
Generation of quality of care indicators based on
systematic reviews
Osorio D1 , Aller M1 , Roqué M2 , Foradada C3 , Vives A4 , Rigau D2 , Bonfill X2
1
Institut d’Investigació Biomèdica Sant Pau (IIB Sant Pau), Spain;
2 Iberoamerican Cochrane Centre, Spain; 3 Consorci Sanitari Parc Taulı́,
Spain; 4 Consorci Sanitari de Terrassa, Spain
Background: Systematic and explicit methods to combine scientific
evidence with professional consensus for assessing the appropriateness
of care have been developed to a limited extent. Obstetric care during
childbirth is suitable to further develop such methods because it is
both a field with a high production of systematic reviews (SRs) and
where many indicators currently used may be questionable or may
need to be reassessed/reexamined. Objectives: To generate a set
of indicators to assess the appropriateness of obstetric care related to
childbirth based on evidence from SRs. Methods: We reviewed all
the SRs from the Cochrane Pregnancy and Childbirth Group (Cochrane
Library Issue 3, 2009) and searched for Clinical Evidence and main
guidelines to identify further SRs. We selcted interventions supported
by high quality evidence (of a benefit or harm) and from which a strong
recommendation could be obtained using GRADE methodoloy. A group
of experts partipated in the recommendation generation and provided
feedback on the indicators’ content validity and reliability. Results:
We identified 303 SRs. After excluding SRs about non-acute care,
not delivered in hospital or without clear evidence (either in favour or
against), we select 48 SRs that generated 21 strong recommendations
based on high quality evidence. The group of experts considered
valid and reliable a total of 18 quality indicators: eight referred to
prepartum care, eight referred to delivery, one referred to postpartum
and one referred to incomplete miscarriage. These quality indicators
are currently being tested for their validity, reliability and feasibility
in a pilot study. Conclusions: It is possible and feasible to develop
evidence-based quality indicators to measure the appropriateness of
care in Obstetrics. High-quality SRs that explicitly express support for or
against an intervention are reliable and accessible tools for generating
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
indicators of quality of care, once considered to be the nuances of
clinical practice, and help to make the indicators coherent.
P3A306
Graphic display for effective reporting of evidence quality
tables
Khan K1 , Mignini L2 , Borowiack E3 , Ross C4 , Kowalska M3 , Zapalska A3 ,
Mol B5 , Meads C1 , Walczak J3
1
for Health Sciences, Barts and The London School of Medicine and
Dentistry, Queen Mary University of London. UK; 2 Centro Rosarino de
Estudios Perinatales, Rosario, Argentina; 3 Arcana Institute, Plk Dabka,
Cracow, Poland; 4 Department of Obstetrics and Gynaecology, Radboud
University Nijmegen Medical Centre, Nijmegen, Netherlands; 5 Department
of Obstetrics and Gynaecology, Academic Medical Center, University of
Amsterdam, Amsterdam, Netherlands
Background: When generating guidelines, quality of evidence is
frequently reported in tabulated form capturing several domains, e.g.
study design, risk of bias, and heterogeneity. Increasingly this is
done using the GRADE (Grading of Recommendations Assessment,
Development and Evaluation) approach. A more user-friendly format is
needed. One example is a graphic display plotting the several domains
used in evidence grading on equiangular spokes starting from the same
point, the data length of each spoke proportional to the magnitude
of the quality, which succinctly captures tabulated information.
Objectives: To allow easy identification of deficiencies, outliers and
similarities in evidence quality for individual and multiple comparisons
and outcomes, paving the way for their routine use alongside tabulated
information. Methods: We tabulated findings separately for tocolytic
agents compared to placebo and those compared to beta agonists. For
each comparison and outcome pair evidence was initially graded by
the study design. We assigned all evidence a ‘high’ level of quality if it
was based on randomised controlled trials. If any of the reasons below
applied to the body of evidence for each comparison-outcome pair, the
quality level was downgraded by one level (if the reason was classified
as serious) or by two levels (if the reason was classified as very serious):
bias, inconsistency, indirectness, imprecision. Results: Figures 1 and
2 provide a visually striking display of the strengths and weaknesses
of the evidence across the spectrum of comparisons and outcomes
which, when tabulated, is dry, cumbersome to read, and difficult
to assimilate. Conclusions: There is considerable opportunity for
confusion when moving from evidence syntheses to recommendations.
The graphic approach we have developed can make the job of evidence
assimilation for decision-making easier. When guideline statements are
accompanied by these plots, this approach also has the potential for
improving the credibility of the recommendations made as the strength
of the evidence is more clearly displayed.
P3A307
Mapping research gaps in interventions related
to Millennium Development Goal (MDG) 5-A (maternal
mortality) in the ‘Implication for Research’ section
of Cochrane reviews
Higgins V1 , Chapman E2 , Reveiz L2 , Cuervo LG2
1 International Heath Department, School of Nursing and Health Studies,
Georgetown University, WDC, USA; 2 Public Policy and Research, Health
Systems Based on Primary Health Care, Pan American Health Organization.
WDC, USA
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
166
Figure 1 (P3A306). Graphic display of evidence quality on tocolytic efficacy compared to placebo or no treatment in randomized trials.
Comparisons vs placebo or no treatment2
Magnesium
Outcomes1
Atosiban3
Beta agonists
Rank according to % of respondents rating critical or important
3
2
1
95% Perinatal morbidity6
0
3
2
1
95% Safety to mother7
0
3
91% Perinatal mortality
91% Stop birth before 34 weeks
85% Undelivered at 24 hours
82% Undelivered at 48 hours
55% Stop birth before 37 weeks
Indomethacin
NO Donor5
&
Ca Blocker5
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
3
2
2
2
1
1
1
0
0
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
Design
3
Imprecision
2
1
Risk of Bias
0
Indirectness
No data
reported
Inconsistency
Each graph represents the quality domains shown on concentric spokes. For each of the spoke, the length
represents the magnitude of quality ranging from very low at the centre of the plot to high at its maximum
length. Concentric lines rippling out of the centre show quality increasing to low and then to moderate before
reaching the maximum value of high. Evidence quality tabulated in appendix 2, tables A-D.
1
Importance of outcome colour coded for ease of display. Ranking of importance undertaken through formal
survey of over 500 practitioners asking them to provide responses on a scale anchored between critical at one
extreme and not at all important at the other.
2
Presented in alphabetical order except Ca Blocker which is presented along with Magnesium sulphate.
3
Atosiban trade name Tractocile®, which is now not protected by patent.
4
NO = Nitric oxide.
5
Ca = Calcium channel.
6
Based on neonatal admission to intensive care unit.
7
Based on maternal drug reaction requiring treatment cessation.
Figure 2 (P3A306). Graphic display of evidence quality on tocolytic efficacy compared to Beta agonists in randomized trials.
Comparisons vs Beta agonists2
Outcomes1
Rank according to % of respondents rating critical or important
95% Perinatal morbidity6
95% Safety to mother7
91% Perinatal mortality
91% Stop birth before 34 weeks
85% Undelivered at 24 hours
82% Undelivered at 48 hours
55% Stop birth before 37 weeks
Atosiban3
Ca Blocker4
Indomethacin
NO Donor5
Magnesium
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
3
2
1
0
Design
3
Imprecision
2
1
0
Indirectness
Risk of Bias
Inconsistency
No data
reported
Each graph represents the quality domains shown on concentric spokes. For each of the spoke, the length
represents the magnitude of quality ranging from very low at the centre of the plot to high at its maximum
length. Concentric lines rippling out of the centre show quality increasing to low and then to moderate before
reaching the maximum value of high. Evidence quality tabulated in appendix 2, tables E-I.
1
Importance of outcome colour coded for ease of display. Ranking of importance undertaken through formal
survey of over 500 practitioners asking them to provide responses on a scale anchored between critical at one
extreme and not at all important at the other.
2
Presented in alphabetical order except.
3
Atosiban trade name Tractocile®, which is now not protected by patent.
4
Ca = Calcium channel.
5
NO = Nitric oxide.
6
Based on neonatal admission to intensive care unit.
7
Based on maternal drug reaction requiring treatment cessation.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
167
Introduction: Achieving MDG-5A requires a concerted effort and
political will for the implementation of strategies and cost-effective
interventions based on scientific evidence to improve the health of
women. Cochrane reviews could be useful to the identification of
research priorities. Objective: To assist research sponsors and
developers in achieving MDG-5A by mapping and prioritizing relevant
research gaps identified in the ‘Implication for Research’ section of
Cochrane reviews. Methods: Phase 1: We used an advanced search
strategy in PUBMed limited to the Cochrane Database of Systematic
reviews, and included reviews assessing interventions likely to reduce
maternal mortality if they were published or updated during or after
2006. We evaluated the ‘Implication for Research’ section of each
review and developed questions in ‘PICO’ (spell out) format according
to what Cochrane reviewers presented, then categorized questions
according to intervention type (Table 1). Phase 2: A group of experts
will prioritize the list of identified questions by selecting and ranking
each question based on specific criteria. We will verify that no new
clinical trials address these questions, by identifying randomized clinical
trials listed in PUBMED, the Cochrane Library, and the International
Clinical Trials Registry Platform (ICTRP). We will provide a list of
unanswered questions that remain relevant to the achievement of
MDG5-A (maternal mortality). Results: We identified 695 references
during phase 1 and included 203 reviews that met our inclusion criteria.
More than 300 questions were structured and classified by intervention
type (Table 1). An analysis of advanced phases of this study will be
presented at the Cochrane Colloquium. Conclusion: By organizing
and presenting pre-identified knowledge gaps, we will provide helpful
information for research sponsors, funding and development agencies,
and researchers interested in addressing MDG-5A. We expect this
contribution to act as a catalyst for the development of the essential
research needed for the reduction of maternal mortality.
Table 1 (P3A307). PICO questions according to intervention.
Number (%) of PICO questions (N = 312)
Type of Intervention
General
31 (9,9%)
Pregnancy Prevention
20 (6.4%)
Education/Training of Women
11 (3.5%)
Specific
237 (76.0%)
Post-Partum Hemorrhage
25 (8%)
Abortion/Miscarriage
25 (8%)
Hypertensive Disorders
27 (8.7%)
Infection/Sepsis
11 (3.5%)
Caesarean Section
16 (5.1%)
Preterm birth
11 (3.5%)
Other Direct Causes
65 (20.9%)
Indirect Obstetric Causes
57 (18.3%)
Health policy and systems
42 (13,5%)
Governance
1 (0.3%)
Financial
0 (0%)
P3A308
A citation analysis of Cochrane systematic reviews
in Chinese Science Citation Database
Delivery
23 (7.5%)
Implementation
18 (5.7%)
Chen Y1 , Wei D2 , Li J2 , Xu H2 , Zhu S2
1 Evidence-Based Medicine Center of Lanzhou University, China; 2 School
of Basic Medical Sciences of Lanzhou University, China
No Questions Listed
2 (0.6%)
Background: Although The Cochrane Database of Systematic Reviews
has an Impact Factor of 5.653 and is ranked 11th out of 132 in the
ISI category, ‘Medicine, General & Internal,’ in 2009, we know
little about citation of Cochrane systematic reviews in Chinese articles.
Objectives: To investigate the citation of Cochrane systematic reviews
in Chinese articles. Methods: We searched the Chinese Science
Citation Database (CSCD) using the term ‘Cochrane Database of
Systematic Reviews’ from sources cited. Then we screened all relevant
papers. Results: There were 1184 Cochrane Systematic Reviews
cited by 1403 Chinese papers which were published in 191 medical
journals. The total citations are 2005 and citation per paper is 1.7.
The most cited paper is ‘Nicotine replacement therapy for smoking
cessation’ (12 times). The top three citing journals are Chinese Journal
of Evidence Based Medicine, Chinese Journal of Practical Gynecology
and Obstetrics, and Chinese Journal of Clinical Rehabilitation, which
have cited more than 500 CSRs. Conclusions: Citation per paper
of The Cochrane Database of Systematic Reviews in China is only
a third of the average number. It is necessary to further promote
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
the dissemination and utilization of Cochrane Database of Systematic
Reviews in China.
P3A309
A systematic review of the quality and types of instruments
used to assess KTE implementation and impact
Irvin E1 , Cole D2 , Keown K1 , Kramer D3 , Brenneman Gibson J1 , Kazman
Kohn M4 , Mahood Q1 , Slack T1 , Amick III B1 , Phipps D5 , Garcia J3 ,
Morassaei S1
1
Institute for Work & Health, Canada; 2 University of Toronto, Canada;
3 University of Waterloo, Canada; 4 St Michael’s Hospital, Canada; 5 York
University, Canada
Background: Knowledge transfer and exchange (KTE) is a burgeoning
practice at research organizations worldwide. However the effectiveness of KTE practices has not been routinely/consistently evaluated.
The need for such evaluation has been voiced (Lavis et al, 2003; Eccles
et al, 2005). One potential reason for the lack of evaluation is the
paucity of tools and methods for evaluation across various disciplines.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
168
Figure 1 (P3A309). Flowchart of documents progressing through review steps.
Objectives: To answer the following question: Are there reliable,
valid and/or useful instruments to apply in the assessment of KTE
implementation and its impact? Methods: A team of researchers
and stakeholders followed a systematic review process developed by
the Institute for Work & Health using a Cochrane-like approach and
a descriptive synthesis. The basic review steps include: formulation
of research question and search terms; decision-maker/stakeholder
meetings; literature search; relevance review; quality appraisal;
data extraction; evidence synthesis; decision-maker/stakeholder key
message generation; and report preparation. Results: The search
was conducted in MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, CAB
Abstracts, LISA, Social Science Abstracts, and Business Source Premier.
The search resulted in 9998 unique titles and abstracts (see Figure 1).
We found 54 quantitative articles that described instruments for the
evaluation of KTE approaches. Data extraction focused on instruments
and measurement properties. Conclusions: Our synthesis of 54
quantitative studies revealed a variety of instruments used to evaluate
KTE approaches. Overall, the description of the measurement properties
was not consistent and, often, details of measurement properties were
lacking or missing. However, a subset of articles, provided descriptions
of instrument development and measurement properties and show
promise as potentially useful tools in the evaluation of KTE approaches.
Key messages from this review include the need to continue instrument
development work and to clearly present the measurement properties
of the instruments used.
P3A310
The impact of systematic reviews on health care policy
Background: Despite increasing emphasis on the use of research to
inform decision making in health care there is concern that systematic
reviews do not have the expected impact on policy and practice and
more work may be needed to enable us to maximise their potential.
Objectives: The aim of this study was to increase understanding of the
likely impacts of systematic reviews on policy and identify factors that
might facilitate their influence. Methods: Nine systematic reviews
published between 1998 and 2008 and used established methods
for the evaluation of research impact, including bibliometrics and
documentary review, to examine whether or not these reviews had
influenced policy development were reviewed. Data from these analyses
were combined with an overview of the literature to identify factors
that might increase impact. Results: The reviews had influenced
the development of national and international policy, although much
of the impact was at a ‘micro’ level in the form of guidelines for
practice. There was considerable variation in the impact of the reviews.
Reviews evaluating fluid resuscitation and road safety interventions
showed the greatest evidence of impact, while a review of qualitative
studies on barriers to fall prevention showed the least amount of
impact. Differences might be explained by time since publication, type
of question, importance to policy makers, the nature and strength
of the evidence, the purpose of the review and the networks and
strategies used for dissemination. Conclusions: Systematic reviewers
should consider the desired impacts of their work early in the review
process and detail appropriate strategies for increasing impact in the
review protocol. This might include specifying methods to address
applicability to particular contexts and devising active strategies for
dissemination.
Frances Bunn1
1
Injuries Group, UK
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
169
P3A312
Determining the role of patient values and preferences
in the development and review of preference-sensitive
practice guidelines
MacLean S1 , Akl E2 , Johnston B1 , Guyatt G1
1 Department of Clinical Epidemiology and Biostatistics, McMaster
University, Canada; 2 Department of Medicine, State University of New York
at Buffalo, NY, USA Department of Clinical Epidemiology and Biostatistics,
McMaster University, Hamilton, ON, Canada
Background: While systematic summaries of the available evidence on
benefits and harms are essential for the development of clinical practice
guidelines, evidence alone is insufficient. Treatment recommendations
involving a trade-off of desirable and undesirable consequences of
alternative course of actions require value and preference judgments.
Previous studies have identified a wide range of possible strategies for
integrating patient values and preferences into guideline development.
Objectives: To assess how clinical practice guidelines incorporate
patient values and preferences into the processes of developing
recommendations and stating them. Methods: We selected guidelines
published between 2007 and 2009 which addressed five conditions
for which incorporation of values and preferences would be important:
breast cancer, prostate cancer, colorectal cancer, end of life care,
and childbirth. We searched the National Guideline Clearinghouse,
the Guidelines International Network, the Scottish Intercollegiate
Guidelines Network, and the National Institute for Health and Clinical
Excellence, using a combination of subject or condition headings, and
keywords. One reviewer carried out the initial search, and two reviewers
independently adjudicated the final inclusion. Results: In the sample of
60 eligible guidelines, we identified 1357 recommendation statements.
Less than 5% of all recommendation statements were explicitly
informed by evidence that pertained to patient values and preferences
(.58%), or included recommendations in support of involving patients
in clinical decision-making (4.9%). Of the 60 guidelines, 24 (40%)
reported that patients were involved in guideline development and/or
review. Twelve guidelines (20%) reported that patient advocacy
group members, or members of patient charity organizations, were
invited to participate, and 15 (25%) guidelines involved community
members in the development and/or review of guidelines. Conclusion:
Patient values and preferences are under-represented in guideline
development.
P3A313
A US hospital-based health technology assessment center
to improve the quality, safety and cost-effectiveness of
patient care
Mitchell M1 , Williams K1 , Umscheid C1
1 Center for Evidence-based Practice, University of Pennsylvania Health
System, USA
Table 1 (P3A313).
Reports requested by
2006–08
2009–11
Chief Medical Officers
12
9
Clinical Department
6
23
Purchasing Committee
12
15
Pharmacy & Therapeutics Comm.
4
14
Quality Improvement Groups
6
12
Nursing Administration
1
3
CDC (US government)
4
3
Administrative Department
1
3
Payer
1
0
Clinical Care Review Committee
0
1
Other
0
3
scientific evidence and applying it to decision making about purchasing,
formularies, and clinical practice. CEP is staffed by two hospitalist
co-directors trained in epidemiology, three HTA analysts, physician and
nurse liaisons, a librarian, a health economist, and an administrator,
totaling 5.5 full time equivalents. Results: Over 150 evidence
reports have been completed to date, 44 in the most recent 12
months. Internal clients requesting reports include clinical departments,
quality administrators and hospital committees. Topics have included
processes of care like the use of heparin versus saline for catheter
flushing; and high-cost and emerging technologies like telemedicine
in critical care. Reports review existing guidelines and systematic
reviews first, and review primary studies when previously published
reviews do not offer sufficient evidence. Local utilization and cost data
are incorporated so reports can be tailored to our medical center’s
needs. CEP then works to implement findings, including integrating
them into computerized clinical decision support, and measures their
impact using administrative and clinical data. Evidence reviews are
shared publicly through the National Guideline Clearinghouse, the
HTA database, and peer-reviewed publications. CEP also offers
education through workshops, a resident elective, and courses for
medical and graduate students. In addition, CEP has developed
collaborations with payors, government organizations, and private
industry. Conclusion: An evidence-based practice center within an
academic medical center can educate clinicians and support a culture
of evidence-based decision-making.
Purpose: Most existing centers for health technology assessment
(HTA) are associated with payers or government agencies. They
most frequently review and analyze emerging and costly technologies.
But hospitals often have to make decisions about processes of care
that impact not only cost, but also on quality and safety of patient
care. Methods: Our academic medical center created a Center for
Evidence-based Practice (CEP) in 2006 for the purpose of gathering
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
170
P3A314
Simple ‘quick’ searches to answer questions arising
in clinical practice
Jordan J1 , Stevenson K2 , Lewis R3
1 Arthritis Research UK Primary Care Centre, Keele University, UK;
2
University Hospital of North Staffordshire, UK; 3 Health Library, Univeristy
of North Staffordshire, UK
Background: Clinicians have limited time to search for literature to
answer questions arising in day-to-day practice. Pre-appraised evidence
summaries are more readily available via sources such as NHS Evidence
and the TRIP Database. However, are these easy for clinicians to use
and will they help clinicians to be able to find answers to their questions
quickly? Objectives: The aim of this project was to assess the ability
of simple searches to answer questions that arise in clinical practice and
highlight some of the barriers that might be encountered. Methods:
We selected clinical questions from Critically Appraised Topics (CATs)
in Musculoskeletal Care produced by allied health professionals and
researchers using traditional structured searches. Questions included
three intervention effectiveness questions, one question on access to
musculoskeletal primary care and a question on the effectiveness of
outpatient appointment duration. Simple searches were carried out for
each of these topics on Clinical Knowledge Summaries, NHS Evidence,
PubMed Clinical Queries and the TRIP Database and compared to the
results of the original CATs. Remember to spell out what these acronyms
mean. Results: Only two of the intervention questions were answered
by the simple search; one of these by a pre-appraised resource, Best
Bets. No new literature was found that had not been found in the
full CAT search. However, the full CAT search did retrieve literature
not found in the simple search but this was based on lower quality
evidence. Some resources searched produced a large volume of hits and
literature had to be appraised and interpreted, which took additional
time and skills. Conclusions: Evidence to answer clinicians’ questions
about the effectiveness of interventions can sometimes be found from
simple literature searches. However, this is not always a quick process
and some judgement and appraisal skills are still required to select the
most appropriate literature. Many questions remain unanswered about
both simple and traditional searching methods.
P3A315
A multi-faceted provider education program designed for
consistent, evidence-based clinical practice change within
an integrated care delivery system
Okawa G1
1 Kaiser Permanente, USA
Background: An ongoing challenge in medical education is demonstrating its ability to help implement evidence-based care, enable
sustainable clinical practice change, and reduce unnecessary variations
in care delivery. In 2009, an integrated care delivery system in the
U.S. launched a multi-faceted organization-based provider education
program strategically designed to improve the consistency and quality
of care. Objectives: The primary objective was to design a provider
education program integrating multiple strategies for guideline implementation and medical education to improve the quality, consistency,
and sustainability of care. This required clarifying practice standards
using the best available evidence, gaining agreement across clinical
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
departments, creating multi-faceted medical education interventions,
and developing operational processes to optimize the care delivery
system. Methods: Executive leadership identifies key clinical quality
gaps and provides sponsorship for the planned interventions.
Evidence-based methods are used to formulate clinical practice
recommendations based on the best available evidence. Multi-faceted
medical education is designed to communicate the key clinical
recommendations. Operational processes, audit and feedback, clinical
champions, and point-of-care decision support are created to sustain
practice change. Objective measures are used to identify baseline
performance and to monitor for process improvement over time. Be
careful about switching from past tense to present tense. Results:
Initial 6-month data shows improvement in several process measures
following medical education interventions. Examples include a 4%
improvement in the number of diabetic patients with a HBA1C of
<9%, a 3.4% improvement in the number of patients achieving
a target LDL <100, and a 14.4% improvement in the number of
referred patients with a complete hematuria evaluation. Conclusions:
Medical education, designed strategically, may be effective within a
large integrated care delivery system in improving the consistency and
quality of clinical care.
P3A316
Effectiveness of quality improvement strategies in
improving the management of diabetes: A systematic
review and meta-regression
Tricco A1 , Ivers N2 , Moher D3 , Turner L3 , Galipeau J3 , Halperin I4 , Vachon
B3 , Ramsay T3 , Manns B5 , Tonelli M6 , Shojania K4 , Grimshaw J3
1 Li Ka Shing Knowledge Institute of St. Michael’s Hospital, Canada;
2
Women’s College Hospital, Canada; 3 Ottawa Hospital Research Institute,
Canada; 4 University of Toronto, Canada; 5 University of Calgary, Canada;
6 University of Alberta, Canada
Background: Diabetes mellitus has reached epidemic levels and
quality of care gaps are persistent. Quality improvement (QI) strategies
are commonly implemented at great cost, but the best approach to
improve the provision of diabetes care remains uncertain. Objectives:
To evaluate the effects of QI strategies on the management of diabetes.
Methods: Systematic review of published and unpublished studies
in MEDLINE (1966 to July 2010), Cochrane Effective Practice and
Organisation of Care (EPOC) database (July 2010), and references
of included trials. Randomized controlled trials (RCTs) examining
11 pre-defined QI strategies targeting healthcare professionals for
managing adult outpatients with diabetes were included. RCTs
lacking data (on glycemic control (HbA1c), blood pressure (BP), or
write-out LDL cholesterol), RCTs focusing only on interventions at the
patient level, and RCTs not written in English were excluded. Two
reviewers independently extracted data and appraised risk of bias
using the Cochrane EPOC tool. Results: Data from 145 RCTs and 20
companion reports were included. Compared with usual care, the QI
strategies statistically improved HbA1c (117 studies, mean difference
[MD] −0.4% [95% confidence interval {CI} −0.5 to −0.3%]), LDL
(45 studies, MD −4.1 mg/dL [−5.8 to −2.5 mg/dL]), systolic BP (66
studies, MD −3.7 mmHg [−4.6 to −2.6 mmHg]), and diastolic BP (61
studies, MD −1.6 mmHg [−2.4 to −0.7 mmHg]). After adjusting for
sample size and other factors, QI strategies with the most significant
improvement included case management, team changes, and patient
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
171
education. Conclusions: Some QI strategies were more successful
than others in improving outcomes. Clinicians and policy-makers can
use these results to tailor the choice of intervention according to the
desired outcome, available resources, and local healthcare context.
P3A317
Exploring the experiences with a new policy for managing
conflict of interest in guideline developers
Neumann I1 , Akl E2 , Guyatt G3
1
Department of Medicine, Pontificia Universidad Catolica de Chile,
Santiago, Chile; MScC Health Research Methodology, Department of
Clinical Epidemiology and Biostatisitcs, McMaster University, Hamilton,
Canada; 2 Department of Medicine, State University of New York at Buffalo,
NY, USA; Department of Clinical Epidemiology and Biostatistics, McMaster
University, Hamilton, Canada; 3 Department of Clinical Epidemiology and
Biostatistics, McMaster University, Hamilton, Canada
Background: The American College of Chest Physicians (ACCP)
9th edition of the Antithrombotic guidelines (AT9) has adopted a
conflict of interest (COI) policy that gives the primary responsibility to
methodologists, equally emphasises intellectual and financial conflicts
of interest, and excludes conflicted individuals from decision-making
regarding recommendations. A qualitative study found that prior to
the implementation of the policy, the methodologists and content
experts were uneasy regarding their respective roles. Also, they
disagreed about the potential effect of the new strategy on the quality
of the guideline. Objectives: To explore the experience of the
methodologists and content experts with the implementation of the
COI policy during AT9 guideline development process. Methods: We
will conduct semi-structured interviews with both methodologists and
content experts. We will ask about their experience with regards to:
(1) giving primary responsibility to a methodologist; (2) emphasis on
intellectual conflict of interest; (3) restrictions based on conflicts of
interest; and (4) perceived impact on the quality of the guideline.
Results: The project is underway and we will present results in
October. Exploring the experience of the individuals affected by the
new COI policy will help understand its feasibility, whether it achieved
utilization of the insight of experts while ensuring that conflicts do not
influence recommendations, and if there are refinement that should be
considered for future guidelines.
P3A318
Judiciary branch and scientific evidence: An analysis of
judicial decisions on galsufase
Marin dos Santos D1 , Atallah A1 , Dallari Jr. H1 , Barroso L1
1 Brazilian Cochrane Centre and Federal University of São Paulo, Brazil
Background: The judiciary branch, in Brazil and other countries, has
been an active actor in public health policies, especially regarding
drug policies. In the present paper, we evaluate the judicial
response to cases regarding Galsulfase, a recombinant form of human
N-acetylgalactosamine 4-sulfatase, designed for the treatment of
patients suffering from mucopolysaccharidosis type VI. Objectives:
To identify the prevalence of scientifically grounded judicial decisions
and the knowledge of Brazilian judges about evidence-based medicine.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Methods: We searched the electronic database of the Supreme Court
(STF), the Superior Court of Justice (STJ) and all Federal Regional
Courts (TRFs), looking for the keywords ‘naglazyme’ OR ‘galsulfase’.
Results: We found 31 judgments, including collegial judgments
(5; 16.12%) and monocratic decisions (26; 83.87%). Of these, 9
were excluded due to procedural reasons (29.03%) and 22 met the
eligibility criteria. Out of them, 20 (90.09%) decisions determined the
government to supply galsulfase. Only 1 decision (4.54%) considered
it improper to deliver the drug due to lack of evidence about its
effectiveness. And 1 decision (4.54%) determined the performance of
forensic expertise. Out of the 22 decisions examined, 18 (81.81%)
were not substantiated by scientific evidence. Other 2 (9.09%) referred
to evidence-based medicine (EBM), but not considered it as a plea for
the decision-making. Only 2 decisions (9.09%) considered EBM as the
basis of the decision-making process. On the other hand, 7 decisions
(31.81%) referred to medical expert opinion. Incompatible outcomes
were associated to galsulfase in 7 decisions. Conclusions: The
judiciary ignores scientific evidence as an aid to the decision-making
process. In order to preserve the right to health and sustainability of the
system, it is necessary that judicial decisions find ground on high-grade
medical evidence. Therefore, it is imperative that the Judiciary branch
becomes more pragmatic and less ideological.
P3A319
High impact collaborations: Examples from the Cochrane
HIV/AIDS group
Horvath T1 , Kennedy GE1 , Rutherford GW1
1 Cochrane HIV/AIDS Group, University of California, San Francisco, USA
Background: Policymakers need high-quality evidence to make
appropriate decisions. United States (US) and international agencies
have sought help from the Cochrane HIV/AIDS Review Group (CRG) in
developing such evidence for guidelines and other policy statements.
Objectives: To ensure that HIV/AIDS policy and guidelines are based
on the best evidence. Methods: We review our experience with
providing systematic reviews for state, national and international
agencies and how they have been used. Results: Between 2001
and 2011 our group has been commissioned to conduct 42 systematic
reviews for state, national and international agencies, including the
US Centers for Disease Control and Prevention (7), the World Health
Organization (31), the California Department of Health Services (3) and
the US Council on Foreign Relations (1). Since 2008, we have conducted
22 reviews (including more than 70 GRADE profiles) for the World
Health Organization. These evidence assessments have been used
to develop worldwide guidelines for antiretroviral therapy of adults,
adolescents and children, prevention of mother-to-child transmission,
prevention of sexual transmission among men who have sex with men,
prevention of sexual transmission among couples, and prevention,
diagnosis and treatment of cryptococcal infection. Conclusions: For
a decade, evidence generated by our CRG has helped guide regional,
national and global HIV/AIDS guidelines.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
172
P3A320
Do Cochrane reviews inform the development of clinical
practice guidelines? The case of hypertension
Al-Ansary L1 , Tricco A2 , Adi Y3 , Al-Ghonaim M4 , Bawazeer G5 , Perrier L2 ,
Tashkandi M2 , Straus S6
1
Dept of Family and Community Medicine, College of Medicine, King Saud
University, Saudi Arabia; 2 Li Ka Shing Knowledge Institute, University of
Toronto, Canada; 3 Shaikh Bahamdan’s Research Chair for Evidence-Based
Health Care and Knowledge Translation, King Saud University, Saudi
Arabia; 4 Dept of Internal Medicine, College of Medicine, King Saud
University, Saudi Arabia; 5 College of Pharmacy, King Saud University,
Saudi Arabia; 6 Li Ka Shing Knowledge Institute, university of Toronto,
Canada
Background: The Institute of Medicine has redefined clinical practice
guidelines (CPGs) as statements that include recommendations
intended to optimize patient care that are informed by a systematic
review (SR) of evidence. It is unclear whether commonly used CPGs
for hypertension meet this standard despite the availability of many
systematic reviews on the management of this condition. The Cochrane
Library, the foremost ‘provider’ of SRs has over 35 reviews on the
management of hypertension. Objectives: To determine the extent to
which recently-developed hypertension CPGs are informed by Cochrane
(or non-Cochrane) SRs. Methods: We searched MEDLINE, EMBASE,
guidelines’ websites and Google for recently published (2006–2010)
CPGs on the general management of hypertension in any practice
setting and that were written in English or have an English abstract.
Two reviewers assessed each guideline for quality using the AGREE
tool and determined whether they were informed by available SRs.
Results: Ten CPGs were identified; 8 included references. Seven CPGs
cited a total of 59 non-Cochrane SRs and/or meta-analysis. Of these,
5 were cited by 3 CPGs and 10 by 2 CPGs. Only 4 CPGs referred to
a total of 14 of the 35 Cochrane reviews (CRs); no CR was cited in
more than one CPG. The country CPGs citing most were: Japan (8)
and Malaysia (3). The relationship between the evidence generated by
the reviews and the corresponding recommendation could not always
be recognized. Conclusions: Despite the increased production of SRs
in general and CRs in particular, few guidelines refer to available SRs.
The reasons for lack of inclusion of SRs need to be explored. And the
need for the Cochrane Collaboration to consider the practical means
by which its knowledge can be translated clearly stands out.
P3A321
Impact of an evidence based pediatrics electronic journal
González MP1 , Buñuel JC1 , González de Dios J1 , Esparza MJ1 , Aizpurua P1
1 Spanish Association of Pediatrics, Spain
Background: In order to give the best care to patients, health
professionals need to integrate the highest quality scientific evidence
with clinical expertise and the opinions of the patients. However
they have to deal with a huge amount of publications. Evidencias en
Pediatrı́a (EeP) (http://www.evidenciasenpediatria.es/) is a secondary
publication of the Spanish Association of Paediatrics with the purpose
of alerting clinicians of important advances in paediatric topics. EeP
reviews 70 journals. The articles selected are summarised in abstracts
and then commented by clinical experts. EeP has been included
in the Cochrane Plus, and recently it has been supported by the
Latin-American Association of Paediatrics (ALAPE). Objectives: To
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
describe the impact of the electronic journal EeP. Methods: Since
March 2010, EeP has been in a new web site. The social bookmarking
has been implemented by creating links to Facebook and Twitter pages
of EeP. We have used Google Analytics to extract the most outstanding
information on the impact and acceptance of the journal. Results: An
amount of 142,377 visits have been received from April 2010 to May
2011, with 51,000 returning visitors (RV) (35.6%). RV meet the criteria
of regular good clients of the site: more number of pages visited (3.12
pages/visit), more time on the site per visit (4.53 minutes) and smaller
bounce rate (51,83%). The sources of traffic used were: referring
sites (48.32%), search engines (28.2%)% (Google was the most used,
26.66%) and direct traffic (23.4%). The referring sites most used
were the paediatric websites, followed by the social webs Facebook
and Twitter. The majority of visitors come from Spanish-speaking
countries, mainly Spain, Mexico and Argentina. Conclusions: EeP is
the first electronic journal of evidence-based paediatrics in Spanish in
the world. It receives a large number of total visits and specially of
returning visitors, who come mainly from Spanish-speaking countries.
P3A322
Do Cochrane reviews drive clinical decision-making at the
point of care?
Moja L1 , Banzi R2 , Cinquini M2 , Moschetti I2 , Pecoraro V2 , Tagliabue L1 ,
Liberati A3
1
University of Milan, Italy; 2 Mario Negri Institute, Milan, Italy; 3 University
of Modena and Reggio-Emilia, Italy
Background: ‘Without Cochrane Reviews, people making decisions
are unlikely to be able to access and make full use of existing healthcare
research’, the Cochrane Collaboration website reports. Although
systematic reviews (SR) are increasingly popular as a form of convenient
synthesis of evidence to support clinical decision-making, it is difficult
to measure to what extent may back practitioners up in their daily
practice. A proxy of this achievement is whether Cochrane Reviews are
utilized at the point of care. Point-of-care summaries provide physicians
with comprehensive, condensed evidence in easily digestible formats.
Objectives: We compared Cochrane and non-Cochrane SR citations
in point-of-care summaries. Our null hypothesis was that the citation
hazard of a relevant sample of Cochrane and non-Cochrane reviews
did not differ. Methods: We selected the top five point-of-care
summaries for coverage of medical conditions, editorial quality and
evidence-based methodology. As samples of relevant non-Cochrane
SRs, we selected all the SRs signalled by two literature surveillance
journals (ACP Journal Club and Evidence-Based Medicine Primary Care
and Internal Medicine). For Cochrane SRs we selected those labelled
as ‘Conclusion changed’ in the Cochrane Library. Parallel sampling ran
from April to December 2009. We measured the occurrence and timing
of Cochrane and non-Cochrane SR citation in point-of-care summaries
from June 2009 to May 2010. We assessed the updating cumulative
rate using Kaplan-Meier survival analyses. Cox model was used to
calculate hazard ratios (HR) between the non- and Cochrane review
citation. Results: The analysis included 128 SRs, 68 non-Cochrane
(53%) and 60 Cochrane (47%). The difference between the nonand Cochrane review citation speed was not significant (HR 1.13, IC
95% 0.87 to 1.46, p = 0.36). Both were cited with similar patterns
by point-of-care summaries. Conclusions: This finding belies down
the criticism that Cochrane reviews are undervalued as a source of
evidence for bedside clinical decision making.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
173
P3A323
Mr. Pickwick and his child went on a field trip and
returned empty handed: the widening gap between
non-randomised studies and evidence for circadian rhythm
sleep disorders management in a paediatric population
Khan S1 , Heussler H2 , McGuire T3 , Pache D3 , Dakin C2 , Cooper D2 ,
Norris R4 , Flenady V5 , Charles B1
1
School of Pharmacy, The University of Queensland, Brisbane, Australia;
2 Sleep and Respiratory Unit, Mater Children Hospital, Brisbane, Australia;
3 Pharmacy- Practice and Development, Mater Health Services, Brisbane,
Australia; 4 Australian Centre for Paediatric Pharmacokinetics, Mater
Children Hospital, Brisbane, Australia; 5 Translation Research into Practice
(TRIP), Mater Medical Research Institute, Mater Health Services, Brisbane,
Australia
Background: Therapeutic protocols in the management of circadian
rhythm sleep disorders (CRSDs) in the paediatric population are currently
inadequate. There is a lack of randomised controlled trials (RCTs) and
an ever increasing number of non-RCTs in this specialised niche.
However, including information from non-randomised studies may
adversely affect therapeutic recommendations. Such concerns have
led the Cochrane collaboration to focus on evaluation of results
from RCTs, to the virtual exclusion of other forms of evidence.
Objectives: To determine the number of excluded studies on melatonin
for the management of CRSDs in visually impaired children and
to describe a general framework by which information from nonrandomised studies might be integrated into evidence. Methods:
1) A retrospective, observational study for determining the number
of RCTs and non-RCTs/observational studies for melatonin in the
management of CRSDs in visually impaired children through the
Cochrane Collaboration [Cochrane Developmental, Psychosocial and
Learning Problem Group (CDPLG)] was conducted. 2) A framework
was developed to include non-randomised studies into evidence: (a)
Conceptualisation of question under study (determining the internal
and external validity; differentiating the efficacy and effectiveness
trials); (b) Developing a conceptual model (based on CRSDs therapeutic
outcome measures); (c) Recommendation of proposed for inclusion
of non-RCTs / Observational studies. Results: A total of 9 studies
of various designs were evaluated (3 case series; 3 non-RCTs; 2
randomised, but design-flawed studies and 1 N-of-1 trial). None met
the Cochrane inclusion criteria. A framework for potential inclusion of
such studies is provided. Conclusions: A null evidence report, based on
inclusion criteria, runs the ethical risk of excluding potentially relevant
practice-changing evidence. It thus unfairly evaluates the current
practice as lacking a sufficient evidence basis. The proposed framework
may assist to strengthen the quality and documentation of evidence
from non-randomised studies in highly specialized therapeutic areas.
P3A324
Including diverse study types in systematic reviews
to address multiple questions about public health
interventions: The case of a review of school effects
on health
Harden A1 , Bonell C2 , Jamal F1 , Wells H2
1 University of East London, Institute for Health and Human Development;
2 London school of hygiene and tropical medicine, department of social and
environmental health research
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: There is increasing interest in addressing multiple
questions in systematic reviews using a wider range of evidence,
particularly in relation to broad and multi-faceted interventions such
as those typically implemented within public health. Objectives: This
study explores how diverse types of evidence including trials, process
evaluations, theory, ‘non-intervention’ quantitative and qualitative
studies can be combined using a review on the effects of schools
and school environment interventions on health as a case example.
Methods: Exhaustive searches were conducted and the results
screened to identify relevant studies. Each study type underwent
separate critical appraisal, data extraction and synthesis using specially
developed tools tailored to each type of study. All five syntheses
were integrated to draw conclusions and make recommendations.
Results: Each type of study was used to address a different question
within the review. Trials were used to estimate the effectiveness
of interventions and process evaluations to address questions about
intervention acceptability, feasibility and context. The quantitative
non-intervention studies were used to examine the range of school-level
factors which influence health and might form the focus of future
intervention studies (these studies examined whether differences in
health between schools could be explained by school level factors).
The qualitative studies examined how students viewed the impact of
schools on their health. Conclusions: This approach to combining
diverse types of evidence in a single review proved especially valuable
here as the interventions under study – those which aim to change the
school environment – are at a relatively early stage of development.
Current intervention studies have not addressed the full range of
potential determinants of school health. Findings from the different
study types were integrated to: build new theory for how schools affect
health; suggest potential new avenues for intervention research; and
refine logic models for interventions.
P3A325
Measuring organizational readiness for knowledge
translation in chronic care: A mixed methods systematic
review on theories and instruments
Gagnon M1 , Légaré F2 , Ouimet M3 , Estabrooks CA4 , Roch G1 ,
Labarthe J5 , Ghandour EK5 , Tremblay N5 , Grimshaw J6
1
Research Center of the Centre Hospitalier Universitaire de Québec,
Québec, Canada and Faculty of Nursing, Université Laval, Québec,
Canada; 2 Research Center of the Centre Hospitalier Universitaire de
Québec, Québec, Canada and Department of Family Medicine, Université
Laval, Québec, Canada; 3 Research Center of the Centre Hospitalier
Universitaire de Québec, Québec, Canada and Department of Political
Science, Université Laval, Québec, Canada; 4 Faculty of Nursing, University
of Alberta, Edmonton, Alberta, Canada; 5 Research Center of the Centre
Hospitalier Universitaire de Québec, Québec, Canada; 6 Ottawa Hospital
Research Institute, Ottawa, Canada and Faculty of Medicine, University of
Ottawa, Ottawa, Ontario, Canada
Background: Organizational readiness for change (ORC) assesses
organizational members’ collective motivation and capability to implement change. However, the theorization of ORC lacks consensus
and the available ORC instruments have shown limited validity and
reliability. Objectives: We aim to develop an evidence-based,
comprehensive, and valid instrument to measure organizational readiness (OR) for knowledge translation (KT) in chronic care. The existing
evidence on ORC will be reviewed and synthesized as the basis for
the development of a comprehensive, bilingual OR for KT instrument.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
174
Methods: Phase 1: We are conducting a mixed-methods systematic
review on theories and instruments assessing ORC in health care. We
will synthesize the findings in a conceptual map. A bibliography and
a database of the ORC instruments will be prepared based on their
psychometric appraisal. A pan-Canadian Delphi study will be carried
out for the contextual assessment of these instruments. Phase 2:
Individual interviews and focus groups will be conducted with key
stakeholders for further development of the proposed instruments.
A final bilingual (French/English) OR questionnaire will be tested
in the field of chronic care to measure KT regarding the adoption
of comprehensive, patient-centered and system-based chronic care
models. Results: The initial review process retained 98 articles from
a database of 2903 articles for full-text evaluation. After evaluation
by two independent evaluators, 57 met the inclusion criteria and
were critically appraised using quality criteria specific to quantitative,
qualitative, and mixed-methods designs. Preliminary findings suggest
a lack of consensus on the theoretical domains involved in ORC and
limited evidence of ORC instruments’ validity. Conclusions: This study
provides a comprehensive synthesis and aims at creating a consensus
on the theoretical underpinnings and the instrumentation of ORC.
The final product, a validated, comprehensive, bilingual instrument to
assess ORC for KT will be useful for supporting the implementation of
evidence-based chronic care practices.
P3A326
Use of Cochrane Reviews in NICE Clinical Guidelines
Alderson P1 , Tan T1
1 NICE, UK
Background: It is routine practice when developing clinical guidelines
for NICE to look for relevant Cochrane reviews. The NHS funds
NICE and makes a substantial contribution to the funding of the
Cochrane Collaboration. We wanted to see how many reviews have
been used. Objectives: To collect citation rates for Cochrane
reviews in NICE clinical guidelines and map cited reviews to guideline
questions. Methods: We used free text searching for the word
’Cochrane’ in all NICE clinical guidelines published before March 2011
to identify citations of Cochrane reviews. We examined the use of
the review as for background only or as a source of evidence for the
guideline development group. We have also mapped, where possible,
the questions addressed in NICE clinical guidelines, and are cross
referencing this with the reviews cited. Results: NICE had published
116 clinical guidelines by the end of February 2011. Ten have been
replaced by more up to date guidelines, leaving 106 guidelines. There
were 731 citations of Cochrane reviews in the guidelines, ranging
from 0 to 44 with a mean of almost 7. We judged that 23 of the
citations were used purely as background information. 46 of the
currently registered CRGs had reviews cited. We will also suggest how
our mapping exercise can be used to identify where Cochrane reviews
could have been better used, and as a source of information for CRGs
when prioritising reviews. Conclusions: Cochrane reviews are widely
cited in NICE clinical guidelines, emphasising the very important role
of the reviews in helping to inform guideline recommendations.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P3A327
Developing a web-based international register of clinical
prediction rules for primary care
Keogh C1 , Wallace E1 , Fahey T1
1 Royal College of Surgeons in Ireland, Ireland
Background: Clinical medicine is becoming increasingly interested in
clinical prediction rules (CPRs) and their potential for improving patient
care. However, identifying CPRs for primary care from electronic
databases is difficult because several terms are used interchangeably
to describe them. Objectives: The aim of the current work is to
develop an international register of CPRs relevant to primary care to be
distributed through the Cochrane Primary Health Care Field. Methods:
The register is being developed by identifying relevant articles from
various sources including: (1) the MEDLINE database [1980–2008]
using an electronic search string, specifically developed and tested
in-house; (2) other electronic resources (e.g. JAMA rational clinical
examination series); and (3) contacting authors in the field for published
and unpublished work. Relevant articles are classified according to a
set of criteria including condition specific codes, level of evidence and
quality assessment. Results: To date, the register consists of 300
articles that contain CPRs relevant to primary care, covering 16 different
broad clinical areas. The largest number of articles was identified for
cardiovascular conditions (n = 93) and the least identified for eye
conditions (n = 1). These broad areas are further classified according
to specific conditions. The register identified CPRs across all levels of
evidence. The majority of articles were concerned with validating rules
(narrow validation n = 99, broad validation n = 53). Few articles
were concerned with formal impact analysis (n = 8). Assessment of the
quality of the published articles produced mixed results. Conclusions:
Although the register will need to be regularly updated, it offers many
potential benefits for primary care practice. Classification of articles
allows the clinician to objectively assess the utility of the CPR for use
with patients. The register will also assist researchers to identify clinical
areas in which CPRs have not yet been derived, as well as CPRs that
have not undergone formal impact analysis.
P3A328
Synthesis of recommendations for clinical use
of biomarkers in oncology
Trevisiol C1 , Pregno S2 , Rosin C3 , Fabricio ASC4 , Gion M3
1
Istituto Oncologico Veneto IOV – IRCCS, Padua, Italy; 2 Cochrane Italian
Center, Italy; 3 Regional Center for Biomarkers, Department of Clinical
Pathology, Azienda ULSS 12 Veneziana, Venice, Italy; 4 ABO Foundation/
Regional Center for Biomarkers, Department of Clinical Pathology, Azienda
ULSS 12 Veneziana, Venice, Italy
Background: the widely observation of inappropriate use of biomarkers in clinical practice and limited economic resources require
strategies to translate the knowledge on their clinical utility and
limitations to their users. Objectives: to produce synthesis of
recommendations for clinical use of biomarkers in 17 neoplasias for
all clinical spectrum together with AGREE critical appraisal, having the
purpose to represents systematic comparison of guidelines addressing
similar topic areas and facilitate their use by end-user in a independent
and critical way. Methods: the project was developed in the
last 2 years adopting a transparent, explicit, and systematic EBM
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
175
approach, involving a working group of 60 professional figures with
expertises in different health and economic fields. The search of
the guidelines was conducted following systematic methodological
criteria and the selection was conducted by 2 independent reviewers
using pre-settled criteria. Recommendations regarding biomarkers with
the corresponding strength of evidence and recommendation rating
schemes were extracted and recorded in tables and then synthesised
into synoptical tables. The critical appraisal was conducted using
the AGREE-I instrument by at least three independent professional
figures from the working group. In addition, more in-depth original
methodological tables were produced to describe the strengths and
limitations of the body of selected evidence. Results: from over 3500
abstracts in the English and Italian language published in the last seven
years (using MEDLINE and the principal guidelines databases) 52 EBM
guidelines were selected and critically appraised between February and
June 2010. The synoptical tables and the AGREE evaluation were
published in a document in October 2010. Conclusions: the final
document synthetises the available recommendations for each clinical
spectrum and at the same time supplies a synthetic methodological
quality appraisal of the guidelines. This approach allows the users
to make critical and informed decisions on biomarkers use in clinical
practice oncology.
P3A329
Collaborative knowledge networks (CKN) for guideline
development: A Kaiser Permanente and BMJ evidence
centre case study
Davino-Ramaya MD C1 , Brunnhuber MD K2 , Haynes MPH J1 , Simpkins MD
C2 , Tom MS G1 , Minhas MB ChB R2 , Robbins MD C1
1 Kaiser Permanente USA; 2 British Medical Journal Evidence Center UK
Background: As we experience one the greatest explosions in
scientific knowledge of our time, guideline developers seek to
explore creative and innovative strategies to address the challenges
of resource constraints. Developing a short-term Collaborative
Knowledge Network (CKN) provides the necessary foundation and
serves as a catalyst for the development of a clinical practice
guideline (CPG) through intellectual sharing, capacity expansion and
quality enhancement. Objectives: Emphasizing effective knowledge
translation and communicating evidence, we present a case study of
the CKN between infectious disease experts, the Kaiser Permanente
(KP-USA) guideline development team, and the evidence team at
BMJ Evidence Centre (BMJEC-UK) during the development of a new
sexually transmitted disease (STD) CPG. Methods: We describe the
structure of the KP/BMJEC-CKN; the flow of information between
key stakeholders; and the measures set in place to ensure high
levels of quality, transparency, and rigor in communicating the
requirements of the guideline development team, clinical knowledge
of the guideline panel, and management of the large amount of
data and information resulting from an elaborate evidence synthesis
process. Results: The partnership provides insight into the benefits and
challenges of an international cross-collaboration in evidence synthesis
during the development of a HIV/STD CPG. It demonstrates the need
for: – agreement on methodological approach including GRADE; – early
identification/definition of roles; – on-going collaboration; multidirectional communication; – transparent, rigorous and regular documentation - aligning and setting clear timelines of evidence creation and
guideline team work (problem formulation/clinical questions, evidence
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
summaries/systematic reviews); – use of agreed templates; – rapid
feedback and query loops; – process sharing/evaluation. Conclusions:
Collaborative Knowledge Networks for guideline development have
the potential to increase quality and capacity of guideline developing
organizations, to strengthen guideline methodology, to provide
transparent and systematic processes while serving as a repository
of evidence synthesis for global availability to develop localized clinical
decisions.
P3A330
Reporting guidelines for systematic review protocols
Moher D1 , Shamseer L1 , Clarke M2 , Ghersi D3 , Liberati A4 , Petticrew M5 ,
Shekelle P6 , Stewart L7
1 Ottawa Hospital Research Institute, Canada; 2 Queen’s University Belfast,
Ireland; 3 World Health Organization, Switzerland; 4 University of Modena,
Italy; 5 London School of Hygiene & Tropical Medicine, UK; 6 RAND
Corporation, USA; 7 Centre for Reviews and Dissemination, UK
Background: Systematic review protocols are seldom published
(with some exceptions, such as the protocols for Cochrane reviews).
Furthermore, for systematic reviewers interested in publishing their
protocols, there is limited guidance currently. Where protocols are
accessible, they show what was – and was not -planned for the
review which might not be clearly understandable in published reports.
Objectives: To develop a guideline to aid authors when preparing and
reporting protocols. This will extend the PRISMA Preferred Items for
Reporting Systematic Reviews and Meta-Analyses (PRISMA) Statement,
to provide guidance for protocols (PRISMAP). Methods: Development
of PRISMAP will follow the process for reporting guideline development
designed by the EQUATOR (Enhancing the QUAlity and Transparency
Of health Research) group. Potential checklist items were identified
through the Delphi process and extensive consensus-building during the
development of PROSPERO – an international register for systematic
review protocols. The PROSPERO items will be debated and further
refined into a checklist for the reporting of systematic review protocols
during a June 2011 meeting of international experts, facilitated by the
developers of PROSPERO and experts in the development of reporting
guidelines. The draft guideline with accompanying checklist will be
shared at the 2011 Cochrane Colloquium. Potential Impact: The
availability of a tool to help Cochrane and other systematic reviewers
create and report protocols will hopefully improve the quality of both
protocols and the subsequent reviews. PRISMAP might also make it
easier for readers and peer reviewers to identify selective reporting
biases in systematic reviews.
P3A331
An extrapolation checklist to answer ‘Can I use the results
of this systematic review in a disadvantaged population?’
Ueffing E1 , Petticrew M2 , Welch V1 , Clarke M3 , Gruen R4 , Guyatt G5 ,
Kristjansson E1 , Mercer S6 , Pardo Pardo J1 , Tugwell P7
1
University of Ottawa, Canada; 2 London School of Hygiene and Tropical
Medicine; 3 Queens University Belfast, Northern Ireland; 4 Monash
University, Australia; 5 McMaster University, Canada; 6 CDC Guide to
Community Preventive Services, USA; 7 Medicine & Epidemiology and
Community Medicine, Faculty of Medicine, University of Ottawa, Canada
Research Chair in Health Equity, Canada
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
176
Background: There is a need for guidance on how knowledge
users can apply the results from Campbell and Cochrane reviews to
disadvantaged groups. Knowledge users are under pressure to consider
the effects of policies and programs on health equity. Objectives:
To develop an extrapolation checklist to help decision-makers and
practitioners decide when the results from systematic reviews are
applicable to disadvantaged populations/settings. Methods: We
developed an extrapolation checklist by drawing on items from
existing checklists and consulting healthcare practitioners, researchers,
consumers and policy-makers. We held a pilot session with Cochrane
methodologists, review authors, and staff to test this extrapolation
checklist with specific scenarios which asked them to apply results of
Cochrane reviews to specific disadvantaged populations (e.g. women,
people in low- and middle-income countries, the poor). We then
refined the checklist and the scenarios. Results: We created 2
checklists: one for populations (to be used by policy-makers) and one
for patients (to be used by practitioners). Feedback during a workshop
and from one-on-one consultations suggested further refinement to
operationalize the checklists as a decision aid, to take into account
the relative and absolute effects, quality of evidence, values of the
intended recipients of the intervention (population or patients), and
other issues such as costs. Conclusions: Our work suggests that using
a systematic review involves complex decisions which could benefit
from structured decision support tools, similar to decision support tools
developed for patients.
P3A332
Systematic review of the impact of reporting guidelines
on publication quality
Moher D1 , Palepu A2 , Shamseer L1 , Turner L1 , Altman D3 , Hirst A3 ,
Hoey J4 , Schulze K5 , Simera I3
1 Ottawa Hospital Research Institute, Canada; 2 McMaster University,
Canada; 3 Centre for Statistics in Medicine, UK; 4 University of Toronto,
Canada; 5 Family Health International, USA
Background: Reporting of health research is often inadequate.
To this end, many reporting guidelines (RGs), aimed at improving
the quality of health research reports, have been developed for
reporting a wide variety of specific types of research. A RG typically
consists of a checklist, flow diagram or explicit text to guide authors
on what should be included in a research report. Despite their
emergence, RGs are underused and published health research continues
to be poorly reported. Synthesizing evidence of their impact on the
quality of reporting will help identify the most effective guidelines and
motivate their use. Objective: To systematically review and, where
possible, synthesize evidence on the impact of RGs on the overall
transparency of health research reporting. Methods: This systematic
review will investigate the impact of 81 RGs identified by an earlier
systematic review, as well as newly eligible guidelines identified by
the EQUATOR Network. A search strategy has been designed to
identify evaluations of each RG. We will explore characteristics of
RGs and journals associated with their use. Analysis methods will
follow those of a recently completed systematic review of evaluations
of the CONSORT Statement – a reporting guideline for randomized
trials. Preliminary findings for this review will be presented at the
Cochrane Colloquium. Potential Impact: This project aims to provide
evidence to help guide decision making for journal editors. Knowledge
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
translation activities will include a meeting of journal editors to help
develop next steps for RG implementation based on the results of this
review, since it is likely a prudent policy to endorse and adhere to
those RGs for which there is some evidence on their effectiveness.
Ultimately, this review will help increase the uptake of RGs by different
user groups and contribute to the improvement in health research
reporting.
P3B333
Developing a typology of activities for linking research
evidence to research use
Tripney J1 , Kenny C1 , Gough D1
1 EPPI-Centre, SSRU, Institute of Education, London
Background: One of the main objectives of using systematic methods
for primary research synthesis is to inform policy and practice decision
makers. However, the research on methods to increase the uptake
of research by decision makers is at an early stage of development.
The research is held back by a lack of theory and a lack of concrete
measures of interventions and outcomes (see for example, Grimshaw
2004). Objectives: To develop a typology for activities that link
research evidence to policy making, facilitating future studies on
research utilization (as part of a European Commission funded study).
Methods: A simple model of the evidence to policy system was
developed based on previous work by Best (2010) Levin (2004), and
Nutley et al (2003). A classification system for describing the content
of activities and the mechanism by which they linked research to
policy was developed from: (i) theories of evidence use; (ii) the system
developed by Nutley et al (2003); (iii) a survey of such activities in
Europe. Results: A typology of 27 activity types and 9 mechanisms that
enabled the link between research evidence to policy was developed.
Conclusions: The typology provides practical means for clarifying the
nature of knowledge translation activities and for more focused and
fruitful studies on research production and use. The typology will be
further refined though application to further examples of research to
policy linking activities Worldwide.
P3B334
Agree instrument: A good instrument for family practice
training
Stein AT1 , Khan GS2
1 Grupo Hospitalar Conceição/Ufcspa/Ulbra, Brazil;
Conceição/Ulbra, Brazil
2 Grupo
Hospitalar
Background: The Appraisal of Guidelines for Research & Evaluation
(AGREE) Instrument was developed to address the issue of variability in
guideline quality. Underutilization of evidence-based research is often
described as a gap between ‘what is known’ and ‘what is currently
done’ in practice settings. Objective: to introduce the AGREE
instrument for family physicians in a residency programme. Methods:
The Community Health Service of Conceição Hospital Group in Porto
Alegre, Brazil has been training family physicians and other health
professionals for over 20 years. There is a need to improve primary
health care in Brazil. The standard of care in this service has been
well established by various audit researches and several guidelines
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
177
have been produced by this group, although no evaluation has been
performed with regards to the quality of these guidelines. There are
15 preceptors who have evaluated a Clinical Practice Guideline (CPG)
which has been produced for the Brazilian Medical Association in liaison
with the Brazilian Association of Family Practice on Management of
Dementia in Primary Health Care. A workshop has been conducted to
introduce the AGREE instrument and a questionnaire evaluating their
perception on the importance of this instrument in the process of care.
Results: Family physicians have found that the AGREE instrument has
been very useful on the evaluation of the quality of CPG. They have
identified that clinical practice guidelines when rigorously developed
using a transparent process that combines scientific evidence, clinician
experimental knowledge, and patient values are essential for a good
standard of medical care. Conclusion: The process of using the AGREE
instrument for family physicians has emphasized on establishing the
process of knowledge translation and a good strategy for training of
family physicians in Brazil.
P3B335
Access to The Cochrane Library: Results of Canada’s
National Pilot License Initiative
Squires J1 , Grimshaw J2 , Schaafsma ME3
1
Ottawa Hospital Research Institute, Canada; 2 Ottawa Hospital Research
Institute and The Canadian Cochrane Centre, Canada; 3 The Canadian
Cochrane Centre, Canada
Background: The Cochrane Library is one of the most reliable sources
of health information about ‘what works’ in healthcare. Yet access in
Canada is limited; approximately 10% of Canadians have access to The
Cochrane Library. In response, a national pilot license was instituted
that gave all Canadians ‘one-click’ access to The Cochrane Library for
one year. Objective: To determine the impact of a national license
on use of The Cochrane Library in Canada. Methods: We performed:
(1) an interrupted time series study of use of The Cochrane Library in
Canada and (2) a cross-sectional anonymous survey of users of the
Library (n = 1192). Monthly use of The Cochrane Library (in terms of
the number of: full-text downloads, access denials, abstracts viewed,
and searches conducted) for the pilot period was compared with the
expected numbers based on the 12 months preceding the pilot. The
survey assessed characteristics of users of The Cochrane Library and
their attitudes towards the Library. Results: Use of The Cochrane
Library increased as a result of the pilot license. During the pilot, the
number of full-text downloads and abstracts viewed were 65% (95%
CI 34%, 96%) and 7% (95% CI 0%, 15%) higher than expected,
respectively. The number of access denials and searches conducted
decreased; there were 83% fewer denials (95% CI −97%, −70%)
and 13% fewer searches (95% CI −20%, −6%) than expected.
Attitudes towards the Library were positive with 88% and 91%
of survey respondents reporting the Library as important to making
good healthcare decisions and for all Canadians to have access to,
respectively. Conclusions: Canadians want better access to high
quality health knowledge. The provision of a national pilot license to
The Cochrane Library in Canada significantly increased its use and, as a
result, has the potential to positively impact the healthcare Canadians
receive.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P3B336
Do value judgments play a role in evidence-based
recommendations when information is based on proof
of concept?
Hutton B1 , Lee K1 , O’Grady T1 , Shukla V1
1 Canadian Agency for Drugs and Technologies in Health, Canada
Background: The Canadian Expert Drug Advisory Committee (CEDAC)
delivers evidence-informed formulary recommendations for drug
reimbursement based on data from clinical systematic reviews and
critiques of health economic information. Past work suggests
that value judgments play a role in evidence interpretation and
recommendations. Objectives: To explore the role of value judgments
in CEDAC recommendations. Methods: CEDAC recommendations
from 2009–2010 were reviewed. Recommendations were evaluated
which had (1) a partial or full listing, where evidence consisted of
only placebo controlled trials; and (2) a do not list recommendation,
where evidence consisted of only active comparator trials. Information
was gathered on factors potentially influencing the recommendation,
such as: drug novelty (i.e. first drug for indication or first in drug
class); indication risk and rarity; insufficiency of evidence; study
quality; economic considerations; and harms concerns. Results: Nine
partial or full listing recommendations were identified based on only
placebo controlled trials, and 8 ‘do not list’ recommendations, based
only on active comparator trials, were identified. Of the nine ‘list’
recommendations, factors were related to therapeutic need (3/9) and
potential for cost savings (6/9). Reasons underlying the 8 ‘do not
list’ recommendations related to cost/cost-effectiveness (8/8; increased
cost or uncertainty around cost-effectiveness), insufficiency of evidence
versus comparators (7/8; lack of studies versus key comparators, lack
of meaningful improvements over other active therapies, inconsistency
of findings across studies), and study quality (6/8; issues regarding
blinding or withdrawals/missing data, appropriateness of non-inferiority
margin, other design issues). For the ‘do not list’ category, recommendations were largely made in consideration of ≥2 criteria. Conclusions:
The absence of active comparator trials may not be the determinant
of a recommendation as value judgments related to the reliability of
clinical trial data, therapeutic need, and other factors play a role in
CEDAC recommendations.
P3B337
Proof of value decisions from proof of concept data: A
review of Canadian formulary review recommendations
Shukla V1 , O’Grady T1 , Clifford T1 , Hutton B1
1
Canadian Agency for Drugs and Technologies in Health, Canada
Background: Regulatory approval of drugs is often based on trials
designed to demonstrate proof of concept. However, such studies
are not typically designed to simultaneously demonstrate proof of
value in a formulary recommendations setting. Objectives: To
enumerate types of inadequate information in drug submissions which
received a ‘do not list’ recommendation from the Canadian Expert
Drug Advisory Committee (CEDAC). Methods: CEDAC ‘do not list’
recommendations from 2009–2010 were reviewed. Recommendations
with cited reasons involving inadequate information were identified,
and recorded. A summary of these reasons was compiled. Results: Of
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
178
Table 1 (P3B337). Types of Inadequate Information in CEDAC ‘‘do not list’’ recommendations (2009–10).
Frequency as a Cited Factor (% of total # of recommendations)
Submissions based on
only placebo-controlled
trials (n = 8)
Submissions based on
only active comparator
trials (n = 6)
Submissions based on a combination
of placebo controlled and active
comparator studies (n = 5)
Lack of active comparator trials
5 (63%)
1 (17%)
3 (60%)
Insufficient evidence of meaningful benefit vs placebo
2 (25%)
0 (0%)
2 (40%)
Insufficient evidence of meaningful benefit vs active therapy
0 (0%)
5 (83%)
1 (20%)
Failure to provide data for a key outcome measure
2 (25%)
1 (17%)
2 (40%)
Use of a surrogate outcome considered to be limited value
3 (38%)
0 (0%)
0 (0%)
Uncertainty of response durability over time
1 (13%)
0 (0%)
0 (0%)
Limited study population generalizability
1 (13%)
0 (0%)
1 (20%)
Cited Influential Criteria
26 ‘do not list’ recommendations, a total of 19 (73%) were associated
with one or more forms of inadequacy of information. Amongst
them, 10/19 (53%) were associated with inadequate evidence of a
clinically meaningful benefit compared to either placebo or an active
comparator, 9/19 (47%) were associated with a lack of trials versus
any active comparator or a particular active comparator of interest,
5/19 (26%) failed to report data for a key outcome measure, 3/19
(16%) used a surrogate outcome considered to be of limited value,
and 2/19 (11%) were associated with limited generalizability in terms
of the study populations. Table 1 summarizes and highlights variations
in the types of information inadequacies that were noted according
to the type of evidence used in the submission. Conclusions:
Proof of concept studies don’t always provide sufficient information to
show proof of value, and represent a challenge for recommendation
panels. In such situations, value judgments play an important role
due to uncertainties regarding the information available. Carefully
planned trial designs involving relevant comparators, evaluating
outcomes relevant to the interests of regulators, physicians and
patients, will increase the likelihood of a recommendation for formulary
listing.
P3B338
Lowering lipids enhances the effect of statins in preventing
cardiovascular outcomes
Contreras J1 , Schmid C1 , Alsheikh-Ali A2
1 Institute for Clinical Research and Health Policy Studies, Tufts Medical
Center, USA; 2 Institute of Cardiac Sciences, Sheikh Khalifa Medical City,
UAE
Background: Multiple randomized clinical trials have demonstrated
that lowering low density lipoprotein (LDL) cholesterol reduces cardiovascular (CV) outcomes (death, myocardial infarction (MI), stroke).
Meta-analyses to date have focused on treatment comparisons with
respect to single outcomes, not addressing the potential competing
risks among these correlated events to compare multiple treatments
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
across multiple outcomes. Objectives: To assess the effect modification
of changes in LDL and other lipid measures on a network of treatment
effects applied to a set of correlated outcome measures using multiple
treatments meta-analysis (MTM). Methods: Data from 23 large
randomized controlled trials comparing lipid lowering agents, statins,
fibrates and conventional treatment at various doses and with different
amounts of follow up were combined. Each trial reported some or all of
6 mutually exclusive outcomes: fatal or non-fatal MI, fatal or non-fatal
stroke, or other fatal or non-fatal events. We applied a new Bayesian
multinomial multiple treatments meta-regression model to estimate the
effect of the different treatments on the different outcomes at different
lipid levels. This model could account for the missing information
on some outcomes in some trials. Results: When LDL cholesterol
was reduced by 30 mg/dl, the average risk reduction was 26% for
MI, 18% for stroke and 10% for death. The reductions were further
reduced for each outcome by 5% for every additional 10 mg/dl.
Smaller effects were found for other lipid measures. Reductions
were greatest for high dose statins. Conclusions: Meta-analysis of
large statin trials demonstrates that statins effectively reduce different
CV events at different rates and that these rates depend on the
amount of lipid reduction achieved in the trials and on the doses
employed.
P3B339
Resource se compared to evidence of effectiveness as
an indirect measure of clinical quality and a basis for
improvement
Harris J1
1
The Permanente Medical Group, USA
Background: Analysis of large health care claims databases indicate
that many tests and treatments are done for indications not supported
by evidence, or in quantities in excess of those supported by evidence.
Findings of this sort, coupled with variance in care for groups of cases of
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
179
similar diagnoses, suggest that health care efficiency and effectiveness,
key elements of quality, could be improved by adhering to practices
supported by evidence of effectiveness. Objectives: 1. To review and
evaluate systematic reviews of tests and treatments for low back pain. 2.
To present examples of care in excess of evidentiary support for low back
complaints. 2. To suggest means to reduce inappropriate and variable
use of health care resources. Methods: We identified a number of
systematic reviews and evidence-based guidelines for effective care of
low back pain. We assessed them and collected recommendations
and time frames for tests and treatments. We analyzed very large
medical claims databases from the states of California, Texas and Ohio
and compared the tests and treatments used per case to evidence of
effectiveness. Results: There is good evidence for the effectiveness or
lack thereof of imaging, pharmaceutical treatment, physical modalities,
and procedures for low back pain, radiculopathy, and other common
diagnoses. Our studies revealed testing and treatment far in excess
of evidentiary support. Higher levels of treatment are associated with
worse outcomes, including disability absence. We also found some
evidence of improvement in these patterns following organizational or
public policy interventions, but in some cases it was not sustained.
Conclusion: Evidence-based best practices for back care are not
uniformly applied. There is wide variation in practice patterns. Effective
means to improve the consistency and appropriateness of practice are
needed to improve outcomes.
P3B340
A Canadian example of facilitating the uptake of syntheses
into public health practice
Docorby K1 , Tirilis D1 , Husson H1 , Robeson P1 , McCrae L1 , Greco L1
1 McMaster University, Canada
Background: The Canadian Institutes of Health Research (CIHR)
are mandated to ensure the results of the research it funds lead to
improved health and health systems. However, current knowledge
translation (KT) efforts may be insufficient to promote the uptake of
these CIHR funded reviews into policy and practice in Canada. The
Cochrane Collaboration, review groups and systematic review authors
face similar challenges in facilitating the uptake of reviews into health
policy and practice. Objectives: The purpose of this project is to
identify high quality CIHR funded syntheses relevant to public health
practice, and to promote the uptake of this evidence into policy and
practice. Methods: Eleven methodologically rigorously completed
CIHR funded reviews were identified with another 12 reviews expected
to be completed during 2011. Short summaries highlighting what
the evidence is and corresponding implications for policy and practice
will be written for each review. The multi-pronged KT strategy to
promote uptake of the evidence includes tailored, electronic messages,
webcasts, and topic specific online discussion groups moderated by a
knowledge broker. Results: We will evaluate these KT strategies by
tracking how often the summaries and full text reviews are accessed at
www.health-evidence.ca. Conclusions: At the time of the Cochrane
Colloquium in October 2011, we will be nearing the completion of the
KT strategy for this project and will be able to report on the extent
to which we have been able to reach and engage the public health
community in these CIHR funded syntheses. Lessons learned from this
project will provide useful insight to those intent on promoting the
uptake of Cochrane reviews.
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
P3B341
Bringing best evidence to front line clinicians
Tejani A1 , Adlparvar C2
1
Fraser Health Pharmacy Services, Vancouver, Canada; 2 Therapeutics
Initiative & Hypertension Review Group, Vancouver, Canada
Background: Continuing education activities for clinicians are often
not based on best evidence and have relied on pharmaceutical industry
support, which can be problematic. The Therapeutics Initiative has
developed a strategy to overcome these issues. Objectives: 1. To
provide ethically sound source of knowledge based on systematic
reviews that is not dependent on industry funding. 2. To establish
a sustainable and collaborative knowledge exchange environment
between physicians and pharmacists. 3. To develop tools to transfer
evidence from systematic reviews into practice that accommodate
different learning styles and different access to educational opportunities.
Methods: We formed of a team of knowledge-users, academic
researchers, and educators to develop a comprehensive multi-faceted
continuing education strategy. We created learning environment that
enables physicians and pharmacists to learn together. 3) developed
the following educational tools that cater to the needs of different
clinicians with different warning styles and different access to continuing
education opportunities. • website with access to all educational tools
• medical evidence podcasts • evidence summaries • case-based
workshops • critical appraisal courses. We developed an educational
strategy that is primarily grant funded and has no pharmaceutical
industry input. Results: There were three educational events with 15
topics and over 250 practitioners in which over 90% of participants
agreed; • the information was practical, • it was productive to work
with colleague, • provided excellent information and • would attend
again. Material for these events was used subsequently for over 50
presentations with smaller groups at hospitals in British Columbia (BC)
Canada. There were over 20 presentations onteaching clinicians how
to critically appraise the literature, four therapeutic letters which tere
senw to over 14000 physicians and pharmacists in BC and over 5000
online users five podcast which elaborate on the systematic reviews
and newsletters. Conclusions: The Therapeutics Initiative has been
successful in providing best evidence to front-line clinicians in an
ethically sound environment.
P3B342
Evidence to support process improvement: What’s
available, and how to proceed
Harris J1
1 The Permanente Medical Group and the Care Management Institute, USA
Background: Many processes of health care could be improved.
Adherence to evidence-based best practices, and reduction in unwanted
variation in care and prevention should significantly improve population
health and better manage care provided through health care systems.
Objectives: 1. To review the availability and quality of evidence
that could support important process improvement (PI) efforts in
clinical areas such as organ failure in sepsis, ventilator-acquired
pneumonia, central line infections, acute coronary syndrome and stroke,
and prevention of communicable disease, cancer, and cardiovascular
disease. 2. To describe evidence and efforts to implement
evidence-based practices consistently in a large health care delivery
system. Methods: We searched the Cochrane Library, major data
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
180
bases, and guidelines for studies and systematic reviews on healthcare
process improvement, preventive services, and PI implementation for
preventive screening, immunizations, and care for sepsis, coronary
disease, stroke, and pneumonia. We characterized the quality
of the evidence and reviewed process improvement projects at
Kaiser Permanente in these areas. Results: Evidence is generally
observational for clinical process improvement, immunizations and
screening. There was good evidence for the testing and treatment
portions of process improvement protocols, but less so for the entire
integrated process. Nevertheless, positive results were demonstrated
in most areas. Conclusions: Trials are most often focused on specific
treatments rather than processes of care. Processes occur over time,
with a number of sequential or parallel components. The quality
and availability of evidence to support health care processes varies
significantly and can require chains of evidence, which are indirect by
definition. Outcomes can include positive or adverse events, resource
use, and operational efficiency as well as intermediate and final clinical
states. Even when there is good evidence, how to have providers to do
the right thing consistently remains an open question.
P3B343
Cochrane Journal Club: Going from strength to strength?
Curtis T1
1
Wiley-Blackwell, John Wiley & Sons, Chichester, UK
Background: In October 2009, Cochrane and Wiley-Blackwell
launched the Cochrane Journal Club (CJC) http://www.cochranejournal
club.com/ to promote an increase in awareness of research, educate
students to critique and use research findings, and to aid in the
translation of research into practice. By the time of the Colloquium, the
CJC will have been published monthly for almost two years. In order
to continue to meet the needs of the users the CJC needs to reflect on
the number and type of users and whether the initial objectives are still
being met. Objectives: To analyze data for the first 18 months to find
out who is using the Cochrane Journal Club in terms of numbers to
the site, job type and country. To survey a random sample of users to
find out whether CJC met their educational requirements. Methods:
Since the first issue visitor-data, including country and job type, has
been collected. Users are also invited to ‘join the club’ and register. It
is a sample of these users who will be invited to provide information
regarding educational use via an online survey. Results: Eighteen
months of visitor data will be analysed and presented; early indications
are that over 3,200 unique users visit the site each month, with just
under 1000 users returning each month. To date, just over 4,200 have
registered to join the Cochrane Journal Club and to receive email alerts
about new journal club topics. Conclusion: Early indications are that
the Cochrane Journal Club user-numbers have exceeded expectations,
but through the online user survey we need to ensure that the content
supplied meets and continues to meet the educational requirements of
these users.
P3B344
Who is listening? A qualitative study of Cochrane podcast
users
Stewart G1 , Mavergames C2
1 Wiley-Blackwell, Chichester, UK; 2 German Cochrane Centre, Freiburg,
Germany
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Podcasts from the Cochrane Library were first launched
in January 2008. Since then, the number of listeners worldwide has
increased substantially; more than 100,000 accesses per month were
achieved in 2010. Previous quantitative studies have provided evidence
to show that the Cochrane podcasts have grown in popularity but have
not demonstrated why users are choosing to listen. A qualitative study
will enable the podcasts to develop further and continue to serve the
needs of the listeners. Methods: A user survey targeted toward
existing users of the Cochrane podcasts will be conducted. Social
networking outlets including Facebook and Twitter will be used in
order to ask listeners a number of questions relating to their usage of
the Cochrane podcasts. A small number of previous podcast authors
will be asked to share their views on the current format. Results:
The results from the survey will give a more detailed impression of the
users of the Cochrane podcasts. The strengths and weaknesses of the
current podcast format that the users identify will be used to improve
the service for listeners. Conclusions: How the podcasts are produced
and presented in the future will be influenced by the outcome of the
user survey.
P3B345
Cochrane review of school-based interventions for the
prevention of human immunodeficiency virus (HIV) and
sexually transmitted infections (STIs) in adolescents: Why
biological outcomes are important
Mason-Jones A1 , Mathews C2 , Kagee A3 , Lombard C4
1
Specialist Scientist, Health Systems Research Unit, South African Medical
Research Council, Senior Lecturer, Adolescent Health Research Unit,
University of Cape Town; 2 Associate Professor, Department of Family
Medicine and Public Health, University of Cape Town, Chief Specialist
Scientist, Health Systems Research Unit, Medical Research Council;
3 Department of Psychology, Stellenbosch University, South Africa;
4 Department of Biostatistics,
South African Medical Research
Council
Background: Self-report outcome measures of sexual behaviour are
the mainstay of most HIV/STI prevention trials amongst adolescents.
All reviews to date have focussed on sexual risk behaviour as the
primary outcome. This is partly due to the large sample sizes required
to demonstrate a reduction in incidence of HIV or STIs. However,
self-report measures are prone to recall and social desirability bias
(Plummer 2004; Langhaug 2010) and the relationship between such
outcomes and HIV or STIs is not known. Biological measures, such
as incidence of HIV or STIs are the most convincing indicators of the
success of prevention interventions. Objective: To summarise the
effect of school-based HIV prevention interventions on the incidence
and prevalence of HIV and STIs and on sexual risk behaviour among
adolescents. Methods: We conducted a comprehensive search
for randomised controlled trials, published between 1985 and 2010,
evaluating school-based HIV infection prevention interventions in which
the unit of randomisation was individual students, classrooms, schools
or communities, and in which the outcome measures included HIV
and/or other STI incidence or prevalence. Results: Three recent large
randomised controlled trials of HIV prevention interventions among
adolescents and youth in sub-Saharan Africa have included biological
measures of STIs as their primary outcomes in addition to self-reported
sexual risk behaviour (Cowan 2010; Ross 2007; Jewkes 2008). These
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
181
trials demonstrate that it is possible to include biological outcomes
in adolescent HIV prevention trials. Conclusions: Future systematic
reviews and individual intervention studies should include biological
outcome measures. The development and validation of acceptable and
valid biological measures should be prioritized and included, where
feasible, in future studies and trials of interventions to reduce sexual
health risk.
Statistical methods
Day 22 Oct (P3B346–P3B381)
P3B346
Trial sequential analyses of meta-analyses from a Cochrane
review of targeting intensive versus conventional
glycaemic control in patients with type 2 diabetes
Hemmingsen B1 , Lund S2 , Gluud C1 , Vaag A3 , Almdal T2 ,
Hemmingsen C1 , Wetterslev J1
1
Copenhagen Trial Unit, Centre of Clinical Intervention Research,
Rigshospitalet, Copenhagen, Denmark; 2 Steno Diabetes Center, Gentofte,
Denmark; 3 Department of Endocrinology, Rigshospitalet, Copenhagen,
Denmark
Background: Cumulative meta-analysis may increase the risk of type
1 and 2 errors due to sparse data and repetitive testing on accumulating
data. To limit risk of random errors, meta-analyses may be analysed with
trial sequential analysis (TSA) using heterogeneity-adjusted required
information size to determine when convincing evidence is reached.
Methods: Meta-analyses of the effects of targeting intensive versus
conventional glycaemic control on mortality, non-fatal myocardial
infarction, and severe hypoglycaemia in patients with type 2 diabetes
were conducted. TSA was applied to limit the overall type 1 error
to 5% and type 2 error to 20%. We used a heterogeneity-adjusted
required information size calculated for a 10% relative risk reduction
(RRR) of mortality and of non-fatal myocardial infarction and a 30%
RRR of severe hypoglycaemia corresponding to numbers needed to
treat (harm) of 50–100. Results: The relative risk reduction of
all-cause mortality and non-fatal myocardial infarction were 1.02, 95%
CI 0.91 to 1.13; I2 = 30% (Figure 1) and 0.85, 95% CI 0.76 to
0.95; I2 = 0% (Figure 2). TSA showed that a 10% RRR or more
can be rejected for all-cause mortality (Figure 3). TSA showed lack
of firm evidence for a benefit of targeting intensive glycemic control
for non-fatal myocardial infarction. Only 27,958 participants (44%) of
the heterogeneity-adjusted required information size of 63,446 have
been accrued to detect a 10% RRR of non-fatal myocardial infarction
(Figure 4). The RR of serious hypoglycaemia was 2.18, 95% Cl 1.47 to
3.23; I2 = 73% (Figure 5). TSA showed firm evidence for at least a
30% increase of severe hypoglycaemia with intensive glycaemic control
(Figure 6). Conclusion: Intensive glycaemic control does not seem
to reduce all-cause mortality, and if so then less than 10%. A 10%
RRR of non-fatal myocardial infarction could not be confirmed. TSA
confirmed that intensive glycaemic control increases the RR of severe
hypoglycaemia with 30%.
Figure 1 (P3B346).
Figure 2 (P3B346).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
182
Figure 3 (P3B346).
Figure 4 (P3B346).
Figure 5 (P3B346).
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
183
Figure 6 (P3B346).
P3B347
A comparison of subgroup analyses in grant applications
and publications
Boonacker CW1 , Hoes AW1 , van Liere-Visser K2 , Schilder AG3 ,
Rovers MM4
1 Julius Center for Health Sciences and Primary Care, University Medical
Center Utrecht, Netherlands; 2 ZonMw, Netherlands Organisation for Health
Research and Development, The Hague; 3 Department of Otorhinolaryngology, Wilhelmina Children’s Hospital, University Medical Center Utrecht,
Netherlands & Julius Center for Health Sciences and Primary Care, University
Medical Center Utrecht, Netherlands; 4 Julius Center for Health Sciences and
Primary Care, University Medical Center Utrecht, Netherlands & Department
of Otorhinolaryngology, Wilhelmina Children’s Hospital, University Medical
Center Utrecht, Netherlands
Background: One of the most important recommendations of the
available guidelines on studying and reporting subgroup analyses,
is to pre-specify subgroups rather than define them post hoc. We
therefore studied both grant proposals and their publications and
compared the subgroup analyses that were pre-specified in the grant
proposal to those that were finally published. Objectives: To compare
grant applications and final publications regarding subgroup analyses.
Methods: Grants awarded by the ‘Netherlands Organization for
Health Research and Development’ from 2001, were studied. We
analyzed whether or not projects mentioned subgroups in their grant
application and related publications (i.e. the final report and scientific
publications). The main outcome measure was the proportion of studies
in which the publications were completely in agreement with the grant
proposal, i.e., subgroups that were pre-specified in the grant proposal
were reported and no new subgroup analyses were introduced in the
publications. Of all individual subgroups that could be identified in
the included projects, we analyzed if they were pre-specified or a
post-hoc finding. Results: In 20 (25%) of the 79 included projects,
publications were completely in agreement with the grant proposal. Of
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
the 149 pre-specified subgroups, 46 (31%) were reported in the final
report or scientific publications, and 143 of the 189 (76%) reported
subgroups were based on post-hoc findings. For 77% of the subgroup
analyses in the publications, there was no mention whether these
were pre-specified or post-hoc. Justification for subgroup analysis and
methods to study subgroups were rarely reported. Conclusion: There
is a large discrepancy between grant applications and final publications
regarding subgroup analyses. Both non-reporting of pre-specified
subgroup analyses and reporting of post-hoc subgroup analyses are
common.
P3B348
The impact of underpowered studies in meta-analyses
reported by Cochrane Reviews
Turner R1 , Bird S1 , Higgins J1
1 MRC Biostatistics Unit, UK
Background: Most meta-analyses include data from one or more
small studies, which would not themselves have power to detect an
intervention effect. The relative influence of adequately powered and
underpowered studies in published meta-analyses has not previously
been explored. Objectives: To examine the amount of power available
in studies included in meta-analyses reported by Cochrane reviews, and
to investigate the impact of underpowered studies on meta-analysis
results. Methods: Our analyses included 14,886 meta-analyses of
binary outcomes extracted from 1,991 Cochrane reviews. For each
study in each meta-analysis, we calculated the power available to
detect a 30% relative risk reduction, based on mean study arm size
and prevalence of the outcome. Associations between meta-analysis
characteristics and power were examined. In a subset of the data set,
meta-analyses were repeated with underpowered studies excluded.
Results: In 81% of meta-analyses in this data set, all studies
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
184
included were underpowered (power < 50%) to detect a relative risk
reduction of 30%. Only 10% of meta-analyses included at least two
adequately powered studies (power ≥ 50%). The amount of power
typically available varied across medical areas and outcome types. In
meta-analyses including at least two adequately powered studies and
at least one underpowered study, differences between results based
on adequately powered studies alone and results based on all studies
were small on average. The standard error of the intervention effect
increased by a median of 12% (inter-quartile (IQ) range −1% to
35%) when underpowered studies were omitted. The between-study
heterogeneity estimate decreased by a median of 27% (IQ range
100% decrease to 14% increase). Conclusions: In the majority of
meta-analyses reported by Cochrane reviews, underpowered studies
make up the entirety of the evidence. However, for topics in which
at least two adequately powered studies are available, underpowered
studies often contribute little information, which raises questions about
scientific efficiency.
P3B349
Quantifying, displaying and accounting for heterogeneity
in the meta-analysis of RCTs using standard and
generalised Q statistics
Bowden J1 , Tierney J2 , Copas A2 , Burdett S2
1 MRC Biostatistics Unit; 2 MRC Clinical Trials Unit
Background: Clinical researchers have often preferred to use a
fixed effects model for the primary interpretation of a meta-analysis.
Heterogeneity is usually assessed via the well known Q and I2 statistics,
along with the random effects estimate they imply. In recent years,
alternative methods for quantifying heterogeneity have been proposed,
that are based on a ‘generalised’ Q statistic. Methods: We review
18 IPD meta-analyses of RCTs into treatments for cancer, in order
to quantify the amount of heterogeneity present and also to discuss
practical methods for explaining heterogeneity. Results: Differing
results were obtained when the standard Q and I2 statistics were used
to test for the presence of heterogeneity. The two meta-analyses with
the largest amount of heterogeneity were investigated further, and on
inspection the straightforward application of a random effects model
was not deemed appropriate. Compared to the standard Q statistic, the
generalised Q statistic provided a more accurate platform for estimating
the amount of heterogeneity in the 18 meta-analyses. Conclusions:
Explaining heterogeneity via the pre-specification of trial subgroups,
graphical diagnostic tools and sensitivity analyses produced a more
desirable outcome than an automatic application of the random effects
model. Generalised Q statistic methods for quantifying and adjusting
for heterogeneity should be incorporated as standard into statistical
software. Software is provided to help achieve this aim.
P3B350
Individual participant data meta-analysis of prognostic
factor studies: State of the art?
Riley R1 , Abo-Zaid G2 , Sauerbrei W3
1 University of Birmingham, UK; 2 University of Birmingham, UK; 3 University
Medical Center Freiburg, Germany
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: Prognostic factors are associated with the risk of a
subsequent outcome in people with a given disease or health condition.
They have a broad array of uses; for example, they help define disease at
diagnosis; they inform individual treatment strategies; and they inform
the design and analysis of clinical trials. Studies to identify prognostic
factors are abundant in the literature, and meta-analysis is needed to
combine them. Methods: Meta-analysis using individual participant
data (IPD), where the raw data are synthesised from multiple studies,
has been championed as the gold-standard for synthesising prognostic
factor studies. In this talk I will examine the feasibility and conduct
of this approach, using a systematic review of currently published
IPD meta-analyses of prognostic factors studies. Results: Forty-eight
published IPD meta-analyses of prognostic factors were identified; only
three were published before 2000 but thereafter a median of four
articles exist per year. These show that having IPD is advantageous,
allowing modelling assumptions to be checked; variables to be analysed
on their continuous scale; non-linear relationships to be examined; and
results to be adjusted for other variables. However, the IPD approach
also raises many challenges, such as large cost and time required to
obtain and clean IPD; unavailable IPD for some studies; different sets
of prognostic factors in each study; and variability in study methods
of measurement. Their conduct can also be improved; continuous
variables are often categorised without reason and more sophisticated
methods to meta-analyse them are required; and publication bias is
rarely examined. Conclusions: IPD meta-analysis of prognostic factor
studies is a huge improvement over an aggregate data meta-analysis
approach. However, many practical, methodological and clinical
problems still remain. The gold-standard is a prospectively planned
meta-analysis using IPD.
P3B351
The effect of the assessment goal on the interpretation of
subgroup findings and the corresponding conclusions
Bender R1
1 Department of Medical Biometry, Institute for Quality and Efficiency in
Health Care, Germany
Background: The interpretation of subgroup analyses in clinical
research is still challenging. Beside statistical criteria such as preor post-specification, multiplicity, power, and the applied statistical
technique, the reason for conducting subgroup analyses plays a role.
Moreover, the interpretation of subgroup findings may be dependent
on the context in which the corresponding results are assessed.
Objectives: To discuss the effect of the assessment goal on the
interpretation of subgroup analyses and the corresponding conclusions
in the context of drug approval and reimbursement. Methods: By
means of an example it is discussed and demonstrated that the different
perspectives of drug approval and reimbursement can lead to different
conclusions regarding subgroup analyses. Results: Noninferiority to
an established drug is a plausible basis for licensing a safe new drug.
However, reimbursement at higher costs requires an additional clinical
benefit, which means that superior efficacy of the new drug compared
to the established drug has to be demonstrated. In the situation of
similar efficacy of the new and the established drug in one subgroup
and superior efficacy of the new drug in a second subgroup, it would be
adequate to approve the new drug for the whole population but restrict
reimbursement to the second subgroup of patients. Conclusions:
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
185
Different assessment goals of different institutes or authorities may
lead to different conclusions regarding subgroup analyses. The specific
context of decisions has to be considered in the interpretation of the
credibility and importance of subgroup findings.
P3B352
Exploring treatment by covariate interactions in mixed
treatment comparison meta-analysis: Individual patientlevel covariates versus aggregate trial-level covariates
Donegan S1 , Williamson P1 , D’Alessandro U2 , Tudur-Smith C1
1 University of Liverpool, UK; 2 Prince Leopold Institute of Tropical Medicine,
Belgium
Background: Mixed treatment comparison (MTC) meta-analysis
allows several treatment options to be compared simultaneously in a
single analysis while utilising both direct and indirect evidence. Three
key assumptions underlie the methodology: similarity; homogeneity;
and consistency. Assessment of the assumptions is vital to ensure
the results of the MTC meta-analysis are interpreted appropriately. A
previously proposed assessment method is the inclusion of potential
treatment effect modifying covariates in the MTC meta-analysis models.
For conventional pair-wise meta-analysis, important benefits regarding
the investigation of treatment by covariate interactions, gained
from using individual patient data rather than aggregate data for
meta-analysis, have previously been described. Objectives: We
aim to compare individual patient data MTC meta-analysis models that
include patient-level covariates with aggregate data MTC meta-analysis
models that include study-level covariates. Methods: Two types of
random effects MTC models for dichotomous outcomes were applied:
models based on the individual patient data from the trials’ original
datasets and models based on the ‘aggregate’ event rates from each
trial. For each type of MTC model, three different model specifications
were applied that made increasingly stronger assumptions regarding
the treatment by covariate interactions. We compared the two types
of models through application to a real dataset. To ensure that any
differences in the results would be due to the model specifications,
we used the real individual patient dataset to generate the aggregate
data. Results: We found that the treatment effects and drug rankings
from MTC models that include patient-level covariates and those from
models that included study-level covariates differed. The inclusion of
patient-level rather than study-level covariates produced more precise
results. Conclusions: When assessing the similarity assumption of a
MTC model, including patient-level covariates is more favourable than
including study-level data when the covariate distributions vary within
trials.
P3B353
Is it reasonable to pool estimates of intervention effect
estimated from different analytical methods for
continuous outcomes? A meta-epidemiological study
McKenzie JE1 , Deeks JJ2
1
School of Public Health and Preventive Medicine, Monash University,
Australia; 2 Public Health, Epidemiology and Biostatistics, University of
Birmingham, United Kingdom
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
Background: In randomised trials, continuous data are often
measured before and after the intervention. Common analytical
methods include analysing final values (FV), analysing change scores
(CS) and analysis of covariance (ANCOVA). These approaches provide
unbiased estimates of the same true underlying intervention effect.
Theoretically, pooling effects from a mix of these methods should
yield unbiased estimates of pooled intervention effect. However,
the meta-analytic estimates may be biased if the trialists selectively
report results. Objectives: To investigate if the pooled estimate
of intervention effect and its standard error were modified by the
analytical methods employed in the component trials. Methods:
Fifty-four meta-analyses were selected from The Cochrane Database
of Systematic Reviews (2004, Issue 4). Twenty met the inclusion
criteria of at least four comparisons from parallel group trials with
enough data available to extract at least one effect estimate, with
a mix of analytical methods across the comparisons. Pooling effects
using a mix FV and CS were compared with: pooling effects from i)
one analytical method (either all FV or all CS), ii) a mix of FV, CS, and
ANCOVA, iii) all ANCOVA. A ‘meta-meta-analysis’ was undertaken to
investigate if the pooled effect was modified by the use of either FV or
CS analyses. Results: Analyses are suggestive of selective reporting
of analytical method by baseline imbalance. However, the impact of
this on the random effects meta-analytical estimates was less clear.
Inconsistency of effects (I2 ) varied: greater consistency was observed
with approach i) and greater inconsistency with approach iii). Standard
errors, p-values, and between trial heterogeneity were similar between
the approaches. Conclusions: Pooling effects from a mix of analytical
methods is appealing, however it may be problematic. The inclusion
of ANCOVA estimates did not result in expected gains in terms of
increased precision of the meta-analytic estimates.
P3B354
IPD meta-analysis of time-to-event data: One-stage versus
two-stage approaches to estimating the hazard ratio
Bowden J1 , Tierney J2 , Simmonds M3 , Copas A2 , Higgins J1
1 MRC Biostatistics Unit; 2 MRC Clinical Trials Unit; 3 Queen Mary University
of London
Background: Meta-analyses of individual patient data (IPD) provide
a strong and authoritative basis for evidence synthesis. IPD are
particularly useful when the outcome of interest is the time to an
event. Methodological developments now enable the meta-analysis of
time-to-event IPD using a single model, allowing treatment effect and
across-trial heterogeneity parameters to be estimated simultaneously.
This differs from the standard approaches used with aggregate data,
and also predominantly with IPD. Methods: Facilitated by a simulation
study, we investigate what these new ‘one-stage’ random-effects
models offer over standard ‘two-stage’ approaches. Results and
Conclusions: We find that two-stage approaches represent a robust,
reliable and easily implementable way to estimate treatment effects
and account for heterogeneity. Nevertheless, one-stage models can
be used to provide a deeper insight into the data. Software for fitting
one-stage Cox models with random effects using REML methodology
is made available, and its use demonstrated on an IPD meta-analysis
assessing post operative radio therapy for patients with non-small cell
lung cancer.
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
186
P3B355
The use of an ordered multinomial logit model in network
analysis: An example in psoriatic arthritis
Craig D1 , Epstein D2 , Yang H1 , Bojke L2 , Sculpher M2 , Woolacott N1
1 CRD, University of York. UK; 2 CHE, University of York, UK
Background: Bayesian network meta-analyses (NMA) are an
extension of classic meta-analysis, but where a meta-analysis includes
only direct evidence, NMAs draw on both direct and indirect evidence.
The lack of head-to-head trials for newer drugs has seen an increase
in the use of NMAs. When appropriate trials are to be synthesised,
model choice for NMA is extensive. Researchers should not be
limited to commonly reported methods, but should select models
which appropriately capture the relationship of the outcomes being
synthesised. Objectives: To demonstrate the appropriate application
of a hierarchical multinomial ordered logit model for a network
meta-analysis in the context of a recent systematic review and
decision model. Methods: A NMA was used to estimate the
probability that patients have a Psoriasis Area and Severity Index
(PASI) 50/75/90 response or American College of Rheumatology
(ACR) 20/50/70 response. These outcomes represent percentage
reduction, have an order and are clearly related. The models were
implemented in WinBUGS. The hierarchical multinomial ordered logit
model, allowed an estimation of the probability of achieving one level of
response compared with another as a linear function of the explanatory
variables. Two separate analyses were undertaken for the outcomes of
PASI 50/75/90 response and ACR 20/50/70 response. We estimated
the probability that patients have a PASI 50/75/90 or ACR20/50/70
response by means of a cumulative logistic model. All priors were
non-informative. Alternative modelling scenarios/assumptions were
assessed. Results and Conclusions: The models presented provided
flexible methods to ensure that the relationship between the ordered
outcomes was appropriately maintained and synthesised. In the
absence of head-to-head trials, and outcomes that are clearly related,
researchers need to look beyond the standard models in our toolkits
and use methods that deal appropriately with the clinical outcome of
interest.
P3B356
Classification of methods for evaluation of treatment
selection markers: A systematic review
Tajik P1 , Mol BW2 , Bossuyt PM1
1
Department of Clinical Epidemiology, Biostatistics and Bioinformatics,
Academic Medical Center, University of Amsterdam; 2 Department of
Obstetrics and Gynecology, Academic Medical Center, University of
Amsterdam
Background: There is an increasing interest in the evaluation of
biomarkers useful for treatment selection. However, the literature on
the methods for the evaluation of these markers is diverse and there is no
consensus about their classification and nomenclature. Objectives: To
systematically review and generate a classification of study methods to
identify and/or validate treatment selection markers, and to standardize
the nomenclature. Methods: We searched MEDLINE (1950 to Jan
2011), EMBASE (1980 to Jan 2011), Cochrane Methodology Register
and MathSciNet (up to the Jan 2011), and handsearched the citations
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
of included studies. We included methodology reports that described
one or more study design or analysis plans related to identification
and validation of treatment selection markers. Results: Our search
identified 1758 abstracts, among which 82 were found to be probably
eligible. After full-text review, 36 studies were included. The following
dimensions were identified for the classification of methods: trial
phase, patient flow, randomization type, sample size requirements,
timing, the number of evaluated markers, analysis plan, and summary
statistics. For each study design advantages and limitations were
described and the efficiency of methods were compared with each
other. Using identified dimensions, all methods were classified and
identical methods received a unique terminology. Conclusions:
We propose a standard nomenclature and classification system for
study designs of treatment selection marker evaluation. Using this
classification system could help researchers in designing future studies
and reduce reporting inconsistencies.
P3B357
Adverse events of antiepileptic drugs, across indications:
Can randomised controlled trial data from non-epilepsy
indications be included in meta-analysis for AEDs used in
epilepsy?
Tudur-Smith C1 , Marson A1
1
University of Liverpool
Background: Adverse Events (AEs) are reported in randomised
controlled trials (RCTs) of epilepsy and non-epilepsy trials using the
same antiepileptic drug (AED). Meta-analysis of these trials would
increase study power. Objectives: To determine if AE outcomes from
RCTs of AEDs across non-epilepsy indications (neuropathy or migraine)
can be meta-analysed with data from epilepsy trials. Methods: We
searched databases for RCTs meeting inclusion criteria. AEDs included
were topiramate, gabapentin, valproate, oxcarbazepine, lacosamide,
lamotrigine, carisbamate, zonisamide and pregabalin. Extracted
data was analysed using RevMan V5.0. Common AEs analysed
were; dizziness, ataxia, headache, fatigue, nausea, somnolence,
withdrawals due to AE and any AE. Summary statistics of effect
size were calculated using the Mantel-Haenszel method. Statistical
heterogeneity was assessed using a random effects model. To test
between indications we used a fixed effects model and calculated
an I2 statistic. Results: We included 106 RCTs for analysis.
Effect sizes varied with drug and outcome. When dizziness was
analysed, test between indications showed no statistical heterogeneity
(I2 = 0%) for gabapentin, topiramate, lacosamide and lamotrigine.
However, heterogeneity was significant (I2 = 59%) for trials of
oxcarbazepine. When somnolence was the AE outcome, heterogeneity
was insignificant for oxcarbazepine (I2 = 8%), lacosamide (I2 = 0%)
and topiramate (I2 = 0%), but significant for gabapentin (I2 = 56%)
and lamotrigine (I2 = 60%). When nausea was analysed there
was no heterogeneity (I2 = 0%) for lacosamide, oxcarbazepine
and topiramate, but significant heterogeneity existed for gabapentin
(I2 = 41%) and lamotrigine (I2 = 40%). In instances where there
was significant heterogeneity, the size of relative risk was greater
in the non-epilepsy indications. Conclusions: AEs of AEDs from
non-epilepsy trials could be used in meta-analysis given the absence of
statistical heterogeneity for some interventions and outcomes. Effect
sizes were larger in the non-epilepsy indications overall. Further
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
187
meta-regression would unmask any dose effect on heterogeneity and
effect size.
P3B358
Exploring the effects of different treatment doses by
‘blurring’ the nodes of a treatment network: Application
to a Cochrane overview for multiple sclerosis
Del Giovane C1 , Mavridis D2 , Vacchi L3 , Filippini G3 , Salanti G2
1 University of Modena – Italian Cochrane Center and Department of
Hygiene and Epidemiology, University of Ioannina School of Medicine,
Ioannina Greece; 2 Department of Hygiene and Epidemiology, University
of Ioannina School of Medicine, Ioannina Greece; 3 Neuroepidemiology
Unit, Fondazione I.R.C.C.S. Istituto Neurologico Carlo Besta, Milano,
Italy
Background: Several immunotherapies exist for the management
of multiple sclerosis and their relative efficacy can be evaluated via
Multiple-Treatments Meta-analysis (MTM). Some agents are
administered at different doses, but the impact of increasing the
dose on delaying the disease progression is still unclear. Therefore it is
uncertain whether different doses of the same agent can be considered
to form a common ‘node’ in the treatment network. Objectives:
a) to explore a series of MTM models where the uncertainty about
the impact of dose is embedded in the definition of each node in the
treatment network b) to evaluate the assumptions that dose effects are
the same, randomly different but exchangeable or follow a monotonic
pattern. Methods: We developed Bayesian models considering the
different node definitions. We started with the network where each
agent, whatever the dose, defines a node. We gradually ‘blurred’
the nodes so that different dose effects of the same agent are:
random but exchangeable; isotonic or have a linear pattern around
the agent’s mean effect. We ended up with the network where
each different dose defines an independent node. Each model has
different implications for the assumption of consistency of effect (either
at agent level or at dose level). The models were compared using
goodness of fit criteria; changes in heterogeneity and inconsistency
were monitored. Results: Models for linear and independent dose
effects had poor fit whereas exchangeable dose effects models had
the best fit and lowest heterogeneity. This suggests that the dose
effects of the same agent approximate the mean agent effect and
hence a dose-response association is unlikely. Conclusions: The
‘node-blurring’ approach can be a useful exploratory tool when there
is doubt whether similar interventions should be grouped under the
same node in a network; a dilemma which is frequently encountered
in MTM.
P3B359
Meta-analysis of diagnostic studies: SROC modelling – the
profile log-likelihood approach based upon the
proportional hazard model
Charoensawat S1 , Böhning D2
1
Udon Thani Rajabhat University, Thailand; 2 University of Reading, United
Kingdom
Background: The number of meta-analysis of diagnostic studies is
increasing and models that deal with the summary receiver operating
Copyright c2011 The Cochrane Collaboration.
Published by John Wiley & Sons, Ltd.
characteristics (SROC) approach have become popular. These models
have become complex and difficult to use which led us to develop
a simple model, the proportional hazard model (PHM). Objective:
To improve SROC modelling based upon PHM by finding the most
efficient estimate of the diagnostic accuracy parameter θ. Methods:
A simulation study was done to investigate the performance of
the profile log-likelihood (PMLE), the adjusted profile log-likelihood
(APMLE) and the iterative weighted least square (IWLS) approaches.
The performances of approaches were found from score function
and the Fisher information I(θ). The heterogeneity was considered
and the APMLE with the random effect was developed. Maximum
likelihood estimation was used to find the estimate of θ and tau2 ,
two parameters in the model. Also, the variance-covariance matrix
was used to determine the inverse of I(θ) and I(tau2 ) and to find the
estimate of θ and tau2 of I(θ). Meta-analysis of diagnostic studies of
exercise echocardiography (ECHO) and exercise single photon emission
compute tomography (SPECT) imaging in the diagnosis of coronary
artery disease was exemplified under homogeneity and heterogeneity.
Results and Conclusion: PMLE turned out to be an easy approach
to eliminate nuisance parameter from the model but a normalizing
constant was missing. APMLE has been suggested for correcting the
shortcoming of the PMLE. In the simulation study, likelihoods behaved
similarly, but the APMLE was the most efficient estimator. In our
example of meta-analysis of diagnostic studies, θ of ECHO was smaller
which meant it had higher diagnostic accuracy than the SPECT device,
since the smaller the θ the higher the diagnostic power. p − values
became non-significant when tau2 was incorporated.
P3B360
How to improve the credibility of meta-analysis? How
to avoid mistakes in choosing the statistical method of
estimation? The Bayesian and traditional approaches
Nikodem M1 , Walczak J1 , Borowiack E2 , Siedmiogrodzki K1 ,
Zapalska A2 , Khan K3 , Meads C4 , Mol B5 , Oude Rengerink K5 ,
Thangaratinam S3 , Zamora J6
1 CASPolska Association, Poland; 2 Arcana Institute, Poland; 3 Queen
Mary, University of London, Great Britain; 4 Queen Mary, University of
London; 5 Academic Medical Center Amsterdam, Netherlands; 6 Hospital
Universitario Ramon y Cajal, Madrid, Spain
Background: There are at least ten different statistical methods of
meta-analysis. It can be surprising how different results of estimation
can be obtained just by applying different method. Therefore, if we
want to run the meta-analysis properly then we have to solve the
crucial problem of choosing the most credible statistical method. This
is especially important in the case of small sample size, low event rate
or large discrepancy between event rates in groups of comparison.
Objectives: To find out which methods are the most credible and
precise for particular data type. Moreover to underline which methods
should not be used to avoid substantial errors in estimation. Our main
aim is to answer the question of how to deal with following problems:
a. insufficient sample size, b. low event rate. Methods: Initially
a systematic review on existing statistical methods was conducted.
It was followed by an expanded analysis of credibility and precision
of statistical methods. Ten statistical methods (including alternative
methods such as Bayesian ones combining results of nonrandomized
with randomized trials) were examined on different data sets. Overall
Cochrane Database Syst Rev Suppl 3–233 (2011)
DOI:10.1002/14651858.CD000003
188
Figure 1 (P3B360).
not disappear completely, even in very large datasets. Conclusions:
The authors conclude that the proposed method to incorporate
previously published univariable predictor-outcome associations as
evidence in new multivariable prediction analyses is superior to
established approaches and is especially worthwhile when relatively
limited individual patient data are available.
P3B362
Aggregating published prediction models with individual
patient data: A comparison of different approaches
124 sub-cases (different sample size, event rate, heterogeneity, etc.)
were considered. Bias, confidence interval coverage and statistical
power were the measures of credibility and precision. Results:
Possible datasets were categorized and for particular categories the
optimal methods are highlighted and warnings of possible mistakes
are underlined. All results are presented in the form of guidelines.
Conclusions: There is no universal method which is credible for every
set of data, therefore the choice of method should be done carefully and
depending on the data characteristics. For several cases, the Bayesian
approach (with or without inclusion of non-randomized trials) was
highly ranked, hence it should be taken under greater consideration.
P3B361
Meta-analytical approaches for diagnostic and prognostic
prediction research: Incorporating published univariable
associations in multivariable analysis of individual patient
data
Debray T1 , Koffijberg H1 , Lu D2 , Vergouwe Y3 , Steyerberg EW2 , Moons
KGM1
1
Julius Center for Health Sciences and Primary Care, University Medical
Center Utrecht, Netherlands; 2 Department of Public Health, Erasmus
MC, Netherlands; 3 Julius Center for Health Sciences and Primary Care,
University Medical Center Utrecht, Netherlands – Department of Public
Health, Erasmus MC, Netherlands
Background: Diagnostic and prognostic literature is overwhelmed
with studies reporting univariable associations. Currently, methods
to incorporate such information in multivariable analysis of individual
patient data (IPD) are