Download A Critical Analysis of the Scientific Evidence Behind International

A Critical Analysis of the Scientific Evidence Behind
International Guidelines Related to Cardiac Arrhythmias
Markus Roos, MSc, MD; Jeannette Brodbeck, PhD; Andrea Sarkozy, MD;
Gian Battista Chierchia, MD; Carlo De Asmundis, MD; Pedro Brugada, MD, PhD
Downloaded from http://circep.ahajournals.org/ by guest on May 9, 2017
Background—Guidelines have become very important in assisting with decision making in clinical practice. However, few
studies have analyzed the level of evidence (LOE) underlying guidelines critically. This study aims to assess the
accuracy of the referenced literature that has led to recommendations with a level of evidence A (LOE-A) rating.
Methods and Results—The latest updates of the practice guidelines related to arrhythmia posted on the European Society
of Cardiology (ESC) web site were analyzed. The referenced literature for LOE-A recommendation was analyzed to
reassess the proposed grading scheme for LOE-A. Furthermore, the clearly defined positive (Class I) and negative (Class
III) recommendations with correctly referenced LOE-A were assessed. A median of 5.4% of all recommendations per
guideline (interquartile range 4.9% to 9.7%) were categorized as LOE-A, but only 3.7% (IQR 3.4% to 4.9%) were
accurately referenced as LOE-A. In total, 27 of 698 recommendations (median 1.2% per guideline [IQR 0.95% to 3.7%])
were correctly referenced as Class I or III LOE-A recommendations implying definite evidence-based positive or
negative conclusion.
Conclusions—Our findings raise the question of the accuracy of LOE-A in medical guidelines in general and highlight the
importance of a critical use of all recommendations. Moreover, they underline the need for improving the
guideline-writing process. Further randomized double-blinded and/or crossover-designed studies should focus on areas
with a gap in the evidence, such as existing but not yet convincing (LOE-B) or conflicting (Class II) evidence. (Circ
Arrhythm Electrophysiol. 2011;4:202-210.)
Key Words: arrhythmia 䡲 guideline 䡲 clinical practice 䡲 level of evidence 䡲 evidence based
G
uidelines such as the American College of Cardiology/
American Heart Association/European Society of Cardiology (ACC/AHA/ESC) guidelines on cardiovascular disease
provide criteria and recommendations for decision making with
regard to diagnosis, management, and treatment in specific areas
of healthcare. Despite their self-declared limitations, they are
often perceived as evidence based and valid and have become
the final arbiters of care for many clinicians. However, findings
published in 1999 showed that guidelines did not adhere well to
established methodological standards in terms of their identification and evaluation of the scientific evidence.1 In recent years,
there has been an improvement in methodological standards
including a more sophisticated grading system and a more
transparent link between the evidence and the grade of the
recommendations. However, the question still remains about the
methodological rigor in the adherence to the grading system
when linking the evidence to recommendations.2
used. Definitions of the classes of recommendation are as
follows:
Class I: conditions for which there is evidence and/or a
general agreement that a given procedure/therapy is beneficial, useful and effective.
Class II: conditions for which there is conflicting evidence
and/or a divergence of opinion about the usefulness/
efficacy of performing the procedure/therapy.
Class IIa: weight of evidence/opinion is in favor of
usefulness/efficacy.
Class IIb: usefulness/efficacy is less well established by
evidence/opinion.
Class III: conditions for which there is evidence and/or a
general agreement that a procedure/therapy is not useful or
effective and in some cases may be harmful.
A recommendation may therefore be based on scientific
evidence OR on opinion/consensus of the writing committee.
In contrast, a level of evidence (LOE) to support a given
recommendation is strictly based on available scientific
literature. The weight of evidence is ranked from highest (A)
to lowest (C) as follows:
Editorial see p 119
Clinical Perspective on p 210
Currently a grading scheme based on class of recommendation and level of evidence for a given recommendation is
Received June 16, 2010; accepted February 10, 2011.
From the Heart Rhythm Management Center (M.R., A.S., B.C., C.A., P.B.), Vrije Universiteit Brussel, Belgium; Medical Research Council, Cognition
and Brain Sciences Unit (J.B.), Cambridge, United Kingdom.
Correspondence to Markus Roos, Heart Rhythm Management Center, Free University of Brussels (UZ Brussel) VUB, Laarbeeklaan 101, 1090
Brussels. [email protected]
© 2011 American Heart Association, Inc.
Circ Arrhythm Electrophysiol is available at http://circep.ahajournals.org
202
DOI: 10.1161/CIRCEP.110.958181
Roos et al
Scientific Value of Arrhythmia Guidelines
203
No. of References assessed for
f LEO-A
for
LEO A iin allll 5 Guidelines:
G id li
457
Class I: 13 References
Supraventriuclar arrhythmias: 21
Class IIa: 2 References
Class IIb: 6 References
Class I: 9 References
S
Syncope:
14
Class III: 5 References
Class I: 109 References
Ventricualr arrhythmias & SCD: 116
Class IIa: 5 References
Class III: 2 References
Class I: 189 References
Atrial fibrillaon:282
Class IIa: 58 References
Class III: 35 References
Class I: 4 References
Cardiac pacing: 24
Class IIa: 14 References
Downloaded from http://circep.ahajournals.org/ by guest on May 9, 2017
Class IIb: 6 References
Figure 1. Number of references assessed per guideline for all recommendations with a LOE-A. LOE-A indicates level of evidence A:
Data were derived from multiple randomized clinical trials or meta-analyses; SCD, sudden cardiac death; ACC/AHA/ESC, American
College of Cardiology/American Heart Association/European Society of Cardiology; Class I, conditions for which there is evidence
and/or a general agreement that a given procedure/therapy is beneficial, useful and effective; Class II, conditions for which there is
conflicting evidence and/or a divergence of opinion about the usefulness/efficacy of performing the procedure/therapy; Class IIa, weight
of evidence/opinion is in favor of usefulness/efficacy; Class IIb, usefulness/efficacy is less well established by evidence/opinion; Class
III, conditions for which there is evidence and/or a general agreement that a procedure/therapy is not useful or effective and, in some
cases, may be harmful; supraventricular arrhythmias, ACC/AHA/ESC guidelines for the management of patients with supraventricular
arrhythmias7; syncope, guidelines for the diagnosis and management of syncope (version 2009)8; ventricular arrhythmias and SCD,
ACC/AHA/ESC 2006 guidelines for management of patients with ventricular arrhythmias and the prevention of SCD9; atrial fibrillation,
ACC/AHA/ESC 2006 guidelines for the management of patients with atrial fibrillation: full text10; and cardiac pacing, guidelines for cardiac pacing and cardiac resynchronization therapy.11
Level of evidence A: data derived from multiple randomized
clinical trials or meta-analyses.
Level of evidence B: data derived from a single randomized
trial or nonrandomized studies.
Level of evidence C: only consensus opinion of experts, case
studies, or standard of care.
Thus, only a recommendation Class I LOE-A and a Class III
LOE-A, respectively, are based on highest scientific evidence
and suggest clearly positive or negative conclusions. However, shortcomings of recommendation Class I LOE-A and a
Class III LOE-A are that they are based on patients who are
referred and recruited to participate in randomized studies
and may not represent the population to whom the results are
applied.
All recommendations of Class IIa or Class IIb, independent
of their level of evidence, reflect divergent evidence or
opinions. Recommendations Class I or III, based on an
intermediate level of evidence (LOE-B) or expert opinion
(LOE-C), are less likely to be reliable and reproducible. They
are more prone to be biased by financial or intellectual
interests than recommendations on the highest level of
evidence.2–5
A recent study evaluated the distribution of recommendations and level of evidence of 16 ACC/AHA clinical practice
guidelines.6 Results showed that recommendations are largely
developed from lower levels of evidence or expert opinion
(LOE-C, 48%) and only 314 of 2711 recommendations (11%)
are classified as LOE-A. Furthermore, 41% of all recommendations were of no conclusive evidence (Class II). The
percentage of Class I LOE-A and Class III LOE-A recommendations with a definite scientific evidence-based statement was only 9.5% and 0.6%, respectively. However, the
authors reported only the distribution across classes of recommendations and LOE without any judgment for the accuracy of the referenced literature underlying the LOE.
Our study extends the existing literature by performing a
systematic review of the currently available ESC clinical
practice guidelines related to arrhythmia with intent to evaluate (1) the accuracy of the referenced literature underlying
the recommendations of LOE-A and (2) to summarize the
clear definite positive (Class I LOE-A) and negative (Class
III LOE-A) statements as an estimation of the scientific
evidence on which these 5 guidelines are based. We postulate
that by strictly following the grading scheme, the proportion
of the correctly referenced LOE-A recommendations is lower
than that categorized in the guidelines.
Materials and Methods
The following 5 full-text practice guidelines related to arrhythmia
posted on the ESC web site on February 07, 2010 (http://www.
escardio.org/guidelines-surveys/esc-guidelines/Pages/GuidelinesList.
aspx) were analyzed:
ACC/AHA/ESC guidelines for the management of patients with
supraventricular arrhythmias.7
ESC/Heart Rhythm Society guidelines for the diagnosis and management of syncope (version 2009).8
ACC/AHA/ESC 2006 guidelines for management of patients with
ventricular arrhythmias and the prevention of sudden cardiac
death.9
204
Circ Arrhythm Electrophysiol
Table 1.
April 2011
Baseline Characteristics for the 5 ESC Guidelines Related to Arrhythmia
Guidelines
Supraventricular
arrhythmias
Year of
Publication
No. of
References in
the Full Text
References
10 Years
or Older*
No. (% Total)
References
5 Years
or Older*
No. (% Total)
References
Less than 5
Years Old*
No. (% Total)
No. of
Committee
Members
No. of
Reviewers
Mean No. of
Conflict per
Committee
Member
Maximal No.
of Individual
Conflict
2003
537
212 (39)
392 (73)
145 (27)
14
45
n/a
n/a
Syncope
2009
213
51 (24)
121 (57)
92 (43)
29
31
1
7
Ventricular
arrhythmias
and SCD
2006
1087
455 (42)
759 (70)
328 (30)
15
29
7
19
Atrial fibrillation
2006
910
325 (36)
587 (65)
323 (35)
14
40
5
11
Cardiac pacing
2007
384
144 (37)
228 (59)
156 (41)
12
16
n/a
n/a
626
237 (38)
417 (67)
209 (33)
17
32
4
12
Mean (% of total)
Downloaded from http://circep.ahajournals.org/ by guest on May 9, 2017
Supraventricular arrhythmias indicates ACC/AHA/ESC guidelines for the management of patients with supraventricular arrhythmias7; syncope, guidelines for the
diagnosis and management of syncope (version 2009)8; ventricular arrhythmias and SCD, ACC/AHA/ESC 2006 guidelines for management of patients with ventricular
arrhythmias and the prevention of sudden cardiac death9; atrial fibrillation, ACC/AHA/ESC 2006 guidelines for the management of patients with atrial fibrillation: full
text10; cardiac pacing, guidelines for cardiac pacing and cardiac resynchronization therapy.11 n/a indicates not applicable; ACC/AHA/ESC, American College of
Cardiology/American Heart Association/European Society of Cardiology; SCD, sudden cardiac death.
*At the time of publication of the specific guideline.
ACC/AHA/ESC 2006 guidelines for the management of patients
with atrial fibrillation: full text.10
ESC guidelines for cardiac pacing and cardiac resynchronization
therapy.11
First, to obtain the number of recommendations and the distribution of LOE within each class of recommendation, all guideline texts
were manually analyzed to assess their general characteristics.
Second, 457 relevant references for all recommendations with a
LOE-A were investigated to reassess the accuracy of the referenced
literature underlying each LOE-A recommendation (see Figure 1 for
an overview). The type of study, the methods, and the results for
each relevant reference were analyzed. According to the requirements for a level of evidence A, only (a) results of randomized trials
(open-labeled or blinded, as well as post hoc analyses of primarily
randomized conditions of those trials) directly addressing a given
recommendation and (b) meta-analyses of randomized trials were
accepted as adequate. Consequently, references of nonrandomized
studies, animal studies, reviews, editorials, guidelines, as well as
results from registries/surveys/databases were excluded. Third, the
actual level of evidence behind a given guideline was derived by
counting the Class I and III recommendations with correctly referenced LOE-A. Because 4 of 5 of the guidelines reviewed are joint
guidelines between the ESC and ACC/AHA or Heart Rhythm
Society, the results and conclusions of our study may have an impact
on all 4 cardiology societies.
Statistics
Descriptive data were reported as frequencies, means, and standard
deviations. Because individual guidelines vary widely in their numbers
of recommendations, the median of the percentage reported for each
guideline was shown. Two-sided t tests for independent samples were
used for continuous variables. All statistics were computed with SPSS
software (SPSS Inc, Chicago, IL). All probability values are 2-sided,
with values of ⬍0.05 considered significant.
Results
General Characteristics of the Analyzed
Guidelines Related to Arrhythmia
Three of 5 clinical practice guidelines related to arrhythmia
posted on the ESC web site were developed in conjunction
with the ACC/AHA Task force7,9,10 or Heart Rhythm Society.8 One guideline was released only by the ESC.11 The
general characteristics of all 5 guidelines are summarized in
Table 1. Each guideline contained a mean of 626 references,
of which 38% were older than 10 years at the time of
publication of the specific guideline, whereas 33% were less
than 5 years old. On average, 17 committee members and 32
reviewers were involved in establishing a guideline. Conflicts
of interest were listed in only 3 of the 5 guidelines8 –10; the
mean number of conflicts per committee member was 4.
Distribution Across Classes of Recommendation
and LOE
The distribution of all 698 recommendations with regard to
classes of recommendations and LOE is shown as an overview in Figure 2 and subdivided by guidelines in Table 2. A
median of 87.6% were positive Class I and II recommendations (interquartile range [IQR] 87.1% to 93.2%) compared
with 12.4% negative Class III recommendations (IQR, 6.8%
to 12.9%, P⫽0.01). Almost all recommendations were classified as having low or intermediate evidence (LOE-C and
LOE-B, median 94.6%, IQR 90.3% to 95.1%). Only 5.4% of
the recommendations were categorized as having highest
evidence (LOE-A, IQR, 4.9% to 9.7%, P⫽0.002). The
distribution per guideline for the classifications I to III of
recommendations categorized as LOE-A is depicted in Table 3.
Accuracy of the Referenced Literature Underlying
the LOE-A Recommendations
The number of relevant references assessed for all recommendations of LOE-A is shown as an overview in Figure 1.
The reassessment of the accuracy of the referenced literature
underlying every individual recommendation of LOE-A is
shown in detail in Table 4. The table reveals that 67.8% (40
of 59) of LOE-A recommendations were correctly referenced
according to the proposed grading scheme for LOE-A and
were indeed “derived from multiple randomized clinical trials
or meta-analyses” (Tables 3 and 4).
Roos et al
No. of Recommendaons
of all 5 Guidelines: 698
LOE-A: 39
Class I: 300
LOE-B: 114
LOE-C: 147
LOE-A: 10
Class IIa: 191
LOE-B: 69
LOE-C: 112
LOE-A: 4
Class IIb: 129
LOE-B: 33
LOE-C: 92
LOE-A: 6
Class III: 78
LOE-B: 21
Downloaded from http://circep.ahajournals.org/ by guest on May 9, 2017
LOE-C: 51
Figure 2. Distribution of recommendations and level of evidence. Class I indicates conditions for which there is evidence
and/or a general agreement that a given procedure/therapy is
beneficial, useful, and effective; Class II, conditions for which
there is conflicting evidence and/or a divergence of opinion
about the usefulness/efficacy of performing the procedure/therapy; Class IIa, weight of evidence/opinion is in favor of usefulness/efficacy; Class IIb, usefulness/efficacy is less well established by evidence/opinion; Class III, conditions for which there
is evidence and/or a general agreement that a procedure/therapy is not useful or effective and in some cases may be harmful;
LOE-A, level of evidence A: data derived from multiple randomized clinical trials or meta-analyses; LOE-B, level of evidence B:
data derived from a single randomized trial or nonrandomized
studies; LOE-C, level of evidence C: only consensus opinion of
experts, case studies, or standard of care.
Recommendations With a Definite Conclusion
A median of only 1.2% (IQR 0.95% to 3.7%, n⫽27) clearly
defined positive (Class I) and negative (Class III) recommendations with correctly classified highest level of evidence
remained after excluding Class IIa and IIb recommendations
with conflicting evidence or divergence of opinion (Tables 3
and 4). Only these 27 of 698 recommendations showed
definite evidence-based conclusions. They covered a maximum of 3 similar areas per guideline, and totaled 8 positive
and 2 negative directions/instructions:
The 10 Evidence-Based Recommendations for
Arrhythmia Treatment
1. Reduce morbidity and mortality with cardiac resynchronization therapy through the use of a biventricular
pacemaker in symptomatic heart failure patients with left
ventricular ejection fraction ⱕ35% and QRS ⱖ120 ms.
2. Reduce total mortality with implantable cardioverter
defibrillator therapy for primary and secondary prevention in patients with left ventricular dysfunction because
of prior myocardial infarction with left ventricular ejection fraction 30% to 40% or symptomatic heart failure.
2a. Reduce total mortality with implantable cardioverter
defibrillator therapy for secondary prevention in
patients with nonischemic dilated cardiomyopathy
and significant left ventricular dysfunction.
Scientific Value of Arrhythmia Guidelines
205
3. Avoid Class Ic antiarrhythmic drugs in patients with a
history of myocardial infarction.
4. Treat elderly patients experiencing ventricular arrhythmias in the same manner as younger individuals.
5. Use antithrombotic therapy (aspirin/vitamin K antagonists) to prevent thromboembolism in all patients with
atrial fibrillation according to their risk stratification for
stroke and bleeding.
6. Administer dofetilide, flecainide, ibutilide, or propafenone,
but avoid digoxin and sotalol for pharmacological cardioversion of atrial fibrillation.
7. Avoid antiarrhythmic agents to maintain sinus rhythm in
patients with atrial fibrillation who are at risk for developing an arrhythmia with this drug.
8. Treat patients with oral ␤-blockers to prevent postoperative atrial fibrillation.
9. Avoid ␤-blocker therapy as a treatment for the prevention of syncope.
10. Control the rate of stable atrial flutter with a nondihydropyridine calcium channel blocker.
Incorrectly Categorized LOE-A Recommendations
Nineteen of 59 recommendations (18 Class I) were incorrectly categorized as LOE-A. We divided these into 3 major
categories:
A. Ten recommendations were not based on any randomized
study because of possible difficulties that might have been
encountered if they had been randomized in clinical trials:
four of them were obvious statements; for example,
“Acute management of cardiac arrest: Cardiopulmonary
resuscitation (CPR) should be implemented immediately
after contacting a response team.”9 Four recommendations would have resulted in ethical issues if they had been
randomized into trials; for example, “Withdrawal of any
offending drugs and correction of electrolyte abnormalities are recommended in patients presenting with torsades
de pointes.”9 Two of the recommendations were diagnostic procedures, which are more difficult to randomize in
clinical trials compared with studies that investigate a
therapeutic outcome, eg, “Resting 12-lead ECG is indicated in all patients who are evaluated for ventricular
arrhythmias.”9
B. Seven of 19 recommendations could have been randomized, but none or only one study was referenced,
eg, “INR should be determined at least weekly during
initiation of therapy and monthly when anticoagulation
is stable.”10 Three of these 7 recommendations are no
longer of interest for research, eg, “Atrial or transesophageal pacing is recommended for conversion of
stable atrial flutter.”7
C. Finally, 2 recommendations, although being accurately supported by randomized studies, were based
on other guidelines as tertiary source, instead of the
original research articles as required for LOE-A, eg,
“In patients with ischemic cardiomyopathy with
severely depressed left ventricular ejection fraction
or HF, implantable cardioverter defibrillator therapy
is indicated according to current guidelines for implantable cardioverter defibrillator-cardiac resynchronization therapy implantation.”8
206
Circ Arrhythm Electrophysiol
Table 2.
April 2011
Number and Distribution Across Classes of Recommendation and Level of Evidence
Positive
Positive
Negative
Low (LOE-C)
Intermediate
High (LOE-A)
Recommendations Recommendations Recommendations
Evidenced
(LOE-B) Evidenced
Evidenced
Total No. of
(Class I)
(Class IIa and IIb)
(Class III)
Recommendations Recommendations Recommendations
Recommendations No. (% of Total)
No. (% of Total)
No. (% of Total)
No. (% of Total)
No. (% of Total)
No. (% of Total)
Guidelines
Supraventricular
arrhythmias
147
61 (41.5)
76 (51.7)
10 (6.8)
84 (57.2)
55 (37.4)
8 (5.4)
Syncope
105
53 (50.4)
39 (37.2)
13 (12.4)
55 (52.4)
47 (44.8)
3 (2.8)
Ventricular
arrhythmias
and SCD
217
103 (47.4)
100 (46.1)
14 (6.5)
127 (58.5)
69 (31.8)
21 (9.7)
Atrial fibrillation
147
57 (38.8)
71 (48.3)
19 (12.9)
76 (51.7)
48 (32.7)
23 (15.6)
Cardiac pacing
82
26 (31.7)
34 (41.5)
22 (26.8)
60 (73.2)
18 (21.9)
4 (4.9)
Total
698
Summary of
guidelines median
(IQR), %
300
320
78
402
237
59
41.5 (38.8–47.4)
46.1 (41.5–48.3)
12.4 (6.8–12.9)
57.2 (52.4–58.5)
32.7 (31.8–37.4)
5.4 (4.9–9.7)
Downloaded from http://circep.ahajournals.org/ by guest on May 9, 2017
Class I indicates conditions for which there is evidence and/or a general agreement that a given procedure/therapy is beneficial, useful, and effective; Class II,
conditions for which there is conflicting evidence and/or a divergence of opinion about the usefulness/efficacy of performing the procedure/therapy; Class IIa, weight
of evidence/opinion is in favor of usefulness/efficacy; Class IIb, usefulness/efficacy is less well established by evidence/opinion; Class III, conditions for which there
is evidence and/or a general agreement that a procedure/therapy is not useful or effective and in some cases may be harmful; LOE-A, level of evidence A: data derived
from multiple randomized clinical trials or meta-analyses; LOE-B, level of evidence B: data derived from a single randomized trial or nonrandomized studies; LOE-C,
level of evidence C: only consensus opinion of experts, case studies, or standard-of-care. Additional abbreviations are as in Table 1.
Despite the importance of these 19 recommendations in
clinical practice, there is still no justification for their being
ranked as LOE-A.
rized as having highest level of evidence (LOE-A). Among
those, one third were not at all or not adequately supported by
randomized clinical trials or meta-analyses. This highlights a
lack of accuracy in the rating of the scientific evidence
underlying even those recommendations that are supposed to
have the strongest empirical evidence. Our study raises the
question about the accuracy of categorized LOE-A in guidelines for other health topics. It supports suggestions for the
reform of the guideline-writing process because recommendations with incorrectly attributed LOE-A may reflect a lack
of critical reviews of the guideline document within the
guideline committee, as well as the fact that guidelines do not
undergo the usual procedures of independent reviews in the
publication process.5 Some of the recommendations incor-
Discussion
The present study analyzed not only the classes of recommendations and level of evidence in 5 currently available
international guidelines related to cardiac arrhythmias, but
also reassessed the referenced literature underlying these
recommendations and extracted the accurately referenced
recommendations with a definite conclusion.
As previously found most recommendations were positive
(Class I and II) and of intermediate (LOE-B) or low (LOE-C)
evidence.6 Only some of the recommendations were categoTable 3.
Number and Distribution Across Classes of Recommendation of Level of Evidence A
Guidelines
Class I LOE-A Class IIa LOE-A Class IIb LOE-A Class III LOE-A
No. of all
No./Total Class No./Total Class No./Total Class No./Total Class
Recommendations
I (%)
IIa (%)
IIb (%)
III (%)
Stated LOE-A
No./Total (%)
Correctly
Correctly
Referenced
Referenced as
Class I or III
LOE-A No./Total LOE-A No./Total
(%) (Table 4)
(%) (Table 4)
Supraventricular
arrhythmias
147
4/61 (6.6)
1/43 (2.6)
3/33 (9.1)
0/10 (0.0)
8/147 (5.4)
5/147 (3.4)
1/147 (0.68)
Syncope
105
2/53 (3.8)
0/20 (0.0)
0/19 (0.0)
1/13 (7.7)
3/105 (2.9)
1/105 (0.95)
1/105 (0.95)
Ventricular
arrhythmias
and SCD
217
19/103 (18.4)
1/65 (1.5)
0/35 (0.0)
1/14 (7.1)
21/217 (9.7)
8/217 (3.7)
8/217 (3.7)
Atrial fibrillation
147
13/57 (22.8)
6/42 (14.3)
0/29 (0.0)
4/19 (21.1)
23/147 (15.6)
22/147 (14.9)
16/147 (10.9)
Cardiac pacing
82
1/26 (3.8)
2/21 (9.5)
1/13 (7.7)
0/22 (0.0)
4/82 (4.9)
4/82 (4.9)
1/82 (1.2)
4
6
Total
Summary of
guidelines median
(IQR), %
698
39
6.6 (3.8–18.4)
Abbreviations and footnotes as in Tables 1 and 2.
10
2.6 (1.5–9.5)
0 (0–7.7)
7.1 (0–7.7)
59
40
27
5.4 (4.9–9.7)
3.7 (3.4–4.9)
1.2 (0.95–3.7)
Roos et al
Table 4.
Supraventricular
arrhythmias
Recommendation
With LOE-A
1
2
3
4
5
6
7
Downloaded from http://circep.ahajournals.org/ by guest on May 9, 2017
8
1
2
3
Ventricular
arrhythmias
and SCD
207
Reassessment of LOE-A per Recommendation
Guideline
Syncope
Scientific Value of Arrhythmia Guidelines
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Guideline Chapter
Acute management of stable
SVT
Acute management of stable
SVT
Conversion of stable atrial
flutter
Conversion of stable atrial
flutter
Conversion of stable atrial
flutter
Conversion of stable atrial
flutter
Conversion of stable atrial
flutter
Rate control of stable atrial
flutter
Beta-blocker in vasovagal
syncope
Ischemic cardiomyopathy with
severely depressed LVEF
Nonischemic cardiomyopathy
with severely depressed LVEF
Resting electrocardiogram
Ambulatory
electrocardiography
Electrocardiographic
techniques and measurements
Management of cardiac arrest
Torsades de pointes
Torsades de pointes
LV dysfunction due to prior MI
LV dysfunction due to prior MI
LV dysfunction due to prior MI
LV dysfunction due to prior MI
Endocrine disorders and
diabetes mellitis
Dilated, nonischemic
cardiomyopathy
Neuromuscular disorders
Heart failure, secondary
prophylaxis
Heart failure due to CAD,
primary prophylaxis
Long QT syndrome
Lipids
Elderly patients
Digitalis toxicity
Drug-induced long QT
syndrome
Sodium channel
blocker-related toxicity
Class of
Recommendation
Total No. of
References
for This
Recommendation
No. of References
of Randomized
Studies and/or
Meta-Analyses
Correctly
Referenced for
LOE-A (⫹/⫺)
Correctly
Referenced
Class I or III
LOE-A (⫹/⫺)
I
3
0
⫺
⫺
I
1
1
⫺
⫺
I
5
1
⫺
⫺
IIa
2
2
⫹
⫺
IIb
2
2
⫹
⫺
IIb
2
2
⫹
⫺
IIb
2
2
⫹
⫺
I
4
4
⫹
⫹
III
5
4
⫹
⫹
I
9
0
⫺
⫺
I
9
0
⫺
⫺
I
I
7
3
0
0
⫺
⫺
⫺
⫺
IIa
5
0
⫺
⫺
I
I
I
I
I
I
III
I
7
1
1
10
10
3
2
11
0
0
0
0
2
3
2
0
⫺
⫺
⫺
⫺
⫹
⫹
⫹
⫺
⫺
⫺
⫺
⫺
⫹
⫹
⫹
⫺
I
4
3
⫹
⫹
I
I
7
3
0
3
⫺
⫹
⫺
⫹
I
4
2
⫹
⫹
I
I
I
I
I
1
5
21
1
5
0
4
11
0
0
⫺
⫹
⫹
⫺
⫺
⫺
⫹
⫹
⫺
⫺
I
5
0
⫺
⫺
(Continued)
208
Table 4.
Circ Arrhythm Electrophysiol
April 2011
Continued
Guideline
Atrial fibrillation
Recommendation
With LOE-A
1
2
3
4
5
6
7
8
9
10
Downloaded from http://circep.ahajournals.org/ by guest on May 9, 2017
11
12
13
14
15
16
17
18
19
20
21
22
23
Cardiac pacing
1
2
3
4
Guideline Chapter
Class of
Recommendation
Total No. of
References
for This
Recommendation
No. of References
of Randomized
Studies and/or
Meta-Analyses
Correctly
Referenced for
LOE-A (⫹/⫺)
Correctly
Referenced
Class I or III
LOE-A (⫹/⫺)
I
I
I
I
I
I
IIa
I
24
22
21
28
0
19
14
32
13
10
8
5
0
8
5
29
⫹
⫹
⫹
⫹
⫺
⫹
⫹
⫹
⫹
⫹
⫹
⫹
⫺
⫹
⫺
⫹
IIa
17
15
⫹
⫺
III
10
10
⫹
⫹
I
6
6
⫹
⫹
I
10
8
⫹
⫹
I
6
6
⫹
⫹
I
10
9
⫹
⫹
IIa
8
8
⫹
⫺
III
5
5
⫹
⫹
III
5
5
⫹
⫹
I
6
6
⫹
⫹
IIa
9
7
⫹
⫺
IIa
6
6
⫹
⫺
III
15
15
⫹
⫹
I
5
3
⫹
⫹
IIa
4
3
⫹
⫺
IIa
9
4
⫹
⫺
IIa
5
4
⫹
⫺
IIb
6
3
⫹
⫺
I
4
2
⫹
⫹
Preventing thromboembolism
Preventing thromboembolism
Preventing thromboembolism
Preventing thromboembolism
Preventing thromboembolism
Preventing thromboembolism
Preventing thromboembolism
Pharmacological cardioversion
of afib of up to 7-day duration
Pharmacological cardioversion
of afib of up to 7-day duration
Pharmacological cardioversion
of afib of up to 7-day duration
Pharmacological cardioversion
of afib of up to 7-day duration
Pharmacological cardioversion
of afib of up to 7-day duration
Pharmacological cardioversion
of afib of up to 7-day duration
Pharmacological cardioversion
of afib of up to 7-day duration
Pharmacological cardioversion
of afib of up to 7-day duration
Pharmacological cardioversion
of afib of up to 7-day duration
Pharmacological cardioversion
of afib of up to 7-day duration
Pharmacological cardioversion
of afib present ⬎7 Days
Pharmacological cardioversion
of afib present ⬎7 Days
Pharmacological cardioversion
of afib present ⬎7 Days
Pharmacological maintenance
of sinus rhythm
Preventing postoperative atrial
fibrillation
Preventing postoperative atrial
fibrillation
Choice of pacing mode in AV
block
Choice of pacing mode in AV
block
Cardiac pacing in hypertrophic
cardiomyopathy
Cardiac resynchronization
therapy
SVT indicates supraventricular tachycardia; ICD, internal cardioverter defibrillator; VT, ventricular tachycardia; VF, ventricular fibrillation; LV, left ventricular; MI,
myocardial infarction; CAD, coronary artery disease; afib, atrial fibrillation; d, days; AV, atrioventricular. Additional abbreviations and footnotes as in Tables 1 and 2.
rectly ranked as LOE-A cannot be analyzed in randomized
studies, eg, because of ethical reasons. In these cases, surveys
that are actually ranked as LOE-C have been used to
retrospectively underline the importance of such recommendations. As an alternative, recommendations with consistent
evidence derived from surveys could be summarized in a new
category of LOE-S in order to not compromise correctly
supported recommendations with LOE-A and attach more
scientific importance than simply LOE-C.
As an indicator for improvement in the guideline
process, the number of inappropriate uses of LOE-A is
decreasing in the most recent guidelines.10,11 In addition,
Roos et al
Downloaded from http://circep.ahajournals.org/ by guest on May 9, 2017
the guidelines for the management of patients with atrial
fibrillation10 quantified the number needed for the LOE-A
label to 3 to 5 randomized studies or meta-analyses, which
is more difficult to reach and could be added in updates of
guidelines.
More than a third of the references were older than 10 years
at the time of release of the analyzed guidelines. This is
surprising because 3 of 5 guidelines were updates. Although
some older milestone studies remain pivotal, some other trials
from the early 1990s used a less sophisticated design, or
treatments, procedures, or instrumentation that are no longer
in use. As a consequence, a list of research topics attached at
the end of the guideline text could serve as an indicator for
areas in which studies with a new methodological approach
are needed.
Ninety-nine percent of the recommendations lack sufficient scientific support and are situated in a gray zone of
evidence, which include Class I and Class III recommendations of LOE-B and LOE-C and all Class II recommendations
independent of the level of evidence. Class I and Class III
recommendations of LOE-B (13% and 2.3%, respectively)
imply the existence of only one randomized (or even a
nonrandomized) trial. Further studies might yield similar
results, leading to a LOE-A label, or might yield opposite
results, switching the recommendation from Class I to Class
II. Updates in guidelines led to the highest increase in the
category of recommendations with conflicting evidence
(Class II),6 suggesting that additional studies lead to more
conflicting results. The crux of all Class II recommendations
(46.1% in our study) independent of the LOE is that
conflicting evidence has to be assessed by expert opinion
to look into the methodology and the outcomes of the
studies. Finally, all LOE-C recommendations (58.5% in
our study) are by definition “only consensus opinion of
experts or standard-of-care.”
All recommendations in the gray zone are prone to a
potential bias such as financial and intellectual interests of the
committee members.2–5,12 Additionally, methodological
shortcomings may lead to biased recommendations. Decision
making in the guideline-writing process often does not follow
clearly defined and transparent methodological standards.
Less than 10% of the guidelines used and described formal
methods of combining expert opinion.1 For example the
definition of usefulness/efficacy is not precisely quantified in
terms of quality of life, number needed to treat, cost-benefit,
cost of year of life gained, or other cost benefit analysis, as
well as clinical relevance. There is a difference between the
level of evidence that can be derived from large randomized
trials with hard end points and results from smaller randomized trials with surrogate markers or meta-analyses. Furthermore, interpretation of systematic reviews with meta-analyses
includes a subjective component. This can lead to discordant
conclusions13 or even to different recommendations between
guidelines.2,6 In addition, general agreement and consensus
opinion of experts in the decision-making process is not
clearly defined, ie, whether the full or the simple majority of
committee members has to agree.
As a consequence of these shortcomings, a reform of the
guideline process has been suggested by different authors.
Scientific Value of Arrhythmia Guidelines
209
Sniderman and Furberg proposed to post a prefinal version of
a guideline on the internet to give an opportunity to exchange
scientific opinion before final decisions are taken.5 Hirsch
and Guyatt suggested that unconflicted methodologists instead of potentially biased experts should have the final
responsibility for recommendations.2
In order to further improve future guideline-writing processes, the most recent ESC guidelines for the diagnosis and
treatment of acute and chronic heart failure 200814 summarize
the major “gaps in the evidence” in an attempt to focus future
clinical research on important issues that have not been
adequately addressed. However, the definition of “gap in the
evidence” is defined as absence of any recommendations in a
certain area. Therefore, it differs from our definition which
includes nonconclusive evidence.
A main limitation of this study is the focus on Class I and
III recommendations with LOE-A. We did not carry out a
detailed exploration of Class IIa and IIb recommendations
with LOE-A which imply much more expert opinion than
classifications I or III. Furthermore, to analyze whether
recommendations with LOE-B are really “derived from
single randomized or nonrandomized trials” was beyond the
scope of this article. Because there are more LOE-B than
LOE-A recommendations, a critical interpretation of guidelines in general might have even more of an impact than just
focusing on LOE-A recommendations alone.
Conclusions
Our findings raise the question of the accuracy of LOE-A in
medical guidelines in general and highlight the importance of
a critical use of all recommendations. Moreover, they underline the need for improving the guideline-writing process.
Although guidelines are improving by adding recommendation classifications and grading schemes for LOE, the points
mentioned above highlight the fact that guidelines are mostly
driven by consensus or expert opinion. Even recommendations with LOE-A need to be reassessed for sufficient
empirical support. In the absence of highest correctly referenced evidence (Class I and III recommendations of LOE-A)
clinicians should be even more cautious in implementing
recommendations into clinical practice, also taking into
consideration primary data, their own knowledge, and a
multidisciplinary approach. Further randomized, doubleblinded, and/or crossover-designed studies should focus on
areas with a gap in the evidence, such as existing but not yet
convincing (LOE-B) or conflicting (Class II) evidence.
Acknowledgments
We thank Matt Owens and Mabelle Young for proofreading.
Sources of Funding
Jeannette Brodbeck was supported by a research fellowship from the
Swiss National Science Foundation.
Disclosures
Pedro Brugada discloses that his institution has received research
grants and himself speaker fees from: Medtronic, Boston Scientific,
St Jude Medical, Biosense Webster and Biotronik.
210
Circ Arrhythm Electrophysiol
April 2011
References
Downloaded from http://circep.ahajournals.org/ by guest on May 9, 2017
1. Shaneyfelt TM, Mayo-Smith MF, Rothwangl J. Are guidelines following
guidelines? The methodological quality of clinical practice guidelines in
the peer-reviewed medical literature. JAMA. 1999;281:1900 –1905.
2. Hirsh J, Guyatt G. Clinical experts or methodologists to write clinical
guidelines? Lancet. 2009;374:273–275.
3. Choudhry NK, Stelfox HT, Detsky AS. Relationships between authors of
clinical practice guidelines and the pharmaceutical industry. JAMA. 2002;
287:612– 617.
4. Detsky AS. Sources of bias for authors of clinical practice guidelines.
CMAJ. 2006;175:1033, 1035.
5. Sniderman AD, Furberg CD. Why guideline-making requires reform.
JAMA. 2009;301:429 – 431.
6. Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC Jr. Scientific
evidence underlying the ACC/AHA clinical practice guidelines. JAMA.
2009;301:831– 841.
7. Blomstrom-Lundqvist C, Scheinman MM, Aliot EM, Alpert JS, Calkins
H, Camm AJ, Campbell WB, Haines DE, Kuck KH, Lerman BB, Miller
DD, Shaeffer CW, Stevenson WG, Tomaselli GF. ACC/AHA/ESC
guidelines for the management of patients with supraventricular arrhythmias: a report of the American College of Cardiology/American Heart
Association Task Force on Practice Guidelines and the European Society
of Cardiology Committee for Practice Guidelines (Writing Committee to
Develop Guidelines for the Management of Patients With Supraventricular Arrhythmias). 2003. American College of Cardiology Web Site.
http://www.acc.org/clinical/guidelines/arrhythmias/sva_index.pdf.
Accessed February 7, 2010.
8. Moya A, Sutton R, Ammirati F, Blanc JJ, Brignole M, Dahm JB, Deharo
JC, Gajek J, Gjesdal K, Krahn A, Massin M, Pepi M, Pezawas T, Granell
RR, Sarasin F, Ungar A, van Dijk JG, Walma EP, Wieling W, Abe H,
Benditt DG, Decker WW, Grubb BP, Kaufmann H, Morillo C, Olshansky
B, Parry SW, Sheldon R, Shen WK, Vahanian A, Auricchio A, Bax J,
Ceconi C, Dean V, Filippatos G, Funck-Brentano C, Hobbs R, Kearney
P, McDonagh T, McGregor K, Popescu BA, Reiner Z, Sechtem U, Sirnes
PA, Tendera M, Vardas P, Widimsky P, Acarturk E, Andreotti F,
Asteggiano R, Bauersfeld U, Bellou A, Benetos A, Brandt J, Chung MK,
Cortelli P, Da Costa A, Extramiana F, Ferro J, Gorenek B, Hedman
A, Hirsch R, Kaliska G, Kenny RA, Kjeldsen KP, Lampert R, Molgard H,
Paju R, Puodziukynas A, Raviele A, Roman P, Scherer M, Schondorf R,
Sicari R, Vanbrabant P, Wolpert C, Zamorano JL. Guidelines for the
diagnosis and management of syncope (version 2009): the Task Force for
the Diagnosis and Management of Syncope of the European Society of
Cardiology (ESC). Eur Heart J. 2009;30:2631–2671.
9. Zipes DP, Camm AJ, Borggrefe M, Buxton AE, Chaitman B, Fromer M,
Gregoratos G, Klein G, Moss AJ, Myerburg RJ, Priori SG, Quinones MA,
Roden DM, Silka MJ, Tracy C, Priori SG, Blanc JJ, Budaj A, Camm
AJ, Dean V, Deckers JW, Despres C, Dickstein K, Lekakis J, McGregor
K, Metra M, Morais J, Osterspey A, Tamargo JL, Zamorano JL, Smith SC
10.
11.
12.
13.
14.
Jr, Jacobs AK, Adams CD, Antman EM, Anderson JL, Hunt SA, Halperin
JL, Nishimura R, Ornato JP, Page RL, Riegel B. ACC/AHA/ESC 2006
guidelines for management of patients with ventricular arrhythmias and
the prevention of sudden cardiac death: a report of the American College
of Cardiology/American Heart Association Task Force and the European
Society of Cardiology Committee for Practice Guidelines (Writing Committee to Develop guidelines for management of patients with ventricular
arrhythmias and the prevention of sudden cardiac death) developed in
collaboration with the European Heart Rhythm Association and the Heart
Rhythm Society. Europace. 2006;8:746 – 837.
Fuster V, Ryden LE, Cannom DS, Crijns HJ, Curtis AB, Ellenbogen KA,
Halperin JL, Le Heuzey JY, Kay GN, Lowe JE, Olsson SB, Prystowsky
EN, Tamargo JL, Wann S, Smith SC Jr, Jacobs AK, Adams CD,
Anderson JL, Antman EM, Halperin JL, Hunt SA, Nishimura R, Ornato
JP, Page RL, Riegel B, Priori SG, Blanc JJ, Budaj A, Camm AJ, Dean V,
Deckers JW, Despres C, Dickstein K, Lekakis J, McGregor K, Metra M,
Morais J, Osterspey A, Tamargo JL, Zamorano JL. ACC/AHA/ESC 2006
guidelines for the management of patients with atrial fibrillation: full text:
a report of the American College of Cardiology/American Heart Association Task Force on practice guidelines and the European Society of
Cardiology Committee for Practice Guidelines (Writing Committee to
Revise the 2001 guidelines for the management of patients with atrial
fibrillation) developed in collaboration with the European Heart Rhythm
Association and the Heart Rhythm Society. Europace. 2006;8:651–745.
Vardas PE, Auricchio A, Blanc JJ, Daubert JC, Drexler H, Ector H,
Gasparini M, Linde C, Morgado FB, Oto A, Sutton R, Trusz-Gluza M.
Guidelines for cardiac pacing and cardiac resynchronization therapy: The
Task Force for Cardiac Pacing and Cardiac Resynchronization Therapy of
the European Society of Cardiology. Developed in collaboration with the
European Heart Rhythm Association. Eur Heart J. 2007;28:2256 –2295.
Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines:
go gently into that good night. JAMA. 2009;301:868 – 869.
Shrier I, Boivin JF, Platt RW, Steele RJ, Brophy JM, Carnevale F,
Eisenberg MJ, Furlan A, Kakuma R, Macdonald ME, Pilote L, Rossignol
M. The interpretation of systematic reviews with meta-analyses: an
objective or subjective process? BMC Med Inform Decis Mak. 2008;8:19.
Dickstein K, Cohen-Solal A, Filippatos G, McMurray JJ, Ponikowski P,
Poole-Wilson PA, Stromberg A, van Veldhuisen DJ, Atar D, Hoes AW,
Keren A, Mebazaa A, Nieminen M, Priori SG, Swedberg K, Vahanian A,
Camm J, De Caterina R, Dean V, Dickstein K, Filippatos G, FunckBrentano C, Hellemans I, Kristensen SD, McGregor K, Sechtem U, Silber
S, Tendera M, Widimsky P, Zamorano JL. ESC Guidelines for the
diagnosis and treatment of acute and chronic heart failure 2008: the Task
Force for the Diagnosis and Treatment of Acute and Chronic Heart
Failure 2008 of the European Society of Cardiology. Developed in collaboration with the Heart Failure Association of the ESC (HFA) and
endorsed by the European Society of Intensive Care Medicine (ESICM).
Eur Heart J. 2008;29:2388 –2442.
CLINICAL PERSPECTIVE
Guidelines have become very important in assisting with decision making in clinical practice. However, few studies have
critically analyzed the level of evidence (LOE) supporting the guidelines. This study assessed the accuracy of the
referenced literature that has led to recommendations with a level of evidence A (LOE-A). The latest updates of the practice
guidelines related to arrhythmia posted on the European Society of Cardiology web site were analyzed. Only 27 of 698
recommendations were correctly referenced as class I or III LOE-A, implying definite evidence-based positive or negative
recommendations. These observations highlight the importance of a critical use of guidelines in clinical practice. Further
randomized double blinded and/or crossover designed studies should focus on areas with a gap in the evidence, such as
existing but not yet convincing (LOE-B) or conflicting (Class II) recommendations.
A Critical Analysis of the Scientific Evidence Behind International Guidelines Related to
Cardiac Arrhythmias
Markus Roos, Jeannette Brodbeck, Andrea Sarkozy, Gian Battista Chierchia, Carlo De
Asmundis and Pedro Brugada
Downloaded from http://circep.ahajournals.org/ by guest on May 9, 2017
Circ Arrhythm Electrophysiol. 2011;4:202-210; originally published online March 3, 2011;
doi: 10.1161/CIRCEP.110.958181
Circulation: Arrhythmia and Electrophysiology is published by the American Heart Association, 7272 Greenville
Avenue, Dallas, TX 75231
Copyright © 2011 American Heart Association, Inc. All rights reserved.
Print ISSN: 1941-3149. Online ISSN: 1941-3084
The online version of this article, along with updated information and services, is located on the
World Wide Web at:
http://circep.ahajournals.org/content/4/2/202
Permissions: Requests for permissions to reproduce figures, tables, or portions of articles originally published
in Circulation: Arrhythmia and Electrophysiology can be obtained via RightsLink, a service of the Copyright
Clearance Center, not the Editorial Office. Once the online version of the published article for which
permission is being requested is located, click Request Permissions in the middle column of the Web page
under Services. Further information about this process is available in the Permissions and Rights Question and
Answer document.
Reprints: Information about reprints can be found online at:
http://www.lww.com/reprints
Subscriptions: Information about subscribing to Circulation: Arrhythmia and Electrophysiology is online at:
http://circep.ahajournals.org//subscriptions/