Download 1-3 - Universiteit Utrecht

Quality of reporting in non-inferiority vaccine trials; a systematic review.
Master thesis
Robine Donken BSc. (3736660)
Master Biomedical Sciences; Epidemiology
Universiteit Utrecht
Daily supervision:
Second reviewer:
Dr. M.J. Knol (Rijksinstituut voor Volksgezondheid en Milieu)
Dr. R.H.H. Groenwold (UMC Utrecht)
QUALITY OF REPORTING IN NON-INFERIORITY VACCINE
TRIALS; A SYSTEMATIC REVIEW
Donken R 1, Knol MJ 1
1 Epidemiology and Surveillance Unit, Centre for Infectious Disease Control, National
Institute for Public Health and the Environment (RIVM), Bilthoven
Abstract
Background: A non-inferiority (NI) design aims to demonstrate that the experimental arm is
not worse than a comparator within a pre-specified margin. A margin of a risk difference of
10% or a Geometric Mean Titer (GMT) ratio from 1.5 to 2.0 is implicitly recommended for
vaccine trials. This study explores which NI margins were used in vaccine studies and how
they were determined.
Methods: A systematic literature search was performed on February 21st, 2013. This resulted
in 260 eligible articles, 148 articles were selected randomly; data was collected by the use of
a data extraction form. This thesis had an explorative character and therefore descriptive
statistics were used to show the determination of the NI margins and which NI margins were
used.
Results: From the selected articles, 48 were excluded because they did not met the predefined
criteria. In 81/100 articles a NI margin on difference was determined. In 71% of reviewed
articles a margin of 10% was used, 19% of the articles stated margins lower than 10%
(range 3-7.5%), whereas in 10% of the articles margins larger than 10%(range 11.5-50%)
were stated as the NI margin. In 55/100 articles a NI margin on the GMT ratio was stated. In
60% of the articles a margin of 0.67/1.5 was used, in 33% the stated margins were 0.5/2.0. A
large amount (85%) of the examined articles did not discuss how the NI margin was
determined. Only five articles referred to the FDA or EMA guidelines, eight articles referred
to methodological articles and two articles referred to previous articles specific for the
vaccine. In 82/100 articles the conclusion corresponded to the results, in fourteen the
conclusion could not be checked because the effect estimates were not shown, in four articles
the conclusion of the authors was not in line with the results.
Conclusion: The majority of reviewed articles showed no determination of the NI margin. Not
all articles showed all their estimates on NI. Reporting on design and results of NI vaccine
trials could therefore be improved. Current guidelines on NI trials contain no explicit margin
for vaccine trials. The choice of the NI margin is essential for the outcome of the NI trials,
although determination is a great challenge. Regulatory offices might help with presenting
guidelines which contain an explicit margin (for a specific vaccine/disease).
Introduction
A trial to evaluate the efficacy of a new drug can have different designs. A superiority design
aims to demonstrate that an experimental treatment is better than a comparator. A noninferiority (NI) design aims to demonstrate that the experimental arm is not worse than a
comparator within a pre-specified margin.[1-3] NI designs might be useful in situations when
the efficacy of the new drug is supposed to be the same as the comparator but the new drug
might have additional benefits, such as less adverse events or reduced costs. NI designs can
also be used to show indirectly the efficacy of a new drug over placebo, when the use of a
placebo is not ethical.[3, 4] Two fundamental issues exist when using an NI design. First is
the possibility to develop biocreep, this occurs when a new treatment is stated as non-inferior,
because it does not exceed the margin of NI, although it could be slightly less well performing
than its comparator. When a new study is designed and the new treatment is now used as a
comparator, the second new treatment might be non-inferior but could again be less effective
than the first comparator and therefore ultimately worse than the placebo. Secondly the use of
the NI design can enhance the evaluation of drugs that do not have important efficacy or
safety advantages over the standard treatment, these treatments are the so called “me-toodrugs”.[3]
The NI study uses a margin, any difference between the comparator and the new treatment
larger than this margin is unacceptable and called inferior.[3] The choice of the NI margin is
essential for the outcome of the NI trial, although determination is complicated. There is some
general guidance on the determination of the NI margin. The size of the NI margin should be
based on a combination of statistical and clinical considerations.[4] The method to determine
the NI margin as described by the U.S. Department of Health and Human Services, Food and
Drug Administration (FDA) is as follows. First the total assumed effect of the active
comparator over placebo needs to be determined and a conservative estimate of this effect
should be taken, to assure that the test drug does have an effect greater than zero. This
conservative estimate is for example the lower limit of the confidence interval of the
difference in effect when comparing the active control to placebo. Secondly, the largest
clinically meaningful difference or the preserved fraction should be determined, for instance
50%. [2, 3] For example when a new treatment is compared to an active comparator. This
active comparator has compared to placebo a risk difference of 15% (95%CI 10%-20%). The
10% is now used as the conservative effect estimate. Using a preserved fraction of 50%,
results in a NI margin of 5%.
Vaccine studies for measuring efficacy and safety can also be done by a NI design. Points
unique to NI vaccine trials are that most vaccines have high effectiveness and work
preventive. Using the method described above, high effectiveness would lead to a large
margin. However, for preventive interventions you prefer a small NI margin, i.e. one is not
willing to lose some of the effects. Thereby it is difficult to study hard endpoints on efficacy,
for example when the incidence is low; correlates of protection i.e. antibody concentrations
are therefore often used as an endpoint. These are the immune response rate and the geometric
mean antibody concentration (GMC, or geometric mean titer(GMT)).[5] In literature, it is
suggested that the NI margin on difference between treatment and comparator should be
10%.[6] For GMT ratio suggestions on preferable margin are less clear, although should be
between 1.5 and 2.0. [3] Despite these recommendations, explicit NI margins for vaccine
trials are not mentioned in the current guidelines of the regulatory offices.[2, 7]
Because no hard guidelines on which NI margin to use for vaccine trials exists, the aim of this
thesis is to evaluate which NI margins are used in NI vaccine trials and how they were
determined. Thereby it was also explored which study characteristics influenced the choice of
the NI margin.
Methods
Systematic search
We performed a systematic search for NI vaccine trials in PubMed and Cochrane Library on
February 21st, 2013. The used search string can be found in box 1. In total 970 articles were
retrieved as can be seen in figure 1, with 385 duplicates. From the articles which were eligible
for full-text analysis (n=260), 148 were selected randomly for this thesis. We excluded
articles that were not written in Dutch or English, studies that were not performed on humans,
studies that were not performed in a trial setting, studies in which no infectious disease was
studied, studies with a superiority design, abstracts from presentations and articles on
bioequivalence studies. These last ones might have come up in the search because of the
words equivalence and equivalent in the search term.
Box 1. The search term which was used to search for NI vaccine trials in PubMed and Cochrane.
For PubMed the search filter Clinical Trial was used.
(non-inferior*[tiab] OR noninferior*[tiab] OR NI[tiab] OR equivalence[tiab] OR equivalent[tiab])
AND (vaccin*[tiab] OR immunogenicity[tiab] OR (immune[tiab] AND response[tiab] AND rate[tiab])
OR (immune[tiab] AND response[tiab]) OR (geometric[tiab] AND mean[tiab] AND
concentration[tiab]) OR (geometric[tiab] AND mean[tiab] AND titer[tiab]) OR GMC[tiab] OR
GMT[tiab])
Data extraction
A data extraction form was designed and used to obtain the required data from the articles.
The data extraction form can be found in appendix 1. RD extracted the data from the articles
and in case of doubt the articles were discussed with MK. All articles in which RD believed
the conclusion was not in line with the results were also discussed with MK.
Data extracted from the articles were general characteristics on the articles such as the author,
year of publication, journal in which the article was published, infectious disease studied,
study population, treatment arms, analysis population and measured endpoints. Also
characteristics which are specific for NI trials were obtained including whether the concern
was on safety or efficacy, the NI margin, the way in which the NI margin was determined, the
effect estimates and their correspondence to the NI margin and the conclusion of the authors.
Both NI margin on difference and NI margin on GMT ratio were extracted from the articles.
The GMT ratio is an effect measure which is specific to research in the field of
immunogenicity. Because antibody levels are not normally distributed these values are logtransformed and then back transformed to the original scale. When the difference between
two log-transformed values is back transformed to the original scale a ratio is obtained, this is
the GMT ratio.
Biocreep
We checked within our dataset whether a potential for biocreep existed. This was done by
searching whether brand names or compositions of new treatments were used as a comparator
in later NI trials. When indeed a new treatment was later used as a comparator, the design,
aim and effect estimates were analyzed to see whether there is potential for biocreep.
Data analysis
Frequencies were calculated to describe the data. A relation between different diseases,
different aims, whether studies were published before or after 2006 and the reported NI
margin were analyzed using cross-tabs. Statistical analyses were performed using SPSS 19
(SPSS Inc, USA).
Pubmed
#494
Cochrane Library
# 446
Total #940
EXCLUSION:
- Duplicates # 380
Before selection on
title #560
EXCLUSION:
- New treatment not a vaccine / Bioequivalence #213
- Not a human study #16
- Not a trial #12
- No infectious disease studied #4
Before selection on
abstract
#317
After abstract
selection
# 260
↓
RANDOM SAMPLE
#148
EXCLUSION:
- Language # 9
- Superiority /no NI design #10
- Not a human study #2
- Not a (comparative) trial #18
- New treatment not a vaccine/ Bioequivalence # 13
- Duplicate #2
- Not an article #2
- No infectious disease studied #1
EXCLUSION:
- No NI margin defined #32
- Superiority/no NI design #13
- Not a (comparative) trial #2
- New treatment not a vaccine #1
Data extracted
from 100
articles
Figure 1. The selection process of the used articles.
Results
In figure 1 an overview of the selection process of articles on NI vaccine trials is shown. For
the analyses in this thesis, 100 articles are included. From the sample of 148, 48 articles were
excluded on the basis of predefined exclusion criteria.
Articles analyzed
The analyzed articles were published between 1996 and 2013, where most studies were
published in the last five years. General characteristics of the trials are presented in table 1.
Most of the trials (n=37) were published in Vaccine. In most of the trials it was studied
whether combined or concomitant administration was as effective or safe as separate
administration of vaccines (n=36) or a new vaccine was tested against an already available or
licensed vaccine (n=32). Only two of the reviewed trials also had a placebo arm.
Table 1. General characteristics of reviewed trials (n=100)
Number
Journal*
Vaccine
The Pediatric Infectious Disease Journal
Human Vaccines
Pediatrics
Journal of Infectious Diseases
International Journal of Infectious Diseases
Other
37
19
7
5
3
3
26
Comparison in trial
Combined vs. separate administration of vaccines
New vs. already available vaccine
Different doses tested
Different schedules of administration tested
Different administration routes tested
Essential change in production process
Lot-to-lot consistency
36
32
15
7
5
3
2
Studied treatment arms
One treatment vs. comparator(s)
Multiple treatments vs. comparator(s)
Multiple treatments alone
81
17
2
* Journals are only named when 3 or more articles on NI vaccine trials were published, otherwise they
were stated under other
Design and analysis of NI vaccine trials
The NI margin used most (71%) in the reviewed vaccine trials was a difference of 10%
(figure 2a). For GMT ratio the variability between used NI margins was much smaller,
0.67/1.5 is the most commonly used (33%) GMT ratio, although in almost 19% a ratio of
0.5/2.0 is used (figure 2b). We also examined whether the used margins in articles differed
between different aims. Remarkable was that all GMT ratios when comparing separate versus
combined administration were between 0.5 and 0.67, the studies who aimed to evaluate
difference between new and an already licensed vaccine also used 0.25 and 0.33. (Fisher’s
Exact test p=0.16)
a. NI margins on difference
b. NI margin on GMT ratio
Figure 2. NI margins used in vaccine trials.
Because an article can contain multiple NI margins, these numbers will add up to more than one
hunderd. No margin presented means that the article did not contain a margin for difference, but for
GMT ratio or the other way around. All articles contained some NI margin.
Determination of the margin was explained in only fifteen of the articles. Five articles referred
to the guidelines of either European Medicines Agency (EMA) or FDA. Eight articles
determined the NI margin by the use of methodological papers on margin determination. Two
articles determined their margin based on previous articles examining this (type of) vaccine.
None of these sources stated an exact margin and none of the NI trials showed a calculation of
the NI margin. Before publication of the extension on the CONSORT statement for NI trials
in 2006 (n=20), only 5% of the articles stated how they determined their margin. This
increased to 18% in the articles published in 2006 and thereafter (n=80) (χ²=1.961 p=0.16).
Which specific margins were used to study vaccines for specific diseases is shown in table 2.
Remarkably, while the most used margin was 10%, all HPV studies (n=3) used a margin of
5%, for rabies (n=1) a margin of 50% was used and for rubella (n=7) the mostly used margin
(57%) was smaller than 10%. Most GMT ratios (95%) were between 0.5 and 0.67.
Table 2. Used NI margins per studied disease
Margin on difference
Smallpox
Meningococcal
Malaria
Rotavirus
Tick-borne encephalitis
Hib
Hepatitis B
Pertussis
Pneumococcal
Diphteria
Tetanus
Polio
Influenza
Varicella
Yellow Fever
Mumps
Tuberculosis
Hepatitis A
Measles
Rubella
Rabies
HPV
N <10% 10% >10%
1
0% 100%
0%
10
0% 100%
0%
2
0% 100%
0%
3
0% 100%
0%
1
0% 100%
0%
22
0% 95%
5%
20 10% 90%
0%
32
6% 88%
6%
7
0% 86% 14%
33 12% 85%
3%
33 12% 85%
3%
26 23% 77%
0%
12
0% 75% 25%
4
0% 75% 25%
3
33% 67%
0%
7
43% 57%
0%
2
0% 50% 50%
6
50% 50%
0%
8
50% 50%
0%
7
57% 43%
0%
1
0%
0% 100%
3 100%
0%
0%
Margin on GMT
Diphteria
Tetanus
Polio
Hepatitis B
Pertussis
Hepatitis A
Malaria
Measles
Yellow Fever
Mumps
Rubella
Varicella
Pneumococcal
HPV
Rotavirus
Rabies
Hib
Influenza
Meningococcal
Tick-borne encephalitis
N
14
10
12
7
16
1
2
6
2
3
5
5
6
3
2
1
8
19
6
1
<0.5 0.5≤0.67 0.67≤1
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
0%
100%
0%
5%
95%
0%
17%
83%
0%
100%
0%
0%
In grey are the mostly used NI margins for the specific disease study.
The analysis methods of the NI vaccine trials are shown in table 3. The most commonly used
analysis population (n=59) was the per-protocol (PP) population. Almost 80% of the articles
had a NI objective stated as their primary objective. In 93 studies the NI was on efficacy.
Table 3. Analysis characteristics of the analyzed trials
Number
Analysis method
Both
Per protocol* (PP)
Intention to treat* (ITT)
Not Clear
19
59
5
28
* ITT analyses all randomized patients, PP analyses only those patients who were treated accordant to
the protocol
Conclusion of NI vaccine trials
Only one of the NI vaccine trials concluded inferiority, all the others concluded NI, or even
superiority. We checked whether the conclusion was in accordance to the displayed results. In
the majority (n=82) of articles the results and conclusion were totally in line, in nine of these
only p-values or figures were shown, so whether the CI included the NI margin could not be
checked based on the displayed results. In fourteen articles the conclusion could not be
checked because not all estimates were shown. In four articles the conclusion drawn by the
authors was not (completely) in line with the presented results. Two of these articles are
discussed anonymously in box 2.
Box 2. Articles where the conclusion was not (totally) in accordance to the displayed results.
Article 1
This study compares a new vaccine in different doses with an already available vaccine.
The used NI margin is -10%.
The difference for treatment(T) minus comparator(C) for the highest dose was:
- 2% [95% Confidence Interval (CI) -12%; 9% ]
The authors stated:
“T at the highest dose was found to be NI to the C.”
“T has a similar vaccination success rate, antibody responses and safety profile to the C. “
As the lower boundary of the CI exceeds the NI margin, we think that the conclusion drawn by the authors is
not correct, because the NI margin lies within the CI.
Article 2
This study compares a lower with a higher dose, which were administered differently.
The used NI margin is -5%.
The difference for T minus C for the third strain was:
-0.92% [95% CI -5.01%; 2.43]
The authors stated:
“For each strain, the 95% CI of the difference between the 2 groups lay above the clinically acceptable limit
of -5%, demonstrating NI.”
Although the lower boundary of the CI exceeds the NI margin minimally, the researchers think that the
conclusion for this strain is not NI.
Biocreep
The researchers found four examples of potential for biocreep in their dataset, these examples
are discussed in box 3.
Box 3. Examples with potential risk for biocreep.
Example 1
In 2001, an article was published on a NI trial where a new combination vaccine (Vaccine1) against
diphtheria, tetanus, pertussis, polio and hepatitis B was compared with an already licensed vaccine
(LicensedVaccine). The NI margin was set at 10% for seroprotection rates post dose 3. Post dose 3 the
seroprotection against all diseases was NI.
In 2012, another article was published on a NI trial where a new combination vaccine (Vaccine2) against the
same diseases as in the previous article was tested against an active comparator (Vaccine1), the study vaccine
of the first trial. The NI margin was 10% for seroprotection rates post primary immunization. In this study
only NI for diphtheria was examined and found.
Difference found in the first study:
Vaccine 1-LicensedVaccine (T-C) for diphtheria: 2.1% [90%CI -2.8 ; 7.5]
Difference found in the second study:
Vaccine 2- Vaccine 1 (T-C) for diphtheria: -2.76 [95%CI -4.55; 1.18]
In this situation the potential for biocreep is low, because in the first trial the new drug might be slightly
better, so when Vaccine 2 is not as good as Vaccine 1 but NI, it will not per se mean that it is worse than the
LicensedVaccine.
Example 2
In this article from 2012 coadministration was compared against separate administration of different
vaccines, pneumococcal vaccine and influenza vaccine. In the abstract the authors stated that
“Coadministration of the pneumococcal vaccine and the influenza vaccine was well tolerated but associated
with pneumococcal virus lower antibody responses and is of unknown clinical significance.” NI was found
for all strains because all CI of GMT ratios exceeded 0.5, although eight of thirteen strains showed indeed
lower antibody responses. When in further research coadministration is used as an active comparator, this is
potential for biocreep.
Discussion
Mostly (71%) used NI margin in vaccine trials turned out to be 10%. For GMT ratios the most
(33%) used margin was 0.67/1.5, although 0.5/2.0 was also frequently (19%) used. In 85% of
reviewed articles it was unclear how the NI margin was determined. In the articles who did
mention a source on which they based their determination, in all cases none of the
methodological articles contained an exact margin, when referred to a previous NI trial,
neither of these contained an explanation or calculation for the used margin. Not only
reporting on the design of the NI trials was weak, also in reporting the results of NI vaccine
trials, there is room for improvement. In 23 of the reviewed articles the reporting on the
results was incomplete in accordance to the extension for NI trials of the CONSORT
statement. In four of the reviewed articles the conclusion of the authors was not (totally) in
line with the presented results.
None of the official guidelines on NI (vaccine) trials contains an exact NI margin for vaccine
trials. A common recommendation for NI margin on difference is 10%, however in the
current guidelines of regulatory offices this information is lacking, while it is mentioned on
websites, methodological articles or previous guidelines. [6, 8] In previous Committee for
Medicinal Products for Human Use (CHMP, part of EMA) documents it was stated that “In
individual trials, delta can often be set to about 10 percentage point, but will need to be
smaller for high seroprotection rates.”[8] In later versions of this document, this explicit
margin is no longer discussed. The NI margin should be pre-specified in the study protocol.
[2, 7] In 2006 an extension of the CONSORT statement specific for NI trials was published,
this was updated in 2012. The checklist states that the NI margin and the rationale of choice
should be specified in the reporting article. [9, 10] The majority of reviewed articles did not
show how they determined their margin. This might be an indication for regulatory offices to
come up with clearer guidelines on how to determine the NI margin or what margin to use for
a specific vaccine. On the other hand academic journals should stimulate authors to use the
extension of the CONSORT statement to give readers a clear idea where the used NI margin
comes from.
Vaccine preventable diseases differ in their severity and vaccines in safety and efficacy. We
inspected whether there was a pattern between severity of prevented disease and the used
margin. Due to the small samples, inspection was performed visually. No clear pattern was
found. We also checked whether there was a difference between the used margins in studies
with different aims. The range of used NI margins for difference were larger for studies who
examined combined versus separate administration of vaccines, whereas the spread in used NI
margins for GMT ratios was larger for vaccine trials that examined the use of a new vaccine
to an already licensed vaccine. Here also no clear pattern was found. The variability might be
due to the lack of clear and explicit guidelines on which NI margin to use for vaccine trials.
The CONSORT statement states that for the primary and secondary objectives effect
estimates and confidences intervals (CI) should be showed, special for NI trials is that figure
with the CI and NI margin might be useful.[10] We aimed to evaluate whether the conclusion
stated by the authors was in accordance to the displayed results. In fourteen articles, the
estimates were not shown and therefore we could not check the accordance between results
and conclusion. In seven articles the conclusion was in line with the displayed p-values and in
two articles the conclusion was in line with the displayed figures. In a total of 23 articles the
reporting on the results on NI was incomplete, therefore as a reader it was hard to determine
whether it was indeed NI. Academic journals should stimulate the authors to publish all
relevant effect estimates and CI on NI trials to enable the reader to check the conclusion.
Four of the reviewed articles draw conclusions which were in our opinion not in line with the
results. Two of these articles draw the wrong overall conclusion, while in the full-text article
their findings were correct. In the other two articles the CI overlapped the NI margin, and
those were therefore not NI, although they were stated as such. In one article the exceedance
was small, only 0.01%, although the NI margin was exceeded.
Assay sensitivity is the ability to distinguish an effective therapy from an ineffective therapy.
The easiest way in which assay sensitivity could be assessed is by adding a placebo arm
besides an active comparator. Although using a placebo is not always ethical. [3] Of the
reviewed articles, only two had a placebo arm. Previous reviews found the ITT analysis as
mostly used in NI trials. [1, 11, 12] Guidelines on NI trials state that PP is often suggested for
the primary analysis, although there are some concerns on informative censoring. Because
both methods have restrictions, it might be worthwhile to publish both ITT and PP. [2] This is
supported by Sanchez et al. who stated that in case of non-compliance both ITT and PP could
be anticonservative and therefore it is more informative to present them both, also to check if
they have similar conclusions. [13] Mostly used analysis population found in this review was
the per protocol population. NI was mostly used as a primary objective. This makes the
analyzed results and conclusions more interpretable, because the sample size calculations are
based on the primary objective.
Only one of the reviewed articles concluded that the obtained estimates were not NI, but
inferior. Saad et al. demonstrated that NI trials were significantly more likely to demonstrate
positive results than superiority trials and positive results occurred in about 80% of the NI
trials. [12] It is already known that positive trial outcomes are more likely to be published.
[14] Whereas almost all reviewed articles had a positive outcome, these high scores of
positivity might be a indication for publication bias. It might be worthwhile to consider
further exploration of the publication bias in NI in comparison to superiority trials.
This review has some limitations. Firstly only PubMed and Cochrane Library were searched,
so articles in other databases might have been missed. In the searched databases only title and
abstract were searched for the used search terms. The researchers think the influence of this
method is limited, since in the extension on the CONSORT statement it was mentioned that
the title should able identification of the trial as a NI trial. For PubMed the search filter
“Clinical Trials” was used, the most recent articles might therefore have been missed, because
they were not labeled yet. Secondly the used sample of 148 articles which was used for this
thesis might not be representative for all published NI vaccine trials. For now due to time
considerations this sample was used, however for publication of this article, all 260 articles
will be analyzed. The most important strength of this review is as far as can be determined the
first review which focusses specifically on NI vaccine trials and the used margins.
Conclusion
The current guidelines on NI trials contain no explicit margins for vaccine trials. In the
majority of papers the way in which the NI margin is determined is unclear. Thereby not
always all results on NI are shown by the authors of the reviewed articles. Academic journals
and authors should be stimulated to improve the reporting on the design and results of NI
vaccine trials. On the other hand, determination on the NI margin should be simplified by
better and more explicit guidelines of regulatory offices.
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
Wangge, G., et al., Room for improvement in conducting and reporting non-inferiority randomized
controlled trials on drugs: a systematic review. PLoS One, 2010. 5(10): p. e13550.
FDA, Guidance for Industry; Non-inferiority Clinical Trials, 2010.
Rothmann, M., Wiens, B., Chan, I., Design and Analysis of Non-Inferiority Trials2012: Chapman &
Hall.
Soonawala, D. and O.M. Dekkers, ['Non-inferiority' trials. Tips for the critical reader. Research
methodology 3]. Ned Tijdschr Geneeskd, 2012. 156(19): p. A4665.
Plikaytis, B.D. and G.M. Carlone, Statistical considerations for vaccine immunogenicity trials. Part 2:
Noninferiority and other statistical approaches to vaccine evaluation. Vaccine, 2005. 23(13): p. 160614.
D'Agostino, R.B., Sr., J.M. Massaro, and L.M. Sullivan, Non-inferiority trials: design concepts and
issues - the encounters of academic consultants in statistics. Stat Med, 2003. 22(2): p. 169-86.
CHMP, Note for guidance of the clinical evaluation of vaccines, 2005.
CHMP, Note for guidance on the clinical evaluation of new vaccines, 1999.
Piaggio, G., et al., Reporting of noninferiority and equivalence randomized trials: an extension of the
CONSORT statement. JAMA, 2006. 295(10): p. 1152-60.
Piaggio, G., et al., Reporting of noninferiority and equivalence randomized trials: extension of the
CONSORT 2010 statement. JAMA, 2012. 308(24): p. 2594-604.
Le Henanff, A., et al., Quality of reporting of noninferiority and equivalence randomized trials. JAMA,
2006. 295(10): p. 1147-51.
Saad, E.D. and M. Buyse, Non-inferiority trials in breast and non-small cell lung cancer: choice of
non-inferiority margins and other statistical aspects. Acta Oncol, 2012. 51(7): p. 890-6.
Matilde Sanchez, M. and X. Chen, Choosing the analysis population in non-inferiority studies: per
protocol or intent-to-treat. Stat Med, 2006. 25(7): p. 1169-81.
Rising, K., P. Bacchetti, and L. Bero, Reporting bias in drug trials submitted to the Food and Drug
Administration: review of publication and presentation. PLoS Med, 2008. 5(11): p. e217; discussion
e217.
Appendix 1:
Date of data extraction:
First author:
Date of publication:
Journal:
Vaccine:
Comparator:
Data Extraction Form
Study domain:
Date of study:
Primary endpoint:
Secondary endpoint:
Endpoint measured at:
Analysis of:
Antibody Threshold:
Series:
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
1)__________________________3)___________________________
2)__________________________4)___________________________
________________________________________________________
____________________________□ not clear
________________________________________________________
________________________________________________________
________________________________________________________
short/long term immune response
____________________________ IU/ml
□ Priming
□ Booster
Study Objectives:
NI used as
NI concern on
Treatment arms:
Placebo controlled
□ Primary Obj.
□ Safety
□ T vs single C
□ Yes
Margin of NI:
Method to determine:
Analysis method:
Primary
Secondary
□ Secondary obj. □ Both
□Not clear
□ Efficacy
□ Costs □ Both S&E □ Not Clear
□ T vs multi Cs □ T vs C vs P □ T vs Cs □ Other
□ No
_____________________________________________ □ Not Clear
_____________________________________________ □ Not Clear
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
□ ITT □ PP
□ ITT □ PP
Effect estimate, confidence interval and NI margin:
Conclusion:
Comments:
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
________________________________________________________
Appendix 2: Summary
The regulations of the Board of Examiners of the Graduate School of Life Sciences (art. 5.2.6
and 5.3.5) state that every thesis should contain a laymen summary.
When a new drug or treatment is developed, the efficacy could be showed in a study with a
controlled setting. A superiority study, often comparing the new drug to a placebo, evaluates
whether the new drug is better than the placebo. Sometimes the use of a placebo in a study is
unethical or unfeasible, for example when a commonly used and licensed drug exists for that
specific disease. In those cases the already available drug could be used as an active
comparator. The newly developed drug might not necessarily be better than this already
available drug, but should have other benefits, such as being cheaper or having fewer side
effects. When a study tries to show that a new drug does not perform worse than the active
comparator, its design is called non-inferiority (NI). Showing that two drugs are equally
effective is statistically impossible. A NI study therefore evaluates if the new drug is not
worse than the already available drug within a specific margin, called the NI margin. The size
of this NI margin should be based on statistical and medical considerations and any difference
larger than the NI margin is an unacceptable difference and called inferior.
To study the efficacy and safety of new developed vaccines, the NI design is also applicable.
There is some guidance from regulatory offices on how to determine the NI margin but no
explicit rules on specific margins for vaccines. For this thesis we explored which margins
were used in vaccine studies and how they were determined.
A systematic search in the online literature databases PubMed and Cochrane Library was
performed, in order to get a complete overview as possible from all published articles on NI
vaccine trials. First, papers were selected based on their title, then by abstract and finally on
full-text. In total 260 full-text articles were found suitable and for this thesis 148 were
selected at random. Of the selected articles 100 met the predefined criteria. On a standardized
form general characteristics of the articles, as author and journal and information about the NI
design and margin were noted.
In NI vaccine studies it is difficult to study the number of infected as an end point, which
would normally be done in efficacy studies on infectious diseases. Therefore so called
correlates of protection are used, which is a measure for the immunity against a specific
disease. Two different correlates of protection are used often, the difference in immune
response and the geometric mean titer (GMT) ratio.
The most used NI margin for the difference in immune response was found to be 10%,
reoccurring in 71% of the studied articles. The most used GMT ratio was 0.67/1.5,
reoccurring in 60% of the studied articles. In 85% of the articles it was unclear how the
authors determined the NI margin. This is contrary to recommendations on how to report on
NI studies. Of the articles that referred to other sources, such as guidelines or previous
articles, none of their sources contained an explicit explanation on the NI margin.
We also checked whether, in our opinion, the conclusion of the authors was in line with the
results shown in the article. In 82% of the papers, the conclusion was in line with the results.
In fourteen articles not all outcomes were shown by the authors and therefore the conclusion
could not be verified, in four articles the conclusion of the authors was not in line with the
results.
Design and reporting of NI vaccine trials is a great challenge. Current guidelines of regulatory
offices contain no explicit NI margin for vaccine trials. Academic journals and their authors
should be stimulated to improve reporting on the design and results of NI vaccine studies. On
the other hand, the determination of the NI margin could be simplified by more specific
guidance from the regulatory offices.