Download substantial evidence - Regulatory Affairs Professionals Society

SUBSTANTIAL EVIDENCE:
Substantially Black
and White
By Moulakshi Roychowdhury, PharmD, JD
18
August 2011
Those of us who work in the regulatory advertising and promotions field often describe the
regulations under which we operate and the
US Food and Drug Administration (FDA) interpretation of those regulations as “gray.” We
continuously scour FDA enforcement websites
hoping to gain an inkling of what FDA’s current
thinking is on a wide range of advertising and
promotion topics such as risk presentation and
social media. We attend conferences and hang on
each regulator’s every word looking to be enlightened and move from gray to black and white.
One exception to this gray zone is the
“substantial evidence” standard. A thorough
evaluation of enforcement trends indicates
FDA has been unwavering in its application
of this standard to both drug approvals and
drug promotion. Of the 14 letters issued by the
Division of Drug Marketing, Advertising, and
Communications (DDMAC) in the first five
months of 2011, 10 (71%) cited the need for substantial evidence when purporting promotional
product claims.
Similarly, of the 52 letters issued by DDMAC
in 2010, 42 (81%) cited lack of substantial evidence. In each of these letters, DDMAC has
qualified these citations with statements such as:
”Promotional materials are misleading if they
represent or suggest that a drug is more effective than has been demonstrated by substantial
evidence.”1,2
By sending these letters, DDMAC sent a
loud and clear message it sees no flexibility in
promotion with regard to the requirement for
substantial evidence. But what does substantial
evidence mean and how it is applied?
Applicable Regulations
With new drug applications (NDA), the Federal
Food, Drug, and Cosmetic Act, 21 U.S.C. §355(d)
defines “substantial evidence” as:
“Evidence consisting of adequate and wellcontrolled investigations, including clinical
investigations, by experts qualified by scientific
training and experience to evaluate the effectiveness of the drug involved...“
Using this definition, FDA extends the substantial evidence standard specifically to advertising
in 21 Code of Federal Regulations, 21 (CFR) 202.1
(e)(3)(ii)(b) as follows:
“For which there exists substantial evidence of
safety and effectiveness, consisting of adequate
and well-controlled investigations… on the basis
of which it can fairly and responsibly be concluded by such experts that the drug is safe and
effective for such uses.”
This is further elaborated in 21 CFR 314.126,
which provides a comprehensive list of criteria
for adequate and well-controlled investigations
for a New Drug Application (NDA).
Accordingly, when determining whether
a study will provide substantial evidence for
either drug approval or promotion, the following
criteria must be satisfied by the study and study
design:
• The study must provide a clear statement of objectives.
• A valid control group must exist
to demonstrate a drug effect independent of other influences (e.g.,
placebo-controlled).
• The study must select a patient population with a specific condition or which
is at-risk of disease.
• The baseline should be comparable
(generally accomplished through
randomization).
• Steps must be taken to minimize bias
(e.g., blinding).
• There must be sufficient detail to allow
for appropriate assessment and analysis of whether the data truly constitute
evidence.
Substantial evidence requires at least two
adequate and well-controlled studies, each
convincing on its own, to establish effectiveness. Occasionally, data from one adequate and
well-controlled investigation with additional
confirmatory evidence may be sufficient if FDA
determines that this establishes effectiveness.3
As per FDA guidance, “Reliance on only
a single study will generally be limited to
situations in which a trial has demonstrated a
clinically meaningful effect on mortality, irreversible morbidity, or prevention of a disease with
potentially serious outcome and confirmation of
the result in a second trial would be practically
or ethically impossible.” A claim supported by
two strong studies will always be considered
more favorably and be more likely to constitute
substantial evidence.4
Substantial Evidence Pitfalls
While the above regulations are very clear, if we
do not understand their application, the question
remains whether DDMAC’s application of the
“substantial evidence” standard in promotion is
just as black and white. An analysis of DDMAC’s
enforcement letters over the years, and particularly in the last few years, suggests that DDMAC
consistently and rigidly applies this standard.
A review of the enforcement letters identified a number of areas that were repeatedly
deemed to not constitute substantial evidence:
post-hoc subgroup analyses, meta-analyses,
open-label studies, secondary endpoints and
inconsistency with pre-specified statistical analysis plan (SAP) and endpoints.
Regulatory Focus
19
Post-hoc Subgroup Analyses
Currently, use of post-hoc analyses to make
claims appears to be a red-flag for DDMAC.
Post-hoc analyses generally do not have clearly
defined endpoints with pre-specified statistical
analysis plans (SAP), and are observational in
nature. Consideration of countless characteristics
or factors can be considered, causing significant
tests to become difficult to interpret. This study
design allows for confounding variables and bias
and is generally unreliable. 5
These issues were highlighted in a DDMAC
enforcement letter issued to a company for a
professional journal advertisement. While the
studies cited were “randomized, double-blind,
and prospective” studies, the claims stemmed
from the resulting data being retrospectively
analyzed. Specifically, the claim came from a
post-hoc subgroup analysis.
DDMAC stated, “P-values are meaningless
in this situation and cannot be used to demonstrate statistically significant differences between
treatment groups … When looking for differences between treatment groups, the study must
be designed to look for these differences prospectively, and must be sufficiently powered.”6
Meta-analyses
A perception of bias has led to a general distrust of claims stemming from meta-analyses
conducted by pharmaceutical companies.
Meta-analyses combine multiple studies and
summarize the results. They also increase the
sample size and the power to study effects of
interest. The caveats include selection bias, as
unfavorable studies can potentially be omitted
and patient populations may vary vastly from
study to study. Essentially, meta-analyses are
not well-controlled studies and have on occasion produced results inconsistent with those of
large, well-controlled trials.7 DDMAC is mindful of these caveats and has shared these same
concerns.8
For example, when a certain company used
a meta-analysis of 43 double-blind studies to
claim efficacy in a sales aid, DDAMC stated in
an enforcement letter that the claim was not
supported by substantial evidence. In this letter, DDMAC reiterated, “A claim of superiority
should be based on a comparison of the maximum dose of two products in two adequate,
well-designed, head-to-head clinical trials.”9
In addition, meta-analyses of “across label”
comparisons are not considered to be substantial
evidence. DDMAC issued an enforcement letter
to a company that utilized a meta-analysis of
18 different product labels comparing relative
abuse liability to support a claim of superiority.
This was a real problem, as the FDA-approved
indications for the products being compared
were not the same. DDMAC noted that a disclaimer was included in the piece clarifying that
the claim stemmed from a meta-analysis and
20
August 2011
not a head-to-head clinical trial, but said that
the disclaimer did not “mitigate the misleading
implication.”10
Open-label Studies
Open-label studies are those in which both the
patients and investigators are aware of which
treatment is being administered, and they are
especially problematic when the endpoints being
evaluated are based on subjective patient or
investigator responses, as this allows for biases.
This concern about subjective responses
was underscored in a DDMAC enforcement
letter issued for a journal advertisement which
touted a claim supported by a “randomized,
open-label, multi-center study of 3097 subjects
with two treatment arms.” Although the study
was randomized, it still allowed for bias because
the endpoint being evaluated (depression) was
subjective. DDMAC further opined, “Blinding is
intended to minimize the potential biases resulting from differences in management, treatment,
or assessment of patients, or interpretation of
results that could arise as a result of subject or
investigator knowledge of the assigned treatment
… while nominally statistically significant in this
very large study (over 3,000 patients), cannot be
relied upon as substantial evidence.”11
Similarly, another company was cited for
a visual aid that made efficacy claims based on
the results of an open-label, uncontrolled clinical
study. DDMAC explained, “Results from a single
open-label clinical trial with no control group do
not constitute substantial evidence… to support
this, or any other, efficacy claim.”12
Secondary Endpoints
Secondary endpoints are tricky, as they are often
dependent on the primary endpoints. Generally,
if a study misses its primary endpoint, this
renders the secondary endpoint useless. Also,
if a study’s primary endpoint is met, then the
secondary endpoints must be pre-specified and
statistically designed to meet significance. Elaine
Hu, of DDMAC, clarified, “Generally, only primary and key secondary endpoints can support
claims; primary and key secondary endpoints are
endpoints that have a pre-specified analysis plan
with explicit allocation of alpha (the possibility of
making a false assumption).”13
DDMAC issued an enforcement letter to a
company for a professional reprint carrier that
claimed superiority to another drug based on
a secondary endpoint of an open-label, randomized study because the study “failed to
demonstrate statistical significance on the primary endpoint and has not been replicated.”
DDMAC explained that, “In the absence of a
finding of statistical significance for the primary
endpoint, any further analysis conducted on
this study, including secondary endpoints and
subpopulations, are all considered exploratory
and do not generally constitute substantial
evidence.”14
Similarly, DDMAC issued another letter for
a product website that included efficacy claims
based on a secondary endpoint. However, the
study cited only achieved significance on one of
the two primary endpoints, rendering the results
of the secondary endpoint invalid. DDMAC
explained that “Generally, claims based on secondary endpoints are problematic when, as is the
case here, the study fails to meet statistical significance on all primary endpoints … in the absence
of a statistically significant improvement in
overall response rate AND time to (primary endpoint), it is misleading to make claims based on
secondary endpoints from the clinical study.”15
Inconsistency with Pre-specified
Statistical Analysis Plan (SAP) and
Endpoints
Essentially, through enforcement, DDMAC has
made it clear that it is not sufficient that a claim
stems from the results of a well-controlled study,
but that the claim being made must be consistent
with what the study is designed to measure. For
instance, if a study is prospectively and statistically designed to measure a difference between
placebo and treatment at four weeks, then claiming a benefit at two weeks is inappropriate.
This was illustrated in a DDMAC enforcement letter issued for a direct-to-consumer (DTC)
patient brochure that claimed effects of a drug
on skin at specific time points (specifically, two
weeks and one month). However, the primary
efficacy endpoint was designed to measure a difference at week 10. The other time points were
not pre-specified and were analyzed retrospectively. That made the shorter time points invalid,
and they were considered to not constitute substantial evidence.16
Very recently, DDMAC issued an enforcement letter to a company for a detail aid that
contained a number of comparative claims that
suggested overall superiority over time. Each of
these claims stemmed from two studies. While
they were well-controlled, the studies focused on
one specific time point of comparison. However,
the products were each respectively approved
for symptom relief over an extended period of
time, and not solely one time period. Therefore,
superiority over one time point would not constitute substantial evidence to support a claim of
overall superiority. DDMAC ruled that the piece
was misleading as it suggested that the drug
was “more effective than another drug, when
this has not been demonstrated by substantial
evidence.”17
Real-world Navigation of the
Substantial Evidence Standard:
Advice for Industry
Given that the application of the substantial
evidence standard may be black and white, but
with a number of pitfalls, ensuring that a claim
is supported by substantial evidence may require
a step-by-step approach. Provided below are
suggested questions to ask when navigating this
analysis:
1. Does the proposed claim stem from the
data provided in the FDA-approved
label? If yes, there may be substantial
evidence.
2. If not, is the proposed claim consistent
with the indication and related clinical
outcomes? If no, end analysis here. If
yes, continue on to question 3.
3. If yes, is there substantial evidence to
support the claim (generally, exclude
post-hoc subgroup analyses, metaanalyses, open-label studies and
secondary endpoints with failed primary endpoints)?
4. If making a comparative claim, are there
two adequate and well-controlled studies supporting the claim?
5. If making a single product claim, are
there two adequate and well-controlled
studies supporting the claim? If there is
only one study, is there strong confirmatory evidence?
22
August 2011
Those of us who work on promotional review
committees soon learn that people who do not
fervently comb through DDMAC enforcement
letters on a daily basis are not always aware of
the paramount need for substantial evidence to
support promotional claims. Without this knowledge, frequently, by the time pieces are routed
for promotional review, a great deal of expense,
time and energy has already been invested in
developing claims that may not be appropriate
for promotion. To avoid wasting these resources,
it may be helpful to train those involved in the
creation of these pieces ahead of time so they are
aware of how to avoid pitfalls.
Furthermore, which professionals are typically the most well-versed in study designs and
the clinical validity of study results? The prime
candidates are those who specialize in medical
affairs. To help companies avoid enforcement
due to lack of substantial evidence, medical
colleagues should be actively engaged in the
review of promotional materials. In evaluating
for substantial evidence, medical may be able
to determine whether endpoints measured are
well-defined and reliable, whether the data are
meaningful, and whether they can be interpreted
without bias to reach a specific claim, to name
a few. In essence, medical may be very valuable
in the critical evaluation and determination of
whether the characteristics of an adequate and
well-controlled study are present as provided
in 21 CFR 314.126. Further, these experts should
also review DDMAC enforcement letters to be
aware of the distinction between the “substantial
evidence” standard as applied by DDMAC versus what may be appropriate in clinical practice.
Lastly, it may be beneficial for regulatory
professionals responsible for promotional review
and marketing to be involved with regulatory
affairs and development during the early stages
of designing studies to provide guidance on
study designs and set realistic expectations with
commercial organizations. In addition, regulatory personnel should have clear discussions
with commercial designers early on. They should
talk about what information is important to communicate to physicians and/or consumers in a
particular disease state and ensure that studies
(if possible) are designed adequately to measure
these endpoints, and that there exists a pre-specified SAP. In addition, DDMAC suggests that
sponsors discuss these desired claims with the
appropriate review division to determine how to
best plan for the interpretation of study findings
to avoid conducting studies that are not able to
support the sponsors’ desired claims.18
Stay Within the Black and White
Lines
In summary, the substantial evidence standard
is one area where DDMAC is unwavering.
The countless DDMAC enforcement letters
underscore the necessity for substantial evidence
when making promotional claims, and it is our
responsibility as regulatory professionals to communicate this message and educate our industry
colleagues.
References
1.
Patel S. DDMAC Untitled Letter to Dow Pharmaceutical
Sciences Inc. for Acanya. Jun 16 2011.
2.
Panholzer L. DDMAC Untitled Letter to Allergan Inc. for
Acuvail. 25 August 2010.
3.
Hu Cunningham E. Levels of Evidence to Support
Promotional Claims for Prescription Drugs. Presentation
at Drug Information Association Meeting. 23 February
2011.
4.
Guidance for Industry: Providing Clinical Evidence of
Effectiveness for Human Drug and Biological Products. May
1998.
5.
Sleight P. Debate: Subgroup analyses in clinical trials: fun
to look at- but don’t believe them! Current Control Trials in
Cardiovascular Medicine. 2000 1(1): 25-27
6.
Abrams T. DDMAC Warning Letter to Pfizer Inc. for
Zyvox. 20 July 2005.
7.
Walker E, et al. “Meta-Analysis: Its Strengths and
Limitations.” Cleveland Clinical Journal of Medicine. 75(6).
431-439. Jun 2008.
8.
Op cit 30.
9.
Abrams T. DDMAC Warning Letter to Sankyo Pharma
for Benicar. 6 January 2006.
10. Patel S, et al. DDMAC Untitled Letter to Takeda
Pharmaceuticals North America Inc. for Rozerem. 28
January 2010.
11. Abrams T. DDMAC Warning Letter to Wyeth
Pharmaceuticals Inc. for Effexor XR. 10 December 2007.
12. Abrams T. DDMAC Warning Letter to Galderma
Laboratories for Tri-Luma Cream. 18 August 2009.
13. Op cit 30.
14. Olin K. DDMAC Untitled letter to Sanofi Aventis for
Taxotere. 16 April 2009.
15. Abrams T. DDMAC Warning Letter to Eisai Inc. for
Dacogen. 6 November 2009.
16. Patel S. DDMAC Untitled Letter to Sanofi Aventis for
BenzaClin. 22 January 2010.
17. Op cit 1.
18. Op cit 30.
Author
Moulakshi (Mou) Roychowdhury, PharmD, JD, currently
works at Biogen Idec as Associate Director, Regulatory Affairs
- Advertising, Labeling, and Promotion. She is a graduate of
the Ernest Mario School of Pharmacy, Rutgers University and
Brooklyn Law School. Roychowdhury has worked in similar
capacities at Bayer HealthCare Pharmaceuticals and Genzyme
Corporation advising the companies on promotional compliance with FDA regulations.
Regulatory Focus
23