Download Guide Materials for ASPECT-R Tool

Training for ASPECT-R
A Study Pragmatic-Explanatory
Characterization Tool – Rating
Version 1.0
CA Bossie, L Alphs
October 2014
ASPECT-R ©2014 Janssen Pharmaceuticals, Inc.
GUIDE MATERIALS for ASPECT-R Tool ©2014 Janssen Pharmaceuticals, Inc.
Users may reproduce the ASPECT-R Tool or GUIDE MATERIALS without further permission, providing they
respect copyright by quoting the names of the authors, the title, and the source in all reproduced materials.
1
Introduction
• This slide deck will serve as the training for an ASPECT-R
rater.
• The rater should allow approximately 2 hours. It will cover the
following:
1. Background relevant to the development of ASPECT-R
2. An overview of the ASPECT-R tool
3. Components of the ASPECT-R tool
4. Steps on how to use ASPECT-R
5. Example: Rating a study with ASPECT-R
2
1. Background relevant to the
development of ASPECT-R
3
1. Background relevant to the development of ASPECT-R
Background - Definitions
• Explanatory (also known as efficacy) studies:
– Examine the effects of a treatment under ideal conditions to
determine if it works and is safe
– To minimize confounds, these trials are almost always done
under highly controlled and well-defined conditions
• Pragmatic (also known as effectiveness) studies:
– Examine the effects of a treatment to determine if it works and is
safe under usual clinical or real-world conditions
4
1. Background relevant to the development of ASPECT-R
Background - Considerations
• Study conditions that may differ for explanatory and
pragmatic approaches broadly include:
– Population of interest
– Treatment setting
– Study-related interventions
– Outcome measures
• Both approaches have value and neither is intrinsically
superior to the other
• Many trials have aspects of both design approaches
5
1. Background relevant to the development of ASPECT-R
Background - Rationale
• The increasing importance and interest in real world data for
stakeholders in our evolving healthcare environment drives
the need for understanding pragmatic:explanatory
characteristics of trials
– Does the treatment work in patients treated with concomitant
medications? with comorbid conditions?
– Is the treatment safe and effective in patients with clinical rather
than formal ICD diagnoses?
• It is difficult to address all of these questions with a traditional
explanatory approach to demonstrating efficacy in highly
controlled clinical trials
6
1. Background relevant to the development of ASPECT-R
Background - Evolution
• ASPECT-R (A Study Pragmatic-Explanatory Characterization
Tool - Rating) is a tool whose development was informed by
PRECIS (A Pragmatic-Explanatory Continuum Indicator
Summary)
• PRECIS was developed to assist researchers designing trials
that are more pragmatic or explanatory (Thorpe et al, 2009)
– It consists of subjective ratings on 10 study design features along
a continuum of explanatory (efficacy) to pragmatic (effectiveness)
– Study responsible individuals independently rate the 10 domains
for comparison, discussion, and modification of study design
• The rater may wish to read the Thorpe et al, 2009 article at
this time
7
1. Background relevant to the development of ASPECT-R
Background - Evolution
• PRECIS has 10 study design domains each rated via an
unmarked Visual Analog Scale
• PRECIS results can be displayed via radar graphs
Thorpe et al. 2009
8
1. Background relevant to the development of ASPECT-R
Background - Evolution
• The Pragmascope, based on PRECIS, added a rating system
with rating considerations to PRECIS (Tosh et al, 2011). See
example below
• The rater may wish to read Tosh et al, 2011 at this time
9
1. Background relevant to the development of ASPECT-R
Background - Evolution
ASPECT-R expands and refines PRECIS / Pragmascope by:
• Re-examining the components
• Developing an anchored rating system with more detailed
definitions to aid in the reliability of the ratings
10
2. An overview of the ASPECT-R tool
11
2. An overview of the ASPECT-R tool
ASPECT-R Overview
Objective
• To aid in the development of study designs or to
aid in understanding the pragmatic:explanatory
characteristics of existing studies.
12
2. An overview of the ASPECT-R tool
ASPECT-R Overview
Six key study domains
Key Study Domains
1
2
Participant Eligibility Criteria
Intervention Flexibility
2a Experimental
2b Comparison
3
Medical Practice Setting / Practitioner Expertise
4
5
6
3a Experimental
3b Comparison
Follow-up Intensity / Duration
Primary Trial Outcomes
Participant Compliance
Informed by the original domains and concept of PRECIS (Thorpe et al. 2009) and
modified to focus only on those relevant to the pragmatic:explanatory continuum
13
2. An overview of the ASPECT-R tool
ASPECT-R Overview
Definitions of terminology
1. Participant eligibility criteria:
considers the intended treatment population of interest
identified by authors of the study (defined in title, objectives
and/or conclusions)
2. Intervention:
experimental and comparator treatment, including dose,
dosing interval, and posology; concomitant treatments
3. Medical Practice Setting/Practitioner Expertise:
includes the practitioner, treatment team, their experience,
available resources, health care delivery system, standards of
care, local cultural practices that may influence medical
delivery or outcomes
14
2. An overview of the ASPECT-R tool
ASPECT-R Overview
Definitions of terminology
4. Follow-Up Intensity & Duration:
includes frequency and length of visits and the number and
scope of assessments
5. Outcome(s):
measure(s) by which the interventions’ effects are assessed
6. Participant Compliance:
degree to which the subjects are encouraged to follow the
study related directions
15
2. An overview of the ASPECT-R tool
ASPECT-R Overview
Scoring
• Scores:
Each domain is rated from 0-6 (extremely explanatory to extremely
pragmatic); descriptive anchors are available for each rating of each
domain (shown later)
Score
Descriptor
0
Extremely explanatory
1
Very explanatory
2
3
Explanatory
Elements of both designs
4
Pragmatic
5
Very pragmatic
6
Extremely pragmatic
• Timing:
ASPECT-R takes ~30 minutes to complete for a given study; this
can vary according to the quality and clarity of the source
documents and the rater’s knowledge
16
2. An overview of the ASPECT-R tool
ASPECT-R Overview
Visualization of scores
• Ratings are graphed via a radar or “spider” diagram
17
2. An overview of the ASPECT-R tool
ASPECT-R Overview
Rater criteria
• Raters should have experience and/or knowledge of clinical
research and study designs
• While raters need not have advanced degrees, expertise is
required regarding the study’s population of interest relevant
to: epidemiology, clinical characteristics, course of illness,
and treatment regimens, modalities, and response
• Ideally, a broad understanding of treatment practices in this
therapeutic area around the world is useful
• Raters must be trained on ASPECT-R, it’s methodology, and
definitions related to its use
18
2. An overview of the ASPECT-R tool
ASPECT-R Overview
Information sources for ratings
• Published manuscript(s) describing methods, findings,
subanalyses, and conclusions
• Other related study documents, i.e., protocol, study report
• Clintrials.gov or any publically available study website
• Direct contact with investigators
• Sources used for ratings should be documented
19
3. Components of the ASPECT-R tool
20
3. Components of the ASPECT-R tool
ASPECT-R:
The tool components
• An excel file containing multiple worksheets provides information
and guidance to raters, allows for documentation of the ratings,
and graphs the ratings
• The top of each worksheet has a section to list the study being
rated, the rater’s name, and the source documents used
• The first worksheet requires the rater to define the study objective
or question, and the study’s population of interest
3. Components of the ASPECT-R tool
ASPECT-R:
The tool components
• A separate worksheet is provided for rating each domain
• Each worksheet is specific for each domain with columns for:
– “Domain Description” - defining a pragmatic vs explanatory domain
– “Rating Considerations” for rating each domain
– “Rating Anchors” for each domain
– “Rating” column for each domain
• A “Rationale” section is included to document the rationale
underlying the rating
• The embedded radar graph auto-populates to show the domain
ratings as they are entered
3. Components of the ASPECT-R tool
Using ASPECT-R
Example worksheet
Domain
Rating
Considerations
Domain
Description
Rationale for
Score
Rating
Anchors
Rating
Graph
Output
23
4. Steps on how to use ASPECT-R
24
4. Steps on how to use ASPECT-R
Using ASPECT-R
Study & rater information
Preparation
• An ASPECT-R tool (excel file with 9 worksheets) is
needed for each study to be rated
• After opening the file, enable macros
• Raters will fill in all “pink” shaded cells in each
worksheet
25
4. Steps on how to use ASPECT-R
Using ASPECT-R
Identify study, rater, study objective
Step 1 – Identify study and rater
• On each worksheet, fill in top rows with the study,
source documents, and rater name
Step 2 – Identify study objective
• Using the first worksheet, fill in the Study Question
and the Population of Interest
26
4. Steps on how to use ASPECT-R
Using ASPECT-R
Identify study, rater, study objective
Steps 3 to 7 are described next, and will be
repeated for each domain
27
4. Steps on how to use ASPECT-R
Using ASPECT-R
Prepare to rate a domain
Step 3
• Open the worksheet for corresponding domain and
read both the Domain and Domain Description
columns
28
4. Steps on how to use ASPECT-R
Using ASPECT-R
Prepare to rate a domain
Step 4
• Read the Rating Considerations columns
NOTE: this column provides important information the
rater should carefully think about when rating
29
4. Steps on how to use ASPECT-R
Using ASPECT-R
Prepare to rate a domain
Step 5
• Read all Rating Anchors
before rating a domain
30
4. Steps on how to use ASPECT-R
Using ASPECT-R
Documenting the rating rationale
Step 6
• In free-text Rationale Section, document facts or information
from the study design contributing to your rating; document
sources used
– Personal knowledge of study design can be used for ratings;
it must be documented here
– If it is known that the study was used for a regulatory
submission where a high degree of scrutiny is required, this
can be used to influence ratings and should be documented
confidential
31
4. Steps on how to use ASPECT-R
Using ASPECT-R
Rating a domain
Step 7
• Using source material for the study and information you
reviewed in ASPECT-R for a given domain, choose a rating that
best fits the design for that domain
• Enter this in the Rating Column
confidential
32
4. Steps on how to use ASPECT-R
Using ASPECT-R
The graphic
• A radar graph is generated for each study – domain
ratings are automatically populated as they are
entered in the Rating Column of the tool
33
4. Steps on how to use ASPECT-R
Using ASPECT-R
Rating a domain
• If there is insufficient information to rate a
domain, it should not be rated (=NA rating) and
noted as such in the Rationale section
34
4. Steps on how to use ASPECT-R
Using ASPECT-R
Overview of steps
1
• Identify study, source materials, and rater in each
worksheet of the ASPECT tool
2
• Identify study objective in the first worksheet of the
ASPECT tool
3
• Open a domain worksheet and read Domain &
Domain Description
4
5
• Read Rating Considerations
• Read all Rating Anchors
6
• In free-text Rationale Section, document facts or
information contributing to rating; document sources
7
• Rate the domain and insert rating in ASPECT tool
rating column
Repeat 3-7
for each
domain
35
4. Steps on how to use ASPECT-R
Using ASPECT-R:
Details on domains & rating anchors
The following slides present the information exactly
as it appears the ASPECT tool for each domain
• Domain name
• Domain descriptions
• Domain rating considerations
• Rating anchors - structured domain evaluation
confidential
36
4. Steps on how to use ASPECT-R
Participant Eligibility Criteria
Domain Description
Explanatory: Selection criteria are applied that restrict
study individuals to a defined and specific subgroup of
the population of interest.
1.
Participant Eligibility
Criteria
Pragmatic: Individuals eligible for the study are fully
representative of the population of interest.
Domain Descriptions modified from Thorpe et al, 2009
37
4. Steps on how to use ASPECT-R
Participant Eligibility Criteria
Domain Description
Explanatory: Selection criteria are applied that restrict
study individuals to a defined and specific subgroup of
the population of interest.
1.
Participant Eligibility
Criteria
Pragmatic: Individuals eligible for the study are fully
representative of the population of interest.
38
4. Steps on how to use ASPECT-R
Participant Eligibility Criteria
Domain Rating Considerations
● Consider to whom the reader is expected to generalize the result (ie, the intended
treatment population). It is usually to persons with a defined disease/syndrome or a
subpopulation of that disease/category.
● Examine information provided in the title, stated objective(s), and conclusion(s)
regarding the population of interest.
● Determine if a "convenience" sample is used and if so how well it generalizes to the
population of interest; note whether the sites selected for the study are representative
of medical practice in the region of interest.
● Consider inclusion / exclusion criteria that would limit the generalizability of results
relative to the population of interest. Consider restrictions on age, ICD diagnosis,
symptom status, comorbidities, duration of illness, illness severity, prior treatments or
hospitalizations, country, etc.
39
4. Steps on how to use ASPECT-R
Participant Eligibility Criteria
Anchors
0 = Specific, protocol-mandated selection criteria for the eligible study population or non-random
site selection restrict confident generalization of study results to 10% or less of the hypothesisdefined population of interest.
1 = Specific, protocol-mandated selection criteria for the eligible study population or non-random
site selection restrict confident generalization of study results to >10% to ≤25% of the hypothesisdefined population of interest.
2 = Specific, protocol-mandated selection criteria for the eligible study population or non-random
site selection restrict confident generalization of study results to >25% to <50% of the hypothesisdefined population of interest.
3 = Specific, protocol-mandated selection criteria for the eligible study population, or non-random
site selection, restrict confident generalization to approximately 50% of the hypothesis-defined
population of interest.
4 = Specific, protocol-mandated selection criteria for the eligible study population and site
selection, allow confident generalization of study results to >50% to <75% of the hypothesis-defined
population of interest.
5 = Specific, protocol-mandated selection criteria for the eligible study population and site
selection, allow confident generalization of study results to 75% to <90% of the hypothesis-defined
population of interest.
6 = Specific, protocol-mandated selection criteria for the eligible study population and site
selection, allow confident generalization of study results to 90% or more of the hypothesis-defined
population of interest.
4. Steps on how to use ASPECT-R
Intervention Flexibility
Domain Description
2.
Intervention Flexibility
2a. Experimental
2b. Comparator
Explanatory:
Inflexible experimental (2a) or comparator (2b)
intervention with strict instructions for every aspect of
use.
Pragmatic:
Instructions on how to apply the experimental (2a) or
comparator (2b) intervention are highly flexible, so as to
reflect real world use and practice.
Domain Descriptions modified from Thorpe et al, 2009
41
4. Steps on how to use ASPECT-R
Intervention Flexibility Domain
Rating Considerations
• Consider whether the study protocol assigns, mandates, or restricts the experimental
(2a) or comparison (2b) intervention and aspects of its use beyond that of real world
practice.
• For a pharmacological intervention, consider instructions on: dose, when/how to
increase or decrease dose, specific time of dosing, administration with or without food,
interval between dosings, site for injection, or how to handle missed doses.
• Consider whether the comparator is placebo instead of the best alternative active
comparator (2a only).
• A chart review study would generally impose no restrictions on the intervention and, as
such, reflects real world practice.
• NOTE: Do NOT consider frequency and timing of study assessments here. These are
considered in Domain 4: “Follow-up intensity/Duration.”
42
4. Steps on how to use ASPECT-R
Intervention Flexibility
Anchors
0 = Use of the experimental (2a) or comparison (2b) intervention is carefully and precisely
defined for nearly all aspects of its use in the study.
1 = Use of the experimental (2a) or comparison (2b) intervention is carefully and precisely
defined for most aspects of its use in the study.
2 = Use of the experimental (2a) or comparison (2b) intervention is defined such that many
aspects of its use are constrained by the study protocol. Use is somewhat more constrained than
that outlined in existing (or anticipated) product label.
3 = Use of the experimental (2a) or comparison (2b) intervention is defined such that restrictions
are limited to the constraints of the existing (or anticipated) product label, or to constraints of
another standardized and broadly used definition for the use of the intervention.
4 = Use of the experimental (2a) or comparison (2b) intervention is defined such that it allows for
limited use beyond the constraints of the existing (or anticipated) product label or beyond that of
another standardized definition of the intervention.
5 = Use of the experimental (2a) or comparison (2b) intervention is defined such that it allows for
considerable use beyond the constraints of the existing (or anticipated) product label or another
standardized definition of the intervention. Minimal constraints remain.
6 = Use of the experimental (2a) or comparison (2b) intervention is limited only by considerations
of what the practitioner personally considers to be ethical and good clinical practice.
43
4. Steps on how to use ASPECT-R
Medical Practice Setting /
Practitioner Expertise
Domain Description
Explanatory: The experimental (3a) or comparator (3b)
intervention is applied only by seasoned practitioners in the
3.
Medical Practice Setting field of interest, and in practice settings where the care
/ Practitioner Expertise delivery system and providers are highly experienced in
managing the types of patients enrolled in the trial.
3a. Experimental
3b. Comparator
Pragmatic: The full range of practitioners in the full range
of clinical settings are eligible to participate in the trial.
Domain Descriptions modified from Thorpe et al, 2009
44
4. Steps on how to use ASPECT-R
Medical Practice Setting /
Practitioner Expertise
Domain Description
Explanatory: The experimental (3a) or comparator (3b)
intervention is applied only by seasoned practitioners in the
3.
Medical Practice Setting field of interest, and in practice settings where the care
/ Practitioner Expertise delivery system and providers are highly experienced in
managing the types of patients enrolled in the trial.
3a. Experimental
3b. Comparator
Pragmatic: The full range of practitioners in the full range
of clinical settings are eligible to participate in the trial.
45
4. Steps on how to use ASPECT-R
Medical Practice Setting /
Practitioner Expertise
Domain Description
• Consider the range of practitioners relative to their: skill sets, training
regarding use of the intervention, specialized training for ratings or other
research procedures, requirements for special certifications, and experience
or specialized skills with the intervention or research activities.
• Consider the standard of medical care provided by study practitioners and
the expertise of staff at the site/setting where the study has been conducted
relative to that available to all patients included in the population of interest.
46
Medical Practice Setting /
Practitioner Expertise
Anchors
0 = Practitioners in the study are limited to those with very well defined or implied skill sets that are held by very few
practitioners in the real world who might ever use the experimental (3a) or comparator (3b) intervention. Extensive studyspecific training or expertise is required to qualify as an investigator for the trial.
1 = Practitioners in the study are limited to those with very well defined or implied skill sets that are held by a small
number of practitioners in the real world who might ever use the experimental (3a) or comparator (3b) intervention.
Substantial specialized study-specific training or expertise is required to qualify as an investigator for the trial.
2 = Practitioners in the study are limited to those with very well defined or implied skill sets that are held by a meaningful
proportion (but less than half) of practitioners in the real world who might ever use the experimental (3a) or comparator
(3b) intervention. Some study-specific training or expertise is required to qualify as an investigator for the trial.
3 = Practitioners in the study are limited to those with skill sets that are held by about half of all practitioners in the real
world who might ever use the experimental (3a) or comparator (3b) intervention, but not by a sizeable subgroup. Some
study-specific training or expertise is required to qualify as an investigator for the trial.
4 = Practitioners in the study are those with skill sets that are held by a substantial proportion (more than half) of
practitioners in the real world who might ever use the experimental (3a) or comparator (3b) intervention. Little to no
study-specific training or expertise is required to qualify as an investigator for the trial.
5 = Practitioners included in the study are those with skill sets that are held by most practitioners in the real world who
might ever use the experimental (3a) or comparator (3b) intervention. Little to no study-specific training or expertise is
required to qualify as an investigator for the trial.
6 Practitioners included in the study are those with skill sets that are held by all or nearly all practitioners in the real world
who might ever possibly use the experimental (3a) or comparator (3b) intervention. No study-specific training or expertise
is required to qualify as an investigator for the trial.
4. Steps on how to use ASPECT-R
Follow-up Intensity / Duration
Domain Description
4.
Follow-up Intensity /
Duration
Explanatory: Study participants are followed with more
frequent visits and more extensive data collection than
would occur in clinical practice, regardless of their clinical
need.
Pragmatic: Follow-up is limited to usual clinical practice.
Domain Description modified from Thorpe et al, 2009
48
4. Steps on how to use ASPECT-R
Follow-up Intensity / Duration
Domain Rating Considerations
• Consider the protocol-defined visit frequency, structure, time commitment
and/or intensity of contact.
● Consider the number of interventions and assessments required to
complete the evaluation and the time necessary to complete them(ie,
specialized tests, rating scales, additional therapies, etc.) relative to what is
available in standard practice.
● Consider the invasiveness of the intervention relative to that used in
standard practice, ie, requirements for pharmacokinetic sampling, magnetic
resonance imaging, hospitalization.
49
4. Steps on how to use ASPECT-R
Follow-up Intensity / Duration
Anchors
0 = Visit frequency/structure/duration/intensity, as well as the duration of the study, are very
explicitly and precisely defined by the study protocol. These visit parameters are much greater than
those which would be seen in normal clinical practice.
1 = Visit frequency/structure/duration/intensity, as well as the duration of the study, are explicitly
defined and go considerably beyond those outlined in available treatment guidelines or standard
practice.
2 = Visit frequency/structure/duration/intensity, as well as the duration of the study, are defined
such that they are more constrained than those outlined in available treatment guidelines or
standard practice.
3 = Some elements of the visit frequency/structure/duration/intensity or the duration of the study,
but not all, are more constrained than those outlined in available treatment guidelines or standard
practice.
4 = Visit frequency/structure/duration/intensity, as well as the duration of the study, are defined
such that they conform to available treatment guidelines or standard practice.
5 = Visit frequency/structure/duration/intensity, as well as the duration of the study, are defined
such that flexibility is allowed beyond that found in available treatment guidelines or standard
practice.
6 = No constraints are put on visit frequency/structure/duration/intensity or follow-up period.
50
4. Steps on how to use ASPECT-R
Primary Trial Outcome(s)
Domain Description
Explanatory: The outcome is known to be a direct and
immediate consequence of the intervention. Its
relationship to an important clinical outcome has not been
established. The outcome may require specialized training
or testing not normally used in general clinical practice.
5.
Primary Trial Outcomes Pragmatic: The primary outcome is an objectively
measured, clinically meaningful outcome to the study
participants. It does not rely on central adjudication, is one
that can be assessed under usual conditions, and does not
require special training or tests beyond those used in
general clinical practice.
Domain Description modified from Thorpe et al, 2009
51
4. Steps on how to use ASPECT-R
Primary Trial Outcome(s)
Domain Rating Considerations
● Consider that more explanatory outcome measures may require specialized training to complete an adequate
assessment. It may be necessary to complete such assessments in specialized settings with specialized
instrumentation or personnel who have unique skill sets (eg, ability to assess persons on specialized symptom scales).
Information generated from them may be important for the driving question of the clinical trial, but has limited value
as an outcome in general clinical practice (eg, measurement of brain cortical volume in a depression trial).
● Consider that more pragmatic outcomes include measures that are clinically obvious and generally accepted as
clinically important, such as hospitalization or death.
● Consider all trial outcomes that support the primary study question, giving the most weight to those predefined as
the study primary endpoint.
● Scales used to assess outcomes should be evaluated with regard to how informative they are to clinical practice.
Surrogate markers of clinical response, like a triglyceride level, would be considered more explanatory. Measures of
death or all cause hospitalization would be considered more pragmatic. A total score on a scale measuring psychosis
would be intermediate.
● Consider how much the outcomes are influenced by the treatment settings or systems of care used in the study and
whether they can be easily generalized to customary clinical practice.
52
4. Steps on how to use ASPECT-R
Primary Trial Outcomes
Anchors
0 = In the aggregate, trial outcome measures support an overall result for which no direct link to
a clinically important outcome has been established.
1 = In the aggregate, trial outcome measures support an overall result for which only a very
limited link to a clinically important outcome has been established.
2 = In the aggregate, trial outcome measures support an overall result for which only a limited
link to a clinically important outcome has been established.
3 = In the aggregate, trial outcome measures support an overall result that represents an
established outcome, but one which requires extrapolation regarding its clinical importance.
4 = In the aggregate, trial outcome measures support an overall result that represents a wellestablished clinical outcome requiring moderate interpretation regarding its clinical importance.
5 = In the aggregate, trial outcome measures support an overall result that represents a wellestablished clinical outcome requiring limited interpretation regarding its clinical importance.
6 = In the aggregate, trial outcome measures support an overall result that represents a wellestablished and important clinical outcome requiring no complex interpretation.
53
4. Steps on how to use ASPECT-R
Participant Compliance
Domain Description
6.
Participant Compliance
Explanatory: Study participants compliance with the
intervention is monitored closely, may be a pre-requisite
for study entry, and both prophylactic and rescue strategies
are used.
Pragmatic: There is unobtrusive or no measurement of
compliance, and no strategies for rescue beyond normal
clinical practice.
Domain Description modified from Thorpe et al, 2009
54
4. Steps on how to use ASPECT-R
Participant Compliance
Domain Descriptor
6.
Participant Compliance
Explanatory: Study participants compliance with the
intervention is monitored closely, may be a pre-requisite
for study entry, and both prophylactic and rescue strategies
are used.
Pragmatic: There is unobtrusive or no measurement of
compliance, and no strategies for rescue beyond normal
clinical practice.
55
4. Steps on how to use ASPECT-R
Participant Compliance
Domain Rating Considerations
● Consider study protocol directives that assure participant compliance with
the intervention.
● Protocol directives to consider include: protocol-defined observation of the
compliance with the intervention, use of blood drug levels, patient report via
interview, patient diary, caregiver report via interview, caregiver support, pill
counts, documentation, or if no relevant information is gathered.
● Consider that retention efforts will likely have an effect on compliance.
● Consider protocol mandates disallowing or requiring co-therapeutic
interventions in each arm.
● Consider that requirements for informed consent can correspond to a
protocol directive that may impact compliance (rating of 4 or lower).
56
4. Steps on how to use ASPECT-R
Participant Compliance
Anchors
0 = Protocol directives exist such that participant compliance with the intervention is required to
continue in the study. This is confirmed and well documented. Patients are excluded for noncompliance.
1 = Protocol directives exist such that participant compliance with the intervention is strongly
encouraged and documented, but not required in order to continue in the study.
2 = Protocol directives exist such that participant compliance with the intervention is strongly
encouraged and documented.
3 = Protocol directives exist such that participant compliance with the intervention is strongly
encouraged; documentation of compliance is not required.
4 = Some protocol directives exist that may impact participant compliance with the intervention;
documentation of compliance is not required.
5 = Few protocol directives exist that may impact participant compliance with the intervention;
documentation of compliance is not required.
6 = No protocol directives exist to assure or document participant compliance with the
intervention.
57
5. Example: Rating a study with
ASPECT-R
58
5. Example: Rating a study with ASPECT-R
Example: Rating the CATIE Study
• Raters have two CATIE references available as the
source material
• Reminder: raters complete the “pink” shaded cells
in each worksheet.
Step 1 – Identify study and rater
Study:
Effectiveness of antipsychotic drugs in patients with chronic schizophrenia (CATIE) Phase 1
Rater:
Source documents (manuscript, Stroup et al. Schizo Bull 2003;29:15-31;
Alphs, Bossie protocol, etc):
Lieberman et al, NEJM 2005;353:1209-22.
59
5. Example: Rating a study with ASPECT-R
Example: Rating the CATIE Study
Study Objective
Step 2 – Identify study objective on the first worksheet
Steps 3 to 7 as described earlier (slide 35) are followed to
rate each domain. The subsequent slides provide the rationale
used to reach the rating for each domain for the CATIE study.
60
5. Example: Rating a study with ASPECT-R
Example: Rating the CATIE Study
1 Participant Eligibility Criteria
Rationale
There were 4 inclusion criteria and 13 exclusion criteria per Stroup et al, 2003 in Tables 1-2. Two of
these criteria identify the population of interest by diagnosis and by the exclusion of first episode
patients. Each of the other criteria impose a limitation on the selection of the population of
interest (relative to "all comers"). In most cases, this limitation is likely minor (ie, the age range
inclusion of 18-65). Of note, many of the selection criteria imposed are based on safety concerns,
GCP, or other ethical factors relevant to running a clinical study. While they are appropriate and
necessary for running a clinical study, it should be recognized that they each impose some
restriction on the selection of the population of interest, and thereby reduce the generalizability. In
total, the limitations imposed by these 15 criteria are estimated to allow confident generalization
to >50% to >75% of the population of interest, resulting in an ASPECT-R score of 4.
Graph auto-populates
when rating entered
61
5. Example: Rating a study with ASPECT-R
Example: Rating the CATIE Study
2a Intervention Flexibility - Experimental
Rationale:
Initiation doses were directed to be per label. The subsequent dosing was flexible based on
clinician judgment and consistent with label. However, some of the assigned drugs are
directed by label to be taken twice daily Therefore, for those drugs that can be taken either
once daily or twice daily, there was randomization to once or twice daily dosing to help
maintain the blind. This was judged to not be more or less restrained than the label and
assigned as ASPECT-R score of 3.
62
5. Example: Rating a study with ASPECT-R
Example: Rating the CATIE Study
2b Intervention Flexibility - Comparison
Rationale:
See intervention Flexibility for the Experimental Arm.
Assigned ASPECT-R score of 3.
Graph auto-populates
when rating entered; the
tool averages the scores
for domains 2a & 2b
63
5. Example: Rating a study with ASPECT-R
Example: Rating the CATIE Study
3a Medical Practice Setting - Experimental
Rationale:
Practitioners had to be trained on several scales including the Positive and Negative
Syndrome Scale, Clinical Global Impressions-Severity scale, Calgary depression scale, a
neurocognitive battery of tests, extrapyramidal symptom rating scales (Simpson Angus
Rating Scale, Barnes Akathisia Scale, Abnormal Involuntary Movements Scale), Short-Form12 quality of life scale, Insight and Treatment Attitudes Questionnaire, Drug Attitude
Inventory, Service Use and Resource Form. Thus, the setting/practitioner expertise was
judged to require very well defined skill sets held by very few practitioners in the real
world who might use the intervention, with extensive study-specific training or expertise
required. This domain was assigned as ASPECT-R score of 0.
64
5. Example: Rating a study with ASPECT-R
Example: Rating the CATIE Study
3b Medical Practice Setting - Experimental
Rationale:
See Medical Practice Setting for the Experimental Arm.
Assigned ASPECT-R score of 0.
Graph auto-populates
when rating entered; the
tool averages the scores
for domains 3a & 3b
65
5. Example: Rating a study with ASPECT-R
Example: Rating the CATIE Study
4 Follow-up Intensity / Duration
Rationale
The study's 18-month duration involved assessments at months 1, 3, 6, 9, 12, 15, and 18. The
visit frequency was judged to reflect what one might encounter in standard clinical practice.
At these visits, patients were assessed on a number of scales and measures as defined by the
schedule of assessments in the protocol (including the Positive and Negative Syndrome Scale,
Clinical Global Impressions-Severity scale, Calgary depression scale, a neurocognitive battery
of tests, extrapyramidal symptom rating scales [Simpson Angus Rating Scale, Barnes Akathisia
Scale, Abnormal Involuntary Movements Scale], Short-Form-12 quality of life scale, Insight
and Treatment Attitudes Questionnaire, Drug Attitude Inventory, Service Use and Resource
Form, adverse event reports, weight, ECGs, labs. Taken together, the follow up
intensity/duration was judged to be much greater than standard clinical practice and assigned
a ASPECT-R score of 0.
Graph auto-populates
when rating entered
66
5. Example: Rating a study with ASPECT-R
Example: Rating the CATIE Study
5 Primary Trial Outcome
Rationale
The primary outcome was discontinuation of treatment for any reason, which is an
important clinical outcome requiring no complex interpretation. The key secondary
outcomes include reasons for discontinuation, Clinical Global Impressions-Severity, and
Positive and Negative Syndrome Scale; the first two requires no complex interpretation, but
latter does require interpretation regarding its clinical relevance. Taken together, it was
judged that this represents an outcome requiring limited interpretation regarding its clinical
importance, and was assigned an ASPECT-R rating of 5.
Graph auto-populates
when rating entered
67
5. Example: Rating a study with ASPECT-R
Example: Rating the CATIE Study
6 Participant Compliance
Rationale
As described in Stroup et al, an adherence enhancement therapy intervention (adapted
from Kemp et al. 1996) was employed for CATIE to guide and enhance patient and family
education on medication adherence. A 3-step cognitive behaviorally-oriented approach was
used. This was judged to be a protocol directive to strongly encourage compliance (there is
no indication of documentation of compliance) and assigned an ASPECT-R score of 3.
Graph auto-populates
when rating entered
68
Limitations & Considerations
• Inter-rater reliability (IRR) assessment currently ongoing.
• Rater expertise required, as noted on rater slide.
• Study design or other relevant information may be poorly
documented or unavailable. Ratings may sometimes reflect
information available vs actual study design.
• Some domains may be somewhat overlapping, sharing nonunique contributions.
• Does not consider the quality of the study conduct, design, or
interpretation relative to objective.
69
Questions?
Cyndi Bossie: [email protected]
Larry Alphs: [email protected]
ASPECT-R ©2014 Janssen Pharmaceuticals, Inc.
GUIDE MATERIALS for ASPECT-R Tool ©2014 Janssen Pharmaceuticals, Inc.
70
References
• Thorpe KE, Zwarenstein M, Oxman AD, et al. A pragmaticexplanatory continuum indicator summary (PRECIS): a tool to
help trial designers. J Clin Epidemiol 2009; 62:464-475.
• Tosh G, Soares-Weiser K, Adams CE. Pragmatic vs explanatory
trials: the Pragmascope tool to help measure differences in
protocols of mental health randomized controlled trials. Dialogues
Clin Neurosci 2011;13:209-215.
• Lieberman JA, Stroup TS, McEvoy JP. Effectiveness of
antipsychotic drugs in patients with chronic schizophrenia. N Engl
J Med 2005;353:1209-1223.
• Stroup TS, McEvoy JP, Swartz MS, et al. The National Institute of
Mental Health Clinical Antipsychotic Trials of Intervention
Effectiveness (CATIE) Project: Schizophrenia trial design and
protocol development. Schizophrenia Bulletin 2003:29(1):15-31.
ASPECT-R ©2014 Janssen Pharmaceuticals, Inc.
GUIDE MATERIALS for ASPECT-R Tool ©2014 Janssen Pharmaceuticals, Inc.
71