Download Medication Management Pilot Study

 Report to The Vermont Legislature Medication Management Pilot Study
In Accordance with Act 63 (2011), Sec. E.311,
An Act Relating to Making Appropriations
for the Support of Government
Submitted to:
House Committee on Health Care
Senate Committee on Health and Welfare
Submitted by:
Harry Chen, MD
Commissioner
Prepared by:
University of Vermont Area Health Education Center (AHEC)
Report Date:
January 15, 2014
108 Cherry Street
PO Box 70
Burlington, VT 05402
healthvermont.gov
MEDICATION MANAGEMENT PILOT STUDY
Grant #03420-5852S
Legislative Report to the Vermont Department of Health (VDH)
January 2012 – December 2013
Submitted to:
Debra Wilcox, RN, BSN, MSPH
Director of Planning and Healthcare Quality
Vermont Department of Health
P.O. Box 70
Burlington, VT 05402
Prepared by:
Amanda G. Kennedy, PharmD, BCPS
Associate Professor of Medicine
UVM College of Medicine Office of Primary Care
Arnold 5, UHC Campus
1 South Prospect Street
Burlington, VT 05401
Program contact information:
Elizabeth Cote
Director, UVM College of Medicine Office of Primary Care
University of Vermont
UHC Campus--Arnold 5
1 South Prospect Street
Burlington, VT 05401
(802) 656-2179
Table of Contents
Executive Summary
Purpose of the Report
Act 63 (2011) Sec. E.311
Project Specific Aims
Results of Deliverables: Planning Phase
Results of Deliverables: Implementation and Data Collection Phase
Results of Deliverables: Evaluation Phase
Summary of Major Findings
References
3
4
4
4
6
9
10
18
19
Executive Summary
This grant supported the University of Vermont College of Medicine’s Office of Primary Care to
conduct a population-based medication management pilot study in collaboration with selected primary
care practices and pharmacists over a two year period. The broad goal of this pilot was to improve
care for Vermonters, specifically related to optimizing medications in primary care. To support this
goal, the pilot implemented and evaluated several approaches to pharmacist collaboration with
primary care. The primary measure of success was the identification of real and potential problems
relating to medications, commonly referred to as “drug therapy problems.”
The approaches in the project reflected the pharmacists’ backgrounds and the types of practices
served. Seven demonstration sites were enrolled representing five Vermont counties. The pharmacist
types included academic, community, and hospital pharmacy. The practices included private,
federally-qualified health center (FQHC), practices that train medical residents, and hospital-owned.
The pharmacists identified over 700 drug therapy problems related to dosing, adherence,
unnecessary drug therapy, etc. Identification of drug therapy problems occurred through three
strategies: 1) direct patient care, 2) population-based strategies, and 3) prescriber and patient
education. Common population-based strategies included dose adjustments and discontinuing
unnecessary medications. The problems identified by pharmacists involved medications common in
primary care, such as cardiovascular or diabetes medications, medications for mental health
indications (e.g. depression), and high-risk medications (e.g. anticoagulants). Pharmacists’
recommendations to correct drug therapy problems were accepted by prescribers 86% of the time. Prescribers, staff, and patients overwhelmingly perceived the pharmacists as having value. It is
estimated that $2.00 in cost was avoided for every $1.00 spent on a pharmacist.
Major Recommendations
 Integrating pharmacists into primary care is feasible, reveals important drug therapy problems
that impact Vermonters, and results in overall cost-avoidance. Offering pharmacy services to
primary care practices is recommended, including direct patient care (especially during
transitions of care), population-based management, and patient and prescriber education.
 Sustainability will require the identification of alternative funding mechanisms that do not rely on a
fee-for-service approach. This is essential to preserve a pharmacist’s ability to provide
population-based management and educational services in addition to direct patient care.
 While larger practices may benefit from a full-time pharmacist, smaller practices may be served
well with a pharmacist as a shared resource. A ratio of 1 pharmacist full-time equivalent (FTE) to
5 physician FTEs, or 1 pharmacist to 8,000-10,000 patients is recommended.
 Pharmacists should ideally be full-time in primary care, even if serving multiple practices. At a
minimum, individual pharmacists should be 0.5 FTE.
 It is recommended that pharmacists be either specialty trained in primary care (e.g. primary care
residency) or employed by the same organization as the practice(s). For pharmacists without
specialty training in primary care, prior job experience interacting with physicians as part of an
inter-professional team (e.g. participation in hospital rounds) is recommended.
 Pharmacists must have full permissions to access and document findings in the practices’
electronic health records. It is unlikely pharmacists can be successful in fully integrating into
primary care practices without these permissions.
 It is recommended to follow the recently published Guidelines for Pharmacists Integrating into
Primary Care Teams, published by the Canadian Pharmacists Journal, November 2013.[1]
Purpose of the Report
The purpose of this report is to provide an overview of results for the Medication Management Pilot
Study and to respond to the reporting requirements detailed in Act 63. This report details all project
phases and covers a timeframe of January 2012 through December 2013.
Act 63 (2011) Sec. E.311
POPULATION-BASED MEDICATION MANAGEMENT PILOT PROJECT
a) As part of the evidence-based education program established in subchapter 2 of chapter 91 of
Title 18, the Department of Health, in collaboration with the Department of Vermont Health
Access and the University of Vermont Office of Primary Care, shall establish a populationbased medication management pilot project to include a collaborative pharmacist practice
using principles consistent with the Vermont Blueprint for Health.
b) The Department of Vermont Health Access shall fund the pilot project from the fee established
in 33 V.S.A. § 2004 and shall transfer funds to the Department of Health for implementation of
the pilot.
Sec.15a. POPULATION-BASED MEDICATION MANAGEMENT PILOT: REPORT
a) January 15, 2014, the Department of Health, University of Vermont Office of Primary Care,
and the Joint Fiscal Office shall provide a report to the House Committee on Health Care and
the Senate Committee on Health and Welfare describing and evaluating the effects of the
population-based medication management pilot program.
b) The report shall describe how the pilot project is implemented, including which medications
were targeted. The report shall assess the pilot program in terms of improvements to patient
care and increases in evidence-based prescribing through improvements to prescriberpharmacist communication and collaboration.
Project Specific Aims
This grant supported the UVM Office of Primary Care to conduct a population-based medication
management pilot study in collaboration with selected primary care practices and pharmacists over a
two year period. The broad goal of this pilot study was to improve care for Vermonters, specifically
related to optimizing medication regimens in primary care settings. To support this goal, the pilot
described and evaluated varying models for pharmacist collaboration with primary care practices.
Health information technology, including population-based medication reports from primary care
practices’ electronic health records, served as a tool for the pharmacist collaborations.
The selected demonstration sites included primary care practices from multiple Vermont counties. The
evaluation of the pilot study identified the strengths and weaknesses of various pharmacist/primary
care collaborative models related to costs, clinical outcomes, patient and healthcare professional
satisfaction, and model sustainability. This pilot study was collaborative between pharmacists and
primary care practices and was supported by a broad foundation of stakeholders, using the expertise
and opinions of the Vermont Blueprint for Health and Vermont Academic Detailing Program Advisors,
including the Department of Vermont Health Access and the Vermont Department of Health.
The specific aims of this project were to:
1. Design and implement several models of pharmacist collaboration in primary care
2. Evaluate the impact of the pharmacist models across multiple outcomes
3. Develop and pilot a population-based approach to managing medications in primary care
using health information technology
4. Collaborate with stakeholders to inform the process and disseminate findings
Background Reports
A full project protocol, including the documents listed below and background references, is available
from Dr. Kennedy upon request. For a complete history of this project please request the following
documents:
• Generic Drug Voucher Pilot: An evidence-based advisory report to the Vermont Department
of Health, submitted April 30, 2010.
• Implementation of Option 3 of UVM’s Generic Drug Voucher Advisory Report, submitted to the
Vermont Department of Health on June 29, 2011.
• Medication Management Pilot Study Phase 1 (Planning) Progress Report, submitted to the
Vermont Department of Health on July 31, 2012.
• Medication Management Pilot Study Phase 2 (Implementation and Data Collection), submitted
to the Vermont Department of Health on July 30, 2013.
Table 1. Attachment A: Deliverable Overview
ID
Phase
Deliverable
1A
1 (Planning)
List of demonstration site participants
1B
1 (Planning)
Summary of baseline data for each demonstration site
1C
1 (Planning)
Documentation of orientation attendance by participants
1D
1 (Planning)
Description of the pharmacy network progress and plans
1E
1 (Planning)
Summary of practical applications from literature for primary care
practices
2A
Phase 2 (Implementation Description of the models piloted including which medications
and Data Collection)
were targeted
2B
Phase 2 (Implementation Overview of types of data collected
and Data Collection) 2C
Phase 2 (Implementation Description of the pharmacist integration into Community Health
and Data Collection) Teams for each demonstration site
3A
Phase 3 (Evaluation)
Evaluation focused on:
• Costs
• Improvements to patient care
• Patient satisfaction
• Primary care provider and pharmacist satisfaction
• Increases in evidence-based prescribing through
improvements to prescriber-pharmacist communication and
collaboration.
• Strengths and weaknesses of various pharmacist models of
collaboration between primary care practices and
pharmacists
• Sustainability of the collaborative models
• Data about how health information technology may be
utilized to support population-based medication
management in primary care
• Plan for continued collaboration with broadly-represented
stakeholders to share best practices and foster integration
of pharmacy services into primary care practices and the
Vermont Blueprint for Health.
Results of Deliverables: Planning Phase
The results of the Medication Management Pilot are presented in order of
deliverables required by Attachment A of the grant (Table 1). Please note that the
results presented here may vary from previously submitted progress reports, as
these results represent final project data unless otherwise indicated.
1A-1B: Demonstration Sites and Baseline Data
In total, seven demonstration sites were enrolled representing five Vermont
counties (Table 2). However not all sites contributed the same length of time to the
study. Milton Family Practice exited the project in the fall of 2012, due to the
pharmacist leaving her academic position. Porter Hospital did not begin collecting
data until spring 2013. No sites are currently collecting data.
Table 2. Description of Demonstration Sites and Baseline Data
County Practice* Type Ownership
Pharmacist
Addison Porter-owned
practices Internal &
Family Med Hospital: Porter Medical
Center
Caledonia Corner Medical
Center Family Med Hospital: Northeastern
Vermont Regional
Hospital (NVRH)
Chittenden Aesculapius
Medical Center Internal Med
1/13-8/13
Approx.
Annual
Visits***
--
Hospital: NVRH 5/12-6/13
19,100
Hospital: FAHC
5/12-6/13
22,500
Hospital: Porter
Medical Center
Project
Dates**
Hospital: Fletcher Allen
Health Care (FAHC)
Chittenden Given Burlington
Chittenden Milton Family
Practice Franklin Cold Hollow
Family Practice
Internal Med
Resident
Hospital: FAHC
Hospital: FAHC
5/12-8/13
23,000
Family Med
Resident Hospital: FAHC
Academic: Albany
College of Pharmacy
& Health Sciences
5/12-10/12
30,000
Family Med Private
Community: Rite Aid 10/12-5/13
--
Rutland Brandon Medical
Center Family Med FQHC: Community Health Community: CHCRR
5/12-8/13
12,200
Centers of the Rutland
Region (CHCRR)
*We gratefully acknowledge Fletcher Allen Pharmacy Director, Karen McBride, for her generous donation of a Fletcher Allen
pharmacist to open Aesculapius Medical Center as an additional demonstration site.
**Please note: Phase 1 began in January 2012, however not all sites were prepared to collect data before the Phase 2 start
date of July 2012.
***Porter Medical Center and Cold Hollow Family Practice contributed less than 6 months of data to the project and have not
been included in the final data analysis. 1C-1D: Orientation and Pharmacist Network
All enrolled sites and pharmacists were invited to attend an orientation meeting at the University of
Vermont on January 30, 2012 (meeting minutes are available upon request). This meeting included
introductions, a project overview, proposed evaluation measures, and a general discussion. This
meeting was well attended and helped set a foundation for the pharmacists to begin partnering with
the physicians and other prescribers in the practices. Dr. Kennedy maintained close contact with
practices throughout the project.
The project pharmacists met weekly throughout the project by online conference call. The purpose of
meeting weekly was to develop a pharmacist network for sharing of best practices, protocols, barriers,
successes, etc. Three technologies were used to support the network. First, the online conferencing
service, GoToMeeting, was used for the weekly meetings. This online platform allowed the
pharmacists from across the state to meet “live” without the need to travel. Second, a Google Blog
was established for sharing meeting minutes and for facilitating discussions among the pharmacists.
The blog was set up as a private blog, with access granted only to the project pharmacists and
investigators. Although the pharmacists were aware not to disclose Protected Health Information
(PHI) in the blog, they were encouraged to present de-identified challenging cases for discussion.
Lastly, an online file sharing tool, Dropbox, was set up for the participating pharmacists. The Dropbox
allowed the pharmacists to share protocols, relevant articles, educational materials, and other
electronic files with each other. Sharing resources was encouraged as a way to maximize efforts and
efficiency. Dropbox was a valuable tool and information repository for the pharmacists, as protocols
and documents could be shared easily across institutions and practices.
The pharmacist network was also extended to faculty at Albany College of Pharmacy and Health
Sciences. Michael Biddle, PharmD, BCPS, Assistant Professor of Pharmacy Practice, regularly
participated in the network. Dr. Biddle maintains a faculty practice site at Richmond Family Medicine
in Richmond, Vermont.
1E: Summary of the Literature
Primary care patients in the United States consume a staggering number of medications. Seventy-five
percent of all primary care visits include prescribing or continuing medications.[2] Nearly a quarter of
all children under 18 years use one prescription medication, 90% of all adults 65 years and older use
one prescription medication, and nearly 40% of all adults 65 years and older use five or more
prescription medications.[3] Among adults who use prescriptions medications, one study estimated
46% of patients also use over-the-counter medications and 52% use dietary supplements.[4] There is
a critical need for primary care medication management services, given the association between the
number of medications used and frequency of drug-related problems, such as adverse events,
medication errors, drug interactions, drug-disease interactions, and non-adherence.[5]
Pharmacists are being increasingly recognized for their services beyond dispensing. Evidence
suggests pharmacists in ambulatory settings reduce hospital and emergency room admissions,
decrease the use of non-scheduled health services, decrease the number and cost of drugs, and
improve prescribing.[6, 7] Pharmacists have been shown to optimize patient outcomes such as blood
pressure[8-15], cholesterol[13, 14, 16], diabetes[17, 18], and smoking cessation.[13, 14, 19]
Additionally pharmacists have improved patient safety by reducing medication errors[20], improving
laboratory monitoring for medications[21], dose adjusting for renal dysfunction[22], stopping
medications[23], reducing inappropriate prescribing[24], improving adherence[25], and reducing
costs.[23, 26]
With primary care adopting innovative care models, including patient-centered medical homes, and
Accountable Care Organizations (ACO), there are new opportunities to integrate the unique skills of
pharmacists as part of the primary care team. To integrate successfully, pharmacists must practice at
the full scope of their license, have support from primary care providers, patients, and other
stakeholders, and have payment mechanisms that align with the highest yield activities.[27-29]
Pharmacist Role in Primary Care
A growing body of evidence supports pharmacists’ activities in primary care. We loosely define the
role of a pharmacist in primary care in three ways: providing direct patient care, providing populationbased medication management services, and prescriber education. Smith et. al. integrated
pharmacists into four federally qualified health centers and one private practice in Connecticut.[30]
Over the course of one year the pharmacists intensively worked with 88 Medicaid patients, with an
estimated savings of $1,600 per patient. Kozminski et al. integrated pharmacists into four patientcentered medical homes in Pittsburgh, Pennsylvania.[31] The pharmacists provided direct patient
care, including comprehensive medication therapy management and medication reconciliation during
transitions of care. Qualitative results from prescribers and staff included positive overall benefits of
the integrated pharmacists, perceived time-savings, and improved workflow. It took about six months
for pharmacists to feel they had integrated into the practices.
Academic detailing is an evidence-based prescriber education and support strategy for improving
prescribing.[32, 33] Academic detailing draws on the strategy of person-to-person social marketing
and provides prescribers with the most up-to-date, evidence-based information, to support effective
prescribing.[34] Pharmacists have long been recognized as an ideal group for providing academic
detailing. Combining academic detailing with direct patient care and population-based medication
management is likely synergistic for optimizing patient outcomes.
Population management is a relatively novel, but important task for pharmacists practicing in primary
care. One study found population management in combination with academic detailing for generic
substitution, therapeutic interchange, and targeting drugs based on evidence-based guidelines
improved generic utilization, blood pressure control, and cost avoidance for deep vein thrombosis.[35]
Qualities of Successful Pharmacists
In November 2013, guidelines for pharmacists integrating into primary care were published in the
Canadian Pharmacists Journal.[1] The guidelines suggest 10 recommendations. The guidelines also
stress that pharmacists’ personalities influence the level of success achieved in primary care
integration, with a suggestion that individual pharmacists should have personality traits that include
assertiveness and confidence.
Table 3. Recommendations for Pharmacist Integration into Primary Care[1]
Number
1
2
3
4
5
6
7
8
9
10
--
Recommendation
Determine the needs and priorities of the team and its patients
Develop a pharmacist job description
Educate the team about the pharmacist role
Educate yourself about the roles of the other team members
Ensure clinic infrastructure supports the pharmacist role
Be highly visible and accessible
Ensure your skills are strong
Provide proactive care and take responsibility for patient outcomes
Regularly seek feedback from the team
Develop and maintain professional relationships
Pharmacists should possess the qualities of assertiveness and confidence
Pharmacist Workforce
An opinion statement of the American College of Clinical Pharmacists suggests pharmacists in
primary care should practice with a ratio of 1 pharmacist full time equivalent (FTE) to 5,000-6,000
patients.[36]
Results of Deliverables: Implementation and Data Collection Phase
2A-2B: Description of the Implemented Models Targeted Medications, and Data Collected
The different models enrolled in the project reflect the pharmacists’ backgrounds and the types of
practices served (Table 4). The pharmacist types included academic, community, and hospital
pharmacy. Within community pharmacy, we selected a community pharmacist working for a federallyqualified health center and a traditional community or retail pharmacist. Academic, community, and
hospital pharmacy represent the major pharmacist workforce opportunities in Vermont. The
pharmacists provided services to three different types of practice sites: individual primary care
practice sites, individual practices that primarily train medical residents, and all practices owned by a
hospital.
Table 4. Piloted Models
Academic
pharmacist
One practice
FQHC
Community
pharmacist
Chain Drug Store
Community
pharmacist
Brandon
Medical Center
Cold Hollow
Family Practice
Hospital
pharmacist
Aesculapius
Medical Center
Corner
Medical Center
One resident practice
All hospital-owned
practices
Milton
Family Practice
Given Burlington
Porter
Medical Center
The pharmacists were required to engage with the primary care practices in three ways: populationbased medication management, individual or direct patient care, and prescriber education. The
amount of time spent in each task was left to the pharmacists and the primary care practices.
Targeted medications included those common to primary care (e.g. cardiovascular, diabetes, mental
health) or medications associated with a high risk of adverse events (e.g. anticoagulants, insulin,
medications associated with FDA warnings).
This project collected data using numerous methods:
• Assurance™ software: This HIPAA-compliant software enabled pharmacists to document their
direct patient care interventions related to optimizing medications.
• IVents: This Fletcher Allen-specific tool was integrated into PRISM, the hospital’s electronic
health record, allowing the Fletcher Allen project pharmacists to efficiently document their
interventions.
• Microsoft Excel: Pharmacists created spreadsheets to track population-based activities and
pharmacist time.
• Patient Surveys: The pharmacists piloted a medication adherence survey and satisfaction
survey.
• Online Surveys: At entry and exit to the project, practice providers and staff were encouraged
to provide their perceptions of having a pharmacist in primary care.
• Interviews: At project exit, Dr. Kennedy conducted interviews with selected prescribers, staff,
and project pharmacists to understand the strengths and weaknesses of pharmacists in
primary care.
2C: Pharmacist Integration into Blueprint Community Health Teams (CHTs)
Dr. Kennedy organized meetings between the pharmacists and their Community Health Team
contacts. The Fletcher Allen site pharmacists met with Pam Farnham, RN, on June 20, 2012 and
August 1, 2012. The Corner Medical pharmacists met with Laural Ruggles, MBA, on June 29, 2012.
The Brandon Medical Center pharmacist met with Mary Lou Bolt on July 12, 2012 and March 28,
2013. The Cold Hollow pharmacist met with Candace Collins on October 25, 2012. Dr. Kennedy
participated in each of these meetings. Although no formal relationships were established (e.g. formal
processes for referring patients), both the pharmacists and CHTs were interested in understanding
how formal collaborations might develop in future projects.
Results of Deliverables: Evaluation Phase
3A: Evaluation
Improvements to patient care and increases in evidence-based prescribing through improvements to
prescriber-pharmacist communication and collaboration.
Identification of drug therapy problems occurred through three routes: direct patient care, populationbased strategies, and education. See Table 5. Common population-based strategies included dose
adjustments and discontinuing unnecessary medications.
Table 5. Pharmacist Activity Types
Activity
N
(%)
Direct Care
336
(47.5)
Population-based
276
(38.9)
Education
96
(13.6)
Total
708
(100)
The pharmacists identified 708 drug therapy problems related to dosing, adherence, unnecessary
drug therapy, etc. See Table 6. The problems identified by pharmacists involved medications common
in primary care, such as cardiovascular (e.g. blood pressure, cholesterol) or diabetes medications and
medications for mental health indications (e.g. depression). See Table 7. Pharmacists’
recommendations to correct drug therapy problems were accepted by prescribers 86% of the time,
where data about acceptance is known. Of the 49 recommendations not accepted, 47/49 (96%) were
population-based and 2/49 (4%) were related to direct patient care.
Table 6. Pharmacist-identified Drug Therapy Problems
Drug Therapy Problem
N
(%)
Dosage Too High
150
(21.2)
Adherence
105
(14.8)
Unnecessary Drug Therapy
100
(14.1)
Specific Interventions Missing
(e.g. curbside consult)
Different Drug Needed
96
(13.6)
72
(10.2)
Education
53
(7.5)
Need Additional Drug Therapy
50
(7.1)
Drug Information Question
39
(5.5)
Adverse Drug Reaction
24
(3.4)
Dosage Too Low
19
(2.7)
Total
708
(100)
Table 7. Medication Categories
Medications
N
(%)
Mental Health and Insomnia
179
(25.3)
CV and Diabetes
177
(25.0)
GI
105
(14.8)
Asthma/Allergy
72
(10.2)
Anticoagulants
31
(4.4)
Immunizations
21
(3.0)
Benzos and Opiates
16
(2.3)
Other
107
(15.1)
Total
708
(100)
Costs
The cost evaluation to date is preliminary, as the data are being independently reviewed by another
pharmacist to verify the results. Additionally the cost evaluation only included the 5 study sites that
contributed at least 6 months of data (i.e. Cold Hollow and Porter Medical Center were not included).
Pharmacist Salary
Five sites partnered with a pharmacist one day per week (0.2 FTE) for 12 months is equivalent to one
full-time pharmacist. A typical hospital pharmacist’s annual salary (1.0 FTE plus 33% fringe benefits
rate) is approximately $176,690.
Costs Avoided
Costs avoided were calculated for the recommendations that were accepted by physicians (N=309) or
where the acceptance data were missing (N=350). Recommendations not accepted by prescribers
(N=49) were not included in this analysis.
We used a Veterans Administration (VA) perspective specific to primary care for the cost analysis, as
these data were rigorously structured and may be representative of an Accountable Care
Organization. Each drug therapy problem was categorized as having an outcome using the categories
provided by Lee et al. See Table 8. Data were inflated to 2013 dollars.[37] Applying the VA approach
to the interventions recommended by a pharmacist in the current project resulted in a total cost
avoidance of $356,294. See Table 9.
Table 8. Examples of Outcomes
Outcome
Example
Untreated
diagnosis
Patient on simvastatin 80 mg for >1 year. Patient's LDL has not been at goal of <100 since
2008. Recommended to change to a more potent statin (rosuvastatin 10 mg daily). This change
was made. Repeat lipid panel 3 months later revealed LDL = 80.
Prevent or manage
adverse drug event
Enoxaparin dose unclear. The discharge instructions stated 50 mg twice daily and she was on
120 mg once daily in the hospital. Based on her weight and renal function, 40 mg once daily is
recommended for DVT prophylaxis. Specialist physician adjusted the dose to 40 mg once daily.
Average of any
intervention
85 year old female. Reviewed med list with patient. Patient admits to regularly taking all meds
each day and understands indications. Answered questions regarding losartan and association
with increasing potassium (including potassium content in foods).
Avoid drug
interaction
Patient on simvastatin 40 mg + gemfibrozil 600 mg BID. Lipid panel revealed TG=265 (ranged
from 210-350 since 2003). Patient also has uncontrolled diabetes that could be impacting TG
control. Suggested d/c of gemfibrozil. Gemfibrozil was discontinued.
Adjust dose or
frequency
Suggested re-assessment of zolpidem dose/hx of treatment. Consider decrease in dose to 5
mg. MD decreases dose to 5 mg and is considering discontinuing.
Drug not
indicated
Chart review and did not reveal evidence of Zollinger-Ellison Syndrome, recurrent PUD,
prevention of NSAID induced peptic ulcer, or GERD with esophagitis, ongoing symptoms, or
complications such as Barrett's esophagus. Recommended tapering of their proton pump
inhibitor. Physician agreed with the recommendation.
Information
Only
Question from physician: How often is thrombocytopenia seen in patients on plaquenil that’s
attributable to the med? All medications in the 4-aminoquinoline class including plaquenil can
cause thrombocytopenia. Most information is based on long term use and FDA post marketing
reports. Thrombocytopenia occurs in approximately 1% of patients who take plaquenil.
Table 9. Cost Evaluation
Recommendations
Costs Avoided per
Recommendation
$1,919.69
44
Total Costs
Avoided
$84,466
Prevent or manage adverse drug event
$695.71
157
$109,226
Average of any intervention
$567.21
161
$91,321
Avoid drug Interaction
$411.46
62
$25,511
Adjust dose or frequency
$375.11
96
$36,011
$94.75
103
$9,759
$0.00
36
$0
659
$356,294
Untreated diagnosis
Drug not indicated
Information only
Total
N
Reference for cost data: Lee AJ, et al. Clinical and economic outcomes of pharmacist recommendations
in a VA medical center. Am J Health Syst Pharm. 2002 Nov 1;59(21):2070-7. PMID: 12434719
Return on Investment (ROI)
Preliminary analyses suggest $2.00 in cost was avoided for every $1.00 spent on a pharmacist
($356,294/$176,690). If the analysis assumed the missing data would be accepted at 86% (i.e. the
same proportion as the known data) the estimated cost avoidance would be $323,779 with an
estimated R0I of $1.83 ($323,779/$176,690). One of the 5 sites contributed only 6 months of data
(N=64 recommendations). If those recommendations were doubled to estimate 1 year of data, and the
missing data were considered accepted at 86%, the estimated cost avoidance would be $357,191
with an estimated R0I of $2.02 ($357,191/$176,690).
Patient satisfaction
Pharmacists were encouraged, but not required, to mail satisfaction surveys to the patients that
participated in direct patient care activities with a pharmacist. See Table 10. The satisfaction survey
was adapted from those used previously in the literature.[38-40] Five patients returned surveys.
Although too few surveys were collected to permit a statistical analysis, all were very positive about
their interactions with a pharmacist in primary care.
Table 10. Patient Satisfaction with Pharmacists’ Care (N=5 patients)
Question
Score
1.
I am satisfied with the care that I received at my medication consult
4.4
2.
The pharmacist explained things in a way that was easy to understand
4.4
3.
The medication consult increased my understanding of my medications
4.4
4.
The medication consult increased my understanding of my medical conditions
4.2
5.
The medication consult provided me with information that I had not learned from my other
health care providers
4.2
6.
I was comfortable speaking to the pharmacist and asking her questions
4.8
7.
I would recommend a medication consult to friends and relatives
4.6
8.
I am satisfied with the amount of time that I spent with the pharmacist
4.4
9.
I would be willing to pay a copay to meet with a pharmacist for a medication consult
Score: Strongly Agree=5, Agree=4, Neutral=3, Disagree=2, Strongly Disagree=1
4.2
Patient Comments
 I learned a lot from you. I am a lot better!
 The concern and friendly attitude towards me personally [what was best about the visit]
 Being honest [what was best about the visit]
 They were great all the way around. They really made me understand and taught me how to
use my medicine which was huge!
 I feel everyone taking meds should have to do this. The doctors do not know or do not tell you
how to use your medicine.
Primary care provider and pharmacist satisfaction
Prescribers, staff, pharmacists, and Blueprint Community Health Teams and staff were invited to
participate in baseline and follow-up surveys to evaluate satisfaction with pharmacists as part of the
primary care team. Sixty-seven total surveys were returned; 45 individual baseline surveys, 14
individual follow-up surveys, and 8 surveys that included both baseline and follow-up by the same
physician. See Tables 11 and 12. A sample of physicians and all of the pharmacists have been
interviewed to further understand the strengths and weaknesses of a pharmacist in primary care.
Analyses of the interview data are ongoing.
Table 11. Survey Respondents
Role
Physician
Nurse or Medical Assistant
Physician Resident
Practice Staff
N
26
10
9
8
(%)
(38.8)
(14.9)
(13.4)
(11.9)
Community Health Team or
Blueprint Administration
Nurse Practitioner
Pharmacist
Practice Manager
Total
5
3
3
3
67
(7.5)
(4.5)
(4.5)
(4.5)
(100.0)
Table 12. Prescriber Experience
Prescriber Experience
0-5 years
6-10 years
11-15 years
16+ years
Total
N
13
3
8
14
38
(%)
(34.2)
(7.9)
(21.1)
(36.8)
(100.0)
Survey participants were encouraged to include comments about their perceptions of working with
pharmacists in primary care. Table 13 includes all of the comments received in the follow-up surveys.
Prescribers overwhelmingly perceive the pharmacists as having value to their practice and
their patients.
Table 13. Follow-up Comments
Role
Comment
Physician
Excellent program that promoted development of relationships with pharmacist, as well as improving
patient care
Physician
Having a pharmacist in our office was a fantastic experience. The entry of a pharmacist into our clinic coexisted at a time when we also received assistance for our patients from the community health team.
From the patient perspective there may have been some overlap in what the pharmacist and CHT nurse
could provide (med reconciliation, etc.). I often have multiple issues to discuss with a patient in an office
visit and by previewing which of my patients could possibly discontinue or wean their PPIs this made me
more likely to address this issue in my office visits. As I was intimately aware of the role of our
pharmacist, I knew of her availability but this may not have been the case with other faculty members. I
hope we have the opportunity to have a pharmacist again in the near future as patients are best cared for
by a team approach. We have many refugees in our clinic and I think a pharmacist could fill a unique role
in bridging the cultural divide of educating this vulnerable population on their medication management.
Physician
I found the pharmacist to be extremely competent and helpful. However, I found it difficult to incorporate
her talents / knowledge into my daily practice. I typically would think of medication questions on the fly,
and if this did not happen to be on a Thursday afternoon, I would use UptoDate or a retail pharmacist to
answer my question. I had one question having to do with how to find the least expensive drug option for
a patient for a particular condition, and I was told by a colleague that this would not be an appropriate
reason to consult the pharmacist. This was a surprise to me, in that I would think even relatively
straightforward questions would still be a learning opportunity (e.g. perhaps she would have had a good
way to look such questions up, etc.). I think a list of good examples of how to utilize her skills would have
been helpful. For example consider consults for elderly patients with > 8 or 10 medications to reduce
poly-pharmacy. The one consult I was able to come up with was a patient with several meds to be taken
at different times of the day, some with and some without meals, and some to avoid using together. The
pharmacist was able to construct a regimen. Again, I feel the concept is excellent, I just did not feel I used
her presence optimally.
Physician
I wish we could meet an hour every week. The educational value is immense.
Physician
It seems like even small improvements in medication prescribing yield big dollars in savings, and can
justify the expense of having a pharmacist on site.
Physician
Medicine is a very dynamic field with changes in principles and protocols happening constantly. One of
the biggest areas in medicine to change is pharmacology. Drugs that used to be deemed as "safe" now
have many important drug interactions or dosing limits that providers can be challenged to keep up with.
Having a pharmacist as part of our team was a very valuable asset. The ability to have the pharmacist
review complicated medication lists or to do outreach to patients identified as high risk based on planned
data searches is one of if not the biggest patient safety measures we can apply. I hope that there is a
way to support having a pharmacist available as part of our teams going forward.
Physician
The pharmacists were valuable people on the team. Feedback from patients was positive and I felt that it
improved their understanding of meds and disease. Her in-services for providers were great. Good
feedback for me regarding FDA changes.
Physician
There is huge potential to use pharmacists to both improve the quality of care and lower costs. We need
to do more of this!
Physician
Resident
Excellent project and extremely helpful to have an on-site pharmacist for both patient care/safety,
provider education. Very informative to be able to discuss medications and problem-shoot with
pharmacist while seeing patients.
Physician
Resident
It was great to have a pharmacist in clinic. She brought medication interactions and dosing issues to my
attention that otherwise would have gone unrecognized. She was also an excellent resource for
questions when I was concerned about prescribing a certain medication for fear of an adverse effect or
interaction. She was able to quantify risk and discuss other options in a manner that an online resource
cannot.
Nurse
Practitioner
It is difficult to bring on another player in the EMR on site. If we had a pharmacist available as a resource,
especially for the complicated medical patient, that would be perfect
Nurse
Practitioner
Very helpful to have the resource for sorting out implications of new interactions and recommendations on
management of chronic health issues
Pharmacist
Group patient education visits were very successful especially when located at the local physician
practice.
Practice
Manager
More guidance from the pharmacist community on how they see their roles in a community practice would
have been beneficial. There was too much, "Well, how would you use a pharmacist in your primary care
practice?"
Nurse or
Medical
Assistant
We have really enjoyed working with (pharmacist). She has discovered numerous med problems on
patients that were recently discharged, reconciling meds post discharge, and thanks to her we have
avoided many medication problems. She has worked closely with our residents and patients around
medications. She is very knowledgeable and happy to help in any way she can.
Four physicians answered both the baseline and follow-up surveys, allowing for a pre-post
comparison. The survey results demonstrate that after working with a pharmacist, there are
improvements in all perceptions of pharmacists in primary care, including usefulness, ease of working
with a pharmacist, trust, patient outcomes, and pharmacist role in a practice.
Table 14. Paired Analyses (Results from 4 physicians)
Question
Baseline
Follow-up
Result
I find working with a pharmacist useful in my job.
6.5
6.8
Improved
I find it easy to work with a pharmacist.
6.5
7.0
Improved
I trust the pharmacist to work with my patients or my practice’s patients.
6.8
7.0
Improved
The pharmacist improves the care of my patients or my practice’s patients.
6.5
6.8
Improved
I understand the role of the pharmacist in my practice.
6.3
6.5
Improved
*Baseline questions were asked as “I will find…” Scale: Strongly Agree=7, Agree=6, Somewhat Agree=5, Neutral=4,
Somewhat Disagree=3, Disagree=2, Strongly Disagree=1
Strengths and weaknesses of various pharmacist models of collaboration between primary care
practices and pharmacists
Strengths
 Diverse pharmacists were successfully integrated into diverse medical home practices. These
results suggest broad generalizability for pharmacist integration. The most successful sites
included specialty-trained academic pharmacists and pharmacists who were employed by the
same organization as the practice.
 Pharmacists successfully identified important drug therapy problems using populationbased methods.
 Pharmacists demonstrated cost avoidance quickly, within 6-12 months.
 Prescribers and staff perceived the pharmacists as having value to the practice and the
patients they served.
 The pharmacist network was important for sharing of best practices.
Weaknesses
 Health information technology was a barrier to population activities at times.
 The selected practices were diverse, but did not represent every Vermont county and were
limited in number.
 This project did not explore the role of the independent community pharmacist and explored
the role of a chain drug store community pharmacist in a limited scope.
 The pharmacists did not attempt to bill for their services and therefore the feasibility of fee-forservice billing is unknown.
 The part-time (one day per week) nature of the project was difficult for pharmacists. In general,
the pharmacists perceived one day per week to be inefficient and limiting in the services
provided.
 Understanding the pharmacists’ roles in the practices took about 6 months, even when the
prescribers and pharmacists had existing working relationships.
 Working across organizations presented additional challenges to integration.
Sustainability of the collaborative models
Sustainability is highly dependent on a funding mechanism for the pharmacists’ salaries. Currently
there is no sustainable mechanism to keep the pharmacists in primary care beyond the scope of what
was paid for by the grant. Given the value of the population-based and educational efforts of the
pharmacists, two activities not paid for by fee-for-service billing, it is unlikely that fee-for-service billing
is the optimal route to a sustainable model for pharmacist integration.
Based on the current project, it is highly recommended that any future project or funding mechanism
support pharmacists providing direct patient care (including transitions of care), population-based
medication management, and patient and prescriber education to primary care practices.
Vermont Workforce Estimates
Vermont currently has 414 physician FTEs that include Family Medicine, Internal Medicine, and
Pediatrics. Currently recommended ratios for pharmacist services in primary care are in the range of 1
pharmacist FTE to 5,000-6,000 patients.[36] Based on the findings of this pilot, Vermont’s practice
environment, and the inclusion of population-based approaches would allow 1 pharmacist to manage
up to 10,000 patients. Assuming 1 FTE physician has a patient panel of about 2,000 patients, 1
pharmacist FTE can support about 5 physician FTEs.
Assuming 414 Vermont physician FTEs, we estimate 83 pharmacist FTEs (414/5) are required
to support all of Vermont’s primary care patients. See Table 15. If focusing solely on Family
Medicine and Internal Medicine, we estimate 64 pharmacist FTEs (318/5) are required.
Table 15. Vermont Physician Workforce
Primary Care Specialty
MD/DO FTEs
2012
Family Medicine
202
Internal Medicine
116
Pediatrics
Total
96
414
Vermont Area Health Education Centers (AHEC) Program. The Vermont Primary Care Workforce: 2012 Snapshot.
http://www.uvm.edu/medicine/ahec/documents/AHEC_PCREPORT_1_16.pdf
Data about how health information technology may be used to support population-based medication
management in primary care
Efficiently identifying drug therapy problems using population-based approaches requires an
electronic health record capable of producing the needed reports, personnel capable of producing
reports in a timely manner, and pharmacist full-permission to access and document findings in the
electronic health record. Health information technology was a barrier to population-based approaches
at times.
Our findings suggest that pharmacists’ full permission to access and document findings in the
practices’ electronic health records is the most efficient means for documenting and communicating
with prescribers (rather than using an outside vendor for documentation).
Plan for continued collaboration with broadly represented stakeholders to share best practices
and foster integration of pharmacy services into primary care practices and the Vermont
Blueprint for Health.
See summary of major findings and recommendations.
Summary of Major Findings
 Pharmacists were successfully integrated into medical home practices and identified important
drug therapy problems. The most successful sites included specialty-trained academic
pharmacists and pharmacists who were employed by the same organization as the practice.
 Using population-based approaches to identifying drug therapy problems is novel and proved
important and feasible.
 Efficiently identifying drug therapy problems using population-based approaches requires
personnel and an electronic health record capable of producing the needed reports, and
pharmacist full-permission to access and document findings in the electronic health record.
Health information technology was a barrier to population-based approaches at times.
 Our findings suggest that pharmacists’ full permission to access and document findings in the
practices’ electronic health records is the most efficient means for documenting and
communicating with prescribers (rather than using an outside vendor for documentation).
 The potential impact of pharmacists in primary care is large, based on the number of drug
therapy problems identified and costs avoided in this demonstration.
 Prescribers and staff perceived the pharmacists as having value to the practice and the
patients they served.
 Working part-time for a practice was a barrier for pharmacists. Prescribers commented they
wanted access to a pharmacist as a shared resource in primary care.
 Prescribers accepted the recommendations of pharmacists 86% of the time, where data about
acceptance is known.
 The estimated return on investment was $2 avoided for every $1 spent on a pharmacist.
Recommendations
 Integrating pharmacists into primary care is feasible, reveals important drug therapy problems
that impact Vermonters, and results in overall cost-avoidance. Offering pharmacy services to
primary care practices is recommended, including direct patient care (especially during
transitions of care), population-based management, and patient and prescriber education.
 Sustainability will require the identification of alternative funding mechanisms that do not rely
on a fee-for-service approach. This is essential to preserve a pharmacist’s ability to provide
population-based management and educational services in addition to direct patient care.
 While larger practices may benefit from a full-time pharmacist, smaller practices may be
served well with a pharmacist as a shared resource. A ratio of 1 pharmacist full-time
equivalent (FTE) to 5 physician FTEs, or 1 pharmacist to 8,000-10,000 patients is
recommended.
 Pharmacists should ideally be full-time in primary care, even if serving multiple practices. At a
minimum, individual pharmacists should be 0.5 FTE.
 It is recommended that pharmacists be either specialty trained in primary care (e.g. primary
care residency) or employed by the same organization as the practice(s). For pharmacists
without specialty training in primary care, prior job experience interacting with physicians as
part of an inter-professional team (e.g. participation in hospital rounds) is recommended.
 Pharmacists must have full permissions to access and document findings in the practices’
electronic health records. It is unlikely pharmacists can be successful in fully integrating into
primary care practices without these permissions.
 It is recommended to follow the recently published Guidelines for Pharmacists Integrating into
Primary Care Teams, published by the Canadian Pharmacists Journal, November 2013.[1]
The University of Vermont College of Medicine’s Office of Primary Care continues to be in
conversation with the Vermont Department of Health regarding these findings and recommendations.
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
Jorgenson, D., et al., Guidelines for pharmacists integrating into primary care teams. Can
Pharm J (Ott), 2013. 146(6): p. 342-52.
CDC. National Center for Health Statistics. National Hospital Ambulatory Medical Care Survey:
2010 Outpatient Department Summary Tables. 2010 November 21, 2013].
CDC. National Center for Health Statistics. Health, United States. 2012 November 21, 2013].
Qato, D.M., et al., Use of prescription and over-the-counter medications and dietary
supplements among older adults in the United States. Jama, 2008. 300(24): p. 2867-78.
Viktil, K.K., et al., Polypharmacy as commonly defined is an indicator of limited value in the
assessment of drug-related problems. Br J Clin Pharmacol, 2007. 63(2): p. 187-95.
Beney, J., L.A. Bero, and C. Bond, Expanding the roles of outpatient pharmacists: effects on
health services utilisation, costs, and patient outcomes. Cochrane Database Syst Rev,
2000(3): p. CD000336.
Nkansah, N., et al., Effect of outpatient pharmacists' non-dispensing roles on patient outcomes
and prescribing patterns. Cochrane Database Syst Rev, 2010(7): p. CD000336.
Margolis, K.L., et al., Effect of home blood pressure telemonitoring and pharmacist
management on blood pressure control: a cluster randomized clinical trial. Jama, 2013.
310(1): p. 46-56.
Weber, C.A., et al., Pharmacist-physician comanagement of hypertension and reduction in 24hour ambulatory blood pressures. Arch Intern Med, 2010. 170(18): p. 1634-9.
Carter, B.L., et al., The potency of team-based care interventions for hypertension: a metaanalysis. Arch Intern Med, 2009. 169(19): p. 1748-55.
Carter, B.L., et al., Physician and pharmacist collaboration to improve blood pressure control.
Arch Intern Med, 2009. 169(21): p. 1996-2002.
Hunt, J.S., et al., A randomized controlled trial of team-based care: impact of physicianpharmacist collaboration on uncontrolled hypertension. J Gen Intern Med, 2008. 23(12): p.
1966-72.
Santschi, V., et al., Impact of pharmacist care in the management of cardiovascular disease
risk factors: a systematic review and meta-analysis of randomized trials. Arch Intern Med,
2011. 171(16): p. 1441-53.
Santschi, V., et al., Pharmacist interventions to improve cardiovascular disease risk factors in
diabetes: a systematic review and meta-analysis of randomized controlled trials. Diabetes
Care, 2012. 35(12): p. 2706-17.
Simpson, S.H., et al., Effect of adding pharmacists to primary care teams on blood pressure
control in patients with type 2 diabetes: a randomized controlled trial. Diabetes Care, 2011.
34(1): p. 20-6.
Pape, G.A., et al., Team-based care approach to cholesterol management in diabetes mellitus:
two-year cluster randomized controlled trial. Arch Intern Med, 2011. 171(16): p. 1480-6.
Choe, H.M., et al., Proactive case management of high-risk patients with type 2 diabetes
mellitus by a clinical pharmacist: a randomized controlled trial. Am J Manag Care, 2005. 11(4):
p. 253-60.
Jameson, J.P. and P.J. Baty, Pharmacist collaborative management of poorly controlled
diabetes mellitus: a randomized controlled trial. Am J Manag Care, 2010. 16(4): p. 250-5.
Dent, L.A., K.J. Harris, and C.W. Noonan, Randomized trial assessing the effectiveness of a
pharmacist-delivered program for smoking cessation. Ann Pharmacother, 2009. 43(2): p. 194201.
Murray, M.D., et al., Effect of a pharmacist on adverse drug events and medication errors in
outpatients with cardiovascular disease. Arch Intern Med, 2009. 169(8): p. 757-63.
Raebel, M.A., et al., Improving laboratory monitoring at initiation of drug therapy in ambulatory
care: a randomized trial. Arch Intern Med, 2005. 165(20): p. 2395-401.
Bhardwaja, B., et al., Improving prescribing safety in patients with renal insufficiency in the
ambulatory setting: the Drug Renal Alert Pharmacy (DRAP) program. Pharmacotherapy, 2011.
31(4): p. 346-56.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
Jameson, J., G. VanNoord, and K. Vanderwoud, The impact of a pharmacotherapy
consultation on the cost and outcome of medical therapy. J Fam Pract, 1995. 41(5): p. 469-72.
Hanlon, J.T., et al., A randomized, controlled trial of a clinical pharmacist intervention to
improve inappropriate prescribing in elderly outpatients with polypharmacy. Am J Med, 1996.
100(4): p. 428-37.
Lee, J.K., K.A. Grace, and A.J. Taylor, Effect of a pharmacy care program on medication
adherence and persistence, blood pressure, and low-density lipoprotein cholesterol: a
randomized controlled trial. JAMA, 2006. 296(21): p. 2563-71.
Strand, L.M., et al., The impact of pharmaceutical care practice on the practitioner and the
patient in the ambulatory practice setting: twenty-five years of experience. Curr Pharm Des,
2004. 10(31): p. 3987-4001.
Smith, M., D.W. Bates, and T.S. Bodenheimer, Pharmacists belong in accountable care
organizations and integrated care teams. Health Aff (Millwood), 2013. 32(11): p. 1963-70.
Collaborative, P.-C.P.C. Integrating Comprehensive Medication Management to Optimize
Patient Outcomes. 2012 November 21, 2013]; Available from:
http://www.pcpcc.org/sites/default/files/media/medmanagement.pdf.
Giberson S, Y.S., Lee MP., Improving Patient and Health System Outcomes through
Advanced Pharmacy Practice. A Report to the U.S. Surgeon General.2011: U.S. Public Health
Service. Office of the Chief Pharmacist.
Smith, M., M.R. Giuliano, and M.P. Starkowski, In Connecticut: improving patient medication
management in primary care. Health Aff (Millwood), 2011. 30(4): p. 646-54.
Kozminski, M., et al., Pharmacist integration into the medical home: qualitative analysis. J Am
Pharm Assoc (2003), 2011. 51(2): p. 173-83.
O'Brien, M.A., et al., Educational outreach visits: effects on professional practice and health
care outcomes. Cochrane Database Syst Rev, 2007(4): p. CD000409.
Avorn, J. and S.B. Soumerai, Improving drug-therapy decisions through educational outreach.
A randomized controlled trial of academically based "detailing". N Engl J Med, 1983. 308(24):
p. 1457-63.
Soumerai, S.B. and J. Avorn, Principles of educational outreach ('academic detailing') to
improve clinical decision making. Jama, 1990. 263(4): p. 549-56.
Devine, E.B., et al., Strategies to optimize medication use in the physician group practice: the
role of the clinical pharmacist. J Am Pharm Assoc (2003), 2009. 49(2): p. 181-91.
Nigro, S.C., et al., Clinical Pharmacists as Key Members of the Patient-Centered Medical
Home: An Opinion Statement of the Ambulatory Care Practice and Research Network of the
American College of Clinical Pharmacy. Pharmacotherapy, 2013.
Lee, A.J., et al., Clinical and economic outcomes of pharmacist recommendations in a
Veterans Affairs medical center. Am J Health Syst Pharm, 2002. 59(21): p. 2070-7.
Larson, L.N., J.P. Rovers, and L.D. MacKeigan, Patient satisfaction with pharmaceutical care:
update of a validated instrument. J Am Pharm Assoc (Wash), 2002. 42(1): p. 44-50.
Moczygemba, L.R., et al., Patient satisfaction with a pharmacist-provided telephone
medication therapy management program. Res Social Adm Pharm, 2010. 6(2): p. 143-54.
Shimp, L.A., et al., Employer-based patient-centered medication therapy management
program: evidence and recommendations for future programs. J Am Pharm Assoc (2003),
2012. 52(6): p. 768-76.