Download Standardizing VBP`s Patient Satisfaction Data into Percentile Ranks

Standardizing VBP's Patient Satisfaction Data
into Percentile Ranks Using Reference
Distribution
Jenhao Jacob Cheng, PhD, MS
Alice Li, MS
Nikolas Matthes, MD, PhD, MPH
Quality Indicator Project
Press Ganey Associates, Inc.
[email protected]
APHA Annual Meeting, Denver CO, November 10 2010
Background
– The 2010 Heath Care Reform Bill mandates the
implementation of Medicare’s value-based Purchasing
(VBP) program for fiscal year 2013
– VBP evaluates hospitals’ performance by 17 clinical
measures for process of care and 9 patient satisfaction
measures on a rate-based (percent) scale
– Prior year’s performance serves as the baseline reference
from which the benchmarks (median and 95th percentile)
are derived for assessment year period
– Patient satisfaction data must be converted into
percentile rank (0-100) based on baseline data to
facilitate the scoring process
2
Percentile Rank
– People usually get confused between percentile rate (in
original data scale) and percentile ranks (always 0-100)
– Typically we are interested to know a data distribution
by percentile rates on some commonly used percentile
ranks (e.g., 25, 50, 75)
– Sometimes we convert the data into percentile ranks,
like in this study, to reflect the position in the reference
distribution for comparison purpose
– Therefore, percentile rank is a standardized score to tell
how good you are for any of the target measures
– Determining percentile ranks within one dataset is easy
but is more challenging when two datasets involved
3
Scoring for Patient Satisfaction
First 8 measures are assigned a score up to 10 points based
on percentile ranks and up to 20 additional points are
awarded for the lowest percentile rank
Patient Satisfaction Measure
Rate
Percentile Rank
Score
1. Nurse Communication
0-1
0 - 100
0 - 10
2. Doctor Communication
0-1
0 - 100
0 - 10
3. Cleanliness and Quiet
0-1
0 - 100
0 - 10
4. Staff Responsiveness
0-1
0 - 100
0 - 10
5. Medications Communication
0-1
0 - 100
0 - 10
6. Pain Management
0-1
0 - 100
0 - 10
7. Discharge Information
0-1
0 - 100
0 - 10
8. Overall Hospital Rating
0-1
0 - 100
0 - 10
0 - 100
0 - 20
9. Lowest Percentile Rank
Total
0 - 100
4
Study Design
– Retrospective cohort study based on hospitallevel data.
– Study population: 3,725 hospitals with HCAHPS
data for both 2007 and 2008 from Hospital
Compare website, maintained by Centers for
Medicare and Medicaid Services (CMS)
– For demonstration purposes only one patient
satisfaction measure, Overall Hospital Rating,
has been used.
5
Methodology
– We investigated three methods to assign
percentile ranks based on reference distribution:
• Cross Ranking (CR)
• Percentile Mapping (PM)
• Distribution Modeling (DM)
– Each method is individually evaluated for
accuracy, variation, and efficiency
– Pairwise comparison is performed for each pair
of methods to evaluate agreement with each
other
6
Cross Ranking
– Ranking each data point repeatedly within the entire
reference distribution, implemented by SAS Proc SQL
– When using properly, SQL can be viewed as a rank-based
statistical tool for data analysis
– The feature of Cartesian product allows SQL to handle
the repeated ordering of each data point (Yi) in reference
distribution (X)
– Percentile rank of each data point is calculated by
Count (X<Yi) / N
– N x N processing
7
SQL Code: Cartesian Product
select
y.rate,
sum(case when x.rate<=y.rate then 1 else 0 end) as num,
count(*) as den,
100*(calculated num / calculated den) as pctl
from x, y
group by id, y.rate;
8
Percentile Mapping
– Create a percentile lookup table which contains 101 pairs
of “rank” and “rate” listings, implemented by SAS Proc
UNIVARIATE (or Oracle percentile function)
– Mapping each data point (rate) to the nearest percentile
value (rate) to obtain the corresponding percentile rank
– Mapping can be implemented by a compound SQL
technique called “Nearest Neighbor Join” which involves
two “Unequal Join” procedures in opposite directions
– Two unequal joins not only make sure the nearest
mapping is obtained but also take care of the potential
outlier in either direction of unequal join
– N x 101 processing
9
SQL Code: Nearest Neighbor Join
select rate, pctl from
(
select y.rate, max(look.pctl) as pctl, min(abs(y.rate-look.rate)) as diff
from y, look
where y.rate>=look.rate
group by id, y.rate
union
select y.rate, min(look.pctl) as pctl, min(abs(y.rate-look.rate)) as diff
from y, look
where y.rate<=look.rate
group by id, y.rate
)
having diff=min(diff);
10
Distribution Modeling
– Compute the empirical cumulative distribution function
(CDF) based on Kernel density estimation, implemented
by SAS Proc KDE
– Fitting CDF data by piecewise linear regression with 9
knots, implemented by SAS Proc REG
– Note that the rounded predictions represent percentile
ranks
– Increase # of knots or include quadratic functions can
improve the model fit but the gain is limited
– Cubic functions are not suggested to avoid fitting a
decreasing curve section
11
Modeling CDF Curve
12
Comparison Analysis
– Accuracy of each method is evaluated by Kendall’s t
coefficient for rank correlation between percentile rank
and original data (rate)
– Variation of each method is evaluated by the standard
deviation
– Efficiency of each method is evaluated by computation
time using SAS 9.2 on a reference desktop computer
– Pairwise agreement is evaluated by root mean square
error (RMSE) between each pair of methods
13
Comparison Results
Individual Comparison
Pairwise Comparison
Method
Kendall t
SD
Time [sec]
Method
RMSE
CR
0.9996
28.71
34.7
CR-PM
0.68
PM
0.9997
28.67
0.25
CR-DM
0.89
DM
0.9998
28.22
0.13
PM-DM
0.79
14
Conclusion
– DM is the most parsimonious and efficient method to
convert data into percentile ranks based on reference
distribution
– Its statistical properties in terms of accuracy and
variation are also a bit superior but requires more
mathematical knowledge to understand the approach
– CR can be easily implemented by standard IT solution like
SQL but is much more computationally intensive and this
may prevent it to be used as an online tool
– PM presents an in-between choice which is still efficient
for online applications but not that difficult for nonstatistician (IT) to understand and implement it
15
Discussion
– Statistical properties like accuracy and variation are not
very different since they simply reflect how smoothly the
percentile ranks are calculated
– Therefore, they can be more different when N is smaller
and/or many ties exist
– On the other hand, efficiency is more distinguishing in
this study when N is large and vise versa
– PM is conceptually similar to DM but its calculation is
more like CR in a discrete domain (while DM in
continuous domain)
– Pairwise comparison also confirms that CR and PM are in
more agreement
16