Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
© 2011 Jones and Bartlett Publishers, LLC Donald E. Lighter, MD, MBA, FAAP, FACHE Professor, University of Tennessee Statistical Thinking and Analysis © 2011 Jones and Bartlett Publishers, LLC Deming – Theory of Profound Knowledge • Systems thinking – – – – – • System is more than a sum of its parts Understanding that the parts interact to produce the end product Coordination and collaboration of parts increase productivity of system Interconnected subsystems and processes affect each other Process variation – – All processes have variation that can either be inherent in the process or due to external influences – Variation is a major source of nonconforming output leading to reduced quality and higher cost – Identifying and reducing sources of variation is a major undertaking for performance improvement initiatives • Theory of knowledge – – Information must be tempered by experience and theory to become knowledge – Effective managers combine experience and theory to create organizational knowledge • Psychology – – Understanding the variation in people is as important as understanding the variation in processes – Successful managers use human psychology to effectively coordinate, collaborate, and motivate workers to optimize system outcomes © 2011 Jones and Bartlett Publishers, LLC Statistical thinking • Ron Snee (1986): “… statistical thinking is used to describe the thought processes that acknowledge the ubiquitous nature of variation and that its identification, characterization, quantification, control, and reduction provide a unique opportunity for improvement. ” “ ….Every enterprise is made up of a collection of interconnected processes whose input, control variable, and output are subject to variation. This leads to the conclusion that statistical thinking must be used routinely at all levels of the organization.” Snee, R. D. (1986). "In Pursuit of Total Quality." Quality Progress 20(8): 25-31. © 2011 Jones and Bartlett Publishers, LLC Shewhart’s concept of process variation • Common cause • Special (attributable) cause – Variation is based on the process materials and procedures; – Variation is predictable using mathematics related to probability and chance; – Variation is irregular, i.e. shows no particular pattern; and – High and low values within the measurements are statistically indistinguishable – Variation due to new, unanticipated, emergent, or previously unknown factors within the system; – Variation that is entirely unpredictable, even using statistical probability techniques; – Variation that is outside historical trends; and – Variation that indicates an underlying change in the system or some previously unidentified factor. © 2011 Jones and Bartlett Publishers, LLC Translated into statistical thinking: • Common Cause variation – Data points from process fall inside control limits – Data points are statistically indistinguishable • Special Cause variation – Data points from process fall outside control limits (3 standard deviations) – < 0.3% (3 chances per 1000) probability of occurrence • Tampering – Deming’s concept of treating common cause variation like a Jones special cause © 2011 and Bartlett Publishers, LLC Why is this stuff important? Let’s Review Some Data Collection Rules… © 2011 Jones and Bartlett Publishers, LLC Data collection principles • Understand need for information collected • Collect everything you think might be needed • Least invasive methods of collection • Operational definition of each data element • Appropriate format for analysis • Before starting, review the study to ensure correctness © 2011 Jones and Bartlett Publishers, LLC Common data sources in health care Source Claims Pro Con •Data used to pay claims •Analyzed for errors by edits in payer computer system •Data entry errors – insurer, provider •Paucity of information (limited clinical info) •Inconsistent payments for same services •Upcoding •Capitation effects Medicaid •Consistent coding systems within state •Population fairly uniform •Same as above •Varying types of plans around the US •Tendency to upcode more pronounced Medicare •Relatively consistent data set •Edits tend to reduce coding errors •Upcoding still a problem •Payment schedules vary by region, more than by specialty Provider Billing Systems •Source data from point of care •Usually consistent within a practice •Broad variation in coding between practices •Coding variation also for same services •Variety of formats •Original source data from point of care •Complete record of clinical encounter •Expensive to review •Variation in recording •Handwriting •Variety of recording conventions •Measures customer opinions directly •Often can be done simply •Lack of scientific approach, leading to bias •Selection bias •Validation Patient Charts Surveys © 2011 Jones and Bartlett Publishers, LLC Statistical process control (SPC) – a method to understand variation • Shows trends in the process mean over time • Evaluates process variability at each point in time • Provides graphic evidence that process is in control (or not) at each point © 2011 Jones and Bartlett Publishers, LLC Two primary types of data • Attributes – Counts of individual items – Examples? • Continuous (variables) – Variables along a measurement scale – Real numbers, no “gaps” between measures © 2011 Jones and Bartlett Publishers, LLC Types of Control Charts • Attribute data charts – p and np charts – c and u charts • Continuous data charts – IX-MR charts – X-bar and R charts – X-bar and s charts © 2011 Jones and Bartlett Publishers, LLC © 2011 Jones and Bartlett Publishers, LLC Commonly used control charts Control Charts for Attributes Data © 2011 Jones and Bartlett Publishers, LLC Attribute chart selection • p-charts – Proportions of nonconformities – Example: C-section rates • np-charts – Numbers of nonconformities – Example: maternal deaths • c-chart – Nonconformities per inspection unit, constant number of inspection units – Examples: housekeeping errors per room; missed appointments per day • u-chart – Nonconformities per inspection unit, like c, BUT… – Used when the number of inspection units varies © 2011 Jones and Bartlett Publishers, LLC Attributes data limb of decision tree… © 2011 Jones and Bartlett Publishers, LLC Example c-chart 20 15 Mean LCL 10 UCL Nonconformities 5 10 7 6 5 4 3 2 0 1 Number of Nonconformities c-chart for XYZ Clinic Day of Study © 2011 Jones and Bartlett Publishers, LLC u-charts u-chart for St. Elsewhere Food Service Note “wavy” control limit line – why? 0.0700 0.0600 UCL 0.0400 LCL 0.0300 u 0.0200 Mean u 0.0100 Day of Data Collection © 2011 Jones and Bartlett Publishers, LLC 7 -0.0100 4 0.0000 1 u-value 0.0500 Commonly used control charts Control Charts for Continuous Variables © 2011 Jones and Bartlett Publishers, LLC Continuous (variables) data limb of decision tree… © 2011 Jones and Bartlett Publishers, LLC IX-MR chart creation • Example: ALOS for a hospital – Data obtained from a hospital over 24 months – Calculate mean of all samples, plot as center line – Calculate MR, average moving range – Control limits = + D4 * MR-bar (D4 is the “correction factor”, see Table 5.7, p 191 in the text) – Plot on graph Remember: software does this work for you… © 2011 Jones and Bartlett Publishers, LLC IX-MR chart 40 IX Chart 30 24.7170 Average 20 10 7.3636 0 -10 -9.9897 -20 Range Date/Time/Period 30 25 20 15 10 5 0 MR Chart Note out of control MR chart point! 21.3133 6.5238 Date/Time/Period/Number IX chart has sample size of 1 Moving range is the difference between successive points and is surrogate for standard deviation (with correction factor) © 2011 Jones and Bartlett Publishers, LLC IX-MR chart • What’s important? – MR Chart – what does it mean if the MR is out of control? – IX Chart – what does it mean if an IX value is out of control? – What other analyses could we do? © 2011 Jones and Bartlett Publishers, LLC Common control charts depend on… • “Reasonable” conformity of the data set to a Gaussian (normal, bell shaped) distribution • Most analysis programs will provide a histogram of the data to determine if data are normally distributed IX-MR Histogram 12 10 0.08 24.71 -9.99 0.07 0.06 8 Number 0.05 6 0.04 0.03 4 0.02 2 0.01 0 -9.986920881 0 -3.046697983 3.893524915 10.83374781 17.77397071 24.71419361 Note bimodal distribution of data, indicating reason for MR chart in previous slide to be out of control; thus, IX-MR may not be appropriate for this data set © 2011 Jones and Bartlett Publishers, LLC What if the MR is out of control? • Determine special cause using root cause analysis and eliminate • Re-run the analysis with special cause eliminated • Track data through more cycles to ensure that attributable cause was correctly identified • Other options: – Data transformation, e.g. natural log of each point – Usually better to identify special © 2011 Jones and Bartlett Publishers, LLC cause Other types of continuous variable charts © 2011 Jones and Bartlett Publishers, LLC X-bar and Range Chart • Similar to IX-MR chart, except: – Subgroup size = 2 – 9 – Measure of variation is range • Procedure: – Mean of each subgroup plotted – Mean of those means is centerline – Range of each subgroup plotted – Mean of those ranges is centerline – D4 is used to adjust ranges to control limits – A2 is used to create X-bar control limits © 2011 Jones and Bartlett Publishers, LLC X-bar-R chart 50 Phlebotomist Time - Notify to Draw Average Time (X-bar) 40 30 20 10 Day of Study 60 Subgroup Range 4.5943 0 R Chart - Phlebotomist Time 40 20 0 0.0000 Day of Study Note the R chart is in control © 2011 Jones and Bartlett Publishers, LLC The histogram is “reasonably” normally distributed… Histogram 40 35 59.54 -22.82 30 Number 25 20 15 10 5 0 -22.8158868-14.57998216-6.344077532 1.8918271 10.12773173 18.36363636 26.599541 34.83544563 43.07135026 51.30725489 59.54315952 67.77906415 © 2011 Jones and Bartlett Publishers, LLC The last commonly used continuous variable chart… © 2011 Jones and Bartlett Publishers, LLC X-bar and s chart • Similar to others, except – Subgroup size >9 – Measure of variation = sample standard deviation • Procedure – Mean of each subgroup plotted – Mean of those means is centerline – s of each subgroup calculated and plotted – Mean of those s-values is centerline – B4 and B3 are used to adjust s to control limits – A3 is used to create X-bar control limits © 2011 Jones and Bartlett Publishers, LLC Airflow Example • Airflow measurements on a clinical unit • Ten measurements a day, spaced throughout the day • Subgroup size = 10 • Subgroup time period = 1 day • Measurements then plotted on x-bar s chart © 2011 Jones and Bartlett Publishers, LLC The Airflow Example 37.43 UCL 33.66 CL 29.89 LCL 6.62731 UCL 3.86207 CL Xbar SD 1.09683 LCL 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Day Note similarity to X-bar-R chart © 2011 Jones and Bartlett Publishers, LLC 21 22 23 24 25 Data conversion… • Used when raw data are not normally distributed • Used when raw data sample sizes are not uniform • Types of conversions – Lognormal – Arcsin – z-score • How do we calculate z-scores? © 2011 Jones and Bartlett Publishers, LLC Example Z-score plot Z-Score Chart 4.000 UCL 3.000 2.000 1.000 0.000 Mean -1.000 -2.000 -3.000 LCL -4.000 Z scores are x-values divided by the standard deviation © 2011 Jones and Bartlett Publishers, LLC In summary… • Data types - attribute vs. continuous variables determine type of control chart • Control charts have center line (average of control chart means) and upper and lower control limits (+3s) • For attribute charts, data points are nonconformity values or rates • For continuous variable charts, data points are sample values or averages of sample values • Measures of variation for control charts are corrected using bias correction tables © 2011 Jones and Bartlett Publishers, LLC Other useful analyses ANOM ANOVA Regression © 2011 Jones and Bartlett Publishers, LLC Rankings are used in health care • Concept of rankings – How are they used? – Are they valid? – What about control limits? • Measures falling within control limits are common cause - statistically indistinguishable • Can’t be ranked! – Time factor - most rankings are for specific period of time • Physician or provider profiles – experiences? © 2011 Jones and Bartlett Publishers, LLC Ranking – some approaches to validation • 95% Confidence Intervals – Not time series based, usually single point in time – Help establish the level of variation in the measurement used for the ranking (higher variation, less predictive ability from ranks) – Still difficult to identify outliers © 2011 Jones and Bartlett Publishers, LLC Percentiles • Often used for comparisons – Examples • • • • Percent mortality post-op Nosocomial infection rates Error rate for claims entry Others? • Problems with percentages – Denominator size may vary, making comparisons potentially invalid – Case mix adjustment not often done to adjust for sampling bias © 2011 Jones and Bartlett Publishers, LLC Now for something a little different… Analysis of Means! • Not time series data • Used for attribute (count) data with unequal subgroup sizes – Rate of particular measure of count data – Examples? • • • • C-section rates Antibiotic utilization rates Infection rates post-op Others? • Does provide adjustment for issues like case mix, if done correctly © 2011 Jones and Bartlett Publishers, LLC ANOM example: C-section rates ANOM Chart - Comparison of Proportion Data 0.600 0.500 Proportion 0.400 Proportions 0.300 Lower Common Cause Limits Upper Common Cause Limits 0.200 0.100 0.000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Subject Number C-section rates among providers doing deliveries UCCL = upper control limits for each provider LCCL = lower control limits for each provider Control limits adjusted for opportunities, i.e. cases, that provider treats © 2011 Jones and Bartlett Publishers, LLC 27 28 29 30 Analysis of variance (ANOVA) • Test hypotheses about differences between two or more means • Used in DOE to determine if changes in mean in one intervention subgroup statistically differ from other intervention subgroups • See Example 5.6 (p 220) © 2011 Jones and Bartlett Publishers, LLC Regression analysis • Test hypotheses of relationships between a response variable (Y) and one or more predictor variables (X) • Determination of statistical significance of relationships (r value) • Sign of coefficient (b) for predictor variable determines if effect is positive or negative • R2 value provides predictive level of model (i.e. how much of the variation in Y is due to the selected predictor variables) © 2011 Jones and Bartlett Publishers, LLC Types of regression • Simple linear regression – relates one xvariable to one dependent y-variable Linear Model 120 100 80 y = 4.007x + 8.663 60 R² = 0.9793 40 20 0 0 5 10 15 20 © 2011 Jones and Bartlett Publishers, LLC 25 30 Types of regression • Multiple regression – One dependent variable with multiple predictor variables – Graphic output is a multidimensional surface, so usually not provided – Output includes: • Coefficients (b) and levels of significance (pvalue) for each x-value • r value • R2 value © 2011 Jones and Bartlett Publishers, LLC Design of Experiments A scientific approach to improvement © 2011 Jones and Bartlett Publishers, LLC DOE – when evidence is needed • Method for validating processes and determining which factors are most important • Just like in science class – multiple runs, varying “factors” (predictor variables) at different “levels” • Statistically valid approach to identify “main effects” (primary effect of each factor) and “interaction effects” (effects caused by combinations of factors) • Optimization of experiment is desirable to ensure identification of salient factors © 2011 Jones and Bartlett Publishers, LLC