Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Usability & Human Factors Unit 7: Decision Support Systems: a Human Factors Approach Outline Understanding Human Decision Making Clinical Decision Support Systems (CDSS) Computer Provider Order Entry Systems and CDSS • Promise and Pitfalls • Factors that facilitate and impede CDSS tools Barriers Improving Design Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 2 Patient Safety Landmark Institute of Medicine Report: • 98,000 preventable deaths due to human error • Eighth leading cause of death • Adverse drug events (ADEs) are estimated to injure or kill more than 770 000 people in hospitals annually ADEs: any harm resulting from medication whether due to adverse drug reaction or medical error Complexities of medication management pose a significant safety risk for hospitalized patients. Each phases of medication process, namely prescribing, dispensing, administration, and monitoring, provide opportunities for confusion or error Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 3 Human Factors Approach Focus Goal • Human beings and their interactions with products/equipment, tasks and environments Component 15/Unit 7 • Design systems and system components to match the capabilities and limitations of humans who use them; optimize working and living conditions Health IT Workforce Curriculum Version 1.0/Fall 2010 4 Understanding Decisions Decision involves 3 components: • Choice options & courses of actions • Beliefs about objective states, processes & events in the world, including outcomes states & means to achieve them • Desires, values or utilities that describe the consequences associated with the outcomes of each action-event combination Good decisions effectively choose means available to achieve the goals Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 5 Medical Decision Making Research Two sets of objectives: • 1. Understand how clinicians & patients make decisions in experimental & "realworld" settings • 2. Develop ways to facilitate the decision process (paper-based guidelines, computerassisted decision-support technologies and training in decision methods) Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 6 Heuristics and Biases Heuristics are rules of thumb for making decisions • Intuitive and rapid • Error prone Biases systematic deviations from normative standards • Representativeness • Availability Biases impact the process of decision making & have been well-documented in health-related decisions Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 7 Hindsight Bias Occurs when decision makers inflate the probability of a prior judgment (e.g., diagnose a patient) on the basis of subsequent available information Studies: Estimate the probability of a diagnosis for a given clinical case: 4 potential diagnoses • With/without knowledge of correct diagnosis • Probability judgment should be the same • W knowledge, physicians inflate probability Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 8 Hindsight Bias: So What? Learning • If physicians assume they would have predicted a clinical outcome, they may fail to learn from a case. • Unusual or noteworthy cases presented at grand rounds Error attribution • Given hindsight, errors can seem glaringly obvious • Failure to appreciate the context of an error episode or other mitigating factors Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 9 Confirmation Bias Bayes’ Theorem: Normative theory for hypothesis testing • Probability of a diagnosis is determined by combining prior probability of event with the information value (relative likelihood) of each piece of relevant information Component 15/Unit 7 Confirmation bias: Overconfidence in one’s judgment causes decision maker to favor one hypothesis over another • Selectively attend to data and not give adequate weight to alternatives Health IT Workforce Curriculum Version 1.0/Fall 2010 10 The Cost of Confirmation Bias Desire to confirm a favorite son hypothesis may contribute to inefficiency • Ordering of additional laboratory tests of limited diagnostic values Results of tests serve to reinforce bias (clinician’s confidence) without actually altering likelihood of diagnosis Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 11 Classic DM Problem (Eddy, 1982) • • • • Estimate the probability that a woman has breast cancer given that she has a positive mammogram on the basis of the following information: – The probability that a patient has breast cancer is 1%. (This provides the prior probability) – If the patient has breast cancer, the probability that the radiologist will correctly diagnose it is 80% (This provides the sensitivity or hit rate) – If the patient has a benign lesion (no breast cancer), the probability that the radiologist will misdiagnose it is 9.6% (This provides the false positive rate). What is the probability that a patient with a positive mammogram actually has breast cancer? Probability of breast cancer is only 7.8%, while Eddy reports that 95 out of 100 95 out of 100 doctors estimated this probability to be between greater than 75% Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 12 Framing Effect Alternative representations of a problem can give rise to different judgments and preferences Component 15/Unit 7 Preference for a particular course of action is different when a problem is posed in terms of potential gain rather potential loss even though the underlying situation is identical Health IT Workforce Curriculum Version 1.0/Fall 2010 13 Survival vs. Mortality • McNeil et al (1982) presented a hypothetical lung cancer decision scenario to physicians and patients – The treatment options were radiation therapy, which had an immediate higher survival (lower mortality) rate, but a lower 5 year survival rate. – Frame 1: treatments were described in terms of survival rates – Frame 2: treatments were described in terms of mortality rates • Results: – Survival frame, clear preference for surgery, – Mortality frame, the two choices were preferred almost equally. • One possible explanation is that the positive framing leads to more risk averse choices, while the negative framing increases risk-seeking decision making Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 14 DM in Naturalistic Settings Embedded in a broader social context Decision-action cycle affected by monitoring and feedback High volume and multiple streams of information Situation assessment and serial evaluation of options Changing, ill-defined, or competing goals Substantial stress, time pressure, high risk Multiple players coordinate decisions and actions Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 15 Decision Support Systems Interactive computer-based systems help individuals use communications, data, documents, knowledge and models to solve problems and make decisions DSS are auxiliary systems • Intended to assist human decision makers rather than replace them • Not a fully automated system Designed for specific types of organizations including banks, insurance companies and hospitals Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 16 Clinical Decision Support Systems Provides clinicians, staff, patients, with knowledge and person-specific information, intelligently filtered or presented at appropriate times, to enhance health and health care. • Uses patient data to generate case-specific advice Primary purpose assist clinicians at point of care Designed to aid decision making for prevention, screening, diagnosis, treatment, drug dosing, test ordering, and/or chronic disease management Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 17 Star Trek Tricorder: The Ultimate Clinical Decision Support Tool Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 18 Forms of CDSS Advice Alerts Reminders Structured order forms Pick lists Patientspecific dose checking Guideline support Medication reference information Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 19 The Case for Clinical Decision Support Knowledge base regarding effective medical therapies continues to improve • Practice of medicine continues to lag behind CDS achieve the following objectives: • • • • • • Reduced medication errors and adverse medical events Improved management of specific acute and chronic conditions Improved personalization of care for patients Best clinical practices consistent with medical evidence Cost-effective and appropriate prescription medication use Effective communication and collaboration across clinical/prescribing/dispensing/administering settings • Better reporting and follow-up of adverse events Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 20 Degrees of CDSS Computerization The computer: • Offers no assistance • Offers a complete set of action alternatives • Narrows the selection • Suggests one action alternative • Executes that selection if the human approves Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 21 Degrees of Computerization Continued (6-10) The computer: • Allows human a restricted time to veto before automatic execution • Executes automatically, then informs human • Informs after execution only if asked • Informs him if computer decides to • The computer decides everything and acts autonomously, ignoring the human Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 22 Computerized Provider Order Entry Systems (CPOE) Supports electronic entry of clinical orders for the treatment of patients • Medication • Investigative tests Automate the medication ordering process Orders communicated over a network to the medical staff or to the departments • Pharmacy • laboratory • radiology) responsible for fulfilling the order Typically includes decision support tools Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 23 Promise of Order-Entry Systems Ordering of drugs with computer support is a promising application for reducing medication errors • Most potential adverse events in patients occur at the stage of drug ordering CPOE offers real-time decision support, alerts and reminders Improvements in response time, efficiency of dispensing and delivery of medication Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 24 Some Advantages of CPOE Systems Faster to reach pharmacy Less subject to error associated with similar drug names Easily integrated into medical records and decision-support systems Easily linked to drug-drug interaction warnings Able to link to ADE reporting systems Well suited for training and education Claimed to generate significant economic savings With online prompts, CPOE systems can • Link to algorithms to emphasize cost-effective medications • Reduce under prescribing and over prescribing • Reduce incorrect drug choices Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 25 ADrug-Drug Interaction Scenario “When ordering a new medication, a prescriber may not be aware that two drugs interact, or may not be keeping in mind the other medications that the patient is taking. As an example, consider the case of a hospitalized patient who is being treated with venlafaxine (Effexor) for chronic depression and develops an infection with a drug resistant bacterium requiring treatment with linezolid, a new antimicrobial agent. The interaction between linezolid and venlafaxine (serotonin syndrome -- altered mental status, including agitation, confusion and coma, neuromuscular hyperactivity, and autonomic dysfunction) is very severe but may not be known to the practitioner. While writing the order for linezolid, an alert screen can warn the practitioner that these two drugs should not be used together. The alert screen may offer the prescriber the opportunity to cancel the order, to discontinue the existing medication that interacts with the newly ordered medication, or to order a test that could detect the interaction or monitor therapy. The alert screen may prompt the physician to have a conversation with the patient regarding potential side effects of the medications. Any of these consequences of the decision support software could be beneficial.” Kuperman et al, 2007. Online JAMIA Data Supplement doi: 10.1197/jamia.M2170 J Am Med Inform Assoc 1 January 2007 vol. 14 no. 1 29-40 Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 26 Challenges with Order Entry Steep learning curve Disruption of workflow and organization roles and routines Perceived increase in completion time • Decreases with gains in expertise Can introduce new sources of error Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 27 CPOE Paradox CPOE identified as important intervention to reduce prescribing errors and yet the evidence-base for their effectiveness is limited Some studies have shown electronic prescribing with CPOE significantly increases prescribing quality in hospital inpatients Also introduces new types of errors which have been introduced following CPOE system implementation Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 28 Cognitive Evaluation of Interaction with a CDSS Cognitive evaluation of an interaction with a CPOE system • Focus on decision support for heparin dosing Objective: • characterize the interaction in terms of effectiveness • changes to ordering behavior • opportunities for error attributable to the interaction process Representational format of information in alerts affects performance and errors • Timing of trigger in relation to workflow • (Horsky et al, 2005) Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 29 Weight-Based Heparin Ordering Clinical scenario for the administration of weight-based heparin • Decide on giving bolus followed by IV drip • Calculate both doses Component 15/Unit 7 CPOE triggers decision-support alert • Calculates dose automatically Health IT Workforce Curriculum Version 1.0/Fall 2010 30 Methods Entry of hospital admission orders Cognitive walkthrough • Step-by-step task analysis – goals, actions • Completed by 2 researchers • Characterize information to complete task Component 15/Unit 7 • Completed by 7 clinicians • Subjects instructed to think aloud • Screen progression captured on video • Transcripts of comments coded for analysis Health IT Workforce Curriculum Version 1.0/Fall 2010 31 CPOE Screen Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 32 Weight-based IV Heparin Protocols Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 33 Results - Presentation Salience Recognition of DS purpose Visual salience of calculated values • 2 clinicians confused patient-specific dose calculator with a general guideline • 6 clinician engaged in extended reading of text to derive alert meaning • Values cleared from screen when users needed to enter them – memory recall Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 34 Results - User Behavior Knowledge of DS availability • 1 subject computed both doses before DS was triggered – no indication of DS function Dose estimated before DS • 6 subjects used system-calculated value only as reference – extra time, cognitive effort Transparency of computation • 3 subjects guessed system is using “80/18” formula to compute weight-based dose Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 35 Summary Suboptimal presentation format • Extra time, few realized benefits of DS • Inconsistent with workflow Different representational form would enable a quick perceptual judgment could reduce this extra cognitive effort. • Leaving only the calculated dose in the frame with a clear description of how the result was computed • To take advantage, user needs to be able to invoke this feature on demand. • May identify inefficient and error-prone screen configurations Propose specific interface improvement • Higher rate of decision support use Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 36 36 Role of CPOE Systems in Facilitating Medical Errors Koppel et al (2005) conducted multi-faceted study investigating medication errors associated with CPOE use CPOE system facilitated 22 types of medication error. • Many appeared to occur with great frequency Errors classified into: • Information errors generated by fragmentation of data and failure to integrate the hospital’s information systems • human-machine interface flaws reflecting machine rules that do not correspond to work organization or usual behaviors Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 37 Information Errors: Fragmentation and Systems Integration Failure 1 Assumed Dose Information • House staff rely on CPOE displays to determine minimal effective or usual doses. • dosages listed in display based on the pharmacy’s warehousing and not clinical guidelines. • For example, normal dosages are 20 or 30 mg, the pharmacy might stock only 10-mg doses, so 10-mg units are displayed on the CPOE screen • Clinicians select inappropriate doses Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 38 Information Errors: Fragmentation and Systems Integration Failure 2 Medication Discontinuation Failures • Ordering new or modifying existing medications is usually a separate process from canceling • Without discontinuing the current dose, physicians can increase or decrease medication • Add new but duplicative medication • Medication-canceling ambiguities exacerbated by interface and multiple-screen displays of medications • Viewing 1 patient’s medications may require 20 screens Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 39 Human-Machine Interface Flaws Patient Selection • Easy to select wrong patient file because names and drugs are close together, the font is small, and, patients’ names do not appear on all screens. Unclear Log On/Log Off • Physicians can order medications at computer terminals not yet "logged out" by the previous physician • result in either 1) unintended patients receiving medication or 2) patients not receiving the intended medication Failure to Provide Medications After Surgery • When patients undergo surgery, CPOE cancels their previous medications. • Physicians must reenter CPOE and reactivate each previously ordered medication Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 40 Automation Bias Clinicians using automated decision follows the aid directs regardless of knowledge to the contrary Errors of Omission: • Less vigilant in checking drug orders because they assume the computer will have already done the work Errors of Commission: • Continue with a dangerous drug order because the computer did not alert them that the order was unsafe Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 41 Anti-Automation Bias • Errors of dismissal, where computer advice is ignored • Clinicians routinely disable or ignore the alarms or alerts on clinical monitoring devices – Legitimate reasons such as high false alarm rates [or repetition of the same alarms – Less valid reasons such as not wanting to be interrupted Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 42 Barriers to Prescriber DecisionMaking and Clinical Workflow Russ and colleagues (2009) observed medication prescribing during routine patient care Through inductive qualitative analysis, they identified 15 barriers associated with medication alerts Facilitate medication ordering by improving the style and content of the medication alert display Next 2 pages describes 11 such barriers Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 43 15 Barriers to Prescriber Decision-Making BARRIER DESCRIPTION 1. Poor Screen Display Alert display does not support alert resolution and/or prescriber workflow 2. Inadequate Alert Specification Alert does not provide information on why it was triggered and/or the potential problem 3. Actual or Perceived Lack of Evidence Alert is not evidence-based, does not provide a reference to evidence that does exist 4. Unclear Level of Risk Alert does not provide clear information on relative risk of harm for a given patient 5. Redundancy Repeated alerts within the same encounter or over multiple encounters for a given patient Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 44 Barriers Continued 6. Low Alert Signal to Noise Ratio Numerousness of alerts leads to information overload, prescriber desensitization, and potential for missing key alerts 7. Inadequate Allergy Logic Alert system does not distinguish between true allergies and bothersome, but non-serious, side effects 8. Duplicate Workload Alert duplicates other required work processes 9. Paper Prescriptions & Limited CPOE Some medications are not or cannot be entered electronically, and therefore, are not reviewed by the alert system 10. Unclear System Alert system does not adequately reveal its Capabilities capabilities/limitations to the prescriber; full functionality of the alert system is ambiguous Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 45 Barriers Elaborated Poor Screen Display • Much of the alert text was in all capital letters • multiple alerts grouped in one pop-up window • Scroll box to view multiple alerts Inadequate Alert Specification • alerts did not show all clinically-relevant information needed for decision-making Unclear Level of Risk • Significance of alert may be unknown (e.g., drug allergy versus sensitivity). • Difficult decision whether to discontinue course of treatment Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 46 Human Factors and Information Management Situation Awareness • What must be known in order to complete a cognitive task Mental Workload • High MWL occurs when a person’s mental capacity is exceeded • when the mental demands imposed on the clinician because of information overload, for example, exceed the clinician’s ability to keep it all straight. Poor SA and high MWL impair memory, problem identification and decision making Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 47 Situation Awareness Construct used in decision making research to characterize: • awareness of what is happening around you • and understanding what the information means to you now and in the future Three levels • Perception of elements in the environment (e.g., cues/stimuli from patient [pulse, color, weight change], chart, EHR, nurse) • Comprehension of the meaning of those elements (by integrating the disparate pieces of information and determining what is salient) • Projection of future status so that decisions can be made If SA is poor, results in impaired decision making Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 48 Mental Workload As information management problems increase, MWL increases Time pressure makes it more important that CDS automation be easy to use and useful • Under time pressure with less time and patience to navigate through poorly designed technology • Under time pressure, people can adapt and still perform well by exerting more mental effort or by concentrating harder • Under more significant mental workload, individuals can no longer adapt or compensate in order to maintain cognitive performance • Demands imposed by the system (e.g., clinician needing to remember the important facts of the most recent patient visit while starting the next patient’s visit) exceed the attentional resources or mental capacity of the person Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 49 CPOE/CDSS Design Recommendations Interfaces explicitly map to workflow patterns of clinicians • CDSS systems must support, rather than impede, clinical workflows through speedy, available, and usable algorithms that provide parsimonious, clear, concise, and actionable warnings and advice Clues in interface to optimally support users in medication ordering Reduce layers of screens (to a maximum of 3 layers) to facilitate users navigation Alerts should be timed properly-the moment a clinician would himself search for this information Alerts displayed in a more prominent position on the screen Organize screen-elements into logical groups, visually separated by space and alignment Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 50 50 Designing for Better Workflow All information should be available in a clinical information system Clinical systems should help clinicians to see the right amount of the right type of data wherever and whenever needed Clinical information should be accessible in the shortest possible amount of time Data from disparate sources should be aggregated for completeness • Clinicians are not forced to go to multiple different systems to obtain important information Clinical systems should reduce to a reasonable minimum the number of steps required to obtain any information Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 51 Concluding Thoughts • Computer-based decision support systems offer great promise for reduction of errors in medicine and facilitation of quality patient care • Results on system efficacy have thus far been equivocal – CDSS can lead to new types of errors • Adherence to usability/human factors principles can lead to superior design and enhanced performance Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 52 References Coiera E, Westbrook J, Wyatt J. The safety and quality of decision support systems. Methods Inf Med. 2006;45 Suppl 1(suppl 1):20–5.me06010020 Eddy, D. M. (1982). Probabilistic reasoning in clinical medicine: Problems and opportunities. In D. Kahneman, P. Slovic & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases (pp. 249-267). Cambridge, England: Cambridge University Press. Horsky, J., Kaufman, D. R., & Patel, V. L. (2005). When you come to a fork in the road, take it: strategy selection in order entry. AMIA Annu Symp Proc, 350-354. Institute of Medicine Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press, 2001. J.A. Osheroff, J.M. Teich, B. Middleton, E.B. Steen, A. Wright and D.E. Detmer, A roadmap for national action on clinical decision support, J Am Med Inform Assoc 14 (2) (2007), pp. 141–145. Karsh B-T. Clinical practice improvement and redesign: how change in workflow can be supported by clinical decision support. AHRQ Publication No. 09-0054EF; Rockville (MD): Agency for Healthcare Research and Quality; June 2009. Khajouei R, Jaspers MW: CPOE system design aspects and their qualitative effect on usability. Stud Health Technol Inform 2008 , 136:309-14 Koppel, R., Metlay, J. P., Cohen, A., Abaluck, B., Localio, A. R., Kimmel, S. E., et al. (2005). Role of computerized physician order entry systems in facilitating medication errors. JAMA, 293(10), 1197-1203. Kuperman, G. J., Bobb, A., Payne, T. H., Avery, A. J., Gandhi, T. K., Burns, G., et al. (2007). Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc, 14(1), 29-40. McNeil, B. J., Pauker, S. G., Sox, H. C., Jr., & Tversky, A. (1982). On the elicitation of preferences for alternative therapies. N Engl J Med, 306(21), 1259-1262. Reckmann, M. H., Westbrook, J. I., Koh, Y., Lo, C., & Day, R. O. (2009). Does computerized provider order entry reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc, 16(5), 613-623. Russ, A. L., Zillich, A. J., McManus, M. S., Doebbeling, B. N., & Saleem, J. J. (2009). A human factors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow. AMIA Annu Symp Proc, 2009, 548552. Teich JM, Osheroff JA, Pifer EA, Sittig DF, Jenders RA, Panel CDSER. Clinical decision support in electronic prescribing: Recommendations and an action plan. J Am Med Inform Assoc 2005;12(4):365-76. Component 15/Unit 7 Health IT Workforce Curriculum Version 1.0/Fall 2010 53