Download Slide 1 - The University of Utah

Situation Awareness Assessment to Determine the Usability of a Graphical Anesthesia
Display During a Simulated Anesthetic: Study Methods
David Lamborn, MSII, Parley Williams, MSII, Noah Syroid, M.S., Dwayne Westenskow, Ph.D.
Department of Anesthesiology, University of Utah, Salt Lake City
Methods and Results (Continued)
Introduction
Potent intravenous sedatives, analgesics and muscle relaxants are administered by
anesthesiologists for induction and maintenance of general anesthesia. Titration of these
anesthetic agents is particularly challenging due to the limited amount of available patient
information, (e.g., blood pressure, heart rate, respiratory rate, and pO2 levels) available to guide
clinicians’ medication and dosing decisions. They have little guidance in terms of visualizing
the pharmacology of the agents in real-time. A presentation of drug pharmacology could be
used by clinicians to anticipate the changes in drug effect-site concentrations and the
corresponding patient responses.
A multidisciplinary research team at the University of Utah has developed a real-time graphical
display showing model-based predictions of drug pharmacology (i.e. pharmacokinetics,
pharmacodynamics and drug-drug synergism) intended for use by anesthesiologists in the
operating room. The intent of the graphic display is to assist clinicians when titrating anesthetic
agents in the operating room that have differing pharmacologic characteristics and complex
interactions (e.g., varying times to reach peak effect, and drug-drug synergism). The foundation
of the display is based upon synergistic drug interaction response surfaces (Figure 3), which
provide model-based predictions of a typical patient’s level of sedation, analgesia and muscle
relaxation. For example, the models are able predict a probability of a typical patient responding
to laryngoscopy as a function of the drug concentrations for a sedative (e.g., propofol) and an
analgesic (e.g., remifentanil, sufentanil or fentanyl). The primary features of the drug display
are illustrated in Figure 1.
Use of displays like the one under study may heighten clinical situation awareness, which
includes an anesthesiologist’s awareness and understanding of the health, sedation and
analgesic status of patient. An anesthesiologist with a high degree of situation awareness is
expected to make more effective decisions in complex, dynamic situations1. There are three
levels of situation awareness2:
Vertical bars represent drugs
given by bolus
Simulation
Colored lines show the effect site
concentration of drugs normalized to their ec50’s. All lines
White
show
dose-effect
appear lines
dashed
in the
future
relationships, including drugdrug synergism
Horizontal bars represent
drugs given by infusion
Gray bands represent probabilities of patient response.
The top band in the middle graph
represents the 50-95% probability of no response to intubation. The smaller band below
it denotes the 50-95% probability of adequate post-op pain
management. In the top graph,
the band depicts the 50-95%
probability of unconsciousness
This is the 50% of T1 line, and
represents the point where the
first twitch in a train of four is expected to lose 50% of its strength.
Figure 1: A screenshot from the actual display. The display is broken into 3 graphs: the top graph is the sedation graph, in
the middle is the analgesia graph, and at the bottom of the display is the NMB graph. Each gives the clinician pharmacokinetic and pharmacodynamic information about their respective class of drugs. The axes of the graphs are labeled
with normalized effect site concentration on the y-axis and time on the x-axis. Note how time is displayed: the break in the
middle of the screen represents the present, everything to the left of the break represents the past (extending 30 minutes),
and everything to the right of the break represents a model-based prediction of future events (extending 10 minutes). The
most salient features of the display are labeled above.
Study methods and research tools were developed to train and evaluate anesthesiologists when
using a PkPd display during a simulated anesthetic.
Figure 2: The patient simulator (METI HPS ver 5.55, Sarasota, FL )
provided a realistic, yet controlled environment for assessing the display’s usability and feasibility. The mannequin was intubated and
connected to two infusion pumps, one with propofol the other with
remifentanyl. Other monitors, an anesthesia machine and a fully
stocked anesthesia cart were used in order to make the simulation
lifelike.
Figure 3: Response surfaces showing predicted drug effect or normalized
probability of patient response to stimuli. The display uses mathematical
algorithms with drug doses as inputs to predict effect site concentrations
and drug effects. The display takes complicated 3-D pharmacologic modeling
and makes it more understandable in an intraoperative setting.
Training
A training tool, a simulation, and an assessment tool was developed. With extensive training
and experience, anesthesiologists have a solid understanding in anesthetic pharmacology. Yet
the typical anesthesiologist has had little exposure to PkPd information (such as that on the
display in Figure 1) that is presented to them in a real-time clinical situation. Thus, it is
necessary to train clinicians prior to their use of the display. A 15 minute scripted PowerPoint
presentation was developed to describe the most salient features in the display. Emphasis was
placed upon the pharmacologic concepts such as how the display graphically shows the
profound synergistic interaction between propofol and an opioid.
During the simulation, study participants were encouraged to use the display in order to make decisions
regarding their administration of anesthesia to the simulated patient. The simulation was designed to be
periodically paused at randomized intervals (standardized across all participants), where investigators
probed the participating anesthesiologists’ situation awareness using the SAGAT.
1. Situation Awareness Assessment: 55 questions were designed for investigators to probe participant
understanding of the most salient features of the display, in context of a realistic simulation. Of the 55
questions 11, 10, and 34 probed level 1, 2 and 3 situation awareness, respectively. To get a question
correct, a participant’s answer would have to agree with predetermined criteria for a correct response.
For example, a typical question assessing level 1 situation awareness was “how is a bolus of propofol
represented on the display?” If a participant’s response made reference to 1) a vertical yellow bar
within 2) the sedation graph, the response was considered correct. Similarly, a typical question
probing level 2 situation awareness was “What is the y-axis measuring on the display?” An example of
a probe used to assess level 3 situation awareness was “According to what you see on the display,
please comment on the patients level of sedation now and where you expect it to be ten minutes into the
future.” Similar criteria were drafted for questions assessing both level 2 and 3 situation awareness that
had to be fulfilled for any response to be scored as correct. Scripted prompts and answers were drafted
to guide participants or train them on any misunderstood components of the display. Correct responses
were given a score of 1 point on a scoresheet with no possibility of partial credit.
One effective means for evaluating new monitors and visualizations is via assessment of
situation awareness obtained by clinicians when using them. The Situation Awareness Global
Assessment Technique (SAGAT)3 utilizes freezes in a simulated scenario where participants
are queried by investigators on their understanding of the clinical situation to ascertain levels of
their situation awareness in context of the display.
Methods and Results
Situation Assessment Tools
Three were designed for use during and after the simulated anesthetic:
1. Level 1: The perception of important factors in the environment
2. Level 2: An understanding what those factors mean
3. Level 3: An ability to predict how the system will behave in the future
The major aim of the present study is to analyze clinician situation awareness yielded by the
SAGAT when using the pharmacokinetic-pharmacodynamic (PkPd) display, and find “holes” in
their understanding that were due to non-intuitive characteristics and features. Identifying areas
in the display that are difficult to understand constitutes an important step in increasing the
display’s safety and usability in an intraoperative setting.
A 45 minute, highly-realistic, scripted simulation of a total intravenous anesthetic was implemented
following the presentation. The simulation was designed to have an anesthesiologist take over an
anesthetic 10 minutes after induction of anesthesia and placement of an endotracheal tube in the patient.
During the simulation, the anesthesiologist had to titrate a sedative (propofol), analgesics (remifentanil
and fentanyl), and a muscle relaxant (rocuronium) for a surgical procedure involving a laparoscopic
hernia repair that changed to an open hernia repair. The simulated anesthetic and surgery was
performed in a realistic mock operating room using a realistic patient simulator (METI HPS ver. 5.55,
Sarasota, FL.) as shown in Figure 2. At predetermined points during the simulation, hemodynamic
variables (e.g., heart rate and blood pressure) were adjusted according to a relative level of painful
stimulus, and the levels of sedation and analgesia titrated by the anesthesiologist. If the display showed
over-sedation/analgesia, the simulated patient would become bradycardic and hypotensive; similarly,
the patient would become tachycardic and hypertensive if the display showed under-sedation/analgesia.
Following the simulation, a NASA Task Load Index (TLX) was administered as well as a short
questionnaire to obtain feedback about the simulation and the strengths, weaknesses, and general
usability of the display. Responses to the questions posed during and after the simulation were scored
and analyzed.
2. NASA-TLX: This standard questionnaire consists of 6 scales of 0-9 (0 being low) and is intended for
the participant to self-assess his/her mental demand, physical demand, temporal demand, perceived
performance, effort and frustration level. The results indicate potential difficulty, challenge, and
frustration with regards to the use of the display
3. Post-simulation Questionnaire: This consisted of 4 scales of 0-9, and 3 open ended questions. Scales
measured participants’ experience levels with the simulated case, attitudes about the realism of the
simulation, and how helpful the display was in managing the case. The questions asked participants to
comment on how the display managed the case, as well as on the strengths and weaknesses of the
display.
Discussion
References
1. Endsley MR: Situation awareness global assessment technique (SAGAT): Air to air tactical version user guide. Hawthorne (CA), Northorp Corp., 1990
2. Endsley MR: Measurement of situation awareness in dynamic systems. Human Factors 1995; 37: 65-843.
3. Endsley MR: Situation awareness analysis and measurement. Manwah (NJ), Lawrence Erlbaum Assoc., Inc., 2000
Training, simulation and research tools have been designed and developed for evaluating
anesthesiologists’ situation awareness. After training and evaluating anesthesiologists, the results from
the SAGAT questions, the NASA-TLX, and the Post-simulation Questionnaire were compiled and
analyzed to identify: 1) the adequacy of the training for the PkPd Display, and 2) the inherent strengths
and weaknesses of the PkPd display when used during a realistic simulation of a general totalintravenous anesthetic. The results of the study will drive changes in the design of the PkPd Display
prior to its implementation into clinical practice.