Download the full sized image

Asking a Great Question: A Librarian Teaches Questioning Skills to First Year Medical
Students
Nancy E. Adams
ABSTRACT. In a single one-hour session, first-year medical students were taught a framework
for differentiating between lower-order questions that lead to knowledge of facts and higherorder questions that lead to integration of concepts and deeper learning, thereby preparing them
for problem-based learning (PBL). Students generated lists of questions in response to an
assertion prompt and categorized them according to Bloom’s Taxonomy. These data were
analyzed in addition to data from the course exam, which asked them to formulate a higher-level
question in response to a prompt. Categorizing questions according to Bloom’s Taxonomy was a
more difficult task for students than was formulating higher-order questions. Students reported
that the skills that they learned were used in subsequent PBL sessions to formulate higher-order
learning objectives that integrated new and previously-learned concepts.
KEYWORDS.
Bloom’s taxonomy, critical thinking, questions, medical students, problem based learning
Author.
Nancy E. Adams, MLIS ([email protected]) is Associate Director / Coordinator of
Education & Instruction, George T. Harrell Health Sciences Library, Penn State Hershey, 500
University Dr., Mail Code H127, Hershey, PA 17033-0850.
Running head: “Asking a Great Question”
1
INTRODUCTION
The ability to ask questions is an important skill for medical students and is considered a key to
critical thinking.1 Asking good questions motivates learning and increases engagement because
the learners identify their own educational needs and interests.2 The ideal future physician is
inquisitive and curious about others’ thinking and her own—in other words, approaches her
environment with an attitude of questioning.3 The type of question asked drives learning.
Questions asked about factual details will lead to factual knowledge, whereas questions that
involve analysis and require the learner to integrate concepts that were previously thought to be
unrelated can lead to deeper conceptual learning. Because of this, some have even proposed that
the medical education reward system be changed so that learners who ask the best questions are
recognized, rather than those who give the best answers.4
How learners ask questions is a topic of interest to medical librarians. One of the medical
librarians’ professional tasks is to help library users express their information needs in the form
of questions, which can be considered a tool for finding and then delving into the literature on a
topic. The ability to ask a well-defined question is an important part of information literacy and
is addressed by information literacy learning objectives and standards published by the
Association for College and Research Libraries (ACRL): “The information literate student
defines and articulates the need for information.”5 Question formulation skills are subsumed in
the ACRL Framework for Information Literacy for Higher Education in the “Research as
Inquiry” and “Searching as Strategic Exploration” threshold concepts.6
2
However, although the ability to ask questions is a critical skill for medical students, little
has been published in the journal literature on specific ways to teach medical students how to ask
questions for learning. Much has been published on formulating an answerable clinical question,
and teaching this skill is a feature of most evidence-based medicine courses.7 However, the
purpose of such questions is for clinical decision making, and they may be qualitatively different
from questions that are asked for the purpose of gaining knowledge and insight for medical
school success, such as to stimulate learning in problem-based learning sessions.
Educational interventions teaching third-year medical students how to ask better
questions in a clinical context have been described, showing that students could be trained to ask
open-ended rather than closed questions and that training in asking questions improved critical
thinking skills.8,9 Wolpaw, Côté, Papp, and Bordage show that when students ask questions of
their preceptors during case presentations, they elicit just-in-time teaching focused on their
uncertainties.10 These studies are all related to learning in a clinical context and do not address
learning such as that taking place in the problem-based learning (PBL) group. Other authors
have described interventions in which students write questions for written examinations, but
exam questions may also be qualitatively different than questions formulated for the purpose of
stimulating dialog within problem-based learning groups or for guiding self-directed learning.11,
12
The purpose of this project was to teach first-year medical students a framework for
differentiating between lower-order questions that lead to knowledge of facts and higher-order
questions that lead to integration of concepts and deeper learning, thereby preparing them for
PBL. Active learning principles were incorporated with the goal of increasing student
engagement in the session.
3
METHODS
The session was part of the Profession of Medicine course at the Penn State College of Medicine,
a required course for all first-year medical students during their first two weeks of medical
school. The author was invited to serve on a small team of faculty and students to design the
course due to her involvement in the medical curriculum teaching principles related to evidencebased medicine—a domain highly related to critical thinking—and an academic background in
the principles of adult learning. The goal of the course was to introduce medical students to key
competencies for success in medical school, including teamwork, professionalism, quality
improvement, self-directed learning (SDL) and problem-based learning (PBL) strategies, and
self-care. The “Asking a Great Question” session was part of a half-day session devoted to SDL
and critical thinking skills in 2013 (the inaugural year of the course) and 2014; the session was
largely unchanged between the two years. Students, other faculty members, and the librarian
perceived that the session was successful in 2013, but no data was collected as evidence;
therefore, in 2014 the decision was made to collect data in order to measure the effectiveness of
the session in helping learners ask higher order questions using Bloom’s Taxonomy and to see
how this knowledge was applied to later coursework such as PBL sessions.
The setting of the session was a lecture hall with approximately 145 students in
attendance. Students were assigned a brief pre-reading, “Asking a Great Question,” which was
posted to the course management system. The document described the qualities of open
questions, which require complex answers beyond the citing of facts, and closed questions,
which can easily be answered with a yes or no statement or a specific piece of information. It
4
included information on Bloom’s Taxonomy of Cognitive Learning Objectives, a well-known
framework in education which presents cognitive skills in order of complexity: knowledge,
comprehension, application, analysis, synthesis, and evaluation.13 (A later revision of the
Taxonomy adds a dimension describing type of knowledge and presents the skills in order as:
remember, understand, apply, analyze, evaluate, create; the earlier version was used for this
exercise since it is more widely used.14)
Bloom’s Taxonomy begins with knowledge, the retention of facts, and progresses to
analysis, ability to compare and contrast concepts or break a concept down into components, and
then finally to evaluation, the ability to make judgments about quality and prioritize. Bloom’s
Taxonomy can also be considered a guide to the types of questions that learners could ask when
identifying knowledge gaps for self-directed learning. For example, “Where is the saphenous
vein located?” is an example of a closed question that might be categorized as a knowledge
question according to Bloom’s Taxonomy, whereas “What are the results of damage to the
saphenous vein?” is an example of a more complex, open question that addresses the skill of
analysis and would lead to deeper learning of concepts. A chart of the Taxonomy listing the
skills and examples of questions that a physician might ask in clinical practice utilizing each skill
was included along with several other practice questions that students could categorize according
to Bloom’s Taxonomy.
Since the students prepared by reviewing content before entering the classroom, the onehour large-group lecture session was used for group learning activities and discussion. After brief
introductory remarks, students formed groups of three to four students and were presented with
this assertion to consider: Medical errors are usually attributed to personal failure, but are often
the result of systems failure. Other statements could be used; this statement was selected because
5
it addressed health care quality improvement, one of the themes of the Profession of Medicine
course, and it foreshadowed a PBL case about causes of medical errors that they would receive
the following week. Groups were given approximately seven minutes to brainstorm and record
questions about the medical error statement described above. Groups recorded questions
verbatim, without discussion or judgment on any of the questions. Questions in the form of an
interrogative statement beginning with action verbs such as describe, compare, and justify could
also be listed. Students applied their knowledge of Bloom’s Taxonomy by categorizing the
questions on their lists as open or closed and deciding which category each question fit in the
Taxonomy, noting particularly the balance between lower-order questions (knowledge,
comprehension, and application) and higher-order questions (analysis, synthesis, and evaluation).
Examples of questions that groups recorded for each level of Bloom’s Taxonomy are as follows:

Knowledge: What is a medical error?

Comprehension: Compare and contrast personal failure and systems failure.

Application: Describe three examples of medical errors and their consequences,
differentiating between personal and systemic errors.

Analysis: If a physician performs wrong-site surgery, what are the possible sources of
personal failure and systems error?

Synthesis: Describe a plan to prevent system failures.

Evaluation: Justify why systems errors contribute more toward medical errors than does
personal failure.
As described by Bloom’s Taxonomy, questions differ in the knowledge and skills
required to answer them. It is also important to consider that questions serve different purposes,
such as research questions, questions to drive self-directed learning such as in problem-based
6
learning groups, or for making clinical practice decisions. Since the audience was a group of
newly entering medical students, each group was then asked to consider all of the questions on
its list and decide which of those questions were most likely to generate the most useful learning
for success in medical school, if pursued in the context of a problem-based learning session
about medical errors. Groups were then encouraged to trade lists to see the variety of questions
generated by others.
Following the small-group learning activities, the librarian led a large-group discussion to
further consider the value of different types of questions for learning. Students were asked to use
an audience response system to vote on which of four specific questions would lead to the
deepest learning for the same “medical error” scenario as above. Respondents explained their
reasoning to the group. Other questions considered in group discussion were: Did you notice a
preference for lower-order or higher-order questions, and why? Are higher-order questions more
important than lower-order questions? Must lower-order questions be addressed before moving
on to higher-order questions? What new insights about group processes related to asking
questions were gained during this exercise? Students engaged in a lively discussion.
RESULTS
Data were collected to assess the impact of the 2014 session on first-year medical students’
ability to use Bloom’s Taxonomy to ask higher-order questions. Approval was obtained from the
Penn State College of Medicine Institutional Review Board for all data collection.
The lists of questions generated by groups during the session were collected and entered
into a spreadsheet. Forty-one groups submitted question lists with an average of 11.8 questions
7
submitted per group (range, 5-35). The questions that each group chose as the “best for the
purposes of learning in medical school” were analyzed. Groups were asked to vote on their top
three questions of this type, but not all groups submitted three questions. The questions were
categorized—115 in all‒ according to Bloom’s Taxonomy using the detailed descriptions found
in Bloom’s original work.13 The questions were also compared with lists of key words or action
verbs associated with cognitive objectives for each level of the taxonomy. Lists of this type are
widely available on the web.15 This process was repeated several times for each question on the
list to achieve confidence in the analysis. The questions could be categorized as following,
according to Bloom’s Taxonomy: 28 knowledge questions; 30 comprehension questions; 2
application questions; 15 analysis questions; 25 synthesis questions; and 13 evaluation questions.
Two were characterized as too vague to determine. Of this “best for learning” subset of
questions, students categorized 79 of them according to Bloom’s Taxonomy, but there was
agreement between students’ and the librarian’s categorizations for only 28 of those questions
(35%).
A summative assessment at the end of the two-week course included a brief case scenario
on a patient with sickle cell crisis as a prompt for the exam question: “Formulate two concise
questions for this case that would lead to higher-order learning outcomes according to Bloom’s
Taxonomy.” The 136 student responses were compared with Bloom’s Taxonomy using the same
method as above to look for higher-order questions—those aimed at a response of analysis,
synthesis, or evaluation. For example, correct student responses could show evidence of seeking
relationships between two separate entities, such as between disease and symptom or evidence of
applying general principles to the specific patient in the case. Incorrect responses were those
requiring only knowledge, comprehension, or application to answer, such as any questions about
8
signs and symptoms of sickle cell crisis in general or side effects of medications, for example.
Students received one point for each higher-order question written. The average score was 1.8
out of 2.0 (range 0-2).
Longer-term learning outcomes of the intervention were also assessed using students’
self-report of application of the skills in a subsequent course. Approximately 14 weeks following
the intervention, students received a course evaluation survey at the end of their Scientific
Principles of Medicine (SPM) course in which the following optional survey question was asked:
In your Profession of Medicine course during the first week of medical school, you
participated in a session, “Asking a Great Question” in which various types of questions
were discussed and categorized into higher and lower order questions. Can you
remember a specific instance during the SPM course, perhaps during a PBL session,
when you recalled and applied the knowledge and skills that you learned about question
asking? If so, please describe what happened and how you applied the question-asking
knowledge and skills.
Responses to this open-ended question were assessed qualitatively to look for themes.
Out of 118 students who submitted a course evaluation survey, 62 students responded to this
question, for a 53% response rate. Of the 62 respondents, 38 students (61%) described the
question-asking session as useful for their learning in their subsequent science course. The other
24 students (39%) who responded indicated that they could not remember a time when they
applied the question-asking knowledge or skills, that it was not useful, or marked the question as
“N/A” (not applicable).
Respondents indicated that they applied the knowledge and skills learned during the
“Asking a Great Question” session several ways. Of the respondents, 25 (40%) stated that
9
question-asking skills were used by their problem-based learning (PBL) group to formulate
“higher-order” learning objectives for group members to pursue independently. Eleven
respondents (18%) indicated that they applied the skills on their own, during their personal study
time or in their individual research for PBL. Two respondents (3%) stated that they used what
they learned about asking “higher-order” questions in finding relevant journal articles to share
with their PBL group about case concepts, and two respondents (3%) found what they learned
useful during group study for the course. Some respondents indicated that it was useful in more
than one category.
Responses were also scrutinized for descriptions of how the knowledge and skills gained
during the session helped them in their group and individual study. The strongest theme from this
analysis was connecting and integrating: students described the utility of higher-order questions
in helping them to integrate their learning across topics and over time in the course. Some
representative responses supporting the theme of higher-order questions as tools for connecting
and integrating concepts are as follows:
In reviewing lecture content, I tried to come up with questions I would write if I were
writing the exam. A lot of those questions eventually became higher order questions
comparing concepts I saw multiple times throughout the course (e.g., methotrexate in the
metabolism and cancer topic weeks). I think our PBL group did the same thing with some
concepts that could be compared across the weeks (e.g., relating back to the myasthenia
gravis case during a later discussion of autoimmunity).
Yes—our PBL group always strove for higher order questions to the point of becoming
almost frustrating. However, because we were challenged to think of the material in a
10
different way (e.g., how all the information connects together for a higher order
question), I think we learned more about the material overall.
Instead of discussing a neuromuscular junction, we started asking questions and
discussing what could go wrong and how it would affect movement if anything goes
wrong during signal transmission, such as impaired ACh vesicle release, adding ACh
esterase suicide inhibitor.
Rather than coming up with [learning objectives] that merely asked for definition, we
were able to devise more detailed questions that helped to connect the clinical relevance
with the hard science.
DISCUSSION
The question-asking session was conceived, designed and taught by a librarian as part of a course
providing entrée to medical school for first-year students. This project serves as an example of
how medical students can be taught the skill of asking higher-order questions that can help them
to connect concepts and achieve deeper learning. Critical thinking skills such as asking questions
are highly related to the evidence-based medicine skills and information literacy skills that health
sciences librarians are already involved in teaching, so extending librarians’ teaching
responsibilities to the critical thinking domain, while challenging, is appropriate.
Results of the assessments showed that categorizing questions according to Bloom’s
Taxonomy was a more difficult task for students than was formulating a “higher-order” question.
11
This was shown by their success on the exam question, with almost all students able to write at
least one higher-level question, compared to the discrepancy between the students’ and the
librarian’s categorizations of questions. While the exercise of categorizing questions can help
students practice differentiating various types of questions, the ability to formulate “higherorder” questions is more important, as it is a skill that is of immediate application to learning
throughout medical school and beyond. Students showed more proficiency in this task, although
this study does not provide evidence that the proficiency was a result of the intervention.
The qualitative data from the free-response question clarifies why the intervention was
successful for those students who did use the skills and knowledge learned: its utility in
formulating learning objectives in PBL and in asking questions that lead to integration of
learning. This student-reported data links the skills learned in the session with their application to
PBL cases. Many students reported that asking higher-order questions helped them to connect
seemingly disparate concepts across scientific and clinical domains and over time in the
curriculum. Integration of learning is important for medical students who must build a coherent
knowledge base from a voluminous amount of information that they encounter in the curriculum.
There are several limitations to this study. Data from a single session is presented here;
data should be collected from future groups of learners who participate in a similar session. The
qualitative data gauging application of these skills in PBL sessions are based on student selfreport only; collecting data from PBL facilitators regarding their observations of student
questioning behaviors would have strengthened the study. Finally, future research studies with
randomized control and intervention groups and a pre-test administered would be required to
show that this intervention does indeed lead to a statistically and educationally significant
improvement in students’ question-asking ability and overall performance in PBL.
12
CONCLUSION
PBL is a widely used modality in many medical schools, but new medical students may need
coaching in how to approach learning in the self-directed manner that PBL requires. Research
shows that first-year medical students’ independent study time related to PBL is largely confined
to researching the learning issues that are generated during PBL sessions.16 Therefore, first-year
medical students’ report that their PBL groups used the knowledge and skills taught during the
intervention to devise higher-order learning objectives that integrated new and previous concepts
(such as analysis, synthesis, and evaluation questions) is an important finding. Because the skills
were used to formulate better learning objectives (i.e., ask better questions), this intervention
might serve to improve overall learning in PBL for first-year medical students.
REFERENCES
1. Krupat, Edward, Jared M. Sprague, Daniel Wolpaw, Paul Haidet, David Hatem, and Bridget
O'Brien. “Thinking Critically About Critical Thinking: Ability, Disposition or Both?”
Medical Education 45, no. 6 (June 2011): 625-635.
2. Wolpaw, Terry M., DanielR. Wolpaw, and KlaraK. Papp. “SNAPPS: A Learner-centered
Model for Outpatient Education.” Academic Medicine 78, no. 9 (September 2003): 893-898.
3. Papp, Klara K., Grace C. Huang, Laurie M. Lauzon Clabo, et al. “Milestones of Critical
Thinking: A Developmental Model for Medicine and Nursing.” Academic Medicine 89, no. 5
(May 2014): 715-20.
13
4. Huang, Grace C., Laurie R. Newman, and Richard M. Schwartzstein. “Critical Thinking in
Health Professions Education: Summary and Consensus Statements of the Millennium
Conference 2011.” Teaching and Learning in Medicine 26, no. 1 (January 2014): 95-102.
5. Association of College and Research Libraries. Information Literacy Competency Standards
for Higher Education. January 2000.
http://www.ala.org/acrl/sites/ala.org.acrl/files/content/standards/standards.pdf.
6. Association of College and Research Libraries. Framework for Information Literacy for
Higher Education. February 2, 2015. http://www.ala.org/acrl/standards/ilframework.
7. Maggio, Lauren A., Nancy H. Tannery, H. Carrie Chen, Olle ten Cate, and Bridget O'Brien.
“Evidence-based Medicine Training in Undergraduate Medical Education: A Review and
Critique of the Literature Published 2006-2011.” Academic Medicine 88, no. 7 (July 2013):
1022-8.
8. Egan, Mari E. “Soliciting Feedback by Asking Questions That Promote Thinking Among
Medical Students: A Pilot Study” (Master’s thesis, University of Illinois at Chicago, 2001).
9. Loy, Gary. L., Mark. H. Gelula, and Louis A. Vontver. “Teaching Students To Question.”
American Journal of Obstetrics and Gynecology 191, no. 5 (November 2004): 1752-6.
10. Wolpaw, Terry, Luc Côté, Klara K. Papp, and Georges Bordage. “Student Uncertainties
Drive Teaching During Case Presentations: More So With SNAPPS.” Academic Medicine
87, no. 9 (September 2012): 1210-1217.
11. Kolluru, Srikanth. “An Active-Learning Assignment Requiring Pharmacy Students to Write
Medicinal Chemistry Examination Questions.” American Journal of Pharmaceutical
Education 76, no. 6 (August 2012): 112.
14
12. Papinczak, Tracey, Awais S. Babri, Ray Peterson, Vaughan Kippers, and David Wilkinson.
“Students Generating Questions for Their Own Written Examinations.” Advances in Health
Sciences Education: Theory and Practice, 16, no. 5 (December 2011): 703-710.
13. Bloom, Benjamin. S. Taxonomy of Educational Objectives: The Classification of
Educational Goals. New York: David McKay Company, 1956.
14. Anderson, Lorin W., and David R. Krathwohl. A Taxonomy for Learning, Teaching, and
Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. New York:
Longman, 2001.
15. Center for Teaching and Learning, University of North Carolina-Charlotte. “Writing
Objectives Using Bloom’s Taxonomy.” Accessed April 20, 2015.
http://teaching.uncc.edu/learning-resources/articles-books/best-practice/goalsobjectives/writing-objectives.
16. Dolmans, Diana, Henk G. Schmidt, and Wim H. Gijselaers. “The Relationship Between
Student-generated Learning Issues and Self-study in Problem-based Learning.” Instructional
Science 22, no. 4 (July 1994): 251-267.
15