Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Asking a Great Question: A Librarian Teaches Questioning Skills to First Year Medical Students Nancy E. Adams ABSTRACT. In a single one-hour session, first-year medical students were taught a framework for differentiating between lower-order questions that lead to knowledge of facts and higherorder questions that lead to integration of concepts and deeper learning, thereby preparing them for problem-based learning (PBL). Students generated lists of questions in response to an assertion prompt and categorized them according to Bloom’s Taxonomy. These data were analyzed in addition to data from the course exam, which asked them to formulate a higher-level question in response to a prompt. Categorizing questions according to Bloom’s Taxonomy was a more difficult task for students than was formulating higher-order questions. Students reported that the skills that they learned were used in subsequent PBL sessions to formulate higher-order learning objectives that integrated new and previously-learned concepts. KEYWORDS. Bloom’s taxonomy, critical thinking, questions, medical students, problem based learning Author. Nancy E. Adams, MLIS ([email protected]) is Associate Director / Coordinator of Education & Instruction, George T. Harrell Health Sciences Library, Penn State Hershey, 500 University Dr., Mail Code H127, Hershey, PA 17033-0850. Running head: “Asking a Great Question” 1 INTRODUCTION The ability to ask questions is an important skill for medical students and is considered a key to critical thinking.1 Asking good questions motivates learning and increases engagement because the learners identify their own educational needs and interests.2 The ideal future physician is inquisitive and curious about others’ thinking and her own—in other words, approaches her environment with an attitude of questioning.3 The type of question asked drives learning. Questions asked about factual details will lead to factual knowledge, whereas questions that involve analysis and require the learner to integrate concepts that were previously thought to be unrelated can lead to deeper conceptual learning. Because of this, some have even proposed that the medical education reward system be changed so that learners who ask the best questions are recognized, rather than those who give the best answers.4 How learners ask questions is a topic of interest to medical librarians. One of the medical librarians’ professional tasks is to help library users express their information needs in the form of questions, which can be considered a tool for finding and then delving into the literature on a topic. The ability to ask a well-defined question is an important part of information literacy and is addressed by information literacy learning objectives and standards published by the Association for College and Research Libraries (ACRL): “The information literate student defines and articulates the need for information.”5 Question formulation skills are subsumed in the ACRL Framework for Information Literacy for Higher Education in the “Research as Inquiry” and “Searching as Strategic Exploration” threshold concepts.6 2 However, although the ability to ask questions is a critical skill for medical students, little has been published in the journal literature on specific ways to teach medical students how to ask questions for learning. Much has been published on formulating an answerable clinical question, and teaching this skill is a feature of most evidence-based medicine courses.7 However, the purpose of such questions is for clinical decision making, and they may be qualitatively different from questions that are asked for the purpose of gaining knowledge and insight for medical school success, such as to stimulate learning in problem-based learning sessions. Educational interventions teaching third-year medical students how to ask better questions in a clinical context have been described, showing that students could be trained to ask open-ended rather than closed questions and that training in asking questions improved critical thinking skills.8,9 Wolpaw, Côté, Papp, and Bordage show that when students ask questions of their preceptors during case presentations, they elicit just-in-time teaching focused on their uncertainties.10 These studies are all related to learning in a clinical context and do not address learning such as that taking place in the problem-based learning (PBL) group. Other authors have described interventions in which students write questions for written examinations, but exam questions may also be qualitatively different than questions formulated for the purpose of stimulating dialog within problem-based learning groups or for guiding self-directed learning.11, 12 The purpose of this project was to teach first-year medical students a framework for differentiating between lower-order questions that lead to knowledge of facts and higher-order questions that lead to integration of concepts and deeper learning, thereby preparing them for PBL. Active learning principles were incorporated with the goal of increasing student engagement in the session. 3 METHODS The session was part of the Profession of Medicine course at the Penn State College of Medicine, a required course for all first-year medical students during their first two weeks of medical school. The author was invited to serve on a small team of faculty and students to design the course due to her involvement in the medical curriculum teaching principles related to evidencebased medicine—a domain highly related to critical thinking—and an academic background in the principles of adult learning. The goal of the course was to introduce medical students to key competencies for success in medical school, including teamwork, professionalism, quality improvement, self-directed learning (SDL) and problem-based learning (PBL) strategies, and self-care. The “Asking a Great Question” session was part of a half-day session devoted to SDL and critical thinking skills in 2013 (the inaugural year of the course) and 2014; the session was largely unchanged between the two years. Students, other faculty members, and the librarian perceived that the session was successful in 2013, but no data was collected as evidence; therefore, in 2014 the decision was made to collect data in order to measure the effectiveness of the session in helping learners ask higher order questions using Bloom’s Taxonomy and to see how this knowledge was applied to later coursework such as PBL sessions. The setting of the session was a lecture hall with approximately 145 students in attendance. Students were assigned a brief pre-reading, “Asking a Great Question,” which was posted to the course management system. The document described the qualities of open questions, which require complex answers beyond the citing of facts, and closed questions, which can easily be answered with a yes or no statement or a specific piece of information. It 4 included information on Bloom’s Taxonomy of Cognitive Learning Objectives, a well-known framework in education which presents cognitive skills in order of complexity: knowledge, comprehension, application, analysis, synthesis, and evaluation.13 (A later revision of the Taxonomy adds a dimension describing type of knowledge and presents the skills in order as: remember, understand, apply, analyze, evaluate, create; the earlier version was used for this exercise since it is more widely used.14) Bloom’s Taxonomy begins with knowledge, the retention of facts, and progresses to analysis, ability to compare and contrast concepts or break a concept down into components, and then finally to evaluation, the ability to make judgments about quality and prioritize. Bloom’s Taxonomy can also be considered a guide to the types of questions that learners could ask when identifying knowledge gaps for self-directed learning. For example, “Where is the saphenous vein located?” is an example of a closed question that might be categorized as a knowledge question according to Bloom’s Taxonomy, whereas “What are the results of damage to the saphenous vein?” is an example of a more complex, open question that addresses the skill of analysis and would lead to deeper learning of concepts. A chart of the Taxonomy listing the skills and examples of questions that a physician might ask in clinical practice utilizing each skill was included along with several other practice questions that students could categorize according to Bloom’s Taxonomy. Since the students prepared by reviewing content before entering the classroom, the onehour large-group lecture session was used for group learning activities and discussion. After brief introductory remarks, students formed groups of three to four students and were presented with this assertion to consider: Medical errors are usually attributed to personal failure, but are often the result of systems failure. Other statements could be used; this statement was selected because 5 it addressed health care quality improvement, one of the themes of the Profession of Medicine course, and it foreshadowed a PBL case about causes of medical errors that they would receive the following week. Groups were given approximately seven minutes to brainstorm and record questions about the medical error statement described above. Groups recorded questions verbatim, without discussion or judgment on any of the questions. Questions in the form of an interrogative statement beginning with action verbs such as describe, compare, and justify could also be listed. Students applied their knowledge of Bloom’s Taxonomy by categorizing the questions on their lists as open or closed and deciding which category each question fit in the Taxonomy, noting particularly the balance between lower-order questions (knowledge, comprehension, and application) and higher-order questions (analysis, synthesis, and evaluation). Examples of questions that groups recorded for each level of Bloom’s Taxonomy are as follows: Knowledge: What is a medical error? Comprehension: Compare and contrast personal failure and systems failure. Application: Describe three examples of medical errors and their consequences, differentiating between personal and systemic errors. Analysis: If a physician performs wrong-site surgery, what are the possible sources of personal failure and systems error? Synthesis: Describe a plan to prevent system failures. Evaluation: Justify why systems errors contribute more toward medical errors than does personal failure. As described by Bloom’s Taxonomy, questions differ in the knowledge and skills required to answer them. It is also important to consider that questions serve different purposes, such as research questions, questions to drive self-directed learning such as in problem-based 6 learning groups, or for making clinical practice decisions. Since the audience was a group of newly entering medical students, each group was then asked to consider all of the questions on its list and decide which of those questions were most likely to generate the most useful learning for success in medical school, if pursued in the context of a problem-based learning session about medical errors. Groups were then encouraged to trade lists to see the variety of questions generated by others. Following the small-group learning activities, the librarian led a large-group discussion to further consider the value of different types of questions for learning. Students were asked to use an audience response system to vote on which of four specific questions would lead to the deepest learning for the same “medical error” scenario as above. Respondents explained their reasoning to the group. Other questions considered in group discussion were: Did you notice a preference for lower-order or higher-order questions, and why? Are higher-order questions more important than lower-order questions? Must lower-order questions be addressed before moving on to higher-order questions? What new insights about group processes related to asking questions were gained during this exercise? Students engaged in a lively discussion. RESULTS Data were collected to assess the impact of the 2014 session on first-year medical students’ ability to use Bloom’s Taxonomy to ask higher-order questions. Approval was obtained from the Penn State College of Medicine Institutional Review Board for all data collection. The lists of questions generated by groups during the session were collected and entered into a spreadsheet. Forty-one groups submitted question lists with an average of 11.8 questions 7 submitted per group (range, 5-35). The questions that each group chose as the “best for the purposes of learning in medical school” were analyzed. Groups were asked to vote on their top three questions of this type, but not all groups submitted three questions. The questions were categorized—115 in all‒ according to Bloom’s Taxonomy using the detailed descriptions found in Bloom’s original work.13 The questions were also compared with lists of key words or action verbs associated with cognitive objectives for each level of the taxonomy. Lists of this type are widely available on the web.15 This process was repeated several times for each question on the list to achieve confidence in the analysis. The questions could be categorized as following, according to Bloom’s Taxonomy: 28 knowledge questions; 30 comprehension questions; 2 application questions; 15 analysis questions; 25 synthesis questions; and 13 evaluation questions. Two were characterized as too vague to determine. Of this “best for learning” subset of questions, students categorized 79 of them according to Bloom’s Taxonomy, but there was agreement between students’ and the librarian’s categorizations for only 28 of those questions (35%). A summative assessment at the end of the two-week course included a brief case scenario on a patient with sickle cell crisis as a prompt for the exam question: “Formulate two concise questions for this case that would lead to higher-order learning outcomes according to Bloom’s Taxonomy.” The 136 student responses were compared with Bloom’s Taxonomy using the same method as above to look for higher-order questions—those aimed at a response of analysis, synthesis, or evaluation. For example, correct student responses could show evidence of seeking relationships between two separate entities, such as between disease and symptom or evidence of applying general principles to the specific patient in the case. Incorrect responses were those requiring only knowledge, comprehension, or application to answer, such as any questions about 8 signs and symptoms of sickle cell crisis in general or side effects of medications, for example. Students received one point for each higher-order question written. The average score was 1.8 out of 2.0 (range 0-2). Longer-term learning outcomes of the intervention were also assessed using students’ self-report of application of the skills in a subsequent course. Approximately 14 weeks following the intervention, students received a course evaluation survey at the end of their Scientific Principles of Medicine (SPM) course in which the following optional survey question was asked: In your Profession of Medicine course during the first week of medical school, you participated in a session, “Asking a Great Question” in which various types of questions were discussed and categorized into higher and lower order questions. Can you remember a specific instance during the SPM course, perhaps during a PBL session, when you recalled and applied the knowledge and skills that you learned about question asking? If so, please describe what happened and how you applied the question-asking knowledge and skills. Responses to this open-ended question were assessed qualitatively to look for themes. Out of 118 students who submitted a course evaluation survey, 62 students responded to this question, for a 53% response rate. Of the 62 respondents, 38 students (61%) described the question-asking session as useful for their learning in their subsequent science course. The other 24 students (39%) who responded indicated that they could not remember a time when they applied the question-asking knowledge or skills, that it was not useful, or marked the question as “N/A” (not applicable). Respondents indicated that they applied the knowledge and skills learned during the “Asking a Great Question” session several ways. Of the respondents, 25 (40%) stated that 9 question-asking skills were used by their problem-based learning (PBL) group to formulate “higher-order” learning objectives for group members to pursue independently. Eleven respondents (18%) indicated that they applied the skills on their own, during their personal study time or in their individual research for PBL. Two respondents (3%) stated that they used what they learned about asking “higher-order” questions in finding relevant journal articles to share with their PBL group about case concepts, and two respondents (3%) found what they learned useful during group study for the course. Some respondents indicated that it was useful in more than one category. Responses were also scrutinized for descriptions of how the knowledge and skills gained during the session helped them in their group and individual study. The strongest theme from this analysis was connecting and integrating: students described the utility of higher-order questions in helping them to integrate their learning across topics and over time in the course. Some representative responses supporting the theme of higher-order questions as tools for connecting and integrating concepts are as follows: In reviewing lecture content, I tried to come up with questions I would write if I were writing the exam. A lot of those questions eventually became higher order questions comparing concepts I saw multiple times throughout the course (e.g., methotrexate in the metabolism and cancer topic weeks). I think our PBL group did the same thing with some concepts that could be compared across the weeks (e.g., relating back to the myasthenia gravis case during a later discussion of autoimmunity). Yes—our PBL group always strove for higher order questions to the point of becoming almost frustrating. However, because we were challenged to think of the material in a 10 different way (e.g., how all the information connects together for a higher order question), I think we learned more about the material overall. Instead of discussing a neuromuscular junction, we started asking questions and discussing what could go wrong and how it would affect movement if anything goes wrong during signal transmission, such as impaired ACh vesicle release, adding ACh esterase suicide inhibitor. Rather than coming up with [learning objectives] that merely asked for definition, we were able to devise more detailed questions that helped to connect the clinical relevance with the hard science. DISCUSSION The question-asking session was conceived, designed and taught by a librarian as part of a course providing entrée to medical school for first-year students. This project serves as an example of how medical students can be taught the skill of asking higher-order questions that can help them to connect concepts and achieve deeper learning. Critical thinking skills such as asking questions are highly related to the evidence-based medicine skills and information literacy skills that health sciences librarians are already involved in teaching, so extending librarians’ teaching responsibilities to the critical thinking domain, while challenging, is appropriate. Results of the assessments showed that categorizing questions according to Bloom’s Taxonomy was a more difficult task for students than was formulating a “higher-order” question. 11 This was shown by their success on the exam question, with almost all students able to write at least one higher-level question, compared to the discrepancy between the students’ and the librarian’s categorizations of questions. While the exercise of categorizing questions can help students practice differentiating various types of questions, the ability to formulate “higherorder” questions is more important, as it is a skill that is of immediate application to learning throughout medical school and beyond. Students showed more proficiency in this task, although this study does not provide evidence that the proficiency was a result of the intervention. The qualitative data from the free-response question clarifies why the intervention was successful for those students who did use the skills and knowledge learned: its utility in formulating learning objectives in PBL and in asking questions that lead to integration of learning. This student-reported data links the skills learned in the session with their application to PBL cases. Many students reported that asking higher-order questions helped them to connect seemingly disparate concepts across scientific and clinical domains and over time in the curriculum. Integration of learning is important for medical students who must build a coherent knowledge base from a voluminous amount of information that they encounter in the curriculum. There are several limitations to this study. Data from a single session is presented here; data should be collected from future groups of learners who participate in a similar session. The qualitative data gauging application of these skills in PBL sessions are based on student selfreport only; collecting data from PBL facilitators regarding their observations of student questioning behaviors would have strengthened the study. Finally, future research studies with randomized control and intervention groups and a pre-test administered would be required to show that this intervention does indeed lead to a statistically and educationally significant improvement in students’ question-asking ability and overall performance in PBL. 12 CONCLUSION PBL is a widely used modality in many medical schools, but new medical students may need coaching in how to approach learning in the self-directed manner that PBL requires. Research shows that first-year medical students’ independent study time related to PBL is largely confined to researching the learning issues that are generated during PBL sessions.16 Therefore, first-year medical students’ report that their PBL groups used the knowledge and skills taught during the intervention to devise higher-order learning objectives that integrated new and previous concepts (such as analysis, synthesis, and evaluation questions) is an important finding. Because the skills were used to formulate better learning objectives (i.e., ask better questions), this intervention might serve to improve overall learning in PBL for first-year medical students. REFERENCES 1. Krupat, Edward, Jared M. Sprague, Daniel Wolpaw, Paul Haidet, David Hatem, and Bridget O'Brien. “Thinking Critically About Critical Thinking: Ability, Disposition or Both?” Medical Education 45, no. 6 (June 2011): 625-635. 2. Wolpaw, Terry M., DanielR. Wolpaw, and KlaraK. Papp. “SNAPPS: A Learner-centered Model for Outpatient Education.” Academic Medicine 78, no. 9 (September 2003): 893-898. 3. Papp, Klara K., Grace C. Huang, Laurie M. Lauzon Clabo, et al. “Milestones of Critical Thinking: A Developmental Model for Medicine and Nursing.” Academic Medicine 89, no. 5 (May 2014): 715-20. 13 4. Huang, Grace C., Laurie R. Newman, and Richard M. Schwartzstein. “Critical Thinking in Health Professions Education: Summary and Consensus Statements of the Millennium Conference 2011.” Teaching and Learning in Medicine 26, no. 1 (January 2014): 95-102. 5. Association of College and Research Libraries. Information Literacy Competency Standards for Higher Education. January 2000. http://www.ala.org/acrl/sites/ala.org.acrl/files/content/standards/standards.pdf. 6. Association of College and Research Libraries. Framework for Information Literacy for Higher Education. February 2, 2015. http://www.ala.org/acrl/standards/ilframework. 7. Maggio, Lauren A., Nancy H. Tannery, H. Carrie Chen, Olle ten Cate, and Bridget O'Brien. “Evidence-based Medicine Training in Undergraduate Medical Education: A Review and Critique of the Literature Published 2006-2011.” Academic Medicine 88, no. 7 (July 2013): 1022-8. 8. Egan, Mari E. “Soliciting Feedback by Asking Questions That Promote Thinking Among Medical Students: A Pilot Study” (Master’s thesis, University of Illinois at Chicago, 2001). 9. Loy, Gary. L., Mark. H. Gelula, and Louis A. Vontver. “Teaching Students To Question.” American Journal of Obstetrics and Gynecology 191, no. 5 (November 2004): 1752-6. 10. Wolpaw, Terry, Luc Côté, Klara K. Papp, and Georges Bordage. “Student Uncertainties Drive Teaching During Case Presentations: More So With SNAPPS.” Academic Medicine 87, no. 9 (September 2012): 1210-1217. 11. Kolluru, Srikanth. “An Active-Learning Assignment Requiring Pharmacy Students to Write Medicinal Chemistry Examination Questions.” American Journal of Pharmaceutical Education 76, no. 6 (August 2012): 112. 14 12. Papinczak, Tracey, Awais S. Babri, Ray Peterson, Vaughan Kippers, and David Wilkinson. “Students Generating Questions for Their Own Written Examinations.” Advances in Health Sciences Education: Theory and Practice, 16, no. 5 (December 2011): 703-710. 13. Bloom, Benjamin. S. Taxonomy of Educational Objectives: The Classification of Educational Goals. New York: David McKay Company, 1956. 14. Anderson, Lorin W., and David R. Krathwohl. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. New York: Longman, 2001. 15. Center for Teaching and Learning, University of North Carolina-Charlotte. “Writing Objectives Using Bloom’s Taxonomy.” Accessed April 20, 2015. http://teaching.uncc.edu/learning-resources/articles-books/best-practice/goalsobjectives/writing-objectives. 16. Dolmans, Diana, Henk G. Schmidt, and Wim H. Gijselaers. “The Relationship Between Student-generated Learning Issues and Self-study in Problem-based Learning.” Instructional Science 22, no. 4 (July 1994): 251-267. 15