Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Writing Effective Test Questions for the Basic and Clinical Sciences Carolyn L. Cambor, M.D. Department of Pathology Jennifer R. Kogan, M.D. Department of Medicine FAPD Series September 29, 2010 Materials adapted from the National Board of Medical Examiners Objectives Describe the item types currently used by the NBME & the rationale for their use Describe the steps in writing A-type items • • • • • • • Basic Sciences Clinical Sciences Identify technical flaws in test items & understand the importance of avoiding them Advance your skills in writing A-type test items If time permits, describe and practice writing R-type test items 2 Steps in Test Development Test purpose Testing time and method of administration Test standardization Test content Number of items Item format Develop items Item selection and evaluation 3 Why learn how to write test items? Ensure that your items are: Clear & understandable Test what you want to test Fair Provide reproducible results Avoid technical “flaws” Create confusion Add unnecessary difficulty May aid the “testwise student” 4 “Alphabet Soup” of Item Types True-false X (simple true/false) K (complex true/false) “1,2,3”, “1,3”, “2,4” “all” C (A/B/Both/Neither) In general, not recommended; not used in Mod1,2 If used, must be clear, unambiguous with options that are 100% T or 100% F One-best answer A (4 or more options) B (4 or 5 option matching sets in sets of 2–5 items) R (extended matching items in sets of 2-20 items) 5 A-Type Item: Components Stem Longest part of the item Sufficient information to answer the lead-in question. Lead-in The question being asked Options The correct answer and 3-4 distractors 6 A-type Item Stem: A 65-year-old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty flexing his leg. Lead-in: Which of the following muscles is most likely to have been injured? Options: A. Gluteus maximus* B. Gluteus minimus C. Hamstrings D. Iliopsoas E. Obturator internus Answer Distractors 7 Good, A-Type Items Have a Distinctive Shape Stem Lead In Option A (distractor) Option B (answer) Option C (distractor) Option D (distractor) Option E (distractor) 8 Poorly shaped A-type item Short stem & lead-in a. Long option b. Long option c. Long option d. Long option 9 6 Rules for Writing A-type Items Basic Sciences 1) Focus on important topics. Ideally based on your objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty 10 6 Rules for Writing A-type Items: Basic Sciences 1) Focus on important topics. Ideally based on your objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty 11 6 Rules for Writing A-type Items: Basic Sciences 1) Focus on important topics. Ideally based on your objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty 12 Example – Recall or Application? What area of the brain is supplied by blood from the posterior inferior cerebellar artery? A) Location 1 B) Location 2 C) Location 3 D) Location 4 Recall item: rote memory of isolated fact 13 Example – Recall or Application? A 62-year-old man develops left sided limb ataxia, Horner’s syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations. Which artery is most likely to be occluded? A) Artery A B) Artery B C) Artery C D) Artery D Application of knowledge: Reach a conclusion, make a prediction, select a course of action 14 6 Rules for Writing A-type Items: Basic Sciences 1) Focus on important topics. Ideally based on your objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty 15 Cover the Options Rule Stem Lead in Options a-e 16 Example – Cover the Options A 62-year-old man develops left sided limb ataxia, Horner’s syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations. Which artery is most likely to be occluded? A) Artery A B) Artery B C) Artery C D) Artery D 17 Example – Cover the Options A 62-year-old man develops left sided limb ataxia, Horner’s syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations. Which artery is most likely to be occluded? A) Artery A B) Artery B C) Artery C D) Artery D 18 Writing the Stem Avoid “Which of the following statements is correct?” “Each of the following statements is correct EXCEPT…” Because Unfocused stems Don’t follow “cover the options” rule Heterogeneous options mix of epidemiology, genetics, mechanisms etc 19 Poorly worded stem: True statements about cystic fibrosis (CF) include: 1. 2. 3. 4. The incidence of CF is 1:2000 Children with CF usually die in their teens. Males with CF are sterile CF is an autosomal recessive disease 20 Patient vignettes are good for “stems” Patient vignettes (stem) brief for basic sciences • should not require patient care expertise • age gender site of care presenting complaint duration • +/- family history • +/- results of diagnostic tests; initial treatment, etc material needs to be taught with clinical relevance Lead in Clearly focused question 21 Patient Vignette Templates: Basic Sciences A (patient description) has a (type of injury, location). Which of the following structures is most likely to be affected? A (patient description) has (signs & symptoms). These observations suggest that the disease is a result of the (absence or presence) of which of the following (enzymes, mechanisms)? A (patient description) dies of (disease). Which of the following is the most likely finding on autopsy? 22 Patient Vignette Templates: Basic Sciences Which of the following is the most likely mechanism of the therapeutic effect of this (drug class) in patients with (disease)? (time period) after a (event, such as a trip or meal w/ certain foods) a (patient or group description) became ill with (symptoms and signs). Which of the following (organisms, agents) is most likely to be found on analysis of (food)? 23 Physiology: Patient Vignette During an operation, the arterial PCO2, and pH of an anesthetized patient are monitored. The patient is being ventilated by a mechanical respirator, and the initial values are normal (PCO2=40 mm Hg; pH=7.42). If the ventilation is decreased, which of the following is most likely to occur? pH Arterial pCO2 A. B. C. D. E. F. decrease decrease decrease increase * increase increase A. B. C. D. E. F. decrease increase no change decrease * increase no change 24 Microbiology: Patient Vignette At a banquet, the menu included fried chicken, homefried potatoes, peas, chocolate eclairs and coffee. Within 2 hours most of the diners became violently ill, with nausea, vomiting and abdominal pain. Analysis of the contaminated food is most likely to yield large numbers of which of the following organisms? A. B. C. D. E. Escherechia coli Proteus mirabilis Salmonella typhimurium Staphylococcus aureus* Streptococcus faecalis 25 Lab Vignettes are good for Basic Sciences “Lab”-vignettes laboratory examples research examples biochemical pathways drug metabolism 26 Biochemistry: Lab Vignette A II I E B III C IV D V IV In the branched metabolic pathway, a different single enzyme catalyzes each of the individual steps. The enzyme that would be expected to be most severely inhibited by compound V is enzyme: A. B. C. D. E. A B * C D E 27 Pharmacology: Lab Vignette Drug Y has a volume of distribution of 75 L in both younger and older adult men. In younger adults, it has a clearance rate of 15L / h, 50% of which is via the liver and 50% via the kidneys. For younger men, the maintenance regimen is 100 mg every 6 hours. Which of the following regimens will produce essentially the same steady-state concentration in an older man, whose creatinine clearance is half that of younger men, but whose hepatic function is unimpaired? A. B. C. D. E. F. 75 mg every 3 hrs 75 mg every 6 hrs * 75 mg every 9 hours 100 mg every 3 hours 100 mg every 6 hours 100 mg every 12 hours 28 Basic Sciences: Integrated Items Physiology / Pharmacology Microbiology / Pharmacology Pathology / Pharmacology Pathology / Pathophysiology Stem describes one component, options ask about another 1 stem is used for 2+ questions, on different topics Guidelines: Team preparation Avoid “cueing” Avoid “hinging” 29 Integrated Items - Example An unresponsive 58 year old woman is brought to the ED after collapsing at a local shopping center. More history. Physical signs describe a neurologic deficit. 1. The dilated, unreactive left pupil is most consistent with injury to the left List of anatomic structures 2. The extensor posturing on the right is most consistent with injury to the left List of anatomic structures 3. Which of the following herniation syndromes is most consistent with her clinical presentation? List of herniation syndromes 30 Patient Vignettes: Clinical Sciences Include – age, gender – site of care – presenting complaint – duration – patient history – physical findings – +/- diagnostic studies – +/- initial treatment Stems should – not be completely based on real patients – include reference material when it would be realistic in practice – not use the patient’s or doctor’s own words – not include patients who lie 31 Non-vignettes in the Clinical Sciences The most likely renal abnormality in children with nephrotic syndrome and normal renal function is A. B. C. D. E. Acute poststreptococcal glomerulonephritis Hemolytic-uremic syndrome Minimal change disease Focal and segmental glomerulosclerosis Schonlein-Henoch purpura A B C D E Hi 1 0 99 0 0 Lo 8 1 90 1 0 32 Short Vignette A 2 year old boy has a 1 week history of edema. His blood pressure is 100/60 mmHg and there is generalized edema and ascites. Labs show Cr 0.4 mg/dL, albumin 1.4 g/dL and cholesterol of 569 mg/dL. UA shows 4+ protein and no blood. The most likely diagnosis is A B C D E Hi 0 0 98 2 0 Lo 5 2 82 8 1 33 Long Vignette A 2 year old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mmHg, pulse 110/min respirations 28/min. Exam shows swelling of eyes, abdominal distention and a positive fluid wave. Labs show Cr 0.4 mg/dL, albumin 1.4 g/dL and cholesterol of 569 mg/dL.UA shows 4+ protein and no blood. The most likely diagnosis is A B C D E Hi 0 1 98 1 0 Lo 10 9 66 10 5 34 Writing the Lead-In: Clinical Sciences Diagnosis Which of the following is the most likely diagnosis? Which of the following is the most appropriate next step in diagnosis? Management Which of the following is the most appropriate next step in patient care? Mechanisms of disease Which of the following is the most likely pathogen? Which of the following is the most likely explanation for the findings? 35 6 Rules for Writing A-type Questions 1) Focus on important topics. Ideally based on course / activity objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty 36 Writing the Options Homogeneous in content Grammatically consistent with stem Similar in construction and length Alphabetized or logically ordered Incorrect/inferior to the correct answer choice affects question difficulty Plausible/attractive to uninformed no distracter should be obviously incorrect Each should be chosen by some 37 Easy Distractors Who was the primary author of the Declaration of Independence? A. B. C. D. E. Abraham Lincoln Thomas Jefferson* Franklin Roosevelt King George II Catherine the Great 38 More Difficult Distractors Who was the primary author of the Declaration of Independence? A. B. C. D. E. George Washington Thomas Jefferson* Alexander Hamilton Benjamin Franklin James Madison 39 6 Rules for Writing A-type Questions 1) Focus on important topics. Ideally based on course / activity objectives. 2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees. 3) Assess application of knowledge, not recall. 4) Provide sufficient information in the stem and pose a clear question in the lead-in. “Cover the options rule” 5) Use homogeneous distractors 6) Avoid technical flaws and unnecessary difficulty 40 Technical Item Flaws Issues Related to “Testwiseness” Issues related to Irrelevant Difficulty 41 Assess Your “Testwiseness” Testwise Analysis The primary purpose of the stam is to remove the carm denton menace stam bar 43 Testwise Analysis The primary purpose of the stam is to remove the carm denton menace stam bar “Word repeats” in stem and answer 44 Testwise Analysis Which of the following pairs has won the greatest number of Abby awards? Jones and Smith Smith and Taylor Smith and White White and Allen 45 Testwise Analysis Which of the following pairs has won the greatest number of Abby awards? Jones and Smith Smith and Taylor Smith and White White and Allen Convergence: correct answer has the most in common with other choices 46 Testwise Analysis How many pounds of pressure are exerted by a callam? 2.6 150 260 2600 47 Testwise Analysis How many pounds of pressure are exerted by a callam? 2.6 150 260 2600 Convergence: Correct answer has the most in common with other choices 48 Testwise Analysis The stanon is aided by a instel immon octal port 49 Testwise Analysis The stanon is aided by a instel immon octal port Grammatical cues: only “port” follows grammatically from the stem 50 Testwise Analysis The stanon frequently overheats because all grestels are bilious no immon are directly fectitious octals are usually casable ports are always critical 51 Testwise Analysis The stanon frequently overheats because all grestels are bilious no immon are directly fectitious octals are usually casable ports are always critical Absolute terms: never found in the correct answer 52 Testwise Analysis Stammation normally occurs when the anstels rupture immon falls and the denton is in place octal rotates easily ports pass over the carm 53 Testwise Analysis Stammation normally occurs when the anstels rupture immon falls and the denton is in place octal rotates easily ports pass over the carm Longest answer is usually the correct answer 54 Testwiseness: Grammatical Cues The option(s) does not flow from the stem Example The minor differences among organisms of the same kind are known as A. Heredity B. Variations C. Adaptation D. Natural selection 55 Testwiseness: Logical Cues A subset of options are collectively exhaustive Example Crime is A. Equally distributed among the social classes B. Overrepresented among the poor C. Overrepresented among the middle class and rich D. Primarily an indication of psychosexual maladjustment E. Reaching a plateau of tolerability for the nation 56 Testwiseness: Absolute Terms Terms such as ‘always’ or ‘never’ are used in options Example In patients with advanced dementia, Alzheimer’s type, the memory defect A. Can be treated adequately with lecithin B. Could be a sequela of early parkinsonism C. Is never seen in patients with neurofibrillary tangles C. Is never severe D. Possibly involves the cholinergic system 57 Testwiseness: Long Correct Answer The correct answer is longer, more specific, or more complete than the other options. Example: Secondary gain is A. Synonymous with malingering B. A frequent problem in obsessive-compulsive disorder C. A complication of a variety of illnesses and tends to prolong many of them D. Never seen in organic brain damage 58 Testwiseness: Word Repeats A word or phrase is included in the stem and correct answer. Example: A 58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called A. Depersonalization B. Derailment C. Derealization* D. Focal memory defect 59 Testwiseness: Convergence The correct answer includes the most elements in common with the other options Example: Local anesthetics are most effective in the A. Anionic form, acting from inside the nerve membrane B. Cationic form, acting from inside the nerve membrane* C. Cationic form, acting from outside the nerve membrane D. Uncharged form, acting from inside the nerve membrane E. Uncharged form, acting from outside the nerve membrane 60 Technical Item Flaws Issues Related to “Testwiseness” Issues related to Irrelevant Difficulty 61 Irrelevant Difficulty: Options are long, complicated or doubled Systematic geography differs from regional geography in that A. Systematic geography deals, in the main, with physical geography, while regional geography concerns itself essentially with the field of human geography B. Systematic geography studies a region systematically while regional geography is concerned only with descriptive account of a region C. Systematic geography studies a single phenomenon in its distribution over the earth in order to supply generalizations for regional geography, which studies the arrangement of phenomena in one given area* 62 Irrelevant Difficulty: Numeric data are not stated consistently Following a second episode of infection, what is the likelihood that a woman is infertile? A. Less than 20% B. 20% to 30% C. Greater than 50% D. 90% E. 75% 63 Irrelevant Difficulty: Frequency terms in options are vague Severe obesity in early adolescence A. Usually responds dramatically to dietary regimens B. Often is related to endocrine disorders C. Has a 75% change of clearing spontaneously D. Shows a poor prognosis E. Usually responds to pharmacotherapy and intensive psychotherapy 64 Irrelevant Difficulty: Language in options not parallel In a vaccine trial, 200 2-year-old boys were given a vaccine against a certain disease and then monitored for five years for occurrence of disease. Of this group, 85% never contracted the disease. Which of the following statements concerning these results is correct? A. No conclusions can be drawn since no follow-up was made of non-vaccinated children B. The number of cases (I.e. 30 cases over five years) is too small for statistically meaningful conclusions C. No conclusions can be drawn because the trial involved only boys D. Vaccine efficacy (%) is calculated as 85-15/100 65 Irrelevant Difficulty: Options in an non-logical order The population of Denmark is A. 2 million B. 15 million C. 4 million D. 7 million 66 Irrelevant Difficulty: “None of the Above” used as option Which city is closest to New York City? A. Boston B. Chicago C. Dallas D. Los Angeles E. None of the above 67 Irrelevant Difficulty: Complicated Stems and Options Arrange the parents of the following children with Down’s syndrome in order of highest to lowest risk of recurrence. Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are (see next slide) 68 Complicated Stems and Answers I. II. III. IV. V. 46, XX, -14, +T (14q21q)pat 46, XX, -14, +T(14q21q) de novo 46, XX, -14, +T (14q21q) mat 46, XX, -21, +T (14q21q)pat 47, XX, -21, +T (21q21q) parents not typed A. B. C. D. E. III, IV, I, V, II IV, III, V, I, II III, I, IV, V, II IV, III, I, V, II III, IV, I II, V 69 Irrelevant Difficulty “Window Dressing” “Red Herrings” 70 Basic Science Examples for Discussion Which one of the following sets of laboratory studies is most consistent with a diagnosis of anemia of chronic inflammation? (Inc means increased level, Dec means decreased level) Ferritin A. B. C. D. Inc Dec Inc Inc TIBC Inc Inc Dec Dec Serum Iron Soluble Transferrin Receptor Dec Dec Dec Dec Dec Inc Inc Normal 72 A 16 year old girl presents with a sore throat of 4 months duration. She describes feeling a lump in her throat. Physical exam reveals a 1 cm cystic lesion at the base of the tongue. This developmental lesion most likely arises as a remnant of which of the following anatomic structures? A. B. C. D. E. Auditory tube Branchial arches Facial fusion lines Rudimentary thymus Thyroglossal duct 73 All of the carbons in cholesterol are derived from: A. B. C. D. E. choline sphingosine acetyl CoA HMG-CoA reductase CO2 74 What is the most likely inheritance pattern of the following pedigree? A. B. C. D. E. Autosomal recessive Autosomal dominant X-linked recessive X-linked dominant Mitochondrial 75 Clinical Science Examples A 64 year old male is admitted to the hospital with 3 days of progressive shortness of breath. He has a medical history of poorly controlled hypertension. He denies tobacco or alcohol use. On physical examination he is afebrile, pulse 100, respiratory rate 26, blood pressure 180/110 mmHg and room air oxygen saturation 91%. Examination of the lungs reveals bilateral crackles and his cardiac exam reveals a diffuse, laterally displaced point of maximal impulse, a soft S3 and loud S4. Abdominal exam and extremity exams are normal. A chest Xray reveals bilateral alveolar infiltrates and an electrocardiogram is remarkable for left ventricular hypertrophy. Which of the following is the next best step in management? A. B. C. D. E. Administer intravenous furosemide for diuresis Administer oral morphine Start oral beta blocker Administer chewable aspirin None of the above 77 A 72 year old female presents to her physician with progressive bilateral knee pain for 3 months. The pain is worse with ambulation and increases in severity during the day. She denies trauma or pain in other joints. On physical examination she is afebrile. Her body mass index is 31. Her knees are warm bilaterally but without effusion. There is bilateral crepitus. There is no instability. The remainder of her examination is normal. Plain xrays of the knees demonstrates narrowing of the joint space, marginal osteophytes and subcondral sclerosis. Which of the following is the most likely diagnosis? A. Osteoporosis B. Osteoarthritis C. Osteogenesis imperfecta D. Rheumatoid arthritis E. Pseudogout 78 A 42 year old female presents to her physician for a routine medical examination. She has no specific concerns. Her blood pressure is noted to be 170/80 mm Hg. Which of the following sounds auscultatory findings corresponds to the systolic and diastolic readings? A. The systolic reading is the first appearance of the Korotkoff sounds and the diastolic reading is the disappearance of the Korotkoff sounds B. The systolic reading is the regular appearance of the Korotkoff sounds and the diastolic reading is the muffling of the Korotkoff sounds. C. The systolic reading is the regular appearance of the Korotkoff sounds and the diastolic reading is the disappearance of the Korotkoff sounds D. The systolic reading is the first appearance of the Korotkoff sounds and the diastolic reading is the muffling of the Korotkoff sounds. 79 A 42 year old female presents to her physician for a routine medical examination. She has no specific concerns. Her blood pressure is noted to be 170/80 mm Hg. Which of the following sounds best corresponds to the blood pressure reading? A. B. C. D. 170 First Korotkoff sound Regular Korotkoff sounds Regular Korotkoff sounds First Korotkoff sounds 80 Disappearance of Korotkoff sounds Muffling of Korotkoff sounds Disappearance of Korotkoff sounds Muffling of Korotkoff sounds. 80 Practice Time! R-Type Items Extended Matching Extended Matching: R-type Items Four components 1. 2. 3. 4. A theme An option list A lead-in statement At least 2 item stems 83 R-type item-Theme: Cerebrovascular anatomy A. Left anterior cerebral artery B. Right anterior cerebral artery C. Left middle cerebral artery D. Right middle cerebral artery E. Left posterior cerebral artery F. Right posterior cerebral artery G. Left lenticulostriate arteries H. Right lenticulostriate arteries For each patient with neurological abnormalities presented below, select the artery that is most likely to be involved from the list above. 1. A 72-year-old right-handed man has weakness and hyperreflexia of the right lower limb, an extensor plantar response on the right, normal strength of the right arm, and normal facial movements. Answer: A 2. A 68-year-old right-handed man has right spastic hemiparesis, an extensor plantar response on the right, and paralysis of the lower two-thirds of his face on the right. His speech is fluent, and he has normal comprehension of verbal and written commands. Answer: G 84 Sample Lead-ins and Topics for Option Lists For each of the following patients, select the most likely (cause). Underlying mechanism of disease, medications, toxic agents… For each of the following patients select the (eg, nerve) that is most like to be (abnormal, defective, deficient, nonfunctioning). -lists of nerves, muscles, enzymes, hormones, proteins, types of cells, pathologic processes, neurotransmitters For each of the following patients, select the (finding) that would be expected. Laboratory results, physical signs… 85 More sample lead-ins & topics for option lists For each of the following patients, select the [ eg, drug] that should be administered. For each of the following patients with [chief complaint], select the most likely diagnosis. For each of the following patients, select the most appropriate next step in patient care. (drugs, lab tests, disposition choices) 86 Options Sets in R-Type Items Arteries Nerves Muscles Amino acids Peptides Hormones Enzymes Cell components Cell types Blood components Diagnoses Karyotypes Proteins Lipids Pathogens Cytokines Anatomic structures Metabolic defects Organelles Drugs/drug classes ECM components Diagnostic tests 87 Writing the Item Stems for R-Type Items Patient vignettes Use for basic & clinical science questions Homogeneous in construction Include same demographic features in each one All the same age group Make sure only one best answer and usually 4 distractors 88 Steps in writing R-type items Identify the theme first Write the lead-in statement List all the options Write the vignettes (stems) Review the item Be sure there is only ONE single best answer for each stem Be sure each item has several distractors 89 R-Type Item: Good Example Theme: Fatigue Lead in: For each patient with fatigue, select the most likely diagnosis. Options: Many causes of fatigue (anemia, CHF, depression, infections etc) Vignettes: 2 or more that correspond to diagnoses in your list 90