Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Creating and Analyzing Test Items February 12, 2011 Barbara Penn, PhD, RN, BC Director of Member Education, AACN Rationale for Well-Developed Tests Determining readiness to learn Assessing student learning Validating learning progress (formative) Measuring accomplishment (summative) Evaluating curriculum and instruction Comparing groups of test takers Familiarizing learners with testing Preparing students for NCLEX-RN© NCLEX-RN© z Licensure examination developed by NCSBN Re-evaluated every 3 years z 2010 Test Plan Based on 2008 practice analysis Frequency and priority of >150 nursing tasks Captures entry-level nursing practice Nurse Educator version w/ item writing suggestions z Computerized adaptive testing (CAT) Since 1994 Variable number of items: 75-265 z www.ncsbn.org 2010 NCLEX-RN© Test Plan z Safe and Effective Care Environment Management of Care: 16-22% Safety and Infection Control: 8-14% z z z Health Promotion and Maintenance: 6-12% Psychosocial Integrity: 6-12% Physiological Integrity Basic Care and Comfort: 12% Pharmacological and Parenteral Therapies: 13-19% Reduction of Risk Potential: 10-16% Physiological Adaptation: 11-17% Concepts Embedded Throughout Nursing process z Caring z Communication and documentation z Teaching and learning z The Language of Assessment Assessment = Process of obtaining information about student learning Measurement = Process of assigning a score that represents a characteristic Evaluation = Value judgment that attaches meaning to assessment data McDonald, 2007 Domains of Learning Cognitive Test, paper, presentation, discussion Psychomotor Demonstration of skill, product Affective Attitude/values self-assessment, behavior Cognitive Domain Bloom, 1956 Knowledge Comprehension Application Analysis Synthesis Evaluation Recall facts Understand ideas Use in a new situation Break down to understand Put together a new product Judge against specific criteria Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001 ¾ Remembering - define, state, identify, tell, name, list, recall, recognize, label ¾ Understanding - order, describe, restate, paraphrase, explain, interpret, summarize, translate ¾ Applying - determine, compute, conclude, give an example, illustrate ¾ Analyzing - categorize, classify, differentiate, distinguish, examine, infer ¾ Evaluating - appraise, prioritize, defend, give your opinion, conclude, decide, support ¾ Creating - design, change, formulate, write, invent, suggest, predict, reconstruct, find an unusual way http://edtech.tennessee.edu/~bobannon/classifications.html Cognitive Domain Bloom, 1956 Anderson & Krathwohl, 2001 Knowledge Remembering Recall facts Comprehension Understanding Understand ideas Application Applying Use in a new situation Analysis Analyzing Break down into parts Synthesis Evaluating Judge against specific criteria Evaluation Creating Put together a new product Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001 ¾ Remembering - define, state, identify, tell, name, list, recall, recognize, label ¾ Understanding - order, describe, restate, paraphrase, explain, interpret, summarize, translate ¾ Applying - determine, compute, conclude, give an example, illustrate ¾ Analyzing - categorize, classify, differentiate, distinguish, examine, infer ¾ Evaluation - appraise, prioritize, defend, give your opinion, conclude, decide, support ¾ Creating - design, change, formulate, write, invent, suggest, predict, reconstruct, find an unusual way http://edtech.tennessee.edu/~bobannon/classifications.html Psychomotor Domain Dave, 1970 Imitation Manipulation Precision Articulation Naturalization Repeat after demonstration Perform after instruction Add accuracy, refinement Add sequence, speed High proficiency; habit Affective Domain Krathwohl, 1964 Receiving Responding Valuing Organization Characterization Aware, sensitive React overtly Accept, prefer Conscious values Behavior shows values Types of Test “Items” z Selected Response - Multiple choice - Matching - True-False z Constructed response - Completion - Short Answer - Essay NCLEX-RN© Items z z z z z Place respondents in realistic situations (not a lot of detail) Elicit higher level thinking Are typically at application level or higher 1-2 minutes per item recommended Vary in format Multiple choice (1 or >1 answer) Short answer or completion Drag and Drop Audio Hot spots Ordered response Chart/exhibit The Test Blueprint Objectives Æ Content Æ Evaluation Test Blueprint is Linked to Validity z z z z Validity: Test measures intended content Indirectly indicates fairness of a test Not quantified; rather determined by evidence Types Criterion validity – r/t variables external to the test Face validity - what a test appears to measure; not sufficient evidence of validity by itself Content validity - ensured by closely following the test blueprint which reflects well-written objectives Creating a Blueprint z z z z z z z z Draft overview of course objectives, units, or content (in collaboration with colleagues) Estimate relative teaching time Transfer these proportions to the test plan Consider all steps in the nursing process Include all possible levels of the cognitive domain Space tests so that content is proportional across the semester Consider the school’s social calendar Share with students Ways to Depict Course Organization By Objectives By Topics Course Goal 1 Course Objective 1 Unit 1 Topic 1 (2 hr) Topic 2 (3 hr) Class 1 Objectives 1-4 Class 2 Objectives 5-8 Unit 2 Class 3 Objectives 9-12 Class 4 Objectives 13-20 Class 5 Objectives 21-23 Course Goal 2 Unit 3 Class 6 Objectives 21-26 Class 7 Objectives 27-30 Course Goal 3 Unit 4 Class 8 Objectives 31-35 (etc.) Course Objective 2 Topic 3 (1.5 hr) Topic 4 (2 hr) Topic 5 (4 hr) Course Objective 3 Topic 6 (3 hr) Topic 7 (2 hr) Topic 8 (1 hr) (etc.) Simple Course Test Plan Date Test Content 2/15 3/21 4/20 5/2 Unit 1 Unit 2 Unit 3 Final Weeks 1-5 Weeks 6-10 Weeks 11-15 Week 16 + Comprehensive Items Weight 60 60 60 120 20% 20% 20% 40% Novotny & Griffin, 2006, page 113 Two-Way Table of Specifications Content/Outcomes Know Comp Appl Totals (20%) Antipsychotic agents (25%) 2 Antianxiety agents (25%) 2 Antidepressant agents (25%) 2 Antimanic agents (12.5%) 1 Antiparkinsons agents (12.5%) 1 Totals 8 (40%) (40%) 4 4 4 2 2 16 4 4 4 2 2 16 10 10 10 5 5 40 Billings and Halstead, 1998, p. 387 Three-Way Table of Specifications Outcomes/Process Assessment (40%) Diagnosis (10%) Planning (10%) Intervention (20%) Evaluation (20%) Totals Know Comp Appl (20%) (40%) (40%) Totals P,A,A,D,D,M P,P,A,D,M,PA P,A,D,PA P A,D,M PA A,D P P,A P,A,A,D,M,PA P,PA,D P,A,D,D,M 16 4 4 8 8 16 16 40 8 P=antipsychotic agents; A=antianxiety agents; D=antidepressive agents; M=antimanic agents; PA=antiparkinson agents Billings and Halstead, 1998, p. 388 Test Blueprint Example 1 of 3 (Total Number of Items) Used with permission of Dr. Jeanne Wissmann, Graceland University z z Total Lecture Hours/Course: 12 weeks X 3 lecture hrs = 36 hrs Total Exam Items/Course: three 50-item Unit Exams + one 100-item Final Exam = 250 items Content Lecture Hrs Exam Items Unit Exam Items Final Exam Items Unit I 9 (25%) 60-65 50 10-15 Unit II 9 (25%) 60-65 50 10-15 Unit III 9 (25%) 60-65 50 10-15 Unit IV 9 (25%) 60-65 0 60-65 36 250 150 100 Total Test Blueprint Example 2 of 3 (Number/Distribution of Items) Used with permission of Dr. Jeanne Wissmann, Graceland University Learning Outcomes (Unit I Objectives) # of items 1. Apply the overall therapeutic objective of drug therapy to an understanding of the ideal properties of drugs and to specific drug selection by prescribers and in the preparation, administration, and evaluation of prescribed medications to clients. 1 2. Appreciate that the nurse’s role in medication administration includes the six rights of drug administration and other actions taken prior to, during, and following medication administration 3 3. Distinguish between generic and trade drug names, over-the-counter and prescribed medications, Schedules I-V for Controlled Substances, and Phases I-IV of Clinical Testing for FDA approval of New Drugs. 2 4. Demonstrate an understanding of the terms (side effect, adverse effect, toxicity, nephrotoxicity, neurotoxicity, cardiotoxicity, ototoxicity, idiosyncratic effect, carcinogenic effect, teratogenic effect, allergic reaction, iatrogenic disease, physical dependence, additive effects, and medication error) and the nurse’s role in preventing, recognizing, and responding to each of these possibilities. 4 Test Blueprint Example 3 of 3 (Types of Items) Used with permission of Dr. Jeanne Wissmann, Graceland University Unit Objective [Goal?] Behavioral Evidence [Objective?] Appreciate that the nurse’s role in medication administration includes the six rights of drug administration and other actions taken prior to, during, and following medication administration •Recognize errors in the use of the six rights •Identify client findings preadministration which indicate a need to hold a prescribed medication •Identify precautions to be taken during administration •Identify client findings postadministration that indicate therapeutic effects. Targets (e.g., Thinking Skill, Nursing Process) Analysis Assessment Item Format MC Timing – Planning Exams Develop the blueprint z Prepare items z Select items to use z Peer review test z Types of Test Items Matching True-False Short Answer Essay Multiple Choice Evidence of Learning New information is linked to what the learner already knows. z The learner can use new information; not just remember facts. z Recognition = Recall “A small boat found in the Far East, propelled by a single oar at the rear, and often covered by a roof of straw mats.” Multiple Choice A patient with chronic cancer pain has been receiving morphine sulfate 100 mg/hour intravenously for two months . Yesterday the dose was increased to 125 mg/hour. The nurse will need to be observant for which common side effect? a. respiratory depression b. constipation c. nausea d. agitation Terms: Stem Options: Distractor Key (Answer) Distractor Distractor Distractors z z z z Directly influence student performance on the test Are intended to be attractive to low scorers who have not mastered the content being tested Should be clearly incorrect Must be plausible Advantages of Multiple Choice Learner is able to – Analyze phenomena – Apply principles to new situations – Comprehend concepts and principles – Discriminate between fact and opinion – Interpret cause-and-effect relationships – Interpret data – Make inferences – Solve problems Burton, et.al, 1991 Limitations of Multiple Choice Requires teacher time and skill Learner is unable to – Articulate explanations – Display thought processes – Furnish information – Organize personal thoughts – Produce original ideas – Provide examples – Perform a task Burton, et.al, 1991 Higher Reliability with MC Tests z z z z z z Not as susceptible to guessing as with T-F Increased number of items offsets guessing effect Scoring is more accurate than short answer Samples a broader range of content than essay Not affected by scorer bias as with essay Immune to bluffing and effects of writing ability as with short answer and essay Burton, et.al., 1991 Constructing a Multiple Choice Test z z z z z Phrase the stem as a question if possible Make the stem longer than the options Develop the correct answer first Ensure there is only one correct answer Be careful about multiple answers Scoring issues; Most experts recommend only 1 z z z Keep options as brief as possible Make options conceptually congruent Consider the number of options Constructing a Multiple Choice Test z z z z z Include directions to the students (purpose of the test, time allotted, basis for responding, value of items) Use simple, precise, unambiguous wording Make each item independent from others Avoid grammatical and typographical errors Space evenly; place items on same page Constructing a Multiple Choice Test z z z z z Arrange items from simple to complex Consider grouping related information Balance the placement of the correct answers Plan sufficient time: 30 sec-1 minute to respond to each item (depending on difficulty) 60 minutes = 50 (4-option) MC items Use at least 25 items for the test 1 in 942,651 chance to score 70% by guessing Encourage Critical Thinking z z z z Ask students to draw on their nursing knowledge, apply concepts Use novel, unique clinical scenarios “Which of these actions would be most appropriate for the nurse…” Use best answer rather than correct answer McDonald, 2007 Example: Knowledge (Remembering) In a patient with myocardial infarction, which should be monitored during titration of nitroglycerine? a. pulse rate b. blood pressure c. heart rhythm d. ST segment Nursing Spectrum, January 29, 2007, p. 19 Example: Comprehension (Understanding) Which of the following actions, if performed by the nurse, would be considered negligence? a. The nurse obtains a Guthrie blood test on a 4-day-old infant. b. The nurse massages lotion on the abdomen of a 3-year-old diagnosed with Wilm’s tumor. c. The nurse instructs a 5-year-old asthmatic to blow on a pinwheel. d. The nurse plays kickball with a 10-year-old with juvenile arthritis. http://www.4tests.com/exams/questions.asp?exid=8515025&googlebot=78 Example: Application (Applying) On the first postoperative day after an open reduction and internal fixation of the tibia with application of a long leg cast, the patient begins complaining of severe pain uncontrolled by intravenous medications. What is the priority nursing intervention? a. Administer a narcotic bolus per protocol b. Assess neurovascular status c. Lower the leg to increase arterial flow d. Raise the leg to decrease venous return Billings and Halstead, 1998, p. 394 Example: Analysis (Analyzing) A 70-year old woman is admitted to the hospital with a diagnosis of dehydration. Laboratory values: serum sodium=165 mmol/L; serum osmolarity=320 mOsm/L. Which type of fluid replacement is most likely to be administered? a. Hypertonic b. Hypotonic c. Isotonic Billings and Halstead, 1998, p. 394 Examples: Evaluating z z When a client who has a [dx] is being treated with [medication], which of these laboratory data would indicate that the client’s condition is improving? Which of these client observations is the most reliable indicator that the client has the correct understanding of how to [give injection, follow diet, avoid side effects]? McDonald, 2007 Avoid Common Mistakes in Developing MC Items Aim for Higher than Knowledge/Remembering Which care delivery model is represented when a unit uses RNs, LPNs, and UAPs to provide direct care? a. Primary nursing. b. Team nursing. Loretta Manning, ICAN Publishing. Used with permission. M-u-c-h Better Æ Analyzing Which assignment would be most appropriate to delegate to the LPN? a. A client in acute renal failure with a potassium level of 5.9 mEq/L. b. A four-day post operative GI surgical client with a nasogastric tube. c. A newly diagnosed diabetic client who needs to learn about insulin. d. An order to administer 2 units of blood to a newly admitted client. Loretta Manning, ICAN Publishing. Used with permission. Critique Which of these is an example of the ninth cranial nerve? a. Glossopharyngeal b. Olfactory c. Spinal accessory d. Trigeminal Don’t Settle for Remembering The physician has just completed a liver biopsy. Immediately following the procedure, the nurse will position the client a. On his right side b. In Fowler's position c. Supine d. In Sims' position Adapted from http://www-nix.oit.umass.edu/~helene/Nclexfinal.htm Ask for More than Knowledge of Facts/Trivia (Use “memory-plus” items) The physician has just completed a liver biopsy. Immediately following the procedure, the nurse will position the client a. On his right side to promote hemostasis b. In Fowler's position to facilitate ventilation c. Supine to maintain blood pressure d. In Sims' position to prevent aspiration http://www-nix.oit.umass.edu/~helene/Nclexfinal.htm Or… describe a range of nursing actions from which to choose Which of these measures should a nurse include in the care plan for a client who had a liver biopsy 1 hour ago? a. Monitor the neurovascular status of the client’s affected extremity. b. Advise the client to maintain a right side-lying position. c. Check the patient’s urine for occult blood. d. Instruct the client to move slowly when arising from a sitting position. McDonald, 2007 Make options conceptually congruent Conditions that impair the accuracy of the pulse oximetry include all of the following except? a. Temperature of the finger b. Position of the probe c. Raynaud’s Disease d. Quality of the respirations Loretta Manning, ICAN Publishing. Used with permission. Increase level of thinking The pulse oximeter alarms indicating the oxygen saturation is at 86%. What nursing intervention should have the highest priority? a. Administer oxygen via mask. b. Check the position of the probe. c. Complete a cardiovascular assessment. d. Reset the pulse oximeter alarm. Loretta Manning, ICAN Publishing. Used with permission. Emphasize Qualifiers (only, best, first, not, except) Signs and symptoms of hypernatremia would include all of the following, except a. restlessness b. muscular weakness c. tachycardia d. thirst Use CAPS, bold, underline, and/or italics Confusing: Negative stem + Negative options When physically restraining a client, a nurse should consider all of these standards of care EXCEPT a. obtaining the client’s consent for the restraint b. Using the least restrictive device for the shortest time c. not keeping restraints on continuously d. applying restraints when nonrestrictive alternatives are not effective McDonald, 2007 Use Positive Stem & Options Which of these standards of care should a nurse include when caring for a client who is physically restrained? a. Obtaining a signed consent from the client b. Using the most restrictive device available. c. Removing restraints at regular intervals d. Using a nonrestrictive alternative in addition to the restraint McDonald, 2007 Don’t Give Away the Answer! What is the process by which an oral medication is greatly metabolized by the liver before it reaches the systemic circulation? a. Metabolism b. Absorption c. Distribution d. Elimination Loretta Manning, ICAN Publishing. Used with permission. Test at Entry-level; “Need to know” You are taking the history of a 14 year old girl who has a BMI of 18. The girl reports inability to eat, induced vomiting and severe constipation. Which of the following would you most likely suspect? a. Multiple sclerosis b. Anorexia nervosa c. Bulimia d. Systemic sclerosis http://www.testprepreview.com/modules/nclex.htm More appropriate A client with asthma has received an order for propranolol (Inderal). What is the highest priority nursing action before initiating the order? a. Assess breath sounds 90 minutes after administering the medication. b. Evaluate heart rate to determine a problem with tachycardia. c. Monitor the blood pressure prior to administering the medication. d. Verify the accuracy for this medication order Loretta Manning, ICAN Publishing. Used with permission. Test Only One Major Point or Concept per Item A 93 year-old female with a history of Alzheimer’s Disease Gets admitted to an Alzheimer’s unit. The patient has exhibited signs of increased confusion and limited stability with gait. Moreover, the patient is refusing to use a wheelchair. Which of the following is the most appropriate action for the nurse? a. Recommend the patient remain in her room at all times b. Recommend family members bring pictures to the patient’s room c. Recommend a speech therapy consult to the doctor d. Recommend the patient attempt to walk pushing the wheelchair for safety Source: http://www.testtrade.com/nclex1.pdf Put Common Information in the Stem (Redundant Options) The contraction stress test (CST) is used frequently in the management of postdate pregnancies. The main reason that a CST is performed is which of the following? a. To determine whether the fetus is ready to be delivered b. To determine whether the placenta is adequately oxygenating the fetus c. To determine whether the fetus can tolerate normal labor prior to elective induction d. To determine whether the uterus will be able to contract efficiently enough to establish an adequate labor pattern. Better: A contraction stress test (CST) is done primarily to determine whether: a. the fetus is ready… b. the placenta is adequately… c. the fetus can tolerate… d. the uterus will be able… BUT… Don’t Emphasize Options A contraction stress test (CST) is done primarily to determine whether: a. the fetus is ready… b. the placenta is adequately… c. the fetus can tolerate… d. the uterus will be able… Avoid cues from wording of options When brushing the teeth of patients who have severe disabilities and are prone to gagging and choking, caregivers should use which of the following on the brush? a. hydrogen peroxide b. a minty mouthwash c. a pea-sized amount of toothpaste d. enough toothpaste to cover the entire brushing surface AJN, June 2009 Avoid Cues from Opposite Options A client has a Sengstaken-Blakemore tube connected to low wall suction. When the client develops respiratory distress, which of these actions should a nurse take? a. Inflate the tube’s esophageal balloon b. Deflate the tube’s esophageal balloon c. Decompress the tube’s gastric balloon d. Increase the amount of wall suction McDonald, 2007 Start Options with 4 Different Words Diane, a 28-year-old female, is admitted to the hospital with a diagnosis of anorexia nervosa. She is 5'6" tall and weighs 90 pounds. She appears weak and lethargic, with very poor skin turgor. Diane is placed on a high-protein, high-carbohydrate diet. The primary function of carbohydrates in the body is to: a. Develop a cushion around vital organs b. Break down and transport proteins c. Supply energy to the body d. Maintain fluid balance http://www.nclex-pass.com/sample_questions.htm Or Use the 2 +2 Approach Melena is blood loss in the stool which is: a. less than 20 ml b. greater than 20 ml c. less than 50 ml d. greater than 50 ml Source: http://www.nursingstudy.com/nclex/nclex_practice_questions.html Make Options Roughly the Same Length 1. All of the following tasks may be delegated to the LPN except: a. developing a patient teaching plan regarding the patient's diet, exercise, and medications. b. monitoring Mrs. B's blood sugars via Accucheck. c. performing dressing changes in the infected foot. d. administering the patient's pain medication. http://www.nclex-pass.com/sample_questions.htm After the lungs, the kidneys work to maintain body pH. The best explanation of how the kidneys accomplish regulation of pH is that they a. Secrete hydrogen ions and sodium. b. Secrete ammonia. c. Exchange hydrogen and sodium in the kidney tubules. d. Decrease sodium ions, hold on to hydrogen ions, and then secrete sodium bicarbonate. http://www-unix.oit.umass.edu/~helene/Nclexfinal.htm Avoid Using “Loaded” Words To avoid infection after receiving a puncture wound, you would advise someone to a. always go to the doctor to get a tetanus shot b. request an antibiotic only if the wound is painful c. ensure that no foreign object has been left in the wound d. never wipe the wound with alcohol unless it is bleeding Avoid Using “None of the Above” Most nosocomial infections of the urinary tract are caused by a. Pseudomonas b. Staphylococcus c. Klebsiella d. None of the above z z Why is it not a good idea when “none” is the correct answer? One exception = math calculations Be Careful with “All of the above” z z Not a good substitute for well developed distractors Cues students - How? XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXX? a. XXXXXXXXX b. XXXXXXX c. XXXXXXXXXX d. XXXXXXXXXXXXX e. All of the above Don’t Use Abbreviations A nurse working a surgical unit, notices a patient is experiencing SOB, calf pain, and warmth over the posterior calf. All of these may indicate which of the following medical conditions? a. Patient may have a DVT b. Patient may be exhibiting signs of dermatitis c. Patient may be in the late phases of CHF d. Patient may be experiencing anxiety after surgery http://www.testtrade.com/nclex1.pdf Avoid bias z z z z Avoid using gender or names Don’t stereotype members of a group Refer to race, culture, religion, marital status, or sexual orientation only if pertinent Avoid terminology unfamiliar to other cultures – – “Stop smoking cold-turkey” “Take the medication before you retire” McDonald, 2007 Use Logical Order and Even Intervals Hypoglycemia is a blood glucose of less than a. 60 b. 80 c. 100 d. 120 Avoid inconsistent responses Henry and colleagues’ survey of nursing home residents indicated that what percentage of them had not seen a dentist in more than 5 years? a. 10% b. 25% c. 33% d. 50% AJN, June 2009, p.51 Avoid overlapping options When should medication X be given? a. Before breakfast b. On a full stomach c. With meals d. Before going to bed Fray, 1995 Clarify “Should” v. “Would” Loberta Jackson, a 21-year-old college student is admitted to a medical unit with diagnosis of uncontrolled diabetes and an acute hypoglycemic reaction. Loberta says, “I’ve been feeling "sick to my stomach, like I was coming down with the flu " for the past 48 hours. She has continued to take her usual daily insulin. noting that Loberta’s admission blood glucose is 53, which question would you ask first? a. "Have you been under a great deal of stress lately, Loberta? b. "Were you having difficulty sleeping after this illness started?” c. "Have you eaten anything in the past 48 hours?" d. "Did you take any medications for this illness other than your insulin ?“ Adapted from http://www.nclex-pass.com/sample_questions.htm Clarify “Should” v. “Would” A 32 year-old male with a complaint of dizziness has an order for Morphine via. IV. The nurse should do which of the following first? a. Check the patient’s chest x-ray results. b. Retake vitals including blood pressure. c. Perform a neurological screen on the patient. d. Request the physician on-call assess the patient. http://www.testtrade.com/nclex1.pdf Multiple response MC - NCLEX RN The nurse is caring for a client who has a wound infected with methicillin-resistant Staphylococcus aureus (MRSA). Which of the following infection control precautions should the nurse implement? a. Wear a protective gown when entering the client’s room. b. Put on a particulate respirator mask when administering medications to the client. c. Wear gloves when delivering the client’s meal tray. d. Ask the client’s visitors to wear a surgical mask when in the client’s room. e. Put on a face shield before irrigating the client’s wound. 2006 NCLEX Examination Candidate Bulletin Combined or Complex Multiple Choice The fluid imbalance known as edema is commonly associated with 1. Allergic reactions 2. Congestive heart failure 3. Extensive burns 4. Protein deficiency a. 1, 2, and 3 b. 1 and 3 c. 2 and 4 d. 4 only e. 1, 2, 3, and 4 Burton, et. al., 1991 Consider the Nursing Process and Cognitive Domain Levels A 22 year-old patient in a mental health lock-down unit under suicide watch appears happy about being discharged. Which of the following is probably happening? a. The patient is excited about being around family again. b. The patient’s suicide plan has probably progressed. c. The patient’s plans for the future have been clarified. d. The patient’s mood is improving. Nursing Process = Analysis Cognitive Level = Comprehension (Understanding) http://www.testtrade.com/nclex1.pdf Critique Briefly describe the electrical conduction system of the heart. Critique Based on the article that was assigned to read on Monday, what diuretic was discussed throughout the article? a. furosemide (Lasix) b. hydrochlorothiazide (HCTZ) c. mannitol (Osmitrol) d. spironolactone (Aldactone) Source: Loretta Manning, ICAN Publishing. Used with permission. Item Analysis z z z z Functional Analysis Reliability -- KR-20 Difficulty – p-value Discrimination -- PBI Functional Analysis Assessment of degree to which wrong answers were selected by students z z z Response frequencies High and low performers Non-distractors Functional Analysis Response Frequencies Nurs 471 Exam 1, Question 6 Response Frequency Percent A 9 12.50 B* 52 72.22 C 8 11.11 D 3 4.17 E 0 0.00 Total 72 100.00 Functional Analysis High and Low Performers Item 1 2 3 4 5 Correct Group Responses Total Upper 27% Lower 27% 86.67% 100.00% 62.50% 93.33% 100.00% 75.00% 40.00% 87.50% 12.50% 73.33% 87.50% 75.00% 100.00% 100.00% 100.00% McGahee & Ball, 2009 Functional Analysis Non-distractors Exam 390 A, Question 7 Response Frequency Percent A 0 0.00 *B 57 100.00 C 0 0.00 D 0 0.00 E 0 0.00 Totals 57 100.00 Reliability Refers to consistency of test results or “reproduceability” How well test is measuring similar characteristics Not a measure of the test per se Three types: test-retest – same test given again to same group parallel forms – same person gets 2 versions of test internal consistency – from a single administration http://www.som.tulane.edu/ome/helpful_hints/test_analysis.pdf McGahee and Ball, 2009 Reliability – Internal Consistency How well the exam measures a single cognitive factor Reported in a range: 0.0 -1.0 Kuder-Richardson – 20 (KR-20) used with dichotomous items (correct or incorrect) best choice for use with MC tests Desired KR-20 = 0.50-0.85 Coefficient Alpha best used with surveys or attitude measures http://www.som.tulane.edu/ome/helpful_hints/test_analysis.pdf McGahee and Ball, 2009 Degree of Difficulty – p-value z z z z z z % or proportion of students answering correctly Reflects ease of item Reported 0.0-1.0 Higher number = easier item Reflects the group taking the test; not the test itself Desired p-value = 0.50 -0.80 Discrimination – Point Biserial z z z z Basic indicator of item quality Correlation between how well students did on that item and their total test score Typically, upper 1/3 and lower 1/3 assessed Goal: – – z Positive PBI for the correct answer (high achievers chose this) and negative (or near zero) PBI for each distractor (low achievers chose these) Desired PBI = > 0.20 Point Biserial Response A *B C D E Exam 1, Question 6 Frequency Percent 9 12.50 52 72.22 8 11.11 3 4.17 0 0.00 Total 72 100.00 PBis -0.28 0.31 -0.12 -0.05 - Item Analysis Response *A B C D E Exam 2, Question 32 Frequency Percent 66 91.67 0 0.0 4 5.56 2 2.78 0 0.0 Total 72 100.00 PBis 0.13 -0.13 -0.03 Questions? Advice to Colleagues? Selected References Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. New York: Addison Wesley, Longman, Inc. Burton, S.; Sudweeks, R.; Merrill, P.; & Wood, B. (1991). How to prepare better multiple choice items: Guidelines for university faculty. Brigham Young University Testing Services and Department of Instructional Science. Retrieved January 22, 2010 from http://testing.byu.edu/info/handbooks/betteritems.pdf Frary, R.B. (1995). More multiple-choice item writing do’s and don’ts. Practical Assessment, Research, and Evaluation 4(11). Retrieved January 22, 2010 from http://PAREonline.net/getvn.asp?v=4&n=11 Selected References Gaberson, K.B. & Oermann, M.H. (2005). Evaluation and Testing in Nursing Education, 2nd ed. NY: Springer Kehoe, J. (1995). Writing multiple-choice test items. Practical Assessment, Research, and Evaluation 4(9). Retrieved January 22, 2010 from http://PAREonline.net/getvn.asp?v=4&n=9 McDonald, M. E. (2007). The Nurse Educator’s Guide to Assessing Learning Outcomes, 2nd Ed. Boston: Jones and Bartlett Publishers McDonald, M. E. (2008). Developing trustworthy classroom tests. In Mastering the Teaching Role: A Guide for Nurse Educators, B. Penn, ed. Philadelphia: F.A. Davis McGahee, T.W., & Ball, J. (2009). How to read and really use an item analysis. Nurse Educator, July/August, v. 34, n. 4, pp. 166-171. Selected References Morrison, S. (2004). Test construction and analysis: Can I do it? In Teaching Nursing: The Art and Science, Volume 1, Caputi and Englemann, eds. Glen Ellyn, IL: College of DuPage Press, pp. 366-387 National Council of State Boards of Nursing. (2007). NCLEX-RN© Examination: Test Plan for the National Council Licensure Examination for Registered Nurses. Retrieved August 15, 2009 from http://www.ncsbn.org/RN_Test_Plan_2007_Web.pdf Novotny, J.M. & Griffin, M.T. (2006). A nuts-and-bolts approach to teaching nursing, 3rd ed. New York: Springer Publishing, pp. 111133. Oermann, M.H. & Gaberson, K.B. (2009). Evaluation and testing in Nursing Education, 3rd edition. New York: Springer. Selected References Rasmussen, L., Speck, D., & Twigg. P. (1998). Developing and using classroom tests. In Teaching in Nursing: A guide for faculty, 3rd edition, Billings & Halstead, eds. Philadelphia: W.B. Saunders, pp. 385-405 Measurement and Evaluation Center, The University of Texas at Austin. (2003). Test item analysis and decision making. Retrieved January 22, 2010 from http://www.utexas.edu/academic/mec/research/pdf/itemanalysis handout.pdf Tulane School of Medicine. Test and item analysis. Retrieved January 22, 2010 from http://www.som.tulane.edu/ome/helpful_hints/test_analysis.pdf