Download Creating and Analyzing Test Items

Creating and Analyzing
Test Items
February 12, 2011
Barbara Penn, PhD, RN, BC
Director of Member Education, AACN
Rationale for Well-Developed Tests
Determining readiness to learn
Assessing student learning
Validating learning progress (formative)
Measuring accomplishment (summative)
Evaluating curriculum and instruction
Comparing groups of test takers
Familiarizing learners with testing
Preparing students for NCLEX-RN©
NCLEX-RN©
z
Licensure examination developed by NCSBN
Re-evaluated every 3 years
z
2010 Test Plan
Based on 2008 practice analysis
Frequency and priority of >150 nursing tasks
Captures entry-level nursing practice
Nurse Educator version w/ item writing suggestions
z
Computerized adaptive testing (CAT)
Since 1994
Variable number of items: 75-265
z
www.ncsbn.org
2010 NCLEX-RN© Test Plan
z
Safe and Effective Care Environment
Management of Care: 16-22%
Safety and Infection Control: 8-14%
z
z
z
Health Promotion and Maintenance: 6-12%
Psychosocial Integrity: 6-12%
Physiological Integrity
Basic Care and Comfort: 12%
Pharmacological and Parenteral Therapies: 13-19%
Reduction of Risk Potential: 10-16%
Physiological Adaptation: 11-17%
Concepts Embedded Throughout
Nursing process
z Caring
z Communication and documentation
z Teaching and learning
z
The Language of Assessment
Assessment = Process of obtaining
information about student learning
Measurement = Process of assigning a
score that represents a characteristic
Evaluation = Value judgment that
attaches meaning to assessment data
McDonald, 2007
Domains of Learning
Cognitive
Test, paper, presentation, discussion
Psychomotor
Demonstration of skill, product
Affective
Attitude/values self-assessment,
behavior
Cognitive Domain
Bloom, 1956
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Recall facts
Understand ideas
Use in a new situation
Break down to understand
Put together a new product
Judge against specific criteria
Revised Bloom’s Taxonomy
Anderson and Krathwohl, 2001
¾
Remembering - define, state, identify, tell, name, list, recall,
recognize, label
¾
Understanding - order, describe, restate, paraphrase, explain,
interpret, summarize, translate
¾
Applying -
determine, compute, conclude, give an example,
illustrate
¾
Analyzing - categorize, classify, differentiate, distinguish,
examine, infer
¾
Evaluating - appraise, prioritize, defend, give your opinion,
conclude, decide, support
¾
Creating - design, change, formulate, write, invent, suggest,
predict, reconstruct, find an unusual way
http://edtech.tennessee.edu/~bobannon/classifications.html
Cognitive Domain
Bloom, 1956
Anderson & Krathwohl, 2001
Knowledge
Remembering
Recall facts
Comprehension
Understanding
Understand ideas
Application
Applying
Use in a new situation
Analysis
Analyzing
Break down into parts
Synthesis
Evaluating
Judge against specific
criteria
Evaluation
Creating
Put together a new product
Revised Bloom’s Taxonomy
Anderson and Krathwohl, 2001
¾
Remembering - define, state, identify, tell, name, list, recall,
recognize, label
¾
Understanding - order, describe, restate, paraphrase, explain,
interpret, summarize, translate
¾
Applying - determine, compute, conclude, give an example,
illustrate
¾
Analyzing - categorize, classify, differentiate, distinguish,
examine, infer
¾
Evaluation - appraise, prioritize, defend, give your opinion,
conclude, decide, support
¾
Creating - design, change, formulate, write, invent, suggest,
predict, reconstruct, find an unusual way
http://edtech.tennessee.edu/~bobannon/classifications.html
Psychomotor Domain
Dave, 1970
Imitation
Manipulation
Precision
Articulation
Naturalization
Repeat after demonstration
Perform after instruction
Add accuracy, refinement
Add sequence, speed
High proficiency; habit
Affective Domain
Krathwohl, 1964
Receiving
Responding
Valuing
Organization
Characterization
Aware, sensitive
React overtly
Accept, prefer
Conscious values
Behavior shows
values
Types of Test “Items”
z
Selected Response
- Multiple choice
- Matching
- True-False
z
Constructed response
- Completion
- Short Answer
- Essay
NCLEX-RN© Items
z
z
z
z
z
Place respondents in realistic situations (not a
lot of detail)
Elicit higher level thinking
Are typically at application level or higher
1-2 minutes per item recommended
Vary in format
Multiple choice (1 or >1 answer)
Short answer or completion
Drag and Drop
Audio
Hot spots
Ordered response
Chart/exhibit
The Test Blueprint
Objectives Æ
Content Æ
Evaluation
Test Blueprint is Linked to Validity
z
z
z
z
Validity: Test measures intended content
Indirectly indicates fairness of a test
Not quantified; rather determined by evidence
Types
Criterion validity – r/t variables external to the test
Face validity - what a test appears to measure;
not sufficient evidence of validity by itself
Content validity - ensured by closely following the test
blueprint which reflects well-written objectives
Creating a Blueprint
z
z
z
z
z
z
z
z
Draft overview of course objectives, units, or
content (in collaboration with colleagues)
Estimate relative teaching time
Transfer these proportions to the test plan
Consider all steps in the nursing process
Include all possible levels of the cognitive domain
Space tests so that content is proportional across
the semester
Consider the school’s social calendar
Share with students
Ways to Depict
Course Organization
By Objectives
By Topics
Course Goal 1
Course Objective 1
Unit 1
Topic 1 (2 hr)
Topic 2 (3 hr)
Class 1 Objectives 1-4
Class 2 Objectives 5-8
Unit 2
Class 3 Objectives 9-12
Class 4 Objectives 13-20
Class 5 Objectives 21-23
Course Goal 2
Unit 3
Class 6 Objectives 21-26
Class 7 Objectives 27-30
Course Goal 3
Unit 4
Class 8 Objectives 31-35 (etc.)
Course Objective 2
Topic 3 (1.5 hr)
Topic 4 (2 hr)
Topic 5 (4 hr)
Course Objective 3
Topic 6 (3 hr)
Topic 7 (2 hr)
Topic 8 (1 hr)
(etc.)
Simple Course Test Plan
Date
Test
Content
2/15
3/21
4/20
5/2
Unit 1
Unit 2
Unit 3
Final
Weeks 1-5
Weeks 6-10
Weeks 11-15
Week 16 +
Comprehensive
Items
Weight
60
60
60
120
20%
20%
20%
40%
Novotny & Griffin, 2006, page 113
Two-Way Table of Specifications
Content/Outcomes
Know Comp Appl Totals
(20%)
Antipsychotic agents (25%)
2
Antianxiety agents (25%)
2
Antidepressant agents (25%) 2
Antimanic agents (12.5%)
1
Antiparkinsons agents (12.5%) 1
Totals
8
(40%) (40%)
4
4
4
2
2
16
4
4
4
2
2
16
10
10
10
5
5
40
Billings and Halstead, 1998, p. 387
Three-Way Table of Specifications
Outcomes/Process
Assessment (40%)
Diagnosis (10%)
Planning (10%)
Intervention (20%)
Evaluation (20%)
Totals
Know
Comp
Appl
(20%)
(40%)
(40%)
Totals
P,A,A,D,D,M
P,P,A,D,M,PA P,A,D,PA
P
A,D,M
PA
A,D
P
P,A
P,A,A,D,M,PA
P,PA,D
P,A,D,D,M
16
4
4
8
8
16
16
40
8
P=antipsychotic agents; A=antianxiety agents; D=antidepressive agents;
M=antimanic agents; PA=antiparkinson agents
Billings and Halstead, 1998, p. 388
Test Blueprint Example 1 of 3
(Total Number of Items)
Used with permission of Dr. Jeanne Wissmann, Graceland University
z
z
Total Lecture Hours/Course: 12 weeks X 3 lecture hrs = 36 hrs
Total Exam Items/Course: three 50-item Unit Exams +
one 100-item Final Exam = 250 items
Content
Lecture Hrs
Exam Items
Unit Exam
Items
Final Exam
Items
Unit I
9 (25%)
60-65
50
10-15
Unit II
9 (25%)
60-65
50
10-15
Unit III
9 (25%)
60-65
50
10-15
Unit IV
9 (25%)
60-65
0
60-65
36
250
150
100
Total
Test Blueprint Example 2 of 3
(Number/Distribution of Items)
Used with permission of Dr. Jeanne Wissmann, Graceland University
Learning Outcomes (Unit I Objectives)
# of
items
1.
Apply the overall therapeutic objective of drug therapy to an understanding of the ideal properties of
drugs and to specific drug selection by prescribers and in the preparation, administration, and
evaluation of prescribed medications to clients.
1
2.
Appreciate that the nurse’s role in medication administration includes the six rights of drug
administration and other actions taken prior to, during, and following medication administration
3
3.
Distinguish between generic and trade drug names, over-the-counter and prescribed medications,
Schedules I-V for Controlled Substances, and Phases I-IV of Clinical Testing for FDA approval of
New Drugs.
2
4.
Demonstrate an understanding of the terms (side effect, adverse effect, toxicity, nephrotoxicity,
neurotoxicity, cardiotoxicity, ototoxicity, idiosyncratic effect, carcinogenic effect, teratogenic effect,
allergic reaction, iatrogenic disease, physical dependence, additive effects, and medication error)
and the nurse’s role in preventing, recognizing, and responding to each of these possibilities.
4
Test Blueprint Example 3 of 3
(Types of Items)
Used with permission of Dr. Jeanne Wissmann, Graceland University
Unit Objective
[Goal?]
Behavioral Evidence
[Objective?]
Appreciate that the
nurse’s role in
medication
administration
includes the six
rights of drug
administration and
other actions taken
prior to, during, and
following medication
administration
•Recognize errors in the use
of the six rights
•Identify client findings preadministration which indicate
a need to hold a prescribed
medication
•Identify precautions to be
taken during administration
•Identify client findings postadministration that indicate
therapeutic effects.
Targets (e.g.,
Thinking Skill,
Nursing Process)
Analysis
Assessment
Item
Format
MC
Timing – Planning Exams
Develop the blueprint
z Prepare items
z Select items to use
z Peer review test
z
Types of Test Items
Matching
True-False
Short Answer
Essay
Multiple
Choice
Evidence of Learning
New information is linked to what the
learner already knows.
z The learner can use new information;
not just remember facts.
z
Recognition = Recall
“A small boat found
in the Far East, propelled
by a single oar at the rear,
and often covered by a
roof of straw mats.”
Multiple Choice
A patient with chronic cancer pain has
been receiving morphine sulfate 100
mg/hour intravenously for two months .
Yesterday the dose was increased to 125
mg/hour. The nurse will need to be
observant for which common side effect?
a. respiratory depression
b. constipation
c. nausea
d. agitation
Terms:
Stem
Options:
Distractor
Key (Answer)
Distractor
Distractor
Distractors
z
z
z
z
Directly influence student performance on
the test
Are intended to be attractive to low scorers
who have not mastered the content being
tested
Should be clearly incorrect
Must be plausible
Advantages of Multiple Choice
Learner is able to
– Analyze phenomena
– Apply principles to new situations
– Comprehend concepts and principles
– Discriminate between fact and opinion
– Interpret cause-and-effect relationships
– Interpret data
– Make inferences
– Solve problems
Burton, et.al, 1991
Limitations of Multiple Choice
Requires teacher time and skill
Learner is unable to
– Articulate explanations
– Display thought processes
– Furnish information
– Organize personal thoughts
– Produce original ideas
– Provide examples
– Perform a task
Burton, et.al, 1991
Higher Reliability with MC Tests
z
z
z
z
z
z
Not as susceptible to guessing as with T-F
Increased number of items offsets guessing effect
Scoring is more accurate than short answer
Samples a broader range of content than essay
Not affected by scorer bias as with essay
Immune to bluffing and effects of writing ability as
with short answer and essay
Burton, et.al., 1991
Constructing
a Multiple Choice Test
z
z
z
z
z
Phrase the stem as a question if possible
Make the stem longer than the options
Develop the correct answer first
Ensure there is only one correct answer
Be careful about multiple answers
Scoring issues; Most experts recommend only 1
z
z
z
Keep options as brief as possible
Make options conceptually congruent
Consider the number of options
Constructing
a Multiple Choice Test
z
z
z
z
z
Include directions to the students (purpose of
the test, time allotted, basis for responding, value
of items)
Use simple, precise, unambiguous wording
Make each item independent from others
Avoid grammatical and typographical errors
Space evenly; place items on same page
Constructing
a Multiple Choice Test
z
z
z
z
z
Arrange items from simple to complex
Consider grouping related information
Balance the placement of the correct
answers
Plan sufficient time: 30 sec-1 minute to
respond to each item (depending on difficulty)
60 minutes = 50 (4-option) MC items
Use at least 25 items for the test
1 in 942,651 chance to score 70% by guessing
Encourage Critical Thinking
z
z
z
z
Ask students to draw on their nursing
knowledge, apply concepts
Use novel, unique clinical scenarios
“Which of these actions would be most
appropriate for the nurse…”
Use best answer rather than correct answer
McDonald, 2007
Example: Knowledge (Remembering)
In a patient with myocardial infarction, which
should be monitored during titration of
nitroglycerine?
a. pulse rate
b. blood pressure
c. heart rhythm
d. ST segment
Nursing Spectrum, January 29, 2007, p. 19
Example: Comprehension
(Understanding)
Which of the following actions, if performed by the
nurse, would be considered negligence?
a. The nurse obtains a Guthrie blood test on a 4-day-old infant.
b. The nurse massages lotion on the abdomen of a 3-year-old
diagnosed with Wilm’s tumor.
c. The nurse instructs a 5-year-old asthmatic to blow on a
pinwheel.
d. The nurse plays kickball with a 10-year-old with juvenile
arthritis.
http://www.4tests.com/exams/questions.asp?exid=8515025&googlebot=78
Example: Application
(Applying)
On the first postoperative day after an open
reduction and internal fixation of the tibia with
application of a long leg cast, the patient begins
complaining of severe pain uncontrolled by
intravenous medications. What is the priority
nursing intervention?
a. Administer a narcotic bolus per protocol
b. Assess neurovascular status
c. Lower the leg to increase arterial flow
d. Raise the leg to decrease venous return
Billings and Halstead, 1998, p. 394
Example: Analysis
(Analyzing)
A 70-year old woman is admitted to the hospital
with a diagnosis of dehydration. Laboratory
values: serum sodium=165 mmol/L; serum
osmolarity=320 mOsm/L. Which type of fluid
replacement is most likely to be administered?
a. Hypertonic
b. Hypotonic
c. Isotonic
Billings and Halstead, 1998, p. 394
Examples: Evaluating
z
z
When a client who has a [dx] is being treated
with [medication], which of these laboratory
data would indicate that the client’s condition
is improving?
Which of these client observations is the
most reliable indicator that the client has the
correct understanding of how to [give
injection, follow diet, avoid side effects]?
McDonald, 2007
Avoid Common Mistakes in
Developing MC Items
Aim for Higher than
Knowledge/Remembering
Which care delivery model is represented
when a unit uses RNs, LPNs, and UAPs to
provide direct care?
a.
Primary nursing.
b.
Team nursing.
Loretta Manning, ICAN Publishing. Used with permission.
M-u-c-h Better Æ Analyzing
Which assignment would be most appropriate to delegate
to the LPN?
a. A client in acute renal failure with a potassium
level of 5.9 mEq/L.
b. A four-day post operative GI surgical client
with a nasogastric tube.
c. A newly diagnosed diabetic client who needs
to learn about insulin.
d. An order to administer 2 units of blood to a
newly admitted client.
Loretta Manning, ICAN Publishing. Used with permission.
Critique
Which of these is an example of the ninth
cranial nerve?
a. Glossopharyngeal
b. Olfactory
c. Spinal accessory
d. Trigeminal
Don’t Settle for Remembering
The physician has just completed a liver
biopsy. Immediately following the procedure,
the nurse will position the client
a. On his right side
b. In Fowler's position
c. Supine
d. In Sims' position
Adapted from http://www-nix.oit.umass.edu/~helene/Nclexfinal.htm
Ask for More than Knowledge of
Facts/Trivia
(Use “memory-plus” items)
The physician has just completed a
liver biopsy. Immediately following the
procedure, the nurse will position the client
a. On his right side to promote hemostasis
b. In Fowler's position to facilitate ventilation
c. Supine to maintain blood pressure
d. In Sims' position to prevent aspiration
http://www-nix.oit.umass.edu/~helene/Nclexfinal.htm
Or… describe a range of nursing
actions from which to choose
Which of these measures should a nurse include in the
care plan for a client who had a liver biopsy 1 hour
ago?
a. Monitor the neurovascular status of the client’s
affected extremity.
b. Advise the client to maintain a right side-lying
position.
c. Check the patient’s urine for occult blood.
d. Instruct the client to move slowly when arising from a
sitting position.
McDonald, 2007
Make options
conceptually congruent
Conditions that impair the accuracy of the pulse
oximetry include all of the following except?
a. Temperature of the finger
b. Position of the probe
c. Raynaud’s Disease
d. Quality of the respirations
Loretta Manning, ICAN Publishing. Used with permission.
Increase level of thinking
The pulse oximeter alarms indicating the
oxygen saturation is at 86%. What nursing
intervention should have the highest priority?
a. Administer oxygen via mask.
b. Check the position of the probe.
c. Complete a cardiovascular assessment.
d. Reset the pulse oximeter alarm.
Loretta Manning, ICAN Publishing. Used with permission.
Emphasize Qualifiers
(only, best, first, not, except)
Signs and symptoms of hypernatremia would
include all of the following, except
a. restlessness
b. muscular weakness
c. tachycardia
d. thirst
Use CAPS, bold, underline, and/or italics
Confusing: Negative stem +
Negative options
When physically restraining a client, a nurse should
consider all of these standards of care EXCEPT
a. obtaining the client’s consent for the restraint
b. Using the least restrictive device for the shortest
time
c. not keeping restraints on continuously
d. applying restraints when nonrestrictive
alternatives are not effective
McDonald, 2007
Use Positive Stem & Options
Which of these standards of care should a nurse
include when caring for a client who is physically
restrained?
a. Obtaining a signed consent from the client
b. Using the most restrictive device available.
c. Removing restraints at regular intervals
d. Using a nonrestrictive alternative in addition
to the restraint
McDonald, 2007
Don’t Give Away the Answer!
What is the process by which an oral
medication is greatly metabolized by the
liver before it reaches the systemic
circulation?
a. Metabolism
b. Absorption
c. Distribution
d. Elimination
Loretta Manning, ICAN Publishing. Used with permission.
Test at Entry-level; “Need to know”
You are taking the history of a 14 year old girl who has a
BMI of 18. The girl reports inability to eat, induced
vomiting and severe constipation. Which of the
following would you most likely suspect?
a. Multiple sclerosis
b. Anorexia nervosa
c. Bulimia
d. Systemic sclerosis
http://www.testprepreview.com/modules/nclex.htm
More appropriate
A client with asthma has received an order for
propranolol (Inderal). What is the highest priority
nursing action before initiating the order?
a. Assess breath sounds 90 minutes after
administering the medication.
b. Evaluate heart rate to determine a problem with
tachycardia.
c. Monitor the blood pressure prior to administering
the medication.
d. Verify the accuracy for this medication order
Loretta Manning, ICAN Publishing. Used with permission.
Test Only One Major Point
or Concept per Item
A 93 year-old female with a history of Alzheimer’s Disease
Gets admitted to an Alzheimer’s unit. The patient has exhibited
signs of increased confusion and limited stability with gait.
Moreover, the patient is refusing to use a wheelchair. Which of
the following is the most appropriate action for the nurse?
a. Recommend the patient remain in her room at all times
b. Recommend family members bring pictures to the
patient’s room
c. Recommend a speech therapy consult to the doctor
d. Recommend the patient attempt to walk pushing the
wheelchair for safety
Source: http://www.testtrade.com/nclex1.pdf
Put Common Information in the Stem
(Redundant Options)
The contraction stress test (CST) is used frequently in the
management of postdate pregnancies. The main reason that a
CST is performed is which of the following?
a. To determine whether the fetus is ready to be delivered
b. To determine whether the placenta is adequately
oxygenating the fetus
c. To determine whether the fetus can tolerate normal labor
prior to elective induction
d. To determine whether the uterus will be able to contract
efficiently enough to establish an adequate labor pattern.
Better:
A contraction stress test (CST) is done
primarily to determine whether:
a. the fetus is ready…
b. the placenta is adequately…
c. the fetus can tolerate…
d. the uterus will be able…
BUT…
Don’t Emphasize Options
A contraction stress test (CST) is done
primarily to determine whether:
a. the fetus is ready…
b. the placenta is adequately…
c. the fetus can tolerate…
d. the uterus will be able…
Avoid cues from wording of options
When brushing the teeth of patients who have
severe disabilities and are prone to gagging and
choking, caregivers should use which of the
following on the brush?
a. hydrogen peroxide
b. a minty mouthwash
c. a pea-sized amount of toothpaste
d. enough toothpaste to cover the entire
brushing surface
AJN, June 2009
Avoid Cues from Opposite Options
A client has a Sengstaken-Blakemore tube
connected to low wall suction. When the client
develops respiratory distress, which of these actions
should a nurse take?
a. Inflate the tube’s esophageal balloon
b. Deflate the tube’s esophageal balloon
c. Decompress the tube’s gastric balloon
d. Increase the amount of wall suction
McDonald, 2007
Start Options with 4 Different Words
Diane, a 28-year-old female, is admitted to the hospital with
a diagnosis of anorexia nervosa. She is 5'6" tall and
weighs 90 pounds. She appears weak and lethargic, with
very poor skin turgor. Diane is placed on a high-protein,
high-carbohydrate diet.
The primary function of carbohydrates in the body is to:
a. Develop a cushion around vital organs
b. Break down and transport proteins
c. Supply energy to the body
d. Maintain fluid balance
http://www.nclex-pass.com/sample_questions.htm
Or Use the 2 +2 Approach
Melena is blood loss in the stool which is:
a. less than 20 ml
b. greater than 20 ml
c. less than 50 ml
d. greater than 50 ml
Source: http://www.nursingstudy.com/nclex/nclex_practice_questions.html
Make Options Roughly
the Same Length
1. All of the following tasks may be delegated
to the LPN except:
a. developing a patient teaching plan regarding the patient's
diet, exercise, and medications.
b. monitoring Mrs. B's blood sugars via Accucheck.
c. performing dressing changes in the infected foot.
d. administering the patient's pain medication.
http://www.nclex-pass.com/sample_questions.htm
After the lungs, the kidneys work to maintain body
pH. The best explanation of how the kidneys
accomplish regulation of pH is that they
a. Secrete hydrogen ions and sodium.
b. Secrete ammonia.
c. Exchange hydrogen and sodium in the
kidney tubules.
d. Decrease sodium ions, hold on to hydrogen
ions, and then secrete sodium bicarbonate.
http://www-unix.oit.umass.edu/~helene/Nclexfinal.htm
Avoid Using
“Loaded” Words
To avoid infection after receiving a puncture
wound, you would advise someone to
a. always go to the doctor to get a tetanus shot
b. request an antibiotic only if the wound is
painful
c. ensure that no foreign object has been left in
the wound
d. never wipe the wound with alcohol unless it
is bleeding
Avoid Using “None of the Above”
Most nosocomial infections of the urinary
tract are caused by
a. Pseudomonas
b. Staphylococcus
c. Klebsiella
d. None of the above
z
z
Why is it not a good idea when “none” is the correct
answer?
One exception = math calculations
Be Careful with “All of the above”
z
z
Not a good substitute for well developed distractors
Cues students - How?
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXX?
a. XXXXXXXXX
b. XXXXXXX
c. XXXXXXXXXX
d. XXXXXXXXXXXXX
e. All of the above
Don’t Use Abbreviations
A nurse working a surgical unit, notices a patient
is experiencing SOB, calf pain, and warmth over
the posterior calf. All of these may indicate which
of the following medical conditions?
a. Patient may have a DVT
b. Patient may be exhibiting signs of dermatitis
c. Patient may be in the late phases of CHF
d. Patient may be experiencing anxiety after
surgery
http://www.testtrade.com/nclex1.pdf
Avoid bias
z
z
z
z
Avoid using gender or names
Don’t stereotype members of a group
Refer to race, culture, religion, marital status,
or sexual orientation only if pertinent
Avoid terminology unfamiliar to other cultures
–
–
“Stop smoking cold-turkey”
“Take the medication before you retire”
McDonald, 2007
Use Logical Order and Even Intervals
Hypoglycemia is a blood glucose of less
than
a. 60
b. 80
c. 100
d. 120
Avoid inconsistent responses
Henry and colleagues’ survey of nursing
home residents indicated that what
percentage of them had not seen a dentist
in more than 5 years?
a. 10%
b. 25%
c. 33%
d. 50%
AJN, June 2009, p.51
Avoid overlapping options
When should medication X be given?
a. Before breakfast
b. On a full stomach
c. With meals
d. Before going to bed
Fray, 1995
Clarify “Should” v. “Would”
Loberta Jackson, a 21-year-old college student is admitted to a
medical unit with diagnosis of uncontrolled diabetes and an
acute hypoglycemic reaction. Loberta says, “I’ve been feeling
"sick to my stomach, like I was coming down with the flu " for the
past 48 hours. She has continued to take her usual daily insulin.
noting that Loberta’s admission blood glucose is 53, which
question would you ask first?
a. "Have you been under a great deal of stress lately, Loberta?
b. "Were you having difficulty sleeping after this illness started?”
c. "Have you eaten anything in the past 48 hours?"
d. "Did you take any medications for this illness other than your
insulin ?“
Adapted from http://www.nclex-pass.com/sample_questions.htm
Clarify “Should” v. “Would”
A 32 year-old male with a complaint of dizziness
has an order for Morphine via. IV. The nurse
should do which of the following first?
a. Check the patient’s chest x-ray results.
b. Retake vitals including blood pressure.
c. Perform a neurological screen on the patient.
d. Request the physician on-call assess the
patient.
http://www.testtrade.com/nclex1.pdf
Multiple response MC - NCLEX RN
The nurse is caring for a client who has a wound infected with
methicillin-resistant Staphylococcus aureus (MRSA). Which of the
following infection control precautions should the nurse implement?
a. Wear a protective gown when entering the client’s room.
b. Put on a particulate respirator mask when administering
medications to the client.
c. Wear gloves when delivering the client’s meal tray.
d. Ask the client’s visitors to wear a surgical mask when in the
client’s room.
e. Put on a face shield before irrigating the client’s wound.
2006 NCLEX Examination Candidate Bulletin
Combined or Complex Multiple
Choice
The fluid imbalance known as edema is commonly associated with
1. Allergic reactions
2. Congestive heart failure
3. Extensive burns
4. Protein deficiency
a. 1, 2, and 3
b. 1 and 3
c. 2 and 4
d. 4 only
e. 1, 2, 3, and 4
Burton, et. al., 1991
Consider the Nursing Process and
Cognitive Domain Levels
A 22 year-old patient in a mental health lock-down unit
under suicide watch appears happy about being discharged.
Which of the following is probably happening?
a. The patient is excited about being around family again.
b. The patient’s suicide plan has probably progressed.
c. The patient’s plans for the future have been clarified.
d. The patient’s mood is improving.
Nursing Process = Analysis
Cognitive Level = Comprehension (Understanding)
http://www.testtrade.com/nclex1.pdf
Critique
Briefly describe the electrical conduction
system of the heart.
Critique
Based on the article that was assigned to
read on Monday, what diuretic was
discussed throughout the article?
a. furosemide (Lasix)
b. hydrochlorothiazide (HCTZ)
c. mannitol (Osmitrol)
d. spironolactone (Aldactone)
Source: Loretta Manning, ICAN Publishing. Used with permission.
Item Analysis
z
z
z
z
Functional Analysis
Reliability -- KR-20
Difficulty – p-value
Discrimination -- PBI
Functional Analysis
Assessment of degree to which wrong answers
were selected by students
z
z
z
Response frequencies
High and low performers
Non-distractors
Functional Analysis Response Frequencies
Nurs 471 Exam 1, Question 6
Response
Frequency
Percent
A
9
12.50
B*
52
72.22
C
8
11.11
D
3
4.17
E
0
0.00
Total 72
100.00
Functional Analysis High and Low Performers
Item
1
2
3
4
5
Correct Group Responses
Total
Upper 27% Lower 27%
86.67%
100.00%
62.50%
93.33%
100.00%
75.00%
40.00%
87.50%
12.50%
73.33%
87.50%
75.00%
100.00%
100.00%
100.00%
McGahee & Ball, 2009
Functional Analysis Non-distractors
Exam 390 A, Question 7
Response
Frequency
Percent
A
0
0.00
*B
57
100.00
C
0
0.00
D
0
0.00
E
0
0.00
Totals
57
100.00
Reliability
Refers to consistency of test results or “reproduceability”
How well test is measuring similar characteristics
Not a measure of the test per se
Three types:
test-retest – same test given again to same group
parallel forms – same person gets 2 versions of test
internal consistency – from a single administration
http://www.som.tulane.edu/ome/helpful_hints/test_analysis.pdf
McGahee and Ball, 2009
Reliability – Internal Consistency
How well the exam measures a single cognitive factor
Reported in a range: 0.0 -1.0
Kuder-Richardson – 20 (KR-20)
used with dichotomous items (correct or incorrect)
best choice for use with MC tests
Desired KR-20 = 0.50-0.85
Coefficient Alpha
best used with surveys or attitude measures
http://www.som.tulane.edu/ome/helpful_hints/test_analysis.pdf
McGahee and Ball, 2009
Degree of Difficulty – p-value
z
z
z
z
z
z
% or proportion of students answering correctly
Reflects ease of item
Reported 0.0-1.0
Higher number = easier item
Reflects the group taking the test; not the test itself
Desired p-value = 0.50 -0.80
Discrimination – Point Biserial
z
z
z
z
Basic indicator of item quality
Correlation between how well students did on that
item and their total test score
Typically, upper 1/3 and lower 1/3 assessed
Goal:
–
–
z
Positive PBI for the correct answer (high achievers chose
this) and
negative (or near zero) PBI for each distractor (low
achievers chose these)
Desired PBI = > 0.20
Point Biserial
Response
A
*B
C
D
E
Exam 1, Question 6
Frequency Percent
9
12.50
52
72.22
8
11.11
3
4.17
0
0.00
Total 72
100.00
PBis
-0.28
0.31
-0.12
-0.05
-
Item Analysis
Response
*A
B
C
D
E
Exam 2, Question 32
Frequency Percent
66
91.67
0
0.0
4
5.56
2
2.78
0
0.0
Total 72
100.00
PBis
0.13
-0.13
-0.03
Questions?
Advice to Colleagues?
Selected References
Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for learning,
teaching, and assessing: A revision of Bloom’s taxonomy of
educational objectives. New York: Addison Wesley, Longman, Inc.
Burton, S.; Sudweeks, R.; Merrill, P.; & Wood, B. (1991). How to
prepare better multiple choice items: Guidelines for university
faculty. Brigham Young University Testing Services and
Department of Instructional Science. Retrieved January 22, 2010
from http://testing.byu.edu/info/handbooks/betteritems.pdf
Frary, R.B. (1995). More multiple-choice item writing do’s and don’ts.
Practical Assessment, Research, and Evaluation 4(11). Retrieved
January 22, 2010 from http://PAREonline.net/getvn.asp?v=4&n=11
Selected References
Gaberson, K.B. & Oermann, M.H. (2005). Evaluation and Testing in
Nursing Education, 2nd ed. NY: Springer
Kehoe, J. (1995). Writing multiple-choice test items. Practical
Assessment, Research, and Evaluation 4(9). Retrieved January
22, 2010 from http://PAREonline.net/getvn.asp?v=4&n=9
McDonald, M. E. (2007). The Nurse Educator’s Guide to Assessing
Learning Outcomes, 2nd Ed. Boston: Jones and Bartlett Publishers
McDonald, M. E. (2008). Developing trustworthy classroom tests. In
Mastering the Teaching Role: A Guide for Nurse Educators, B.
Penn, ed. Philadelphia: F.A. Davis
McGahee, T.W., & Ball, J. (2009). How to read and really use an item
analysis. Nurse Educator, July/August, v. 34, n. 4, pp. 166-171.
Selected References
Morrison, S. (2004). Test construction and analysis: Can I do it? In
Teaching Nursing: The Art and Science, Volume 1, Caputi and
Englemann, eds. Glen Ellyn, IL: College of DuPage Press, pp.
366-387
National Council of State Boards of Nursing. (2007). NCLEX-RN©
Examination: Test Plan for the National Council Licensure
Examination for Registered Nurses. Retrieved August 15, 2009
from http://www.ncsbn.org/RN_Test_Plan_2007_Web.pdf
Novotny, J.M. & Griffin, M.T. (2006). A nuts-and-bolts approach to
teaching nursing, 3rd ed. New York: Springer Publishing, pp. 111133.
Oermann, M.H. & Gaberson, K.B. (2009). Evaluation and testing in
Nursing Education, 3rd edition. New York: Springer.
Selected References
Rasmussen, L., Speck, D., & Twigg. P. (1998). Developing and
using classroom tests. In Teaching in Nursing: A guide for
faculty, 3rd edition, Billings & Halstead, eds. Philadelphia:
W.B. Saunders, pp. 385-405
Measurement and Evaluation Center, The University of Texas at
Austin. (2003). Test item analysis and decision making.
Retrieved January 22, 2010 from
http://www.utexas.edu/academic/mec/research/pdf/itemanalysis
handout.pdf
Tulane School of Medicine. Test and item analysis. Retrieved
January 22, 2010 from
http://www.som.tulane.edu/ome/helpful_hints/test_analysis.pdf