Download The Commandments of Test Item Writing - LeBlanc

Improving your classroom
assessments
Dr. Patty LeBlanc
Southeastern University
Agenda
 Introductions
 Objectives
of the Seminar
 Purpose of Educational Assessments
 Educator and Educational Standards
 Bloom
 Test Blueprints
 Item Writing Commandments
 Item
Critique Activity
 Item Analysis
 Example from grad students
 Authentic Assessment
 Rubrics
 Summary
 Action Plan
 Questions?
Objective of the Seminar

To help teachers design valid
assessments of instruction that inform
instruction
What is the purpose of testing?
Participant Ideas
To encourage students and
improve teaching




To assess the extent to which students have met
the intended learning outcomes, and to look for
and evaluate unintended outcomes
To recognize achievements and diagnose learning
difficulties so that students learn to build on their
strengths and overcome or cope with their
weaknesses
To refine instruction and learning experiences in
order to improve both individual and class
learning
To help students set meaningful and realistic
learning goals and assume responsibility for their
own learning.
To communicate meaningful information to students,
parents, and school authorities about student
learning

To give realistic and helpful feedback
about achievement, capabilities, behavior,
attitudes, and dispositions
 To put teachers, students, and
parents/guardians in touch with one
another about progress over time
 To provide guidance for educational and
vocational choices
 To report learning achievement to school
authorities such as school boards and
governments
Florida Educator Accomplished
Practices in Assessment
The effective educator consistently:
 Analyzes and applies data from multiple assessments and measures to
diagnose students’ learning needs, informs instruction based on those
needs, and drives the learning process;
 Designs and aligns formative and summative assessments that match
learning objectives and lead to mastery;
 Uses a variety of assessment tools to monitor student progress,
achievement and learning gains;
 Modifies assessments and testing conditions to accommodate learning
styles and varying levels of knowledge;
 Shares the importance and outcomes of student assessment data with
the student and the student’s parent/caregiver(s); and
 Applies technology to organize and integrate assessment information.
Educational Standards
Florida Sunshine State Standards
 Common Core Standards
 Other

“Backward” Design
The standards dictate the lesson
objectives…
 The objectives dictate the instruction
 The assessment must align with the
objectives, standards, and instruction in
order to be valid.

Blessed AssurAnce…

A fantastic way to ensure that your test
items align with your objectives: A Test
Blueprint!
Content Validity

Think matrix of instruction and assessment:
Objective
1
2
Item
1
X
Item
2
Item
3
Item
4
X
Item
5
X
X
3
4
X
X
Bloom’s Taxonomy of the
Cognitive Domain
Another blueprint…
Item #
Standard/Objective
Knowledge/Comp
Above
Knowledge/Comp
Comment
Sung to the tune of the Farmer in the
Dell
The teacher follows the standards…
 The teacher creates the objectives…
 The teacher writes the test…
 The teacher teaches the lesson…
 The students learn a lot…
 The parents beam with pride…

Comunicación y Gerencia
The Commandments
of Test
ItemWriting
Writing
Item
Commandments
By W.James Popham
Click to add Text
General Item Writing Items
Thou shalt not

provide opaque directions to students
regarding how to respond to your
assessment instruments.
Thou shalt not

employ ambiguous or wordy statements in your
assessment items.
The term hypothesis, as used in research, as defined as:
a. A conception or proposition formed by speculation or deduction or by abstraction and
generalization from facts, explaining or relating an observed set of facts, given
probability by experimental evidence or by factual or conceptual analysis but not
conclusively established or accepted.
b. A statement of an order or relation of phenomena that so far as is known is invariable
under the given conditions, formulated on the basis of conclusive evidence or tests
and universally accepted, that has been tested and proven to conform to facts.
*c. A proposition tentatively assumed in order to draw out its logical or empirical
consequences and so test its accord with facts that are known or may be determined,
of such a nature as to be either proved or disproved by comparison with observed or know
facts.
Better…
The term hypothesis, as used in research, is
defined as:
a. An assertion explaining an observed set of facts
that has not been conclusively
established.
b. A universally accepted assertion explaining an
observed set of facts.
*c. A tentative assertion that is either proved or
disproved by comparison with an observed set of
facts.
Thou shalt not

Provide unintentional clues as to the correct
response.
What meter is used to measure resistance?
a. Voltmeter
b. Ohmmeter
c. Ammeter
d. Oscilloscope
Thou shalt not

employ complex or give-away syntax in
your assessment items.
In item response theory, the one-parameter
model assumes that each item …….
a. discriminates equally well.
b. students perform equally well.
c. students score the same across items.
d. guessing affects all items the same.
Thou shalt not

use vocabulary that is more advanced than
required.
As the level of fertility approaches its nadir, what is the most likely
ramification for the citizenry of a developing nation?
 a decrease in the labor force participation rate of women
 a dispersing effect on population concentration
 a downward trend in the youth dependency ratio
 a broader base in the population pyramid
 an increased infant mortality rate
Thou shalt

Use plausible distractors
Which of the following artists is known for
painting the ceiling of the Sistine Chapel?
a. Warhol
b. Flintstone
*c. Michelangelo
d. Santa Claus
Thou Shalt

Avoid using "always" and "never" in
the stem or the distractors.
Thou shalt

Rarely us negative statements…
Which of the following is NOT a symptom of
osteoporosis?
 decreased bone density
 frequent bone fractures
 raised body temperature
 lower back pain
Binary Choice Items [True/False]

One of several types of Selected
Response Items
Thou shalt not

Phrase items so that a superficial
analysis by the student suggests a
wrong answer.
Which of the following artists is known for painting the
ceiling of the Sistine Chapel?
a. Warhol
b. Flintstone
*c. Michelangelo
d. Santa Claus
Thou shalt
NEVER use double negatives.
True or False: There is no advantage to
not using specific determiners in true-false
items.
Thou shalt
Have an approximately equal number of
items representing the two categories
being tested.
 Keep item length similar for both
categories being tested.

Thou Shalt

Stick with absolute truths or falsehoods.
True or False: Nearsightedness is
hereditary in origin.
Thou Shalt Not

Use extreme qualifiers.

All sessions of Congress are called by the President.
(F)
The Supreme Court frequently rules on the
constitutionality of law. (T)
An objective test is generally easier to score than an
essay test. (T)


Multiple Choice Items
Thou shalt

Write the stem so that it consists of a
self-contained question or problem.
California:
a. Contains the tallest mountain in the United States
b. Has an eagle on its state flag.
c. Is the second largest state in terms of area.
*d. Was the location of the Gold Rush of 1849.
Better…
What is the main reason so many people moved
to California in 1849?
a. California land was fertile, plentiful, and
inexpensive.
*b. Gold was discovered in central California
c. The east was preparing for a civil war.
d. They wanted to establish religious settlements.
Thou shalt

Include only one concept in each
statement and only one correct answer.
The United States should adopt a foreign policy based
on:
a. A strong army and control of the North American continent.
b. Achieving the best interest of all nations.
c. Isolation from international affairs.
*d. Naval supremacy and undisputed control of the world’s sea
lanes.
Thou shalt

NEVER use ‘all of the above’ alternatives, only
occasionally use ‘none of the above’
alternatives [to increase item difficulty].
Alternative
Weakness
“All of the above”
Distractor can be eliminated by noting that one of the other alternatives
is incorrect
“None of the above”
Answer measures the ability to recognize incorrect answers rather than
correct answers [remember educational psychology 101?].
Distractor does not appear plausible to some students.
Well-designed MC items CAN
measure HOTS
A common goal of the Salt March in India, the
Boxer Rebellion in China, and the Zulu
resistance in southern Africa was to:
A. overthrow totalitarian leaders
B. force upper classes to carry out land reform
programs
C. remove foreign powers
D. establish Communist parties to lead the countries
Numbers of Items is Important!


Number of 4-Alternative
MC Items






2
5
10
15
20
25
Chance of Scoring 70% or
Higher by Blind Guessing
•
•
•
•
•
•
1 out of 16
1 out of 64
1 out of 285
1 out of 8,670
1 out of 33,885
1 out of 942,651
Pop Quiz

Can you identify the problem with this
item?
Due to budget cutbacks, the university library
now subscribes to fewer than _?_ periodicals.




25,000
20,000
15,000
10,000
Matching Items
Most effective when you need to measure the learner’s
ability to identify the relationship or association between
similar items. They work best when the course content
has many parallel concepts, for example:
 Terms and Definitions
 Objects or Pictures and Labels
 Symbols and Proper Names
 Causes and Effects
 Scenarios and Responses
 Principles and Scenarios to which they apply
Thou shalt

Use homogeneous lists.
Hitler
Mulroney
Tank
1939
A. Year in which WWII began
B. A Canadian Prime Minister
C. A German dictator during
WWII
D. An armored vehicle used
originally to break the trench
war stalement in WWI
Thou shalt

Randomly assign correct answers to
alternative positions.
Logical order of responses
Example
Numerical
a. 1939
b. 1940
c. 1941
d. 1942
Alphabetical
a. Changing a from .01 to .05.
b. Decreasing the degrees of freedom.
c. Increasing the spread of the exam scores.
d. Reducing the size of the treatment effect.
Sequential
a. Heating ice from -100°C to 0°C.
b. Melting ice at 0°C.
c. Heating water from 0°C to 100°C.
d. Evaporating water at 100°C.
e. Heating steam from 100°C to 200°C.
Thou shalt not

let the length of alternatives supply
unintended clues.
Which of the following is the best indication of high morale in a
supervisor’s unit?
a. The employees are rarely required to work overtime.
*b. The employees are willing to give first priority to attaining
group objectives,
subordinating any personal desires they may have.
c. The supervisor enjoys staying late to plan the next day.
d. The unit gives expensive birthday presents to each other.
Thou shalt

Use relatively brief lists, placing the
shorter words or phrases at the right.
Thou shalt

Employ more responses than premises.
Thou Shalt

Provide good directions!
Directions: Match the following.
____1. City dwellers
A. Wild animals
____2. Hunter-gatherers
B. Farm
____3. Pastoral nomads
C. Apartments
D. Graze animals
Better…
Directions: On the line to the left of each definition, write the letter of
the term in the right hand column that is defined. Use each term only
once. There are more responses than items. Each correct response
is worth 2 points.
_1. Live in areas of high population density.
A. Pastoral nomads
__2. Move from place to place in search of wild animals.
__3. Move from one place to another with grazing animals.
__4. Till land for cash crops.
B. Ranchers
C. Hunter-gatherers
D. City dwellers
E. Farmers
Thou shalt
Describe the basis for matching and the
number of times responses may be used
in the instructions, as well as the point
values for each item.
 Place all premises and responses for an
item on a single page.
 Tip: Be sure to decide ahead of time
whether you will give partial credit.

Matching HOTS item











Directions: Number (1-8) the following events in the history of
ancient Egypt in the order in which they occurred, using 1 for the
earliest event.
_____Egypt divided; ruled by Libyan kings, Nubian pharaohs,
Assyrians, and Persians
_____Seizure of power by Hyksos kings
_____Upper and Lower Egypt are united by Menes
_____Alexander the Great conquers Egypt
_____Reunification of Egypt under pharaoh Mentuhotop II
_____Rise of feudal lords leads to anarchy
_____Thutmose III expands empire to the Euphrates
_____Many kings with short reigns; social and political chaos
Short Answer/Essay Items

Aka Constructed Response Items
Short Answer/Fill in the Blanks
Thou Shalt

Ask a direct question.
Undesirable:
 Alloys are ordinarily produced by…
Desirable:
 How are alloys ordinarily produced?
Thou Shalt

Put a blank at the end of a sentence
rather than the beginning.
Undesirable:
____________ is the measure of central tendency that is most
 affected by extremely high or low scores.
Desirable:
 The measure of central tendency that is most affected by
extremely high or low scores is the ____________.
Thou Shalt

Omit only significant words.
Undesirable:
Every atom has a central ____________
called a nucleus.
Desirable:
Every atom has a central core called a(n)
____________.
Thou Shalt Not

Omit so many words that the meaning in
unclear.
Undesirable:
The ____________ were to Egypt as the ____________ were to
Persia and as ____________ were to the early tribes of Israel.
Desirable:
The Pharaohs were to Egypt as the ____________ were to Persia
and as ____________ were to the early tribes of Israel.
Thou Shalt Not

Use essay items to measure trivial ideas
Thou Shalt









Give very clear directions.
Know the criteria for grading and use a rubric.
Make expectations clear—have a good idea of what
the teacher expects in a good response
Not provide options…
Indicate point values and suggested time limit per item
Take the test yourself and see how you do…
Ask a colleague to take the test and critique it.
Consider using several short essays vs 1 long one.
Analyze student responses to improve the item in
future.
How do I decide???
Another Pop Quiz

Objective or Essay?
Rubrics for Essay Items
How can a classroom teacher improve
his/her tests?

Before the test:




During the test:




Table of specifications/matrix
Pilot testing, especially essay items and rubrics
Peer review
Observation
Timing
Ask the students to rate the test/items
After the test:




Item Analysis
Go over the test with students
Item and Test Revision
Re-examine instruction and re-teach if necessary
That’s a lot of trouble!
Yes, but it is the right thing to do.
 It’s intellectually honest.
 It makes grades easier to defend.

Test Improvement Project

By Christina Salas and Amanda Williams
Authentic Assessment

Aka performance based assessment

Great example: Edutopia
Authentic Assessment vs. Traditional
Assessment
Traditional --------------------------------------------- Authentic
Selecting a Response ------------------------------------ Performing a Task
Contrived --------------------------------------------------------------- Real-life
Recall/Recognition ------------------------------- Construction/Application
Teacher-structured ------------------------------------- Student-structured
Indirect Evidence -------------------------------------------- Direct Evidence
How to Measure Authentic
Assessments
Rubrics
 Cooperatively
 Panel of Experts

Steps in Rubric Development
1.
2.
3.
4.
5.
Determine the learning outcomes.
Determine the criteria
Determine measurable criteria [4,3,2,1]
Develop a matrix
Evaluate overall point value relative to
the task.
Grading cooperative work…
Participation in Class
Discussions

Sit down with a seating chart and mark
contributions to the discussion with a
rating system

For example, 5 being outstanding and 1
being poor, 0 being absent or silent
Grading cooperative work
Group grade
 Recognize that there will always be
loafers, workers, and compulsive worker
bees


See article on grading cooperative projects
Assessment should…









Encourage and improve student learning
Be a blessing for students that embraces, grace, justice, and
encouragement
Be an integral part of learning, closely related to overall learning
goals
Foster excellence in all a students’ gifts on the basis of the
student’s potential as a bearer of God’s image
Emphasize assessment FOR learning as well as assessment OF
learning
Develop plans that enable teachers and students to interact
frequently about students’ progress as well as challenges, with
opportunities to refine or revise
Use a variety of meaningful tasks, products, and performances,
with descriptive feedback to students
Use accurate descriptors for all grades based on the achievement
of learning outcomes
Communicate meaningful and clear information to students,
parents, and authorities.
Other suggestions?
Questions?
Resources

Van Brummelen, H. 2009. Walking with
God in the Classroom, 3rd edition.
Purposeful Design Publicatons.
 Brookhart, S. 2008. Feedback that Fits.
Educational Leadership 65, no.4:54-59.
 Stiggins, R. 1997. Student-centered
classroom assessment. 2nd edition. Merrill.
More Resources
How to prepare effective essay
questions
http://www.uwgb.edu/oira/teachLearn/bet
terTests/betteressays.pdf
 Steven J. Burton, et al, How to Prepare
Better Multiple-Choice Test Items:
Guidelines for University Faculty,
(http://testing.byu.edu/info/handbooks/be
tteritems.pdf).

Still More Resources
A Short Guide to Writing Effective Test
Questions
 Rubric Maker
 Quiz Star

Thank You!!!
To download this presentation:
http://leblancspace.wikispaces.com/home


[email protected]