Download UNIVERSITY OF MINNESOTA

The Evaluation of the Web-based ARTIST
Ann Ooms, Joan Garfield, Bob delMas – University of Minnesota
Assessment Resource Tools for Improving Statistical Thinking (ARTIST)
The goal of the Web-based ARTIST is to help teachers asses statistical literacy, statistical reasoning and statistical thinking.
The ARTIST website provides a variety of assessment resources for teaching first courses in statistics and contains:
 Item Database: a collection of over 1,200 items organized by topic and learning outcome
 11 Assessment Topic Scales and the Comprehensive Assessment of Outcomes in a first Statistics Course (CAOS)
 Information, guidelines, and examples of alternative assessments
 Information on implementation issues related to designing, administering, and evaluating assessments
 Direct links to or copies of articles on assessment in statistics
 References and links to other related assessment resources
 Copies of conference papers and presentations on the ARTIST project and handouts from ARTIST mini-courses
 Information on upcoming ARTIST events
 Information about ways to participate as a scale tester
Funded by NSF CCLI-ASA-0206571
The Iterative Evaluation Model for Improving Online Educational Resources
Components
Component 1
Evaluation planning
Re port
Component 1
Evaluation planning
Component 4
Evaluation of educational
impact
Component 2
Evaluation of educational value
Component 4a
Evaluation of impact on instructors’
perceptions
Component 4b
Evaluation of impact on educational
environment
Component 4c
Evaluation of impact on student outcomes
Component 4d
Evaluation of sustainability of impact
Component 2a
Evaluation of web design
Component 2b
Evaluation of web content
Component 3
Evaluation of use of the
educational resource
Report
Methods and instruments
Report
Evaluation Instruments
Meetings to plan evaluation
Creation of the pre evaluation interview
protocols
Pre evaluation interviews ARTIST users
and non ARTIST users
Reporting on pre evaluation interview
results
Component 2a
Creation of the survey for ARTIST users
Evaluation of web design
Creation of the observation protocol for
frequent Item Database users
Creation of the observation protocol for
first time Item Database users
Reporting on web design
Component 2b
The survey for ARTIST users
Evaluation of web content
Reporting on web content
Component 3
The survey for ARTIST users
Evaluation of use of the
Creation of the interview protocol for
educational resource
frequent ARTIST users
Reporting on use of the educational
resource
Component 4
Creation of the interview protocol for
Evaluation of educational impact frequent ARTIST users
Reporting on educational impact
Pre Evaluation Interview
Protocol ARTIST Users and
non-ARTIST Users
Survey for ARTIST Users
Survey for Non-ARTIST Users
Interview Protocol for
Frequent ARTIST Users
The interviews were conducted
to collect information about
today’s instructors of statistics
assessment practices. The
information was used to tailor the
other evaluation instruments.
Participants were statistics
instructors known by the
educational resource directors.
The survey was created and
administered to collect
information about the web
design, the web content, and
implementations in educational
settings.
All registered ARTIST users
teaching introductory statistics in
the US, who had accessed the
website since August 2004, and
who were teaching that semester
were invited.
The survey was created and
administered to collect
information about the non-users’
characteristics.
Invitations for submitting the
survey were sent by e-mail to the
following list serves: isostat
group, ASA Stat ed, and the
sigmaa listserv.
The interviews for frequent
ARTIST users were conducted to
collect information about
implementations in educational
settings and educational impact.
Participants were selected based
on the number of ARTIST topic
scales and ARTIST items they
had administered to their
students.
Observation Protocol for
Observation Protocol for FirstFrequent Item Database Users
Time Item Database Users
The observations were conducted
to collect information about the
navigatability of the Item
Database. We wanted to learn
where the Assessment Builder
was difficult to use, even for
experienced users.
Participants were asked to
conduct a think-aloud while
performing certain tasks, such as
create a test, add a question to
that test, remove a question.
The observations were conducted
to collect information about the
navigatability of the Item
Database. We wanted to learn
where the Item Database was
difficult to use for first-time
users.
Participants were asked to
conduct a think-aloud while
performing certain tasks, such as
create a test, add a question to
that test, remove a question.
Evaluation Findings
1. Web design: concrete recommendations have been made to improve the web design
2. Web content: recommendations have been made to include large data sets to the Item
Database and to add additional item topics.
3. Use: Many statistics educators are actively using ARTIST and value the resource. The main
reasons for statistics educators to not use ARTIST are time and misconceptions about what
ARTIST has to offer. Because of the evaluation, several non-users became more aware of
goal of ARTIST and started using ARTIST since.
4. Educational impact: instructors are changing assessment practices but there is no impact
yet on student outcomes because, so far, newly developed ideas have not been implemented
by instructors.
ARTIST
https://ore.gen.umn.edu/artist//
Contact information Ann Ooms
[email protected]