Download A3EH05-cds - Department of Computer Science

TU/
e
eindhoven university of
technology
Evaluation of Interoperability of Adaptive
Hypermedia Systems:
testing the MOT to WHURLE conversion in a classroom setting
Alexandra Cristea1, Craig Stewart2, Tim
Brailsford2 and Paul Cristea3
1 - Information System Department, Faculty of Mathematics
and Computing Science, Technical University Eindhoven
2 - School of Computer Science and IT, University of
Nottingham
3 - Digital Signal Processing Laboratory, “Politehnica”
University of Bucharest
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eOutline
eindhoven university of
technology
•
•
•
•
•
Introduction
MOT
WHURLE
MOT2WHURLE conversion
Student Evaluation
– Hypotheses
– Results
• Conclusion
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eIntroduction
eindhoven university of
technology
• Creation of adaptive hypermedia
content:
– Extremely time intensive
– Complex
– Platform lockdown
– Lack of standard tools
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eIntroduction
eindhoven university of
technology
• Ideal creation of adaptive hypermedia
content:
– Automated (this will aid with the time and
complexity factors)
– Interoperable (conversion, common
language)
– Standardised
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eIntroduction
eindhoven university of
technology
• Our answer:
– Develop a series of tools that will allow for
interoperability between different systems
– We use MOT as an authoring system to
author materials for:
• AHA! (an AEH)
• WHURLE (an AEH)
• Blackboard (a non-adaptive commercial
system, uses a pre-adapted methodology)
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eIntroduction
eindhoven university of
technology
• Here we present an evaluation of the
authoring process using MOT and
WHURLE
• A class of 31 students were introduced
to this new authoring paradigm
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eMOT
•
•
•
•
eindhoven university of
technology
My Online Teacher
Based on the LAOS framework
Is a generic AEH delivery system
Also a powerful and simple authoring
system
– Web-form, therefore a non-technical author
can easily use it
– Very flexible, as it is easy to extend
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eMOT
eindhoven university of
technology
• Domain Maps
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eMOT
eindhoven university of
technology
• Goal and Constraint Maps
– AND/OR
– Weights
– Labels
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eWHURLE
eindhoven university of
technology
• An adaptive XML learning environment
• Basic content building block: Chunk
• Structure applied using Lesson Plans
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eMOT2WHURLE
eindhoven university of
technology
• Currently a command line environment
• Conceptually maps the MOT Goal &
Constraints map structure to the
WHURLE Lesson Plan
– A MOT Concept = WHURLE chunks(s)
• It does this using the MOT weights and
labels.
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eMOT2WHURLE
eindhoven university of
technology
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eMOT2WHURLE
eindhoven university of
technology
• These rules are used to determine
which MOT attributes are to be
aggregated into WHURLE chunks.
– ’35’ = visual
– ’75’ = verbal
– ‘0’ = common
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eMOT2WHURLE
eindhoven university of
technology
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eStudent Evaluation
eindhoven university of
technology
• A class of 31 students in the 4th year of study
for a technical Masters degree at the
University of Bucharest, Romania
• All subjected to a week long intensive course
on AH
• Ideal representation for non-AH expert
authors.
• After initial lectures, they were given the
following task:
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eStudent Evaluation
eindhoven university of
technology
1.
2.
3.
4.
5.
Create 2-3 MOT Domain Concept Maps, with
approximately 5-10 concepts on the http://elearning.dsp.pub.ro/mot/ MOT server
The attributes of each concept were: title; keywords;
introduction; text; conclusion and figure. With limits placed
on the type and amount of content in each one (this was
done so as that each group would not spend their limited
time creating a vast corpus of information).
Create a single MOT Lesson (Goal & Constraints Map)
using their Concepts maps.
Alter the lesson so that the weights and labels of each
concept agreed with those described in the second Table
previously.
Run the ‘mot2whurle’ conversion program and copy the
files to WHURLE.
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eStudent Evaluation
eindhoven university of
technology
6.
7.
8.
Check that the WHURLE XML files are well-formed.
Run and login to WHURLE to check that the lesson
matches their design and make any necessary changes.
Finally at the end of the week, each student was asked to
complete a series of questionnaires: three generic SUS
(System Usability Scale) questionnaires, one for each
system (MOT, mot2whurle and WHURLE) and a single
specific questionnaire designed to determine their level of
knowledge about each system, as well as to gather nonstatistical information.
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eEvaluation hypotheses
eindhoven university of
technology
•
The hypotheses that we wished to examine were:
1.
The systems (MOT, mot2whurle, WHURLE) are simple
and intuitive to use, with a minimum amount of
explanation.
2. The students understood the theoretical background
(Adaptive Hypermedia, LAOS, Adaptive Strategies) of
these systems.
3. The students understood the connection between LAOS
and MOT.
4. The students used MOT purely for authoring adaptive
hypermedia, and perceived it as such.
5. The students used WHURLE solely for delivering adaptive
hypermedia, and perceived it as such.
6. Students consider automatic conversion between one-tomany or many-to-many adaptive hypermedia systems
useful.
3rd A3EH workshop
at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eEvaluation Results
eindhoven university of
technology
• SUS scores:
use frequently
– MOT
5
need to learn a lot to use
4,5
4
complex
3,5
3
2,5
2
confident to use
1,5
easy
1
0,5
75%
0
cumbersome
need support
learn quickly
well integrated
inconsistency
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eEvaluation Results
eindhoven university of
technology
• WHURLE
use frequently
need to learn a lot to use
confident to use
66.6%
5
4,5
4
3,5
3
2,5
2
1,5
1
0,5
0
cumbersome
complex
easy
need support
learn quickly
well integrated
inconsistency
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eEvaluation Results
eindhoven university of
technology
• MOT2WHURLE
need to learn a lot to use
confident to use
60.7%
use frequently
5
4,5
4
3,5
3
2,5
2
1,5
1
0,5
0
cumbersome
complex
easy
need support
learn quickly
well integrated
inconsistency
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eEvaluation Results
eindhoven university of
technology
• Hypothesis 1:
– Overall SUS scores:
• MOT:
75%
• WHURLE:
66.6% ±19.1%
• MOT2WHURLE: 60.7% ±19%
±15%
– NB: SUS scores are comparative
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eEvaluation Results
eindhoven university of
technology
• Hypotheses 2-5 extracted from
questionnaires.
• Hypothesis 2:
[The students understood the theoretical background (Adaptive
Hypermedia, LAOS, Adaptive Strategies) of these systems.]
– 70% ±24%
• Hypothesis 3:
[The students understood the theoretical background (Adaptive
Hypermedia, LAOS, Adaptive Strategies) of these systems.]
– 70% ±21.7%
3rd
• Indicates that students on average
understood the theoretical background after
an intensive introduction.
• However the wide SD indicates a great range
abilities.
A3EHof
workshop
at 12 International Conference on Artificial Intelligence, Amsterdam, 2005
th
TU/
eEvaluation Results
eindhoven university of
technology
• Hypothesis 4:
[The students used MOT purely for authoring adaptive
hypermedia, and perceived it as such.]
– 25 students out of 29 (86%) selecting MOT
to be an adaptive hypermedia authoring
system
– Indicates that students understand the
benefits of using MOT as an authoring
system.
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eEvaluation Results
eindhoven university of
technology
• Hypothesis 5:
[The students used WHURLE solely for delivering adaptive
hypermedia, and perceived it as such.]
– 21 students out of 29 (72%) selecting
WHURLE as an adaptive hypermedia
delivery system
– Indicates that students understand the
benefits of using WHURLE as a delivery
system.
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eEvaluation Results
eindhoven university of
technology
• Hypothesis 6:
[Students consider automatic conversion between one-to-many
or many-to-many adaptive hypermedia systems useful.]
– 4.57 (out of 5) approval rating.
– Indicates that students understand the
benefits of the paradigm shift that we are
proposing.
I.E. away from a ‘one-to-one’ paradigm.
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eConclusion
eindhoven university of
technology
• There has been one previous attempt to
convert content between systems: AHA!
& Interbook.
• This is the first (that we are aware of) to
empirically test the actual authoring and
conversion process between two
systems.
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eConclusion
eindhoven university of
technology
• The responses from this study have
validated our efforts to:
– Create a powerful & flexible authoring
environment
– Create an interoperable methodology
– Use a single authoring environment to
deliver content to other AEH delivery
systems
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eConclusion
eindhoven university of
technology
• Major disadvantages:
– The majority of which concerned the lack
of ‘polish’ for the MOT2WHURLE
conversion program.
– This does not invalidate our methodology
but does indicate that additional work
needs to be done.
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
eConclusion
eindhoven university of
technology
• It appears that a “write once, use often”
authoring methodology is desired by
authors.
• MOT is a simple to use, yet flexible
authoring system.
• Conceptual mapping between MOT and
other authoring systems demonstrates
the feasibility of such and approach
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005
TU/
e
eindhoven university of
technology
Questions … ?
3rd A3EH workshop at 12th International Conference on Artificial Intelligence, Amsterdam, 2005