Download AET Ba[...] - PocketKnowledge

Adaptive Educational Technologies and Educational Research:
Opportunities, Analyses, and Infrastructure Needs
Gary Natriello
Teachers College, Columbia University
February 2012
Background Paper Prepared for the National Academy of Education
1
Adaptive Educational Technologies and Educational Research:
Opportunities, Analyses, and Infrastructure Needs1
Introduction
Throughout the world the growing demand for educational opportunities is driven by the
needs of both advanced and developing economies to have greater proportions of citizens
achieve high levels of learning and to be able to continue to learn effectively throughout
their lives. The growth of post-industrial knowledge work (Bell, 1976; Kumar, 2004)
imposes the requirement of widespread access to higher learning on all societies that wish
to participate in the global economy.
The transformation in the conditions of work brought on by advances in computing and
communications technologies also brings with it the means to re-position educational
institutions and practices on a new base of technologies that are at once informationbased, digital, and networked (Computing Research Association; 2005; NSF Task Force
on Cyberlearning, 2008; Johnson, Smith, Willis, Levine, & Haywood, 2011). These
technologies are facilitating the development of new educational materials, tools, and
environments, and they are reshaping the roles of teachers and students. With such
sweeping changes in the possible conditions for educational work, it should not be
surprising that there are also new opportunities for the work of educational researchers.
In this paper I consider the impact of one type of new educational technology and the
associated practices on the opportunities for educational research. In particular, I explore
the new possibilities for research and analysis generated by the growing use of adaptive
educational technologies. I proceed in three stages. First, I consider the research
opportunities made possible by the data generated from adaptive educational
technologies. Second, I review research that has been conducted to date using data from
adaptive technologies. Finally, I explore how research drawing on data from adaptive
learning technologies might evolve and what might be needed to support and facilitate
various evolutionary paths.
Research Opportunities from Adaptive Educational Technologies
To appreciate the research opportunities connected with data generated by adaptive
educational technologies, it is important to have a general understanding of the bases of
such technologies and the data sets that they generate. I approach this task in seven
stages by (1) defining adaptive technologies, (2) reviewing the types of learner
I would like to thank Ryan Baker, Sasha Barab, Randy Bennet, Ken Koedinger, Kurt
VanLehn and the participants in the National Academy workshop for helpful
comments on an earlier version of this paper. The remaining limitations are my
own.
1
2
characteristics often recognized in adaptive systems, (3) specifying what I term the “sites
of adaptation,” (i.e., the features of systems that can be adapted), (4) considering a
comprehensive multi-layer model of adaptation in hypermedia systems, (5) reviewing a
framework for examining the structure of adaption within intelligent tutoring systems, (6)
considering how complex learning activities and environments may be developed with
adaptive features, and concluding with (7) a discussion of the features of the data sets
from adaptive systems that create unique opportunities for educational research.
Adaptive Technologies
Over the past several decades there has been a progression in the development of
computer-based systems that engage students with educational opportunities. The
introduction of such systems as research systems has moved on to include other unique
and proprietary systems. These systems, often initiated in laboratory studies, have moved
to field research sites in select classrooms, and then on to wide scale adoption.
Koedinger and Corbett (2006) describe this kind of progression for cognitive tutors.
There has also been a transition from stand-alone hypermedia systems to web-based
systems available via the Internet. Of course, it was this last development, the
widespread growth of the world wide web in an increasingly networked context, that
brought interactive systems to the population at large for a variety of activities.
Adaptive educational technologies are a special class of interactive systems that take
account of learner performance and adapt accordingly to maximize learning. Systems of
this type attempt to create personalized educational experiences optimized for each
individual student. Brusilovsky and colleagues (Brusilovsky, 1996; Brusilovsky, 1998;
Brusilovsky, 2001) provide important background for understanding the development of
adaptive hypermedia learning systems. Wenger (1987), VanLehn (2006), and Woolf
(2009) explain the foundations of adaptive tutoring systems. Lovette, Meyer, and Thille
(2008) examine the use of adaptive technologies in the context of a university course.
Pierce, Conlan, and Wade (2008) discuss the growing number of adaptive educational
games while observing that such games still constitute a small proportion of all
educational games.
Taking Account of Learner Characteristics
Adaptive systems rely on an explicit model2 of the individual user or learner that includes
information on the individual (e.g., goals, interests, knowledge, recent online behavior)
that is used to distinguish among different users (Wenger, 1987).
Brusilovsky (1996) identified five types of information on individuals that might be
employed in an adaptive system: goals, knowledge, background, hyperspace experience,
and preferences. Brusilovsky (2001) identified two additional types of user information;
Early groups involved in work on adaptive systems included scholars in the User
Modeling, Artificial Intelligence in Education, and Instructional Tutoring Systems
communities.
2
3
namely, interests and individual traits. He also added environmental dimensions (e.g.,
bandwidth) as another kind of information related to adaptation. Considering these eight
types of information on users or learners is helpful background for thinking about
educational research possibilities.
Goals refer to the objectives the user is trying to accomplish in the context of the learning
environment. Adaptive systems typically support a set of possible learner goals. Simple
systems may support only one or a small number of goals or tasks, while more
sophisticated systems may support a hierarchy of related tasks organized to support goal
accomplishment (e.g., Rebelledo-Mendez, du Boulay, and Luckin, 2006).
Learner knowledge refers to the knowledge that a learner possesses in relation to a model
of the structure of a bounded knowledge domain (Brusilovsky, 1996). The knowledge
domain is represented by a structure of concepts and their relationships, and the
knowledge state of the learner is represented in an overlay model (Carr and Goldstein,
1977) that seeks to assess the state of user knowledge in reference to the domain
structure.
Background refers to all of the available information related to the learner’s previous
experience outside the immediate adaptive system. Included under background may be
the user’s learning history.
Hyperspace experience refers to the learner’s familiarity with the structure of online
systems as reflected in the ease of navigation. Such familiarity is distinct from the
learner’s knowledge of the subject, and has bearing on the most appropriate navigation
techniques to present to the learner.3
Preferences refers to the fact that a learner can prefer some areas of a system over others
and some parts of an individual page over others. Brusilovsky (1996) points out that the
user has to inform the system about these preferences and that adaptive systems can then
generalize such preferences in new contexts within the system.
Interests refer to the long-term or enduring interests of the learner (as opposed to the
short-term task or goal). Interests are an increasingly important dimension of information
retrieval systems as these systems combine interests with short-term goals or tasks to
develop models to improve filtering and recommendations.
Traits, as conceptualized by Brusilovsky (2001), refer to user features such as personality
factors, cognitive factors, and learning styles that are stable or only change over long
periods of time. 4
More general experience and familiarity with technical systems (Turkle, 2007),
especially in education (Schofield, 1995), can also play a role.
4
There is controversy among psychologists over use of this term; some reserve the term
“trait” for personal qualities that cannot be changed and prefer to use propensity in
referencing features that Brusilovsky calls traits (Stanford Aptitude Seminar, 2002).
3
4
The immediate environment in which the learner is accessing the adaptive system offers
possibilities for adaptation along several dimensions. Environments can differ, for
example, in terms of the software platform, hardware capacities, and network bandwidth.
Adapting successfully to variables such as these can influence comprehension of the
material presented. The growth of mobile devices adds an additional environmental
dimension connected to the smaller screen and the capacity to identify geographical
location based on the global positioning system. Such location information has ad aptive
potential for delivering information to augment understanding in connection with
observations of the physical world (Azuma, 1997; Höllerer and Feiner, 2004).
The broad range of types of learner information included in adaptive systems can suggest
an equally broad range of types of research (e.g., Arroyo and Wolf, 2005; Baker,
Walonoski, et al., 2008; Beal, Qu, and Lee, 2008). Perhaps more important, the fact that
such information is often collected continuously over the course of the learning
experience means that the granularity and extensiveness of the resulting data sets is
unlike anything typically encountered by educational researchers in the course of most
current investigations.
Sites of Adaptation
With the range of types of learner information that have been specified in adaptive
systems identified, I turn now to what might be called the “sites” of adaptation, that is,
those dimensions of learning systems and platforms that might be adapted in an effort to
meet the needs of learners. Once again, Brusilovsky (1996) provides a useful starting
point in his discussion of adaptive hypermedia.
Brusilovsky (1996) specifies two broad classes of adaptive strategies: adaptive
presentation and adaptive navigation. Adaptive presentation includes three categories:
adaptive multimedia presentation, adaptive text presentation, and adaptation of modality.
Adaptive multimedia presentation is the tailoring of the presentation of content within
online media based on user characteristics. For example, a system might alter the density
and size of multimedia files presented to the learner based on the bandwidth of the
learner’s online connection.
Adaptive text presentation involves presenting learners with different learner models with
different text. Brusilovsky (2001) divides adaptive text presentation into canned text
adaptation and natural language adaptation. The former involves inserting or removing
text fragments, altering fragments, stretchtext,5 sorting text fragments, or dimming
Nevertheless, relatively stable qualities such as these present special challenges to
adaptive systems because they often require special psychological tests to extract.
5
Stretchtext is a technique for providing more detailed text treatment of particular
content to facilitate learner understanding. Stretchtext retains the original text on the
screen to provide a context for the text that is added to aid understanding (Landow,
2006).
5
fragments. The latter may make similar use of fragments by applying natural language
technologies.
Adaptive navigation support entails techniques that assist learners in finding their paths
through content by adapting the way that links are presented in response to different
learner characteristics. Brusilovsky (2001) identifies six techniques of adaptive
navigation: direct guidance, adaptive link sorting, adaptive link hiding (including hiding,
disabling, and removal), adaptive link annotation, adaptive link generation, and map
adaptation.
Direct guidance is showing the learner the best link based on learner characteristics as
represented in the user model. Adaptive sorting positions the best link (according to the
user model) at the top of the list of links. Adaptive hiding restricts the navigation options
available to the learner by hiding links to pages that are not appropriate according to the
user model. Adaptive link annotation provides additional information to the user about
the page behind the link either through additional text or visual cues. Map adaptation
involves adapting the form of hypermedia maps based on the characteristics of the learner
as represented in the user model.6
A Comprehensive View of Adaptive Technologies in Hypermedia Systems
To move toward a comprehensive view of adaptive technologies the literature on
adaptive hypermedia once again provides some accessible treatments. In a review of the
recent progress in the development of adaptive hypermedia systems, Knutov, De Bra, and
Pechenizkiy (2009) provide a useful multi-layer model that summarizes such systems.
They organize their discussion around six major questions of adaptation:
Why do we need adaptation (Why?)
What can we adapt? (What?)
What can we adapt to? (To What?)
Where can we apply adaptation? (Where?)
When can we apply adaptation? (When?)
How do we adapt? (How?) (pp. 7-8)
Knutov, De Bra, and Pechenizkiy (2009) note that reference models used to describe
6
Brusilovsky and Maybury (2002) and Brusilovsky (2004) discuss the challenge of
adaptive navigation support in situations other than assisting users in moving through a
closed body of hypertext. Two situations that press the limitations of adaptive systems
are the open body of resources represented by the web and the complex graphical
dimensions in virtual reality environments. Providing adaptive support in these situations
pose new challenges and opportunities. Brusilovsky (2004) discusses the use of both
content-based and social technologies to address the open body of resources on the web.
Brusilovsky (2004) notes that adaptive navigation support technologies developed for
hypertext have analogs in similar approaches in virtual environments.
6
adaptive hypermedia architectures employ a tower model that represents different
elements as different levels or layers. Along these lines, they discuss five different layers
as they relate to the key questions.
The Goal Model (GM) layer (addressing the “Why?” question) potentially involves not
just goals, but also a hierarchical structure that includes “goals, objectives, tasks,
requirements, workflows” (Knutov, De Bra, and Pechenizkiy, 2009, p. 23). A
generalized view of the goal centered approach might include a “hierarchy of goals and
corresponding tasks comprising this goal, and workflows that need to be followed to
complete a requirement (Knutov, De Bra, and Pechenizkiy, 2009, pp. 23-24).
The Domain Model (DM) layer (addressing the “What?” question) consists of concepts
and the relationships among concepts, most often organized in a hierarchy. When the
Domain Model is structured so that concepts are fine grained and structured
hierarchically it is possible to apply adaptive techniques.
The User Model (UM) layer (addressing the “To What?” question) is made up of entities
that have values stored in a structure (typically a table) that overlays the Domain Model,
in effect creating a map of some attribute (such as user knowledge) over the domain.
Systems can use both domain dependent properties such as user knowledge and domain
independent properties such as user characteristics.
The Context Model (CM) layer (addressing the “Where?” and “When?” questions) refer
to system features that take account of the context in which the adaptation is applied and
the environment in which the application is used. Adaptation processes and decisions
dependent on an application address the “Where?” question. Adaptation processes and
decisions independent of an application (e.g., time, day of the week, network bandwidth)
address the “When?” question.
The Adaptation Model (AM) layer (addressing the “How?” question) deals with
techniques and methods of providing adaptation within a system. Knutov, De Bra, and
Pechenizkiy (2009) identify techniques of content adaptation (inserting/removing
fragments, altering fragments), techniques of adaptive presentation (dimming fragments,
sorting fragments, stretchtext, zoom/scale, layout, link sorting/ordering, link annotation,
combinatorial techniques), and techniques of adaptive navigation (link generation,
guidance, link hiding).
A View of Adaptive Technologies in Tutoring Systems
VanLehn (2006) proposes a framework for examining the structure of adaptation within
intelligent tutoring systems. His framework includes six elements and two processes.
The elements are: (1) task domain, or the competence that the tutoring system is trying to
impart to the student; (2) task, or an activity lasting from several minutes to over an hour;
(3) step, or a user interface interaction taken by the student in order to complete a task;
(4) knowledge component, or any principle, concept, rule, fact, association, or other
fragment of task-specific information into which knowledge can be decomposed; (5)
7
learning event, or the mental construction or application of a knowledge component; and
(6) incorrect, or the designation applied to a step, knowledge component, or learning
event indicating that the tutoring system should say something about it to meet current
instructional objectives.
These elements are organized by two types of adaptive processes: outer loop processes
which determine which tasks the student should do next, and inner loop processes which
determine the steps for students within tasks. VanLehn (2006) identifies four types of
outer loop processes: (1) the student selects the task from a menu, (2) the tutor assigns
tasks in a predetermined sequence, (3) the tutor assigns tasks from a pool of tasks until
mastery is achieved, and (4) macroadaptive learning or selecting tasks based on a match
between the task’s traits and the student’s traits. VanLehn (2006) notes that inner loop
processes can be thought of as services that support student learning and identifies five
common services in intelligent tutoring systems: (1) minimal feedback, typically whether
a step is correct or incorrect; (2) error-specific feedback to help the student understand
why something is wrong and how to avoid the mistake in the future; (3) hints on the next
step to improve student performance moving forward, (4) assessment of knowledge for
instructors, students, or the tutoring system itself; and (5) review of the entire solution
offered by the student.
Multiple Conceptions of Learning Activities
The ways in which learning activities are conceived also carry implications for thinking
about adaptive learning technologies, the kinds of experiences they support, and the kinds
of data they might generate. Chi (2009) distinguishes four types of overt or visible
activities of learners while engaging with a resource such as a text, a tutoring system or a
virtual environment. Passive activities are those in which the learner is not doing
anything while being exposed to the resource. Active activities involve the learner doing
something, usually physically, involving the resource. Constructive activities are those in
which the learner is producing outputs that contain ideas beyond those presented in the
resource. Interactive activities are those in which the learner is dialoguing in a
substantive way on the same topic with another individual or system and not ignoring the
other’s contribution. As we move from passive to interactive learning activities, the
complexity of the activities and the corresponding data gathering opportunities can
become more elaborate and more complex.
Barab, Gresalfi, and Igram-Goble (2010) conceptualize even more elaborate and complex
learning arrangements in their discussion of the possibilities for transformational play in
games and virtual worlds. They draw on four types of content engagement identified in
Grasalfi, Barab, Siyahhan and Christensen (2009) as a way to think about adaptive
educational technologies that can support transactive engagement with adaptive worlds.
They contrast the four types of content engagement as: (1) procedural or learning about
what to do, (2) conceptual or understanding how their tools work in these worlds, (3)
consequential or learning about the impact of their actions on designed contexts, and (4)
critical or reflecting on the impact of their actions on designed contexts. The adaptive
worlds that can facilitate transformational play are responsive to the decisions learners
8
make within them, and the systems generate data on both the adaptations of the worlds
and the transformations of the learners.
Whether or not one prefers the layered tower model of adaptive hypermedia of Knutov,
De Bra, and Pechenizkiy (2009), the framework for intelligent tutoring systems offered
by VanLehn (2006), and/or the multiple views of learning provided by Barab, Grasalfi
and colleagues, they each contribute to thinking about the data generating potential of
adaptive systems. These data generating and gathering opportunities inherent in adaptive
systems provide a good foundation for considering the kinds of research opportunities
that such systems may afford educational researchers.
Unique Features of Data Generated by Adaptive Educational Technologies
Adaptive Educational Technologies generate data sets that offer both new opportunities
and new challenges for educational researchers. The data sets are larger and often more
complex than those typically encountered in educational research. Moreover, they tend
to be unique along the following eight dimensions.
First, the breadth of data generated by adaptive learning technologies can be considerably
greater than the norm for educational research. The adaptive technologies and platforms
can engage very large numbers of learners simultaneously and gather data about that
engagement in ways not otherwise available.
Second, the depth of the data captured by adaptive technologies is more extensive than
what can be gathered in most studies in education. Adaptive technology platforms
provide data on both educational program elements (i.e., materials and interactions) and
student responses. Moreover, they also provide data on the context in which those student
responses were generated.
Third, the granularity of the data generated by adaptive technologies is considerably finer
than that gathered in most educational research. Each element of a program can be
tracked to its delivery point, and every student movement, i.e., keystroke, in response can
be included in the dataset. In fact, somewhat less granular semantic actions are both
easier to handle (Stephens and Sukumar, 2006) and potentially more useful for research
purposes (Mislevy, et al., 2010.)
Fourth, the linkage between student behaviors and the data elements gathered is more
consistently tighter than in other data available to educational researchers. Adaptive
technologies evoke students’ behaviors on an on-going basis to maintain the interaction
with the delivery of the educational program.
Fifth, the data generated by adaptive technologies and platforms is time specific, and the
data sets are inherently longitudinal over small intervals and for potentially long periods
of time. For example, the Pittsburgh Science of Learning Center DataShop has data sets
that cover entire years from middle and high school students who used educational
9
software three times per week (Koedinger, Baker, Cunningham, Skogsholm, Leber, and
Stamper, 2011). Such data collection is typically cost-prohibitive in most educational
research.
Sixth, the data generated by adaptive systems can be multi-dimensional across both
elements of the educational program and elements of student performance, behavior, and
background as suggested by the earlier discussion of the multiple layers of data inherent
in adaptive technologies. This means that multiple investigators or teams can examine
different streams of data in parallel.
Seventh, data from adaptive educational technologies exhibits what I call the “fruit fly
effect” after the preference that many biologists have for using fruit flies in their research
to take advantage of the short life-cycle and rapid progress through the generations. Data
from adaptive educational systems provide shorter, faster iterative cycles of data as the
educational program elements of the system can be adapted quickly in reaction to student
responses.
Eighth, data gathering in adaptive systems is integrated with program delivery in a way
seldom encountered in educational research activities that are generally grafted on (often
over considerable resistance) to the regular business of educational programs.
These features of the data sets generated by adaptive educational technologies offer new
possibilities for educational researchers to examine problems and issues previously
examined in other data, and they open up new kinds of research questions for
consideration. However, such data sets can also seem overwhelming. Fortunately, there
are strategies that can be used to examine such data, and I turn to these next.
Analyses Using Data from Adaptive Educational Technologies
I consider the use of data generated by adaptive educational technologies from two
perspectives. First, I examine the analytic approaches that have been used most
frequently to mine the data produced from these systems, drawing on several typologies
in the field. Second, I review the application areas where data from adaptive systems
have been employed to provide insight and drive action to improve the educational
experience.
Approaches to Utilizing Data from Adaptive Educational Technologies
The segment of the research community that has devoted the most attention to examining
data generated by adaptive educational technologies is known as educational data mining
(Baker and Yacef, 2009). This community is a part of the larger group of scholars
focused on mining data from very large databases. As Baker and Yacef (2009) note,
educational data mining focuses on the unique types of data available from educational
10
platforms and settings, often characterized by multiple hierarchical levels and often
drawing on psychometrics and the learning sciences.7
Romero and Ventura (2007) provide a careful review of work on educational data mining
from 1995 to 2005, a period during which the field emerged and coalesced around a set
of techniques and approaches. They position educational data mining as a step in the
overall process of knowledge discovery in databases (often referred to as KDD),8 and
they consider it a formative evaluation technique. They distinguish three target audiences
for this kind of inquiry: students, educators, and academics and administrators. When
the audience is students (e.g., Tang and McCalla, 2005) the objective is often to
recommend activities to improve their learning. When the audience is educators (e.g.,
Merceron and Yacef, 2004) the objective is to provide feedback on instruction and the
educational offering. When the audience is academics and administrators (e.g., Urbancic,
Skrjanc, and Flach, 2002) the objective is to guide site improvements and the distribution
of resources.
Romero and Ventura (2007) note that it is possible to apply data mining techniques to
data generated by traditional classroom based education, for example, when such
techniques are applied to university data bases. Of greater relevance, they also note that
data mining techniques are ideally suited to the large data sets generated by distance
education activities, including web-based courses, learning management systems, and our
current concern, adaptive systems, or what they and others refer to as adaptive and
intelligent web-based educational systems (or AIWBES).9 Adaptive systems provide
richer and most robust data sets and thus contain both more opportunities and more
challenges for analysis than non-adaptive systems
The review of data preprocessing tasks presented by Romero and Ventura (2007)
provides further insight into the data mining endeavor. They identify eight steps as part
of preprocessing: 1) data cleaning or the removal of irrelevant items and log entries
7
Data mining as an approach to inquiry has generated various sorts of objections in its
relatively short period of ascendance. These include objections regarding privacy
concerns, violation of constitutional rights when used by government agencies, accuracy
of the data, the effectiveness of the procedures (Brasch, 2005), and the atheoretical nature
of much data mining work.
8
Maimon and Rokach (2005) specify the knowledge discovery in databases process as
nine steps: 1) developing an understanding of the application domain, 2) selecting and
creating a data set on which discovery will be performed, 3) preprocessing and cleansing,
4) data transformation, 5) choosing the appropriate data mining task, 6) choosing the data
mining algorithm, 7) employing the data mining algorithm, 8) evaluation, and 9) using
the discovered knowledge.
9
Such systems are the result of the joint evolution of intelligent tutoring systems and
adaptive hypermedia systems. Among the adaptive systems noted are those described in
Brusilovsky and Peylo (2003): SQL-Tutor, German Tutor, ActiveMath, VC-Prolog-Tutor
(all intelligent tutoring systems) and AHA!, InterBook, KBS-Hyperbook, and
WebCOBALT (all adaptive hypermedia systems).
11
including graphics and scripts; 2) user identification or the process of associating pages
accessed to the particular user; 3) session identification or grouping of page references
for a user into user sessions; 4) path completion or inserting page references missing due
to browser or proxy server caching; 5) transaction identification or the disaggregation of
sessions into smaller transactions; 6) data transformation and enrichment or creating new
variables, values, or meanings for log references; 7) data integration or synchronizing
data from diverse streams or sources; and 8) data reduction or reducing dimensionality.
Some of these activities are familiar ones connected to the preparation of a wide range of
data sets in education; but others are quite specific to the task of using system log data
and databases.
Romero and Ventura divide data mining activities into two broad classes: 1) statistics
and visualization, and 2) web mining. Statistics refers to a range of standard descriptive
techniques for log files or databases, while visualization entails rendering complex
student data in visual displays. Web mining refers to extracting information from web
content, structure, and usage patterns. They note that such mining techniques can be used
in an offline fashion, where the identified patterns are used by educators to make
improvements in systems and practices, as well as in an online fashion, where the
patterns discovered are fed directly into an intelligent software system on-the-fly.10
In their review of studies using data mining techniques in education Romero and Ventura
(2007) identify three major kinds of web mining techniques: 1) clustering, classification,
and outlier detection; 2) association rule mining, and sequential pattern mining; and 3)
text mining. Each of these methods has been employed in studies using data from
adaptive educational technologies.
Clustering and classification group things together based on similarities with clustering
being unsupervised and classification being supervised. Outlier detection involves
identifying observations that are unusually small or large relative to others in the dataset.
Romano and Ventura feature examples of each type of mining technique applied to
adaptive systems. Tang and McCalla (2005) used clustering and collaborative filtering in
a paper recommender system designed to personalize and adapt course content. Baker,
Corbett, and Koedinger (2004) used classification to determine if students are gaming a
system and avoiding learning. Muehlenbrock (2005) detected regularities and deviations
to provide information for learners and educators to manage their learning and teaching.
Association rule mining attempts to associate one or more attributes of a dataset with
another attribute by means of an if-then statement of attribute values. Sequential pattern
mining attempts to identify inter-session patterns in a time-ordered set of sessions or
episodes. Freyberger, Hefferman, and Ruiz (2004) used association rules in a search for
a transfer model of learning in intelligent tutoring systems. Morales, Preez, Soto,
Martinez, and Gomez (2006) developed a tool for discovering sequential patterns from
student usage data in an adaptive hypermedia course to recommend links to new students.
10
This is an important distinction that has implications for the shape of future careers in
educational research as I discuss later in this paper.
12
Text mining extends data mining to text data and often involves machine learning and
natural language processing. Tang, Yin, Li, Lau, Li, and Kilis (2000) used a key-word
driven text mining algorithm to select articles for students.
Baker (2009, 2010) presents another framework for considering data mining approaches
in education that is somewhat more expansive. He distinguishes five major methods: 1)
prediction; 2) clustering; 3) relationship mining; 4) discovery with models; and 5)
distillation of data for human judgment.
In prediction the goal is to develop a model that can infer a predicted variable from some
combination of predictor variables. Baker identifies three types of prediction: 1)
classification where the predicted variable is binary or categorical and popular methods
include decision trees, logistic regression, and support vector machines; 2) regression
where the predicted variable is continuous and popular methods include linear regression,
neural networks, and support vector machine regression; and 3) density estimation where
the predicted variable is a probability density function and where density estimators can
be based on various kernel functions.
In clustering the goal is to find data points that naturally group together to split the full
data set into a set of clusters.
In relationship mining the goal is to discover relationships between variables. Baker
(2010) notes four types of relationship mining: 1) association rule mining where the goal
is to find if-then rules; 2) correlation mining where the goal is to find linear correlations
between variables; 3) sequential pattern mining where the goal is to find temporal
relationships between events; and 4) causal data mining where the goal is to infer if one
event was the cause of another.
These first three types of data mining – prediction, clustering, and relationship mining –
map well onto the web mining catgegories designated by Romero and Ventura (2007).
The fourth type moves beyond those discussed in the earlier review. In discovery with
models, a model is developed (perhaps through prediction, clustering, or human
reasoning) and used to guide another analysis. Baker (2010) cites Baker, Corbett, Roll,
and Koedinger (2008), a study of student gaming of the system, for the use of
assessments of the probability that a student knows the current knowledge component
that depend on models of the knowledge components in a domain.
The fifth type of data mining identified by Baker (2010), which he labels as distillation of
data for human judgment, calls on humans to make inferences about data, often with the
benefit of data visualizations, that are beyond the scope of automated data mining
methods. Here Baker (2010) points to the example of work by Corbett and Anderson
(1995) for its use of visualizations of learning curves to distinguish between smooth and
13
spiked patterns.11 Yet another example may be found in the work of Martinez, Kay, and
Yacef (2011) who use visualizations of learner participation in tabletop collaborations to
assist facilitators in detecting problems in group interaction.
Applications of Data Drawn from Adaptive Educational Technologies
In their 2009 review of educational data mining, Baker and Yacef identify four areas
where educational data mining methods have been employed productively. First,
educational data mining has been used to improve student models to allow adaptive
technologies to respond more appropriately to student differences. Such models have
been used to do things such as identifying which students are experiencing low selfefficacy in online systems (Mcquiggan, Mott, and Lester, 2008) and identifying the
factors that predict student failure in college (Superby, Vandamme, and Meskens, 2006).
Pechenizkiy, Trcka, Vasilyeva, van der Aalst, and De Bra (2009) used a process mining
approach to examine the data generated during exams offered through learning
management systems.
Second, educational data mining methods have been used to improve models of the
knowledge structure of content domains automatically. For example, Barnes, Bitzer, and
Vouk (2005) used a matrix based model to extract latent relationships from observed
binary variables, and Pavlik, Cen, and Koedinger (2009) used analyses of learning curves
to generate domain models
Third, educational data mining methods have been used in studies designed to determine
which types of pedagogical support are most effective in producing student learning
gains. For example, Beck and Mostow (2008) investigated the effects of different types
of practice for different types of students. Chi, VanLehn, Litman, and Jordan (2010)
examined pedagogical strategies that lead to effective tutoring experiences for students.
The fourth application area identified by Baker (2009) is in generating evidence to refine
and extend educational theories. For example, Madhyastha and Tanimoto (2009) drew
on the work of Abelson (1968) on cognitive consistency theory to examine the
relationship between consistency and student performance with a goal of developing
guidelines for scaffolding instruction.
Another perspective on analyses drawing on data from adaptive educational technologies
is provided by Castro, Vellido, Nebot, and Mugica (2007) in their review of the
application of data mining techniques to e-learning. They distinguish studies bearing on
11
Yet another variation on the enumeration of data mining methods may be extracted
from the organization of the basic methods chapters in the Handbook of Educational
Data Mining (Romero, Ventura, Pechenizky, and Baker, 2011). In that volume, chapters
are devoted to each of eight types of methods: visualization, basic statistical analysis,
classification, clustering, association rule mining, sequential pattern analysis, process
mining, and modeling hierarchy and dependence.
14
five areas in e-learning. Applications related to the assessment of student performance
have been the subject of a good deal of work (e.g., Hwang, 2003; Mullier, 2003;
Kotsiantis, Pierrakeas, and Pintelas, 2004; Sheard, Ceddia, and Hurst, 2003). Additional
work on models for assessing student knowledge has been done by Corbett and Anderson
(1995), Martin and VanLehn (1995), and Pavik, Cen, and Koedinger (2009), and a
comparison of models singly and in combination has been conducted by Baker, Pardos,
Gowda, Nooraie, and Heffernan (2011).
Research has also focused on applications that provide course adaptation and learning
recommendations as a result of student learning behavior (e.g., Mizue, and Toshio, 2001;
Romero, Ventura, and De Bra, 2004; Shang, Shi, and Chen, 2001). A third area for
research has been on the evaluation of learning material and web-based courses (e.g.,
Drigas, and Vrettaros, 2004; Hwang, Huang, and Tseng, 2004). Studies have also focused
on applications that provide feedback to teachers and students in e-learning courses as a
result of student learning behavior (e.g., Arroyo, Murray, Woolf, and Beal, 2004; Hwang,
1999). A fifth area of research has involved outlier detection in e-learning environments
(e.g. Ueno, 2003; Vellido, Castro, and Nebot, 2006).
Castro, Vellido, Nebot, and Mugica (2007) also highlight several large research projects
that have incorporated data mining methods into e-learning environments. The
ALFANET project (http://alfanet.ia.uned.es/alfanet) includes an e-learning platform with
a component that provides online real-time recommendations and advice to learners
based on previous users’ interactions and results of questionnaires. The AHA! Project
(http://aha.win.tue.nl) adapts the presentation and navigation system of a course based on
the level of knowledge of an individual learner. The LearningOnline Network with a
Computer Assisted Personalized Approach (LON-CAPA) includes an individualized
homework and automatic grading system (www.lon-capa.org). The LExICON project
(http://lexikon.dfki.de/) involves course adaptation based on the students’ navigational
behavior. The ATutor project (www.atutor.ca/) has created a learning content
management system that includes an assessment component with student behavior
tracking.
Romero and Ventura (2010) in the most up-to-date review of the literature available
discuss data mining efforts in terms of eleven educational tasks12:
1) analysis and visualization of data, with a goal of highlighting useful information to
support decision making, (e.g., Bellaachia and Vommina, 2006; Ben-naim, Marcus, and
Bain, 2008; Mostow, Beck, Cen, Cuneo, Gouvea, and Heiner, 2005; Zinn and Scheuer,
2006);
12
Romero and Ventura (2010) also organize the literature in terms of eight categories of
types of data or environment: traditional education, web-based education or e-learning,
learning management systems, intelligent tutoring systems, adaptive educational systems,
tests and questionnaires, texts and contents, and other. Here I include as examples only
those studies involving intelligent tutoring systems and those on adaptive educational
systems.
15
2) providing feedback for instructors, with a goal of supporting course instructors in
decision making on how to improve students’ learning opportunities, (e.g., Beal and
Cohen, 2008; Garcia, Romero, Ventura, and Castro, 2009; Hurley and Weibelzahl, 2007;
Romero, Ventura, and De Bra, 2004; Tsai, Tseng, and Lin, 2001; Vialardi, Bravo, and
Ortigosa, 2008);
3) recommendations for students, with a goal of personalizing their learning activities and
experiences (e.g., Ba-Omar, Petrounias, and Anwar, 2007; Heraud, France, and Mille,
2004; Karampiperis and Sampson, 2005; Kelly and Tangney, 2005; Kristofic, 2005; Lu,
2004; Lu, Li, Liu, Yang, Tan, and He, 2007; Pavlik, Cen, and Koedinger, 2009; Romero,
Ventura, Zafra, and De Bra, 2009; Stamper and Barnes, 2009; Wang, Weng, Su, and
Tseng, 2004; Wang, Tseng, and Liao, 2009);
4) predicting student’s performance, with a goal of predicting performance in the form of
knowledge or some type of score (e.g., Beck and Woolf, 2000; Cetintas, Si, Xin, and
Hord, 2009; Desmarais, Gagnon, and Meshkinfram, 2006; Hamalainen and Vinni, 2006;
Pardos, Heffernan, Anderson, and Heffernan, 2007; Want and Mitrovic, 2002);
5) student modeling, with a goal of developing cognitive models of human users/students
(e.g., Amershi and Conati, 2009; Antunes, 2008; Baker, Corbett, and Aleven, 2008;
Barmes and Stamper, 2008; Chang, Beck, Mostow, and Corbett; Feng and Beck, 2009;
Gong, Rai, Beck, and Heffernan, 2009; Hwang, Chang, and Chen, 2004; Jonsson,
Hasmik, Johns, Mehranian, Arroyo, Woolf, Barto, Fisher, and Mahadevan, 2005;
Mclaren, Koedinger, Schneider, Harrer, and Lollen, 2004; Rai, Gong, and Beck, 2009;
Ritter, Harris, Nixon, Dickison, Murray and Towle, 2009; Robinet, Bisson, Gordon, and
Lemaire, 2007; Rus, Lintean, and Azevedo, 2009);
6) detecting undesirable student behaviors, such as performance errors, disengagement,
and acting out in some way (e.g., Baker, 2007; Jong, Chan, and Wu, 2007;
Muehlenbrock, 2005; Vee, Meyer, and Mannock, 2006; Yudelson, Medvedeva,
Legowski, Castine, Jukic, and Rebecca, 2006);
7) grouping students, with a goal of grouping students according to their characteristics
(e.g., Crespo, Pardo, Perez, and Kloos, 2005; Hamalainen, Suhonen, Sutinen, and
Toivonen, 2006; Kelley and Tangney, 2005; Zakrzewska, 2006).;
8) social network analysis, with a goal of understanding the relationships among
individuals (e.g., Tang and McCalla, 2005);
9) developing concept maps, with a goal of allowing instructors automatically to create
graphs showing the relationships among concepts and the hierarchical nature of
knowledge in a domain (e.g., Simko and Bielikova, 2009);
16
10) constructing courseware, with the goal of assisting instructors and developers
construct courseware and learning content automatically (e.g., Tang, Lau, Yin, Lin, and
Kilis, 2000); and
11) planning and scheduling, with the goal of helping with future course planning and
student course scheduling (e.g., Wang, Cheng, Chang, and Jen, 2008).
Although much of the work being done to analyze the data from adaptive technologies is
being done by scholars working in traditions not closely aligned with educational
research, there is a significant amount of work being done by scholars in the learning
sciences tradition.13 A number of research and development efforts to create adaptive
learning opportunities have been driven by a learning process research perspective; that
is, they have been influenced by the base of knowledge on patterns of human learning
and strategies to apply technologies to leverage those patterns. Inherent in these
approaches is a model of the learner maintained by the adaptive system and the capacity
of the system to adjust itself in response to the range of processes employed by individual
learners (Shute and Zapata-Rivera, 2007).
Progress in the development of models of learning and learners will lead to the
identification of both new learner characteristics and new modeling techniques. For
example, in response to questions posed by Shute and Zapata-Rivera, (2007, p. 287), a
panel of experts in the field identified the following learner variables as promising for
consideration:
Cognitive abilities (e.g., math skills, reading skills, cognitive
developmental level, problem solving, analogical reasoning)
Metacognitive skills (e.g., self-explanation, self-assessment, reflection,
planning)
Motivational and Affective states (e.g., engagement, flow, anxiety level)
Additonal variables (e.g., personality types, learner styles, social skills
such as collaboration, and perceptual skills)
Lee and Park (2007) call attention to systems that include motivational variables, citing
de Vicente and Pain (2002), whose motivation model includes control, challenge,
independence, fantasy, confidence, sensory interest, cognitive interest, effort, and
satisfaction as variables to indicate a learner’s motivational state. Lee and Park (2007)
also highlight approaches and systems that consider metacognitive skills such as
SCIWISE (White, Shimoda, and Frederiksen, 1999), which encourages students to
express their metacognitive ideas and Help-Seeking Tutor Agent (Aleven, Stahl,
Schworm, Fischer, and Wallace, 2003; Aleven, McLaren, Roll and Koedinger (2004),
which supports learners to become better help seekers. The Help-Seeking Tutor Agent is
also covered in more recent reports, including Aleven, McLaren, Roll, & Koedinger
(2006) and Roll, Aleven, McClaren, & Koedinger (2011).
This section draws on the discussion of the role of learning research in adaptive
systems presented in Natriello (2010).
13
17
In their discussion of technology-rich environments, Lajoie and Azevedo (2006) stress
the utility of research and development driven from well-developed learning theories.
They highlight approaches that involve both cognitive and metacognitive tools to support
learners. In particular, they argue that in the wake of the growing development of open
electronic learning environments accompanying the spread of the Internet, students are
increasingly being called upon to regulate their own learning more effectively. This leads
them to call attention to the substantial body of work on self-regulated learning (e.g.,
Boekaerts, Pintrich, and Zeidner, 2000; Butler and Winne, 1995; Paris and Paris, 2001;
Schunk and Zimmerman, 2001; Zimmerman and Schunk, 2001).
Research on the impact of hypermedia environments on student self-regulated learning
provides a good example of work that expands the view of student learning while
simultaneously attempting to create more powerful learning opportunities. The work
attends not only to learning gains, but also to the learning process. Self-regulated student
learning refers to a host of processes (e.g., planning, monitoring, strategy use, handling of
task difficulty and demands) (see, e.g., Greene and Azevedo, 2009) that learners can
employ to become more effective. However, as Lajoie and Azevedo (2006) note, despite
the range of work being done on technology-rich environments from a self-regulation
perspective there has been little research that would allow us to understand the entire
cycle of activities involved in self-regulated learning in technology-rich environments.
Pursuing such work as part of a broader class of efforts to enhance adaptive learning
opportunities could be very fruitful.
The traditions of analyzing data from adaptive educational technologies formed over the
past twenty years, first as these technologies appeared in stand alone systems, and more
recently as they have become almost entirely internet and web based.14 The scholars
engaging in this work are drawn from diverse fields, including computer science,
information science, cognitive science, and others.15 Although mainstream educational
researchers have not engaged in these analyses in large numbers, the opportunities for
such participation are rising, and the skills and perspectives that such scholars bring can
enhance this work moving forward. However, there are barriers to consider that may
slow or impede such participation by educational researchers, and I turn to these and
ways to address them in the next section.
Enhancing Conditions For Research on Adaptive Educational Technologies
14
Additional examples of analyses of adaptive systems are presented in the case studies
contained in Romero, Ventura, Pechenizky, and Baker, (2011).
15 Work involving datasets generated by online systems is also organized under the
term “learning analytics,” and the First International Conference on Learning
Analytics and Knowledge was held in 2011. Papers presented at this initial
conference evidenced a holistic view of the use of data in educational enterprises.
18
I consider approaches to strengthening the participation of educational researchers in
work that makes use of data from adaptive educational technologies along three lines.
First, I take stock of the likely development of such technologies in the near future.
Second, I highlight a set of issues that are emerging around the use of such data. Third, I
present some strategies for moving forward and engaging educational researchers.
The Growth of Adaptive Educational Technologies
There are several developments that suggest that adaptive educational technologies will
grow in number and pervasiveness as a part of the educational sector in the coming years.
There is a growing appreciation from diverse quarters that the cyberinfrastructure for
education will be increasingly important in the development of the education sector
(Computing Research Association; 2005; NSF Task Force on Cyberlearning, 2008;
Johnson, Smith, Willis, Levine, & Haywood, 2011). Along with the growing demand for
all students to learn that can only be addressed through the delivery of more personalized
educational services, investments in broadband services (Federal Communications
Commission, 2010) will provide the necessary foundation for the delivery of rich
interactive educational applications. Indeed, some commentators have predicted that the
transition to an educational system dominated by computer-based, student-centered,
personalized learning will occur within the next decade (Christensen, Horn, and Johnson,
2008).16
Adaptive educational technologies are likely to emanate from research laboratories,
commercial entities such as publishers, and from educators themselves, as well as from
collaborations across multiple parties. The sources and nature of such technologies are
likely to become more diverse and decentralized. In the absence of an infrastructure of
standards and common development platforms, this diversity will extend to the data
architectures underlying the adaptive applications. As a result the challenge of drawing
on data from such systems will be greater than it is at present where such systems exist in
modest numbers. This will occur even as the need for educational researchers to become
skilled in the analysis of such data becomes greater. A host of complicating issues is
likely to emerge, and I turn to them next.
Emerging Issues
Educational researchers are accustomed to confronting a variety of issues as they seek to
gather and examine data on educational processes and systems. Everything from data
access, to confidentiality, to adequacy for addressing particular research questions and
16
Christensen, Horn, and Johnson (2008) cite four conditions that will hasten the
transition to computer-based adaptive technologies: improvements in the quality of
systems, the ability of students and teachers to select personalized learning paths, a
looming teacher shortage as a result of the retirements of the members of the baby-boom
generation, and significant cost declines as the market for adaptive learning grows.
19
more has been handled over the years. In most cases standard procedures and practices
have evolved through professional consensus. Although many of the issues surrounding
the use of data generated by adaptive educational technologies can be addressed through
current procedures, there are some special issues that require attention, and I consider
them here.
Ownership of Data
Although there may always be questions of who owns the data gathered through
educational research, the data generated by adaptive educational technologies pose this
issue in certain unique ways.
An initial challenge is to be able to communicate which data are being gathered and
through what means since the systems log information on individual activity rather
unobtrusively. Adaptive systems could result in the combination of data from different
schools and districts, and populations often in ways that are not transparent either to
learners or to those educators engaging learners with the systems. Thus, there are special
burdens attendant to the question of informed consent of those whose data is collected. 17
The extensiveness of the data gathering possible through adaptive educational
technologies may also pose special problems. As the data streams from such systems
become more diverse and encompassing, there may be new kinds of objections raised
regarding the degree of intrusiveness they entail.
Unlike data from local activities that tend to remain local, and unlike data from standard
assessments that tend to be narrow in focus, data from adaptive systems could travel far
from the point of data collection, moving from individuals to systems to commercial
entities, and they could be used to construct more revealing portraits of individual
learners, raising new concerns about confidentiality. Such ostensibly educational data
could have particular value to future employers and commercial entities more generally,
particularly if blended with personal data gathered in other sectors. Moreover, the risks
to individual privacy are unlikely to be evenly distributed throughout the population
(Raab and Bennett, 1998)
Accessibility of Data
The data generated by adaptive educational technologies, as noted earlier, require special
processing to make them useful for educational research. Such processing is a distinctly
different task from those required to operate the technologies in educational practice, and
as such, it requires special efforts and investments of time and resources. Yet it is not
clear how such resources will be assembled and put to the task. This is a special case of
the more general data archiving and sharing challenge that has so far eluded the
educational research community in the case of data sets generated directly by educational
researchers.
17
Note that online systems of all kinds confront this same issue (Kobsa, 2007).
20
There are several aspects of the likely provenance of adaptive educational technologies
that pose this problem in different ways. For those systems developed by scholars, the
challenge of data archiving and sharing is the same as it has always been. Both limited
resources and the absence of professional norms make such data sharing less common
than might be desired for scientific purposes. For those systems developed by large
software publishing companies, the incentives for data management, archiving, and
sharing beyond the immediate needs of system and service delivery are absent. Indeed,
without clear and short-term gains, commercial sector cost control sensitivities would
seem to work against such investments. For those systems created by small developers,
the resource constraints can be even more severe and the prospect of gains from data
sharing even more remote.
Making all of this more complicated from the systems developer perspective is the
absence of any organizational capacity within the community of educational researchers
to serve as a reasonable point of contact for data set sharing. Indeed, the tens of
thousands of educational researchers are spread across a large number of organizational
settings of various types, and with varying interests. A manager of an adaptive
educational technology would be hard pressed to identify an appropriate avenue for
sharing data to benefit the general cause of educational research and development.
Proprietary Interests
If adaptive educational technologies develop along the lines suggested at least by some,
increasingly such systems may move from being laboratory and experimental systems to
being large-scale services offered by commercial entities. This migration has already
occurred for cognitive tutors for math, SQL-Tutor for database programming, and
Mastering Physics for college level physics. Commercial entities will have proprietary
interests in the adaptive educational technologies at the core of their business operations,
and as a result they may become sensitive to the risks to their intellectual property posed
by sharing the data generated by such systems.18 As commercial firms seek to protect
trade secrets that confer advantage on their products and services, the risks that
accompany the sharing of data with educational researchers may be unacceptable.19
18
For example, the University of Phoenix appears to have recognized the value of the
unique set of systems it has developed to manage and deliver its core educational
business (Selingo, 2005; Sharkey, 2011). The recent purchase of Carnegie Learning and
its adaptive technologies such as the Cognitive Tutor program by the University of
Phoenix’s parent organization Apollo Group (Keller, 2011) only confirms the importance
of adaptive learning technologies and the data they generate for the future of the
university.
19
Note that the value of keeping “trade secrets” may accrue even in the absence of any
unique technical approaches as anyone who has enjoyed fast food with “secret sauce” can
attest.
21
The question of interests is complex in the case of adaptive educational technologies as a
result of the multiple parties involved. For any adaptive system, interests and ownership
rights may accrue to the learners, to the professional educators working with the learners
and working themselves with the adaptive systems, to the school systems engaging the
adaptive educational technology providers, and, of course, to the system providers
themselves. Even if any one of these parties found it in their own immediate interest to
share data with researchers, it is not clear that they would have the complete and
unfettered right to do so.
Interoperability
As students move through various levels, stages, and institutional and non-institutional
(e.g., network) settings for their education, there is no guarantee that the data generated
by the adaptive educational technologies to which they are exposed will either follow
them or be able to be integrated into a full educational data biography. Indeed, even
within a single institution at a single point in time, there is no guarantee that the various
systems and tools being employed in the delivery of education will be able to exchange
data in meaningful ways. Indeed, without serious efforts to create an infrastructure to
support the exchange of data among applications, we are likely to be confronted with data
islands, rich segments of data on parts of the student educational experience, nearly
impossible to combine into a complete picture upon which to base decisions.
The issue of standards for the collection and reporting of educational data is not new.
However, the issue of interoperability or common data standards is likely to become
more pressing as the number of adaptive educational technologies and the number of
providers of such technologies grows over time. Such data can be generated by
commercial platforms, experimental systems, learning management systems, postsecondary course platforms, stand-alone learning applications, and online learning
opportunities in general. Moreover, the challenge represented by the adaptive
technologies is made more intense by the need to recognize the highly contextualized
nature of much of the data generated.
Student Privacy Interests
Because adaptive educational technologies gather information on individual students
through online applications they inherently raise three types of privacy concerns. First,
are the set of concerns related to the status of students as individual citizens or consumers
in a networked world where valuable access to networked resources requires the
exchange of personally identifiable information (Nissenbaum, 2010). Second, are the set
of concerns related to the status of many students as children whose privacy may require
additional protections by virtue of their youth (Pitman and McLaughlin, 2000). Third,
are the set of concerns related to the role of students within educational institutions, a role
that entails the gathering of particular kinds of information in the educational process
(Glenn, 2008). Developers, providers, and adopters of adaptive educational technologies
must confront these multiple layers of concerns for the privacy of student users of such
technologies. Educational researchers intent on using the data generated by adaptive
22
technologies must be accountable for understanding these various privacy concerns and
the procedures for addressing them in the applications that generate data use in their
research.
Strategies for Moving Forward
The issues surrounding the use of data from adaptive educational technologies are
complex and systemic. Attempts to address such issues will require efforts beyond those
of individual developers or educational researchers. Here I consider some strategies to
respond to these issues and create the conditions necessary to engage the community of
educational researchers in serious and sustained research initiatives that make good use of
the data generated by adaptive systems.
Ownership and Control Solutions
The issues surrounding the ownership and control of data, particularly data on individual
learners are among the most serious confronting efforts to make use of the data generated
by adaptive learning technologies in educational research. One approach to these issues
has been proposed by Raymond (2008) in the form of a student data backpack. The
student data backpack would be an electronic data file controlled and managed by parents
and independent of educational institutions. It would be attached to a web-based student
data clearinghouse service, and it would be able to feed data about students into systems
operated by educational institutions.20
A student data backpack, controlled by parents or presumably older learners themselves,
would address at least some of the issues of ownership and informed consent since
parents or learners would have to permit the movement of individual data into
institutional data systems and/or into educational research datasets. The basic student
information file could be enhanced by the release to parents of contextual data of the sort
generated by adaptive learning technologies. Such data could then be sought by
researchers directly from parents or learners. Alternatively, parents or learners could set
sharing permissions for all of their data at the time of transmission into institutional
systems, including adaptive learning systems, and such permission could govern the
availability of data for researchers.
Network Architecture Solutions
It is also possible to address issues of data ownership and sharing through solutions that
make use of some of the properties of networked and distributed data gathering.
Natriello, Pattinsky, Chae, and Cocciolo (2007) piloted a data gathering approach that
relied on local instances of an online course platform as the vehicles for data gathering in
classrooms in multiple schools. The data along with identifying information on
individual students were joined only within the school where data were gathered under
20
A version of the student backpack has recently been introduced by the National
Network of Digital Schools Foundation (http://www.backpacksis.com/).
23
local conditions and with permissions secured locally. Data from multiple schools were
then transmitted to a central data repository only after identifying information was
removed from the data stream. The resulting data archive contained no personally
identifiable information, and reconnecting to such information was only possible at the
level of the local school.
This is only one example of strategies that could be used to collect data at the local level
to be used to guide educational practice and then aggregate such data over large numbers
of educational institutions for research purposes while insuring anonymity. Similar
strategies of selective data assembly could be employed to gather data selectively from
any number of applications while preventing the assembly of data sets complete enough
to reveal individual student identities. In a system of locally located and fully networked
applications permitting the exchange of data for specific purposes, any number of
permutations might be constructed to maximize data access for research while protecting
the rights of data owners at each level.
Repository Solutions
The development of repositories for the data generated by adaptive educational
technologies represents another approach to improving access to such data for
educational researchers. An early model of such a repository is the Pittsburgh Science of
Learning Center DataShop (Koedinger, Baker Cunningham, Skogsholm, Leber, and
Stamper, 2011). The DataShop focuses on assembling datasets bearing on the interaction
between students and educational software such as online courses and intelligent tutoring
systems. The DataShop has developed a set of standards for the organization and
transmission of such data, and it provides researchers with a set of tools for using the
assembled datasets. The PSLC DataShop has gathered 164 datasets from 50 projects
encompassing over 25 million transactions between students and software applications,
and over 150,000 hours of interaction between students and educational software.
The PSLC DataShop might be viewed as the first of a network of such repositories for
data from adaptive educational technologies. Individual repositories in the network
might focus their energies on particular kinds of data or populations, and they might
develop specialized tool sets to aid researchers. With careful coordination the substantial
work of archiving the large datasets generated by adaptive educational technologies
might be distributed while also allowing researchers to combine data and tools from
different repositories for particular inquiries.
Research Tools
Data mining techniques can be made more accessible to educational researchers through
the development of tools designed specifically for the types of datasets generated by
adaptive learning technologies. Romero and Ventura (2007) list a set of such tools for
particular tasks:
24
Visualization – Syergo/ColAT (Avouris, Komis, Fiotakis, Margaritis, and
Voyiatzaki, 2005); GISMO/CourseVis (Mazza and Milani (2005); Listen tool
(Mostow, Beck, Cen, Cuneo, Gouvea, and Heiner, 2005)
Association – Mining tool (Zaiane and Luo, 2001); MultiStar (Silva and Vieira,
2002); Data Analysis Center (Shen, Yang, and Han, 2002); EPRules (Romero,
Ventura, Bra, and Castro, 2003); TADA-ED (Merceron and Yacefm 2005)
Text Mining – KAON (Tane, Schmitz, and Stumme, 2004); iPDF_Analyzer (Bari
and Bensater, 2005)
Classification – TAFPA (Damez, Marsala, Dang, and Bouchon-Meunier, 2005)
As Romero and Ventura (2007) suggest, the further development of tools that make data
mining techniques more accessible to educators and educational researchers will support
additional work with large-scale datasets.
Standards Solutions
A number of the issues making the use of data from adaptive educational technologies
more difficult for educational researchers might be addressed through the creation of
standards for the documentation and transmission of data to and from these systems.
Efforts to develop such standards for the data associated with adaptive systems might
build on existing efforts to develop common standards for education data such as the
National Education Data Model (http://nces.ed.gov/forum/datamodel/index.aspx) and the
Schools Interoperability Framework (http://www.sifinfo.org/us/index.asp).
Regulatory Solutions
Because there is a public interest in education and because public funds will inevitably be
used to purchase adaptive educational technologies and services, it is possible to imagine
one or more regulatory regimes that would encourage firms providing adaptive systems
to collaborate with the educational research community. For example, the federal
government might consider regulatory approaches akin to those used by the Food and
Drug Administration that require providers to present the results of independent research
trials prior to being licensed for sale. Or the state governments might require reviews
such as those now used for textbook approval before state funds could be used to
purchase adaptive educational technologies. The reviews could include criteria requiring
independent studies of the efficacy of the systems. Such regulations would create
incentives for those developing and providing adaptive educational systems to engage the
educational research community. At the very least, regulations could encourage systems
developers to use common standards for the exchange of data.
Regulatory solutions could also be used to address the multiple types of concerns related
to student privacy. The White House recently released a framework for protecting
consumer privacy online that includes a Consumer Privacy Bill of Rights (White House,
25
2012). The rights outlined provide a starting point for thinking about the kinds of privacy
provisions that might be considered in the handling of student data in adaptive
educational applications. These include:







Individual Control: Consumers have a right to exercise control over what personal
data companies collect from them and how they use it.
Transparency: Consumers have a right to easily understandable and accessible
information about privacy and security practices.
Respect for Context: Consumers have a right to expect that companies will
collect, use, and disclose personal data in ways that are consistent with the context
in which consumers provide the data.
Security: Consumers have a right to secure and responsible handling of personal
data.
Access and Accuracy: Consumers have a right to access and correct personal data
in usable formats, in a manner that is appropriate to the sensitivity of the data and
the risk of adverse consequences to consumers if the data is inaccurate.
Focused Collection: Consumers have a right to reasonable limits on the personal
data that companies collect and retain.
Accountability: Consumers have a right to have personal data handled by
companies with appropriate measures in place to assure they adhere to the
Consumer Privacy Bill of Rights. (White House, 2002, p. 1)
The White House has offered these consumer rights as the basis for possible legislation
and corporate codes of practice as well as to align U.S. policies with global standards for
Fair Information Principles (FIPP). However, broad standards applied to citizens in
general are not always viewed as applicable within school settings where there may be
both additional limitations (e.g., Zirkel, 2005) as well as special additional protections.
The youth of many students means that the regulations designed to protect children must
also be considered in thinking about the data gathered by adaptive educational systems.
The Children’s Online Privacy Protection Act (COPPA) specifies the kinds of privacy
protections accorded to those under the age of thirteen. Under the provisions of COPPA
Online providers serving children must have verifiable parental consent for the collection
of personal data from children and must post online the following requirements when
collecting information on children:






The name and contact information of all parties collecting or maintaining personal
information on children
Information on the kinds of personal information collected from children
Information on the uses of the personal information
Whether the information collected is disclosed to third parties
That the parent has the option to agree to the collection of the information without
consenting to disclosing the information to third parties
Limits on the information disclosed by the child to only that which is reasonably
necessary
26

Parent rights to review the child’s personal information, have it deleted, and
refuse to allow any further collection (Federal Trade Commission, 2006).
In the fall of 2011 the Federal Trade Commission proposed changes to COPPA for the
first time since the regulations were published in 2000 (Kardell, 2011). These changes
include expanding the definition of personally identifiable information to include:
geolocation information, media, IP addresses and other items found on computers or
mobile devices; expansion of the definition of what it means to “collect” to include
occasions when an operator encourages a child to provide information; new options for
providing parental consent; new requirements for deleting information immediately after
the time necessary to achieve the purpose for which it was collected; a requirement that
operators ensure that third parties receiving data have procedures in place to protect it.
These provisions highlight the kinds of issues and policies likely to govern the collection
of data from children through adaptive systems.
A final set of concerns and regulations pertain specifically to student data. These are
highlighted in the 1974 federal Family Educational Rights and Privacy Act (FERPA) and
its subsequent revisions. FERPA provides for rights for students and parents to know
what data is being collected, what data is retained and how it is used, limits on alternative
uses, the right to correct or amend records of personal information, and specifies
obligations to guard against misuse. The various provisions of FERPA offer useful
guidelines on individual student data in adaptive systems and access to third party
organizations conducting studies on behalf of educational agencies or institutions for a
variety of educational purposes. The regulations also offer guidance on practices such as
de-identification and anonymization of data, and on the obligations of data stewardship.
Recent amendments to FERPA regulations contemplate the use of student data in
statewide longitudinal data systems and seek to balance privacy rights against the benefits
of research (National Center for Education Statistics, 2010).
The evolving regulatory regimes pertaining to consumers, children, and students provide
some well-developed perspectives to guide thinking about data on individual students
gathered through adaptive educational technologies. Of course, such technologies
present some new challenges of their own for efforts to protect student privacy.
Professional Solutions
As the use of adaptive educational technologies grows, they may become a target for
professional standards in much the same way that such standards have been developed
for testing technologies (AERA, APA, & NCME, 1999). Such standards could create
benchmarks for the research base to support claims relating to adaptive technologies, and
these benchmarks could be used to foster collaboration between systems developers and
educational researchers.
Training Solutions
27
Preparing educational researchers to deal with the very large and complex data sets
generated by adaptive learning technologies will require focused and specialized training
and practice conducting analyses. Fortunately, there is a model for providing such
training on a national basis in the work done by the National Center for Education
Statistics to prepare and encourage educational researchers to use the large datasets
created by government sponsored national surveys. The NCES has sponsored various
institutes over the years both as stand-alone activities and in conjunction with other
professional associations. It has also provided support for scholars to participate in such
training opportunities, and it has enhanced the documentation for and online availability
of data sets from large-scale data gathering efforts.
Another approach would be to support the development of a network of institutionally
based programs that could offer specialized training. The Pittsburgh Science of Learning
Center offers a program that could serve as a prototype. The Science of Learning Center
Summer School has a track devoted to educational data mining of data from adaptive
learning technologies. Those attending the summer school, including researchers,
graduate students, and industry practitioners, receive training in the use of major
educational data mining methods. Participants in the summer program work on a final
project that has led to several peer-reviewed publications. Training experiences like this
one could be offered at university and commercial settings where adaptive educational
technologies are being developed.
Conclusion
I have considered the opportunities for educational research and educational researchers
connected with adaptive educational technologies. I have reviewed some of the research
drawing on data from adaptive technologies that has been conducted over the past decade
or so, largely by scholars outside the mainstream of educational research. I have
discussed the barriers to advancing educational research and educational improvement
through adaptive technologies, and suggested some ways to create conditions more
favorable for engaging educational researchers in this work. I want to close by posing
three possible paths forward, three scenarios that highlight the importance of the
emergence of adaptive educational technologies for the community of educational
researchers. Each scenario offers a plausible future, and each may be seen as desirable
from some perspective. However, each path represents a distinctly different set of
opportunities with associated costs and benefits.
Separate Camps Scenario
As the use of adaptive educational technologies continues to expand at all levels of the
educational system, educational researchers remain relatively uninvolved in their design
and development. As a result, these technologies draw almost exclusively on insights
generated by the mining of the increasingly large-scale datasets generated by expanded
use. They make little use of the theories and perspectives that have animated and
28
dominated scholarship in education. The result is robust practice-driven development
and improvement in systems and approaches.
As adaptive technologies play an increasingly larger and more important role in
educational practice, the educational research community becomes more separate and
detached from the community of educational practitioners who increasingly draw on the
insights generated by their ever-expanding adaptive systems. Educational practitioners
seize on the data provided by adaptive systems to justify and legitimate a full-range of
practices.
In this world of system-generated insight, the educational research community remains
separate and apart from the educational practice community. Educational scholarship
turns to largely philosophical and theoretical activities, and such activities come to
dominate the preparation of future generations of educational scholars. Educational
research continues to have empirical components, but these components are seldom as
large and robust as those inherent in adaptive technology systems.
Although some lament the detachment of educational research from educational practice
and long for a past golden age where research and practice were more closely connected,
others welcome the sense of intellectual freedom that accompanies the greater distance
from the realities of immediate practices and look forward to a more placid position in
the scholarly world.
Bridging Organization Scenario
As the use of adaptive educational technologies continues to expand at all levels of the
educational system, individual educational researchers become increasingly involved in
their design and development. Educational researchers first become connected to
adaptive technologies in several ways. Some researchers find themselves acting as
consultants to development firms and publishers who seek to draw on the theoretical
traditions of the field. Others are enlisted for their skills in data analysis, particularly
those connected with the hierarchical and nested data sets inherent in educational
systems. Still others become familiar with adaptive educational technologies when they
develop research systems as empirical test beds for theoretically driven lines of research.
The initial wave of involvement of individual isolated educational researchers in adaptive
systems leads to efforts to create mediating and facilitating organizations and utilities to
expand and enhance the participation of the educational research community in adaptive
educational technology development. The first research training programs in adaptive
technologies are started with support from external foundations and software publishers
who realize the growing need for a skilled cadre of professionals who can bridge the
worlds of learning and educational theory and adaptive systems. The creation of a
network of adaptive system data archives provides additional encouragement to a new
generation of researchers who blend the skills of educational research and systems
development.
29
Work on adaptive educational technologies comes to play an increasingly large role in
the world of educational research. Early efforts to establish a special interest group on
adaptive systems in the American Educational Research Association quickly transition to
a successful effort to establish a new division, and this division comes to rival the longstanding mainstay divisions in size and influence within the organization. Efforts to split
off and form an entirely new association are forestalled by politically astute association
leadership.21 The results of this uneasy alliance are deemed particularly valuable when
evidence begins to mount regarding the unintended and almost unrecognizable negative
impact of adaptive technologies on certain segments of students for certain learning tasks.
Embedded Research Scenario
As the use of adaptive educational technologies continues to expand at all levels of the
educational system, the systems advance to the point where they are able to gather data,
process it on-the-fly, and apply the results of the analysis all within the application itself.
The subsequent cycles of this process result in further refinements and adjustments to
these embedded “research” processes. Large scale adaptive systems are the first to
incorporate these capacities, but over time even smaller scale systems adopt such design
elements as the techniques become more standardized and well known.
In this world, the work of educational researchers shifts dramatically. The role of
educational researchers is no longer to engage personally in all phases of the research
process (design, data collection, analysis, interpretation, publication, application).
Instead, educational researchers are primarily engaged in the design of research
components to be embedded within adaptive educational technologies. Although basic
research on human learning remains (and remains largely outside of educational
research), educational research outside of the development and refinement of adaptive
learning technologies largely fades away.
The transformation of educational research associated with the rise of embedded research
capacities within adaptive educational technologies results in a major reconfiguration of
the profession. The skill requirements for educational research increase in ways
heretofore unknown to the profession even as the numbers of individuals engaged in
educational research fall by 90%.22
With the development of communities of researchers and related organizations in
the areas of Intelligent Tutoring Systems (ITS), Artificial Intelligence in Education
(AIED), Educational Data Mining (EDM), and Learning Analytics and Knowledge
(LAK), none of which have strong ties to AERA, it may be past the point where AERA
can become a dominant organization in the professional activities of such scholars.
22 It is difficult to grasp fully how this kind of transformation might be plausible, but
recent reports of a similar transformation in the legal profession may be instructive
(Markoff, 2011).
21
30
While some decry the decline in the numbers of educational researchers or the shift in the
location of research positions from the academy to systems providers, others see the
transition precipitated by the rise of research embedded in adaptive educational
technologies as a long-overdue housecleaning of the profession accompanied by an
increase in skill requirements and a new closer affiliation with on-the-ground delivery of
educational services.
These scenarios surely over dramatize the options before us. There are already too many
scholars who identify as educational researchers engaged in developing and studying
adaptive technologies to allow us to think that the entire educational research community
will be shut out of future development. By the same token, the independent and growing
organizations devoted to such technologies do not leave us optimistic about their full
incorporation into the main bodies of the American Educational Research Association.
In some respects, though, the embedding of research components into adaptive
educational systems presents the greatest prospect for a change in the scholarly
landscape. Such a development would alter the conditions under which scholarship in
education is conducted. The prospect of such profound change makes consideration of
the impact of developing adaptive and interactive educational technologies even more
important for scholars in the field.
References
Abelson, R. (1968). Theories of Cognitive Consistency: A Sourcebook. Chicago: Rand
McNally.
Aleven, V., Stahl, E., Schworm, S., Fischer, F., and Wallace, R. (2003). Help seeking and
help design in interactive learning environments. Review of Educational Research 73(2),
277-320.
Aleven, V., McLaren, B., Roll, I., and Koedinger, K. (2004). Toward tutoring help
seeking: applying cognitive modeling to meta-cognitive skills. In Proceedings of the 7th
Internation Conference on Intelligent Tutoring Systems, 227-239.
Aleven, V., McLaren, B., Roll, I., and Koedinger, K. (2006). Toward metacognitive
tutoring: A model of help seeking with a cognitive tutor. International Journal of
Artificial Intelligence in Education, 16, 101-130
American Educational Research Association, American Psychological Association, and
National Council for Measurement in Education. (1999). Standards for Educational and
Psychological Testing. Washington, DC.
31
Amershi, S., and Conati, C. (2009). Combining unsupervised and supervised
classification to build user models for exploratory learning environments. Journal of
Educational Data Mining, 1(1), 18–71.
Antunes, C. (2008). Acquiring background knowledge for intelligent tutoring systems. In
Proceedings of the 1st International Confernence on Educational Data Mining, Montreal,
QC, Canada, 18–27.
Arroyo, I., Murray, T., Woolf, B., and Beal, C. (2004). Inferring unobservable learning
variables from students’ help seeking behavior. Lecture Notes on Computer Science,
3220, 782-784.
Arroyo, I. and Woolf, B. (2005). Inferring learning and attitudes from a Bayesian
network of log file data. In C.K, Looi, G. McCalla, B. Bredeweg, and J. Breuker (eds.),
Proceedings of the Twelfth International Conference on Artificial Intelligence in
Education, pp. 33-40. Amsterdam: IOS Press.
Asuma, R. (1997). A survey of augmented reality. Presence: Teleoperators and virtual
environments, 6, 4, 355-385.
Avouris, N., Komis, V., Fiotakis, G., Margaritis, M., and Voyiatzaki, E. (2005). Why
logging of fingertip actions is not enough for analysis of learning activities. In the
Workshop on Usage Analysis in Learning Systems at the 12th International Conference
on Artificial Intelligence in Education. Amsterdam.
Baker, R. (2010). Data mining for education. In P. Peterson, E. Baker, and B. McCaw,
and, (eds.). International Encyclopedia of Education (3rd edition). vol. 7, pp. 112-118.
Oxford, UK: Elsevier.
Baker, R. (2007). Modeling and understanding students’ off-task behavior in intelligent
tutoring systems. In Proceedings of the SIGCHI Conference on Human Factors in
Computing Systems. San Jose, CA, 1059–1068.
Baker, R., Corbett, A., and Aleven, V. (2008). Improving contextual models of guessing
and slipping with a truncated training set. In Proceedings of the 1st International
Conference on Educational Data Mining, Montreal, QC, Canada, 67–76.
Baker, R., Corbett, A., and Koedinger, K. (2004). Detecting student misuse of intelligent
tutoring systems. Lecture Notes in Computer Science, vol. 3220/2004, 54-76.
Baker, R., Corbett, A., Roll, I., and Koedinger, K.R. (2008). Developing a generalizable
detector of when students game the system. User Modeling and User-Adapted
Interaction, 18, 3, 287-314.
Baker, R. & Yacef, K. (2009). The state of educational data mining in 2009: A review
and future visions. Journal of Educational Data Mining 1, 1, 3-17.
32
Baker, R., Pardos, Z., Gowdam, S., Nooraei, B., and Heffernanm, N. (2011) Ensembling
predictions of student knowledge within intelligent tutoring systems. User Modeling,
Adaptation and Personalization, Lecture Notes in Computer Science, 6787, 13-24.
Baker, R., Walonoski J., Heffernan, N., Roll, I., Corbett, A., and Koedinger, K. (2008).
Why students engage in “gaming the system” behavior in interactive learning
environments. Journal of Interactive Learning Research, 19 (2), 185-224.
Ba-Omar, H., Petrounias, I., and Anwar, F. (2007). A framework for using web usage
mining for personalise e-learning. In Proceedings of the International Conference on
Advanced Learning Technologies. Niigata, Japan, 937–938.
Barab, S., Gresalfi, M., and Ingram-Goble, A. (2010). Transformational play: Using
games to position person, content, and context. Educational Researcher, 39 (7), 525-536.
Bari, M. and Benzater, B. (2005). Retrieving data from pdf interactive multimedia
productions. In Proceedings of the Fifth International Conference On Human System
Learning, Marrakech – Morocco, 321-330.
Barnes, T., Bitzer, D. & Vouk, M. (2005). Experimental analysis of the q-matrix method
in knowledge discovery. Foundations of Intelligent Systems. Lecture Notes in Computer
Science, 3488, 603-611.
Barnes, T. and Stamper, J. (2008). Toward automatic hint generation for logic proof
tutoring using historical student data,” in Proceedings of the 9th International Conference
on Intelligent Tutoring Systems, Montreal, QC, Canada, 373–382.
Beal, C. and Cohen, P. (2008). Temporal data mining for educational applications. In
Proceedings of the 10th Pacific Rim International Conference on Artificial Intelligence:
Trends in Artificial Intelligence. Hanoi, Vietnam, 66–77.
Beal, C., Qu, L. and Lee, H. (2008). Mathematics motivation and achievement as
predictors of high school students’ guessing and help-seeking with instructional software.
Journal of Computer Assisted Learning, 24, 507-514.
Beck, J. and Mostow, J. (2008). How who should practice: Using learning
decomposition to evaluate the efficacy of different types of practice for different types of
students. In Proceedings of the 9th International Conference on Intelligent Tutoring
Systems, 353-362.
Beck, J. and Woolf, B. (2000). High-level student modeling with machine learning. In
Proceedings of the 5th International Conference on Intelligent Tutoring Systems,
Alagoas, Brazil, pp. 584–593.
33
Bell, D. (1976). The coming of post-industrial society: A venture in social forecasting.
New York: Basic Books.
Bellaachia and E. Vommina, (2006). MINEL: A framework for mining e-learning logs in
Proceedings of the Fifth IASTED International Conference on Web-Based Education,
Mexico, 259–263.
Ben-naim, D., Marcus, N., and Bain, M. (2008). Visualization and analysis of
student interaction in an adaptive exploratory learning environment, in Proceedings of the
International Workshop in Intelligent Support of Exploratory Environment. European
Confernce on Technology Enhanced Learning, Maastricht, The Netherlands, 1–10.
Boekaerts, M., Pintrich, P., and Zeidner, M. (2000). Handbook of self-regulation. San
Diego: Academic Press.
Brasch, W. (2005). Fool’s gold in the nation’s data-mining programs Social Science
Computer Review, 23, 4, 401-428.
Brusilovsky, P. (1996). Methods and techniques of adaptive hypermedia. User Modeling
and User-Adapted Interaction, 6, 87-129.
Brusilovsky, P. (1998). Methods and techniques of adaptive hypermedia. Pp. 1-43 in P.
Brusilovsky, A. Kobsa, and J. Vassileva (eds.). Adaptive Hypertext and Hypermedia.
Dordrech: Kluwer.
Brusilovsky, P. (2001). Adaptive hypermedia. User Modeling and User-Adapted
Interaction, 11, 87-110.
Brusilovsky, P. (2004). Adaptive navigation support: From adaptive hypermedia to the
adaptive web and beyond. PsychNology Journal, 2(1), 7-23.
Brusilovsky, P. and Maybury, M.T. (2002). From adaptive hypermedia to the adaptive
web. Communications of the ACM, 45(5), 30-33.
Butler, D. and Winne, P. (1995). Feedback and self-regulated learning: A theoretical
synthesis. Review of Educational Research, 65, 245-281.
Carr, B. and Goldstein, I. (1977). Overlays: A theory of modeling for computer-aided
instruction. AI Lab Memo 406 (Logo Memo 40). Massachusetts Institution of Technology,
Cambridge, MA.
Castro, F., Vellido, A., Nebot, A., and Mugica, F. (2007). Applying data mining
techniques to e-learning problems. Studies in Computational Intelligence, 62, 183-221.
Cetintas, A., Si, L., Xin, Y. and Hord, C. (2009). Predicting correctness of problem
solving from low-level log data in intelligent tutoring systems,” in Proceedings of the 2nd
International Conference on Educational Data Mining, Cordoba, Spain, pp. 230– 238.
34
Chang, K., Beck, J., Mostow, J., and Corbett, A. (2006). A bayes net toolkit for student
modeling in intelligent tutoring systems,” In Proceedings of the 8th International
Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, 104–113.
Chi, M. (2009). Active-constructive-interactive: A conceptual framework for
differentiating learning activities. Topics in Cognitive Science, 1, 73-105.
Chi, M., VanLehn, K., Litman, D., and Jordan, P. (2010). Inducing effective pedagogical
strategies using learning context features. User Modeling, Adaptation, and
Personalization, Lecture Notes in Computer Science, 6075, 147-158.
Computering Research Association. (2005). Cybrainfrastructure for Education and
Learning for the Future: A Vision and Research Agenda. Washington, DC: Computing
Research Association.
Corbett, A.T., and Anderson, J.R (1995). Knowledge tracing: Modeling the acquisition
of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278.
Corcoran T., Mosher, F. & Rogat, A. (2009). Learning Progressions in Science: An
Evidence-Based Approach to Reform. New York: Consortium for Policy Research in
Education, Teachers College, Columbia University.
Crespo, R., Pardo, A., Perez, J., and Kloos, C. (2005). An algorithm for peer review
matching using student profiles based on fuzzy classification and genetic algorithms. In
Proceedings of the International Conference on Innovations in Applied Artificial
Intelligence, Bari, Italy, 685–694.
Damez, M., Marsala, C., Dang, T., and Bouchon-Meunier, B. (2005). Fuzzy decision tree
for user modeling from human-computer interactions. In Proceedings of the 5th
International Conference on Human System Learning, Marrakech, Morocco, 287-302.
Daro, P., Mosher, F., & Corcoran, T. (2011). Learning Trajectories in Mathematics: A
Foundation for Standards, Curriculum, Assessment and Instruction. New York:
Consortium for Policy Research in Education, Teachers College, Columbia University.
De Vicente, A. and Pain, H. (2002). Informing the detection of the students’
motivational state: an empirical study. Lecture Notes in Computer Science, 2362,
933-943.
Desmarais, M., Gagnon, M., and Meshkinfram, P. (2006). Bayesian student models based
on item to item knowledge structures,” in Proceedings of the First European Conference
on Technology Enhanced Learning, Crete, Greece, pp. 1–10.
35
Drigas, A., and Vrettaros, J. (2004) An intelligent tool for building e-learning contentmaterial using natural language in digital libraries. WSEAS Transactions on Information
Science and Applications, 5(1), 11978-1205.
Federal Communications Commission (2010). Connecting America: The National
Broadband Plan. Washington, DC: Federal Communications Commission, March.
Feng, M. and Beck, J. (2009). Back to the future: A non-automated method of
constructing transfer models. In Proceedings of the 2nd International Conference on
Educational Data Mining, Cordoba, Spain, 240–248.
Federal Trade Commission (2006). How To Comply with the Children’s Online Privacy
Protection Rule. Washington, DC: Federal Trade Commission Bureau of Consumer
Protection. http://business.ftc.gov/documents/bus45-how-comply-childrens-onlineprivacy-protection-rule
Freyberger, J., Heffernan, N., and Ruiz, C. (2004). Using association rules to guide a
search for best fitting transfer models of student learning. In Workshop on Analyzing
Student-Tutor Interaction Logs to Improve Educational Outcomes at ITS Conference,
Maceio, Brazil.
Garcia, E., Romero, C., Ventura, S., and Castro, C. (2009). An architecture for making
recommendations to courseware authors using association rule mining and collaborative
filtering. User Modeling and User-Adapted Interaction, 19, 99–132.
Glenn, D. (2008). Huge databases offer a research goldmine and privacy worries.
Chronicle of Higher Education, 54(35), A10.
Gong, Y., Rai, D., Beck, J., and Heffernan, N. (2009). Does self-discipline impact
students’ knowledge and learning? In Proceedings of the 2nd International Conference
on Educational Data Mining, Cordoba, Spain, 61–70.
Greene, J. A., & Azevedo, R. (2009). A macro-level analysis of SRL processes and their
relations to the acquisition of sophisticated mental models. Contemporary Educational
Psychology, 34, 18–29.
Gresalfi, M., Barab, S., Siyahhan, S., and Christensen, T. (2009). Virtual worlds,
conceptual understanding, and me: Designing for consequential engagement. On the
Horizon, 17 (1), 21-34.
Hamalainen, W., Suhonen, J., Sutinen, E., and Toivonen, H. (2004). Data mining in
personalizing distance education courses. In Proceedings of the World Conference on
Open Learning and Distance Education, Hong Kong, 1–11.
36
Hamalainen, W. and Vinni, M. (2006). Comparison of machine learning methods for
intelligent tutoring systems, in Proceedings of the 8th International Conference on
Intelligent Tutoring Systems, Taiwan, pp. 525–534.
Heraud, J., France, L., and Mille, A. (2004). Pixed: An ITS that guides students with the
help of learners’ interaction log. In Proceedings of the International Conference on
Intelligent Tutoring Systems, Maceio, Brazil, 57–64.
Höllerer, T. & Feiner, S. (2004). Mobile augmented reality. In H. Karimi and A.
Hammad (eds.). Telegeoformatics: Location-based computing and services. New York:
Taylor and Francis, pp. 221-258.
Hurley, T. and Weibelzahl, S. (2007). Using MotSaRT to support on-line teachers in
student motivation. In Proceedings of the European Conference on Technology
Enhanced Learning. Crete, Creece, 101–111.
Hwang, G. (1999). A knowledge-based system as an intelligent learning advisor on
computer networks. Journal of Systems, Man, and Cybernetics, 2, 153-158.
Hwang, G. (2003). A test-sheet-generating algorithm for multiple assessment
requirements. IEEE Transactions on Education, 43(3), 329-337.
Hwang, W., Chang, C., and Chen, G. (2004). The relationship of learning traits,
motivation and performance-learning response dynamics,” Computers and Education, 42,
267–287.
Hwang, G., Huang, T., and Tseng, C. (2004). A group-decision approach for evaluating
educational web sites. Computers and Education, 42, 64-86.
Johnson, L., Smith, R., Willis, H., Levine, A., & Haywood, K. (2011). The 2011 Horizon
Report. Austin, TX: The New Media Consortium.
Jong, B., Chan, T., and Wu, Y. (2007). Learning log explorer in e-learning diagnosis,”
IEEE Transactions on Education, 50(3), 216–228.
Jonsson, A., Hasmik, M., Johns, J., Mehranian, H., Arroyo, I., Woolf, B., Barto, A.,
Fisher, D., and Mahadevan, S. (2005). Evaluating the feasibility of learning student
models from data,” in Proceedings of the AAAI Workshop on Educational Data Mining,
Pittsburgh, PA, 1–6.
Karampiperis, P. and Sampson, D. (2005). Adaptive learning resources sequencing in
educational hypermedia systems. Educational Technology and Society, 8(4), 128–147.
Kardell, N. (2011). FTC will propose broader children’s online privacy safeguards.
National Law Review, http://www.natlawreview.com/article/ftc-will-propose-broaderchildren-s-online-privacy-safeguards
37
Keller, J. (2011, August 2). Apollo to buy adaptive learning company for $75 million.
The Chronicle of Higher Education.
Kelly, D. and Tangney, B. (2005). First aid for you: Getting to know your learning style
using machine learning. In Proceedings of the IEEE International Conference on
Advanced Learning Technologies. Washington, DC, 1-3.
Koedinger, K. and Corbett, A. (2006). Cognitive tutors: Technology bringing learning
sciences to the classroom. In R. Sawyer (ed.). The Cambridge Handbook of the Learning
Sciences. New York: Cambridge University Press.
Koedinger, K., Baker, R., Cunningham, K., Skogsholm, A., Leber, B., and Stamper, J.
(2011). A data repository for the EDM community: The PSLC DataShop. In Romero,
C., Ventura, S., Pechenizky, M., and Baker, R. (eds.). Handbook of Educational Data
Mining. Boca Raton, FL: CRC Press, pp. 43-55.
Knutov, E., De Bra, & Pechenizkly, M. (2009). AH 12 years later: A comprehensive
survey of adaptive hypermedia methods and techniques. New Review of Hypermedia and
Multimedia, 15(1), 5-38.
Kobsa, A. (2007). Privacy enhanced personalization. Communications of the ACM,
50(8), 24-33.
Koedinger, K., Baker, R., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J.
(2011). A data repository for the EDM Community: The PSLC Datashop. In C. Romero,
S. Ventura, M. Pschenisky, & R. Baker (Eds.), Handbook of educational data mining (pp.
43-55). Baco Raton: CRC Press.
Kotsiantis, S., Pierrakeas, C., and Pintelas, P. (2004). Predicting students’ performance in
distance learning using machine learning techniques. Applied Artificial Intelligence,
18(5), 411-426.
Kristofic, A. (2005). Recommender system for adaptive hypermedia applications. In
Proceedings of the 1st Student Research Conference on Informatics and Information
Technologies, Bratislava, Slovakia, 2005, pp. 229–234.i
Kumar, K. (1995). From post-industrial to post-modern society: New theories of the
contemporary world. Oxford, UK: Blackwell.
Lajoie, S. P. & Azevedo, R. (2006). Teaching and learning in technology-rich
environments. In P. Alexander & P. Winne (Eds.), Handbook of educational psychology
(2nd ed.) (pp. 803-821). Mahwah, NJ: Erlbaum.
Landow, G. (2006). Hypertext 3.0: Critical theory and new media in an era of
globalization. Baltimore: The Johns Hopkins University Press.
38
Lee, J., & Park, O. (2007). Adaptive instructional systems. In J.M. Spector, M.D. Merrill,
J.V. Merrienboer, & M.P. Driscoll (Eds.), Handbook of research on educational
communications and technology (3rd ed., pp. 469-484). Mahwah, NJ: Lawrence Erlbaum
Associates.
Lovette, M., Meyer, O., and Thille, C. (2008). The open learning initiative: Measuring
the effectiveness of the OLI statistics course in accelerating student learning. Journal of
Interactive Media in Education, 14
Lu, J. (2004). A personalized e-learning material recommender system. In Proceedings of
the 2nd International Conference on Information Technology for Application., Harbin,
China, 374–379.
Lu, F., Li, X., Liu, Q., Yang, Z., Tan, G., and He, T. (2007). Research on personalized elearning system using fuzzy set based clustering algorithm. In Proceedings of the Seventh
International Conference on Computational Science. Beijing, China, 587–590.
Maimon, O. and Rokach, L. (2005). Chapter 1 – Introduction to knowledge discovery in
databases in O. Maimon and L. Rokach (eds.). The Data Mining and Knowledge
Discovery Handbook (pp. 1-14). New York: Springer.
Madhyastha, T. and Tanimoto, S. (2009). Student consistency and implications for
feedback in online assessment systems. In Barnes, T., Desmarais, M., Romero, C., &
Ventura, S. (Eds.) Educational Data Mining 2009: 2nd International Conference on
Educational Data Mining, Proceedings. Cordoba, Spain. July 1-3, 81-90.
Markoff, J. (2011, March 5). Armies of expensive lawyers replaced by cheaper software.
New York Times, A1.
Martin, J. and VanLehn, K. (1995). Student assessment using Bayesian nets.
International Journal of Human-Computer Studies, 42, 575-591.
Martinez, R., Kay, J, and Yacef, K. (2011). Visualizations for longitudinal participation,
contribution and progress of a collaborative task at the tabletop. Proceedings of the
conference on Computer Supported Collaborative Learning, vol. 1, 25-32.
Mazza, R., and Milani, C. (2005). Exploring usage analysis in learning systems: Gaining
insights from visualizations. In the Workshop on Usage Analysis in Learning Systems at
the 12th International Conference on Artificial Intelligence in Education, Amsterdam.
Mclaren, B., Koedinger, K., Schneider, M., Harrer, A., and Lollen, L. (2004).
Bootstrapping novice data: Semi-automated tutor authoring using student log files. In
Prcoeedings of the Workshop on Analyzing Student-Tutor Interaction Logs to Improve
Education Outcomes, Seventh International Conference on Intelligent Tutoring Systems
Alagoas, Brazil, 1–10.
39
Mcquiggan, S., Mott, B., and Lester, J. (2008). Modeling self-efficacy in intelligent
tutoring systems: An inductive approach. User Modeling and User-Adapted Interaction,
18, 81-123.
Merceron, A. and Yacef, K. (2004). Mining student data captured from a web-based
tutoring tool: Initial exploration and results. Journal of Interactive Learning Research,
15, 4, 319-346.
Mislevy, R., Behren, J., Bennett, R., Demark, S., Fresso, D., Levy, R., Robinson, D.,
Rutstein, D., Shute, V., Stanley, K., and Winters, F. (2010). On the roles of external
knowledge representations in assessment design. Journal of Technology, Learning and
Assessment, 8(2), Retrieved from http://www.jtla.org
Mizue, K. and Toshio, O. (2001). N3: Neural network navigation support-knowledgenavigation in hyperspace: The sub-symbolic approach. Journal of Educational
Multimedia and Hypermedia, 10(1), 85-103.
Morales, C., Perez, A., Soto, S., Martinez, C., and Gomez, A. (2006). Using sequential
pattern mining for links recommendations in adaptive hypermedia educational systems.
Current Developments in Technology-Assisted Education. Badajoz, Spain:
FORMATEX, 1016-1020.
Mostow, J., Beck, J., Cen, H., Cuneo, A., Gouvea, E., and Heiner, C. (2005). An
educational data mining tool to browse tutor-student interactions: Time will tell! In
Proceedings of the Workshop on Educational Data Mining, 15–22.
Muehlenbrock, M. (2005). Automatic action analysis in an interactive learning
environment. Usage Analysis in Learning Systems. Twelfth International Conference on
Artificial Intelligence in Education. Amsterdam, The Netherlands, pp. 73-80.
Mullier, D. (2003). A tutorial supervisor for automatic assessment in educational
systems. International Journal of on e-Learning, 2(1), 37-49.
National Center for Education Statistics (2010). Basic Concepts and Definitions for
Privacy and Confidentiality in Student Education Records. Washington, DC: National
Center for Education Statistics, Brief 1, NCDS 2011-601.
http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2011601
Natriello, G. (2010). Adaptive Learning Opportunities. New York: Teachers College.
Natriello, G., Pattinsky, M., Chae, H. and Cocciolo. A. (2007). E-learning systems as
research platforms: Results from the Networked Education Database (NED) Project.
Paper presented at the Annual Meeting of the American Educational Research
Association, Chicago, April.
Nissenbaum, H. (2010). Privacy in context: Technology, policy, and the integrity of
social life. Stanford, CA: Stanford University Press.
40
NSF Task Force on Cyberlearning. (2008). Fostering Learning in a Networked World:
The Cyberlearning Opportunity and Challenge. Washington, DC: National Science
Foundation.
Pardos, Z., Heffernan, N., Anderson, B., and Heffernan, C. (2007), The effect of model
granularity on student performance prediction using bayesian networks,” in Proceedings
of the 11th International Conference on User Modeling, Corfu, Greece, pp. 435– 439.
Paris, S. and Paris, A. (2001). Classroom applications of research on self-regularted
learning. Educational Psychologist, 36(2), 89-101.
Pavlik, P., Cen, H., and Koedinger, K. (2009). Learning factors transfer analysis: Using
learning curve analysis to automatically generate domain models. In Proceedings of the
2nd International Conference on Educational Data Mining, Cordoba, Spain, 121-130.
Pavlik Jr., P., Cen, H., & Koedinger, K. (2009). Performance factors analysis -- A new
alternative to knowledge tracing. In V. Dimitrova & R. Mizoguchi (Eds.), Proceedings of
the 14th International Conference on Artificial Intelligence in Education. Brighton,
England.
Pechenizkiy, M., Trcka, N., Vasilyeva, E., van der Aalst, W., and De Bra, P. (2009).
Process mining online assessment data. In Proceedings of the 2nd International
Conference on Educational Data Mining, Cordoba, Spain, 279-288.
Pierce, N., Conlan, O., and Wade, V. (2008). Adaptive educational games: Providing
non-invasive personalized learning experiences. Second IEEE International Conference
on Digital Games and Intelligent Toy Enhanced Learning. Banff, BC.
Pitman, J. and McLaughlin, B. (2000). Making cyberspace safe for children. Educational
Leadership, 67(6), 67-71.
Raab, C. and Bennett, C. (1998). The distribution of privacy risks: Who needs
protection. The Information Society, 14, 263-274.
Rai, D., Gong, Y., and Beck, J. (2009). Using Dirichlet priors to improve model
parameter plausibility,” in Proceedings of the 2nd International Conference on
Educatiional Data Mining, Cordoba, Spain, 141–150.
Raymond, M. (2008). The student data backpack. In Kanstoroom, M., and Osberg, E.,
(eds). A Byte at the Apple: Rethinking Student Data for the Post-NCLB Era.
Washington, DC: Thomas B. Fordham Institute, pp. 142-158.
Rebolledo-Mendex, G., du Boulay, B., and Luckin, R. (2006). Motivating the learner: an
empirical evaluation. ITS2006, Eighth International Conference on Intelligent Tutoring
Systems. Taiwan. Lecture Notes in Computer Science, 4053, 545-554.
41
Ritter, S., Harris, T., Nixon, T., Dickison, D., Murray, R., and Towle, B. (2009).
Reducing the knowledge tracing space. In Proceedings of the 2nd International
Conference on Educational Data Mining, Cordoba, Spain, 151–160.
Robinet, V., Bisson, G., Gordon, M., and Lemaire, B. (2007). Searching for student
intermediate mental steps,” in Proceedings of the Workshop on Data Mining for User
Modeling at the 11sth International Conference on User Modeling, Corfu, Greece, 101–
105.
Roll, I., Aleven, V., McLaren, B., & Koedinger, K. (2011). Improving students’ helpseeking skills using metacognitive feedback in an intelligent tutoring system. Learning
and Instruction, 21, 267-280.
Romero, C. and Ventura, S. (2007). Educational data mining: A survey from 1995 to
2005. Expert Systems with Applications, 33, 135-146.
Romero, C., Ventura, S., and De Bra, P. (2004). Knowledge discovery with genetic
programming for providing feedback to courseware. User Modeling and User-Adapted
Interaction, 14(5), 425-464.
Romero C., Ventura, S., Bra, P., and Castro, C. (2003). Discovering prediction rules in
aha! Courses. In Proceedings of the User Modeling Conference, 25-34.
Romero, C., Ventura, S., Pechenisky, M., & Baker, R. (2011). Handbook of educational
data mining. Baco Raton: CRC Press.
Romero, C., Ventura, S., Zafra, A., and De Bra, P. (2009). Applying web usage mining
for personalizing hyperlinks in web-based adaptive education systems,” Computers and
Education, 53(3), 828–840.
Rus, V., Lintean, M., and Azevedo, R. (2009). Automatic detection of student mental
models during prior knowledge activation in MetaTutor. (2009). In Proceedings of the
2nd International Conference on Educational Data Mining, Cordoba, Spain, 161–170.
Schunk, D. and Zimmerman, B. (1994). Self-regulation of learning and
performance. Hillsdale, NJ: Erlbaum.
Schofield, J. (1995). Computers and classroom culture. New York: Cambridge.
Selingo, J. (2005, June 17). University of Phoenix owes rapid growth to use of
technology. Chronicle of Higher Education
Shang, Y., Shi, H., and Chen, S. (2001). An intelligent distributed environment for active
learning. ACM Journal of Educational Resources in Computing, 1(2), 308-315.
42
Sharkey, M. (2011). Academic analytics at the University of Phoenix. Proceedings of the
First International Conference on Learning Analytics and Knowledge, 122-126.
Sheard, J., Ceddia, J., and Hurst, G. (2003). Inferring student learning behavior from
website interactions: A usuage analysis. Education and Information Technologies, 8(3),
245-266.
Shen, R., Yang, F., and Han, P. (2002). Data analysis center based on e-learning
platform. In Proceedings of the 5th International Workshop on the Internet Challenge:
Technology and Applications, 19-28.
Shute, V.J. & Zapata-Rivera, D. (2007). Adaptive technologies. In J.M. Spector, M.D.
Merrill, J.V. Merrienboer, & M.P. Driscoll (Eds.), Handbook of research on educational
communications and technology (3rd ed., pp. 279-294). Mahwah, NJ: Lawrence Erlbaum
Associates.
Silva, D. and Vieira, M. (2002). Using data warehouse and data mining resources for
ongoing assessment in distance learning. In IEEE International Conference on Advanced
Learning Technologies, Kazan, Russia, 40-45.
Simko, M. and Bielikova, M. (2009). Automatic concept relationships discovery for an
adaptive e-course. In Proceeding of the 2nd International Conference on Educational
Data Mining, Cordoba, Spain, 171–178.
Stamper, J., and Barnes, T. (2009). Unsupervised MDP value selection for automating
ITS capabilities. In Proceedings of the International Conference on Educational Data
Mining. Cordoba, Spain, 180–188.
Stanford Aptitude Seminar. (2002). Remaking the concept of aptitude: Extending the
legacy of Richard E. Snow. Mahway, NJ: Lawrence Erlbaum Associates.
Stephens, C. and Sukumar, R. (2006). An introduction to data mining. In R. Grover and
M. Vriens. (eds.). Handbook of Marketing Research, (pp. 455-485). Thousand Oaks CA:
Sage Publications.
Superby, J., Vandamme, J., and Meskens, N. (2006). Determination of factors influencing
the achievement of first-year university students using data mining methods. In
Proceedings of the Workshop on Educational Data Mining at the 8th International
Conference on Intelligent Tutoring Systems, 37-44.
Tane, J., Schmitz, C., and Stumme, G. (2004). Semantic resource management for the
web: An e-learning application. In Proceedings of the 13th International World Wide
Web Conference, New York, 1-10.
Tang, C., Yin, H., Li, T., Lau, R., Li, Q., and Kilis, D. (2000). Personalized courseware
construction based on web data mining. In Proceedings of the first international
conference on web information systems engineering. Washington, DC., 204-211.
43
Tang, C., Lau, R., Li, Q., Yin, H., Li, T., and D. Kilis, D. (2000). Personalized
courseware construction based on web data mining,” in Proceedings of the 1st
International Conference on Web Information Systems Engineering, Hong Kong, China,
204–211.
Tang, T. and McCalla, G. (2005). Smart recommendation for an evolving e-learning
system. International Journal on E-Learning, 4, 1, 105-129.
Tsai, C., Tseng, S., and Lin, C. (2001). A two-phase fuzzy mining and learning algorithm
for adaptive learning environment. In Proceedings of the International Conference on
Computer Science, San Francisco, 429-438.
Turkle, S. (2007). Life on the screen: Identity in the age of the internet. New York:
Simon and Schuster.
Ueno M. (2003). LMS with irregular learning processes detection system. In World
Conference on e-Learning in Corporate, Government, Health, and Higher Education,
2486-2493.
Urbancic, T., Skrjanc, M., and Flach, P. (2002). Web-based analysis of data ming and
decision support education. AI Communications, 15, 199-204.
VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial
Intelligence in Education, 16(3), 227-265.
Vee, M., Meyer, B., and Mannock, K. (2006). Understanding novice errors and error
paths in Object-oriented programming through log analysis. In Proceedings of the
Workshop on Educational Data Mining, Taiwan, 2006, 13–20..
Vellido, A., Castro, F., Nebot, A., and Mugica, F. (2006). Characterization of atypical
virtual campus usage behavior through robust generative relevance analysis. In V. Uskov
(ed.). The 5th IASTED International Conference on Web-Based Education. Puerto
Vallarta, Mexico, 183-188.
Vialardi, C., Bravo, J., and Ortigosa, A. (2008). Improving AEH courses through log
analysis. Journal of Universal Computer Science, 14(17), 2777–2798.
Wang, Y., Cheng, Y., Chang, T., and Jen, S. (2008). On the application of data mining
technique and genetic algorithm to an automatic course scheduling system. In
Proceedings of the IEEE Conference on Cybernetics and Intelligent Systems, Chengdu,
China, 400–405.
Wang, W., Weng, J., Su, J., and Tseng, S. (2004). Learning portfolio analysis and mining
in scorm compliant environment. In Proceedings of the 34th ASEE/IEEE Frontiers in
Education Conference. Savannah, GA, 17–24.
44
Wang, Y., Tseng, M., and Liao, H. (2009). Data mining for adaptive learning sequence in
English language instruction. Expert Systems with Applications: An International
Journal, 36, 7681–7686.
Want, T. and Mitrovic, A. (2002). Using neural networks to predict student’s
performance,” in Proceedings of the International Conference on Computers in
Education, Washington, DC, pp. 1–5.
Wenger, E. (1987). Artificial Intelligence and Tutoring Systems. Los Altos, CA: Morgan
Kaufmann Publishers, Co.
White, B., Shimoda, T., and Frederiksen, J. (1999). Enabling students to construct
theories of collaborative inquiry and reflective learning: computer support for
metacognitive development. International Journal of Artificial Intelligence in Education,
10, 151-182.
White House (2012). Consumer Data Privacy in a Networked World: A Framework for
Protecting Privacy and Promoting Innovation in the Global Digital Economy.
Washington, DC: The White House.
Woolf, B. (2009). Building Intelligent Interactive Tutors. Burlington, MA: Morgan
Kaufmann Publishers, Co.
Yudelson, M., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., and Rebecca, D.
(2006). Mining student learning data to develop high level pedagogic strategy in a
medical ITS. In Proceedings of the AAAI Workshop on Educational Data Mining,
Boston, MA, 1–8.
Zaiane, O. and Luo, J. (2001). Web usage mining for a better web-based learning
environment. In the Proceedings of the Conference on Advanced Technology for
Education, Banff, Alberta, 60-64.
Zakrzewska, Z. (2008). Cluster analysis for user’s modeling in intelligent e-learning
systems. In Proceedings of the 21st international conference on Industrial, Engineering
and Other Applications of Applied Intelligent Systems. Poland, 2008, pp. 209–214.
Zimmerman, B. and Schunck, D. (2001). Self-regulated learning and academic
achievement (2nd ed.). Mahway, NJ:L Erlbaum.
Zinn, C. and Scheuer, O. (2006). Getting to know your students in distance learning
contexts. In Proceedings of the 1st European Conference on Technology Enhanced
Learning, 437-451.
Zirkel, P. (2005). Searching students. Principal Leadership, 6(1), 64-68.
45
Appendix – Examples of Adaptive Learning Technologies
1. CAST UDL Lesson Builder
http://lessonbuilder.cast.org/
Developed by the Center for Applied Special Technology - CAST the UDL (Universal
Design for Learning) Lesson Builder is a web-based resource that provides educators
with models and tools to create and adapt lessons that increase access and participation in
the general education curriculum for all students.
Based on research related to three primary brain networks (recognition network, strategic
network, and affective network) and the roles they play in understanding these
differences, UDL provides a framework for creating and implementing lessons with
flexible goals, methods, materials, and assessments that support learning for all students.
The UDL lesson builder site provides resources so people can:
- Learn about Universal Design for Learning (UDL)
- Explore model UDL lesson plans
- Create, save, edit and print personal lesson plans
Once a user creates an account on the site, he/she can proceed to create lesson plans. A
lesson plan template is provided with all the major headings already in place. The user
then enters content in spaces provided under these sections. A "more information" button
is also available under each section, and upon clicking this, a window pops up with
detailed explanation of all what the particular section entails. The template is also
available as a downloadable word document. Model lesson plans and links to relevant
external resources are also available.
Lesson Builder can be considered an adaptive resource within the context of this
discussion in that it provides:
• Multiple means of representation (text, audio and video), to give diverse users
options for acquiring information and knowledge
• Multiple means of action and expression, to enable users demonstrate what they
know
• Multiple means of engagement, to tap into users' interests, offer appropriate
challenges, and increase motivation
The resource thus works for all categories of users - from the novice to the expert. As it is
not strictly a learning resource, however, Lesson Builder's functionality falls short of
offering that extra level of adaptivity where users are assessed (implicitly and explicitly)
to determine their needs and abilities before prescribing paths deemed appropriate for
each individual user. Likewise, user input is not monitored on the fly and content or
learning paths modified appropriately. This is, however, understandable as the purpose of
the tools is not to explicitly teach individuals how to create lesson plans but to enable
46
users create lesson plans in accordance with UDL principles. A lesson plan creator
however gets the opportunity to learn about adaptivity and universal design as work
progresses.
2. Jasper Woodbury Problem Solving Series
http://peabody.vanderbilt.edu/projects/funded/jasper/Jasperhome.html
Designed by the Cognition and Technology Group at Vanderbilt University (CTGV) the
Jasper Woodbury series of adventures represents an example of anchored instruction, and
also of problem-based learning. Targeted at upper elementary through high-school math
learners the Jasper Series are video-based, narrative adventures that pose complex
problems for students to solve using information embedded in the narrative. Students
watch video segments to understand the situation and the problem and then embark on
problem solving tasks using embedded data, teaching resources etc. Students then present
their solutions to their fellow classmates and discuss strengths and weaknesses of
whatever set of solutions they come up with. The ill-structured nature of the problems
ensures that there is no one perfect solution, and this caters to students with different skill
levels in math problem solving strategies.
The Jasper Series typifies a constructed learning application that specifically aims at
teaching math concepts and applications together with problem-solving and criticalthinking skills. Adaptivity comes in the form of user self-selection of path and resources
to complete tasks, and upon solving a problem one way students can still tackle the same
problem through a more advanced path. Conversely, learners trying to adopt problem
solving strategies that are beyond their capabilities will naturally fail until they master the
requisite skills. Other forms of adaptivity including curriculum sequencing (based on
learner ability), adaptive presentation and navigation support are, however, not available.
3. INSPIRE (INtelligent System for Personalized Instruction in a Remote Environment),
INSPIRE is a web-based adaptive educational hypermedia prototype developed by
Papanikolaou and co-workers to support an introductory course on Computer
Architecture offered to second level undergraduate learners of the Department of
Informatics and Telecommunications of the University of Athens. Its development was
informed largely by instructional design and learning style theories.
INSPIRE gives learners the option to select meaningful learning goals, and then using
additional information gathered through learner-system interaction, the system generates
a sequence of authentic and meaningful tasks that match the learning style and
knowledge level of each learner, based on the target goal. Also throughout its interaction
with the learner, INSPIRE dynamically generates learner-tailored lessons that gradually
lead to the accomplishment of a learner’s learning goals. In addition, learners have
instructional control over the system as they have the option to intervene in the lesson
47
generation process, express their opinion about their own characteristics or about the
lesson contents. INSPIRE is therefore fully adaptive (i.e. adapts its output using some
data or knowledge about the learner) and adaptable (supports end-user modifiability,
providing learners control over several functionalities).
4. MasteringPhysics
http://www.masteringphysics.com
MasteringPhysics is web-based hypermedia application developed by Pearson to
accompany “University Physics,” a standard textbook for many first year college level
physics programs. It has been positioned as a dynamic tutoring system that increases
learner engagement and performance. MasteringPhysics is being adopted by educational
institutions worldwide, notable among which are MIT and the University of Sydney.
MasteringPhysics emulates the instructor's office-hour environment, coaching students on
problem solving techniques by asking students simpler sub-questions, providing
immediate and specific feedback on wrong answers and giving specific feedback on
common errors to help explain why a particular answer is not correct. Hints (accessible
only when needed) are also available throughout the course of solving a problem.
As an adaptive teaching tool, MasteringPhysics facilitates one-on-one tutoring,
encouraging (and compelling) learners to go through particular learning paths based on
their input and navigation of the system. Emphasis is therefore on teaching students the
concepts and application of Physics and not merely to solve a problem or accomplish a
task.
5. Apangea Math
http://www.apangea.com/products/apangea_math.htm
Apangea Math is an internet-based math learning support system that provides one-toone differentiated math instruction through a unique integration of proprietary tutoring
technology and live, online certified teachers. Developed by Apangea Learning Inc.
Apangea Math is aligned to National Council of Teachers of Mathematics (NCTM) and
individual state standards. Schools in more than 25 states across the United States are
reported to be using Apangea Math as a supplement to instruction.
Apangea Math represents an innovative way of integrating computer and human tutoring
to provide adaptive math instruction to individual learners. According to the creators,
Apangea Math is based on the understanding that students learn to analyze and solve
word problems by applying a pedagogy derived from contemporary cognitive science,
including principles of active problem solving, elaboration theory, categorization by
prototype, mastery learning, and worked examples. The system thus challenges students
48
to think mathematically by offering flexible learning pathways that teach students both
concepts and procedures.
In terms of adaptivity, as students work, the program monitors student work and provides
feedback tailored to what students need. Students can monitor their progress on a course
map and progress meter. Quizzes and progress scores are used to determine if students
have mastered content or have additional learning needs. When a topic needs to be
revisited, it is automatically placed back into the student's learning pathway to provide
additional learning opportunities. Teachers have access to assessment data in real-time
reports.
6. SMART.FM (IKNOW!)
http://iknow.jp/
Smart.fm (formerly iKnow!) is a social learning and community website created by
Cerego Japan, Inc. The website uses intelligent software algorithms to assist users retain
facts in memory and increase learning speed. Users can create, manage and share lists of
facts to memorize; as well as learn a number of languages. Progress can be monitored
with tests. The data is used to automatically plan a curriculum and learning strategy for
the users.
In addition to the website base, Smart.fm also has an iPhone/iPod touch application, a
Twitter feed, a YouTube account, and a Facebook application.
Smart.fm represents the first phase of a platform that combines personalized learning
applications and content creation tools in a collaborative, social environment where
members study, create, re-mix, share, and manage learning content of any kind.
7. Carnegie Mellon Open Learning Initiative
http://oli.web.cmu.edu/openlearning/
The Carnegie Mellon Open Learning Initiative has produced a set of online courses
utilizing adaptive technologies that are open to the public at large. The courses draw on
research in the learning sciences to provide a complete instructional experience for
students. Learner behavior are tracked constantly and used to drive improvement to the
system.
A major feature of the Open Learning Initiative courses is the series of “mini-tutors” or
simplified versions of the cognitive tutors developed through Carnegie Learning that are
embedded within the course content. The mini-tutors allow students to answer practice
questions and receive feedback. In addition, students complete self-assessments
49
throughout the courses, and the results of these self-assessments are displayed alongside
the performance information to allow students to develop their metacognitive skills as
they move forward.
50