Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Adaptive Educational Technologies and Educational Research: Opportunities, Analyses, and Infrastructure Needs Gary Natriello Teachers College, Columbia University February 2012 Background Paper Prepared for the National Academy of Education 1 Adaptive Educational Technologies and Educational Research: Opportunities, Analyses, and Infrastructure Needs1 Introduction Throughout the world the growing demand for educational opportunities is driven by the needs of both advanced and developing economies to have greater proportions of citizens achieve high levels of learning and to be able to continue to learn effectively throughout their lives. The growth of post-industrial knowledge work (Bell, 1976; Kumar, 2004) imposes the requirement of widespread access to higher learning on all societies that wish to participate in the global economy. The transformation in the conditions of work brought on by advances in computing and communications technologies also brings with it the means to re-position educational institutions and practices on a new base of technologies that are at once informationbased, digital, and networked (Computing Research Association; 2005; NSF Task Force on Cyberlearning, 2008; Johnson, Smith, Willis, Levine, & Haywood, 2011). These technologies are facilitating the development of new educational materials, tools, and environments, and they are reshaping the roles of teachers and students. With such sweeping changes in the possible conditions for educational work, it should not be surprising that there are also new opportunities for the work of educational researchers. In this paper I consider the impact of one type of new educational technology and the associated practices on the opportunities for educational research. In particular, I explore the new possibilities for research and analysis generated by the growing use of adaptive educational technologies. I proceed in three stages. First, I consider the research opportunities made possible by the data generated from adaptive educational technologies. Second, I review research that has been conducted to date using data from adaptive technologies. Finally, I explore how research drawing on data from adaptive learning technologies might evolve and what might be needed to support and facilitate various evolutionary paths. Research Opportunities from Adaptive Educational Technologies To appreciate the research opportunities connected with data generated by adaptive educational technologies, it is important to have a general understanding of the bases of such technologies and the data sets that they generate. I approach this task in seven stages by (1) defining adaptive technologies, (2) reviewing the types of learner I would like to thank Ryan Baker, Sasha Barab, Randy Bennet, Ken Koedinger, Kurt VanLehn and the participants in the National Academy workshop for helpful comments on an earlier version of this paper. The remaining limitations are my own. 1 2 characteristics often recognized in adaptive systems, (3) specifying what I term the “sites of adaptation,” (i.e., the features of systems that can be adapted), (4) considering a comprehensive multi-layer model of adaptation in hypermedia systems, (5) reviewing a framework for examining the structure of adaption within intelligent tutoring systems, (6) considering how complex learning activities and environments may be developed with adaptive features, and concluding with (7) a discussion of the features of the data sets from adaptive systems that create unique opportunities for educational research. Adaptive Technologies Over the past several decades there has been a progression in the development of computer-based systems that engage students with educational opportunities. The introduction of such systems as research systems has moved on to include other unique and proprietary systems. These systems, often initiated in laboratory studies, have moved to field research sites in select classrooms, and then on to wide scale adoption. Koedinger and Corbett (2006) describe this kind of progression for cognitive tutors. There has also been a transition from stand-alone hypermedia systems to web-based systems available via the Internet. Of course, it was this last development, the widespread growth of the world wide web in an increasingly networked context, that brought interactive systems to the population at large for a variety of activities. Adaptive educational technologies are a special class of interactive systems that take account of learner performance and adapt accordingly to maximize learning. Systems of this type attempt to create personalized educational experiences optimized for each individual student. Brusilovsky and colleagues (Brusilovsky, 1996; Brusilovsky, 1998; Brusilovsky, 2001) provide important background for understanding the development of adaptive hypermedia learning systems. Wenger (1987), VanLehn (2006), and Woolf (2009) explain the foundations of adaptive tutoring systems. Lovette, Meyer, and Thille (2008) examine the use of adaptive technologies in the context of a university course. Pierce, Conlan, and Wade (2008) discuss the growing number of adaptive educational games while observing that such games still constitute a small proportion of all educational games. Taking Account of Learner Characteristics Adaptive systems rely on an explicit model2 of the individual user or learner that includes information on the individual (e.g., goals, interests, knowledge, recent online behavior) that is used to distinguish among different users (Wenger, 1987). Brusilovsky (1996) identified five types of information on individuals that might be employed in an adaptive system: goals, knowledge, background, hyperspace experience, and preferences. Brusilovsky (2001) identified two additional types of user information; Early groups involved in work on adaptive systems included scholars in the User Modeling, Artificial Intelligence in Education, and Instructional Tutoring Systems communities. 2 3 namely, interests and individual traits. He also added environmental dimensions (e.g., bandwidth) as another kind of information related to adaptation. Considering these eight types of information on users or learners is helpful background for thinking about educational research possibilities. Goals refer to the objectives the user is trying to accomplish in the context of the learning environment. Adaptive systems typically support a set of possible learner goals. Simple systems may support only one or a small number of goals or tasks, while more sophisticated systems may support a hierarchy of related tasks organized to support goal accomplishment (e.g., Rebelledo-Mendez, du Boulay, and Luckin, 2006). Learner knowledge refers to the knowledge that a learner possesses in relation to a model of the structure of a bounded knowledge domain (Brusilovsky, 1996). The knowledge domain is represented by a structure of concepts and their relationships, and the knowledge state of the learner is represented in an overlay model (Carr and Goldstein, 1977) that seeks to assess the state of user knowledge in reference to the domain structure. Background refers to all of the available information related to the learner’s previous experience outside the immediate adaptive system. Included under background may be the user’s learning history. Hyperspace experience refers to the learner’s familiarity with the structure of online systems as reflected in the ease of navigation. Such familiarity is distinct from the learner’s knowledge of the subject, and has bearing on the most appropriate navigation techniques to present to the learner.3 Preferences refers to the fact that a learner can prefer some areas of a system over others and some parts of an individual page over others. Brusilovsky (1996) points out that the user has to inform the system about these preferences and that adaptive systems can then generalize such preferences in new contexts within the system. Interests refer to the long-term or enduring interests of the learner (as opposed to the short-term task or goal). Interests are an increasingly important dimension of information retrieval systems as these systems combine interests with short-term goals or tasks to develop models to improve filtering and recommendations. Traits, as conceptualized by Brusilovsky (2001), refer to user features such as personality factors, cognitive factors, and learning styles that are stable or only change over long periods of time. 4 More general experience and familiarity with technical systems (Turkle, 2007), especially in education (Schofield, 1995), can also play a role. 4 There is controversy among psychologists over use of this term; some reserve the term “trait” for personal qualities that cannot be changed and prefer to use propensity in referencing features that Brusilovsky calls traits (Stanford Aptitude Seminar, 2002). 3 4 The immediate environment in which the learner is accessing the adaptive system offers possibilities for adaptation along several dimensions. Environments can differ, for example, in terms of the software platform, hardware capacities, and network bandwidth. Adapting successfully to variables such as these can influence comprehension of the material presented. The growth of mobile devices adds an additional environmental dimension connected to the smaller screen and the capacity to identify geographical location based on the global positioning system. Such location information has ad aptive potential for delivering information to augment understanding in connection with observations of the physical world (Azuma, 1997; Höllerer and Feiner, 2004). The broad range of types of learner information included in adaptive systems can suggest an equally broad range of types of research (e.g., Arroyo and Wolf, 2005; Baker, Walonoski, et al., 2008; Beal, Qu, and Lee, 2008). Perhaps more important, the fact that such information is often collected continuously over the course of the learning experience means that the granularity and extensiveness of the resulting data sets is unlike anything typically encountered by educational researchers in the course of most current investigations. Sites of Adaptation With the range of types of learner information that have been specified in adaptive systems identified, I turn now to what might be called the “sites” of adaptation, that is, those dimensions of learning systems and platforms that might be adapted in an effort to meet the needs of learners. Once again, Brusilovsky (1996) provides a useful starting point in his discussion of adaptive hypermedia. Brusilovsky (1996) specifies two broad classes of adaptive strategies: adaptive presentation and adaptive navigation. Adaptive presentation includes three categories: adaptive multimedia presentation, adaptive text presentation, and adaptation of modality. Adaptive multimedia presentation is the tailoring of the presentation of content within online media based on user characteristics. For example, a system might alter the density and size of multimedia files presented to the learner based on the bandwidth of the learner’s online connection. Adaptive text presentation involves presenting learners with different learner models with different text. Brusilovsky (2001) divides adaptive text presentation into canned text adaptation and natural language adaptation. The former involves inserting or removing text fragments, altering fragments, stretchtext,5 sorting text fragments, or dimming Nevertheless, relatively stable qualities such as these present special challenges to adaptive systems because they often require special psychological tests to extract. 5 Stretchtext is a technique for providing more detailed text treatment of particular content to facilitate learner understanding. Stretchtext retains the original text on the screen to provide a context for the text that is added to aid understanding (Landow, 2006). 5 fragments. The latter may make similar use of fragments by applying natural language technologies. Adaptive navigation support entails techniques that assist learners in finding their paths through content by adapting the way that links are presented in response to different learner characteristics. Brusilovsky (2001) identifies six techniques of adaptive navigation: direct guidance, adaptive link sorting, adaptive link hiding (including hiding, disabling, and removal), adaptive link annotation, adaptive link generation, and map adaptation. Direct guidance is showing the learner the best link based on learner characteristics as represented in the user model. Adaptive sorting positions the best link (according to the user model) at the top of the list of links. Adaptive hiding restricts the navigation options available to the learner by hiding links to pages that are not appropriate according to the user model. Adaptive link annotation provides additional information to the user about the page behind the link either through additional text or visual cues. Map adaptation involves adapting the form of hypermedia maps based on the characteristics of the learner as represented in the user model.6 A Comprehensive View of Adaptive Technologies in Hypermedia Systems To move toward a comprehensive view of adaptive technologies the literature on adaptive hypermedia once again provides some accessible treatments. In a review of the recent progress in the development of adaptive hypermedia systems, Knutov, De Bra, and Pechenizkiy (2009) provide a useful multi-layer model that summarizes such systems. They organize their discussion around six major questions of adaptation: Why do we need adaptation (Why?) What can we adapt? (What?) What can we adapt to? (To What?) Where can we apply adaptation? (Where?) When can we apply adaptation? (When?) How do we adapt? (How?) (pp. 7-8) Knutov, De Bra, and Pechenizkiy (2009) note that reference models used to describe 6 Brusilovsky and Maybury (2002) and Brusilovsky (2004) discuss the challenge of adaptive navigation support in situations other than assisting users in moving through a closed body of hypertext. Two situations that press the limitations of adaptive systems are the open body of resources represented by the web and the complex graphical dimensions in virtual reality environments. Providing adaptive support in these situations pose new challenges and opportunities. Brusilovsky (2004) discusses the use of both content-based and social technologies to address the open body of resources on the web. Brusilovsky (2004) notes that adaptive navigation support technologies developed for hypertext have analogs in similar approaches in virtual environments. 6 adaptive hypermedia architectures employ a tower model that represents different elements as different levels or layers. Along these lines, they discuss five different layers as they relate to the key questions. The Goal Model (GM) layer (addressing the “Why?” question) potentially involves not just goals, but also a hierarchical structure that includes “goals, objectives, tasks, requirements, workflows” (Knutov, De Bra, and Pechenizkiy, 2009, p. 23). A generalized view of the goal centered approach might include a “hierarchy of goals and corresponding tasks comprising this goal, and workflows that need to be followed to complete a requirement (Knutov, De Bra, and Pechenizkiy, 2009, pp. 23-24). The Domain Model (DM) layer (addressing the “What?” question) consists of concepts and the relationships among concepts, most often organized in a hierarchy. When the Domain Model is structured so that concepts are fine grained and structured hierarchically it is possible to apply adaptive techniques. The User Model (UM) layer (addressing the “To What?” question) is made up of entities that have values stored in a structure (typically a table) that overlays the Domain Model, in effect creating a map of some attribute (such as user knowledge) over the domain. Systems can use both domain dependent properties such as user knowledge and domain independent properties such as user characteristics. The Context Model (CM) layer (addressing the “Where?” and “When?” questions) refer to system features that take account of the context in which the adaptation is applied and the environment in which the application is used. Adaptation processes and decisions dependent on an application address the “Where?” question. Adaptation processes and decisions independent of an application (e.g., time, day of the week, network bandwidth) address the “When?” question. The Adaptation Model (AM) layer (addressing the “How?” question) deals with techniques and methods of providing adaptation within a system. Knutov, De Bra, and Pechenizkiy (2009) identify techniques of content adaptation (inserting/removing fragments, altering fragments), techniques of adaptive presentation (dimming fragments, sorting fragments, stretchtext, zoom/scale, layout, link sorting/ordering, link annotation, combinatorial techniques), and techniques of adaptive navigation (link generation, guidance, link hiding). A View of Adaptive Technologies in Tutoring Systems VanLehn (2006) proposes a framework for examining the structure of adaptation within intelligent tutoring systems. His framework includes six elements and two processes. The elements are: (1) task domain, or the competence that the tutoring system is trying to impart to the student; (2) task, or an activity lasting from several minutes to over an hour; (3) step, or a user interface interaction taken by the student in order to complete a task; (4) knowledge component, or any principle, concept, rule, fact, association, or other fragment of task-specific information into which knowledge can be decomposed; (5) 7 learning event, or the mental construction or application of a knowledge component; and (6) incorrect, or the designation applied to a step, knowledge component, or learning event indicating that the tutoring system should say something about it to meet current instructional objectives. These elements are organized by two types of adaptive processes: outer loop processes which determine which tasks the student should do next, and inner loop processes which determine the steps for students within tasks. VanLehn (2006) identifies four types of outer loop processes: (1) the student selects the task from a menu, (2) the tutor assigns tasks in a predetermined sequence, (3) the tutor assigns tasks from a pool of tasks until mastery is achieved, and (4) macroadaptive learning or selecting tasks based on a match between the task’s traits and the student’s traits. VanLehn (2006) notes that inner loop processes can be thought of as services that support student learning and identifies five common services in intelligent tutoring systems: (1) minimal feedback, typically whether a step is correct or incorrect; (2) error-specific feedback to help the student understand why something is wrong and how to avoid the mistake in the future; (3) hints on the next step to improve student performance moving forward, (4) assessment of knowledge for instructors, students, or the tutoring system itself; and (5) review of the entire solution offered by the student. Multiple Conceptions of Learning Activities The ways in which learning activities are conceived also carry implications for thinking about adaptive learning technologies, the kinds of experiences they support, and the kinds of data they might generate. Chi (2009) distinguishes four types of overt or visible activities of learners while engaging with a resource such as a text, a tutoring system or a virtual environment. Passive activities are those in which the learner is not doing anything while being exposed to the resource. Active activities involve the learner doing something, usually physically, involving the resource. Constructive activities are those in which the learner is producing outputs that contain ideas beyond those presented in the resource. Interactive activities are those in which the learner is dialoguing in a substantive way on the same topic with another individual or system and not ignoring the other’s contribution. As we move from passive to interactive learning activities, the complexity of the activities and the corresponding data gathering opportunities can become more elaborate and more complex. Barab, Gresalfi, and Igram-Goble (2010) conceptualize even more elaborate and complex learning arrangements in their discussion of the possibilities for transformational play in games and virtual worlds. They draw on four types of content engagement identified in Grasalfi, Barab, Siyahhan and Christensen (2009) as a way to think about adaptive educational technologies that can support transactive engagement with adaptive worlds. They contrast the four types of content engagement as: (1) procedural or learning about what to do, (2) conceptual or understanding how their tools work in these worlds, (3) consequential or learning about the impact of their actions on designed contexts, and (4) critical or reflecting on the impact of their actions on designed contexts. The adaptive worlds that can facilitate transformational play are responsive to the decisions learners 8 make within them, and the systems generate data on both the adaptations of the worlds and the transformations of the learners. Whether or not one prefers the layered tower model of adaptive hypermedia of Knutov, De Bra, and Pechenizkiy (2009), the framework for intelligent tutoring systems offered by VanLehn (2006), and/or the multiple views of learning provided by Barab, Grasalfi and colleagues, they each contribute to thinking about the data generating potential of adaptive systems. These data generating and gathering opportunities inherent in adaptive systems provide a good foundation for considering the kinds of research opportunities that such systems may afford educational researchers. Unique Features of Data Generated by Adaptive Educational Technologies Adaptive Educational Technologies generate data sets that offer both new opportunities and new challenges for educational researchers. The data sets are larger and often more complex than those typically encountered in educational research. Moreover, they tend to be unique along the following eight dimensions. First, the breadth of data generated by adaptive learning technologies can be considerably greater than the norm for educational research. The adaptive technologies and platforms can engage very large numbers of learners simultaneously and gather data about that engagement in ways not otherwise available. Second, the depth of the data captured by adaptive technologies is more extensive than what can be gathered in most studies in education. Adaptive technology platforms provide data on both educational program elements (i.e., materials and interactions) and student responses. Moreover, they also provide data on the context in which those student responses were generated. Third, the granularity of the data generated by adaptive technologies is considerably finer than that gathered in most educational research. Each element of a program can be tracked to its delivery point, and every student movement, i.e., keystroke, in response can be included in the dataset. In fact, somewhat less granular semantic actions are both easier to handle (Stephens and Sukumar, 2006) and potentially more useful for research purposes (Mislevy, et al., 2010.) Fourth, the linkage between student behaviors and the data elements gathered is more consistently tighter than in other data available to educational researchers. Adaptive technologies evoke students’ behaviors on an on-going basis to maintain the interaction with the delivery of the educational program. Fifth, the data generated by adaptive technologies and platforms is time specific, and the data sets are inherently longitudinal over small intervals and for potentially long periods of time. For example, the Pittsburgh Science of Learning Center DataShop has data sets that cover entire years from middle and high school students who used educational 9 software three times per week (Koedinger, Baker, Cunningham, Skogsholm, Leber, and Stamper, 2011). Such data collection is typically cost-prohibitive in most educational research. Sixth, the data generated by adaptive systems can be multi-dimensional across both elements of the educational program and elements of student performance, behavior, and background as suggested by the earlier discussion of the multiple layers of data inherent in adaptive technologies. This means that multiple investigators or teams can examine different streams of data in parallel. Seventh, data from adaptive educational technologies exhibits what I call the “fruit fly effect” after the preference that many biologists have for using fruit flies in their research to take advantage of the short life-cycle and rapid progress through the generations. Data from adaptive educational systems provide shorter, faster iterative cycles of data as the educational program elements of the system can be adapted quickly in reaction to student responses. Eighth, data gathering in adaptive systems is integrated with program delivery in a way seldom encountered in educational research activities that are generally grafted on (often over considerable resistance) to the regular business of educational programs. These features of the data sets generated by adaptive educational technologies offer new possibilities for educational researchers to examine problems and issues previously examined in other data, and they open up new kinds of research questions for consideration. However, such data sets can also seem overwhelming. Fortunately, there are strategies that can be used to examine such data, and I turn to these next. Analyses Using Data from Adaptive Educational Technologies I consider the use of data generated by adaptive educational technologies from two perspectives. First, I examine the analytic approaches that have been used most frequently to mine the data produced from these systems, drawing on several typologies in the field. Second, I review the application areas where data from adaptive systems have been employed to provide insight and drive action to improve the educational experience. Approaches to Utilizing Data from Adaptive Educational Technologies The segment of the research community that has devoted the most attention to examining data generated by adaptive educational technologies is known as educational data mining (Baker and Yacef, 2009). This community is a part of the larger group of scholars focused on mining data from very large databases. As Baker and Yacef (2009) note, educational data mining focuses on the unique types of data available from educational 10 platforms and settings, often characterized by multiple hierarchical levels and often drawing on psychometrics and the learning sciences.7 Romero and Ventura (2007) provide a careful review of work on educational data mining from 1995 to 2005, a period during which the field emerged and coalesced around a set of techniques and approaches. They position educational data mining as a step in the overall process of knowledge discovery in databases (often referred to as KDD),8 and they consider it a formative evaluation technique. They distinguish three target audiences for this kind of inquiry: students, educators, and academics and administrators. When the audience is students (e.g., Tang and McCalla, 2005) the objective is often to recommend activities to improve their learning. When the audience is educators (e.g., Merceron and Yacef, 2004) the objective is to provide feedback on instruction and the educational offering. When the audience is academics and administrators (e.g., Urbancic, Skrjanc, and Flach, 2002) the objective is to guide site improvements and the distribution of resources. Romero and Ventura (2007) note that it is possible to apply data mining techniques to data generated by traditional classroom based education, for example, when such techniques are applied to university data bases. Of greater relevance, they also note that data mining techniques are ideally suited to the large data sets generated by distance education activities, including web-based courses, learning management systems, and our current concern, adaptive systems, or what they and others refer to as adaptive and intelligent web-based educational systems (or AIWBES).9 Adaptive systems provide richer and most robust data sets and thus contain both more opportunities and more challenges for analysis than non-adaptive systems The review of data preprocessing tasks presented by Romero and Ventura (2007) provides further insight into the data mining endeavor. They identify eight steps as part of preprocessing: 1) data cleaning or the removal of irrelevant items and log entries 7 Data mining as an approach to inquiry has generated various sorts of objections in its relatively short period of ascendance. These include objections regarding privacy concerns, violation of constitutional rights when used by government agencies, accuracy of the data, the effectiveness of the procedures (Brasch, 2005), and the atheoretical nature of much data mining work. 8 Maimon and Rokach (2005) specify the knowledge discovery in databases process as nine steps: 1) developing an understanding of the application domain, 2) selecting and creating a data set on which discovery will be performed, 3) preprocessing and cleansing, 4) data transformation, 5) choosing the appropriate data mining task, 6) choosing the data mining algorithm, 7) employing the data mining algorithm, 8) evaluation, and 9) using the discovered knowledge. 9 Such systems are the result of the joint evolution of intelligent tutoring systems and adaptive hypermedia systems. Among the adaptive systems noted are those described in Brusilovsky and Peylo (2003): SQL-Tutor, German Tutor, ActiveMath, VC-Prolog-Tutor (all intelligent tutoring systems) and AHA!, InterBook, KBS-Hyperbook, and WebCOBALT (all adaptive hypermedia systems). 11 including graphics and scripts; 2) user identification or the process of associating pages accessed to the particular user; 3) session identification or grouping of page references for a user into user sessions; 4) path completion or inserting page references missing due to browser or proxy server caching; 5) transaction identification or the disaggregation of sessions into smaller transactions; 6) data transformation and enrichment or creating new variables, values, or meanings for log references; 7) data integration or synchronizing data from diverse streams or sources; and 8) data reduction or reducing dimensionality. Some of these activities are familiar ones connected to the preparation of a wide range of data sets in education; but others are quite specific to the task of using system log data and databases. Romero and Ventura divide data mining activities into two broad classes: 1) statistics and visualization, and 2) web mining. Statistics refers to a range of standard descriptive techniques for log files or databases, while visualization entails rendering complex student data in visual displays. Web mining refers to extracting information from web content, structure, and usage patterns. They note that such mining techniques can be used in an offline fashion, where the identified patterns are used by educators to make improvements in systems and practices, as well as in an online fashion, where the patterns discovered are fed directly into an intelligent software system on-the-fly.10 In their review of studies using data mining techniques in education Romero and Ventura (2007) identify three major kinds of web mining techniques: 1) clustering, classification, and outlier detection; 2) association rule mining, and sequential pattern mining; and 3) text mining. Each of these methods has been employed in studies using data from adaptive educational technologies. Clustering and classification group things together based on similarities with clustering being unsupervised and classification being supervised. Outlier detection involves identifying observations that are unusually small or large relative to others in the dataset. Romano and Ventura feature examples of each type of mining technique applied to adaptive systems. Tang and McCalla (2005) used clustering and collaborative filtering in a paper recommender system designed to personalize and adapt course content. Baker, Corbett, and Koedinger (2004) used classification to determine if students are gaming a system and avoiding learning. Muehlenbrock (2005) detected regularities and deviations to provide information for learners and educators to manage their learning and teaching. Association rule mining attempts to associate one or more attributes of a dataset with another attribute by means of an if-then statement of attribute values. Sequential pattern mining attempts to identify inter-session patterns in a time-ordered set of sessions or episodes. Freyberger, Hefferman, and Ruiz (2004) used association rules in a search for a transfer model of learning in intelligent tutoring systems. Morales, Preez, Soto, Martinez, and Gomez (2006) developed a tool for discovering sequential patterns from student usage data in an adaptive hypermedia course to recommend links to new students. 10 This is an important distinction that has implications for the shape of future careers in educational research as I discuss later in this paper. 12 Text mining extends data mining to text data and often involves machine learning and natural language processing. Tang, Yin, Li, Lau, Li, and Kilis (2000) used a key-word driven text mining algorithm to select articles for students. Baker (2009, 2010) presents another framework for considering data mining approaches in education that is somewhat more expansive. He distinguishes five major methods: 1) prediction; 2) clustering; 3) relationship mining; 4) discovery with models; and 5) distillation of data for human judgment. In prediction the goal is to develop a model that can infer a predicted variable from some combination of predictor variables. Baker identifies three types of prediction: 1) classification where the predicted variable is binary or categorical and popular methods include decision trees, logistic regression, and support vector machines; 2) regression where the predicted variable is continuous and popular methods include linear regression, neural networks, and support vector machine regression; and 3) density estimation where the predicted variable is a probability density function and where density estimators can be based on various kernel functions. In clustering the goal is to find data points that naturally group together to split the full data set into a set of clusters. In relationship mining the goal is to discover relationships between variables. Baker (2010) notes four types of relationship mining: 1) association rule mining where the goal is to find if-then rules; 2) correlation mining where the goal is to find linear correlations between variables; 3) sequential pattern mining where the goal is to find temporal relationships between events; and 4) causal data mining where the goal is to infer if one event was the cause of another. These first three types of data mining – prediction, clustering, and relationship mining – map well onto the web mining catgegories designated by Romero and Ventura (2007). The fourth type moves beyond those discussed in the earlier review. In discovery with models, a model is developed (perhaps through prediction, clustering, or human reasoning) and used to guide another analysis. Baker (2010) cites Baker, Corbett, Roll, and Koedinger (2008), a study of student gaming of the system, for the use of assessments of the probability that a student knows the current knowledge component that depend on models of the knowledge components in a domain. The fifth type of data mining identified by Baker (2010), which he labels as distillation of data for human judgment, calls on humans to make inferences about data, often with the benefit of data visualizations, that are beyond the scope of automated data mining methods. Here Baker (2010) points to the example of work by Corbett and Anderson (1995) for its use of visualizations of learning curves to distinguish between smooth and 13 spiked patterns.11 Yet another example may be found in the work of Martinez, Kay, and Yacef (2011) who use visualizations of learner participation in tabletop collaborations to assist facilitators in detecting problems in group interaction. Applications of Data Drawn from Adaptive Educational Technologies In their 2009 review of educational data mining, Baker and Yacef identify four areas where educational data mining methods have been employed productively. First, educational data mining has been used to improve student models to allow adaptive technologies to respond more appropriately to student differences. Such models have been used to do things such as identifying which students are experiencing low selfefficacy in online systems (Mcquiggan, Mott, and Lester, 2008) and identifying the factors that predict student failure in college (Superby, Vandamme, and Meskens, 2006). Pechenizkiy, Trcka, Vasilyeva, van der Aalst, and De Bra (2009) used a process mining approach to examine the data generated during exams offered through learning management systems. Second, educational data mining methods have been used to improve models of the knowledge structure of content domains automatically. For example, Barnes, Bitzer, and Vouk (2005) used a matrix based model to extract latent relationships from observed binary variables, and Pavlik, Cen, and Koedinger (2009) used analyses of learning curves to generate domain models Third, educational data mining methods have been used in studies designed to determine which types of pedagogical support are most effective in producing student learning gains. For example, Beck and Mostow (2008) investigated the effects of different types of practice for different types of students. Chi, VanLehn, Litman, and Jordan (2010) examined pedagogical strategies that lead to effective tutoring experiences for students. The fourth application area identified by Baker (2009) is in generating evidence to refine and extend educational theories. For example, Madhyastha and Tanimoto (2009) drew on the work of Abelson (1968) on cognitive consistency theory to examine the relationship between consistency and student performance with a goal of developing guidelines for scaffolding instruction. Another perspective on analyses drawing on data from adaptive educational technologies is provided by Castro, Vellido, Nebot, and Mugica (2007) in their review of the application of data mining techniques to e-learning. They distinguish studies bearing on 11 Yet another variation on the enumeration of data mining methods may be extracted from the organization of the basic methods chapters in the Handbook of Educational Data Mining (Romero, Ventura, Pechenizky, and Baker, 2011). In that volume, chapters are devoted to each of eight types of methods: visualization, basic statistical analysis, classification, clustering, association rule mining, sequential pattern analysis, process mining, and modeling hierarchy and dependence. 14 five areas in e-learning. Applications related to the assessment of student performance have been the subject of a good deal of work (e.g., Hwang, 2003; Mullier, 2003; Kotsiantis, Pierrakeas, and Pintelas, 2004; Sheard, Ceddia, and Hurst, 2003). Additional work on models for assessing student knowledge has been done by Corbett and Anderson (1995), Martin and VanLehn (1995), and Pavik, Cen, and Koedinger (2009), and a comparison of models singly and in combination has been conducted by Baker, Pardos, Gowda, Nooraie, and Heffernan (2011). Research has also focused on applications that provide course adaptation and learning recommendations as a result of student learning behavior (e.g., Mizue, and Toshio, 2001; Romero, Ventura, and De Bra, 2004; Shang, Shi, and Chen, 2001). A third area for research has been on the evaluation of learning material and web-based courses (e.g., Drigas, and Vrettaros, 2004; Hwang, Huang, and Tseng, 2004). Studies have also focused on applications that provide feedback to teachers and students in e-learning courses as a result of student learning behavior (e.g., Arroyo, Murray, Woolf, and Beal, 2004; Hwang, 1999). A fifth area of research has involved outlier detection in e-learning environments (e.g. Ueno, 2003; Vellido, Castro, and Nebot, 2006). Castro, Vellido, Nebot, and Mugica (2007) also highlight several large research projects that have incorporated data mining methods into e-learning environments. The ALFANET project (http://alfanet.ia.uned.es/alfanet) includes an e-learning platform with a component that provides online real-time recommendations and advice to learners based on previous users’ interactions and results of questionnaires. The AHA! Project (http://aha.win.tue.nl) adapts the presentation and navigation system of a course based on the level of knowledge of an individual learner. The LearningOnline Network with a Computer Assisted Personalized Approach (LON-CAPA) includes an individualized homework and automatic grading system (www.lon-capa.org). The LExICON project (http://lexikon.dfki.de/) involves course adaptation based on the students’ navigational behavior. The ATutor project (www.atutor.ca/) has created a learning content management system that includes an assessment component with student behavior tracking. Romero and Ventura (2010) in the most up-to-date review of the literature available discuss data mining efforts in terms of eleven educational tasks12: 1) analysis and visualization of data, with a goal of highlighting useful information to support decision making, (e.g., Bellaachia and Vommina, 2006; Ben-naim, Marcus, and Bain, 2008; Mostow, Beck, Cen, Cuneo, Gouvea, and Heiner, 2005; Zinn and Scheuer, 2006); 12 Romero and Ventura (2010) also organize the literature in terms of eight categories of types of data or environment: traditional education, web-based education or e-learning, learning management systems, intelligent tutoring systems, adaptive educational systems, tests and questionnaires, texts and contents, and other. Here I include as examples only those studies involving intelligent tutoring systems and those on adaptive educational systems. 15 2) providing feedback for instructors, with a goal of supporting course instructors in decision making on how to improve students’ learning opportunities, (e.g., Beal and Cohen, 2008; Garcia, Romero, Ventura, and Castro, 2009; Hurley and Weibelzahl, 2007; Romero, Ventura, and De Bra, 2004; Tsai, Tseng, and Lin, 2001; Vialardi, Bravo, and Ortigosa, 2008); 3) recommendations for students, with a goal of personalizing their learning activities and experiences (e.g., Ba-Omar, Petrounias, and Anwar, 2007; Heraud, France, and Mille, 2004; Karampiperis and Sampson, 2005; Kelly and Tangney, 2005; Kristofic, 2005; Lu, 2004; Lu, Li, Liu, Yang, Tan, and He, 2007; Pavlik, Cen, and Koedinger, 2009; Romero, Ventura, Zafra, and De Bra, 2009; Stamper and Barnes, 2009; Wang, Weng, Su, and Tseng, 2004; Wang, Tseng, and Liao, 2009); 4) predicting student’s performance, with a goal of predicting performance in the form of knowledge or some type of score (e.g., Beck and Woolf, 2000; Cetintas, Si, Xin, and Hord, 2009; Desmarais, Gagnon, and Meshkinfram, 2006; Hamalainen and Vinni, 2006; Pardos, Heffernan, Anderson, and Heffernan, 2007; Want and Mitrovic, 2002); 5) student modeling, with a goal of developing cognitive models of human users/students (e.g., Amershi and Conati, 2009; Antunes, 2008; Baker, Corbett, and Aleven, 2008; Barmes and Stamper, 2008; Chang, Beck, Mostow, and Corbett; Feng and Beck, 2009; Gong, Rai, Beck, and Heffernan, 2009; Hwang, Chang, and Chen, 2004; Jonsson, Hasmik, Johns, Mehranian, Arroyo, Woolf, Barto, Fisher, and Mahadevan, 2005; Mclaren, Koedinger, Schneider, Harrer, and Lollen, 2004; Rai, Gong, and Beck, 2009; Ritter, Harris, Nixon, Dickison, Murray and Towle, 2009; Robinet, Bisson, Gordon, and Lemaire, 2007; Rus, Lintean, and Azevedo, 2009); 6) detecting undesirable student behaviors, such as performance errors, disengagement, and acting out in some way (e.g., Baker, 2007; Jong, Chan, and Wu, 2007; Muehlenbrock, 2005; Vee, Meyer, and Mannock, 2006; Yudelson, Medvedeva, Legowski, Castine, Jukic, and Rebecca, 2006); 7) grouping students, with a goal of grouping students according to their characteristics (e.g., Crespo, Pardo, Perez, and Kloos, 2005; Hamalainen, Suhonen, Sutinen, and Toivonen, 2006; Kelley and Tangney, 2005; Zakrzewska, 2006).; 8) social network analysis, with a goal of understanding the relationships among individuals (e.g., Tang and McCalla, 2005); 9) developing concept maps, with a goal of allowing instructors automatically to create graphs showing the relationships among concepts and the hierarchical nature of knowledge in a domain (e.g., Simko and Bielikova, 2009); 16 10) constructing courseware, with the goal of assisting instructors and developers construct courseware and learning content automatically (e.g., Tang, Lau, Yin, Lin, and Kilis, 2000); and 11) planning and scheduling, with the goal of helping with future course planning and student course scheduling (e.g., Wang, Cheng, Chang, and Jen, 2008). Although much of the work being done to analyze the data from adaptive technologies is being done by scholars working in traditions not closely aligned with educational research, there is a significant amount of work being done by scholars in the learning sciences tradition.13 A number of research and development efforts to create adaptive learning opportunities have been driven by a learning process research perspective; that is, they have been influenced by the base of knowledge on patterns of human learning and strategies to apply technologies to leverage those patterns. Inherent in these approaches is a model of the learner maintained by the adaptive system and the capacity of the system to adjust itself in response to the range of processes employed by individual learners (Shute and Zapata-Rivera, 2007). Progress in the development of models of learning and learners will lead to the identification of both new learner characteristics and new modeling techniques. For example, in response to questions posed by Shute and Zapata-Rivera, (2007, p. 287), a panel of experts in the field identified the following learner variables as promising for consideration: Cognitive abilities (e.g., math skills, reading skills, cognitive developmental level, problem solving, analogical reasoning) Metacognitive skills (e.g., self-explanation, self-assessment, reflection, planning) Motivational and Affective states (e.g., engagement, flow, anxiety level) Additonal variables (e.g., personality types, learner styles, social skills such as collaboration, and perceptual skills) Lee and Park (2007) call attention to systems that include motivational variables, citing de Vicente and Pain (2002), whose motivation model includes control, challenge, independence, fantasy, confidence, sensory interest, cognitive interest, effort, and satisfaction as variables to indicate a learner’s motivational state. Lee and Park (2007) also highlight approaches and systems that consider metacognitive skills such as SCIWISE (White, Shimoda, and Frederiksen, 1999), which encourages students to express their metacognitive ideas and Help-Seeking Tutor Agent (Aleven, Stahl, Schworm, Fischer, and Wallace, 2003; Aleven, McLaren, Roll and Koedinger (2004), which supports learners to become better help seekers. The Help-Seeking Tutor Agent is also covered in more recent reports, including Aleven, McLaren, Roll, & Koedinger (2006) and Roll, Aleven, McClaren, & Koedinger (2011). This section draws on the discussion of the role of learning research in adaptive systems presented in Natriello (2010). 13 17 In their discussion of technology-rich environments, Lajoie and Azevedo (2006) stress the utility of research and development driven from well-developed learning theories. They highlight approaches that involve both cognitive and metacognitive tools to support learners. In particular, they argue that in the wake of the growing development of open electronic learning environments accompanying the spread of the Internet, students are increasingly being called upon to regulate their own learning more effectively. This leads them to call attention to the substantial body of work on self-regulated learning (e.g., Boekaerts, Pintrich, and Zeidner, 2000; Butler and Winne, 1995; Paris and Paris, 2001; Schunk and Zimmerman, 2001; Zimmerman and Schunk, 2001). Research on the impact of hypermedia environments on student self-regulated learning provides a good example of work that expands the view of student learning while simultaneously attempting to create more powerful learning opportunities. The work attends not only to learning gains, but also to the learning process. Self-regulated student learning refers to a host of processes (e.g., planning, monitoring, strategy use, handling of task difficulty and demands) (see, e.g., Greene and Azevedo, 2009) that learners can employ to become more effective. However, as Lajoie and Azevedo (2006) note, despite the range of work being done on technology-rich environments from a self-regulation perspective there has been little research that would allow us to understand the entire cycle of activities involved in self-regulated learning in technology-rich environments. Pursuing such work as part of a broader class of efforts to enhance adaptive learning opportunities could be very fruitful. The traditions of analyzing data from adaptive educational technologies formed over the past twenty years, first as these technologies appeared in stand alone systems, and more recently as they have become almost entirely internet and web based.14 The scholars engaging in this work are drawn from diverse fields, including computer science, information science, cognitive science, and others.15 Although mainstream educational researchers have not engaged in these analyses in large numbers, the opportunities for such participation are rising, and the skills and perspectives that such scholars bring can enhance this work moving forward. However, there are barriers to consider that may slow or impede such participation by educational researchers, and I turn to these and ways to address them in the next section. Enhancing Conditions For Research on Adaptive Educational Technologies 14 Additional examples of analyses of adaptive systems are presented in the case studies contained in Romero, Ventura, Pechenizky, and Baker, (2011). 15 Work involving datasets generated by online systems is also organized under the term “learning analytics,” and the First International Conference on Learning Analytics and Knowledge was held in 2011. Papers presented at this initial conference evidenced a holistic view of the use of data in educational enterprises. 18 I consider approaches to strengthening the participation of educational researchers in work that makes use of data from adaptive educational technologies along three lines. First, I take stock of the likely development of such technologies in the near future. Second, I highlight a set of issues that are emerging around the use of such data. Third, I present some strategies for moving forward and engaging educational researchers. The Growth of Adaptive Educational Technologies There are several developments that suggest that adaptive educational technologies will grow in number and pervasiveness as a part of the educational sector in the coming years. There is a growing appreciation from diverse quarters that the cyberinfrastructure for education will be increasingly important in the development of the education sector (Computing Research Association; 2005; NSF Task Force on Cyberlearning, 2008; Johnson, Smith, Willis, Levine, & Haywood, 2011). Along with the growing demand for all students to learn that can only be addressed through the delivery of more personalized educational services, investments in broadband services (Federal Communications Commission, 2010) will provide the necessary foundation for the delivery of rich interactive educational applications. Indeed, some commentators have predicted that the transition to an educational system dominated by computer-based, student-centered, personalized learning will occur within the next decade (Christensen, Horn, and Johnson, 2008).16 Adaptive educational technologies are likely to emanate from research laboratories, commercial entities such as publishers, and from educators themselves, as well as from collaborations across multiple parties. The sources and nature of such technologies are likely to become more diverse and decentralized. In the absence of an infrastructure of standards and common development platforms, this diversity will extend to the data architectures underlying the adaptive applications. As a result the challenge of drawing on data from such systems will be greater than it is at present where such systems exist in modest numbers. This will occur even as the need for educational researchers to become skilled in the analysis of such data becomes greater. A host of complicating issues is likely to emerge, and I turn to them next. Emerging Issues Educational researchers are accustomed to confronting a variety of issues as they seek to gather and examine data on educational processes and systems. Everything from data access, to confidentiality, to adequacy for addressing particular research questions and 16 Christensen, Horn, and Johnson (2008) cite four conditions that will hasten the transition to computer-based adaptive technologies: improvements in the quality of systems, the ability of students and teachers to select personalized learning paths, a looming teacher shortage as a result of the retirements of the members of the baby-boom generation, and significant cost declines as the market for adaptive learning grows. 19 more has been handled over the years. In most cases standard procedures and practices have evolved through professional consensus. Although many of the issues surrounding the use of data generated by adaptive educational technologies can be addressed through current procedures, there are some special issues that require attention, and I consider them here. Ownership of Data Although there may always be questions of who owns the data gathered through educational research, the data generated by adaptive educational technologies pose this issue in certain unique ways. An initial challenge is to be able to communicate which data are being gathered and through what means since the systems log information on individual activity rather unobtrusively. Adaptive systems could result in the combination of data from different schools and districts, and populations often in ways that are not transparent either to learners or to those educators engaging learners with the systems. Thus, there are special burdens attendant to the question of informed consent of those whose data is collected. 17 The extensiveness of the data gathering possible through adaptive educational technologies may also pose special problems. As the data streams from such systems become more diverse and encompassing, there may be new kinds of objections raised regarding the degree of intrusiveness they entail. Unlike data from local activities that tend to remain local, and unlike data from standard assessments that tend to be narrow in focus, data from adaptive systems could travel far from the point of data collection, moving from individuals to systems to commercial entities, and they could be used to construct more revealing portraits of individual learners, raising new concerns about confidentiality. Such ostensibly educational data could have particular value to future employers and commercial entities more generally, particularly if blended with personal data gathered in other sectors. Moreover, the risks to individual privacy are unlikely to be evenly distributed throughout the population (Raab and Bennett, 1998) Accessibility of Data The data generated by adaptive educational technologies, as noted earlier, require special processing to make them useful for educational research. Such processing is a distinctly different task from those required to operate the technologies in educational practice, and as such, it requires special efforts and investments of time and resources. Yet it is not clear how such resources will be assembled and put to the task. This is a special case of the more general data archiving and sharing challenge that has so far eluded the educational research community in the case of data sets generated directly by educational researchers. 17 Note that online systems of all kinds confront this same issue (Kobsa, 2007). 20 There are several aspects of the likely provenance of adaptive educational technologies that pose this problem in different ways. For those systems developed by scholars, the challenge of data archiving and sharing is the same as it has always been. Both limited resources and the absence of professional norms make such data sharing less common than might be desired for scientific purposes. For those systems developed by large software publishing companies, the incentives for data management, archiving, and sharing beyond the immediate needs of system and service delivery are absent. Indeed, without clear and short-term gains, commercial sector cost control sensitivities would seem to work against such investments. For those systems created by small developers, the resource constraints can be even more severe and the prospect of gains from data sharing even more remote. Making all of this more complicated from the systems developer perspective is the absence of any organizational capacity within the community of educational researchers to serve as a reasonable point of contact for data set sharing. Indeed, the tens of thousands of educational researchers are spread across a large number of organizational settings of various types, and with varying interests. A manager of an adaptive educational technology would be hard pressed to identify an appropriate avenue for sharing data to benefit the general cause of educational research and development. Proprietary Interests If adaptive educational technologies develop along the lines suggested at least by some, increasingly such systems may move from being laboratory and experimental systems to being large-scale services offered by commercial entities. This migration has already occurred for cognitive tutors for math, SQL-Tutor for database programming, and Mastering Physics for college level physics. Commercial entities will have proprietary interests in the adaptive educational technologies at the core of their business operations, and as a result they may become sensitive to the risks to their intellectual property posed by sharing the data generated by such systems.18 As commercial firms seek to protect trade secrets that confer advantage on their products and services, the risks that accompany the sharing of data with educational researchers may be unacceptable.19 18 For example, the University of Phoenix appears to have recognized the value of the unique set of systems it has developed to manage and deliver its core educational business (Selingo, 2005; Sharkey, 2011). The recent purchase of Carnegie Learning and its adaptive technologies such as the Cognitive Tutor program by the University of Phoenix’s parent organization Apollo Group (Keller, 2011) only confirms the importance of adaptive learning technologies and the data they generate for the future of the university. 19 Note that the value of keeping “trade secrets” may accrue even in the absence of any unique technical approaches as anyone who has enjoyed fast food with “secret sauce” can attest. 21 The question of interests is complex in the case of adaptive educational technologies as a result of the multiple parties involved. For any adaptive system, interests and ownership rights may accrue to the learners, to the professional educators working with the learners and working themselves with the adaptive systems, to the school systems engaging the adaptive educational technology providers, and, of course, to the system providers themselves. Even if any one of these parties found it in their own immediate interest to share data with researchers, it is not clear that they would have the complete and unfettered right to do so. Interoperability As students move through various levels, stages, and institutional and non-institutional (e.g., network) settings for their education, there is no guarantee that the data generated by the adaptive educational technologies to which they are exposed will either follow them or be able to be integrated into a full educational data biography. Indeed, even within a single institution at a single point in time, there is no guarantee that the various systems and tools being employed in the delivery of education will be able to exchange data in meaningful ways. Indeed, without serious efforts to create an infrastructure to support the exchange of data among applications, we are likely to be confronted with data islands, rich segments of data on parts of the student educational experience, nearly impossible to combine into a complete picture upon which to base decisions. The issue of standards for the collection and reporting of educational data is not new. However, the issue of interoperability or common data standards is likely to become more pressing as the number of adaptive educational technologies and the number of providers of such technologies grows over time. Such data can be generated by commercial platforms, experimental systems, learning management systems, postsecondary course platforms, stand-alone learning applications, and online learning opportunities in general. Moreover, the challenge represented by the adaptive technologies is made more intense by the need to recognize the highly contextualized nature of much of the data generated. Student Privacy Interests Because adaptive educational technologies gather information on individual students through online applications they inherently raise three types of privacy concerns. First, are the set of concerns related to the status of students as individual citizens or consumers in a networked world where valuable access to networked resources requires the exchange of personally identifiable information (Nissenbaum, 2010). Second, are the set of concerns related to the status of many students as children whose privacy may require additional protections by virtue of their youth (Pitman and McLaughlin, 2000). Third, are the set of concerns related to the role of students within educational institutions, a role that entails the gathering of particular kinds of information in the educational process (Glenn, 2008). Developers, providers, and adopters of adaptive educational technologies must confront these multiple layers of concerns for the privacy of student users of such technologies. Educational researchers intent on using the data generated by adaptive 22 technologies must be accountable for understanding these various privacy concerns and the procedures for addressing them in the applications that generate data use in their research. Strategies for Moving Forward The issues surrounding the use of data from adaptive educational technologies are complex and systemic. Attempts to address such issues will require efforts beyond those of individual developers or educational researchers. Here I consider some strategies to respond to these issues and create the conditions necessary to engage the community of educational researchers in serious and sustained research initiatives that make good use of the data generated by adaptive systems. Ownership and Control Solutions The issues surrounding the ownership and control of data, particularly data on individual learners are among the most serious confronting efforts to make use of the data generated by adaptive learning technologies in educational research. One approach to these issues has been proposed by Raymond (2008) in the form of a student data backpack. The student data backpack would be an electronic data file controlled and managed by parents and independent of educational institutions. It would be attached to a web-based student data clearinghouse service, and it would be able to feed data about students into systems operated by educational institutions.20 A student data backpack, controlled by parents or presumably older learners themselves, would address at least some of the issues of ownership and informed consent since parents or learners would have to permit the movement of individual data into institutional data systems and/or into educational research datasets. The basic student information file could be enhanced by the release to parents of contextual data of the sort generated by adaptive learning technologies. Such data could then be sought by researchers directly from parents or learners. Alternatively, parents or learners could set sharing permissions for all of their data at the time of transmission into institutional systems, including adaptive learning systems, and such permission could govern the availability of data for researchers. Network Architecture Solutions It is also possible to address issues of data ownership and sharing through solutions that make use of some of the properties of networked and distributed data gathering. Natriello, Pattinsky, Chae, and Cocciolo (2007) piloted a data gathering approach that relied on local instances of an online course platform as the vehicles for data gathering in classrooms in multiple schools. The data along with identifying information on individual students were joined only within the school where data were gathered under 20 A version of the student backpack has recently been introduced by the National Network of Digital Schools Foundation (http://www.backpacksis.com/). 23 local conditions and with permissions secured locally. Data from multiple schools were then transmitted to a central data repository only after identifying information was removed from the data stream. The resulting data archive contained no personally identifiable information, and reconnecting to such information was only possible at the level of the local school. This is only one example of strategies that could be used to collect data at the local level to be used to guide educational practice and then aggregate such data over large numbers of educational institutions for research purposes while insuring anonymity. Similar strategies of selective data assembly could be employed to gather data selectively from any number of applications while preventing the assembly of data sets complete enough to reveal individual student identities. In a system of locally located and fully networked applications permitting the exchange of data for specific purposes, any number of permutations might be constructed to maximize data access for research while protecting the rights of data owners at each level. Repository Solutions The development of repositories for the data generated by adaptive educational technologies represents another approach to improving access to such data for educational researchers. An early model of such a repository is the Pittsburgh Science of Learning Center DataShop (Koedinger, Baker Cunningham, Skogsholm, Leber, and Stamper, 2011). The DataShop focuses on assembling datasets bearing on the interaction between students and educational software such as online courses and intelligent tutoring systems. The DataShop has developed a set of standards for the organization and transmission of such data, and it provides researchers with a set of tools for using the assembled datasets. The PSLC DataShop has gathered 164 datasets from 50 projects encompassing over 25 million transactions between students and software applications, and over 150,000 hours of interaction between students and educational software. The PSLC DataShop might be viewed as the first of a network of such repositories for data from adaptive educational technologies. Individual repositories in the network might focus their energies on particular kinds of data or populations, and they might develop specialized tool sets to aid researchers. With careful coordination the substantial work of archiving the large datasets generated by adaptive educational technologies might be distributed while also allowing researchers to combine data and tools from different repositories for particular inquiries. Research Tools Data mining techniques can be made more accessible to educational researchers through the development of tools designed specifically for the types of datasets generated by adaptive learning technologies. Romero and Ventura (2007) list a set of such tools for particular tasks: 24 Visualization – Syergo/ColAT (Avouris, Komis, Fiotakis, Margaritis, and Voyiatzaki, 2005); GISMO/CourseVis (Mazza and Milani (2005); Listen tool (Mostow, Beck, Cen, Cuneo, Gouvea, and Heiner, 2005) Association – Mining tool (Zaiane and Luo, 2001); MultiStar (Silva and Vieira, 2002); Data Analysis Center (Shen, Yang, and Han, 2002); EPRules (Romero, Ventura, Bra, and Castro, 2003); TADA-ED (Merceron and Yacefm 2005) Text Mining – KAON (Tane, Schmitz, and Stumme, 2004); iPDF_Analyzer (Bari and Bensater, 2005) Classification – TAFPA (Damez, Marsala, Dang, and Bouchon-Meunier, 2005) As Romero and Ventura (2007) suggest, the further development of tools that make data mining techniques more accessible to educators and educational researchers will support additional work with large-scale datasets. Standards Solutions A number of the issues making the use of data from adaptive educational technologies more difficult for educational researchers might be addressed through the creation of standards for the documentation and transmission of data to and from these systems. Efforts to develop such standards for the data associated with adaptive systems might build on existing efforts to develop common standards for education data such as the National Education Data Model (http://nces.ed.gov/forum/datamodel/index.aspx) and the Schools Interoperability Framework (http://www.sifinfo.org/us/index.asp). Regulatory Solutions Because there is a public interest in education and because public funds will inevitably be used to purchase adaptive educational technologies and services, it is possible to imagine one or more regulatory regimes that would encourage firms providing adaptive systems to collaborate with the educational research community. For example, the federal government might consider regulatory approaches akin to those used by the Food and Drug Administration that require providers to present the results of independent research trials prior to being licensed for sale. Or the state governments might require reviews such as those now used for textbook approval before state funds could be used to purchase adaptive educational technologies. The reviews could include criteria requiring independent studies of the efficacy of the systems. Such regulations would create incentives for those developing and providing adaptive educational systems to engage the educational research community. At the very least, regulations could encourage systems developers to use common standards for the exchange of data. Regulatory solutions could also be used to address the multiple types of concerns related to student privacy. The White House recently released a framework for protecting consumer privacy online that includes a Consumer Privacy Bill of Rights (White House, 25 2012). The rights outlined provide a starting point for thinking about the kinds of privacy provisions that might be considered in the handling of student data in adaptive educational applications. These include: Individual Control: Consumers have a right to exercise control over what personal data companies collect from them and how they use it. Transparency: Consumers have a right to easily understandable and accessible information about privacy and security practices. Respect for Context: Consumers have a right to expect that companies will collect, use, and disclose personal data in ways that are consistent with the context in which consumers provide the data. Security: Consumers have a right to secure and responsible handling of personal data. Access and Accuracy: Consumers have a right to access and correct personal data in usable formats, in a manner that is appropriate to the sensitivity of the data and the risk of adverse consequences to consumers if the data is inaccurate. Focused Collection: Consumers have a right to reasonable limits on the personal data that companies collect and retain. Accountability: Consumers have a right to have personal data handled by companies with appropriate measures in place to assure they adhere to the Consumer Privacy Bill of Rights. (White House, 2002, p. 1) The White House has offered these consumer rights as the basis for possible legislation and corporate codes of practice as well as to align U.S. policies with global standards for Fair Information Principles (FIPP). However, broad standards applied to citizens in general are not always viewed as applicable within school settings where there may be both additional limitations (e.g., Zirkel, 2005) as well as special additional protections. The youth of many students means that the regulations designed to protect children must also be considered in thinking about the data gathered by adaptive educational systems. The Children’s Online Privacy Protection Act (COPPA) specifies the kinds of privacy protections accorded to those under the age of thirteen. Under the provisions of COPPA Online providers serving children must have verifiable parental consent for the collection of personal data from children and must post online the following requirements when collecting information on children: The name and contact information of all parties collecting or maintaining personal information on children Information on the kinds of personal information collected from children Information on the uses of the personal information Whether the information collected is disclosed to third parties That the parent has the option to agree to the collection of the information without consenting to disclosing the information to third parties Limits on the information disclosed by the child to only that which is reasonably necessary 26 Parent rights to review the child’s personal information, have it deleted, and refuse to allow any further collection (Federal Trade Commission, 2006). In the fall of 2011 the Federal Trade Commission proposed changes to COPPA for the first time since the regulations were published in 2000 (Kardell, 2011). These changes include expanding the definition of personally identifiable information to include: geolocation information, media, IP addresses and other items found on computers or mobile devices; expansion of the definition of what it means to “collect” to include occasions when an operator encourages a child to provide information; new options for providing parental consent; new requirements for deleting information immediately after the time necessary to achieve the purpose for which it was collected; a requirement that operators ensure that third parties receiving data have procedures in place to protect it. These provisions highlight the kinds of issues and policies likely to govern the collection of data from children through adaptive systems. A final set of concerns and regulations pertain specifically to student data. These are highlighted in the 1974 federal Family Educational Rights and Privacy Act (FERPA) and its subsequent revisions. FERPA provides for rights for students and parents to know what data is being collected, what data is retained and how it is used, limits on alternative uses, the right to correct or amend records of personal information, and specifies obligations to guard against misuse. The various provisions of FERPA offer useful guidelines on individual student data in adaptive systems and access to third party organizations conducting studies on behalf of educational agencies or institutions for a variety of educational purposes. The regulations also offer guidance on practices such as de-identification and anonymization of data, and on the obligations of data stewardship. Recent amendments to FERPA regulations contemplate the use of student data in statewide longitudinal data systems and seek to balance privacy rights against the benefits of research (National Center for Education Statistics, 2010). The evolving regulatory regimes pertaining to consumers, children, and students provide some well-developed perspectives to guide thinking about data on individual students gathered through adaptive educational technologies. Of course, such technologies present some new challenges of their own for efforts to protect student privacy. Professional Solutions As the use of adaptive educational technologies grows, they may become a target for professional standards in much the same way that such standards have been developed for testing technologies (AERA, APA, & NCME, 1999). Such standards could create benchmarks for the research base to support claims relating to adaptive technologies, and these benchmarks could be used to foster collaboration between systems developers and educational researchers. Training Solutions 27 Preparing educational researchers to deal with the very large and complex data sets generated by adaptive learning technologies will require focused and specialized training and practice conducting analyses. Fortunately, there is a model for providing such training on a national basis in the work done by the National Center for Education Statistics to prepare and encourage educational researchers to use the large datasets created by government sponsored national surveys. The NCES has sponsored various institutes over the years both as stand-alone activities and in conjunction with other professional associations. It has also provided support for scholars to participate in such training opportunities, and it has enhanced the documentation for and online availability of data sets from large-scale data gathering efforts. Another approach would be to support the development of a network of institutionally based programs that could offer specialized training. The Pittsburgh Science of Learning Center offers a program that could serve as a prototype. The Science of Learning Center Summer School has a track devoted to educational data mining of data from adaptive learning technologies. Those attending the summer school, including researchers, graduate students, and industry practitioners, receive training in the use of major educational data mining methods. Participants in the summer program work on a final project that has led to several peer-reviewed publications. Training experiences like this one could be offered at university and commercial settings where adaptive educational technologies are being developed. Conclusion I have considered the opportunities for educational research and educational researchers connected with adaptive educational technologies. I have reviewed some of the research drawing on data from adaptive technologies that has been conducted over the past decade or so, largely by scholars outside the mainstream of educational research. I have discussed the barriers to advancing educational research and educational improvement through adaptive technologies, and suggested some ways to create conditions more favorable for engaging educational researchers in this work. I want to close by posing three possible paths forward, three scenarios that highlight the importance of the emergence of adaptive educational technologies for the community of educational researchers. Each scenario offers a plausible future, and each may be seen as desirable from some perspective. However, each path represents a distinctly different set of opportunities with associated costs and benefits. Separate Camps Scenario As the use of adaptive educational technologies continues to expand at all levels of the educational system, educational researchers remain relatively uninvolved in their design and development. As a result, these technologies draw almost exclusively on insights generated by the mining of the increasingly large-scale datasets generated by expanded use. They make little use of the theories and perspectives that have animated and 28 dominated scholarship in education. The result is robust practice-driven development and improvement in systems and approaches. As adaptive technologies play an increasingly larger and more important role in educational practice, the educational research community becomes more separate and detached from the community of educational practitioners who increasingly draw on the insights generated by their ever-expanding adaptive systems. Educational practitioners seize on the data provided by adaptive systems to justify and legitimate a full-range of practices. In this world of system-generated insight, the educational research community remains separate and apart from the educational practice community. Educational scholarship turns to largely philosophical and theoretical activities, and such activities come to dominate the preparation of future generations of educational scholars. Educational research continues to have empirical components, but these components are seldom as large and robust as those inherent in adaptive technology systems. Although some lament the detachment of educational research from educational practice and long for a past golden age where research and practice were more closely connected, others welcome the sense of intellectual freedom that accompanies the greater distance from the realities of immediate practices and look forward to a more placid position in the scholarly world. Bridging Organization Scenario As the use of adaptive educational technologies continues to expand at all levels of the educational system, individual educational researchers become increasingly involved in their design and development. Educational researchers first become connected to adaptive technologies in several ways. Some researchers find themselves acting as consultants to development firms and publishers who seek to draw on the theoretical traditions of the field. Others are enlisted for their skills in data analysis, particularly those connected with the hierarchical and nested data sets inherent in educational systems. Still others become familiar with adaptive educational technologies when they develop research systems as empirical test beds for theoretically driven lines of research. The initial wave of involvement of individual isolated educational researchers in adaptive systems leads to efforts to create mediating and facilitating organizations and utilities to expand and enhance the participation of the educational research community in adaptive educational technology development. The first research training programs in adaptive technologies are started with support from external foundations and software publishers who realize the growing need for a skilled cadre of professionals who can bridge the worlds of learning and educational theory and adaptive systems. The creation of a network of adaptive system data archives provides additional encouragement to a new generation of researchers who blend the skills of educational research and systems development. 29 Work on adaptive educational technologies comes to play an increasingly large role in the world of educational research. Early efforts to establish a special interest group on adaptive systems in the American Educational Research Association quickly transition to a successful effort to establish a new division, and this division comes to rival the longstanding mainstay divisions in size and influence within the organization. Efforts to split off and form an entirely new association are forestalled by politically astute association leadership.21 The results of this uneasy alliance are deemed particularly valuable when evidence begins to mount regarding the unintended and almost unrecognizable negative impact of adaptive technologies on certain segments of students for certain learning tasks. Embedded Research Scenario As the use of adaptive educational technologies continues to expand at all levels of the educational system, the systems advance to the point where they are able to gather data, process it on-the-fly, and apply the results of the analysis all within the application itself. The subsequent cycles of this process result in further refinements and adjustments to these embedded “research” processes. Large scale adaptive systems are the first to incorporate these capacities, but over time even smaller scale systems adopt such design elements as the techniques become more standardized and well known. In this world, the work of educational researchers shifts dramatically. The role of educational researchers is no longer to engage personally in all phases of the research process (design, data collection, analysis, interpretation, publication, application). Instead, educational researchers are primarily engaged in the design of research components to be embedded within adaptive educational technologies. Although basic research on human learning remains (and remains largely outside of educational research), educational research outside of the development and refinement of adaptive learning technologies largely fades away. The transformation of educational research associated with the rise of embedded research capacities within adaptive educational technologies results in a major reconfiguration of the profession. The skill requirements for educational research increase in ways heretofore unknown to the profession even as the numbers of individuals engaged in educational research fall by 90%.22 With the development of communities of researchers and related organizations in the areas of Intelligent Tutoring Systems (ITS), Artificial Intelligence in Education (AIED), Educational Data Mining (EDM), and Learning Analytics and Knowledge (LAK), none of which have strong ties to AERA, it may be past the point where AERA can become a dominant organization in the professional activities of such scholars. 22 It is difficult to grasp fully how this kind of transformation might be plausible, but recent reports of a similar transformation in the legal profession may be instructive (Markoff, 2011). 21 30 While some decry the decline in the numbers of educational researchers or the shift in the location of research positions from the academy to systems providers, others see the transition precipitated by the rise of research embedded in adaptive educational technologies as a long-overdue housecleaning of the profession accompanied by an increase in skill requirements and a new closer affiliation with on-the-ground delivery of educational services. These scenarios surely over dramatize the options before us. There are already too many scholars who identify as educational researchers engaged in developing and studying adaptive technologies to allow us to think that the entire educational research community will be shut out of future development. By the same token, the independent and growing organizations devoted to such technologies do not leave us optimistic about their full incorporation into the main bodies of the American Educational Research Association. In some respects, though, the embedding of research components into adaptive educational systems presents the greatest prospect for a change in the scholarly landscape. Such a development would alter the conditions under which scholarship in education is conducted. The prospect of such profound change makes consideration of the impact of developing adaptive and interactive educational technologies even more important for scholars in the field. References Abelson, R. (1968). Theories of Cognitive Consistency: A Sourcebook. Chicago: Rand McNally. Aleven, V., Stahl, E., Schworm, S., Fischer, F., and Wallace, R. (2003). Help seeking and help design in interactive learning environments. Review of Educational Research 73(2), 277-320. Aleven, V., McLaren, B., Roll, I., and Koedinger, K. (2004). Toward tutoring help seeking: applying cognitive modeling to meta-cognitive skills. In Proceedings of the 7th Internation Conference on Intelligent Tutoring Systems, 227-239. Aleven, V., McLaren, B., Roll, I., and Koedinger, K. (2006). Toward metacognitive tutoring: A model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16, 101-130 American Educational Research Association, American Psychological Association, and National Council for Measurement in Education. (1999). Standards for Educational and Psychological Testing. Washington, DC. 31 Amershi, S., and Conati, C. (2009). Combining unsupervised and supervised classification to build user models for exploratory learning environments. Journal of Educational Data Mining, 1(1), 18–71. Antunes, C. (2008). Acquiring background knowledge for intelligent tutoring systems. In Proceedings of the 1st International Confernence on Educational Data Mining, Montreal, QC, Canada, 18–27. Arroyo, I., Murray, T., Woolf, B., and Beal, C. (2004). Inferring unobservable learning variables from students’ help seeking behavior. Lecture Notes on Computer Science, 3220, 782-784. Arroyo, I. and Woolf, B. (2005). Inferring learning and attitudes from a Bayesian network of log file data. In C.K, Looi, G. McCalla, B. Bredeweg, and J. Breuker (eds.), Proceedings of the Twelfth International Conference on Artificial Intelligence in Education, pp. 33-40. Amsterdam: IOS Press. Asuma, R. (1997). A survey of augmented reality. Presence: Teleoperators and virtual environments, 6, 4, 355-385. Avouris, N., Komis, V., Fiotakis, G., Margaritis, M., and Voyiatzaki, E. (2005). Why logging of fingertip actions is not enough for analysis of learning activities. In the Workshop on Usage Analysis in Learning Systems at the 12th International Conference on Artificial Intelligence in Education. Amsterdam. Baker, R. (2010). Data mining for education. In P. Peterson, E. Baker, and B. McCaw, and, (eds.). International Encyclopedia of Education (3rd edition). vol. 7, pp. 112-118. Oxford, UK: Elsevier. Baker, R. (2007). Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. San Jose, CA, 1059–1068. Baker, R., Corbett, A., and Aleven, V. (2008). Improving contextual models of guessing and slipping with a truncated training set. In Proceedings of the 1st International Conference on Educational Data Mining, Montreal, QC, Canada, 67–76. Baker, R., Corbett, A., and Koedinger, K. (2004). Detecting student misuse of intelligent tutoring systems. Lecture Notes in Computer Science, vol. 3220/2004, 54-76. Baker, R., Corbett, A., Roll, I., and Koedinger, K.R. (2008). Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction, 18, 3, 287-314. Baker, R. & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining 1, 1, 3-17. 32 Baker, R., Pardos, Z., Gowdam, S., Nooraei, B., and Heffernanm, N. (2011) Ensembling predictions of student knowledge within intelligent tutoring systems. User Modeling, Adaptation and Personalization, Lecture Notes in Computer Science, 6787, 13-24. Baker, R., Walonoski J., Heffernan, N., Roll, I., Corbett, A., and Koedinger, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19 (2), 185-224. Ba-Omar, H., Petrounias, I., and Anwar, F. (2007). A framework for using web usage mining for personalise e-learning. In Proceedings of the International Conference on Advanced Learning Technologies. Niigata, Japan, 937–938. Barab, S., Gresalfi, M., and Ingram-Goble, A. (2010). Transformational play: Using games to position person, content, and context. Educational Researcher, 39 (7), 525-536. Bari, M. and Benzater, B. (2005). Retrieving data from pdf interactive multimedia productions. In Proceedings of the Fifth International Conference On Human System Learning, Marrakech – Morocco, 321-330. Barnes, T., Bitzer, D. & Vouk, M. (2005). Experimental analysis of the q-matrix method in knowledge discovery. Foundations of Intelligent Systems. Lecture Notes in Computer Science, 3488, 603-611. Barnes, T. and Stamper, J. (2008). Toward automatic hint generation for logic proof tutoring using historical student data,” in Proceedings of the 9th International Conference on Intelligent Tutoring Systems, Montreal, QC, Canada, 373–382. Beal, C. and Cohen, P. (2008). Temporal data mining for educational applications. In Proceedings of the 10th Pacific Rim International Conference on Artificial Intelligence: Trends in Artificial Intelligence. Hanoi, Vietnam, 66–77. Beal, C., Qu, L. and Lee, H. (2008). Mathematics motivation and achievement as predictors of high school students’ guessing and help-seeking with instructional software. Journal of Computer Assisted Learning, 24, 507-514. Beck, J. and Mostow, J. (2008). How who should practice: Using learning decomposition to evaluate the efficacy of different types of practice for different types of students. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems, 353-362. Beck, J. and Woolf, B. (2000). High-level student modeling with machine learning. In Proceedings of the 5th International Conference on Intelligent Tutoring Systems, Alagoas, Brazil, pp. 584–593. 33 Bell, D. (1976). The coming of post-industrial society: A venture in social forecasting. New York: Basic Books. Bellaachia and E. Vommina, (2006). MINEL: A framework for mining e-learning logs in Proceedings of the Fifth IASTED International Conference on Web-Based Education, Mexico, 259–263. Ben-naim, D., Marcus, N., and Bain, M. (2008). Visualization and analysis of student interaction in an adaptive exploratory learning environment, in Proceedings of the International Workshop in Intelligent Support of Exploratory Environment. European Confernce on Technology Enhanced Learning, Maastricht, The Netherlands, 1–10. Boekaerts, M., Pintrich, P., and Zeidner, M. (2000). Handbook of self-regulation. San Diego: Academic Press. Brasch, W. (2005). Fool’s gold in the nation’s data-mining programs Social Science Computer Review, 23, 4, 401-428. Brusilovsky, P. (1996). Methods and techniques of adaptive hypermedia. User Modeling and User-Adapted Interaction, 6, 87-129. Brusilovsky, P. (1998). Methods and techniques of adaptive hypermedia. Pp. 1-43 in P. Brusilovsky, A. Kobsa, and J. Vassileva (eds.). Adaptive Hypertext and Hypermedia. Dordrech: Kluwer. Brusilovsky, P. (2001). Adaptive hypermedia. User Modeling and User-Adapted Interaction, 11, 87-110. Brusilovsky, P. (2004). Adaptive navigation support: From adaptive hypermedia to the adaptive web and beyond. PsychNology Journal, 2(1), 7-23. Brusilovsky, P. and Maybury, M.T. (2002). From adaptive hypermedia to the adaptive web. Communications of the ACM, 45(5), 30-33. Butler, D. and Winne, P. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65, 245-281. Carr, B. and Goldstein, I. (1977). Overlays: A theory of modeling for computer-aided instruction. AI Lab Memo 406 (Logo Memo 40). Massachusetts Institution of Technology, Cambridge, MA. Castro, F., Vellido, A., Nebot, A., and Mugica, F. (2007). Applying data mining techniques to e-learning problems. Studies in Computational Intelligence, 62, 183-221. Cetintas, A., Si, L., Xin, Y. and Hord, C. (2009). Predicting correctness of problem solving from low-level log data in intelligent tutoring systems,” in Proceedings of the 2nd International Conference on Educational Data Mining, Cordoba, Spain, pp. 230– 238. 34 Chang, K., Beck, J., Mostow, J., and Corbett, A. (2006). A bayes net toolkit for student modeling in intelligent tutoring systems,” In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, 104–113. Chi, M. (2009). Active-constructive-interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1, 73-105. Chi, M., VanLehn, K., Litman, D., and Jordan, P. (2010). Inducing effective pedagogical strategies using learning context features. User Modeling, Adaptation, and Personalization, Lecture Notes in Computer Science, 6075, 147-158. Computering Research Association. (2005). Cybrainfrastructure for Education and Learning for the Future: A Vision and Research Agenda. Washington, DC: Computing Research Association. Corbett, A.T., and Anderson, J.R (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253-278. Corcoran T., Mosher, F. & Rogat, A. (2009). Learning Progressions in Science: An Evidence-Based Approach to Reform. New York: Consortium for Policy Research in Education, Teachers College, Columbia University. Crespo, R., Pardo, A., Perez, J., and Kloos, C. (2005). An algorithm for peer review matching using student profiles based on fuzzy classification and genetic algorithms. In Proceedings of the International Conference on Innovations in Applied Artificial Intelligence, Bari, Italy, 685–694. Damez, M., Marsala, C., Dang, T., and Bouchon-Meunier, B. (2005). Fuzzy decision tree for user modeling from human-computer interactions. In Proceedings of the 5th International Conference on Human System Learning, Marrakech, Morocco, 287-302. Daro, P., Mosher, F., & Corcoran, T. (2011). Learning Trajectories in Mathematics: A Foundation for Standards, Curriculum, Assessment and Instruction. New York: Consortium for Policy Research in Education, Teachers College, Columbia University. De Vicente, A. and Pain, H. (2002). Informing the detection of the students’ motivational state: an empirical study. Lecture Notes in Computer Science, 2362, 933-943. Desmarais, M., Gagnon, M., and Meshkinfram, P. (2006). Bayesian student models based on item to item knowledge structures,” in Proceedings of the First European Conference on Technology Enhanced Learning, Crete, Greece, pp. 1–10. 35 Drigas, A., and Vrettaros, J. (2004) An intelligent tool for building e-learning contentmaterial using natural language in digital libraries. WSEAS Transactions on Information Science and Applications, 5(1), 11978-1205. Federal Communications Commission (2010). Connecting America: The National Broadband Plan. Washington, DC: Federal Communications Commission, March. Feng, M. and Beck, J. (2009). Back to the future: A non-automated method of constructing transfer models. In Proceedings of the 2nd International Conference on Educational Data Mining, Cordoba, Spain, 240–248. Federal Trade Commission (2006). How To Comply with the Children’s Online Privacy Protection Rule. Washington, DC: Federal Trade Commission Bureau of Consumer Protection. http://business.ftc.gov/documents/bus45-how-comply-childrens-onlineprivacy-protection-rule Freyberger, J., Heffernan, N., and Ruiz, C. (2004). Using association rules to guide a search for best fitting transfer models of student learning. In Workshop on Analyzing Student-Tutor Interaction Logs to Improve Educational Outcomes at ITS Conference, Maceio, Brazil. Garcia, E., Romero, C., Ventura, S., and Castro, C. (2009). An architecture for making recommendations to courseware authors using association rule mining and collaborative filtering. User Modeling and User-Adapted Interaction, 19, 99–132. Glenn, D. (2008). Huge databases offer a research goldmine and privacy worries. Chronicle of Higher Education, 54(35), A10. Gong, Y., Rai, D., Beck, J., and Heffernan, N. (2009). Does self-discipline impact students’ knowledge and learning? In Proceedings of the 2nd International Conference on Educational Data Mining, Cordoba, Spain, 61–70. Greene, J. A., & Azevedo, R. (2009). A macro-level analysis of SRL processes and their relations to the acquisition of sophisticated mental models. Contemporary Educational Psychology, 34, 18–29. Gresalfi, M., Barab, S., Siyahhan, S., and Christensen, T. (2009). Virtual worlds, conceptual understanding, and me: Designing for consequential engagement. On the Horizon, 17 (1), 21-34. Hamalainen, W., Suhonen, J., Sutinen, E., and Toivonen, H. (2004). Data mining in personalizing distance education courses. In Proceedings of the World Conference on Open Learning and Distance Education, Hong Kong, 1–11. 36 Hamalainen, W. and Vinni, M. (2006). Comparison of machine learning methods for intelligent tutoring systems, in Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Taiwan, pp. 525–534. Heraud, J., France, L., and Mille, A. (2004). Pixed: An ITS that guides students with the help of learners’ interaction log. In Proceedings of the International Conference on Intelligent Tutoring Systems, Maceio, Brazil, 57–64. Höllerer, T. & Feiner, S. (2004). Mobile augmented reality. In H. Karimi and A. Hammad (eds.). Telegeoformatics: Location-based computing and services. New York: Taylor and Francis, pp. 221-258. Hurley, T. and Weibelzahl, S. (2007). Using MotSaRT to support on-line teachers in student motivation. In Proceedings of the European Conference on Technology Enhanced Learning. Crete, Creece, 101–111. Hwang, G. (1999). A knowledge-based system as an intelligent learning advisor on computer networks. Journal of Systems, Man, and Cybernetics, 2, 153-158. Hwang, G. (2003). A test-sheet-generating algorithm for multiple assessment requirements. IEEE Transactions on Education, 43(3), 329-337. Hwang, W., Chang, C., and Chen, G. (2004). The relationship of learning traits, motivation and performance-learning response dynamics,” Computers and Education, 42, 267–287. Hwang, G., Huang, T., and Tseng, C. (2004). A group-decision approach for evaluating educational web sites. Computers and Education, 42, 64-86. Johnson, L., Smith, R., Willis, H., Levine, A., & Haywood, K. (2011). The 2011 Horizon Report. Austin, TX: The New Media Consortium. Jong, B., Chan, T., and Wu, Y. (2007). Learning log explorer in e-learning diagnosis,” IEEE Transactions on Education, 50(3), 216–228. Jonsson, A., Hasmik, M., Johns, J., Mehranian, H., Arroyo, I., Woolf, B., Barto, A., Fisher, D., and Mahadevan, S. (2005). Evaluating the feasibility of learning student models from data,” in Proceedings of the AAAI Workshop on Educational Data Mining, Pittsburgh, PA, 1–6. Karampiperis, P. and Sampson, D. (2005). Adaptive learning resources sequencing in educational hypermedia systems. Educational Technology and Society, 8(4), 128–147. Kardell, N. (2011). FTC will propose broader children’s online privacy safeguards. National Law Review, http://www.natlawreview.com/article/ftc-will-propose-broaderchildren-s-online-privacy-safeguards 37 Keller, J. (2011, August 2). Apollo to buy adaptive learning company for $75 million. The Chronicle of Higher Education. Kelly, D. and Tangney, B. (2005). First aid for you: Getting to know your learning style using machine learning. In Proceedings of the IEEE International Conference on Advanced Learning Technologies. Washington, DC, 1-3. Koedinger, K. and Corbett, A. (2006). Cognitive tutors: Technology bringing learning sciences to the classroom. In R. Sawyer (ed.). The Cambridge Handbook of the Learning Sciences. New York: Cambridge University Press. Koedinger, K., Baker, R., Cunningham, K., Skogsholm, A., Leber, B., and Stamper, J. (2011). A data repository for the EDM community: The PSLC DataShop. In Romero, C., Ventura, S., Pechenizky, M., and Baker, R. (eds.). Handbook of Educational Data Mining. Boca Raton, FL: CRC Press, pp. 43-55. Knutov, E., De Bra, & Pechenizkly, M. (2009). AH 12 years later: A comprehensive survey of adaptive hypermedia methods and techniques. New Review of Hypermedia and Multimedia, 15(1), 5-38. Kobsa, A. (2007). Privacy enhanced personalization. Communications of the ACM, 50(8), 24-33. Koedinger, K., Baker, R., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2011). A data repository for the EDM Community: The PSLC Datashop. In C. Romero, S. Ventura, M. Pschenisky, & R. Baker (Eds.), Handbook of educational data mining (pp. 43-55). Baco Raton: CRC Press. Kotsiantis, S., Pierrakeas, C., and Pintelas, P. (2004). Predicting students’ performance in distance learning using machine learning techniques. Applied Artificial Intelligence, 18(5), 411-426. Kristofic, A. (2005). Recommender system for adaptive hypermedia applications. In Proceedings of the 1st Student Research Conference on Informatics and Information Technologies, Bratislava, Slovakia, 2005, pp. 229–234.i Kumar, K. (1995). From post-industrial to post-modern society: New theories of the contemporary world. Oxford, UK: Blackwell. Lajoie, S. P. & Azevedo, R. (2006). Teaching and learning in technology-rich environments. In P. Alexander & P. Winne (Eds.), Handbook of educational psychology (2nd ed.) (pp. 803-821). Mahwah, NJ: Erlbaum. Landow, G. (2006). Hypertext 3.0: Critical theory and new media in an era of globalization. Baltimore: The Johns Hopkins University Press. 38 Lee, J., & Park, O. (2007). Adaptive instructional systems. In J.M. Spector, M.D. Merrill, J.V. Merrienboer, & M.P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 469-484). Mahwah, NJ: Lawrence Erlbaum Associates. Lovette, M., Meyer, O., and Thille, C. (2008). The open learning initiative: Measuring the effectiveness of the OLI statistics course in accelerating student learning. Journal of Interactive Media in Education, 14 Lu, J. (2004). A personalized e-learning material recommender system. In Proceedings of the 2nd International Conference on Information Technology for Application., Harbin, China, 374–379. Lu, F., Li, X., Liu, Q., Yang, Z., Tan, G., and He, T. (2007). Research on personalized elearning system using fuzzy set based clustering algorithm. In Proceedings of the Seventh International Conference on Computational Science. Beijing, China, 587–590. Maimon, O. and Rokach, L. (2005). Chapter 1 – Introduction to knowledge discovery in databases in O. Maimon and L. Rokach (eds.). The Data Mining and Knowledge Discovery Handbook (pp. 1-14). New York: Springer. Madhyastha, T. and Tanimoto, S. (2009). Student consistency and implications for feedback in online assessment systems. In Barnes, T., Desmarais, M., Romero, C., & Ventura, S. (Eds.) Educational Data Mining 2009: 2nd International Conference on Educational Data Mining, Proceedings. Cordoba, Spain. July 1-3, 81-90. Markoff, J. (2011, March 5). Armies of expensive lawyers replaced by cheaper software. New York Times, A1. Martin, J. and VanLehn, K. (1995). Student assessment using Bayesian nets. International Journal of Human-Computer Studies, 42, 575-591. Martinez, R., Kay, J, and Yacef, K. (2011). Visualizations for longitudinal participation, contribution and progress of a collaborative task at the tabletop. Proceedings of the conference on Computer Supported Collaborative Learning, vol. 1, 25-32. Mazza, R., and Milani, C. (2005). Exploring usage analysis in learning systems: Gaining insights from visualizations. In the Workshop on Usage Analysis in Learning Systems at the 12th International Conference on Artificial Intelligence in Education, Amsterdam. Mclaren, B., Koedinger, K., Schneider, M., Harrer, A., and Lollen, L. (2004). Bootstrapping novice data: Semi-automated tutor authoring using student log files. In Prcoeedings of the Workshop on Analyzing Student-Tutor Interaction Logs to Improve Education Outcomes, Seventh International Conference on Intelligent Tutoring Systems Alagoas, Brazil, 1–10. 39 Mcquiggan, S., Mott, B., and Lester, J. (2008). Modeling self-efficacy in intelligent tutoring systems: An inductive approach. User Modeling and User-Adapted Interaction, 18, 81-123. Merceron, A. and Yacef, K. (2004). Mining student data captured from a web-based tutoring tool: Initial exploration and results. Journal of Interactive Learning Research, 15, 4, 319-346. Mislevy, R., Behren, J., Bennett, R., Demark, S., Fresso, D., Levy, R., Robinson, D., Rutstein, D., Shute, V., Stanley, K., and Winters, F. (2010). On the roles of external knowledge representations in assessment design. Journal of Technology, Learning and Assessment, 8(2), Retrieved from http://www.jtla.org Mizue, K. and Toshio, O. (2001). N3: Neural network navigation support-knowledgenavigation in hyperspace: The sub-symbolic approach. Journal of Educational Multimedia and Hypermedia, 10(1), 85-103. Morales, C., Perez, A., Soto, S., Martinez, C., and Gomez, A. (2006). Using sequential pattern mining for links recommendations in adaptive hypermedia educational systems. Current Developments in Technology-Assisted Education. Badajoz, Spain: FORMATEX, 1016-1020. Mostow, J., Beck, J., Cen, H., Cuneo, A., Gouvea, E., and Heiner, C. (2005). An educational data mining tool to browse tutor-student interactions: Time will tell! In Proceedings of the Workshop on Educational Data Mining, 15–22. Muehlenbrock, M. (2005). Automatic action analysis in an interactive learning environment. Usage Analysis in Learning Systems. Twelfth International Conference on Artificial Intelligence in Education. Amsterdam, The Netherlands, pp. 73-80. Mullier, D. (2003). A tutorial supervisor for automatic assessment in educational systems. International Journal of on e-Learning, 2(1), 37-49. National Center for Education Statistics (2010). Basic Concepts and Definitions for Privacy and Confidentiality in Student Education Records. Washington, DC: National Center for Education Statistics, Brief 1, NCDS 2011-601. http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2011601 Natriello, G. (2010). Adaptive Learning Opportunities. New York: Teachers College. Natriello, G., Pattinsky, M., Chae, H. and Cocciolo. A. (2007). E-learning systems as research platforms: Results from the Networked Education Database (NED) Project. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, April. Nissenbaum, H. (2010). Privacy in context: Technology, policy, and the integrity of social life. Stanford, CA: Stanford University Press. 40 NSF Task Force on Cyberlearning. (2008). Fostering Learning in a Networked World: The Cyberlearning Opportunity and Challenge. Washington, DC: National Science Foundation. Pardos, Z., Heffernan, N., Anderson, B., and Heffernan, C. (2007), The effect of model granularity on student performance prediction using bayesian networks,” in Proceedings of the 11th International Conference on User Modeling, Corfu, Greece, pp. 435– 439. Paris, S. and Paris, A. (2001). Classroom applications of research on self-regularted learning. Educational Psychologist, 36(2), 89-101. Pavlik, P., Cen, H., and Koedinger, K. (2009). Learning factors transfer analysis: Using learning curve analysis to automatically generate domain models. In Proceedings of the 2nd International Conference on Educational Data Mining, Cordoba, Spain, 121-130. Pavlik Jr., P., Cen, H., & Koedinger, K. (2009). Performance factors analysis -- A new alternative to knowledge tracing. In V. Dimitrova & R. Mizoguchi (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education. Brighton, England. Pechenizkiy, M., Trcka, N., Vasilyeva, E., van der Aalst, W., and De Bra, P. (2009). Process mining online assessment data. In Proceedings of the 2nd International Conference on Educational Data Mining, Cordoba, Spain, 279-288. Pierce, N., Conlan, O., and Wade, V. (2008). Adaptive educational games: Providing non-invasive personalized learning experiences. Second IEEE International Conference on Digital Games and Intelligent Toy Enhanced Learning. Banff, BC. Pitman, J. and McLaughlin, B. (2000). Making cyberspace safe for children. Educational Leadership, 67(6), 67-71. Raab, C. and Bennett, C. (1998). The distribution of privacy risks: Who needs protection. The Information Society, 14, 263-274. Rai, D., Gong, Y., and Beck, J. (2009). Using Dirichlet priors to improve model parameter plausibility,” in Proceedings of the 2nd International Conference on Educatiional Data Mining, Cordoba, Spain, 141–150. Raymond, M. (2008). The student data backpack. In Kanstoroom, M., and Osberg, E., (eds). A Byte at the Apple: Rethinking Student Data for the Post-NCLB Era. Washington, DC: Thomas B. Fordham Institute, pp. 142-158. Rebolledo-Mendex, G., du Boulay, B., and Luckin, R. (2006). Motivating the learner: an empirical evaluation. ITS2006, Eighth International Conference on Intelligent Tutoring Systems. Taiwan. Lecture Notes in Computer Science, 4053, 545-554. 41 Ritter, S., Harris, T., Nixon, T., Dickison, D., Murray, R., and Towle, B. (2009). Reducing the knowledge tracing space. In Proceedings of the 2nd International Conference on Educational Data Mining, Cordoba, Spain, 151–160. Robinet, V., Bisson, G., Gordon, M., and Lemaire, B. (2007). Searching for student intermediate mental steps,” in Proceedings of the Workshop on Data Mining for User Modeling at the 11sth International Conference on User Modeling, Corfu, Greece, 101– 105. Roll, I., Aleven, V., McLaren, B., & Koedinger, K. (2011). Improving students’ helpseeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction, 21, 267-280. Romero, C. and Ventura, S. (2007). Educational data mining: A survey from 1995 to 2005. Expert Systems with Applications, 33, 135-146. Romero, C., Ventura, S., and De Bra, P. (2004). Knowledge discovery with genetic programming for providing feedback to courseware. User Modeling and User-Adapted Interaction, 14(5), 425-464. Romero C., Ventura, S., Bra, P., and Castro, C. (2003). Discovering prediction rules in aha! Courses. In Proceedings of the User Modeling Conference, 25-34. Romero, C., Ventura, S., Pechenisky, M., & Baker, R. (2011). Handbook of educational data mining. Baco Raton: CRC Press. Romero, C., Ventura, S., Zafra, A., and De Bra, P. (2009). Applying web usage mining for personalizing hyperlinks in web-based adaptive education systems,” Computers and Education, 53(3), 828–840. Rus, V., Lintean, M., and Azevedo, R. (2009). Automatic detection of student mental models during prior knowledge activation in MetaTutor. (2009). In Proceedings of the 2nd International Conference on Educational Data Mining, Cordoba, Spain, 161–170. Schunk, D. and Zimmerman, B. (1994). Self-regulation of learning and performance. Hillsdale, NJ: Erlbaum. Schofield, J. (1995). Computers and classroom culture. New York: Cambridge. Selingo, J. (2005, June 17). University of Phoenix owes rapid growth to use of technology. Chronicle of Higher Education Shang, Y., Shi, H., and Chen, S. (2001). An intelligent distributed environment for active learning. ACM Journal of Educational Resources in Computing, 1(2), 308-315. 42 Sharkey, M. (2011). Academic analytics at the University of Phoenix. Proceedings of the First International Conference on Learning Analytics and Knowledge, 122-126. Sheard, J., Ceddia, J., and Hurst, G. (2003). Inferring student learning behavior from website interactions: A usuage analysis. Education and Information Technologies, 8(3), 245-266. Shen, R., Yang, F., and Han, P. (2002). Data analysis center based on e-learning platform. In Proceedings of the 5th International Workshop on the Internet Challenge: Technology and Applications, 19-28. Shute, V.J. & Zapata-Rivera, D. (2007). Adaptive technologies. In J.M. Spector, M.D. Merrill, J.V. Merrienboer, & M.P. Driscoll (Eds.), Handbook of research on educational communications and technology (3rd ed., pp. 279-294). Mahwah, NJ: Lawrence Erlbaum Associates. Silva, D. and Vieira, M. (2002). Using data warehouse and data mining resources for ongoing assessment in distance learning. In IEEE International Conference on Advanced Learning Technologies, Kazan, Russia, 40-45. Simko, M. and Bielikova, M. (2009). Automatic concept relationships discovery for an adaptive e-course. In Proceeding of the 2nd International Conference on Educational Data Mining, Cordoba, Spain, 171–178. Stamper, J., and Barnes, T. (2009). Unsupervised MDP value selection for automating ITS capabilities. In Proceedings of the International Conference on Educational Data Mining. Cordoba, Spain, 180–188. Stanford Aptitude Seminar. (2002). Remaking the concept of aptitude: Extending the legacy of Richard E. Snow. Mahway, NJ: Lawrence Erlbaum Associates. Stephens, C. and Sukumar, R. (2006). An introduction to data mining. In R. Grover and M. Vriens. (eds.). Handbook of Marketing Research, (pp. 455-485). Thousand Oaks CA: Sage Publications. Superby, J., Vandamme, J., and Meskens, N. (2006). Determination of factors influencing the achievement of first-year university students using data mining methods. In Proceedings of the Workshop on Educational Data Mining at the 8th International Conference on Intelligent Tutoring Systems, 37-44. Tane, J., Schmitz, C., and Stumme, G. (2004). Semantic resource management for the web: An e-learning application. In Proceedings of the 13th International World Wide Web Conference, New York, 1-10. Tang, C., Yin, H., Li, T., Lau, R., Li, Q., and Kilis, D. (2000). Personalized courseware construction based on web data mining. In Proceedings of the first international conference on web information systems engineering. Washington, DC., 204-211. 43 Tang, C., Lau, R., Li, Q., Yin, H., Li, T., and D. Kilis, D. (2000). Personalized courseware construction based on web data mining,” in Proceedings of the 1st International Conference on Web Information Systems Engineering, Hong Kong, China, 204–211. Tang, T. and McCalla, G. (2005). Smart recommendation for an evolving e-learning system. International Journal on E-Learning, 4, 1, 105-129. Tsai, C., Tseng, S., and Lin, C. (2001). A two-phase fuzzy mining and learning algorithm for adaptive learning environment. In Proceedings of the International Conference on Computer Science, San Francisco, 429-438. Turkle, S. (2007). Life on the screen: Identity in the age of the internet. New York: Simon and Schuster. Ueno M. (2003). LMS with irregular learning processes detection system. In World Conference on e-Learning in Corporate, Government, Health, and Higher Education, 2486-2493. Urbancic, T., Skrjanc, M., and Flach, P. (2002). Web-based analysis of data ming and decision support education. AI Communications, 15, 199-204. VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227-265. Vee, M., Meyer, B., and Mannock, K. (2006). Understanding novice errors and error paths in Object-oriented programming through log analysis. In Proceedings of the Workshop on Educational Data Mining, Taiwan, 2006, 13–20.. Vellido, A., Castro, F., Nebot, A., and Mugica, F. (2006). Characterization of atypical virtual campus usage behavior through robust generative relevance analysis. In V. Uskov (ed.). The 5th IASTED International Conference on Web-Based Education. Puerto Vallarta, Mexico, 183-188. Vialardi, C., Bravo, J., and Ortigosa, A. (2008). Improving AEH courses through log analysis. Journal of Universal Computer Science, 14(17), 2777–2798. Wang, Y., Cheng, Y., Chang, T., and Jen, S. (2008). On the application of data mining technique and genetic algorithm to an automatic course scheduling system. In Proceedings of the IEEE Conference on Cybernetics and Intelligent Systems, Chengdu, China, 400–405. Wang, W., Weng, J., Su, J., and Tseng, S. (2004). Learning portfolio analysis and mining in scorm compliant environment. In Proceedings of the 34th ASEE/IEEE Frontiers in Education Conference. Savannah, GA, 17–24. 44 Wang, Y., Tseng, M., and Liao, H. (2009). Data mining for adaptive learning sequence in English language instruction. Expert Systems with Applications: An International Journal, 36, 7681–7686. Want, T. and Mitrovic, A. (2002). Using neural networks to predict student’s performance,” in Proceedings of the International Conference on Computers in Education, Washington, DC, pp. 1–5. Wenger, E. (1987). Artificial Intelligence and Tutoring Systems. Los Altos, CA: Morgan Kaufmann Publishers, Co. White, B., Shimoda, T., and Frederiksen, J. (1999). Enabling students to construct theories of collaborative inquiry and reflective learning: computer support for metacognitive development. International Journal of Artificial Intelligence in Education, 10, 151-182. White House (2012). Consumer Data Privacy in a Networked World: A Framework for Protecting Privacy and Promoting Innovation in the Global Digital Economy. Washington, DC: The White House. Woolf, B. (2009). Building Intelligent Interactive Tutors. Burlington, MA: Morgan Kaufmann Publishers, Co. Yudelson, M., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., and Rebecca, D. (2006). Mining student learning data to develop high level pedagogic strategy in a medical ITS. In Proceedings of the AAAI Workshop on Educational Data Mining, Boston, MA, 1–8. Zaiane, O. and Luo, J. (2001). Web usage mining for a better web-based learning environment. In the Proceedings of the Conference on Advanced Technology for Education, Banff, Alberta, 60-64. Zakrzewska, Z. (2008). Cluster analysis for user’s modeling in intelligent e-learning systems. In Proceedings of the 21st international conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems. Poland, 2008, pp. 209–214. Zimmerman, B. and Schunck, D. (2001). Self-regulated learning and academic achievement (2nd ed.). Mahway, NJ:L Erlbaum. Zinn, C. and Scheuer, O. (2006). Getting to know your students in distance learning contexts. In Proceedings of the 1st European Conference on Technology Enhanced Learning, 437-451. Zirkel, P. (2005). Searching students. Principal Leadership, 6(1), 64-68. 45 Appendix – Examples of Adaptive Learning Technologies 1. CAST UDL Lesson Builder http://lessonbuilder.cast.org/ Developed by the Center for Applied Special Technology - CAST the UDL (Universal Design for Learning) Lesson Builder is a web-based resource that provides educators with models and tools to create and adapt lessons that increase access and participation in the general education curriculum for all students. Based on research related to three primary brain networks (recognition network, strategic network, and affective network) and the roles they play in understanding these differences, UDL provides a framework for creating and implementing lessons with flexible goals, methods, materials, and assessments that support learning for all students. The UDL lesson builder site provides resources so people can: - Learn about Universal Design for Learning (UDL) - Explore model UDL lesson plans - Create, save, edit and print personal lesson plans Once a user creates an account on the site, he/she can proceed to create lesson plans. A lesson plan template is provided with all the major headings already in place. The user then enters content in spaces provided under these sections. A "more information" button is also available under each section, and upon clicking this, a window pops up with detailed explanation of all what the particular section entails. The template is also available as a downloadable word document. Model lesson plans and links to relevant external resources are also available. Lesson Builder can be considered an adaptive resource within the context of this discussion in that it provides: • Multiple means of representation (text, audio and video), to give diverse users options for acquiring information and knowledge • Multiple means of action and expression, to enable users demonstrate what they know • Multiple means of engagement, to tap into users' interests, offer appropriate challenges, and increase motivation The resource thus works for all categories of users - from the novice to the expert. As it is not strictly a learning resource, however, Lesson Builder's functionality falls short of offering that extra level of adaptivity where users are assessed (implicitly and explicitly) to determine their needs and abilities before prescribing paths deemed appropriate for each individual user. Likewise, user input is not monitored on the fly and content or learning paths modified appropriately. This is, however, understandable as the purpose of the tools is not to explicitly teach individuals how to create lesson plans but to enable 46 users create lesson plans in accordance with UDL principles. A lesson plan creator however gets the opportunity to learn about adaptivity and universal design as work progresses. 2. Jasper Woodbury Problem Solving Series http://peabody.vanderbilt.edu/projects/funded/jasper/Jasperhome.html Designed by the Cognition and Technology Group at Vanderbilt University (CTGV) the Jasper Woodbury series of adventures represents an example of anchored instruction, and also of problem-based learning. Targeted at upper elementary through high-school math learners the Jasper Series are video-based, narrative adventures that pose complex problems for students to solve using information embedded in the narrative. Students watch video segments to understand the situation and the problem and then embark on problem solving tasks using embedded data, teaching resources etc. Students then present their solutions to their fellow classmates and discuss strengths and weaknesses of whatever set of solutions they come up with. The ill-structured nature of the problems ensures that there is no one perfect solution, and this caters to students with different skill levels in math problem solving strategies. The Jasper Series typifies a constructed learning application that specifically aims at teaching math concepts and applications together with problem-solving and criticalthinking skills. Adaptivity comes in the form of user self-selection of path and resources to complete tasks, and upon solving a problem one way students can still tackle the same problem through a more advanced path. Conversely, learners trying to adopt problem solving strategies that are beyond their capabilities will naturally fail until they master the requisite skills. Other forms of adaptivity including curriculum sequencing (based on learner ability), adaptive presentation and navigation support are, however, not available. 3. INSPIRE (INtelligent System for Personalized Instruction in a Remote Environment), INSPIRE is a web-based adaptive educational hypermedia prototype developed by Papanikolaou and co-workers to support an introductory course on Computer Architecture offered to second level undergraduate learners of the Department of Informatics and Telecommunications of the University of Athens. Its development was informed largely by instructional design and learning style theories. INSPIRE gives learners the option to select meaningful learning goals, and then using additional information gathered through learner-system interaction, the system generates a sequence of authentic and meaningful tasks that match the learning style and knowledge level of each learner, based on the target goal. Also throughout its interaction with the learner, INSPIRE dynamically generates learner-tailored lessons that gradually lead to the accomplishment of a learner’s learning goals. In addition, learners have instructional control over the system as they have the option to intervene in the lesson 47 generation process, express their opinion about their own characteristics or about the lesson contents. INSPIRE is therefore fully adaptive (i.e. adapts its output using some data or knowledge about the learner) and adaptable (supports end-user modifiability, providing learners control over several functionalities). 4. MasteringPhysics http://www.masteringphysics.com MasteringPhysics is web-based hypermedia application developed by Pearson to accompany “University Physics,” a standard textbook for many first year college level physics programs. It has been positioned as a dynamic tutoring system that increases learner engagement and performance. MasteringPhysics is being adopted by educational institutions worldwide, notable among which are MIT and the University of Sydney. MasteringPhysics emulates the instructor's office-hour environment, coaching students on problem solving techniques by asking students simpler sub-questions, providing immediate and specific feedback on wrong answers and giving specific feedback on common errors to help explain why a particular answer is not correct. Hints (accessible only when needed) are also available throughout the course of solving a problem. As an adaptive teaching tool, MasteringPhysics facilitates one-on-one tutoring, encouraging (and compelling) learners to go through particular learning paths based on their input and navigation of the system. Emphasis is therefore on teaching students the concepts and application of Physics and not merely to solve a problem or accomplish a task. 5. Apangea Math http://www.apangea.com/products/apangea_math.htm Apangea Math is an internet-based math learning support system that provides one-toone differentiated math instruction through a unique integration of proprietary tutoring technology and live, online certified teachers. Developed by Apangea Learning Inc. Apangea Math is aligned to National Council of Teachers of Mathematics (NCTM) and individual state standards. Schools in more than 25 states across the United States are reported to be using Apangea Math as a supplement to instruction. Apangea Math represents an innovative way of integrating computer and human tutoring to provide adaptive math instruction to individual learners. According to the creators, Apangea Math is based on the understanding that students learn to analyze and solve word problems by applying a pedagogy derived from contemporary cognitive science, including principles of active problem solving, elaboration theory, categorization by prototype, mastery learning, and worked examples. The system thus challenges students 48 to think mathematically by offering flexible learning pathways that teach students both concepts and procedures. In terms of adaptivity, as students work, the program monitors student work and provides feedback tailored to what students need. Students can monitor their progress on a course map and progress meter. Quizzes and progress scores are used to determine if students have mastered content or have additional learning needs. When a topic needs to be revisited, it is automatically placed back into the student's learning pathway to provide additional learning opportunities. Teachers have access to assessment data in real-time reports. 6. SMART.FM (IKNOW!) http://iknow.jp/ Smart.fm (formerly iKnow!) is a social learning and community website created by Cerego Japan, Inc. The website uses intelligent software algorithms to assist users retain facts in memory and increase learning speed. Users can create, manage and share lists of facts to memorize; as well as learn a number of languages. Progress can be monitored with tests. The data is used to automatically plan a curriculum and learning strategy for the users. In addition to the website base, Smart.fm also has an iPhone/iPod touch application, a Twitter feed, a YouTube account, and a Facebook application. Smart.fm represents the first phase of a platform that combines personalized learning applications and content creation tools in a collaborative, social environment where members study, create, re-mix, share, and manage learning content of any kind. 7. Carnegie Mellon Open Learning Initiative http://oli.web.cmu.edu/openlearning/ The Carnegie Mellon Open Learning Initiative has produced a set of online courses utilizing adaptive technologies that are open to the public at large. The courses draw on research in the learning sciences to provide a complete instructional experience for students. Learner behavior are tracked constantly and used to drive improvement to the system. A major feature of the Open Learning Initiative courses is the series of “mini-tutors” or simplified versions of the cognitive tutors developed through Carnegie Learning that are embedded within the course content. The mini-tutors allow students to answer practice questions and receive feedback. In addition, students complete self-assessments 49 throughout the courses, and the results of these self-assessments are displayed alongside the performance information to allow students to develop their metacognitive skills as they move forward. 50