Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Project DoCTA: Design and use of Collaborative Telelearning Artefacts Barbara Wasson Frode Guribye Anders Mørch with contributions from Eskil Andreassen Konrad Morgan Rune Baggetun Glenn Munkvold Arnstein Eidsmo Trond Pedersen Knut Ekker Kurt Rysjedal Hege Higraff Helge Underhaug Øyvind Meistad Jo Wake Pedagogisk informasjonsvitenskap Universitetet i Bergen Project DoCTA: Design and use of Collaborative Telelearning Artefacts Skriftserie for Forsknings- og kompetansenettverk for IT i utdanning (ITU). www.itu.uio.no Produsert i samarbeid med Unipub forlag ISBN 82-7947-005-0 ISSN 1500-7707 © 2000 ITU Det må ikke kopieres fra denne boka i strid med åndsverkloven eller med avtaler om kopiering inngått med KOPINOR, interesseorgan for rettighetshavere til åndsverk. Omslag: Nordahl & Jøntvedt Sats og trykk: GCS Multicommunication AS Unipub forlag er en avdeling i Akademika AS, som eies av Studentsamskipnaden i Oslo 2 Preface Om ITUs skriftserie Forsknings- og kompetansenettverk for IT i utdanning (ITU) ble opprettet som en del av KUFs handlingsplan om ’IT i norsk utdanning 199699’, og er nå blitt videreført for en ny 4-års periode under Handlingsplanen ’IKT i norsk utdanning, Plan for 2000 – 2003’. ITUs hovedaktivitet er å igangsette forsknings- og utviklingsprosjekter innen feltet IKT og utdanning. Blant andre aktiviteter skal den også fungere som en nettverksnode mellom ulike forskningsmiljøer i Norge. ITU har gjennom perioden 1997-99 fokusert på begrepene læring og kommunikasjon innenfor skjæringspunktet av teknologi, pedagogikk og organisasjon. Det har vært et omfattende fokus på teknologiens rolle som katalysator for endring innen det tradisjonelle utdanningssystemet. Vi har i dag en rekke erfaringer knyttet til ulike endringsperspektiver, men trenger fortsatt å systematisere og klargjøre hva endringene består i. Eksempelvis trenger vi både kvalitative og kvantitative innsyn knyttet til teknologi i læringssituasjonene. Dette vil være viktige satsningsområder for den nye perioden. Skriftserien omhandler ulike typer tekster som har til felles at de tar opp utfordrende perspektiver relatert til IKT og utdanning. Det gjelder utredningsarbeid, prosjektrapporter og artikkelsamlinger. ITU har, gjennom skriftserien, som siktemål å bidra til systematisk kunnskap om IKT og utdanning, samt å skape debatt og refleksjon om de utfordringer vi står overfor. Vi håper med andre ord at skriftserien kan bidra til å presentere nye perspektiver på fremtidens utdanningssystem. ITU, februar 2000. 3 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Utgivelser i denne serien: Rapport nr. 1 Ola Erstad Innovasjon eler tradisjon Rapport nr. 2 Sten R. Ludvigsen mfl. Elektronisk ransel: Ny teknologi – nye praksisformer Rapport nr. 3 Svein Østerud mfl. Når ideer flyter sammen … Rapport nr. 4 Mona Hovland Jakobsen Skoleveien videre Rapport nr. 5 Barbara Wasson mfl. Project DoCTA: Design and use of Collaborative Telelearning Artefacts Rapport nr. 6 Geir Haugsbakk Interaktivitet, teknologi og læring Rapport nr. 7 Harald Haugen m.fl. SULDAL Rapport nr. 8 Sten R. Ludvigsen og Svein Østerud (red.) Ny teknologi – nye praksisformer. Teoretiske og empiriske analyser av IKT i bruk Rapport nr. 9 Ingeborg Krange mfl. Samarbeidsorientert læring i skolen med distribuert bruk av interaktiv 3D - en evaluering av erfaringer fra prosjektet EduAction 4 Preface Table of Contents 1 Introduction ........................................................................................................ 17 1.1 Research Questions ................................................................................. 18 1.2 Highlights ................................................................................................... 21 1.3 Organisation of the Report ..................................................................... 23 2 Theoretical foundations .................................................................................. 29 2.1 Computer supported collaborative learning ...................................... 29 2.2 Genuine interdependence ..................................................................... 32 2.3 Collaborative telelearning ...................................................................... 33 2.4 Coordination theory ................................................................................. 33 2.5 Awareness ................................................................................................. 37 2.6 Distributed collaborative learning communities ............................... 38 3 Conceptual framework for evaluations ........................................................ 41 3.1 Vygotsky and Cultural-Historical Psychology .................................... 42 Internalisation ........................................................................................... 43 The mediating role of artefacts ............................................................ 43 Zone of proximal development .............................................................. 45 3.2 Basic principles of Activity Theory ...................................................... 45 Unit of analysis .......................................................................................... 46 The activity system .................................................................................. 46 Object-orientedness ................................................................................ 49 Hierarchical structure of activity ......................................................... 50 Internalisation/Externalisation ............................................................. 51 Artefacts and mediation ......................................................................... 52 Development .............................................................................................. 52 3.3 Methodological Issues ........................................................................... 54 3.4 Some general remarks on AT ................................................................. 54 3.5 Activity theory and design ..................................................................... 56 4 Evaluation of Groupware Systems ................................................................ 59 4.1 Overview of the four groupware systems ........................................... 59 Habanero .................................................................................................... 59 LOTUS LearningSpace ............................................................................. 61 5 Project DoCTA: Design and use of Collaborative Telelearning Artefacts BSCW Shared Workspace system ....................................................... 62 TeamWave Workplace ............................................................................ 63 4.2 Test Scenario Design ............................................................................... 64 4.3 Preliminary Analysis ................................................................................. 66 Lotus LearningSpace ............................................................................... 66 TeamWave Workplace ............................................................................ 66 Habanero .................................................................................................... 67 BSCW ........................................................................................................... 69 4.4 Discussion .................................................................................................. 71 The groupware systems .......................................................................... 71 Preliminary observations ........................................................................ 72 Preliminary suggests for what a groupware system should contain ................................................................. 74 5 Teamwave workplace ....................................................................................... 77 5.1 The basis for design ................................................................................ 77 Creating a shared networked place .................................................... 80 Awareness ................................................................................................. 82 5.2 User interface considerations .............................................................. 84 5.3 Production, Communication and Distribution .................................... 85 Production/Knowledge construction .................................................. 86 Communication .......................................................................................... 87 Distribution/Division of Labour .............................................................. 88 5.4 Technical requirements .......................................................................... 89 5.5 Final remarks on TeamWave Workplace ............................................ 90 6 Ideels .................................................................................................................... 93 Norwegian Participation ......................................................................... 95 OPUSi ........................................................................................................... 96 6.1 Evaluation of intra-group collaboration .............................................. 98 The Research Study ................................................................................. 99 Data Collection ........................................................................................ 100 Data Analysis ........................................................................................... 102 Analysing the context using the activity system ............................ 106 Conclusions ............................................................................................. 109 6.2 Changes in Attitude towards Simulation-based Distributed Learning .............................................................................. 112 Pre-questionnaire ................................................................................... 112 Knowledge of computers ...................................................................... 117 7 VisArt .................................................................................................................. 121 7.1 VisArt Design ........................................................................................... 121 The design process ............................................................................... 122 Modelling the VisArt Activity ............................................................... 124 6 Preface 7.2 7.3 7.4 7.5 7.6 7.7 7.8 7.9 7.10 7.11 7.12 Management and Coordination .......................................................... 126 Student Consent to Participate in Research ................................... 129 The Computer Supported Collaborative Learning Assignment ... 131 The Background Literature .................................................................. 131 The VisArt Activity .................................................................................. 132 The Individual Report ............................................................................ 135 Communication VisArt ............................................................................ 136 Assistance in VisArt ............................................................................... 136 VisArt in TeamWave Workplace ......................................................... 137 The Classroom ......................................................................................... 138 The Training Room .................................................................................. 139 Deployment of VisArt ............................................................................. 151 Participants .............................................................................................. 152 Preparation for VisArt ............................................................................ 153 Training ...................................................................................................... 153 The Design Actvitiy ................................................................................ 154 Reflections on the Design .................................................................... 158 Assistance and Help .............................................................................. 158 Administration of VisArt ......................................................................... 161 Evaluation of VisArt ................................................................................ 164 The learners’ experiences ................................................................... 165 Sharing of information ........................................................................... 165 Division of labor ...................................................................................... 166 Joint thinking ........................................................................................... 166 Awareness ............................................................................................... 166 A Usability evaluation of TeamWave Workplace ........................... 168 Research design ..................................................................................... 168 Analysis and Preliminary conclusions ............................................... 171 Effectiveness ........................................................................................... 171 Efficiency ................................................................................................. 173 PePedagogical decisions: .................................................................... 173 Satisfaction .............................................................................................. 177 Conclusions ............................................................................................. 182 The Role of Classical Jungian Personality Factors in Computer Supported Collaborative Learning Environments ........ 183 Review of Previous Personality Research in the use of predicting and understanding user attitudes and behaviour ...................................................................................... 183 Previous work by the author ................................................................ 184 The Myers-Briggs Personality Test .................................................... 184 The Myers-Briggs personality Type Inventory 7 Project DoCTA: Design and use of Collaborative Telelearning Artefacts (MBTI) in Education ................................................................... 185 Method ...................................................................................................... 186 Security and ethical controls .............................................................. 186 Data entry and analysis ........................................................................ 187 Results ....................................................................................................... 188 7.13 Evaluating How Students Organise their Work ............................... 191 Evaluation approach ............................................................................. 192 The data collection ................................................................................ 192 The data .................................................................................................... 194 Preliminary data analysis ..................................................................... 195 Co-ordination ........................................................................................... 196 Communication mode ............................................................................ 198 Feedback .................................................................................................. 200 Summary and preliminary findings ..................................................... 201 7.14 Evaluating the Organisation of a Collaborative Telelearning Scenario .................................................................................................... 202 Methodological aspects ...................................................................... 202 Data collection ........................................................................................ 205 Preliminary data analysis ..................................................................... 205 7.15 Supporting Collaborative Telelearning Research using Server Logs .............................................................................................. 211 Method ...................................................................................................... 212 The TeamWave Workplace Server File ............................................. 212 The TeamWave Workplace Version Control .................................... 213 Two tools for analysing log-files ......................................................... 214 Conclusion ............................................................................................... 220 7.16 Coordination in collaborative telelearning ...................................... 221 Method ...................................................................................................... 222 Analysis ..................................................................................................... 223 Summary .................................................................................................... 224 7.17 Facilitating Help and Training in VisArt ............................................. 225 Data collection ........................................................................................ 225 Preliminary analysis ............................................................................... 226 8 Results and conclusions ............................................................................... 231 8.1 Collaboration Patterns ........................................................................... 231 Finding collaboration patterns ............................................................ 231 Four collaboration patterns ................................................................. 232 Pattern 1: Adaptation ............................................................................. 232 Pattern 2: Coordinated desynchronisation ...................................... 233 Pattern 3: Constructive commenting .................................................. 235 Pattern 4: Informal Language .............................................................. 236 8 Preface Structure of collaboration patterns ................................................... 237 How we plan to use collaboration patterns in further work ........ 238 8.2 Studying students’ online learning .................................................... 240 Methodological issues .......................................................................... 241 Online and offline activities ................................................................. 243 9 Summary and Conclusions ............................................................................ 245 9.1 Summary .................................................................................................... 245 9.2 Conclusions ............................................................................................. 249 References ............................................................................................................. 253 Appendicies ........................................................................................................... 267 9 Project DoCTA: Design and use of Collaborative Telelearning Artefacts List of Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 Figure 12 Figure 13 Figure 14 Figure 15 Figure 16 Figure 17 Figure 18 Figure 19 Figure 20 Figure 21 Figure 22 Figure 23 Figure 24 Figure 25 Figure 26 Figure 27 Figure 28 Figure 29 Figure 30 Figure 31 Figure 32 Figure 33 10 The DoCTA Project’s home page ..................................................... 22 Vygotsky’s mediating triangle (Vygotsky, 1978) ............................ 44 The basic mediational triangle (Cole & Engeström, 1993) ......... 47 Engeström’s (1987) model of the activity system ......................... 48 The hierarchical structure of activity (Leontev, 1978) ................ 51 The nested nature of activity dynamics (Engeström 1987) ....... 53 Integration of a design perspective with Activity Theory (Mørch & Wasson, 1999) ................................................................... 57 The physical setting of the scenario .............................................. 65 The two white boards used by the students in Habanero ........ 68 Part of the class-diagram the students made in MS paint while testing BSCW ............................................................................ 70 Barbara Wasson’s (empty) Room in TeamWave Workplace ..... 80 Room and permissions menus in TeamWave Workplace .......... 81 Classroom – An example of a TW room and its tools .................. 82 Screenshot from the IDEELS’ homepage ....................................... 94 Supporting tools in the scenario ..................................................... 96 Screenshot from OPUSi ...................................................................... 98 The two phases in the collaboration process............................ 103 The collaboration process as an activity system ...................... 107 The hierarchical structure of an activity ..................................... 109 Constraints on the VisArt design ................................................... 122 Top level of VisArt model ................................................................. 125 A second level of the model of VisArt .......................................... 126 Organisation of Assistance in VisArt ............................................ 137 The Classroom in VisArt ................................................................... 138 Training Room in VisArt .................................................................... 139 Welcome to the Training Room ...................................................... 140 Training Assignment 1 room in VisArt ........................................... 141 Training Assignment 1 ...................................................................... 142 An excerpt from the Narg Island Game ....................................... 143 Directions for a Narg Island Game with Distributed Teams .... 144 Training Assignment 4 ...................................................................... 146 VisArt’s Help Room in TeamWave .................................................. 147 Home Page of TeamWave Workplace Help Pages (developed for VisArt) ...................................................................... 148 Preface Figure 34 Figure 35 Figure 36 Figure 37 Figure 38 Figure 39 Figure 40 Figure 41 Figure 42 Figure 43 Figure 44 Figure 45 Figure 46 Figure 47 Figure 48 Figure 49 Figure 50 Figure 51 Figure 52 Figure 53 Figure 54 Figure 55 Figure 56 Figure 57 Figure 58 Figure 59 Figure 60 Figure 61 Figure 62 Figure 63 Figure 64 Figure 65 Figure 66 Figure 67 Figure 68 Help page for the Address Book tool ........................................... 149 Help page for the File Holder tool ................................................. 150 Help page for Navigating between rooms .................................. 151 Team 7’s working room ..................................................................... 154 Team 7’s room for learning about polar bears ............................ 155 Team 10’s group room ....................................................................... 156 Team10’s warehouse ........................................................................ 157 Designed learning room for Team10 ............................................. 157 Ease of getting help/information ................................................... 160 Administrator Interface for server information and sending messages ............................................................................................ 162 Administrator interface for creation of accounts ..................... 162 Administrator interface for creation of groups .......................... 163 TeamWave Workplace tools grouped according to functionality ....................................................................................... 170 An instructional room designed by Team 6 ................................. 172 Extracts from Team 6’s pedagogical decisions file ................... 173 Extracts about asynchronous communication in the interviews ............................................................................................ 175 Extracts about synchronous communication in the usability interviews ........................................................................... 176 Some example questions from the pre-questionnaire .............. 178 Perception of TW regarding usefulness, simplicity and attractiveness. ................................................................................... 178 Difficulty of understanding functions in TW .............................. 179 Stability of TW server ....................................................................... 179 Missing functions in TW ................................................................. 180 Categories of negative and positive comments ........................ 181 Producing a TW learning room as an activity system .............. 195 VisArt instructors and facilitators as an activity system ......... 206 Extractions from the VisArt server.log file ................................... 213 An excerpt from SLA’s group log-file ............................................. 215 Graphs of Team Activity in VisArt .................................................. 216 One of many possible views of TRA .............................................. 218 Using TRA to record analysis of tool use .................................... 220 Experience with collaboration from prior education ................ 226 Experience with collaboration from work life ............................ 227 Usefulness of the training in TW, after completion of the design task ......................................................................................... 227 Help page for the File Holder tool ................................................. 370 Help page for the ToDoList tool ..................................................... 370 11 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 69 Figure 70 Figure 71 Figure 72 Figure 73 12 Help page for the Page function ................................................... 371 Help page for how to Save Rooms ................................................ 371 Administrator interface for administering conferences .......... 372 Administrator interface for administering templates ................ 373 Administrator interface for administering repositories ............ 373 Preface List of Tables Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Table 8 Table 9 Table 10 Table 11 Table Table Table Table Table Table 12 13 14 15 16 17 Table 18 Table 19 Table 20 Table 21 Table 22 Table Table Table Table Table 23 24 25 26 28 Research Questions asked in the DoCTA studies ....................... 20 Project DoCTA Work Package Deliverables .................................. 24 Dependencies between Activities (from Malone & Crowston, 1994) ...................................................... 35 Dependencies between Actors ....................................................... 37 TW tools facilitating production/knowledge construction (Fjuk et al. 1999) ................................................................................... 87 How TW facilitates communication (Fjuk et al. 1999) ................. 88 How TW facilitates distribution/division of labour (Fjuk et al. 1999) ....................................................................................................... 89 Data gathering techniques ............................................................. 100 Distribution of background information by Site ......................... 113 Distribution of ‘computer at home’ and ‘ computer use’ by Site. ..................................................................... 114 Distribution of experience with teamwork, role playing and negotiating games by Site. ..................................................... 114 Evaluation of the simulation by Site ............................................. 115 Evaluation of the simulation by Gender ....................................... 116 Cross-cultural experience by Gender........................................... 116 Experience with teamwork by Gender ......................................... 117 Overall evaluation by Gender ......................................................... 117 Overall evaluation of the simulation by Knowledge of computers ................................................................ 118 A good learning experience (attitude change) * Gender ....... 118 The simulation game will be/was difficult (attitude change) * Gender ............................................................ 119 A good learning experience * The simulation game will be/was difficult ........................................................................... 119 Preparation from the Instructor’s Perspective (as at 24.02.99) ................................................................................... 127 Preparation from Researcher’s Perspective (as at 24.02.99) ................................................................................... 129 The 5 VisArt Activities ...................................................................... 133 Training Assignment 3 ...................................................................... 145 Data gathering techniques ............................................................. 194 Instructor and facilitator roles in VisArt ...................................... 203 Characteristics of Collaboration Patterns ................................... 238 13 Project DoCTA: Design and use of Collaborative Telelearning Artefacts List of Appendicies Appendix Appendix Appendix Appendix Appendix Appendix Appendix Appendix Appendix Appendix DoCTA Publications ................................................................. 268 DoCTA Presentations .............................................................. 270 Assignment for test-scenario (English) .............................. 272 Interview Guide for Evaluation of Groupware Systems .. 275 The IDEELS Scenario .............................................................. 276 Assignment for designing an evaluation of IDEELS ........ 290 Interview Guide for IDEELS (English) .................................. 292 Pre-simulation survey for IDEELS ......................................... 294 Post-simulation survey for IDEELS ....................................... 299 Planning Tables from the Instructor’s View (Phases II-IV) ............................................................................ 303 Appendix K: Planning Tables from the Researcher’s View (Phases I-III) ............................................................................. 305 Appendix L: Planning Tables from the Student’s View (Phases I-III) . 307 Appendix M: VisArt Consent Form ................................................................ 310 Appendix N: Semester Assignment ............................................................. 312 Appendix O: Directions for TeamWave Workplace Installation .......... 317 Appendix P: Designed Learning Rooms from VisArt ................................ 318 Appendix Q: VisArt Profile Questionnaire (Norsk) ................................... 323 Appendix R: VisArt Self-Evaluation Questionnaire .................................. 328 Appendix S: VisArt Pre-questionnaire ........................................................ 331 Appendix T: VisArt Post-questionnaire (Norsk) ....................................... 340 Appendix U: Norweign Version of Training Assignments 1,3 & 4 ......... 354 Appendix V: The Narg Island Game ............................................................ 357 Appendix W: Help Page Screen Shots ......................................................... 370 Appendix Y: Administration Interface in TeamWave Workplace ........ 372 Appendix Y: Usability Study Interview Guide ........................................... 374 Appendix Z: Interview Guide for How Students Organise their Work 380 Appendix AA: Interview Guide for How Instructors and Facilitators Organise their Work (English) ............................................... 384 14 A: B: C: D: E: F: G: H: I: J: Preface Preface Activities of project DoCTA: Design and use of Collaborative Telelearning Artefacts (project leader Barbara Wasson) are aimed at the design and use of technological artefacts to support collaborative telelearning aimed at teacher training. The research is not limited to only studying these artefacts per se, but includes social, cultural, pedagogical and psychological aspects of the entire process in which these artefacts are an integral part. This means that we both provide and study virtual learning environments that are being deployed to students organised in geographically distributed teams. The main research focus is reflected in both the theoretical and methodological approach chosen in the project. The theoretical, or conceptual approach, is rooted in a sociocultural perspective that emphasises an understanding of language, culture and other aspects of social setting. The methodology is influenced by ethnographic studies, favouring naturalistic and qualitative research methods. Three scenarios utilising the Internet were used to engage the students in collaborative learning activities. Scenarios IDEELS and Demeter involved European inter-cultural simulations where the goal was to design a textual artefact (e.g., a treaty or policy statement). A third scenario, VisArt, was designed and developed explicitly for use between the three educational partners and had the goal of designing a visual artefact to be used in teaching a subject of choice. These scenarios were studied from a number of perspectives including ethnographic flavoured studies focused on understanding work organisation, usability studies of groupware systems, evaluations of computer logs, and questionnaire studies. This report documents the activities undertaken in project DoCTA. The majority of the studies were undertaken as part of graduate students’ thesis work. Although all data has been collected, analysis of the data is 15 Project DoCTA: Design and use of Collaborative Telelearning Artefacts still ongoing, and the dissertations are in various stages of being written. Thus, our final discussion in this report is preliminary and can only point to the direction in which our integrative analysis is headed. DoCTA has been administered and co-ordinated by the Department of Information Science at the University of Bergen (Bergen). It has been a collaboration with HiNT (Høgskolen i Nord-Trøndelag), HSH (Høgskolen Stord/Haugesund) and Telenor FOU, Kjeller. The 19 researchers and graduate students involved have various backgrounds including computer science, psychology, sociology and education. Project members included: University of Bergen Nord-Trøndelag College College Barbara Wasson Hege Higraff Knut Ekker Anders Mørch Øyvind Meistad Arnstein Eidsmo Konrad Morgan Trond Pedersen Glenn Munkvold Mike Spector Kurt Rysjedal Eskil Andreassen Helge Underhaug Rune Baggetun Jo Wake Frode Guribye Stord/ Haugesund Lars Vavik Amund Junge Telenor FOU, Kjeller Annita Fjuk As project leader I take the liberty to thank all project members for efforts both during the numerous project activities and, especially, during these last months as we have prepared for the delivery of this final report. In particular I thank my D.Polit student Frode Guribye for all his extra efforts while I am on maternity leave. In addition, we all thank Dr. Ileana de la Teja of LICEF, Télé-université, Montréal who served as a consultant on the design of VisArt. Finally, I would like to thank KUF (ITU programme) for funding DoCTA and supporting 6 of our graduate students with stipends. Barbara Wasson, January, 2000 16 Chapter 1 Introduction 1 Introduction Project DoCTA (Design and use of Collaborative Telelearning Artefacts) aims to bring a theoretical perspective to the design of ICT technologies that supports the sociocultural aspects of human interaction and to evaluate its use. The main, long term, research objectives are to: • take a sociocultural perspective on learning activity focussing on the interpersonal social interaction in a collaborative learning setting (collaboration = coordination + communication) • contribute to collaborative telelearning knowledge about the pedagogical design of learning scenarios, the technological design of the learning environment to support these learning scenarios, and the organisational design for management of such learning environments, including a reflection on teacher and learner roles • study and evaluate the social and cultural aspects of distributed collaborative telelearning environments Through these objectives we aim to improve our understanding of the pedagogy and technology of networked learners, and increase our understanding of learner activity. This will lead to better design, management and affordances (Gibson, 1986; Laurillard, 1987; Norman, 1990) of on-line learning spaces. In the first phase of project DoCTA (June 1998 - December 1999) we focus on the design and use of technological artefacts to support collaborative telelearning aimed at teacher training (Wasson & Mørch, 1999). The research is not limited to only studying these artefacts per se, but includes social, cultural, pedagogical and psychological aspects of the entire process in which these artefacts are an integral part. This means that we both provided and studied virtual learning environments that were deployed to students organised in geographically distributed teams. The main research focus is reflected in both the theoretical and 17 Project DoCTA: Design and use of Collaborative Telelearning Artefacts methodological approach chosen in the project (Wasson & Guribye, 1999; Mørch & Wasson, 1999). The theoretical, or conceptual approach, is rooted in a sociocultural perspective and the methodology was influenced by ethnographic studies, favouring naturalistic and qualitative research methods. These ethnographic flavoured studies were augmented with more traditional computer science oriented usability studies, evaluations of computer logs, and questionnaire studies. Various scenarios utilising the Internet were used to engage the students in collaborative learning activities (Wasson & Mørch, 1999). Through participation, the teachers gained experience with not only collaborative learning, but with collaborative telelearning through the collaborative design of a textual or visual artefact. Scenarios IDEELS and Demeter, involved European inter-cultural simulations where the goal was to design a textual artefact (such as a treaty or policy statement). Scenario VisArt (Wasson, 1999b) had the goal of designing a visual artefact to be used in teaching a subject of choice. The IDEELS and Demeter scenarios resulted in 1 Masters1 dissertation each, Guribye (1999) and Junge, (1999), respectively. VisArt will result in 9 Masters dissertations ¾ these are currently being written. In addition, two researcher studies, one of IDEELS (Ekker, 1999) and one of VisArt (Morgan & Morgan, 1999) have been completed. Although we have begun to integrate the results from the individual studies in order to identify collaboration patterns, there is much work to be done. The remainder of this introduction focuses on DoCTA’s research questions, enumerates our evaluation studies, points out highlights of the first phase and describes the organisation of the report. 1.1 Research Questions Project DoCTA is first and foremost an exploratory study that focuses on understanding interaction in a set of collaborative telelearning scenarios. The community of study includes teachers, learners and facilitators participating in the various collaborative telelearning scenarios. The study provides us with insight into the processes of collaboration enabling us to 1 18 A Masters dissertation is the written document resultant from a Norwegian hovedfagsoppgaver. Chapter 1 Introduction identify collaboration patterns and further our understanding of how instructors, students and other learning facilitators organise their learning and work given the kinds of artefacts they have access to, the kind of learning activity they are presented with, and the kinds of artefacts they will design. While a plethora of research questions could be asked, we restricted this study to two main questions. Given that within our collaborative telelearning scenarios, the contexts vary with respect to the actor characteristics, the type of learning activity, the kinds of artefacts they have access to, and the kind of artefact to be designed, we ask: • How do the students, teachers and facilitators organise their work? • How do patterns of collaboration vary? In order to answer these two primary questions, a number of evaluation studies are being carried out on several levels and from several perspectives. In these evaluations, more specific questions have been asked. Eleven of these studies are part of Masters’ projects and cover: evaluations of suitability of groupware systems for collaborative telelearning; usability studies of TeamWave Workplace (TW); looking at the efficiency of TW from a qualitative perspective using the data logs generated by TW; performing a formative evaluation of how to support collaborative design activities; seeing how TW supports coordination; how to design training and assistance in a collaborative telelearning setting; and, several activity theory studies of how students, instructors and facilitators organise their work. In addition, as mentioned above, two researcher studies, one of attitude changes over time and one on personality factors and choice of collaboration artefacts have been completed. Table 1 summarises the research questions of the various studies. Numerous data and data collection techniques have been used including questionnaires, semi-structured interviews, automatically generated log files, email sent among team members, instructors and facilitators, chat files saved by the teams, documents and artefacts produced by the teams, participant observations, video taping of activities, self-evaluations and the participants own interpretation of the experience. 19 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Table 1 Research Questions asked in the DoCTA studies Domain/theoretical framework Research Question Suitability of Groupware Systems for Collaborative Telelearning Evaluation of the interface and Can students working in collaboration solve a the relative usefulness collaborative design task with Habanero, TeamWave Workplace, BSCW and Lotus LearningSpace? Usability of TeamWave Workplace Usability – effectiveness Can students working in collaboration reach their team goals with TW? Usability – efficiency To what extent do the tools enable students to meet their task? Usability - satisfaction Are students satisfied of TW tools? Efficiency from a qualitative perspective Comparison of anticipated use and actual use of the tools What are the differences between the anticipated use of the tools and the actual use of the tools? Which are the differences of tool use when alone and when others in the room? How are the rooms used? Research methodology What implications do the use of the tools have for electronic data-collection? Formative evaluation Tailoring for instructional design How to support pedagogical room design in TW? User centred design How do students use TW tools to design a room for teaching? What is to be improved? Attitude changes What changes in attitudes towards simulationbased learning situations occur? Personality factors Are there any personality factors involved in the use of tools in telelearning situations? Coordination theory Supporting coordination How to support coordination in a collaborative telelearning environment? User centred design What kind of coordination do students do? 20 Chapter 1 Introduction Activity theory Organisation of work How do the students organise their work? – student perspective Organisation of work – Instructor‘s perspective How instructors and facilitators organise their work Design of training and assistance Training / Assistance effectiveness How effective is the training designed for collaborating and learning to use TW? How to improve training and assistance? 1.2 Highlights During the past one and a half years, there have been many activities around project DoCTA. This section discusses some of the highlights. DoCTA Workshops: Project DoCTA began with a kick-off workshop in Bergen in September 1998. We arranged and hosted an Activity Theory workshop in October 1998 with Dr. Victor Kaptelinin as the invited speaker (2 graduate students from the ITU Project Ny Teknologi og NyePraksisformer also attended this workshop). In November 1998 we had an internal Bergen workshop on the Design of VisArt. Finally, a VisArt Debriefing workshop was held in the late spring of 1999 after the VisArt scenario was completed. This workshop served as a means of debriefing ourselves about the scenario. Publications: Project DoCTA has resulted in 9 publications. One additional paper is under review and another was invited to appear in the IFETS (The Journal of the International Forum of Educational Technology & Society) Special Issue on On-line Collaborative Learning Environments. Appendix A lists these publications. Guribye & Wasson (1999) was awarded an Outstanding Paper award at Ed-Media’99 and Meistad (1999) was one of three winners of the ITU Essay Contest. 21 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Presentations: Seventeen presentations of DoCTA have been presented. These presentations, listed in Appendix B, have presented various aspects of the DoCTA project. Conferences: DoCTA researchers attended 10 different conferences, workshops or graduate summer schools. The project was presented at many of these. Several of the graduate students were able to attend conference tutorials that provided both relevant information and motivation for their Masters thesis work. Masters’ dissertations: The IDEELS and Demeter scenarios resulted in 1 Masters dissertation each, Guribye (1999) and Junge, (1999), respectively. The VisArt scenario will result in 9 Masters dissertations Andreassen (in preparation), Baggetun (in preparation), Higraff (in preparat- Figure 1 22 The DoCTA Project’s home page Chapter 1 Introduction ion), Meistad (in preparation), Pedersen (in preparation), Rysjedal (in preparation), Underhaug (in preparation), and Wake (in preparation). Hovedfagforum: Six of DoCTA’s graduate students received stipends from ITU’s hovedfagforum (graduate student forum). This enabled them to attend conferences and tutorials. Web page: Project DoCTA strives to maintain a project web page. Figure 1 presents the project’s home page. The pages include information about: the four scenarios; the project partners and researchers, publications and presentations, and a set of resources including related projects, conferences (with reviews), relevant literature (with a review), and relevant web pages. We are currently adding an overview of the VisArt scenario that will provide a demo tour through the scenario and include a presentation of some of the visual artefacts created by the students participating in the scenario. 1.3 Organisation of the Report This report is rather substantial and serves as the final report for the first phase of project DoCTA. As our integrative analysis is very preliminary at this point2 , subsequent results will be published as Masters dissertations, or conference or journal papers. This report is divided into three parts. Part I expands the theoretical foundations and conceptual framework described in the DoCTA project application. Part II presents the design, deployment and evaluation of each two of the scenarios, IDEELS and VisArt. Part III presents some of the tentative results on collaboration patterns. In the project application we identified a number of work packages and deliverables. Table 2 lists these work package deliverables and identifies where in the report they can be found3 . 2 3 The project leader has been on maternity leave since July. These are often only summaries or examples of what will be reported in a Masters dissertation. 23 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Table 2 Project DoCTA Work Package Deliverables WP# Deliverable Source Where in Report 2.1 Expanded theoretical framework Hovedfag dissertations, published papers Part I, section 2 2.2 Review of possible tools for use in WP 7 Hege’s dissertation, Barbara’s review Part I, sections 4 & 5 3.1 A set of quantitative and qualitative tools (e.g., questionnaires, interview guides) IDEELS questionnaires, VisArt questionnaires, Interview schemes Appendices D, G-I, M-O, Q-Z, AA 3.2 A description of our qualitative method (including data collection issues, data analysis issues, AT, etc.) Frode’s dissertation, published papers Part I, section 3 Part III, section 7.10 – 7.17 3.4 A scenario 1 evaluation report (IDEELS) Frode’s dissertation, Knut’s study Part II, section 6.1 Part II, section 6.2 3.6 A scenario 3 evaluation report (VisArt ) Hovedfag dissertations, published papers Part II, section 7 3.7 Final integrating evaluation report Group meetings, evaluation studies, collected data Part III, section 8.1 4.1 Documented PedInfo learning EdMedia paper scenario (this scenario changed from the original application) See Wasson (1999b) 4.3 Documented experience with Trond’s experience on the TeamWave Workplace administration of TW software Part II, section 7.8 5.1 Design details of the IDEELS scenario and reflections on the deployment Frode’s dissertation, Part II, section 6 Bergen student’s reflection, HiNT reflections (Glenn, Knut & Arnstein), published papers, Knut’s questionnaire 6.1 Design details of the scenario and reflections on the deployment Amund’s thesis: Junge (1999) 7.1 Design details of the VisArt scenario Scenario tables, VisArt assignment published papers, hovedfag dissertations 24 — Part II, section 7.1-7.5 Appendicies J-O Chapter 1 Introduction 7.2 FAQ-list for commercial tools Kurt’s help pages, help room, tips http://www.ifi.uib.no/docta/ VisArt/help Part II, section 7.5 http:/www.ifi.uib.no/docta/ teamwave/install.html, 7.3 Reflections on deployment Trond’s administration Part II, section 7.6 experience, Helge/Barbara/Trond on help/assistance 7.4 Web Archive All contribute with screen Under development, will shots of your teams’ rooms be accessible from the DoCTA web page http://www.ifi.uib.no/docta 8.1 DoCTA web site All 8.3 International publications List on web & in Appendix Appendix A http://www.ifi.uib.no/docta Part I: Theoretical Foundations and Conceptual Framework Part I expands the theoretical foundations and conceptual framework presented in the project application, presents a review of four groupware systems and concludes with a comprehensive description of TeamWave Workplace. Section 2, theoretical foundations, covers computer supported collaborative learning, genuine interdependence, collaborative telelearning, coordination theory, awareness, and distributed collaborative learning communities. Section 3 presents the conceptual framework that has influenced the evaluations of the scenarios. In particular, Vygotsky and cultural-historical psychology are reviewed and Activity Theory is covered in detail. The section concludes with some general remarks on AT and on it’s implications for design. Section 4 presents an analysis of four groupware systems, Habanerio, Lotus LearningSpace, BSCW Shared Workspace System, and TeamWave Workplace. Finally, a thorough review of TeamWave Workplace in presented in section 5. Part II:The Scenarios Part II has two sections, one for each scenario. Section 6 reports on the IDEELS scenario and section 7 on VisArt. Each section describes its scenario’s design, discusses issues around the deployment of the scenario, and then presents evaluation studies of the scenario. 25 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Part III: Results and Conclusions Part III has 2 sections. Section 8 discusses the results of the project to date. Section 8.1 reports the tentative results of our integrative analysis of collaboration patterns. As indicated in the preface, however, the project provided the opportunity for 11 Masters theses and these are in various stages of completion. Therefore, the integrative analysis of collaboration patterns presented in this report is preliminary. Section 8.2 presents a discussion of methodological issues around virtual ethnography. Section 9 summaries the report and gives some conclusions. 26 Chapter 2 Theoetical foundations Part I Theoretical Foundations, Conceptual Framework, and Groupware Systems 27 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 28 Chapter 2 Theoetical foundations 2 Theoretical foundations BARBARA WASSON There are a number of research areas that have influenced project DoCTA (see Wasson & Mørch, 1999; Wasson, 1999 for details). The two most significant are the conceptual framework offered by sociocultural perspectives and the emerging area of computer-supported collaborative learning, in particular Salomon’s (1992) work on genuine interdependence. Other perspectives that provide inspiration and guidance for this research come from: computer supported collaborative work (CSCW), in particular Gutwin et al.’s (1995) ideas on awareness; coordination science (Malone & Crowston, 1994); sociocultural perspectives (Wertsch, del Río & Alvarez, 1995) on learning and thinking; and, the emerging notion of distributed learning communities. The conceptual framework adopted in the project is described in section 3. This section provides a brief description of the remaining areas. 2.1 Computer supported collaborative learning Computer supported collaborative learning (CSCL) is an emerging paradigm (Koschmann, 1996) for research in educational technology that focuses on the use of information and communications technology (ICT) as a mediational tool within collaborative methods (e.g., peer learning and tutoring, reciprocal teaching, project- or problem-based learning, simulations, games) of learning. It is an approach to ICT in education that emphasises an understanding of language, culture and other aspects of the social setting (Scott, Cole & Engel, 1992). Its intellectual heritage can be found in social constructivism (Doise, 1990), the Soviet cultural-historical psychology (e.g., Vygotsky (1978), Leontiev (1978), Davydov (1988)) and situated cognition (Suchman, 1987; Lave, 1988). CSCL research involves naturalistic observations being used in an exploratory 29 Project DoCTA: Design and use of Collaborative Telelearning Artefacts fashion to permit a more complete understanding of this instructional mode. The emphasis is on the process and not so much focused on outcome. Evaluations often result in descriptive studies which focus on artefacts that support or are produced by teams of learners and usually contain participant accounts of their own work. The influence of CSCL research on DoCTA is evident in the type of research questions we ask and in the choice of conceptual framework for organising our evaluations. Project DoCTA subscribes to the collaborative learning paradigm. Collaborative learning can be seen as a collection of perspectives based on principles of interpersonal interaction that emphasises an understanding of language, culture and other aspects of the social setting (Scott, Cole & Engel, 1992). Traditionally methods such as peer learning and tutoring, reciprocal tutoring, project- or problem-based learning, simulations and games have been used to engage learners in collaboration. Collaborative learning also refers to notions of “socially shared cognition” (Resnick, Levine & Teasley, 1991), of “distributed cognition” (Salomon, 1993) and of “jointly accomplished performance” (Pea, 1993), which emphasise that cognitive development occurs through interactions between students, between students and teachers, as well as between students and rich or knowledgeable environments. ICT support for collaborative learning can take various forms. Fjuk (1998) identifies three perspectives of collaboration that place emphasis on different goals including joint construction, joint negotiation and using teachers and peers as resources. Andriessen & Sandberg (1999) list four different uses of ICT to support collaborative learning including computer-based collaborative tasks, co-operative tools, computer mediated communication, and intelligent co-operative systems. The Center for Innovative Lear ning Technologies (http://cilt.or g/html/ communitytools.html) at the Stanford Research Institute (SRI) takes a tools oriented approach and identifies three classes of interactive digital media for collaboration including tools for collaborative representation, tools for social and knowledge networking, and tools for cognitive scaffolding. This can be summarised in the following perspectives on the use of ICT to support collaborative learning: 1. Collaborative Tasks. In this perspective, the user is provided with an environment that supports student-student collaborative problem solving. Shared problem representation is an important aspect, 30 Chapter 2 Theoetical foundations through, for example, visualisation and modelling. For one example see the work on joint construction of knowledge (e.g., joint problem-solving by mutual refinement). For example, see the work on CoVis (Pea, R., Edelson, D., & Gomez, L., 1994). On the web at http://www.covis.nwu.edu/. 2. Co-operative or Collaborative Tools. In this perspective, a computer serves to reduce the cognitive load or lower-order tasks so the student can focus on higher-order activities. Such tools provide cognitive scaffolding. The computer provides cognitive support to structure learning activity. For an example see the work on CSILE (Scardamalia et al., 1989) where scaffolding in the form of sentence starters (e.g., I think., I know, I wonder, etc.) is provided (http://www.ed.gov/pubs/ EdReformStudies/EdTech/csile.html) Another example is the work on Belvedere (Suthers & Weiner 1995) where students engaging in joint negotiation of alternatives through argumentation, debate and other means are provided with interface support for structuring their arguments. On the web at http://advlearn.lrdc.pitt.edu/belvedere/ index.html 3. Computer-mediated Communication. In this perspective, collaboration occurs over electronic networks. Students rely on peers, teachers, domain experts, etc. as a resource to support their own learning and for external feedback. For example Computer Mediated Communication (CMC) systems or groupware systems (e.g., Teamwave Workplace, http://www.teamwave.com) provide a forum where distributed students can participate in collaborative learning activity. 4. Intelligent Co-operative Systems. In this perspective the computer serves as an intelligent co-operative partner, a co-learner (Dillenbourg & Self, 1992), or a learning companion (Chan & Baskin, 1990). As designers of learning tasks and the technological support to go with them, we must keep in mind that our foremost role is to provide a support environment that makes coordination, communication and collaboration as transparent as possible (Bourdeau & Wasson, 1997). This will enable students to create their own learning community. 31 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 2.2 Genuine interdependence Salomon’s work on CSCL (Salomon, 1992, 1993) provides the most complete approach to the study of CSCL in that it is built upon learning theories, relies on observations, raises strong design issues and gives methodological tools for educational research. Salomon (1992) distinguishes between the effects with a tool and/or collaborating peers and the effects of these. The first, the effects with, are the changes or results afforded during the use of a tool or in collaboration with a partner (i.e., the group can achieve more together than individually). The second, the effect of, is the aim Salomon advocates for education and is the lasting changes, or cognitive residue, that are a consequence of an intellectual partnership. To achieve long-term cognitive residue requires mindful engagement on the part of the learner. Helping the learner reach this level requires that “the whole learning environment, not just the computer program or tool, be designed as a well orchestrated whole. This includes curriculum, teachers’ behaviors, collaborative tasks, mode of peer collaboration and interaction, tasks, learning goals, and the like (Salomon, 1992, pg. 64)”. Salomon’s research focus is on mediation in CSCL, which is a key issue in collaborative telelearning. Salomon (1992, 1993) states that CSCL’s success depends on mindful involvement and personal responsibility on the part of the individual learner, and interdependencies between learners. Of primary importance is interdependence. Furthermore, Salomon argues that effective collaborative learning requires much and well orchestrated interdependence. To avoid group phenomena such as the “free rider” (Kerr & Brunn, 1983), the “sucker effect” (Kerr, 1983), the “status sensitivity” effect (Dembo & McAuliffe, 1987) or the “ganging up on the task” phenomenon (Salomon & Globerson, 1989), the interdependence must be genuine. Genuine interdependence is characterised by Salomon (1992) as: • the necessity to share information, meanings, conceptions and conclusions • a division of labour where roles of team members complement one another in a joint endeavour and the end product requires this pooling of different roles, and • the need for joint thinking in explicit terms that can be examined, changed, and elaborated upon by peers. In project DoCTA, Salomon’s ideas influence the design of the collaborative tasks given to the students — effort is placed in designing 32 Chapter 2 Theoetical foundations tasks that create genuine interdependence between the students. This is illustrated in the design of the VisArt scenario (see Part II). 2.3 Collaborative telelearning The term telelearning is used to designate new forms of distance or of computer mediated learning, where the distance is not only distance in space or time as in traditional distance learning, but the mediation of learning activities served by media such as multimedia shared workspaces, communication, or servers (Bourdeau & Wasson, 1997). Distance learning evolved from a need to ensure equal access to education for all students even when there are obstacles for them to access school or campus (Bourdeau & Bates, 1997). The main features of distance learning are the time and space co-ordinates, where students and professors do not all meet at the same place at the same time. Individual learning, individual tutoring and asynchronous communication are typical features of a distance learning situation, requiring extensive macro- and microinstructional design, and a strong student support system. These features, however, do not dominate in telelearning. Many variations of telelearning can be found in terms of presence, telepresence, meeting in virtual spaces, interactivity with rich multimedia environments and extensive human interactions in a virtual world with no limits except access and language. Collaborative telelearning emphasises the collaborative interaction between students in a virtual world. By following Salomon’s recommendation for genuine interdependence, collaborative telelearning tasks need to create interdependencies between team/group members. In order to mange the interdependencies, team and group members will need to coordinate their efforts. Providing support for this coordination is one challenge for telelearning environments. 2.4 Coordination theory (from Wasson (1997), Wasson (1998) and Wasson & Bourdeau (1997)) Schmidt (1997) argues that the investigation of artefacts for coordinative purposes is a crucial and fertile area for CSCW (computer supported coopeartive work) research. For CSCW researchers, cooperation is 33 Project DoCTA: Design and use of Collaborative Telelearning Artefacts defined as communication plus coordination (Olson et al., 1993). For collaborative learning based on a definition of genuine interdependence, both are certainly essential. Add collaborative learning to a telelearning environment and this requirement for coordination is magnified. Coordination entails the meshing, allocating, relating, and scheduling of activities, actors and resources with respect to each other (Strauss, 1985; Schmidt, 1994). Furthermore, the coordination of group activities requires awareness information (Dourish & Bellotti, 1992). Awareness is “an understanding of the activities of others, which provides a context for your own activity (Dourish & Bellotti, 1992, p. 1)”. Schmidt & Simone (1996) identify the understanding of how computer systems can aid in reducing the complexity of coordinating cooperative activities as a major research issue in CSCW. They argue that this issue was identified and defined early in CSCW history by Holt (1985). He wrote “Whatever has to do with task inter-dependence — coordination — is left to the users to manage as best they can, by means of shared databases, telephone calls, electronic mail, replies to which multiple users have access, or whatever ad hoc means will serve (Holt, 1985)”. Carsten & Sørensen (1996) point out that although many have explicitly addressed and modelled coordination work (Holt, 1988; Malone & Crowston, 1990, 1994; Kaplan et al, 1992; Fitzpatrick et al., 1995), these studies have not been based on empirical studies of artefacts introduced in order to cope with the complexity of the coordination work the actors must handle. In their own field studies (Carsten & Sørensen, 1996) they found that actors themselves chose to increase the formalisation of work processes, despite the problems of increased rigidness and possible loss of control of the work. Mechanisms were invented and adopted to support coordination to deal with complexity. Malone & Crowston (1994) describe coordination theory as an emerging research area focused on the interdisciplinary study of how coordination can occur in diverse kinds of systems. Coordination theory provides a means for specifying (inter)dependencies between, and among, actors, goals, activities, and resources by identifying a dependency type (e.g., shared resource) and a coordination process (e.g., group decision-making) for managing the dependency. In their work, coordination is defined as managing dependencies between activities (Malone & Crowston, 1994). In particular, actors face coordination problems because of dependencies between their activities, and as such, Malone and Crowston have focused on dependence between activities. 34 Chapter 2 Theoetical foundations Drawing on ideas about activity coordination in complex systems from disciplines as varied as computer science, linguistics, psychology, economics, operations research and organisation theory, they present a first version of an analysis that characterises the basic processes involved in coordination. Table 3 gives a taxonomy of dependencies between activities and possible coordination processes for managing them. Table 3 Dependencies between Activities (from Malone & Crowston, 1994) Dependency Examples of coordination processes for managing dependency Shared resources “First come/first serve”, priority order, budgets, managerial decision, market-like bidding Task assignments (same as for “shared resources”) Producer/Consumer relationships Prerequisite constraints Notification, sequencing, tracing Transfer Inventory management (e.g., “Just In Time”, “Economic Order Quality”) Usability Standardisation, ask users, participatory design Design for manufacturability Concurrent engineering Simultaneity constraints Scheduling, synchronisation Task / Subtask Goal selection, task decomposition For example, two activities are dependent if they share a resource. Resources include actors, tools or information. A first come/first serve coordination process would allow the first to ask for a resource (be it, for example, an actor or a computer processor) to receive the resource. Another possibility is that a managerial decision decides who gets the resource, or a marketlike bidding process takes place. Task assignment is a special case where an activity needs to be assigned to an actor and the same coordination processes that applied for a shared resource apply here. Another example is that two activities are dependent if there is a prerequisite constraint between them. In this case some kind of explicit sequencing and tracking processes are needed. The usability dependence implies that the product of one activity must be useable by the consuming activity and this can be ensured by a standardisation process. A simultaneity constraint dependency exists if two activities need to occur at the same time. For example a meeting requires scheduling. 35 Project DoCTA: Design and use of Collaborative Telelearning Artefacts The task/subtask dependency often surfaces when a goal is decomposed into subgoals or activities. Coordination process goal selection is choosing a goal while goal decomposition is the process of choosing activities. Such a taxonomy of dependencies and related coordination processes serves several purposes. First, the taxonomy serves as a way to organise coordination mechanisms. This will enable others to determine if identified dependencies or coordination mechanisms have already been recognised, and if not, to add the new dependency type or new coordination mechanism. Second, if you can identify a dependency, then you can consult the table to find out what coordination process is needed to manage that dependency. The coordination process itself can be broken down into a number of activities as is shown in table 3 where the generic resource assignment coordination mechanism is represented as a four step process comprising: identify necessary resource, identify available resources, choosing resources and marking resources in use (i.e., assign resource). Although Malone & Crowston (1990, 1994) include actors in their theory, the major emphasis is on the role of actors as a resource (an actor is assigned to an activity). The coordination processes they identify deal with managing the dependencies where it is a goal to minimise the number of dependencies in a complex system. In collaborative telelearning, on the other hand, interdependence between learners is desirable4 , and we design tasks that will create actor interdependence. Wasson & Bourdeau (1997) report that viewing collaborative telelearning from a coordination theory perspective offers a means of understanding the inter-relationships between actors and entities and how these relationships can and should be supported. Adopting Salomon’s ideas about genuine interdependence and a coordination science approach, they modelled (inter)dependencies between actors in collaborative telelearning scenarios and they have extended the definition of coordination to be managing dependencies between activities (Malone & Crowston, 1994) and supporting (inter)dependencies between actors. Wasson (1997, 1998) proposes a set of actor (inter)dependencies and related coordination processes for collaborative telelearning. 4 36 Remember Salomon’s arguments that genuine interdependence is need for successful CSCL Chapter 2 Theoetical foundations Table 4 presents an initial taxonomy of (inter)dependencies between actors and gives examples of coordination processes for supporting the (inter)dependencies. Both collaborative interdependence and competitive interdependence can manifest themselves through either a shared goal or a shared activity. Where they differ the most is in the coordination processes required for managing them. Note that several of the coordination processes are found in Salomon’s definition of genuine interdependence Table 4 Dependencies between Actors Dependency Examples of coordination processes for managing dependencies Collaborative Interdependence shared goal team building shared activity team assignment; team building; division of labour; sharing of information, conceptions and conclusions; joint thinking; group decision making; planning; task assignment; scheduling; synchronisation; communication Competitive Interdependence shared activity scheduling; communication; monitoring; competition assignment shared resource same as in Table 3 The importance of this work for DoCTA lies in making sure the technological environment within which the students carry out their tasks provides mechanisms that make coordination as effortless as possible. Otherwise coordination issues can become a bottleneck that hinders students from carrying out their individual and collaborative work. Wasson and Bourdeau’s extended definition is particularly important in the view of collaborative telelearning as supporting a distributed collaborative learning community where the aim is to support the learners in creating that community. 2.5 Awareness In 1992, CSCW researchers Dourish and Bellotti introduced the concept of awareness and defined it as an “understanding of the activities of the others, which provides context for your own activity. (Dourish & Bellotti, 37 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 1992, p. 107)”. Since that time it has been a concept that has received a lot of attention (for example see Gutwin et. al., 1995; Bellotti and Bly, 1996; Palfreyman and Rodden, 1996; Roseman and Greenberg, 1996). Gutwin et al. (1995) present a framework of awareness for collaborative learning which comprises four types of awareness: social awareness, task awareness, concept awareness and workspace awareness. Social awareness refers to awareness of the social connections within the group or team (e.g., What role will I take in the team, what roles will the other members take). Task awareness concerns awareness about how to complete the common task (e.g., What do we know about the task, how much time do we have, what steps must we take to complete the task). With concept awareness the individual student is aware of how a particular activity or piece of knowledge fits into his or her own existing knowledge. Workspace awareness is an up-to-the-minute knowledge of the other students’ interactions with the workspace (e.g., what are they doing, where are they, what has been done so far, what is left to do). They suggest that social awareness, being inter-personal, is best supported implicitly by providing communication opportunities for team members to negotiate their roles. Task and concept awareness are often supported by providing explicit scaffolds to assist with organisation and helping stay on task. Workplace awareness is maintained by providing “tracking information such as other learners’ location in the shared workspace, their actions, the interaction history and their intentions (Gutwin et al., 1995, p. 147). In DoCTA, Gutwin et al.’s framework has been used by the students participating in VisArt to evaluate whether TeamWave Workplace supports his categories of awareness. 2.6 Distributed collaborative learning communities The final area providing inspiration to project DoCTA is summarised as follows “…a distributed collaborative learning community is a ‘place’ that is created by the individual students through their individual and collective actions, …The designers’ role is to support the students’ work of creating that community, and in such a way that the computer systems become integrated parts of the students’ activity (Fjuk, 1998, p. 70)”. 38 Chapter 2 Theoetical foundations Furthermore, Fjuk (1995) concludes that collaborative telelearning applications need to have both a mediating role between the individual learner and the peer-students and between the individual learner and her learning tasks. Thus, collaborative telelearning can be understood as a medium for inter-human interactions and articulation of individual work. As designers of DoCTA learning tasks and the technological support environments, we must keep in mind that our role is to provide a supporting environment that makes coordination, communication and collaboration as transparent as possible (Bourdeau & Wasson, 1997) enabling students to create their own learning community. The environment must also support both individual and collaborative work. 39 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 40 Chapter 3 Conceptual framework for evaluations 3 Conceptual framework for evaluations FRODE GURIBYE The underlying conceptual framework adopted in DoCTA is taken from three different, although closely interrelated approaches, namely: activity theory (Leontev, 1978, Engeström 1987), distributed cognition (Hutchins, 1995), and situated action (Suchman, 1987, Lave, 1988, Mantovani, 1996). One of the goals of Guribye’s (1999; Wasson & Guribye, 1999) research5 is to argue that, together, these approaches make up a rich framework for describing, evaluating and analysing collaborative telelearning scenarios. All three approaches underscore the need to look at real activities in real situations (Nardi, 1996), and always, in some way, include the context in studies of human activity. The rationale for combining these three approaches as the conceptual foundation of this study, is that they all fall under what is called a sociocultural perspective (Wertsch, del Río & Alvarez, 1995), that highlights learning and thinking as phenomenon that cannot be studied in isolation. Rather, they are complex processes situated or distribute in an environment — it is impossible to separate them from the context in which they occur. The different approaches each emphasise slightly different elements of the framework that is important to be aware of in these kinds of studies. Situated action emphasises the emergent, contingent nature of human activity, the way activity grows directly out of the particularities of a given situation (Nardi, 1996). Distributed cognition on the other hand, asserts as a unit of analysis a cognitive system composed of individuals and the artefacts they use (Hutchins, 1991, Nardi, 1996). This approach 5 This work began with his Masters research (Guribye 1999) and is being continued in his Dr.Polit. 41 Project DoCTA: Design and use of Collaborative Telelearning Artefacts underscores the distributed nature of cognitive processes, and the role that different artefacts play in these processes. Activity theory also emphasises the mediating role of artefacts, but stresses that these artefacts carry with them a particular culture and history, thus, focuses on the institutional and cultural elements involved in the learning activity (Kuutti, 1996). In this report we confine our focus to activity theory. Activity theory (AT) builds on a solid intellectual heritage (see e.g., Vygotsky, 1978; Zinchenko, 1995), provides extensive theoretical and conceptual resources, places emphasis on the mediating role of artefacts (including ICT), and offers comprehensive and flexible models to account for and analyse the dynamics and complexity of human activity, action and the context where it is situated. In addition, AT provides theoretical suggestions about general principles underlying human activity that can be used to inform and enlighten the analysis. This section begins by reviewing Vygotsky and cultural-historical psychology that provided the foundations for AT. Then the basic principles of AT are presented followed by a short description of some of the methodological assumptions on which AT studies are based. Finally the section closes with a discussion of the implications of AT for the DoCTA project. The majority of the writings in this section are taken from Guribye (1999), Guribye & Wasson (1999) and Mørch & Wasson (1999). 3.1 Vygotsky and Cultural-Historical Psychology The sociocultural perspective has its theoretical heritage in the ideas of the Russian psychologist, semiotican and pedagogical theorist Lev Vygotsky and his followers (Wertsch, del Río & Alvarez, 1995). This research tradition, referred to as Cultural-Historical Psychology6 , appeared in the former Soviet Union in the nineteen twenties and thirties. Vygotsky, his students and his colleagues were committed to formulating a psychology grounded in the ideas of Marx and Engels (Cole & Scribner, 1978). Thus, the task at hand was to create a theory that was a “psychologically relevant application of dialectical and historical materialism (Ibid, p. 6)”. In formulating a cultural-historical theory of 6 42 The term ‘Cultural-Historical Psychology’ is used when referring to the heritage form Vygotsky, Leontev, Lauria and other Soviet psychologists. When dealing with the appropriation of this tradition in contemporary work and debates (especially in the West) the term ‘Sociocultural’ is applied (See Wertsch, del Río & Alvarez, 1995; Zinchenko, 1995). Chapter 3 Conceptual framework for evaluations higher mental processes, Vygotsky rejected the dominant perspectives and traditions (e.g., behaviourism) in psychology at his time (Vygotsky, 1978; see also Minick, 1996). A brief presentation of some fundamental concepts in the Vygotskian framework is given in the following subsections Internalisation A central tenet in this framework is the social nature of individual development, including higher mental functioning. According to Vygotsky many mental functions appear on two planes; first interpersonally (or inter-psychological) on a social level through social interaction, and later intrapersonally (or intra-psychological) on an individual level (Vygotsky, 1978). This is known as the genetic law of cultural development. The process in which the linking of these levels or where “the internal reconstruction of an external operation (Ibid. p. 56)” occurs, is called internalisation. In light of the dichotomization of “the inner” and “the outer” in cognitive psychology, the concept of internalisation has somehow become a controversial one (see e.g., Marton & Booth, 1997; Rogoff, 1990). This debate seems to question whether the term internalisation presupposes a rather strict distinction between mind and environment and thus is subject to the ontological stance often referred to as Cartesian Dualism. Leading theorists within the sociocultural perspective argue that the term is compatible with a non-dualistic view, and interpret internalisation as “simultaneously a social and an individual process (John-Steiner & Mahn, 1996, p. 10)”. This debate is not elaborated on further in this discussion, and the internalisation process is primarily seen as a confirmation of the view that human thinking and learning are, to a large extent, social processes. Following Vygotsky’s genetic law of cultural development, Säljö (1996) argues that “categories used between people in discursive practices to account for the world (inter-psychological categories), will appear as tools for thinking within individuals (i.e. intrapsychological categories) (p. 84)”. This brings us to the next central idea in the Vygotskian framework, semiotic mediation. The mediating role of artefacts As a further elaboration on the interdependency between the social and individual planes, Vygotsky heavily emphasises mediation and the role of tools in human practices. These tools can be both technical and psychological: 43 Project DoCTA: Design and use of Collaborative Telelearning Artefacts “The most essential feature distinguishing the psychological tool from the technical tool, is that it directs the mind and behaviour whereas the technical tool, which is also inserted as an intermediate link between human activity and the external object, is directed toward producing one or other set of changes in the object itself (Vygotsky, quoted in Daniels, 1996, p. 7)”. The concept of mediation reflects that human action, on both individual and social planes, is mediated by tools and signs (the latter corresponds to psychological tools) (John-Steiner & Mahn, 1996). In other words, “tools serve as mediational means, i.e. they – metaphorically speaking – stand between the individual and the world (Säljö 1996, p 84)”. This process is depicted in figure 2. Figure 2 Vygotsky’s mediating triangle (Vygotsky, 1978) Opposing Pavlov’s stimulus-response model of human behaviour, Vygotsky introduced a mediating link between the stimulus and its response (ÅbergBengtsson, 1998). Though, this should not be interpreted as a mere extension of Pavlov’s model. All the higher psychological processes are mediated through a tool (i.e. via X in figure 2), but in a “much more sophisticated form than that shown in the figure (Vygotsky, 1978, p. 40)”. One of the most important psychological tools is language, which serves as the “prime device for rendering the world intelligible and for communicating our intentions to others (Säljö, 1996, p. 84)”. Other examples of mediational tools include: systems for counting; algebraic 44 Chapter 3 Conceptual framework for evaluations symbol systems; diagrams; maps; and a more recent invention, the computer (John-Steiner & Mahn, 1996). More generally, the theme of mediation as formulated by Vygotsky plays an important role in the sociocultural perspective in that it provides “the link or bridge between concrete actions carried out by individuals and groups, on the one hand, and cultural, institutional, and historical settings, on the other (Wertsch, del Río & Alvarez, 1995, p. 21)”. Zone of proximal development Discussing the relation between learning and development, Vygotsky introduced the concept zone of proximal development (ZPD), which he described as: “the distance between actual developmental level as determined by independent problem solving and the level of potential development as determined through problem solving under adult guidance or in collaboration with more capable peers (Vygotsky, 1978, p. 86)”. This concept helps to explain the way that social and participatory learning takes place (John-Steiner & Mahn, 1996), and can be seen as an argument for collaborative learning. Wells (1996) claims that in light of the ZPD, both the teacher’s role in learning and the role of collaboration in learning are viewed in a new and different way. Since Vygotsky first applied this concept, many different interpretations have been suggested (see e.g. Lave & Wenger, 1991; 1996). As this is not a central concept (except as a theoretical foundation of collaborative learning) for the analysis that is presented in this research, it will not be elaborated any further. Having briefly presented some of the essential ideas of Vygotsky and cultural-historical psychology, focus is now turned to his student and colleague A. N. Leontev who played a crucial role in the development of activity theory as it is known today. 3.2 Basic principles of Activity Theory In the following paragraphs, some of the ideas that constitute the body of knowledge referred to as activity theory (AT) are presented. Activity theory 45 Project DoCTA: Design and use of Collaborative Telelearning Artefacts emerged from cultural-historical psychology7 where, as pointed out above, Vygotsky was a central figure. Vygotsky himself never explicitly examined the concept of activity, but strongly influenced the development of activity theory (Wertsch, 1981). As Kuutti (1996) points out, activity theory is not a theory per se, rather it is “a philosophical and cross-disciplinary framework for studying different forms of human practices as developmental processes, with both individual and social levels interlinked at the same time (p. 25)”. The main interest in AT in this project is that it provides a “powerful sociocultural lens through which we can analyse most forms of human activity (Jonassen & Rohrer-Murphy, 1999, p. 2)”, and that it is suggested as an alternative framework for HCI research (Nardi, 1996), including CSCW and CSCL. Unit of analysis A central tenet in the AT framework is that of unity of consciousness and activity (Kaptelinin, 1996). The idea is that it is through interaction with the environment that the consciousness or the mind as a whole emerges and comes into existence. This interaction, i.e. the activity, is socially and culturally determined. To understand human praxis and consciousness (e.g. thinking and learning) the proper unit of analysis, according to AT, is activity. The concept of activity always includes a minimal meaningful context for understanding individual actions (Kuutti, 1996). According to Engeström (1987), activity “is the smallest and most simple unit that still preserves the essential unity and integral quality behind any human activity (p. 81)”. In focusing on activity as the basic unit of analysis, emphasis is put on the cultural, institutional and social settings in which these activities occur. One can thus argue that AT also provides the necessary conceptual resources for capturing essential elements of these settings. The activity system A common reformulation of Vygotsky’s mediational triangle (as depicted in figure 2) is shown in figure 3. 7 46 See Zinchenko (1995) for a further treatment of the relation between culturalhistorical psychology and the theory of activity. Chapter 3 Conceptual framework for evaluations Figure 3 The basic mediational triangle (Cole & Engeström, 1993) As Cole & Engeström (1993) explain, this model “although useful as schematic ‘minimal structures’ of human cognitive functions”, fails to “account for the collective nature of human activities (p. 7)”. According to Leontev (1978) an activity, or an activity system as he called it, consists of a subject, the object of the activity, and the community in which the subject is constituted, thus stressing the collective nature of activity. The relations between these three central components of an activity are mediated in a reciprocal way (Kuutti, 1996). The relationships between subject and community are mediated by the group’s collection of artefacts on the one hand, and by the community’s rules of interaction on the other. The intermediate term introduced to describe the mediational relation between community and object is ‘division of labour’ and should be understood as “the continuously negotiated distribution of tasks, powers and responsibilities among the participants of the activity system (Cole & Engeström, 1993)”. The activity system, as modelled by Engeström (1987), presented in figure 4, is an expanded version of the mediational triangle to include the collective dimension. 47 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 4 Engeström’s (1987) model of the activity system The model is slightly simplified for the sake of clarity. All the elements (including the “mediators”) of the activity are actually connected, forming a systemic whole (Kuutti, 1996). All the terms should be conceived in their broad sense: “A tool can be anything used in the transformation process, including both material tools and tools for thinking. Rules cover both explicit and implicit norms, conventions, and social relations within a community. Division of labour refers to the explicit and implicit organisation of a community as related to the transformation process of the object into the outcome. Each of the mediating terms is historically formed and open for further development (Kuutti, 1996, p. 28, italics in original)”. Activity can be understood as a form of doing, a social or cultural praxis, and transforming an object into an outcome is what motivates an activity. Individuals usually participate in several activities simultaneously, and these activities are distinguished from each other according to their objects (Kuutti, 1996). 48 Chapter 3 Conceptual framework for evaluations Further elaborations of the ideas that animate activity theory are organised into a set of five basic principles (Kaptelinin & Nardi, 1997). These principles are presented separately in the following subsections, but should be comprehended in a holistic way: “These basic principles of activity theory should be considered as an integrated system, because they are associated with various aspects of the whole activity. A systematic application of one of these principles makes it eventually necessary to engage all the others. For instance, understanding the hierarchical structure of an activity requires an analysis of its object or motive, as well as developmental transformations between actions and operations, and between internal and external components. The latter, in turn, can critically depend on the tools used in the activity (Kaptelinin & Nardi, 1997, p. 3, italics in original)”. Object-orientedness The principle of object-orientedness was identified by Leontev when he drew the connection between an activity realised by a subject and the objects of the external world (Stetsenko, 1990). The concept of “objects of the external world” is not identical with the concept of “thing”: “Whereas a thing may be regarded as any element, any aspect of reality characterised by spatiotemporal physical determinateness, by ‘object’ … we understand a form embodying the sociohistorical experience of mankind. An object is the vehicle of this experience and embodies a specific aspect of human social practice; it is the form in which a physically defined thing functions as people go about their life activities in society. The essence of an object functioning in the social process is constituted not by its physical properties, but by the specific connections and relations that become known in the process of collective activity (Stetsenko, 1990, p. 59)”. The notion of object in activity theory is not limited to physical, chemical or biological properties, but in its wide sense, embracing both socially and culturally determined properties (Kaptelinin & Nardi, 1997). As Jonassen & Rohrer-Murphy (1999) put it, “the object of an activity can be anything, so long as it can be transformed by subjects of the activity 49 Project DoCTA: Design and use of Collaborative Telelearning Artefacts system. Objects might be physical objects, soft objects, or conceptual objects (p. 8)”. In short, an activity is oriented or directed towards an object, which is held by the subject (a person or a group engaged in the activity). This “object (in the sense of ‘objective’) motivates activity, giving it a specific direction (Nardi, 1996, p. 73)”. Hierarchical structure of activity According to Leontev (1978) activities are organised into three hierarchical levels. These levels are based on a very functional criteria – what is a subordinated level (e. g. action, see below) in one activity can be considered the entire activity in another situation. The levels Leontev differentiated between are activities, actions and operations. Kuutti (1996) gives a comprehensible description of these levels: “Activities are longer-term formations; their objects are transformed into outcomes not at once but through a process that typically consists of several steps or phases. There are also need for shorterterm processes: activities consist of actions or chains of actions, which in turn consist of operations (p. 30)”. People engage in activities to fulfil motives of which they may or may not be consciously aware (Kaptelinin & Nardi, 1997). To realise these activities certain actions must be performed. These actions are directed towards a conscious goal, and are related to one another by the same overall objective. The activities form a frame of reference within which the individual actions can be understood. Actions, in turn, are composed of functional sub-units called operations. These operations are automatic processes that are routinised and unconscious. Unlike actions, they are not directed toward a goal, but are carried out automatically, providing an adjustment of actions to the current situation and the prevailing conditions. Leontev offers a classical example of the action-operation dynamics: “When learning to drive a car, the shifting of the gears is an action with an explicit goal that must be consciously attended to. Later, shifting gears becomes operational and can no longer be picked out as a special goal-directed process: its goal is not picked out and discerned 50 Chapter 3 Conceptual framework for evaluations by the driver. Conversely, an operation can become an action when conditions impede an action’s execution through previously formed operations (Leontev, cited in Kaptelinin & Nardi, 1997 p. 2)”. The distinction between both operations and actions, and actions and activities are blurred – there are no firm boarders – movements between the hierarchical levels in both directions are possible (Kuutti, 1996). The hierarchical structure of activity are depicted in figure 5. Figure 5 The hierarchical structure of activity (Leontev, 1978) Internalisation/Externalisation Elaborating on Vygotsky’s concept of internalisation, activity theory differentiates between internal and external aspects of activities. “The subject and the object of an activity are in a reciprocal relationship with each other: the subject is transforming the object, while the properties of the object penetrates into the subject and transform him or her (Kuutti, 1996, p. 32)”. In this internalisation, as Leontev notes, “processes are subjected to a specific transformation: they are generalised, verbalised, abbreviated and, most importantly, become susceptible of further development which exceeds the possibility of external activity (Leontev, cited in Kuutti, 1996, pp. 32-33)”. According to AT, the internal “mental” processes can not be understood without reference to external processes, and thus AT rejects the dualistic concept of an isolated, independent “mind” (Kuutti, 1996). The internal processes are formed at a social level in collaboration and interaction with others (or external artefacts). The other way around, “mental processes manifest themselves in external actions performed by a person, so they can be verified and corrected, if necessary 51 Project DoCTA: Design and use of Collaborative Telelearning Artefacts (Kaptelinin, 1996, p. 109)”. Thus, externalisation is important in collaboration between people. In order to co-ordinate their activities, they must be performed externally (Kaptelinin & Nardi, 1997). Artefacts and mediation As stressed by Vygotsky (1978), artefacts mediate or alter the nature of human activities. All activities contain different artefacts, and these artefacts or tools embody a certain history and culture. “Artefacts themselves have been created and transformed during the development of the activity itself and carry with them a particular culture – a historical residue of that development (Kuutti, 1996, p. 26)”. Thus, artefacts are carriers of cultural knowledge and social experience. This experience and knowledge is manifested in both the structural properties of the tool, and in the way the tool should be used (Kaptelinin & Nardi, 1997). The structural properties are linked to Gibson’s (1979) term affordances that “refers to the perceived and actual properties of a thing, primarily those functional properties that determine just how the thing could possibly be used (Pea, 1993, p. 51)”. Activity theory stresses that an artefact first comes fully into being when it is actually used. Hence, knowing how to use it is an essential part of the artefact (Kaptelinin & Nardi, 1997). The concept of tool mediation implies an asymmetrical model of people and things (e.g. the computer) unlike that of cognitive science (Nardi, 1996), and at the same time invites a serious study of how artefacts are an inseparable and integral part of human activities. Development Activity theory perceives activities not as given or static, but as dynamic processes under continuous change and development (Kuutti, 1996). Thus, to fully understand an activity, its history and development should be taken into consideration. This development occurs at all the different levels in the activity: “New operations are formed from previous actions as participants’ skills increase; correspondingly, at the level of actions the scope of new actions is enlarging, and totally new actions are being invented, experimented with, and adapted as responses to new situations or possibilities encountered in the process of transforming the object. Finally, at the level of activity the object/motive itself (and the whole structure of activity 52 Chapter 3 Conceptual framework for evaluations related to it) is reflected, questioned, and perhaps adapted, reacting to larger changes and other activities (Kuutti, 1996, pp. 33-34)”. In addition, all the elements of an activity are subject to continuous development. These dynamics underlying the activity are represented in figure 6. Figure 6 The nested nature of activity dynamics (Engeström 1987) Activities are not isolated units, “but more like nodes in crossing hierarchies and networks, they are influenced by other activities and other changes in their environments (Kuutti, 1996, p. 34)”. This also emphasises the collective nature of activities, and Engeström describes these activities as “systems of collaborative human practice (Engeström, 1987) ”. A major driving force in the development and change of activities are, according to Engeström (1987; 1991) contradictions. These multileveled contradictions can exist both between and within activities, and will “manifest themselves as problems, ruptures, breakdowns, clashes (Kuutti, 1996, p. 34)”. Viewing activities as developing processes also reveals certain methodological implications. For example, in order to understand how 53 Project DoCTA: Design and use of Collaborative Telelearning Artefacts an artefact is used, one has to study its use over time allowing for the usage to develop (Kaptelinin & Nardi, 1997) and not for example in a single one-time laboratory experiment. 3.3 Methodological Issues Having discussed the five basic principles of activity theory, what remains is to take a closer look at some of the methodological assumptions on which activity theory studies are based. Vygotsky writes: “The search for methods becomes one of the most important problems of the entire enterprise of understanding the uniquely human forms of psychological activity. In this case, the method is simultaneously prerequisite and product, the tool and the result of the study (Vygotsky, 1978, p. 65, italics in original)”. One of the most important methodological commitments of contemporary8 activity theory studies are to examine activities in situ – in the environment where it “naturally” occurs. Nardi (1996) summarises the methodological approach in activity theory into a set of characteristics (p. 95): • A research time frame long enough to grasp the objects of the activity. • Attention to, first, broad patterns of activity, then (within the context of the activity as a whole) more narrow episodic fragments can be analysed. • The use of a varied set of data collection techniques including interviews, observations, video and historical materials. • A commitment to understanding things from the users’ points of view. 3.4 Some general remarks on AT Activity theory was mainly developed within the confines of the former Soviet Union, and has only more recently been increasingly adapted by western (Anglo-American) researchers (see e.g., Wertsch, 1998; 8 54 See Cole & Engeström (1993) for an account of the methodological assumptions of Vygotsky, Leontev and Lauria. Chapter 3 Conceptual framework for evaluations Engeström, Miettinen & Punamäki, 1999). Some research traditions in the west have followed “similar avenues of thought (Kuutti, 1996, p. 25)”. It is possible, for example, to trace parallels between AT and Dewey’s pragmatism (Kuutti, 1996, see also Bredo, 1994) and Mead’s (1968) symbolic interactionism (see Holzman, 1996 for an extensive treatment). An approach closely related to AT is that of Distributed Cognition (cf. Salomon, 1993; Hutchins, 1991; 1995). Nardi (1996) notes, rather informally, that: “activity theory and distributed cognition are very close in spirit … and it is my belief that the two approaches will mutually inform, and even merge, over time, though activity theory will continue to probe questions of consciousness outside the purview of distributed cognition as it is presently formulated (p. 89)”. This is also apparent in Cole & Engeström’s (1993) article “A culturalhistorical approach to distributed cognition”, where they use AT to analyse different notions of distributed cognition. An important feature in the distributed cognition approach is the distribution of cognitive activities between the human mind and the environment or context in which it operates. This implies that you cannot analyse the human cognition and the way in which it is functioning in isolation, but that one has to see intelligence as a feature of both the individual’s mind and the different physical and mental artefacts that are a part of the particular cognitive phenomena of human activities (see e.g, Pea, 1993). The focus is on the mediational means of the artefacts, and the interaction that these artefacts mediate between individuals and the context or environment. Hutchins (1995) argues in his book “Cognition in the Wild” that the boundary between inner and outer, between individual and context, should be softened. According to Hutchins, a more fruitful unit of analysis than the cognitive system of an isolated individual, is to focus on the individual located and actively engaged in a culturally constructed world ¾ thus, leaving us with only a subtle difference in respect to activity theory. 55 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 3.5 Activity theory and design Our experience (Mørch & Wasson, 1999) with AT is that it provides a lens through which we can see the world. It is useful to orient thoughts and research questions. It provides a number of methods/tools shaped by a general theoretical approach. It can be used to guide methodological decisions regarding evaluation (how technology is used) and to a lesser extent regarding design (how technology will be used). In fact, from an Activity Theory perspective we should always talk about redesign because we are “redesigning” work/learning praxis. Adoption of these approaches provides a strong and fruitful conceptual framework that informs both the design and evaluation of collaborative telelearning scenarios. In order to study human activity and evolution of technology in DoCTA, we take as a starting point the “use of technology” and not the technology per se (Mørch & Wasson, 1999). Our working hypothesis is that the interplay between change of technology and change in activity (through the use of technology) is significant. We thus design our case studies to explore this interplay in various ways. Activity theory reminds us to always focus on context and suggests the articulation of the design of artefacts as “change in activity”. From this perspective activities evolve and change and artefacts evolve and change, thus mutually influencing each - but not in the same way. To operationalise this view and to articulate and distinguish activities and artefacts we integrate Activity theory with a design perspective. We see artefact evolution as continual redesign of ready-made application units (Norman, 1990; Mørch, 1995). Furthermore, the evolution of these artefacts needs to be informed by the practice carried out by users when they use the artefacts in genuine human activity. Otherwise, the redesign will be of little practical value. The rationale for this is that evolution of artefacts needs to be informed by the practice carried out by users when they use the artefacts in genuine human activity. Otherwise the redesign will be of little practical value. The diagram in figure 7 is an attempt to put Activity Theory in such a context in order to help us focus our discussion. 56 Chapter 3 Conceptual framework for evaluations Figure 7 Integration of a design perspective with Activity Theory (Mørch & Wasson, 1999) From evolution of artefacts to meaningful tasks (and back again) Figure 7 can be seen as an extension of Carroll and Rosson’s task-artefact cycle developed in HCI (Carroll et al., 1991). The task-artefact cycle describes the interdependence of tasks and artefacts in human activities. There is a reciprocal relationship between the two. When tasks change, artefacts tend to change as well. Carroll and Rosson give as an example the evolution of writing tools. When the typewriter was replaced by the word processor this was not only an evolutionary change in technology — it also changed the task of writing. For example, with a word processor it is now possible to correct a misspelled word without the reader ever knowing that it was once misspelled. What consequences this has for the evolution of groupware and whether or not we can find similar or other kinds of relationships between the other components in the diagram is something we are exploring. 57 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 58 Chapter 4 Evaluation of Groupware Systems 4 Evaluation of Groupware Systems HEGE HIGRAFF The focus in this thesis (Higraff, in preparation) has been on the evaluation of four different groupware systems for use in collaborative telelearning. The evaluation has addressed usability, the interface, and the relative usefulness of each system. The four groupware systems are: TeamWave Workplace (TW), Lotus LearningSpace (LearningSpace), Habanero (H) and Basic Support for Co-operative Work (BSCW). Each system was used by a geographically dispersed pair of students9 to carry out a collaborative telelearning task. Since these groupware systems have very different capabilities, it was quite difficult to find a task that could serve as a benchmark. The four student pairs were asked to design UML diagrams for a dental office information system (this will be explained in more detail in the section on the scenario design). Since two of the systems did not offer a drawing tool, however, the student pairs collaborating through LearningSpace and BSCW used Microsoft Paint to draw their diagrams. The diagrams could then be shared through the system (e.g., by posting the diagrams as messages in BSCW). 4.1 Overview of the four groupware systems Habanero Habanero is a framework developed by NCSA (National Center for Supercomputing Applications), University of Illinois, to help developers design shared applications in an easier fashion than building from scratch. 9 The 8 volunteer were undergraduate students from the Department of Information Science at the University of Bergen. They had no experience with any of the systems. 59 Project DoCTA: Design and use of Collaborative Telelearning Artefacts The framework is supposed to help the users to reuse existing infrastructure for communication and toolkit development. The programmers will be able to develop new applications from examples, or to develop new ones from the application framework. The framework itself provides the co-operation functions (Chabert et. al, 1996). The goal for Habanero is “to provide a cross-platform, generic and extensible synchronous collaborative framework to cover a wide range of collaborative functionality’s for different user communities. (Chabert, 1997, p. x)”. Habanero supports only synchronous co-operation.. It is therefore primarily a real-time system where everybody collaborates at the same time. It does not support asynchronous tasks. Habanero is implemented in Java, which is supposed to make it platformindependent. Also the fact that Java is object-oriented gives room for reuse and code-inheritance. The fact that Java is object-oriented is not important unless one actually has the source-code, and this is understandable primarily to programmers. Users of the applications created with Habanero will typically not be interested in the source code. The original tools include (NCSA Habanero, 1998): • Whiteboard — drawing-tool • Telnet — for remote login on different machines • Savina — web browser • Voting Tool — to vote for different proposals • Chat — text-based chat • Audio Chat — voice-based chat (only for Solaris on Sparc-machines, for now) • mpEdit — viewing and editing text-files During the summer of 1999, four new tools have been. These are: • Habanero Neighbourhood — distributed file-system for file-sharing • ClipNShip — download and use other user’s files, without leaving the session • The GIS Viewer — viewing and manipulating geographic information • The Virtual Network Computing (VNC) — allows remote control of any system running a VNC server Habanero is an application framework — not a complete application. Applications are built by extending the framework by adding application60 Chapter 4 Evaluation of Groupware Systems specific subclasses. One can develop new tools on top of this framework, and one can develop new tools from scratch. There are also examples of existing tools that can be reused and modified. The tools can also be removed, if one finds them of little value. Thus, Habanero can easily be adapted to different purposes. For the purposes of this evaluation study, Habanero is tailored for individual use. LOTUS LearningSpace Lotus LearningSpace (LearningSpace) is a groupware system developed by Lotus Development Corporation. It consists of a platform with different programs attached to it. One can work synchronously through LearningSpace Live, or asynchronously through LearningSpace Forum. A combination of the two Lotus products would give the most flexible environment. The educational goal of LearningSpace is that learning should be possible anytime and anywhere, and they aim to produce products that are innovative, responsive, productive, and competent (LDC, 1998a). In LearningSpace it is possible to do assignments, run tests and exams, as well as collaborate on different levels. The groupware system is flexible enough for people who would like to co-operate with others, but who do not have the time to participate in an ongoing collaboration which demands 100% real-time co-operation. LearningSpace also provides a kind of database in which one can post messages, articles etc. Lotus LearningSpace comprises the following tools: • Schedule — course-outline and tasks from the instructor • MediaCenter — all material for course and tasks • CourseRoom — interactive collaboration environment • Profiles — “homepage” for each student • Assessment Manager — instructors “home-directory” • Application Sharing — instructor can fetch windows-applications • IP Audio and Video — communication-means for use in the ”classroom” It is also possible to do adapt LearningSpace. Two supplemental tool packages are provided for developers and administrators (LDC, 1998b). These are: • Customise — tools to modify the look and feel of LearningSpace and to build custom functions and modules. New features provide unique customisation for different courses on a single LearningSpace server. 61 Project DoCTA: Design and use of Collaborative Telelearning Artefacts • Central — tools to manage the creation, installation, configuration and student enrolment/access processes of LearningSpace courses. In this evaluation study, only LearningSpace Live, the synchronous environment, will be used. BSCW Shared Workspace system BSCW, developed by GMD (German National Research Center for Information Technology), is a groupware system which can either run on a local server, on the web, or on the producer’s (GMD) server. BSCW enables collaboration over the Internet. It is a shared work environment that supports uploading of documents, notification of events (such as arrival of new messages), group-management, etc. The only software needed for using the system is a standard web-browser (GMD, 1999). BSCW is built on the “shared workspace” metaphor. This means that the user has the opportunity to save information in such a way that other members of the group also have access to it. This means each group member has the same permissions to edit, post or delete any document. In many ways it is similar to Lotus LearningSpace. In both groupware systems you post messages on a page (either on web or on a local server) for everybody to read. The primary goal of the BSCW project was to construct a platform that provides the elements needed for collaboration in geographically dispersed groups. They attempt to do this independently of the different infrastructures the users have on their desktop, network or hardware. The project focuses on how to help users collaborate on writing documents. To this end the system supports version-control and both new and old version of the documents are accessible. To prevent several people editing a document at the same time, a lock mechanism is provided (Bentley et. al, 1997). BSCW does not support chating or drawing/painting. Since the task performed in this evaluation study required drawing, the student pair was allowed to use MS Paint in addition to BSCW. 62 Chapter 4 Evaluation of Groupware Systems TeamWave Workplace TeamWave Workplace (TW) is the groupware system based on a room metaphor. It supports both synchronous and asynchronous communication between the participants. According to the developers, TW is a CSCW (computer supported co-operative work) system “designed to help people work together. It is versatile enough to support telecommuting and telework, distance learning, case-based learning, and corporate training (TeamWave Workplace, 1999)”. TW runs on a variety of platforms including Macintosh, Windows95 and Unix (AIX, Sun, Linux). In TW rooms are created for different purposes. Rooms can be created by the instructor or by the participants themselves. A group might create a room where they will collaborate on a task, or an instructor might create a room where assignments can be left and messages can be posted. Access to each room is regulated by its creator. For example, there could be individual access (to a single user), group access (for a defined group of users), or access for everyone. TW provides the participants with the following tools: • Address Book — shared address book • Brainstormer — to generate ideas; ideas are posted in the same sequence as they are suggested • Calendar — shared calendar for making appointments • Concept Map —organising concepts and ideas in a visual interface • Database — a simple database • Doorway — shortcut connection to the other rooms on the server • Fileholder —shared file “container”; files can be stored, collected and opened by anyone • File Viewer — view text simultaneously with other participants • Image Whiteboard — pictures can be opened and edited • Meeting Roster — to arrange meetings • Message Board — newsgroup-like discussions; placed on the Whiteboard • Postit — like yellow post-it notes for exchanging messages and leaving reminders • Slide Tool — slide-show with slides made in MS PowerPoint • To Do List — list of what have to be done • URL Reference — direct link to a web pages • Vote — to cast a vote in a discussion • Web Browser — to view HTML pages with others 63 Project DoCTA: Design and use of Collaborative Telelearning Artefacts TW has been used in several graduate courses in pedagogical information science at the University of Bergen. In this evaluation study, a task room was made for the student pair. 4.2 Test Scenario Design The test scenario was completed over a three day period in October 1999. This section describes the task, the physical setting and the data collected. The task. The volunteers students were given a system modelling task that was connected to an undergraduate course (grunnfag) they were taking. The task (see Appendix C) required that they design a use-case diagram and a class-diagram for an information system to be used in a Dental Centre. Each pair was given a short introduction to the groupware system they were to use. Then they were given the task assignment and could begin to work, each at their assigned computer. The physical setting. To simulate geographical distance between the collaborating students, two computers10 were placed different floors of IFI. One of the students in the pair was video taped and observed while working on the assignment. Figure 8 illustrates the placement of the computer screen, student, camera and observer. Data-collection. During the collaboration task, a video camera was used to record what one of the students said and what they did on the screen (see figure 8 for camera placement). This data will give us an impression of how easy it was to learn the different systems and give us an idea of how long it took them to feel comfortable with the systems. The sessions were relatively short, lasting between one and three hours. The students were also interviewed (see interview guide in Appendix D) after the sessions. Two of the groupware systems created log files and screen-cam and screenshots were used to save pictorial information about the sessions. After the three days of the scenario, there is about 6 hours of videotape, 8 interviews and miscellaneous screen-shots, event logs etc. The video is currently being transcribed and analysed. The full results will be reported in (Higraff, in preparation). A preliminary analysis, however, is presented below. 64 Chapter 4 Evaluation of Groupware Systems Problems during the test scenario. One day the network crashed causing a major problem since the students were not able to connected anymore. This also affected the tests performed the next day. In these two cases the students were not able to complete their assignment — they had, however, completed a high enough percentage of it to warrant inclusion. Both the diagrams were almost finished, and they had shown if and how they were able to do such a task in the different groupware systems. Looking at what they had completed, we decided not to redo the tests. Figure 8 The physical setting of the scenario 65 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 4.3 Preliminary Analysis Lotus LearningSpace This was the first groupware system to be tested. The two students with respect to how fast they started to try the system. Student B started experimenting immediately, while A began by reading the assignment. This resulted in A having problems following B in solving the task. The following11 is a comment from A to B, regarding A having problems following B: A: “Takes time to understand this- I hope I soon catch up regarding drawing the diagrams” B: ”Hang in there! J” After getting going, the students quickly agreed to each make a solution and post it in LearningSpace Live. Working in this manner, it was more like doing the work twice than dividing the work among them selves. Over time, their work style shifted more to A commenting the work B had posted, and then B commenting on what A said or just correcting it, if he believed his was a better proposal. The students finished their assignment in about two hours. During this time they came up with proposals, complaints and positive comments to LearningSpace Live. B uttered a wish for having a shared whiteboard, which LearningSpace Live did not have. He also wanted to use Rational Rose12 as the drawing tool, and he wanted to have a chat tool. TeamWave Workplace In preparation for this scenario a new room on the TeamWave server was created. A concept map with some ready-made icons was placed in the room. We also posted a post-it note with information about the task. It took them approximately 1 hour and 30 minutes to complete the assignment. They had access to all the tools in the program. Having been told up front, however, which TW tools were suitable for drawing, they focused on those tools. They also used the chat tool extensively. 11 12 66 This and subsequent dialogues have been translated from Norwegian to English by the author. Rational Rose is a tool for object-oriented system design used in the undergraduate (grunnfag) course. It provides support for UML-notation and Booch-notation that simplifies designing object-oriented systems. Chapter 4 Evaluation of Groupware Systems During the first few minutes, they spent time reading. Then, they used the chat-tool to make proposals before they posted anything. After a while this pattern changed. They posted, waited for a reaction from their partner, and then continued if they did not hear anything. The following discussion shows their co-operation in the beginning: A: [12:55:25] bruker1 says: Then we have to start renaming the use cases. B: [12:55:32] bruker2 says: What use cases do we have? A: [12:56:10] bruker1 says: What about “Making Appointment”? B: [12:57:28] bruker2 says: I agree. But one can make an appointment through both dentist and assistant. – shall we make sub-cases? A: [12:59:10] bruker1 says: Let us decide that making appointments can only be done through a secretary. B: [12:59:30] bruker2 says: OK The following comments appeared a bit later in the scenario: A: [13:10:33] bruker1 says: I have done a relocation as you can see. Agree? B: [13:10:51] bruker2 says: Ditto! I have made connection “report-dentist”, OK? A: [13:11:01] bruker1 says: Yes As can be seen from the above dialog, the pattern in their comments changed over time. It started with one student giving approval to the other’s proposals, and ended with commenting and encouraging each other etc. The two of them did not have much difficulty collaborating, and also it seemed as if they knew how the other student wanted things to be done. Habanero The two students using Habanero, first read through the assignment before they contacted each other on chat. Both of them co-operated well, but expressed frustration regarding functionality in the groupware system. Habanero is not a program, but a framework with demo tools on it, and it still lacks when it comes to functionality. The students agreed, however, that it was easy to use, and it had a very simple user interface. 67 Project DoCTA: Design and use of Collaborative Telelearning Artefacts They decided to open two whiteboards and alternated between them, see figure 9. They drew a use case-diagram on one of them and the other was used for scratch paper and to create the class-diagram. They divided the tasks between themselves as illustrated in this dialogue: A: Hege> Ok. I try to draw, and you list the entities... B: Habanero> Is it possible to use the function “ copy”, to get more stickmen13 ? A: Hege> Don’t know. Let’s try. B: Habanero> Does the list look ok? A: Hege> Excellent! A: Hege> Have you managed to copy anything? B: Habanero> No, I’m trying to find the help-program to get some hints… B: Habanero> I can’t find anything. Do you want me to draw too? A: Hege> Sure, just get going.. Figure 9 13 68 The two white boards used by the students in Habanero The stick-men had to be made in MS Paint. Chapter 4 Evaluation of Groupware Systems Their strategy, having two whiteboards running at the same time, worked until they started to rearrange the diagrams. They found it impossible to use copy & paste like they were used to, which was, according to them, a major drawback. After this the subjects experienced a network breakdown before having completed their assignment. As they had nearly completed the task, we did not rerun the session. BSCW This groupware system contains no chat tool. To communicate, one posts messages and texts in a database, and uses a web browser as a user interface to the database. This caused some minor delays and a problem of having to reload the page for updates14 . The two students testing this system quickly decided to split the assignment, and carry out one part each. The diagrams were made in Paint, and exported into BSCW. This worked fine for one of the students, but the other one had problems exporting the documents. She could write messages but could not add her documents. I believe this was caused by a fault in the set-up. We were not able to correct this. The problem was partly solved by one subject still being able to post drawings, and the other subject describing what work she had done in her messages. This problem could easily be solved by the mail-program they had available. The students were able to complete the assignments, but it was obvious that there were many things to learn and too little time. They had problems communicating in the beginning; the messages were hidden behind different icons. 14 It is also possible to run the system on a local server. 69 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 10 Part of the class-diagram the students made in MS paint while testing BSCW This is a part of the conversation, taking place through mail messages while creating, with MS Paint, the diagram shown in figure 10: A: from rose, 06 Oct I have made a start on the classdiagram, but I’m not able to send it. I have Employee as a super-class, with four sub-classes Clients have connection to employee? Or is it clinic? B: svar from bruker, 06 Oct Sounds OK, client ought to be connected to employee, not the clinic? Am working on the use case-diagram, I will soon send you a proposal. BSCW is a complex groupware system with a number of opportunities. This results in a high usability threshold when learning the different functions of the groupware systems. Major parts of the groupware system are quite intuitive if one has used Windows earlier. One cannot, however, make full use of the system without proper training. 70 Chapter 4 Evaluation of Groupware Systems 4.4 Discussion The groupware systems If one tries to compare these groupware systems, the first noticeable difference between them is the user-interface. BSCW and LearningSpace are both arranged with one or more pages in which to post messages. In addition, LearningSpace has three other pages or rooms. TW is based on a room-metaphor, and provides a lot of tools. Habanero is a framework with few tools available, but it is organised on a desktop, a metaphor with which most people are familiar. Habanero is not very time-consuming to learn because it is not a fully developed groupware system. To develop this groupware system further, will be at least as time-consuming as learning the other ones. It also requires a well-trained programmer to add the functions needed. Not being entirely completed means that adjustments can be made. One can get a groupware system for each special situation. But it will be time and resource consuming. Generally, the more positive comments were directed towards TW. It was rated as user-friendly and its tools are well arranged. The students using TW only took advantage of a few of the many tools available. I believe they did not realise how complex the groupware system actually was. TW can be used for both synchronous and asynchronous work. This is an advantage if one wants to solve different assignments with such a groupware system. In a pure synchronous system, one is dependent on all the students being able to work on the task simultaneously. For example, with Habanero, what is done in a Whiteboard is not saved and the next person using the program can not see that anything has been done. Working synchronously is not always possible. Many students choose an on-line course believing it is possible to combine them with work or children. And it is possible, as long as the technology supports asynchronous work. BSCW and LearningSpace are both primarily designed for asynchronous work, which was not explicitly tested in the scenario. These two groupware systems are therefore suitable for long lasting group tasks. BSCW would be excellent for people writing a paper together. They can write and immediately post their proposals, and the other persons can see when somebody last edited a piece of text. LearningSpace is suitable for students 71 Project DoCTA: Design and use of Collaborative Telelearning Artefacts working together for a relatively short period of time, and also for teachers wanting to run an entire course over the net. It is possible to post messages to a group, or a person, and one can also arrange exams with this system. The fact that there are four rooms makes it fairly well arranged, as long as the students are told what to post in the various rooms. The groupware system causing the students the most trouble was Habanero. As already stated, it is not a fully developed system, but a framework with a relatively small number of tools. Habanero can only be used for synchronous work. This can be a problem for many students working part-time or having other obligations. The framework has potential, but it would demand quite a lot of hard work by a good programmer to tailor it for a specific situation. One can ask if it is worth it, as long as there are free manufactured products within reach. Preliminary observations One surprise with this study was the relatively low expectations most of the students had for such a system. Even though they are information science students, just the fact that they were able to work together seemed to please and fascinate them. It appears that working together online is much easier if one have access to a chat-tool, where one can see what everybody else writes as they write it. Email is different in that one can not see what the other person has written until the whole message is posted. To post messages is time-consuming and it might lead to a lot of double work. If one can read while the other types, one will be able to see at once what the other person is trying to do. The students started off trying the groupware system they were assigned to use, feeling their way around. After a while, they seemed to develop a certain work-pattern. This is not to imply that they adopted the same pattern, but they changed the way they initially worked. In several of the cases it seemed they became more secure in what they was doing. Instead of having every move confirmed by the other person, they started to do things or draw what they believed to be correct. It was impressive how fast they seemed to adapt to this new way of co-operation. It was also fascinating to see how they started out very formal, but after a few minutes they began to use quite informal language. Several of them 72 Chapter 4 Evaluation of Groupware Systems switched to dialect15 after a short period of formal orthography. It seemed they wanted to “test” the person with whom they worked, and when this person was found OK, they could be informal. The following extracts give an example of this (in this example and the next example, the original Norwegian dialect is given in italics with an English translation in parentheses). This dialog is extracted from the first part of the session. The two are “getting to know” each other: [12:52:40] bruker1 says: [12:53:15] bruker2 says: [12:54:45] bruker1 says: Skal jeg lage flere aktører? Tannlege osv? (Shall I make more actors? Dentist, etc.?) Greit (OK) Enig i at vi har alle ansatte på høyresiden? (Agree that we have all the employees on the right?) The next excerpt is taken from the dialog about half an hour later. The two seems to feel more comfortable with each other. They both use slang or “dialect”. Also, they both use abbreviations and uncompleted sentences: [13:31:39] bruker1 says: [13:31:48] bruker2 says: [13:31:51] bruker1 says: [13:32:14] bruker1 says: [13:32:21] bruker2 says: [13:33:23] bruker1 says: [13:33:49] bruker2 says: [13:34:08] bruker2 says: [13:34:35] bruker1 says: 15 Se! (Look!) (Subject points at new connection-arrow) Kossen? (How?) Vi har laga Inheritance-pil! (We made an inheritance-relation!) Den ligg nederst i “tabellen” av linktyper. (It’s in the bottom of the “table” of link-types) Aha! (Aha!) Ka no då? (What now?) Forbindelsar... (connections...) Ml.klient - tannlege? (bet. Client- Dentist?) Ja, selvfølgelig... E litt sein i hodet i dag... (Yes, of course…My head works a bit slow today…) In Norwegian there are 2 formal languages and an unknown number of dialects. Every little place in Norway has their own dialect. The two students in this example come from somewhere outside Bergen (on the western coast) and somewhere close to Bodø (in the north) – their dialects are very different! 73 Project DoCTA: Design and use of Collaborative Telelearning Artefacts It seemed that although the students learned to use the groupware systems quite fast, they were also learning about how to communicate with one another. This is a consequence that is not directly related to using groupware systems. In the sessions the students found the basic tools needed for their task, but they never got to try the whole groupware system. Thus, the students would have benefited from having had longer training in the different groupware systems. In VisArt, the students were given extensive training in both use of the system and in collaboration. Preliminary suggests for what a groupware system should contain The user-interface is extremely important as it gives a first impression, and gives access to the functionality of the underlying system. If the interface is bad, it will be harder to make use of the system’s functionality. It would be beneficial, if the interface were changeable for different tasks. In that way one could emphasise the more important features needed for the different tasks. In some ways TW allows the students to configure the interface (e.g., their room) with the tools they need. In the test scenario described here, a large number of the systems’ interface features were not needed. A preliminary impression is that the most important tools are: • Chat • Drawing-tool • Text-editor All of the students commented at some point on the need to talk to their partner. When working asynchronously, it is important to post messages, but when working synchronously, time becomes an important aspect. One should get the message instantaneously. An easy way to do this is by a chat-like tool. The volunteers gave different reasons for why they wanted to be able to talk more or less directly to their partner, but the reason more often stated was time-saving needs. We also found that a whiteboard or a drawing-tool was necessary. Of course it is possible to use a tool outside of the groupware system, however, one of the important reasons for using such a groupware system is that one has all-important features collected in one place. Another important tool to have, is some kind of text-editor. Most tasks contain some textual 74 Chapter 4 Evaluation of Groupware Systems aspect, which one would like to edit. If such a tool is not included, one might consider using an existing program, like Microsoft Word or Word Perfect. In many cases, this will be less costly than making a new tool, as a text-editor is quite complex. Also, most people are familiar with these tools. But, as mentioned before, this must be evaluated in each specific case. In order to evaluate whether to use an existing tool or to build an integrated tool, one needs to find answer to the following questions: • What will be more time-consuming? • What will be more expensive? • Who is the user? Each of these tools must be fully functional. For instance, the chat should be simultaneous, and preferably with different colours indicating the different users. The drawing-tool should support basic features as copy/ paste and print. One should also have auto-figures, and if possible, specialised symbols. The editor should provide different fonts, maybe spell-checks etc. Another important issue here, is how specialised such a tool should be. Is it desirable to have a tool that can do all the things needed for a specific case, or is it more important that it is adaptable to several tasks? An example can be the use of Rational Rose (RR) or MS Paint for the assignment in the scenario. MS Paint was chosen because it was more adaptable for all programs. We experienced large difficulties when trying to copy diagrams from RR to the different systems. Paint is also a program with which many people have experience. Also, RR is very specialised, while Paint can be used for several tasks. One has to consider and weigh what is more important, reuse or specialisation. One must also weigh the cost and what purpose one has for the program. Again, this must be considered in each specific case. Of the four groupware systems tested, TeamWave Workplace has more useful tools included for the kind of task performed in the scenario. It has a fully developed text editor, a chat tool and a drawing tool. TW also has a special tool for organising concepts and ideas in a visual interface, the Concept Map. Thus, in our situation the students testing TW had an advantage. Still, one can not say the other systems were not applicable — all the students finished their assignment, but they experienced different conditions. 75 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 76 Chapter 5 Teamwave workplace 5 Teamwave workplace BARBARA WASSON As TW was used in both the IDEELS and VisArt scenarios, a more indepth review of it is given. This section, based on Wasson’s review of TW in (Fjuk, Sorensen & Wasson, 1999), describes in detail this groupware system and highlights its strengths and weaknesses. 5.1 The basis for design TeamWave Workplace (TW) is based on the metaphor of shared networked places. Using real life physical team rooms (Johansen, 1991) that provide a permanent shared working place for teams as inspiration, the notion of virtual team rooms has been adopted in TW. These virtual team rooms provide a permanent shared space where teams distributed over the Internet can have meetings, store documents, share URL links to web sites, coordinate and communicate with one another and can carry out collaborative activities such as brainstorming or participation in a discussion forum. Each team can build a set of different rooms according to need. For example, they might create a common project room and resource room, and in addition, individual rooms for each team member. A TW server maintains the operational and interactional aspects of each collaborating team. The server acts as a communication hub, maintains a database of registered users and a persistence repository which stores information about rooms and their contents. This means that all materials placed in each room are stored on the server and are available for each team member to view at anytime. Each team member downloads a client version and connects to the team server. The server retrieves all information shared by the team, making it available to the client. The 77 Project DoCTA: Design and use of Collaborative Telelearning Artefacts latest version of TW provides server-side licensing16 so all that users need to access TW is a user account (a user name and password created by the server administrator). TW is created and marketed by TeamWave Software that has its roots in the GroupLab’s (http://www.cpsc.ucalgary.ca/grouplab/) TeamRoom project at the University of Calgary, Canada. GroupLab is one of the leading research groups into the end-user and technical requirements of collaborative systems and TeamWave Workplace grew out of a series of research systems developed within the group. One important distinction between TW and other real-time groupware systems is that TW is based on the metaphor of a place (i.e., a room), while most others are based on the metaphor of a meeting. This place metaphor choice resulted in a number of room design requirements for TW (Roseman & Greenberg, 1996): • rooms must be long-lived, and usable by both individuals and groups • rooms and their contents must be fully persistent • rooms should provide interpersonal communication facilities • generic collaboration tools should be automatically made available in each room • special purpose tools can be added to any rooms • rooms must be accessible from anywhere on a network • access to rooms must be limited to a designated group, and • external computer & network information should be easily accessible from within rooms A major strength of TW is that it provides a well-integrated set of varied collaboration tools with a good blend of real-time (synchronous) and asynchronous communication tools which enable anytime team collaboration. Furthermore, TW augments both existing user interaction tools such as email, newsgroups and conferencing, and existing conventional applications such as word processors and spreadsheets. This can be seen as both a strength and weakness depending on one’s point of view. As a strength it means that the user (i.e., student) can continue to use other communication tools and applications with which they are most comfortable and familiar, using TW for the purpose of supporting their team interactions. As a weakness, TW does not support the sharing 16 78 Previous versions required that each client have a user account and a license number to connect. Chapter 5 Teamwave workplace of applications17 such as collaborative editors for document writing, thus synchronous collaborative activities such as co-authoring cannot easily be carried out. Another strength of TW’s integrated approach is that spontaneous as well as pre-planned intra-team interactions are supported. This means that anytime several team members are logged on simultaneously, they can interact without setting up a pre-specified meeting time. Furthermore, the materials left in the room when a team member leaves remain in the room for the other team members to see when they eventually log on. These two features are in contract to the majority of groupware systems that only support pre-planned meetings (with a specified start/end time) and then when the meeting is over, all materials disappear. TW also provides an administration client to be used by whoever is maintaining the server and/or facilitating, for example, a course or a project. The administrator client is used to create and delete individual accounts, assign individuals to teams/groups, as well as change access permissions and grant administrative privileges to a particular user. A set of tools for managing the persistence repository is also provided. From a CSCL perspective, TW enables collaboration and supports genuine interdependence between team members. Team members are able to share information, meanings, thoughts, conceptions and conclusions through their choice of operational tool objects. These thinking tools (signs) in turn facilitate both teammate knowledge construction and collective growth. Furthermore, the thinking tools provide a means for thoughts to be examined, changed and elaborated upon by fellow team members. The collectively oriented actions are mediated by mechanisms enabling or supporting: • creation of a shared networked place • workspace awareness These are described in the subsections below. 17 The latest versions of TW, however, provide for a seamless integration with Microsoft’s NetMeeting. 79 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Creating a shared networked place A shared networked place is created through individual users’ production of tool objects (artefacts) within a room. Upon starting up TW, the user is placed in a user-specified default room. In this default room the user can begin to work, or can either create a new room or move (enter) to another existing room. When newly created (see figure 11), a room consists of a blank shared white board with a pen tray where the user can select colored pens for producing free hand drawings and a chat tool. Figure 11 Barbara Wasson’s (empty) Room in TeamWave Workplace A tool menu is provided for selecting tools for customising the room for use (e.g., to share files, create a list of things to do, etc.). To move to a new room, the user can either enter through a doorway or select a room from the list of all room available to the group (in figure 11 see the list of rooms — Geir Moen’s Fast Room, Classroom, Team11 — which are visible and a scrollbar to access others). For each room, a room menu (see figure 12a) enables different actions to be performed: create a new room; enter another room; delete the current room (only allowed if you are room owner); erase the whiteboard; empty the room; find out the room properties (i.e., who owns the room, etc.); set room permissions; 80 Chapter 5 Teamwave workplace retrieve old versions of the room; save the current version of the room; save the room as a template; and set the room as the default room when logging in. The room owner can set permissions for the room (see figure 12b) such as who can enter the room, create tools in the room, delete tools in the room, change the whiteboard, etc. In figure 12b the column on the left indicates the permission type to be set and the right column gives the choices of which group can be given that permission. Figure 12 Room and permissions menus in TeamWave Workplace As mentioned earlier, each room can be customised by the team to suit their specific needs and tasks by using any of the 19 tools provided in TW. The use of the tools is relatively intuitive just by seeing the name of the tool (e.g., address book, database). The tools include: Address Book, Brainstormer, Calendar, Chat, Concept Map, Database, Doorway, File Holder, File Viewer, Image Whiteboard, Meeting Roster, Message Board, Personalised Message, PostIt, ToDoList, URLRef, Vote, Web Browse, and the on-line help. Figure 13 illustrates a doorway tool (e.g., doorways to Help Room, Training Room, and Team01…Team11 rooms), a calendar tool, an address book tool, a To Do List tool, a URL link tool, a file holder tool (VisArt.rtf), and the chat area. The tools can be used for navigation, production, communication, management and consulting. Navigation tools provide means for navigating between rooms (doorway tool) or to information (URLRef tool). Production tools (brainstormer, concept map) enable team members to collaboratively share ideas, make group decisions, or build knowledge representations. Communication tools provide a way of asynchronously communicating information (file holder, message board, vote, Postit) or synchronously 81 Project DoCTA: Design and use of Collaborative Telelearning Artefacts communicating with one or more team member(s) (chat, page/personalised message). Management tools (address book, calendar, meeting roster, To Do List) allow team members to coordinate their work. Finally, consulting tools (database, file viewer, image whiteboard, web browser, on-line help) provide access to shared information or help. Figure 13 Classroom – An example of a TW room and its tools Awareness As described in section 2.5, Gutwin et al. (1995) present a framework of awareness for collaborative learning which comprises four types of awareness: social awareness, task awareness, concept awareness and workspace awareness. TW pays close attention to workspace awareness by providing a number of features that maintain awareness of other team members, and of other 82 Chapter 5 Teamwave workplace users who are on the same server18 . For example, as illustrated in figure 13, TW displays separate lists of users in the current room and a general list of other users on the server. The current room users list contains either the user’s name or, if desired, a small picture of the user. In figure 13 this can be seen in the Now in Classroom list with a picture of Barbara and Eirin’s name and how long Eirin has been idle. Moving the mouse pointer over the picture reveals whether the user is active or how long they have been idle and indicates the colour of that user’s telepointer on the whiteboard. Clicking the mouse button over the name or picture gives access to the user’s business card which contains more information about the user including a phone and fax number, email address, URL homepage address and (if entered when the user logged on) the current physical location19 of the user and a phone number where they can be reached. It is also through this clicking on the name or picture that one is given a choice of sending email, accessing the user’s home web page or paging the user (i.e., send a short personalized message that to get the user’s attention). The displayed general list of other users on the server, gives their name, the name of the room which they are in and the length of time they have been idle. This list can be seen inn figure 13 in the Other Connected Users list where there are 5 visible names and a scrollbar to read the rest of the list (at the time the screen shot was taken there actually were 12 users listed in this list). A double click on a name in the general list gives the business card of the user. These simple mechanisms (displayed lists of users) play a significant role is providing a general awareness of who is around (the general list) and a more fine-grained awareness of other team members’ actions (current room users list). Another fine-grained sense of awareness of other team members’ actions is provided through telepointer. A telepointer shows up as a coloured dot, one for each team member, on the shared whiteboard and follows each member’s movements. These enables team members to follow what team-mates are looking at. When a tool is moved around on the shared 18 19 These users may be on different teams and have access to different areas of the server. For example a class of students may be divided into various teams for a particular activity and each team assigned their own private team room. In addition, there maybe other rooms which all teams can access (e.g., a room describing the activity, a resource room, a help room, a coffee room, etc.). This feature of TW also facilitates access to each other enabling, for example, a phone call to the person or a walk around the building to where the person is located. 83 Project DoCTA: Design and use of Collaborative Telelearning Artefacts whiteboard, other team members see not only the result, but see who is moving the tool. Thus users see not only each other and what they are doing, but see immediate changes in the room’s artefacts. A room overview radar is used to provide awareness of the location of all room tools on the shared whiteboard. This miniature room overview of the room is necessary to facilitate the use of a shared whiteboard that is bigger than what will fit on a display (remember that not all users will have a large 21 cm display and maybe working on a 12 cm display). A miniature telepointer is also provided on the room overview radar, so as team members move around the room, the radar tracks their actions (Roseman & Greenberg, 1996). Concept awareness supported in TW is directly related to the learners’ collaborative construction of knowledge. For example, concept awareness of a file that has been placed on the server using the file holder tool is provided by giving information about the size of the file, who put it on the server, and the date and time of the last modification when the mouse pointer is moved over the tool. Similarly, a history of each tool can be retrieved through the tool’s menu (a menu for each tool is created when the tool is created and is visible when that tool is activated) that tells the name of the tool creator or modifier, the date, time and size of the information at creation or modification of the tool. In addition to these automatic awareness features, team members can use the provided tools to provide awareness. For example team members can use PostIt notes to indicate work that they have done while the others have not been around and to indicate when they have been present in a room. 5.2 User interface considerations The TW interface seems to support what Fjuk and Øgrim (1997) suggest as an action-oriented approach to computer support for collaborative learning. All actions are connected to the different information objects and the interface provides an intuitive indication of what operations the provided tools support. The TW interface can be described according to Appelet et al.’s (1998) characterisation of a system’s interface: Visibility of the different types of information objects. TW’s use of the shared whiteboard for placement of shared information artefacts provides 84 Chapter 5 Teamwave workplace signs to team members that information is available for individual knowledge construction and for collective growth. When an interactional aspect of an action requires an operation, TW provides a number of mediational tools that are both immediately visible (chat and lists over others in the room and in other rooms, white board, help menu) and easily accessible through the tool menu. Once a tool artefact is created, a menu corresponding to that tool is created and is visible whenever the tool is activated (by clicking on the tool). The list of available rooms (including who is in the room) provides structural information about the teams working area. Visibility of what actions can be performed. Awareness of what operations can be performed by what tools is fairly intuitive in TW. Each created tool artefact’s menu gives a number of operations that can be performed on the tool artefact including: minimise the tool; delete the tool; copy it to another room; copy it to the server; find out its properties; get a version history; and save a version of the tool. In addition, the available operations are either explicit (fill in the slots in a database, or enter an item in the ToDoList) or are found by visiting the tool artefacts menu or by clicking the right mouse button over the tool artefact (e.g., visiting a doorway artefact menu or clicking the right mouse button over a doorway artefact gives the option of entering the room to which the doorway leads, or changing the room to which the doorway leads). Layout of the interface. TW allows users to customise the user interface to reflect individual and team needs. Upon creation of a new room, the user is given an empty whiteboard where various tool artefacts (e.g., information or communication artefacts) can be placed. Each information object is given a name and several operations can be performed on the object (recall descriptions above). TW’s interface appears the same to both novice and experienced users. Since the room begins with an empty whiteboard, it is up to each individual or team to maintain and control the “clutter” of the room (i.e., organise the room for their own individual or team’s use). It is not possible, however, to customise the standard tool menus. 5.3 Production, Communication and Distribution In Fjuk et al. (1999) it was reported how TW supports production/ knowledge construction, communication and distribution/division of labour. This section summarises these results. 85 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Production/Knowledge construction TW supports, in a limited way, searching for information. The construction of a personal domain for information and knowledge is facilitated by the possibility to create one’s own personal room, which can be customised as desired. Limited support is provided for articulating meaning (thought beliefs, knowledge, experiences, and skills) into various information objects (plain text, hypertext, pictures, drawings, schemes, informal notes, etc.). Producing information objects for distributing them in the collaborative community can be either carried out in the individual’s own working environment (e.g., using favourite word processor or spread sheet) and then sharing them through TW, or using some of the tools to provide information objects for team-mates to examine. TW also provides support for distributing and co-ordinating meanings to peers, enables individuals and teams to reflect upon and elaborate information objects, and supports the planning of an individual’s own and collective progress. Tools are provided to support the identification of one’s own role and tasks with respect to the collaborative community’s division of labour. No explicit support is provided for (re)evaluating ones own knowledge and interpretations, although such support can be designed by the individual, a team or a facilitator20 . Table 5 lists the TW tools that support the operationalisation of these interactional aspects of actions. 20 86 For example, in our use in a course at the University of Bergen we created a self-evaluation form on the web which we linked to in TW, that helped the student evaluate their confidence in using TW. Chapter 5 Teamwave workplace Table 5 TW tools facilitating production/knowledge construction (Fjuk et al. 1999) Operational aspect of actions TW tool/feature Digital libraries and corresponding functionality for searching for information Database, URL links, Web Browse, File Viewer Mechanisms for constructing personal domains of selected information objects. Rooms Mechanisms for articulating thoughts White board, concept map, file holder Mechanisms for simple downloading and uploading information objects. File holder, URL holder Mechanisms for simple attachment of information objects in social interactions Mechanisms for making notes Postit, Mechanisms for personal planning Calendar, ToDoList, Self-evaluation tools Postit notes (with comments from peers), URL link (to self-evaluation questionnaire) Communication TW supports the distribution and co-ordination of information among peers. There are several ways to comment on information objects provided by peers and to discuss relevant topics connected to the problem/subject under consideration. Mechanisms for making common decisions are provided. Limited support is provided for negotiating meaning, coproduction/co-authoring of information objects and, while getting an overview of the peers’ action and progress is well supported. TW is focused on providing means of creating a common presentation area and enable users to create content annotations for each other. Table 6 lists the TW tools that support the operationalisation of these interactional aspects of actions. 87 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Table 6 How TW facilitates communication (Fjuk et al. 1999) Operational aspect of actions TW tool/feature Mechanisms that retrieve information about social interactions Telepointers, Immediate update of information objects when changed, PostIt (to record information about who was here and when) Mechanisms that retrieve lists of all actors in the social communities and other actors in the system Address Book Mechanisms that inform about which actors that are currently on-line. General user list, Room user list Mechanisms for interacting both synchronous (e.g., chat) and asynchronous. Chat, Page/Personal Message, Brainstorming, Message Board Mechanisms that retrieve a whole dialogue, Chat (saving to own working area and possibly containing contributions before and sharing with team members), Message after the one specified by the actor. Board, Brainstormer Mechanisms that search across the dialogues. Mechanisms for categorising dialogues item according to keywords, chosen by the actor. Mechanisms for producing joint information objects File holder, Concept Map, White Board Mechanisms for voting among alternative positions and meanings. Vote Mechanisms for providing decision support. Vote, Brainstormer Distribution/Division of Labour TW provides several mechanisms to support the organisation of a team/ project enabling the articulation of responsibility and mutual commitments (who is doing what when) and articulation the use of time (for meetings, deadlines, etc.). Making common plans is supported in a limited sense as is providing awareness of how a particular activity fits into the individuals’ knowledge, doings and progress. Finally, support is provided with respect to up-to-the minute knowledge about peers’ interactions with the collaborative environment. Table 7 lists the TW tools that support the operationalisation of these interactional aspects of actions. 88 Chapter 5 Teamwave workplace Table 7 How TW facilitates distribution/division of labour (Fjuk et al. 1999) Operational aspect of actions TW tool/feature Mechanisms for retrieving an overview of the ToDoList individual’s status in relation to her commitment in the social community. Mechanisms for supporting project management. ToDoList, Vote, Address Book, Meeting Roster Mechanisms for using a joint calendar facility. Calendar 5.4 Technical requirements TeamWave Workplace maintains a web site at http://www.teamwave.com where interested users can download a free Workplace client and get information about connecting to a demonstration server before committing to purchase. To purchase TeamWave Workplace, you buy a workplace license (valid for 1 year of updates) according to the number of users you wish to support and according to whether you buy a regular or educational license. In March 1999, a single regular license cost $78 US and a single educational license cost $40 US while 50 regular licenses cost $1999 US and 50 educational licenses cost $899 US. Purchase before March 15th and there was a 10% discount (there seem to be several such promotions throughout the year). It is also possible to get a demonstration license that will allow you to test TeamWave Workplace for a period of 3 weeks. Clients and servers are available for the following platforms: Windows 95/ 98/NT, MacIntosh, Linux, Solaris (aka SunOS 5.x), SunOS (4.1.4 aka Solaris 1.x), SGI (IRIX 6.2), and AIX. This means that each user can decide for themselves which client version they will download. It is also possible to launch Workplace from a web page. The minimum ram required for the server is 16 Mbytes, but 32 Mbytes is recommended. The required disk space depends on the activity the server sees. Every file or graphic that is put to a workplace server takes space and is not automatically removed when a room is deleted or its contents is deleted. This is due to the versioning feature which allows the recall of prior versions of a room’s contents. A client requires only enough disk space to run which is about 4 Mbytes. As for the server, 16Mbytes of ram is enough but 32Mbytes is better. According to Roseman & Greenberg (1996), TW has been designed to not require a very high bandwidth for network 89 Project DoCTA: Design and use of Collaborative Telelearning Artefacts connections. This means that users connecting over 14.4 modem connections or long distance Internet connection find it reasonable to use TW. 5.5 Final remarks on TeamWave Workplace There are several reasons why TW was chosen for use in the DoCTA scenarios IDEELS and VisArt. First, the focus in TW is on the support of collaboration itself rather than on instruction and scaffolding. The emphasis is first and foremost on the collectively-oriented aspects of collaboration through the facilitation of the creation of a shared networked place and awareness, and secondly on individual aspects (e.g., by enabling the creation of a room for individual use when individual tools can be placed). Furthermore, a major strength of TW is its support for both pre-planned and spontaneous interaction among team members. The second major strength of TW is that it provides an integrated set of general tools that can be placed in customised rooms for team or individual use. The individual continues to work independently on their own system using the tools they are used to and uses TW to share created information with team-mates. A final aspect that has not been discussed to this point is the tailorability and customisation of TW. TW is tailorable by the administrator/facilitator and developers, and customisable by the individual/team and by developers. The administrator can adapt TW for local use by creating teams of individual users and giving the permission to access particular rooms on the server. A facilitator (e.g., a course instructor) can tailor TW to have a set of rooms for the students (e.g., a help room, a training room, a classroom, and team rooms, activity rooms, etc.). As discussed earlier, the individual user or team can customise individual or team rooms as desired according to need and tasks. Finally, as TW has emerged from a University research group, the developers have had a very open attitude toward enabling developers to tailor TW to their own use. Purchase of TW licenses also provides access to a Software Development Kit (SDK) that enables one to extend the functionality of TW by developing their own tools (see http:// www.teamwave.com/newtools.html for hints on how to do this). In VisArt we tried to tailor further development of the application itself and ran into problems. We received help from the TW developers, as there were certain development aspects they needed to do themselves. This can be seen as a drawback with regard to tailorability. 90 Chapter 6 Ideels Part II The Scenarios IDEELS VisArt 91 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 92 Chapter 6 Ideels 6 Ideels Project IDEELS (Intercultural Dynamics in European Education though on-line Simulation) brings together a diverse group of educators and researchers from five tertiary institutions in four European countries who share a common interest in simulation and games (see Appendix E for further information). IDEELS is an EU Socrates curriculum development project with partners at the University of Bremen, Germany (coordinators), the Polytechnic University of Valencia, Spain, the University of Nice, France, and Nord-Trøndelag College (HiNT) and University of Bergen (UiB), Norway (see figure 14). The goals of IDEELS include adding impetus to the curriculum development trend towards content & processbased learning and to enhance European competitiveness by providing students with opportunities to learn essential cross-cultural, linguistic and negotiating skills. A generic simulation game is used to complement existing curricula in a wide range of areas including language learning, negotiation, policy studies, political science, environmental issues, cross-culture communication, law, education, and computer science. In IDEELS simulations, students act as high-level negotiators, consultants, and journalists in a fictional world, working to resolve real problems of importance to the European Community – problems that can only be solved through co-operation at the international level. 93 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 14 Screenshot from the IDEELS’ homepage Teams from the various partner institutions take on different roles in a simulation conducted on two levels: deliberations within a team and negotiations between or among teams. Thus, it can be said that the simulation scenarios are designed to require (inter)dependence among both team members and between teams (Wasson & Mørch, 1999). Teams are given a common mission where the goal is to produce, agree upon and ratify a jointly written document (e.g., come to a consensus and sign either a policy or treaty, or write a set of recommendations). In the November 1998 simulation, teams from the various partner institutions represented a country, a technical consulting company or a newspaper. The overall goal or the common mission these teams had was to produce, agree upon and ratify a jointly written set of recommendations for a design plan for the educational system for Eutropolis, the New Eutropian Capital. The first task each team had to do was to prepare two documents: an Internal Briefing Document, intended to guide the team’s actions 94 Chapter 6 Ideels through the negotiation, and; a Position Paper for “public consumption” stating the team’s initial negotiation position. Other specific tasks that the teams had to engage in included a number of real-time online teleconferences, where the different parties should discuss a pre-selected topic. The teams were also encouraged to engage in an ongoing dialogue by sending each other questions, inquiries and request clarification of different statements. To communicate their policies and conduct a dialogue with the other teams, all participants had access through the World Wide Web to OPUSi a computer-based communication system developed at the Department of Computer Science, University of Bremen for use in IDEELS simulations. OPUSi enables participants to send each other written messages (within a team and between teams), and to participate in real-time online teleconferences. Norwegian Participation Two students from Nord-Trøndelag College participated in the IDEELS simulation during November 1998 were students in the Sociology programme in the Department of Social Sciences. The students participated as a team representing a country, Bardland. An attempt to recruit about 8 or 10 students failed, primarily because the participation did not count towards meeting a course requirement. Since these were volunteer participants, the students did not have any special reading assignments with respect to role-playing. Two teams of students from the University of Bergen participated in the November 1998 scenario as part of a graduate course in pedagogical information science. Six students volunteered to participate in the IDEELS scenario and write a term report based on the experience. The students were divided into two teams of 3 students and each team was assigned a role by the IDEELS facilitator at the University of Bremen. One team played a country, Highland, and the second team were journalists. As the participating Bergen students were geographically spread over Norway, it was decided that OPUSi did not provide enough support for intrateam collaboration, so it was supplemented with both team email addresses and a Teamwave Workplace. In addition, the students had access to their own PC environment with whichever word processor and Internet browser they preferred (see figure 15). 95 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 15 Supporting tools in the scenario The collaborative telelearning scenario that is the focus of this study is not only limited to include the actual IDEELS scenario. As mentioned previously, the students’ participation in the IDEELS scenario was a part of an assignment in a research seminar in pedagogical information science (see Appendix F for the assignment and the course description). In this assignment they were instructed to evaluate the scenario from one of the perspectives covered in the class viz.: psychometrics, distributed cognition, activity theory, or discourse analysis. They were supposed to write a project description for a study, including a research question, the type of data to be collected, what methods to use in this data collection and account for the perspective they had chosen. The result of this assignment should be a written report that was to be handed in for evaluation and be given a grade. The collaborative telelearning scenario is thus used to denote the team’s entire collaboration effort, starting with the participation in IDEELS and, subsequently, collaborating to finish the assignment. OPUSi OPUSi is a web-based communication system that is available from a server at the University of Bremen, Germany, and can be accessed through a common web-browser, such as Microsoft Internet Explorer or Netscape communicator. The OPUSi system is specially designed to support the different activities in the IDEELS scenarios. The user interface of OPUSi is divided into a number of “frames”, where the top frame contains a menu showing the various user features available 96 Chapter 6 Ideels in the system (see figure 16). This navigation bar has a list of options that includes: Internal memos; Message centre; Papers; Library; Conference; and Logout. The “internal memos” option can be used to both send and receive asynchronous messages within a defined user group (a team). The “Message centre”, on the other hand, supports inter-team asynchronous communication. As you can see in figure 16, the “Message centre” contains a “Folder” for each team, and each of these “Folders” (“EuGlobe” in the figure) holds the messages submitted by the actual team. All members of a particular team have the same “view” within the system and thereby have access to the same areas and documents. This means that, for example, in the “Message centre” a specific group can read the messages that are sent to everyone and the messages that are addressed specifically to this group. The “Library” area contains resources and background information (e.g. about the Maastricht Treaty) about certain topics that is addressed and discussed in the scenario. The “Conference” choice gives access to and support online real-time teleconferences. A conference is usually initiated by a team of facilitators in the scenario and has a specific topic, but all teams have the possibility to start a conference. After a conference is ended the written discourse is still available for review. The “Papers” option is dedicated to support collaborative (not simultaneous) writing and editing of the papers each team is supposed to publish, for example, in the “message centre”. In addition, the system contains very basic word processing facilities within the different menu options that enable the users to compose written messages. 97 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 16 Screenshot from OPUSi In other words, OPUSi provides its users with the possibility to engage in synchronous and asynchronous communication, collaborative writing and helps them find relevant resources. 6.1 Evaluation of intra-group collaboration FRODE GUIBYE This section describes an evaluation (Guribye, 1999, Guribye & Wasson, submitted) of the intra-group collaboration of one of the teams from UiB that participated in the IDEELS scenario and subsequently wrote an assignment about their own participation in this collaborative telelearning scenario. This team of students had been assigned the role of journalists who were responsible for publishing a newspaper or magazine. As journalists, they, 98 Chapter 6 Ideels in addition to producing a periodical edition of their publication (at least twice a week), had to regularly interview the other teams and provide reports from the different events and developments in the scenario. In the teleconferences they had a more passive role and were only supposed to join these conferences as observers without any active interventions in the actual discourse. The first thing the team decided was that they wanted to produce an electronic newspaper, called “NewWave” and publish it on the Internet. First the focus of the research study is given, then an overall description of the teams’ collaboration process is presented, and finally some concluding remarks and possible implications of the particular study is given. In this way the goal with this part of the report is to give a “thick description” (Geertz, 1973) of one of the teams that were actively engaged in the IDEELS scenario, and from this description try to give some concluding remarks concerning what aspects one should be aware of when designing a collaborative telelearning scenario. The Research Study The initial empirical investigations in this study, were centred around one tentative research question: How do students organise their work? The design of the scenario provided a setting for the students to carry out their tasks. Prior to the actual study, relevant elements of this context were considered to consist of: actor characteristics, the kind of artefacts provided, the kind of artefacts they were to design and the design of the learning activity. During the study the focus was narrowed down and directed towards some aspects of the way the students organised their work. It was possible to identify more precise questions, such as: “how do the students collaboratively arrive at a particular division of labour?” “what modes of communication do the students choose?” and “what tools do they use to mediate their communication?”. 99 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Data Collection The data collection techniques that were applied in the actual data gathering were mainly selected from the repertoire of techniques available within ethnography (Hammersley & Atkinson, 1983). Some of these techniques had to be especially adapted (as always) to the specific and a bit unusual context and environment. The most extraordinary dimension of this was that large portions of the group’s communication were computer-mediated. This led to a question about how to do computer-mediated “field” observations. The answer is given partly by trying to immerse oneself in the, so called, virtual environment and partly in improvising and being creative when searching for possible sources of information. Table 8 lists the techniques used in this study. Table 8 Data gathering techniques Data gathering techniques Remarks “Field” observations Informal, limited interaction, including the bservations in the virtual environment Collecting email Automatically received from the team’s “mailing list” Interviews In-depth, semi-structured, open-ended, individual Data logs, chat logs From Teamwave, chat logs saved by one of the students and emailed to the researcher Informal interview / email interview With a second group Described chronologically, the empirical study began with brief informal conversations with the group members and occasional drop-in visits in the data-lab where the one student situated in Bergen was working. These visits can be categorised as “field” observations with limited interaction from the researcher. This means that the interventions with the members are mainly confined to seeking clarification and the meaning of ongoing events. During the entire IDEELS scenario, the members of the group communicated frequently using email. All these emails were continuously collected and read. Being able to follow the communication on email as it arrived played an important role in the evolving “field observations”. In 100 Chapter 6 Ideels this way it was possible to get a picture of the progress and the different activities they were involved with at various stages in their collaboration effort. The preliminary and ongoing textual analysis of the information provided in the emails played an essential role in picking out suitable data gathering techniques for further evaluation and in developing a more particular focus for the study. At the same time their emails served as documentation of the communication. Reading and rereading these was a very valuable source when doing the post evaluation trying to reconstruct a picture of the already past events and activities. OPUSi also mediated communication during the scenario. This tool was used mainly to publish produced documents. It soon became clear that OPUSi played an insignificant role in the intra-group collaboration, at least as a mean for active communication. It was rather a reference spot the students only observed and on which they commented. At the very beginning of the scenario the application was occasionally browsed to check for possible intra-group activities. Most of the documents submitted here were directed towards inter-group communication and these documents were usually interchanged via email prior to their submission. Closer to the end of the collaboration process, TW was used quite frequently. Thus, observing and following the students’ activities in this environment came to the fore. Although many of the different tools within TW were used, the chat-logs that were collected from six real-time meetings seem to be most important for evaluation purposes. Interviews were used to follow up the data gathering that went on during the group’s work. These interviews were carried out after the scenario was ended, the assignment was due, and to cut the students some slack after their oral exam. So, approximately a week after the collaboration effort, an interview with each of them was arranged. To derive a structure for the interviews, decide what topics that should be addressed, and to design the main questions AT was used as a conceptual resource. The resultant interview guide can be found in Appendix G. This means that when preparing the interviews by analysing the already gathered data, concepts and ideas from the theoretical framework were actively used to understand what had been observed and to develop new ideas about the way that the students organised their work. Also, the ongoing, preliminary analysis served as a basis for selecting a certain focus in the interviews. The interviews had two functions: to clarify what activities each student 101 Project DoCTA: Design and use of Collaborative Telelearning Artefacts had been engaged in, and to get insight into their view on why the different activities had taken place and why they chose to do them in this particular way. This whole process was very useful for identifying essential elements in their colloquial activity. The three individual, in-depth interviews can be labelled as semi-structured and open-ended. Four main topics or categories were outlined, and some questions about each topic were designed. This structure was used for all the interviews, but many “follow-up-questions” arose spontaneously from the situation or from the specific answer to a question. This description of how the data gathering was performed in this empirical study gives an actual example of what challenges that face researchers studying actors collaborating (partly) over the Internet, and how these challenges might be met. Data Analysis The analytical strategy used in this study included textual analysis (of emails and chat-logs), an analysis of events and process, and a contextual analysis where activity theory was used as a conceptual resource. In addition, some effort was put into analysing the use of the different artefacts. Overall, this strategy aims at gaining a holistic understanding of the collaboration process. Details of the analysis can be found in Guribye (1999). The group’s collaboration process can be divided into two main phases (see figure 17), which have basic differences in terms of the tasks in which they were engaged, the objectives they had, and the collaboration patterns that emerged. The first phase is referring to the period when they were participating in the IDEELS scenario (28/10 – 13/11). The following weeks (14/11 – 30/11) are denoted as the second phase. In this period they were engaged in carrying out tasks related to the assignment. The first few days of this second period can be said to be an ‘intermezzo’, where there was practically no collaborative activity at all could perhaps be described as “an anticlimactic” reaction after the scenario had ended. Overall, the two phases constitute a useful distinction for analytic purposes. 102 Chapter 6 Ideels Figure 17 The two phases in the collaboration process Characteristic to the first phase is the fact that the students were quite confused about both their intended role in the entire scenario, and the lack of a (for them) meaningful objective and significant motivation for their participation. All the same, the students ambitiously began to carry out the tasks and collaborated eagerly in the initial stage Their first introduction to the scenario was given very briefly at one of the lectures in the course they were taking. Beyond this, they only got the instructions and descriptions available from the web page (http:// www.zait.uni-bremen.de/wwwgast/fzhb/ideels/public_html/links.html) for the Eutropolis scenario. The first intra-group communication in relation 103 Project DoCTA: Design and use of Collaborative Telelearning Artefacts to the scenario was in a spontaneous face-to-face meeting, where they discussed their role in the scenario and also decided that they wanted to make a web-based newspaper. After going back to their respective hometowns, they used email to reignite the communication. The first task in the scenario was to publish an “internal briefing document” and a “positioning paper”. One of the group members wrote drafts of these, shared them as email attachments and then the other two members read and commented on these. This became the way they generally collaborated when producing documents for the scenario. In the “internal briefing document” they explicitly assigned internal roles for each of the group members with respect to their status as producing a newspaper. The roles were labelled as web-editor, journalist and editor with the responsibilities for designing and updating the WebPages, designing questions and ”interviewing” the other teams in the first phase, and writing and editing articles for the newspaper, respectively. When they divided the roles between themselves, they did this according to their different skills and experiences. For example, one had done some web design and volunteered as web-editor. These roles and their responsibilities were not rigorously applied to guide their division of labour and collaboration. As the scenario was running, new roles evolved and emerged. It was only the work with the Web Pages that was handled by one student, but this same student also had other roles to fulfil. At a later stage in this phase, they continued to publish some articles and questioned some of the other teams. The collaboration patterns varied as new tasks had to be done. When they were writing articles, one of the students took the initiative. But this student didn’t feel quite confident about his English proficiency, so he wrote them in Norwegian and sent them to the others so they could translate them to English, since the two others had more experience with writing this foreign language. At the same time, the student that wrote the drafts in Norwegian might be characterised as the one who had the best overview over both the scenario and the topics discussed there. So we see clearly that they choose a division of labour where they all get to do what they do best, and in this way pooling their abilities and resources in a (seemingly) optimal way. Another task they were obliged to in the scenario was to follow the conferences. In relation to this task another division of labour was chosen. As all of the group members were quite busy with other assignments, work etc., all of them didn’t have time to attend all of the conferences. Instead, usually, only one student followed each conference, saved it as a 104 Chapter 6 Ideels file and sent this to the others by email. One rather significant event early in the first phase was when one of the students tried to communicate to the other group members through the internal messages in OPUSi. He submitted his message, but had to wait for approximately half an hour before it was displayed in the system. Experiencing this kind of delay in the system, was a contributing factor for not using the internal communication facilities in OPUSi, and for choosing email as the main communication medium, at least in this phase. Their use of email had a peak in the very beginning of phase one. The same day as they published their first “articles” in the scenario, they sent over twenty emails to each other. After this the activity on email stabilised at around three to four email per day. They also had similar experiences with TW. During the first days, two of the students tried to use this tool, but one of them were frequently “kicked-off” due to technical problems. They also had problems with getting into the “room” created for the their group, and after a couple of days they also gave up on using this tool. Actually, the logs from TW indicate that one of the students did not log on to TW until the last week of phase two. Overall, after having started with a very high activity level, some confusion and with a considerable amount of communication, by the last one and a half weeks of the scenario they had reached a more stable division of labour and less communication took place. In the second phase distinctively different collaboration patterns emerged - they used the provided tools differently, had a clearer objective, and had more discussions on specific topics and tasks. This phase also began with a face-to-face meeting in Bergen. At this meeting they tried to get an overview of the task - what had to be done? In one of the interviews this meeting was characterised as very efficient, because they had ideas and suggestions that they could easily exchange and elaborate upon. They also divided the assignment into subtasks and made a preliminary division of the main responsibility for these subtasks between them. This seemed (as perceived by them) to be a fruitful way to start the collaboration with respect to the assignment. After this they went each to his/hers town, and did some work on their own. They then exchanged the produced documents (drafts) with each other (first using email) and made comments on this work in the email body. The last week, they also used TW extensively, having (at least) six long chat-sessions and also used the facilities for asynchronous communication provided in the software. During this last period the 105 Project DoCTA: Design and use of Collaborative Telelearning Artefacts collaboration was really intensified, and they needed a medium for synchronous communication, and had to give the work with the assignment a high priority (as we so often see when students have an assignment due). Since they began using TW the number of emails dropped significantly in relation to some of the peaks we saw in phase one, and they “only” sent a couple of emails each day. Another reason that made it possible for them to collaborate through TW was that the Teamwave server was moved to another server thereby providing a more stable, reliable and accessible service for them. Together with email they used TW to exchange documents. By leaving the documents on the whiteboard they could access each other’s files whenever they wanted. The coordination and version control regarding these files was to a large extent managed by putting up “post-it notes”. They also used the highlight and “track changes” tools in Word to comment on each other’s work. One of them downloaded a Word document one of the others had authored and highlighted or made a strikethrough (which means it should be deleted) on certain sections and made editorial comments to these. This way they worked closely together producing the documents, and had a subtle and dynamic division of labour throughout the last part of the second phase. The objective in this phase was finishing the assignment, and the outcome of the transformation process is the written document. Analysing the context using the activity system To really understand what was going on and how the students organised their work, considerable effort had to be put into uncovering and describing different aspects of the context. In doing this the activity theory framework played an important role. It provides concepts and ideas that make it easier to capture and analyse the context. Figure 18 represents the collaboration process as an activity system employing the general model of activity system proposed by Engeström (1987). In this specific activity system the main components are the individual students as the subjects, the student group as the community, and the object/objective is finishing the assignment. The activity was mediated by a number of artefacts, including language (written and oral), their email systems, TW and all its tools, OPUSi, and the WWW (the browsers and the sites they visited and created). 106 Chapter 6 Ideels Figure 18 The collaboration process as an activity system The relation between the community and the subject was mediated by a set of rules (both implicit and explicit) of interaction. These rules are mostly present in the different cultures permeating the social setting in which the activity took place. Certain rules and norms are more or less a part of the, mostly, tacit body of knowledge that constitute the different cultures, for example a university culture, the student culture or the discipline culture. Interaction on the Internet can also be said to be subject to certain rules and norms, and as all of the students had some experience with this medium and its culture, this can be established as a contributing factor in mediating the relation between community and subject. The widely used and well-known Internet “genre”, including symbols such as “smiley faces”, was to a large extent present in the intra-group communication. This, actually, was somehow problematic at the conferences in the IDEELS scenario. There the participants were supposed to apply a rather formal “tone”, and when the students from this team used (as they usually do when chatting on the Internet) “genre specific signs”, the other teams didn’t understand what this was supposed to mean (many of the other teams had less experience with communication at the Internet), and it was interpreted as “inappropriate”. In the intra107 Project DoCTA: Design and use of Collaborative Telelearning Artefacts group communication, however, everybody was familiar with these symbols, and they were used frequently in both emails and chats. At large, all these norms and rules are not static, but are continuously being constituted and dynamically evolve as they are used in new situations and settings. The relation between the community and object is, on the other hand, mediated through a division of labour and the different roles and responsibilities the students have towards the object. This division of labour is also changing and evolving throughout the collaboration process and is closely related to the way the students divide the assignment into tasks and subtasks. Another distinction made by activity theory is that of the hierarchical structure of an activity (Leontiev, 1978). The longer-term formation of an activity where its object is transformed into an outcome, consists also of shorter-term processes, namely actions and operations (Kuutti, 1996). Analysing this structure in this specific context may contribute to the understanding of certain events and happenings in the collaboration process. The activity, as a whole, can be characterised as “finishing the assignment”, and this is also the objective or motive for the activity (see figure 19). Again, this activity consists of a number of actions directed toward a goal. Examples of actions in this activity is “writing an email to the other group members”, “participating on a conference”, “having an online discussion in TW”, “publishing an article in the scenario”, etc. It is important to note that the actions can be both individual and collaborative. At the lowest level, all these action are realised as chains of operations. As an example, the action “writing an email” consists of operations such as “typing”, “selecting appropriate words”. This hierarchical model of an activity reveals some problems that occurred during the collaboration process. Especially in the first phase, the motive or the objective was a little diffuse for the students. This means that all the actions were not directed towards the same overall object and motive. This is perhaps due to the fact that the students did not perceive a clear purpose (other than “just participating to have something to write an assignment on later”) when participating in the scenario. So, the only motivation left then is the particular goals of each of the actions. In this sense one could say that the actions, in lack of an overall object, have shifted upwards to become activities. At the operation level, some 108 Chapter 6 Ideels Figure 19 The hierarchical structure of an activity technical obstacles (conditions) appeared, such as having to wait half an hour for the internal message to be displayed in the OPUSi system. Then what should have been operations required a fair amount of attention, and were turned into actions directed towards conscious goals. When the mediational tool (e.g. OPUSi) contains hindrances and requires much attention from the users, it is easy to understand that they choose other more familiar, reliable and “internalised” artefacts, such as email. Highlighting these elements of the context makes it possible to better understand the way the students organised their work and is at the same time a frame for comprehending the different actions and events that took place during the collaboration process. Conclusions Through this naturalistic study, significant aspects of the collaboration activity have been identified. When looking at this activity from the angle chosen in this study (strongly emphasising the social setting), it is apparent that – when using technology as a medium for communication and collaboration – the social interaction is to a certain degree influenced by the tools, but not fully determined by them. There are certain characteristics of the social context and how the participants perceive their tasks that are, at least, equally important to take into consideration as the strictly technical aspects when looking at technology in use. Of course, the technology in it self is a prerequisite for being able to engage in computer mediated collaboration. If, at a very basic level, the technology is not available or not reliable one can not make use of it as 109 Project DoCTA: Design and use of Collaborative Telelearning Artefacts a medium. Still, if all the technology (artefacts/tools) is provided and functions properly, its potential users need, first, to have a sufficient incitement and motivation to use it, and secondly, know how to use it, and finally, to “trust” it and feel comfortable with using it. These three considerations are not static and given, but are dynamically evolving as part of complex social and psychological processes, and dependent on the social context in which they occur. When a set of artefacts are provided, forming a “virtual environment” for the collaboration, the activities can be studied at another level. Basically, one can say that (in this case) the students didn’t make use of the different tools if they did not really need to use them. This is not one of the most interesting and surprising findings in this study, but is in a certain sense a basis on which the other findings rely. When looking for collaboration patterns and work organisation in this context, it is important to take into consideration that they organised their work and that the collaboration patterns emerged in a process mediated by the artefacts they used. For example, when they, in the beginning of the first phase, came across some technical and practical problems with TW, email took the role as the main communication medium. As we all know, email is a tool that only supports asynchronous communication, and this turned out to be rather problematic in the sense that they really needed a tool for synchronous communication, such as the chat-tool in TW, to support their collaboration at this stage. As a consequence, the quality of their communication suffered, and it was obvious that not all of them were able to follow every aspect of this discourse when they tried to emulate synchronous communication through email. Apparently, this example illustrates the importance of having a good blend of tools supporting both synchronous and asynchronous communication available in the virtual environment. At the same time, acknowledging that they somehow did solve the problems and tasks at hand, it suggests that the particular medium or tool is influencing, but is not determining the course of the collaboration activity. Using concepts from activity theory, the mediating tools are only one aspect of the entire activity that also appears at several levels (operations – actions – activity). When everything at the operation level functioned properly (technically speaking), the students could focus on their conscious goals and thereby discuss the problems/tasks/topics at hand more 110 Chapter 6 Ideels carefully, without having to pay much attention to the tools, as such. This was the case in phase 2 in the scenario, when TW was used actively and both synchronous and asynchronous communication was supported. Different collaboration patterns became visible and this clearly had a positive influence on the quality of the discourse. As always, this is not the only contributing factor at play in this complex situation. The group had, in this phase, better motivation and a clearer objective in their collaborative activity, and this might have had a positive effect on the discourse. This shows that through the lens of activity theory, it is not sufficient to see only the technology or a tool in isolation. Rather, technology must be seen as an integral part of human activity and human activity is always a complex and dynamic process that needs to be studied in situ. From an epistemological perspective, there are certain ways the findings of this study should be handled. In a qualitative case study, such as this, the generalisations should be made from empirical data to theory and concepts and not to a population of cases. In this way the concern for the uniqueness of the particular setting and participants withstands. Bearing this in mind, some concluding suggestions about what is important to take into consideration when designing collaborative telelearning scenarios are listed. A crucial aspect is the importance of having a variety of technologies (or one integrated technology with a variety of tools) available to support both synchronous and asynchronous interaction in a collaborative activity. In this way the social actors can swap easily between different modes of communication depending on the particular need in the specific situation. Other significant prerequisites include giving the participants sufficient training in both use of the technology – ensuring that they are comfortable with using it, and in how to integrate the technology in the learning activity. One last fundamental, and perhaps obvious, suggestion is to provide the participants with clear objectives and goals for the learning activity, and in this way arrange for proper motivation. 111 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 6.2 Changes in Attitude towards Simulation-based Distributed Learning KNUT EKKER This section reports on an evaluation (http://wwwstaff.hint.no/~kne/ ideels/results) of changes in attitudes towards simulation-based distributed learning in the IDEELS Scenario. All IDEELS participants, not just the Norwegian students at HiNT and Bergen, could participate in the evaluation. This quantitative evaluation of IDEELS was administered with web-based questionnaires. Pre- and post-simulation questionnaires (see Appendicies H & I) were used and each student had a personal id code that was the key to linking the pre- and post-simulation data sets. Pre-questionnaire The pre-simulation survey (http://wwwstaff.hint.no/~kne/ideels/ pre_sim_1998.htm) collected background data on the participants (e.g., age, year of university/college, native language, etc.) and asked questions about their: • attitudes and previous experience in using computer • language related skills and attitudes, and • experience with collaboration and team work Forty-six students answered the pre-questionnaire. Background information on participants The background information of the participants shows that the teams were quite different with respect to age, gender, year in university/college and whether they participated through a required class. Table 9 shows the distribution along these variables. The students in Bergen and Steinkjer were generally older than the students in Valencia and Bremen, and the majority of students were female in Bremen and male at the other sites. Students in Valencia and Bergen/Steinkjer were more often in their 5th year of university/college education whereas in Bremen the majority of students were in their 3rd 112 Chapter 6 Ideels or 4th year of post-secondary education. The majority of the students in Bremen, and all in Valencia, are participating through a language class. Further the students in Bremen are participating as part of an elective class, while most students in Valencia and Bergen/Steinkjer were participating as part of a required class. Table 9 Distribution of background information by Site 6.2.1.1.1 Variable Age Bremen Total (N) 40,0 40,0 20,0 100,0 (5) 69,7 21,2 9,1 100,0 (33) 12,5 37,5 50,0 100,0 (8) 56,5 26,1 17,4 100,0 (46) Male Female 40,0 60,0 75,8 24,2 62,5 37,5 69,6 30,4 1 or 2 rd th 3 or 4 th 5 or later 20,0 80,0 3,1 25,0 71,9 25,0 12,5 62,5 8,9 28,9 62,2 Context in which language class you technical class are participating in interdisciplinary class IDEELS interdisciplinary class 60,0 20,0 20,0 100,0 42,9 57,1 80,0 2,2 6,7 11,1 75,0 25,0 71,7 28,3 Gender Year in University or College Participating through a required class ? 20 – 22 years 23 – 25 years 26 years or older Site Valencia Bergen / Steinkjer st nd yes no 100,0 81,8 18,2 Attitudes and previous experience in using computer Table 10 shows that 75% of the students have a computer at home, and that most students in Valencia or Bergen/Steinkjer have owned home computers for four years or more. Fifty percent of the students or more use the computer for an average of two hours or less per day. 113 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Table 10 Distribution of ‘computer at home’ and ‘computer use’ by Site. Variable Bremen Computer at home I have owned a computer for… Average # of hours a day using computer Yes No Site Valencia Bergen / Steinkjer Total (N) 80,0 20,0 100,0 (5) 96,9 3,1 100,0 (32) 75,0 25,0 100,0 (8) 91,1 8,9 100,0 (45) 3 years or less 4 years or more 75,0 25,0 6,3 93,7 100,0 12,0 88,0 2 hrs or less 3 hrs or more 60,0 40,0 75,0 25,0 50,0 50,0 71,0 29,0 Experience with collaboration and teamwork Table 11 shows the extent of experience with teamwork, role playing and negotiating games distributed by Site. Most students have experience with working in teams, while few students have experience with role playing, and even fewer have participated in negotiating games. Table 11 Distribution of experience with teamwork, role playing and negotiating games by Site. Variable Bremen Site Valencia Bergen / Steinkjer Total Experience with working in teams none / little some / a lot 100,0 (N) 20,0 80,0 100,0 (5) 45,4 54,6 100,0 (32) 12,5 87,5 100,0 (8) 36,9 63,1 100,0 (45) Experience with role playing None / little Some / a lot 80,0 20,0 84,8 15,2 87,5 12,5 84,8 15,2 Experience with negotiating games None / little Some / a lot 100,0 94,0 6,0 100,0 95,6 4,4 Post-questionnaire The number of participants who completed the post-simulation survey (http://wwwstaff.hint.no/~kne/ideels/post_sim_1998.htm) was 36, compared with 46 participants in the pre-simulation survey. In Bergen/ 114 Chapter 6 Ideels Steinkjer the number of participants completing the post-simulation questionnaire was reduced from 8 to 4 and in Bremen there were only 3 participants in the post-simulation survey. Thus, there will be some large percentage-wise differences that may only reflect the fact that there were only 3 or 4 participants in two of the sites. Table 12 shows the evaluation of the simulation distributed by site. Four individuals (13,8 %) in Valencia and two (50 %) in Bergen / Steinkjer were neutral or negative in response to the item “I enjoyed this experience”. Most of the participants felt this was a good learning experience (over 90 % across the three sites), and most of the participants felt this simulation fit well with the course they were taking (more than 73% across the three sites). Table 12 Evaluation of the simulation by Site Variable I enjoyed this experience Bremen agree neutral or disagree (N) 100,0 100,0 (3) Site Valencia Bergen / Steinkjer Total 86,2 13,8 100,0 (29) 50,0 50,0 100,0 (4) 83,3 16,7 100,0 (36) This was a good agree learning experience neutral or disagree 100,0 96,6 3,4 50,0 50,0 91,7 8,3 The simulation fit well agree in the course I was neutral or disagree taking 100,0 75,9 24,1 50,0 50,0 73,6 26,4 Gender In order to find out to what extent gender has an effect on the experiences in the simulation, we present tables 13 – 16. Table 13 shows that female participants were overall more positive to the simulation experience (90 or 100% positive) compared to the males (64 to 88% positive). 115 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Table 13 Evaluation of the simulation by Gender Male Gender Female Total (N) 80,0 20,0 100,0 (25) 90,0 10,0 100,0 (10) 82,9 17,1 100,0 (35) This was a good agree learning experience neutral or disagree 88,0 12,0 100,0 91,5 8,5 The simulation fit well in agree the course I was taking neutral or disagree 64,0 36,0 100,0 72,7 27,3 Variable I enjoyed this experience agree neutral or disagree In table 14 we see that gender does not have any effect on the evaluation of the simulation as a cross-cultural experience. In general, most participants (60 %) felt that cultural differences were easy to identify, but most participants (80 % or higher) did not feel that these cultural differences made it difficult to communicate. Table 14 Cross-cultural experience by Gender Variable Male Gender Female Total Cultural differences agree were easy to see neutral or disagree 60,0 40,0 100,0 (25) 60,0 40,0 100,0 (10) 60,0 40,0 100,0 (35) 4,0 96,0 20,0 80,0 8,6 91,4 (N) It was difficult to agree communicate neutral or disagree because of cultural differences Table 15 shows gender differences in the evaluation of the teamwork in the simulation. All males feel that the teamwork functioned well with respect to participation by all members, while a minority of the female participants (22,2 to 30%) felt that not everyone participated in team decisions. 116 Chapter 6 Ideels Table 15 Experience with teamwork by Gender Male Gender Female Total (N) 100,0 100,0 (25) 22,2 77,8 100,0 (9) 5,9 94,1 100,0 (34) Everyone participated agree in team decisions neutral or disagree and product N) 92,0 8,0 100,0 (25) 70,0 30,0 100,0 (10) 85,8 14,2 100,0 (35) Variable One or two members took over agree neutral or disagree Table 16 shows the overall evaluation by gender and shows that both males and females felt that most, or all, of the goals or tasks were accomplished in the simulation (88 vs 80%, respectively). Table 16 Overall evaluation by Gender Overall evaluation We accomplished .. .. some of our goals or tasks .. most of our goals or tasks .. all of our goals or tasks (N) Male Gender Female Total 12,0 80,0 8,0 100,0 (25) 20,0 70,0 10,0 100,0 (10) 14,3 77,1 8,6 100,0 (35) Knowledge of computers Table 17 shows the effect of background knowledge of computers on the overall experience of participating in the simulation. We see that a larger percentage of the participants with an excellent knowledge of computers prior to the simulation accomplished “all of our goals or tasks” (25%). Of the participants with poor to good knowledge of computers, none accomplished “all of our goals or tasks”. 117 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Table 17 Overall evaluation of the simulation by Knowledge of computers Evaluation Please rate your knowledge of computersOverall poore-good very good exelent Total I We accomplished .. .. some of our goals or tasks .. most of our goals or tasks .. all of our goals or tasks (N) 18,8 81,3 9,1 100,0 (16) 9,1 81,8 25,0 100,0 (11) 12,5 62,5 8,6 100,0 (8) 14,3 77,1 100,0 (35) Attitude change This section summarises the survey findings with respect to the research question on attitude change. Table 18 shows the attitude change with respect to the variable “A good learning experience” by gender. The post evaluation score on this question was subtracted from the pre evaluation score (a value of 1 indicated “strongly agree” to the question “This was a good learning experience (post)” and “I anticipate this will be a good learning experience (pre)”). The table shows the majority did not show any attitude change (54.3 % indicate “no change”). The table also shows differences by gender — males are more likely to indicate an attitude change — about ¼ in either direction (“more negative” or “more positive”) while female participants 70 % indicate no change in attitude towards regarding simulations as a good learning experience. Table 18 A good learning experience (attitude change) * Gender Male This will be/was a good learning experience (attitude change) Gender Total More negative 28.0% No change 48.0% More positive 24.0% 25 100% Gender Female Total 10.0% 70.0% 20.0% 10 100% 22.9% 54.3% 22.9% 35 100% Table 19 shows the attitude change with respect to the variable “The simulation game will be (or was) difficult” by gender. The pre-evaluation score on this 118 Chapter 6 Ideels question was subtracted from the post evaluation score (a value of 1 indicated “strongly agree” to the question “I am concerned the simulation game will be difficult (pre)” and “The simulation game was difficult (post)”. The table shows that almost half (48,6) of the participants felt the simulation was more difficult than expected — but when distributed by gender we see that 60 % of the female participants expressed no or a positive attitude change. Table 19 The simulation game will be/was difficult (attitude change) * Gender Male The simulation game will be/was difficult (attitude change) More negative 52.0% No change 32.0% More positive 16.0% 25 100% Total Gender Female Total 40.0% 30.0% 30.0% 10 100% 48.6% 31.4% 20.0% 35 100% Table 20 shows the distribution of the attitude change variables by each other. If you experienced the simulation as more positive than you expected, you are more likely to express that this was a good learning experience (28.6 % vs. 17.6 %). But just as many expressed that the simulation game was more positive than expected, but found it to be a more negative experience (28,6 %). It should be kept in mind that there were only 7 individuals in this category (“More positive” with respect to the difficulty of the simulation). Table 20 A good learning experience * The simulation game will be/ was difficult The simulation game will be/was difficult (attitude change) More No negative change positive This will be/was a learning experience (attitude change) Total More negative No change More positive 23.5% 58.8% 17,6% 17 18.2% 54.5% 27.3% 11 100% 28.6% 42.9% 28.6% 7 100% More Total 22.9% 54.3% 22.9% 3 100% 119 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Other tables from the quantitative data analysis of Scenario I of Project DoCTA / Project IDEELS may be found at this URL: http:// wwwstaff.hint.no/~kne/ideels/results/. The discussion on attitude change in Project IDEELS shows that quite a few of the participants had a more negative attitude towards the simulation after completing the participation compared with the attitude they had in the outset. This was particularly the case for male participants. Another effect by gender is the fact that female participants were more likely to experience frustration with respect to other team members “taking control” over the team effort. Two conclusions may be drawn from these facts. First, the software utilised (OPUSi) may be improved in order to alleviate some of the frustration experienced. Second, in preparing for future simulations it will be important to focus on integrating all members of the team in order to avoid having one or two participants “taking control”. 120 Chapter 7 VisArt 7 VisArt The VisArt scenario involved students taking courses at three educational institutions, the University of Bergen (UiB), Nord-Trøndelag College (HiNT), and Stord/Haugesund College (HSH). Teams comprised of 3 students, 1 student from each institute, collaborated to design a learning activity in TW. There were no opportunities for the teams to meet face-to-face. TeamWaveWorkplace (TW) was used as the main information and communication technology. The VisArt activity took place during February and March 1999 and provided an opportunity for the DoCTA project to study an authentic collaborative telelearning activity. One week of training in using the TW tools and in collaboration proceeded three weeks of design activity. This section describes how we designed the VisArt scenario, discusses its deployment and presents overviews of the evaluation studies on VisArt. 7.1 VisArt Design VisArt was a somewhat special scenario. From its inception it was designed to serve two purposes, each with different, but complementary goals. As a course assignment the goals were to enable the students to: • participate in an authentic collaborative telelearning activity • evaluate their own participation from a theoretical perspective and, the research goals were to: • study / evaluate a collaborative telelearning activity from a sociocultural perspective • study / evaluate distributed collaboration artefacts • find out how students experience it • find out their own explanations and evaluation of the activity 121 Project DoCTA: Design and use of Collaborative Telelearning Artefacts The design process As indicated earlier, VisArt was to be part of three different courses at Norwegian educational institutions. As the University of Bergen (UiB) was leading the scenario, the VisArt activity, including the evaluation, was designed by the UiB researchers and then sent to the other institutions for reactions. This description describes aspects of the design process. As Fjuk, Sorensen & Wasson (1999) argue, the design of telelearning scenarios is a product of complex interactions between pedagogical, technological and organisational perspectives. Each of these perspectives place constraints on the design. Figure 20 illustrates the major constraints that were placed on the design of the VisArt scenario. In addition, as VisArt was part of a research project, evaluation concerns played a role in the design. Figure 20 Constraints on the VisArt design The design of the VisArt activity took place from early November 1998 until mid-February 1999. This may seem like a long period, but one aspect of telelearning activities is that a lot of preparation goes into designing and preparing an activity. Tasks included in the VisArt design phase were choice of design task, specification of the technological environment, coordination of dates for the activity, design of training, design of help and assistance during the deployment, specification of the design task, and design of evaluation instruments. In addition, coordination and management of all these tasks needed to be carried out. 122 Chapter 7 VisArt The design was initiated as a brainstorming activity at a DoCTA workshop in early November 1998. In the project application it had already been determined that: • teams of students would collaboratively design a visual artefact, and • a team of students would comprise one student from each of the educational institutions Previous work with the other scenarios and evaluations of collaboration technologies, lead us to decide that: • TeamWave Workplace (TW) would be the collaboration technology, and • the students would use their own email system and web browser During the workshop we decided that: • the VisArt students would design a web page for teaching some topic of choice • two of the graduate students would implement a new tool, a web palette, in TW to support this design We decided that the first step was the implementation of web palette and several meetings were held where progress was demonstrated. Unfortunately, due to constraints on the type of tailoring allowed by the TW developers, it proved impossible to implement the tool as we wanted. At the time21 we were trying to add our own tool (i.e., November 1998) TeamWave did not officially support the development of new tools. There were guidelines for how to do so, but it was not easily done (e.g., the api/ documentation they provided was not sufficient enough, and the environment/framework also had “bugs” we could not work around). These factors made it impossible to build our own tool and integrate it into TW. Our only option was to change the design task. In December 1998 we had another group meeting where we brainstormed about other possible design topics and concluded that we would have the students design a teaching/learning room in TeamWave Workplace. In retrospect, this was a much better design task than the original we had chosen. Once the design task had been decided upon, we could begin serious planning and preparations of the VisArt design activity. 21 It is now, however, possible to develop your own tools through the inclusion of a software development kit (SDK). The SDK includes complete documentation, as well as the Tcl/Tk source code for all of the built-in TW tools. 123 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Modelling the VisArt Activity In order to assist in management and coordination of all aspects of VisArt, the activity was modelled22 in a knowledge modelling tool developed at the LICEF Research Centre at Télé-université, Montréal called MOT (modélisation par objets typés—modelling with object types). MOT (http:/ /206.167.88.162/anglais/real/mot.htm) is an object-oriented modelling tool for developing graphical knowledge models for learning environments. The MOT editor enables easy creation of hierarchical graphs that represent knowledge objects and the links between them making the nature and structure of a learning system explicit. MOT is most often used in conjunction with MISA (http://206.167.88.162/anglais/real/ misa.htm) a learning systems engineering method used to produce a learning system which is characterised by a knowledge model, a pedagogical model and a media model. MOT provides the modeller with a set of knowledge types (concept (rectangle), procedure or activity (oval), principle (rectangle with diamond ends, or fact (rectangle with missing corners) and links (C: component, S: specialisation, P: precedence/prerequisite, I/P: input/output, R: regulates/governs, In: instance of). For example, figure 21, shows the top level of the VisArt model. The model can be read as follows: The Visart concept (square) is a component (link C) of both DoCTA and 3 courses. The VisArt concept is input (link I/P) to two processes (ovals) “designing an instructional room in TW” and “evaluating the design of an instructional room in TW”. To indicate that the students are carrying out the design activity, a principle (student designers) regulates/governs (link R) the “designing an instructional room in TW” process. Similarly, researchers regulate/ govern the “evaluating the design of an instructional room in TW” process. Each of the two processes in figure 21 is expanded at a lower level. 22 Ileana de la Teja, a senior researcher at Télé-université with a lot of experience in the design and evaluation of telelearning scenarios, served as a consultant and helped Barbara Wasson model the scenario. Barbara had previous experience in using MOT from a 1 year visit to Télé-université. 124 Chapter 7 VisArt Figure 21 Top level of VisArt model Figure 22 shows the model of “designing an instructional room in TW”. This model can be read as the “training” process is a prerequisite (link P) for the “designing an instructional room in TW” process. The output (link I/P) of the “designing an instructional room in TW” process is the concept “instructional room in TW”. The “designing an instructional room in TW” process is regulated/governed by a “deadline” and “evaluation criteria”. There are several inputs (I/P) to the process including: “tools”, “assistance”, “instructions”, “infrastructure”, and “student designers”. 125 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 22 A second level of the model of VisArt Management and Coordination The completed model of the VisArt activity was used to identify tasks and to create management and coordination tables from three perspectives: the instructor perspective, the student perspective, and the researcher perspective. Each perspective had a table for preparation, training, design activity, and, if necessary, follow-up. These tables were used to orchestrate the activity. Some examples are explained below. Table 21 presents the preparation table from the instructor’s perspective. Each table has a column for Tasks, a Date the task is to be completed, Data sources for the task, the State of source, Comments, and who is Responsible. As can be seen in this table, the instructor has a lot of preparation tasks. These tasks are shared among the head instructor (Barbara) and three graduate student assistants (Helge, Trond & Kurt). 126 Chapter 7 VisArt Table 21 Preparation from the Instructor’s Perspective (as at 24.02.99) Task Date Data source Prepare Assignment: • Course • VisArt 01.02 Word file, Prepared VisArt activity is accessible in TW State of source Comments Responsible VisArt activity Barbara is part 2; each instructor to choose how to use it Prepare Training 15.02 Activities Word files, accessible in TW Prepared Helge Prepare selfevaluation form Form on the web Prepared Helge Prepare Help 15.02 pages on the Web Web page Prepared Kurt Give out Assignment 16.02 Assignment document Prepared Set-up TW on server 08.02 TW Up & running on grevling.ifi.uib.no Set up accounts in TW 23.02 Participant & TO DO Team lists (from Barbara) 15.02 Create classroom 18.02 in TW Set up Teams Each instructor Barbara > to hand out & Arnstein & Lars go through with students Trond Participant lists Trond from Lars & Arnstein Prepared Barbara (w/ Helge & Kurt) 19.02 Team list (login Prepared names & teams) Received Barbara list from Instructors ; Assign login name and team Create team rooms 22.02 in TW Team01 to Prepared Team11 rooms in TW Make sure Barbara permissions are set properly Ask “drift” for email aliases 22.02 Team list Prepared Email list to “drift” Barbara Inform about TW license 24.02 License number Recieved Send email to participants Barbara Inform about start 24.02 of the scenario TO DO To be sent by email to all Barbara Prepare for 24.02 providing assistance TO DO Agree to assistance process Helge w/ Barbara & Trond 127 Project DoCTA: Design and use of Collaborative Telelearning Artefacts For example in the first entry it can bee seen that the assignment had to be prepared (“Prepare Assignment”) by the 1st of February. It had to be a word file that was accessible in TW and Barbara was responsible to develop it. When it was complete, the status was changed to Prepared. Eventually, Barbara would have to “Give out Assignment” and, as indicated, this would be done by Barbara giving the assignment to Arnstein and Lars the two instructors at HiNT and HSH, respectively. It was up to each of them to incorporate the part of the assignment into their course (more on this in section 7.6). As another example, graduate student Trond was responsible for the administration of the TW server. There are two tasks related to this, “Set up the TW server” and “Set up the TW accounts”. Other important tasks are preparation of the training activities (“Prepare Training Activities”) that were the responsibility of Helge (his Masters thesis topic) and preparation, by Kurt, of help pages (“Prepare Help pages on the Web”) on the web. Such an overview of all the tasks made management of the scenario go smoothly. Similar tables, see Appendix J, existed for the training, design activity and follow-up phases. As mentioned previously, there also were tables from the student perspective and the researcher perspective. Table 21 shows the preparation phase from the researcher perspective. The majority of tasks in this perspective address preparation of evaluation instruments such as preand post-questionnaires, and preparing and getting student consent (to participate as a research subject). The remainder of the tables that show the scenario from the researcher and student perspectives can be found in appendices K and L. 128 Chapter 7 VisArt Table 22 Preparation from Researcher’s Perspective (as at 24.02.99) Task Data source State of source Comments Responsible Prepare computer 01.02 supported collaborative learning assignment Date Assignment file (merge together activities) Part 2 of assignment is VisArt Barbara Prepare profile questionnaire 15.02 Profile Prepared questionnaire On web (make Kurt available 24.02) Prepare prequestionnaire 15.02 PrePrepared questionnaire On web (make Kurt available 7.03) Prepare postquestionnaire 15.02 PostPrepared questionnaire On web (make Kurt available 25.03) Prepare student consent form 15.02 Consent form Prepared Send to Lars and Arnstein Getting students’ consent By 24.02 Consent form Prepared Receive from Barbara Lars and Arnstein as well Assign researchers to a team 21.02 Team list Getting students’ profile As needed TO DO Barbara > Lars & Arnstein Barbara > researchers Prepared Profile Prepared questionnaire file Prepare for video 24.02 taping Prepare for interviews Prepared student > Kurt Eskil & Knut TO DO Each researcher This section has presented the design of VisArt and has shown how we managed all of the tasks that needed to be coordinated. It is necessary to spend the needed time on preparing and organising a collaborative telelearning activity. Otherwise one can run the risk of chaos and frustration during the deployment of the activity. Student Consent to Participate in Research The DoCTA project is registered with NSD (Norwegian Social Science Data Services) and their regulations for data collection, storage and deletion are being followed. Thus, in order to collect student data in the VisArt 129 Project DoCTA: Design and use of Collaborative Telelearning Artefacts scenario, the students had to give their consent. A glimpse of the student consent form for participation in the research project is given below. The full consent form can be found in Appendix M. The students were informed about the VisArt activity’s role in the DoCTA project as follows: “Our intention is to gain a better understanding of student learning in this new mode of course delivery, more specifically collaborative learning over the internet with a distributed team of students. As part of a course at each of the teaching institutions, a collaborative telelearning activity, called VisArt, has been designed to have teams comprised of a student from each of the teaching institutions collaborate through an internetbased groupware system called TeamWave Workplace. Through your responses to questionnaires, interviews and through electronic data collection we hope to find more about your opinions of the software, the activities, collaboration, telelearning in general, etc. There are 8 graduate student projects associated with VisArt and we hope that you will agree to provide them with enough data that they can carry out their projects. (VisArt consent form, see Appendix M)”. After indicating what and how data would be collected, the consent form indicates that their participation as a research subject is completely voluntary and will not have any repercussions on their course grade: “Your cooperation in the collection of this information is greatly appreciated, but is completely voluntary. You have the right to choose not to participate in this research project. Your decision will in no way affect your grade. Please indicate by checking in one of the boxes below whether you freely consent and agree to participate in data collection for research in collaborative telelearning. If you have any questions regarding these procedures, please feel free to contact Barbara Wasson at 55 58 41 20 or by email at [email protected]. (VisArt consent form, see Appendix M)” All students at the three institutes agreed to participate as research subjects. The collected data will be stored until the end of the project and then deleted according to NSD guidelines. 130 Chapter 7 VisArt 7.2 The Computer Supported Collaborative Learning Assignment As mentioned previously, UiB had responsibility for preparation of the VisArt scenario. The UiB students would participate in VisArt a graduate course where the first semester assignment would involve participating in the VisArt activity. Participation alone, however, was not enough. Since this is a graduate course and two of the topics the students were learning about were collaborative learning and telelearning, the VisArt activity provided an opportunity for both participation in collaborative telelearning and an opportunity for theoretical reflection. A semester assignment, found in Appendix N and described in more detail below, was designed to cover both opportunities. The assignment, which entailed reading CSCL literature, participating in the VisArt activity, and writing an individual report that individual report that reflected on the process from both a practical and theoretical perspective, was adopted at both HiNT and HSH with minor adjustments (more on this in section 7.6). Each of these explored in more detail below. The Background Literature As computer supported collaborative learning (CSCL) was a topic covered in all three of the courses, it was natural that the students be given CSCL literature. The students were told that the following literature should be read: Dillenbourg, P., Baker, M., Blaye, A. & O’Malley, C. (1995). The Evolution of Research on Collaborative Learning. In P. Reimann H. Spada (Eds). Learning in humans and machines. Towards an interdisciplinary learning science, 189- 211. London: Pergamon. Koschmann, T. (1997). Paradigm shifts and instructional technology: An introduction. In T. Koschmann (Ed.) CSCL: Theory and practice of an emerging paradigm, 1-23. Mahwah, NJ: Lawrence Erlbaum Associates. Salomon, G. (1992). What does the design of effective CSCL require and how do we study its effects? SIGCUE Outlook, Special Issue on CSCL, 21(3), 62-68. 131 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Gutwin, Carl; Stark, Gwen; Greenberg, Saul: Support for Workspace Awareness in Educational Groupware. Proceedings of CSCL’95. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. http:/ /www-cscl95.indiana.edu/cscl95/toc.html. In addition, a number of web pointers to CSCL related Internet sites were given, see Appendix N. The students were encouraged to read as much as possible on CSCL. The VisArt Activity The objectives of the VisArt activity were given as follows: • To provide an opportunity for participation in a distributed collaborative learning activity where team members do not have the opportunity to meet face-to-face. This will give experience in both distributed collaboration and in the use of a groupware system over the Internet. It will hopefully highlight both positive and negative aspects. • If each team member has given consent to participating in the research aspects of this scenario, your team’s experience will also contribute to furthering our understanding of this new learning phenomenon. The students were informed that in the VisArt activity they will be part of a group of 3 students, 1 from Bergen, 1 from HiNT, and 1from HSH, who will form a design team that will: • Organise a collaborative team effort • thinking Salomon’s definition genuine interdependence: 1) sharing information 2) division of labour 3) joint thinking • Carry out the Design Activity in TeamWave Workplace (TW) with the help of: • help pages (http://www.ifi.uib.no/docta/VisArt/help) • assistance (email [email protected]) • Help Room in TW • Produce 2 items: • A document of your pedagogical decisions (e.g., who is the room intended for, what is the content, etc….) • A TW room for teaching/learning 132 Chapter 7 VisArt The VisArt activity consists of 5 major activities as shown in Table 23. Two of the activities, pre-questionnaire and post-questionnaire were part of the researcher’s evaluation of VisArt and are described in section 7.11. Table 23 The 5 VisArt Activities VisArt Activity Date(s) Preparation Week 8 (from February 25) Training on Tools & Collaboration Pre-questionnaire 23 Design Activity Post-questionnaire Week 9 (March 1-5) March 7 Week 10-12 (March 8-26) 24 by March 26 Preparation. The students were told that in preparation for VisArt they need to download TeamWave Workplace and some extension files. In order to make this as easy as possible for the students, Trond Pedersen25 created a web page with directions for downloading. This can be found at http:/www.ifi.uib.no/docta/teamwave/install.html, see also Appendix O. In addition, the students were told to use the logon id that their instructor had given them and were told what their team email was. They were informed that if they experience any problems they should contact Helge Underhaug26 by email at [email protected]. If you have not delivered your consent form to your instructor please do so. If they had agreed to participate in the research, they were asked to fill in a participant’s profile questionnaire that can be found at http:// www.ifi.uib.no/docta/VisArt/profile.html and also in Appendix P. Training. During the training session they were taken through a number of activities where they gained experience in using the technological tools and gained experience in collaborating while at the same time getting to know their team members. A self-evaluation questionnaire (see Appen- 23 24 25 26 Only to be filled in student has consented to participating in the research study. Only to be filled in student has consented to participating in the research study. Trond, a graduate student working on DoCTA, had administrative responsibility for the TW server. Helge, a graduate student working on DoCTA, had responsibility for designing the training and served as head assistant during VisArt. 133 Project DoCTA: Design and use of Collaborative Telelearning Artefacts dix Q and http://www.ifi.uib.no/docta/VisArt/self-evaluation.html) was provided to help them gauge their competence in using the tools. All training took place in TeamWave Workplace Training Room. More detail on the training activities can be found in section 7.5. Pre-questionnaire. The pre-questionnaire that asks about attitudes towards TeamWave Workplace so far needed to be filled in before the design activity began. The questionnaire can be at http://www.ifi.uib.no/ docta/VisArt/pre-questionnaire.html and also in Appendix R. Design Activity. In the VisArt design activity the teams were to design a TeamWave Workplace room for learning about some topic. As we were interested in the process of collaborating – how decisions were made and carried out, the outcome of this activity included the learning room as well as documentation about major pedagogical decisions that were made. More details about the design task were given as follows: 1. The room you chose to design should enable the students to know more about a concept, a procedure, a theory, a process, etc. Some examples of possible topics include a room for: • 6th grade science (e.g., birds, dinosaurs, pollution, etc.) • high school geometry • a university course on programming (e.g., recursion, file structures, etc.) • a theme in an art history course • OR a topic approved by the activity leader ([email protected]) 2. The room could contain a presentation, a game, a tutorial, a questionnaire, an exercise, or some combination of these and utilise a number of the TeamWave Workplace tools. The room you develop should be under your team’s working room. 3. When documenting the pedagogical decisions that your team makes, you should include issues such as characteristics of the student audience (e.g., age, grade, etc.) and the complexity of the subject area. Create a file PedDecteamx (i.e., use the fileholder tool and substitute your team number in teamx – team 1 is PedDec-team1) and leave the file in your working room. 134 Chapter 7 VisArt Post-questionnaire. The post-questionnaire asked about attitudes towards the training, telelearning and TeamWave Workplace after the VisArt activity had been completed. It can be found after at http:// www.ifi.uib.no/docta/VisArt/post-questionnaire.html and in Appendix S. The Individual Report The individual report is the document the UiB students handed in to be counted towards their grade for the semester assignments. This is the part of the assignment that was tailored to fit the courses at HiNT and HSH (see section 7.6). The students were told that the individual report should contain: • an introduction to computer supported collaborative learning and collaborative telelearning (use literature in class + other information you might find on the web, in the library etc.) • a description of the design activity you were asked to participate in, including the tools provided and used (e.g., email, TeamWave Workplace, etc…) • a presentation of your team’s room (maybe take a screen snapshot here) and the pedagogical decisions you made to get to the solution • a discussion of how the team met Salomon’s requirements for genuine interdependence and whether or not TeamWave Workplace supported activities resulting from attempts at meeting Salomon’s requirements • a discussion of Gutwin et al.’s awareness concept and what it means in conjunction with your distributed collaboration through TeamWave – does TeamWave Workplace support awareness? How? • your general reaction to collaborative telelearning (as you experienced it) including: a reflection on the team’s work focusing on your participation and the others’ participation (did you feel you did more work, someone did not pull their weight etc. – such comments will not work against them (you do not need to give names) and are private to the instructor), the process of carrying out the assignment, general comments about the entire assignment, your reaction to TeamWave Workplace (did it help or hinder your work, was it easy to use, hard to use, etc.) 135 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 7.3 Communication VisArt Communication was designed to take place directly through TW or through team email. In TW, both asynchronous and synchronous communication was possible. The Chat and Page functions are designed for synchronous communication (users who are logged onto the TW server at the same time) and tools such as the MessageBoard, PostIt, Whiteboard are designed for asynchronous communication. In order for teams to communicate without logging into the TW server, each team was given a team email account (on the IFI machines) through which the team could contact each other. The students were told that their team email would be collected as data for VisArt evaluation. Thus, in addition to the 3 team members, the team email included one researcher who received all the team’s email and was responsible for its archiving. The head instructor could also use the team email accounts to send messages about, for example, the start of the scenario, approval of the choice of domain, etc. In addition, help assistants were available by email. Assistance in VisArt is discussed in the next section. 7.4 Assistance in VisArt Another aspect to which a lot of attention was paid was how to provide help and assistance to the students. We used a model that was introduced to us by Ileana de la Teja, a senior researcher at Télé-université. Figure 23 presents our design. 136 Chapter 7 VisArt Figure 23 Organisation of Assistance in VisArt In this organisation, all help requests go through a head assistant via email. The head assistant then either answers the help request, or sends it further to either the content assistant or the technical assistant who become responsible for answering the question. If the assistant answering the question feels that the question and answer are relevant to all the students, the answer can be emailed to all students and not just the student asking the question. Figure 23 illustrates the case where a student sends an email question to the head assistant who determines that the mail should be forwarded to the technical assistant. The technical assistant answers the student’s question directly. 7.5 VisArt in TeamWave Workplace Included in the design of VisArt was the preparation of the learning environment in TeamWave Workplace. This section presents some of the rooms that were created for the students including, the Classroom, the Training Room and the Help Room. 137 Project DoCTA: Design and use of Collaborative Telelearning Artefacts The Classroom When a student logs onto the TW server, they find themselves in the Classroom. The Classroom, see figure 24, was created by the head instructor. Figure 24 The Classroom in VisArt In the Classroom there is a To Do List which lists a number to tasks for the students to carry out (e.g., change their password, add their name to the Address Book, go to their Team room and leave a message on a PostIt), etc.). There is an Address Book where students, instructors and researchers can enter contact information, a class Calendar where important deadlines are indicated (e.g., on March 3 Oppgave 4 legges ut means Assignment 4 will be available) and a Message Board where messages can be left for the instructor or researchers. There are also a number of doorways to other rooms including: the Help Room, the Training Room, and Team rooms (Team01 – Team11). There is a File Holder that contains the VisArt assignment (VisArt Oppgave) and four URL links to the profile, pre-, self-evaluation, and post-questionnaires. Entering the Training room Doorway, takes one into the Training Room as described below. 138 Chapter 7 VisArt The Training Room The Training room, see figure 25, was created and maintained by a graduate student27 responsible for training and assistance as part of his Master’s thesis (Underhaug, in preparation). Figure 25 Training Room in VisArt In the Training room, there are Doorways to the Classroom and Help Room, a URL link to the help pages on the web (see section 7.5) and a Self-Evaluation questionnaire (Egen-evaluering), a PostIt note telling the group to agree on when they will do the training, Doorways to 4 training rooms (Oppgave 1 room – Oppgave 4 rom) and a Welcome to the Training Room (Velkommen til Treningsrommet) written directly on the Whiteboard (see figure 26). 27 The training room and majority of the training tasks were designed by Helge Underhaug. 139 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 26 Welcome to the Training Room In the Welcome, figure 26, it says: “The goal of the training period is that you will become comfortable in the use of TeamWave Workplace and meet the others on your team. In order for your collaboration to be as best possible later in the scenario, it is important that you think about the ways in which you can work. This is a special theme in training assignment 4.” After the list of available help, the Welcome continues with “Here you find the doorways to the different assignment rooms. Each assignment is available in 3 formats MS Word, Mac Word and as text (.txt, assignment 2 as .rtf). Assignments 1-3 should be finished by March 3rd at 15:00, when Assignment 4 will become available.” Each of the four assignments was designed for learning either about TW or about collaboration, and were either an individual assignment (1 & 3) or a group assignment (2 & 4). Entering Oppgave 1 rom (Assignment 1 room), see figure 27, takes you to the first assignment. Here you are told to download the version of the assignment you desire and begin. To download the file you use the right mouse button and click on the version you want — choose either “Get file” or “View file”. If you finish assignment 1 and want to begin on another individual assignment, you can go to the Oppgave 3 rom (Assignment 3 room). All of the other three training rooms are arranged similarly. 140 Chapter 7 VisArt Figure 27 Training Assignment 1 room in VisArt Assignment 128 , see figure 28, was designed to help the team members get to know one another. Each team member was to interview another team member and present the information for the third member. Suggestions for which TW tools to use were also given. 28 Norwegian text for assignment 1 and 3-4 can be found in Appendix U. 141 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Assignment 1 Getting to know one another Each person on the team is to interview one of the other team members (i.e., A interviews B who interviews C who interviews After the interview you are to create a textual introduction of your partner to the 3rd team member (i.e., A writes an introduction of B, B writes an introduction of C and C writes an introduction of A). • Carry out the interview as follows: • Go the your group room. • Use Calendar to plan the time for the interview. • Use Chat to carry out the interview. • Create a ToDoList for your group. • Present the interview to the others in your group with the File Viewer. Create your own individual room (Room, New Room) and name it the same as your user name. To limit acess to the room, go the the Permissions menu and set the permissions so that only your team members can enter your room. Create a Doorway back to the classroom from your individual room. Figure 28 Training Assignment 1 Assignment 2 is a team building activity that is based on a personal development game developed by Dr. Konrad Morgan29 . In the adventure game, Narg Island a shipwreck in a Northern Norwegian fjord is presented. Teams compete with one another to choose the best ACTION ALTERNATIVES, and SALVAGEABLE ITEMS. An excerpt of the presentation is given in figure 29. 29 Dr. Morgan, a psychologist, is an associate professor at IFI. 142 Chapter 7 VisArt Figure 29 An excerpt from the Narg Island Game Narg Island was adapted for use30 in a distributed environment by the head instructor, Barbara Wasson. Figure 30 gives the instructions for a distributed version of Narg Island. These directions were available as a file in the Oppgave 2 rom (Assignment 2 room). The complete Narg Island scenario, directions, alternative lists, expert rankings, and scoring tables are available in Appendix V. 30 With the permission of Dr. Morgan. 143 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Assignment 2 Narg Island Group Activity The teams will play Narg Island. Here is what you need to do: a. b. c. d. e. f. g. First, fetch the scenario (NargIsland-95.doc OR NarIsland-Mac.mcw) and read it. Then, fetch a list of ACTION ALTERNATIVES which lists 9 alternatives and a list of SALVAGEABLE ITEMS which lists15 items. These are Excel files: ActionAlternatives-95-xls, SalvageableItems-95.xls, OR ActionAlternatives-Mac. and SalvageableItems- Mac.csv. Then, Individually and without discussing The Situation, the ACTION ALTERNATIVES, or the SALVAGEABLE ITEMS, with anyone, rank (1 for most important, 2 for next most important, etc. with a maximum of 9 for the ACTION ALTERNATIVES and 15 for the SALVAGEABLE ITEMS) each according to how important it is to your survival. Fill your rankings in under STEPs A and I. Once you have Individually filled in your rankings, arrange a CHAT among all the team members where you will come up with a team ranking listing your team decisions under STEPs D and M. Once you have reached this point, check off on the Meeting roster (in the TEAM BUILDING ACTIVITY room) that your team has completed the team rankings. Once all the teams have checked that they are finished, you will be given the EXPERT RANKINGS which you will fill in under STEPs B, E, J and N. STEPS C, F, K and O will be calculated automatically for you. You will probably want to discuss these results with your team. After this you can check your INDVIDUAL SCORE (STEP Q) and your TEAM SCORE STEP R). You will need to fetch file Scores1-95.xls or Scores1-Mac.csv. Hopefully your TEAM SCORE is lower than your INDIVIDUAL SCORE. If you want to calculate the TEAM EFFECTIVENESS SCORES, arrange to meet with the team and calculate STEPs S, T, U, V, W and X. You will need to fetch file Scores2-95.xls or Scores2-Mac.csv. Figure 30 Directions for a Narg Island Game with Distributed Teams Assignment 3 is another individual assignment designed to have the students use the TW tools they most likely had not used yet. Table 24 gives directions for using a number of tools. 144 Chapter 7 VisArt Table 24 Training Assignment 3 Tool Task Web browser Load a web page into TW. Try out the web browser. URL Use the Help function and find the address of TW’s own home pages. Link to them via the URL reference tool. Post-it Use a PostIt note to tell your team members what you think of TW so far. Vote tool If you have not already done so, register you vote in the Classroom. Meeting roster Plan an exciting debate or meeting between some Norwegian politicians. You decide the theme and goals of the debate/ meeting. Record these in the meeting roster tool. Database Create a simple database to record your own CDs. Have at least 3 fields (e.g., name, title, nationality). Legg in 5 album i databasen. Experiment with the different fields. Image whiteboard Draw your house. Try to import a picture and see you the two can be combined. Page Page a team member and tell that you are finished with assignment 3. 145 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Assignment 4 was designed to provide an opportunity for a team to practice collaborating. Figure 31 shows the assignment. Assignment 4 Collaborating to produce a presentation In assignment 4 you will collaborate to choose a theme which you will present for the other groups. You will search the Internet for web pages on this theme and decide as a group which should be part of your presentation. Method for the Assignment: • Brainstorm about a theme you will make a presentation of. Use about 30 mintues. • Individually find web pages that can be used to present your theme. This part can be done for tomorrow. • Together as a team, come to a common understanding of your findings and determine as a group what you should keep/cast. Use about 1.2 hours. è Give a short presentation of your theme by linking to the web pages you have agreed on. This method meets the requirements for genuine interdependence as definded by G. Salomon: 1. the need to share information 2. division of labor among the team members 3. pooling together of minds / joint activity of thinking Salomon argues that for CSCL to succeed, there must be Figure 31 Training Assignment 4 The Help Room Another important room we created in TW was the Help Room shown in figure 32. In this room, the students could find several sources of help. Two Message Boards — one for content questions and one for technical questions — were place in the room. Students could either ask or answer questions 146 Chapter 7 VisArt here. There were also several links to resources on the web. One URL points to pages on Virtual Teamwork. A pointer on the Whiteboard draws the student’s attention to the internal TW help system that is accessed though the Help Menu. This help system is in English. A second URL points to a set of web page that explain how the different tools and functions in TW work. These Help pages on the web are explained in the next section. Figure 32 VisArt’s Help Room in TeamWave The Help Pages The versions of TW that we were using until just before VisArt was deployed did not have any online help. It was decided, therefore, that we would build our own help pages on the web31 . These web pages would be in Norwegian and would explain how the different tools and functions in TW worked. The web pages also gave some tips on how to, for example, Chat, send a message (Page), etc. Figures 33-36 show some of the pages. Just before we were to deploy VisArt, a new release of TW became available and this version included online help. At first it seemed like the efforts to 31 The TeamWave Workplace help pages were developed by Kurt Rysjedal. 147 Project DoCTA: Design and use of Collaborative Telelearning Artefacts develop our own help pages were wasted, but this was not the case. First, our web pages were in Norwegian that was preferable for our Norwegianspeaking students. Second, the pages were accessible on the web and a web window could be open at the same time as a TW window, thus help was available while they tried to do something. Finally, they could access the web pages without opening TW, thus giving them a chance to introduce themselves to TW without having to be on the server. Figure 33 148 Home Page of TeamWave Workplace Help Pages (developed for VisArt) Chapter 7 VisArt Figure 34 Help page for the Address Book tool 149 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 35 150 Help page for the File Holder tool Chapter 7 VisArt Figure 36 Help page for Navigating between rooms Other screen shots can be seen in Appendix W and the entire help system can be explored on the web at http://www.ifi.uib.no/docta/VisArt/help. 7.6 Deployment of VisArt The VisArt activity was deployed for 1 month beginning on February 25th and ending on March 26th. Following 5 days in which the students were to download TW, test their accounts and team email addresses, 1 week of training began. This was followed by three weeks of design activity. This section addresses this deployment and covers the following topics: 151 Project DoCTA: Design and use of Collaborative Telelearning Artefacts participants, some reflections on each of the phases of VisArt, assistance and help, and administration. Participants The 32 students participating in the VisArt activity had different backgrounds, ranged in age from 23 to 68 years, and many had family responsibilities and full time work. The University of Bergen students participated in VisArt through the pedagogical information science graduate course32 “Contemporary Approaches to Pedagogical Information Technology” given in the Spring of 1999. There were 11 students and their backgrounds were a blend of pedagogical information science graduate students with a teacher’s background, and information science graduate students with a social science background. Each student turned in an individual report as described above. The five HiNT students who participated in VisArt were students in an undergraduate course entitled Computer Supported Learning. Four of them chose to participate so they could use the VisArt experience as a case in their final exam paper. This final exam paper replaced the individual report that the UiB students had to produce. Two of them wrote a paper about evaluating TW for pedagogical use. The other two wrote a paper about differences in computer-supported collaborative work (CSCW) and face-to-face collaborative work. The last student participated in VisArt in order to gain experience in using collaborative technologies, but wrote a final exam paper on a different (unrelated) topic. The 16 students at HSH Teacher’s College, were senior undergraduate students training to be teachers who were taking a distance learning course33 on pedagogical information science and were also learning about collaborative learning. As they were taking a distance course and did not meet face-to-face, it was possible to have 2 HSH students and 1 Bergen on 7 of the teams. VisArt was part of the topic Computer Supported Collaborative Learning (CSCL) and they had to write about their participation in their electronic workbook34 (elektronisk arbeidsbok). 32 33 34 http://www.ifi.uib.no/staff/barbara/courses/pivar99.html The course was offered by NVU, the Norwegian Network University (Nettverksuniversitetet) – see http://www.nvu.no/katalog/1999-2000/hsh-alu.html#23 For example see, http://home.sol.no/~aaroenes/Dagbok/CSCL/TeamWave/ TeamWave_4.htm 152 Chapter 7 VisArt The HiNT instructors35 felt that the design decision to establish teams of students who were all in geographically different locations was quite important and worked well. Preparation for VisArt The preparation for VisArt went smoothly for all but one of the Bergen students. This student had a MacIntosh and some effort was required to get his client connected to the UiB server. Due to system constraints at HiNT, the two instructors had to help with installation of TW. At HiNT the client software was installed on the private computer account of each student in order to allow variations (e.g. inserting a portrait/picture) in the configuration files locally. This was also necessary since the workstations (PC) were write-protected. Preparation for the HSH students was somewhat more of a problem as they were only informed of the VisArt scenario when they got the email (only a day or so in advance) from the head instructor that the scenario was starting. The students at both Bergen and HiNT had been told about the upcoming scenario (that there was going to be one, not what it was) in advance. This lack of notice caused a few problems for some of the teams in the Training phase. 7.7 Training HELGE UNDERHAUG The training-phase lasted for ten days, and the main goal for this activity was for the students to get to know TW, to get to know the other members of their team, and to also give them some ideas on how to work and collaborate in TW. The first problem to occur36 was that not all the students were given information about the training phase, and thought that the scenario was to start ten days later. Due to this some students had their holidays at this time (it was a winter holiday week for Norwegian schools) and this of course was a problem for the rest of the group. Second, a few students also had minor difficulties getting attached to the server, and this also delayed the start of the training for some of the groups. 35 36 Arnstein Edismø and Glenn Munkvold Revealed through some of the questions on training in the pre-questionnaire. 153 Project DoCTA: Design and use of Collaborative Telelearning Artefacts An evaluation of the training session is being carried out (see section 7.17) and will be reported on in a Master’s thesis (Underhaug, in preparation). From a design perspective, the time and effort that went into preparing the training for VisArt is immense. Although, two of the four tasks used in the training phase had been developed and used earlier, they had to be adapted and prepared for this scenario. The Design Actvitiy The VisArt design activity seems to have been successful. The students were able to collaborate to create a learning room in TW. There were 11 teams and the topics they chose to design learning rooms for included: endanger species, gothic art, publishing on the internet, triangles, the big bang, travelling in Denmark, renewable energy sources, between the world wars, polar bears, and astronomy. Figures 37 and 38 give examples of the TW working room for Team 7 and their designed learning room on polar bears 7, respectively. Figure 37 154 Team 7’s working room Chapter 7 VisArt In Team 7’s working room shown in figure 37, it can be seen that the team was very organised. The VisArt Task (VisArt Oppgave) has a central place in the room with a doorway to their designed learning room The Polar Bear – King of the Arctic (Isbjørn – Kongen av Arktis). There are a number of URL’s pointing to Internet sources of information on polar bears that were used to design the activities in the learning room (see figure 37). Figure 38 Team 7’s room for learning about polar bears In the learning room for polar bear’s (isbjørn) shown in figure 38, the targeted students are told that they will learn about polar bears. Before beginning the 4 assignments (oppgave 1 – oppgave 4), they can look at some pictures on the web (URL Bilder) of polar bears in their natural habitat. Also they can take a polar bear test (isbjørnn-testen) to check how much they already know about polar bears. There are 4 assignments each with 2 or 3 questions that they will find the answers to. For example, in assignment 2 (oppgave 1) they are to find out “Why don’t polar bears freeze?” and they are asked to “find their secret to keeping warm”. Assignment 3 (oppgave 3) is about enemies of the polar bear and the students are asked to find out how many polar bears there are on the Norwegian/Russian Island of Svalbard. 155 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Team 10 chose to design a learning room on the topic of Norwegian authors in the period between the two world wars (Norske forfatterer i mellomkrigstida). Figures 39-41 illustrate their working process. In figure 39 we see their team room where they have organised the room according to tasks they have been doing. For example along the left side of the room we see 4 sets of doorways to (from top to bottom) individual rooms37 , training assignment rooms, working rooms where they worked on the design task, and finally doors to the learning rooms. Figure 40 shows their working room, which they refer to as their “warehouse” (lager). In this room we can see that there has been a lot of planning going on. There are URL links to relevant web pages on Norwegian authors (e.g., in the box named I bruk we have Sigrid Undset), a number of sound files (lydfiler in the circle), File Viewers with illustrations (Illustrasjonar in the box at the bottom and right of the room), text files (Textar box to the right of the circle), and so on. It appears that this room has been used to store possible information sources to be used in their learning room. The resultant learning room is presented in figure 41. Figure 39 37 Team 10’s group room The names of the team members have been “whited-out” to preserve anonymity. 156 Chapter 7 VisArt Figure 40 Team10’s warehouse Figure 41 Designed learning room for Team10 157 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Reflections on the Design There were several weaknesses with the design that were revealed through formal and informal discussions with the students. One issue was that the final designed room was not evaluated and that there were so few guidelines as to what should be designed. This decision was made somewhat consciously in that we were really more interested in the collaboration process and not so much in the final product. It was assumed that they, as a team, would decide what to design and this would be documented as pedagogical decisions. In retrospect, it would have been possible to give more specific guidelines. The second issue is that the designed room was just that designed — it was not deployed with a group of students. Several of the VisArt students expressed disappointment in that they were not going to get to try their design out on the students for whom it was designed. This was actually an unanticipated point. Due to the type of courses VisArt was embedded in and the time constraints, it was not, nor would it have been possible, to do so. This is an excellent point, however, when looking at the authenticity of the design problem. It would have been wonderful to test the designed rooms on real students, but it would be a challenge and virtually impossible, given the courses we were teaching. Another weakness of the VisArt design as a whole is that we did not formally plan for debriefing of the scenario. In Bergen we actually presented VisArt as part of the DoCTA project after the end of the scenario in order to give the students insight into the types of research studies we were doing on VisArt. At this meeting we discussed the reactions to TW, team collaboration and to the VisArt scenario. At HiNT there was no debriefing although the instructors feel that there should have been. They feel that it would have helped the students understand the experience better. Assistance and Help From the formal assistant’s perspective, assistance during the Design Activity proceeded smoothly. The assistants did not feel overburdened with email questions. This probably was due to the fact that informal, unplanned assistance was available. During the design phase of VisArt, we planned for 3 assistants to collaborate to provide assistance/help to the students. What we did not anticipate was that researchers carrying out observations in TW, thus “visible” to all on the students logged in at 158 Chapter 7 VisArt that time, would be asked for assistance in TW – either though the Page or a Chat function. During the preparation and training phases we tried to keep track of this activity by using a Message board and Database in our own researcher’s private room. This quickly became impossible, as there were just too many requests. In addition, the instructors at the three institutions were often asked for help when they were logged onto the system. Details of these situations are being covered in a Master’s thesis looking at how the instructors and facilitators organise their work (Wake, in preparation) and summarised in section 7.14. Even though the majority of the Bergen students were physically situated in Bergen, little assistance was provided face-to-face. The majority of assistance was provided in TW. All the HSH students were geographically dispersed, so all assistance was provided either in TW or through email. At HiNT, some of the students participating were situated at home while others were in the computer lab at the college (HiNT) during the whole simulation. Generally, assistance by the HiNT instructors was given in the VisArt scenario in the following manner: • Face to face • primarily for solving technical issues (e.g. installing TW) • occasionally when we were in the computer-lab at the same time as the students did their work • Email or Chat in TW • to handle ambiguous elements in the tasks (“What is it we are actually supposed to do”, “Is our choice of topic OK, we have not got any feedback from Barbara” , “When is our solution actually good enough?”) • to handle administrative issues, for example, some of the students were planning surveys / questionnaires for the local course In his usability study, see section 7.11, Rysjedal is concerned with assistance. In the post-questionnaire, the users were asked to respond to the following statement: “ If I had a problem, it was easy to find help/ information on how it could be solved.” As figure 42 shows, there were 8 students that agreed, 10 that were neutral to this statement. Three students disagreed and 3 strongly disagreed. There were none that strongly agreed. This is not a bad result, but still, it could have been better. 159 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 42 Ease of getting help/information Rysjedal also found that the message board in the Help Room was nearly not used at all. His interviews indicate that the on-line help function in TW was not used very much either. What they used most frequently were the help pages on the web. These were mainly used in the beginning, to see how they could install and start using TW. One of the reasons for this was that they could read the help without being logged into TW. In this way the help was available in an earlier moment of time then the internal help in TW. Once they had started to use TW more extensively they felt that they could get the help they needed by simply asking the other users logged into TW. Based on this we can say that it is important that the help is available as early as possible, and that it should be an external application that the users can start without having to start the application itself. This is probably especially important in a groupware system like TW, since some users might be reluctant to log into an environment with other users present if they don’t know anything about how the software is working. Actually, it is possible to start the help in TW without having to start TW itself, but then you have to find the help-file in the folder where TW is installed. The problem is that most users will not look in this folder before they get started. If the help had been available from the TW folder on the startmenu, it would probably have been more frequently used. 160 Chapter 7 VisArt 7.8 Administration of VisArt RUNE BAGGETUN, ØYVIND MEISTAD, TROND PEDERSEN This section discusses the administration of, and our experiences with, the TeamWave Workplace (TW) server. Downloading, installation. An educational server license for 50 users was purchased in the fall of 1998, and TeamWave Workplace (TW) version 3.1 were installed on Grevling, one of the Institute of Information Science’s Unix servers. Grevling is an Axil 320 Sparc Server (NIS+ client). Students participating in the fall graduate course “Contemporary Perspectives on Research in Pedagogical Information Technologies” were given the opportunity to use this groupware application during this course. This provided us with an opportunity to test TW. By the time the VisArt scenario was to start in the Spring of 1999, TeamWave Software Ltd. had released version 4.1 of their product. TeamWave Workplace version 4.1 was downloaded as an compressed (gzip, http://www.gnu.org ) archive (tar, http://www.gnu.org ). After decompression and extraction of the archived file structure, the server was ready to be started. Administration can be carried out through a client provided with the server. Administration. The administrator interface is well arranged and it is easy to configure the server from this module (see figure 43). Common tasks include adding/editing user accounts (see figure 44) or groups (see figure 45), managing rooms on the server and managing the server repository. The repository is where everything is stored. Whenever something is added to a room, a reference to it is made here. Versions of rooms and tools are also stored here. The administrator can selectively purge rooms and their contents depending upon need. Other screen shots of the Administration interface can be found in Appendix X. Inquires from the participants in the VisArt scenario dealt with various subjects. The most common inquiry was about forgotten passwords, which was quite easy to solve. Lost data (even complete rooms) was a more severe problem the administrator had to deal with. A combination of the version control in every room and the server’s repository module, however, made a complete recovery of these data (and rooms) possible. 161 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 162 Figure 43 Administrator Interface for server information and sending messages Figure 44 Administrator interface for creation of accounts Chapter 7 VisArt Figure 45 Administrator interface for creation of groups Supported platforms. TeamWave Workplace supports various operation systems, and there is no need to run the client application from the same platform as the server. In VisArt, most of the students were using Windows 95/98/NT, while the server was running on Solaris. The following platform are currently supported: Windows 95/98/NT, MacIntosh (Fat binary), Linux (version 1.2.13; ELF), Solaris (aka SunOS 5.x), SunOS (4.1.4 aka Solaris 1.x), SGI (IRIX 6.2), AIX (version 4.1). Application size and user installation. Installed and ready to go the Windows 95/98/NT client version is about 3 megabytes in size. I was the same for the MacIntosh version that was used by one of the students. The server application was originally only 1,5 megabytes, but after fifty users had carried out more than 38000 transactions during the VisArt scenario, it grew to more than 110 megabytes. Approximately 40 megabytes of this are files (various documents, spreadsheets, etc.) that the users have uploaded (via the fileholder tool) to the various rooms. Server stability. Some days into the VisArt scenario we experienced a reduction in the server’s performance, and one day the server did not respond to our transaction queries. After confronting TeamWave Soft163 Project DoCTA: Design and use of Collaborative Telelearning Artefacts ware Ltd. with our problem, we still could not figure out what caused this to happen. Without going into detail, it was probably an overrun buffer of some sort. To reduce the poor performance and breakdown of the server, routines that included restarting the server 3 times a day (at 0900, 1500 and 2000) were initiated38 and carried out for about 1 week. Restarting the server this way greatly improved the server’s performance. Eventually ,after unsuccessfully trying to find out from TeamWave why the server was slowing down, the server was upgraded to version 4.2. After this there was no longer a need to restart the server unless there was an explicit need. In summary, our experiences in administering the TW server are positive and we intend to continue to use the product in courses at the Institute. 7.9 Evaluation of VisArt The evaluation of the VisArt scenario is being carried out on several levels and from several perspectives. In addition to the student’s own theoretical reflections (described in section 7.10), VisArt is being evaluated as part of eight Master’s theses. These theses include a usability study of TW, looking at the efficiency of TW from a qualitative perspective using the data logs generated by TW, performing a formative evaluation of how to support collaborative design activities, seeing how TW supports coordination, how to design training and assistance in a collaborative telelearning setting, and several activity theory studies of how students, instructors and facilitators organise their work. In addition, a study on whether there are any personality factors involved in the use of tools in telelearning situations has been carried out by one of the researchers. Numerous data and data collection techniques have been used including questionnaires, semi-structured interviews, log files produced automatically by TW, email send among team members, chat files saved by the teams, documents and artefacts produced by the teams, participant observations, video taping of activities, self-evaluations and the participants own interpretation of the experience. A number of these studies are summarised in the following sections. 38 The researchers volunteered to participate in this restarting process. Trond Pedersen provided directions in how to do so. 164 Chapter 7 VisArt 7.10 The learners’ experiences A summary of the learners’ experiences as described in their individual reports is given in the following. In particular, attention is paid to how their group met Salomon’s criteria for genuine interdependence and how TW supported Gutwin’s ideas on awareness. In general, the students were very satisfied with TW. As one student writes “An important side with TeamWave is that one can work both asynchronously and synchronously. … For example one can use the shared whiteboard synchronously when the users are online at the same time and write on it together, but it is also possible to use the whiteboard asynchronously when the different users log on at different times and work individually on tasks on the whiteboard. …That it supports both forms of work makes the program package flexible and accessible at all times.” Several students wrote that the successful use of TW was not just tied to the ease of use, rather, that it is used in an activity that meets Salomon’s requirements. As one student succinctly put it “I think that a requirement for successful use of it [TW] is that the participants are motivated and have mindful engagement and that the tool [TW] is used for something meaningful.” Sharing of information The majority of the groups had a heterogeneous makeup with the group members having different backgrounds. As one group said, this meant that they had different preconceptions and different experiences with collaboration. They said, that “according to Salomon it is exactly these differences that makes collaboration work…to use each others competence and pull something useful of these competencies through collaboration.” When it came to TW supporting the sharing of information, it was possible to do so both synchronously and asynchronously. One particular group worked 165 Project DoCTA: Design and use of Collaborative Telelearning Artefacts mostly synchronously (with respect to sharing and they meant this was timeconsuming. Often they would meet at a pre-arranged time and have little to say to one another and this was a disadvantage as it required that they bind themselves to meeting at a particular time. It would have been “better if we could have worked a little bit more asynchronously, and instead given each other feedback, for example by using a PostIt, when we were logged on to work with one of our own tasks. TW provides several tools to support this asynchronous communication…but our use of this functioned badly”. Division of labor With regard to division of labour, one of the students said the following: “TW was a good support for division of labour in those cases where it was not necessary to be in our common work area at the same time. If one person started on a task, then left the work area, the next person could come in and continue working on the task since there was a common work area.” Joint thinking One of the teams said that the requirement for joint thinking in explicit terms that can be examined, changed, and elaborated upon by peers was supported in TW through the Chat, Postit and Brainstorm tools. For them “This particular need was satisfied through the many online joint discussions between team members. The fact that team members had the opportunity to express their ideas, conceptions, meanings, etc., in clear and observable terms (by means of Postit and Chat artefacts), enables each one of us to analyze, expand on, or change these as they were being developed.” Awareness The students were in general agreement that TW supported many of the aspects of awareness as identified by Gutwin et al. (1995). As one student points out, it is TW alone that supported social awareness, but the combination with some of the training activities that were designed for 166 Chapter 7 VisArt learning about TW and about each other that supported social awareness. He says “The social awareness tied to expected roles is first and foremost tied to personal relationships. I feel that the training activities we carried out in TeamWave were important for this. Through informal chats we were able to each get a feeling of what we could expect of each other and what type of interaction and tone we would have between us. … It was never explicitly said that “I am … and you are …”, but over time we picked up aspects of each others personaility and found our own roles.” According to the students TW supported workspace awareness by several techniques. The Information Areas provided awareness about the other users presently connected to the server, and indicated who was in the same room. The coloured cursor on the white board made it easy to follow what each of the other users who are in the same room are doing, and if another user is typing on the Whiteboard, PostIt, or Brainstormer, you see what they are typing immediately. One of the other students meant that TW’s Chat tool “provides for some of the social aspects of collaboration. We used it for informal discussions around different themes, both scientific and social. Through the common mileu that a room in TeamWave enables, both task and conceptual awareness are fulfilled.” Another student said that “The opportunities provided by TeamWave Workplace in dealing with the issues in social awareness were generally poor. Support for interactivity in information exchange and negotiation of roles, which is fundamental given the interpersonal nature of social awareness, was limited to artefacts such as Chat and Postit…In the case of concept and task awareness, the requirement within CSCL environments is for explicit artefacts that can be used as scaffolds to help students with task organisation, providde them with concentration aids and step-by-step guidelines for learning tasks. TeamWave’s support for both concept and task awareness is provided by tools such as ToDoList and Concept Map (for task organisation, and Message Board (For structured message capabilities).” 167 Project DoCTA: Design and use of Collaborative Telelearning Artefacts It is early in the evaluation of VisArt to make many conclusions. From a sociocultural research perspective, however, the student’s own reflections are a very important part of evaluation and as illustrated in this section they demonstrated an ability to reflect theoretically on practice. The research reports they submitted in the course contained comments and reflections that were both thoughtful and insightful and will lead to improvements in future versions of the scenario. 7.11 A Usability evaluation of TeamWave Workplace KURT RYSJEDAL In the usability literature there are various definitions of usability. In this study, the definition of usability in the Draft for International Standard ISO/DIS 9241-11 (1996) has been adopted: “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.” Here the effectiveness of a system relates to the work objectives (or goals). The efficiency relates to effectiveness in relation to the resources needed to perform the tasks, and satisfaction, according to ISO 9241, concerns acceptability and comfort (Lif, 1998). This definition was used to design instruments to measure the usability of TeamWave Workplace (TW) in supporting the VisArt collaborative telelearning activity. The purpose of the evaluation was to identify potential problems with the system’s usability. Based on the problems found, some implications for redesign of the system will be proposed. This section reports on this usability study (Rysjedal, in preparation). Research design A usability evaluation of TW should, according to the definition adopted, include an evaluation of the effectiveness of TW, the efficiency of TW, and the users’ satisfaction with TW. Such an evaluation has been undertaken while TW was in use in VisArt, an actual collaborative telelearning scenario. In VisArt, teams composed of students from three 168 Chapter 7 VisArt different Norwegian educational institutions collaborated to design an instructional room in TW. To guide this evaluation three different questions were asked: • Can students working in collaboration reach their team goals with TeamWave Workplace? • To what extent do the tools in TeamWave Workplace enable students to meet their tasks? • Are the students satisfied with TeamWave Workplace? In order to answer the first question on whether or not students working in collaboration through TW can reach their team goals, the goals need to be identified. In this case the team’s main goal was the task objective: to design an instructional room in TW and to document the team’s pedagogical decisions. To see if they managed to reach their main goal entails evaluating the completed instructional room and the document containing their pedagogical decisions. To find out to what extent the tools in TW enabled students to meet their tasks, five of the students were interviewed. The interviews were carried out after the students had completed their assignment, and an interview guide (see Appendix Y) was developed to support the interviews. In the interviews the students were asked about how well they TW supported the following functions (from Paquette et al., 1995): communication, administration, production, presentation, and navigation. These are functions that have to be well supported to make it possible for the students to meet their tasks in an efficient way. Figure 46 groups the TW tools according to their functionality. 169 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 46 TeamWave Workplace tools grouped according to functionality Questionnaires were used to help determine if the students were satisfied with TW. In the preparation phase, the students were given a profile questionnaire (see Appendix P). The profile questionnaire collected background data on the students (e.g., computer experience, collaboration experience, educational background, etc.). A pre-questionnaire (see Appendix R) was used to find out about their first impressions of TW. This questionnaire was given to the students just after the training phase. Three weeks later when they had finished their assignment, a postquestionnaire (see Appendix S) was given. The pre- and postquestionnaires together will be used to identify the students’ degree of satisfaction and see if their opinions changed over time. All questionnaires were web-based and could be accessed from within TW by clicking on a URL holder that opened the questionnaire in their web-browser. This should have made it easy for the students to submit the questionnaires. They were informed by email when the questionnaire was accessible and their instructors also told the students that it was mandatory to answer the questionnaires. Although most of them did answer 170 Chapter 7 VisArt the questionnaires, some of the students did not submit their answers when they were supposed to (e.g., one or two students submitted the pre-questionnaire only days before they submitted the post-questionnaire — there should have been three weeks between). This made it difficult to see if their opinions changed over time. Analysis and Preliminary conclusions All the data have now been collected, but the analysis has not yet been finished. Nevertheless, there are some points that can be drawn out of the analysis so far. This section summarises some of the analyses and reports some preliminary findings. Effectiveness In order to measure the effectiveness of TW in supporting a team’s goals in the VisArt scenario, the designed TW room and decision file need to be examined. The completed instructional rooms and documents with the teams’ pedagogical decisions are preserved on the TW server. In addition, screenshots of the instructional rooms, both during construction and after completion have been taken. With regard to effectiveness, it can most likely be concluded that the students working in collaboration managed to reach their goal with TW. All the teams managed to make an instructional room in TW. They also all made a file documenting their pedagogical decisions. We can, therefore, say that it appears that in this context of use the students managed to effectively achieve their goals. Figure 47 gives a screenshot of a completed room and figure 48 shows an extraction from a pedagogical decision file. 171 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 47 An instructional room designed by Team 6 Figure 47 shows that Team 6 created a project on Publishing on the Internet (Publisering på Internett). They have created a jigsaw collaborative task where 76 groups of 3 students solve a problem. A ToDoList (Arbeidsoppgaver) guides them through the activity. Each student in a group is assigned a number from 1 to 3. All students with the same number are to go to their room (e.g., Elev 1 rom for student 1, Elev 2 rom for student 2 and Elev 3 rom for student 3) where they first solve an individual task. Then all number 1 students (likewise all student 2’s and 3’s) meet to discuss their solutions. Then they return to their own group and present what they have done. Each group produces a final report on the web. These reports can be found by following the URL’s Gruppe 1 – Gruppe 7. 172 Chapter 7 VisArt PePedagogical decisions: * OOur target group is students at secondary school. * We presuppose that the students have already learned to made web-pages (HTML-programming or use of editiors). * The teaching material is not based on a single subject, but is an interdisciplinary task where the students are supposed to gain some knowledge that later can be used within several subjects. * References to the curriculum plan: In the curriculum plan for information-technology in VKI and VKII, presentation of information is an important part of the subject’s definition. Figure 48 Extracts from Team 6’s pedagogical decisions file These rooms and documents need to be evaluated as to whether they meet some criteria39 for an instructional room in TW. As we need to determine what criteria should have been meet, this evaluation has not been carried out yet. It will, however, be reported on in (Rysjedal, in preparation). Efficiency The interviews to determine efficiency were carried out just after the students had finished their assignment. Two of these interviews were face to face interviews, and three of them were telephone interviews. All the interviews were recorded and have been transcribed. Analysis, however, has just begun. 39 There was no specific criteria given in the VisArt assignment, rather the student team should negotiate their own criteria and document it as pedagogical decisions. This lack of criteria is one weakness of the assignment design and should be corrected in future versions. 173 Project DoCTA: Design and use of Collaborative Telelearning Artefacts One advantage of having the interviews face to face was that it was easier to see how the students reacted to the different questions—for example one could see if they had misinterpreted some of the questions. On the other hand, when doing a telephone interview it was easier to keep the interview focused according to the interview guide. One problem in all the interviews was that the respondents were very positive. Observation of the students while they were working on the assignment had revealed that there were some problems with some of the functionality of TW. When they were asked about this in the interview, it was expected that the respondent would mention these problems, but they seldom did unless they were asked specifically about them. Pacquette et al.(1995) argue that in order for the students to achieve their tasks in an efficient way in a telelearning environment, the tools should support the following functions: communication, administration, production, presentation, and navigation. In this section only preliminary results with regard to communication and navigation functionality are discussed. Communication. In TW there are tools, see figure 46, for both synchronous (chat and page) and asynchronous (postit, message board, vote tool and file holder) communication. The tools for communication were the tools that were most frequently used during the collaboration, and also among the tools that the users rated as most important. Still, the communication tools had some efficiency problems. In TW tools can be placed in different rooms, thus, the users had to visit several rooms to see if there were any messages (using the asynchronous communication tools) for them. This could often take some time, and it could also lead to communication failure if the teams had not established some kind of agreement for how and where they would leave their messages. Figure 49 shows some extracts about what the users said about the asynchronous communication in the interviews. 174 Chapter 7 VisArt • It requires that you agree in advance: What will we use to leave messages. So that we do not leave a message one time here, and another time there. That we have a standard for where we should put the message. (…) Maybe we were a little bit weak on that. We didn’t really have a joint bulletin board where we left messages. It was a little bit like we… Left something here and something there. So I guess it could have been done better. • I think there should have been a way to leave messages for people that were offline. That you could reach them, e.g. with some sort of chat-tool when they weren’t there, like ICQ. There should be a function that makes it possible to leave a message for people that are not online and that comes up as soon as they log on. That would have been nice, because then you could reach a person. But for the group it was OK to write on a Postit or in the Calendar, or things like that. So there were tools for that too, you can say. But I also missed the possibility to send mail directly. • If they should check their mail or something, they had to check their mail in addition to TeamWave. So it was obvious that that wasn’t very popular. They wanted all the communication to be gathered in TeamWave, and not so much outside. We disagreed there. Figure 49 Extracts about asynchronous communication in the interviews The main tool for synchronous communication in TW is the chat-box located in each room. In figure 50 we can see some examples of what the users said in the interviews about the synchronous communication. As we can see from these examples many users found the chat to be too slow. The reason for this was that they wanted the tool to transmit character by character, while this tool only transmitted sentence by sentence. Actually, many users started using postit (which is originally a tool for asynchronous communication) instead of chat. Postit transmitted character by character, and even though this tool had some limitations, many users preferred it on several occasions. The reason why many 175 Project DoCTA: Design and use of Collaborative Telelearning Artefacts users preferred the text be transmitted character by character was that they could see the text while it was written. In this way they were aware of what the other users were doing. In most cases they could also see what other users were going to write, so they could start to formulate an answer before the other user had finished writing. • The disadvantage with chat is that you have to sit there and wait and wait, wondering if someone is writing something, (…) we started to use Postit instead, and then we could talk simultaneously as much as we wanted. • When you are using a Postit you could follow what the other person was writing. You can’t do that in chat. You can see while he is writing. You get time to think of what is coming. In this way it promotes understanding, and is quicker. • We tried for a while to use Postit, but I think it was a bit annoying that both should write on the same place. Even if you could see what the other one was writing, you might not know for sure when the person was finished. You felt that maybe you broke in… And what is negative with Chat is that you can’t see. If they write very long sentences in Chat, it takes a very long time to get an answer on your question, so then you might just go on. But actually I think it was better, nevertheless. • In the Chat-tool, for example, it could have been made so that several could communicate at the same time. That you can see what the others are writing, but that it still functions like a chat. Using some of what you have in the Postit-tool, in a way. There you can see what the others are writing. Transfer some of that functionality to Chat. Figure 50 Extracts about synchronous communication in the usability interviews Navigation. Navigation refers to navigation around the rooms in TW. Most users found that the doorway tool in TW made this navigation very efficient. By placing doorways between the rooms they used most often, 176 Chapter 7 VisArt they could navigate between these rooms by just clicking on the doorways. On the other hand, if they wanted to go to a room that didn’t have a doorway, they had to locate the room in a list of all the rooms on the server. This could be more time consuming. Especially as the number of rooms grew large. As one user phrased it40 : “It was easy to find them, but after a while the number of rooms grew very large. So then you used some time to scroll up and down to find your own rooms. It should have been so that… I actually don’t know how TeamWave organises the list of rooms, but it seems very casual.” Satisfaction In the questionnaires many of the questions are statements where the respondents are asked to indicate on a five-point scale to which extent they agree or disagree, see question 3 in figure 51. There are also many questions where the respondents are asked to locate their position between two contrary adjectives on different dimensions. A descriptive analysis has been carried out on these questions, showing an average distribution on the different dimensions. On most of these questions the students were asked to fill in with comments, see question 2 in figure 51. These comments are grouped together in different categories, which makes it easier to determine any trends in the comments. When it comes to satisfaction with TW, the general conclusion that can be made is that most of the students were very satisfied, but there is still some room for improvement. In both the pre- and post-questionnaires the users were asked about their perception of TW regarding usefulness, simplicity and attractiveness. On each of these dimensions they were asked to locate themselves on a five-point scale between a set of adjectives where 1 is the most negative and 5 is the most positive. As we can see from figure 52, the results were fairly positive. Usefulness and simplicity attained a mean of 4 or above on both the pre- and post-questionnaire. Attractiveness was a little bit lower with a mean on 3.61 on the prequestionnaire, and 3.56 on the post-questionnaire. 40 All translations from Norwegian to English have been done by Kurt. 177 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 178 Figure 51 Some example questions from the pre-questionnaire Figure 52 Perception of TW regarding usefulness, simplicity and attractiveness. Chapter 7 VisArt Some of the other aspects of TW we wanted to know something about were aspects like TW’s learnability, stability, and if they felt that it was easy to recover from mistakes. For each aspect the users were asked to which degree they agreed or disagreed to different statements. Some of the results we can draw from the answers to these statements are that TW is quite easy to learn and to use. Still, as we can see from figure 53, there were some functions that some users found difficult to understand how to use. Figure 53 Difficulty of understanding functions in TW Figure 54 Stability of TW server 179 Project DoCTA: Design and use of Collaborative Telelearning Artefacts There was only one statement regarding stability: ”I think the stability in TeamWave was good.” This was a statement that most users disagreed on. We can see from figure 54 that only 5 respondents agreed, while 7 strongly disagreed and 6 disagreed. With this number of users disagreeing with the statement, we can conclude that the stability in TeamWave Workplace was not good enough. There were several statements related to how easy the users felt that they could recover from mistakes. One of these statements was the following: “I missed functions like undo and redo.” Even though 46% of the respondents in a previous statement had agreed that it was easy to correct mistakes, there was only one that disagreed to this statement (see figure 55). This indicates that even though many respondents found it easy to correct mistakes, they still missed functions like undo and redo. Figure 55 Missing functions in TW In the post questionnaire the users were also asked to mention 3 negative and 3 positive aspects about TW. The answers were summarised into one list of negative aspects and one list of positive aspects. We then ended up with a list of 54 negative comments of TW, and another list of 56 positive comments. Many of the comments in the list were similar, or referred to the same type of problem. The comments in the list were therefore grouped into different categories according to which type of problem they referred to. Figure 56 shows which categories the comments were grouped into, and how many comments there were in each category. 180 Chapter 7 VisArt Figure 56 Categories of negative and positive comments Also in this section it was revealed that stability was a problem. In the beginning of the training phase the server went down a couple of times, but during the rest of the period the server was restarted regularly to prevent it from going down while it was in use. Still there are as many as 10 comments that the stability wasn’t good enough. Related to the comments on stability we have 8 comments on slowness in TW. Several users pointed out that it was slow to edit, or move things around in the rooms. There were also several complaints that the tools in TW weren’t good enough or didn’t work. Many of these comments just state that the tools weren’t good enough without telling why. Other comments are more specific and complain that some tools have poor support for other file formats than .txt and gif. There were also some users who commented that they missed the opportunity to use sound and video for communication. The user interface in TW was an aspect that received both positive and negative comments from the users. Most of the positive comments were that it was easy to use and learn. Among the negative comments it is mentioned that that the display has a tendency to get messy. They also 181 Project DoCTA: Design and use of Collaborative Telelearning Artefacts found the editing and saving of work to be a bit cumbersome. There were also some students who felt that the working surface in TW was too small. This is probably due to the fact that the working surface is dependent on the size of the screen. The smaller screen you have, the smaller the portion of the room that is visible in the working area. Conclusions In this study we have undertaken a usability evaluation of TW. We have adopted the definition of usability in the Draft for International Standard ISO/DIS 9241-11 (1996). This definition suggests that a usability evaluation of TW should include an evaluation of the effectiveness, the efficiency and the user’s satisfaction. The analysis of the data in this study has not been finished yet, but the data that have been analysed so far indicate that in the context of use for this study TW was an effective tool. All the teams managed to fulfil their assignment. To find out how efficient TW was we investigated how well the users felt they could perform the following functions: communication, administration, production, presentation, and navigation. Only communication and navigation have been analysed so far. The users were able to perform both these functions very well. Still, it was revealed that there was some room for improvement. In general the users were also very satisfied with TW. They felt that TW was easy to use and used little time to learn how to use it. Despite this general satisfaction there were some negatives. Roughly speaking these were minor issues, but if they had been dealt with, the users’ satisfaction would probably be even better. 182 Chapter 7 VisArt 7.12 The Role of Classical Jungian Personality Factors in Computer Supported Collaborative Learning Environments KONRAD MORGAN & MADDY MORGAN The work described in this section formed part of the evaluation of the VisArt scenario. Although the VisArt scenario investigated many aspects about the use of these tools in collaborative telelearning the study reported in this paper investigated if there were any personality factors involved in the use of these tools in telelearning situations. It is a summary of a full report presented in (Morgan & Morgan, 1999). Review of Previous Personality Research in the use of predicting and understanding user attitudes and behaviour The idea of looking at personality as a factor in human computer interaction is not new. Van Muylwijk (1983) was among the first to recognise that personality traits would have a major impact on both behaviour and attitudes when using technology. This idea of looking at personality and interaction styles was followed up by both Van der Veer (1985) and Singleton (1989). Van der Veer proposed a more general approach where various personality factors would have some affect on attitudes towards and use of the system. In contrast Singleton proposed intelligence as a single factor which he believed would predict the degree of success in using computer systems and in having positive attitudes towards computer systems. Later investigations into the role of personality included a series of experimental studies by Van-Hoe (1990) who attempted to look at the role of personality and preferences for menu characteristics and Weil (1990) who tried to find links to computer phobia. Neither Van-Hoe or Weil were successful in their attempts to find links between personality styles and computer use or attitudes. This may have been because they were simply looking at more high level personality factors or that the tasks they selected were so far removed from day to day experience that classical personality theory was overwhelmed by system characteristics. By the early 1990’s computer systems had advanced enough to allow for computer mediated communication which permitted more naturalistic 183 Project DoCTA: Design and use of Collaborative Telelearning Artefacts communication styles. It is not therefore surprising that when Adrianson (1991) investigated the role of extroversion in computer mediated communication he found significant differences connected with the extroversion factor but that these were much weaker than in normal face to face communication. We must again recognise that in the early 1990’s that although CMC provided naturalistic communication it did not support video or audio conferencing except in rare research settings. Previous work by the author In contrast to some of the less successful studies described in our review, we have found significant differences in computer attitudes and behaviour associated with major personality factors (Morgan & MacLeod, 1992). Specifically we found possible relationships between a stated preference for command line systems and extreme scores on a high order control scale, and stated preferences for graphical interfaces and extreme scores on high order extrovert scales. These promising findings prompted us to undertake further research, leading to the current study where we investigated the computer based attitudes and behaviour of students using computer mediated collaborative learning environments. The Myers-Briggs Personality Test The Myers-Briggs Type Indicator (MBTI) is a self-report personality inventory designed to give people information about their Jungian psychological type preferences. The measure was developed by Isabel Briggs Myers and Katherine Cook Briggs in the early 1940s to try and make C. G. Jung’s theory of human personality understandable and useful in everyday life, increasing popularity in these types of studies due to its non judgmental nature. The MBTI results indicate the respondent’s likely preferences in four dimensions: • • • • Extroversion (E) OR Introversion (I) Sensing (S) OR Intuition (N) Thinking (T) OR Feeling (F) Judging (J) OR Perceiving (P) 184 Chapter 7 VisArt In general results on the Indicator use letters representing each of the preferences as indicated above. There are 16 possible ways to combine the preferences, resulting in 16 MBTI types: ISTJ, ISTP, ESTP, ESTJ, ISFJ, ISFP, ESFP, ESFJ, INFJ, INFP, ENFP, ENFJ, INTJ, INTP, ENTP, and ENTJ Though many factors combine to influence an individual’s behaviours, values, and attitudes, the four-letter type descriptions summarise underlying patterns and behaviours common to most people of that type. The Myers-Briggs personality Type Inventory (MBTI) in Education The history of the use of the Myers-Briggs Type Inventory within education is relatively long. As early as the late 1960’s Richek (1969) had proposed that the MBTI might be a suitable instrument to determine the best teachers with regard to teaching style and material presentation to students. Although the history of the MBTI within education is long it took some considerable time for it to gain widespread support. Early evaluations compared the MBTI and other personality measures such as Cattell’s 16PF in the role of predicting successful learning styles and grade point averages (Eison, 1985) and although researchers such as Lorr (1991) had problems recognising the usefulness of the MBTI on the whole by the early 1990’s there was growing support and recognition for both the validity and reliability of the MBTI in education (Murray, 1990 ; Brown et al., 1991). Since that time although there have been some studies which raised concerns that the MBTI was being taken out of context from Jungian theory (Garden, 1991) and that it might not truly reflect unconscious desires (Barbuto, 1997) it has been found to be one of the best predictors for many aspects of education and educational technology (Eison, 1985; Jackson et al., 1996). Some of these studies which involved large samples (n > than 1000) have found the MBTI to be amongst the best personality descriptors and, reassuringly for the study reported in this paper, that there was no support for the separate gender scoring commonly recommended for the MBTI (Jackson et al., 1996). This is of importance because the subjects on the VisArt personality scenario were anonymous with no record of 185 Project DoCTA: Design and use of Collaborative Telelearning Artefacts their gender, so all subjects were scored in their personality types as males. It is unlikely given the large meta-analysis performed by Jackson et al. (1996) that this has any significant implication on our findings. However, this would need to be confirmed by future replication. Method As part of the VisArt scenario within the DoCTA project 26 students were asked to take part in an evaluation of the TW. This evaluation concentrated on the use of various component parts within TW. As well as an online collaborative activity within TW the students were required to complete three questionnaires, a profile questionnaire, which recorded their background, their views on group work and their experience with technology. A pre-session questionnaire was also administered which enquired on the clarity of the task that had been explained to the students, their preconceptions about TW and the components of the Address Book, the Brainstormer, Calendar, Concept Map, Database, Doorway, File Holder, File Viewer, Image Whiteboard, Meeting Roster, Message Board, Post-It, To Do List, URL Ref, Vote, Web browser, Chat, Page and Whiteboard. After completing their tasks within TW the subjects were asked to complete a post-session questionnaire. This questionnaire attempted to record the student’s views about the clarity of the task they were given and the work tools. In addition they were asked questions about how effective they felt the teamwork was within TW and what was the nature of their interactions with the other subjects within the collaborative working environment. The findings from these questionnaires are not the topic for this paper and are dealt with elsewhere, specifically as the Masters dissertation for Kurt Rysjedal (Rysjedal, in preparation). Instead this study focused on the correlations which appeared from this data and a personality inventory (form G Self-Scorable version of the MTBI, 1987) administered to the subjects prior to the experimental session. Security and ethical controls The project described in this section conformed to and was registered with the Norwegian Data Security Agency (NSD). In accordance with NSD’s specifications every subject in this study was assigned a unique 186 Chapter 7 VisArt subject number. These same subject numbers were used to identify their behaviour online in the TW environment and their responses to the profile, pre-session and post-session questionnaires. At no time did the experimenter know the identities of the subjects. Data entry and analysis The MBTI personality surveys were hand coded according to the self assessment schemas provided by the Myers Briggs corporation. These codings resulted in a raw score in each of the major dimensions of personality recorded by the Myers Briggs personality test, namely, extroversion (E) or introversion (I) Sensing (S) or Intuition (N), Thinking (T) or Feeling (F), Judging (J) or Perceiving (P). Normally these raw scores are converted into a single set of four primary personality measures such that the respondent is provided with a four-letter combination representing their personality preferences. As we described in the review of the Myers Briggs personality test above, this process leads to 16 possible ways to combine the preferences and result therefore in the 16 MBTI types. Rather than using these simple letter codings we took the raw numerical scores in the primary eight preferences and used these to provide each respondent within our survey with a score for their preferences in extroversion, introversion, sensing, intuition, thinking, feeling, judging or perceiving. We did this because the simple letter types lose the strength or degree of the respondent’s preference on the four major personality dimensions recorded by the Myers Briggs personality test. It is also possible within the Myers Briggs personality test for an individual to be rated for example as an introvert (I) when in fact their extroversion score was nearly balanced to their introversion score. This is because a single number difference results in the larger score being taken as the primary personality dimension. By just using the raw scores within the four dimensions we therefore retain the original information about the respondents personality. After the data had been manually decoded into the primary scores of the four dimensions of the Myers Briggs personality inventory, they were then entered into a statistical package (Minitab v.10) combined with the data from the profile, pre-session and post-session questionnaires. An exploratory analysis was then performed on the data looking for significant correlations, using a Pearson correlation test, between the primary eight measures of personality provided by the MBTI. 187 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Results In the following results section we will discuss the findings from the Pearson correlations between the eight primary MBTI types and the responses from the profile, pre-session and post-session questionnaire data. Within each of the questionnaires we will summarise the findings associated with four primary personality dimensions. Summary of Findings Within our study we appear to have found support for the theoretical involvement of personality factors within behaviour and attitudes in computer mediated collaborative working environments. Our work is based upon a relatively small sample (n = 26) and therefore our findings can only be preliminary and exploratory within the area of personality and computerised behaviour and attitudes. However, we do appear to have found some strong and cohesive relationships between Jungian personality factors and human behaviour in a novel environment. When we examine our findings we can see clear patterns within each of the major personality dimensions. It does not appear unlikely that these differences would follow over into predictable generic guidelines for the creation and use of collaborative computer based learning systems. In particular the major difference between Extrovert and Introvert seems to have been extremely clear within our sample such that Extroverts were actively looking forward to the group activity before their session while Introverts reported strong negative anticipation. The importance of social interaction in such a collaborative learning environment cannot be overestimated and while it was predictable that Extroverts would be intensive Chat users and Introverts be reluctant or infrequent Chat users it was an interesting finding that Extroverts were more likely to have their own website and to have significantly more positive views towards the use of web browsers. The social interaction differences between Extroverts and Introverts was also found to be reflected in their preferred learning style such that Extroverts found the social part of the interaction between student and teacher to be a vital part of their learning experience whereas Introverts clearly preferred to work and learn alone. This difference was reflected not only in humanhuman interaction (teacher- student) but also was hinted at in terms of 188 Chapter 7 VisArt artificial agents within the Help support of TW so that Extroverts found such agent based Help systems to be very useful and Introverts found them to be unhelpful. These fundamental differences meant that TW was predominantly geared or designed towards an Extrovert based learning style and may have disadvantaged the Introvert students so that the Introverts reported having problems learning how to use TW and felt that TW was unstable. In contrast Extroverts felt that TW was easy to use and quick to learn. As a possible side effect to the match between TW and the Extroverts learning style they reported finding TW significantly more stable as a product than their comparative Introvert users. Within the Sensor/ Intuitive dimension (S/N MBTI types) we found a significant difference in the reported attitude and use of several of the TW tools. In addition to finding TW difficult to learn the Sensors also avoided using the Message Board, Post It Notes and Chat tools preferring instead to use the Address Book, BrainStorming, Calendar, Concept Map and Database. In total contrast those individuals who were high on the Intuitive dimension often used the Chat system and thought that TW was enjoyable and easy to learn. Although these Intuitive individuals used the Post It tool frequently they tended to significantly avoid the use of the favoured tools of the Sensor. Given these findings future development of computer based collaborative learning environments might find profitable research in investigating the effect of adapting the tools based on the preferences of individuals at the extremes of the S/N dimension. From the evidence we have seen in this study there appears to be a clear split between tools which support individuals high on the S scale and those high on the N scale. Only future studies of these factors will be able to isolate the specific aspect of each of these tools responsible for the differences we have found. The remaining two dimensions of Thinker/Feeler and Judger/Perceiver appear more concerned with ways in which group work is viewed and performed. However we should be cautious of the fact that it appears that individuals high on the Thinker end of the Thinker/Feeler scale and the Perceiver end of the Judger/Perceiver scale were older and significantly more experienced in group work. It might be that the differences in attitudes towards group work reported within our study were a reflection of the age and prior experience of our subject population. Given that possibility it does appear that Thinkers feel that group work involves more effort and greater responsibility than their 189 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Feeler co-workers. Likewise individuals high on the Judger scale tended to enjoy group work within TW while Perceivers, who were more experienced of group work, disliked the idea and experience of group working in TW and felt that luck was more of an element in the success of such collaborative efforts. Conclusions In this study we have investigated the possible link between MBTI types and the attitudes and behaviour of students using computer supported collaborative learning environments. We have found statistically significant differences between the major personality factors in terms of the use of the learning environment and attitudes towards the various components of the online collaborative learning system. Based on our findings we strongly support the idea that the MBTI can provide a useful tool in configuring such online learning environments to student’s personalities and preferred learning styles. Of the MBTI types investigated in our study the dimensions of Extrovert/ Introvert and Sensor/Intuitive appear to be most promising as major predicting factors in learning styles and system component preferences. These can be summarised such that Extrovert / Introvert dimension appears very useful in determining primary learning and interacting style and preference and the Sensing / Intuitive dimension appears most useful as a predictive tool for the use or avoidance of certain system components. The remaining dimensions Thinkers / Feelers and Judgers / Perceivers appear more related to attitudes toward group work and may reflect some previous experience within our subject population. Although Myers-Briggs and Jung did not include maturity of the individual within their personality types in our subject population there does appear to be a strong trend within the Thinker/Feeler and Judger/ Perceiver dimensions correlated with greater age and experience. Within our sample the older and more experienced individuals disliked group work and were more likely to report system based problems linked to group based activities. It is uncertain if these findings related to the Thinker/Feeler and Judger /Perceiver scales will be reflected in a more general subject population. As a result of this study it is to be hoped that future research will investigate methods in which MBTI type tests for Extrovert / Introvert and Sensor/ 190 Chapter 7 VisArt Intuitive can be directly linked to real time online changes within the collaborative learning environment to more closely match students preferred interaction styles and preferred learning tools. 7.13 Evaluating How Students Organise their Work ESKIL ANDREASSEN This Activity Theory-oriented evaluation has been carried out as a part of the VisArt scenario and is being written as a Master’s dissertation (Andreassen, in prepraration). More specifically, the subject of this study is the activity undertaken by the students in Team3, designing a TW room for teaching 6th graders about endangered animal species, a task41 of their own choice, within the boundaries of the VisArt design activity. The data collection was carried out between March 1st and April 11th. The students were geographically distributed over Norway and were not able to meet face-to-face. This distributed nature of the collaboration process influenced the way the students collaborated, and an important part of the study is to document how they organised their work in order to overcome this challenge. The research question is: How do the students organise their work? This tentative research question, whose scope has been narrowed down during the data gathering, will be specified further during the data analysis. The focus will be on trying to detect how the participants co-ordinated their activities, how they divided the labour between them, and what kind of tools that were used in the collaboration process. 41 “Task” is to be understood as the design task the students chose themselves, the task of designing a TW-room for teaching 6th grades about endangered animal species. The “activity” is the process of collaboration running from March 1 st to April 11th, i.e. the period framing the first and last contact between the students. 191 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Evaluation approach The goal of the evaluation has been to conduct a naturalistic study of how participants in collaborative telelearning organise their work and learning activities (Guribye & Wasson, 1999). An ethnographic approach was chosen for three different reasons. There is already an emerging tradition of using qualitative methods or a naturalistic approach (Savenye & Robinson, 1997) to investigate different phenomenon within the area of Computer Supported Collaborative Learning (CSCL) (Koschmann, 1997). Also, the choice of Activity Theory (Leont’ev, 1978, Engeström, 1987) as theoretical approach in this study calls for the use of qualitative methods of data collection (Jonassen & Rohrer-Murphy, 1999, see also Nardi, 1996). In addition, Yin (1994) suggests, that “when asking a “how” or “why” question about contemporary events with little or no control over the behavioural events (p. 9)”, an exploratory case study design is called for. These taken together, and given that the goal is to understand how the students organise their work, and to see what tacit practices that underlie the accomplishment of work practises, the use of an ethnographic study could be fruitful (Pettinari & Heath, 1998). Ethnography (see Hammersley & Atkinson, 1983, Denzin & Lincoln, 1994; Harper, 1998) influences the design of the evaluation approach including the choice of data collection methods and analysis techniques, in other words, the evaluation is an iterative process where an ongoing analysis guides the data collection emphasis in successive phases. The data collection During the scenario, different data gathering techniques were used, providing different data sources. The most important technique was electronic collection of communication, both emails and logs of TW chat-sessions. The students had a total of nine TW chat sessions, five in which all three students took part, four in which only two participated. Seven of the nine chat sessions were held before the students decided on the task, only two after. This, which can be seen as a shift in communication mode, is discussed below. The TW chat logs were saved and forwarded by the students. There seem to be a few “holes” in the chat logs, i.e. parts of logs are missing. This is mainly due to the fact that the students forgot to save the chats 192 Chapter 7 VisArt when moving from one TW room to another, or that they experienced technical problems when saving. These “holes”, of course, might influence the validity of the data in that it can weaken the understanding of the collaboration process as such. To observe the students collaborating, immersing oneself in the virtual environment, in this case entering the TW room during sessions without directly interfering or taking part in the work, provided a good opportunity for taking field notes and watching the students “in action”, deepening the understanding of the process. A non-participatory role, a sort of passive presence, was adopted when observing the students, and the actual interaction with the students was minimal, limited to one question from one student regarding the saving of chat logs. Whether this passive presence influenced the students in a substantial way is hard to say, but when asked in the post scenario interviews, they stated this presence was of little or no influence on their work and collaboration effort. A total of 20 emails were sent between the students, nine using the team address (automatically forwarded), and eleven between two students without a copy to the third person. After the scenario had finished, mail that was originally sent without using the team address was, by request, forwarded by the students. Based on the reconstruction of the mail communication, it is reason to believe that all emails have been obtained. The kind of mail being sent, addressed to the team or to an individual, was evenly dispersed throughout the scenario. Post scenario interviews with all three students, one face-to-face and two telephone interviews, were also conducted. The interview guide (see Appendix Z) that was based on the preliminary findings of the data analysis and phenomenon observed during the data collection, loosely guided these interviews. The choice of Activity Theory as a theoretical framework, emphasising aspects like, division of labour, tool mediation, and rules, also made suggestions about what to look for during the data gathering and what to ask when interviewing the students. Informal conversations with the student situated in Bergen during the scenario provided additional information about the collaboration process, allowing for clarifying questions, and inquiries about the work already carried out, as well as the planned progress, to be made. The interviews were a valuable supplement to the data gathered during the scenario in that they presented the opportunity to ask questions to 193 Project DoCTA: Design and use of Collaborative Telelearning Artefacts clarify indistinct aspects of the collaboration effort. In addition, the student brought up new themes that helped in understanding the process. The data Qualitative methods, or a naturalistic approach, have been dominant in investigating the collaboration process. The data collected consist of chat logs from TW sessions, emails, transcribed interviews, and artefacts created in TW by the students. In addition, the transcripts of the electronically collected log files from TW Workplace, which includes periodic, chronological recordings of all artefacts in the environment, as well as statistical data of who was logged on when, have been made available. This might turn out to be valuable supplementary information in trying to understand the “time aspect” of the collaboration process; who was logged on when, were they logged on at the same time, and, if they were, what rooms did they work in, what changes did they make? The techniques, how the data were gathered, and the kinds of data produced are summarised in table 25, below. Table 25 Data gathering techniques Data gathering Techniques Gathered by Type of data Electronic collection of chat logs Saved and sent by students Chat logs Field observation during TeamWave sessions Taking field notes during observation Field notes Electronic collection of email Received automatically as a member of the team mailing list Emails Informal interviews Recorded and later transcribed Transcripts of interviews Collection of artefacts produced by the students Saved or received by researcher Artefacts and documents produced TeamWave logs Collected electronically Transcripts of TeamWave logs 194 Chapter 7 VisArt Preliminary data analysis As stated above, the focus of this study is on how the students organised their work. From an activity theoretical perspective the division of labour, tool mediation, societal rules, and the object of the activity are all important components of an activity. One can organise these components of an activity, like the collaboration process in VisArt, into an activity system (Engeström, 1987). This activity system can be used in the process of developing a deeper understanding of the activities under study (Bakardjieva, 1998). In figure 57, below, the team’s effort of producing a TW room for teaching 6th graders about endangered animal species, has been put into such an activity system. Figure 57 Producing a TW learning room as an activity system When looking for ways an activity such as this has been organised, the activity system might give clues about what aspects to emphasise. Questions such as: How did the students co-ordinate their work? In what way did they divide the labour? What rules facilitated, or constrained their activities? How did they take the community into account? How did they decide on the design task?, Did the goals and motives change in the course of the scenario? A few of these aspects will be treated more thoroughly in the following sections. 195 Project DoCTA: Design and use of Collaborative Telelearning Artefacts The collaboration process can be divided into four phases: • Phase 1: the VisArt training period (March 1st to 7th) • Phase 2: the first part of the Design Activity, before the students decided on their task ( March 8th to 12th ) • Phase 3: the latter part of the Design Activity, i.e. from having decided on the task till the end of the scenario (March 13th to 26th) • Phase 4: the time after the scenario had ended (March 27th to April 11th). Co-ordination In order to finish the assignment the students had to work together, and hence, had to co-ordinate their work. Co-ordination is thus to be understood as the efforts undertaken by the students in order to act together as smoothly as possible. This includes appointing meetings in TW, updating each other on new or changed appointments, dividing the labour, and keeping each other posted on who has done what. During the second phase of the scenario, before they decided on the design task, teaching 6th graders on endangered animal species, the students met regularly for synchronous collaboration sessions. The meetings were arranged by using mainly email, but TW was also used to make sure that everybody was notified42 . Email from Lucy I have made a note on the Message Board in Team03 suggesting that we meet to agree on how to solve the assignment. My proposal is 6 PM tomorrow Tuesday March 9th. We have three weeks to finish the task, or am I wrong? This kind of “double communication” occurred in phase one and the initial part of phase two, and was not used in the latter parts of the process. Given the fact that this double communication only occurred in the initial stages of the activity, one possible explanation is a lack of mutual understanding of, or mutual agreement on, forms of collaboration. Not knowing whether everybody would log on to TW before the suggested meeting, sending a mail in addition to the note on the message board, could act as a reassurance that everybody was notified. 196 Chapter 7 VisArt Later, new meetings in TW were arranged at the end of each session, Lucy: I suggest that we work individually and that we meet Monday or Thursday morning. Anna: Tomorrow? Lucy: Yes, if we must meet before Monday at 5 PM? Michael: I think tomorrow is too soon. I’ll go for Monday at 5 PM. Anna: (………) Monday is better for me too. Lucy: OK, Monday it is. If , for some reason, one student was not present in the chat sessions or the meetings had to be rearranged, the students left messages in their TW-room, or sent emails, as a standalone action, not as double communication. Another possible interpretation of the disappearance of the double communication is that the students developed an increased insight in the technical functioning of TW and email as the scenario ran. One example of lack of technical understanding is a reply made by Lucy to an email sent by Michael, using the team email address (which had all the participants as recipients). Hello. I hope you are feeling better. Anna and I have done training task 1 and 2, the 2nd and 3rd of March. I will place a note for Anna in our TeamWave room telling her that you have sent me an email. I’ll see you in the Team03 room in TeamWave. Lucy clearly does not understand the function of the team3 address, making an additional note in the Team03 room, informing Anna of Michael’s mail. Although this does not go directly to “double communication” it may be an indicator of poor technical understanding, leading to double communication. The establishment of routines for collaborating, combined with increased technical understanding and skills, may be possible explanations for the absence of double communication in the latter parts of the activity. 42 Eskil has translated this extract, and all the other extracts. For the sake of anonymity, the names of the students are fictitious. 197 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Communication mode Also, the collaboration mode seemed to shift from synchronous to asynchronous as the scenario went on, with a noticeable change before and after the decision on the task. As Anna expressed it in a post-it note in TW, two days after the decision was made: It’s really not necessary that we meet again until we have done what we agreed on Monday, is it? We don’t have to be online at the same time to work on our rooms. We can look at what the others have done when we are logged on, and leave a post-it, like this one, if we want to comment on anything. What do you think? And, when being asked during an interview, what she thought of the synchronous meetings on TW, she elaborated on the same subject. I don’t think it worked out at all, we met to often, (………), is it really that important to be online on the same time to discuss what to do? Of course, when deciding on the task, we had to be online using the brainstorm-tool, but in the project itself, we could have managed with a Message Board, we could leave messages and make comments. At least that’s what I think. After deciding on the task, the students divided the labour between them, each being responsible for one part. Their idea was that they should work on their own creating the rooms, and that the others should drop in and comment on that work that had been done, and propose changes they thought should be made. This may be a possible explanation to the shift in communication mode, also suggested in the post-it note and extract from the interview with Anna, above. Division of labour After deciding on “splitting” the task, and such, the world into six parts, the five continents and Norway, they divided the task between them in the following way: Michael: I don’t want to do Norway! Lucy: I can do Norway (I’ll see what I can find) Michael: I can do Africa 198 Chapter 7 VisArt Anna: I’d like to do Asia and America Michael: Norway may represent Europe? Lucy: We agreed on being more thorough on Norway. I guess that part is enough for one. Michael: Yes. Michael: I can do Africa + Australia This part of the chat session reflects the actual division of labour, but the main reasons for this division, and which student was assigned which part, are to be found in the part of the chat where the students are trying to decide and agree on the task. Lucy expresses an interest in Norway, and what she calls “social conditions”, and elaborates on her interest in the interview; (……) I wanted to take more than the science part into consideration, you know animals and things like that, that it has a social side. I live in Nord-Trøndelag and it has been quite a few discussions via the media concerning “predator policy” and things like that; and goes on: (……) I found that it was okay to relate it to Norwegian conditions, because I had nothing to contribute with in relation to other parts of the world. This does not go directly to the division of labour, but may be regarded as one aspect playing a part in the assignments of work areas, in that it reflects what can be termed as “personal interests”. Lucy being the only one expressing a wish about what she wanted to do, was granted her wish. The other students also gave answers to questions regarding the division of labour, that might support such an interpretation. EA: How did you divide the labour between you? Michael: It went by itself really, you know…, it depended on interest, maybe. It went really fast, dividing the labour, it presented no problem EA: So it was based on personal interest and… Michael: Yes EA: So there were no discussions as to who should do what? Michael: No 199 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Although not very extensive, Michael’s answers might point in the same direction as Lucy’s statements, that the assignment of subtasks depended on personal interests. Anna also made similar claims, but she, in addition, commented on what she saw as the reason for the physical division of the task into subtasks, each student being assigned two parts of the world: (……) and we found out that if we were to divide the task into six parts, six parts of the world, we could do two each. And then be given free leash within those two rooms, and then we could do the main room jointly, or how we wanted, in this manner of speaking it was a pretty okay division of labour. At least I think so. This division of labour may have contributed to a co-operative form of work, rather than a collaborative form, (cf. Dillenbourg, Baker, Blaye & O’Malley, 1996), in that the work was “divided among the participants each being responsible for a portion of the problem solving, rather than the work being a mutual engagement of all participants in a co-ordinated effort to solve the problem together” (Roschelle & Teasley, in press, cited in Dillenbourg, Baker, Blaye & O’Malley, 1996:190). This division of labour, inviting to a co-operative kind of work, may also have played a part as the communication mode shifted from synchronous to asynchronous in the course of the scenario. The students did not have to work synchronously, as they were to produce a room on their own, not as a joint effort. Although the division of labour opened for individual problem solving, the students recognised the value of feedback and agreed on providing this during the scenario. Feedback In spite of their agreement on helping each other and commenting on each other’s work, feedback hardly ever occurred. When being asked why this didn’t happen, the students answered that it was mainly because they wanted to finish the assignment as fast as possible, and that when they didn’t get any response when actually trying to comment on the work, they soon gave up. Lucy answered, when asked if the others commented on her work: 200 Chapter 7 VisArt No, they didn’t. I only felt that when we were finished, that they found it OK. But I really didn’t expect a lot of comments during the process anyway, because I believed that they felt that they had enough with their own part. But I looked at what they had done, because I felt I had to, ha-ha, to say what I thought about it (……) Anna said that she tried to make comments, but that she never “got any response” and that “ no one ever commented on her work”. Michael claimed, as stated above, that he liked working on his own when designing “his room”, but that he could have wanted to meet the others more frequently during the third phase of the scenario “to get some feedback” on his work. A possible explanation for this lack of feedback can be found in the interview with Michael: EA: Why didn’t you do that? (Give feedback) Michael: It was time pressure I guess. That we were to be finished by a certain day Anna’s statement in a interview may support such an interpretation; Anna: (… …) maybe the others also felt that it was to time consuming, that one had to concentrate on one self rather than the others, yes. They, again, ascribe this time pressure both to the new TW mediated way of collaborating, which they found time consuming, and the size of the task, which they found to large, and therefore reduced in the last days before the deadline of the design activity. Summary and preliminary findings In order to study and elucidate the organising of the VisArt collaborative activity of Team3, a naturalistic evaluation approach has been applied. Different qualitative data gathering techniques have been used providing a variety of data sources, ranging from electronically collected TW chat logs to transcribed informal interviews. 201 Project DoCTA: Design and use of Collaborative Telelearning Artefacts The preliminary data analysis has dealt with aspects like co-ordination, communication mode, division of labour, and feedback. During the entire scenario the students met regularly in TW, and co-ordinated their actions by using TW or email. Sometimes both TW and email were used as a means of co-ordination, providing a form of double communication. This form of communication disappeared as the scenario went on, maybe as a result of the establishment of regular meetings and patterns for collaboration. The task decision marked a noticeable line of demarcation in the communication mode. Before deciding on the task the rate of synchronous meetings and communication were higher than after the decision had been made. The asynchronous nature of the post decision work, may have its root both in the fact that the need for synchronous meetings were diminished, and that each student was assigned her/his own area of responsibility, contributing to a co-operative, rather than a collaborative form of work. In spite of a mutual agreement on providing feedback on each other’s work, this hardly ever occurred. Time pressure and a feeling that one had to concentrate on what oneself was doing, are probably the main reasons for the lack of feedback. 7.14 Evaluating the Organisation of a Collaborative Telelearning Scenario JOWAKE The particular perspective in this Master’s study (Wake, in preparation) is to evaluate the VisArt scenario from the instructor and facilitator viewpoint. More specifically, an understanding of how the instructors and facilitators organise their work is pursued. Activity Theory (Leontev, 1978, Engestrøm, 1987) is used as a conceptual framework to guide the data collection and analysis. Methodological aspects The data gathering techniques employed to address the research question can be characterised as qualitative methods. The reason for taking this particular approach, is partly justified by the theoretical foundations in 202 Chapter 7 VisArt the DoCTA project, partly by the phenomenon at hand, and mostly by the research question. First, the theoretical perspective chosen, Activity Theory, necessitates qualitative methods of data collection (Jonassen, 1998). There is also an emerging tradition of using qualitative methods, or a naturalistic approach, to investigate phenomenon within the field of CSCL (Savenye & Robinson, 1997). Second, the research focus on the intra-group collaboration of a group of six instructors/facilitators — or the people and artefacts in interaction — suggests that an ethnographic approach might be useful (Pettinary & Heath, 1998). The data gather techniques used were semi-structured interviews and collection of mail sent between the instructors and fascilitators during the scenario. Participant observation was also used to a certain extent, and the use of TW log files is currently being considered. Interviews of all the five instructors/facilitators that were assigned a formal and pre-planned role in the scenario were undertaken, and their responsibilities can best be characterised as “administratively responsible”, “technically responsible”, or “responsible for training”. In addition to the head instructor in Bergen, there were three, instructors located at HSH and HiNT43 who were responsible for integrating the collaborative telelearning activity into their course, and preparing their students for participation in VisArt. These roles are summarised in Table 26. Table 26 Instructor and facilitator roles in VisArt Formal role Main, pre-planned responsibility Head instructor Responsible for the overall administration of the scenario. Help/ training facilitator Responsible for training the students before the scenario, and receiving/answering/forwarding requests for assistance from the students during the scenario. Technical facilitator Responsible for maintaining the TW server, and helping the students with technical problems. HiNT instructor Responsible for preparing the students at HiNT for the scenario, and integrating it into the semester courses for his students. Stord instructor Responsible for preparing the students at HiNT for the scenario, and integrating it into the semester courses for his students. 43 Although there was only one formal instructor role at HiNT, the role was actually shared by two persons. 203 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Semi-structured interviews were performed using an interview guide (see appendix AA), that was designed to reveal the particular instructor’s views or opinions. The theoretical perspective chosen largely influenced the questions asked. The same questions were, in general, asked to all the instructors/facilitators. The questions investigated the division of labour, the tools or vehicles of communication and work organisation, the conception of roles and goals, and any sociocultural rules or norms. Information about the particular social context in which the learning activity took place was also gathered. To gain an understanding of how activities in VisArt were organised, all the email, sent between the students, instructors, and facilitators during the scenario was collected. As described in an earlier section, the design of assistance designated one head assistant, referred to here as the help/ training facilitator. This person was responsible for preparing the students for the scenario by training them in use of TW and in coordination and in getting them acquainted with each other. The help/training facilitator was also designated to receive all mail queries from the students who had questions. If the help/training facilitator could not answer a question, he categorised it passed on to either the technical expert (technical facilitator) or the content expert (head instructor). One particular aspect revealed through the interviews is connected to the question of instructor and facilitator roles. VisArt was, as mentioned, a collaborative telelearning scenario, where the students were to collaborate in constructing a visual artefact for teaching a subject of choice. In addition, however, it was also part of the DoCTA research project. The instructors and facilitators all had roles as researchers, collecting data about the students’ activities, a role supplementary to being an instructor. When asking questions about their activities in the scenario in an interview, it was sometimes a challenge to extrapolate the answers as being an “instructor” or a “researcher” answer. In addition to the instructors’ dual roles, there were a number of graduate students also following the student groups, collecting data, and observing their collaboration in TW. These graduate students had no formal role as instructors or facilitators, but a preliminary glance at the data indicates that they may have, in practice, been perceived and functioned as more “experienced learners” for the students. The students asked them questions via the Chat and Page tools, if they noticed they were online. This may have affected the outcome of the scenario, in that their presence 204 Chapter 7 VisArt may have affected the students’ work, and was not a planned part of the VisArt activity. The implications of this, however, are not considered here. Data collection The semi-structured interviews were held after the completion of the scenario and lasted anywhere from thirty minutes, to one hour and thirty minutes. The instructors and facilitators from University of Bergen were interviewed face-to-face. The instructors from HSH and HiNT were interviewed by telephone. HiNT had, as pointed out, two persons officiating as instructors, and they were interviewed together as a group. All the interviews were transcribed in full length, and their contents are being evaluated. The mail sent from students to the help/ training facilitator during the scenario, amounted to approximately 80. There was also email sent directly to the head instructor (i.e., not through the help/training facilitator). All the email will be treated with respect to content, numbers, and time of occurrence in the scenario. Preliminary data analysis Activity theory has influenced the data gathering and analysis in this study. In this section, a preliminary evaluation is presented with examples extracted from the data. An Activity System Engestrøm (1987)’s system for the analysis of activity, in relation to the collected data, is presented in figure 58. The basic components of the activity are the subject, the object and the community. Their relations are, in principle, mediated. Tools, both physical and psychological mediate the relation between the subject and the object. A division of labour mediates the relation between the object, or what is being worked on, and the community. The relation between the subject and the community is mediated by sociocultural rules. The example of mediating artefacts given in this particular activity system is by no means extensive, and depends on level of analysis. This level of analysis is an overview of VisArt as a whole, from the instructors’ point of view. The subjects in this particular focus on the VisArt scenario, 205 Project DoCTA: Design and use of Collaborative Telelearning Artefacts are the instructors and facilitators, and the object is the VisArt scenario, which they want to transform. Tools that mediate that relation are for example TW and e-mail, but also less “visible” tools such as language (written and spoken). Sociocultural rules mediate the relation between the subject and the community, and rules here can be for example “professional” culture. The community is the social “surroundings”, or environment of the subject, and can be the teaching institution, the funding organisation, and the group of instructors/facilitators. The objectcommunity relation is mediated by division of labour, and this means how the instructors/facilitators decide to divide the workload among themselves. Administering TW, creating the help system, or giving out assignments are examples. The outcome of the activity, is the completion of VisArt. Figure 58 VisArt instructors and facilitators as an activity system Division of labour The division of labour, which corresponds to the division of responsibilities between the instructors, was very much planned in advance. This is illustrated by one instructor’s response: 206 Chapter 7 VisArt “So it was the help/training facilitator who had the overall responsibility for the contact with the students, and he was supposed to separate questions from technical ones to questions about content. The technical facilitator was technically responsible, and I was responsible for content. The help/training facilitator catalogued the questions. I was surprised not to receive more mail, but he took responsibility... If he couldn’t answer, he passed them on.” In this excerpt, the head instructor is explaining how they had planned on helping the students during the scenario. A help-system, mediated by email, was designed. The students were to send questions to the help/ training facilitator, who was to read them, and pass them on to the assistant (head instructor or technical facilitator) who was assigned responsibility for the area in question. As it turned out, he answered most of the questions himself. One focus that will be examined in more detail in my dissertation, is how the division of labour was in reality in relation to how it was planned in advance. Artefacts The artefact that mediated the communication between the students and the instructors/facilitators, was mainly email. Communication in TW also occurred, both between instructors, and between students and instructors, if the instructors happened to be logged on simultaneously with the students. Between the instructors themselves, Email also seemed to be the dominant vehicle of communication between instructors, although the telephone was also used. “In order to clarify problems related to running of the TW program, talking to the technically responsible person, I used e-mail. And I have also spoken to “Stord instructor” on the telephone... Here, it was mostly face to face. I guess these are the three forms I have used.” For the HiNT instructors who were physically located at the same place, face-to-face communication in the form of scheduled meetings or occasional meetings in the “corridor” was also mentioned Sociocultural rules Naturally, it isn’t easy to pinpoint culture or tacit, intra-group norms or rules when oneself is an immersed part of that group or culture. The 207 Project DoCTA: Design and use of Collaborative Telelearning Artefacts instructors, however, seemed to have a mutual understanding of the required work and the division of labour. Tasks that were assigned to a particular instructor, were this person’s sole responsibility: “Being part of it, it is hard to say. But you have your own tasks. Or areas of responsibility, rather. … If I was withheld from doing it, it was my responsibility to ask someone else to fill in.” One might label this “professional”, in the broad, cultural, understanding of the word. Goals and motives Understanding the goals and motives of the instructors and the objective of scenario itself is revealed in the data. After analysing the interview data, it became clear that the instructors had a personal interest in the groupware technology. For example, one of the says: “I suppose it has been a genuine interest in groupware, or collaboration technology. (...)What I found particularly interesting in this tool is the fact that the people are very much aware of each other, bearing in mind the ”whiteboard”, where they made drawings together and worked simultaneously. (...) The fact that the tool had many functions making people aware of each other’s presence was very interesting in itself, because this is new to me. I also participated in IDEELS...” Other goals arise from an interest in collaborative learning, and in trying this out in practice through the deployment of VisArt. The objective for VisArt was, as given in the assignment, to give the students practical experience in collaborative learning, mediated by an Internet tool, and to theoretically reflect on the experience. There was a divergence of perceptions with the instructors. The divergence can be described as a difference in opinion of the importance of the product that the students were to produce — the visual artefact. Some perceived this as secondary to the main goal, while others were preoccupied with the quality of this artefact more than the collaboration process. Issues in telelearning scenarios: organisational, pedagogical and ethical In the following section, a number of specific issues arising in the data are discussed. These will be expounded further in the dissertation in preparation (Wake, in preparation). 208 Chapter 7 VisArt Virtual versus Real Environments. Being an instructor, or a student, in a virtual learning such as TW, will differ from a “real” learning environment, in the classical sense, with regards to communication and learning to be mediated by tools (mental or physical). This may affect the student—instructor relationship, the student—student relationship, and certainly the student/instructor—tool/technology relationship. These relationships require specific considerations in the planning of the learning activity. Focus on Collaboration Process. Although there was a general satisfaction with the completion of the scenario, the focus was on tool use and the collaboration process, rather than the end product. In addition, the objective set for the students was to gain experience in a collaborative telelearning activity, rather than producing a visual artefact for teaching a subject of choice. Scenario Planning. As one interviewed instructor pointed out, time used in planning the telelearning activity such as VisArt exceeds, by far, that of a learning activity where information and communication technology (ICT) is not a mediating instrument. Resource Demanding. The instructor interviews revealed that VisArt has been much more demanding on resources both in time spent as instructors and in facilitating the technological environment for the students, as compared to a non-telelearning course. This is contrary to the popular perception of technology as “self-running”. Instructors as Researchers. The VisArt instructors also had interest in the activity as researchers. Some of the instructors reported in the interviews that they felt themselves “drawn” into the activity. For example, most of the instructors ran TW for the duration of the working day, and spent a number of hours online every day,. While online they observed the activity of the students, and were available for questions and tutoring. In addition, they commented that they did not only receive email or requests for advice during regular working hours, but also in the evenings and on the weekends. This is not usually the case in an ordinary semester course. This “24-hour demand” for availability can put an amount of stress on them if they are not careful. Restarting the TW Server. Although it has been pointed out that TW is not a particularly difficult groupware system to administer (see section 7.6), the server stalled during the beginning of the scenario. This was 209 Project DoCTA: Design and use of Collaborative Telelearning Artefacts due to an overload on the system’s buffer file — a result of an extensive workload. In order to avoid a server crash, three daily restarts were required. As the students worked at hours that suited them best (not nine to four), including during the weekends, the TW server needed to be operative at all times. Thus a schedule for restarting the TW server was done planned and either a VisArt graduate student or the head instructor restarted the server three times a day. Importance of Facilitators. One instructor stated that the assistants/ facilitators, and the technical support, was far more important than the instructors in a telelearning scenario. Perhaps this can indicate that it is important to have a certain number of assistants, rather than a large group of instructors. Relationships. The instructor—student relation, largely mediated TW and e-mail, was affected by the distributed nature of the learning situation. It would seem, from the data gathered for this study, that the personal contact with the students that follows from a classroom situation, or being in the same room, where it is possible to see facial expressions and hear the tone of a voice, is obviously weakened by collaborating and communicating through an Internet tool. It is difficult for the instructor to monitor the internal collaboration process in the group and to appreciate their level of activity (both as a group and as individual students). This makes it more difficult to give personal feedback, to help the students whose activity level perhaps is below expected, and to lever group progress. In some respect they only saw the artefacts produced by the group in the TW rooms (i.e, the product of the collaboration process) rather than the collaboration in process per se. They did not, for example, have access to the intra-group mail communication. Having access to it might have helped their understanding of the intra-group collaboration process, but it would also increase the workload on the instructors, and would also raise certain ethical questions. Standing and Staring in Silence. Some of the instructors expressed hesitation in entering the group rooms in order to observe the activity of the students. They were afraid that it would interfere with the group collaboration process. One instructor, in an interview, rather humorously pointed out that one could compare entering a group room in TW while students collaborated, with going into a room where a small group of people are working without knocking on the door, and just standing there not saying anything, just staring. One can speculate that these feelings 210 Chapter 7 VisArt can be to the strong metaphorical concept of rooms adopted in TW — that they are perceived as someone’s personal space, and one is hesitant to intrude. 7.15 Supporting Collaborative Telelearning Research using Server Logs ØYVIND MEISTAD The focus of the Master’s thesis described here44 is on how automatically generated log-files can contribute to research in understanding collaboration patterns (Meistad, in preparation) in telelearning environments. One challenge for research within collaborative telelearning is what and how to collect and analyse data in a distributed environment. Traditional qualitative methods such as structured and unstructured interviews, participant observations, and video recording need to be tailored to distributed participants. Once the obstacles in collecting this type of data are overcome, questions about how to interpret and analyse the data persist. Collection of electronically logged data is on the other hand relatively straightforward. As Garton et al. (1999) point out, the restrictions are often “the amount of server storage space and the integrity of researchers and programmers in their study design (p. 92)” and electronic data gathering “replaces issues of accuracy and reliability with issues of data management, interpretation, and privacy (p. 92)”. Generally, the amount of electronically logged data can easily be overwhelming and again, questions of interpretation and analysis pose real problems. There exists a long tradition of using logged data in various usability studies focusing on identifying events from log files such as the number of errors occurring or when a menu is selected with the mouse etc., and graphically visualising the results (e.g., Fjeld et al. 1998, Guzdial 1993, Okada & Asahi 1999). Another interesting approach is to use the logged data in conjunction with video analysis to effectively retrieve parts of a video recording (Badre et al. 1995). Common in these types of studies is that the focus is on the interaction between human and computer. Another, 44 This section is based on Meistad & Wasson (submitted). 211 Project DoCTA: Design and use of Collaborative Telelearning Artefacts somewhat different tradition is to use logged data to measure the usage of various Internet resources such as FTP-servers, web-servers, etc. This kind of information is likely to be most interesting for the Internet providers providing the resources, but is also starting to gain attention from researchers (e. g. Newhagen & Rafaeli 1997, Garton et al. 1999). While most usability studies focus on the interaction between human and computer, the focus of the last kind is on the human communicating with a remote computer, mediating his or her communication through his or her local computer. The tradition of using logged data for identifying patterns of collaboration mediated by computers between two or more geographically dispersed persons appears to be non-existent as no references to others with the same or a related approach have been found. This study explores this possibility and tries out various techniques, with a mere goal of examining whether or not the approach is promising. Method The research was conducted using automatically generated logged data, generated by TeamWave Workplace (TW) during the VisArt scenario. TW comes with the usual logging mechanisms such as recording that a user logs in and out of the server, moves between rooms etc., as well as a nontraditional feature, a repository. All rooms are saved in the repository, and it thus becomes possible to recreate earlier states of any given room using an option called version control. Both these mechanisms (the server log and version control) are used in conjunction with two prototypical tools developed by Mesitad (in preparation) and described below. The TeamWave Workplace Server File All major interactions with the TW server are electronically logged in a server log-file called server.log. The file is automatically generated by the TW server on its first start-up, and is then updated as the various users interact with the server. During the VisArt scenario the file grew from 0 KB to over 2.2 MB with 38,207 lines. Figure 59 shows some extractions from the VisArt server.log file. The extractions describe some of the author’s actions when he logged into the VisArt server for the first time. 212 Chapter 7 VisArt Fri Feb 19 15:25:08 1999 oyvind entered Overview-room Fri Feb 19 15:25:26 1999 oyvind left Overview-room Fri Feb 19 15:25:26 1999 oyvind entered Klasse-diagram Fri Feb 19 16:32:35 1999 oyvind left Klasse-diagram Fri Feb 19 16:32:35 1999 connection closed for oyvind Figure 59 Extractions from the VisArt server.log file As one can see in figure 59, each line starts with information about date and time. Thereafter, the user is identified and the user’s particular action is recorded. For example, the second entry tells that on Friday, February 19th 1999, at 15:25:26, oyvind left the Overview-room. In addition to logging information about users entering and leaving rooms (as shown in figure 59), the TW server logs users logging in and out of TW, information about the server starting up, the creation of rooms, the writing of rooms, and the server shutting down. Although the structure of the server.log file is rather simple, the often large number of users involved (e.g. in VisArt: 32 students, 3 instructors and 10 researchers) creates such a large file that it is difficult to glean information from the file. Therefore, a software tool that interprets the entries has been developed. The TeamWave Workplace Version Control Every room created in TW resides on the TW server, and is stored in a repository. When a TW user selects the menu option to save the current room, or leaves a room when no other person is visiting the room, the room is saved to the repository. Every tool created in each room is also stored in the repository. As a room is saved, tools that have been changed, or added, or moved around, etc. since the previously saved version of the room, are saved. When a user enters a room, the last version of the room and the last version of the tools residing in the room is automatically retrieved from the repository. TW’s version control enables earlier versions of the current room, or earlier versions of a specific tool to be retrieved from the repository. The average number of versions stored in the repository for a single room at 213 Project DoCTA: Design and use of Collaborative Telelearning Artefacts the end of VisArt was approximately 50. The version control option enables a researcher to recreate the state of various rooms and tools during one’s analysis. Thus, long after the completion of the scenario, one is still able to follow how the scenario evolved for a particular team. The potentially large number of available versions of one room or tool, does, however, make such an analysis difficult. Therefore a software tool, TRA, was developed to help researchers keeping track of how the various rooms evolved over time. Two tools for analysing log-files Currently there is a lack of software designed for research on automatically generated logs with the intention of identifying collaboration patterns. To support the evaluations of scenarios where TW was used, it was necessary to develop two prototypical tools for the analysis. The tools, the Server.log analyzer (SLA), and the TWWRoom Analyzer (TRA), were both developed in Java 1.1 using the Java Foundation Classes 1.03 as graphical user interface components (when available in VisualAge for Java, the tools will be modified to use the generally available Java Foundation Classes 1.10). The programs differ from each other in that SLA reads in one log-file, sorts out a researcher specified group of persons, and then uses the information to calculate some results from the log. TRA, on the other hand, is a tool to help researchers in classifying the use of various tools inside TW, organising their classifications, and registering how the use of the tools evolves over time. In this study TRA was used in conjunction with the TW version control and the output from SLA. Server log analyzer SLA takes as its starting point the server.log file. The VisArt scenario was completed by 11 teams and SLA focuses on identifying the times at which a team could have possibly collaborated synchronously in TW (i.e., two or more of the team’s members must have been logged on at the same time). To find out what types of collaboration was possible for a given team at a given time, SLA reads through the server.log using a researcherspecified names of the team members. SLA creates a log for each group member, as well as a group log consisting of every entry from the individual logs, appearing in the same order as found in the server.log. The 214 Chapter 7 VisArt resulting logs (4 in this case) contain redundant information. SLA removes unnecessary information from the group log, and adds some statistics to the group log. An excerpt from such a log, (usernames and room names have been changed) is shown in figure 60. ———————————————— - Wed Mar 17 , 1999 ———————————————— 10:24:43 - Subject02 entered Teamx 10:28:27 - Subject02 left Teamx after 3 minute(s) and 44 second(s). 10:28:27 - connection closed for Subject02 *** subject02 - time in TeamWave current session: 4 minutes. *** subject02 - time in TeamWave all sessions: 12 hour(s) and 31 minutes. 12:23:49 - Subject02 entered Teamx 12:31:39 - Subject02 left Teamx after 7 minute(s) and 50 second(s). 12:31:39 - connection closed for Subject02 *** subject02 - time in TeamWave current session: 8 minutes. *** subject02 - time in TeamWave all sessions: 12 hour(s) and 39 minutes. 12:58:30 - Subject02 entered Teamx 13:02:05 - Subject03 entered Teamx 14:01:48 - Subject01 entered Teamx 14:24:31 - Subject03 left Teamx after 82 minute(s) and 26 second(s). 14:24:31 - Subject03 entered TeamRoomX 14:25:15 - Subject03 left TeamRoomX after 0 minute(s) and 44 second(s). Figure 60 An excerpt from SLA’s group log-file This log is used to generate a graph that illustrates the times at which the various group members were logged in to TW. If the graph shows that two or more group members were logged in at the same time, they had the possibility to collaborate synchronously. Figure 61 shows a graph for one of the groups from VisArt. The left column of the graph denotes the time of the day. Each day starts on the bottom at 0, and goes up past 23 and up to 0 again. The bottom axis specifies the date that is displayed. If a date is missing (e. g. Feb 26 and Feb 27), it means that none of the group members logged in that day. 215 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 61 Graphs of Team Activity in VisArt Interesting information may be found looking at the graphs. First they tell us that from February 25 - March 24, there were only three times when it was possible for the whole group to collaborate synchronously. VisArt started on February 25 when the group members were told to download the TW client, and log in to the server. They were given a few days to carry out this task. March 1 - 7 was a week of training in use of TW and collaboration, and March 8 - 26 was the period of carrying out the design task. By examining the graphs, it becomes apparent that subject02 did not log in to TW before the fourth day of the training period. Is also becomes apparent that the formerly active subject01, did not log in once between March 4 and March 15. Consequently, even asynchronous collaboration is highly unlikely during this period. To explain why these more or less negative patterns occurred, is difficult using a simple log-file as the only kind of data. It is important, however, that the researcher is aware that one of the three team members disappeared for 10 days in the middle of a 4 week collaborative learning task. This is valuable information that enables the situation to be investigated more closely using, for example, interviews, and / or questionnaires. 216 Chapter 7 VisArt Server log analyser, version 2 SLA is a fairly static application, and the type of output it provides is limited compared to the possibilities. A new, more flexible and dynamic version will be implemented based on the outcome of this project. Using SLA in conjunction with TRA in an ongoing analysis, however, identified a few urgent needs, thus a few version 2 functions have been implemented in order to give the desired output from SLA. Due to limitations in the graphical user interface (GUI) of SLA, the GUI of SLA version 2 (SLA2) will be completely rewritten. Thus, the tool currently exists in two versions, of which both have been used in conjunction with TRA. The most important new function is the ability to sort the server.log based on the rooms involved. SLA focuses on groups of persons, and filters away all other information present in the server.log. For example, information that a researcher is present in the room at the same time as the members of a team would be discarded. This is, however, important information needed by TRA. Therefore SLA2 is implemented in such a way that SLA2 does not filter away such information. TWW Room Analyser TWW Room Analyser (TRA) is designed to be used in an analysis of collaborative learning environments that use a room metaphor or another analogous structure such as a web page. In this study, TRA helps researchers in classifying the use of various tools inside TW, organising their classifications, and registering how the use of the tools evolves over time. TRA, see figure 62, requires the researcher to use the TW version control and the output of both SLA and SLA2 to obtain input data for TRA. Ideally it should be possible to retrieve most of the information needed by TRA from the TW repository, but that path has not yet been investigated. Even then, some of the information needed by TRA, such as a classification of the uses of the tools, would still have to be identified by a researcher. Once the data has been input to TRA, the researcher can use TRA to track changes to rooms and tools over time. 217 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Figure 62 One of many possible views of TRA TRA, shown in figure 62, is a dynamic tool that is used to describe the learning environment of a VisArt team. TRA comprises several graphical user interface components that are displayed at different times. One important component is a tabbed pane that helps the researcher quickly navigate between various room states at different levels of detail. In figure 62, three tabbed panes are visible: the room tabbed pane where the room “Team05” is active; the version tabbed pane where version “5” is active; and the tools tabbed pane where the “Add new tool” tab is active. Table 27 summaries the main components of TRA telling when each component is visible, and describing each component’s input and output. In the table, one cell denotes one of the main components of TRA. Table 27 should therefore be studied in conjunction with figure 62. 218 Chapter 7 VisArt Table 27 The main components of TRA Persons influencing the LE LE include the following rooms - Always visible - Always visible - List of users influencing the current team’s LE - List of all rooms used by the current team - Possible to add new persons - Possible to add new rooms Tabbed pane (no 1) is always visible. A tab is created for each room from the room list above Properties tab is always visible, and when active enables several attributes (e. g. where to save the current analysis, time between each automatic save, whether to backup analysis) to be set Room tab is always visible, and when activated enables several attributes (such as who created and when created, etc.) of the current room to be set. The available tools and available versions list become visible. Available tools: - Visible when a room tab is activated - List tools available in the current room - Possible to add new tools Available versions: - Visible when a room tab is activated - Lists versions of the current room - Possible to add new versions Tabbed pane (no 2): is visible only when one of the room tabs is activated in the tabbed pane (no 1) Summary tab when active displays information about the state of the room (e. g. number of tools, type of tools, if a tool is used asynchronously or not, etc.) Version tab when activated displays information about the version (e. g. when created by whom, extractions from log-files concerning the version, researcher comments to version, etc.) Tabbed pane (no 3) is visible only when a version tab (from tabbed pane (no 2)) is activated Add new tool tab when activated enables tools found in the current version of the room to be added to the list of tools and a tool tab is created for the tool. If a tool existed in the previous version of a room, it is possible to remove it from the current room. The researcher can also copy in parts of log-files concerning the version, and write down some general comments concerning the version Tool tab when activated displays information about the tool type (e. g. doorway, postIt), who created it, if it was used synchronously, asynchronously, etc. In addition, each tool can be classified by a researcher according to the type actor (e. g. leaner, tutor, manager); the roles taken; the modulator of the actor (i. e. individual, team, or entire group) etc. according to Téleuniversité’s model for a virtual campus environment (Paquette et. al., 1995). It is also left space for other classifications, as well as general comments about the use of the particular tool. TRA can present information about any number of rooms. For each room there may be any number of room versions, with each room version containing any number of tools. As TRA is currently being used during analyses of VisArt, it is possible new features may be required. If so, then TRA will be revamped and the user interface may change. The 219 Project DoCTA: Design and use of Collaborative Telelearning Artefacts TRA feature where the designer can record the use of the tools in a particular room is shown in figure 63 (see the bottom pane with the Designer tag and window for explaining tool use. Figure 63 Using TRA to record analysis of tool use Conclusion The dissertation work described here has resulted in the development of two prototypical tools that take automatically generated data as their input. SLA does so as a fairly automated process, whereas TRA depends on a great deal of interaction with the researcher. Information produced by SLA is currently useful for identifying the times at which a team had the potential to collaborate asynchronously or synchronously, as well as well as if anybody else was logged in to the same room as one or more of the team members. To say that it is possible to collaborate is, however, 220 Chapter 7 VisArt not to say that collaboration occurred. Thus, SLA’s output can be used to focus other methods that are being used to interpret the interaction between users of TW through its various tools. TRA was developed as an aid for studying the learning environment of a team (e.g. the rooms involved, the persons influencing the environment, the available tools, etc.) Incorporating Télé-universite’s tool classification model for a virtual campus environment (Paquette et. al., 1995) in TRA, it suggests how the particular uses of the tools found in the team’s rooms may be classified. Using these tools in conjunction with other data gathering methods such as interviews or video observation, the researcher will be able to develop a richer picture of collaboration patterns. 7.16 Coordination in collaborative telelearning RUNE BAGGETUN The challenge of collaboration between geographically separated team members in a learning context is the focus of this section. In VisArt participants collaborated using TW as well as a personal email application. VisArt was studied in order to find out how the participants coordinated their work using this suite of tools. Wasson (1998) identified a set of interdependencies in collaborative telelearning (see table 4) and the VisArt scenario provided the opportunity to study coordination in a situation were the participants are faced with these (inter)dependencies. Building on the concept of coordination mechanism (Schmidt & Simone, 1996) we decided to focus our research on the artefacts and procedures45 used for coordinating distributed work. A coordination mechanism is defined Schmidt & Simone define as “a mechanism that, by means of a set of conventions and prescribed procedures and supported by a symbolic artefact with a standardised format, stipulates and mediates the coordination of the distributed activities of collaborating ensembles” 45 A procedure is seen as a resource an individual can turn to, rather than a strict rule. 221 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Method Data from 2 teams46 was gathered and studied. Some of the other researchers also provided relevant data from other teams that they were studying. In order to gain a deeper understanding of the issues being explored, the activities of 6 students (2 teams of 3) were closely followed using a number of techniques. Virtual Field Study. The distribution of the participants gave little opportunity to conduct a normal field study, but the features of TW gave instead the opportunity to conduct a “virtual” field study. When logged on I could observe the actions of team member inside the virtual world (i.e., the TW rooms), as well as track any changes made by others since the last time they logged on. What could not be observed was what actions were taken outside TW. For example, it was impossible to know if the team member from HSH surfed on the web while talking, in TW, to his/ her team companions. Interviews. Three types of interviews were conducted. Three of the interviews were conducted as “In-Depth, Open Ended Interviews (Patton, 1987)”. The second type was informal open-ended interviews. For example, talking casually to team members in the hallway or at lunch. Several times a participant who had some opinions to share or something else that they wanted to tell approached me. The last type of interviews conducted were “context interviews” conducted in TW using the Chat tool. They are being referred to as “context interviews” because the interviews were conducted with team members while they were working (i.e., in context). All the context interviews were informal and were saved as a text file using a special feature of the Chat tool in TW. Automatic Data Logs. One important feature in TW is the automatic logging of data. Almost all user actions are logged. The log analyser tools SRA and TRA, as described in section 7.15, were used in this study. Using these tools it was possible to see each team member’s movements inside TW (e.g., from room to room), to see how long each team member was logged on, and to see who was logged on at the same time in a room. This gave an indication of the synchronous/asynchronous nature of the collaboration within a team. 46 Rune was assigned 2 teams to follow. 222 Chapter 7 VisArt Lotus Screencam. Lotus Screencam enables a researcher to record what is happening inside a TW room without the researcher having to be present. Screencam records movies of screen activity through a full video capture that includes all mouse-pointer movements and other screen events. Screencam was used to record screen activity in several TW rooms. These recordings are a rich source of data that can be studied over and over again to find new subtleties in the on-line interaction. Analysis The concept of coordination mechanism can be used to determine how the group invented, used and adopted mechanisms that supported the coordination work during the scenario. A preliminary analysis has thus far, revealed three collaboration patterns. These are: • No initial discussion regarding the use of tools • What was obvious for some was not obvious for others • Different expectations regarding whole sequences of related actions Each of these is briefly presented below from a coordination mechanism perspective. No initial discussion There was no agreement or discussion of how to use the different tools before actually using them. This led to some unfortunate implications. For example, the team members on one team used different tools for the same purpose and time was spent checking the different tools. For example, group messages were left on several different tools in several different rooms instead of only on an agreed upon tool in one room. From a coordination mechanism perspective we can see that although an artefact exists (e.g., a message board), it is not very useful unless an agreed upon procedure for use, i.e., a coordination mechanism, is adopted. What was obvious for one was not obvious for others With procedures missing and no conventions developed, the use of tools was dependent upon the team member’s own skill and ideas about when and how to use the different tools. This again led to some unfortunate implications. Some team members put a lot of effort into using a message board for communicating messages to the team while another team member used the 223 Project DoCTA: Design and use of Collaborative Telelearning Artefacts message board just to get information or post an answer. In addition, a completely different tool, i.e., the PostIt note, was used to communicate messages, and it placed in a different room (in this case it was placed in the individual’s room and not the team room). What was obvious for some was not as obvious for all the team members. In this case the whole team should have used the one message board in the team room as intended by the one who created it. Again we see that the missing procedure for how to use the tool created trouble. The procedure developed was, in this case, personal. It seemed that each team member developed personal “styles” of coordinating their own effort to accommodate the others way of working. Different expectations Different expectations regarding whole sequences of related actions were different among the team members. Because of this simple action such as dividing labour or discussing topics was not as efficient as it could have been. For example, in a chat session a question or suggestion from a team member would often get no response because the others thought that being silent was the same as saying “I agree”. The one with the suggestion or question was, however, accustomed to getting feedback. From her experiences in Internet chat groups she expected at least to get a smiley (or some other feedback) otherwise she would feel ignored or that her suggestion was a bad one that the other did not want to discuss. The no feedback strategy in a virtual environment means no information at all because the lack of other cues such as facial expressions and body movements are missing. We see (again) that missing procedures are causing problems. Summary Even though there were procedures missing for how to use tools, both teams collaborated well. Using mostly the chat facility in TW they agreed both on what to do when they were online, and what to do before the next meeting. The more interesting findings in this study that will be elaborated further in my dissertation (Baggetun, in preparation) were: • Personal styles were developed when there was a lack of conventions • We need to determine how to help users develop good procedures and detecting bad ones • We need to find out how to help the students maintain a balance between personal styles and agreed upon procedures. 224 Chapter 7 VisArt 7.17 Facilitating Help and Training in VisArt HELGE UNDERHAUG The main focus of this Master’s thesis is the design and deployment of and assistance in VisArt. The students began VisArt with a one-week training period that provided training in both use of the TW tools and in collaboration. Assistance was provided from the start to the end of VisArt. An evaluation study of the training and assistance asks the following two questions: How effective was the training and assistance designed for learning to use the TW tools and learning to collaborate? How can the training and assistance be improved? Section 7.7 described the training design and introduced the training assignments. The organisation of help and assistance was described in section 7.5. This section describes the data collection and presents a very preliminary analysis of some of the data. Data collection Several data collection techniques were used, including: • • • • • email that was sent to the head assistant email that was sent within 2 groups interviews with two members from one of the groups notes taken while observing collaboration/group work within TW questions regarding training and assistance included in the pre- and post-questionnaires • a web-based self evaluation test • counters on the TW web help pages • saved chat’s The email that was sent to the head assistant was collected in order to analyse what difficulties the students experienced during both the training and the design activity. The team email collected from two teams will be used to determine what they communicated about outside TW, and if and how they asked for help from their team members. 225 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Notes taken while observing the 2 teams collaborating in TW should contribute to understanding the other data. In addition, Chat’s from these sessions have been saved. An interview with one of the teams was held at the end the scenario to supplement the observation notes and the chat’s. The self-evaluation test was developed to help the students determine what they had learned through the training phase and to help identify if they perhaps should do more training before starting the design activity. This gives us data on how they felt their progress was during and after the training phase. The pre- and post-questionnaires also contained questions regarding the training — how they experienced the training and asked if they had any comments on possible improvements. Preliminary analysis A very preliminary analysis reveals that the majority of the students had some experience with collaboration / teamwork from both prior education and work life. This must be taken into consideration when looking at what the students learned about collaboration in the training phase. Figures 64 and 65 present background questions on collaboration from the profile-questionnaire that was filled out before the students started the training phase. Figure 64 Experience with collaboration from prior education A question about the usefulness of the training in TW given before the design activity was included in the post-questionnaire. The question was included in the post-questionnaire47 in order to give the students the 47 As opposed to the pre-questionnaire which was also given after the training but before the design activity. 226 Chapter 7 VisArt opportunity to finish the design task before judging if the training ha been useful. Figure 66 shows that the majority of the students felt the training was useful. Figure 65 Figure 66 Experience with collaboration from work life Usefulness of the training in TW, after completion of the design task The tendency so far seems to be that most students were rather pleased with the training phase. Some comments on the post-questionnaire also indicate that it could have been done better: “There should have been better time on the training” “It was a bit short, I got help from a fellow student. This made it easier and more effective”. Both students in these examples complain about the length of the training phase. This could be attributed to insufficient information about the upcoming scenario that caused some students to be delayed in starting. 227 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Or, it could be attributed to the minor difficulties some had with getting attached to the server. This last reason is also supported by the fact that in the first week of the training, 24 emails requesting help with technical matters were sent to the head assistant. Further analysis will try to elucidate the dissatisfaction with the training. 228 Chapter 8 Results and conclusions Part III Results and Conclusions 229 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 230 Chapter 8 Results and conclusions 8 Results and conclusions This part of the report discusses the results of the first phase of project. Two issues that have been central to the project are the identification of collaboration patterns and challenges in studying students’ online learning. These two issues are discussed below. Section 8.1 presents the very preliminary results of our integrative analysis that aims to identify collaboration patterns in collaborative telelearning environments. In section 8.2 a brief, but important, discussion on methodological issues for studying online environments is presented. 8.1 Collaboration Patterns Collaboration patterns define sequences of interaction among members of a team (such as students) that satisfy established criteria for collaborative behaviour. We have already reported (see sections 2.2-2.4) on several such collaborative behaviours: genuine interdependence, division of labour, sharing information, awareness of others, joint thinking, etc. Although collaboration patterns can be found in physical as well as virtual environments, in DoCTA we have focused on how collaboration patterns manifest themselves in virtual environments. Finding collaboration patterns In the VisArt scenario we have searched for collaboration patterns by analysing interaction data from data logs, videotapes, observations, and interviews both between students and between students and facilitators (instructors and assistants). We have identified several instances that we believe can be characterised as collaboration patterns (Wasson & Mørch, to appear). Four of them will be explained and discussed below. However, 231 Project DoCTA: Design and use of Collaborative Telelearning Artefacts the reader should keep in mind that our results thus far are preliminary48 and that we expect to report more on this later. We see great promise in identifying collaboration patterns and expect them to be useful in further work on virtual learning environments. After the four patters are described, a collaboration pattern structure that is shared by all the collaboration patterns we have identified is presented. The section concludes with a short discussion future work. Four collaboration patterns Four collaboration patterns are described and discussed below. The patterns have been identified in the VisArt scenario by analysis of the data reported earlier. We use examples from the data to illustrate the patterns by showing concrete, but representative, activities. Pattern 1: Adaptation Synopsis: This pattern describes how students gradually adapted to each other’s practices when working together to solve a common problem. There were no established protocols for how to use the different tools in TW. The students used a wide variety of tools when communicating with each other (e.g., a message board, shared rooms, chat, post-it notes, etc.). Indeed, several of the tools could actually be used to accomplish the same end and since the students preferred to use tools they felt comfortable with they would choose differently. After an initial phase of interaction, however, the more or less arbitrary tool use pattern changed. Here is an excerpt from an interview that illustrates this change: Sam: “… but also Pat had a tendency to use post-it notes inside his room” Interviewer: “…Yes, they are really popular.” Sam: “We had to run into his room and see what new post-it notes had been posted there instead of looking at the message board or the group room.” In this excerpt one of the students (Sam) has adapted to another student’s (Pat) way of communicating by placing post-it notes in his private room and 48 Remember that the majority of the data collected is part of a set of Master’s theses that are in various stages of completion. 232 Chapter 8 Results and conclusions waiting for people to come by and read them instead of using the shared room (group room). The shared room had been explicitly designed to be a common meeting area. In spite of this, the student (and later the others) adapted to one student’s ways of using a communication tool and ignored the initial set-up of the public room. The facilitator of the above scenario (i.e., the observer who noticed it) summarises it in the following way: “A person enters into his own room looking for messages. He then takes a roundtrip through the other team member’s rooms to see if any messages have been posted there. He then returns to his own room”. This adaptation to a new team behaviour that followed as a result and persisted throughout the scenario we have identified as the adaptation pattern. We call this a collaboration pattern because we have found similar interaction sequences in the other teams as well. Pattern 2: Coordinated desynchronisation Synopsis: This pattern describes how coordination of activities between team members changes after they have identified a common goal. Many of the teams would start working synchronously, for example by having a simulated real-time meeting. In the meeting the team members would agree upon a goal to pursue, divide the work and then work more or less asynchronously to accomplish it. The latter kind of working meant fewer same-time meetings but required regular “coordination points” to ensure that the divided work would progress towards the common goal. TW tools supported this kind of coordination —they served as coordination mechanisms. Here is an excerpt from a session that started with a conventional (realtime) discussion of meeting times: S1: I vote that we work individually and that we meet Monday or Thursday morning. S2: Tomorrow? S1: Yes, we must meet before Monday at 5pm. S3: I think tomorrow is too soon. I’ll go for Monday at 5pm. S2: (…) Monday is better for me too. S1: OK, Monday it is. 233 Project DoCTA: Design and use of Collaborative Telelearning Artefacts When a meeting time later had to be rearranged because one or more of the students could not make it they would leave messages for each other in their TW rooms, or they would send emails. The data from later activities shows a gradual shift from synchronous to asynchronous communication. This is further evidenced by a comment from S2, expressed as a post-it note in TW: “It’s really not necessary that we meet again until we have done what we have agreed on Monday, is it? We don’t have to be online at the same time to work on our rooms. We can look at what the others have done while we are logged on, and leave a post-it, like this one, if we want to communicate on anything. What do you guys think?” After having deciding upon the goal, the students would divide the work among themselves, each being responsible for one part and work asynchronously on this. A reason for this shift from synchronous to asynchronous communication can, in part, be explained by the following comment by the facilitator: “They (the students) would work on their own — creating the rooms — and the others would drop in and comment on the work that had been done and proposing changes they thought should be made.” Not only were the students working on their own in an asynchronous fashion; they were coordinating the activity by giving feedback to each other. We also see the Adaptation pattern at work in this example. The asynchronous communication is carried out in a situation satisfying the conditions of the Adaptation pattern as described above. This instance of the Coordinated desynchronisation pattern uses the Adaptation pattern as one of its parts, and we can say that the larger pattern is composed of a smaller pattern. This part/whole structure can be found in many examples and is similar to how patterns are organised in other “pattern languages” (e.g., Alexander et al’s (1977) architectural patterns and Gamma et al’s (1995) software design patterns). 234 Chapter 8 Results and conclusions Pattern 3: Constructive commenting Synopsis: This pattern describes commenting behaviour. Comments that are neutral (e.g., just to the point) are perceived to be less useful than comments that are also constructive (e.g., suggesting what to do next) or supportive (e.g., encouraging). This pattern is related to Pattern 2 and often used as one of its components. We found that lack of feedback was perceived to be problematic in several of the teams. Students wanted to get feedback from each other on the artefacts they have made in their rooms. Feedback is also required to keep each other up to speed and moving in the right direction towards the common goal. This could be accomplished by using the tools such as a message board, or post-it notes, etc. A chat dialog that took place between students illustrates this pattern. While collaborating in TW to create a class diagram (see section 4), two students were using the chat tool to make proposals to each other and waiting for responses before continuing. Initially – student B asks for permission to execute her proposals: B: What use cases do we have? A: What about “Making Appointment”? B: I agree. But we can have two kinds of appointments: through dentist and through assistant. Will this require two sub-cases? A: Let’s assume that making appointments can only be done through a secretary B: OK Later this interaction sequence (of getting a confirmation before continuing) changed. One would post a diagram and keep on working until they heard something from the other person. The following excerpt illustrates this as well as the linguistic mechanisms used to coordinate the activity. Later – both of them are commenting and encouraging each other: A: I have changed the diagram. Do you agree? B: Yes, it is great! You have created a connection between “report” and “dentist” A: Yes … A: You impress me! 235 Project DoCTA: Design and use of Collaborative Telelearning Artefacts The initial commenting is somewhat of a hindrance (time consuming) since feedback is requested and given at every atomic step in the interaction. However, in the latter case the student A is given the opportunity to change something that has built up over some time (a diagram) and the changes are likely to have more impact compared to the previous version of the diagram. The author of the changed diagram will therefore be subject to more severe commenting than in the one-step-onecomment case. When commenting of the latter kind is constructive and encouraging it is perceived to be more supportive of collaborative behaviour. Thus, we call this pattern Constructive commenting. Pattern 4: Informal Language Synopsis: This pattern describes how interaction often starts in a formalistic style and gradually becomes more informal as team members get to know each other. Frequent use of slang words or dialects local to the community working together is common in instances of this pattern. This dialog is extracted from a dialogue between 2 students (bruker1 and bruker2) who are, as in the previous example, creating a class diagram together. In this first part, the two are “getting to know” each other49 : [12:52:40] bruker1 says: Skal jeg lage flere aktører? Tannlege osv? (Shall I make more actors? Dentist, etc.?) [12:53:15] bruker2 says: Greit (OK) [12:54:45] bruker1 says: Enig i at vi har alle ansatte på høyresiden? (Agree that we have all the employees on the right?) The next excerpt is taken from the dialog about half an hour later. The two seem to feel more comfortable with each other. They both use slang or “dialect” 50 . Also, they both use abbreviations and uncompleted sentences: 49 50 In this excerpt and the next example, the original Norwegian dialect is given in Italics with an English translation in parentheses. In Norwegian there are 2 formal languages and an unknown number of dialects. Every little place in Norway has their own dialect. The two students in this example come from somewhere outside Bergen (on the western coast) and somewhere close to Bodø (in the north) – their dialects are very different! 236 Chapter 8 Results and conclusions [13:31:39] bruker1 says: Se! (Look!) (Subject points at new connectionarrow) [13:31:48] bruker2 says: Kossen? (How?) [13:31:51] bruker1 says: Vi har laga Inheritance-pil! (We made an inheritance-relation!) [13:32:14] bruker1 says: Den ligg nederst i “tabellen” av link-typer. (It’s in the bottom of the “table” of link-types) [13:32:21] bruker2 says: Aha! (Aha!) [13:33:23] bruker1 says: Ka no då? (What now?) [13:33:49] bruker2 says: Forbindelsar... (connections...) [13:34:08] bruker2 says: Ml.klient - tannlege? (bet. Client- Dentist?) [13:34:35] bruker1 says: Ja, selvfølgelig... E litt sein i hodet i dag... (Yes, of course…My head works a bit slow today…) When the students had “warmed up” to each other, they were more comfortable with each other. They would often shift to an informal language and use slang words and local dialects during the interaction. In the excerpt above one of the students used the word “Kossen” (How) while communicating using the chat tool. This term is not common in written Norwegian, but it is common in regional speech. By using the word Kossen, the student (bruker2) identifies himself as someone coming from a certain region in Norway (in this case Bodø). This makes the interaction much more personal and informal. Structure of collaboration patterns We anticipate that a common structure for all the collaboration patterns can be useful. First, it will simplify later reuse since patterns can be stored in an indexed catalogue of patterns. This similar to how patterns are organised in other pattern languages (e.g., architecture and software). Second, it becomes easier to search for new patterns when the characteristics of existing patterns have already been established. We tentatively propose that each pattern has the following eight characteristics: (1) index, (2) name, (3) short description, (4) related patterns, (5) example, (6) conditions, (7) mediating artefact(s), and (8) references. Each of the items is briefly explained in Table 28 next page. 237 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Table 28 Characteristics of Collaboration Patterns Characteristic Description Index The list of patterns is enumerated to ease later retrieval Name Each pattern has a unique name (1-3 words) Short description A sentence or two giving the essence of the pattern Related patterns Patterns above and below in the organisational hierarchy of patterns Example Shows an instance of the pattern (as in the previous section) Condition(s) In which situation(s) does the pattern apply Mediating artefact(s) What tool(s) are used (if any) when executing the pattern References Where has this pattern been published or reported (if at all) As mentioned above, this is a tentative structure that will continue to evolve as we carry on our analyses of the collected data. How we plan to use collaboration patterns in further work In this section we have described our preliminary analyses through which we have identified and structured four collaboration patterns. We consider this a first step towards a new and promising avenue for further work. Some steps we plan to take in the future include: • Document and organise the collaboration patterns other researchers have reported51 • Find out what computational mechanisms in groupware tools are supportive of / hindrance to collaboration patterns. This is the topic for a DoCTA Masters thesis. Asking “How do the various collaboration patterns put different demands on the design and use of groupware?” the focus of Pedersen’s research (Pedersen, in preparation) is to analyse how the collaboration patterns identified above are (or may be) supported by groupware applications. He will furthermore investigate what communication modes should exist for collaborative work to be effective. 51 Although they may not be using the term “collaboration pattern”. 238 Chapter 8 Results and conclusions • Find out how electronic (intelligent) agents can identify incomplete collaboration patterns and then advise the participants in the activity of this discovery so that it can be completed (assuming the agent is right) or discussed (assuming the agent is wrong). This is also the topic of Silje Jondal’s Masters thesis. • Find out whether collaboration patterns really are useful. This may include interviewing “collaboration experts” (if they exist) and asking them to interpret and analyse our patterns to see whether or not they make any sense. However, it is most likely that collaboration patterns emerge because of particular characteristics of the actors, their social environment and the pedagogical and technological constraints so we will need to carefully analyse our empirical data and try to find out when which collaboration patterns worked for whom. This may enable us to suggest “efficient” collaboration patterns for a given situation. • What are the implications of these collaboration patterns on pedagogical, technological and organisational design? • How can collaboration patterns inform our training of students in collaboration? When we began discussing collaboration patterns among ourselves, several questions arose. These questions still do not have answers (some thoughts around the first one were given above). Two examples are: How are collaboration patterns influenced by the student’s social situation (whether they have family, work full time, etc.)? How can we to distinguish between modes of interaction such as student/student, student/facilitator, and student-tool (HCI)? In the first instance, it is most likely that external situations will put strains on collaboration patterns, for example, causing some parts of a pattern to remain uncompleted because the student has to work from 9-5. This may create the need for modified (relaxed) patterns, credit for partial fulfilment of a pattern, etc. In the section instance, the patterns we have found are primarily for student-student interaction. Student-tool interaction is included in the mediating artefact characteristic of the collaboration pattern structure. This characteristic may or may not be present. A more detailed look at the data will enable us to identify which tools were involved. In addition, a closer look at student-facilitator interactions is underway. 239 Project DoCTA: Design and use of Collaborative Telelearning Artefacts This preliminary work on collaboration patterns is, however, already influencing the analyses of one of the Masters theses. In his preliminary analysis, see section 7.13, Andreassen (in preparation) dealt with aspects like co-ordination, communication mode, division of labour, and feedback and identified a few aspects of work organisation that occurred in VisArt, as above. The emergence of an explicitly formulated common goal seemed to reduce the need for same-time meetings, leading to a shift in communication mode, from largely synchronous to asynchronous. This shift was accomplished also through a process identified as coordinated desynchronisation. In addition, a more asynchronous form of work organisation may be ascribed to the division of labour. This, and coordinated desynchronisation, may, but need not, lead to a co-operative rather than a collaborative form of work (cf. Dillenbourg,, Baker, Blaye & O’Malley, 1996. Each team member having his/her own area of responsibility, without the final outcome being dependent on the results of the work of the other team members, may be unfortunate in that it reduces both the need and will to provide feedback. Feedback, leading to reflection on the work and learning of both oneself and others, may be seen as an integral part of a concept of collaborative learning, and settings promoting collaborative work organisation should therefore be provided. In the initial phases of a collaboration effort like this, a sort of double communication, when more than one tool is used to inform other team members about a, for example, a changed meeting time, might occur. This sort of communication may be reduced or disappear with improved technical understanding or changed work co-ordination over time, but might be avoided with sufficient training and examples on how different tools can be used for co-ordination purposes. Finally, this work in DoCTA on identifying collaboration patterns is in its infancy. We feel, however, that it is promising and that our continued scrutiny of the collected data will most likely reveal additional patterns. 8.2 Studying students’ online learning A major challenge for today’s researchers studying telelearning (or online) learning environments is how to design their studies. A simple question of what and how to collect and analyse data becomes a major obstacle. 240 Chapter 8 Results and conclusions Even “traditional” ethnographic studies that collect qualitative data do not readily suit these distributed ICT environments. As Steve Jones explains in the Preface to the collection on Doing Internet Research: Critical Issues and Methods “It [the book] is a result of discussion with many scholars from a wide variety of disciplines who believe, as do I, that simply applying existing theories and methods to the study of Internet-related phenomena is not a satisfactory way to build our knowledge of the Internet as a social medium. Consequently, this is not a book that will (at least not in any direct way) help people to use the Internet as a research tool. Rather, its goal is to assist in the search for, and critique of, methods with which we can study the Internet and the social, political, economic, artistic, and communicative phenomena occurring within, through, and in some cases, apart from but nevertheless related to, the Internet. (Jones, 1999, p. x).” Methodological issues have been a central discussion point in project DoCTA as we struggle to come to grips with what and how to evaluate our scenarios. As the reader has seen (if one managed to read all the studies presented in Part II J) we have carried out a wide variety of evaluations. These evaluations complement one another and have provided us with a multifaceted picture of the telelearning environments. Interpretation of this picture has only just begun! As the “field” of studying online learning environments is in its infancy, there are no “off the shelf” methods and techniques to apply. We have tried to be keenly aware of the limitations of our studies and the challenges of adapting methods to these types of studies. This section reflects on some of our data collection techniques and shares our experiences. Methodological issues During this project a major challenge has been to find and adopt a strategy for examining students involved in collaborative telelearning. In this project the strategy applied has included of a number of different techniques and methods, ranging from online questionnaires, keeping a record of and receiving copies of the students email, using the server logs generated from the software used, and online observations and interviews. 241 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Online observations This technique consisted of immersing oneself in the virtual environment and following the collaboration in the actual medium. This means that the researchers regularly followed the students’ activities in the virtual environment and in this way tracked changes made in this environment. Observing them when they were collaborating synchronously, and attempting to follow the different paths of communication and the collaboration process, was also an important aspect. There were some considerations that the researchers had to make in relation to these observations. When the students were logged on they immediately became aware of the researchers’ presence in the environment, and often asked them questions regarding the organisation of the scenario, problems they were facing or to clarify certain aspects of the task they were given. Consequently, the researchers that were present, in some cases, also took on the role of facilitator. A possible reason for this might be attributed to the design of the actual groupware system that was used (TeamWave Workplace). It is designed to support a workspace awareness (Gutwin et. al., 1995) so that it is easy to know who else is logged on and what they are actually doing and “where” they are located within the virtual environment. In relation to these considerations the observations should, at large, be classified as a certain form of participant observations, emphasising that the researchers had a special role in the scenarios and that the interventions by the researchers most often were initiated by specific questions from the students. Online interviews Some of the researchers have carried out online interviews both using the groupware system and email. The interviews that took place in TW were rather unstructured and were done synchronously, while the email interviews were asynchronous and fairly structured. Both methods produced interesting data. Collecting email and chats A technique that also has been used is collection of emails and chat-logs. The email was mainly gathered by setting up common team addresses (one for each team) that included a researcher in the list. Another method consisted of simply asking the students to save and forward the chat-logs 242 Chapter 8 Results and conclusions (as these were not saved automatically) and email. As the different researchers were collecting these kinds of data, they found it much easier to arrange the collection of the data with an informant (i.e., one of the participating students) they could meet face-to-face. An important lesson here is that the researcher has to be rather flexible when s/he is looking for possible sources of information and be actively in contact with the informants in order to gather the necessary empirical data and get a rich picture of the online activities. Server logs The communication that was mediated through the groupware system (TW) automatically leaves electronic traces in the server logs. These traces can be exploited when doing an analysis of the collaborations in this virtual environment. The data being logged, is not just statistical data recording “who is logged on when”, but includes a periodic chronological recording of all artefacts in the environment. This means that we can recreate versions of the environment to study the use of artefacts over time and the creation and development of the artefacts produced in the collaboration process (Guribye & Wasson, 1999). Overall this is a very valuable source of information when doing online research. The details of how this information was dealt with are treated in section 7.14 of this report and in Meistad & Wasson (1999). Online and offline activities As all of the techniques mentioned above are important when studying students’ online learning, one should not assume that the students’ online and offline experiences can be strictly divorced (Kendall, 1999). In fact Jones (1999) argues that “to study it [on line] as if it were somehow apart from the “off-line” world that brought it into being would be a gross mistake…on-line experience is at all times tethered in some fashion to off-line experience. (p. xii)”. The implications of this duality is that we need to have an understanding of how their online experience fits into their offline life. In DoCTA we strive to maintain this view, but it is challenging. It is important, however, to address issues related to both online and offline activities and how these are interrelated and mutually influence each other in order to gain an understanding of the collaboration process. 243 Project DoCTA: Design and use of Collaborative Telelearning Artefacts One implication of this is that the data collection should not, if at all possible, be limited to the sorts of techniques described above, but one should try to seek other sources of information as well. These might include techniques such as face-to-face interviews, telephone interviews, and offline observations (where the actual students are physically situated). Data collection in DoCTA included all of these techniques. 244 Chapter 9 Summary and Conclusions 9 Summary and Conclusions This section concludes the report. Section 9.1 gives a summary of the issues we have addressed in the first phase of DoCTA. The report ends with a reflection on how we have meet our objectives in section 9.2 9.1 Summary DoCTA is a large project with many issues being addressed. This section attempt to summarise some of the issues we have explored. The theoretical foundations presented in section 2 have influenced the design and evaluations of the IDEELS and VisArt scenarios. In particular, Salomon’s (1993) work on genuine interdependence and Gutwin et al.’s (1995) work on awareness affected the design of the collaboration assignments for both IDEELS and VisArt. Furthermore, both of these perspectives influenced the evaluation of TeamWave Workplace presented in section 5. Coodination theory serves as the foundation for a number of our studies on how students coordinate their work in these online learning environments. Section 3 presented the conceptual framework that has influenced our evaluations. In particular Activity Theory has played a role in several studies. Activity Theory provides a lens through which we can look at the world. It is useful to orient thoughts and research questions. It provides a number of methods/tools shaped by a general theoretical approach. It can be used to guide methodological decisions regarding evaluation (how technology is used) and to a lesser extent regarding design (how technology will be used). Engeström’s (1987) triangle (i.e., activity system), employed in several analyses (see sections 6.1, 7.13 & 7.14) provides concepts and ideas that 245 Project DoCTA: Design and use of Collaborative Telelearning Artefacts make it easier to capture and analyse the context of the activity. In particular it is useful to draw focus towards the division of labour, use of artefacts, sociocultural rules, community, the object of the activity, and the goals and motives of the subject. By first understanding the activity system, one is better able to focus an evaluation on particular aspects of the activity. Guribye, in describing his evaluation study of IDEELS (Guribye, 1999; Guribye & Wasson, submitted), found Leontiev’s (1978) distinction of the hierarchical structure of an activity useful in revealing some problems that occurred during the collaboration process (see section 6.1). Finally, Guribye argues that in order to gain a thorough understanding of the role of technological artefacts (including ICT), it is not sufficient to see only the technology or a tool in isolation. Rather, these artefacts must be seen as an integral part of human activity and human activity is always a complex and dynamic process that needs to be studied in situ. This can be done in a structured and informed way by using AT as an analytical framework The evaluations of the groupware systems presented in sections 4 and 5 can serve as a starting point for those interested in choosing a collaborative technology for use in a learning environment. Higraff’s evaluation of 4 groupware systems (see section 4) concluded with a preliminary set of suggestions for what groupware systems should contain. The thorough review of TeamWave Workplace in section 5 is complemented with the usability study of TeamWave Workplace presented in section 7.11 and the study on coordination mechanisms presented in section 7.16. Section 6 presented the IDEELS scenario and described two evaluation studies. Guribye’s study, section 6.1, focused on intra-group collaboration in a team of students participating in the IDEELS simulation game. His study, in many ways, has influenced the design of the evaluation studies of the VisArt scenario. Ekker (section 6.2) was interested in changes in attitudes towards simulation-based distributed learning. He was able to draw two conclusions. First, the software utilised (OPUSi) may be improved in order to alleviate some of the frustration experienced. Second, in preparing for future simulations it will be important to focus on integrating all members of the team in order to avoid having one or two participants “taking control”. VisArt is presented in detail in section 7. All phases from design and deployment to evaluation have been documented. Our description of the design process should be of interest to those preparing to design their 246 Chapter 9 Summary and Conclusions first collaborative learning scenario. The numerous evaluation studies of VisArt complement one another and it will be a challenge for us to integrate the results when the individual anlyses are completed. A short reflection on some of the preliminary observations, however, can be made. For example, Wake’s evaluation of how the VisArt instructors and faciliators organised their work (see section 7.14) draws out aspects of deployment to be considered. Regarding practical experiences, the interviewed instructors/facilitators all focused on the amount of time used in the planning and running of a telelearning activity. They felt that the resources needed were quite extensive. Unforeseen technical problems can occur, and there is high demand of feedback made by the students via e-mail and other electronic communication tools. A major motivation for many students engaging in telelearning activities is that it is flexible, that is, they can work at hours that suit them best. This puts pressure on the instructors as they want to provide feedback as fast as possible, but they can not be accessible around the clock. Emerging from these time demands is a need for several assistants — for both technical and content aspects. One desirable quality of groupware tools used in collaborative telelearning activities, is tied to the extent the instructors can get feedback on the student’s collaboration process in the activity. Wake’s study found that the facilitators felt that their ability to follow, in TW, the students’ progress in the learning activity was difficult and thus their ability to give feedback was limited. In ordinary classroom activity, the instructor can follow the activity level of the students relatively easily. In an online environment where students and instructors are not necessarily in the environment at the same time (or in different places in the environment), one is dependent on students’ self-reports, requests for assistance, and inspection of the used and produced artefacts. This results in the instructor having problems in, for example, giving adequate support to individual students, or teams, that are struggling. Although little analysis has been carried out in Underhaug’s (see section 7.17) evaluation of the training and assistance in VisArt, the tendency so far seems to be that most students were rather pleased with the training phase. Some comments on the post-questionnaire also indicate that it could have been done better. Several of the comments point to the length of the training phase. As Underhaug points out this could be attributed to insufficient information about the upcoming scenario that caused some students to be delayed in starting. Or, it could be attributed to the minor difficulties some 247 Project DoCTA: Design and use of Collaborative Telelearning Artefacts had with getting attached to the server. Further analysis will try to elucidate the dissatisfaction with the training and these results will be used to suggest improvements that can be made in future training situations. As we have discussed in section 8.2, studies of online learning environments are challenging. We have tried to meet this challenge by focusing studies from several perspectives. Even a more “traditional52 ” usability study by Rysjedal (see section 7.11) focused on studying the technology in use thus was challenged by how to collect the data to best answer questions about efficiency, effectiveness and satisfaction. His preliminary conclusions tend toward a positive picture of TeamWave Workplace from a usability perspective. One could wonder why a research project focused on sociocultural studies of collaborative learning would include a usability study of technology in its repertoire. We would argue that if the chosen collaborative technology hinders a user’s work, then creating all the right social and cultural conditions would not be enough to avoid frustration and annoyance on the part of the user. This became even clearer in Guribye’s study of IDEELS where the OPUSi system was so “unusable” that it clearly hindered their work practices and created frustration. Guribye’s findings are also supported by Ekker’s conclusions (see above). In his investigation of how personality factors effected the attitudes and behaviour of students participating in VisArt, Morgan (see section 7.12) found statistically significant differences between the major personality facts in terms of the use of the learning environment and attitudes towards the various components of TW. Such findings suggest that future development of collaborative telelearning environments might find it profitable to directly link personality type tests directly to real time online changes within the collaborative learning environment to more closely match students preferred interaction styles and preferred learning tools. Combining the results of Morgan’ study with Guribye’s and Ekker’s might enable us to make some interesting observations about how individual student’s will experience online learning. This, however, is presently speculation, and an integrated analysis would be necessary before any conclusions could be drawn. In his research on how to use electronic data logs to support collaborative telelearning research, Meistad (see section 7.15) has developed two tools 52 “Traditional” is used to indicate that there is a tradition of using usability studies to determine whether or not a software system enables a user to carry out their tasks. Traditionally these were laboratory-based studies. 248 Chapter 9 Summary and Conclusions that present unwieldy log data in a format that is useful for determine collaboration opportunities and classifying tool use. Using the two tools in conjunction with other data collection methods will enable researchers to develop a richer picture of collaboration patterns. Meistad’s tools have been used by some of the other DoCTA researchers to use the data logs to complement their own data. For example, Baggetun (see section 7.16) found that using these tools made it possible to see each team member’s movements inside TW (e.g., from room to room), to see how long each team member was logged on, and to see who was logged on at the same time in a room. This gave an indication of the synchronous/asynchronous nature of the collaboration within a team. Finally, we leave behind an archive of artefacts for others to use. As summarised in table 2, these include: a review of collaborative technologies; a set of qualitative and quantitative instruments (such as questionnaires and interview guides); reflections on the use of Activity Theory; examples of training assignments; examples of a design activity and how it has been integrated into our courses; documented experience in the use of TeamWave workplace in an actual collaborative telelearning activity including how we designed the rooms to support the activity and examples of the resultant designed learning rooms; a set of TeamWave Workplace help pages on the web; reflections on administration of TeamWave workplace; and reflections on the deployment of the VisArt scenario. We have published and presented VisArt in several forums and will continue to do so as the preliminary analyses are expanded and more concrete conclusions can be made. The DoCTA web site is currently being expanded to include a demonstration of the VisArt scenario and will periodically be updated as our activities continue. 9.2 Conclusions The report began with a presentation of the objectives of DoCTA and the research questions to be answered. The main, long term, research objectives are to: • take a sociocultural perspective on learning activity focussing on the interpersonal social interaction in a collaborative learning setting (collaboration = coordination + communication) 249 Project DoCTA: Design and use of Collaborative Telelearning Artefacts • contribute to collaborative telelearning knowledge about the pedagogical design of learning scenarios, the technological design of the learning environment to support these learning scenarios, and the organisational design for management of such learning environments, including a reflection on teacher and learner roles • study and evaluate the social and cultural aspects of distributed collaborative telelearning environments In DoCTA, we have maintained a focus on learning activity and have attempted to integrate this into both the designs and evaluations we have undertaken. We have focused on both coordination and communication issues when designing our training and in designing our evaluation studies. The majority of the evaluation studies we have carried out have roots in a sociocultural perspective. The design and deployment of IDEELS and VisArt has not only contributed to our understanding of the pedagogical, technological and organisational aspects of collaborative telelearning environments, but has had an impact on those who have participated as students. These teachers in training have been given first hand experience in using a collaborative technology and in coordinating and communicating with a team to collaboratively design a learning artefact. A spin-off result has been that several of the students have expressed an interest in using the designed learning activity and the technology (i.e., TeamWave Workplace) in their own teaching. The numerous evaluation studies that we have carried out not only contribute to our understanding of the social and cultural aspects of collaborative telelearning environments, but equally important, they have also addressed methodological issues related to studying online environments. As pointed out in section 8.2, the challenges around studying and understanding the social processes occurring within, and around (Jones, 1999), such environments are a contemporary research challenge. Two research questions have stood central as we strive to make sense of these collaborative telelearning environments: How do students, teachers and facilitators organise their work? What collaboration patterns emerge? 250 Chapter 9 Summary and Conclusions Although we are in the early stages of understanding how the participants in our scenarios organised their work, we feel we are beginning to make progress. One complete study, Guribye (1999), has documented how one group of students organised their work in the IDEELS scenario. Results of this study have informed the studies of Andreassen (in preparation) and Wake (in preparation) who take a similar approach to studying students work organisation and instructor/facilitator work organisation in VisArt, respectively. Collaboration patterns are beginning to emerge from these and the other evaluation studies. Although we are unfortunately not as far along in our integrative analysis as we would have liked to be, we feel that the progress we have made thus far is promising. Future directions for work on collaboration patterns have been enumerated in section 8.1. A final aspect of the DoCTA research project that warrants mention is the fostering of enthusiasm for ICT and learning research both at the Department of Information Science and in the research milieu in Bergen in general. We have had little problem in attracting graduate students to the project and are proud to have initiated, in at least three of the Masters students, an interest in pursuing doctorate studies in this area. The opportunity afforded us to work on such an encompassing project has been well worth the efforts. Finally, we conclude by saying that the research ITU funding has enabled us to engage in has been both exciting — and challenging! 251 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 252 References References Adrianson, L. Hjelmquist, E. (1991) Group processes in face to face and computer mediated communication. Behaviour and Information Technology, 10(4), 281-296. Alexander, C.A., Ishikawa, M. and Silverstein, S. (1977). A Pattern Language: Towns, Buildings, Construction. Oxford Univerity Press. New York, Andreassen, E. F. (in preparation). Evaluating how students organise their work in a collaborative telelearning scenario: An Activity Theoretical Perspective. Masters dissertation, Department of Information Science, University of Bergen, Norway Andriessen & Sandberg, (1999). Where is education heading and how about AI? International Journal of Artificial Intelligence in Education, 10, 130-150. Badre, A. N., Guzdial, M., Hudson, S. E. and Santos, P. J. (1995). A user interface evaluation environment using synchronized video, visualizations and event trace data. Software Quality Journal 4, pp 101-113. Baggetun, R. (in preparation). Coordination Work in Collaborative Telelearning. Masters dissertation, Department of Information Science, University of Bergen, Norway Barbuto, J.E. Jr. (1997). A critique of the Myers Briggs Type Indicator and its operationalization of Carl Jung’s psychological types. Psychological Reports, 80(2), 611-625. Bellotti, V. and Bly, S., (1996). Walking away from the desktop computer: Distributed Collaboration and Mobility in a Product Design Team. Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW’96), 209-218. 253 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Bentley, R., Appelt, W., Busbach, U., Hinrichs, E., Kerr, D., Sikkel, K., Trevor, J. & Woetzel, G. (1997). Basic Support for Cooperative Work on the World Wide Web, International Journal of Human Computer Studies: Special issue. Bentley, R., Horstmann, T., & Trevor, J. (1997). The World Wide Web as Enabling Technology for CSCW: The Case of BSCW, Computer Supported Cooperative Work (CSCW), 6 (2/3), 111-134. Amsterdam: Kluwer Academic Publishers. Bourdeau, J. & Bates, A (1997). Instructional Design for Distance Learning. In R. Tennyson. S. Dijsktra, N. Seel, and F. Schott (Eds.) Instructional Design: International Perspectives, 369-397. Hillsdale, NJ: LEA Publishers. Bourdeau, J. & Wasson, B. (1997). Orchestrating collaboration in collaborative telelearning. In Proceedings of the 8th World Conference on Artificial Intelligence in Education, pages 565567. Amsterdam: IOS Press. Bredo, E. (1994). Reconstructing Educational Psychology: Situated Cognition and Deweyian Pragmatism. In Educational Psychologist, 29(1), 23-35. Brown, Virginia L.; DeCoster, David A. (1991). The Myers Briggs Type Indicator as a developmental measure: Implications for student learners in higher education. Journal of College Student Development, 32(4): 378 379 Carroll, J.M., Kellogg, W.A., & Rosson, M. B. (1991). The taskartefact cycle. In J. M. Carroll (Ed.) Designing Interaction: Psychology at the Human-Computer Interface, 74-102. Cambridge, UK: Cambridge University Press. Carstensen, P.H. & Sørensen, C. (1996). From the social to the systematic? An analysis of mechanisms supporting coordination work in design. CSCW Journal, 5(4), 387-413. Cattell, R. (1936). A guide to mental testing. London. Chabert, A. (1997). Habanero and the Vrteach Educators, http:// havefun.ncsa.uiuc.edu/habanero/Whitepapers/vrteacher.html 254 References Chabert, A., Grossman, Ed., Jackson, L. & Pietrovicz, S. (1996). Synchronous collaborative framework and environment, http:// www.ncsa.uiuc.edu/SDG/Software/Habanero/Whitepapers/ ecscw-habanero.html Chabert, A., Grossman, E., Jackson, L., Pietrovicz, S. & Seguin, C. (1998). Java Object-Sharing in Habanero. Communications of the ACM, 6, 69-76. Chan, T-W. & Baskin, A.B. (1990). Learning Companion Systems. In C. Frasson, G. Gauthier, (eds.), Intelligent Tutoring Systems: at the crossroad of Artificial Intelligence and Education, 6-33. Ablex, Norwood: NJ.. Cole, M. & Engeström, Y. (1993). A Cultural-Historical Approach to Distributed Cognition. In G. Salomon, (Ed.) Distributed Cognitions, Psychological and Educational Considerations. 1-46. Cambridge, Mass.: Cambridge University Press. Cole, M. & Scribner, S. (1978). Introduction. In Cole, M., JohnSteiner, V., Scribner, S., & Souberman, E. (Eds.). L. S. Vygotsky, Mind in Society: The Development of Higher Psychological Processes,1-16. Cambridge, MA: Harvard University Press. Daniels, H. (1996). An Introduction to Vygotsky. London: Routledge. Davydov, V. (1988). Learning activity: The main problems needing further research. Activity Theory, 1(1-2), 29-36. Decortis, F., Noirfalise, S. & Saudelli, B. (1996). Activity Theory as framework for cooperative work. COTCOS – T. M. R programme, WP1. Dembo, MH, McAuliffe, T.J. (1987) .Effects of Perceived Ability and Grade Status on Social Interaction and Influence in Cooperative Groups. Journal of Educational Psychology,79(4), 415-423. Dillenbourg, P., Self, J.A. (1992). People Power: A Human – Computer Collaborative Learning- System. Lecture Notes in Computer Science, 608, 651-660. Doise, W. (1990). The development of individual competencies through social interaction. In H.C. Foot, M.J. Morgan & R.H. Shute (Eds.) Children helping children, 43-64. Chichester: J. Wiley and Sons. 255 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Dourish, P., and Bellotti, V.(1992), Awareness and Coordination in Shared Workspaces, Proceedings of the Conference on Computer-Supported Cooperative Work, 107-114. Eidsmo, A., Munkvold G. & Ekker, K. (1999). (Nord-Trondelag College, Norway): Experiences with TeamWave and OPUSi software as pedagogical tools in CSCL (computer supported cooperative learning). http://www.unice.fr/RI/Journees-TIC/ (Démonstrations) Eison, J.A., Pollio, H.R. (1985). A multidimensional approach to the definition of college students’ learning styles. Journal of College Student Personnel. 26(5), 434-443. Engeström, Y (1987). Learning By Expanding: An activitytheoretical approach to developmental research. Helsinki: Orienta-Konsultit Oy Engeström, Y. (1991). Activity theory and individual and social transformation. Activity Theory 7(8), 6-17. Engeström, Y., Miettinen, R. & Punamäki, R-L. (1999). Perspectives on Activity Theory. Cambridge, UK: Cambridge University Press. Fitzpatrick, G., Tolone, W.J., Kaplan, S.M. (1995). Work, locales and distributed social worlds. In H. Marmolin, Y. Sundblad & . Schmidt (Eds.) Proceedings of the Third European Conference on Computer-Supported Cooperative Work (ECSCW’95), 1-16. Dordrecht: Kluwer Academic Publishers. Fjeld, M., Schluep, S., Rauterberg, M. (1998). Automatic, action driven classification of user problem solving strategies by statistical and analytical methods: a comparative study. In F. E. Ritter and R. M. Young (eds.) Proceedings of the Second European Conference on Cognitive Modelling, 98-103. Nottingham: Nottingham University Press. Fjuk, A. (1998). Computer Support for Distributed Collaborative Learning. Exploring a Complex Problem Area. Dr. Scient. Thesis. University of Oslo, Department of Informatics. Fjuk, A., Sorensen, E. & Wasson, B. (1999). ICT-mediated learning and competence development in companies. NIN pre-project final report. Telenor FOU, Kjeller, Norway. 256 References Fjuk, A & Øgrim, L. (1997). Towards transcendent thinking in design of distributed collaborative learning. Accepted for publication in European Journal of Open and Distance Learning. Gamma, E.R., Helm, R., Johnson, R. & Vlissides, L. (1995). Design Patterns: Elements of Re-usable Object-Oriented Software. Reading MA: Addison-Wesley. Garden, A.M. (1991). Unresolved issues with the Myers Briggs Type Indicator. Journal of Psychological Type, 22, 3-14. Garton, L., Haythorntwaite, C. & Wellman, B. (1999). Studying online social networks. In S. Jones (Ed.) Doing Internet research: Critical issues and methods for examining the net, 75-106. London: Sage Publicaions, Inc. Gibson, J. (1986). The ecological approach to visual perception. Hillsdale NJ: Lawrence Erlbaum Associates. GMD (1999). What is BSCW? http://bscw.gmd.de/index.html Guribye, F. (1999). Evaluating a collaborative telelearning scenario: A sociocultural perspective. Masters dissertation, Department of Information Science, University of Bergen, Norway. Guribye, F. & Wasson B. (1999). Evaluating Collaborative Telelearning Scenarios: A Sociocultural Perspective. In B. Collis & R. Oliver (Eds.) Proceedings of Educational Multimedia & Educational Telecom ’99, 1264-1265. Charlottesville, VA: AACE. (ISBN : 1880094-35-5) (Outstanding Paper Award). http://www.uib.no/ People/stud1068/documents/edmedia.html Guribye, F. & Wasson, B. (submitted). Using Activity Theory to Understand Work Organisation in a Collaborative Telelearning Scenario. Submitted to the International Journal on Learning Environments Research, Kluwer. Gutwin, C., Stark, G. and Greenberg, S. (1995). Support for Workspace Awareness in Educational Groupware. in Proceedings of the ACM Conference on Computer Supported Collaborative Learning 147-156. Hillsdale NJ: Lawrence Erlbaum Associates. Guzdial, M. (1993). Deriving software usage patterns from log files. Technical Report #GIT-GVU-93-41. Graphics, Visualization, and Usability Center, College of Computing, Georgia Insitute of Technology. 257 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Habanero (1998). Habanero tools. http://www.ncsa.uiuc.edu/SDG/ Software/Habanero/Tools/ Holt, A. (1985).. Coordination technology and petri nets. In G. Rozenberg (Ed.) Advances in Petri Nets 1985, 222, 278-296. Holt, A.W. (1998). DIPLANS – A new Language for the Study and Implementation of Coordination. ACM Transactions in Information Systems. 6(2), 109-125. Holzman, L. H. (1996). Pragmatism and Dialectical Materialism in Language Development. In H. Daniels (Ed.). An Introduction to Vygotsky. London: Routledge. Hutchins, E. (1991). The social organisation of distributed cognition. In L. Resnick, J. Levine, & S. Teasley (Eds.). Perspectives on Socially Shared Cognition, 238-287. Washington, DC: American Psychological Association Press. Hutchins, E. (1995). Cognition in the Wild. Cambridge, MA: MIT Press. Jackson, Stacy L.; Parker, Christopher P.; Dipboye, Robert L. (1996). A comparison of competing models underlying responses to the Myers Briggs Type Indicator. Journal of Career Assessment. 4(1), 99-115. Johansen, R., Sibbet, D., Benson, S., Martin, A., Mittman, R. & Saffo, P. (1991). Leading Business Teams. New York: Addison-Wesley. JohnSteiner, V, Mahn, H. (1996). Sociocultural approaches to learning and development: A Vygotskian framework Educational Psychologist, 31( 3-4), 191-206. Jonassen, D. & Rohrer-Murphy, L. (1999). Activity Theory as a Framework for Designing Constructivist Learning Environments. Educational Technology (47). Jones, S.G. (Ed.) (1999). Doing Internet Research: Critical Issues and Methods for Examining the Net. London: SAGE Publications Ltd. Jung, C. (1953 –1971) Collected works. London. Junge, A. (1999). Hva er The Demeter Project ? Masters dissertation, Department of Information Science, University of Bergen, Norway 258 References Kaplan, S.M., Tolone, W.J., Bogia, D.P. & Bignoli, C. (1992). Flexible, active support for collaborative work with conversation builder. In J. Turner & R. Kraut (Eds.) Proceedings of the Conference on Computer-Supported Cooperative Work (CSCW’92), 378-385. New York: ACM Press. Kaptelinin, V. & Nardi B. A. (1997). The Activity checklist: A tool for representing the “space” of context. Working paper. Department of Informatics, Umeå University. Kendall, L. (1999) Recontextualizing “Cyberspace”: Methodological considerations for on-line research, 57-74. In S.G. Jones (Ed.) Doing Internet Research: Critical Issues and Methods for Examining the Net. London: SAGE Publications Ltd. Kerr, N.L. (1983). Motivation losses in small groups: A social dilemma analysis. Journal of Personality and Social Psychology, 45, 819828. Kerr, N.L. & Bruun, S.E. (1983). Dispensability of member effort and group motivation losses: Free rider effects. Journal of Personality and Social Psychology, 44, 78-94. Koschmann, T. (1996). Paradigm shifts and instructional technology: An introduction. In T. Koschmann (Ed.) CSCL: Theory and Practice of an Emerging Paradigm, 1-23. Mahwah, NJ: Lawrence Erlbaum Associates. Kuutti, K. (1996). Activity theory as a Potential framework for humancomputer interaction research. In B.A. Nardi (Ed.) Context and Consciousness: Activity Theory and Human-computer Interaction, 17-44. Cambridge, MA: MIT Press. Laurillard, D. (1987). Computers and the Emancipation of Students – Giving Control to the Learner. Instructional Science, 16(1), 3-18. Lave, J. (1988) Cognition in Practice. Cambridge University Press. Lave, J. & Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation. NY: Cambridge University Press. Lave, J. & Wenger, E. (1996). Practice, Person, Social World. In H. Daniels (Ed.). An Introduction to Vygotsky. London: Routledge. 259 Project DoCTA: Design and use of Collaborative Telelearning Artefacts LDC (1998a). White Paper LearningSpace, Solutions for Anytime Learning”, Lotus Development Corporation, 55 Cambridge Parkway, Cambridge, MA 02142 LDC (1998b). Fact Sheet. Lotus LearningSpace Release 2.5 Distributed Learning Solution. Lotus Development Corporation, 55 Cambridge Parkway, Cambridge, MA 02142 Leont’ev, A. N. (1978). Activity, Consciousness, Personality. Englewood Cliffs, NJ: Prentice Hall. Lorr, M. (1991). An empirical evaluation of the MBTI typology. Personality and Individual Differences 12(11), 1141-1145. Malone, T. & Crowston, K. (1994). The Interdisciplinary study of coordination. ACM Computing Surveys, 26(1), 87-119. Mantovani, G. (1996). New Communication Environments: From Everyday to Virtual. London: Taylor & Francis Marton, F. & Booth, S. (1997). Learning and awareness. Hillsdale, N.J.: Lawrence Erlbaum Associates. Mead, G. H. (1968). Mind, Self, and Society. Chicago: Chicago University Press. Meistad, Ø. (in preparation). Collaborative telelearning: Using log-files to identify collaboration patterns. Masters dissertation, Department of Information Science, University of Bergen. Meistad, Ø. & Wasson, B. (submitted). Supporting collaborative telelearning research using server logs. Submitted to of Educational Multimedia & Educational Telecom 2000. Minick, N. J. (1996). The Development of Vygotsky’s Thought: An Introduction to Thinking and Speech. In H. Daniels (Ed.). An Introduction to Vygotsky. London: Routledge. Morgan, K., Macleod, H. (1992). Results from exploratory investigations into the possible role of personality factors in computer interface preference. Paper presented to the UK Joint Research Council’s Second Interdisciplinary Workshop on Mental Models, Robinson College, University of Cambridge, March 1992 260 References Morgan, K. & Morgan, M. (1999). Religion and the Internet: the cultural impact of information technology on culture and religious beliefs. Proceedings of the VI European Congress of Psychology, Rome, Italy July 4-9 1999. Morgan, K. & Morgan, M. (in preparation). The Role of Classical Jungian Personality Factors in Computer Supported Collaborative Learning Environments. Murray, J.B. (1990). Review of research on the Myers Briggs Type Indicator. Perceptual and Motor Skills, 70(3, Pt 2), 1187-1202. Mørch, A. (1995). Application Units: Basic Building Blocks of Tailorable Applications. Proceedings 5th International East-West conf. on HCI, Moscow, July 1995. Lecture Notes in Computer Science 1015, 68-87. Amsterdam: Springer Verlag. Nardi, B. A. (1996). Studying Context: A comparison of activity theory, situated action models and distributed cognition. In B.A. Nardi, (Ed.) Context and Consciousness: Activity Theory and Humancomputer Interaction, 69-102. Cambridge, MA: MIT Press. Newhagen, J. E., and Rafaeli, S. (1997). Why communication researchers should study the Internet: A dialogue [On-line]. Journal of Computer-Mediated Communication, 1(4). Available: http:// www.ascusc.org/jcmc/vol1/issue4/rafaeli.html. Norman, D.A. (1990). Human Error and the Design of Computer Systems – Commentary. Communications of the ACM, 33(1), 4. Okada, H., Asahi, T. (1999). GUITESTER: A log-based usability testing tool for Graphical User Interfaces. IEICE Transactions on information and systems. Volume E82. Olson, J., Card, S., Landauer, T., Olsen, G. Malone, T. & Legget, J. (1993). Computer-supported cooperative work: Research issues for the 90s. Behavior and Information Technology, 12 (2), 115-129. Palfreyman K. A. and Rodden T. (1996). A Protocol for User Awareness on the World Wide Web, In Proceedings of CSCW’96 Cooperating Communities,130 – 139. ACM Press. Paquette, G, Ricciardi-Rigault, C., Bourdeau, J., Paquin, C. & Liegeois, S. (1995). Modelling a Virtual Campus Environment for Interactive Distance Learning. Proceeding of EdMedia’95. 261 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Pea, R. (1993). Practices of distributed intelligence and designs for education. In G. Salomon (Ed) Distributed Cognitions: Psychological and Educational Considerations, 47-87. Cambridge, Mass: Cambridge University Press. Pea, R., Edelson, D., & Gomez, L. (1994). Distributed collaborative science learning using scientific visualization and wideband telecommunications. Paper presented at the symposium entitled Multimedia information systems for science and engineering education: Harnessing technologies at the 160th Meeting of the American Association for the Advancement of Science (AAAS), San Francisco. Pedersen, T. (in preparation). Technological artefacts and collaboration patterns: evaluating the use of groupware in a collaborative telelearning scenario. Masters dissertation, Department of Information Science, University of Bergen, Norway. Resnick, L., Levine, J. & Teasley, S. (1991, Eds.). Perspectives on socially shared cognition. Washington, DC: APA Press. Richek, H.G. (1969). Jung’s typology and psychological adjustment in prospective teachers: A preliminary investigation. Alberta Journal of Educational Research. 15(4), 235-243. Rogoff, B. (1990). Apprenticeship in Thinking: Cognitive Development in Social Contexts. New York: Oxford University Press. Rysjedal, K. (in preparation). Usability Evaluation of a Groupware Tool. Masters Dissertation. Department of Information Science, University of Bergen, Norway Jonassen, D.H. & RohrerMurphy, L. (1999). Activity theory as a framework for designing constructivist learning environments. ETR&D- Educational Technology Research and Development, 47(1), 61-79. Roseman, M. & Greenberg, S. (1996). TeamRooms: Networked Places for Collaboration. Proceedings of ACM CSCW’96 Conference on Computer Supported Cooperative Work, ACM Press. 262 References Salomon, G. (1992). What does the design of effective CSCL require and how do we study its effects ? SIGCUE Outlook, Special Issue on CSCL, 21(3), 62-68. Salomon, G. (Ed.) (1993). Distributed Cognitions, Psychological and Educational Considerations. Cambridge: Cambridge University Press. Salomon, G, Globerson, T. (1987). The Computer as a Zone of Proximal Development – Internalizing Reading Related Metacognitions from a Reading Partner. Journal of Educational Psychology. 81( 4), 620-627. Sardamalia, M, Bereiter, C, McLean, RS, Swallow, J, Woodruff, E, (1987). Computer Supported Intentional Learning Environments. Journal of Educational Computing Research, 5(1), 51-68. Schmidt, K (1997). Of maps and scripts. The status of formal constructs in cooperative work. Proceedings of the ACM Conference on Supporting Group Work (Group’97), 138-147. New York: ACM Press. Schmidt, K. (1994). Cooperative Work and its Articulation Requirements for Computer Support. Travail Humain, 57( 4), 345-366. Schmidt, K. & Simone, C. (1996). Coordination mechanisms: Towards a conceptual foundation of CSCW systems design. Computer Supported Cooperative Work: The Journal of Collaborative Computing, 5, 155-200. Scott, T. Cole, M. Engel, M. (1992). Computers and Education - A Cultural Constructivist Perspective. Review on Research in Education, 18, 1992, 191-251. Singleton, W. T. (1989). The mind at work: Psychological ergonomics. Cambridge, England UK: Cambridge University Press. Stetsenko, A. P. (1990). The Role of the Principle of Objectrelatedness in the Theory of Activity: Criticism from “Without” and from “Within”. In V.V. Davydov, & D.A. Leont’ev, (Eds.). The Activity Approach to Psychology: Problems and Perspectives. Moscow: USSR Academy of Pedagogical Sciences. 263 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Strauss, A. (1985). Work and the division of labor. The Sociological Quarterly, 26(1), 1-19. Suchman, L. (1987). Plans and situated actions. Cambridge University Press Suthers, D. & Weiner, A. (1995). Groupware for Developing Critical Discussion Skills. Proceeding of Computer Supported Collaborative Learning 1995, October 17-20, 1995, Indiana University, Bloomington, IN. Säljö, R. (1996). Mental and physical artifacts in cognitive practices. In P. Reimann & H. Spada (Eds). Learning in humans and machines. Towards an interdisciplinary learning science, 83-96. London: Pergamon. TeamWave Workplace (1999). TeamWave Workplace. http:// www.teamwave.com/advantages/overview.html Underhaug, H. (in preparation). Facilitating Help and Training in a Collaborative Telelearning Scenario. Masters dissertation, Department of Information Science, University of Bergen, Norway. Van der Veer, G.C., Tauber, M.J., Waern, Y., Van Muylwijk, B. (1985). On the interaction between system and user characteristics. Behaviour and Information 4(4), 289-308. Van Hoe, R., Poupeye, K., Vandierendonck, A., de Soete, G. (1990). Some effects of menu characteristics and user personality on performance with menu driven interfaces. Behaviour and Information Technology 9(1), 17-29. Van Muylwijk, B., Van der Veer, G., Waern, Y. (1983). On the implications of user variability in open systems: An overview of the little we know and of the lot we have to find out. Behaviour and Information Technology 2(4), 313-326. Vygotsky, L. S. (1978)., Mind in Society: The Development of Higher Psychological Processes M.Cole, V. John-Steiner, S. Scribner, & E. Souberman (Eds.). Cambridge, MA: Harvard University Press. 264 References Wake, J. (in preparation). Evaluating the Organisation of a Collaborative Telelearning Scenario; An Activity Theoretical Perspective. Masters dissertation, Department of Information Science, University of Bergen, Norway. Wasson, B. (1997). Advanced educational technologies: The learning environment. Computers in Human Behaviour, 13(4), 571-594. Wasson, B. (1998). Identifying Coordination Agents for Collaborative Telelearning. International Journal of Artificial Intelligence in Education, 9, 275-299. Wasson, B. (1999). Design and evaluation of a collaborative telelearning activity. In C. Hoadley (Ed) Proceedings of CSCL99 Designing new media for a new millennium: Collaborative Technology for Learning,Education, and Training, 659-666. ACM Press. Wasson, B. & Mørch, A. (1999). DoCTA: Design and Use of Collaborative Telelearning Artefacts. In B. Collis & R. Oliver (Eds.) Proceedings of Educational Multimedia & Educational Telecom ’99, 534-539. Charlottesville, VA: AACE. Wasson, B. & Mørch, A. (to appear). Identifying collaboration patterns in collaborative teleleraning scenario. IFETS Special Issue on Online Collaborative Learning Environments. Weil, M.M., Rosen, L.D., Wugalter, S.E. (1990). The etiology of computerphobia. Computers in Human Behavior 6(4), 361-379. Wertsch, J. V. (Ed.) (1981). The Concept of Activity in Soviet Psychology. Armonk, NY: M.E. Sharpe. Wertsch, J. V. (1998). Mind as Action. Cambridge: Cambridge University Press. Wertsch, J. V., del Río, P. & Alvarez, A. (1995). Sociocultural studies: history, action and mediation,. In J.V. Wertsch, P. del Río & A. Alvarez. (Eds.) Sociocultural Studies of Mind, 1-34. Cambridge: Cambridge University Press. Zinchenko, V. P. (1995). Cultural-historical psychology and the psychological theory of activity: retrospect and prospect. In J.V. Wertsch, P. del Río, & A. Alvarez. Sociocultural Studies of Mind, 37-55. Cambridge University Press. 265 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Åberg-Bengtsson, L. (1998). Entering a Graphicate Society: Young Children Learning Graphs and Charts. Doctoral thesis. Gothenburg: University of Gothenburg. 266 Appendicies Appendicies Appendix A: DoCTA Publications Appendix B: DoCTA Presentations Appendix C: Assignment for test-scenario Appendix D: Interview Guide for Evaluation of Groupware Systems Appendix E: The IDEELS Scenario Appendix F: Assignment for designing an evaluation of IDEELS Appendix G: Interview Guide for IDEELS Appendix H: Pre-simulation survey for IDEELS Appendix I: Post-simulation survey for IDEELS Appendix J: Planning Tables from the Instructor’s View (Phases II-IV) Appendix K: Planning Tables from the Researcher’s View (Phases I-III) Appendix L: Planning Tables from the Student’s View (Phases I-III) Appendix M: VisArt Consent Form Appendix N: Semester Assignment Appendix O: Directions for TeamWave Workplace Installation Appendix P: Designed Learning Rooms from VisArt Appendix Q: VisArt Profile Questionnaire Appendix R: VisArt Self-Evaluation Questionnaire Appendix S: VisArt Pre-questionnaire Appendix T: VisArt Post-questionnaire Appendix U: The Narg Island Game Appendix V: Norweign Version of Training Assignments 1,3 & 4 Appendix W: Help Page Screen Shots Appendix X: Administration Interface in TeamWave Workplace Appendix Y: Usability Study Interview Guide Appendix Z: Interview Guide for How Students Organise their Work Appendix AA: Interview Guide for How Instructors and Facilitators Organise their Work 267 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix A: DoCTA Publications Guribye, F. (1999). Evaluating a collaborative telelearning scenario: A sociocultural perspective. Masters dissertation, Department of Information Science, University of Bergen, Norway. Available as EIST Research Report 4.(http://www.ifi.uib.no/docta/thesis/) Guribye, F. & Wasson, B. (1999). Evaluating collaborative telelearning scenarios: A sociocultural perspective. In B. Collis & R. Oliver (Eds.) Proceedings of Educational Multimedia & Educational Telecom ’99, 1264-1265. Charlottesville, VA: AACE. (http:// www.ifi.uib.no/staff/frode/publications/edmedia99.html) (Outstanding Paper Award) Meistad, Ø, (1999). Skolens bruk av kulturgjenstander med innebygd intelligens. Essay, ITU- Interactive Conference (ITU- Konferansen, 1999), Oslo, Norway, October 8th. (http://www.ifi.uib.no/docta/ resources/ITUessay99.html) Meistad, Ø. & Wasson, B. (to appear). Supporting collaborative telelearning research using server logs. To appear at the International Conference on Educational Multimedia & Educational Telecom 2000, Montreal, June. Mørch, A. (1999). Project DoCTA: Knowledge-Building in Collaborative Telelearning, Position paper at the Workshop on Collaborating on the Design and Assessment of KnowledgeBuilding Environments in the 2000’s, CSCL’99, Stanford CA, 12-15 December. (http://www.cs.colorado.edu/%7Egerry/ publications/conferences/1999/cscl99/kbe_workshop/ morch.html) Mørch, A. & Wasson, B. (1999). Dynamics of groupware use in a collaborative telelearning scenario. Position paper at Workshop on «Evolving use of groupware» at ECSCW’99, Copenhagen DK, 12 September, 1999. 268 Appendicies Wasson, B. (1999a). Design and evaluation of a collaborative telelearning activity. In A. Gulbrandsen (Ed.) UPED Report Series nr. X/99, Programme for Research on Learning and Instruction, University of Bergen. Wasson, B. (1999b). Design and evaluation of a collaborative telelearning activity. In C. Hoadley (Ed) Proceedings of CSCL’99 Designing new media for a new millennium: Collaborative Technology for Learning,Education, and Training, 659-666. ACM Press. Wasson, B. & Mørch, A. (1999). DoCTA: Design and Use of Collaborative Telelearning Artefacts. In B. Collis & R. Oliver (Eds.) Proceedings of Educational Multimedia & Educational Telecom ’99, 534-539. Charlottesville, VA: AACE. (http://www.ifi.uib.no/ staff/barbara/papers/edmedia99.html) Wasson, B. & Mørch, A. (to appear). Identifying collaboration patterns in collaborative teleleraning scenario. IFETS Special Issue on Online Collaborative Learning Environments. Submitted Paper Guribye, F. & Wasson, B. (submitted). Using Activity Theory to Understand Work Organisation in a Collaborative Telelearning Scenario. Submitted to the International Journal on Learning Environments Research, Kluwer. 269 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix B: DoCTA Presentations Guribye, F. (1999, December). Design and evaluation of a collaborative telelearning activity aimed at teacher training. Presented at CSCL’99. San Francisco. Mørch, A. (1999, December). Project DoCTA: Knowledge-Building in Collaborative Telelearning, Presented at the Workshop on Collaborating on the Design and Assessment of KnowledgeBuilding Environments in the 2000’s, CSCL’99 , Stanford CA. Munkvold, G., Eidsmo, A. & Ekker, K. (November, 1999). Experiences with TeamWave Workplace and OPUSi software as pedagogical tools for CSCL. Journées d’études internationales, Nice et Sophia Antipolis, France. Baggetun, R. & Guribye, F. (1999, November) The Use of TeamWave Workplace to Support Collaborative Learning in VisArt. Presented at Workshop on ICT- Mediated Learning, Dragefjellet, University of Bergen, Norway. Morch, Anders (1999, October). DoCTA/VisArt: Use of Groupware in Design of Visual Artefacts. ITU- Interactive Conference (ITUKonferansen), Oslo, Norway, October 8th. Guribye, Frode (1999, October). Virtual Ethnography. ITU-Interactive Conference (ITU- Konferansen), Oslo, Norway, October 8th. Meistad, Øyvind (1999, October). Log-files as a Supplement to the Evaluation of Collaborative Telelearning Scenarios. ITU- Interactive Conference (ITU- Konferansen), Oslo, Norway, October 8th. Munkvold, G., Eidsmo, A. & Ekker, K. (September, 1999). Virtual Collaboration: experiences from an international and a national project: Project Ideels and Project Docta. Presentation at the NordTrøndelag College, Research and Development Day. 270 Appendicies Mørch, Anders (1999, September). Activity Theory: Basic Concepts and their Evolution. Invited presentation at Workshop on Evolving use of Groupware at European Conference on Computer Supported Cooperative Work (ECSCW99), Copenhagen, Denmark, September 12th. Spector, Mike & Guribye, Frode (1999, August). Theoretical Foundations for the design of collaborative distance learning / VisArt. European Conference for Research on Learning and Instruction (EARLI’99). Göteborg, Sweden, August 26th. Guribye, Frode (1999, June). Evaluating collaborative telelearning scenarios: A sociocultural perspective. International Conference on Educational Multimedia and Telecommunications, Seattle, WA, June 23rd. Guribye, Frode (1999, June). Project DoCTA: Design and use of Collaborative Telelearning Artefacts. International Conference on Educational Multimedia and Telecommunications, Seattle, WA, June 21st. Wasson, Barbara (1999, June). VisArt: A collaborative telelearning scenario. Invited speaker at the LINGO Project Workshop, University of Bergen, June 4th. Wasson, Barbara (1999, April). Project DoCTA. ITU Group Leader Workshop, University of Oslo, April 26th. Wasson, Barbara (1999, April). Reflections on Activity Theory. ITU Group Leader Workshop, University of Oslo, April 26th. Wasson, Barbara (1998, October). Project DoCTA. ITU Konferanse: Fra tradisjon til innovasjon - IKT og utdanning, University of Oslo, October 13th. Guribye, Frode (1998, October). Project DoCTA. Human Centred Technology Workshop, University of Sussex, Brighton, UK, October 3rd. 271 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix C: Assignment for test-scenario (English) “Ytrebygda Tannlegesenter” is a private dental clinic with 18 employees. There are 4 secretaries, 4 dental-assistants, 5 dentists and 5 assistants. Every dentist has a personal assistant, but sometimes it will be necessary for the assistant to have a day off. It will then be possible to switch with some other assistant, if time allows. The clients have a regular dentist, whom the consult. But sometimes it can be necessary to seek out different competence. One wishes to provide all kind of dental treatment on the clinic. As for now, they have about 2500 regular patients, and a number of emergency-patients who only come this one time. They are also provided with a number of new patients, often children of their regular customers. It is not desirable to get too many new customers. A number of the patients are long-term customers who come once a month to get braces or orthodontic treatment. There has been some trouble with the manual archives, especially when a patient has received treatment from different dentists. The files have a tendency to stay in the different dentists’ own little archives, instead of being put in the main archive in the secretaries’ office. This leads to much redundancy of information, and that not all-important information reaches the archive. This can lead to serious consequences on what is to be done to the patients. One wishes to have a better archive-system, which is more accessible and more informative. Also, there is no complete overview of the work-schedule of the different dentists, this because the dentists give their patients new appointments without informing the secretaries. This can lead to double booking of appointments and great frustration with the patients and secretaries. The same problems are to be found by the dental-assistants. In turn, these problems lead to disputes between dentists and dental-assistants because nobody knows exactly when the others have given the patients a new appointment. One wishes to have better overview, as to the patients to be more dispersed between the different dentists. This will lead to better time managing. 272 Appendicies The board of the clinic has decided to grant some funding to get an improved system. Exactly how much is not yet decided, but they want to look at several different proposals before they make up their mind. Give a proposal of how these problems can be solved. Make a use-case diagram and a class diagram. Use UML-notation if possible. It is not necessary to include methods and variables. Oppgave til test-scenario (Norsk) Bedriften ”Ytrebygda Tannlegesenter” er en privatdrevet tannklinikk med 18 ansatte. Man har 4 sekretærer, 4 tannpleiere, 5 tannleger og 5 assistenter. Hver tannlege har sin egen assistent, men enkelte ganger kan det være nødvendig for assistentene å ha fri. Man kan da bytte med noen av de andre assistentene dersom det er mulig tidsmessig. Klientene har sin egen faste tannlege, men noen ganger kan det være aktuelt med annen kompetanse enn det den faste tannlegen har. Man ønsker på klinikken å kunne gi alle former for tannbehandling. For tiden har man en krets på omtrent 2500 faste klienter, og en del hastesaker som kun er engangstilfeller. Man får også en del nye klienter etterhvert, da gjerne barn av dem som allerede er faste kunder. Man ønsker ikke å ta inn så veldig mange flere faste kunder. En rekke av kundene er langtidskunder som kommer hver måned på grunn av streng/tannstillings-behandling. Man har hatt en del problemer med de manuelle arkivene, da spesielt når en pasient har vært hos flere ulike tannleger på senteret. Mappene har en tendens til å havne i den enkelte tannleges eget lille arkiv, i stedet for i hovedarkivet som befinner seg på sekretærenes kontor. Dette fører også til mye dobbeltlagring av informasjon, og at ikke all informasjon havner i mappene. Slike ting kan gi alvorlige konsekvenser på det som gjøres med pasientene. Man ønsker et bedre system på arkivene, slik at de blir mer tilgjengelig, og mer informative. Man har heller ikke noe fullgod oversikt over alle tannlegenes arbeidsplan, da de gjerne gir pasientene ny time, uten å informere sekretærene om dette. Det kan gi dobbelt-bestilte timer, til pasientenes og sekretærenes store frustrasjon. De samme problemene går igjen hos tannpleierne. 273 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Det blir også en del gnissninger mellom tannpleiere og tannleger, ettersom ingen vet helt når de andre har gitt pasientene timer. Man ønsker å kunne ha bedre oversikt, slik at pasientene blir jevnere spredd ut på de ulike tannlegene. Dette vil også gi bedre disponering av tid. Tannlegekontoret har besluttet å sette av en del midler i forbindelse med å få et forbedret system. Akkurat hvor mye er ikke klart, men de ønsker å se på en del ulike løsninger før de bestemmer seg. 274 Appendicies Appendix D: Interview Guide for Evaluation of Groupware Systems (English) 1. What expectations did you have to a groupware-system before conducting this test? 2. Did this program fulfil your expectations? Why/ Why not? 3. What surprised you more, solving a task like this, using the program? 4. Did you feel you could solve the task in the way you are used to, or was it more/ less demanding? 5. 6. 7. 8. 9. Did you feel you had more limitations or more possibilities? How did you find the user interface of the program? Did you find the program difficult or easy to understand? Have you any suggestions for changes/improvements of the program? Did you miss something in this program? Intervju-guiden (Norsk) 1. Hvilke forventninger hadde du til et gruppevare-program i forkant av denne testen? 2. Levde dette programmet opp til dine forventninger? Hvorfor / hvorfor ikke? 3. Hva overrasket deg mest med å løse oppgaver på denne måten (vha programmet)? 4. Følte du at du kunne løse oppgavene slik du er vant med, eller ble det mer/mindre krevende? 5. 6. 7. 8. 9. Følte du at du fikk begrensninger, eller flere muligheter? Hva synes du om brukergrensesnittet i programmet? Fant du programmet vanskelig eller enkelt å forstå? Forslag til endringer/forbedringer? Var det noe du savnet i programmet? 275 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix E: The IDEELS Scenario Scenario: Designing the Tertiary Education System How IDEELS Simulations Work In IDEELS simulations, students act as high-level negotiators, consultants, and journalists in a fictional world, working to resolve real problems of importance to the European Community – problems which can only be solved through cooperation at the international level. In so doing, participants model real interactions between nations and organizations. The simulations are conducted on two levels: deliberations within a team and negotiations between or among teams. Within a team, students familiarize themselves with their assigned profile (country, consultant group, or journalists) and work together to formulate the positions they will present in the multilateral negotiations. They then communicate their policies and conduct a dialogue with the other country teams and consultant groups by sending written messages over a computer-based communications system, OPUSi. In addition, teams confer with each other during real-time, online teleconferences. While the focus of the exercise is multilateral, participants should keep in mind that bilateral discussions are also a central component of the negotiations. The goal is to produce and ratify a jointly-written document (in this case, a set of recommendations) that addresses the problems set forth in the scenario. As a participant in an IDEELS simulation, you will be a member of a team representing one of the following: a country, a team of technical consultants, or a team of journalists. Work with your teammates to prepare your positions carefully, looking for a balance between national, regional, or local strategic considerations and international (Federation) interests. 276 Appendicies In the simulation, which is set in the early years of the 21st century, you will engage in the following processes or phases: Phase 1: Pre-Negotiation Your team will prepare two documents: 1) an Internal Briefing Document and 2) a Policy Statement Both of these should be ready to submit at the beginning of the simulation. The Internal Briefing Document is intended to guide your team’s actions during the negotiations; it outlines your positions on the issues, your team’s interests, negotiating goals, and strategies for achieving these goals. This document is confidential and should not be revealed to other teams. The Policy Statement , in contrast, is for public consumption and sets out your team’s initial negotiating position – its views concerning the issues to be discussed, their significance for your country or organization as well as for Eutropia, and ideas that you would like to see included in the final document. Phase 2: Opening Positions By noon CET on the Thursday before the negotiations begin (November 4), all teams should have finished writing their Internal Briefing Documents and Policy Statements. The Internal Briefing Document should be put in “Papers” in OPUSi. A copy of this Internal Briefing Document is to be e-mailed to the simulation moderator: [email protected] but not to the other teams. Policy Statements, on the other hand, should be sent to all other teams (via the OPUSi Message Centre) by the same time. Phase 3: Dialogue Once you have read the other teams’ Policy Statements, you should immediately begin writing messages (Message Centre) to other teams to request clarification of any points that are not clear or to suggest areas of possible agreement. The Eutropian Commissioner will schedule a teleconference during the first week to open the negotiations. 277 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Phase 4: Negotiation Throughout the simulation, teams will communicate with each other via the Message Centre and will participate in bilateral and multilateral teleconferences. Most of the real work of the negotiations takes place via the Message Centre’s asynchronous messaging system. The scheduled teleconferences provide opportunities for negotiators to discuss specific issues with all participants simultaneously. Phase 5: Closure The agreed-upon recommendations are to be submitted to the Eutropian Commissioner and ratified at a final teleconference. Phase 6: Debriefing Both during and following the simulation, you will have opportunities to reflect on and discuss your participation in the simulation. Guidelines for Writing the Internal Briefing Document A well-conceived briefing document is essential for the success of an IDEELS simulation. Writing a briefing document gives you a chance to learn more about the topics and issues that are central to the simulation. It also allows teams to formulate their policy goals and plan negotiating strategies before the simulation begins. This guide will help you prepare your internal briefing document. I. Overall format and organisation of the Internal Briefing Document A. The Internal Briefing Document is the product of a group effort. Therefore, it should be submitted as a unified document rather than as separate submissions from individuals or from issue area working groups. 1. The final text should be contained in a single document. This document should be written and revised in “Papers” with a subject line “Internal Briefing Document - Top Secret!” 278 Appendicies 2. The format, i.e. margins, etc., of the document should be consistent throughout. B. The Internal Briefing Document should be as professional a document as possible. It should contain the following components: 1. A table of contents. 2. An executive summary or overview, which succinctly presents the following: a. The overall national policy goals. b. The broad policy goals for each issue area. c. A brief description of the overall national negotiating strategy. C. Finally, the briefing document should not contain any policy recommendations that are contradictory. NOTE: The Internal Briefing Document for teams of journalists should include the following information: What kind of readers do you want to appeal to? What kind of publication is likely to be taken seriously by high-level negotiators and high-powered technical consulting groups? What is your publication’s editorial stance on education-related issues? When and how often will you publish your newspaper (e.g. daily, three times a week, or once a week, with special editions or news bulletins as events require)? NOTE for technical consultant groups: To a large extent, the company’s profile, as well as your objectives and goals, are set down in your team profile and in the call for proposals. Your goal is to have your technology proposal endorsed by the Task Force on Education and Training (TaskFETE) by the end of the simulation. Your internal briefing document should cover the areas in which you intend to make specific recommendations, e.g., elementary, secondary, tertiary institutions, open and distance learning, etc., as well as the strategies you intend to use to reach your goal. II. The format and organisation of the individual sections Each section addressing a substantive issue area should contain the following information, presented in this format. A. Identify the substantive issue. 1. What is the significance of the issue for the Eutropian Federation? 2. How is this issue important for your country or organisation? 279 Project DoCTA: Design and use of Collaborative Telelearning Artefacts B. Policy objectives. 1. What are the policy objectives of your country or organisation? 2. If these objectives cannot be reached, what are the secondary or compromise objectives? C. Negotiating strategies. 1. What are the various strategies that could be pursued in the negotiations? A comprehensive paper will provide a variety of strategies. 2. What are the advantages and disadvantages of each strategy? D. Final recommendation 1. What is your team’s the final policy recommendation, in terms of goals and strategies? Guidelines for Writing the Policy Statement The Policy Statement is in many ways a public version of the Internal Briefing Document. In it, you state your country’s policies on the various issues (what you favour and what you oppose), but you do not reveal your negotiation strategies NOTE to journalist teams: Present your Policy Statement in the form of the inaugural issue of your newspaper, to be published on the first day of the negotiations. In it, you introduce your publication, describe your editorial stance and announce your publication schedule. Indicate how you intend to cover the negotiations. Describe the relationship you would like your publication to have with its readers. You might want to use the opportunity to invite letters to the editor or other pieces of writing, for example. Indicate who you hope will be interested in reading your publication. Technical consultant groups: Your Policy Statement is a chance to advertise your company. Use this opportunity to profile your company for the others. Tell them what kind(s) of technical expertise you can provide to the country teams as they negotiate a design plan for the Eutropolis education system. 280 Appendicies I. Overall Format and Organization of the Policy Statement A. The Policy Statement is the product of a group effort. Therefore, it should be published (i.e. sent to all other teams) in a single, unified document rather than as a number of separate messages from individual team members or from issue area working groups. 1. While the final document should be contained in a single file, it can be written and revised in any number of separate sections in “Papers” and then copied into a single message in the “Message Centre”. Send this message to ALL teams, with the following subject line: “Policy Statement.” 2. The format, i.e. margins, etc., of the document should be consistent throughout. B. The final Policy Statement should be as professional a document as possible. It should contain the following components: 1. A table of contents. 2. An executive summary, or overview, which succinctly presents the following: a. The overall national policy goals. b. The broad policy goals for each issue area. C. Finally, the Policy Statement should not contain any policy recommendations that are contradictory. II. The Format and Organization of the Individual Sections Each section should address a substantive issue area and should contain the following information: A. Identify the substantive issue. 1. As your delegation sees it, what is the significance of the issue for the Eutropian Federation? 2. Explain how important this issue is for your country and why it is so important. B. How does your delegation propose to deal with or resolve this issue? In addition, the position paper should include a bibliography listing any references to secondary sources. The importance of including references for quoted or paraphrased material cannot be overstated. It allows your readers to check the original references and it demonstrates academic integrity. 281 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Eutropia 282 Appendicies Journalists You are members of a team of journalists (Eutropian or national) who publish an electronic newspaper or magazine. It is your job to report and comment regularly on the progress of the Task Force’s negotiations as well as on the work being done by the technical consultants. Your first task is to organise yourselves as a team. This must be done before the simulation starts. Choose a name for your publication and decide how often you intend to publish. As soon as you have chosen a name, send a message to the simulation moderator indicating what your publication’s new name is. The moderator will then change your group’s name in OPUSi. Decide how you want to organise your team; your team’s tasks include activities such as news reporting, editorial writing, doing background research, editing articles and commentaries, interviewing members of the Task Force on Education and Training, sifting through the existing documents and communications for newsworthy stories. You can run your organisation in any way you see fit. Remember, though, that the more professional it is, the more seriously you will be taken by your readers. Prepare a confidential statement outlining your publication’s editorial stance (bias) and the strategies you intend to use to achieve your publishing goals. This document is the equivalent of the Internal Briefing Document prepared by the country teams. Send a copy of this to the simulation moderator. Your second document, the equivalent of the country teams’ Policy Statements, is the first (inaugural) edition of your publication. In it, you announce the launching of your publication and introduce it to the other participants in the simulation. Consider including some of the following 283 Project DoCTA: Design and use of Collaborative Telelearning Artefacts kinds of “market-relevant” information about your organisation: size, type of publication, political or editorial orientation (bias), circulation (number of readers). Consider “real world” models such as “The European,” “Le Monde,” the “Chronicle of Higher Education,” CNN’s “Impact,” MSNBC’s “Dateline” or the BBC’s “Broadcasting House.” Make your profile as realistic as possible – this will add to your publication’s credibility. Publish your inaugural issue on the Thursday prior to the start of the negotiations (4 November). Read the profiles of the various countries and consultants before the negotiations begin. As soon as the Policy Statements have been published, contact the other groups. Ask questions about their positions on various issues, based on what they have said in their Policy Statements and what you know about them from the team profiles. Make your questions specific, and direct them to individual countries whose profiles suggest they might have something interesting to say on that topic. Once you have launched your publication, your next issue (you may combine this with your inaugural issue) should be based on the country profiles and the basic socio-economic and education-related data contained in them. Your second issue might be based on the countries’ Policy Statements. If countries are late in sending their Statements, there is nothing against your mentioning this in your publication, but bear in mind that your success as journalists depends to a great extent on your ability to establish and maintain good relationships with your readers and with your information sources (contacts). During the simulation, you will have the following tasks to accomplish: 1. Produce an edition of your periodical at least twice a week. This will contain at least two broad categories of text: (a) reporting on events and developments and (b) editorial commentary on these or other items and issues. Decide what your deadlines are – on what days and by what time will you put out your periodical? 2. Regularly interview each country and technical consultant team to obtain information and opinion on current issues. 284 Appendicies 3. When events warrant, publish a special issue covering a breaking news story. You may also publish issues devoted to a particular education-related theme or a particular aspect of the Task Force’s work. 4. Invite your readers to contribute “op ed” pieces (views which disagree with positions your publication has taken) and “letters to the editor.” Make a clear distinction between reporting news (“facts,” events or others’ opinions) and offering editorial commentary (presenting your organisation’s own opinion or point of view). In addition to writing messages to the other groups, here are some actions you can take: interviewing countries, organizing press conferences and providing background information or documents related to the education issues that are the focus of the simulation. Please do not “invent” major facts or events without first checking with your facilitator. S/He will let you know as quickly as possible if a fact or event is viable. Remember that such things should be credible; otherwise they tend to reflect badly on your organisation. From time to time you may receive “leaked” information (sent to you by others involved in the negotiations, for example) that you would not normally have access to; you may also receive “wire service reports” or “tips” that give you leads to follow up. Again, the credibility of your periodical depends largely on the professional quality of your reporting and commentary. You might like to consider looks as well as content from a professional angle. If you would like your publication to be Web-based, contact the simulation moderator. Space will be provided on a Bremen-based server. Do not use a local server for this, as doing so would reveal your real location. Top 285 Project DoCTA: Design and use of Collaborative Telelearning Artefacts InfoTech Unlimited: Technical Consultants Highland Government: Highland is a parliamentary democracy. Work: Over half of the labour force (54.5%) works in the service sector (roughly half of these in the public sector), another 30% work in manufacturing and about 15.5% in agriculture. Unemployment has proved to be a stubborn problem, and is currently around 18.5% nationally, with at least one region experiencing levels as high as 26%. Strict austerity programs in the early 1990’s allowed Highland to reduce budget deficits and government indebtedness to acceptable levels so Highland could join the monetary union, and budgetary discipline has kept the national debt at just under 60% of the gross domestic product since that time. Government pressure has also kept wages lower than elsewhere in the Federation, which might have attracted new investment if the infrastructure had been adequate. Instead, by the turn of the century Highland’s gross domestic product was still less than 70% of the Federation average per capita. Not surprisingly, Highland receives more money from the Federation in the form of subsidies than it contributes; 8% of the national budget is currently supplied by the Federation. At the end of the 1990’s, in an attempt to reduce Highland’s dependence on Federation funds by revitalizing the Highland economy, the government began to increase spending to improve the country’s infrastructure and to renew its industrial base. 286 Appendicies The campaign to make Highland a more attractive location for new investment included a program of incentives to encourage the use of information technologies in the workplace. However, Highlanders initially resisted, fearing that further computerisation would cause even more unemployment. Currently, there are only 56 computers per 100 employees - the lowest ratio in all Eutropia. Society: Highland has a total population of 9,983,050. The overwhelming majority of Highlanders are Catholic; with the exception of a small community of citizens of African descent (less than 5%) who moved to Highland during decolonization the population is also ethnically homogeneous. Highlanders, as a rule, identify more strongly with the local district in which they live than with Highland’s national government, and district governments have more autonomy than is the case in most Eutropian member countries. On some issues, Highland behaves like a federation within the Eutropian Federation; responses to social problems such as crime or substance abuse frequently have a distinctly local character. Education System: Comprehensive schooling is compulsory until the age of 16, after which apprenticeships or, alternatively, vocational or technical training are available for those who do not complete the final two years of comprehensive schooling to qualify for university. Subjects taught and teaching methods used are prescribed by provincial culture and education ministries and there has been little encouragement or tolerance of innovation in teaching. Attempts to increase the national government’s influence on education policy-making and to set national performance standards have met with limited success. Because of this, wealthy Highlanders tend to send their children abroad for their secondary and tertiary education. A significant number of these later decide to live and work as expatriates in other countries, leading to a serious “brain drain” problem in Highland. 287 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Classical languages are required of students preparing to study at university, and bilingual education has been introduced on a limited basis in the border districts, but foreign languages are otherwise optional and offered only in the last four years of the school curriculum. Hokish is the official language of instruction. Highland is home to some of Euturopia’s oldest universities. In the 1980’s and 90’s, as Highland struggled to meet Federation criteria for joining the monetary union, Highland’s tradition-rich educational system paid a heavy price. Austerity measures and budget restrictions repeatedly led to severe cutbacks in the already modest resources available for education. Teachers and professors alike were poorly paid and demoralized, equipment was hard to come by and buildings were poorly maintained. Today the situation is showing signs of improving as the economy improves. Class sizes are still intolerably large at all levels, particularly at universities, library holdings are outdated, and computer equipment is in short supply, but the government has finally understood that competitiveness in Eutropian and global markets requires a well-educated, technology-literate workforce. As a result, Highland has begun to invest in facilities and information technology. In order to have more money available for national education initiatives, Highland’s government wants to keep spending for the Eutropolis education system at an absolute minimum. On the other hand, the Government has had to acknowledge the existence of a seemingly insoluble problem regarding finance: as standards in schools improve, more people are seeking admission to tertiary institutions, putting an immense strain on the current practice of funding for all who gain a place in a tertiary establishment - and this at a time when more investment in equipment and personnel is being demanded. One proposed solution is to make tertiary education available to as many as possible, as cheaply as possible, via an “Open University,” with material transmitted via television, radio and the Internet, and making considerable use of teleconferences. Two problems arising from this are the question of entrance requirements and whether qualifications won in such a way would be fully accepted by industry and employers as the equivalent of more traditional qualifications. 288 Appendicies Literacy: 85% of the population over the age of 15 can read and write; this is the lowest literacy rate in Eutropia. Literacy rates are significantly higher for males (90%) than for females (81%). The discrepancy is even more noticeable when it comes to computer literacy; 55% of Highland’s male population between the ages of 7 and 45 is computer and Internet literate, compared to 38% of the female population. One hopeful sign: Highlanders under 25 years of age accept and use computer technology at a much higher rate than the national statistical average. Electronic Information Infrastructure: Highland has a generally adequate integrated network of coaxial cables, open wire, microwave radio relay, satellite earth stations and submarine cables to off-shore islands. Highland’s mountainous terrain means improvements to the infrastructure are costly and technically challenging. 289 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix F: Assignment for designing an evaluation of IDEELS Contemporary Perspectives on Research in Pedagogical Information Technology Research Seminar - Fall 1998 Semester Assignment – 3 In the semester oppgave you will evaluate your chosen scenario from one of the perspectives covered in the class this semester: psychometrics, distributed cognition, activity theory, discourse analysis. This perspective gives you boundries for the type of research question you propose (problem stillingen) the type of data you need to collect, how you collect it and how to analyse it. When coming up with a research question, you could begin by discussing among yourselves the types of data you can collect and how it can be used (e.g., structured interviews, participant observations, video taping etc.) and the types of questions you can ask of the data. What I am trying to say is that do not just think of it from the top down (question —> data collection etc), rather think ALSO from a bottom up (this data —> what questions). When it comes to what to include in the written report, you should have sections such as: 1. the scenario (describe the scenario, the learning environment including the actors involved (i.e., students, teachers, facilitators, technical helpers, tutors, the school or university, parents, rectors etc.), the artefacts (i.e., the actual CSILE system, the technologies in IDEELS - email, teamwave, OPUSi, the internet, your own editor etc. -, the OthoDL 290 Appendicies database etc.) and so on. When you start writing this part you will probably begin to get some ideas as to the types of data you can collect (created artefacts, electronic data logs of email, interviews with people, videotapes of people using the system etc..)). 2. the theoretical perspective (here you will demonstrate that you understand the main aspects of the perspective - describe it in general, its theoretical roots, the types of research carried out here etc., where the theory focuses (e.g., distributed cognition focuses on people interacting through artefacts) etc. 3. the research question you will ask. Motivate why you want to ask this question. 4. Data to be collected - describe the data you need and how you will get it. If you will have a questionnaire, make up the questionnaire, if a structured interview design the questions, if an unstructured interview describe what topics you hope to get out of it and a few leading questions you will use etc. 5. How will you analyse the data to answer the question you have asked. Again, for the report length, the content is more important - that you include the above points is a minimum. Try to keep the report between 10 and 20 pages. Hand in 3 copies of the report On the 1st exam day the group will present their projects to the entire class. This is part of the exam. There will be allocated 15 minutes per group. 291 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix G: Interview Guide for IDEELS (English) Interview #: Date: Interviewee: General • How has the assignment work and the group functioned? Division of labour • Who did what? * Why? * How? • How did you arrive at the particular division of labour? * Process? * (example, responsibility for webpages) • Did everyone agree on the chosen division of labour? * division of workload Communication • What communication forms did you choose * Face-to-face, computer mediated, telephone * What computer tools did you use? Why? * General on TW as a communications tool (use / lack of use) Integration • How did the assignment work function in relation to your other everyday tasks? • How did this effect the assignment work/group collaboration? * in particular: division of labour and forms of communication Goal Setting • What did you understand the goals of IDEELS to be? * generally and specifically 292 Appendicies Intervjuguiden (Norsk) Interview #: Date: Informant: Generelt • Hvordan har arbeidet med oppgaven og gruppen fungert? Arbeidsdeling • Hvem har gjort hva (hvilke oppgaver bestod arbeidet av? hvorfor?) • Hvordan kom dere frem til den arbeidsdelingen? Prosess. (eks webside) • Var alle enige i den arbeidsdelingen som ble valgt? (fordeling av arbeidsmengde) Kommunikasjon • Hvilke former for kommunikasjon valgte dere? Hvorfor? (face-to-face, telefon, datamediert?) • Hvilke verktøy brukte dere? Hvorfor? • Kan du si litt om teamwave? (bruk / mangel på bruk) Integrasjon • Hvordan fungerte det å arbeide med denne oppgaven i forhold til resten av dine daglige gjøremål? • Hvordan påvirket dette samarbeidet og valg av arbeidsdeling og kommunikasjonsformer? Målsetting • Hva oppfattet du som målsettingen i å være med i IDEELS? (generelt og spesifikt) 293 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix H: Pre-simulation survey for IDEELS 294 Appendicies 295 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 296 Appendicies 297 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 298 Appendicies Appendix I: Post-simulation survey for IDEELS 299 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 300 Appendicies 301 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 302 Appendicies Appendix J: Planning Tables from the Instructor’s View (Phases II-IV) II Training Task Date Giving instructions 18.02 for training Data source State of source Comments Responsible Instructions Prepared available in TW Helge Put out files at pre-specified times 24.02- Files (Expert Prepared 5.03 Rankings, Permission on Activity 4 room) Helge Follow training (general) 24.02- T W 5.03 Follow training activities 24.025.03 Helge Activity 1 24.02 - Activity 1 room Prepared 3.03 Helge Activity 2 24.02 - Activity 2 room Prepared 3.03 Helge Activity 3 24.02 - Activity 3 room Prepared 3.03 Helge Activity 4 3.025.03 Receiving Assistance Requests students 24.02- E-mail from 5.03 students *Agreed upon Helge method for answering Distributing assistance?’s if necessary 24.02- E-mail from 5.03 students Send forward Helge > if cannot Barbara or answer! Trond > * Answering content ?s 24.02- E-mail from 5.03 Helge Barbara Answering technical ?s 24.02- E-mail from 5.03 Helge Trond Activity 4 room Prepared Maintain message 24.02- Message boards with ?’s 5.03 boards in Help Room Keep server running Just to seen in general Prepared 24.02 - TW server on 5.03 grevling.ifi.uib.no Available from 3.02 Barbara, Lars, Arnstein Helge *Need routine Helge (w/ for checking Barbara message & Trond) boards Trond > “drift” 303 Project DoCTA: Design and use of Collaborative Telelearning Artefacts III Designing an instructional room Task Date Data source State of source Comments Following up the design of an instructional room Responsible All researchers Receiving Assistance Requests students 24.02- E-mail from 5.03 students *Agreed upon Helge method for answering Distributing assistance ?’s if necessary 24.02- E-mail from 5.03 students Send forward Helge > if cannot Barbara or answer! Trond > * Answering content ?s 24.02- E-mail from 5.03 Helge Barbara Answering technical ?s 24.02- E-mail from 5.03 Helge Trond Maintain message 24.02- Message boards with ?’s 5.03 boards in Help Room Keep server running Prepared *Need routine Helge (w/ for checking Barbara & message Trond) boards 24.02- TW server on 5.03 grevling.ifi.uib.no Trond > “drift” IV Follow-up Task Date Data source State of source Comments Responsible Receive course assignment 12.04 Individual student assignments Each instructor responsible for receiving their own student’s assignments Barbara, Arnstein, Lars Received Tell class about the To Be research in DoCTA Decided Barbara Take down TW server Trond 304 To Be Decided Appendicies Appendix K: Planning Tables from the Researcher’s View (Phases I-III) I Preparation Task Data source State of source Comments Prepare computer 01.02 supported collaborative learning assignment Date Assignment file Prepared (merge together activities) Part 2 of Barbara assignment is VisArt Prepare profile questionnaire 15.02 Profile Prepared questionnaire On web (make Kurt available 24.02) Prepare 15.02 pre-questionnaire PrePrepared questionnaire On web (make Kurt available 7.03) Prepare postquestionnaire 15.02 PostPrepared questionnaire On web (make Kurt available 25.03) Prepare student consent form 15.02 Consent form Prepared Send to Lars Barbara > Lars and Arnstein & Arnstein Getting students’ By consent 24.02 Consent form Prepared Receive from Barbara Lars and Arnstein as well Assign researchers to a team Team list 21.02 Getting students’ profile Prepare for video taping Prepare for interviews Prepared Profile Prepared questionnaire file 24.02 Responsible Barbara > researchers student > Kurt Eskil & Knut Each researcher 305 Project DoCTA: Design and use of Collaborative Telelearning Artefacts II Training Task Date Follow what is going on 25.02- T W 7.03 Data source State of source Comments Prepared Getting the self- 25.02- Email to Helge Prepared evaluation results 7.03 Interviewing students 25.02- TW, emails 7.03 Responsible All researchers Helge According to Each interviewing Researcher needs Collecting assistance requests Email from students Helge w/ Barbara & Trond Getting logs TW TW >Øyvind, Getting versions control TW TW >Øyvind, Selecting teams for following 7.03 Data logs for phase 2 Select teams Øyvind to follow in phase 3 III Designing an instructional room Task Date Data source State of source Follow what is going on 8.0326.03 TW Comments Prepared Responsible All researchers Getting the self- 8.03evaluation results 26.03 Email to Helge Make available Helge to other researchers Interviewing students 8.0326.03 TW, emails According to Each interviewing Researcher needs Collecting assistance requests 8.0326.03 Email from students Make available Helge w/ to other Barbara & researchers Trond Getting logs 8.03 26.03 TW Make available to other researchers TW >Øyvind, Getting versions control 8.03 26.03 TW Make available to other researchers TW >Øyvind, Collecting chat messages 8.0326.03 Emailed Chat file Remind students to send chats 306 Each researcher Appendicies Appendix L: Planning Tables from the Student’s View (Phases I-III) I Preparation Activity Date Documents/ instruments State of Source Comments Responsible Sign an agreement 16.02 to participate in research Consent agreement Prepared Paper version Barbara > instructors > students Get TW account 24.03 List of accounts Prepared Email list to all participants Barbara > students Get team list 24.03 List of teams Prepared Email list to all participants Barbara > students Getting instructions 16.02 Instructions (Explaining 3 phases, their purpose, the deadlines, the assistance provided etc.) Assignment document completed & delivered Electronic form Barbara > of VisArt part in instructors > Classroom student > Completing profile 25.02 questionnaire Profile Prepared questionnaire Electronic form student > on web; Kurt (collects, e-mailed auto puts in file) matically to Kurt 307 Project DoCTA: Design and use of Collaborative Telelearning Artefacts II Training Activity Date Getting instructions 25.02 for training Documents/ instruments State of Source Instructions for Prepared the training session Comments Responsible In TW training Helge > room student Doing training activities 25.02 - Training documents 5.03 Prepared Activity 1 25.02- Training 4.03 document Prepared In TW activity 1 room Helge Activity 2 25.02- Training 4.03 document Prepared In TW activity 2 room Helge Activity 3 25.02 - Training 4.03 document Prepared In TW activity Helge 3 room Activity 4 4.03 –5.03 Training document Prepared room In TW activity 1 Helge Prepared on web; e-mailed automatically to Helge Electronic form student > Helge Completing the self-evaluation tool Self-evaluation test Participate in an interview 5.03 Getting assistance 25.02- Help and 5.03 resources 308 Interview Guide Helge TO DO According to Each each researcher researchers needs TO DO Help rooms; Student > Email > Helge Helge > Barbara or Trond > Student Appendicies III Designing an instructional room Activity Date Documents/ instruments Completing prequestionnaire 8.03 Designing an instructional room Getting assistance Comments Responsible PrePrepared questionnaire Electronic form on web; e-mailed automatically to Kurt Student > Kurt (sort questions) > Interested researchers 8.0327.03 Assignment Document Paper copy and in TW Classroom Barbara 8.0327.03 Help Room Prepared and resources; Assistance plan TW Help Room; Student > Assistance plan Helge > ready Barb or Trond > Student Completing post- 26.03 questionnaire Postquestionnaire Electronic form on web; e-mailed automatically to Kurt Participating in an interview Interview guides 8.0327.03 State of Source Prepared TO DO Student > Kurt (sort questions) > Interested researchers Selected Responsibility students only of individual as needed researchers 309 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix M: VisArt Consent Form Student Consent to Participate in Research A research team headed by Dr. Barbara Wasson is currently conducting a research project DoCTA: Design and use Of Collaborative Telelearning Artefacts, that is investigating the design and use of telelearning activities with distributed students. The institutes involved in the research include: the University of Bergen, Stord/Haugesund College and Nord-Trøndelag College and Telenor FOU (Kjeller). Our intention is to gain a better understanding of student learning in this new mode of course delivery, more specifically collaborative learning over the internet with a distributed team of students. As part of a course at each of the teaching institutions, a collaborative telelearning activity, called VISART, has been designed to have teams comprised of a student from each of the teaching institutions collaborate through an internet-based groupware system called TW. Through your responses to questionnaires, interviews and through electronic data collection we hope to find more about your opinions of the software, the activities, collaboration, telelearning in general, etc. There are 8 graduate student projects associated with VisArt and we hope that you will agree to provide them with enough data that they can carry out their projects. The activity has been designed with a preparation phase (installing software, etc.), a one-week training phase and a three-week activity phase. During each phase you will be asked to fill in questionnaires, perhaps participate in an interview, and agree to have your team email (you will have a special team account to use for project planning, etc. that will go to a researcher as well, any private email can be sent using your usual email account and WILL NOT be collected) and team electronic chats (to 310 Appendicies be saved by you and sent to a researcher) collected. All information collected will be kept strictly confidential and will be used for research purposes only. Publication of results and final reports will in no way identify individual students and your person number will not be collected. This research is solely for the advancement of knowledge about the conditions under which people learning given this new technological media. There are no other motives. The data collected will in no way inform your course grade. In addition, the project is registered with NSD and their regulations for data collection, storage and deletion are being followed. If you are willing to participate in this study, please sign this consent form and return it to your instructor. Your cooperation in the collection of this information is greatly appreciated, but is completely voluntary. You have the right to choose not to participate in this research project. Your decision will in no way affect your grade. Please indicate by checking in one of the boxes below whether you freely consent and agree to participate in data collection for research in collaborative telelearning. If you have any questions regarding these procedures, please feel free to contact Barbara Wasson at 55 58 41 20 or by email at [email protected]. The research is funded by KUF’s IT I utdanning programme. • yes, I do consent to participate • no, I do not consent to participate Name & Institute (please print):_________________________________ Signature:_______________________ Date:________________ 311 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix N: Semester Assignment IVP 532, Spring 1999 Assigment 1: Computer Supported Collaborative Telelearning This is both an individual and collaborative assignment. It is individual in that each student will hand in a report and receive an individual mark. To write the report, however, entails participating in a collaborative telelearning activity called VisArt. VisArt is part of the DoCTA (Design and use of Collaborative Telelearning Artefacts) research project which is focused on computer supported collaborative learning (CSCL). DoCTA is lead and coordinated by Barbara Wasson and the VisArt scenario is a collaboration between the University of Bergen (UiB), Høgskolen I Nord-Trøndelag (HiNT) and Høgskolen Stord/Haugesund (Stord). During the collaborative activity you will be assigned to a team consisting of 3 students, one from each of UiB, HiNT and Stord. The team will collaborate through some specially tailored groupware called TeamWave Workplace. You should have turned in your consent form before starting this assignment so we know whether you and your team will contribute to the data collection. The assignment has 3 main parts: 1. Read the background literature on CSCL and awareness. 2. Participate in the VisArt Collaborative Activity. 3. Write an individual report that reflects on the process from both a practical and theoretical perspective (details of the report to follow). 1. Literature on CSCL The following literature should be read: Dillenbourg, P., Baker, M., Blaye, A. & O’Malley, C. (1995). The Evolution of Research on Collaborative Learning. In P. Reimann H. Spada (Eds). Learning in humans and machines. Towards an interdisciplinary learning science, 189- 211. London: Pergamon. 312 Appendicies Koschmann, T. (1997). Paradigm shifts and instructional technology: An introduction. In T. Koschmann (Ed.) CSCL: Theory and practice of an emerging paradigm, 1-23. Mahwah, NJ: Lawrence Erlbaum Associates. Salomon, G. (1992). What does the design of effective CSCL require and how do we study its effects? SIGCUE Outlook, Special Issue on CSCL, 21(3), 62-68. Gutwin, Carl; Stark, Gwen; Greenberg, Saul: Support for Workspace Awareness in Educational Groupware. Proceedings of CSCL’95. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. http://wwwcscl95.indiana.edu/cscl95/toc.html. Some pointers to CSCL on the web: PedInfo ’98: http://www.ifi.uib.no/staff/barbara/courses/pivar98a2projects.html Outlook: http://www-cscl95.indiana.edu/cscl95/outlook/ Outlook.Contents.html CSCL’95: http://www-cscl95.indiana.edu/cscl95/fullprg.html CSCL’97: http://www.oise.utoronto.ca/cscl (click on plenary sessions & parallel sessions) Collaboration Projects: http://www.hypernews.org/HyperNews/ get/www/collaboration.html 2. VisArt Collaborative Activity Objectives: • To provide an opportunity for participation in a distributed collaborative learning activity where team members do not have the opportunity to meet face-to-face. This will give experience in both distributed collaboration and in the use of a groupware system over the Internet. It will hopefully highlight both positive and negative aspects. · If each team member has given consent to participating in the research aspects of this scenario, your team’s experience will also contribute to furthering our understanding of this new learning phenomenon. 313 Project DoCTA: Design and use of Collaborative Telelearning Artefacts In the VisArt activity you will: • Be part of a group of 3 (see list: 1 Bergen, 1 HiNT, 1 Stord) who will form a design team (each team will be assigned a number). This same number will be used to identify the team’s working room in TeamWave Workplace (e.g., team 1 will use the Team1 room) and the team’s common email address (e.g., team 1 will have [email protected]). • Organise a collaborative team effort • thinking Salomon’s definition genuine interdependence: 1) sharing information 2) division of labour 3) joint thinking • Carry out the Design Activity with the help of: • help pages (http://www.ifi.uib.no/docta/VisArt/help) • assistance (email [email protected]) • Help Room • Produce 2 items: • A document of your pedagogical decisions (e.g., who is the room intended for, what is the content, etc….) • A TeamWave Workplace room for teaching/learning The VisArt activity consists of 5 major parts: VisArt Activity Date(s) Preparation Week 8 (from February 25) Training on Tools & Collaboration Week 9 (March 1-5) Pre-questionnaire March 7 Design Activity Week 10-12 (March 8-26) Post-questionnaire by March 26 Preparation: In preparation for VisArt you will need to download TeamWave Workplace and some extension files that you need. Directions for this preparation can be found at http://www.ifi.uib.no/docta/teamwave/. Use the logon id that your instructor has given you. Your team’s email should be working as well. If you experience problems, email [email protected]. If you have not delivered your consent form to your instructor please do so. If you have agreed to participate in the research, you will need to fill in a participant’s profile questionnaire that can be found at http:// www.ifi.uib.no/docta/VisArt/profile.html. 314 Appendicies Training: During the training session you will be taken through a number of activities where you will gain experience in using the technological tools and gain experience in collaborating while at the same time getting to know your team members. A self-evaluation questionnaire will be provided to help you gauge your competence in using the tools. All training will take place in TeamWave Workplace Training Room and you will get more information about this week 8. Pre-questionnaire: The pre-questionnaire that asks about your attitude towards TeamWave Workplace so far needs to be filled in. The questionnaire is can be found from March 5 th at http://www.ifi.uib.no/docta/VisArt/prequestionnaire.html . Please fill it before March 8th. Design Activity: In the VisArt design activity your team is to design a TeamWave Workplace room for learning about some topic. We are most interested in your process of collaborating – how did you make your decisions and carry out them out. Therefore, the outcome of this activity will include the learning room as well as documentation about major pedagogical decisions that you made. More details about the design task follow: 1. The room you chose to design should enable the students to know more about a concept, a procedure, a theory, a process, etc. Some examples of possible topics include a room for: • 6th grade science (e.g., birds, dinosaurs, pollution, etc.) • high school geometry • a university course on programming (e.g., recursion, file structures, etc.) • a theme in an art history course • OR a topic approved by the activity leader ([email protected]) 2. The room could contain a presentation, a game, a tutorial, a questionnaire, an exercise, or some combination of these and utilise a number of the TeamWave Workplace tools. The room you develop should be under your team’s working room. 315 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 3. When documenting the pedagogical decisions that your team makes, you should include issues such as characteristics of the student audience (e.g., age, grade, etc.) and the complexity of the subject area. Create a file PedDecteamx (i.e., use the fileholder tool and substitute your team number in teamx – team 1 is PedDec-team1) and leave the file in your working room. Post-questionnaire: The post-questionnaire that asks about your attitude towards the training, telelearning and TeamWave Workplace after the VisArt has been completed needs to be filled in. It can be found after March 25th at http://www.ifi.uib.no/docta/VisArt/post-questionnaire.html. 3. Individual Report The individual report is the document you will hand in to be counted towards your grade for the semester assignments (i.e., it will count ½ of the semester assignment grade). The individual report should contain: • an introduction to computer supported collaborative learning and collaborative telelearning (use literature in class + other information you might find on the web, in the library etc.) • a description of the design activity you were asked to participate in, including the tools provided and used (e.g., e-mail, TeamWave Workplace, etc…) • a presentation of your team’s room (maybe take a screen snapshot here) and the pedagogical decisions you made to get to the solution • a discussion of how the team met Salomon’s requirements for genuine interdependence and whether or not TeamWave Workplace supported activities resulting from attempts at meeting Salomon’s requirements • a discussion of Gutwin et al.’s awareness concept and what it means in conjunction with your distributed collaboration through TeamWave – does TeamWave Workplace support awareness? How? • your general reaction to collaborative telelearning (as you experienced it) including: a reflection on the team’s work focusing on your participation and the others’ participation (did you feel you did more work, someone did not pull their weight etc. – such comments will not work against them (you do not need to give names) and are private to the instructor), the process of carrying out the assignment, general comments about the entire assignment, your reaction to TeamWave Workplace (did it help or hinder your work, was it easy to use, hard to use, etc.) 316 Appendicies Appendix O: Directions for TeamWave Workplace Installation 317 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix P: Designed Learning Rooms from VisArt Below you will find a number of examples of the rooms designed in the VisArt scenario. Astronomy: 318 Appendicies Threatened Species: Renewable Energy: 319 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Gothic Architecture: Publishing on the Internet: 320 Appendicies Triangles and Pythagoras: The Big Bang: 321 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Propaganda: 322 Appendicies Appendix Q: VisArt Profile Questionnaire (Norsk) 323 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 324 Appendicies 325 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 326 Appendicies 327 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix R: VisArt Self-Evaluation Questionnaire 328 Appendicies 329 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 330 Appendicies Appendix S: VisArt Pre-questionnaire 331 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 332 Appendicies 333 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 334 Appendicies 335 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 336 Appendicies 337 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 338 Appendicies 339 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix T: VisArt Post-questionnaire (Norsk) 340 Appendicies 341 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 342 Appendicies 343 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 344 Appendicies 345 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 346 Appendicies 347 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 348 Appendicies 349 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 350 Appendicies 351 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 352 Appendicies 353 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix U: Norweign Version of Training Assignments 1,3 & 4 (Norsk) Oppgave 1 Velkommen til din første TeamWave oppgave. Denne oppgaven består i at hver person i gruppen skal intervjue en av de andre gruppemedlemmene (f.eks. A intervjuer B som intervjuer C som intervjuer A). Du skal samle informasjon og deretter presentere dette for den 3. personen i gruppen, (f.eks. A skriver en presentasjon av B, B skriver en presentasjon av C og C skriver en presentasjon av A). Forslag til gjennomføring: • Gå til rommet tilhørende din gruppe. • Bruk ”Calendar” for å planlegge tid for intervju. • Bruk ”Chat” for å gjøre intervjuet. • Lag en ”To-do list” for gruppen din. • Presenter intervjuet til de andre i gruppen din ved hjelp av en ”File Viewer”. • Lag ditt eget rom (Room, New Room). Dette kaller du det samme som brukernavnet ditt. For å begrense innsyn til rommet ditt går du til Permissions og ”setter” at bare din gruppe kan gå inn i dette rommet. • Lag en ”Doorway” tilbake til klasserommet fra ditt personlige rom. Når du og gruppen din er ferdig med oppgave 1, kan dere begynne på oppgave 2. Dersom du skulle få tid til overs mens dere holder på med oppgave 1 eller 2, kan du begynne på oppgave 3, som er en individuell oppgave. 354 Appendicies Oppgave 3 Disse oppgavene løser du i ditt personlige rom, dersom ikke annet blir anbefalt. Verktøy Oppgave Web browser Hent en web-side inn i TeamWave Workplace. Prøv hvordan denne nett-losen fungerer. URLRef. Bruk ”Help”-funksjonen og finn adressen til TeamWave sin hjemmeside. Legg så denne inn i en URLRef. og prøv den. Post-it Fortell dine gruppemedlemmer hva du synes om TeamWave så langt ved hjelp av en post-it lapp (i grupperommet). Vote tool Dersom du ikke har gjort det ennå, avlegg din stemme i “Classroom”. Meeting roster Planlegg det du vil anse som å være en/et spennende debatt/møte mellom noen av Norges politikere, du bestemmer tema/mål for møtet. Database Lag en enkel database for cd-samlingen din (tre felter, f.eks. Navn, Tittel, Nasjonalitet). Legg in 5 album i databasen. Eksperimenter med ulike typer felter. Image whiteboard Tegn huset ditt! Prøv også å importere et bilde, og se hvordan du kan kombinere dette. Page Fortell dine gruppemedlemmer at du er ferdig med Oppgave 3 Oppgave 4 I oppgave 4 skal dere sammen finne frem til et hvilket som helst tema som dere skal presentere for de andre gruppene ved å tilby en eller flere linker til sider som omhandler det valgte tema. Forslag til fremgangsmåte: • Foreslå og bli enig om tema, ”brainstorming”, ca. 30 min. • På egenhånd finne sider på www som inneholder informasjon om det tema dere har valgt. Dette kan gjøres til neste dag. • I fellesskap finne frem til hvilke linker som bør benyttes, ca. 1-2 timer. è Gi en kort presentasjon av deres tema ved hjelp av linker til de sidene dere ble enige om. 355 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Denne arbeidsprosessen følger de krav som G. Salomon setter til ”genuine interdependence”. 1. dele informasjon, (the need to share information ) 2. arbeidsdeling, (division of labor among the team members) 3. felles tenkning, (pooling together of minds / joint activity of thinking) Det er G. Salomons oppfatning at for å lykkes med CSCL, (Computer Supported Collaborative learning) må “interdependency” være tilstede. For det videre arbeid i VisArt anbefales det at dere i størst mulig grad prøver å arbeide sammen på denne måten. 356 Appendicies Appendix V: The Narg Island Game Narg Island53 53 Reworked and Developed from AVIAT’s Caribbean Island Survival II Exercise 357 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 14.45 HRS on Sunday 5th of June. Sailing on the 15metre yacht ‘The Pel’ in the centre of a 10-km wide stretch of one of the most beautiful and remote Norwegian Fjords. The boat is close to the southern tip of Narg Island, which is a bare expanse of rock located approximately in the middle of the wide Fjord. Bjørn, the skipper is shouting to you over the noise of a sudden severe storm Here take the wheel and keep her away from the rocks” Suddenly the wheel goes loose in your hands. The rudder has snapped! “Oh No! There goes the rigging…. “ “Loose the sail before she breaks!” Over the sound of the storm everyone hears the noise of the mast breaking “Stand Clear everyone…” “Push the mast out overboard before it knocks anyone out!” “Someone get below and start the engines.” Nils-Erik emerges from below looking worried “The engine compartment is completely flooded and we are taking on water fast” “The radio is dead. So are all the electrics!” 358 Appendicies “Quick get your lifejackets on. Abandon ship!” The rudderless ship is now at the mercy of the storm, with 3 to 4 metre waves and 70 km/hr winds, which are pushing you towards the cruel rocks of Narg Island. Bjørn, the skipper shouts: “Stand by… we are going to hit the island!” “Everyone jump overboard and head for the shore!” 15.03 HRS on Sunday 5th of June. As Bjørn predicted the boat hits and is smashed into the rocks of Narg Island. The cruel waves destroy everything. Even the small inflatable lifeboat is destroyed. The boat sinks rapidly into the extreme depths of the Fjord. Thankfully you manage to scramble up the rocks in the raging storm. “Help Me. “ cries Solveig as she is covered in blood from cuts over her arms and legs. “Quick lets take cover out of the storm in the sheltered cove on the other side of these rocks!” “My God! This is the worst storm I’ve ever seen!” “Look! There is Bjørn floating in the water. Quick help me get him up out of the water!” “Does anyone know first aid?” “He’s out cold. Quick stand back! I’ll have to try mouth to mouth!” “Look at his arm… its bent over double and black.” “That looks bad.” “Does anyone have any ideas what we can do?” 359 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 15.12 HRS on Sunday 5th of June. All of your group are out of the raging seas and are sheltering from the worst of the storm in a small cove. Your leader, Bjørn, looks in bad shape. After mouth to mouth resuscitation he has started breathing but his arm is obviously badly broken and is swelling. He has not regained consciousness and seems to be having a fever. The rest of you are in better shape with just some deep cuts and feeling very wet and cold. Before “The Pel” went to its watery grave you each managed to grab some items from the boat and have added some other things from the wreckage remains that smashed against the rocks after the boat went down (a list of the items are given on page 8) You wanted an adventure holiday, but not like this! It has been 7 days since you rented the boat from Bergen harbour for a 14-day bare-boat charter in the fjords. Bare-boating is chartering a boat without a professional skipper and crew. Since leaving Bergen, ‘The Pel’ has visited several points along the coast always heading north for the remote beauty in that area. Most recently you sailed around the northern tip of Hammerfest and spent last night in the small town of Børselv. This morning, all of you spent two hours scuba diving in dry suits off Maelstrom Rock near Narg Island. Suddenly, a violent storm came up, causing enough concern for Bjørn to decide to weigh anchor and head for the nearest harbour (uninhabited) 30 km further at the far end of the fjord. The cove in which you are sheltering is about 400 metres from the rocks where the boat was lost. The water bordering the island is very deep (500m); the cliffs and rocks around the edge of the island are bare and dangerous. They range in height from 200 to 300 metres. The beach is about 30 metres long and 8 metres deep, rising sharply into rocky and rugged terrain. The only inhabitants of the islands are the colonies of huge gulls on the top of some of the higher cliffs. 360 Appendicies Narg Island is a very remote place. Only occasionally does a cruise or any other boat visit this end of the fjord, which is uninhabited and somewhat foreboding. Sometimes on summer weekends Maelstorm Rock will attract some of the more experienced scuba divers that want a deep dive in icy waters. Narg Island is mountainous with sheer cliffs surrounding most of the perimeter of the island, rising from the sea to about 200 metres. It is a very rocky and rugged terrain with little vegetation. There are several small scrubby trees that tell of very severe weather. Animal life is limited to a large colony of gulls and the odd seal which suns itself on the low beaches during warm weather. There is no fresh water on the island. Air temperature in the area for June is typically 15-22 degrees centigrade during the day and 7-10 degrees centigrade at night with 12 to 18 knot winds. The relative humidity is about 60 to 65 per cent. During storms, the temperature is known to suddenly drop some 20 degrees or more. In the very worst summer weather snow is not unknown!! Water temperature in June is 10 to 13 degrees. There is some pretty 361 Project DoCTA: Design and use of Collaborative Telelearning Artefacts constant fog in this part of the fjord, and this time of the year brings some heavy rainfall and brief storms, which may last up to 48 hours. Sunrise is 3.10 a.m. and sunset is 23.15 p.m. “Jan, what possessed Bjørn to choose this place?” “Simple. The island provides some shelter from the storm. When Bjørn realised he would not make it to the end of the fjord he hoped we could hide from the worst of the storm behind the island.” “At least we are all out of the water!” “Yes that is true. Now we need to decide what we must do next. Any suggestions?” “I wish Bjørn was awake - he would know what to do” “I think we should wait for the storm to settle and then send the best swimmer out to swim for the mainland and then walk for help” “You are crazy! That water is freezing and its 120km walk to the nearest habitation” “Athletes swim the fjords all the time. 120 km is only 3 days walking. I’m for making an escape” “I think we should make a raft and try and float our way to land.” “You can both drown if you like. I’m staying put” “Not me, I’m staying put, at least until the storm ends.” 15.34 HRS on Sunday 5th of June. You are all still in the cove sheltering from the storm. The winds seem to be dropping, but so does the temperature. What was a sunny warm day now feels freezing and it has started pouring with rain. Bjørn does not look good and you are all worried about him. 362 Appendicies You are all dressed in lightweight summer clothes or bathing suits. One of you has a pair of deck shoes on, the rest are barefoot. Collectively, your possessions are: 12670 NOK in bills, 124 NOK in coins, 2 handkerchiefs, 1 scarf, one pair of nail scissors and one pack of cosmetics with a mirror. Given this situation, the resources that you have, the action alternatives which you have and the salvaged items (all listed on pages 7 and 8) what will you do? You should assume the following: 1. You are the people in the situation – only the six described. 2. All salvaged items are in working order, but wet. 3. You will try to survive, keeping all six of you alive. 4. The captain of the boat has a broken arm and concussion. Instructions Now that you have survived the shipwreck, it is critical that you decide how to deal with the situation. First Individually and without discussing The Situation, the ACTION ALTERNATIVES, or the SALVAGEABLE ITEMS, with anyone, rank each according to how important it is to your survival (Steps A to I). Then After everyone has finished the individual work, by group consensus, rank each ACTION ALTERNATIVE and each SALVAGEABLE ITEM according to how important your group thinks it is to your groups survival: Achieving group consensus means arriving at sufficient agreement, not necessarily unanimity (D to M). 363 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Action Alternatives STEP A STEP B STEP C Your Expert Difference Rank Rank A to B 0 364 STEP G Your Total ACTIONALTERNATIVES STEP D STEP E STEP F Team Expert Difference Rank Rank D to E 0 Develop techniques for signalling 0 0 Send representatives to nearest inhabited island 0 0 Do nothing until the storm has subsided 0 0 Search for food and water 0 0 Seek shelter in the cove 0 0 Give first aid to the injured 0 0 Look for vantage points for signalling 0 0 Send out two explorers to research the island 0 0 Decide on a leadership strategy and start planning what needs to be done 0 0 0 STEP G Team Total 0 Appendicies Salvageable Items STEP I Your Rank 0 STEP J STEP K Expert Difference Rank I toJ SALVAGEABLE ITEMS STEP M STEP N STEP O Team Expert Difference Rank Rank M to N 0 Two six packs of beer 0 0 Various pieces of stranded rigging wire 5’ to 20’ long 0 0 One scuba mask 0 0 Four bottles of SPF 5 suntan lotion 0 0 6’ x 14’ blue canvas canopy 0 0 One standard size plastic cooler 0 0 Yachtsman’s guide to the Fjords 0 0 Two orange life jackets 0 0 80’ nylon rope 0 0 Seven cans of food – Meat balls , baked beans, peas, chicken chunks, fish soup and sardines 0 0 SwissArmy knife with multiple blades, including a can opener 0 0 One pair of flippers 0 STEP L Your Total 0 Four wrist watches, two of which are underwater types 0 0 One dinghy paddle 0 0 Three float seat cushions 0 0 0 STEP P Team Total 0 365 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Expert Rankings The Expert Svein ‘IsBjørn’ Tanjun has 36 years of active military service including many years as senior instructor at the elite Norwegian Survival, Evasion, Resistance and Escape school. He has taught courses on survival in the northern Scandinavian climate to the US Army Rangers, US Airborne, US Navy SEALs, The Canadian Winter Warfare Unit, The British Royal Marines and various other European NATO elite units. The Expert’s Basic Guidelines Stay Put & Keep Busy! Most rescues have been made when the lost or stranded party stays in the same spot and makes themselves easily found. Experience has shown that this action has many advantages, enabling you to conserve food, water and other supplies while maintaining all of your stores and focusing solely on surviving and making yourself visible Make a Fire Making a fire for warmth and for signaling is of vital importance Find or Create Shelter The need for adequate shelter cannot be over emphasised. One cold windy night in the open without fire or shelter could be your last. The type of shelter that you construct will depend on the season and terrain in which you find yourself. You should be adaptable and able to construct several types of shelter depending on your circumstances Water About 80% of your body is water, and your vital life processes center around the balance of intake and output of this most vital of elements. Water is needed for normal functioning and when loss of water exceeds intake, dehydration results. It is critical to maintain this balance in order to avoid losing bodily efficiency. 366 Appendicies Food One of the most important things to know about food is that it is not as important as you might think. Depending on your activity, you can go without food for several weeks. Food can in fact work to your detriment if you are low on your most critical resource, water. However eventually you must eat and frequently, if you are stranded, this means you must generate your own food sources Create Some Signals When you are stranded your chief objective should be to make yourself as visible as possible to rescue parties. Once you have secured your surroundings make sure to immediately develop your “signalling network” Fire Light by night Smoke By day Mirrors or Reflective Items These are used to signal with when rescue is in sight. Sound or Noise Production These are used to attract attention in case vocal cords are dried and not functioning. Carving a whistle can be a worthwhile distraction while you wait to be rescued. The Experts Action Rankings 1. Give First Aid to the Injured 2. Decide on a Leadership Strategy and Start Planning What Needs to be Done 3. Send out Two explorers to research the Island 4. Seek Shelter in the Cove 5. Develop Techniques for Signalling 6. Look For Vantage points for signalling 7. Search for Food and Water 8. Do Nothing Until the Storm has Subsided 367 Project DoCTA: Design and use of Collaborative Telelearning Artefacts The Experts Salvageable Items Ranking 1. 6 x 14 blue canvas canopy Shelter and water collection. Parts may be cut from the main canopy to make solar water stills. This with the knife below is the most useful items on this list. With these two items the chances of survival and rescue reach 86%. 2. Swiss Army knife with multiple tools Best all round tool for survival. Shiny blade surface can act as a signal device. 3. One Dinghy Paddle Acts as a support for the canvas. 4. Seven cans of food Can be eaten, but once opened can also act as water containers. 5. 80' of nylon rope Can tie the canvas together - also acts as wicks for lights. 6. Two orange life jackets When set on fire these will make a large smoke trail which can be seen for many kilometres 7. Three flotation cushions Likewise can be burnt to raise a smoke trail. 8. One Scuba Mask The glass can be used as a light signal by catching the suns rays from the glass. Might also be useful to smash and have additional knives. The water is so deep and cold death by hypothermia would be inevitable for anyone attempting to use the mask to swim. 368 Appendicies 9. Various Pieces of rigging wire Might try to use these for traps, but frankly the chances of catching anything large enough to reward the effort is unlikely. 10. One standard size plastic cooler Good for storing water and fresh food - but since both will be very rare you might use it as a basket to carry firewood from the trees and driftwood. 11. One pair of flippers Could be used for digging or as smoke producing fuel. The water is so deep and cold death by hypothermia would be inevitable for anyone attempting to use the flippers to swim. 12. Four Bottles of Suntan Oil You could use these as fuel to raise smoke from the fire or as fuel for wick based lights using lengths of the nylon cord as a wick. 13. Two Six Packs of Beer If you poured the beer away and used the cans to store water they might be useful. The danger in drinking the beer is the dehydration caused by the alcohol. 14. Four Wrist Watches Might be useful for signalling using the watch glass. 15. Yachtman’s guide to the fjords Might be used for fuel for starting the fire or toilet paper. The danger would be if the book tempted people to try and escape. It is almost certain they would perish in the icy waters of the fjord. 369 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix W: Help Page Screen Shots Figure 67 Help page for the File Holder tool Figure 68 Help page for the ToDoList tool 370 Appendicies Figure 69 Help page for the Page function Figure 70 Help page for how to Save Rooms 371 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix Y: Administration Interface in TeamWave Workplace These are additional screen shots of the Administration Interface in TeamWave Workplace Figure 71 372 Administrator interface for administering conferences Appendicies Figure 72 Administrator interface for administering templates Figure 73 Administrator interface for administering repositories 373 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix Y: Usability Study Interview Guide Goal of the interview The goal of the interview is to get feedback on how efficient TeamWave has been with respect to achievement of goals and what was needed/ required to reach these goal. Various functions First, I will present various functions that I think a groupware tool should provide. Then, you will get to give your feedback with respect to how you think Teamwave fulfilled these functions. The functions are: Communication: The tool should faciltate both synchronous and asynchronous communication between the team members. Administration: The tool should provide functions to support the team in administration tasks such as scheduling meetings, distribution of work tasks etc. Production: The tool should support the team in production of artefacts such as text documents, images, web ages etc. Presentation: The tool should facilitate the presentation and exchance of various types of information between the team members. Navigating: The tool should provide guidance for how to use the tool and how to manouver within it. 374 Appendicies Synchronous communication • To what extent do you think TeamWave made it possible for the group to communicate synchronously? • What was the quality of this communication? • Which tools did you use for synchronous communication? • Do you have any suggestions for ways the synchronous communication could have been improved? Asynchronous communication • Did TeamWave make it possible for the team to communicate asynchronous? • What was the quality of this communication? • Which tools did you use for asynchronous communication? • Do you have any suggestions for ways the asynchronous communication could have been improved? Administration • Do you think that TeamWave supported the team in the administration of the team work? • If no: • How did you administrate the teamwork? • Otherwise: • Which tools did you use to support administrative tasks? • Are you satisfied in the way the team administrated the work, or did you experience any problems? Production • Did you, through the support of TeamWave, manage to produce the elements you wanted to include in the task (such as documents, images etc.)? • Which tools did you use to support these tasks? • How well did these tools support production? • Please report tools that you felt were missing, if any Presentation • To what extent did TeamWave support you in presenting information for other team members? • Which tools did you use as aids when presenting information? • How well did these tools perform? 375 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Navigating • Was it ok to navigate within the various TeamWave rooms? • How did you manage in finding the other members of your team (easy or difficult)? Help • To what extent did you use TeamWave’s help system? • Did you use and of the web based help pages? TeamWave in general • Are there any other aspects of TeamWave that you want to comment on? (not necessarily the specific tools, but rather some words on your experiences in using TeamWave as a GroupWare tool. Finally • Did you make use of the team addresses when sending email to each others, or did you also make use of private email addresses? • Are you aware of any insecurities within the team with respect to what was meant with ”collaboration” (such as everyone having to be logged on TeamWave simultaneously to be able to take part in the collaboration)? 376 Appendicies Norsk Formålet med intervjuet Det jeg ønsker å se på i dette intervjuet er hvor effektivt dere synes TeamWave har vært. Når det gjelder effektivitet vil jeg se på både effektivitet i forhold til måloppnåelse og effektivitet i forhold til hva som kreves for å nå dette målet (efficiency). Ulike funksjoner Jeg vil først ta for meg ulike funksjoner som jeg mener et gruppeverktøy bør ha, og se på hvor godt du synes TeamWave fylte disse funksjonene. Disse funksjonene er: Kommunikasjon: At verktøyet gjør det mulig for gruppemedlemmene å kommunisere både synkront og asynkront. Administrasjon: At verktøyet støtter gruppen i administrering av gruppearbeidet, f.eks avtale møter, fordeling av oppgaver o.l. Produksjon: At verktøyet gjør det mulig for gruppen å produsere noe, f. eks tekstdokumenter, bilder, websider o.l. Presentasjon: At verktøyet gjør det mulig for gruppemedlemmene å presentere ulike typer informasjon for hverandre. Navigering: At verktøyet støtter gruppemedlemmene i å finne frem internt i verktøyet. (evt også eksternt?) Synkron kommunikasjon • I hvilken grad synes du at TeamWave gjorde det mulig for gruppen å kommunisere synkront? • Hvor godt synes du denne kommunikasjonen fungerte? • Hvilke verktøy brukte dere for å kommunisere synkront? • Har du noen forslag til hva som kunne vært gjort for at den synkrone kommunikasjonen kunne fungert bedre? 377 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Asynkron kommunikasjon • Synes du at TeamWave gjorde det mulig for gruppen å kommunisere asynkront? • Hvor godt synes du denne kommunikasjonen fungerte? • Hvilke verktøy brukte dere for å kommunisere asynkront? • Har du noen forslag til hva som kunne vært gjort for at den asynkrone kommunikasjonen kunne fungert bedre? Administrasjon • Synes du TeamWave støttet gruppen i administreringen av gruppearbeidet? • Hvis nei: • Hvordan administrerte de gruppearbeidet? • Ellers: • Hvilke verktøy brukte dere til dette? • Er du fornøyd med måten gruppen fikk administrert gruppearbeidet eller fikk dere noen problemer? Produksjon • Klarte dere ved hjelp av TeamWave å produsere de elementene dere ønsket å ha med i oppgaven? (f.eks. dokumenter, bilder o.l.) • Hvilke verktøy brukte dere til dette? • Hvor godt synes du disse verktøyene fungerte til dette? • Var det noen verktøy/funksjoner du savnet? Presentasjon • I hvilken grad synes du at TeamWave gjorde det mulig for deg å presentere informasjon for andre? • Hvilke verktøy brukte dere til dette? • Hvor godt synes du disse verktøyene fungerte? Navigering • Gikk det greit å navigere mellom de ulike rommene i TeamWave? • Hvordan synes du det var å finne fram til de andre medlemmene i gruppen din? (lett eller vanskelig) Hjelp • I hvilken grad brukte dere hjelpesystemet internt i TeamWave? • Brukte dere de web-baserte hjelpesidene? 378 Appendicies Generelt om TeamWave • Er det noen andre aspekt ved TeamWave enn de vi har snakket om til nå du ønsker å kommentere? (Da tenker jeg ikke nødvendigvis på de spesifikke verktøyene, men mer generelt om hvordan du synes det har vært å bruke TeamWave for å løse oppgaven deres.) Avslutning • Brukte dere team-adressen når gruppen sendte mail til hverandre, eller brukte dere også private mail-adresser? • Følte du at det var noe usikkerhet i gruppen om hva som menes med collaboration? (f.eks. at alle måtte være logget på TW samtidig for å samarbeide) 379 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix Z: Interview Guide for How Students Organise their Work (English) Interview #: Date: Interviewee: Interwiew Type: General • How has the assignment work and the group functioned? * With respect to geographical distribution * TW vs face-to-face meetings * Difference from other collaboration forms - advantages/ disadvantages Division of labour, roles, rules • Who did what? * Why? * How? • How was the division of labour decided? * Discussion? * Previous experience/education? * Wishes? * Agreement? • Meets and collaboration * Group vs individual * Showing up at meetings - problems? (Especially in the first phase, did this have implications for collaboration in later phases?) * Why did you work so much on your own? (Easier, collaboration problems?) • Interdependencies vs flexability 380 Appendicies Communication • Communication forms * Mail, TW - Others? Why? * TW – advantanges, disadvantages, problems * Server problems – How did the server work? * Geographical distribution – What did this mean? • General on TW as a communications tool Integration • How did the assignment work function in relation to your other everyday tasks? * Job, family, school, free time activities • How did this effect the assignment work/group collaboration? * Division of labour * Meeting frequency/Meeting times * Communication forms * Comprehensiveness of the assignment (did you have to reduce the comprehensiveness?) Own Background • Experience, knowledge, education * Was your knowledge valuable, was it used? * Were you able to contribute things that the others did not know about? * Did you/your group make use of knowledge the others had? * Genuine interdependence? Goal Setting • What did you understand the goals of VisArt to be? • What was your goal for participation in VisArt? Was it changed underway? If yes, when, why? • Engagement, what kept you engaged/not engaged in the work?? Evaluation of VisArt / Own experience • What is your experience with VisArt? Help Functions • Did you use any help functions? 381 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Intervjuguiden (Norsk) Intervju #: Dato: Informant: Intervjutype: Generelt • Hvordan har arbeidet med oppgaven og gruppen fungert? * Med hensyn til geografisk spredning * TW vs face-to-face møter * Forskjell fra andre samarbeidsformer - fordeler/ulemper Arbeidsdeling, roller, regler • Hvem har gjort hva? * Hvorfor? * Hvorda? • Hvordan ble arbeidsdelingen fastsatt? * Diskusjon? * Tidligere erfaring/utdannelse? * Ønsker? * Enighet? • Møter og samarbeid * Gruppe vs enkeltperson * Oppmøte - problemer? (Spesielt i oppgavens første del, hadde dette betydning for samarbeidet i senere faser?) * Hvorfor så mye arbeid på egenhånd? (Enklere, samarbeidsproblemer?) • Interdependencies vs fleksibilitet Kommunikasjon • Kommunikasjonsformer * Mail, TW - andre? Hvorfor? * TW - fordeler, ulemper, problemer * Serverproblemer - hvordan virket det inn? * Geografisk spredning - hva betyr det? • Generelt om TW som kommunikasjonsverktøy 382 Appendicies Integrasjon • Hvordan fungerte arbeidet med oppgaven i forhold til andre gjøremål i hverdagen? * Jobb, familie, skole, fritidsaktiviteter • Hvordan påvirket dette arbeidet med oppgaven/gruppearbeidet? * Arbeidsdeling * Møtefrekvens/-tidspunkter * Kommunikasjonsformer * Oppgavens omfang (Reduserte oppgavens omfang etterhvert) Egen bakgrunn • Erfaringer, kunnskap, utdanning * Ble din kunnskap utnyttet, ble den brukt? * Bidro du med ting de andre ikke kunne noe om? * Dro du/gruppen nytte av kunnskap de andre satt inne med? * Genuine interdependence? Målsetting • Hva oppfattet du som målsettingen med VisArt? • Hva var din målsetting med å delta i VisArt? Ble den endret underveis? Hvis ja, når, hvorfor? • Engasjement, hva gjorde at du engasjerte deg / ikke engasjerte deg i arbeidet? Vurdering av VisArt/egne erfaringer • Hva er dine erfaringer med VisArt? Hjelpefunksjoner Brukte du noen hjelpefunksjoner under arbeidet? 383 Project DoCTA: Design and use of Collaborative Telelearning Artefacts Appendix AA: Interview Guide for How Instructors and Facilitators Organise their Work (English) Interview #: Date: Interviewee: 1. General How has working with the organisation of VisArt functioned? - What has been going on, in general - Who has been involved - Planning and background 2. Division of labour: Who has been doing what in the scenario? (Generally speaking) - How did you arrive at the particular division of labour. - What have been your tasks? (Specifically.) - What has been your role in the scenario? - Can you give examples of tasks that have been routinized in the duration of the scenario? 3. Communication What forms of communication have been used? - What tools were used in communication? - Were there any specific challenges in communicating with the District Colleges? - Did you use TeamWave to inform yourself about the work of the students? - Did you use TeamWave to communicate with the other instructors? 4. Rules, tacit/explicit: Have there been any specific rules of interaction between the students? - Tacit/ Explicit 384 Appendicies 5. Social context: What other work has occupied you during the scenario? - How has it influenced the work on VisArt? - What was your background before taking part in the scenario? 6. Goal Setting What was your motivation for taking part in the scenario? - What was the goal for the VisArt scenario? Intervjuguide for VisArt Interview #: (Norsk) Dato: Informant: 1. Generelt: Hvordan har arbeidet med organiseringen av VisArt-scenariet fungert? - Hva har foregått, generelt sett? - Hvem har vært involvert? - Planlegging og bakgrunn 2. Arbeidsdeling: Hvem har gjort hva i scenariet? (generelt.) - Hvordan ble man enige om fordelingen av arbeidet? - Hva har vært din oppgave? (spesifikt.) - Hvilken rolle har du hatt i scenariet? - Kan du gi eksempel på typer av arbeidsoppgaver som har blitt rutinisert i gjennomføringen av scenariet? 3. Kommunikasjon: Hvilke kommunikasjonsformer ble benyttet? - Hvilke verktøy ble benyttet i kommunikasjonen? - Var det spesielle utfordringer knyttet til å kommunisere med høyskolene? - Ble TeamWave brukt til å gi tilbakemelding om hva studentene foretok seg ? - Ble TW brukt til å kommunisere instruktørene imellom? 385 Project DoCTA: Design and use of Collaborative Telelearning Artefacts 4 Regler, uformelle/formelle Har det eksistert spesielle regler instruktørene imellom? - Uttalte/uuttalte 5. Kontekst: Hvilke andre gjøremål har informanten hatt i løpet av scenariet? - Hvordan har det påvirket arbeidet med VisArt? - Hvilken bakgrunn har informanten før deltakelse i scenariet? 6. Målsetning Hva var din målsetning med å delta i VisArt scenariet? - Hva var målsetningen med VisArt scenariet? 386