D4.4 Final evaluation report

(includes both the assessment of the result and the strategic evaluation)

– M23

Status: Submitted

Download

Deliverable

Abstract

This document describes the summative and strategic evaluations for the AtGentive project. The project consists of the development of attention-aware versions of the Ontdeknet eLearning platform for children aged 8-15 (to produce AtGentSchool) and INSEAD’s collaborative learning platform for adults (to produce AtGentNet).

Summative evaluation began with usability testing of the software, using expertconducted heuristic evaluations, to ensure a suitable level of usability. The purpose of using heuristics as a software evaluation tool is to validate the software as being to appropriate usability standards separate to its effects on the users. The primary phase of summative evaluation comprises an extended period of direct user testing. AtGentSchool was tested over a six week period by classes in five schools in the Czech Republic (one or two classes per school). The AtGentSchool platform creates virtual learning relationships between subject experts and students, and provides guidance to support individuals to learn together based upon common interests.

AtGentNet was tested over a five month period by business people, across Europe and beyond, enrolled in a business-related distance learning course organised by the Swedish Trade Council (STC). Learners attend seminars and meet with local tutors who provide expert advice and assistance to help learners set and achieve practical goals. The process is supported by the AtGentNet platform which provides both access to information and a social forum.

The criteria for success were based upon three scenarios, implemented for each system (AtGentSchool and AtGentNet). These scenarios relate to the guidance of learning, the (re)direction of attention and user-notification. Research questions were generated from these scenarios, leading to five key indicators: Attention, Performance, Satisfaction, Learning and Collaboration. These indicators are used as part of the summative evaluation to assess the benefits gained by incorporating attentive agents.

In terms of overall success, the data indicate that use of an animated agent for children can successfully promote Performance, Satisfaction and Collaboration, with no apparent detriment to other factors. The use of scenarios as design elements proved very effective in designing for children. For adults, careful and subtle perceptual enhancements appear to be a better approach than animated agents. The data here indicate that performance and collaboration may be enhanced, but only where the motivation pre-exists. Perceptual-based attention support does not engender motivation, compared to the motivation provided to children by the animated agent in AtGentSchool.

Strategic evaluation examines the real contribution of AtGentive in the outside world. The primary project outputs are identified, along with their strengths and weaknesses. Overall, the project approach of attacking the problem from several perspectives corresponding to the strengths and expertise of individual participants has proved very fruitful. AtGentive has met its principle objectives to successfully design and run two pilot studies which advance the support and understanding of attention with regard to educational software. While the support is situated in particular applications, knowledge gained may have implications in a wider context. This work has contributed across several disciplines in the research community – teaching and learning, collaborative systems, human- computer interface, and intelligent agents to mention but a few – and offers a basis for continuing research in this new and expanding area.

Table of Content

1. INTRODUCTION ........................................................1
1.1 General introduction to AtGentive...................................1
1.2 Introduction to this document.......................................2
1.3 Introduction to the Conceptual Framework............................2
1.4 Introduction to Platforms...........................................4
1.4.1 AtGentSchool .....................................................4
1.4.2 AtGentNet.........................................................5
1.5 Introduction to Summative Evaluation................................7
1.5.1 Approach .........................................................7
1.5.2 Methodology.......................................................8
1.5.3 Heuristics .......................................................8
1.5.4 Evaluation Framework..............................................8
1.5.5 User evaluation ..................................................8
1.5.6 Ethical Issues ..................................................10
1.5.7 Additional experiments...........................................10
1.6 Introduction to Strategic Evaluation...............................11
2. SCENARIOS TO PILOT STUDY............................................12
2.1 AtGentSchool.......................................................12
2.1.1 Scenario S1 – Guidance of Learning ..............................12
2.1.2 Scenario S2 – Idle User .........................................12
2.1.3 Scenario S3 – External events ...................................12
2.1.4 Implementation...................................................12
2.1.5 The animated Agent...............................................17
2.1.6 Continual development............................................17
2.1.7 User Evaluation Pilot............................................17
2.1.8 Timetable of events..............................................18
2.2 AtGentNet..........................................................18
2.2.1 Scenario N1 – (Initial) learning guidance........................18
2.2.2 Scenario N2 – Notify tutor of user (in)activity..................19
2.2.3 Scenario N3 – Notification of events.............................19
2.2.4 Implementation...................................................19
2.2.5 The animated Agent...............................................20
2.2.6 Continual development............................................21
2.2.7 User Evaluation Pilot............................................21
2.2.8 Timetable of events..............................................21
3. TOOLS FOR SUMMATIVE EVALUATION .....................................23
3.1 Description of users and tasks.....................................23
3.1.1 Description of users - AtGentSchool .............................23
3.1.2 Description of tasks - AtGentSchool .............................23
3.1.3 Description of users - AtGentNet ................................23
3.1.4 Description of tasks - AtGentNet.................................24
3.2 Derivation of the Key Indicators...................................24
3.3 Context, Interaction, Attitudes and Outcomes ......................26
3.4 ISO 9241 for Satisfaction and Performance .........................27
3.4.1 Overview ........................................................27
3.4.2 Effectiveness, Efficiency, Satisfaction .........................28
3.5 Heuristics ........................................................28
3.6 Questionnaires.....................................................30
3.6.1 AtGentSchool ....................................................30
3.6.2 AtGentNet........................................................31
3.7 Log files .........................................................31
3.8 Interviews and Focus Groups .......................................32
3.9 Pilot Evaluation criteria .........................................32
4. SUMMATIVE EVALUATION - RESULTS .....................................33
4.1 Heuristic Evaluations..............................................33
4.1.1 AtGentSchool ....................................................33
4.1.2 AtGentNet........................................................33
4.2 Pilot study – AtGentSchool ........................................33
4.2.1 Teachers’ experience ............................................33
4.2.2 Results from Pilot ..............................................41
4.2.3 Analysis ........................................................55
4.3 Pilot study - AtGentNet ...........................................60
4.3.1 Results from Pilot ..............................................60
4.3.2 Analysis ........................................................74
4.4 Overall conclusion.................................................77
5. ADDITIONAL EXPERIMENTS ON ATTENTION FOR ONLINE LEARNING ............79
5.1 Acceptance of agents’ instructions.................................79
5.2 Animated agent’s gestures verbal discrepancy.......................79
5.3 Animated agent’s gestures and guidance of user’s attention on screen ...80
5.4 AtGentNet eye-tracking study.......................................80
5.5 General applicability of the conceptual framework..................81
5.6 General applicability of the Reasoning Module .....................82
6. STRATEGIC EVALUATION................................................83
6.1 Project Objectives ................................................83
6.2 Project Outputs ...................................................84
6.3 Key Assessors for evaluating project impacts ......................84
6.4 Applying the Key Assessors.........................................85
6.4.1 Literature survey (State of the Art).............................85
6.4.2 Conceptual framework.............................................86
6.4.3 ASKME module ....................................................86
6.4.4 Reasoning module ................................................87
6.4.5 Agents (animated characters).....................................88
6.4.6 AtGentSchool ....................................................89
6.4.7 AtGentNet........................................................90
6.4.8 Results of pilots ...............................................90
6.4.9 Physiological experiment results ................................91
6.4.10 Publications – papers / web site................................91
6.4.11 Student experience on project...................................92
6.5 Strategic Evaluation – Meeting the Project Objectives .............92
6.6 Strategic Evaluation - Conclusion .................................93
7. REFERENCES .........................................................94

8. APPENDIXES...........................................................1
8.1 Appendix 1 - Usability heuristics ..................................2
8.1.1 Established heuristics ...........................................2
8.1.2 Additional AtGentive heuristics ..................................3
8.2 Appendix 2 – AtGentSchool - Abstract concepts and questions for Questionnaires..4
8.3 Appendix 3 – AtGentNet - Abstract concepts and questions for Questionnaires.....9
8.4 Appendix 4 – AtGentSchool – Results of the Heuristic evaluation.....12
8.4.1 General comments .................................................12
8.4.2 Established heuristics (Nielsen, 2006)............................12
8.4.3 Additional AtGentive heuristics ..................................14
8.4.4 Other ............................................................16
8.5 Appendix 5 – AtGentNet – Results of the Heuristic evaluation .......17
8.5.1 General comments .................................................17
8.5.2 Established heuristics (Nielsen, 2006)............................17
8.5.3 Additional AtGentive heuristics ..................................21
8.5.4 Other ............................................................22
8.6 Appendix 6 – AtGentSchool – Pre- and post-test questions ...........23
8.7 Appendix 7 – Summary of Academic Dissemination Activities ..........25
8.7.1 White papers / conference submissions / publications:.............25
8.7.2 Conferences (conferences, presentations, posters): ...............26
8.7.3 Non-academic level presentations .................................27
8.7.4 Others ...........................................................27
8.7.5 Web presence......................................................28
8.7.6 Exhibitions.......................................................28
8.8 Appendix 8 – Acceptance of agents’ instructions (OBU)...............29
8.9 Appendix 9 – Animated agent’s gestures verbal discrepancy (OBU).....32
8.10 Appendix 10 – Animated agent’s gestures and guidance of user’s attention on screen (UTA) ...33
8.11 Appendix 11 – AtGentNet eye-tracking study (UTA) ..................34
8.12 Appendix 12 – Evaluation of the level of general applicability of the conceptual framework: restoring context (AUP)...40
8.13 Appendix 13 – Evaluation of the level of general applicability of the Reasoning Module(AUP) ...45
8.14 Appendix 14 – Additional pedagogical Analysis (AUP)................48

back to WP4, Deliverables