Evaluation of Learning Technology Implementation

Learning technologies present the opportunity to augment or replace the role which was traditionally played by the teacher. In a learning-led educational context the effectiveness of teaching, whether by the teacher, or through interaction with media, can be readily evaluated by the degree to which it contributes to learning. The use of level descriptors, such as Biggs’ SOLO taxonomy, provides a powerful tool to assess the outcomes of learning, and by implication the effectiveness of the learning activity. Contextual factors which correlated with the SOLO levels, and encourage deep approaches to learning, can be used to improve effectiveness. This paper outlines the SOLO taxonomy, indicates implications for implementation, and discusses the correlating factors which enhance students’ approach to learning. Purposes and scope of evaluation 'Evaluation is any activity that throughout the planning and delivery of innovative programmes enables those involved to learn and make judgements about the starting assumptions, implementation processes and outcomes of the innovation concerned' Stern [1988] Under this definition the principal purpose of evaluating learning technologies is to provide the designer or user with enough evidence on which to make confident judgements regarding the effectiveness of the innovation. From these judgements actions can be based which will result in the redesign or adaptation of the implementation to improve its performance. Judgements of the 'effectiveness of the innovation' are in effect judgements of the extent to which the innovation meets its identified aims its fitness for purpose. It is important therefore that these intentions are thought through and clarified at the design stage of the project good evaluation practice is designedin, not appended, to projects. There are three stages in innovative projects about which judgements may be made: intentions, implementation and outcomes. The match between the outcomes and intentions provides a measure of the success of the innovation. If outcomes do not match the intentions, then it is readily observed that something needs to be done: perhaps implementation is inappropriate, or perhaps the starting assumptions should be revisited and revised. Even when outcomes closely match the intentions, and the project is counted successful there may be assumptions, behind the stated aims of the innovation, often implicit and unstated. An innovation may be judged as 'fit for purpose' while the fitness of the purpose itself is questionable. It is important therefore to be as explicit as possible about the starting assumptions, in order that the evaluation of design and implementation can be most useful. To take more concrete examples: a learning package might be evaluated against a number of intended outcomes. These might include: • ease of use can the target users navigate and interact with the product in the expected way? • efficiency does the product provide efficient coverage of the appropriate curricular unit? Can more multiple learners access the content than by other means? • preferences do learners enjoy it? will they choose to use it if it is offered? to what extent does it cater for variety of user? • attractiveness is it sufficiently attractive to persuade purchasers/tutors to choose it over alternatives • cost effectiveness is it cost effective in comparison with alternative means of achieving the same intended outcomes? However, at another level there are assumptions about the nature of learning and learners, which will shape and define the answers to all these evaluative questions. It is to these assumptions that I wish to draw attention below assumptions about learning at tertiary level which I believe are shared by most teachers, and which provide a necessary aspect of evaluation of any learning technology, alongside any of the aspects outlined above.