Using IT to assess IT: Towards greater authenticity in summative performance assessment

An applied Information Technology (IT) course that is assessed using pen and paper may sound incongruous but it is symptomatic of the state of high-stakes assessment in jurisdictions such as Western Australia. Whereas technology has permeated most aspects of modern life, including schooling, and more has been demanded of education systems in terms of outcomes and participation, methods of summative assessment have changed little and are seriously out of alignment with curriculum, pedagogy and the needs of individuals and society. This paper reports on an analysis of some of the data from a component of a study into the feasibility of using digital technologies to achieve greater authenticity in summative performance assessment in the Applied Information Technology (AIT) course in Western Australian secondary schools. In the first phase of the study a sample of 115 students completed a digital portfolio and a computer-based exam that were both externally assessed using online tools and by two methods of marking, with the results analysed using Rasch modelling software. A traditional analytical method and a comparative pairs method of marking were investigated. The study found that both the digital portfolio and computer-based exam were implemented without significant technical difficulty and were well accepted by the students and teachers. The work output in digital form was readily accessed from an online repository by external markers using a standard web browser. The two methods of marking provided highly reliable scores, with those from the comparative pairs method being the more reliable. A number of questions of validity and manageability were raised and the strengths and weaknesses of the two forms of assessment revealed. It was concluded that it was feasible to implement either form of assessment for high-stakes purposes, with a resulting improvement in alignment and authenticity.