Publish and be Damned? Publicising student feedback and the quality process
چکیده
Most higher education institutions have collected feedback data from their students since the late 1980s, yet the issue of publication of such data has recently caused widespread controversy within the British higher education system. Although publication of student feedback can provide useful information for stakeholders such as prospective students, such a requirement might also be invidious and damaging to institutions of higher education. With the recent announcement by the Cooke committee that feedback, along with other documentation about HEIs, should be made publicly available, this paper is a timely examination of the issues surrounding the publication of student feedback and investigates some of approaches (and lessons) that currently exist. This paper examines the advantages, drawbacks and barriers to publishing student feedback from the viewpoint of different institutions. It analyses the different ways in which HEIs publish feedback from their students and the different rationales for doing so, examines the different methods used in publishing feedback and what elements are published. Introduction The announcement by Sir Ron Cooke’s task force in November 2001 that student feedback should be amongst the information that higher education institutions make publicly available has met with at best guarded approval and not a little opposition from university officials and academics (HEFCE, 2001b; Elliot Major, 2001; Baty, 2001). For several commentators, however, publication of such data is essential in making higher education more accountable. For example, it was argued some years ago that Accountability not only requires that institutions are open to scrutiny but that the outcomes of scrutiny are published. Publication must, ultimately, be in a form that is readily assimilated by all stakeholders. (Colling and Harvey, 1995, p.30) The Better Accountability report of 2000 argued, in addition, that streamlined information flows assist in creating greater accountability (HEFCE, 2000). Few seem particularly unhappy about the collection of feedback data, yet why does the issue of publication cause such controversy? What is meant to be made public? What is it appropriate to make public? What are the issues surrounding publication of feedback? Are there advantages? What are the implications? Once such questions have been tackled, and agreement reached, there are still some fundamental questions about practicalities. Who should have responsibility for making it public? What approaches to publicising feedback currently exist, and what has been learnt from these? What exactly is to be made publicly available? The original joint HEFCE, QAA, UUK and SCP proposal was for the publication of student views on a range of issues including courses, facilities and general university environment, in addition to summaries of external examiners’ reports and programme specifications. The publication of feedback was one of the five key points for discussion, and is intended To provide students, employers and others with reliable and consistent information about quality and standards at each higher education institution (HEFCE, 2001a, p. 3). The consultation document recommended that a range of issues should be considered in relation to public information (HEFCE, 2001a, p. 6), and student feedback was but one of the many sets of information that should be made available. The Cooke Report claimed that publication of feedback would provide information on arrangements for academic and tutorial guidance, support and supervision; library services and IT support; suitability of accommodation, equipment and facilities for teaching and learning; perceptions of the quality of teaching and the range of teaching and learning methods; assessment arrangements; quality of pastoral support. Student satisfaction surveys, argued the Task Force, should at least be available internally as a basis for institutional audit discussions. It is recognised in the report that feedback from students is widely recognised as being a valuable measure of the quality of programmes, and should therefore take a more prominent role in the quality assurance process. Arguably – two approaches that can run concurrently: publication of student experiences of institutions and faculties; internal publication of modular feedback? Types of feedback Most institutions of higher education across the world do now collect feedback from their students, but there is very little standardisation, except where a generic model has been adopted. Hence, although student feedback is frequently referred to, it is not always clear what is meant by it. There are a variety of different types of student feedback, as described by a number of authorities. According to Harvey (2001a, p. 1), feedback refers to students’ expressed opinions about the service they received as students, and includes perceptions about Learning and teaching; Learning support facilities; Learning environment; External aspects of being a student. There are two different approaches to collecting such data, although most higher education institutions ask their students to comment on their satisfaction with their whole learning experience. First, institutions may simply collect student views on range of facilities and teaching experience. This has now been standard practice in most institutions of higher education since the late 1980s (Harvey, 1999). Second, institutions may develop a full cycle of data collection, reporting and action, although this, as Harvey notes (2001), is far less common. Fears and reservations There is no doubt that the prospect of publicising student feedback has met with only reserved approval from higher education institutions themselves. Cooke’s own committee recognised that the issue was controversial: ‘Many had reservations about the suggestion that summaries of the results of internal processes ... might be published’(HEFCE, 2001c, p. 4). The reason for such doubts were twofold. First, respondents ‘saw this as potentially compromising the rigour and candour of those processes’. Second, some respondents argued that publication of such information would be of ‘limited utility to stakeholders anyway’. Even during a workshop on Student Satisfaction, held at the University of Central England (UCE) in February 2002, the majority of delegates, who came from a variety of professional and academic backgrounds, and supported the development of a full feedback cycle, were only reservedly in favour of publication of student feedback. There are five principal issues of concern that normally arise in discussions of public information and the use of student feedback: that publicising feedback would lead to the creation of league tables; that the whole process would simply be an exercise in cost cutting; that the system is recriminatory; that students’ views are prejudiced and therefore unreliable as a benchmark of quality; the burden would be too expensive. Experience has demonstrated that there is a very real danger that the publication of student feedback would create league tables, as universities receive ‘scores’. Teaching staff in the Australian higher education system have been deeply resentful of publication of this sort, where league tables are available annually in the press (Gosling and D’Andrea, 2001, p.14). Student feedback in this case allows for the creation of statistical data. Statistical values can, of course, be deeply misleading. Without careful explanation and breakdown, they cannot provide stakeholders or public at large with a realistic view of the state of individual institutions or departments. In particular, such league tables can be used, as they are currently, as a basis on which to organise funding distribution. Arguably, this already occurs through the RAE system. The result in this case is continued deprivation for universities that already suffer from funding difficulties. Second, there is a fear amongst academic staff that the publication of feedback is part of an exercise in cost-cutting. Departments or universities that score badly may more easily be deprived of funds, reduced in size or even closed altogether (Trow, 1994, p. 20). Currently, departments of non-main-stream disciplines that traditionally attract fewer students suffer financially, and their future in many universities is uncertain. Publication of student feedback, it is feared by some academics, might be used to reinforce this process. Third there is a fear amongst academics that the use of student feedback in general can be used as an instrument of recrimination, both against departments and more importantly, against individual lecturing staff (Rustin, 2000, p. 86, 98). It is feared that the publication of student feedback would simply reinforce this process. Commentators have argued that the quality process can only be successful if it is seen to be non-recriminatory (Harvey et al, 1997; Gosling and D’Andrea, 2001). Finally, it is argued by some academics that as student views are prejudiced, they are therefore unreliable as one of the chief benchmarks of quality assessment of higher education (Todd, 2002). Clearly, individual animosities may affect lecturer appraisals, but recent studies suggest that there are deeper influences. Gender in particular, has been shown to influence student evaluations than has previously been assumed (Basow, 1994). The publication of student views could therefore, it has been argued, ‘undermine the balance between fair comment and inbuilt bias’ (Baker, 2002). Overall, fears are derived principally from two sources. Publicising student feedback may be used as a recriminatory tool, one that could be used to punish already pressured academics, and as a means for allocating funds, thereby depriving institutions that score badly of much needed resources. An extra financial burden? Cooke’s report notes that ‘Some respondents saw an increase in burden in the proposals in the consultation document, and pointed to ... the costs of data collection’ (HEFCE, 2001b, p. 15). In answer to this, the report argued that it is already common practice for higher education institutions ‘to conduct student feedback surveys through questionnaires and that the collection of data would not in itself constitute a new burden’. Concerns have been raised, however, by such organisations as the Better Regulation Task Force (BRTF), that such a requirement would impose a significant and ‘unnecessary’ financial burden on already stretched institutions (BRTF, 2002, p. 15). New costs would certainly be incurred as the ‘preparation of results for publication would be a new task for most higher education institutions’ (HEFCE, 2001b, p. 15). Should a full feedback and action cycle be adopted it would undoubtedly entail significant financial and staff investment (Harvey et al, 1997). Full-time researchers would be required in order to carry through a full annual survey and write a suitable report. The cost, however, need not be overbearing. Some institutions, such as UCE or Sheffield Hallam University, possess a team that, in addition to the annual survey and subject evaluations, works on a variety of other sponsored projects and thereby bring in supplementary income. Other institutions employ a small staff that is dedicated to undertaking the annual survey. The Vaal Teknikon in South Africa, for example, employs two full-time staff to conduct its annual satisfaction survey (Brits, 2002). Information overload? Respondents to Cooke’s consultation document who were nervous about publicising feedback also pointed to ‘issues of information overload’. In the experience of UCE’s Centre for Research into Quality, students are unwilling to devote time to completing feedback questionnaires for two main reasons. First, some institutions require the completion of many questionnaires and overevaluation can be a cause of ‘questionnaire fatigue’ (Harvey, 2002, p. 11). Certainly, this issue is a significant and destructive one for researchers undertaking satisfaction feedback surveys (Bowes et al, 2001, p. 2). Some institutions already conduct surveys on a range of issues and find that response rates are generally low, even after reminder letters have been sent out. Where surveys are conducted more sparingly, a general student satisfaction survey seems to be more enthusiastically received. Second, the length of questionnaires has proved to be an important factor in encouraging or discouraging students to complete them. Experience of satisfaction surveys has also shown that shorter questionnaires achieve greater response rates than longer ones. Conversely, the value of feedback to the user, in particular potential students, is far from clear. University applicants are confronted with an increasingly dazzling array of information, but it is unclear how far they will be willing or able to make efficient use of the available material. League Tables Quality assessment is inherently fraught with the danger of creating league tables. Publication of results will, perhaps inevitably, create ‘leagueing’. As mentioned above, this has already occurred in Australia, and in general, results are believed by many academics to harm rather than improve the higher education system (Bowden, 2002). It has already produced a similar effect in Britain with the publication of the RAE tables. League tables are most likely to arise when results are statistical, simply because numerical figures may be merged and overall results be calculated, without reference to practical considerations. In the Student Satisfaction model, however, the danger of league table creation is at least alleviated if not completely eliminated by the use of ‘A-E tables’ rather than statistical information, as it discourages the appearance of league tables. Whilst departments or facilities may ‘score’ badly on individual points, such as the time taken by tutors to return feedback, library loan periods, student-union décor, or car-parking, no overall ‘average’ can be reasonably calculated. The use of the A-E table is, therefore, too diffuse to allow overall comparisons to be made between institutions as a whole, but can provide valuable information on specific issues affecting students and their total learning experience.