The Wayback Machine - https://web.archive.org/web/20030309125404/http://www.hefce.ac.uk:80/pubs/hefce/1996/m19_96.htm
HEFCE logo Section banner
Section banner

HEFCE

Quality Assessment and Quality Improvement: An analysis of the Recommendations made by HEFCE Assessors

Reference M 19/96

Published June 1996

The electronic version of this document contains: the Contents; Lists of Annexes, Tables and Figures; Foreword and Executive Summary only. Printed copies of the complete publication are available from External Relations, HEFCE, price 7.00.

CONTENTS

FOREWORD

EXECUTIVE SUMMARY

1 INTRODUCTION

2 RECOMMENDATIONS MADE BY ASSESSORS OF 'SATISFACTORY' PROVISION

2.1 Summary of recommendations made
2.2 Similarities and differences in the recommendations made to institutions according to former higher education sector
2.3 Similarities and differences between subjects in the pattern of recommendations

3 RECOMMENDATIONS MADE BY ASSESSORS OF 'UNSATISFACTORY' PROVISION

4 ISSUES MOST COMMONLY RAISED BY ASSESSORS IN THEIR RECOMMENDATIONS

4.1 Teaching and learning strategy, methods and structure
4.2 Assessment strategy, methods and structure
4.3 Departmental and institutional quality assurance
4.4 Staff development
4.5 Student experience
4.6 Dissemination
4.7 Commentary

5 RECOMMENDATIONS ON INDIVIDUAL SUBJECTS

5.1 Anthropology
5.2 Applied Social Work
5.3 Architecture
5.4 Business and Management
5.5 Chemistry
5.6 Computer Science
5.7 English
5.8 Environmental Studies
5.9 Geography
5.10 Geology
5.11 History
5.12 Law
5.13 Mechanical Engineering
5.14 Music
5.15 Social Policy and Administration

6 CONCLUSIONS

METHODOLOGICAL ANNEX

A1 The quality assessment method
A2 The sample
A3 Method of analysis
A4 Development of an analytic framework
A5 Intercoder reliability
A6 Phase 2: analysis
A7 Phase 3: analysis of a subset of reports
Appendix 1: The coding frame

LIST OF TABLES

Table 1 Summary of recommendations: distribution (%) according to aspects of provision and subjects
Table 2 Subjects with the highest and the lowest proportion of recommendations in different aspects of provision
Table A1 Subject and institutional breakdown of 'satisfactory' reports

LIST OF FIGURES

Figure 1 Summary of recommendations: distribution (%) across all institutions
Figure 2 Summary of recommendations: distribution (%) according former HE sector
Figures 3 - 9 Subject differences (%) in the pattern of recommendations
Figure 10 Summary of recommendations on 'unsatisfactory' provision: distribution (%) according to aspects of provision
Figure 11 Anthropology recommendations
Figure 12a Applied Social Work recommendations
Figure 12b Applied Social Work recommendations by former HE sectors
Figure 13 Architecture recommendations
Figure 14a Business and Management recommendations
Figure 14b Business and Management recommendations by former HE sectors
Figure 15a Chemistry recommendations
Figure 15b Chemistry recommendations by former HE sectors
Figure 16a Computer Science recommendations
Figure 16b Computer Science recommendations by former HE sectors
Figure 17a English recommendations
Figure 17b English recommendations by former HE sectors
Figure 18 Environmental Studies recommendations
Figure 19 Geography recommendations
Figure 20 Geology recommendations
Figure 21a History recommendations
Figure 21 b History recommendations by former HE sectors
Figure 22a Law recommendations
Figure 22b Law recommendations by former HE sectors
Figure 23a Mechanical Engineering recommendations
Figure 23b Mechanical Engineering recommendations by former HE sectors
Figure 24a Music recommendations
Figure 24b Music recommendations by former HE sectors
Figure 25 Social Policy and Administration recommendations

FOREWORD

This document presents the results of the second in a series of projects sponsored by the Quality Assessment Division (QAD) of the Higher Education Funding Council for England (HEFCE) to explore overall conclusions from reports of quality assessments carried out over the last three years. This project's principal aim is to gain an overview of the recommendations for improvement, and to identify any striking characteristics or patterns.

The analysis, by a research team from the Quality Support Centre of the Open University, draws on the assessment reports graded 'satisfactory' and 'unsatisfactory' under the old HEFCE rating. Reports graded 'excellent' were analysed in the first study of the series, entitled 'Report on Quality Assessment: 1992-95'.

Both documents should be read in conjunction with the subject overview reports, written after the assessment of each academic subject. The cross-subject and cross-institutional comparisons of recommendations for improving quality contained in this document can be considered alongside the more detailed analyses of good practice and areas for improvement in the subject overview reports.

The conclusions to reports on satisfactory and unsatisfactory provision included two sections: commendations on good practice and elements regarded as excellent; and recommendations for action where quality could be improved. This analysis concentrates on the second and makes no reference to the many strengths identified in individual reports.

This document rightly signals that institutions themselves are responsible for acting on assessors' recommendations. Nevertheless, it will be of wider interest to the academic community, and to subject associations and professional bodies. In some subjects the distribution of recommendations is significantly different from the general pattern.

This report, together with the earlier report on excellent provision and research projects presently being commissioned by the HEFCE, reflects a strong commitment by the Council to exploit the potential of the quality assessment process to lead to improvements in teaching and learning practice. I welcome this document as a significant contribution to that objective.

Dr Paul M Clark
Director of Quality Assessment

HEFCE

EXECUTIVE SUMMARY

Assessment of teaching quality in higher education institutions in England and Northern Ireland is undertaken by the Quality Assessment Division of the Higher Education Funding Council for England (HEFCE) using a two stage process of institutional selfassessment and external peer review of provision at the subject level. During the period under review, each subject provider prepared a self-assessment which could include a claim to be providing excellent quality of education ('claim for excellence'). On the basis of analysis of the self assessments by academic and professional peer assessors, some providers received a three-day assessment visit carried out by an assessment team.

The assessment method addresses the wide range of influences that shape the experiences and achievements of students. It includes consideration of the breadth of teaching, learning and assessment activities, students' achievements, the curriculum, the application of learning resources (library, equipment, IT, laboratory), student support and guidance, and academic management at the subject level. The assessment visit includes scrutiny of students' work, direct observation of a wide range of teaching and learning activities, and discussions with staff and students.

Following each assessment visit, a Quality Assessment Report is published. The report gives the assessment outcome and includes a conclusion which highlights points of strength and excellence, and areas identified by the assessors for institutional attention.

Provision in 15 subjects ('units of assessment') was assessed in England and Northern Ireland between February 1993 and June 1995. The fifteen subjects were: Anthropology, Applied Social Work, Architecture, Business and Management Studies, Chemistry, Computer Science, English, Environmental Studies, Geography, Geology, History, Law, Mechanical Engineering, Music, and Social Policy and Administration.

In the period covered by this Report, the Council used a three-point assessment scale leading to an overall summative judgement of Excellent, Satisfactory or Unsatisfactory quality of education. No judgement of excellent or unsatisfactory quality was made without an assessment visit. Judgements of satisfactory quality were made either with or without an assessment visit.

In 1995 the Council published a Report of Quality Assessment 1992-1995 on the characteristics which peer assessors have associated with education judged to be excellent. The report suggested some common features, but also some differentiation in the characteristics of excellent education between subjects and between the two former sectors of higher education.

The purpose of the present project, as set out by the Council in the Invitation to Tender, was

"to examine the recommendations made by peer assessors for the improvement of the education that was graded satisfactory or unsatisfactory, and to identify the extent to which there are common or divergent elements between subjects and/or between institutions in the two former sectors of higher education (UFC and PCFC)".

Reports on provision judged to be satisfactory contain many points of strength and excellence which are not reported here. This project is concerned only with analysis of the areas identified by assessors for institutional attention in order to enhance the quality of the existing provision.

Over the period from the inception of quality assessment in 1992 to the introduction of a revised assessment method in April 1995, 553 assessment visits took place, of which 45% (249 reports) led to judgements of excellent quality, nearly 53% (292 reports) of satisfactory quality and just over 2% (12 reports) of unsatisfactory quality. At the time of the research, 11 assessment reports of unsatisfactory provision and 287 assessment reports of satisfactory provision were available. It is the recommendations contained in these 298 reports that provided the basis for the analysis described in this report.

The 287 assessment reports of satisfactory provision covered 15 subject fields and institutions from the former Universities Funding Council (UFC) sector (118 reports), the former Polytechnics and Colleges Funding Council (PCFC) sector (157 reports) and the further education(FE) sector (12 reports). they contained some 1806 recommendations to institutions of ways in which the quality of education might be improved. The 11 assessment reports of unsatisfactory provision covered 8 subject fields and concerned one former UFC sector institution, six former PCFC sector institutions and three FE institutions. These recommendations provide valuable guidance as to the areas and issues which need to be addressed in order to improve the quality of education provided by HEFCE funded institutions.

This report presents an overview of assessors' recommendations. The coding of recommendations utilised the six aspects of provision introduced in 1995 by the Council as part of its revised assessment method. The six aspects are: Curriculum Design, Content and Organisation; Teaching Learning and Assessment; Student Progression and Achievement; Student Support and Guidance; Learning Resources; Quality Assurance and Enhancement. Well over 90% of the recommendations made related to these six aspects. A few recommendations fell outside the six aspects and most of these were coded under a generic heading of Organisational Context and Policy.

FINDINGS

The assessment reports identify a number of issues that peer assessors identify as enabling improvement to the existing quality of satisfactory education: Overall, there are strong similarities between subjects in the spread of recommendations between the six aspects of provision. Two aspects of provision account for are 50% of all recommendations. These are the aspects of Teaching, Learning and Assessment and quality Assurance and Enhancement. In practice these two aspects are closely connected. Many individual recommendations refer to both. This interconnectedness should serve as a reminder that quality assurance within higher education institutions is not a separate administrative activity but a vital part of central academic processes of teaching, learning and assessment.

Within each aspect there are common areas to do with the formulation of appropriate strategies for teaching, learning and assessment, the selection of appropriate methods, the monitoring of performance, the use of student feedback, staff development and the dissemination of good practice. Appropriate quality assurance mechanisms at departmental level have an often central role to play in achieving the essential linkage between these and other areas of provision. There are also some interesting differences between subject areas in the recommendations made, particularly in the proportions concerned with curriculum matters and learning resources.

In all subjects with the exception of Chemistry, English and Mechanical Engineering the aspect of provision which received the most recommendations was Teaching, Learning and Assessment. In the three exceptions, Quality Assurance and Enhancement was the most commonly referred to aspect.

There were significant differences between subjects in the frequency with which the Curriculum Design, Content and Organisation aspect was subject to recommendations. The subjects with the highest percentage of recommendations in this aspect were Environmental Studies, Social Policy and Administration, Geography and History. The subjects with the lowest proportion of recommendations in the Curriculum aspect were Applied Social Work, Computer Science and Mechanical Engineering.

The subjects with the highest proportion of recommendations in Curriculum Design, Content and Organisation (Geography, Social Policy and Administration, Environmental Studies) also have the highest proportion of recommendations in Teaching, Learning and Assessment.

Three subjects (Chemistry, Mechanical Engineering and Computer Science) which have low proportions of recommendations in these two aspects are found at the top of the list of subjects with a high proportion of recommendations in Quality Assurance and Enhancement.

Another group of subjects contained a high proportion of recommendations in Quality Assurance and Enhancement. These were Applied Social Work, Chemistry, Mechanic . Engineering, Computer Science, Law and Business and Management. It is worth noting that none of these subjects overlaps with the Curriculum group. Subjects with relatively low proportions of recommendations in Quality Assurance and Enhancement are Architecture and Environmental Studies.

Recommendations to former PCFC sector institutions are more likely to concern Learning Resources (17% compared with 10% in former UFC sector institutions) whereas recommendations to former UFC sector institutions are more likely to concern Quality Assurance and Enhancement (27% compared with 21% in former PCFC sector institutions).

In the former UFC sector institutions, Quality Assurance and Enhancement is subject to almost as many recommendations as Teaching Learning and Assessment (27% compared to 29%).

There is a slightly different pattern to the recommendations in the relatively small number of reports to FE institutions. Nearly a third of all recommendations (32%) concern Quality Assurance and Enhancement while 21% concern Learning Resources and 20% concert Teaching, Learning and Assessment.

Recommendations for the 11 unsatisfactory quality assessment reports reflect almost exactly the distribution of recommendations in satisfactory reports. Over 50% of coded recommendations were made in the aspects Teaching, Learning and Assessment and Quality Assurance and Enhancement.

The unsatisfactory reports on former PCFC sector institutions contain a higher proportion of coded recommendations in the aspects Curriculum Design, Content and Organisation (23% against 18% for unsatisfactory provision as a whole), and Learning Resources (22% against 16%) than the report on the former UFC sector institution. In this latter aspect, staffing was the most frequently mentioned key feature in the reports of the former PCFC sector institutions. The PCFC reports also contain a lower proportion of coded recommendations in Quality Assurance and Enhancement (18% against 23% for unsatisfactory provision overall), and Organisational Context and Policy (2% against 5%).

In relation to the three unsatisfactory reports of the FE institutions, the most frequently coded recommendations were to be found in Teaching, Learning and Assessment and Quality Assurance and Enhancement (27 coded recommendations out of a total of 43).