Proposals for a revised approach to assessing and rating quality

Key points of this proposal

• The provider submission would be self-evaluative, setting out the provider’s understanding of the quality experienced by its students and the evidence to support this.
• The submission would focus on students’ experiences and, for TEF assessments, improvements in those experiences rather than descriptions of policies, processes or activities.
• The provider would be required to include certain information but beyond this could determine evidence that is relevant, robust and reliable, supported by OfS guidance.
• The submission would cover a four-year period. This would apply across all providers regardless of how frequently they are assessed.
• Submissions should integrate evidence relating to subcontracted provision, and separately contextualise validated-only provision.
• We would rely on Ofsted reports of apprenticeship provision where possible as evidence about the quality of apprenticeship students’ experience.
• A structured template would be recommended but not required.
• A TEF student experience submission would be limited to 20 pages.
• A minimum requirements only student experience submission would be limited to 12 pages.
• If a provider is selected for an enhanced assessment, there would be an additional three-page annex to address the risks that we have identified as needing the visit.

Explanation

Content of submissions

208. Following stage one we decided that the provider should determine the evidence, as relevant to its context, that demonstrates how it meets the student experience criteria. We also decided that providers will be guided to supply evidence that demonstrates the impact and effectiveness of their approaches to delivering a high-quality or outstanding academic experience for their students.
209. We now make further proposals about the approach to submissions that build on the stage one decisions, and reflect other relevant proposals in this consultation.
210. Under Proposal 1, we have proposed that providers would be able to apply for a TEF assessment, or would be assessed only against minimum requirements that would determine if the level of quality that students experience at the provider is either Satisfactory or Unsatisfactory.
211. Under Proposal 7, we set out criteria that focus on the quality of what students experience rather than the provider’s activities or processes. Under Proposal 8, we explain why we consider that the NSS indicators should normally carry no more than half the weight in the assessment of the student experience.
212. We propose that the student experience submission should be self-evaluative and should provide a range of evidence that goes beyond contextualising the NSS indicators. Instead of describing provider activities or processes, submissions should focus on the outcomes of these processes in terms of the quality experienced by students and the evidence that demonstrates the level of quality experience by students. For TEF assessments, this should include where that quality exceeds minimum requirements.
213. All providers will be expected to demonstrate that their students experience at least satisfactory quality. Providers being assessed only against minimum requirements would need to demonstrate satisfactory quality. Providers applying for a TEF rating would not be expected to separately demonstrate that minimum requirements are met and that higher levels of quality are delivered. Instead, their submissions should explain the level of quality experienced by students and the evidence that supports that evaluation. Evidence consistent with high or outstanding quality would also demonstrate that satisfactory requirements are met.
214. These proposals would mean that providers applying for a TEF rating should explain their understanding of the level of quality experienced by their students and set out a range of evidence to support their conclusions. This should include any important variations across different groups of students or areas of provision, and evidence of the impact of actions taken to improve quality. To support this approach, TEF submissions should address three underlying questions:
• What level of quality do students experience, and what evidence does the provider have to demonstrate this?
• How consistently do students experience that quality across the provider, and what evidence supports this?
• Are students experiencing improvements in quality over time, and what evidence does the provider have to demonstrate this?
215. For providers undergoing an assessment against only the minimum requirements, submissions should address the underlying question:
• What evidence does the provider have to demonstrate that students consistently experience satisfactory quality across the provider?
216. Providers could discuss the actions they have taken, or the policies and processes they have put in place, but these would be considered as context rather than evidence of what students experience. The submission should focus on the experiences of students and provide a range of evidence used by the provider to understand the quality of their experience. Where a submission describes the provider’s activities, policies or processes, it should show the impact on the student experience and the scale of that impact across the student population.
217. The submission could contextualise the NSS indicators but should also provide a range of relevant evidence that addresses the criteria that goes beyond the NSS indicators. In Proposal 8 we map the NSS indicators to the six student experience criteria. However, the questions within the NSS do not cover all elements of the corresponding criteria. We therefore propose that the provider submission should place particular emphasis on supplementing the NSS indicators with evidence about those elements of the criteria that are not covered by the NSS. These include, for example, course coherence and challenge, rigorous assessment, and careers support.
218. Providers would determine what evidence is most appropriate for their context, drawing on their existing approaches to monitoring and evaluating quality. Evidence could be quantitative or qualitative, internal or external. The draft guidance includes information about the general characteristics of stronger or weaker evidence, and examples of types of potential evidence. It also sets out evidence that must be provided in the submission.
219. For example, we consider that Professional, Statutory and Regulatory Body (PSRB) findings can provide valuable evidence about the quality of provision and may be particularly relevant where they identify issues that could affect students’ educational experience or outcomes, support that provision is satisfactory, or highlight outstanding provision.
220. We propose that providers should include information about PSRB activity relevant to the provision being assessed. This should include circumstances where a PSRB has identified concerns about quality, standards or outcomes and has required, expected or recommended actions to address those concerns.
221. We would not normally expect providers to report recommendations about enhancement where the PSRB concluded that its requirements were fully met and no remedial action was necessary.
222. The draft guidance includes further details, at Part 5.5. In determining their evidence, providers would be guided to consider the following characteristics:
• how relevant, robust and reliable the evidence is
• how it is contextualised
• how it demonstrates impact.
223. These proposals recognise that providers differ in size, mission, course offerings, student populations and approaches to monitoring and evaluating quality. We consider there is no single set of policies, processes or organisational arrangements that will necessarily deliver high quality. Submissions would therefore be expected to focus on the outcomes of these processes in terms of the quality experienced by students, rather than on a prescribed set of processes. We would expect submissions to focus on the evidence providers use themselves to understand those outcomes, rather than prescribed types of evidence. This supports institutional autonomy and diversity while maintaining a consistent focus on students’ interests.

Scope of the submission

224. Following stage one, we decided that providers awarded Gold ratings would normally be reassessed after five years, Silver after four years and Bronze after three years. In Proposal 2, we propose that providers awarded a Requires improvement rating would normally be reassessed after two years; and that the intervals for providers with Satisfactory and Unsatisfactory outcomes would be equivalent to TEF ratings of Bronze and Requires improvement respectively.
225. Although time between assessments will vary, depending on the ratings awarded, we propose that the provider submission should always cover the most recent four years prior to the submission. We consider that a standard timeframe would make the basis of assessment clearer for providers and support greater consistency between assessments. It would also allow assessors to consider performance over time and the impact of actions taken to improve the student experience.
226. Under this approach:
• NSS indicators would be based on the most recent four years of data
• provider submissions would cover the most recent four academic years
• as set out in Proposal 10, TEF student submissions would primarily present views of current students but could include views from previous cohorts over the last four academic years.
227. In Proposal 6 we propose that subcontracted provision would carry equal weight in the assessment to directly taught provision. We therefore propose providers would be expected to include evidence about subcontracted provision that is proportionate to the scale and nature of that provision and that addresses any material differences in the experiences of students studying through those arrangements.
228. In Proposal 6 we propose a different approach to assessing validated-only provision. Where NSS indicators for validated-only provision are materially below benchmark, providers would have an opportunity to provide contextual information or further evidence about how they ensure a satisfactory quality student experience for these students.
229. For higher education apprenticeship provision, we propose to take account of any relevant findings from Ofsted inspections published within the previous four years. We consider that Ofsted reports provide important evidence about the quality of apprenticeships, particularly in relation to course content, course delivery, certain aspects of assessment, support and learning resources and could help reduce duplication and burden. However, the Ofsted inspection judgements do not map directly to the student experience criteria and ratings, and do not fully address all aspects of our quality conditions, particularly student engagement.
230. We propose that providers would therefore need to address the student experience criteria for apprenticeships, if there are any contradictory themes in their Ofsted report and other evidence, such as NSS indicators. Where Ofsted identifies concerns relevant to the student experience criteria, we would normally expect providers to address those findings in their student experience submission. Positive findings may also provide evidence in support of the quality of apprenticeship provision. We would also expect providers to draw on and respond to relevant outcomes from external quality assurance reports concerning apprenticeship provision in their student experience submission.

Structure and page limit

231. We propose making templates available that are structured around the student experience criteria. Providers would be encouraged, but not required, to use the relevant template. The purpose of the templates would be to help providers present evidence clearly and ensure all criteria are addressed. The draft guidance at Part 5 will also provide further information about what kinds of evidence would be relevant and worked examples of stronger and weaker evidence, to help providers.
232. For a provider’s TEF student experience submission, we propose a maximum length of 20 pages. Together with the proposed page limit of ten pages for the student outcomes submission, this would maintain an overall limit broadly in line with the (25 pages) total length for provider submissions used previously. Feedback from the stage one consultation was that the overall limit was generally appropriate, allowing providers adequate space to provide robust evidence. We consider the page limit for student experience should be higher than for student outcomes because there are a wider range of criteria to address for student experience, and because less weight would normally be placed on the student experience data. We propose to allow for a small increase in overall length of the submission to account for the expectation that providers address satisfactory and high quality.
233. For an assessment against minimum requirements only we propose a maximum length of 12 pages for the provider’s student experience submission. This length reflects that these assessments would not require providers to demonstrate that their students experience quality that exceeds minimum requirements.
234. We propose that submissions should be sufficiently clear and self-contained to enable assessment without routine requests for additional information. However, if assessors have concerns, they may seek additional evidence; in exceptional cases they may seek clarification or further information where this is necessary to make a judgement that could materially affect the aspect rating or outcome. Assessors would also access publicly available information to support their assessment.
235. Finally, we propose submissions should provide references to sources of evidence that support key claims made in the submission, in a similar way to the previous TEF. We would take a proportionate, risk-based approach to verifying that these sources of evidence support those claims. The purpose of this referencing and verification is to provide assurance that claims made in the submission are supported by verifiable evidence.