Proposals for a revised approach to assessing and rating quality

Key points of this proposal

• Assessment outcomes would reflect student success both during study and beyond study, ensuring strength in one area cannot fully compensate for weaker performance in the other.
• A Requires improvement rating, or an Unsatisfactory outcome, would indicate unsatisfactory rates of student success either during or beyond their studies.
• A Bronze rating, or Satisfactory outcome, would indicate satisfactory rates of student success both during and beyond study.
• A Silver rating would indicate consistently high rates of student success, and Gold consistently outstanding rates of student success.
• Higher ratings would require consistently strong performance across the provider; weaknesses in significant areas of provision could limit the rating awarded.
• We would assess student outcomes against three criteria: rates of continuation, completion and positive post-study outcomes.
• We would form a judgement on each criterion, based on the available data indicators and relevant contextual evidence submitted by the provider.
• The criteria for continuation and completion would take account of absolute performance as well as performance relative to benchmarks.
• Continuation and completion rates at least five percentage points above the minimum threshold would normally indicate high rates, as would performance in line with the provider’s benchmark.
• Continuation and completion rates of at least 95 per cent would normally indicate outstanding performance, as would performance materially above the provider’s benchmark.
• For continuation and completion, providers could demonstrate high or outstanding performance through either strong benchmarked performance or strong absolute outcomes.
• The criterion for post-study outcomes would take account of benchmarked performance across three measures, placing more weight on the more positive measures, and not relying on any one measure to determine the judgement.
• An unsatisfactory judgement for post-study outcomes would normally require progression and at least one other post-study measure to be substantially below benchmark and not justified by context.

Explanation

Definitions of TEF ratings and minimum requirements outcomes

319. Table 11 proposes definitions for the student outcomes TEF rating categories. They reflect the decisions taken following stage one about the purpose and structure of the future TEF. In particular, the rating categories align with condition B3, with Bronze representing outcomes that meet minimum requirements, Silver representing high-quality outcomes, Gold representing outstanding outcomes, and Requires improvement indicating that outcomes are not meeting minimum requirements.
320. We consider the ratings should reflect both success during study and positive outcomes after study, because both matter to students when choosing and undertaking higher education. Students expect to be supported to continue and complete their studies successfully, and to gain value from their higher education through positive outcomes after graduation. We therefore consider that higher ratings should require strong performance in both areas, rather than allowing strong performance in one to compensate for weaker performance in the other. We also consider that weak outcomes in either area (study outcomes or post-study outcomes) should lead to a lower rating, even if outcomes in the other area are strong.
321. We also consider that we should only award higher ratings where strong outcomes are delivered consistently across a provider’s provision and student population. While some variation may be expected, we consider the rating should be constrained by substantially weaker outcomes in subjects, modes of study or groups of students representing a significant proportion of the student population.
322. This reflects both the expectation that students should experience consistently strong outcomes and the role of TEF in driving improvement. By linking ratings to consistency, we aim to encourage providers to identify and improve weaker areas of provision and poorer outcomes for particular groups of students, rather than relying on stronger performance elsewhere within the provider to achieve a high rating.
323. Further detail about how we propose to take account of consistency in the assessment is set out in Proposal 17.

Table 11: Proposed rating definitions for student outcomes

324. Where a provider is assessed for only minimum requirements for the student outcomes aspect, we propose that it receives one of the following outcomes, which align with Bronze and Requires improvement TEF ratings:
• Satisfactory: There are satisfactory rates of student success both during and beyond study, across the provider taking account of student and course characteristics.
• Unsatisfactory: There are unsatisfactory rates of student success during or beyond study, in one or more significant areas.
325. However, where a provider’s student outcomes data is too limited or the indicators are not statistically reliable enough to support one of these two judgements, we would instead indicate that student outcomes were not assessed.

Student outcomes assessment criteria

326. We propose that judgements on each of three assessment criteria would underpin the assessment outcomes for each aspect. We propose these criteria would be continuation, completion and post-study outcomes, taking context into account.
327. We decided following stage one that we would use benchmarked indicators to assess student outcomes, and take account of minimum numerical thresholds for continuation and completion to assess whether providers meet minimum requirements. We also said we would consider further how to take account of absolute performance alongside benchmarked performance, when assessing student outcomes.
328. We are proposing a different approach to judging continuation and completion from that for post-study outcomes. This reflects the fact that continuation and completion are subject to minimum numerical thresholds under condition B3, whereas post-study outcomes would be assessed using a basket of benchmarked indicators and contextual evidence rather than minimum numerical thresholds. As a result, continuation and completion judgements would take account of both absolute and benchmarked performance, while post-study outcomes would be judged primarily through benchmarked performance.

Continuation and completion

329. As we have developed the detail of the approach, we now consider that assessment of continuation and completion should take account of both benchmarked performance and absolute performance, not just for considering whether minimum requirements are met.
330. As decided following stage one, continuation and completion rates that are below minimum thresholds and not justified by context would be considered unsatisfactory. We do not propose that rates below the provider’s benchmarks would be considered unsatisfactory, as long as they meet the minimum requirements.
331. We recognise that some providers achieve continuation or completion outcomes that are substantially above the minimum requirements, even where those outcomes are below their benchmark. We therefore propose that we would judge continuation and completion outcomes to be high where, taking account of context, they are either:
     1. In line with the provider’s benchmarks. This is because performance would be in line with similar students on similar courses elsewhere, in a sector that generally achieves high rates of successful student outcomes, which are above minimum expectations.
     2. Substantially above the minimum thresholds. This is because substantially exceeding minimum thresholds would indicate performance beyond the minimum requirements. We propose that continuation and completion rates that are at least five percentage points above the relevant minimum threshold, with at least strong statistical certainty, would normally indicate high rates of continuation or completion, even if a provider is below its benchmark. This would recognise strong student outcomes in absolute terms as well as performance relative to benchmark.
332. We also propose that continuation and completion outcomes would be judged to be outstanding where, taking account of context, they are either:
     1. Materially above the provider’s benchmark. This is because performance would be materially better than for similar students on similar courses elsewhere, in a sector that generally achieves high rates of successful student outcomes.
     2. Very high in absolute terms. We propose to define very high absolute continuation or completion rates as at least 95 per cent, and consider these in themselves to be outstanding.
333. Previously in the TEF, if a provider had a very high benchmark value – such that it would have been difficult to exceed it – we treated performance in line with benchmark as outstanding. For consistency with the proposed approach to assessing high absolute rates of continuation and completion and to make the criteria simpler, we consider it appropriate to move to using very high absolute thresholds. Under this approach, outcomes above the very high thresholds would normally indicate outstanding performance, even where benchmarks have not been exceeded.
334. This approach to considering both absolute and benchmarked rates of continuation and completion allows these judgements to build on the minimum requirements of condition B3 and take full account of context. It recognises that benchmarks help assess how outcomes compare with those that might reasonably be expected given a provider’s students and provision. Absolute measures help us recognise where outcomes are particularly weak or strong in their own right, and support the clear minimum standards that apply to continuation and completion. Together, they allow us to take into account both context and actual outcomes.

Post-study outcomes

335. As decided following stage one, we will use a basket of benchmarked post-study measures. This recognises that positive post-study outcomes can take different forms and that each measure has particular strengths and limitations. Considering the measures together mitigates the risk that weaknesses in any one measure have an undue influence on the judgement. We therefore propose to base the post-study outcomes criterion on the overall pattern of performance across the available measures and across the provider’s provision, taking account of relevant context. No single measure would determine the judgement in isolation.
336. Consistent with this rationale, in considering positive performance, greater weight would be placed on the highest-performing measure or measures. This would allow particularly strong performance on one measure or form of positive post-study outcome to support a high or outstanding judgement without requiring all measures to reach the same level. However, the criterion would also take account of the performance of the other measures, so that the strongest measure would need to be supported by the overall pattern of outcomes and not materially undermined by contrary evidence. An example of how this would work in practice is set out in Table 12.

Table 12: Example of post-study criteria in practice

337. This would recognise the diversity of positive post-study outcomes, while avoiding reliance on any one measure in isolation.
338. The consistency of performance across the provider’s provision would also be important for higher ratings. A high or outstanding judgement would require the strongest performance to be consistent across provision, while allowing for some variation.
339. An unsatisfactory judgement would not be based on a single poor post-study measure. It would normally require multiple measures to be substantially below benchmark in one or more substantive areas of provision, and that performance not to be justified by context. We propose that one of the measures would normally need to be progression. We consider that only reaching an unsatisfactory judgement where there are poor rates of progression recognises the newness of the other two post-study measures and that providers have had longer-standing visibility of progression indicators and therefore greater opportunity to improve this outcome. Requiring another measure to also be substantially below benchmark would mitigate the risk of relying on a single measure resulting in an unsatisfactory judgement, as the judgement would be supported by a wider pattern of post-study outcomes.
340. The post-study measures would all be benchmarked and we do not propose to introduce absolute thresholds for post-study outcomes. We explain in Proposal 12 why we are not proposing to set minimum thresholds for post-study outcomes. For the same reasons we would not use absolute thresholds to judge whether these outcomes are substantially above minimum requirements.
341. We have considered whether to set very high absolute thresholds for post-study outcomes as well as continuation and completion outcomes, which could indicate outstanding performance. We do not propose to do so. This is because continuation and completion are subject to established minimum requirements under condition B3, and the distributions of these measures mean that very high levels of performance are achievable and can be meaningfully distinguished in absolute terms. By contrast, analysis of post-study outcomes indicates these outcomes are less likely to reach comparably high absolute levels, and may be more affected by factors outside a provider’s control. We therefore consider that benchmarked performance provides a more appropriate basis for distinguishing performance in this area.