Proposals for a revised approach to assessing and rating quality

Key points of this proposal

• Condition B3 would include a new requirement for providers to deliver positive post-study outcomes for students.
• We would assess compliance with the post-study outcomes requirement through a basket of benchmarked post-study indicators and relevant contextual information, rather than minimum numerical thresholds for a single measure.
• Condition B3 would also include a new requirement for providers to deliver positive outcomes for students undertaking modular study. Until specific measures and thresholds are developed, we would use this requirement to identify and address serious concerns about outcomes for modular learners.
• We would revise minimum numerical thresholds for some continuation and completion indicators for part-time and apprenticeship provision.
• We would assess providers applying for registration for compliance with initial condition B3 according to an updated Regulatory advice 20, aligned to the approach used in TEF assessments.

Explanation

288. Following stage one, we decided to develop an integrated quality system. This includes:
     1. Integrating the assessment of whether providers satisfy the minimum requirements of condition B3 into the TEF assessment of student outcomes.
     2. Aligning the TEF ratings with the relevant B conditions, so that ratings will show whether or how far the provider exceeds the minimum requirements.
289. We also decided to simplify condition B3 by removing the progression indicator and associated numerical thresholds.
290. We have now developed proposals for how assessment of condition B3 would in practice be integrated into the TEF assessment of student outcomes. We are therefore proposing some further revisions to condition B3. These are intended to ensure an integrated and coherent approach to assessing student outcomes.
291. We also make proposals for revising some of the B3 thresholds for continuation and completion, which we are committed to reviewing, and propose to update Regulatory advice 20 to align it to the new approach.

Positive post-study outcomes

292. We are proposing a new requirement within condition B3 that providers deliver positive post-study outcomes for their students.
293. Following stage one, we decided to remove the progression indicator and its associated numerical thresholds from condition B3. We said that progression would continue to have an important role in the assessment of student outcomes, because positive outcomes beyond higher education are fundamentally important to students and are a key purpose of higher education. But we recognised the limitations of using a single post-study measure assessed against a minimum threshold to determine whether a provider was meeting minimum requirements, and recognised that the outcomes achieved by graduates are influenced by a range of factors.
294. As we have developed the detailed proposals, we have reflected further and have changed our approach. Rather than relying on a single progression measure and numerical thresholds, we now propose to assess post-study outcomes using a basket of benchmarked measures covering progression to further study, use of skills and earnings, alongside relevant contextual evidence submitted by the provider.
295. We consider that this approach provides a broader and more reliable assessment of graduate outcomes than the previous progression measure. It would reduce reliance on any single indicator and allow assessors to consider different forms of positive post-study outcomes. By using benchmarked measures and contextual evidence, rather than minimum numerical thresholds, we would be able to take greater account of the factors that influence student success after their studies: for example, taking account of the region a graduate is employed would allow us to take account of differences in regional earnings.
296. We therefore consider that the concerns that led us to remove progression and the associated thresholds from condition B3 following stage one are substantially addressed by the approach now being proposed. We no longer propose that compliance with condition B3 should be determined through a single post-study measure or minimum numerical thresholds. Instead, we propose a broader assessment based on multiple indicators and contextual evidence.
297. Because of the importance of post-study outcomes for students, we consider that poor post-study outcomes should be capable of contributing to a rating of Requires improvement. We do not think it would be coherent for those outcomes to be capable of leading to a rating of Requires improvement while having no role in determining whether minimum requirements for student outcomes are being met. We therefore propose that condition B3 should include a requirement that providers deliver positive post-study outcomes.
298. Rather than a requirement to meet numerical thresholds for individual post-study indicators, condition B3 would include a broad requirement that providers deliver positive post-study outcomes, expressed in a similar way to other B conditions. Compliance would be assessed using the basket of benchmarked post-study indicators, together with any relevant contextual information provided by the provider.
299. The broad requirement to deliver positive post-study outcomes would apply to all modes and levels of study, including postgraduate study. We intend to review the applicability of the post-study indicators for postgraduate study outcomes before incorporating postgraduate study into the second cycle of assessments. We may consider variations to these measures to ensure they are fit for purpose, and expect to consult on the details of how we will assess postgraduate study before the second cycle.
300. Considering a basket of indicators would avoid overreliance on any one measure of post-study outcomes to determine non-compliance. Instead, assessors would consider the overall picture presented by the provider’s post-study indicators and take account of relevant contextual evidence from the provider. If a provider has indicators for more than one of the post-study measures that are substantially below benchmark, represent a substantial proportion of their students, and are not justified by additional contextual information, this would normally lead assessors to conclude that the provider has not delivered positive post-study outcomes. 

301. We have considered whether it would be appropriate to set minimum thresholds for the post-study indicators. Our view is that benchmarked performance provides a better basis for judging post-study outcomes, because a wide range of factors influence these outcomes, especially subject of study, geographic employment factors and students’ prior attainment. Using benchmarks allows us to take account of these factors in a systematic way and provides a better indication of the provider’s contribution to students’ post-study outcomes. The IFS report recommended using a measure based on difference from an institution-specific benchmark rather than difference from an absolute threshold, for the same reasons.⁷
302. We do not consider this proposal to be inconsistent with retaining minimum thresholds for continuation and completion. Continuation and completion measure whether students successfully progress through and complete their studies. They are measures of minimum student success during study and relate to outcomes that providers have more direct influence over. We therefore consider it reasonable to expect all providers to achieve at least a common minimum standard and to define that through numerical thresholds.
303. Post-study outcomes are measures of success beyond higher education and are influenced by a wider range of factors outside of the provider’s control. While providers play an important role in preparing students for future success, we do not consider that sector-wide numerical thresholds would represent appropriate minimum expectations across all types of students, courses and providers. Instead, we consider the more relevant regulatory test is whether graduates are achieving outcomes that are reasonably consistent with what would be expected, given their characteristics and other factors. In our view, benchmarked performance provides a more appropriate basis for applying that test than common absolute thresholds.
304. We would though be interested in views about whether we should instead set minimum absolute thresholds for progression, use of skills and earnings.
305. We consider that this proposal substantially addresses the issues that led us to remove progression thresholds from condition B3 following stage one, while ensuring that post-study outcomes are given appropriate consideration and play an important role within the quality system. The basket of measures considered together provide a richer and more reliable picture of what students achieve after their studies than the single progression measure, and the use of benchmarks rather than absolute thresholds provides us with a better measure of the provider’s contribution to those outcomes.

Positive outcomes for modular study

306. We also propose introducing a requirement within condition B3 relating to students undertaking modular study. This proposal reflects the growing importance of modular learning following the introduction of the LLE.
307. We consider that students studying on standalone modules should be able to expect positive outcomes in the same way as students studying full qualifications. At this stage, however, we do not have the data to construct student outcome measures for modular study and we need to understand patterns of participation before considering what thresholds might be appropriate.
308. We will be collecting data from 2027-28 to inform the development of a module completion measure and will consult separately on our approach. The exact date for the introduction of a module completion measure and minimum thresholds for potential inclusion in condition B3 will depend on data quality, but these are not likely to be in place until 2028-29 or 2029-30. A new modular completion measure would therefore not be included in TEF until the second cycle of assessments, starting in 2030-31. We will engage with the sector on whether there are other appropriate student outcome measures for modular study beyond completion (for example progression to other modules) for inclusion in the second cycle of TEF.
309. We think it is important that we can intervene if there is very poor performance and, until we have consulted on a completion measure for modular study, we are therefore proposing a broad requirement within condition B3 that providers deliver positive outcomes for students undertaking modular study. We would assess compliance on the basis of the available evidence about whether students successfully complete the modules they undertake.
310. In considering whether a provider is delivering positive outcomes for modular learners, we would take account of the provider’s context and the characteristics of the provision. We propose that we would consider module completion in relation to the performance of other providers, and alongside the completion rates for students studying towards the parent qualification. We recognise that completion of individual modules may not always be directly comparable to completion of a full course, and would therefore not expect equivalent outcomes in every circumstance.
311. Our intention is not to intervene whenever completion rates for modular learners differ from those of students undertaking full qualifications. Rather, we intend the requirement to enable intervention where the evidence suggests that outcomes for modular learners are materially poor. For example, where a substantial proportion of students enrolling on modules consistently fail to complete them, and the provider cannot demonstrate a reasonable explanation, this may indicate that positive outcomes are not being delivered.
312. We would only expect to take compliance action in extreme circumstances and following engagement with a provider to understand any issues.

Revised thresholds for continuation and completion

313. We have reviewed the minimum numerical thresholds for continuation and completion in line with the published approach. We propose revisions to the following thresholds.

Notes

[7] See Institute for Fiscal Studies, Using graduate earnings data in the regulation of higher education providers.