Proposals for a revised approach to assessing and rating quality

Key points of this proposal

• Where we identify indicators of concern, we would expect the provider to submit relevant contextual evidence about these indicators.
• Relevant contextual information would fall into three categories: highly distinctive factors not captured by the indicators, demonstrable improvements in outcomes, and exceptional circumstances that have impacted student outcomes.
• Assessors would consider contextual information about course closures, partnership changes or restructuring, where providers can demonstrate that these changes have improved outcomes.
• To demonstrate improvements, providers would need to show improvements in outcomes using the same measures and definitions as the published indicators.
• Where we identify indicators of concern, the provider would have a limit of ten pages to submit relevant contextual evidence.
• Providers applying for a TEF rating would have an additional ten pages to submit relevant contextual evidence that may support a higher rating.
• In their TEF submissions, we would expect providers to address material variation in outcomes across their provision where relevant.

Explanation

Identifying indicators of concern

360. As part of the initial risk assessment when we select providers for assessment each year, we would identify any student outcomes that constitute indicators of concern. We would notify providers of any indicators of concern at the start of the assessment and invite them to provide relevant contextual information as part of their assessment submission. We propose to allow ten pages for this.
361. We would normally select indicators where continuation or completion is below the relevant numerical threshold and is not justified by benchmarked performance, or where at least two post-study outcomes are substantially below benchmark.
362. We would normally focus on indicators or split indicators representing a substantial proportion of a provider’s students, including indicators covering at least 20 per cent of the student population. However, we may also consider smaller groups where performance gives rise to particular concern, for example because outcomes are substantially below the relevant threshold, or because it relates to particular disadvantaged student groups (which we explain further in Proposal 17).
363. We propose to place greater weight on benchmarked performance when deciding whether below-threshold continuation or completion outcomes warrant further scrutiny. Where an indicator is below a numerical threshold but broadly in line with or above the benchmark, we would normally regard this as sufficient justification and would not normally seek further explanation from the provider. Before requesting additional contextual information, we may also consider relevant regulatory intelligence we already hold, including information obtained through previous assessments.
364. We consider this approach provides a transparent and proportionate way of integrating B3 assessments within cyclical quality assessments, by focusing additional scrutiny only on areas that are potentially of concern and where available evidence does not already provide a reasonable explanation.

Scope of submissions

365. If we have identified indicators of concern, we would expect the provider’s submission to provide relevant contextual information to address them.
366. In line with the stage one decisions, providers applying for a TEF rating could use their discretion to identify any other indicators for which they wish to provide contextual information. However, the proposed assessment approach places emphasis on consistency of outcomes across provision and student groups, as set out in Proposal 13 and Proposal 17. We would therefore expect providers to consider whether there are material differences in outcomes across their provision, including by subject, mode, level, student group or partnership arrangement, and consider supplying any relevant contextual information in relation to weaker areas of performance, as these could have a limiting effect on the judgements.
367. Where the provider does not supply contextual information in relation to an indicator, the assessors would form a judgement based on the data alone, which itself takes account of context through benchmarking.
368. For subcontracted provision, lead providers should address outcomes in the same way as for directly taught provision. Because the lead provider has direct responsibility for the quality and outcomes of subcontractual provision, the fact that provision is delivered through a partner would not, by itself, explain weaker outcomes. We recognise that some providers have multiple subcontractual arrangements in place and that there may be differences in outcomes between different partners.
369. For validated-only provision, providers may include contextual information under a separate heading. As set out in Proposal 6, this information would contribute to a judgement about whether rates of student success for validated provision are satisfactory or unsatisfactory, but would not form part of the evidence used to determine a TEF rating.

Relevant contextual information

370. As decided following stage one, we will assess student outcomes on the basis of data indicators and additional context relating directly to those indicators, including actions that have led to demonstrable improvement. We propose to define relevant contextual information as:
• highly distinctive contextual factors not captured by the indicators
• demonstrable improvements in outcomes that are not yet fully reflected in the published indicators
• exceptional or unforeseen events that have had a material effect on student outcomes.
371. We expand on these definitions in paragraphs 373 to 383 and in the Part 3 of the draft guidance, where we also provide some examples of what we would and would not consider relevant contextual information.
372. We propose to apply these definitions of contextual information to all indicators used in our assessments, i.e. continuation, completion and post-study indicators. This is because our experience from previous assessments suggests that this provides an appropriate framework for identifying contextual information that could justify weaker student outcomes data. However, we recognise that our basket of post-study indicators introduces new measures that we have not previously considered in assessments. We would therefore welcome views on any additional contextual factors that we could take into account when assessing post-study indicators.

Highly distinctive factors not captured by the indicators

373. Following stage one we decided we would consider historical factors other than those already accounted for in benchmarking. We now propose to define this as highly distinctive contextual factors not captured by the indicators, which could include:
• highly distinctive student or course characteristics that are not adequately reflected in the benchmarking approach
• demonstrably positive outcomes that are not captured because of the way positive outcomes are defined in our measures.
374. We would expect the threshold for this type of evidence to be high. A wide range of delivery models are in place across the sector, and the existence of a distinctive mission, student population or course portfolio would not be sufficient on its own without clear supporting evidence. The provider would need to demonstrate that the factor is genuinely unusual, is not already reflected in benchmarking and has a material effect on the interpretation of the indicators. Explaining that a certain measure is not appropriate for the provider’s students, without providing evidence that the relevant outcomes were in fact positive for its students, would not be sufficient.

Demonstrable improvements in outcomes

375. Following stage one we decided we would consider actions a provider has taken to improve outcomes, the impact of which is demonstrable. We propose to define this as demonstrable improvements in outcomes that are not yet fully reflected in the published indicators. We would only treat improvement as demonstrable where the provider can show measurable improvement in outcomes using the same definitions and methods used in the published indicators, and would consider the extent of any measurable improvement when reaching a view on a provider’s rating for student outcomes.
376. We recognise that improvements within particular courses, subjects or student groups may sometimes be evident to providers but less visible in the published indicators that do not include year-by-year splits within each category. In these circumstances, providers could present more granular calculations of outcomes. These may be considered relevant where they relate to a significant proportion of students, and the improvement is material.
377. Where a provider can demonstrate that closing a course, ending a partnership or making another structural change has led to improved outcomes in the remaining provision, we would consider this as relevant contextual information. In considering it, we would take account of whether there is any ongoing risk to students. For example, we may seek to understand the reasons for course closure or partnership termination where a substantially similar course or partnership has subsequently been opened, including to ensure that such action has not been taken to avoid regulatory action.
378. We recognise that an additional year of student outcomes data will become available after submissions have been made, but before assessments are completed. We propose to take account of this where assessors’ initial view points towards a Bronze or Requires improvement TEF rating, or an Unsatisfactory outcome, so that recent improvements are considered before such judgements are made. For efficiency, we do not propose to do this where our initial view points towards a Silver or Gold TEF rating or Satisfactory outcome. This is because we are not proposing that any negative consequences would flow from a Silver or Gold rating, so the risk of those providers being materially prejudiced by not including the additional year’s data is reduced. For providers undergoing an assessment against only minimum requirements, Satisfactory would be the highest achievable outcome.
379. We think this approach strikes an appropriate balance between ensuring all relevant information is assessed for the providers where the impacts of the assessment are likely to be greater, and ensuring that the overall assessment process in a particular year is not unduly delayed or extended.
380. Evidence of actions, interventions or improvement activity would not by itself demonstrate improvement. Providers would need to show that those actions have already resulted in better outcomes. Where improvements take longer to emerge in published data, they would be considered in future assessment cycles.

Exceptional circumstances

381. Providers may submit contextual information about exceptional or unforeseen events that have had a direct and material effect on student outcomes. The event should be genuinely exceptional, rather than one that affects the sector as whole, and should generally fall outside the provider’s normal operating environment. It should also not already be accounted for in our benchmarking approach. We would not expect this category to cover challenges that arise as part of the normal operation of a higher education provider, such as managing relationships with staff and employment unions.
382. We would generally expect such events to be outside of the provider’s control. Where there is a compelling case, we may also take account of exceptional circumstances that are not wholly outside the provider’s control, but could not reasonably be foreseen or planned for.
383. Providers should explain the event, the students or provision it affected, and why it had a material impact on the outcomes shown in the data.

Standards of evidence

384. We propose to apply consistent standards when assessing contextual information. For any of the categories of contextual information, the provider would need to set out its calculations of outcomes using the appropriate definitions in our measures, and specify what has been calculated in a way that we can verify through our own analysis of the data. This approach means that a provider’s internal measures using alternative definitions of positive outcomes or methods of calculation that differ from those in the published indicators would not be considered as relevant contextual information. We are proposing this to support consistency and robustness of assessments for all providers.
385. We may by exception request further information during an assessment where necessary. However, submissions should normally contain all evidence needed for assessors to reach a judgement. We would normally seek additional information only where assessors are considering a Bronze or Requires improvement rating or Unsatisfactory outcome, and require clarification on certain points contained in the provider submission or additional supporting information before reaching a judgement.
386. We think this approach is appropriate because we are not proposing that any negative consequences would flow from a Silver rating, so the risk of those providers being materially prejudiced by not being able to submit additional information is reduced. We considered whether it would be appropriate to seek additional information for providers where assessors are considering a Silver rating, but have balanced that against the risk that this would led to undue delay in the overall assessment cycle and have reached a view that this would not be an efficient approach.
387. Where providers reference or link to externally published information as part of their submission, we propose to apply the same approach to verification and use of such material as set out in the student experience section.

Page limits

388. Where we have informed a provider that we have identified indicators of concern, we propose that it would have up to ten pages to provide contextual information to address these indicators.
389. Providers applying for a TEF student outcomes rating would have an additional ten pages for their student outcomes submission. This would enable providers to submit relevant contextual evidence that could support a rating above the level of Bronze.
390. We consider this to represent an appropriate balance between giving providers sufficient scope to contextualise their indicator performance and ensuring assessments can be carried out efficiently, avoiding excessive burden on assessors.
391. We are interested in views on whether these proposed page limits are appropriate, including whether they provide sufficient flexibility for providers to present relevant evidence in a clear and proportionate way.