Proposals for a revised approach to assessing and rating quality

Proposal 18: Published outcomes and enhancement

456. Where a provider does not have a TEF rating or minimum requirements outcome for an aspect, we propose to make this explicit in the published presentation of outcomes. Our view, informed by the student research, is that clearly stating that the aspect was not assessed is preferable to making no reference to that aspect, which could otherwise be interpreted negatively.
457. Because some assessments may take longer to finalise than others, particularly where providers make representations on provisional ratings (that is, a provider asks us to review a provisional rating or outcome before it is finalised), outcomes would normally be published in two batches each year. This is likely to be in September and January. We consider this preferable to delaying publication until every provider in a cohort has completed the assessment process, as earlier publication allows students to access information sooner and enables providers to benefit from any positive outcomes without unnecessary delay.
458. We decided, following stage one, that we will continue to publish TEF 2023 ratings that remain valid alongside ratings from the new scheme, with clear labelling and explanation. We will do this through a single published record of TEF 2023 ratings that remain valid alongside new outcomes, and through publication on Discover Uni and UCAS services. When providers receive ratings and outcomes under the future system, those outcomes will replace their TEF 2023 outcomes where relevant. We will support this approach with clear labelling, explanations and distinct branding to help users understand the differences between the previous and the new TEF, and minimum requirements outcomes.

Publication of minimum requirements outcomes

459. For providers that do not apply for TEF ratings, we propose we would publish:
• The outcome of the assessment for student experience (conditions B1, B2 and B4) and student outcomes (condition B3). These outcomes would be Satisfactory or Unsatisfactory.
• The year of the assessment.
• The assessment report.
• Any relevant regulatory action.
460. We propose that providers assessed only against minimum requirements would have a different published presentation from TEF-rated providers.

Publication of outcomes by providers

461. We have considered what our expectations should be on providers in terms of publishing their own TEF ratings or minimum requirements assessment outcomes. We propose to require providers to publish their future TEF ratings or minimum requirements assessment outcomes in a consistent way. We consider that such requirements would further improve the visibility and consistency of quality information and would support informed student choices.
462. The publication requirements would be established through revised ongoing conditions of registration.
463. If we change or remove a provider’s TEF rating or outcome, the provider would be required to remove publication of the superseded or withdrawn rating or outcome and, where applicable, publish the new one.
464. We propose that to ensure accurate and consistent information for students, providers would be required to:
1. Publish their future TEF aspect ratings, using the standardised logos, on relevant course webpages within scope of the assessment and that are taught directly by the provider. For the first cycle this would normally be on all pages relevant to undergraduate level courses that are taught directly by the provider.
2. Publish ratings in a timely manner.
3. Publish the ratings for both aspects.
4. Always publish the ratings alongside any other information from the assessment report that a provider may choose to publish.
5. Avoid implying the ratings cover provision that was not in scope of the assessment.
6. Include links to the relevant OfS TEF outcome page.
465. Although subcontracted-out courses are also within the scope of the assessment, we consider that students would be more interested in the ratings of the provider that teaches these courses. We therefore would not require providers to publish their TEF ratings on webpages for courses that they subcontract out.
466. Similarly, a provider (a ‘delivery partner’) that teaches courses through a subcontractual arrangement would display its own rating, but not that of the ‘lead provider’ that subcontracts out the teaching.
467. Where a provider continues to hold a valid TEF 2023 rating, the TEF 2023 guidance regarding publication would continue to apply. Publication of these ratings would therefore continue to be optional for the provider. We have considered the impact of this on the visibility and consistency of student information during the transitional period. Because we are prioritising the early assessment of all providers holding a TEF 2023 Bronze or Requires improvement aspect rating, after the first year of the future system, only Silver and Gold TEF 2023 ratings will continue to be valid. We think these providers are more likely to choose to publish their ratings, mitigating a potential negative impact on student information.

TEF assessment reports

468. Following stage one, we decided that we will publish TEF assessment reports in full, subject to normal publication considerations. We now propose further detail about the content of the reports, with the intention of serving multiple audiences including students, providers and other stakeholders. We propose to publish an assessment report for each provider that provides a clear summary of judgements, a more detailed explanation of the assessment findings and reasons for the ratings awarded, and recommendations for improvement where applicable. Reports would follow a standard template.
469. In terms of the summary, each report would include a ‘scorecard’ of judgements against the assessment criteria to help students understand the assessment outcomes. For example, it would state whether ‘Course content’ is outstanding or of high quality, satisfactory or unsatisfactory. This may also include summary information about the consistency of quality or outcomes across different groups of students or areas of provision where that variation was material to the assessment outcome. For example, the ‘scorecard’ might indicate that ‘Course quality’ is satisfactory overall and of high quality for some areas of provision.
470. The scorecard would aim to provide a clearer picture of what has informed the ratings and offer more detail about a provider’s strengths and areas for improvement than could be conveyed through a rating alone. We intend the scorecard to be useful, clear and relevant to a student audience in particular. We intend to undertake user testing with students to inform the content, design and presentation of the scorecard. Through user testing we would consider whether to include:
• a summary of the provider’s published TEF data alongside the scorecard (This is because previous research with students has indicated students would be interested in succinct, clear data to help understand the ratings, but that the published dashboards are too complex for this audience.)
• the judgement about validated-only provision in the summary, where applicable.
• include in the summary, the restrictions associated with lower TEF ratings, such as student recruitment limits.
471. While we would develop the scorecard through user testing, we are also interested in views about this approach and how to make the information as useful as possible to students.
472. Further detail in the report would explain how the assessment team reached its judgements. This would include the interpretation of indicator evidence, consideration of provider and student submissions and any evidence gathered in an enhanced assessment, and the way in which variations between student groups or areas of provision were taken into account. The aim is to provide sufficient information for providers to understand the rationale for the outcomes and identify areas of strength and improvement.
473. We are also considering including recommendations for improvement within assessment reports. These recommendations would identify areas where students, or particular groups of students, would benefit from improvements. However, it would not prescribe how providers should address them. Recommendations could be included for providers at any rating level, although they may be most relevant where ratings are lower. Where recommendations are made about improvement for disadvantaged student groups, we would expect them to be addressed in a provider’s next access and participation plan.
474. Assessment reports would also include any related restrictions that follow as a result of a provider’s TEF ratings.
475. Where a provider’s TEF outcome changes following a quality investigation or other regulatory activity, we would not amend the original assessment report but would publish supplementary information alongside the report to explain the revised outcome. We would ensure that any such supplementary information is presented clearly and prominently so that students and other users are able to understand the provider’s current position. We consider that published information should reflect the provider’s current position and that retaining the report without supplementary information may be misleading, particularly for students. We discuss the circumstances that could lead to changes to TEF ratings in more detail under Proposal 21.
476. Following the decisions taken in stage one, and subject to further decisions about the stage two proposals, we would make consequential revisions to Regulatory advice 21: Publication of information so that it reflects the information we would publish arising from the future quality system. This would include updating information about what we would normally publish in relation to provider participation, ratings and outcomes, assessment reports and associated publications.

Comments on TEF materials for publication

477. We decided following stage one that providers would have the opportunity to comment on the factual accuracy of their assessment report before publication. We now propose that before publication, the provider would also receive its provider and student submission (if applicable) that we intend to publish. The provider would be able to comment on factual inaccuracies and identify information that should not normally be published, such as confidential, commercially sensitive or personally identifiable information.
478. Providers would normally have 14 days to provide comments. We would consider whether factual corrections or redactions are appropriate before publication and would make publication decisions in accordance with its published approach.

Minimum requirement outcome reports

479. We propose that assessment reports for minimum requirements outcomes would focus on findings against the relevant criteria and explain the reasons for the outcomes reached. The approach would be similar to TEF assessment reports but with judgements on each of the criteria and the overall aspects limited to Satisfactory or Unsatisfactory.
480. Providers would have the opportunity to comment on the factual accuracy of their assessment report before publication. Providers would normally have 14 days to provide comments. We would consider whether factual corrections or redactions are appropriate before publication and would make publication decisions in accordance with our published approach.

Supporting enhancement

481. Alongside publication of TEF ratings and assessment reports and data indicators, we will work with providers, students and other stakeholders to support enhancement and continuous improvement. This may include producing thematic reports, case studies and other activities that help providers learn from TEF findings and from each other.
482. In our stage one consultation outcomes, we said that we would consult in stage two on sample-based visits, which could potentially provide assurance about the operation of the quality system and support enhancement activity across the sector.
483. We propose a programme of up to ten sample-based visits per year. These visits would not form part of the TEF assessment of providers and would focus on high-quality provision experienced by students.
484. Rather than revisiting all the findings of the TEF assessment, we expect that these visits would seek to identify good practice in relation to certain criteria or areas of provision. We expect that we would publish examples of practices contributing to high quality to recognise best practice and to support quality enhancement across the sector.
485. These visits would also seek to understand the efficacy of the TEF assessment process in recognising areas of high quality experienced by students. We would use this information to improve the TEF process.
486. We anticipate that we would work with the sector to develop the details of sample-based visits.

Notes

[8] See Applicants’ use of TEF information.