Proposals for a revised approach to assessing and rating quality

Key points of this proposal

• Providers may apply for TEF ratings in student experience, student outcomes or both, depending on whether they have sufficient data for those aspects.
• Where a provider has sufficient data for both aspects, it would have to apply for ratings in both aspects rather than only one.
• Where a provider has sufficient data in only one aspect, it could apply for a TEF rating in that aspect and we would assess it for meeting minimum quality requirements in the other aspect.
• We would aim to schedule providers for assessment when they do have sufficient data to apply for at least one TEF aspect rating.
• We would assess providers without sufficient data in either aspect for the minimum requirements.
• Revised tests would determine if a provider has sufficient data in student experience and student outcomes.
• For student outcomes, continuation, progression and at least one other post-study indicator would need to be available and provide sufficiently strong evidence to support a TEF rating.
• We would improve data sufficiency for student experience by extending the NSS to all undergraduate students on one-year courses once in-year data is available across the sector.
• Before sector-wide, in-year data is available, further education colleges that provide the necessary in-year data through the Individualised Learner Record (ILR) would be able to opt into including students on one-year courses in the NSS.
• We will consult at a later date on an alternative approach to rating the student experience for providers that want to apply for a TEF rating but do not have sufficient data for a TEF rating.

Explanation

Aspects that could be rated

102. Following stage one, we decided that providers would receive TEF ratings for student experience, student outcomes, or both, depending on whether sufficient data is available to support assessment.
103. We also said we would consult further on the definition of sufficient data and on measures to increase the number of providers for which sufficient data is available.
104. We still believe a provider must have enough data for us to award TEF ratings. This data allows assessment teams to compare providers and consistently identify different levels of quality above the minimum required.
105. In Proposal 2 we set out that we would seek to schedule providers for assessment when they have sufficient data in at least one aspect and, as far as possible, in both aspects. Once selected for assessment, we propose that providers may then apply for TEF ratings as follows:
• If the provider has sufficient data for both aspects, it would be able to apply for TEF ratings in both aspects. We consider that it would be undesirable to allow the provider to choose to apply for a rating in only one aspect. This is because allowing providers to choose which aspect to have rated could result in selective participation and give students an incomplete picture of quality. Requiring ratings in both aspects would support transparency and comparability.
• If the provider has sufficient data in only one aspect, it would be able to apply for a TEF rating in that aspect. The provider would be assessed for meeting minimum quality requirements in the other aspect.

Sufficient data

106. We propose refinements to the data sufficiency tests consulted on in stage one. The aim is to ensure that as many providers as possible could apply for a TEF rating with data that can support reliable assessment.
107. We would take the same broad approach to determining data sufficiency as originally proposed. This would involve testing:
     1. Whether there are indicators that:
        1. Cover a substantial proportion of the provider’s students
        2. Provide sufficiently strong statistical evidence to inform a judgement about performance.
     2. Whether there are enough indicators that meet these tests to inform a judgement on the aspect as whole.
108. We set out in Annex C further detail about the revised definitions of these tests and the impact they have on the number of providers that would have sufficient data to apply for a TEF rating.
109. For the student experience indicatorswould increase the number of providers that would be able to apply for student experience ratings. We expect a further increase once an additional year of NSS data becomes available and when students on one-year courses are brought within the scope of the NSS.
110. At stage one, we proposed that there would be enough student outcomes data if the continuation measure and at least one other measure—completion, progression, use of skills or earnings—gave a clear picture of performance, supported by strong enough statistical evidence. However, this could mean assessing some providers using only continuation and completion data.
111. We have considered further how post-study outcomes should inform the assessment. In Proposal 12 we explain why we think the student outcomes ratings should reflect both success during study and positive outcomes beyond study. We also set out the reasons why we do not think a single measure should determine judgements about post-study outcomes in isolation.
112. We therefore propose that there must be enough student outcomes data to assess performance clearly. This means data must be available for continuation, progression and at least one other measure of what students do after their studies. We believe TEF ratings for student outcomes should consider both what happens during students’ studies and what they do afterwards. This will make the ratings more relevant to students and easier to understand and use. The proposed criteria for rating student outcomes (see Proposal 13) reflect this approach.
113. Based on analysis of current data, the revised requirements for student outcomes would reduce the number of providers able to apply for a student outcomes rating. However, this is a small number of providers and those affected represent a small proportion of undergraduate students. Providers without sufficient data for a student outcomes rating would continue to be assessed against the minimum requirements for condition B3. In a small number of cases, data may be too limited to support a minimum requirements outcome and we explain in Proposal 13 how we would approach these providers. The proposal would also affect providers that register in future, as it would likely take them longer to build up sufficient data to apply for a student outcomes rating, and would be more likely to only be able to apply for a student experience rating in their first assessment.
114. Under this proposal, providers would not be able to apply for a TEF student outcomes rating until at least progression and another post-study outcomes indicator become available. However, we would still assess these providers against condition B3 through the quality assessment process. This means we would continue to investigate concerns about student outcomes, even if there is not enough evidence to award a TEF rating.

Increasing the availability of data

115. We have also considered how to improve data availability over time so that more providers can apply for TEF ratings. About half of the providers without enough NSS data mainly have students on one-year courses. These students are not included in the NSS survey. We propose to extend the NSS to include one-year courses, to increase the number of providers that can apply for student experience ratings. We estimate that this could allow us to rate up to 50 more providers for student experience.
116. To extend the NSS to one-year courses we need to identify students within the scope of the survey that have registered in the current academic year. To do that, we would need in-year student data. The first live in-year student data collection via Jisc, delivered through the In-year Student data programme, will be in 2028-29. We propose to include all undergraduate students on one-year courses within the NSS once in-year student data is available for all providers. This would be no earlier than the 2029 survey and, depending on timings, potentially in 2030.
117. The providers affected by this issue are mostly further education colleges and many colleges already return in-year data through the ILR. This means we could start surveying students on one-year courses at colleges with good quality in-year higher education data earlier.
118. Until in-year data is available for all providers, we propose allowing further education colleges to choose whether to include their one-year courses in the NSS. They could do this if they provide the required in-year data through the ILR.
119. In this interim period, we would survey students on one-year courses in the same survey window and using the same questionnaire as other students. As numbers are likely to be relatively low until we include all providers, to benchmark the data effectively we would need to use a population that includes all students, rather than just students on one-year courses. However, because we use level of study as a benchmarking factor this ensures that we will use only students on similar types of course in the comparison.

Providers without sufficient data for either aspect

120. We recognise that some providers, particularly newer or smaller providers, may not have sufficient data in either aspect. Data availability for these providers would improve over time as newer providers operate for longer, providers recruit additional cohorts of students, or benefit from extensions to NSS coverage. For newer providers, we may have data from before they registered with us. For example, they may previously have delivered courses under a subcontract. We would consider this data when deciding whether there is enough data to assess the provider.
121. In Proposal 2 we set out how we would aim to schedule providers for assessment when sufficient data is available for TEF ratings, including deferring some providers into a fourth year if that would improve data sufficiency. Where sufficient data for TEF ratings in either aspect remains unavailable for an extended period, we would assess providers only against the minimum quality requirements for the relevant aspect or aspects. This would make sure that we assess all providers, including newer, smaller and specialist providers that may not have enough evidence for a TEF rating.
122. We recognise that providers may want to apply for a TEF rating even if they have not had enough data for both areas over a long time. If this is the case, we would consult later on an alternative assessment approach that could enable providers to apply for a TEF student experience rating without sufficient NSS data. The approach would likely involve provider and student submissions as well as a visit. We would anticipate consulting on this in time to implement such an approach during the fourth year.