Proposals for a revised approach to assessing and rating quality

Key points of this proposal

• Where we select a provider for an enhanced assessment (see Proposal 3), we would gather additional evidence and normally conduct a visit as part of its quality assessment.
• We would gather additional evidence relating to the risks identified in the initial risk assessment to decide whether potential concerns can be ruled out or substantiated.
• Additional evidence gathering would normally focus on one or more defined themes.
• We would inform providers of the identified risks, when we notify them of their quality assessment.
• The assessment team would conduct visits and gather additional evidence during the provider submission window, and providers would have the opportunity to respond in an annex to their submissions.
• Assessment teams would consider the additional evidence alongside all other evidence when reaching judgements.
• We would incorporate findings based on the additional evidence gathering into the assessment report rather than report separately.

Explanation

Purpose of enhanced assessments

264. Following stage one we explained why we had moved away from an approach in which every provider participating in the TEF would receive a visit as part of its assessment. We remain of the view that requiring assessment visits for all providers would not be proportionate. It would significantly increase the cost and burden of the system, lengthen assessment timescales and reduce the number of providers that could be assessed each year.
265. As we have developed the proposals, we consider it would be appropriate to gather additional evidence, including through visits, for some providers. This would be where we identify risks to quality (as described in Proposal 3).
266. We propose that enhanced assessments would gather additional information where the initial risk assessment identifies potential risks relating to the quality of the student experience and the available evidence may not provide sufficient assurance that a provider is delivering satisfactory quality. The purpose of the enhanced assessment would be to enable the assessment team to gather additional evidence to test the issues identified through the risk assessment. It would enable the team to either rule out concerns, or determine that quality is unsatisfactory, when considered alongside the NSS indicators and submissions. If concerns are ruled out, there would be no limit on the rating that could be awarded.
267. We consider that limiting visits and other additional evidence gathering within the quality system on the basis of risk provides a more proportionate balance between assurance, burden and cost than requiring routine visits to every provider. It would allow additional evidence to be gathered where necessary without introducing routine visits for all providers.
268. Quality investigations to assess specific concerns about meeting minimum quality requirements may also be conducted. Proposal 3 sets out how the initial risk assessment might lead to an investigation instead of a cyclical quality assessment. Proposal 21 explains the circumstances in which concerns identified through a quality assessment may lead to a more detailed investigation that might include further visits to the provider.

Timing, focus and method of enhanced evidence gathering

269. We propose that, when providers are notified that they have been selected for a quality assessment, we would also inform them if we will conduct an enhanced assessment. We would inform providers about:
• the risks identified
• the additional evidence required
• whether a visit would be undertaken.
270. We propose that the provider would supply the additional information, and the assessment team would carry out visits, during the window for the provider submission. The additional information we require would be existing information held by the provider, such as data on academic misconduct. We would normally give the provider three weeks to provide this information. Visits would most likely take place during October. This approach would, if necessary, give the provider an opportunity to respond to the assessment team’s observations in its submission. It would also enable the assessment team to have available all the evidence to inform its assessment, before beginning its detailed assessment.
271. We considered whether it would be better to run the assessment visits after assessors have reviewed provider submissions. This could enable the assessment team to use the provider submission to focus their visit on a narrower range of questions. However, we do not propose this approach because it would be impractical for assessors to conduct the visits at the same time as carrying out large numbers of assessments. It would delay the determination of ratings which would not be in the student interest.
272. Enhanced assessments, including visits, would focus on gathering evidence about criteria related to the areas of risk identified through the initial risk assessment. We considered whether the visits should be individually tailored to every pattern of risk identified at each provider. However, we consider that a more structured approach would improve consistency, transparency and efficiency.
273. We therefore propose that enhanced assessments would normally focus on one or more defined themes:

1. Academic challenge and assessment rigour: Where available information suggests that students may be reporting positive experiences or outcomes despite a risk that courses are insufficiently challenging or that assessment lacks sufficient rigour.
2. Capacity to deliver quality: Where risk factors suggest that changes in provision may have created risks relating to staffing, learning resources, student support or course delivery.
3. Oversight of higher-risk provision: Where risk factors suggest increased quality risks because of particular delivery models, partnership arrangements or other areas requiring stronger oversight.

274. Enhanced assessments would not normally explore more than two themes.
275. Where concerns appear broader, more numerous or potentially serious, systemic or widespread, a quality investigation is likely to provide a more appropriate mechanism rather than an enhanced assessment.
276. Enhanced assessments would normally include a visit to the provider. The visit would be conducted by the assessment team that will review all the evidence and make recommendations on the provider’s assessment outcomes, as described in Proposal 16.
277. Depending on the circumstances, the enhanced assessment could also involve:
• requests for additional information
• requests for supporting evidence
• review of samples of provision
• access to relevant systems or materials.
278. The information requested would depend on the risks identified. For example, this could include information relating to staffing, academic withdrawal rates, student attendance and engagement, student support, learning resources, assessment practices, academic misconduct, assessed work or other matters relevant to the identified risks. Following the visit, the assessment team might request further evidence or clarifications from the provider.
279. The focus of each visit would depend on the themes that the assessment team is exploring. Assessors would gather evidence relevant to the areas of risk and seek to establish whether potential concerns are substantiated or can be ruled out. For example, if exploring the theme of academic challenge and assessment rigour, the assessors may place greater emphasis on observing teaching activities, reviewing assessed work and assessment design.
280. A summary of the main observations from the visit would be shared with the provider to comment on factual accuracy, prior to the provider’s submission deadline. If necessary, the provider would also be able to respond to those observations as an annex to its submission.
281. Evidence gathered through the additional information request and visit would then form part of the overall evidence base for the TEF or minimum requirements assessment and would be considered alongside all other evidence. The assessment report would explain how the additional evidence, including evidence from the visit, informed the assessment team’s judgements.
282. A provider would be required to facilitate and cooperate with the visit and this would be a requirement of condition B6. We propose that failure to cooperate with the visit would likely result in a breach of the condition.

Enhanced assessment of providers with subcontractual partnerships

283. In Proposal 6 we set out that the students taught by a provider and those studying through subcontracted arrangements would be equally weighted in the provider’s assessment. The provider’s indicators and submission would include information about all these students as set out in Proposal 8, Proposal 9, Proposal 14 and Proposal 15.
284. Where we conduct a visit to a provider that also subcontracts delivery of teaching (the ‘lead’ provider), we would not normally visit delivery partners as part of a lead provider’s visit. This is because it could duplicate assessment activity and would not be proportionate in all cases. Instead, where we identify risks related to a lead partner’s subcontracted provision, we are likely to request additional information from the lead provider and take account of evidence from relevant assessments of delivery partners. We may also:
• schedule its main delivery partner or partners for assessment in the same year
• conduct an enhanced assessment and visit the delivery partner(s) as part of its own assessment, if risks are identified at the delivery partner(s)
• provide the lead partner’s assessment team with the notes of the visit to the delivery partner(s) to take into account when assessing the lead partner.
285. We recognise that subcontractual arrangements vary considerably in scale and nature. We would therefore normally take this approach only where a partnership covers a significant proportion of a provider’s students.
286. Where we identify risks and conduct an enhanced assessment of a subcontractual delivery partner, we would not visit the lead provider as part of this assessment.
287. We do not propose to take this approach for validated-only provision. Under Proposal 6 we set out a different approach to assessing validated-only provision that relies solely on data and provider submissions.