Proposals for a revised approach to assessing and rating quality

Key points of this proposal

• Each year, we would carry out an initial risk assessment of every provider selected for a quality assessment based on factors that may indicate risks to quality, including information from our regulatory work, notifications, student outcomes and indicators of students’ experiences.
• The risk assessment would aim to identify risks and potential concerns about compliance with quality conditions to establish whether a quality investigation is needed, or if additional evidence would be needed to assess the student experience.
• Providers would be categorised into one of five risk categories ranging from no or low risk to high risk.
• Most providers would proceed through a desk-based assessment.
• Where we identify particular risks, we may gather additional evidence and conduct a visit as part of an enhanced assessment of the student experience.
• Where we identify concerns about a provider’s student outcomes indicators not meeting minimum B3 requirements, we would invite the provider to submit relevant contextual information about these indicators.
• We would inform providers at the start of the process about any identified risks, any requests for additional information and whether a visit would be required.
• Providers would not be able to apply for a TEF rating while a quality investigation is underway.

Explanation

Initial risk assessment

77. Following stage one we decided to introduce a risk monitoring tool that sets out the factors associated with increased risks to quality. We decided we will make use of the risk factors and indicators to inform decisions about when we should assess a provider in the TEF. We have published the quality risk monitoring document alongside this consultation.²
78. We propose carrying out an initial risk assessment for each provider selected for a quality assessment. This would help us decide how closely to assess the provider, whether we need more evidence and whether concerns about compliance mean that we should carry out a quality investigation instead.
79. We consider that we could appropriately assess most providers using a standard package of desk-based evidence. This would normally include:

Table 3: Standard evidence requirements

80. However, we do not think that a standard evidence package will always provide sufficient assurance about quality where we identify particular risks. We may need to explore these risks through additional evidence and assessment.
81. For example, in some cases NSS indicators may provide a positive picture of quality while other information available to us could suggest potential risks or concerns. Additional evidence and assessment would provide a mechanism for the assessment to examine identified risks and test whether we can rule out potential concerns about quality. We prefer this approach to starting a quality investigation whenever we identify risks. It allows us to examine and resolve many concerns through the regular assessment process.
82. The initial risk assessment would consider information, including:

• • notifications
  • reportable events
  • regulatory intelligence from external organisations and agencies
  • NSS indicators
  • student outcomes indicators
  • previous regulatory findings and compliance activity
  • information about the size and shape of provision
  • other factors within the OfS quality risk factors document.

83. We would look at the risk factors together and use our judgement to decide how closely to examine the provider. The risk assessment would not in itself show that a provider is failing to meet minimum requirements and would not predetermine the outcome of the provider’s assessment.
84. We would not normally base the risk assessment on a single risk factor or isolated data point. However, in some circumstances a single source of information may indicate concerns that are sufficiently serious to justify a higher-risk categorisation.
85. We have decided after stage one that we will update the risk factors over time. We will also consider which indicators could help us identify risks to examine during an assessment. For example, we plan to discuss with professional, statutory and regulatory bodies what information they could share with us, such as pass rates.

Risk categories

86. To help us make consistent decisions, we would categorise the level of risk identified in the initial risk assessment. This risk categorisation would not be a judgement about quality itself and we would not publish it. Its purpose would be to inform decisions about the most appropriate level of scrutiny. The initial risk assessment would place a provider in one of five broad risk categories.

Table 4: Risk assessment categories

87. In deciding on the risk category, we would consider both the number of identified risks and their significance. A single serious concern from specific intelligence may result in a higher-risk categorisation than several lower-level concerns.
88. A categorisation that a provider is high risk would be more likely where:
    1. multiple sources of information point towards the same or a similar concern, for example it has a high volume of notifications as well as multiple risk factors
    2. the information available suggests a potentially significant impact on students
    3. progress to resolve previous serious concerns remain untested.

Determining the level of scrutiny

89. The outcome of the risk assessment would inform the level of scrutiny applied to the assessment, or whether a quality investigation is more appropriate:
    1. Providers in Categories 1 and 2 would normally proceed through a standard assessment.
    2. Providers in Categories 3 may proceed through an enhanced assessment, involving additional evidence gathering.
    3. Providers in Categories 4 would normally proceed through an enhanced assessment, involving additional evidence gathering.
    4. Providers in Category 5 would normally be subject to a quality investigation before proceeding through assessment or becoming eligible to receive a TEF rating.
90. If we identify more providers with risks or concerns than we can investigate in a year, we would prioritise those with the highest risks and the greatest potential impact on students.
91. Where we identify risks that we can explore by gathering more evidence, we may undertake an enhanced assessment.
92. An enhanced assessment would gather more evidence about the risks to the student experience that we had found. It would establish whether those risks can be satisfactorily explained or indicate concerns about compliance.
93. An enhanced assessment would not assume that concerns exist. Its purpose would be to provide additional assurance.

Quality investigations

94. Some concerns about the quality of the student experience cannot be appropriately addressed through a minimum requirements or TEF assessment, even where enhanced evidence gathering is used.
95. We will therefore continue to undertake quality investigations where information available to the OfS suggests that concerns about compliance require more detailed, urgent or targeted examination.
96. A quality investigation may be appropriate where:

• • information suggests serious concerns about compliance with quality conditions
  • concerns appear systemic or widespread
  • concerns require urgent investigation
  • a quality assessment identifies issues that are likely to require regulatory intervention if substantiated.

97. Where we decide that a quality investigation is required, this would normally take place before the provider proceeds through its cyclical assessment and is able to apply a TEF rating.
98. An investigation would look at whether a provider meets the minimum quality requirements, whether the concerns about compliance are supported by evidence and whether we need to take regulatory action. Proposal 21 explains how the findings of a quality investigation could affect TEF ratings and minimum requirements outcomes.
99. If we start an investigation, we propose that, in exceptional and very serious cases, we could suspend or remove any TEF rating above Bronze until the investigation ends. A provider with credible and serious quality concerns based on specific information or evidence would not be eligible to apply for a TEF rating. We therefore do not think it should continue to receive the benefits of a higher rating while we investigate those concerns.

Concerns about student outcomes

100. Where our initial risk assessment identifies student outcome indicators that may be below the minimum B3 requirements, we would inform the provider of these and invite it to submit relevant contextual information about these indicators. We would assess any concerns about compliance with condition B3 in the same way, whether or not the provider applies for TEF ratings.
101. Proposal 15 sets out further detail about how we would identify student outcomes indicators of concern, and what we would consider relevant contextual information.