Proposals for a revised approach to assessing and rating quality

Key points of this proposal

• All providers would be assessed by a small assessment team consisting of academic assessors, student assessors and an OfS staff member.
• A lead assessor from each team would present the team’s recommendations to the Quality Assessment and TEF panel.
• The Quality Assessment and TEF panel would consist of two subpanels, which would consider assessment team recommendations, calibrate and make decisions about ratings and outcomes for half the providers.
• Each subpanel would comprise experts from different types of providers, student representatives and senior OfS staff, and would be chaired by a member of the OfS executive.
• Providers could make representations about Requires improvement and Bronze ratings or Unsatisfactory outcomes, where they identify a procedural irregularity or, for TEF ratings, where more recent NSS data shows a marked improvement in performance.
• Representations would be reviewed by a different assessment team who were not involved in the original assessment, and the final decision would be made by the other subpanel.

Explanation

392. The structure of the assessment teams is illustrated in Figure 2.

393. Following stage one we decided that assessments would be carried out by assessors with expertise in developing and delivering of higher education in diverse provider contexts, and experience of being and representing students. We also said we would consider further whether OfS staff should contribute as assessors. We now propose who would be involved in assessments and how recommendations and decisions about TEF ratings or minimum requirements outcomes would be made.
394. We propose that small assessment teams would conduct assessments. Assessment teams would review all the evidence in detail and make recommendations on the ratings and outcomes to a Quality Assessment and TEF panel that would decide the ratings and outcomes. We would seek to ensure that teams collectively possess expertise relevant to the nature, scale and characteristics of the providers being assessed. This may include experience of specialist provision, particular delivery models or relevant subject areas.
395. Each assessment team would comprise at least two academic assessors, at least one student assessor and an OfS staff member. This approach would bring together expertise in higher education delivery, student experience and regulatory requirements. Where practicable, assessment teams could include two student assessors, to strengthen the student-centred nature of assessments.
396. Student assessors would normally be current students or recent graduates with relevant experience of student representation or higher education quality improvement activities. Relevant experience may include institution-wide representative roles, membership of provider committees, course representative positions or comparable student engagement activities.
397. We propose that OfS staff would be full members of assessment teams and would contribute to all discussions and judgements. We consider there are several benefits of this proposal. OfS staff assessors would support consistency in the interpretation of indicators and assessment criteria, and understanding of diverse provider contexts. They would provide expertise in applying regulatory requirements and assessing whether providers meet these. We think this is important given that we are proposing to operate a single assessment framework, and the regulatory consequences linked to TEF ratings and minimum requirements outcomes. They could also draft the assessment reports on behalf of the assessment team, helping to limit other assessors’ workloads.
398. Within each team one of the academic assessors would be the lead assessor. The lead assessor would lead the team to form an evidence-based recommendation. They would approve the assessment report drafted by the OfS staff member, and present their team’s recommendations to the panel.
399. To support assessment across a diverse higher education sector, we have already decided to recruit a large and evolving pool of academic and student assessors through open recruitment. We consider that maintaining a larger assessor pool will provide flexibility in forming assessment teams, support the management of conflicts of interest and help ensure that appropriate expertise is available for each annual cohort of providers.
400. We would support assessors through training and calibration activities and would provide common guidance and procedures to support a consistent approach to the interpretation of indicators, evidence and assessment criteria.
401. We will aim to ensure representation from across the diversity of the higher education sector, including different provider types, modes of provision and educational contexts. We intend to recruit a broader range of student assessors compared with previous TEF assessments. To support this, we have considered suggestions made in response to our stage one consultation and intend to work with representative bodies, mission groups and other provider networks to encourage applications from a broad range of candidates.
402. To encourage a diversity of applicants, we will seek to limit the workload expected of assessors, for example through OfS staff assessors drafting the reports on behalf of the team. We will also provide some flexibility in the caseload expected of individual assessors. We recognise the practical challenges that staff from smaller providers in particular may have in taking part as assessors. To help ensure representation from these institutions, we propose that as well as academic staff we may recruit assessors with experience gained in leadership, quality management and professional services roles that contribute directly to the student academic experience.

Decision making

403. A Quality Assessment and TEF panel would consider recommendations from assessment teams and make decisions on TEF ratings and minimum requirements outcomes. The panel would operate as two subpanels, which would each include academic members from a range of provider types, student representatives and senior OfS staff responsible for quality assessment delivery. Lead assessors from assessment teams would attend meetings to present and discuss recommendations, but would not be members of the panel responsible for the decisions. Members of the panel would be appointed separately from assessors, to ensure an appropriate breadth of expertise to support consistency and make decisions about the ratings and outcomes.
404. We consider that the use of a panel to decide ratings and outcomes would support calibration and help ensure consistency across assessments, particularly where cases are finely balanced. Panel members would operate within the same common framework of published criteria, guidance and procedures, supported by training and calibration activities. We consider that operating the panel through two subpanels would help ensure members have sufficient time to consider a large volume of recommendations, and would also enable representations about ratings and outcomes to be considered independently of the original assessment.
405. The panel would be established as a committee of the OfS with two co-chairs. Co-chairs would be members of the OfS executive, and would each chair a subpanel. This would help ensure that ratings, outcomes and associated restrictions and interventions align with our regulatory requirements and approach.
406. We have considered how to ensure consistency across the two subpanels in an efficient way. We propose to appoint two academic members as deputy chairs and they, along with some senior OfS staff members, would rotate between the subpanels to support calibration and consistency across the two subpanels.
407. We consider that the proposed structure establishes a clear distinction between assessment by teams of people with an appropriate mix of expertise, and decision making by a committee of the OfS. This would ensure that ratings and outcomes are informed by independent expertise while the final decisions and associated consequences remain our responsibility as regulator.

Representations on ratings and assessment outcomes

408. Following stage one we decided that providers would be able to make representations only about a provisional Bronze or Requires improvement rating for an aspect, before a final decision is reached. We now propose providers would also have an opportunity to make representations about an Unsatisfactory outcome and its associated regulatory consequences.
409. We propose that providers would have 28 days to submit these representations.
410. Representations would normally be given weight only where either:
     1. There has been a procedural irregularity in the application of the published guidance (for example, incorrect application of the criteria or material evidence submitted by the provider was not taken into account).
     2. More recent NSS data published after the provider’s submission shows a marked improvement in specific areas judged to be lower quality.
411. We would not normally give weight to representations that try to strengthen the original submission with additional context or evidence, or revisit matters that have already been considered through the assessment process. Nor would we normally give weight to representations that introduce additional evidence that was available at the time of submission, challenge the exercise of expert judgement where published guidance has been followed, or question the design of the TEF. We consider that providing clarity on where representations would normally be given weight supports transparency, fairness and efficient decision making.
412. We have considered the implications of an additional year of data becoming available after the submission was made, but before the outcomes have been decided. We propose that, for consistency and efficient use of OfS resources, there is a standard timeframe in scope of the assessment. As set out this would be the four years immediately prior to submission. Any change in performance after the submission has been made would not normally be relevant to the assessment.
413. However, we recognise that a rating of Requires improvement or Bronze, or an Unsatisfactory outcome, could have significant impacts on a provider. We propose that where the latest data (published after the submission but before final outcomes are decided) shows a marked improvement, this should be taken into account before awarding one of these ratings or outcomes.
414. Data on student outcomes is typically available in the spring, during the assessment process. We propose that during the assessment, if assessors are minded to recommend a provisional rating of Requires improvement or Bronze, or an outcome of Unsatisfactory, for student outcomes, they would consider whether the latest year of data shows a marked improvement in student outcomes. If so, they would take this into account during the assessment. We do not therefore propose to give weight to representations that refer to improvements shown in the latest year of student outcomes data, because this will already have been taken into account.
415. The later year of NSS data is typically available in June and there would not, on our proposed timeline, be time for assessors to take this into account during the assessment. We therefore propose that if the provider receives a provisional student experience rating of Requires improvement or Bronze or outcome of Unsatisfactory, and the latest NSS data shows a marked improvement in the areas judged to be of lower quality, it could raise this in representations for consideration before the final rating is awarded.
416. Where we receive representations, they would be considered initially by assessors who were not part of the original assessment team. The mix of assessors would depend on the nature and scope of the representations made. They would make recommendations to the panel on whether the provisional rating should be changed, or the outcome, individual criterion judgements or both. If so, they would amend the assessment report accordingly. The subpanel that would consider the recommendations and make the final decisions would be the subpanel that did not make the provisional decision. We consider this approach ensures independent consideration of representations, while maintaining consistency through a common decision-making structure.