Proposals for a revised approach to assessing and rating quality

Key points of this proposal

• The earnings measure would treat graduates as having a positive outcome if they earn above a specified threshold in any of the three years after graduation, or are studying for a higher education qualification three years after graduation.
• The use of skills measure would treat agreement to the Graduate Outcomes (GO) survey question about whether graduates are using what they learned during their studies as a positive outcome.
• We would benchmark the earnings and use of skills measures using broadly the same factors as the existing progression measure, to take account of student characteristics, subject of study and geographic factors.
• We would present student outcomes data in a combined view of taught and subcontracted-out provision, as well as separate views for taught, subcontracted-out and validated-only provision.
• We would present continuation and completion indicators by mode and level of study to reflect that minimum thresholds are set for each mode and level.
• We would also ‘split’ the indicators subject, course characteristics and student characteristics, including enhanced disability splits.
• We would present performance in relation to the provider’s benchmarks, as well as absolute numerical thresholds for continuation and completion.

Explanation

342. As decided following stage one, we will use a basket of three post-study measures alongside continuation and completion to inform the assessment of student outcomes.

Additional post-study measures

343. We decided to use three post-study measures to avoid overreliance on any one of them. We decided to introduce two additional post-study measures to supplement progression: use of skills and earnings. We set out in this section a summary how we propose to define and benchmark them. Annex E includes further detail on the definitions and Part 4.9 of the draft guidance includes proposed benchmarking factors for the new measures.
344. The use of skills measure would use responses to the GO survey question ‘I am utilising what I learnt during my studies in my current activity/study/work.’ Responses of ‘Agree’ or ‘Strongly agree’ would count as positive outcomes, regardless of the activity the graduate was undertaking. We are proposing this as our analysis shows that across all the different main activities being carried out, there are graduates who say they are using their skills and graduates who say they are not.
345. We propose the earnings measure would treat graduates as having achieved a positive outcome where they either:
• earn above a threshold based on average UK earnings for 25- to 29-year-olds (including graduates and non-graduates)
• are studying for a higher education qualification, three years after graduation.
346. We propose to assess earnings using Longitudinal Education Outcomes (LEO) data. To recognise that graduates’ earnings often develop over time, we would include only graduates who are in employment or higher education study three years after graduation. However, for those graduates we would use their highest recorded earnings from any of the first three years after graduation, rather than earnings in year three alone.
347. We consider this approach provides a fairer picture of graduate success. Some graduates have higher earnings in years one or two than in year three because of later changes in working patterns, career choices or personal circumstances. Using the highest earnings recorded across the three-year period helps reduce the impact of these factors and partly mitigates limitations in the LEO data, which does not identify whether graduates work full-time or part-time. Our analysis suggests this approach particularly increases the number of female graduates who would be treated as having a positive outcome, compared with only considering earnings in year three.
348. We considered using earnings measured four or five years after graduation. The Institute for Fiscal Studies recommended considering the highest earnings between three and five years after graduation for regulatory purposes. However, we consider that a longer period would increase the lag between study and assessment. In our view, three years strikes a better balance between allowing graduates time to establish themselves in employment and ensuring the measure remains relevant to the provision currently being delivered.
349. We also considered including all graduates, regardless of whether they were in work or study three years after graduation. This would mean that graduates who are unemployed would count negatively within the measure. However, evidence suggests that many graduates who are not earning are engaged in other activities rather than seeking employment. We therefore do not think it would be appropriate to treat all non-earners as negative outcomes, and propose to restrict the measure to those who are in employment or higher education study three years after graduation. 
350. We propose initially to restrict both the use of skills and earnings measures to UK-domiciled graduates in the same way as the existing progression measure. In the case of the earnings measure, this is because the LEO data is based on UK tax records, and in the case of the use of skills measure because of the lower levels of response from overseas students to the GO survey, which make it more difficult to draw reliable conclusions from the response data. For the progression and use of skills measures, we would keep this position under review and consider extending coverage if response rates to the GO survey from overseas students improve in future.
351. Indicators for both measures would be benchmarked to take account of broadly the same factors as the existing progression measure: subject of study, level of study, geography of employment (geography of earnings for the earnings measure), student entry qualifications, year of qualification, and a composite measure of students’ socioeconomic characteristics (ABCS). A list of the proposed factors is available in Part 4.9 of the draft guidance and in our benchmarking analysis, published alongside the draft guidance. The analysis identifies these factors as having a material impact on use of skills and earnings, and supports using factors that are consistent with those used for the progression measure. We also consider there is value from a user perspective in taking a consistent approach where there is not a compelling reason to do otherwise.

Structure of the indicators

352. We would structure the student outcomes indicators and splits in the same way as the NSS indicators, as set out in Proposal 8, with one exception: we propose to change the level at which we present the continuation and completion indicators.
353. In Proposal 13 we explain why we consider continuation and completion outcomes should be assessed through both absolute and benchmarked performance. We have considered how best to construct and present these indicators to enable this. We currently set minimum absolute thresholds for each combination of mode and level of study, and we therefore present the B3 continuation and completion indicators by mode and level of study. To enable the assessments to consider both absolute and benchmarked performance, we propose also to produce the overall benchmarked indicators for continuation and completion by each mode and level. This would make it more straightforward for providers and assessors to consider how each indicator performs against both absolute thresholds and the provider’s benchmarks. For example, it would enable providers and assessors to identify where an indicator is below benchmark but substantially above the minimum threshold.
354. We would present the overall NSS and post-study indicators for all undergraduate students by mode of study, rather than by level and mode of study. This is consistent with our current approach to constructing the TEF indicators.