Benchmarking

The core principles relating to the factors being considered for benchmarking are:

• The selection of benchmarking factors should be fit for purpose, evidence-based and robust, conforming to recognised best practice in the production of statistical information. In particular:
  • Details of the selection process should be published for the benefit of providers and other users or interested parties.
  • The selection of benchmarking factors should vary across different student outcome and experience indicators only when there is a clear and valid rationale.
  • The number and definition of benchmarking factors selected should not compromise the statistical integrity of the broader benchmarking approach.
• Benchmarking factors should be applicable to, and available for, all types of providers across England that are delivering the higher education provision for which the indicator is measuring students’ outcomes or experience.
• Benchmarking factors should contribute to an overall benchmarking approach which supports fair comparison of indicators across the higher education sector. A candidate benchmarking factor should therefore have relevance to help explain the context or differing characteristics of a provider’s students or provision.
• The benchmarking approach should neutralise the effect of characteristics on a provider’s performance where this is consistent with policy objectives. This approach guards against inadvertently creating incentives for providers to change their behaviour in terms of the students they recruit or the range of provision they offer in ways that could undermine our ability to meet our duties around access and participation, and competition. It does not imply that it is acceptable for some student groups to receive lower quality provision, but recognises that this is currently the case, and the risks of not controlling for it. The benchmarking approach should only neutralise the effect of characteristics where there is such a risk of negative unintended consequences, as otherwise it risks creating perverse incentives.
• Benchmarking factors should primarily reflect structural factors that contribute to variations in student outcomes or experience which are outside of a provider’s control, or undesirable for it to control for. This means that characteristics of the provider will not normally act as benchmarking factors.
• In selecting the range of benchmarking factors to apply for a given indicator, the need to preserve the statistical integrity of the broader benchmarking approach requires that consideration should be given to limit the number of factors on the basis of:
  • The size of the population for which the effect occurs: it is unlikely that a factor where the effect is limited to a small population will be selected where there are other factors with similar effects that have broad applicability.
  • The distribution of the population for which the effect occurs: it is unlikely that a factor where the effect is limited to a population concentrated in a small subsection of providers will be selected where there are other factors with similar effects that have applicability to a wider cross-section of provision.
  • The nature of the other candidate factors: where there are a number of similar candidate factors (for example, measures of disadvantage), it will normally be the case that only the one that has the greatest effect should be selected so that a balance of factors is achieved.
• The factors used in benchmarking should be reviewed at regular intervals, to check that the evidence for, and applicability of, the approach remains current and fit for purpose, and to consider the impact achieved by previous benchmarking exercises.

The availability and data quality of candidate benchmarking factors should be considered in relation to the principles as follows:

• The quality of data items considered as candidate benchmarking factors should be understood and judged to be of sufficiently high quality for use in a benchmarking exercise. The data items should normally be collected in a consistent and fair way across the sector; it should have a good sample base and use transparent definitions.
• Where possible, benchmarking factors should be drawn from existing data sources. Any proposal to collect further data for the purpose of a benchmarking factor should be carefully considered against the principles for data burden included within the OfS data strategy.

The principles for the statistical properties that candidate benchmarking factors should demonstrate are:  

• Statistical models that seek to account for a range of characteristics should identify a remaining correlation between the benchmarking factor and the student outcome or experience that is being measured.
• Once other factors have been accounted for, statistical modelling should identify that the performance being measured is not uniformly distributed across the attributes within a benchmarking factor, and that differences between these attributes are non-trivial.
• A benchmarking factor should not be uniformly distributed across providers or performance units; rather, the factor should differentially affect the benchmarks that are calculated, meaning that factors which are distributed unevenly across providers or performance units should be considered as stronger candidates to be used as benchmarking factors.
• Where possible, a benchmarking factor should be a direct measure, rather than a proxy.
• As far as possible, the selection of benchmarking factors should limit the extent to which a benchmark value can be determined by a single provider. The selection of a benchmarking factor (and the subsequent grouping of attributes within it) should not compromise the statistical integrity of the broader benchmarking approach.

Once benchmarking factors have been selected, the principles for defining groupings of the attributes within the benchmarking factor are:  

• The grouping of attributes within benchmarking factors should be fit for purpose and determined through consideration of sound evidence.
• The number of categories formed when grouping attributes within benchmarking factors should be the minimum for the benchmarking factor to be effective. The number and definition of the groupings should not compromise the statistical integrity of the broader benchmarking approach.
• The grouping of attributes within benchmarking factors should avoid creating groups in which numbers of students possessing those attributes are either very small or very large in the sector overall. The effect of creating groups that are known to be very small or very large at individual provider level should be acknowledged where they cannot be avoided.
• The attributes that form a grouping should share a consistency of student backgrounds, outcomes or behaviours with respect to the indicator to which they refer. The consistency of attributes should be understood from the evidence of statistical analysis.
• The grouping of attributes within benchmarking factors should make practical sense, to form coherent groups which share a qualitative similarity.
• The grouping of attributes within benchmarking factors should vary across indicators only when there is a clear and valid rationale. Where variations are necessary, those deviations should use other groupings that exist elsewhere in a sector-wide hierarchical view of the benchmarking factor in question, at a more aggregated or disaggregated level according to need.
• The grouping of attributes within benchmarking factors should be reviewed periodically to ensure that it continues to comply with these principles.