A set of guiding principles have been established for selecting and applying the factors used in benchmarking when assessing higher education providers.
The principles are intended to be guiding rather than binding, and we expect to consider the merits of any benchmarking factor's selection on a case by case basis.
Where possible, new benchmarking values should not be published before giving the participating providers an opportunity to comment on errors of fact in the data from which the benchmarking factors are formed
The guiding principles are outlined below.
Principles for the selection of benchmarking factors
- The benchmarks provided in association with performance indicators should use factors that allow the benchmarking approach to take account of context and differing provider characteristics, thereby supporting fair comparison of indicators across the higher education sector.
- The selection of benchmarking factors should be evidence-based and robust, conforming to recognised best practice in the production of statistical information. Details of the selection process should be published for the benefit of providers and other users or interested parties.
- Benchmarking factors should be applicable to higher education provision delivered across all of the UK by higher education providers of all types, to the extent that they are included in the coverage of the associated indicator.
- Benchmarking factors should be outside a higher education provider’s control, or undesirable for it to control for.
- Use of data for benchmarking purposes must not knowingly create perverse incentives or lead to perverse behaviour.
- Once other factors have been accounted for, statistical modelling should identify a remaining correlation between a benchmarking factor and what is being measured.
- Once other factors have been accounted for, statistical modelling should identify that the performance being measured is not uniformly distributed across the attributes within a benchmarking factor, and that differences between these attributes are non-trivial.
- A benchmarking factor should not be uniformly distributed across providers or performance units; rather, the factor should differentially affect the benchmarks that are calculated.
- Where possible, a benchmarking factor should be a direct measure, rather than a proxy.
- As far as possible, the selection of benchmarking factors should limit the extent to which a benchmark value can be determined by a single higher education provider. The selection of a benchmarking factor (and the subsequent grouping of attributes within it) should not compromise the statistical integrity of the broader benchmarking approach.
- Benchmarking factors should normally have longevity, with these statistical properties observed to continue over time.
- Data used for the benchmarking factors should be of high quality, collected in a consistent and fair way across the sector; it should have a good sample base and use consistent and transparent definitions.
- Where possible, benchmarking values should not be published before giving the participating higher education providers an opportunity to comment on errors of fact in the data from which the benchmarking factors are formed.
- Where possible, benchmarking factors should be drawn from existing data sources. Dialogue with the sector or its representatives should form part of a careful consideration of any proposal to collect further data for use as a benchmarking factor. Within this, the additional burden of data collection should be compared with the anticipated use and usefulness of the data.
Principles for defining groupings of attributes within benchmarking factors
- The grouping of attributes within benchmarking factors should be determined through consideration of sound evidence.
- The number of categories formed when grouping attributes within benchmarking factors should be the minimum for the benchmarking factor to be effective. The number and definition of the groupings should not compromise the statistical integrity of the broader benchmarking approach.
- The attributes that form a grouping should share a consistency of student backgrounds, outcomes or behaviours with respect to the indicator to which they refer. The like-with-like grouping of attributes should be based on the evidence of statistical analysis.
- The grouping of attributes within benchmarking factors should avoid creating groups in which numbers of students possessing those attributes are either very small or very large.
- The grouping of attributes within benchmarking factors should make practical sense, to form coherent groups which share a qualitative similarity.
- The grouping of attributes within benchmarking factors should vary across indicators only when there is a clear rationale. Where variations are necessary, those deviations should use other groupings that exist elsewhere in a hierarchical view of the benchmarking factor in question; at a more aggregated or disaggregated level according to need.
- The grouping of attributes within benchmarking factors should be reviewed periodically to ensure that it continues to comply with these principles.