Measuring completion rates for performance linked funding

Purpose

This paper provides an opportunity for sector discussion on options for:

• measuring the qualification completion rate in the Student Achievement Component (SAC) performance-linked funding framework; and
• measuring the programme completion rate in the Industry Training Organisation (ITO) performance-linked funding framework.

Background to performance-linked funding

In March 2010, the Government announced it would introduce an element of performance-linked funding to the tertiary education system. This will commence from 2011.

Performance-linked funding is intended to:

• provide incentives for performance improvements for Tertiary Education Organisations
• improve educational outcomes for students and employers
• improve value for taxpayers’ money
• while at the same time minimising any disruption to well performing TEOs.

Key features of performance-linked funding

The key features of performance-linked funding have now been agreed and can be read on the performance-linked funding page of the TEC website. These include:

• the percentage of funding at risk
• the timing of the implementation of the performance-linked funding mechanism
• the groupings of New Zealand Qualification Framework (NZQF) levels
• which of the educational performance indicators will be used for each grouping of NZQF levels and the weightings on each indicator
• the adjustment for part time learning to be applied in the SAC performance-linked funding framework, and
• how thresholds will be set.

However, there are a number of options that could be used to measure the qualification and programme completion rates.

The following documents sets out three possible options, and asks for your feedback and views on these options.

Introduction to measuring completion rates

In September 2010 the TEC published educational performance information for SAC-funded tertiary providers. This included a qualification completion rate based on data for 2009 only.

The qualification rate that was used shows the proportion of tertiary providers’ students who complete qualifications as measured by the EFTS-weighted qualification completion rate.

Measuring qualification completions in this way may result in apparent peaks and troughs in performance when compared to previous years for some tertiary providers. Feedback has indicated that these peaks and troughs may not be directly related to a tertiary providers’ performance.

The TEC has also looked at the current ITO performance information and modelled the effects of the programme completion rate. This has shown that the same scenario can occur when measuring the programme completion rate for ITOs.

Factors which can have an impact on the variability of the qualification and programme completion rates from one year to the next include fluctuations in enrolments, and the use of a synthetic cohort.

The synthetic cohort measure of qualification and programme completions is a simple comparison between a single year of enrolments and completions that occur within that year.

This approach has been used for the performance-linked funding framework because it provides a proxy for current performance, and is simpler and more transparent than tracking individual students over a number of years.

Proposed options for performance-linked funding

The three options the TEC is investigating for measuring SAC qualification completion rates and ITO programme completion rates are as follows:

Option A: Continue to measure qualification and programme completion rates from the previous year’s performance (as used for publishing educational performance information).

Option B: Measure qualification and programme completion rates on a rolling average over three years.

Option C: Measure qualification and programme completion rates on a weighted rolling average putting more weight on performance in the most recent year (e.g. this would mean putting 20% weighting on 2009; 30% on 2010 and 50% on 2011 completion rates, to impact on 2012 funding).

Discussion of options

Option A: Continue to measure qualification and programme completion rates from the previous year’s performance.

The graph below shows how measuring qualification and programme completion rates from the previous year’s performance can lead to peaks and troughs in performance year on year.

 

In order to reduce the year on year changes, which may be related to fluctuations in enrolments, and the use of a synthetic cohort, the following two options have been proposed.

Option B: Measure qualification and programme completion rates on a rolling average over three years.

A refinement that could be made to counter the fluctuations in the qualification and programme completion rates would be to calculate them using a rolling three year average rather than a single year. 

Modelling of rolling average completion rates smoothes out peaks and troughs in TEOs’ qualification and programme completion rates over time, particularly those caused by changing enrolment patterns.

The graph below shows the effect a three-year rolling average would have on the same tertiary providers’ data as used in Graph 1.

 

Advantages and disadvantages of using a rolling average completion rate
The advantage of smoothing out variability means that tertiary providers will have additional funding stability in a trough year.

One disadvantage of a rolling average is that, when a tertiary providers’ performance improves in a particular year, the effects of this are diluted by the previous two year’s performance.  This could be mitigated in part by Option C. 

Option C: Measure qualification and programme completion rates on a weighted rolling average putting more weight on performance in the most recent year (e.g. 20% weighting on 2007; 30% on 2008 and 50% on 2009).

This graph uses the same tertiary providers and the same data as the previous two examples.

Advantages and disadvantages of using a weighted rolling average completion rate
More emphasis is placed on the most recent years’ performance.  Year on year peaks and troughs are also smoothed out.

It is important to note that both Options B and C would mean that educational performance from 2009 and 2010 will have an impact on how much funding a TEO retains in 2012. 

Measuring qualification and programme completion rates as a rolling average for performance linked funding would be inconsistent with the way it is measured for other purposes, for example Plan commitments and publishing performance information. 

Feedback

The TEC is looking for feedback from tertiary providers and individuals in the tertiary education sector about what option you think is the most effective method for measuring qualification and programme completion rates.

The TEC prefers Option C (weighted rolling average) as this has a smoothing effect on the completion rate whilst still incentivising and rewarding performance improvement. 

Submitting feedback

Please email any feedback on the options outlined in this document to the TEC Service Centre by 5pm on Friday 10 December 2010. 

Emails should be sent to servicecentre@tec.govt.nz with ‘PLF Feedback’ in the subject line.

The TEC expects to confirm its approach to measuring SAC qualification completion rates and ITO programme completion rates for use in performance-linked funding by the end of 2010.