Setting performance targets in pay for performance programmes: what can we learn from QOF?

How do payers set targets?

Most pay for performance schemes use either relative targets or absolute targets. The relative target approach compares performance between providers and is used in tournament style programmes, where higher performers are rewarded and low performers may even be penalised—for example, Medicare’s physician value based payment modifier.7 Payers must decide what proportion of providers to reward but don’t make judgments on acceptable levels of performance. This approach has several drawbacks for providers: as they do not know the targets in advance they lack a firm reference point to guide quality improvement efforts, and high levels of achievement may not be rewarded, which is particularly unfair when the distribution of performance scores is narrow and differences between providers are clinically insignificant.

With absolute targets, also known as criterion referenced targets, payers determine performance targets in advance, which requires more work on evaluation. This approach is less directly competitive because all providers can “win” or “lose” depending on their level of achievement, and payers must therefore accept greater uncertainty about programme costs. In contrast to relative targets, where targets are adjusted automatically, absolute targets need to be reset in response to provider performance so that they remain appropriate.

Quality and Outcomes Framework: points mean prizes

The UK’s primary care pay for performance programme—the Quality and Outcomes Framework (QOF)—was introduced in 2004 to provide quality related payments to family practices in addition to their existing capitation payments8 and encourage better practice.

The scheme uses a criterion referenced approach, with absolute targets set across several domains (box 1). Most of the targets relate to management of chronic disease and public health measures (such as smoking cessation and immunisation), but organisational and patient experience factors are also included. As an example, table 1⇓ shows the current QOF targets (“indicators”) for the clinical domain of stroke and transient ischaemic attack.

QOF sets a lower and upper threshold for each indicator. Practices can earn an increasing number of points depending on their level of achievement between the lower and upper threshold. The lower threshold sets the minimum level of achievement required to receive any remuneration and the upper threshold the achievement to receive maximum remuneration with a linear scale between (figure⇓). In the absence of robust evidence on performance, thresholds were initially set through discussion between the Department of Health and the British Medical Association (BMA), the representative body for doctors. All lower thresholds were set at 25%, and upper thresholds were set at 50% to 90%, depending on the estimated difficulty of achieving individual indicators.

Example of QOF clinical indicator: reported achievement scores for English general practices on DM17 (diabetic patients with total cholesterol level ≤5 mmol/L), 2004-05 to 2012-13

• Download figure
• Open in new tab
• Download powerpoint

Because the targets are absolute and size of payments varies over a range of performance, all practices performing below the upper threshold have an incentive to make incremental improvements. However, the financial incentive to improve diminishes as the upper threshold is approached and disappears once it is exceeded. This may partly explain why the practices with the lowest baseline achievement rates improved at the fastest rate,10 although similar patterns have been observed in other incentive schemes.11 12 It could also explain the lack of improvement in achievement rates after the first three years of the scheme (figure⇑).

Difficulties of setting QOF thresholds

Meeting QOF targets has become an established part of clinical practice in the UK. Performance is routinely monitored through practices’ clinical computing systems and over 90 000 practice years of data are now publicly available. Until recently this wealth of information was not used to set thresholds. In 2006-07 (the third year of the scheme) most minimum thresholds were arbitrarily raised to 40% and upper thresholds were raised by 5-20 percentage points for 12 of 50 clinical indicators, and lowered by 10 percentage points for one. No other changes were made until 2011, despite average practice achievement rates exceeding the upper thresholds for 66 out of 71 clinical indicators.13

There were three main reasons for this inertia. Firstly, the purpose of payment thresholds was not well defined at the outset. For example, upper thresholds were intended to represent the “maximum practically achievable level to deliver clinical effectiveness,”14 but no formal definition was provided and it was unclear whether thresholds were intended to maintain performance levels or to encourage improvement. Secondly, changes to QOF must be agreed in annual contract negotiations between the Department of Health and the BMA, and BMA representatives sought to protect practice incomes by resisting threshold increases. Contract changes are also applied uniformly across all practices, precluding piloting. Thirdly, the annual QOF review process prioritised clinical evidence on management of chronic disease over evidence on the effect of incentives on clinician behaviour, so while detailed recommendations were made for developing quality indicators,15 performance thresholds were not substantially revised.

The National Institute for Health and Care Excellence (NICE) took over the annual QOF review process in 200916 and explored alternative methods for setting targets. On reviewing national performance, researchers from the University of Manchester proposed basing upper thresholds on the 75th percentile of achievement in the previous year and lower thresholds on the fifth percentile of achievement (box 2).17 The aim was to calibrate targets against historical performance so that they remained appropriate, while retaining a criterion based approach that potentially allowed all practices to obtain maximum rewards. This approach was rejected in the 2010 contract negotiations. Instead upper thresholds for 2011-12 were increased by one percentage point for two indicators and lower thresholds were left unchanged. The following year lower thresholds were increased by 5-10 percentage points for most indicators and upper thresholds were increased by 4-10 percentage points for 13 indicators. In most cases upper thresholds remained below the average practice achievement rates.

In 2012 the Department of Health imposed a series of changes to the general practitioners’ contract, including setting upper thresholds for QOF indicators at the 75th percentile of achievement. To allow practices time to adapt, the new thresholds were applied to 20 indicators in 2013-14, with the remaining indicators due to follow in 2014-15.9 Under this new system upper thresholds would increase by an average of 14.7 percentage points (range 2-46 percentage points).18

However, two design decisions diluted the effect of these changes to the upper threshold. Firstly, lower thresholds were fixed at 40 percentage points below the upper thresholds. Because the distribution of achievement for most QOF indicators is compressed over a narrow range most practices could therefore earn close to the maximum payment without increasing achievement. For example, for indicator DM17—the proportion of patients with diabetes and a cholesterol concentration ≤5 mmol/L—median achievement in 2012-13 was 81.3%, which was within four percentage points of the 75th percentile of achievement (85.1%); hence most practices would earn at least 90% of maximum remuneration without further improvement (table 2⇓). Secondly, the base payment per point was increased by 17%, from £133.76 to £156.92. The combined effect of these changes is that under the new system practices would, on average, earn 87.5% of their existing income from the clinical indicators if they did not make any improvements. In monetary terms, this represents a mean loss of £8324 a year, with practices in the most deprived areas losing less.18 Although this is not a trivial threat to income, it is much less than that posed by the original NICE working group proposal (table 2⇓).

BMA representatives were nevertheless resistant to the changes19 and succeeded in postponing the threshold changes planned for 2014-15 by a year and in removing several unpopular indicators from the scheme.20 Most of the money earmarked for these indicators (representing 23.8% of total QOF funding) will be redirected into increased capitation payments.

Does increasing targets improve performance?

If providers are primarily motivated by extrinsic (financial or reputational) rewards, increasing upper thresholds should improve performance. Whether this improvement is meaningful or is secured through gaming (for example, by inappropriately excluding patients) will depend on the provider. If intrinsic (professional or altruistic) motivations drive performance—which is consistent with practices overachieving relative to the existing thresholds—raising thresholds may have a limited effect, although practices may still respond to new benchmarks for high quality care.

As the adjustments to QOF thresholds have so far been limited it is difficult to draw conclusions on the effect of the changes. Although thresholds for several QOF indicators were raised in 2006-07 and 2012-13, they remained below most practices’ existing levels of achievement. Raising the thresholds seemed to produce little response from the minority of low achieving practices affected by them, particularly in 2012-13 (figure⇑). To date the change that has had the greatest effect on performance was the increase in the threshold for CHD10 (prescription of β blockers for patients with coronary heart disease) from 50% to 60% in 2006-07. In 2005-06, 1748 practices (21%) reported achievement rates of between 50% and 60%. Mean reported achievement for these practices, which had increased by just 0.3 percentage points in the year before the threshold increase (2004-05 to 2005-06), increased by 11.1 percentage points in the year after (from 55.7% to 66.8%). However, actual prescribing rates increased by just 1.9 percentage points; the apparent improvement in performance was largely the result of practices excluding more patients (exception reporting rates increased from 15.8% to 25.8%).

Assessment of the effects of threshold changes has been further hindered by the fact that they are applied to all UK practices, preventing comparative analysis. However, in 2006-07, a natural experiment arose when the upper threshold for immunising patients with coronary heart disease against influenza was raised from 85% to 90%, while thresholds for immunising patients with three other conditions (chronic obstructive pulmonary disease, diabetes, and stroke) remained unchanged. The small increase in the upper threshold was associated with a 0.41 percentage point increase in immunisation rates and a 0.26 percentage point increase in exception reporting rates for patients with coronary heart disease, relative to increases for patients with other conditions. Increases in immunisation rates were greater for practices with lower baseline performance.21

Conclusions

QOF was introduced to deal with an existing emphasis on volume rather than quality of care that ran “counter to general practitioners’ professionalism, the interests of the NHS, and the interests of patients.”14 However, performance thresholds were set relatively low and not adequately adjusted, allowing most practices to secure close to maximum remuneration despite wide variations in performance. Because payments are adjusted for list size and disease prevalence, QOF effectively became a weighted capitation payment with minimum quality requirements, rather than a discriminatory quality payment. The NHS now has two choices: retreat from financial incentives and secure value by alternative means or reinvigorate pay for performance. Recent contract changes move in both directions at once; reallocating a quarter of QOF funding to core capitation payments reduces the share of remuneration linked to quality measures, while setting upper thresholds at the 75th percentile of achievement creates a more discriminating quality scheme.

Future decisions on reforming pay for performance should be based on assessments of the benefits and costs of incentive programmes. This will require not only clinical evidence on individual quality indicators but evidence on how incentive structures affect provider behaviour and patient outcomes. Opportunities to generate evidence on the effects of changing performance thresholds have repeatedly been missed in QOF, with arbitrary changes simultaneously applied to all practices. Given that the framework directly affects the care of over 12 million patients and represents an investment of over £1bn a year, the need to trial changes to incentive structures is as great as the need to trial clinical interventions. In the absence of trials, causal effects are best estimated by exploiting natural experiments, and the phasing of the latest threshold changes over two years will, at least, provide such an opportunity.