Background The Quality and Outcomes Framework (QOF) is a pioneering attempt to improve the quality of primary care in the UK through the use of financial rewards. Despite its achievements, there are concerns that the QOF may offer poor value for money.
Aim To assess the cost-effectiveness of QOF payments.
Design of study Economic analysis.
Setting England, UK.
Method Cost-effectiveness evidence was identified for a subset of nine QOF indicators with a direct therapeutic impact. These data were then applied to an analytic framework to determine the conditions under which QOF payments would be cost-effective. This framework was constructed to assess the cost-effectiveness of QOF payments by modelling the incentive structure using cost-effectiveness thresholds of £20 000 and £30 000 per quality-adjusted life year (QALY) gained, to represent good value to the NHS. It used 2004/2005 data on the QOF performance of all English primary care practices.
Results Average indicator payments ranged from £0.63 to £40.61 per patient, and the percentage of eligible patients treated ranged from 63% to 90%. The proportional changes required for QOF payments to be cost-effective varied widely between the indicators. Although most indicators required only a fraction of a 1% change to be cost-effective, for some indicators improvements in performance of around 20% were needed.
Conclusion For most indicators that can be assessed, QOF incentive payments are likely to be a cost-effective use of resources for a high proportion of primary care practices, even if the QOF achieves only modest improvements in care. However, only a small subset of the indicators has been considered, and no account has been taken of the costs of administering the QOF scheme.
Concern with both the quality of health care1 and the failure of payment systems to provide appropriate incentives for efficiency and equity2 has focused attention on pay-for-performance programmes as a potential solution. Such programmes for health care have been adopted in Australia, Canada, Germany, the Netherlands, and New Zealand,3 and programmes in the US have been tried in both public and private sectors.4 Systematic reviews of pay-for-performance programmes suggest that they can have positive effects, but they can also have unintended consequences, and important questions about their optimal design, effectiveness, implementation, and value for money remain unanswered.2,4
In April 2004, a new General Medical Service (GMS) contract was introduced into UK primary care.5 The contract was backed by a significant investment, estimated to total £8 billion in the first 3 years.6,7 It included a major pay-for-performance scheme, the Quality and Outcomes Framework (QOF). This seeks to secure higher-quality primary care by offering financial incentives to general practices for the achievement of specific indicators. Whereas effectiveness evidence was taken into account when designing the scheme and selecting indicators, cost-effectiveness was not an explicit selection criterion.7 Evaluations of the QOF suggest that the scheme may deliver modest improvements in the quality of care,8 and could help to reduce health inequalities.9 However, the cost-effectiveness of the QOF has been questioned.10 Focusing on QOF indicators expected to have a direct therapeutic impact, this research explored the potential cost-effectiveness of this pay-for-performance approach.
The research comprised three main stages: first, developing an analytic framework; second, reviewing the literature to identify cost-effectiveness evidence for the indicators; and third, estimating the cost-effectiveness of the QOF payments.
To assess the cost-effectiveness of providing financial incentives to change clinical practice, an analytic framework was constructed. Developed from previous research,11 the framework took account of the cost-effectiveness of the treatment incentivised by the QOF indicator; the incentive payment; and the resulting change in the proportion of eligible patients treated in accordance with the indicator (the ‘utilisation level’).
The framework is based on standard principles of cost-effectiveness analysis, whereby alternative interventions for a given condition are compared on the basis of differences in their health benefits and in their costs to the health service. To do this, benefits in the form of quality-adjusted life-years (QALYs) are transformed on to a monetary scale using cost-effectiveness thresholds of between £20 000 and £30 000 per QALY gained, which are commonly used in the UK to define value for money.12 If the monetary value of the additional health gain outweighs the additional cost of the treatment, then the intervention is considered to be a cost-effective use of resources. Therefore, the larger the health gain and the lower the additional cost, the more likely a treatment is to be cost-effective.
Under the QOF, payments are based on points achieved, which vary by indicator and with the proportion of eligible patients treated between lower and upper ‘utilisation’ thresholds (for example, from 25% to 90%). Total points are then adjusted for practice size and disease prevalence relative to national average values. QOF payments therefore depend on the number of patients treated, irrespective of whether treatment was initiated before or after the introduction of the QOF. Some patients will already have been receiving the treatment before the payments were introduced. For these patients, there is no additional health gain or treatment cost but only the cost of the extra incentive payment for their general practice as part of the QOF. If all patients fell into this category, then the incentive payments would never be cost-effective as they would entail additional cost without any additional gain in health benefit. However, for patients who are treated as a direct result of the QOF's introduction, there would be added health benefits of the treatment. For QOF payments to be cost-effective, the monetary value of this health gain in previously untreated patients must exceed both the additional cost of the treatment (incurred for these patients) and the QOF payments made for both previously treated and untreated patients.
How this fits in
The new 2004 contract was backed by significant investment, estimated to total £8 billion in the first 3 years, equivalent to around 40% of GP contract spend. The contract includes a pay-for-performance scheme, known as the Quality and Outcomes Framework (QOF), which seeks to secure higher-quality primary care by offering financial incentives for the achievement of specific indicators. Research suggests that the scheme may deliver modest improvements in the quality of care and could help to reduce health inequalities, but the cost-effectiveness of the QOF has been questioned. This study uses an analytic framework to model data on QOF performance, treatment cost-effectiveness, and incentive payments. It shows that, for the majority of indicators that are assessable, incentive payments are likely to represent good value for money for the NHS.
For QOF payments to be good value, three conditions must hold. First, the intervention they incentivise must itself be cost-effective. Second, the intervention must lead to an increase in the number of eligible patients receiving the intervention. Third, the incentive payment must be proportionate to the net health gain achieved.
Reviews of the literature were conducted to identify and appraise the cost-effectiveness evidence relating to those interventions covered in the nine therapeutic QOF indicators considered. These indicators were chosen as it was felt that they had a direct therapeutic effect and there was greater likelihood that evidence on cost-effectiveness would be available from the literature. The selection criteria for published cost-effectiveness studies required that they: (i) were UK based; (ii) were recently published (2001 or later); (iii) reported health outcomes in terms of both QALYs and costs; and (iv) fitted the indicator in terms of the appropriateness of the patient population, intervention, and comparator. The quality and relevance of the economic evaluations found from the literature were evaluated based on these criteria: if the study matched these criteria closely, then it was considered ‘robust’; otherwise it was designated as ‘indicative’. Costs taken from the literature were transformed to £ sterling using purchasing power parity estimates if required,13 and inflated to 2005 prices using the Hospital and Community Services Pay and Price Index.14
Estimation of the cost-effectiveness of the indicators
Results from the literature review were combined with payments based on practice utilisation levels for the indicators in 2004/2005, with utilisation data available for all primary care practices in England (n = 8576).15–22 The analysis assumed that payments would be made for 5 years. Although utilisation data for each year of QOF were available, there was no evidence on the actual changes in utilisation due to QOF because there was no prior audit of baseline performance, and so the actual cost-effectiveness of the QOF indicators could not be determined. Instead, the potential cost-effectiveness of the QOF was estimated, contingent upon two statistics: first, the overall proportional change in utilisation as a result of QOF which was required for the payments to be a cost-effective use of resources as a whole; and second, the proportion of primary care practices for which a particular change in utilisation would represent a cost-effective use of resources. These statistics were derived using the cost-effectiveness evidence from the literature, data on QOF payments, and the analytical framework. Further details of the methodology are available in Appendix 1 and the final report to the sponsors: http://www.york.ac.uk/inst/che/
Table 1 summarises the QOF indicators examined, information on post-QOF utilisation levels, average payments per treated patient, and evidence from the literature review. The mean payment per treated patient made to primary care practices under the QOF ranged from £0.63 (for a cervical screening indicator, CS1) to £40.61 (for a coronary heart disease indicator, CHD11). At the end of the first year of the QOF, the mean utilisation level (the percentage of eligible patients receiving the intervention) ranged from 63.2% (for CHD10) to 90.0% (for CHD9). The least cost-effective indicator subject to the value for money thresholds was DM21 (diabetic retinopathy), which cost £15 654 per QALY gained. However, some indicators not only offered health improvements but also decreased health service costs (that is, DM15 and CHD9).
Figure 1 presents the proportional change in utilisation that the QOF must bring about for a QOF payment to be cost-effective. This is generated by using the formula from the analytic framework, which combines data on the estimated cost-effectiveness of the indicator's treatments, the QOF payment, and the reported attainment levels of practices. Individual figures for each indicator are provided, with results given under both the lower and higher cost-effectiveness thresholds. The y axis on each individual figure shows the percentage of practices for which a particular hypothetical proportional change (x axis) would be cost-effective. These figures show that, for many indicators, relatively small increases in utilisation are required before the payments would be considered cost-effective in most practices (for example, BP5), whereas larger increases would be required in some indicators (for example, LVD3). For each indicator, the numbers in the boxes present the overall mean proportional change in utilisation required for the payments to be a cost-effective use of resources for the healthcare system as a whole. For example, reading from the figure at a cost-effectiveness threshold of £20 000 per QALY, an increase in utilisation of around 5% in each practice for the indicator LVD3 would mean that payments were cost-effective in around 50% of practices, while an increase of 20% would result in the payments being cost-effective in almost all practices. However, for the payments to be cost-effective for the healthcare system overall, a total increase of just 4.2% is needed. This suggests that some very cost-effective practices are offsetting the costs in those practices that are not as cost-effective.
The figures also reveal wide variation between the indicators in the proportional changes necessary for QOF payments to be cost-effective. For many indicators, only a very small proportional change is needed. For example, the proportional increase in utilisation required for one of the coronary heart disease indicators (CHD10) is only 0.06% for a threshold of £20 000 per QALY, and 0.04% for a threshold of £30 000 per QALY. In nearly 100% of primary care practices, QOF payments for CHD10 would be cost-effective following only a very small proportional change in utilisation under either threshold. In contrast, much larger proportional changes are required for other indicators. For example, another indicator from the same clinical group, CHD11, would require a change of almost 20% at a threshold of £20 000 per QALY, or 11% at a threshold of £30 000 per QALY, for the payments to be cost-effective over all practices. The wide variation between the indicators is driven by the cost-effectiveness of the indicator as such (that is, the intervention's cost per QALY) as well as the price received per patient treatment. The lower the cost per QALY and the lower the price received per patient, all other things being equal, the lower the increase in utilisation required for the payments to be a cost-effective use of resources.
Sensitivity analyses were also conducted to explore the effect of the increased price per point paid under the 2006/2007 contract compared to the 2004/2005 contract. The higher price meant that larger changes in utilisation were required for indicators to be cost-effective, although in most cases the required changes were still small.
Summary of main findings
For most therapeutic indicators assessed in this study, QOF payments would be cost-effective provided the QOF achieved very small improvements in care. Moreover, the incentive payments are likely to be a cost-effective use of resources for a high percentage of English primary care practices.
Strengths and limitations of the study
Although this study considered only a subset of all the indicators, the authors believe it is unique in its attempt to evaluate the cost-effectiveness of QOF indicators. Through an extension of previous work,11 this study also provides an analytic framework for determining the cost-effectiveness of both existing and new QOF indicators.
There are a number of limitations to this work. Firstly, the indicators evaluated were those with a direct therapeutic effect and they represent a small subset of all QOF indicators (nine out of 149 indicators). These indicators were drawn from areas of clinical activity where cost-effectiveness evidence was most likely to be available. However, even within this subset, robust evidence on incremental costs and health effects was available for just four indicators. Although the cost-effectiveness of other QOF indicators is unclear, this does not imply that these represent poor value for money, but rather that new studies are needed to evaluate their cost-effectiveness. The majority of indicators represent structures or processes (112/135 in the 2006 contract) and thus have no direct therapeutic effect, such as maintaining a register of patients or providing training for new staff. Practices will incur costs but there may be no direct health gain for patients. Nonetheless, these indicators may be beneficial insofar as they either provide indirect health gains through facilitating the achievement of therapeutic goals, or improve the overall quality of care in ways that are valued by patients. For the overall cost-effectiveness of the QOF to be determined, the value for money represented by these ‘non-therapeutic’ indicators also needs to be estimated.
A second limitation is that the collection and processing of the QOF data incurs large administrative costs that were not accounted for in the study analyses. These are largely fixed costs borne by central government, and it is unclear how these should be apportioned to each indicator. Nonetheless, omitting these costs could overestimate indicators' cost-effectiveness. While these costs may be significant in the short run, for example as a result of set-up costs, it is reasonable to assume that such costs may decrease in the long run, and thus are less likely to result in the indicators being cost-ineffective.
Finally, the true opportunity cost of the QOF indicators remains opaque. Although the use of cost-effectiveness thresholds gives an indicative value of care displaced by the indicators, exactly what general practice activities are displaced locally and their true value is unknown. This is one reason why a national disinvestment programme has merits.23
Comparison with existing literature
A lack of data on the change of utilisation made it impossible to quantify the actual cost-effectiveness of the QOF. Consequently, the study analysis evaluated the potential cost-effectiveness of QOF indicators for a range of changes in utilisation. Previous studies have explored changes in utilisation associated with the QOF. Hippisley-Cox et al considered the proportional change in achievement from 2001 to 2006 for 19 indicators, only one of which (BP5) matched those considered by the present study.24 Eighteen of these 19 indicators showed positive proportional increases in utilisation over the study period, with the changes ranging from −7% to 356%. However, the analysis did not control for previous upward trends in utilisation, and so may have overestimated the effect of introducing QOF payments. Campbell et al analysed changes in the quality of care from 1998 to 2005, based on bundles of indicators that included some QOF indicators.8 Controlling for trends in utilisation, the study found an increase in the rate of improvement in quality of care for asthma, diabetes, and coronary heart disease between 2003 and 2005, which may have been the result of QOF payments. In addition, systematic reviews of pay-for-performance schemes that are targeted at doctors suggest that financial rewards can improve the quality of care.2,25 On the basis of these studies, it would seem reasonable to conclude that the QOF is likely to have effected changes in utilisation that would make payments cost-effective. However, the QOF may also have had unintended consequences, such as encouraging inappropriate ‘exception reporting’ of patients to improve apparent performance.26,27 Although more recent evaluations indicate that such practices are not widespread,3 the consequences of any exception reporting would need to be taken into account when assessing the QOF's overall impact on population health. Other unintended consequences may include perverse incentives; for example whereby it becomes financially advantageous to focus therapeutic efforts on those with milder disease rather than those with the greatest capacity to benefit, and long-term effects on doctors.28 Therefore, there is the long-term concern that doctors may focus their efforts on incentivised activities to the detriment of other activities that may be equally beneficial.
Implications for future research and clinical practice
This work provides an analytical framework that can help prioritise potential new indicators. If longitudinal data on utilisation were collected and new indicators piloted to account for trends in performance that would have taken place without incentives, then the change attributable to the QOF could be estimated more accurately. In addition, the framework could be useful for informing which payment levels would be cost-effective.
The framework reported in this paper will, indeed, be used to inform healthcare policy in the NHS. From April 2009, the National Institute for Health and Clinical Excellence (NICE) was made responsible for overseeing future developments of the QOF.29 The framework will help to inform the methodology for assessing the cost-effectiveness of existing and prospective indicators.
We are indebted to our colleagues from the Centre for Reviews and Dissemination for their expert assistance in undertaking the literature searching, and are also grateful to the National Primary Care Research and Development Centre (NPCRDC) at the University of Manchester for providing us with full access to the QOF datasets. The views expressed in this paper are those of the authors and not necessarily those of the Department of Health.
Appendix 1. Methodology
1. The overall proportional change in utilisation as a result of QOF required for the payments to be a cost-effective use of resources
This analysis was based on individual practice attainment levels and assumed that all practices would have the same proportional increase from their baseline following the introduction of the QOF. Therefore, to calculate the baseline, a practice with an observed attainment level of X% and a required proportional increase of Y% for cost-effectiveness would have had a baseline level of X/(1 + Y)%.
2. The proportion of primary care practices for which a particular change in utilisation would represent a cost-effective use of resources
Here it was determined for each practice whether the payments were cost-effective based on a given proportional change in attainment based on the method given in 1 above.
The project was funded by the Policy Research Programme of the Department of Health in England.
The authors have stated that there are none.
Discuss this article
Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss
- Received September 22, 2009.
- Revision received November 25, 2009.
- Accepted February 5, 2010.
- © British Journal of General Practice, 2010.