Abstract
Background Depression is a major cause of chronic ill-health and is managed in primary care. Indicators on depression severity assessment were introduced into the UK Quality and Outcomes Framework (QOF) in 2006 and 2009. QOF is a pay-for-performance scheme and indicators should have evidence to support their use; potential unintended consequences should also have been considered.
Aim To review the effectiveness of routine assessment of depression severity using structured tools in primary care, and to determine the views of GPs and patients regarding their use.
Design Systematic review.
Method Studies were identified by searching electronic databases; study selection, data abstraction, and quality assessment were carried out by one reviewer, with checks from other authors and GRADE (grading of recommendations, assessment, development and evaluation) tables completed for included effectiveness studies.
Results Eight studies met the eligibility criteria. There was very low-quality evidence that assessing severity in a structured way at diagnosis using a validated tool led to interventions that were appropriate to the severity of depression. Patients and GPs had different perceptions of the assessment of depression at diagnosis, with patients being more positive. GPs highlighted unintended consequences. There was low-quality evidence that structured assessment at follow-up led to increased rates of remission and response, but changes to management were not seen. Patients used this assessment to measure their own response to treatment.
Conclusion Any estimate of the effect of structured assessment of depression severity in UK general practice is uncertain. GPs consider routine use of questionnaires as incentivised by the QOF has unintended consequences, which could adversely affect patient care.
- depression
- health care
- incentive
- performance measures
- primary health care
- quality assurance, reimbursement, Severity of Illness Index
INTRODUCTION
The Quality and Outcomes Framework (QOF) for the UK was introduced in April 2004 and is a pay-for-performance scheme: it provides financial incentives to implement interventions across a range of clinical and health-improvement indicators.1 Pay-for-performance schemes are now widely used in different healthcare systems and there is evidence that they can improve health outcomes for patients.2 However, it is important that indicators have an evidence base to support their use and that any potential unintended consequences are identified and rectified, otherwise they may have adverse effects on care.3
Depression is a major cause of chronic ill-health and is largely managed in primary care.4,5 Two indicators in the depression domain of the QOF are related to depression severity assessment: one at diagnosis and one at follow-up. These were introduced into the QOF in April 2006 and 2009 respectively, using an expert panel process.6
The indicators are:
DEP4: in those patients with a new diagnosis of depression, recorded between the preceding 1 April to 31 March, the percentage of patients who have had an assessment of severity at the time of diagnosis, using an assessment tool validated for use in primary care;
DEP5: in those patients with a new diagnosis of depression and assessment of severity recorded between the preceding 1 April to 31 March, the percentage of patients who have had a further assessment of severity 4–12 weeks (inclusive) after the initial recording of the assessment of severity. Both assessments should be completed using an assessment tool validated for use in primary care.
Their rationale for inclusion is national guideline recommendations to assess severity in patients with depression, to determine appropriate interventions and improve the quality of care.4,5,7 Severity assessment as close as possible to diagnosis (DEP4) enables a discussion with the patient about relevant treatment options. Further assessment (DEP5) enables continued monitoring and determination of the treatment response. Its rationale is that depression is often a chronic disease, yet treatment is often episodic and short lived.7 The assessment tools recommended are any of three severity measures validated for use in primary care: the Patient Health Questionnaire (PHQ-9), the Beck Depression Inventory, second edition (BDI-II), and the Hospital Anxiety and Depression Scale, Depression subscale (HADS-D). The underlying principle of all suggested measures is that a higher score indicates greater severity, requiring different types of intervention.
The National Institute for Health and Clinical Excellence (NICE) has managed the development of QOF indicators from April 2009. The process has a number of significant changes that should lead to the QOF acting as a vehicle for quality improvement, and delivery of more rigorously developed indicators. Key changes include a more explicit guideline recommendation-driven indicator-development process; the consideration of cost effectiveness and clinical effectiveness; and piloting of indicators in a representative sample of UK general practices prior to any recommendation for use.8 There is also an expectation that the QOF will continue to develop,9 and that existing indicators will be retired and new indicators introduced when certain criteria are met; these include retirement based on changes to evidence that suggest an indicator is likely to be ineffective.10
How this fits in
Since 2004, the UK’s Quality and Outcomes Framework (QOF) has incentivised the assessment of severity of depression in primary care. However, it is not known whether the QOF indicators are effective in improving outcomes for patients and whether they have unintended consequences. This systematic review shows that it is very uncertain whether using the QOF indicators for depression leads to improved health outcomes for patients. It also shows that GPs consider there are associated unintended consequences that could adversely affect patient care.
Since NICE has managed the QOF programme, it has received stakeholder suggestions that the QOF depression indicator set should be reviewed, owing to concerns that there is limited evidence these indicators lead to improved health outcomes, and the use of these indicators may have unintended consequences.11 NICE’s independent QOF Indicator Advisory Committee therefore recommended that a review of the evidence was undertaken for the QOF depression severity indicators.
This study systematically reviewed the evidence for the effectiveness of assessing depression severity using structured tools in UK primary care, and unintended consequences of the use of two QOF depression severity indicators as reported by GPs and patients.
METHOD
Four review questions were formulated:
Does structured assessment of severity at diagnosis improve depression-related outcomes, or processes of care?
What is the experience of GPs and patients assessing the severity of depression at diagnosis as incentivised by the QOF indicator, with specific reference to unintended consequences?
Does structured assessment of severity after diagnosis improve depression-related outcomes or processes of care?
What is the experience of GPs and patients assessing the severity of depression after diagnosis, as incentivised by the QOF indicator, with specific reference to unintended consequences?
Review protocols were developed and reviewed by a panel consisting of GP academic advisers. Details are provided in the Appendix (available from the authors).
Studies were included if they were primary studies of the effectiveness of assessing depression severity, either at diagnosis or at follow-up in a primary care population, or reported the views or experience of GPs or patients. For studies of effectiveness, the review aimed to include randomised controlled studies only; however, owing to the lack of such studies, observational studies were included where relevant. For studies of GP and patient views, qualitative studies or surveys were included. Because of concerns about applicability, studies reporting the views of GPs or patients were restricted to UK primary care only. Studies published in abstract only, or written in languages other than English were excluded.
MEDLINE®, Embase, PsycINFO, the Cochrane Database of Systematic Reviews, Cochrane Central Database of Controlled Trials, Database of Abstracts of Reviews of Effects, and Health Technology Assessment (HTA) databases were searched from inception to June 2012. See Appendix (available from the authors) for details of the search strategies. Grey literature sources were also searched for any relevant audits.
Studies were assessed for inclusion by a single reviewer and the final list of included studies was checked by GP experts and academic advisers. No relevant, additional studies were suggested. See Appendix (available from the authors) for a full list of excluded studies. Data were extracted from included studies by a single reviewer, and checked for accuracy by another reviewer. Risk of bias was assessed using GRADE (Grading of Recommendations, Assessment, Development and Evaluation) methodology (www.gradeworkinggroup.org),12 a system for appraising and summarising the quality and strength of recommendations. In this system, the following features are assessed for evidence found for each relevant outcome:
study design (as proxy for bias);
limitations in the methodological quality of the study;
consistency of an effect across studies (defined as inconsistent when heterogeneity of results exists, but no plausible explanation is identified); and
directness (the degree to which the results directly address the question posed).
Quality of evidence reflects the extent to which confidence in an estimate of the effect is adequate to support a particular recommendation.
The use of GRADE is consistent with the methods used by NICE in developing recommendations on the effectiveness of interventions in its clinical guideline programme.13 Owing to the nature of the data and outcomes, meta-analysis and statistical assessments of inconsistency and publication bias were not possible; results and judgements are therefore reported qualitatively. See Appendix (available from the corresponding author) for full GRADE tables.
A narrative review of the qualitative evidence is reported; GRADE is not currently formally developed for such use.
RESULTS
Evidence review
A total of 2978 references, including duplicates, were identified through systematic searching and asking expert advisers. Full text was ordered for 132 articles, based on their title and abstract. Searches for published audits found no new relevant references (Figure 1).
Flow diagram of the process.
Eight studies, reported in 10 publications (Figure 1), met the eligibility criteria (for the full review protocol and inclusion/exclusion criteria, see Appendix, available from the corresponding author).14–23
Studies related to the assessment of severity at diagnosis are detailed in Tables 1 and 2 and studies related to the assessment of severity at follow-up are shown in Tables 3 and 4.
Summary of included studies for DEP4 assessment — effectiveness of assessment of severity
Summary of included studies for DEP4 experience
Summary of included studies for DEP5 assessment — effectiveness of assessment of severity
Summary of included studies for DEP5 experience
For further details of how the evidence is graded, see the NICE guidelines manual.13 GRADE profiles for the effectiveness studies are presented in Tables 5 & 6.
GRADE profile 1 assessment at diagnosis
GRADE profile 2 assessment at follow-up
Assessment of severity at diagnosis
Assessment of depression severity at diagnosis, and its use to inform treatment is generally considered to be good practice,4 and increased severity of depression as assessed at diagnosis using structured tools or GP judgement is associated with higher rates of treatment and referral (based on very low-quality evidence). However, no evidence was found on whether structured assessment of severity and subsequent treatment based on the assessment resulted in improved outcomes, such as depression remission or improved quality of life.
Patients and GPs had different perceptions of assessing depression severity at diagnosis,15,18 with patients generally being more positive than GPs. For example, patients considered structured assessment to be an efficient addition to clinical judgement, while GPs perceived clinical judgement to be more important than an objective measure. GPs considered that routine use of questionnaire severity measures as incentivised by the QOF had a number of unintended consequences, specifically compromising the doctor–patient relationship, threatening holistic practice and intuition, interfering with the consultation process, and being mechanistic and intrusive.18,20 Concerns were also raised about the need to adapt questionnaires to different patient groups and how this affected the validity of the tools.20
Assessment of severity at follow-up
As at diagnosis, professional opinion supports assessment of depression severity at follow-up, and its use to evaluate treatment effectiveness. The score for assessment of severity broadly reflected patients’ accounts of the severity of depression over time. Severity scores were associated with changes in management; however, this was not a consistent finding across studies. Structured assessment of severity after diagnosis was associated with increased rates of remission and response, but changes in management were not seen. A proposed explanation was that the intervention may have influenced patient behaviour and thus led to improved outcomes.
As at diagnosis, GPs noted concerns about the need to adapt the questionnaires to different patient groups and how this affected the validity of the tools.20 However, some patients used the severity assessment at follow-up to measure and monitor their own treatment response and recovery process.19
DISCUSSION
Summary
Very limited evidence was found of the effectiveness of assessing depression severity using structured tools as incentivised in UK primary care by the QOF. No evidence was found on whether structured assessment of severity (either at diagnosis or follow-up) and subsequent treatment based on the assessment resulted in improved health outcomes, such as depression remission or improved quality of life. The assessment of depression severity at diagnosis, and its use to inform treatment, is generally considered to be good practice, and increased depression severity as assessed at diagnosis using structured tools or GP judgement is associated with higher rates of treatment and referral. As at diagnosis, professional opinion supports the assessment of severity of depression at follow-up, and its use to evaluate the effectiveness of treatment. The assessment of severity score broadly reflected patients’ accounts of the severity of their depression over time. This review shows that any estimate of the effect of the use of structured tools to assess severity in UK general practice is very uncertain and should not form the basis for a strong (‘should do’) recommendation for clinical practice.13,24 This uncertainty is reflected in the NICE depression clinical guidelines, where it is recommended that when assessing a person with suspected depression, the use of a validated measure (for example, for symptoms, functions and/or disability) be considered to inform and evaluate treatment, rather than being recommended as a ‘should do’.4,5
GPs considered that the routine use of depression severity structured tools as incentivised by the QOF had a number of unintended consequences, specifically compromising the doctor–patient relationship, threatening holistic practice and intuition, and interfering with the consultation process. In contrast, patients were more positive, seeing the tools as efficient and structured supplements to medical judgement and as evidence that GPs were taking their problems seriously, through full assessment of their patients’ depression.
Strengths and limitations
The main strengths of this review are its recognition of the need for setting-specific (UK primary care) evidence, its inclusive approach to the types of interventions studied, and its use of GRADE profiles. Although no randomised controlled trials were found, the researchers were able to consider observational evidence and, through the use of GRADE, assess confidence in the results, in a transparent and systematic way.
One limitation is that the study focused on the use of structured assessments in primary care alone; there may be evidence from other settings, such as hospital-based clinics, that care is improved through their use. However, it is not clear whether or how such evidence could be extrapolated to primary care populations.
Comparison with existing literature
This is the first systematic review to appraise the evidence of the effectiveness of assessing depression severity using structured tools, and any unintended consequences in UK primary care related to the use of two QOF depression severity indicators as reported by GPs and patients.
While making recommendations on this area, the NICE depression guidelines did not conduct a systematic review on the method of assessment of severity, and the NICE recommendations were made using expert opinion with documented consultation with professionals and wider stakeholders. However, it is notable that the NICE depression guidelines do not make a strong recommendation for the routine use of structured assessment tools in the assessment of depression severity, rather they state that clinicians should ‘consider using a validated measure to inform and evaluate treatment’.4,5 Such a weak recommendation, which makes explicit the need for clinical judgement and does not advocate the routine use of such tools, is consistent with the findings of this review. The conclusions of this review on effectiveness are supported by a psychometric assessment of the discriminatory performance of the three structured depression tools (PHQ-9, HADS-D and BDI-II) in UK general practice.25 This shows that none of these tools have optimal cut-off values with likelihood ratios that are adequate to inform clinical practice, and they are therefore inappropriate for use to assess depression severity in general practice.25
The qualitative studies reviewed here are specific to depression and the UK but the findings based on GP interviews are consistent with the wider international literature on pay-for-performance schemes. Studies of unintended consequences of indicators used in such schemes have shown they can lead to changes in the nature of the consultation/office visit, threats to the physician–patient relationship, and threats to professional autonomy.3
Implications for practice and research
The current QOF depression severity indicators (DEP4 and DEP5) incentivise routine use of structured assessment tools to assess depression severity at diagnosis and follow-up. This systematic review shows that it is very uncertain whether this leads to improved health outcomes for patients. These indicators, therefore, do not meet the criteria that indicators used in pay for performance should lead to improved health outcomes for patients and should not have major unintended consequences.26 Given these findings, the recommendations of NICE’s independent QOF Indicator Advisory Committee, that these indicators should be retired from the QOF,11 is consistent with this evidence review.
It should be noted that these indicators were developed using the previous QOF expert panel process,6 and the evidence base was not the subject of independent external review through public consultation, nor were the indicators piloted prior to their adoption (which should have allowed unintended consequences to have been identified). The new NICE QOF process uses an explicit guideline recommendation-driven approach,8 with piloting of indicators prior to any recommendation on their use.9,27 There is therefore scope to develop new QOF indicators on depression that have evidence to support their use to improve health outcomes for patients and are piloted prior to introduction, to minimise the risk of unintended consequences.
Acknowledgments
We would like to thank the GP academic advisers for their help with the study protocol and the late Helen Lester for her comments on the draft version of the paper.
Notes
Funding
Elizabeth J Shaw, Terence Lacey, and Daniel Sutcliffe are employees of NICE which is funded by the Department of Health to develop QOF indicators for publication on the NICE menu of indicators. Tim Stokes was an employee of NICE at the time the systematic review was conducted.
Provenance
Freely submitted; externally peer reviewed.
Competing interests
The authors have declared no competing interests.
Discuss this article
Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss
- Received May 17, 2012.
- Revision received June 21, 2012.
- Accepted November 26, 2012.
- © British Journal of General Practice 2013
REFERENCES
In this issue
Jump to section
Keywords
More in this TOC Section
Related Articles
Cited By...
