Are NHS primary care performance indicator scores acceptable as markers of general practitioner quality?

Introduction

IN 1999, the Department of Health for England set up a system to monitor the performance of primary care organisations (PCOs). The system comprised of 20 measures that were intended to reflect the progress of primary care in implementing the government's National Health Service (NHS) modernisation agenda. The 20 measures are shown in Box 1.

In August 2002, the results of the first annual assessment of PCO performance using these indicators was published.1 Since these measures are intended to reflect the progress of primary care in implementing the government's NHS modernisation agenda, we discussed the performance indicator scores obtained by Birmingham PCOs with local health professionals and general practitioners (GPs).

The overwhelming response was that there were far too many indicators, and that many were clearly of no relevance to general practice. We therefore decided the following:

• To ascertain whether we could identify a subgroup out of these 20 indicators that GPs would consider valid indicators of their performance within the PCO.

• To develop a single composite score from these ‘GP validated’ indicators.

• To establish whether this composite indicator was likely to be a valid tool for measuring and comparing the combined performance of GPs in PCOs.

Methods and results

We listed the 20 NHS indicators in random order and approached 25 GPs who were both in active practice and recognised leaders of general practice education across the West Midlands Deanery. These doctors included the Deputy Postgraduate Dean, the Regional Director of GP Education, area directors, educational advisors, course organisers, GP tutors and GP trainers, coming from a region with a larger population than Scotland and encompassing the rurality of Herefordshire, the urban deprivation of the Black Country and the affluence of Sutton Coldfield and Solihull. We asked them individually to identify all those indicators that, in their opinion, were most likely to reflect high quality GP practice, particularly those for which GPs could have personal control of the service that the indicator sought to measure. We then asked them, again without any conferring, to place eight of these indicators in rank order and return their results to us anonymously. A pilot of this methodology had previously ascertained that participants were unable to rank order more than eight choices.

We found that just seven indicators comprised 73% of the indicator choices selected and these were chosen by three-quarters of all responders (Table 1). Five indicators were not selected by any of the 24 participating GPs: 26–week outpatient waits, 18–month and 15–month in-patient waits, surgery rates for cataract removal, and accidental deaths.

Using these seven indicators we calculated a ‘composite GP performance indicator score’ for each PCO. The composite GP performance indicator score was derived in the following way. We used the Department of Health 2001–2002 data to obtain the median score of the 302 PCOs in England for each of these seven indicators. If a PCO's score was better than the national median value for an indicator we assigned the PCO one point. We repeated this point allocation process for each of the seven GP performance indicators. (See Table 2 for national median scores and examples of high and low scoring PCOs.) For each PCO, we added these individual scores to produce a composite score, which would have to lie between 0 and 7. We then plotted a graph which showed the number of PCOs observed with composite indicator score (Figure 1). Thirty-six PCOs achieved high scores of either 6 or 7, 35 had low scores of 0 or 1, and 231 had middling scores of 2, 3, 4, or 5.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Expected and observed composite indicator scores for 302 primary care organisations.

We also plotted the graph that we would expect if the performance of all PCOs was equal, and differences in their performances were owing to chance or sampling variation. Put simply, common sense (and binomial theory) suggests that:

• The average composite score for any PCO must be 3.5.

• The majority (165) of PCOs should have composite scores of 3 or 4.

• Fewer (49) PCOs should have to have a composite score of either 2 or 5.

• Even fewer (19) PCOs should have low composite scores of 0 or 1, and a similar number have high composite scores of 6 or 7.

Figure 1 shows that only 126 PCOs — 39 fewer than expected — had scores of 3 and 4. More interestingly, about twice as many PCOs had scores of 0 and 1 (35), and of 6 or 7 (36) than expected theoretically. These results suggest that:

• The performance of about half of the 71 PCOs with very high or low scores can be attributed to chance and should therefore not attract criticism or praise.

• The performance of the other — approximately 35 — PCOs are likely to be related to specific PCO related factors. The implication must be that the performance of these PCOs can justifiably be criticised or praised.

• Unfortunately, using this data alone we have no way of deciding which PCOs have low or high scores owing to ‘luck’ and which have low or high scores owing to practice-related factors.

Discussion

Nobody likes composite performance scores (‘It is like mixing apples and oranges’). Nevertheless, governments have shown an increasing desire to collect indices of performance to justify organisational and funding changes. These are of little value if they are not easily understood and credible to those whose performance they purport to measure. We believe that our method, a 7-indicator composite score, meets these criteria. Furthermore these scores are:

• Simple. Calculating a composite score for a PCO by using the criteria of whether each indicator value is above the median is an easy statistical ‘sign test’ technique.2

• Easily interpreted. We found that GPs and other health professionals could follow the simple steps of our method without feeling that they had been blinded by abstruse statistical or mathematical theory. We presented these results to the West Midlands area directors of general practice and there was unanimous agreement on the simplicity of the method.

• Relevant. Practitioners have doubted the value, relevance and the evidence base of government inspired performance indicators.3^-6 First sight of the 2001–2002 list, with its emphasis on hospital waits as markers for primary care quality, did little to dispel this attitude. We believe that by allowing GPs to select the indicators that they feel able to control, they have given professional credibility to the composite indicator score. Even the few GPs evaluating the study who were not totally convinced of the fairness of the indicators, requested that we use the methodology to calculate the composite scores of the practices selected to train GP registrars in the Deanery. It is important to remember that these composite scores are not based on arbitrary ‘thresholds’ or ‘targets’, but a real score derived from the median performance of all PCOs. They could be used to put PCOs in bands, comparable to the government's star rating system, but they are internal comparisons rather than the basis of league tables, which practitioners find neither helpful nor acceptable.

• Face validity. In its drive to improve recruitment into general practice, the government has introduced a ‘golden hello’ inducement payment to new GP principals.7 This is paid either at ‘normal’ or ‘enhanced’ rates, depending on the recruitment need of each PCO, as judged by the Department of Health. Only eight of the 35 (23%) PCOs with high composite scores offer the enhanced payment, compared with 33 out of the 36 (92%) low scoring PCOs. Furthermore, several of the indicators chosen by our group of clinicians have been used in the formulation of the General Medical Services contract in the determination of the global sum, additional services and quality payments.8 The Department of Health has announced a new set of indicators for primary care and removed two — Internet access and the results of the National Patient Survey — from the previous list, neither of which reached the shortlist chosen by our Delphi group of GP educators.

We noted that none of the low scoring PCOs registered a score for cervical cytology, in contrast with all but three of the high scorers. All of the high scorers except one were above the mean for influenza immunisation, whereas only one of the low scorers was above the mean. Initial analysis suggests that low scoring PCOs tend to be in conurbations and reflect socioeconomic deprivation; this is in line with previous research into family health services authorities performance indicators where morbidity, socioeconomic characteristics, and secondary care supply were identified as confounding factors, explaining between a third and a half of the variation in admission rates across health authority areas.9 Although Roland and his colleagues at the National Primary Care Research and Development Centre, University of Manchester, have been researching into quality markers in individual practices,10 we feel that, especially with increasing central emphasis on managed primary care, scores like ours can be used to encourage joint PCO and practice endeavours to demonstrate improving quality of care.