Developing a framework of, and quality indicators for, general practice management in Europe

Abstract

Objectives. To develop a framework for general practice management made up of quality indicators shared by six European countries.

Methods. Two-round postal Delphi questionnaire in the setting of general practice in Belgium, France, Germany, The Netherlands, Switzerland and the United Kingdom. Six national expert panels, each consisting of 10 members, primarily primary care practitioners and experts in the field of quality in primary care participated in the study. The main outcome measures were: (a) a European framework with indicators for the organization of primary care; and (b) ratings of the face validity of the usefulness of the indicators by expert panels in six countries.

Results. Agreement was reached about a definition of practice management across five domains (infrastructure, staff, information, finance, and quality and safety), and a common set of indicators for the organization of general practice. The panellist response rate was 95%. Sixty-two indicators (37%) were rated face valid by all six panels. Examples include out of hours service, accessiblility, the content of doctors' bags and staff involvement in quality improvement. No indicators were rated invalid by all six panels.

Conclusions. It proved to be possible to develop a European set of indicators for assessing the quality of practice management, despite the differences in health care systems and cultures in the six different countries. These indicators will now be used in a quality assessment procedure of practice management in nine European countries. While organizational indicators are part of the new GMS contract in the UK, this research shows that many practice management issues within primary care are also of relevance in other European countries.

Engels Y, Campbell S, Dautzenberg M, van den Hombergh P, Brinkmann H, Szécsényi J, Falcoff H, Seuntjens L, Kuenzi B and Grol R. Developing a framework of, and quality indicators for, general practice management in Europe. Family Practice 2005; 22: 215–222.

Introduction

Practice organization has the propensity to diminish or enhance the quality of clinical care.¹ While evidence that good practice management (structure) is important for good clinical performance (process) is limited^2,3 and a well-organized practice is not a guarantee for high quality clinical care or outcome, it provides the opportunity for individuals to receive it.⁴ Berwick put it in day-to-day terms: “… a result lost, a specialist who cannot be reached, a missing requisition, a misinterpreted order, a vanished record, a long wait for a CT-scan; these are all-too-familiar examples of waste, rework, complexity and error in a doctor's life …”.⁵ Moreover, patient service aspects, such as a good accessibility, patient involvement and time for care are a proxy for the care given by the practice.^6,7 Indicators on practice management would enable consumers and providers of care to compare practices. However, clinical indicators are widely overrepresented over practice management indicators in research and assessment of primary care.^8–15

European unification requires quality indicators that allow comparisons of health care facilities. Several countries have developed tools to assess the organization of general practice. In the UK, approximately 20% of the indicators in the Quality and Outcomes Framework of the new GMS contract relate to organizational aspects of care. In The Netherlands, the visitation instrument for practice management (VIP) is widely implemented and presently used for practice accreditation.¹⁶ Except for the Europep questionnaire for patient satisfaction with general practice care¹⁷ no instruments are available to compare the organization of primary care across countries.

While there is agreement within Western Europe on the importance of general practice, the financing and role of primary care within wider health care systems varies.^18–20 For example, in some countries the practitioner has as a gatekeeper role whereas in other countries patients have direct access to specialist doctors (Box 1). There are also differences in practice size and the availability of practice managers or practice nurses.²¹ Therefore, we started a European Practice Assessment research project (EPA) to study whether it is possible to develop a common framework and set of indicators of practice management, which is applicable across several European countries. For example, which aspects of practice management are shared and valued by the participating European countries?

Methods

GPs, researchers and experts in the field of quality in primary care from six European countries (Belgium, France, Germany, The Netherlands, Switzerland and the UK (England and Wales) were invited by RG at WOK, and agreed, to take part as partners in EPA (Appendix 1).

A literature review was undertaken to search for instruments, tools and methods for practice assessment, originating from various sources and countries.^16,22–32 (Appendix 2). With this information, in well prepared and structured workshops during three consecutive days in 2002, the participants agreed a definition of practice management and a framework of preliminary indicators. This set was translated into the various languages.

We then conducted a two-round postal Delphi procedure between June 2002 and January 2003. This is an accepted consensus method used to determine the extent of agreement on an issue, and is an accepted method for developing indicators where research evidence is lacking.^9,12 The partners created six national expert panels, each composed of 10 panellists, predominately GPs, but also practice managers (UK and NL). All but one of those who were invited to take part accepted.

In the first round panel members in each country were sent the preliminary set of indicators in questionnaire form and asked to rate the indicators for clarity (1 = not clear at all; 9 = very clear) and usefulness (1 = not useful at all; 9 = very useful). Panellists were also invited to rephrase unclear indicators.

Panellists were instructed to rate an indicator high on usefulness if: (1) it corresponded with a basic quality level, which all practices should meet; or (2) if it referred to a higher quality level that would be met only in very good practices; or (3) if it was associated with an innovative quality level that was exceptional at the moment but that could become the optimal quality level in the coming years. They were asked to give a low rating on usefulness to indicators that: (1) were too ambiguous or represented an unrealistically high quality level and were thus not being met in any practice; or (2) did not correspond with the material, social or cultural conditions of general practice in their country; or (3) were not in accordance with the regulations of general practice in their country.

In the second round panellists received feedback on the median scores in the first round and were invited to rate the indicators again for usefulness.

Analyses

Analyses were based on the Rand Appropriateness Method.^33,34 Indicators with a national median rating on the usefulness scale of 7, 8 or 9 without disagreement were considered face valid for that panel. Disagreement is defined as 30% or more of ratings in both the 1–3 tertile and the 7–9 tertile. Indicators scored with a national median of 1–3 without disagreement were considered invalid. All other indicators were rated equivocal. Only indicators that were rated valid by all six panels were included in the European set of indicators. We computed the number of indicators rated face valid and rated invalid by all countries and per country.

Results

Definition of practice management

The participants agreed on the following definition of practice management: systems (structures and processes) meant to enable the delivery of good quality patient care. Starting from this definition and the available literature, a theoretically based framework was developed containing five domains of practice management: infrastructure; staff; information; finance; and quality and safety. Each domain was divided in to several dimensions and a draft set of 171 indicators was created across these dimensions (Table 1).

As a result of the first Delphi round with the six panels, two indicators were added, 44 reworded, and five indicators were discarded. There were therefore 168 indicators in the second round.

The response rate in the second round on the usefulness of the 168 indicators was 95% (90 to 100%; n = 57 panellists overall). Sixty-two indicators (37%) were rated face valid by all six panels (Table 2). The key aspects of the panel ratings can be summarized as follows:

Infrastructure. Good accessibility of the premises, particularly for disabled patients, as well as a clean and well maintained practice, are important indicators. This is also true for the availability of emergency equipment and drugs, a refrigerator for medicines and a complete doctor's bag with no expired drugs and with an inventory list to keep it up-to-date. However, there was no agreement on protocols for checking and supplying equipment and drugs. An adequate telephone system as well as computers protected by a firewall and anti virus software were rated valid by all panels. The panels disagreed on the need for a separate emergency telephone line. For good accessibility and availability the panels considered it important to have good access by telephone, to have an appointment system, to provide home visits for patients who are physically not able to travel to the practice, and to have easy access to out-of-hours services when contacting the practice outside normal hours. No consensus was reached on having protocols for advice given by telephone by non-physicians.

Staff. A signed contract and appropriate qualifications for all staff were rated valid. All but one panel agreed on the necessity of job descriptions and annual appraisals. Structured team meetings as well as defining and understanding responsibilities within the team also got high ratings. There was no agreement on indicators about the education and training of staff, although almost all panels agreed that having an induction programme for new staff adds to quality. The panels disagreed on the value of ‘personal learning plans’. A pleasant working atmosphere for the staff was considered an important quality indicator, as well as having a policy that enables staff to offer suggestions for improvement.

Information. There was consensus on structured and complete medical record keeping, as well as on the annual review of repeat prescriptions by a GP. The panels did not agree on coding diagnoses or episodes (e.g. ICPC or read codes). Proper storing of medical records, as well as privacy of conversations at the reception desk and in the consultation room were rated highly. Well structured referral letters with a copy kept in the medical record got a high rating, as did receiving information from out of hours services quickly and keeping an up-to-date directory of local health care providers. A procedure that ensures incoming clinical information to be seen by the GP and a procedure for filing it in the medical record were both rated highly. A practice information sheet with the names of the GPs, address and consulting hours etc. should also be available. The availability of clinical guidelines and scientific information was considered important by most of the panels, as well as having a range of information leaflets for patients.

Finance. Producing a detailed annual plan was not rated valid by all panels nor was keeping full detailed records of finances. However, clearly defined financial management had a high rating as did ensuring that every GP and member of the clinical staff are insured to cover liability. This was also true for producing an annual financial report.

Quality and safety. Involvement of all staff in quality improvement had a high rating in each panel, but there was virtually no agreement on detection of quality and safety problems such as undertaking clinical audits, having a critical incident registration, and involving patients (a patient participation group, a suggestion box or a complaint procedure). All panels agreed about the importance of smoking forbiddance and procedures for the prevention of infections (having a steriliser, using protective equipment when dealing with blood or fluids).

Differences in ratings between panels/countries

Overall, the English/Welsh panel rated the most indicators face valid (142, 85%), and the French and German panels the least (103, 61%). No indicators were rated invalid by all six panels (Table 2).

Discussion

Summary of main findings

We defined a framework for conceptualizing practice management from a European perspective and developed a set of face valid indicators for assessing the quality of management in general practice in six European countries. Despite the differences in the health care systems and the role of general practice within each of the six countries we found remarkable agreement between the various countries regarding the criteria for good practice management. The resulting set of 62 face valid indicators reflected considerable overlap in vision and content. However, the remaining 109 indicators on which no agreement was reached reflect interesting differences in health care systems. For example, in some countries panellists rated indicators low because the items or procedures are already so generally accepted that they would not discriminate between practices. We found for example a low rating for medical registration in The Netherlands. Another example was indicators regarding recalling groups of patients, which were not rated valid in France because this is not allowed in this country.

No indicators on the availability of written protocols (15) were rated face valid by all panels. In particular, panels of countries with predominately small practices did not find it necessary to have protocols, because the communication lines are often one to one. Also indicators about systems (2) or agreements (11) were not rated face valid by all panels. The reason given, especially in countries with few group practices, was that written papers do not provide any guarantee for implementation in daily practice. Practice staff often solve problems or make arrangement by direct communication with other staff rather than relying on written documents.

The fact that the English/Welsh and The Netherlands panels rated the largest number of indicators face valid is not a surprise as they have a more formal practice management structure within their primary health care system (gatekeeper role of GP, patient lists etc). There is also more cooperation between GPs and there are fewer single-handed practices. These two countries also have more national initiatives for quality assurance (peer review, guideline development, accreditation) for both medical care and practice organization.²⁰ The French and German panels rated the least number of indicators face valid, reflecting their lesser organized general practice care.

Limitations

Consensus techniques have limitations.⁹ Firstly, the common set of indicators cannot be seen as a comprehensive set of indicators for the assessment of practice management either in a European context or in any of the six participating countries. Rather, it merely represents consensus amongst the six panels in defining quality of practice management. Given the heterogeneity of primary care in Europe, a consensus building exercise, while highlighting where agreement exists, may overlook important local issues in the process.

Secondly, panels rating the least number of indicators face valid (France and Germany) had a greater influence on the final common set than those panels rating the highest number of indicators face valid (England/Wales and The Netherlands). Had analyses been based solely on the overall aggregate ratings of all 57 panellists within one pan-European panel, a greater number of indicators (138, 82%) would have been rated face valid (Table 3). However, distinct panels allowed the process to be more sensitive and warranted that the core set kept its relevance for each country. Using the set of the pan-European panel would have resulted in a more complete set of indicators but with less relevance particularly for France and Germany.^9,35

Thirdly, our purpose was to compose a set of indicators relevant to the health systems of the participating countries. Therefore, we did not weigh for country size or level of national organizational development of primary care in the Delphi procedure; small countries or countries that are frontrunners in the field of practice organization had the same number of panellists as larger countries or countries that are lagging behind from an organizational point of view, and the ratings of all panels had an equal weight.

Fourthly, panel composition in consensus methodologies is a fundamental factor in determining the legitimacy of the findings.¹² Care was taken to ensure that panels reflected a range of expertise by choosing GPs and in appropriate countries (The Netherlands and UK) practice managers, as these are the disciplines that are involved in practice management. The panels contained men and women, and acknowledged leaders in primary care. All panels contained ten members, which is within the recommended range of 7 to 15 to permit sufficient diversity.³⁶ Nevertheless, they could not be said to be representative within each country. Moreover, the framework and the indicators which the panels had to rate, had been developed by the research partners of the participating countries as part of a European network, who all have specialist expertise in primary care/practice organization. Therefore, the outcome of the study was determined by the framework of practice management developed by the research partners.

Lastly, the process of translation did not adhere completely to formal translation procedures. But in each country more than one partner took part in the translation, which incorporated backwards translation procedures.

Implications for quality assessment

The usefulness of this framework and set of indicators will only be clear after further research establishes its validity, acceptability and feasibility. The indicator set needs to be assessed on a national and on an international level.⁹ Ideally, the set will be useful to provide feedback for practices to reflect on their performance. The common set could be used for a number of other purposes, such as supporting professional quality improvement activities, practice accreditation, research, contracting practices, enhancing transparency about service quality and for enabling patients to make better-informed decisions. The main purpose of the common set may be the demonstration of differences within and between countries.

Conclusion

This set of quality indicators gives insight in to the essential aspects of general practice management across these six European health care systems. The practical considerations of applying these indicators will need careful consideration before they can be seen as valid performance measurement tools. The research instruments that have been developed based on these indicators are currently being tested in nine countries in 30 practices.

Declaration

Funding: Bertelsmann Foundation Germany.

Ethical approval: n/a.

Conflicts of interest: none.

Appendix 1

Members of the EPA collaboration

Centre for Quality of Care Research (WOK), Universities of Nijmegen and Maastricht, The Netherlands: Richard Grol, Yvonne Engels, Maaike Dautzenberg and Pieter van den Hombergh; Bertelsmann Foundation, Gütersloh, Germany: Henrik Brinkmann, Andreas Esche and Jan Böcken; AQUA Institute, Göttingen, Germany: Joachim Szecsenyi, Ferdinand Gerlach, Björn Broge and Petra Wippenbeck; Société de Formation Thérapeutique du Généraliste, Paris, France: Marianne Samuelson and Hector Falcoff; Swisspep Institute for Quality and Research in Healthcare, Gümligen, Switzerland: Beat Künzi and Walter Oswald; Scientific Society of Flemish General Practitioners, Berchem (Antwerp), Belgium: Luc Seuntjens and Nicole Boffin; Department of Primary Care, University of Wales Swansea Clinical School: Glyn Elwyn and Melody Rhydderch; National Primary Care Research and Development Centre, University of Manchester, UK: Stephen Campbell and Martin Marshall.

Appendix 2

Instruments for the assessment and improvement of the organization of general practices

Aiming for Excellence in General Practice: a comprehensive set of RNZCGP standards for general practice.²²

Europep: A European instrument aiming at involving patients effectively in (improving) care and aiming at strengthening their role.²³

Insight 360 degrees: A computerised feedback tool for personal and organizational development in general practice. It collects information from several sources in order to be able to prioritise areas of improvement.²⁴

Maturity Matrix: A formative self-assessment tool, based on an externally facilitated small group process, that can be applied as a tool for internal and external assessment in primary care organizations. It can be used to benchmark an organization against others, in order to set targets, or to determine one's own position against the position of others.²⁵www.medicine.swan.ac.uk/publicationsframe.html

Quality of Care in general practice: An instrument for the evaluation of quality management in family practice, based on the theoretical concept of the Excellence Model of the European Foundation of Quality Management (EFQM) and adapted for use in small-scale family practices.^26,27

Standards for General Practice (2nd edn): A document that defines minimum acceptable standards for accreditation of general practices in Australia. The Royal Australian College of General Practitioners. Australia, 2000.²⁸

Swisspep Quali Doc: A balanced scorecard tool, with questionnaires for patients and all practice members, to give primary care practices an individual practice profile^29,30 (www.swisspep.ch).

The Accredited Professional Development (APD) programme of the RCGP (UK) offers ongoing support for GPs' continuing professional development (CPD) as part of their everyday practice (www.rcgp.org.uk).

The Family Practice Management Practice Self-Test: An easy way to gauge how well the practice is doing in everything from quality of care to quality of claims.³¹

VIP (Visit Instrument to assess Practice management): a detailed tool, based on staff and patient questionnaires, a practice visit and feedback to improve management in general practice.¹⁶

Warr-Cook-Wall work satisfaction questionnaire.³²

References

Author notes

aCentre for Quality in Care Research (WOK), Nijmegen the Netherlands, bFor the EPA working party (see Appendix 1) and cNational Primary Care Research and Development Centre, University of Manchester, UK