Skip to main content

Main menu

  • HOME
  • ONLINE FIRST
  • CURRENT ISSUE
  • ALL ISSUES
  • AUTHORS & REVIEWERS
  • SUBSCRIBE
  • BJGP LIFE
  • MORE
    • About BJGP
    • Conference
    • Advertising
    • eLetters
    • Alerts
    • Video
    • Audio
    • Librarian information
    • Resilience
    • COVID-19 Clinical Solutions
  • RCGP
    • BJGP for RCGP members
    • BJGP Open
    • RCGP eLearning
    • InnovAIT Journal
    • Jobs and careers

User menu

  • Subscriptions
  • Alerts
  • Log in

Search

  • Advanced search
British Journal of General Practice
Intended for Healthcare Professionals
  • RCGP
    • BJGP for RCGP members
    • BJGP Open
    • RCGP eLearning
    • InnovAIT Journal
    • Jobs and careers
  • Subscriptions
  • Alerts
  • Log in
  • Follow bjgp on Twitter
  • Visit bjgp on Facebook
  • Blog
  • Listen to BJGP podcast
  • Subscribe BJGP on YouTube
Intended for Healthcare Professionals
British Journal of General Practice

Advanced Search

  • HOME
  • ONLINE FIRST
  • CURRENT ISSUE
  • ALL ISSUES
  • AUTHORS & REVIEWERS
  • SUBSCRIBE
  • BJGP LIFE
  • MORE
    • About BJGP
    • Conference
    • Advertising
    • eLetters
    • Alerts
    • Video
    • Audio
    • Librarian information
    • Resilience
    • COVID-19 Clinical Solutions
Editorial

Measuring performance quality in general practice: is international harmonization desirable?

Richard Grol and Michel Wensing
British Journal of General Practice 2007; 57 (542): 691-692.
Richard Grol
Roles: Professor, Director
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michel Wensing
Roles: Senior Lecturer
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Info
  • eLetters
  • PDF
Loading

The call for transparency about quality of care provided to patients has become stronger in recent years in most developed countries. Society demands that care providers, GPs included, account for their performance, and is increasingly prepared to pay for better care. The development in accountability faces critical debate, such as questions about its impact on improving practice, and claims that it is a threat to professionalism, that it promotes strategic behaviour and even fraud, and increases bureaucracy and costs of health care.1–3

As public reporting of performance and pay for performance are becoming more of a reality in health care, it is important to observe and evaluate these new ‘accountability approaches’.

Seen from the other side of the North Sea, the pay-for-performance initiative in general practice in the UK (Quality and Outcomes Framework [QOF] and GMS contract) is one of the most interesting quality improvement experiments for GPs in the world today. We in the rest of Europe learn from this experiment. However, we sometimes do things differently, and also ask ourselves why the UK does not take on board some of the relevant experiences gained within Europe on measuring performance in general practice.

One of the crucial issues related to any performance assessment, whatever its objectives, is the validity and reliability of the indicators and measurement instruments used. In particular, when indicators are used for comparison, incentives, or certification, they should be ‘correct and fair’ and avoid unjustified harm to practitioners.4 Indicators should meet the highest standards for quality, their features should be tested, the scores should be corrected for case-mix differences. Those assessed should have the possibility to check data before use by others.

There is vast literature on criteria for indicators and instruments,5,6 including papers focused on measuring patients' views of care.7 However, in different countries many current initiatives related to performance assessment, public reporting, and pay for performance fail to meet such requirements. Based on a large number of projects undertaken in the Netherlands and other European counties, we identified some steps in the development and validation of performance indicators and instruments to measure performance in primary care:

  • Development of a logical and consistent framework covering all aspects of the field to be assessed with all relevant stakeholders involved. This framework needs to assure good coverage of the field and acceptance by all stakeholders. Strategic behaviour, such as aiming at high performance for only those aspects of care that are measured, needs to be prevented. For instance, indicators reflecting patient experiences need to cover a variety of aspects of primary care services, including accessibility of the practice, professional performance in consultations with the patient, and involvement of patients in decision making and in debates about practice development. Indicators for patient experiences should be developed together with patients to ensure that they reflect patients' priorities.

  • Specific indicators can be derived from the vast international literature on indicators, evidence-based clinical guidelines, and other sources. A rigorous procedure (for example, the Rand modified Delphi method) is required to select the most relevant indicators. The selection of indicators should be guided by the goals of performance assessment. For example, a review identified six instruments for measuring patient views in the context of formative feedback.8 In most of our indicator projects we observed that many ‘obvious’ indicators do not meet these criteria. Therefore, indicators need to be selected using robust methods.

  • Feasible and reliable instruments need to be selected or developed to collect data and measure the indicators. Decisions are required as to whether new instruments should be developed and what type of instruments are suitable (for example, checklists, questionnaires, observations, prospective self-recording).

  • A test in real practice, such as an audit, is a crucial step to determine whether instruments and their items meet criteria for construct and criterion validity, reliability, case-mix control, sensitiveness to change, and acceptability by those assessed. As this test will normally lead to a further reduction of the indicators selected, it is not wise to implement the indicators widely at first.

  • Finally, it is crucial to consider the impact that indicators will have on the delivery of care after wide-scale implementation. Even a rigorously developed indicator may have undesired side effects or show little room for improvement.

In this issue of the BJGP two articles provide evidence of the failure to meet these requirements in some widely used instruments for measuring patient experiences with health care. Garratt et al9 reviewed four questionnaires to measure patient experiences and satisfaction with primary care out-of-hours services: two from the UK and two from the Netherlands. They conclude that all questionnaires had limitations regarding validation.

The article by Hankins et al10 in this issue of the Journal reviewed the value of instruments now used as part of the QOF. It concludes that the two instruments approved by the QOF, which are used to determine part of the financial bonus, appear to have insufficient research evidence of validity and reliability.

Serious consideration of these issues in the ongoing revision of the QOF indicators is essential in our opinion. Current developments outside the UK also need to be taken into consideration. A critical question could be why the QOF has not yet adopted some of the indicators and instruments developed in a European research collaboration in primary care (the TOPAS-Europe association).

The EUROPEP instrument, developed by TOPAS, is probably the most widely used and best validated patient satisfaction instrument for general practice available. This tool was developed and validated using a rigorous procedure. This included a needs assessment of thousands of patients in 11 European countries, including the UK, which showed many similarities between patient expectations of primary care.11 A systematic process was used to develop the instrument (framework development, inventory of available indicators and items, and a selection process), and there has been wide testing in 16 European countries, including the UK.12–14

The EUROPEP is now used as the national instrument to measure patient experience with primary care in some countries, such as Denmark and Switzerland. After several years of use, the EUROPEP has been recently modified by an international working group and a user manual has been developed. It will be tested again in some countries before wider implementation is recommended.

The instrument is short (23 items) and makes international comparison possible. Will the UK continue to drive on the left side of the patient experience road, or harmonize with the rest of Europe in the use of this patient satisfaction instrument?

EUROPEP is used in conjunction with another important instrument that could be included in the QOF: the European Practice Assessment (EPA).15–17 The EPA focuses on practice management and organisation and was developed using the same rigorous validation methods as the EUROPEP. It was developed within a collaboration between researchers and practitioners from general practice from 10 European countries including the UK. In addition to questions for GPs and nurses and visitor observations, the EPA contains a number of questions for patients about factual experiences with their primary care practice; for example, concerning their experiences with accessibility or coordination of care.

It would be great if UK general practice could compare itself with achievements in other countries using such standardised instruments, to see where improvements in measuring and enhancing services for their patients are possible.

An interesting finding from a small group of practices in different countries was that the UK has the highest scores for the topic ‘incident reporting’ (an indicator in the QOF), while it has the lowest score on ‘essential drugs in doctors' emergency bag’ (not in the QOF).17

The newest development of the TOPAS collaboration concerns the validation and testing of a set of European primary care indicators related to cardiovascular risk and disease management (EPA-Cardio) in nine European countries. The first publication of the indicators and their validity and use in practice will soon follow.

Many quality assessment schemes for primary care have been developed around the world in the last decade. The QOF is one of the most intriguing and ambitious ones. However, the validation of the indicators and instruments used (for example, for measuring patient experiences) may be improved. Collaboration and exchange of developments and expertise between different countries could help to take this difficult undertaking forward, and make harmonization and comparison of measures and data between countries possible.

  • © British Journal of General Practice, 2007.

REFERENCES

  1. ↵
    1. Mangin D,
    2. Toop L
    (2007) The Quality and Outcomes Framework: what have you done to yourselves? Br J Gen Pract 57(539):435–437.
    OpenUrlFREE Full Text
    1. Werner R,
    2. Asch D
    (2005) The unintended consequences of publicly reporting quality information. JAMA 293(10):1293–1344.
    OpenUrl
  2. ↵
    1. Roland M
    (2007) The Quality and Outcomes Framework: too early for a final verdict. Br J Gen Pract 57(540):525–527.
    OpenUrlFREE Full Text
  3. ↵
    1. Russell E
    (1998) The ethics of attribution: the case of health care outcome indicators. Soc Sci Med 47(9):1161–1169.
    OpenUrlCrossRefPubMed
  4. ↵
    1. Streiner DL,
    2. Norman GR
    (1995) Health measurement scales: a practical guide to their development and use (Oxford Medical Publications, Oxford).
  5. ↵
    1. Campbell SM,
    2. Braspenning J,
    3. Hutchinson A,
    4. Marshall M
    (2002) Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care 11(4):358–364.
    OpenUrlAbstract/FREE Full Text
  6. ↵
    1. Wensing M,
    2. Elwyn G
    (2003) Methods for incorporating patients'views in health care: validity, effectiveness and implementation. BMJ 326(7394):877–879.
    OpenUrlFREE Full Text
  7. ↵
    1. Evans RG,
    2. Edwards A,
    3. Evans S,
    4. et al.
    (2007) Assessing the practising physician using patient surveys: a systematic review of instruments and feedback methods. Fam Pract 24(2):117–127.
    OpenUrlCrossRefPubMed
  8. ↵
    1. Garratt AM,
    2. Danielsen K,
    3. Hunskaar S
    (2007) Patient satisfaction questionnaires for primary care out-of-hours services: a systematic review. Br J Gen Pract 57(542):741–747.
    OpenUrlAbstract/FREE Full Text
  9. ↵
    1. Hankins M,
    2. Fraser A,
    3. Hodson A,
    4. et al.
    (2007) Measuring patient satisfaction for the Quality and Outcomes Framework. Br J Gen Pract 57(542):737–740.
    OpenUrlAbstract/FREE Full Text
  10. ↵
    1. Grol R,
    2. Wensing M,
    3. Mainz J,
    4. et al.
    (1999) Patients' priorities with respect to general practice care: an international comparison. European Task Force on Patient Evaluations of General Practice (EUROPEP). Fam Pract 16:4–11.
    OpenUrlCrossRefPubMed
  11. ↵
    1. Grol R,
    2. Wensing M,
    3. Mainz J,
    4. et al.
    (2000) Patients in Europe evaluate general practice care: an international comparison. Br J Gen Pract 50(460):882–887.
    OpenUrlAbstract/FREE Full Text
    1. Wensing M,
    2. Vedsted P,
    3. Kersnik J,
    4. et al.
    (2002) Patient satisfaction with availability of general practice: an international comparison. Int J Qual Health Care 14(2):111–118.
    OpenUrlCrossRefPubMed
  12. ↵
    1. Wensing M,
    2. Baker R,
    3. Szecsenyi J,
    4. Grol R
    (2004) Impact of national health care systems on patient evaluations of general practice in Europe. Health Policy 68(3):353–357.
    OpenUrlCrossRefPubMed
  13. ↵
    1. Grol R,
    2. Dautzenberg M,
    3. Brinkmann H
    , eds (2004) European Practice Assessment, Quality management in primary care (Verlag Bertelsmann Stiftung, Gütersloh).
    1. Engels Y,
    2. Dautzenberg M,
    3. Campbell S,
    4. et al.
    (2006) Testing a European set of indicators for the evaluation of the management of primary care practices. Fam Pract 23(1):137–147.
    OpenUrlCrossRefPubMed
  14. ↵
    1. Engels Y,
    2. Campbell S,
    3. Dautzenberg M,
    4. et al.
    (2005) Developing a framework of, and quality indicators for, general practice management in Europe. Fam Pract 22(2):215–222.
    OpenUrlCrossRefPubMed
Back to top
Previous ArticleNext Article

In this issue

British Journal of General Practice: 57 (542)
British Journal of General Practice
Vol. 57, Issue 542
September 2007
  • Table of Contents
  • Index by author
Download PDF
Email Article

Thank you for recommending British Journal of General Practice.

NOTE: We only request your email address so that the person to whom you are recommending the page knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Measuring performance quality in general practice: is international harmonization desirable?
(Your Name) has forwarded a page to you from British Journal of General Practice
(Your Name) thought you would like to see this page from British Journal of General Practice.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Measuring performance quality in general practice: is international harmonization desirable?
Richard Grol, Michel Wensing
British Journal of General Practice 2007; 57 (542): 691-692.

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero

Share
Measuring performance quality in general practice: is international harmonization desirable?
Richard Grol, Michel Wensing
British Journal of General Practice 2007; 57 (542): 691-692.
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One
  • Mendeley logo Mendeley

Jump to section

  • Top
  • Article
    • REFERENCES
  • Info
  • eLetters
  • PDF

More in this TOC Section

  • New concepts in iron deficiency anaemia
  • Funding for general practice in the next decade: life after QOF
  • Mental health help-seeking behaviours in young adults
Show more Editorial

Related Articles

Cited By...

Intended for Healthcare Professionals

BJGP Life

BJGP Open

 

@BJGPjournal's Likes on Twitter

 
 

British Journal of General Practice

NAVIGATE

  • Home
  • Current Issue
  • All Issues
  • Online First
  • Authors & reviewers

RCGP

  • BJGP for RCGP members
  • BJGP Open
  • RCGP eLearning
  • InnovAiT Journal
  • Jobs and careers

MY ACCOUNT

  • RCGP members' login
  • Subscriber login
  • Activate subscription
  • Terms and conditions

NEWS AND UPDATES

  • About BJGP
  • Alerts
  • RSS feeds
  • Facebook
  • Twitter

AUTHORS & REVIEWERS

  • Submit an article
  • Writing for BJGP: research
  • Writing for BJGP: other sections
  • BJGP editorial process & policies
  • BJGP ethical guidelines
  • Peer review for BJGP

CUSTOMER SERVICES

  • Advertising
  • Contact subscription agent
  • Copyright
  • Librarian information

CONTRIBUTE

  • BJGP Life
  • eLetters
  • Feedback

CONTACT US

BJGP Journal Office
RCGP
30 Euston Square
London NW1 2FB
Tel: +44 (0)20 3188 7400
Email: journal@rcgp.org.uk

British Journal of General Practice is an editorially-independent publication of the Royal College of General Practitioners
© 2023 British Journal of General Practice

Print ISSN: 0960-1643
Online ISSN: 1478-5242