Abstract
Background While practice-level or team accreditation is not new to primary care in the UK and there are organisational indicators in the Quality and Outcomes Framework (QOF) organisational domain, there is no universal system of accreditation of the quality of organisational aspects of care in the UK.
Aim To describe the development, content and piloting of version 1 of the Primary Medical Care Provider Accreditation (PMCPA) scheme, which includes 112 separate criteria across six domains: health inequalities and health promotion; provider management; premises, records, equipment, and medicines management; provider teams; learning organisation; and patient experience/involvement, and to present the results from the pilot service evaluation focusing on the achievement of the 30 core criteria and feedback from practice staff.
Design of study Observational service evaluation using evidence uploaded onto an extranet system in support of 30 core summative pilot PMCPA accreditation criteria.
Setting Thirty-six nationally representative practices across England, between June and December 2008.
Method Study population: interviews with GPs, practice managers, nurses and other relevant staff from the participating practices were conducted, audiotaped, transcribed, and analysed using a thematic approach. For each practice, the number of core criteria that had received either a‘good’or‘satisfactory’rating from a RCGP-trained assessment team, was counted and expressed as a percentage.
Results Thirty-two practices completed the scheme, with nine practices passing 100% of core criteria (range: 27–100%). There were no statistical differences in achievement between practices of different sizes and in different localities. Practice feedback highlighted seven key issues: (1) overall view of PMCPA; (2) the role of accreditation; (3) different motivations for taking part; (4) practice managers dominated the workload associated with implementing the scheme; (5) facilitators for implementation; (6) patient benefit — relevance of PMCPA to quality improvement; (7) recommendations for improving the scheme.
Conclusion Version 1 of PMCPA has been piloted as a primary care accreditation scheme and shown to be relevant to different types of practice. The scheme is undergoing revision in accordance with the findings from the pilot and ongoing consultation.
INTRODUCTION
There are over 8500 general practices in England, all providing primary medical care services. However, there is no universal system of accreditation of the quality of organisational aspects of care and no contractual levers to promote organisational quality beyond the voluntary indicators within the Quality and Outcomes Framework (QOF) organisational domain, which had a mean practice achievement rate in 2007/2008 of 94.5%.1
Patients have the right to expect and receive high-quality care. Structural features of primary care organisation do not guarantee the provision of safe and quality care; however, they can influence the processes of clinical care.2–4 For example, if the necessary equipment or skills are not available, this can affect patient safety; if there are no available appointments, this will influence patient access to care.
Accreditation is a model of external assessment of healthcare providers that is being increasingly used worldwide.5–7 Practice-level or team accreditation is not a new concept to primary care in the UK, with previous initiatives such as Quality Team Development, Quality Practice Award, and practice team accreditation.8,9 These previous schemes have all been voluntary programmes using standards set above the expected minimum. Recent UK government policy documents such as the Next Stage Review have focused on organisational issues of primary care and expressed support for provider accreditation schemes that focus on measures to ensure continuous improvement in the quality of primary and community care.10
In 2008, an accreditation scheme called Primary Medical Care Provider Accreditation (PMCPA), focusing on organisational issues of primary care, was developed and piloted by the authors, in conjunction with the Royal College of General Practitioners (RCGP). The aims of the pilot were to evaluate the experiences of a representative sample of general practices across England in implementing PMCPA, and to analyse the uptake and achievement of the core criteria in each of the domains. This paper focuses on the content and concept of PMCPA, how PMCPA was developed and piloted, and the uptake and achievement of the core criteria within the scheme. It does not describe how it might be implemented since this is the subject of ongoing discussion.
METHOD
Development of the accreditation scheme
PMCPA focuses on organisational rather than clinical issues, although criteria that are organisational but support clinical issues were included. It was developed using the methodology and criteria of the RCGP Quality Team Development scheme,8 as a starting point, supplemented by criteria from other international accreditation schemes in primary care. These included ideas from the European Practice Assessment scheme,11–13 and New Zealand Practice Accreditation Scheme.14 Criteria were also derived from recent primary care policy documents including the RCGP Road Map,15 and Standards for Better Health.16 This enabled criteria to reflect currently accepted good practice and also key issues for the future development of quality primary care practice. Six separate quality domains were identified based on this evidence base (Table 1).
Each criterion was allocated to an appropriate domain and reviewed by an accreditation development group, and divided into those that were deemed summative (yes/no) and those that were formative/developmental in nature. Summative assessment represents a judgment of‘pass or fail’, which must be underpinned by a clear idea of what constitutes acceptable performance when judged by external assessors. In comparison, formative or educational assessment is an internal reflective process to promote improvement and attainment of a goal, which is judged as evidence of improvement rather than a simple pass or fail. Each criterion was underpinned by guidance, provided to participants, about the evidence necessary to meet the criterion.
Each domain was created to include a balance of core summative and developmental formative criteria. Indicators in the organisational component of the QOF were excluded, as were criteria that covered legal and contractual requirements in version 1 of PMCPA. The overall framework of PMCPA including the six domains and the number of core (n = 30) and developmental (n = 82) criteria is shown in Table 1. The full set of core criteria and associated evidence is shown in Appendix 1. This paper describes the results from the service evaluation, focused on the 30 core criteria for which data were collected in all participating practices.
Prior to participation in PMCPA, practices were asked to sign a declaration that they complied with a list of statutory, contractual, and legal requirements. Practices participating in the pilot scheme were asked to self-assess and provide written documentation against all 30 core criteria and three or four randomly selected developmental criteria. Practices self-assessed and uploaded written documentation in support of each criterion, using a web-based extranet system created for the pilot. This also enabled documents to be time stamped in terms of uploading. The extranet suite also had a facility for staff to communicate directly with staff in other participating practices in their primary care trust (PCT).
After the pilot period, practices were visited by one of the three authors and asked to describe their experience and views of PMCPA as part of the service evaluation. Eighty-three practice staff took part in these feedback sessions (31 GPs, 33 practice managers, and 13 practice nurses, as well as six other staff such as information technology/audit personnel).
Sample and method
The study aimed to recruit a total of 40 practices — 10 each in four purposively chosen PCTs, selected to provide a mix of rural/urban and deprived/affluent areas. In each PCT area, a stratified random sample was selected to be representative of all practices in the PCT in terms of deprivation, practice list size, and 2007 QOF score.
If a practice in a particular sampling cell (that is, small practice in a deprived area) declined to participate, a practice with a similar profile from the same sampling cell was invited to take part. A criteria and information pack was distributed to each participating practice in May 2008. The pack detailed information on the scheme, the evidence required, and guidance to meet the core and developmental criteria, as well as user instructions for the extranet site. Practices were allocated 15 weeks, from 15 June to 30 September 2008, to prepare the self-assessment report and submit written evidence via the web-based extranet site.
An RCGP-trained assessment team consisting of a clinician, a practice manager, and a lay assessor, was appointed by the respective PCT for each practice. The assessor team reviewed the relevant evidence prior to the visits, which took place between October and December 2008. RCGP assessor training included calibration to help ensure validity, reliability, repeatability, and equity of assessment. At each practice, all 30 core criteria were rated as‘good’or‘satisfactory’, representing a pass of a summative criterion, or‘borderline’or‘unsatisfactory’, representing a fail of the summative criterion.
Analyses
For each practice, the number of core criteria (out of all 30) that had received either a‘good’or‘satisfactory’rating were counted and expressed as a percentage of the number of criteria that were attempted/applied to each practice.
Practices were classed as being either‘single-handed’or‘group’practices. Single-handed was defined as those practices with no more than 1.5 full-time equivalent GPs. In addition, to examine the effect of list size, practices were divided into three roughly equal groups based on list size: less than 4000 patients (11 practices); 4000 to 6999 (13 practices); and 7000 or more (12 practices). Index of Multiple Deprivation (IMD) scores were used to examine the impact of location on practice means.
Since the distribution of practice total core criteria scores was highly skewed and the sample size fairly small, the data were analysed using non-parametric statistical methods. Median scores were used as the measure of central tendency, rather than means. Relationships between total criteria scores and the categories of practice type and size were investigated using quantile regression, to allow examination of relationships with and without control for IMD scores as a covariate. Regression coefficients were tested for statistical significance using a non-parametric method based on 1000 bootstrap replications. All analyses were undertaken in Stata (version 10.1).
Key concepts were identified from these feedback session notes for each practice and summarised for each practice by using an open coding method. Codes that had common elements were merged to form categories. These categories are reported in this paper.
RESULTS
A total of 80 practices were invited to take part, and a sample of 36 practices were recruited (45%): 10 in Haringey, London (from 21 invited: 48%); eight in Nottinghamshire (18: 44%); nine in Oldham (19: 47%); and nine in Warwickshire (22: 41%). The characteristics of this sample of 36 practices compared to all practices in England are shown in Table 2.
Four practices withdrew during the piloting process and the remaining 32 practices fully completed the pilot (eight in Haringey; eight in Nottinghamshire; nine in Oldham; and seven in Warwickshire). Thirty of these 32 practices received an assessment visit by their PCT-appointed, three-person team. Two practices were unable to complete assessment visits due to difficulties timetabling a visit with assessors. There were therefore 30 practices with an assessment rating of each criterion.
Thirty-four practices uploaded 1817 documents on to the extranet site (mean 53.4, range 1–165); two practices did so before withdrawing from the pilot. Over 80% of documents were uploaded by practice staff in the last month, with most practices uploading documents during the last week, and some in the last 24 hours of the pilot; although it is known anecdotally that this did not necessarily reflect when the majority of the work took place.
Overall achievement
At most practices, all 30 criteria were deemed to be applicable by assessors and assigned a rating. Where all 30 criteria applied and were rated, eight practices scored 30/30 using a rating of‘good’or‘satisfactory’as a pass. Three of these practices were small practices (list size <4000 patients), two medium-sized practices (list size: 4000–7999 patients), and three were larger practices (list size: ≥8000 patients). Of the practices passing 30/30 core criteria, two were in areas with a low deprivation index, five in areas of medium deprivation index, and one in an area of high deprivation.
In some practices, certain criteria were not given a rating by assessors; for example, core criterion 9 related to minor surgery and four practices stated they did not provide a minor surgery service. As such, the denominator for ascribing an overall score to a practice was not always 30. One additional practice scored 28/28. This meant that nine practices (30%) passed 100% of the core criteria for which they were given a rating by their assessors. On this basis, practices passed between 27% and 100% for the core criteria for which they were given an assessment rating. Only two practices had scores of ≤65%, while half of all practices scored ≥90% within the 15-week timeframe.
The frequency with which each of the 30 core criteria were rated good/satisfactory or borderline/unsatisfactory/not applicable across the evaluation sample of practices is shown in Appendix 1
Only one core criterion relating to confidentiality (CC19) was passed as ‘good’ or ‘satisfactory’ by all participating practices.
Achievement by practice characteristics
Across the 30 core criteria, single-handed practices had a lower median pass (good or satisfactory) score (73.3%) compared to group practices (93.3%). However, there were no statistical differences in achievement between different-sized practices (P = 0.13). Single-handed practices were more likely to be located in areas with a higher IMD score (Mann–Whitney U test, P<0.01), but adjusting for IMD did not affect the (lack of) relationship with total core criteria scores (P = 0.17). Median total criteria scores in relation to practice size were 73.3 for single-handed practices, 96.6 for medium-sized practices, and 86.7 for larger practices. Although the mean score was greater for medium-sized practices, under quantile regression the differences between the group means were not significant (P = 0.11), and remained non-significant after adjustment for IMD scores (P = 0.11). Figure 1 shows the individual practice total core criteria pass scores by practice deprivation for single-handed and group practices.
Practice feedback
Practice feedback highlighted seven key issues: (1) overall view of PMCPA; (2) the role of accreditation; (3) different motivations for taking part; (4) practice managers dominated the workload associated with implementing the scheme; (5) facilitators for implementation; (6) patient benefit — relevance of PMCPA to quality improvement; and (7) recommendations for improving the scheme.
Overall view of PMCPA
All practices felt that PMCPA was relevant to and aligned with family practice priorities, reflected quality in primary care, and was a worthwhile use of practice time. PMCPA was seen as promoting improvements in organisational standards. However, the main risk identified was that externally imposed standards could be seen as a‘tick-box’exercise, with organisations seeking to meet the target without necessarily reflecting on how the issues contained within the standard affect their own setting. PMCPA was not seen as just a tick-box exercise by most practices but as an opportunity to change and benefit the practice.
Many managers and doctors, however, felt that the scheme was too heavily populated by criteria where the evidence for external assessment focused on demonstrating the existence of a protocol or procedure. These health professionals stated that PMCPA should focus more on the evidence of change of benefit to the practice.
The role of accreditation
There was general consensus about the value of accreditation, but different interpretations of its role. Some practices fed back that the role of accreditation is to show adherence to an acceptable standard in terms of compliance or conformance with an accepted set of standards. Others explicitly emphasised focusing on formative practice-specific quality improvement as a reflective exercise rather than a box-ticking exercise. The RCGP was, however, seen as the arbiter of professional standards, and their leadership of the scheme was seen as key by almost all practices.
Motivations for taking part
Practices had different reasons for taking part in the accreditation pilot. This was most often described as a team-development exercise focusing on practice quality improvement, but also to fix a perceived problem such as current standards or deficiencies in team working, or to demonstrate how good they perceived their practice to be.
Workload
Practice managers carried out 90–95% of the actual workload. GPs tended to make the initial decision to be involved and then largely had a hands-off approach and were used as a‘checking in point’by most managers. The workload was higher than expected in most practices, although almost all practices emphasised that this reflected the 15-week duration of the pilot, the timing (over the summer months), and the fact that the pilot was implemented at a time when many practices were busy with other concurrent demands such as the information management and technology direct enhanced service.
Facilitators for implementation
While most of the workload was undertaken by managers within practices, both doctor and team engagement at some level were critical to success, and those practices that did not complete PMCPA were usually those where only the manager was truly engaged in the scheme. A combination of team work, designated roles and responsibilities within the team, protected time to work on PMCPA, engagement and ownership by all team members, leadership (from a manager and/or doctor), and doctor buy-in were crucial factors in implementing PMCPA as a learning organisation.
Patient benefit — relevance of PMCPA to quality improvement
Many participants explained how using PMCPA as a learning exercise within the practice, with a focus on internal reflective quality-improvement strategies, as well as focusing on the targets for external assessment, led to direct patient benefits relevant to their practice population. In many practices, staff explained how having to write practice protocols in order to meet the core criteria formalised tacit knowledge. Other examples included facilitating changes in the structures and processes of care that improved patient safety, making services more responsive to the needs of patients, and a greater community focus. A minority of managers and doctors contextualised the role of schemes like PMCPA from the perspective of‘looking through the eyes of patients’.
Examples of how practices used PMCPA to make their practice more responsive to the needs of their patients included thinking about how to work better with local interpreting services, creating a histology safety net to improve the recall system after minor surgery, training healthcare assistants to understand abnormal results picked up in new patient checks, creating fail-safe systems for non-collection of prescriptions, and a variety of initiatives to improve information giving to and consultation with patients.
Recommendations for improving the scheme
Practice feedback on how to improve PMCPA emphasised 2–3 years as a realistic timeframe for completing the scheme and a focus on using evidence of implementation and learning rather than the simple presence of a written protocol. The experiences of piloting PMCPA and listening to feedback in practices of different sizes clearly showed the need to make PMCPA, and the evidence that is presented to meet standards, practice specific and flexible. For example, evidence used to pass a criterion might not be the same in every practice. Many participants also recommended that PMCPA needs to have a greater emphasis on patient benefit and patient responsiveness.
DISCUSSION
Summary of main findings
PMCPA has been developed as a criteria- and evidence-based accreditation scheme based on the RCGP Quality Team Development model and other international accreditation schemes. It is a scheme that seeks to be applicable equally to all providers working in primary care. Within a representative sample of practices, 30% of participating general practices passed 100% of the core criteria and half of all practices scored 90% or above, despite the short 15-week pilot period. There were no statistical differences in achievement levels between practices of different sizes. All participating practices fed back that PMCPA was relevant to and aligned with primary care priorities.
Strengths and limitations of the study
The strengths of this pilot include the representative nature of the sample of practices. Despite the short time period for piloting, 32 of the 36 recruited practices completed the scheme. However, the sample size of participating practices was relatively small, hence only fairly substantial differences between different kinds of practices would have been detectable. This paper has also only reported the summative components of PMCPA and not the formative aspects of continuous quality improvement inherent within the developmental criteria, because data were not collected in all practices for these criteria.
Comparison with existing literature
While accreditation has acquired different meanings in different healthcare settings,17 it nevertheless represents official recognition, acceptance, or approval of demonstrable compliance against set standards.18 There are five ways in which assessment against set standards might be used.6 These are quality control (mandatory, externally set, minimum predetermined acceptable standards), mandatory regulation (legal or safety standards), continuous quality improvement (to show ongoing excellence above a minimum standard), information giving (to enable comparison between providers by patients and policy makers), and marketing (to showcase a standard of service available). A complex picture emerges from reviews of healthcare accreditation schemes worldwide but two key features are commonplace — promoting change, and professional development.5 The piloted version of PMCPA incorporates both quality control with summative assessment and quality improvement with practice-specific formative assessment. Scrivens recommended over a decade ago that accreditation schemes need to move away from just external peer review and incorporate continuous quality improvement.19 It is increasingly recognised worldwide that a quality-improvement strategy at a primary care practice level needs a balance between quality assurance/regulation (summative) and educational (formative internal improvement) approaches.8,14,20
PMCPA echoes previous quality initiatives in the UK where primary healthcare teams generated quality improvement when they had the time and resources to learn, but work and plan together with clear objectives and under the leadership of a manager or doctor.21
Implications for clinical practice and future research
There are concerns about the effectiveness and appropriateness of practice accreditation, despite mounting international recognition of its importance.6,17 In the UK context, where a number of different quality-improvement initiatives are currently in development, the white paper Our Health, Our Care, Our Sayadvocated the need to coalesce current administrative requirements on primary care into one scheme, thereby reducing overall burden.22 It will also be important that accreditation is not just a paper exercise but leads to practice-specific and relevant quality improvement and patient benefit. There is a need for a mix of professional, clinical, managerial, and financial approaches to quality improvement.23 In the UK there is an increasing plurality of primary care providers from‘traditional’general practices to general or personal medical services (PMS), PMS plus, out-of-hours providers, and a range of private provider models. A primary care provider accreditation scheme will also need to be responsive to these different modes of delivery.
Quality team development in primary care was found to have positive benefits for practices but participants tended to be a self-selecting innovative minority of practices.8 The challenge now is how best to roll out PMCPA so that it is engaged with by the majority of practices. If PMCPA is to be implemented in a meaningful way, it will need to align and integrate with other ongoing and planned quality-improvement strategies in the UK. In particular, to avoid duplication of effort and data gathering, PMCPA will need to support the registration requirements of the Care Quality Commission, revalidation of doctors, and revisions of the QOF.
PMCPA is currently being revised in light of the findings from the pilot and to ensure the scheme is integrated with and aligned to other national initiatives. However, the findings from this pilot suggest that PMCPA was a positive experience for the participating practices and that practice size and locality do not affect engagement or achievement.
Acknowledgments
The authors would like to thank Dr Janet Hall (Accreditation Development Group), Dr Meeta Kathoria, Dr Evangelos Kontopantelis, Dr David Reeves, Professor Nigel Sparrow, Rehema Shabaya, Dr Bill Taylor (Accreditation Development Group), Caroline Turnbull, and the practice staff in all participating practices, as well as all members of the PMCPA Stakeholder Committee.
Appendix 1. PMCPA core criteria and participating practices pass rates (n = 30)
Notes
Funding body
This work was done at the National Primary Care Research and Development Centre, which receives funding from the Department of Health. Also Supported by the Royal College of General Practitioners. The views expressed are those of the authors and not necessarily those of the Department of Health or the RCGP.
Ethics committee
North West Research Ethics Committee.
Competing interests
The authors have stated that there are none.
Discuss this article
Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss
- Received July 29, 2009.
- Revision received September 4, 2009.
- Accepted January 27, 2010.
- © British Journal of General Practice, 2010.