Abstract
Background The aim of appraisal is to provide an opportunity for individuals to reflect on their work to facilitate learning and development. Appraisal for GPs has been a contractual requirement since 2004 in Scotland, and is seen as an integral part of revalidation.
Aim To investigate the outcomes of GP appraisal in terms of whether it has prompted change in medical practice, education and learning, career development, attitudes to health and probity, how GPs organise their work, and their perception of the overall value of the process.
Design of study A cross-sectional postal questionnaire.
Setting GP performers in Scotland who had undertaken appraisal.
Method The questionnaire was based on the seven principles outlined in Good Medical Practice, a literature review, and previous local research. The survey was conducted on a strictly anonymous basis with a random, representative sample of GPs.
Results Fifty-three per cent (671/1278) responded. Forty-seven per cent (308/661) thought that appraisal had altered their educational activity, 33% (217/660) reported undertaking further education or training as a result of appraisal, and 13% (89/660) felt that appraisal had influenced their career development. Opinion was evenly split on the overall value of appraisal.
Conclusion Appraisal can have a significant impact on all aspects of a GP's professional life, and those who value the process report continuing benefit in how they manage their education and professional development. However, many perceive limited or no benefit. The renewed emphasis on appraisal requires examination of these findings and discussion of how appraisal can become more relevant.
INTRODUCTION
The requirement to undergo an annual appraisal became a contractual obligation for all UK GPs following the implementation of the new GP contract in April 2004. The Scottish GP Appraisal Scheme was instituted in January 2003, details of which are published elsewhere.1 The aims of the process reflected the statement of the Chief Medical Officer in 1999 when he advocated the introduction of an appraisal process for doctors that was not primarily concerned with detecting underperformance but would help them consolidate and improve on good performance.2 While feedback about experience of the process is routinely gathered following the appraisal, this study aimed to move beyond this limited feedback by evaluating the process in terms of the perceived impact in a number of important areas of a doctor's professional life.
Previous research on GP appraisal has focused on its acceptability to the participants.3–6 Although largely positive, concerns exist about the degree of engagement with the process of peer appraisal, with some evidence of collusion and evasion.7 However, the development of a structured appraisal process in medicine has been examined and demonstrated to have significant benefit on the delivery of care in a hospital setting.8
Research has attempted to predict the impact of possible outcomes for GPs following appraisal.9 As the Scottish appraisal system was designed to be consistent with Good Medical Practice,10 this research went further and asked GPs what impact they felt appraisal had had on their medical practice, education and learning, career development, attitudes to health and probity, and how they organise their work. In addition, their opinions were sought on the overall value of the process.
The research was undertaken shortly after the publication of the review of professional regulation, Good Doctors, Safer Patients,11 undertaken by the Chief Medical Officer for England. The resulting white paper Trust, Assurance and Safety proposes fundamental changes to the current model of appraisal, and it is appropriate to examine what impact the existing model of appraisal is perceived to have had in these areas.12
METHOD
A random 25% sample, stratified by a health board, of all 4926 Scottish GPs who had undergone appraisal since 2004 was drawn from the Scottish appraisal database, using random number tables. A one in four sample was chosen as sufficient to detect important differences between age groups and regions, based on an assumed 50% response rate to the questionnaire. The complete populations in smaller health boards (Borders, Orkney, Shetland, and Western Isles) were sampled to increase the likelihood of a good response rate in these areas, giving a total sample of 1278.
The survey was conducted on a strictly anonymous basis. Each GP was sent a postal questionnaire in September 2006 and one reminder 3 weeks later. An online version of the questionnaire was posted on the Scottish appraisal website,13 as a means of generating additional responses. It was recognised that there was a possibility of online responses duplicating postal responses, but it was thought this was unlikely to have a significant effect.
To enable returns to be monitored for a subsequent reminder exercise, a reply-paid postcard containing an address label was included with the survey. The postcard was returned separately to confirm completion of the survey. This allowed identification of non-responders, and a reminder to be sent.
Creating the questionnaire
The questionnaire was based on the seven principles of good practice as outlined by the General Medical Council in Good Medical Practice,10 a literature review, and a previous internal NHS Education for Scotland report on significant issues raised in GP appraisal in Scotland.14 A pilot version of the questionnaire was tested with a convenience sample of 20 GPs and local appraisal advisers. This led to several iterations of the survey before a final version was created. A machine-readable survey form (Appendix 1) was created using TeleForm® software. Data were verified and validated by a dedicated e-forms team, before being exported into Microsoft® Excel and SPSS 15.0 for Windows for analysis.
How this fits in
Previous research in appraisal has focused on process issues and showed general satisfaction with the system. There has been research on the predicted impact of appraisal but none evaluating actual perceived outcomes. This study provides evidence that GPs perceive that appraisal has had a varying impact on important outcomes, based on the seven principles of Good Medical Practice.
RESULTS
Response rate
A total of 671/1278 (53%) responses were received, including 14 online responses. The response rate was broadly comparable to other such studies.15 A comparison of the distribution of responders by age, sex, and health board with statistics from the Scottish Information Services Division16 suggests these were representative of the Scottish GP population.
Profile of responders
The distribution of responders by practice type was representative of Scottish general practice. Most worked full-time (61%, 400/660), with 34% part-time, 2% job share, and 3% having another working arrangement. Responders worked on average 7.2 clinical sessions a week, which is comparable to a recent Scottish GP workload study.17
Number of appraisals
Ninety-eight per cent (658/671) had experienced at least one appraisal, 85% two appraisals, 30% three appraisals, and 4% had undertaken four.
Impact of the appraisal scheme on clinical practice
The survey was designed to evaluate the perceived impact of the appraisal process on the elements defined in Good Medical Practice.10 Responders were asked to indicate the degree of change they attributed to appraisal by choosing from five options: ‘not at all’; ‘in a small way’; ‘a fair amount’; ‘quite a lot’; and ‘a great deal’. The data were examined both at an individual, component question level, and by group in terms of the seven main headings outlined in Good Medical Practice.
Analysis of responses (Table 1) indicated that, for most items, the three ‘higher’ response options tended to be chosen relatively infrequently, responses being concentrated towards the lower end of the response scale (that is, ‘not at all’ and ‘in a small way’). It was therefore decided to aggregate responses to the three higher options (‘a fair amount’, ‘quite a lot’, and ‘a great deal’) into a single composite category. This offered the incidental advantage of making the data easier to understand when presented in tabular form.
Table 1 Change attributed at an individual question level.
Examining the data at individual question level identified the specific areas where appraisal has had most and least influence. The main areas of impact were: in undertaking significant event analysis, identifying knowledge gaps, and keeping professional knowledge up to date. There was less impact perceived in the areas of communication with colleagues and patients, team management, and managing personal health.
Figure 1 summarises the reported change perceived by GPs on each of the seven sections of Good Medical Practice as represented by an average across all questions in each respective section. It illustrates that while appraisal has had most impact in the areas of ‘maintaining good medical practice’ and ‘probity’, one-third felt that it had no perceived impact, and less than one-fifth felt it had a discernible impact in the areas of ‘working with colleagues’, ‘relationships with patients’, and ‘health’.
Figure 1 Comparison of distribution of percentage change resulting from appraisal averaged for all questions within each section of Good Medical Practice.10
Impact on GP education and learning
Just under half of GPs (46.6%, 308/661) thought that appraisal had altered the type of educational activity in which they had participated. The most commonly reported type of activity undertaken was e-learning (34.6%, 232/671), followed by practice-based learning (27.7%, 186/671), and protected learning time events (23.1%, 155/671).
The introduction of appraisal was compared with a previous incentive to promote learning and development, the postgraduate educational allowance (PGEA). Approximately one-third (34.5%, 225/653) felt that appraisal was more beneficial than PGEA, another third (31.4% (205/653) preferred PGEA, with the remainder being neutral or unsure (34.1%, 223/653).
A key component of the appraisal system is the creation of a personal development plan. There were mixed views on the role of personal development plans, with half of responders stating that they had influenced their learning (49.6%, 324/653), and half indicating they had not (50.4%, 329/653). Many of those influenced by personal development plans commented that they had provided them with focus, structure, and organisation.
The comments received describe a number of reasons for appraisal's impact on education and learning. A number of responders recorded that they were already engaged in many of the activities in the appraisal process prior to its introduction, and therefore had less scope to change. The introduction of the new contract is also likely to have been a significant factor in how GPs planned their development and the educational activities they undertook. Other responders bemoaned the lack of clarity surrounding the purpose of appraisal, and it is likely that this perception was heightened by the confusion surrounding the future of revalidation, and possibly reduced the level of engagement with appraisal. Some responders were suspicious of its true purpose, perceiving it as a summative process. This perception has persisted despite attempts to ensure clarity about the developmental ethos and purpose of appraisal.18
Impact on career development
Almost one-third of responders (32.9%, 217/660) reported that they had undertaken further education or training directly as a result of appraisal, including acquiring new skills and knowledge such as undertaking IT training or a diabetes course.
Thirteen per cent (89/660) felt that appraisal had influenced their career development. Several described how they had undertaken additional activities such as becoming appraisers and trainers, but some had rationalised their activities by limiting the number of professional tasks and roles they undertook.
Eleven responders commented that the demands of appraisal had led them to bring forward their retirement plans.
General impact of appraisal
Thirty-nine per cent of GPs (257/664) stated that appraisal had prompted the introduction of new practice policies, procedures, and guidelines. It had less impact in changing the ways GPs managed their workload (23.8%, 157/661) or managed patient consultations (15.2%, 101/663).
The appraisal process had similar impact in changing responders feelings on their practice as an organisation (22.4%, 148/662), and how they related to their colleagues (22.6%, 150/663). Those who felt appraisal to be beneficial in this area commented that positive outcomes included improved team working, better awareness, communication, cohesion, and organisation within the practice, and a greater appreciation and understanding of their peers, for example their learning needs.
Overall value of the appraisal system
Opinion was evenly divided as to the perceived value of the appraisal process (Table 2). There is evidence that the value GPs attach to or derive from subsequent appraisals declines, but GPs who found the first appraisal valuable generally continued to do so, and those who did not also persisted in their view.
Table 2 Perceived value of appraisal.
Spearman's rank correlation was used as an appropriate method to assess associations among ordered categorical variables.19 No correlation was found between GP status, age, sex, membership of the Royal College of General Practitioners, and perceived value of appraisal. In contrast, a positive association was found between perceived change in outcomes and value — those who rated appraisal as valuable or extremely valuable reported greater change in outcomes (Spearman's rank correlation ranged from 0.36 to 0.64, all P<0.001).
DISCUSSION
Summary of main findings
These findings raise important issues for those who plan and deliver GP appraisal. The results demonstrate that appraisal can have a significant impact on a number of areas that support the development of GPs and facilitate learning. However, for many responders, participation in appraisal had either limited or no perceived impact on their learning and development.
Appraisal is considered by a significant number of GPs to be of particular value with regard to its impact on how they maintain their clinical skills and knowledge, and how they structure their education and plan their development. This is a key aim of GP appraisal and it is reassuring that a large number of doctors are benefiting in this way. However, given the extensive literature on the apparent inaccuracy of strategic self-assessment,20 there must be some doubts about the effects of this on actual clinical practice. However, this caveat might be equally applied to the ability to accurately assess the positive influence of interventions.
In other areas such as organisational issues and career development, appraisal had less impact, which may simply reflect that fewer responders had concerns in these areas. This may also be true for issues of personal health and probity. The reported extent of the impact on managing health reflects the view that a small but significant minority of GPs have personal health issues that have an impact or could have an impact on patient care.21 As such, these results can be interpreted as appropriate.
Strengths and limitations of the study
This is the first attempt to explore the impact as perceived by participating doctors of any GP appraisal scheme, beyond simple satisfaction with the process.
There are some limitations to the research. The non-response rate raises the question of generalisability,15 and potentially responders may have been more or less enthusiastic than the whole sample about appraisal. The survey relied on self-reporting by responders which was not triangulated with other data. Similarly, there are no other data to compare results against as this is the first research of this type. In terms of evaluating the effectiveness of appraisal, the study did not set out to judge the outcomes. It is difficult to evaluate outcome measures for perceived impact, which necessarily are aspirational. It is more appropriate to regard this study as providing a baseline for future research.
In terms of attribution, GPs may ascribe a change in behaviour to appraisal, while it may have happened anyway, in the absence of appraisal. For example, significant event analysis was a core category of appraisal in Scotland prior to its inclusion in the Quality and Outcomes Framework (QOF), and its inclusion in the QOF may have contributed to the level of uptake attributed to appraisal in this study. Similarly, other reported changes such as the increased use of e-learning may simply reflect a general trend.
The Likert scale employed assumes no negative effect of appraisal on the participants. A negative to positive scale might have been more revealing. However, this was not considered necessary because of the largely positive literature about appraisal at the time of the study design.
Comparison with existing literature
Although there is comparatively little evidence of how education influences practice,22,23 it has been shown that targeted interventions and interventions that relate to clinical practice are most successful.24 The appraisal process employed a similar model, introducing a standard framework with explicit criteria relating to the seven areas of Good Medical Practice. However, this standardised approach may lack the necessary sophistication to address all individual learning styles and developmental needs, or to provide an appropriate level of challenge for all participants.
Implications for future research
The appraisal process has an undoubted cost in terms of time and resources. In Scotland it is recommended that the appraisal process should involve a session of preparatory time on behalf of both the appraiser and appraisee, with a further session devoted to completing the process. This represents the annual average workload of several GPs. The process also requires local and national administrative support. We owe it to patients and all other stakeholders to demonstrate that a process that impinges on time devoted to actual patient care is of value and can fulfil the aims of the appraisal process to promote the development of GPs. A first step to achieving this would be to open up a dialogue with all stakeholders to explore how the appraisal process could develop to have a greater impact on patient care, and how this might be achieved in a cost-effective manner. Similarly, this study did not set out to explore the pastoral aspect of appraisal, which is regarded by many as an implicit benefit of a developmental appraisal process. Significant potential exists to explore sensitive areas such as morale and the impact of complaints.
This research was undertaken prior to the publication of the white paper Trust, Assurance and Safety,12 which sets out a programme for reform of the regulation of health professionals. There is a clear indication that the appraisal process will require the inclusion of more objective evidence matching specified criteria with greater subsequent examination of the outcomes of appraisal. The results of this research indicate that appraisal can have an impact and benefit for GPs in all areas of their professional life. In some of the areas addressed, the reported limited impact may be entirely appropriate, and equally, other benefits may be unacknowledged, as exemplified by the Johari window model.25 However, the findings demonstrate that there is a clear requirement to attempt to ensure that appraisal becomes increasingly relevant for all GPs, that the links with professional regulation are clear, and that the resources it employs are used wisely. Further in-depth qualitative research has therefore been commissioned. It is hoped that this will answer many of the issues raised by this initial study to allow the appraisal process to build on the identified benefits.
