Verifying appraisal evidence using feedback from trained peers: views and experiences of Scottish GP appraisers

Completion of core appraisal tasks

Postal questionnaire survey

Participants and appraisal materials

A total of 80/164 GP appraisers agreed to participate (49%). Of the 80, 67 (84%) submitted appraisal materials for external peer review. All 67 participants returned completed questionnaires (100%).

Participants' reported apprehension and knowledge and skill levels before and after peer feedback

Table 2 outlines statistical differences in participants' reported levels of apprehension about submitting SEA (P = 0.52), audit (P = 0.14), and video (P = 0.04) for external peer feedback before and after the study. The differences in appraisers' estimated knowledge and skills levels before and after receiving feedback on SEA (P<0.01), audit (P = 0.06) and video consultations (P = 0.18) are also detailed in Table 2.

Significant event analysis

A majority of potential participants allocated to the SEA group (91.2%; 44/48) submitted completed reports (Table 3). There was unanimous agreement (n = 44) that submitting one report every 5 years for external peer review would be feasible for most GPs. Responders believed the peer feedback received was fair (n = 42); would facilitate improvement in their SEA practice (n = 32); and would be acceptable to most colleagues (n = 42). The majority agreed (n = 42) that external feedback on SEA would add value to the existing appraisal system.

Criterion audit

Fifteen of 18 expected criterion audit reports (83.3%) were received (Table 4). Most responders were unconcerned about their audit report being reviewed by external peers (n = 14) and believed this would also be acceptable to GP colleagues (n = 15). Feedback received provided specific advice on how to improve audit skills (n = 15) and would facilitate improvements in related practices for the majority (n = 12). A majority agreed (n = 14) that the existing appraisal system would be enhanced by external feedback on criterion audits.

Videotaped consultations

A total of 8/14 appraisers (57.1%) submitted videos of consultations for review (Table 5). Five responders believed completing a consultation video was feasible as part of normal general practice. Peer feedback on consulting skills was perceived by all participant appraisers to be fair and educationally useful and to have the potential to have an impact on the consulting skills of most GP colleagues. There was a unanimous perception that peer-reviewed video consultations would add value to the appraisal process.

Summary of main findings

This study found strong support from participants for the feasibility, acceptability, and educational gain associated with the external and independent review of three core appraisal activities by trained colleagues. Many expressed a certain level of apprehension about receiving external feedback on their appraisal activities, but this decreased slightly after participation, particularly for those submitting a video of consultations. Additionally, there was a general perception that review of core appraisal activities by trained colleagues would ‘add value’ to the current appraisal system in Scotland.

Peer feedback received on SEA and criterion audit reports helped responders to understand more about these techniques, and engendered a reported commitment to practise it more effectively. There was a perceived increase in estimated knowledge and skills levels associated with each activity after peer feedback had been received, particularly for SEA. However, a large minority experienced low-to-moderate education gains from participation. This could be explained by the fact that most GP appraisers should be experienced in SEA and audit. Related evidence and knowledge are also required for many educational and governance reasons,1 including training practice accreditation and to benefit from the Quality and Outcomes Framework.25 Additional knowledge will also be gleaned by appraisers when reflecting on SEA and audit reports with appraisees. It should also be acknowledged that a proportion of GPs will be highly effective in applying both techniques, and that peer feedback, while reinforcing excellent performance in these cases, may offer minimal educational gain. Given this background, there is a strong perception from appraisers that other GP colleagues would benefit from developmental feedback by trained peers on SEA and audit attempts, which reflects previous research evidence.17^–20^,22

A comparatively small number of appraisers submitted consultation videos for peer review. Although there was strong agreement that other GPs would benefit from this method of receiving feedback on their consultations skills, there was less agreement on whether this would be acceptable to most GPs. Indeed previous work has demonstrated that the process can be effectively undertaken by a GP without previous experience in this area,21 however, peer review of video consultation is often perceived to be much more challenging and stressful than participating in audit or SEA.26 The process can often take longer and be technically difficult to organise and execute, while patient consent is also required. Perhaps most importantly it is also perceived to be more professionally threatening than the other two activities.26 However, there was strong positive agreement on the potential value of external peer review of video consultations educationally and when linked to the GP appraisal system.

Strengths and limitations of the study

A large minority of all Scottish appraisers participated. However, the study population was self-selecting and potentially biased as the views and experiences of non-participating appraisers and other GP groups may differ from those reported. The study context is specific to NHS Scotland, but the underlying principle linking external peer review and appraisal is potentially relevant to other medical specialties and healthcare systems. Most appraisers who agreed to participate did so, but a substantial minority failed to submit video consultations, which points to a potential difficulty in linking this activity with the peer review model as part of normal general practice. Written feedback was provided by peer reviewers. However, in some situations, face-to-face feedback might be more desirable but this also creates a feasibility issue.

Comparison with existing literature

Given the increasing importance being accorded to the links between appraisal and revalidation,8^–10 more attention will inevitably be paid to the standard of evidence being submitted. Ultimately, someone has to make a judgement as to whether it is good enough. Appraisers have consistently voiced their discomfort at being given this task.6^,7 This external peer review system allows the judgements to be made prior to an appraisal with a built-in sampling process of ‘quality control’ ensuring fairness across the country, that is, a national standard. However, the feasibility of a national external peer review system to support appraisal is open to question. For such a system to function, additional resources will be required. One option is to consider the vision that all GPs are potential peer reviewers — with appropriate training — with local ‘experts’ taking on the role of quality assuring the process. If it is to be practicable, much wider discussion is needed. The status quo, however, may not be an option.

The literature on the limited value of self-assessment is clear.27 The benefit, therefore, of providing additional peer review should be not only desirable but — given the stakes — essential. The adaptation of the Hammond model of Cognitive Continuum Theory underpins this justification by encouraging a more rational (rather than merely intuitive) approach to decision making on material submitted for appraisal and inclusion in a revalidation folder.

A call for external verification of evidence will not be straightforward — GP appraisers are in the frontline and, in their opinion in this study, those who took part felt the advantages outweighed the disadvantages by suggesting it gave added value to the appraisal process. In their proposal to improve the appraisal process, the Royal College of General Practitioners suggests that over a 5-year period GPs should present evidence of a specific number of SEA and completed audit cycle reports for discussion during appraisals.28 This represents an increase in the quantity of evidence that is currently required. However, it is unclear how this overcomes the perceived inadequacy of the present approach, which is that the appraisal system lacks a robust method to verify the quality of evidence that is already known to be variable.17^–20^,22

Implications for future research

Future research on this model should focus on shedding light on the acceptability and feasibility of peer reviewing videos of consultations on a large-scale basis. More confirmatory work on strengthening the inter-rater reliability of video consultation peer review is also required. Overall, there is a need to explore with decision makers if and how independent peer review can make a meaningful contribution to improving the robustness of evidence required for appraisal in support of revalidation.