Significant event analysis (SEA) is a qualitative method of clinical audit that has been proposed as a tool for reflective learning, managing healthcare risk, and enhancing patient safety.^1–3 In recognition of the requirements of appraisal, revalidation and clinical governance in the UK, undertaking an event analysis is now considered a core activity for all general medical practitioners (GPs).^4–6

Peer assessment of one element of an individual GP’s work may provide a critical evaluation of performance in this area by professional colleagues.^7–9 While the use of a reliable and structured instrument for this purpose is considered necessary, validity problems with existing instruments have been highlighted.¹⁰ There is, however, growing acceptance of the need for verifiable evidence of performance, especially with regard to appraisal and revalidation.⁶ Despite reservations about submitting one’s work for external evaluation,¹¹ professional judgement by a peer is increasingly seen as a valid method of making informed judgements on performance.^7,8,10

A voluntary educational model for the peer assessment of SEA reports has been available to all GPs in the west region of NHS Education for Scotland (NES) since 1998 and has previously been described.^12,13 A submitted report is sent to two GP assessors who independently review it using an appropriate instrument and provide formative feedback on improvement, if required.¹⁴ The model exists as a means of promoting the SEA technique and acting as a proxy indicator for determining if an event analysis is satisfactory or not. One session of postgraduate educational allowance (PGEA) was awarded for each submission (box 1).

The willingness of GPs to openly engage in the discussion and analyses of significant events either internally or externally is largely unknown. The dynamics in some practices may militate against this as it may be difficult to even raise sensitive issues such as significant events, which would clearly prevent further investigation. The reasons why GPs would be prepared to subject their own event analyses to external peer assessment are also unclear. Given the potential rewards for patient safety and health care, these issues merit more in-depth investigation.

This study therefore set out to explore the influences and perceived benefits behind the willingness of a select group of GPs to participate in SEA and educational peer assessment as part of their continuing professional development.

METHODS

Study sample

One hundred and five GPs from Greater Glasgow primary care trust had participated in our peer assessment model for SEA. We limited our study to the 84 GPs who had two or more experiences of the model and sent an initial letter to them to gauge interest in attending group interviews. Thirty four respondents were willing to participate and a further letter was sent inviting them to attend sessions on two specified dates in March 2003. A convenience sample was decided upon because of the limitations of purposively selecting caused by relatively small numbers and the pre-arranged interview dates.

Focus group interviews

Two group interviews were held in a neutral and accessible venue. One researcher acted as moderator (PB) in both sessions and assisted the discussion with the aid of a brief topic guide (box 3). Participants were encouraged to speak freely about their experiences of all aspects of SEA and peer assessment. The discussions were lively and we were satisfied that every participant was able to make an adequate contribution. Each session lasted 70 minutes and was audiotaped and transcribed with permission. Contemporaneous field notes were also taken.

Analysis of transcripts

The transcripts were analysed for content. Each transcript was examined independently by PB and JM and data were systematically coded, categorised and initial themes identified. Categories and themes were modified and reduced by merging and linking them after joint discussion between both researchers. Our intention was to present viewpoints rather than to quantify and report numerical data. The validity of the findings was further enhanced through review of the transcripts and cross checking of the identified categories and themes by ED.

RESULTS

Of the 34 GPs willing to participate, 21 were able to attend the two group sessions on the dates specified, with 10 attending the first session and 11 attending the second. Twelve participants were female, six were current GP trainers, and 12 were principals in training practices, giving a satisfactory mix in terms of the sex, professional, and academic status of those who participated in the SEA peer assessment model.

Six principal themes were identified:

• External pressures

• Educational incentives

• Practice benefits

• Emotional aspects

• Peer assessment issues

• Selectivity of SEA submissions

External pressures

Appraisal, training practice accreditation, and gaining the RCGP Practice Accreditation Award were the motivators for most GPs to undertake SEA. They agreed that external verification led to the formalisation of the SEA process. Analytical consideration of significant events has always happened but was previously more informal. Formally documenting these discussions was viewed as taking this a stage further, leading to more in-depth analysis being achieved than discussion alone. This was perceived as more likely to lead to change or learning being demonstrated than if a report was not written up.

“I think people would do it [SEA] but, if it wasn’t for PGEA or accreditation or a training practice visit, very few people would actually go to the trouble of actually writing it down.” (Focus group 1)

Educational incentives

Disagreement was evident over whether one PGEA session was sufficient reward for participating in peer assessment. Some found event analysis straightforward, describing how flexible it was because it can be done at a time and place of their choosing. In their experiences, events were analysed and resolved quickly and without difficulty. Others argued that the degree of difficulty and time taken was dependent on the complexity of the event. They had put greater effort into discussing and resolving the event, implementing change and writing up the analysis. In contrast, they believed the reward of a single PGEA session was insufficient.

The act of writing up an event analysis was viewed as educational because it involved demonstrating learning and insight. Participants agreed SEA facilitates quick team based learning because the process has immediacy about it. It was one way of sharing significant event details and alerting the team to what has gone wrong. Team involvement stimulated interest levels in event analyses, providing an opportunity to demonstrate immediate learning by highlighting best practice.

“It teaches you to involve the team appropriately but doesn’t involve blame but does involve people acting on something positively.” (Focus group 2)

Practice benefits

SEA is perceived positively as a practical, flexible, and team based way to resolve or minimise system failures, which it was agreed appeared to work successfully. There was strong feeling for the idea that taking a multidisciplinary approach to SEA led to improved teamwork in the practice and an increase in individual personal support to those directly involved or affected, especially when something had gone wrong. “Quite a lot of us have support teams so it is helpful for us to write it [the event] down and discuss it at the team discussion because it is quite often multiple factors that contribute to significant events.” (Focus group 2)

Linking a practice complaint with a subsequent event analysis was advocated by some to keep patients informed that something constructive was being done. They believed this provided more credible evidence that the complaint was being taken seriously.

Emotional aspects

GPs described tensions between undertaking certain event analyses as an individual and then sharing it with the practice team. Some were wary of informing others of these events because they themselves found them to be too personally upsetting. They described strong emotions in terms of feelings of vulnerability, guilt or blame for what happened. They were reticent about exposing their professionalism to immediate colleagues and other team members. It was clear from other participants that they too could empathise with this emotional position.

“I think there are some significant events that I, and I’m pretty sure most of us, would be wary of bringing to a group and probably select what we bring on the basis that you don’t want it to be a very negative experience or to lay ourselves open to criticism.” (Focus group 2)

Many participants described a cathartic element in undertaking an event analysis and, more especially, in writing this up. There was a feeling that the act of writing down what happened and why led to a sense of “soul purging” and “closing the door” on the event which may not have happened if it had not been documented.

Peer assessment issues

Gaining a PGEA session was the main reason for participating in peer assessment. However, many participants also admitted this was a secondary issue because the reports were already completed for accreditation purposes, so it seemed sensible to submit these for peer assessment.

A number of GPs described a need to be reassured about their knowledge of SEA as their prime reason for participating in peer assessment. This was also a secondary reason for many of those who coveted the educational reward and who were curious to know how their peers assessed their reports. Participants discussed the added value and validity accorded to their SEA reports, as well as feelings of personal satisfaction when informed their peers had judged them to be satisfactory.

“It is at least useful to know that some of your peers have looked at what you have done and thought that’s a reasonable way of approaching the particular significant event and done in a way which is proper.” (Focus group 1)

“I suppose that what I was saying is that it gives some validation to the fact that somebody else says that it has been done properly.” (Focus group 1)

There were mixed feelings about the educational feedback provided by peers. Some participants were disappointed at the advice provided for improving unsatisfactory event analyses. They felt the feedback was being critical of their event analysis rather than informing the process. Others would have preferred more in-depth feedback even for analyses considered satisfactory.

There was confusion about the feedback in terms of the requirement for demonstrating or considering change as part of both an event analysis and what is required for peer assessment. These GPs acknowledged this was linked to a lack of knowledge about the model and that they were not fully aware of how it worked, even though they may have submitted potentially sensitive events.

Selectivity of SEA submissions

Certain significant events would never be submitted for peer assessment or even written up, but dealt with verbally “in-house” because of the sensitivities involved. Interestingly, the participants in one focus group were initially adamant they would have no concerns about submitting any type of event analysis for peer assessment. However, as this was explored, it became apparent that controversial event analyses would indeed be “off limits” for most GPs. The other GP group was immediately clear that they considered peer assessment submissions to be highly selective.

“It is quite artificial because, if you made a total buffoon of yourself—let’s say, then you probably wouldn’t send it [SEA] into the department, you would select the sort of things where there is a tweak required that would not be appropriate for public scrutiny.” (Focus group 1)

Participants feared making the “department” (NHS Education) aware of certain types of significant event. Some GPs from training practices would rather the department did not know about events concerning partnership issues as they suspected that this may influence their future re-accreditation. Others considered events that could be construed as serious errors or which involved personal practice issues to be too professionally embarrassing to submit for peer assessment.

Strong concern was expressed by some GPs that the department could not be trusted to keep serious event analyses confidential. They feared professional or even public exposure, possibly leading to further investigation and potential litigation. Other participants did not object to these views, which may indicate that they had similar fears. Most GPs also agreed significant events of a highly personal nature should remain private, but chose not to elaborate on any examples.

“I suspect that some of the worst disasters are not the ones they choose to write up and submit because they are too embarrassing or whatever.” (Focus group 1)

GPs are aware that the analyses of positive significant events (those that highlight good practice) are promoted, but generally agreed that the benefits associated with addressing them were questionable. Most agreed that it was probably beneficial to analyse these events but did not do so. They believed that “problem” events should always be prioritised and that practice workloads ensured these took precedence over positive events.

“Bad ones—that is where the challenge lies. I think it has to do with this learning, you learn from the bad ones so that when you get the good ones it just makes you feel good.” (Focus group 2)

DISCUSSION

For this group of GPs, external pressures and verification influenced both their decision to analyse significant events and the depth to which they were analysed. SEA was not an onerous activity for some, but others believed this depended on the complexity of the event. Linking a complaint investigation with an event analysis was perceived by some as one way of informing patients that complaints were treated seriously. Writing up an event analysis was viewed as an educational and emotional process that could act as a form of personal catharsis. Event analyses were submitted for peer assessment for the educational reward, but were highly selective because of concerns about litigation or personal embarrassment. Feedback from peers was believed to enhance knowledge and the validity of the event analysis, which provides an element of reassurance.

Strength and limitations of the study

The focus group environment allowed participants to be relaxed and confident among their peers. This helped in achieving the high level of agreement in the themes independently generated by both researchers, which was enhanced by external verification by an experienced qualitative researcher.

The study was limited by the inability to purposively sample caused by the low availability of participants. Two large focus group sessions were held, which may have affected the range and quality of contributions. The delay in typing transcripts militated against getting participants to verify data accuracy. The results should be viewed in the context of the select study group involved and so interpreted with caution. Importantly, the peer assessment instrument used in this educational model may be limited in terms of its validity and reliability. A further weakness is that the model is centred on GPs and PGEA, despite frequent multidisciplinary input into SEA reports. We decided, however, that it was more relevant to explore peer assessment with this group rather than others.

Findings in relation to existing research

Pressure from an accreditation body has been promoted as one way of focusing attention on the effectiveness of audit.^15,16 Our finding that the external verification process ensures a more structured approach to SEA appears to support this. Bradley has questioned the reliability of informal discussion of significant events in terms of facilitating learning and change.¹⁷ Recent research has also shown that a large minority of GPs failed to demonstrate the knowledge to undertake an event analysis, which indicates a possible educational concern.¹⁸

General medical practice is characterised by uncertainty and complexity.¹⁹ Our finding of differences in the perceived difficulties in resolving significant events reflects similar work on undertaking event analyses.²⁰ The varying ability of GPs to undertake event analyses satisfactorily and implement change when necessary has also been exposed by peer assessment.¹²

Discussing positive significant events is viewed as important in building self-confidence and self-esteem.²¹ Our findings suggest that, although study participants are aware of such events, they do not believe there is value in analysing them. This contradicts the perceived wisdom about what constitutes a significant event and the learning opportunities inherent in sharing good practice.^1,22

The finding that some GPs saw SEA as assisting in the investigation of complaints supports work published previously.¹³ The technique is also perceived as a means of facilitating immediate team learning and needs assessment, which have been confirmed or proposed in previous studies.^2,23–25 An unexpected finding was that analysing and writing up an event may act as a form of personal catharsis.

Despite reassurances about confidentiality, some GPs would be highly selective in the events submitted for peer assessment. Participants were candid about this and cited it as a limitation of our model. Similar GP concerns about the requirement to notify certain significant events as part of a national reporting system have been described.²⁶

Implications of the study for clinicians or policy makers

The ability of a GP registrar to undertake audit is summatively assessed by informed peers in the UK as preparation for independent practice.²⁷ Valid and reliable peer assessment, based on educational principles, is one possible method of verifying the ability of GPs to perform SEA satisfactorily. This may provide an element of assurance that policy makers and primary care organisations may wish to consider. If adopted, GPs may have no option but to submit aspects of their work for external verification as one way of proving their fitness to practise medicine. The GP appraisal system may be an insufficient verifier in this respect, as it is unclear if appraisers have the knowledge to determine if an event analysis is adequate or needs improvement.

Given the concerns and highly selective submissions of GPs in our confidential, voluntary and educational model, there may be similar implications for the mandatory requirement to report certain significant events to PCOs and the associated work of the National Patient Safety Agency.²⁸

Possible future research

The role of SEA as a tool for enhancing patient care and safety is in its infancy. A whole range of research is required to determine what contribution the technique can make to improving health care. The role of peer assessment as a proxy for judging professional performance is an attractive proposition, but also requires in-depth study. Apparent interest in helping the UK General Medical Council to define “verifiable evidence” for clinical audit and revalidation may be informed by some of this work, but this also merits much wider research.⁶ Potential future research areas are shown in box 4.

CONCLUSION

Our small exploratory study has reported mainly positive feedback from a select group of GPs on the merits of SEA and associated peer assessment. Notwithstanding its limitations, our results show that the SEA peer assessment model described is valued. It may also have the potential to inform the understanding and application of the technique in a select and motivated group of GPs, with possible consequences for the enhancement of patient care and safety. Primary care, appraisal, and regulatory bodies may find this message of interest in their quest to be assured that SEA activity undertaken by all GPs is verifiably effective.