Central or local incident reporting? A comparative study in Dutch GP out-of-hours services

Design

The study was a non-equivalent controlled, quasi-experimental field study.15 Baseline incident-reporting data were collected at the end of 2006 relating to the period before the intervention (2005 and 2006). At the end of December 2006, the LIRP was implemented in OHS1. In OHS2 and OHS3, which served as controls, the current practice of reporting incidents to the central committee (CIRP) was continued. Follow-up data were obtained at the end of 2008 on all incidents reported in 2007 and 2008.

Setting

GPs' out-of-hours services in the Netherlands provide acute primary care in the evenings, nights, and weekends.10 Features of the participating OHSs are described in Table 1 (numbers from 2008).

Outcome measures

The outcome measures were the number of incident reports, type of incidents reported, and type of reporters, and an estimation of potential harm for the patient for each reported incident. Furthermore, qualitative data about any improvement measures initiated centrally or locally were obtained.

Intervention

A LIRP was implemented in OHS1. The LIRP was based on a ‘plan-do-act-check’ learning cycle,16 similar to the LIRP in the ‘SPIEGEL’ study.14 All caregivers at the OHS were asked to report all incidents. An incident was defined as any unintended or unexpected event that could have led or did lead to harm for one or more patients receiving care.8 On paper forms, which were put into an ‘incident mail box’ on the OHS, the reporters were asked to write a narrative about what happened and to indicate the date, time, and place of the incident, who was involved, and whether there was any harm to the patient. A local, multidisciplinary incident-reporting procedure committee was trained to screen and analyse the incident reports. Incidents were selected for analysis by first assigning a risk score (0 to 4), based on an estimate of potential harm and the frequency of occurrence. The committee was advised to analyse incidents with a risk score of 2 or higher, based on PRISMA (Prevention and Recovery Information System for Monitoring and Analysis)17 and root cause analysis18 techniques. They were also responsible for feedback to reporters and to the organisation, and for development of improvement measures when appropriate.

In OHS2 and 3, the usual procedure and routine around incident reporting, CIRP, was continued. In the CIRP, incident reports were sent by mail to an advisory committee of the board of directors of the GP OHSs collaboration.

In the LIRP, the incident reports were assessed by physicians, medical nurses, and a chauffeur, who actually worked on that particular OHS location, instead of professionals who did not work in the OHS in which the incident occurred. In addition, the incident reports were processed in a much shorter feedback loop: 2–3 weeks, instead of 8–12 weeks for the CIRP.

In the first year after the introduction of LIRP, the research team supplied quarterly anonymised feedback information from the incident reports to the LIRP committee and, when requested, advised concerning current incident analyses. At the end of 2007 it was decided by the OHS management to continue LIRP. From that time on, LIRP was executed without the support of the study team.

Data collection

All incident reports of the LIRP and CIRP were anonymised and gathered in a research database. The research team categorised the incidents, using a classification derived from the literature,19,20 adapted for practical use in the Dutch situation.14 Potential harm was independently classified using an ORCE (Observe, Record, Classify, Evaluate) procedure,21 by two members of the research team. Differences were solved by discussion.

Data about the implementation of improvement measures were collected at the end of the research period by studying the year reports of each OHS and of the CIRP and LIRP committees. The location managers were asked to check and confirm the implementation of the proposed improvement measures. Furthermore, one researcher conducted open interviews with the location managers and members of the CIRP and the LIRP committees about their opinions on central and local incident reporting procedures, and concerning implementation barriers of the improvement measures.

Data analysis

Quantitative data were analysed with SPSS (version 15). For comparing the ‘before’ and ‘after’ data, χ² or Fisher exact tests were used as appropriate. The data on the implementation of improvement were qualitatively assessed in the study team, by constant comparison.22

Improvement measures

Qualitative analysis revealed that before the intervention the improvement measures were characterised by generically formulated recommendations about the incident, such as ‘GP should pay more attention to …’. Furthermore, after the CIRP had assessed the incident report, it was regularly advised that improvement measures should be developed locally. Often such measures had already been initiated on an ad hoc basis by the local management, when receiving this recommendation from the CIRP.

After the intervention, the improvement measures taken in OHS1 in 2007 and 2008 were formulated in more specific language and could be implemented much more easily than before the implementation of the LIRP. The managers thought that the fact that the measures originated from an incident from their own OHS location considerably facilitated implementation. Examples of these measures were a protocol for informing and preparing GPs who were new in OHS1, and improving the strategy of quickly administrating patient data on the telephone.

Interestingly, the intervention in OHS1 had a general spin-off. It enabled the CIRP to stimulate the local-improvement initiatives not only in OHS1 but also in the other OHS locations of the collaboration. Moreover, the CIRP started focusing on trends and recurring problems instead of on incident reports only. For instance, a prospective risk analysis23 of the process of assigning home visits by the CIRP was triggered by several incident reports from different OHSs. In addition, a root cause analysis was performed in response to six incidents with tissue glue for small cut wounds, reported in the last 2 years from four OHSs. Both inquiries were started in 2007, and the resulting recommendations were implemented in all OHS locations, including the intervention OHS, 1 year later.

When comparing the local-improvement measures and the improvement measures initiated by the CIRP after the intervention, the local measures were implemented in a much shorter time frame.

Summary of main findings

The results from the present study indicate that a LIRP in a GPs' OHS with a fast track from incident report to improvement measures, yields many more incident reports than the CIRP in which reports are assessed in a longer cycle and outside the actual ‘working environment’.

In addition, locally initiated improvements seemed to be more practical and implemented more quickly.

Limitations of the study

The study may have some limitations. The participating OHS locations were not randomly but purposefully selected. The intervention was implemented in OHS1 because its location management had a positive attitude to incident reporting, which at that time was still a sensitive issue. It should be noted, however, that management of OHS1 was not involved in patient care during the out-of-hours operation. In addition, the actual reporters did not have any influence on the selection of the OHS for intervention. Furthermore, the control OHSs were comparable with the intervention OHS in geographical and socioeconomic status of the population, and also in organisational and patient features. Finally, the three OHSs had a comparable willingness to report incidents in the period before the introduction of LIRP. Therefore, it is unlikely that the selection of the intervention site seriously influenced the study findings.

Furthermore, because the number of incident reports in the control OHSs was small, the comparison of distributions concerning the type of incident reports, type of reporters, and potential harm should be cautiously interpreted.

Comparison with existing literature

To the authors' knowledge, this study is the first to experimentally compare two different procedures for reporting and analysing incidents in general practice. Other studies concerning the number and/or nature of incident reports in general practice have been mostly descriptive.5,7,20,24–30

The large increase in the number of incident reports in the first year after the introduction of the LIRP was followed by a smaller number of incident reports in the second year of follow-up. As the support of the study team was withdrawn in the second year of follow-up, this tailing off suggests that the team itself contributed to the effect of the intervention. However, the input of the research team was limited, and the increase could also be considered as a starting effect, as observed in many other implementation studies.31 Even so, in the second year of implementation, the number of reports in OHS1 still remained seven times higher compared to baseline.

In contrast to earlier publications, the present study concerns the reporting process up to and including the design and implementation of improvements in a GP setting. Closing the reporting cycle up to feedback and visible improvement actions is recognised as an important feature of effective incident reporting.32 Several publications have indicated that regular, timely, and meaningful feedback is important in an incident-reporting procedure.33–35 The increase of incident reports in 2007 and 2008 in the present study may be explained by these attributes of the LIRP. The fact that practical improvement measures were implemented more quickly in the LIRP than in the CIRP may also have contributed to this increase. Furthermore, the authors believe that the LIRP enabled the professionals to control the assessment of their incident reports, as the reports remained within their own OHS. This, together with the emphasis on systematic analysis and organisational learning,36 may have convinced caregivers to increase their level of incident reporting compared to the former CIRP.

Implementation of a LIRP is associated with extra costs for administration and analysis. Obviously, the benefits of the resulting improvement measures should outweigh these costs. When interviewed, management and staff indicated that the LIRP was feasible and improved patient safety (data not shown).

Implications for clinical practice and future research

This study suggests that the willingness to report incidents in a GP out-of-hours setting increases with a LIRP compared to a CIRP. It may also result in faster implementation of improvements. A central overview, however, would collate information from various LIRPs to identify trends, which would extend the opportunities for analysis and learning. This implies that the best way of dealing with incidents in order to learn from them, is to develop a system in which both central and local incident-reporting procedures are combined.