Developing checklists to prevent diagnostic error in Emergency Room settings

Introduction

Diagnostic errors represent a largely unaddressed concern in all healthcare settings, including the Emergency Room (ER) [1, 2]. According to current estimates, one in every 10 diagnoses is likely to be wrong and one in every 1000 ambulatory encounters engenders the risk of harm from a diagnostic error [3, 4]. In aggregate, diagnostic errors are estimated to account for 40,000–80,000 deaths annually in the US [5]. In the ER setting, studies of malpractice claims identify multiple breakdowns in the diagnostic process, and cognitive factors were involved in 96% of these instances [6].

Recent reviews have summarized both the system-related and cognitive interventions that have been proposed to reduce diagnostic error [7–11]. A wide range of well-justified interventions has been considered, including personnel-focused programs, educational interventions, structured process systems, second reviews, and technology-based solutions [9]. Relatively few of these have been evaluated rigorously in practice, although second review programs are now increasingly common in both pathology and radiology programs, and decision support tools to help with differential diagnosis [12, 13], along with online access to medical knowledge resources are increasingly being utilized in direct patient care.

Although the diagnostic process is clearly influenced by many factors inherent in healthcare systems, diagnosis is fundamentally a cognitive process, often based on intuition or gestalt [14]. Successful diagnosis depends on clinical reasoning to synthesize all of the available information and relevant knowledge to arrive at plausible diagnostic possibilities. With an ever-expanding number of diseases to be considered (8000–12,000 according to estimates), the variability in how the disease will manifest and be described and perceived, the substantial uncertainty attending every step of the process, and ultimately the inherent limitations of human memory and cognitive processing, diagnosis is a challenge. The typical ER, with the pressures of workload and time, repeated distractions, and constant multitasking, contributes to this challenge.

Checklists have been developed to address complex and error-prone processes, especially for situations where the outcome is too critical to rely on human memory [15]. Although checklist usage in medical settings is a relatively new endeavor [16], they have already produced impressive reductions in the infection rate from central line placement [17, 18], and in surgical complications [19, 20]. However, the process of diagnosis is not as well defined, and it is not known if checklists will be as effective in this application. The goal of this project was to develop and pilot test a checklist to assist with diagnosis and ultimately help reduce the risk of diagnostic error. To complement symptom-specific checklists that have been previously developed and trialed [21, 22], we sought to develop a more general checklist, and performed initial evaluations of both of these products in an ER setting.

Methods

Checklist development

Development of a symptom-specific set of checklists for primary care has been previously described [21, 22]. We focused on developing a usable “general” diagnostic process checklist:

• Version 1: To derive a novel, general checklist for diagnosis we used published checklist design principles and recommendations [23–25] and input from a content-matter expert advisory panel (see acknowledgements) to modify a draft prototype (Version 1, see Appendix) we had previously developed as a starting point [22, 26]. We conducted 45–60 min, structured individual telephone discussions with the seven subject matter experts to obtain input on the ideal components of a general checklist, feedback on Version 1, suggestions on the proposed process for refining and piloting the checklist, and key outcomes to consider. Interviews were transcribed and common themes and major suggestions were identified.

• Version 2a and 2b: Based on input from the subject matter experts and the participating ER physicians, two different general checklists were developed, designated 2a and 2b (see Appendix, Checklists 2a and 2b). Experts recommended we include a complete set of symptom-specific checklists along with the general checklist [22]. Both were then published as a spiral-bound 4×6 inch booklet. Physicians at the two ER sites used the booklet during two work shifts. We then performed cognitive testing to obtain feedback through interviews in person or by telephone. Based on this input and further consultation with the subject matter experts, a revised final “Version 3” checklist was developed.

• The Version 3 checklist, along with reformatted and updated versions of the symptom-specific checklists, (Figure 2A, B) were again published as a spiral-bound booklet sized to fit in a lab coat pocket. Over the next 2 months, the 15 participating physicians were encouraged to use the booklet containing Version 3 twice per shift, focusing on patients they believed had high risk symptoms or “don’t miss” diagnoses. Usage was specifically encouraged for patients with chest or abdominal pain. Physicians were also encouraged to review the appropriate symptom-specific checklist with the patient, and were shown a video demonstrating this approach (http://tiny.cc/ElyDemo or http://www.youtube.com/watch?v=uHpieuyP1w0).

Physicians completed a post-shift questionnaire (Supplemental Data, which accompanies the article at http://www.degruyter.com/view/j/dx.2014.1.issue-3/issue-files/dx.2014.1.issue-3.xml – Checklist Impact Evaluation) examining how and why they used the checklist and whether this had any impact on their clinical decisions.

An overview of the development and evaluation process is illustrated in Figure 1 and the final (Version 3) checklist and one example of a symptom-specific checklist are illustrated in Figure 2.

Figure 1

The process used to develop and evaluate a general checklist for diagnosis.

Figure 2

Checklist examples.

(A) The Version 3 General checklist. (B) The John Ely specific checklists for abdominal pain. (C) The John Ely specific checklist for chest pain.

Results

Discussion

The majority of diagnostic errors in medicine reflect cognitive errors [29], and thus the goal of this project was to develop a checklist that would help optimize the clinical reasoning process and help physicians recognize situations that predispose to these errors. To develop a general checklist, we began with some of the major shortcomings identified in the medical literature, then used an iterative process of evaluation and refinement based on input from subject matter experts and end-users. Rather than serving as a how-to guide, the resulting general checklist may cue physicians to pause and reflect in the process for formulating a diagnosis in the ER setting, and to recognize situations prone to cognitive biases.

We found that using the resulting general checklist along with a set of previously developed symptom-specific checklists was generally well accepted by practicing emergency room physicians. Using the checklists was reported to be helpful in confirming initial impressions, and in almost a third of instances the checklist helped the physician consider important diagnoses they had not previously considered. Using the checklists was identified as having prevented several important diagnostic errors, and there were no reports that usage induced new errors. The checklist also encouraged physicians to engage patients in ways that help prevent diagnostic errors or help catch these errors at an early stage, to avoid or minimize harm (“Make sure the patient knows when and how to get back to you if necessary”).

Although both the general and the specific checklists were judged to be valuable, users preferred using the specific checklists for most situations. Shimizu and colleagues have evaluated the same symptom-specific checklists that we used, along with a general debiasing checklist (our Version 1 checklist) in 188 Japanese medical students asked to evaluate five cases of increasing clinical complexity [30]. Using the general checklist did not increase the proportion of students choosing the correct diagnosis. In contrast, using the symptom-specific checklist was beneficial, at least for the more difficult cases. A similar result was found in a study using checklists to reduce nursing errors in programming IV infusion pumps: a highly specific, step-by-step checklist outperformed a general checklist that encouraged nurses to think critically and consider the ‘five rights’ of medication administration [31]. In our study, physician users were not receptive to a step-by-step approach that emphasized basic elements of the diagnostic process, viewing this as too simplistic. A general checklist that instead focused on high-risk situations therefore evolved, and the results confirmed that high-risk situations were one of the most common reasons to review the checklists. This acknowledges the reality that the synthesis phase of diagnosis, where diagnostic hypotheses are generated from integrating all the available information, is simply not amenable to the step-by-step approach that most checklists employ. Thus, the use of general and symptom specific checklists in the ER setting needs to be explored further, because they appear to be serving different purposes and each may play a role in unique situations. We also believe that the specific checklists used in this study, designed for patients seen in primary care, could be improved by customizing them for patients with urgent and emergency conditions, which often invokes a different differential diagnosis set. Finally, symptom-specific checklists in general are not ideal for patients presenting with multiple problems, or patients whose differential diagnosis is influenced by contextual factors. For example, the differential diagnosis for a healthy patient with weakness would be prioritized differently than the same differential for a patient on chemotherapy for metastatic disease.

A critical issue regarding interventions to improve diagnostic reliability is the extent to which reductions in errors and error-related harm are offset by unintended consequences. In the case of checklists for diagnosis, the goal of prompting consideration of a wider differential diagnosis could potentially lead to additional diagnostic testing and consultation, and thus additional costs in terms of time, money, and possibly complications of these investigations. Although our study was not powered to investigate this definitively, there did appear to be a tendency for providers to order additional diagnostic evaluations post-exposure, albeit the effect appeared to be small, and to some extent offset by cases where tests and consults were cancelled after checklist review. Other negatives were occasionally noted by users, including a sense in some cases that checklist usage ‘slowed me down’ or was not worth the time invested.

Although the success of checklists to improve surgical safety and to reduce central line infections seems adequately established, the impact of checklists in other medical settings has been mixed [16]. The possibility that checklists could improve diagnostic performance is suggested by the findings of Shimizu discussed above, and recent studies by Sibbald et al. who evaluated the impact of a simple checklist for interpreting electrocardiograms [32]. Even though this checklist included simply the standard steps of ECG interpretation, using the checklist improved accuracy in a group of expert interpreters (senior Cardiology fellows) without increasing the cognitive load or causing expert reversal. Similarly, use of a checklist improved the reliability of identifying key findings on the cardiovascular examination [33].

Our study had several limitations. It was not designed to evaluate whether checklist usage reduced the incidence of diagnostic error or error-related harm, the major outcomes of interest. We also make no assertion that the final general checklist that evolved from our process is the final tool or is superior to any others, and several other general checklists and ‘tips sheets’ for diagnosis exist. A key issue regarding checklist usage in ERs is whether it should be used in every case, or just when use is judged to be “appropriate”. We did not mandate checklist use in every patient, even though many diagnostic errors arise from situations where the diagnosis is established quickly and automatically using ‘System 1’ cognitive processing, and physicians seem generally to be unaware of which of their diagnoses are incorrect [34–36]. We would predict that usage in every case would decrease diagnostic errors and might also increase diagnostic testing, hypotheses that would need to evaluated in subsequent studies. Generalizing our findings is limited by the small size of the study sites (2) and physicians (15).

The ultimate success of checklists to improve clinical performance will reflect the complex interaction of many different socio-adaptive influences beyond simply ticking off boxes [37–39]. Improving communication amongst team members, education, providing feedback after usage, using clinical champions, endorsement by local leaders, and establishing a cultural expectation of improved safety are among the other elements relevant to whether such checklists can decrease diagnostic error and harm. Stakeholder buy-in is also critically important [24] and the checklists must be appropriately customized in relation to local needs.

In conclusion, the general checklist derived in this project has been extensively evaluated and refined, and in pilot studies was perceived as being helpful in the diagnostic process. Whether checklists can reduce diagnostic error will require further study. Given the complex nature of the diagnostic process, improving diagnostic reliability will no doubt require a set of multi-faceted interventions. Our pilot study suggests that checklists for diagnosis should be considered for further development, implementation and evaluation, perhaps with additional complementary strategies to reduce diagnostic error.