An approach to evaluating the accuracy of DXplain

https://doi.org/10.1016/0169-2607(91)90004-DGet rights and content

Abstract

DXplain is a computer-based decision support system which generates a differential diagnosis (ddx) from a given list of clinical manifestations (Barnett et al., J. Am. Med. Assoc. 258 (1987) 67–74). An approach was developed to evaluate the accuracy of the ddx's produced by DXplain. The first step involves the collection of 65 benchmark cases drawn from a variety of sources and authors. Despite their diverse origins, the cases share in common that they are all clinical cases upon which a consulting physician might be asked to produce a differential. This helps to ensure that the evaluation of the system will be done in an environment similar to that in which the system is actually used. In the second step, all cases are reviewed by five board-certified physicians (experts) as well as DXplain. For each case, the evaluators (experts and DXplain) produce a rank-ordered ddx list along with an indication of how strongly each disease was felt to be supported by the case findings. A scoring technique was devised which rewards concordance with rthe gold standard: a consensus of the evaluators' ddx lists. Each evaluator receives a score which is proportional to the degree of agreement achieved with the consensus on the dds submitted. Preliminary results on a trial evaluation of 46 cases indicate that DXplain, on average, did well in agreeing with the consensus. Agreement was achieved both in regard to the specific diagnoses listed in the ddx and the degree to which the diseases were felt to be supported by the case findings. A discussion of some important issues in the evaluation of knowledge-based systems is undertaken.

References (10)

  • G.O. Barnett et al.

    DXplain — An Evolving Diagnostic Decision-Support System

    J. Am. Med. Assoc.

    (1987)
  • J.A. Hupp et al.

    DXplain- a computer based diagnostic knowledge base

  • G.O. Barnett et al.

    DXplain: Experience with Knowledge Acquisition and Program Evaluation

  • S.J. Nelson et al.

    Evaluating RECONSIDER. A computer program for diagnostic prompting

    J. Med. Syst.

    (1985)
  • J.A. Reggia et al.

    Computer-aided assessment of transient ischemic attacks. A clinical evaluation

    Arch. Neurol.

    (1984)
There are more references available in the full text version of this article.

Cited by (32)

View all citing articles on Scopus
View full text