Abstract
Background Diagnostic echocardiography has poor access for patients with suspected heart failure. Pre-echocardiography screening with electrocardiograms (ECGs) is recommended as a means of targeting this scarce resource. There are data to support this policy when ECGs are interpreted by cardiologists but not by GPs.
Aim To assess the value of GP-reported ECGs as a pre-echocardiography screening test for left ventricular systolic dysfunction (LVSD).
Design of study Cross-sectional study of GPs' ECG reporting skills.
Setting General practice, NHS in Scotland.
Method A randomly selected, stratified sample of 123 Scottish GPs reviewed 180 ECGs (100 abnormal, 50 normal and 30 duplicate) from 150 patients with suspected heart failure. Forty-one patients had LVSD on echocardiography. GPs were required to categorise ECGs as normal or abnormal.
Results Mean sensitivity was 0.94 (95% CI = 0.92 to 0.95). Mean specificity 0.58 (95% CI = 0.56 to 0.60). Mean positive predictive value (PPV) was 0.47 (95% CI = 0.46 to 0.48). Mean negative predictive value (NPV) was 0.96 (95% CI = 0.95 to 0.97). Mean likelihood ratio was 2.39 (95% CI = 2.28 to 2.50). Seventy of 123 (57%) GPs achieved sensitivity of 0.9 and specificity of 0.5 for the detection of LVSD.
Conclusion Most Scottish GPs have the skills to perform pre-echocardiography screening ECGs in patients with suspected LVSD. However, differences in ECG reporting performance between individual GPs will result in widely varying referral rates for echocardiography and differences in the detection rate of LVSD. The implications of these findings need to be considered when heart failure diagnostic services are being developed.
INTRODUCTION
Heart failure, caused by left ventricular systolic dysfunction (LVSD), is a major health problem with a prevalence of between 0.4 and 3.2%.1 Accurate diagnosis using echocardiography is a prerequisite to safe cost-effective treatment.2,3 However, the number of patients consulting GPs with symptoms suggestive of LVSD greatly exceeds the capacity of existing echocardiography services. A screening test is required to identify the subgroup of patients in whom echocardiography is cost-effective.
In 2003, the National Institute for Health and Clinical Excellence (NICE) recommended that the electrocardiogram (ECG) and B-type natriuretic peptide tests should be used as pre-echocardiography screening tests for LVSD.3 NICE guidance did not indicate whether B-type natriuretic peptide testing alone, ECG alone, or a combination of both should be performed. A recent systematic review and meta-analysis concluded that the accuracy of both tests is comparable and that either can be used; there is no evidence to justify the use of both together.1
Patients with an abnormal recording should be referred for echocardiography. In practice the ECG would be interpreted by the GP during the initial assessment of patients with symptoms suggestive of heart failure. However, there is currently no evidence that the pre-echocardiography ECG is a reliable screening test for LVSD when reported by GPs.
How this fits in
Poor access to diagnostic echocardiography is the major impediment to cost-effective treatment of patients with left ventricular systolic dysfunction (LVSD). Pre-echocardiography electrocardiogram (ECG) screening is useful in targeting this scarce resource when performed by cardiologists. Most GPs have the skills needed to perform pre-echocardiography screening using ECGs. However, differences in ECG-reporting performance between individual GPs would result in varying referral rates for echocardiography and differences in the detection rate of LVSD. These findings have implications for the development and implementation of diagnostic heart failure services.
The aim of this study was to assess the value of GP-interpreted ECGs as a pre-echocardiography screening test in a large randomly selected group of GPs from practices throughout Scotland.
METHOD
Population studied
A random sample of 500 GPs was identified from the current list of Scottish GP principals (3784 GPs in total) which is held by the Information and Statistics Division (ISD Scotland). Members of the project team were removed from the list. Approval to contact individual GPs was obtained from health boards.
The study was performed between November 2004 and July 2005. Each GP was contacted by letter, and asked to complete a previously piloted questionnaire to obtain information about cardiology experience, whether they had obtained the MRCP number of ECGs read per month, and confidence in reporting ECGs. This information was used to stratify the study population into three subgroups with low (group 1), medium (group 2) and high (group 3) levels of ECG interpreting experience and confidence. Of the responders, 32% (132/418) had low scores, 46% (193/418) medium, and 22% (93/418) had high scores.
Researchers aimed to recruit 140 GPs with a representative range of confidence and experience in ECG interpretation. Invitations were issued in batches until 44 GPs with low scores, 65 with medium, and 31 with high experience and confidence scores had agreed to participate or until all GPs in the subgroup had been invited.
Study procedure
Each GP who agreed to participate was asked to review 180 ECGs from patients with suspected heart failure, 30 of which were duplicate ECGs. These patients had been referred to the community-based open access echocardiogram service from September 2002 to August 2003.4 Fifty of the ECGs had previously been reported by a cardiologist as normal, 50 had a minor abnormality, and 50 a major abnormality. Of the 150 ECGs, 41 corresponded to LVSD on echocardiography. Forty LVSD cases showed major ECG abnormalities and one had a minor abnormality. Major abnormalities were defined as previous myocardial infarction, left ventricular hypertrophy, left bundle branch block, atrial fibrillation, and left axis deviation.5 Minor abnormalities included sinus brady/tachycardia, first-degree heart block, right bundle branch block, right axis deviation, poor R-wave progression, atrial ectopics, ventricular ectopics, and non-specific ST-T wave changes. Thirty duplicate ECGs were given to GPs to assess consistency of reporting (10 normal, 10 with a minor abnormality, and 10 with a major abnormality). The order of the ECGs was mixed before being given to GPs.
GPs were informed that all ECGs were taken from patients with symptoms suggestive of LVSD. They were asked to indicate on a multiple-choice response sheet whether, in their judgment, the ECG was normal or abnormal, without consulting colleagues or textbooks. They were not asked to interpret the ECGs to achieve a particular level of sensitivity or specificity for the detection of LVSD. A fee of £120 was paid to each GP who returned a completed multiple choice response sheet. Data were double entered into an Excel spread sheet by two data entry staff working independently. Inconsistencies in data entry were corrected by referring to the original data. Procedures and materials were piloted in a small study of six GPs who were excluded from the final project.6
Statistical methods
Performance measures in detecting LVSD were sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio. Results were presented as means and 95% confidence intervals (95% CI). κ statistics were calculated for consistency in reporting the same ECGs on a second occasion. One-way analysis of variance (ANOVA) was used to compare mean performance levels between groups of GPs defined by experience and confidence. Statistical analysis was performed using SPSS (version 10.1).
RESULTS
Response rate
There was an 84% (418/500) response rate to the initial confidence and experience questionnaire. Of those who were invited to interpret ECGs, 123 GPs interpreted ECGs and returned completed response sheets (group 1, low: n = 35; group 2, medium: n = 59; and group 3, high: n = 29; Figure 1). These proportions reflect the spectrum of experience and confidence in the study GP population.
Sensitivity and specificity for LVSD
Mean sensitivity of GP-reported ECGs as a pre-echocardiography screening test for LVSD was 0.94 (95% CI = 0.92 to 0.95). ECG screening had a lower sensitivity when performed by less experienced and less confident GPs: group 1 mean sensitivity 0.90 (95% CI = 0.87 to 0.93); group 2 mean sensitivity 0.95 (95% CI = 0.94 to 0.97); group 3 mean sensitivity 0.95 (95% CI = 0.92 to 0.97) ANOVA F = 6.41, P = 0.002 (Figure 2).
Mean specificity of GP-reported ECGs as a pre-echocardiography screening test for LVSD was 0.58 (95% CI = 0.56 to 0.60). There was no significant difference in specificity for LVSD between the groups with different levels of experience/confidence, ANOVA F = 1.62, P = 0.20 (Figure 2).
Predictive values for LVSD
The prevalence of LVSD was 27.3% (41/150) in the population of patients whose ECGs were interpreted in this study. Mean PPV of GP-reported ECGs as a pre-echocardiography screening test for LVSD was 0.47 (95% CI = 0.46 to 0.48). Mean NPV of GP-reported ECGs as a pre-echocardiography screening test for LVSD was 0.96 (95% CI = 0.95 to 0.97). In a population with LVSD prevalence of 10%, the PPV would be 0.21 (95% CI = 0.20 to 0.22) and the NPV would be 0.99 (95% CI = 0.987 to 0.991). With a LVSD prevalence of 20%PPV would be 0.37 (95%CI = 0.36 to 0.38) and NPV would be 0.92 (95% CI = 0.90 to 0.93). The mean likelihood ratio (sensitivity/[1-specificity]) was 2.39 (95% CI = 2.28 to 2.50).
Mean κ statistic for intra-rater reliability was 0.69 (95% CI = 0.65 to 0.72) demonstrating good consistency in ECG reporting by individual GPs. There was no significant difference in κ statistic between experience/confidence groups (ANOVA F = 0.43, P = 0.65).
Assessment of current performance
The outcome of screening for LVSD by individual GPs is shown in Figure 3. Differences in ECG reporting performance between GPs would result in variation in referral rates for echocardiography and differences in the detection rate of LVSD. The current data indicate that 70/123 (57%) GPs achieved a sensitivity of 0.9 and specificity of 0.5 for the detection of LVSD.
Experience of and confidence with ECG interpreting, as assessed by the initial questionnaire, was a poor predictor of screening performance in individual GPs. Members of all three groups failed to achieve a sensitivity for LVSD > or ≥0.9 or specificity for LVSD > or ≥0.5. The proportions not achieving this standard were as follows: group 1 = 21/35 (60%; 95% CI = 43 to 77%), group 2 = 20/59 (34%; 95%CI = 21 to 46%), group 3 = 12/29 (41%; 95% CI = 22 to 60%).
DISCUSSION
Summary of main findings
Pre-echocardiography ECG screening for LVSD by a representative sample of Scottish GPs has a mean sensitivity of 0.94 (95%CI = 0.92 to 0.95) and a mean specificity of 0.58 (95% CI = 0.56 to 0.60). Quality of screening, which is dependent on ECG reporting skill, varies between individual GPs. Fifty-seven per cent of GPs achieve a sensitivity of 90% and a specificity of 50% for the detection of LVSD.
Strengths and limitations of the study
Participants are a representative sample of GPs. They were randomly selected from the current list of Scottish GP principals and stratified to ensure that they reflect the range of ECG-reporting experience and confidence in the GP population. ECGs were all obtained from patients consulting their GPs with symptoms suggestive of heart failure.
It is necessary to consider the effect of case mix when comparing the current results with those of other pre-echocardiography screening studies. This study included 41 ECGs from patients with LVSD among the 150 ECGs in the study. Of these, 40 had a major ECG abnormality and one (2.4%) had a minor abnormality. This contrasts with a study where ECGs showed minor abnormalities or were normal in 6% of LVSD cases,5 and others where 10–12% of cardiologist-reported ECGs were normal.7,8 The absence of LVSD cases with normal ECGs will result in an overestimate of the sensitivity of GP screening in the current study. The inclusion of only one case with a minor ECG abnormality may also result in a small overestimate of the sensitivity of GP screening. GPs may have more difficulty distinguishing minor abnormalities from normal, than major abnormalities from normal. They are more likely to miss a case of LVSD with a minor ECG abnormality. The possibility that some GPs may have consulted textbooks or colleagues while interpreting ECGs cannot be excluded.
Comparison with existing literature
The potential of the ECG as a pre-echocardiography screening test for LVSD has been demonstrated in studies where ECGs were interpreted by hospital specialists.5,7,9,10 Prior to this study, data were limited on pre-echocardiography screening for LVSD with GP-reported ECGs,6,11,12 and it was not possible to make informed or evidence-based decisions about its place in a national strategy for the diagnosis of heart failure. In the present study, where pre-echocardiography ECG screening for LVSD was performed by a representative sample of Scottish GPs, mean sensitivity and specificity are comparable to those published for cardiologist-reported ECGs (pooled sensitivity 0.90 [95% CI = 0.88 to 0.92] and pooled specificity 0.58 [95% CI = 0.56 to 0.60]).13
Implications for clinical practice
Results from the current study indicate that overall screening performance is satisfactory. However, differences in the quality of ECG reporting between individual GPs will result in variation in referral rates for echocardiography and in detection rates of LVSD. Managers with a responsibility for implementing the NICE guideline3 need to be aware of these results. They may consider it appropriate to allow referral of patients in whom there was a high clinical suspicion of LVSD, even when the GP considers the ECG to be normal. Training that uses appropriate educational packages could increase the effectiveness of pre-echocardiography screening using GP-reported ECGs, although there is no evidence to support this at present. Alternatively, improved access to B-type natriuretic peptide testing, which is being evaluated in primary care patients14 and has been recommended for use in NHS in Scotland,13 could be a preferred means of achieving more uniform patterns of referral for echocardiography.
Acknowledgments
We would like to thank Margaret Feeney and Marie Pitkethly of EastRen for administrative and secretarial support, and to our GP colleagues for participating in the project.
Notes
Funding body
This project was funded by NHS in Scotland Chief Scientist Office. Sanjay Jeyaseelan was funded by the British Heart Foundation
Ethics committee
This study received MREC approval as a ‘no local investigator’ study from Northwest Multi-centre Research Ethics Committee at their meeting of 8 June 2004 (04/MRE08/2)
Competing interests
The authors have stated that there are none
- Received May 2, 2006.
- Revision received July 6, 2006.
- Accepted October 25, 2006.
- © British Journal of General Practice, 2007.