Abstract
Background Accurate prediction of the course of an acute cough episode could curb antibiotic overprescribing, but is still a major challenge in primary care.
Aim The authors set out to develop a new prediction rule for poor outcome (re-consultation with new or worsened symptoms, or hospital admission) in adults presenting to primary care with acute cough.
Design and setting Data were collected from 2604 adults presenting to primary care with acute cough or symptoms suggestive of lower respiratory tract infection (LRTI) within the Genomics to combat Resistance against Antibiotics in Community-acquired LRTI in Europe (GRACE; www.grace-lrti.org) Network of Excellence.
Method Important signs and symptoms for the new prediction rule were found by combining random forest and logistic regression modelling. Performance to predict poor outcome in acute cough patients was compared with that of existing prediction rules, using the models’ area under the receiver operator characteristic curve (AUC), and any improvement obtained by including additional test results (C-reactive protein [CRP], blood urea nitrogen [BUN], chest radiography, or aetiology) was evaluated using the same methodology.
Results The new prediction rule, included the baseline Risk of poor outcome, Interference with daily activities, number of years stopped Smoking (> or <45 years), severity of Sputum, presence of Crackles, and diastolic blood pressure (> or <85 mmHg) (RISSC85). Though performance of RISSC85 was moderate (sensitivity 62%, specificity 59%, positive predictive value 27%, negative predictive value 86%, AUC 0.63, 95% confidence interval [CI] = 0.61 to 0.67), it outperformed all existing prediction rules used today (highest AUC 0.53, 95% CI = 0.51 to 0.56), and could not be significantly improved by including additional test results (highest AUC 0.64, 95% CI = 0.62 to 0.68).
Conclusion The new prediction rule outperforms all existing alternatives in predicting poor outcome in adult patients presenting to primary care with acute cough and could not be improved by including additional test results.
INTRODUCTION
Acute cough is one of the main reasons for consulting in primary care, with an incidence of 30 to 50 cases per 1000 patients per year.1 Although antibiotic treatment for acute cough has been shown to have little or no effect — both overall and in patients with comorbidities — and the majority of acute cough cases are caused by a self-limiting lower respiratory tract infection (LRTI), antibiotics are prescribed to >50% of patients.2–4 This inappropriately high level of antibiotic prescribing is explained by the difficulty in identifying patients who might benefit from antibiotic treatment (for example, those suffering from a bacterial LRTI or pneumonia).5 The best way forward is to identify early and manage differently those at high risk of an adverse outcome in which the risk for complications might outweigh the risk for unnecessary treatment, while adopting a ‘wait and see approach’ for the others who are expected not to need treatment, hence adjusting treatment according to prognosis rather than diagnosis.6,7
Existing prognostic prediction rules include the Pneumonia Severity Index (PSI; Appendix 1), CRB (confusion, respiratory rate, blood pressure), CURB (confusion, urea nitrogen, respiratory rate, blood pressure), CRB-65 (65 years of age and older), and CURB-65.8–10 These prediction rules were developed to predict mortality in patients presenting to the emergency department with community-acquired pneumonia (CAP), but CRB-65, CURB-65, and PSI could also be used to predict mortality from CAP in outpatients.11,12 However, because death from CAP is very uncommon in outpatients, several authors suggested that other outcomes be considered.13–15 Therefore, the authors of this study developed a prognostic prediction rule to predict poor outcome (that is, re-consultation with new or worsened symptoms, or hospital admission) in adults presenting to primary care with acute-cough, aiming to enable GPs to reassure patients at low risk and provide appropriate advice for patients at high risk. The performance to predict poor outcome in acute cough patients for the new and existing prediction rules (PSI step 1, CRB, CURB, CRB-65, and CURB-65) was compared and the improvement of the new prediction rule’s performance was evaluated by including additional test results (C-reactive protein [CRP] or blood urea nitrogen [BUN]), chest radiography, and aetiology.
METHOD
Data
Data on the presence of poor outcome (re-consultation with new or worsened symptoms, or hospital admission) in adults presenting to primary care with acute cough were collected within the Genomics to combat Resistance against Antibiotics in Community-acquired LRTI in Europe (GRACE; www.grace-lrti.org) Network of Excellence.5 Patients who had no outcome reported (4.4%) were excluded from analyses. To avoid computational issues, the authors selected countries with >15 poor-outcome patients for further analysis — that is, Belgium, Germany, the Netherlands, Poland, Spain, and the UK (Table 1). The working data contain information on 105 variables recorded for 2604 patients. Included covariates cover information that is available to the GP on the day of consultation, concentrations of CRP and BUN, chest radiography, aetiology, and reported outcome (more information available from the authors on request). Bacterial pathogens that were tested for include Streptococcus pneumoniae, Haemophilus influenzae, Mycoplasma pneumoniae, Chlamydophila pneumoniae, Bordetella pertussis, and Legionella pneumoniae. Viral pathogens tested for include rhinovirus, influenza virus, coronavirus, respiratory syncytial virus, human metapneumovirus, parainfluenza virus, adenovirus, polyomavirus, and bocavirus.
Number of total patients and patients with poor outcome for countries included in the working data
How this fits in
In adults presenting to primary care with acute cough, accurate prediction of poor outcome could curb antibiotic overprescribing. The performance of existing prediction rules to predict poor outcome in these patients is very poor. The new prediction rule, RISSC85, presented in this study outperforms these alternatives, and could not be improved by including additional test results. It could help reduce antibiotic overprescribing by enabling clinicians to reassure their patients.
Development of the new prediction rule
Missing covariate information was imputed, on a country-specific basis due to heterogeneity between countries, using multiple imputation by chained equations (five imputations).16,17 To account for the difference in baseline risk of poor outcome, countries were grouped according to the observed proportion of poor outcome patients (A: <15% [Spain], B: 15–25% [Belgium, Netherlands, Poland, UK], C: >25% [Germany]). A conditional random forest approach was then used to identify the most important variables for each imputed dataset.18 The number of variables selected for inclusion in an imputation-specific logistic regression model was chosen based on the number of included patients (Table 1). The authors removed non-significant variables in a backwards fashion (α = 0.10), included interaction terms between remaining fixed effects, and removed non-significant interaction terms (α = 0.05).
Variables that were significant in at least two imputation-specific models were retained in the group-specific model, which was reduced in a backwards fashion (α = 0.05). Variables that were significant in at least one group-specific model were retained in the general model, which was then reduced in a backwards fashion (α = 0.05).19 The prediction rule was constructed using the final general (‘clinical’) model and its pooled parameter estimates, with the optimal cut-off value determined using the Youden index.20 The procedure followed is illustrated in Appendix 2.
Validation of the new prediction rule
The stability of the new prediction rule was evaluated using cross-validation. For this procedure, the data were split into three sets of equal size by sorting by country and assigning every first (second, and third) observation to the first (second, and third) dataset. Two sets were used to conduct backwards model building, starting from the general model, and to obtain pooled parameter estimates. The third set, together with the pooled estimates, was used to determine the area under the receiver operator curve (AUC), which is the probability that for each (randomly) chosen pair (one patient with and one patient without poor outcome), the one with poor outcome is correctly identified. This AUC can range from 0.5, corresponding to no discriminative ability, to 1, corresponding to perfect discrimination.21 This procedure was repeated three times, such that each set was used to determine the AUC once. Empirical bootstrapping was used to obtain 95% confidence intervals (CI).
Evaluation of the new prediction rule
The new prediction rule’s performance to predict poor outcome was compared with that of five existing prediction rules (PSI step 1, CRB, CURB, CRB-65, and CURB-65) using their AUCs.8–10 The improvement in discriminative ability obtained by including information on biomarkers (CRP or BUN), chest radiography, and aetiology was evaluated using the AUCs after inclusion of these components (separately). Empirical bootstrapping was used to obtain 95% CIs.
Sensitivity, specificity, positive, and negative predictive values (PPV and NPV, respectively), with and without inclusion of CRP, BUN, chest radiography, and aetiology, were compared. A cut-off for poor outcome of 20% was selected, and a sensitivity analysis around this threshold was conducted, using thresholds of 15% and 25%.
To provide a tool for clinical practice, the authors derived a simplified prediction rule by dichotomising continuous covariates (diastolic blood pressure and number of years stopped smoking) and retaining only the covariate level with the highest impact on the predicted odds of poor outcome for multilevel covariates (further details available from authors on request).
RESULTS
Countries with <15 poor-prognosis patients were excluded from the analyses (France: seven out of 30 patients; Italy: zero out of 18 patients; Slovakia: five out of 139 patients; Slovenia: six out of 73 patients; and Sweden: eight out of 103 patients). The analyses included 2604 patients, of which 521 experienced poor outcome, divided over six countries (Table 1). Baseline characteristics for these patients are available from the authors on request.
Variable-importance plots for all imputed datasets are available from the authors on request. The final model for group A shows that the odds of a poor outcome are affected by the presence of lung diseases other than asthma or chronic obstructive pulmonary disorder (P = 0.0031), the presence of coughing attacks (P = 0.0308), and the presence of crackles upon physical examination by the GP (P = 0.0022). The final model for group B shows that the odds of a poor outcome are affected by the use of antidepressants (P = 0.0204), the severity of interference with daily activities (P = 0.0016), the number of years since the patient stopped smoking (P = 0.0069), and the severity of sputum as assessed by the patient (P = 0.0005). The final model for group C shows that the odds of a poor outcome are affected by the patient’s smoking status (stopped smoking < or >45 years ago, P = 0.0090), and diastolic blood pressure (> or <85 mmHg, P = 0.0038). Pooled odds ratios (ORs) for group-specific models are available from the authors on request.
The clinical model contains variables related to the patient’s context (baseline Risk of poor outcome, P<0.0001), the patient’s symptoms obtained during an interview with the GP (severity of Interference with daily activities, P<0.0001), the patient’s general information obtained through the patient diary (the number of years since the patient stopped Smoking [< or >45 years ago], P = 0.0045), self-assessment of symptoms obtained through the patient diary (severity of Sputum as assessed by the patient, P = 0.0047), the patient’s signs upon clinical examination by the GP (presence of crackles, P = 0.0117), and diastolic blood pressure (> or <85 mmHg; P = 0.0020). Pooled ORs are reported in Table 2. The final prediction rule (RISSC85) was obtained using pooled parameter estimates. The optimal Youden cut-off was 0.18, which implies that a patient is classified to be at low risk for poor outcome when the predicted probability is below this threshold, and at high risk when it is above this threshold.
Pooled odds ratios for parameters in the ‘clinical’ model
Validation of the new prediction rule
The three-fold cross-validation approach reveals that, out of the nine predictors present in the full general model, similar variables were kept in the clinical model and the three reduced general models, with the three most significant variables present in all models. AUCs for all reduced general models were comparable (data available from the authors on request), indicating that the stability of the clinical model is acceptable.
Evaluation of the new prediction rule
Comparing the AUCs of five existing prediction rules (PSI stage I, CRB, CURB, CRB-65, and CURB-65) and RISSC85 demonstrates that RISSC85 outperforms all existing prediction rules (Table 3).
Area under the receiver operator curve (AUC) and 95% bootstrap confidence intervals (CI) for the new (RISSC85) and five existing prediction rules
Adding continuous CRP concentration to the clinical model resulted in an OR for poor outcome of 1.010 (95% CI = 0.990 to 1.031) per 10 mg/L rise in concentration. Adding continuous BUN concentration to the model resulted in an OR for poor outcome of 0.970 (95% CI = 0.803 to 1.185) per 10 mg/dL rise in concentration. The AUC of the clinical model did not improve significantly after addition of CRP or BUN (Table 4). Because of the limited added value of continuous CRP and BUN, they were not analysed further as dichotomised covariates.
Area under the receiver operator curve (AUC) and 95% bootstrap confidence intervals (CI) for the new prediction rule (RISSC85) alone, and with the inclusion of parameters
Including chest radiography resulted in an OR for poor outcome of 0.927 (95% CI = 0.623 to 1.380) if pneumonia is detected on the radiograph. Adding bacterial aetiology resulted in an OR for poor outcome of 1.324 (95% CI = 1.047 to 1.677) if a bacterial agent was detected. Addition of viral aetiology resulted in an OR for poor outcome of 0.821 (95% CI = 0.672 to 1.004) if a viral agent was detected. Addition of other information on aetiology resulted in an OR for poor outcome of 1.247 (95% CI = 0.882 to 1.765) if only a single bacterial agent was detected, 0.745 (95% CI = 0.583 to 0.952) if only a single viral agent was detected, 1.162 (95% CI = 0.511 to 2.646) if multiple bacterial agents (but no viral agent) were detected, 1.281 (95% CI = 0.780 to 2.109) if multiple viral agents (but no bacterial agent) were detected, and 1.161 (95% CI = 0.824 to 1.634) if both viral and bacterial agents were detected. The AUC of the clinical model did not improve significantly after adding chest radiography or aetiology (Table 4).
Performance of the prediction rule with and without additional covariates (CRP, BUN, chest radiography, bacterial, viral, and general aetiology) was comparable, with sensitivities between 62% and 63%, specificities between 57% and 59%, PPVs between 27% and 28%, and a NPV of 86% (further information available from the authors on request). Using a 15% threshold resulted in sensitivities between 85% and 87%, specificities between 27% and 32%, PPVs between 23% and 24%, and NPVs between 89% and 90%. Using a 25% threshold resulted in sensitivities between 32% and 36%, specificities between 80% and 82%, PPVs between 30% and 32%, and an NPV of 83%.
Selection of the covariate level with the highest impact resulted in an AUC of 0.59 (95% CI = 0.57 to 0.62) (Table 4). With a score of ≥3 indicating poor outcome, the simplified RISSC85 has 43% sensitivity, 73% specificity, 28% PPV, and 84% NPV. Using 2 or 4 as a threshold for poor outcome resulted in 89% and 9% sensitivity, 24% and 96% specificity, 22% and 32% PPV, and 90% and 81% NPV, respectively (Table 5).
Diagnostic risk classification of poor outcome according to the simplified RISSC85 in 2001 patients with acute cough, and sensitivity, specificity, positive, and negative predictive values for different thresholds
DISCUSSION
Summary
Poor outcome occurred in 521 (20%) of the 2604 adult patients presenting to primary care with acute cough. All important predictors for poor outcome in these patients are readily available to primary care clinicians, as RISSC85 is based on information related to the patient’s baseline risk of poor outcome, severity of interference with daily activities, number of years stopped smoking < or >45 years, severity of sputum at the day of consultation, presence of crackles, and diastolic blood pressure < or >85 mmHg. It is a bit unusual that, though a person indicating sputum to be a severe problem has an increased odds of poor prognosis, a person rating sputum as a very severe problem does not have an increased odds of poor prognosis. The only somewhat plausible explanation the authors put forward is that patients rating their symptoms as extreme were exaggerating, whereas patients rating their symptoms as severe were modest but actually really ill. The performance of RISSC85 was moderate (sensitivity 62%, specificity 59%, PPV 27%, NPV 86%, AUC 0.63 (95% CI = 0.61 to 0.67]), but it outperformed all existing prediction rules used today in predicting poor outcome in adult patients presenting to primary care with acute cough, and its performance could not significantly be improved by including information on BUN, CRP, chest radiography, and aetiology. This indicates that, currently, RISSC85 is the best available option to predict poor outcome in adult patients presenting to primary care with acute cough.
Strengths and limitations
This study is the first in which a prognostic prediction rule for adult acute cough patients in primary care was developed, and uses one of the largest datasets to date. Up to now, prediction rules that were developed to predict mortality in patients presenting to the emergency department with CAP were used instead, corroborating the need to develop a new prediction rule. There were very few hospital admissions, so the outcome does not reflect major complications, but mostly individuals returning with bothersome, new, or worsening symptoms.
The number of included poor-outcome cases, and the total number of included patients, were rather low in some countries (<15 and <150, respectively, in Slovakia, Sweden, Slovenia, France, and Italy). Therefore, these countries were excluded from the analysis, and this study focused on a limited number of countries.6 The prediction rule can, however, still be used in other countries, by estimating their baseline risk for poor outcome using literature or personal experience, after which the country can be classified into group A, B, or C. If computing this risk is not possible, the country could be assumed to have an average baseline risk for poor outcome (15–25%).
Although the authors included a lot of variables in building this prediction model, note that they were not able to include some covariates that have previously been shown to increase the risk of hospitalisation or death from pneumonia, for example, high blood glucose levels and the use of proton pump inhibitors.22,23
Comparison with existing literature
Currently, there are no prediction rules for poor outcome in adults presenting to primary care with acute cough. The only alternative available is the use of prognostic prediction rules that were developed to predict mortality in patients presenting to the emergency department with CAP (for example, PSI and CRB).9,10 Although the use of these prediction rules has been demonstrated to predict mortality in outpatients,11,12 the authors showed that they perform poorly in predicting poor outcome, as defined here for adult patients presenting to primary care with acute cough (Table 3).
The new prediction rule could potentially be improved further and, though other authors suggest the use of, for example, CRP5 to improve the predictive ability for pneumonia, this study found that the inclusion of diagnostic markers (CRP, BUN), chest radiography, and aetiology did not significantly improve the model’s predictive ability.
Van Vugt et al showed that the GP’s ability to diagnose pneumonia, based on their clinical judgement, is better than the prediction model based on signs and symptoms.24 The covariate indicating whether the history taking was suggestive of pneumonia or not was, however, not retained as an important predictor in any of the group-specific models, indicating that the clinical judgement on the presence of pneumonia does not contribute to the prognostic assessment of an adult patient with acute cough. And, though Teepe et al showed that patients with mild unsuspected pneumonia benefited from amoxicillin treatment,25 the covariate ‘Intervention’ (indicating whether the patient received amoxicillin or placebo) was not retained in any of the group-specific models, indicating that amoxicillin does not provide protection against poor outcome in adult patients with acute cough.
Implications for research and practice
Although the predictive ability of RISSC85 is suboptimal, it is the best currently available option to predict poor outcome in adult patients presenting to primary care with acute cough. Given that PPV is only 27% while NPV is 86%, this tool will be more suitable for reassuring patients with acute cough that, given their symptoms, the risk of poor outcome is low. GPs could hence use the simplified RISSC85 to differentiate between patients where a ‘wait and see approach’ is appropriate, and careful reassurance is the preferred treatment strategy, and those more at risk for poor outcome, who could then be more explicitly advised about key symptoms and signs that require re-consultation, and possibly offered a delayed prescription.26
Obtaining CRP, BUN, aetiology, and chest radiography has no added prognostic value, and hence can be avoided when the motive is purely predicting poor outcome.
Before this prediction rule can be fully trusted for use in practice, an external validation in the form of an implementation study would be needed to determine whether it can be used to improve patient management, for example, avoiding adverse events.
Acknowledgments
The authors would like to thank all the clinicians and patients who consented to be part of GRACE, without whom this study would not have been possible. The authors are grateful to key members of the GRACE project group whose hard work has made this study possible, including Niels Adriaenssens, Jordi Almirall, Curt Brugman, Slawomir Chlabicz, Mel Davies, An De Sutter, Patricia Fernandez, Maciek Godycki-Cwirko, Iris Hering, Kerenza Hood, Greet Ieven, Anna Kowalczyk, Christine Lammens, Marieke Lemiengre, Frank Leus, Katherine Loens, Artur Mierzecki, Michael Moore, Magdalena Muras, Gilly O’Reilly, Núria Sánchez Ruano, Tom Schaberg, Matteu Serra Prat, Jackie Swain, Antoni Torres, Robert Veen, and Tricia Worby.
Appendix 1. The Pneumonia Severity Index (PSI)
| Step 1 | |
| Age >50 years | |
| Congestive heart failure | |
| Neoplastic disease, cerebrovascular disease, renal disease, and liver disease | |
| Confusion | |
| Pulse ≥125 beats/minute | |
| Respiratory rate ≥30 breaths/minute | |
| Systolic blood pressure <90 mmHg | |
| Oral temperature <35°C or ≥40°C | |
| None of these characteristics present = > PSI score category 0 | |
| Otherwise proceed to step 2 | |
| Step 2 | |
| Characteristic | Points |
| Age for men | age (years) |
| Age for women | age (years) – 10 |
| Nursing home resident | + 10 |
| Neoplastic disease | + 30 |
| Liver disease | + 20 |
| Congestive heart failure | + 10 |
| Cerebrovascular disease | + 10 |
| Renal disease | + 10 |
| Altered mental status | + 20 |
| Respiratory rate ≥30 breaths/minute | + 20 |
| Systolic blood pressure <90 mmHg | + 20 |
| Temperature <35°C or ≥40°C | + 15 |
| Pulse ≥125 beats/minute | + 10 |
| Arterial pH <7.35 | + 30 |
| Blood urea nitrogen ≥11 mmol/l | + 20 |
| Sodium <130 mmol/l | + 20 |
| Glucose ≥14 mmol/l | + 10 |
| Haematocrit <30% | + 10 |
| Partial pressure of arterial oxygen <60 mmHg | + 10 |
| Or oxygen saturation <90% on pulse oximetry pleural effusion | + 10 |
| PSI score category | |
| 0 Class I — very low mortality | |
| ≤70 Class II — low mortality | |
| 71–90 Class III — intermediate mortality | |
| 91–13 0 Class IV — high mortality | |
| >130 Class V — very high mortality |
Appendix 2. Flow chart of the analysis.
Notes
Funding
The Genomics to combat Resistance against Antibiotics in Community-acquired LRTI in Europe (GRACE) was funded by the European Community’s Sixth Framework Programme (grant agreement 518226). Work in the UK was also supported by the National Institute for Health Research (NIHR), in Barcelona by 2009 SGR 911 Ciber de Enfermedades Respiratorias (Ciberes CB06/06/0028), and in Belgium by the Research Foundation, Flanders (FWO; G.0274.08N). Financial support from the Methusalem financing programme of the Flemish Government is also gratefully acknowledged. This publication has been financially supported through the European Science Foundation, in the framework of the Research Networking Program TRACE (www.esf.org/trace). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Niel Hens acknowledges support from the University of Antwerp scientific chair in evidence-based vaccinology, financed in 2009–2017 by an unrestricted gift from Pfizer and GSK.
Ethical approval
The study was approved by ethics committees in all participating countries. Patients who fulfilled the inclusion criteria provided written informed consent.
Provenance
Freely submitted; externally peer reviewed.
Competing interests
The authors have declared no competing interests.
Discuss this article
Contribute and read comments about this article: bjgp.org/letters
- Received July 14, 2017.
- Revision requested October 20, 2017.
- Accepted January 2, 2018.
- © British Journal of General Practice 2018
REFERENCES
In this issue
Jump to section
More in this TOC Section
Related Articles
Cited By...
