Validity of British Thoracic Society guidance (the CRB-65 rule) for predicting the severity of pneumonia in general practice: systematic review and meta-analysis

Search strategy

A search strategy was designed for PubMed in order to retrieve all relevant articles. MeSH terms ‘community acquired infection’, ‘respiratory tract infection’, ‘pneumonia’, ‘severity indices’, and ‘prognosis’ were combined, and potential studies were retrieved. This subject-specific search was combined with a methodological filter for clinical prediction rules.10

PubMed was searched from 1966 to June 2009. In addition, Embase, the Cochrane Library, and Medline databases were searched for relevant studies. Google Scholar was used to track references to each selected study, and hand searches of references in the included studies were also performed. Authors of published studies were also contacted for additional data and for knowledge of any unpublished studies.

Selection

Inclusion criteria for the systematic review were as follows: (1) study design – cohort study; (2) patient population – adult patients (≥16 years) with a primary diagnosis of CAP; (3) explanatory variables – CRB-65 score calculated; (4) setting of care – community-based or hospital-based patients; and (5) outcome measure – death within 30 days. Studies that used the same dataset for more than one publication were included once in the meta-analysis.

Validity assessment

Quality assessment was independently performed by two researchers following the methodological standards of McGinn for validation studies of clinical prediction rules.5

Data abstraction

Abstracts of potential studies were independently screened by two researchers and those considered were read fully in duplicate and their suitability for inclusion in the study was independently determined by both researchers. Disagreements were managed by consensus and the involvement of a third researcher.

Statistical methods

The initial derivation study of the CRB-65 rule was used as the predictive model to which all validation studies were compared.3 The number of deaths as predicted by the CRB-65 severity index was compared to the observed number of deaths in each of the validation studies, across the three strata of risk in the CRB-65 rule – low risk (score = 0), intermediate risk (score = 1 or 2), and high risk (score = 3 or 4) (Figure 1). In order to calculate the predicted score, the proportionate mortality estimate from the original derivation study was applied according to the three risk strata – low risk (mortality = 1.2%), intermediate risk (mortality = 8.2%), and high risk (mortality = 31.3%),3 an approach previously used in a community-based validation study.8 The predicted:observed risk ratios for these three risk strata are presented separately for patients who were treated in the community or in hospital. Results are presented as risk ratios (RRs) with 95% confidence intervals (CIs). RR<1 indicated an under-prediction of death by the CRB-65 rule (the observed number of deaths is greater than the predicted number) and RR>1 indicated an over-prediction of death by the CRB-65 rule (the observed number of deaths is less than the predicted number). RR = 1 indicated a perfect calibration between observed and predicted deaths. Assessment of publication bias was also made by means of generating a funnel plot, and statistical testing for publication bias.

Review Manager 5 software from the Cochrane Collaboration was used to perform the analysis, determine heterogeneity, and produce forest plots. RRs were calculated with 95% CIs using the Mantel-Haenszel statistical method. A random-effects analysis was performed, and heterogeneity was described by the I² statistic.

Description of included studies

The search identified 14 studies with a total of 397 875 patients (Figure 2).8,11–23 Table 1 summarises the characteristics of the included studies. Additional data were provided from three studies,13,15,23 and clarification of included overlapping subjects obtained from two other studies.12,16 Only one study was performed entirely in a primary care setting,8 although two other studies did include data from patients who were treated as outpatients.12,14 The included studies ranged in size from 105 patients to 388 406 patients, with a broadly equal sex balance. Two studies limited the inclusion for age to ≥65 years.8,23 Most studies also had mortality at 30 days as an outcome measure, corresponding directly in timing to the original derivation study.3 However, two studies reported in-hospital mortality and one reported mortality at 42 days.12,23 Thirty-day mortality was obtained from the authors of one of these studies,23 and in-hospital mortality was used as an approximation to 30-day mortality in the other two. The quality-assessment standards of the included studies are summarised in Figure 3. No publication bias exists in the studies (P>0.05) (Appendix 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Selection of studies for inclusion in the meta-analysis.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Summary diagram of the quality assessment of the included studies.

Descriptive statistics of CRB-65 risk strata

Of the 397 875 patients included in the studies, 66 827 (17%) were classified as low risk (score = 0), 283 902 (71 %) intermediate risk (score = 1 or 2), and 47 146 (12%) high risk (score = 3 or 4) (Table 2). However, the percentage of patients in the three risk strata varied substantially depending on the setting of care, with community-based individuals being categorised into lower-risk categories. Observed events were much less frequent in community-based settings (Table 2).

Validation of CRB-65 risk strata

For lower-risk individuals, there are no events in community-based patients with consequent over-prediction; in low-risk hospital-based patients there is relative under-prediction, but the 95% CI around this estimate is compatible with accurate prediction (Figure 4). In intermediate-risk groups, over-prediction persists in community-based subjects, and observed 30-day mortality more closely matches predicted mortality for hospital-based patients, with a tighter CI (RR 0.91, 95% 95% CI 0.71 to 1.17) (Figure 5). For high-risk patients, the magnitude of over-prediction attenuates in community-based subjects, while observed mortality matches that predicted by CRB-65 in hospital-based patients (RR 1.01, 95% CI = 0.87 to 1.16) (Figure 6).

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

Low-risk group (score = 0) predicted and observed deaths in community and hospital settings.

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

Intermediate-risk group (score = 1 or 2) predicted and observed deaths in community and hospital settings.

• Download figure
• Open in new tab
• Download powerpoint

Figure 6

High-risk group (score = 3 or 4) predicted and observed deaths in community and hospital settings.

Sensitivity analysis

The study by Ewig et al has a significantly larger number of included participants than all others, and the results of that study dominate the meta-analysis of the hospital-based patients.22 Sensitivity analysis performed excluding this study shows similar results but with broader CIs, with a low-risk-group RR of 1.25 indicating over-prediction of CRB-65 but with a wider CI (0.60 to 2.59). The intermediate-risk group and high-risk group RR estimates remain similar, being RR 0.98, (95% CI = 0.79 to 1.22) and RR 1.03 (95% CI = 0.87 to 1.21) respectively.

Summary of main findings

This systematic review shows that application of CRB-65 accurately predicts 30-day mortality in hospitalised patients across three strata of risk. In community settings, CRB-65 appears to over-predict the probability of 30-day mortality across all strata of predicted risk (Figures 4–6).

Strengths and limitations of the study

Pooling of data across 14 separate validation studies enables assessment of the performance of CRB-65 in different clinical settings, addressing validity, generalisability, and precision of the estimates across the three different strata of risk. Several limitations of this study should be considered when interpreting the results. The low event rate, particularly in communitybased studies, makes precise estimates about CRB-65 performance less certain. Observed events (Table 2) are considerably lower than estimated in the original derivation study (Figure 1) on which clinical guidelines are based.3,4 Individual patient data could not be obtained, so it was only possible to estimate predicted risk by applying the original risk estimates to the strata in each of the validation studies, an approach adopted in a community-based validation study.8 However, as CRB-65 relies on unequivocal clinical characteristics, it is unlikely that misclassification of patients occurred. The hospital-based patients category is dominated by one large study that contributed 98% of the total patient dataset;22 however, sensitivity analysis indicates that results are similar when this study is excluded from the meta-analysis. One study only measured patients who suffered from pneumococcal pneumonia,19 but rather than exclude this study, it was included albeit that the event rate is low and is likely to produce an under-prediction of risk. While more studies validating CRB-65 in primary care are needed, it is likely that a large number of patients will need to be recruited to enable precise estimates of performance to be made.9 The outcome of interest – 30-day mortality – is the final clinical endpoint but more frequently occurring intermediate outcomes might be more relevant to clinical care in a primary care setting, such as admission to hospital, clinical deterioration, or re-consultation. Alternatively, quality of life scores or duration of illness may be more indicative of morbidity associated with CAP, and are more likely to be outcomes of more immediate relevance to patients. Lastly, some studies assessed hospital mortality rather than 30-day mortality and this could account for differences in the performance of the rule between different studies.

Comparison with existing literature

CRB-65 is one of several clinical prediction rules that assess the severity of CAP. Other clinical prediction rules that address complications of CAP include the Pneumonia Severity Index (PSI) and CURB-65. The PSI was the first clinical prediction rule derived and validated as a prognostic indicator for CAP; however, its use of 20 different clinical variables makes calculation complex.24 In addition, the inclusion of laboratory values limits its use in primary care settings, since many GPs wil not have immediate access to such investigations. CURB-65 includes the measurement of urea, and this also limits is timeliness and value in primary care.25 The PSI has been compared to the CURB-65 rule in a number of studies,17,29,26–29 and the CPRs have been found to perform similarly.25 These preliminary results indicate that the greater number of variables in the PSI does not significantly strengthen the power of the rule, and therefore their inclusion may only stand as a hindrance to its use.

A number of other clinical prediction rules have been derived with promising initial results, including SCAP,28 SOAR,23 and CORB,13 along with one clinical prediction rule derived entirely in primary care.30,31 The hospital-derived CPRs all contain some or all of confusion, tachypnoea, hypotension, and age in their scoring, and therefore have similarities to CRB-65; however, these rules differ by including other criteria (such as oxygenation) and by the outcomes they predict: development of severe CAP,28 requirement of ventilatory or inotropic support,13 or death.13,23 The primary care-derived clinical prediction rule has different criteria to CRB-65 (diagnosis, age, comorbid conditions, medications, and recent hospitalisation), and the additional outcome measure of hospitalisation along with death. Although preliminary results are promising, these clinical prediction rules have yet to be validated to a great extent, and therefore their use in clinical practice is more uncertain.9 The use of generic sepsis and early-warning scores has also been proposed but preliminary results indicate that they are less effective than pneumonia-specific rules.11

In terms of comparative prognostic value when comparing CURB-65 with CRB-65, it appears that additional urea measurement does not substantially improve the predictive value of CURB-65.12–15,17,20,23 This, along with the simplicity of CRB-65, indicates that it is emerging as potentially the most useful severity score for CAP.

Implications for future research and clinical practice

The inclusion and exclusion criteria for this meta-analysis were broad and reflect those used in the validation studies themselves but also do prove to be a limitation in the analysis. Significant common comorbid illnesses are known to have an impact on severity;2,31–34 their impact was not assessed in the validation studies. Other known predictors of severity such as nursing home residency, recent hospitalisations, and recent antibiotics were also not addressed.2,31,32,34 However, the derivation cohort for CRB-65 specifically included patients with these predictors, and the clinical prediction rule was developed adjusting for these prognostic markers. If included separately in a severity assessment alongside CRB-65, these factors may improve prediction; however, their presence in both the derivation and validation cohorts indicates that the components of CRB-65 have independent predictive characteristics.

Another additional way in which prediction may be improved is with the use of near-patient tests such as C-reactive protein or procalcitonin. These tests have been shown to be an adjunct in the diagnosis of CAP, and preliminary results indicate they may also have predictive value for severity.16,18,35,36 A role for these tests may be found as an additional predictor of severity alongside CRB-65, but it is more likely they could be used as a further discriminator for the appropriate site of care after application of CRB-65.

In the context of clinical management strategies, over half of patients in community settings were classified as low risk, with over 40% classified as intermediate risk (Table 2). Within this group, CRB-65 was found to over-predict the risk of death. However, the event rate level is low and the extent of over-prediction by CRB-65 remains uncertain. It is possible that investigation of other clinical predictors or near-patient tests may improve prediction, but this requires further study. The addition of other risk factors such as comorbidity would also enable different thresholds to be set in community settings, reflecting the lower absolute number of events compared with hospital-based settings (Table 2).

In hospital-based patients, the severity scores were more evenly distributed, with two-thirds in the intermediate-risk group and one-fifth in the high-risk group. As a prognostic tool, CRB-65 appears to be more clinically useful in hospital settings. As clinical medicine is being driven by evidence-based standards of care,37 prognostic rules are likely to become the benchmark against which acceptable standards of care are judged. In this context, as this systematic review shows, it is important that caution is used when extrapolating evidence from studies that have recruited from hospital-based rather than community-based settings. Accurate risk stratification of likely prognosis requires confirmatory validation in similar populations of patients and settings of care.7

Finally, future work with the CRB-65 rule will allow for more robust determination of its usefulness in both hospital and community-based practice. Assessment of the impact of the rule on antibiotic prescribing, especialy in the context of a low-risk patient, will be worthwhile. Also, determining an appropriate threshold for referral from primary care to hospital may be established, since the community-based results indicate that the rule may over-predict mortality in this setting, particularly in the low- and intermediate-risk groups (Figures 4 and 5).

In conclusion, CRB-65 has been found to accurately predict severity of pneumonia and resultant deaths in a hospital setting across al strata of risk. CRB-65 has not been validated sufficiently in primary care settings and preliminary findings suggest over-prediction, so its value as a prognostic indicator in the community remains uncertain.