Abstract
Background Clinical prediction rules (CPRs) to identify children with serious infections lack validation in low-prevalence populations, which hampers their implementation in primary care practice.
Aim To evaluate the diagnostic value of published CPRs for febrile children in primary care.
Design and setting Observational cohort study among febrile children (<16 years) who consulted five GP cooperatives (GPCs) in the Netherlands.
Method Alarm signs of serious infection and clinical management were extracted from routine clinical practice data and manually recoded with a structured electronic data-entry program. Eight CPRs were selected from literature. CPR-variables were matched with alarm signs and CPRs were applied to the GPC-population. ‘Referral to emergency department (ED)’ was used as a proxy outcome measure for ‘serious infection’. CPR performance was assessed by calibration analyses, sensitivity, specificity, and area under the ROC-curve (ROC-area).
Results A total of 9794 GPC-contacts were eligible, 54% male, median age 2.3 years (interquartile range 1.0–4.6 years) and 8.1% referred to ED. Frequencies of CPR-variables varied from 0.5% (cyanosis, drowsy) to 25% (temperature ≥40°C). Alarm signs frequently included in CPRs were ‘ill appearance’, ‘inconsolable’, and ‘abnormal circulatory or respiratory signs’. The height of the CPR’s predicted risks generally corresponded with being (or not being) referred to the ED in practice. However, calibration-slopes indicated that three CPRs underestimated the risk of serious infection in the GPC-population. Sensitivities ranged from 42% to 54%, specificities from 68% to 89%. ROC-areas ranged from 0.52 to 0.81, with best performance of CPRs for children aged <3 months.
Conclusion Published CPRs performed moderately well in the primary out-of-hours care population. Advice is given on how to improve translation of CPRs to primary care practice.
- adolescent
- bacterial infection
- child, preschool
- child
- decision support techniques
- fever
- infant
- primary health care
INTRODUCTION
Most children who present to primary (out-of-hours) care have fever as one of their main complaints.1,2 Febrile children are at risk of serious infections, such as meningitis or pneumonia, which are important causes of morbidity and mortality.3–5 However, the prevalence of serious infections in primary care is low6 and physicians have the challenging task of distinguishing children at high risk of serious infections from those with self-limiting disease. Only a few studies on the identification of serious infections in primary care have been published so far.6,7 Consequently, practice guidelines are mainly based on consensus of expert opinion and scientific evidence collected from secondary and tertiary emergency care studies.8,9 To complement practice guidelines, clinical prediction rules (CPRs) could be powerful tools to improve clinical decision making on the basis of combinations of clinical signs and symptoms.10 However, most published CPRs for serious infections have been developed predominantly at hospital emergency departments (EDs),11–13 with the lack of external validation in low-prevalence populations hampering their implementation in primary care practice.11,13,14 The present study aimed to assess the applicability and diagnostic value of published CPRs for serious infections in febrile children consulting primary out-of-hours care.
METHOD
Study design
As part of an observational study, semi-structured, routine clinical practice data were prospectively collected on children who had presented to out-of-hours primary care (workdays from 5 pm to 8 am, and the entire weekend) with fever. The diagnostic value of published CPRs for serious infections was assessed, defining ‘referral to ED’ as the outcome measure.
Study setting and selection of patients
The out-of-hours healthcare system in the Netherlands and data collection of this study have been published previously.15 In summary, all contacts of children <16 years that had taken place at five GP cooperatives (GPCs) of the Rotterdam Rijnmond-district (collaboration of >250 GP-practices) between March 2008 and February 2009 were selected. Eligible contacts were those concerning children who had a face-to-face consultation with the GP and reported fever as the reason for contact, had fever within the 24 hours before contact, or had a (rectal or tympanic) temperature >38°C measured at the GPC. Re-contacts for the same problem within 7 days of the initial presentation were excluded from the main analyses.
How this fits in
Published clinical prediction rules (CPRs) to identify children with serious infections lack validation in low-prevalence populations, which hampers their implementation in primary care practice. The present study shows that published CPRs perform only moderately well in the primary out-of-hours care population with limited rule-out value. Improved translation of CPRs to primary out-of-hours care could be achieved by the introduction of structured documentation of vital signs, application of inflammatory marker point-of-care tests, introduction of follow-up contacts, and the identification of predictors with diagnostic value in the primary care setting specifically.
Extraction of relevant clinical signs
Clinical features indicative of serious infections were derived from one systematic review12 and two published guidelines on management of febrile children.8,16 Features were included that:
had a high predictive value (positive likelihood ratio >5.0 or negative likelihood ratio <0.2);
were mentioned in at least two of the three data sources;
did not represent a diagnosis; and
were not prone to high inter-observer variability (for example, auscultatory sounds).17
The selected features were grouped into 18 ‘alarm signs’ (Appendix 1). For eligible contacts, it was manually recoded whether alarm signs were ‘present’, ‘absent’, or ‘not mentioned’ in the patient record, using the data-entry computer program Embarcadero Delphi XE (version 15.0). Clinical management by the GP was recoded as ‘referral to ED’, ‘follow-up appointment at GP(C)’, or ‘no follow-up’.
Selection of CPRs and translation to the primary out-of-hours care population
Eligible CPRs were selected on the basis of two important reviews on this topic,11,12 and an additional literature search of published CPRs for serious infections in children (published between 1975 and 2012, and relevant to developed countries; Appendix 2). Selected CPRs were deemed to:
have clinical signs and symptoms as predictors;
have no more than one laboratory test as a predictor variable, because these are unavailable at the GPC;
have a composite outcome of serious infections; and
advise on management strategies or give a risk score.
Eight of 32 CPRs extracted from literature were included in the final analyses (Table 1).6,18–23 Variables of the selected CPRs were matched with the alarm signs in the GPC-dataset. In case alarm signs were not entirely identical to the original CPR variables, best proxy variables were used. When CPR variables were missing in the GPC-dataset, it was assumed these were absent.
The unavailability of data on exact diagnoses hampered the verification of outcome diagnoses in the routine clinical practice dataset. As in primary care, identification of febrile children at risk of a serious infection (that is, requiring specialist assessment) is often more important than confirming the exact diagnosis,14,24 ‘referral to ED’ was used as a proxy for ‘serious infection’. This proxy was validated among a subset of GP-referred febrile children who presented to the ED of the nearby Sophia Children’s Hospital during the out-of-hours period (January 2006 to July 2009; N = 376).25 It was observed that 66% of these GP-referred children required some form of extensive diagnostic interventions (such as, blood culture or lumbar puncture), extensive therapeutic interventions (such as, intravenous [IV]-medication or aerosol treatment), or hospitalisation, indicating the presence of a serious febrile illness. Only 395 (4%) of 9794 GPC-contacts had a second contact for the same complaint within 7 days, of which 67 (0.7%) were referred to the ED. Figures were comparable for children who had or had not been prescribed antibiotics at first consultation.
Missing data
As clinical information was obtained from routine practice data, there were some missing values. A consensus-meeting with one GP, two paediatricians, and two residents (general practice and paediatrics) decided, for the purpose of this study, to deal with missing values in two way. As GPs are taught to recognise alarm signs in febrile children, it was assumed that alarm signs were always documented when present and, consequently, when alarm signs were ‘not mentioned’ in the patient record, these were considered to be ‘absent’. Also for continuous variables (such as, temperature and duration of fever), missing values were replaced by mean values.
Statistical analyses
Patient characteristics and frequencies of alarm signs were analysed using descriptive statistics. For some CPRs based on multivariable logistic regression models, two separate but closely related variables were combined into one alarm sign (such as ‘tachypnoea’ and ‘chest wall retractions’ into ‘shortness of breath’) or categorical variables (such as mild, moderate, or severe) were dichotomised into ‘present’ or ‘absent’ for application in the GPC-population. For such CPR variables the β-coefficients were recalculated as a weighted mean on the basis of the original β-coefficients and the number of patients with the variable present at derivation.
Selected CPRs were applied to the eligible GPC-population, within the age ranges for which the rules were originally derived. For CPRs which predict a high or low risk of serious infection (CPRs 1–3), calibration was assessed by calculating the percentage of referral in the predicted high- and low-risk groups. For CPRs which gave a continuous risk prediction (CPRs 4–8), first the linear predictor was calculated, which is the sum product of regression coefficients of the rule and the variable values (lp = α + β1·x1 + β2·x2 + … + βi·xi, in which α is the intercept and β1βi are the regression coefficients of the variables x1xi). Calibration was assessed by calculating the observed frequency of referral among patient groups based on percentiles of the predicted risk (that is, lp-outcome) and calculation of calibration slopes. Discriminative ability was assessed by calculating sensitivity, specificity, and likelihood ratios for CPRs 1–3 and areas under the receiver operating characteristic curve (ROC-area) for CPRs 4–8. Statistical analyses were performed with SPSS PASW software (version 17.0.2).
RESULTS
Description of the GPC-population
In total, 9794 GPC-contacts were eligible for analyses (Figure 1). General patient characteristics and clinical features are shown in Table 2. Vital signs such as heart rate, respiratory rate, and oxygen saturation were documented in only 2% of the patients (data not shown). Referral to the ED was required for 794 (8.1%) contacts.
Performance of CPRs applied to the GPC-population
Eight CPRs were applicable to the primary out-of-hours care setting (Table 1). Observed frequencies of CPR variables were generally low in the GPC-population and ranged from 0.5% (cyanosis/drowsy) to 25% (temperature ≥40°C; Table 3). For CPRs 1–3, observed referral frequencies varied from 4% to 38% among the patients predicted as ‘low-risk’ and from 13% to 78% among those predicted as ‘high-risk’ (Table 4). For CPRs 4–8, the distribution of patients over the percentile groups of the predicted risk differed considerably per CPR, with the lowest percentile groups accounting for minimally 6% to maximally 44% of the total population (Table 5). The observed percentage of children referred to the ED was generally low for those in the lowest percentile groups (range 0–20%). Observed referral frequencies of children in the upper percentile groups ranged from 17% (CPR 5) to 100% (CPR 7). Calibration slopes varied from 0.17 (CPR 6) to 2.05 (CPR 8), with three rules having a slope >1, which indicates that the effects of CPR variables were larger in the GPC- population than in the derivation-population (Appendix 3). Sensitivities of CPRs 1–3 ranged from 42% to 54%, and were lower than those reported in derivation settings. In contrast, specificities of CPRs 2 and 3 were higher (>86% versus 39% and 35% at derivation), as were positive and negative likelihood ratios (Table 4). Discriminative abilities of CPRs 4–8 varied widely, but were to some extent comparable with ROC-areas reported in the derivation studies (Table 5). CPR 6 had the lowest ROC-area of 0.52, whereas the two rules developed for young children showed the best discriminative abilities with ROC-areas of 0.77 (CPR 4) and 0.81 (CPR 7).
DISCUSSION
Summary
The present study demonstrated that published CPRs for serious infections, mainly derived at hospital EDs, performed only moderately well in the primary out-of-hours care setting using ‘referral to ED’ as the outcome measure. Most CPR variables were observed to be reported positively in a low frequency in the GPC-population. Limited rule-out value was found for CPRs that classified children into high- or low-risk groups. Use of CPRs which gave a continuous risk prediction was too moderate to be directly applicable to clinical primary care practice.
Strengths and limitations
This study is the first to assess the diagnostic value of several published CPRs for serious infections in an urban, multi-ethnic, out-of-hours primary care cohort of nearly 10 000 contacts of febrile children. It is anticipated that these results will be valuable to many other countries where primary care is provided by GPs or where out-of-hours care has similarly shifted towards large-scale cooperatives.26 As prospective studies on serious infections in low prevalence settings are logistically challenging and time consuming, routine clinical practice data were used in this study. Consequently, the results should be considered in the context of some difficulties and limitations elicited by this ‘second best’ approach.14,24
First, ‘referral to ED’ was used as a proxy outcome measure for ‘serious infection’. It is acknowledged that using a differently defined, although well-correlated, outcome measure may be suboptimal and may have resulted in an overestimation of the prevalence of serious infections in the study population (for example, reflected by calibration slopes >1). For each of the selected CPRs, however, ‘serious infection’ has been defined differently. As for primary care physicians, identifying children at high risk of serious infections is more important than knowing exact diagnoses, ‘referral to ED’ may be an outcome, which captures all of these different definitions of serious infections. This proxy was validated but some bias cannot be excluded, as for some children clinical management may have been based on factors other than the presence of alarm signs only (for example, physician’s experience, demanding/concerned parents, or the need for diagnostic certainty).15 On the other hand, some children may have been adequately treated with antibiotics rather than being referred by the GP.
Secondly, there was a considerable number of missing (‘not mentioned’) values. As assumptions on the mechanism of ‘missingness’ may be diverse, missing values were replaced on the basis of clinical rationale for the purpose of this study. To evaluate potential bias arising from this approach, sensitivity analyses were performed with missing values imputed on the basis of correlations between missing values and available information of other variables.27 For CPRs 4–8, these secondary analyses showed similar ROC-areas but calibration slopes closer to 1 (that is, better model fit). For CPRs 2 and 3, particular higher sensitivities were observed (73% and 62%, respectively) and lower negative likelihood ratios (0.4 and 0.6, respectively), which indicates that the main analyses may ‘err’ on the safe side by underestimating rather than overestimating the CPR’s performances.
Comparison with existing literature
The present finding that most CPRs performed only moderately well in the low prevalence setting may, next to the methodological limitations of this study, be explained by the specific characteristics of primary care practice itself. GPs constantly have to balance the risk of missing a serious infection versus unnecessary referral.7 This difficulty particularly accounts for children with an unclear presentation (‘grey area’),14 which was previously demonstrated in studies among children hospitalised for meningococcal disease28–31 and malpractice lawsuits.32,33 Clinical decision support by CPRs may be helpful in this diagnostic dilemma; however, in the low prevalence setting, high rule-out value should be achieved to reduce the number of false negative patients.7,24 Unfortunately, in this study it was shown that the published CPRs which predicted a high or low risk of serious infection showed insufficient rule-out value (low sensitivities and high negative likelihood ratios). Most CPRs that gave a continuous risk prediction could not discriminate well between the middle percentile-groups of the predicted risk, and observed referral frequencies varied considerably (Appendix 3).
Possible explanations for this discrepancy in performance of CPRs at the GPC and ED, may be that in the primary care setting observed frequencies of CPR variables are low, vital signs are barely measured,34 and additional diagnostic tests, such as inflammatory markers, are unavailable. These issues reduced the heterogeneity of CPR variable outcomes, and thus of the predicted risks for each patient in the GPC population (that is, less spread of the predicted risks). Unavailability of predictor variables in the GPC setting may have negatively influenced the performance of some CPRs more than others. Also, some predictor variables are likely to be better predictors of serious infection in the ED setting than the GPC setting.
Unfortunately, the only CPR derived in a primary care setting itself6 showed no diagnostic value in the present GPC-population. Previously, others had also demonstrated only marginal rule-out value for this CPR,13,35 which further underscores the importance of external validation before implementation in clinical practice.10,14
Implications for research and practice
How do the present results support the translation of existing CPRs to primary care settings? Given the importance of vital signs in most CPRs, structured vital sign measurements at GPCs would be advised. The present study suggests that published CPRs should be updated with variables available in and relevant to the primary care setting.36 In this way, CPRs may better discriminate between seriously ill children and those in the ‘grey area’ who have a less clear clinical presentation. The strong diagnostic value of inflammatory markers, as already demonstrated in adult primary care,37–39 may favour their implementation in primary care practice to further improve diagnostic discrimination. However, it should be noted that no inflammatory marker has perfect discriminative ability on its own and clinical observation remains essential (as is reflected by CPRs combining clinical variables with inflammatory markers). Finally, a follow-up period after the initial GPC-contact could contribute to differentiation between evolving serious infections from self-limiting viral disease.
In light of the methodological difficulties, the present study shows that published CPRs performed moderately well in the primary out-of-hours care population with limited rule-out value. Most CPR variables were observed to be reported positively in a low frequency in the GPC-population. Improved translation of CPRs to primary out-of-hours care could be achieved by introduction of structured documentation of vital signs, application of inflammatory marker point-of-care tests, introduction of follow-up contacts, and identification of predictors with diagnostic value in the primary care setting specifically.
Acknowledgments
We acknowledge EJ van Dijk for providing the data collected at the GPC and thank T Krecinic, Z Gocmen, M Hofhuis, and M Rotsteeg for their contribution to the data management of this study.
Appendix 1. Grouping of alarm signs for serious infection
Grouped alarm signs (as coded in the GPC-database) | Total selection of alarm signs |
---|---|
Parental concern | Parental concern |
Ill appearance | Clinician’s instinct something is wrong Clinically ill appearance |
ABC-instability | ABC-instability |
Unconsciousness | Unconsciousness |
Drowsy | Child is drowsy Somnolence Reactivity/functional status (decreased) Hypotonia |
Inconsolable | Child is inconsolable Irritability Changed crying pattern Child is moaning |
Abnormal circulation | Abnormal skin colour (pale, mottled, ashen) Capillary refill time >2 seconds Tachycardia |
Cyanosis | Cyanosis Oxygen saturation <95% |
Shortness of breath | Shortness of breath Nasal flaring Rapid breathing Changed breathing pattern |
Meningeal irritation | Meningeal irritation Neck stiffness Bulging fontanelle |
Neurological signs | Focal neurological signs Paresis/paralysis Seizures/fits |
Vomiting and diarrhoea | Vomiting (>2 times in disease period) Diarrhoea (>2 times in disease period) |
Dehydration | Dry mucous membranes Sunken eyes Decreased skin elasticity Reduced urine output Hypotension (APLS) Poor feeding |
Extremity problems | Swelling of limb or joint Non-weight bearing limb Not using an extremity |
Signs of urinary tract infection | Urinary frequency Dysuria Tummy ache (without other focus for fever) |
Petechial rash | Petechial rash Purpura |
Temperature ≥40°C | Measured at home or at GP cooperative |
Duration of fever | Duration of fever (>38.0°C) in days |
ABC = airway, breathing, circulation
Appendix 2. Electronic search strategy for clinical prediction rules
Pubmed (Decision Trees[mesh] OR decision rule*[tw] OR decision tree*[tw] OR prediction rule*[tw] OR predictive rule*[tw] OR decision model*[tw] OR prediction model*[tw] OR predictive model*[tw] OR decision analysis model*[tw] OR risk score*[tw]) AND (child[mesh] OR child[tiab] OR children[tiab] OR pediatric*[tw] OR infant*[tw]) AND (“Arthritis, Infectious”[Mesh] OR “Bone Diseases, Infectious”[Mesh] OR “Community-Acquired Infections”[Mesh] OR “Respiratory Tract Infections”[Mesh] OR “Sepsis”[Mesh] OR “Skin Diseases, Infectious”[Mesh] OR “Soft Tissue Infections”[Mesh] OR “Urinary Tract Infections”[Mesh] OR “Meningitis”[Mesh] OR meningitis[tw] OR serious infection*[tw] OR serious bacterial infection*[tw] OR severe bacterial infection*[tw] OR severe infection*[tw] OR “Gastroenteritis”[Mesh]) |
Embase (‘Decision Tree’/de OR ((decision* OR predict* OR risk*) NEAR/3 (rule* OR model* OR algorithm* OR aid OR score* OR tree*)):de,ab,ti) AND (child/exp OR (child* OR pediatric* OR infant*):de,ab,ti) AND (‘infectious arthritis’/exp OR ‘hematogenous osteomyelitis’/exp OR ‘communicable disease’/exp OR ‘respiratory tract infection’/exp OR ‘sepsis’/exp OR ‘skin infection’/exp OR ‘soft tissue infection’/exp OR ‘urinary tract infection’/exp OR ‘meningitis’/exp OR ‘gastroenteritis’/exp OR (serious* NEAR/3 infection*):de,ab,ti) |
Appendix 3. Distribution of contacts over the percentiles of the predicted risk and frequency of referral within groups
Percentiles of the predicted risk | CPR4 Pantell et al 19 | CPR 5 Bleeker et al 20 | CPR 6 Berger et al 21 | CPR 7 YICSSG 22 | CPR 8 Brent et al 23 | |||||
---|---|---|---|---|---|---|---|---|---|---|
n | Referral to ED, % | n | Referral to ED, % | n | Referral to ED, % | n | Referral to ED, % | n | Referral to ED, % | |
0–10th | – | – | 893 | 6 | – | – | 7 | 0 | 610 | 6 |
10–20th | 65 | 20 | 228 | 8 | – | – | – | – | – | – |
20–30th | – | – | 744 | 5 | 1052 | 12 | – | – | – | – |
30–40th | 16 | 75 | 281 | 7 | – | – | 65 | 40 | – | – |
40–50th | 6 | 67 | 554 | 4 | 29 | 45 | – | – | 6926 | 4 |
50–60th | 48 | 25 | 811 | 9 | 322 | 11 | – | – | – | – |
60–70th | 7 | 86 | 566 | 4 | – | – | – | – | – | – |
70–80th | 20 | 65 | 464 | 13 | 685 | 10 | 20 | 100 | 291 | 34 |
80–90th | 24 | 92 | 705 | 9 | 35 | 46 | 10 | 80 | 1657 | 11 |
90–100th | 18 | 94 | 563 | 26 | 259 | 17 | 12 | 100 | 278 | 67 |
Total | 204 | 49 | 5809 | 9 | 2382 | 13 | 114 | 58 | 9762 | 8 |
CPR = clinical prediction rule. ED = emergency department. YICSSG = Young Infants Clinical Signs Study Group.
Notes
Funding
This work was supported by an unrestricted grant from Europe Container Terminals B.V. The funder had no role in study design, the collection, analyses or interpretation of data, writing the report or the decision to submit the manuscript for publication. Rianne Oostenbrink is supported by a personal grant of the European Society for Paediatric Infectious Diseases (ESPID Fellowship 2009).
Ethical approval
The institution’s medical ethics committee reviewed the study and the requirement for informed consent was waived (MEC-2012–378).
Provenance
Freely submitted; externally peer reviewed.
Competing interests
The authors have declared no competing interests.
Discuss this article
Contribute and read comments about this article: bjgp.org/letters
- Received June 2, 2014.
- Revision requested June 23, 2014.
- Accepted July 28, 2014.
- © British Journal of General Practice 2015