Article Text
PDF
Abstract
Objectives To derive and validate a clinical score to risk stratify children presenting with acute infection.
Study design and participants Observational cohort study of children presenting with suspected infection to an emergency department in England. Detailed data were collected prospectively on presenting clinical features, laboratory investigations and outcome. Clinical predictors of serious bacterial infection (SBI) were explored in multivariate logistic regression models using part of the dataset, each model was then validated in an independent part of the dataset, and the best model was chosen for derivation of a clinical risk score for SBI. The ability of this score to risk stratify children with SBI was then assessed in the entire dataset.
Main outcome measure Final diagnosis of SBI according to criteria defined by the Royal College of Paediatrics and Child Health working group on Recognising Acute Illness in Children.
Results Data from 1951 children were analysed. 74 (3.8%) had SBI. The sensitivity of individual clinical signs was poor, although some were highly specific for SBI. A score was derived with reasonable ability to discriminate SBI (area under the receiver operator characteristics curve 0.77, 95% CI 0.71 to 0.83) and risk stratify children with suspected SBI.
Conclusions This study demonstrates the potential utility of a clinical score in risk stratifying children with suspected SBI. Further work should aim to validate the score and its impact on clinical decision making in different settings, and ideally incorporate it into a broader management algorithm including additional investigations to further stratify a child's risk.
This paper is freely available online under the BMJ Journals unlocked scheme, see http://adc.bmj.com/info/unlocked.dtl
Statistics from Altmetric.com
Introduction
Serious bacterial infections (SBIs) are a leading cause of morbidity and mortality among children worldwide. A recent review of child deaths in England, Wales and Northern Ireland confirmed that the largest single cause of death in children dying of an acute physical illness was infection, and found evidence that healthcare professionals in both hospital and primary care had difficulty identifying serious illness in children.1 In another study of over 400 children with meningococcal sepsis in the UK, only half were referred to hospital at their first primary care consultation, usually after a delay of several hours following the first symptoms and/or signs of sepsis.2 Early recognition of SBIs is associated with better treatment outcomes3 4 but is challenging as many of the early clinical features also occur in self-limiting illnesses. In developed countries, discriminating serious infection from self-limiting illness is further complicated by the low prevalence of serious infection in children presenting to hospital emergency departments as well as primary care settings.5 Better clinical tools are therefore required to improve the recognition of children with serious acute illness, particularly SBIs.1 6 7
What is already known on this topic
▶ Serious bacterial infections (SBI) are a leading cause of childhood illness and death in the UK and worldwide.
▶ In clinical practice, discriminating children with SBI from those with other self-limiting infections is challenging.
▶ A simple, validated clinical tool to risk stratify and guide further management of children with suspected SBI would greatly improve their care.
What this study adds
▶ The high specificity for SBI of some clinical signs supports their use in clinical guidelines to identify seriously ill children.
▶ The utility of a simple clinical score to stratify children according to SBI risk is clearly demonstrated.
▶ Combining this clinical tool with laboratory tests in a broader management algorithm may further improve SBI diagnosis pathways.
Clinical prediction rules utilising key clinical features to generate a diagnostic or prognostic score are increasingly used to risk stratify patients in a wide variety of clinical situations, particularly those in which diagnostic decisions are complicated, and in some areas have had a major impact on clinical service delivery.8,–,15 We aimed to develop a simple clinical score with the ability to risk stratify children with possible SBIs presenting to a paediatric emergency department.
Methods
Patients and setting
As part of a large prospective study of patient care pathways, detailed clinical data were collected prospectively on all children presenting to the Queen's Medical Centre Emergency Department in Nottingham between September 2000 and March 2001, and September 2001 and March 2002, with the exception of neonates and children requiring immediate emergency resuscitation at presentation. A triage nurse recorded vital signs prior to assessment by emergency department clinical staff. All clinical data including investigations, management and final diagnoses were directly entered onto a standard proforma. Completed proformas were then scanned using Formic electronic data capture software to minimise data entry errors. Study clinicians checked the data for completeness, resolved data gaps and inconsistencies by re-review of the clinical notes, and recorded additional clinical data on children who were admitted. Children who re-attended hospital within 1 week of discharge from either the emergency department or the ward were identified from the electronic patient register, their notes reviewed, and final diagnoses and SBI classification amended in the light of their second presentation. A consultant paediatrician re-reviewed the patient records of all those admitted to check the accuracy of the data, particularly in relation to the final diagnosis recorded.
Children were eligible for inclusion in this analysis if their differential diagnosis at presentation included acute infection. We excluded from the analysis children for whom data were insufficient to confidently assign outcome, or who had missing dates of birth.
Clinical definitions
Automated measurements were used for temperature (tympanic thermometer), pulse rate, blood pressure and pulse oximetry. Tachypnoea, tachycardia and hypotension were defined according to UK Advanced Paediatric Life Support guidelines16; children for whom no blood pressure recordings were available were assumed not to be hypotensive for the purpose of the analysis. We adopted pragmatic definitions of mild hypoxia (SaO2<95% and/or documented oxygen therapy) and severe hypoxia (SaO2<90% and/or SaO2<92% despite documented oxygen therapy), and defined a ‘risk factor for infection’ as a documented history of any condition known to increase a child's risk of invasive bacterial infection. Developmental delay was included separately as a potential risk factor, since several studies have documented an increased rate and severity of acute illness including infection among this group.17 18 Other definitions reflected those commonly used in clinical practice (table 1); data collected routinely on all children included level of consciousness, capillary refill time, hydration status, and presence and type of rash.
Univariable associations between clinical variables and risk of serious bacterial infection (SBI) in children presenting to the emergency department
SBI was defined a priori as admission to hospital plus any of the following (in the absence of an alternative non-infective or non-bacterial diagnosis to explain the clinical and laboratory findings): positive bacterial cultures from blood or another normally sterile site in the appropriate clinical context, radiological signs of pneumonia, clinical meningitis plus a cerebrospinal fluid polymorphonuclear leukocytosis, acute febrile purpura, deep collection(s) requiring intravenous antibiotics±surgical drainage, a white blood cell count ≥20×109/l, a C reactive protein ≥120 mg/l, or a final diagnosis of septic arthritis, osteomyelitis, empyema or mastoiditis.
Statistical methods
Analyses were performed using Stata v 10 (StataCorp LP). The distribution of each variable was summarised with respect to SBI, and crude OR derived. The sensitivity, specificity, positive and negative predictive values (PPV, NPV) and positive and negative likelihood ratios (LR+, LR−) were reported for each variable.
We randomly assigned each child to either a ‘derivation’ or a ‘validation’ dataset, such that ∼85% of children were in the derivation set. Multivariate logistic regression models were then derived using the derivation set to develop a clinical prediction score for SBI. We included in the model only variables with at least a weak association with SBI in the univariate analysis (likelihood ratio test (LRT) p value ≤0.1), first confining this to categorical variables (which are more easily applied in clinical practice), and then repeating the analysis using the available continuous data to explore to what extent this improved the model.
Backwards stepwise logistic regression was performed first using ‘standard’ selection criteria, such that variables not significantly associated with SBI (Wald p value <0.5; t test p value ≤0.1 for continuous variables) are sequentially dropped from the model, and then using ‘sign OK’ selection,19 whereby variables are eliminated from the model only if the sign (+ or −) of the regression coefficient (β) is discordant with known or expected clinical associations (since standard selection criteria may not be optimal in the analysis of smaller datasets15 19,–,21).
Each model took the form: SBI=α+β1X1+β2X2+β3X3+…+βiXi, where α is a constant, βi are the regression coefficients for each variable (Xi) in the model, and Xi are coded as 0 or 1 (for binary variables), integers (for ordered categorical variables) or continuous numbers (for continuous variables). For each model we generated a score for each child, such that the score=β1X1+β2X2+β3X3+... +βiXi. The ability of each score to discriminate children with SBI was quantified in the derivation dataset, and validated in the validation dataset, by calculating the area under the receiver operator characteristics (ROC) curve (AUC) of score as a predictor of SBI.
Finally, the best model was selected to derive a simpler score for clinical use. Criteria for model selection included both discriminant ability (defined by the AUC) and model simplicity. We generated a simplified clinical prediction score using weightings defined by the β coefficients of the chosen model, multiplied if necessary by a constant, and rounded as far as possible to the nearest integer. The discriminant ability of this final ‘SBI score’ was tested by plotting an ROC curve of the score as a predictor of SBI in the entire dataset (derivation and validation sets combined). Risks of SBI were presented stratified by score.
Ethics approval
Ethics approval was granted by the Nottingham Research Ethics Committee and the ethics committee of the London School of Hygiene and Tropical Medicine.
Results
There were 2341 eligible children, of whom 17 were excluded because insufficient data were available to confidently determine outcome. A further 373 children were excluded because their date of birth and therefore their age could not be positively determined from the hospital records due to uncertainty in interpreting their handwritten unique patient identifier number on the proforma; the proportion of children with SBI did not differ significantly between those for whom age data were and were not available (p=0.47).
The final analysis therefore included 1951 children. Their median age was 19 months (range 1 month to 15 years) and 55.4% were boys. Five hundred and forty-six (28.0%) children were admitted at first presentation, of whom 72 had SBI. Thirty-six children re-attended hospital within a week of their initial presentation and within the same illness episode, 18 of whom were admitted and 2 of whom had SBI. The principal final diagnoses of the 74 (3.8%) children with SBI included pneumonia (52), sepsis without a clear focus (14), soft tissue infections (5), urinary sepsis (2) and tonsillitis plus sepsis (1).
Several clinical variables were strongly associated with SBI in the univariate analysis (table 1), including signs of a child's a neurologic, haemodynamic and respiratory state, and a known risk factor for infection. Although hypotension was not significantly associated with SBI, it is important to note that blood pressure data were only available for 103 (5.3%) children.
The predictive values of each variable are summarised in table 2. Apart from tachypnoea (sensitivity 71.6%), the sensitivity of most clinical signs was poor. However, some clinical signs, such as those of reduced level of consciousness/alertness and impaired perfusion were highly specific, and demonstrated more clinically useful positive likelihood ratios. The low prior probability of SBI in this population explains the apparently high negative predictive values of many variables, reflected in the poor utility of most negative likelihood ratios.
Predictive value of clinical variables for serious bacterial infection
After random weighted allocation of children to the derivation and validation sets, the derivation set comprised 1600 (85.2%) children without SBI and 64 (86.5%) children with SBI. The models with the best discriminative ability were those derived using ‘sign OK‘ selection. For the ‘sign OK’ model confined to categorical variables, the AUCs in the derivation and validation sets were 0.76 (0.68 to 0.82) and 0.85 (0.71 to 0.99), respectively. Inclusion of continuous data in the model did not markedly improve performance (AUC 0.76 and 0.86 in the derivation and validation sets, respectively), so the simpler model confined to categorical variables was selected. Regression coefficients and adjusted ORs and 95% CIs for each variable included in this model are given in table 3. A modified SBI score was derived based on the β coefficients of this model, such that:
Regression coefficients (β), and adjusted OR and 95% CI for variables included in the chosen model
The AUC for SBI score as a predictor of SBI in the derivation and validation sets combined was 0.77 (95% CI 0.71 to 0.83; figure 1). Table 4 shows the SBI score in tabular form more conveniently for clinical use, and risks of SBI stratified by SBI score are shown in table 5.
Receiver operator characteristics (ROC) curve of serious bacterial infection score as a predictor of serious bacterial infection in the entire dataset.
Serious bacterial infection (SBI) score for prediction of serious bacterial infection
Risks of significant bacterial infection (SBI) stratified by SBI score
Discussion
By exploring the feasibility of developing a clinical prediction score for risk stratification of suspected serious bacterial illness in children, this study addresses an important challenge in everyday clinical practice in the UK and other developed countries.1 2 5 7 Among children presenting to emergency care in whom infection was suspected (where the prior probability of SBI was 5%), our SBI score was able to rule out SBI with reasonable confidence and precision in three quarters of children in the dataset to whom the score was applied (SBI score ≤5 associated with <2% risk of SBI), to identify children at the highest risk of SBI (a third of children with an SBI score >8 had SBI) and to risk stratify the remaining children into groups with increasing risk.
We modelled continuous data as categorical and continuous variables in separate models. Although less suitable for the development of a clinical prediction score for clinicians, continuous variables maximise information15 22 and can be utilised in computer-assisted diagnostic pathways. However, their inclusion in the model did not improve discriminative ability sufficiently to justify the increased complexity.
Previous studies in developed countries have attempted to derive clinical scores as a tool to identify seriously ill children at presentation.23,–,35 Most were carried out in populations with a higher incidence of SBI prior to the introduction of routine vaccination against Haemophilus influenza type b (Hib) and Streptococcus pneumoniae; others were designed to predict the need for admission to intensive care.9 Moreover, promising initial reports of diagnostic performance have not always been consistently reproducible.27 36 37 This probably reflects a combination of the inherent over-optimism of data-driven model predictions based on small datasets,15 19 21 38,–,40 failure to validate models and/or scores in an independent dataset,15 19 and spectrum bias arising from the use of case–control designs.41,–,43 In a recent study based mainly in primary care, a clinical algorithm was derived with the ability to rule out SBI with reasonable confidence.44 However, these results have not been validated in an independent dataset, and negative predictive values are likely to be lower in emergency department settings where the prior probability of SBI is higher.
The strengths of our study include the large number of children studied, detailed review of all clinical and laboratory data to assign outcome, validation of the model in an independent part of the dataset, and inclusion of all children presenting with suspected SBI in a cohort study design, which suggests the predictive value of our SBI score should be generalisable to other similar settings. Furthermore, focusing on children with suspected SBI rather than all children with serious illness allowed inclusion of clinical risk factors specific to infection, and may have helped improve model specificity. It is important to stress that while the SBI score should be generalisable to emergency departments in the developed world, it is unlikely to be applicable to developing world settings where the prevalence of SBI among paediatric presentations is much higher and there may be other important risk factors for infection such as malnutrition and HIV.5
A potential limitation of the study concerns the definition of SBI. No single, reliable gold standard exists for the diagnosis of invasive bacterial infections in children. Although cultures from blood and other sterile sites are highly specific, sensitivity is limited.45,–,47 The definition of SBI used has been formulated to take account of all available clinical and laboratory information in defining each clinical syndrome, including a child's response to treatment, and is designed to capture all children with invasive or otherwise SBIs who require admission to hospital. A potential criticism is the risk of incorporation bias, since clinicians are not blind to the admission clinical variables studied, which are likely to influence admission decisions; inclusion of clinical and laboratory data from the entire admission in assigning outcome is likely to only partly mitigate this bias. Nevertheless, we believe the definition adopted represents a pragmatic working definition of SBI. Importantly, we were also able to identify those children discharged from hospital who re-attended within a week, and to amend their SBI classification as required. While we cannot exclude the possibility that other children may have re-presented elsewhere, the nearest emergency department, which does not have a dedicated paediatric section, is 18 miles away, so the chance of this is small. Similarly, since sudden childhood death is rare in this population and most cases are brought to hospital, the chance of a child in this study dying at home and not re-attending hospital is also small.
Missing data also limit the power of our analysis. Although the proportion of data missing for individual variables was generally small, this became more significant in the multivariate analysis. Blood pressure is not routinely measured in paediatric clinical practice except in the sickest children, so data were missing for most patients. Nevertheless, blood pressure may have an important role in predicting SBI that this analysis would not be able to identify. The number of children for whom age data were missing is unfortunate as it further limits study power; however, this is unlikely to have introduced any bias since the underlying problem (poor legibility of the proforma) is unlikely to be related to the clinical presentation and was not associated with SBI.
Finally, measurement error is also possible, since emergency department staff assessed each clinical sign using routine clinical tools. For some variables this is likely to be minimal since automated measurements were used. Arguably, even for other variables this may be seen as a strength of the study, since it increases its generalisability to the routine clinical setting.
The implications of this study for clinical practice are threefold. First, the specificity of some clinical signs for SBI (such as reduced activity or level of consciousness, prolonged capillary refill time and reduced skin turgor) support the National Institute for Health and Clinical Excellence clinical guidelines7 which use these clinical features as ‘red flags’ to identify seriously ill children. The presence of these features in a febrile child should prompt a careful clinical assessment. Second, it clearly demonstrates the potential utility of a clinical score to stratify children according to their risk of SBI. Such a score might be incorporated into electronic decision support and medical records. Finally, it underlines the need to use such a clinical score as part of a broader algorithm to guide management decisions (including additional investigations, plans for review or admission for observation) to further stratify a child's risk. Future research should explore the benefit of including in the model other risk factors such as immunisation status, routine blood pressure measurement, and other potential markers of sepsis such as limb pain or cool peripheries,2 investigate how laboratory tests such as C-reactive protein, procalcitonin and white blood cell count may best be incorporated into a clinical algorithm to refine a child's risk status, and further validate the performance and impact14 48 of the score in different acute paediatric settings, including primary care.
Acknowledgments
The authors would like to thank Dr Terence Stephenson for his contributions to the initial data collection.The study was carried out on behalf of the Royal College of Paediatrics and Child Health (RCPCH) working group on Recognising Acute Illness in Children (RAIC). MT is a member of University of Oxford Department of Primary Health Care which is part of the NIHR National School of Primary Care Research. ML is a member of the Leicester Collaboration for Leadership in Applied Health Research and Care (CLARCH) team and Director of the Health Education Research and Development Unit (HERADU).
References
Footnotes
-
Funding AJB is supported by a Wellcome Trust research training fellowship (081697). The authors wish to thank the Well Child Medical Charity for their funding of the studies in Nottingham and of this study.
-
Competing interests None.
-
Ethics approval This study was conducted with the approval of the Nottingham Research Ethics Committee and the ethics committee of the London School of Hygiene and Tropical Medicine.
-
Provenance and peer review Not commissioned; externally peer reviewed.