Data collection

The study was performed in primary care in Flanders, Belgium. All children consulting a GP, paediatrician or the emergency department, not referred by another physician at the moment of their inclusion in the study, were considered to be consulting primary care. First, two hospitals with a paediatric emergency department were contacted for collaboration in the study, each in a geographically distinct area. Secondly, GPs working close to these hospitals were recruited for participation in the study on a voluntary basis. Data collection started on 1 January 2004, and ended on 30 November 2004. Every physician participated during four separate months, equally distributed over the year 2004 to ensure data collection in every season.

Patients aged 0–16 years with an acute illness for a maximum of 5 days were consecutively included in the study. Children were excluded if the acute episode was caused by a merely traumatic or neurological illness, intoxication, psychiatric or behavioural problems without somatic cause, or an exacerbation of a chronic condition. If children were entered twice in the study by the same physician within 5 days, the second registration was considered a repeated measurement on the same subject and was subsequently excluded from the analysis. Finally, physicians were excluded if the assumption of consecutive inclusion was violated (inclusion of fewer than five children in 1 month).

Index tests

Presenting signs and symptoms from history taking and physical examination were recorded on a predefined form. The signs and symptoms were chosen based on a systematic review (Van den Bruel et al, unpublished data, 2007) and on the results of a qualitative study.11

‘Body temperature’ was defined as the highest body temperature measured by the parents or the physician. Before analysis; 0.5°C was added to temperatures measured under the axilla,12 or with a tympanic thermometer.13

‘Something is wrong’ was defined as a subjective feeling of the physician that things were not right. Similar, although not identical, was the sign ‘different illness’, which was defined as a statement by the parents that this illness was different from previous illnesses.

‘Dyspnoea’ was defined as difficult or laboured breathing, ‘tachypnoea’ as breathing frequency of ≥40 per minute, ‘changed breathing’ as any change as compared to normal breathing.

‘Impaired peripheral circulation’ was present when the capillary refill took more than 3 seconds ‘Meningeal irritation’ was based on the presence of neck stiffness, Kernig's sign, Brudzinsky's sign 1 or 2, and a bulging fontanelle, or irritability on manipulation of the head or legs in children aged <1 year old. ‘Petechiae’ were present in cases of a non-blanching rash. The signs ‘irritable’ and ‘drowsy’ were used in the analysis separately, and combined as one variable, ‘changed behaviour’, on the basis of previous research.11

All presenting signs and symptoms were coded as ‘yes’ when present, ‘no’ when absent and ‘?’ when they could not be evaluated, for example headache in a baby.

Participating physicians also noted a working hypothesis for each child at the time of recording.

All procedures were pretested in a small number of practices.

Reference standard

Serious infections were defined as admission to hospital with one of the following infections: pneumonia (infiltrate on chest X-ray); sepsis (pathogen in haemoculture); viral or bacterial meningitis (pleocytosis in cerebrospinal fluid and identification of bacteria or a virus); pyelonephritis (≥105/ml pathogens of a single species and white blood cells in urine and serum C-reactive protein elevation); cellulitis (acute, suppurative inflammation of the subcutaneous tissues); osteomyelitis (pathogen from bone aspirate); and bacterial gastroenteritis (bacterial pathogen in the stool). Sepsis and meningitis were combined a priori as one diagnostic category.

Two different and complementary methods were used to establish the final outcome of the children included in the study. First, hospitalisation was verified for all children by checking hospital records from the 10 regional hospitals in the areas. As a back-up, every participating physician completed a follow-up form after every registration period and at the end of the study, on which any known serious infection had to be reported.

From all children thus identified, all available evidence from clinical, laboratory, radiology, and other tests was collected and presented to a panel of two professors of paediatrics, one paediatrician in a regional hospital, and one professor of general practice. The panel was blinded to the diagnosis of the treating physicians; decisions were made by consensus. Children were considered as not having suffered from a serious infection if no serious infection was identified from hospital records or during follow-up.

Analyses

First, the accuracy of the presenting signs and symptoms was analysed for any serious infection and for each diagnostic category separately. In case of an empty cell in the 2 × 2 table, 0.5 was added to every cell. Analyses were performed with STATA (version 8).

Secondly, a classification and regression tree analysis (CART)14 was performed to create a triage instrument, using ‘rpart’ of the R package (www.r-project.org/). CART is a form of binary recursive partitioning. The term ‘binary’ implies that each group of patients, represented by a ‘node’ in a decision tree, can only be split into two groups. Thus, each node can be split into two child nodes, in which case the original node is called a parent node. The term ‘recursive’ refers to the fact that the binary partitioning process can be applied over and over again. Thus, each parent node can give rise to two child nodes and, in turn, each of these child nodes may themselves be split, forming additional ‘children’. The term ‘partitioning’ refers to the fact that the dataset is split into sections or partitioned.

CART analysis has a number of advantages over other classification methods, including multivariable logistic regression. First, it is inherently non-parametric. In other words, no assumptions are made regarding the underlying distribution of values of the predictor variables. Secondly, the interpretation of results summarised in a tree is very simple. It is much simpler to interpret than the multivariable logistic regression model, making it more practical in a clinical setting. The tree produces positive and negative predictive measures, and other measures such as sensitivity, specificity, and likelihood ratios can easily be derived. Additionally, the inherent ‘logic’ in the tree is easily apparent, and makes clinical sense.

An important feature of the analysis in this study was that the signs and symptoms could be used to either include or exclude the possibility of a serious infection, thus exploiting the asymmetry of tests. Also indeterminate test results, that is, signs that were scored as ‘?’, were considered during the analysis.

Sensitivity and negative predictive value of the trees were maximised by introducing a weighing factor of 75 to the misclassification of a serious infection. The minimum number of observations in a node in order for a split to be attempted, and in any terminal ‘leaf’, was set at 100 in both, to obtain sensible and robust splits and accurate predictions. This method deals effectively with missing data through surrogate splits. The selection of the final tree was based on a 50-fold cross-validation procedure, thereby validating the classification trees internally.

Description of the population

In total, 121 physicians participated in the study, of which 113 were GPs and eight paediatricians; 66% were male, with an average of 17 years of clinical practice experience (range 2–35 years). The eight paediatricians also recruited patients at two different emergency departments.

The analyses were based on 3981 patients; the patient flow is illustrated in Figure 1. Children were on average 5.0 years old (range = 0.02–16.9 years) and 2131 were boys (53.5%). Hospital records were retrieved for 196 children, of which 48 were admitted for reasons other than an acute infection (for example, scheduled surgery) and 117 for an acute, but non-serious infection (predominantly gastroenteritis). A serious infection was diagnosed in 31 children (prevalence 0.78%; 95% confidence interval [CI] = 0.53 to 1.11), with 16 cases of pneumonia, five cases of pyelonephritis, nine cases of sepsis or meningitis and one case of cellulitis. There were no cases of bacterial gastroenteritis for which hospital admission was required. No patient died during the study period. The average age of children with a serious infection was 2.2 years (range having pneumonia; the other 10 were diagnosed as = 0.06–14.1 years), 24 (74.2%) were male. Other having a non-serious infection. Five children out of baseline characteristics are listed in Table 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Patient flow chart

Working hypothesis

Physicians labelled six of the 16 children correctly as having pneumonia; the other 10 were diagnosed as having a non-serious infection. Five children out of nine were correctly identified as having sepsis or meningitis, three children were diagnosed with another serious infection, and one was diagnosed with a non-serious infection. Pyelonephritis was diagnosed correctly in only one child; the other four children were missed, as was the one child with cellulitis. Overall, physicians diagnosed 12 of 31 children correctly at the time of registration (38.7%).

Apart from the working hypothesis, physicians found 310 children to be seriously ill, of which 17 had a serious infection.

Bivariable analyses

Overall, the diagnostic accuracy of presenting signs and symptoms is limited (Supplementary Table 1). Sensitivities are low: only body temperature ≥38°C has sensitivity over 80%. Specificities are higher, with maximum specificity of 99.9% for the symptoms ‘cyanosis’ and ‘meningeal irritation’. Odds ratios (ORs) range from 62 for the sign ‘something is wrong’ to 0.19 for the sign ‘headache’.

The probability of a serious infection increases with increasing body temperature. But, two children presented with a normal body temperature lower than 37.5°C: one child with pneumonia and one with cellulitis.

Signs of an upper respiratory tract infection do not exclude a serious infection: 21 children of the 31 with a serious infection showed signs of upper respiratory infection. Coughing was present in 14 children of the 16 with pneumonia; however, coughing was also present in 47% of the children with a non-serious infection. Crepitations and tachypnoea, two classical signs for the diagnosis of pneumonia, were present in eight children with pneumonia, dyspnoea in 11 children, and decreased breathing sounds or dullness on percussion in five. Only two children did not have any sign suggesting pneumonia: crepitations, tachypnoea, dyspnoea, or dullness on percussion.

Meningeal irritation was present in one child, and impaired peripheral circulation in two of the nine children with sepsis or meningitis, leading to low sensitivity. In contrast, specificity was very high, and positive predictive value sufficient to take further action when present.

Multivariable analysis for any serious infection

The variable ‘something is wrong’ was the first knot in the primary tree (tree 1 in Table 2, Figure 2), and correctly classified 20 children with a serious infection. The following five steps using four variables (age is used twice) added 10 correctly classified children with a serious infection. One child with a serious infection (pyelonephritis), was missed by the tree, and 454 children had a false-positive result. This corresponds to a sensitivity of 96.8% (95% CI = 83.3 to 99.9%), specificity 88.5% (95% CI= 87.5 to 89.5%), positive predictive value 6.20% (95% CI = 4.2 to 8.7%), and negative predictive value 100.0% (95% CI = 99.8 to 100.0%).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Classification tree for any serious infection. Prior probability of serious infection is 0.8% (n = 31).

Analyses were repeated excluding ‘something is wrong’. This second tree mainly used the same variables as the first tree, but the sign ‘illness is different’ as stated by the parents has replaced the sign ‘something is wrong’ (Supplementary Figure 1; tree 3 in Table 2).

Multivariable analysis for pneumonia

For the diagnosis of pneumonia, the classification tree used only two presenting signs and symptoms: ‘dyspnoea’ and ‘something is wrong’ (tree 5 in Table 2), classifying 15 of 16 cases of pneumonia correctly, and only 268 children testing false positive.

Excluding the sign ‘something is wrong’, it was replaced by ‘illness is different’ as shown in Supplementary Figures 2 and 3 (tree 7 in Table 2). With this tree, only one child was missed as in the previous tree, but the number of children testing false positive was higher. Applying these trees to children under the age of 4 years, the positive predictive value increased and the specificity was slightly lower (trees 6 and 8 in Table 2).

Multivariable analysis for sepsis or meningitis

The tree for sepsis or meningitis used only one sign: ‘something is wrong’; eight cases were identified and one case was missed; a false-positive result occurred in 115 children (tree 9 in Table 2).

When the sign ‘something is wrong’ was excluded, the tree used ‘illness is different’ as shown in supplementary figures 4 and 5, (tree 10 in Table 2); two cases of sepsis or meningitis were missed, thus lowering sensitivity, and 188 children tested false positive.

Summary of main findings

The prevalence of serious infections, for which hospitalisation was required, was low (0.78%). Depending on the practice population, a primary care physician will encounter a serious infection in a child, for which hospitalisation is required, two or three times a year. In contrast, the need for triage is high, as acute illnesses in children are extremely common.

All signs and symptoms had sensitivity below 90%. Specificities were better, even over 99% in cases of the classic, textbook signs such as peripheral circulation, cyanosis, convulsions, meningeal irritation, and petechiae.

Contrary to individual signs, the classification trees had high sensitivity and were all superior to the physicians' working hypotheses. The test characteristics of the trees compare favourably to those of other triage instruments, of which the Ottawa ankle rules is one of the best-known examples.15

The sign ‘something is wrong’ was paramount in every classification tree. In this statement, the physician synthesises results from various sources of information and finds something is not right; he or she has a gut feeling about it. However, it is not known which signs the physician based this conclusion on. Possibly, some signs and symptoms are counted twice: the physician finds the child has dyspnoea as such, and concludes that something is wrong, partly based on the same dyspnoea.

However, especially in primary care where physicians need to triage patients and refer them to secondary care if necessary, this gut feeling could prove very useful. No difference was found when the diagnostic value of ‘something is wrong’ was stratified according to the physician's experience of more or less than 10 years, but confidence intervals were wide. In essence, the study is not sufficiently powered for these secondary analyses.

Excluding the sign ‘something is wrong’, it was replaced by a similar, although not identical, sign: the statement of the parents that this illness is different from previous illnesses. It is fair to assume that this sign is a synthesis of information and may equally depend on the parents' experience. This is illustrated by the fact that only one parent of the three infants under the age of 3 months with a serious infection stated that this illness was different. Although the classification trees did not miss a serious infection in any of these infants, using the sign ‘different illness’ in this of population very young children should be done with caution.

Strengths and limitations of the study

The most important strength of the present study is the prospective design, including all eligible children consecutively, which is considered the optimal design for diagnostic accuracy studies.16^,17 In addition, several serious infections were considered as outcome, as triage would be done for a variety of serious infections.

However, verification of the outcome had to rely on information obtained from hospital records and during follow-up, as most patients are seen only once for an acute illness, and additional testing is rare in primary care. Although it is possible that not every child with a serious infection was identified, it is reasonable to assume that this probability was made as low as possible by the measures taken. In addition, presenting signs and symptoms could have been the reason for additional testing and subsequently led to a diagnosis of a serious infection. This may increase sensitivity and specificity.18

Comparison with existing literature

The present study included unselected children from all ages, as the authors believed a primary care physician would need to triage every child, regardless of its age. In contrast, previous studies have focused on infants or young children, and selected patient populations. For example, Pantell et al evaluated the accuracy of signs and symptoms in infants of aged 3 months or younger for the diagnosis of bacteraemia or bacterial meningitis and found age and ‘very ill appearance’ the best clinical predictors.19

A similar population of children was used to derive the Baby Check Cards: infants under the age of 6 months. Here, a serious illness was defined if infants had a positive body fluid bacterial culture, a positive chest X-ray, or if significant treatment was required in hospital. The combination of either drowsiness on history or examination, pallor on history or examination, chest wall recession, temperature >38°C, and a lump being present, identified 82.5% of all babies deemed subsequently to be seriously ill.20 Bleeker et al, created a prediction rule for serious infections in children aged 1–36 months, referred by the GP for fever without a cause, and included duration of fever, poor micturition, vomiting, age, temperature <36.7°C or ≥40°C at examination, chest-wall retractions, and poor peripheral circulation in the model.1

Implications for future research or clinical practice

The classification trees were validated internally to correct for optimism. However, external validation is necessary before implementation in clinical practice, especially in studies with a limited number of cases as in the present one,21^,22 as results from validation studies can be worse or better than the original results.23 Future studies should also assess whether physician characteristics influence diagnostic accuracy. Considering the low prevalence, multicentre studies may be necessary.

In clinical practice, finding any of the classic textbook signs should be a reason for further action, as their positive predictive value is higher than the prior probability of disease. But, the absence of these signs is no argument for ruling out a serious infection. In contrast, the classification trees miss only one or two cases: in other words, a serious infection can be excluded in the vast majority of children with an acute illness on the basis of a few simple clinical tests. But, it is less clear what the primary care physician should to do with every child that tests positive to any of the decision trees. In fact, the positive predictive value is low, leading to a substantial number of false positives. It is not the intention to promote immediate referral for all of those children, except for those suspected of sepsis or meningitis. For the others, a review a few hours later or additional testing such as a chest X-ray or blood samples would be a reasonable option. Future studies will need to provide the evidence underpinning these choices.

The diagnostic value of the individual signs and symptoms is limited, although some signs have high specificity. Combining a limited number of signs and symptoms in classification trees, very few cases are missed. The sign ‘something is wrong’, as stated by the physician, is the strongest predictor for a serious infection.