Article Text
PDF
Abstract
Objective The identification of severe bacterial infection (SBI)in children with fever without source (FWS) remains a diagnostic problem. The authors previously developed in their Swiss population a risk index score, called the Lab-score, associating three independent predictors of SBI, namely C reactive protein (CRP), procalcitonin (PCT) and urinary dipstick. The objective of this study was to validate the Lab-score in a population of children with FWS different from the derivation model.
Methods A prospective study, conducted in Padova, on 408 children aged 7 days to 36 months with FWS was recently published. PCT, CRP, white blood cell count (WBC) and urinary dipstick were determined in all children. The Lab-score was applied to this population and the diagnostic characteristics for the detection of SBI were calculated for the Lab-score and for any single variable used in the Italian study.
Results For the identification of SBI, the sensitivity of a score ≥3 was 86% (95% CI 77% to 92%) and the specificity 83% (95% CI 79% to 87%). The area under the receiver operating characteristic curve for the Lab-score (0.91) was significantly superior to that of any single variable: 0.71 for WBC, 0.86 for CRP and 0.84 for PCT. The Lab-score performed better than other laboratory markers, even when applied to children of different age groups (<3 months, 3–12 months and >12 months). The results obtained in this validation set population were comparable with those of the derivation set population.
Conclusions This study validated the Lab-score as a valuable tool to identify SBI in children with FWS.
Statistics from Altmetric.com
In young children with fever without source (FWS), one challenge is to identify those with a severe bacterial infection (SBI) among a majority suffering from a benign viral infection. The commonly used screening method to discriminate non-toxic children at risk of SBI combines a clinical evaluation associated with laboratory variables; a white blood cell count (WBC) with differential, and a urine analysis.1,–,4 More recently, the determination of C reactive protein (CRP) and procalcitonin (PCT) concentrations has been reported to have a better diagnostic accuracy.5,–,8
However, the considerable overlap of these variables in patients with and without SBI limits their discriminative ability when applied as single predictors. We have recently developed a laboratory risk index score called the “Lab-score” combining three markers—PCT, CRP and urinary dipstick.9 This risk index score was the most accurate tool differentiating children with and without SBI. However, the relationship between predictors and outcome may depend on the characteristics of the derivation sample and could change when tested in a different population. Consequently, internal validation is not a guarantee for generalisability and is not a substitute for external validation.10 Therefore, before implementation, the score should be validated externally on a different population and clinical setting.11 Thus, the aim of our study was to validate the Lab-score on a large population of children with FWS enrolled in a previous Italian study conducted in a paediatric emergency department.12
Methods
The purpose of this study was to validate the Lab-score on an external population. This population has to be comparable to the derivation set: it must have similar inclusion and exclusion criteria to those used in the derivation one. But, it has to include children from a different time period and location. We therefore used the data of 408 children with FWS, prospectively enrolled in the emergency department of the Children's Hospital in Padova, Italy, between 1 May 2004 and 31 October 2005. Padova Children's Hospital provides primary and secondary care for a metropolitan area of 350 000 people (45 000 younger than 15 years) and tertiary care for a regional and extraregional population. The vaccination rate for children living in the regional area was approximately 90% for Haemophilus influenzae B and 40% for Streptococcus pneumoniae, at the time of the study. We refer to the paper by Andreola et al12 for a detailed description of the methodology of their work, which included infants younger than 3 years of age who underwent blood analysis because they were more likely to have an SBI (ie, all infants aged 7 days to 3 months old with rectal temperature >38°C and children 3–36 months old, ill/toxic appearing or with fever (rectal temperature) >39.5°C). They excluded from the study those children with a history of (1) antibiotic use within the 48 h before admission to the hospital, (2) vaccination during the previous 2 days, (3) known immunodeficiencies, (4) any chronic pathology or (5) fever lasting longer than 5 days. They recorded for all patients the complete clinical history, demographic information, degree and duration of fever, physical examination and clinical evaluation using the Infants Observation Score.13 According to the guidelines in use at the time of the study, they obtained from all patients the WBC, absolute neutrophil count (ANC), quantitative CRP concentration and urine dipstick; in addition, a serum sample was also collected and stored at −20°C for later determination of PCT level. Toxic-appearing children had a full sepsis work-up. Infants from 1 week to 90 days of age and ill-appearing children aged 3–36 months received a blood culture and two consecutive urine cultures. Well-appearing children aged 3–36 months received a blood culture when displaying WBC >15 000 cells/mm3 or ANC >10 000 cells/mm3, and two consecutive urine cultures if urine analysis was positive for leucocyte esterase and/or nitrite test.
What is already known on this topic
▶ Current US guidelines in the management of young children with fever without source (FWS) are rarely followed by paediatricians because of time constraints.
▶ Biological markers such as procalcitonin (PCT) and C reactive protein (CRP) have been shown to be quick and reliable predictors of severe bacterial infection (SBI).
▶ A risk index score of SBI associating PCT, CRP and urinary dipstick has been recently published and showed to be superior to any individual markers.
What this study adds
▶ This risk index score has been now validated in a large external cohort of young children with FWS.
▶ This risk index score of SBI is a quick and useful tool for the management of FWS in emergency departments.
The investigators followed-up all patients by telephone contact or clinical assessment by a paediatrician within the next 72 h. They registered the final diagnosis at the end of follow-up. They classified children on the basis of their final diagnosis into two groups: patients with severe bacterial infections (SBI group) or without severe bacterial infections (non-SBI group). They considered the following diagnosis as SBI: (1) bacteremia—recovery of a single bacterial pathogen using standard blood culture techniques; (2) acute pyelonephritis—growth of a single bacterial urinary tract pathogen at ≥105 colony-forming units/ml in two consecutive urine samples and presence of a renal hypocaptation at dimercaptosuccinic acid (DMSA) scan performed within the first week after admission; (3) lobar pneumonia—presence of focal infiltrate on chest radiography observed by the pediatric radiologist in a blinded manner; (4) bacterial meningitis—positive cerebrospinal fluid culture; (5) bone or joint infections—local isolation or isolation in blood culture of a microorganism and (6) sepsis defined according to Levy et al.14 Children with negative cultures or clinical improvement without antibiotic therapy or with detection of a focal infection at follow-up were classified in the non-SBI group. After having carefully reviewed the data of these 408 children, we decided to exclude in our analysis two children with osteomyelitis, due to the absence of a bacterial pathogen recovered from blood or bone biopsy.
We recently developed a laboratory risk index score for SBI, called the Lab-score, based on data from 202 children aged less than 3 years with FWS.9 We classified children on the basis of their final diagnosis into two groups, patients with SBI or without SBI, and we used the same diagnostic criteria as the Italian study. Briefly, this score took into account only predictive variables independently associated with SBI in this group of children, namely PCT, CRP and urinary dipstick. The relative weight of each variable was based on the OR in univariable analysis. Two points were attributed to PCT and CRP above the cut-off values of 0.5 ng/ml and 40 mg/l, respectively, and 4 points for values of PCT above 2 ng/ml and for CRP above 100 mg/l. One point was attributed for a positive urine dipstick (ie, positive leucocyte esterase and/or positive nitrite). Consequently, Lab-score values ranged from 0 to 9 points (table 1). We calculated that a cut-off value of 3 points best differentiated children with and without SBI with a sensitivity of 94% (95% confidence interval (CI) 82% to 99%), and a specificity of 81% (95% CI 72% to 88%). To validate our Lab-score on a different population, we applied it to the data obtained from the study performed in Italy.
Lab-score
Statistical analysis
We compared demographic characteristics between our population and the Italian cohort using the Mann–Whitney U test for continuous values and the Fisher's exact test for frequencies with Stata 7.0. The diagnostic performance of the Lab-score and the other laboratory variables considered were compared using a receiver operating characteristic (ROC) analysis with MedCalc 9.5.
We determined the sensitivity, specificity and negative and positive predictive values for the detection of a SBI both for the different laboratory variables and for the Lab-score using the cut-off points derived from our previous studies.15 16
We determined the predictive values of the Lab-score for all patients and for subsets of patients of different age groups: <3 months, 3–12 months and >12 months of age. For all variables, we also calculated positive and negative likelihood ratios.
Informed consent was obtained from the parents or legal guardians for the additional blood sampling. The study protocol was approved by the Padova Hospital Ethics Committee.
Results
Four hundred and six patients were considered for analysis. Two hundred and three (50%) were female. The median age was 9.6 months (range 0.2–36); 106 (26%) were younger than 3 months, 138 (34%) aged from 3 months to 12 months and 162 (40%) were older than 12 months.
SBI was diagnosed in 92 (22.7%) children and non-SBI in 314 children. Among SBI, the main diagnoses were pyelonephritis in 50 (12.3%) and pneumonia in 24 (5.9%) children (table 2). Six (1.5%) patients had occult bacteremia. In the non-SBI group of children, the diagnoses were focal bacterial infection in 64 (16%), proved viral infection in 36 (9%) and probable viral infection in 214 (53%).
Comparison of the final diagnosis of the children with fever without source in the validation and in the derivation population
The median age and the diagnosis distribution were comparable between our derivation set population and these 406 children used as the validation set (table 2).
A higher proportion of children were seen after 24 h of the onset of fever in the Italian population compared to our original population (65% vs 46%, p<0.001).
The area under the ROC curve (AUC) for the Lab-score, 0.91 (95% CI 0.87 to 0.93), was significantly higher than the AUC for PCT, 0.84 (95% CI 0.80 to 0.87) (p=0.002), than the AUC for CRP, 0.86 (95% CI 0.82 to 0.89) (p=0.02), and than the AUC for WBC, 0.71 (95% CI 0.66 to 0.75) (p<0.001) (figure 1).
Receiver operating characteristic analysis for Lab-score, procalcitonin (PCT), C reactive protein (CRP) and white blood cell count (WBC) for prediction of severe bacterial infection.
The diagnostic characteristics of the Lab-score and the other laboratory variables are reported in table 3. The sensitivity of a positive score (≥3) for the identification of SBI was 86% (95% CI 77% to 92%) and the specificity 83% (95% CI 79% to 87%). The diagnostic accuracy of the Lab-score was superior to any single marker, such as WBC (sensitivity 52%, specificity 75%), CRP (sensitivity 73%, specificity 81%) and PCT (sensitivity 75%, specificity 76%). Nine cases of pyelonephritis, two cases of pneumonia, one case of arthritis and one case of bacteremia had a score below 3. The bacteremia case was a neonate of 7 days with a group B Streptococcal-positive blood culture. The patient presented with a very short fever duration (<8 h), a low CRP (5 mg/l), but an increased PCT to 1.9 ng/ml, which was, however, just below the cut-off value of 2 ng/ml to get a positive score.
Sensitivity, specificity, positive and negative predictive and likelihood ratio values of the Lab-Score, WBC, CRP and PCT for SBI detection
We then calculated the likelihood ratios and post-test probabilities for a Lab-score inferior or equal/superior to 3. The results showed 0.17 and 5.1 for a negative and a positive likelihood ratio, respectively. Taking into account a pretest probability of 22.7%, a negative Lab-score (<3) yielded a post-test probability of 4.8% and a positive Lab-score (≥3) resulted in a post-test probability of 60% (figure 2). The Lab-score discriminated children at risk of SBI better than all other variables taken separately. In the age subgroup analysis, there is a trend towards higher sensitivity (p=0.03) and lower specificity (p=0.04) with increasing age (table 3).
Normogram for applying likelihood ratios (LR) for Lab-score and white blood cell count (WBC) in a population with a pretest probability of 23%.
Discussion
This study demonstrates that the Lab-score developed for young children with FWS in our population9 can be applied in other settings with similar study population characteristics with the same accuracy. Indeed, as we previously showed, the combination of PCT, CRP and urinary dipstick, variables included in the Lab-score, is significantly superior to any single variable and to WBC count, in terms of sensitivity but also specificity and predictive values for SBI detection in children less than 3 years of age.
Even though the populations of the derivation and validation set belong to different countries, it is noteworthy how the SBI prevalence (26.7% vs 22.7%) and the diagnosis distribution are very similar: occult bacteremia (both at 1.5%), focal bacterial infection (12.8% vs 15.8%) and probable viral infection (60.4% vs 61.6%) (table 2). Having proved that the aetiology of FWS in young children is comparable in different places, at least in countries with similar socio-economic levels, the Lab-score can be applied as a useful tool for the management of these children, even in settings other than the one of origin. As the good performance of the Lab-score in the derivation set could only be due to chance, the external validation of this prediction rule is an important step to demonstrate its wider efficacy and applicability.11
In this validation study, the Lab-score sensitivity was slightly lower: 86% (95% CI 77% to 92%) and the specificity slightly higher 83% (95% CI 79% to 87%) than previously reported but remained in the 95% CI of the derivation population set.9 Although we showed that the sensitivity of the Lab-score seemed to increase with the age of the child, going from 78% in infants less than 3 months of age to 97% in those more than 12 months of age, and that the specificity decreased with age going from 90% in infants less than 3 months to 77% beyond the age of 1 year, no statistical difference could be found between the derivation and the validation population set regarding the mean age of the studied populations who could have explained the difference of the sensitivity and the specificity between the two populations.
As we have demonstrated the validity of the Lab-score in a different setting, what could be the main advantages of its use in the management of young children with FWS? First, current published guidelines still propose to use a clinical score and the WBC as screening methods to identify SBI in non-toxic children with FWS.1 3 17 However, even though a trained paediatrician performs a clinical score, it remains subjective and variable with time. In contrast, the Lab-score is an objective tool that takes into account laboratory values at a specific time point. Furthermore, the predictive values of the Lab-score were considerably higher than those of WBC to identify SBI. By calculating the likelihood ratios to determine post-test probabilities, we showed that a WBC >15 000 cells/mm3 increased the probability of SBI from 23% to only 38% in the studied population, whereas a Lab-score ≥3 increased this probability from 23% to 60%. Moreover, the Lab-score performed better than WBC also for ruling out SBI (figure 2). In accordance with other studies, we conclude that WBC is not a good predictor of SBI7 18,–,21 or even of bacteremia.22
Second, the good specificity of the Lab-score for the detection of SBI especially in children during the first year of life enables a reliable selection of children who need antibiotic treatment. It is actually commonly accepted that antibiotics are too often prescribed to young children with viral infections,23 and reducing antibiotic prescription is now a general goal for diminishing microbial resistance and treatment costs.24 In a recent meta-analysis on antimicrobial control strategies, studies using rapidly available ancillary tests were associated with the greatest reduction in antibiotic use25 and hospitalisation. If children in the validation set received antibiotics based on a positive Lab-score, only 33% would have been treated, compared to the 67% who were given antibiotics in the derivation population, according to clinician's decisions. The use of the Lab-score could thus substantially reduce antibiotic prescription and could achieve cost saving without compromising patient care.
Finally, if algorithms are used to select patients who are the most likely to benefit from antibiotic treatment, then they must be accurate and applicable in all medical settings where time pressure is important. If WBC count and differential can be obtained in less than 30 min in most emergency departments, they are seldom obtained in this time frame by office practitioners, who do not have skilled technicians. It is therefore not surprising that compliance with the actual guidelines is low and varies widely between private office settings and hospital emergency departments. Thus, according to many authors, the recommendations are inadequate and do favour overhospitalisation and overprescription of antibiotics.26,–,28 On the contrary, the Lab-score is very time-sparing since results are available in less than 20 min and it is simple to use since anyone can perform these tests without referring to an external laboratory. CRP and PCT values are obtained from less than 200 µl of blood by rapid determination tests and urine dipstick by direct reading. Moreover, the score itself is simple to calculate and easy to remember.
Potential limitations of our study should be considered. As for the derivation set population, the validation of this score has been performed in a population recruited in the emergency departments of a reference tertiary care hospital. Therefore, the incidence of SBI was high (22.7%). This bias could influence the positive and negative predictive values of the Lab-score, but not its sensitivity and specificity. As the majority of children with FWS are seen primarily by private practitioners, who refer sicker children to reference hospitals, the incidence of SBI is supposed to be lower in the general population. Thus, to extrapolate the predictive values of the Lab-score to a standard population of children with a 10% incidence of SBI, we calculated the post-test probabilities using likelihood ratios, which remain uninfluenced by the incidence of the disease. In this situation, the post-test probability would be 1.9% for a score <3, and 37% for score ≥3. However, it is necessary to further validate the Lab-score in a population of children seen in general practice to assess its feasibility, its performance and how it can influence antibiotic prescriptionand referral to hospital for management and hospitalisation. Furthermore, the sensitivity and specificity of the score could vary if this population of children had a different proportion of SBI than the population of reference.
In conclusion, this study validated the performance of the Lab-score on an external population and confirmed its superiority compared with WBC and to the single markers as predictor of SBI in children less than 3 years of age with FWS. However, as the sensitivity of this score is not 100%, close follow-up should be ensured in order to identify the small proportion of children with SBI not initially detected by a positive score. Using the Lab-score to identify children at risk of SBI might allow a substantial reduction in antibiotic prescription. Finally, further studies are needed to establish its performance in settings other than referral hospitals.
References
Footnotes
Competing interests None.
Ethics approval The study protocol was approved by the Padova Hospital Ethics Committee.
Provenance and peer review Not commissioned; externally peer reviewed.