Completeness of main outcomes across randomized trials in entire discipline: survey of chronic lung disease outcomes in preterm infants
BMJ 2015; 350 doi: https://doi.org/10.1136/bmj.h72 (Published 26 January 2015) Cite this as: BMJ 2015;350:h72
- John P A Ioannidis, professor1,
- Jeffrey D Horbar, professor2, chief executive and scientific officer3, coordinating editor4,
- Colleen M Ovelman, trials search coordinator4,
- Yolanda Brosseau, managing editor4,
- Kristian Thorlund, associate professor5,
- Madge E Buus-Frank, director 6, instructor7,
- Edward J Mills, visiting associate professor8,
- Roger F Soll, professor2, coordinating editor4
- 1Departments of Medicine, Health Research and Policy, and Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Palo Alto, CA, USA
- 2Department of Pediatrics, University of Vermont, College of Medicine, VT, USA
- 3Vermont Oxford Network, Burlington, VT, USA
- 4Cochrane Neonatal Review Group, Burlington, VT, USA
- 5Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, ON, Canada
- 6Quality Improvement and Education, Vermont Oxford Network, University of Vermont,
- 7Geisel School of Medicine at Dartmouth, Burlington, VT, USA
- 8Stanford Prevention Research Center, Stanford University School of Medicine, Palo Alto, CA, USA
- Correspondence to: John P A Ioannidis, 1265 Welch Road, Medical School Office Building, Room X306, Stanford, CA 94305, USA jioannid{at}stanford.edu
- Accepted 2 December 2014
Abstract
Objective To map the availability of information on a major clinical outcome—chronic lung disease—across the randomized controlled trials in systematic reviews of an entire specialty, specifically interventions in preterm infants.
Design Survey of systematic reviews.
Data sources Cochrane Database of Systematic Reviews.
Study selection and methods All Cochrane systematic reviews (as of November 2013) that had evaluated interventions in preterm infants. We identified how many of those systematic reviews had looked for information on chronic lung disease, how many reported on chronic lung disease, and how many of the randomized controlled trials included in the systematic reviews reported on chronic lung disease. We also randomly selected 10 systematic reviews that did not report on chronic lung disease and 10 that reported on any such outcomes and identified whether any information on chronic lung disease appeared in the primary reports of the randomized controlled trials but not in the systematic reviews.
Main outcome measures Whether availability of chronic lung disease outcomes differed by type of population and intervention and whether additional non-extracted data might have been available in trial reports.
Results 174 systematic reviews with 1041 trials exclusively concerned preterm infants. Of those, 105 reviews looked for chronic lung disease outcomes, and 79 reported on these outcomes. Of the 1041 included trials, 202 reported on chronic lung disease at 28 days and 200 at 36 weeks postmenstrual; 320 reported on chronic lung disease with any definition. The proportion of systematic reviews that looked for or reported on chronic lung disease and the proportion of trials that reported on chronic lung disease was larger in preterm infants with respiratory distress or support than others (P<0.001) and differed across interventions (P<0.001). Even for trials on children with ventilation interventions, only 56% (48/86) reported on chronic lung disease. In the random sample, 45 of 84 trials (54%) had no outcomes on chronic lung disease in the systematic reviews, and only 9/45 (20%) had such information in the primary trial reports.
Conclusions Most trials included in systematic reviews of interventions on preterm infants are missing information on one of the most common serious outcomes in this population. Use of standardized clinical outcomes that would have to be collected and reported by default in all trials in a given specialty should be considered.
Introduction
Many randomized controlled trials report only a portion of their primary and secondary outcomes.1 2 3 4 5 This creates substantial potential for bias in the available evidence.6 7 Trials can be misinterpreted when crucial information is missing. Selective reporting further distorts the systematic reviews and meta-analyses of the evidence. The impact of missing information on outcomes is even more influential when the respective outcomes are clinically the most important ones for the patients and setting examined. Some outcomes are so important that all trials, and thus also all systematic reviews, should consider, collect data, and report results on them. Their absence of documentation in both single trials and systematic reviews would be suspect.
Empirical studies probing the selective and partial availability of outcome information to date have been based largely on comparisons of protocol level or registry level information against study publications.1 2 3 4 5 An interesting complementary approach would be to examine all the systematic reviews and meta-analyses that have been performed in an entire medical specialty in which some specific outcome is considered to be ubiquitously important regardless of the intervention being tested. Ideally, such an empirical evaluation would be performed in a specialty in which systematic reviews have extensively covered the randomized evidence across its breadth and many systematic reviews are available. In this regard, perinatal medicine has one of the strongest and longest traditions of systematic reviews across biomedicine: the pregnancy and perinatal medicine module was the first large collection of systematic reviews8 and its successor, the Cochrane Neonatal Review Group, had a leading role in the development and growth of the Cochrane Collaboration. The large number of systematic reviews offers an opportunity to analyze outcome availability in an entire specialty.
We focused on the outcome of bronchopulmonary dysplasia or chronic lung disease because this is a common and important clinical outcome for preterm infants. Both terms are used to describe chronic respiratory insufficiency in newborn infants, and we will use chronic lung disease for clarity. Various definitions have been used in reporting these endpoints. Northway and colleagues first described a pattern of severe lung injury in premature infants who were exposed to supplemental oxygen and positive pressure ventilation at 28 days of age.9 Studies conducted through the 1980s often report on this outcome. As smaller and more immature infants survived, the 28 day outcome became less consequential. Shennan and colleagues reported on outcome at 36 weeks postmenstrual age and correlated that to poor pulmonary outcome at follow-up.10 The need for oxygen at 36 weeks postmenstrual age is now the standard definition. In fact, over 25% of very low birth weight infants have an oxygen requirement at 36 weeks postmenstrual age.11 Thus chronic lung disease is often the primary outcome of many trials in this population. Chronic lung disease is associated with increased respiratory morbidity and an increased risk of neurodevelopmental impairment during childhood.12 13 Even when not chosen as a primary outcome, chronic lung disease is still an outcome that is essential to capture in clinical trials, regardless of type of intervention, given its frequency and its importance for long term outcomes. Therefore, we evaluated what proportion of systematic reviews of interventions in preterm infants published by the Cochrane Neonatal Review Group and their included trials reported on this pivotal outcome.
Methods
We considered all systematic reviews completed by the Cochrane Neonatal Review Group and published in the Cochrane Library up to November 2013. We excluded reviews that dealt solely with conditions affecting term infants. Of the remaining reviews, we examined how many involved only preterm infants and how many involved both preterm and term infants. Our main analysis focused on systematic reviews in which only preterm infants were included and thus information on chronic lung disease would be considered useful to capture and clinically important to report by default. We then recorded how many of the eligible reviews sought information on chronic lung disease as an outcome (as stated in the methods section) and how many of those further reported on chronic lung disease as an outcome in their results section.
A systematic reviews was counted as having reported a chronic lung disease outcome if at least one trial was included (that is, it was not an “empty” review), if a meta-analysis of at least one comparison was done, and at least one of the comparisons had chronic lung disease included in the list of outcomes.
For each eligible exclusively preterm systematic review that did not report on chronic lung disease in their results we extracted information on the CDSR number, the title of the review, the type of population, the interventions(s) assessed, whether they looked for chronic lung disease as an outcome (as stated in the methods section), the total number of trials in the review, the different comparisons of interventions examined in the review, and the number of trials for each comparison. For eligible reviews that also reported on chronic lung disease in the results, we extracted the same information and also extracted information for each comparison on the number of trials (and the respective total number of participants) that had information on chronic lung disease at 28 days, at 36 weeks postmenstrual age, and at any time point of follow-up.
The populations of preterm infants were divided into three broad categories: preterm infants (with unspecified respiratory condition), preterm infants with respiratory distress (but not requiring assisted ventilation), and preterm infants requiring ventilator support (infants requiring ventilator support including conventional ventilation and high frequency ventilation). Chronic lung disease is extremely important for the two latter categories, but even in the first category information on chronic lung disease would have been essential to capture to provide a complete picture about how the infants fared clinically.
Interventions were categorized based on a high order classification for type of intervention as well as specific interventions and potential modifiers. High order classifications included environment, feeding, intravenous fluid, drugs, blood transfusion, nutrition, respiratory support (various methods of delivering oxygen without ventilation), inhaled nitric oxide, surgical procedures, and ventilation. These high order topics were further refined to specific interventions. For example, the high order classification “environment” would include interventions such as body position, isolette (a type of incubator) environment, temperature, developmental interventions, and light therapy. Finally, modifiers such as dose, route of administration, and timing of intervention were added as a further refinement of the intervention.
We present descriptive statistics about the proportion of reviews, trials, and participants with outcome data on chronic lung disease as defined in various ways (oxygen requirement at 28 days postnatal age, oxygen requirement at 36 weeks postmenstrual age, oxygen requirement at any postnatal age). We also assessed whether the availability of such data differs according to type of population and according to category of interventions assessed. Significance in the N×2 tables was appraised with the Fisher-Freeman-Halton exact test.
Missing information on chronic lung disease outcomes in a systematic review might be caused by problems in the review’s data abstraction rather than because of lack of reporting in the included randomized controlled trials themselves. To examine this possibility we also randomly selected from the overall pool of included reviews 10 that did not report on any chronic lung disease outcomes and 10 that did. We then read the full text manuscripts of all the trials included in these 20 systematic reviews and identified whether or not information on chronic lung disease was available in the primary trial reports.
Results
Eligible systematic reviews
As of 6 November 2013, 302 systematic reviews had been completed by the Cochrane Neonatal Review Group and published in the Cochrane Library. These included 10 reviews (with 44 trials) that reported only on conditions affecting term infants and 118 (with 555 trials) that studied populations of both term and preterm infants (figure)⇓. We excluded these 128 reviews from our analysis as we would not necessarily expect chronic lung disease to be an outcome in them. Our analysis thus focused on 174 systematic reviews (with 1041 trials) that solely included preterm infants, which, by default, would be expected to report chronic lung disease as an outcome. Of the 174 reviews, 105 (60%) sought chronic lung disease outcomes, 79 (45%) sought and reported on some chronic lung disease outcomes, 26 (15%) sought but did not report any chronic lung disease outcomes, and 69 (40%) did not seek and did not report on any chronic lung disease.
Flow chart of examined and eligible systematic reviews that evaluated interventions in preterm infants
Availability of chronic lung disease outcomes
Among the 95 systematic reviews that did not report on chronic lung disease outcomes even though they pertained to an exclusively preterm population, 130 different comparisons were considered. Appendix 1 gives details of these 95 reviews. Of these 95 reviews, 26 with data from 103 trials had explicitly sought outcomes on chronic lung disease, but none had been reported (in two reviews there were no trials identified); these reviews included 45 different eligible comparisons with 95 trial entries. Another 69 systematic reviews (86 comparisons) with 386 trials (375 trial entries) had not sought chronic lung disease outcomes.
Among the 79 systematic reviews that reported on chronic lung disease outcomes, 116 different comparisons were considered. Appendix 2 gives details of these reviews. The 79 reviews considered 552 trials (536 trial entries used in eligible comparisons) with 78 449 participants. Data on chronic lung disease at 28 days were available from 202 trials with 35 861 participants, and data from chronic lung disease at 36 weeks postmenstrual age were available from 200 trials with 38 124 participants. Any data on any chronic lung disease outcome were available from 320 trials with 56 788 participants.
When we combined data from all 174 evaluated eligible reviews, 320/1041 (31%) of the considered trials had information on any chronic lung disease outcome.
Availability of chronic lung disease outcomes in subgroups of populations and interventions
Table 1⇓ shows the association between type of population and class of intervention and the availability of results on chronic lung disease (any data, at any follow-up). As shown there were significant differences (P<0.001 for all comparisons with Fisher-Freeman-Halton exact test) across populations and across different interventions, regardless of whether we analyzed seeking or reporting of data on chronic lung disease in systematic reviews, or reporting of chronic lung disease in specific trials. Specifically, almost all reviews (92-95%) determined chronic lung disease when the population had respiratory distress or required respiratory support, while chronic lung disease outcomes were sought in only 46% of systematic reviews in which there was no explicit mention of respiratory support or distress. Eventually, two thirds of trials in reviews of infants with respiratory support or distress reported chronic lung disease outcomes, as opposed to only 17% in other reviews. Seeking chronic lung disease ranged from 7-100% across different types of interventions, and reporting chronic lung disease ranged from 0-100% at the level of systematic reviews. Similarly, the proportion of trials with chronic lung disease outcome information varied from 0-93% across different interventions. The highest percentages, however, were seen in two interventions with few trials (blood transfusion and inhaled nitric oxide). Among interventions for which many trials were available, even in the case of ventilation interventions only 56% of the respective trials reported information on chronic lung disease.
Associations between availability of any outcome information on chronic lung disease (CLD) and population/intervention features in eligible systematic reviews and trials that evaluated interventions in preterm infants. Figures are numbers (percentage)
Chronic lung disease information available in trials but not included in systematic reviews
Ten randomly selected systematic reviews that sought information on chronic lung disease (table 2⇓) and 10 that did not (table 3⇓) included 84 trials, of which only 39 had any relevant outcome information entered in the systematic reviews. Of the 20 trials included in the reviews that did not even seek chronic lung disease as an outcome, two trials had, in fact, reported on chronic lung disease in their primary reports. Of the 64 trials in the random sample of reviews that did report some information on chronic lung disease, 46 trials reported on chronic lung disease in their primary publications, but for seven of the 46 (15%) this information had not been included in the systematic review. For nine (20%) of the 45 trials with missing CLD information in the systematic review this information could be retrieved in the primary trial reports. There were no trials that had missing information on chronic lung disease but had captured a composite outcome of all major morbidities that might have included chronic lung disease. Among the 36 other trials with missing information also in the primary trial report, the sample size ranged from 12 to 153 (mean 39) and the type of population studied was preterm with respiratory support (n=5), preterm with respiratory distress (n=2), or simply preterm infants (n=29); seven of the 36 trials had been published only in abstract or short (letter) form rather than as full articles.
In depth evaluation of 10 systematic reviews that had sought outcomes on chronic lung disease (CLD) in preterm infants: outcomes available in primary trial reports but missed in systematic reviews
In-depth evaluation of 10 systematic reviews that did not seek outcomes on chronic lung disease (CLD) in preterm infants: outcomes available in primary trial reports but missed in systematic review
Discussion
Principal findings
This examination of a large number of trials in a particular specialty through systematic reviews shows that many reviews and trials are missing information on an important outcome. Of the 174 systematic reviews on preterm infants, less than two thirds sought information on chronic lung disease, a major outcome of interest in this population; less than half reported on this outcome; and less than a third of the considered randomized trials had information on this outcome. When we limited our search to specific definitions of chronic lung disease reflecting specific follow-up, such as 28 days or 36 weeks postmenstrual age, the proportion of trials with available information was around 20%. This improved when we focused on sample size as about half of the randomized infants had available data on chronic lung disease with each of these specific outcomes and about 70% had information on some chronic lung disease outcome. This suggests that much of the missing information stemmed from small trials. Proportions of 30-50% of missing information among the totality of randomized infants, however, are still high. We also documented large differences in the availability of chronic lung disease information according to type of population and type of intervention. Nevertheless, even in trials with preterm infants with respiratory distress or support, information on chronic lung disease was missing in a third of the trials, and the missing proportion was even higher in trials in which the intervention was related to the type of ventilation used. Our evaluation also suggests that only a fifth of the missing information was available in the primary trial publications but was missed by the systematic reviews.
Placing results in context
Our findings suggest that information on a major routinely relevant outcome of interest in trials of premature infants is often not reported in randomized controlled trials and their reviews and is often not even sought. Chronic lung disease might not be considered a primary outcome, especially in preterm infants who do not have respiratory distress or in some interventions that tend to be given in relatively more stable infants. Nevertheless, even in these children and settings, chronic lung disease is a useful outcome to capture and essential in describing the overall clinical fate of the study population. One other potential explanation for our findings might be that many of these trials had small sample sizes and thus were underpowered to detect clinically meaningful differences on chronic lung disease. This outcome is so important, however, that it makes no sense not to report on it, even if the numbers of preterm infants with chronic lung disease in the trial are small or even zero. This is crucial information to assess the overall effectiveness and safety of tested interventions. For systematic reviews it would also make sense to seek this outcome information, regardless of whether single trials report it or not. Our estimates of “missingness” of this type of outcome information are probably even conservative given that we excluded many systematic reviews in which no information was given on chronic lung disease and both term and preterm infants had been mixed; for many of the trials or subgroups thereof included in these reviews, chronic lung disease would be a highly relevant outcome. Of the 118 systematic reviews that included both preterm and term infants, only 10 reported something on chronic lung disease (data not shown in detail).
Even among the trials that provide information on chronic lung disease, we noted substantial variability on whether this is defined as oxygen requirement at 28 days postnatal age, 36 weeks postmenstrual age, or with other definitions and time points. Analyses on specific time points could utilize data from only a fifth or fewer of all potentially useful trials. This creates a large gap of evidence and opportunity for the summary evidence to be distorted by selective reporting. Missing outcome information seems to affect smaller trials more prominently, and most systematic reviews in this specialty include only relatively small trials. Thus the threat of erroneous conclusions from selective reporting of small trials with favorable results is considerable.14
The significant differences in availability of data on chronic lung disease according to type of population and type of intervention are probably not surprising. Chronic lung disease is more likely to be reported when there is respiratory disease in the background of the population and for more aggressive interventions that are given in sicker infants. Even in these settings and interventions, however, the proportion of missing information was substantially large.
The evaluation of the primary trial reports from a random sample of the systematic reviews showed that most of the problem of missing information on chronic lung disease is not because systematic reviewers failed to record existing information. Perusal of the primary trial publications yielded relevant information on chronic lung disease outcomes for only 20% of the trials that had no such outcomes included in the systematic review. One other study has found that discrepancies in the outcomes listed in the protocol and full systematic review occurred in 39% of reviews on cystic fibrosis.15 Thus there is some room for improving the completeness of recording and reporting of evidence in systematic reviews. Most of the problem, however, seems to be at the primary trial level. Most of the trials analyzed here predate the advent of registration, so we cannot assess whether chronic lung disease had been considered as an outcome and had been registered but was subsequently not reported. Based on studies comparing registry entries and reported results, this scenario is possible.16 17 Regardless of whether the outcome was considered or not, however, as data are eventually not available, the end result is the same.
Limitations
Our empirical evaluation has some limitations. First, though we focused on a single type of outcome, other outcomes are also important for preterm infants—for example, mortality or retinopathy of prematurity or necrotizing colitis or cerebral bleeding. There is evidence from other studies that mortality is an outcome that is often not reported in clinical trials, despite its obvious major importance and recommendations to report it routinely.18 Among other non-lethal outcomes, chronic lung disease is the most common among the severe complications in this population, and evidence on other major outcomes is likely to be even more fragmented and less often available. Second, we focused on a specific discipline because it allowed us to examine in a standardized way the availability of information on a single major outcome across hundreds of reviews and over a thousand trials. Similar analyses might need to be performed in other disciplines, but probably no other discipline has more extensive comprehensive coverage by systematic reviews than what exists for evidence on interventions in preterm infants.
Conclusions and policy implications
Despite these caveats, our empirical evaluation shows the large extent of missingness of information on a prime outcome of interest in randomized controlled trials across an entire medical discipline. Such incomplete outcome information constitutes avoidable research waste2 and a potential threat to the validity of the evidence. Randomized trials can be difficult and expensive to conduct, and there is an ethical imperative to make the best use of them.7 Not collecting or not reporting key information on the most important outcomes is a lost opportunity. Key questions might remain unanswered or misleading answers might be obtained if the reported data on these outcomes differ systematically from the data that are not collected or not reported.
Efforts to remedy the situation could be both retrospective and prospective. Retrospective retrieval of missing unreported information might be difficult, if not impossible, especially for trials published many years ago, unless some other record of the trial has been kept by default, such as at a regulatory agency.19 Prospectively, we suggest that each medical discipline should reach consensus on whether certain outcomes exist that are important regardless of which intervention is being tested. The Core Outcome Measures for Efficacy Trials (COMET) initiative has performed a systematic review20 of Delphi, consensus, and other efforts to derive core outcome sets across diverse disciplines, and a website is tracking such efforts.21 The concept of core outcomes has been probed in diverse populations and diseases, ranging from rheumatologic conditions to asthma.22 23 24 Once core outcomes are chosen there should be a requirement that trials should routinely collect information and report it in a standardized fashion. The results could be published in clinical trial reports and/or in registries of results such as clinicaltrials.gov.25 Of course, the use of standardized outcomes does not negate the need to report all major morbidity events, regardless of whether they consist of well known prespecified core outcomes or infrequent and unanticipated outcomes. Such an approach can markedly improve the availability of data and the standardization of information for clinical outcomes that matter.
What is already known on this topic
Selective outcome reporting is a major threat to the validity of results from both clinical trials and systematic reviews
Empirical studies comparing protocols against published studies suggest that many outcomes are selectively reported
What this study adds
An evaluation of all Cochrane systematic reviews in an entire specialty showed that less than half of the reviews on preterm infants reported on chronic lung disease (the most serious outcome in this population), and data were given for only 31% of the trials
When outcome data on chronic lung disease were not reported in the systematic reviews, usually they were also missing in the primary trial reports
Notes
Cite this as: BMJ 2015;350:h72
Footnotes
Contributors: All authors participated in the design and interpretation of the study analyses and results. JPAI had the original idea and drafted the first protocol and the manuscript. All other authors contributed to the drafting of the paper. CMO and RFS collected data and along with JPAI, they organized and analyzed the data. JPAI is guarantor.
Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. METRICS is funded by a grant from the Laura and John Arnold Foundation.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. JH is co-editor of the Cochrane Neonatal Review Group.
Ethical approval: Not required.
Data sharing: All raw data are available from the authors on request.
Transparency statement: The guarantor affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies are disclosed.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.