Core Outcome Sets
=================

* James Webbe
* Ian Sinha
* Chris Gale

*   Outcomes research
*   Measurement

## The importance of outcomes in research

‘Clinical trials are only as credible as their outcomes.’1 

Clinical trials guide clinical practice. They do this by demonstrating beneficial or detrimental effects of an intervention on patients: the outcomes of the trial. The reliability of trial findings can be reduced by poor research practices such as inadequate allocation concealment or lack of blinding.2 The CONSORT standards were developed to alleviate the problems caused by poor trial reporting.3 Problems can also be caused by the selection, measurement and reporting of outcomes for clinical trials:

*   Heterogeneity: When trials addressing similar clinical questions measure different outcomes, or the same outcome at different points or using different tools, it is difficult to combine results in systematic reviews or meta-analyses. A recent study of neonatal Cochrane reviews demonstrated that half were inconclusive, with heterogeneity of outcomes an important contributor.4 

*   Selective reporting: Selective reporting of outcomes on the basis of the results is widespread.5 This can introduce bias to such a degree that after selective reporting is addressed, the conclusions of Cochrane reviews have been shown to change from *evidence of benefit* to *no evidence of benefit*.6 Preregistration of trials, and of the outcomes to be reported, is increasingly practised to minimise this.7 

*   Relevance: When the outcomes of a trial are not relevant to research users (patients, parents and clinicians) statistically significant results may be clinically meaningless and such trials will not translate into improved clinical care. In paediatrics, outcomes in clinical trials are often chosen to meet the needs of researchers,8 with children rarely involved in outcome selection.9 

A solution to these problems is the development of Core Outcome Sets.

## Core Outcome Sets

A Core Outcome Set is an agreed, standardised group of outcomes to be reported by all trials within a research field.10 There are a number of benefits from Core Outcome Sets:

*   They reduce heterogeneity and facilitate meta-analysis.

*   They reduce the risk of reporting bias, and thus ensure that all trials contribute outcome data to meta-analyses.

*   By involving a wide range of stakeholders, such as patients, parents and health professionals, it is more likely that clinically relevant outcomes are identified.

Core Outcome Sets should not stifle innovation. Rather than prescribing outcome selection, they simply represent a minimum group of outcomes that should be reported, commonly as secondary outcomes. An example of a recent paediatric Core Outcome Set is shown in box 1. In this case, the core outcomes have been identified and further work is in progress to determine how they should be measured in a standardised fashion.

### Box 1 An example of a paediatric Core Outcome Set

An example of a Core Outcome Set [11]:

The recommended therapeutic core outcome measures for paediatric outpatients with acute diarrhoea include:

*   diarrhoea duration

*   degree of dehydration

*   need for hospitalisation

*   proportion of patients recovered by 48 hours

*   adverse effects associated with therapy.

## Identifying a Core Outcome Set

The first step in developing a Core Outcome Set is to define the scope: population, condition and types of intervention.10 This information can be used to search existing databases to ensure a suitable Core Outcome Set does not already exist. The COMET (Core Outcome Measures in Effectiveness Trials) initiative maintains a database of Core Outcome Sets.12 If no appropriate Core Outcome Set is found, developing one involves two steps:

1.  identifying potential outcomes

2.  ranking outcomes to determine a ‘core’ set.

Developing a Core Outcome Set should involve stakeholders including patients, parents, family members, clinicians, allied healthcare professionals and researchers. Patients and public involvement is increasingly common: 89% of current and ongoing Core Outcome Set registered on the COMET database involved them.13 Meaningful involvement of stakeholders is commonly done through a multiprofessional steering group.

### Identifying potential outcomes

Identification of outcomes used in clinical research is commonly by systematically reviewing clinical trials within the relevant clinical field to extract primary and secondary outcomes. Such reviews often identify considerable outcome heterogeneity14 within a specialty, further supporting the need for a Core Outcome Set.

A variety of methods have been used to identify outcomes considered important by other stakeholders (patients, parents, children and other health professionals). These include primary qualitative research—in-depth interviews or focus groups15—undertaken for the purpose of the Core Outcome Set, and systematic reviews of published qualitative research.16 Such approaches are required because clinical trials have been repeatedly shown to miss outcomes considered important by these other stakeholder groups.17 18 

### Ranking outcomes to determine a ‘core’ set

Different techniques have been used to reach consensus regarding which outcomes are ‘core.’ These range from relatively informal methods, such as semistructured group discussion at a workshop or meeting, to more formal group work, such as use of the Nominal Group technique.15 Currently the most common method used is modified Delphi technique13 ; this allows the views of a large number of geographically diverse stakeholders to be integrated quickly and efficiently.

In a Delphi process,19 all potential outcomes that have been identified are sent to a panel of representatives from identified stakeholder groups, such as parents, patients and health professionals, usually as an online survey. Each individual then has the opportunity to rank each outcome as important or unimportant to them. After each round, any outcomes that are universally felt to be unimportant are removed. In the following round, participants are given feedback on how other stakeholders ranked the remaining outcomes and have the opportunity to alter their ratings on the basis of this feedback. The aim of the Delphi technique is that after several rounds consensus will be reached on a set of outcomes that all stakeholders agree are important. This process is illustrated in figure 1 (Additional File 1).

![Figure 1](http://ep.bmj.com/https://ep.bmj.com/content/edpract/103/3/163/F1.medium.gif)

[Figure 1](http://ep.bmj.com/content/103/3/163/F1)

Figure 1 
Illustration of the steps in the Delphi process. (Figure courtesy of the COMET Initiative).

## Conclusions

Core Outcome Sets are being developed in paediatrics (box 2) and more widely to improve outcome reporting in clinical trials. Where high-quality Core Outcome Sets have been developed, their use is increasingly mandated by research funders20 and journal editors.21 Within paediatrics and neonatology, Core Outcome Sets provide a mechanism to increase research involvement and participation by groups not traditionally represented (such as parents and patients), reduce research bias and strengthen the evidence base for treatments.

### Box 2 Core Outcome Sets in paediatrics currently in development (according to COMET database)

*   Core Outcomes in Neonatology (COIN project)

*   Outcomes in clinical trials of bronchiolitis

*   Development of a Core Outcome Set for use in infants with Hirschsprung’s disease

*   Systematic review of tools to measure outcomes for young children with autism spectrum disorder

## Footnotes

*   Contributors CG conceived of this article. JW, CG and IS drafted and created the final manuscript.

*   Funding This article was funded as part of aClinician Scientist Fellowship awarded to CG by the MRC (MR/N008405/1)

*   Competing interests None declared.

*   Provenance and peer review Commissioned; externally peer reviewed.

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: [http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/) 

## References

1.   Tugwell P , Boers M . OMERACT conference on outcome measures in rheumatoid arthritis clinical trials: introduction. J Rheumatol 1993;20:528–30.
    
    [PubMed](http://ep.bmj.com/lookup/external-ref?access_num=8478864&link_type=MED&atom=%2Fedpract%2F103%2F3%2F163.atom) 
    
    [Web of Science](http://ep.bmj.com/lookup/external-ref?access_num=A1993KU05800024&link_type=ISI) 

2.   Wood L , Egger M , Gluud LL , et al . Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008;336:601–5.[doi:10.1136/bmj.39465.451748.AD](http://dx.doi.org/10.1136/bmj.39465.451748.AD) 
    
    [Abstract/FREE Full Text](http://ep.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMzYvNzY0NC82MDEiO3M6NDoiYXRvbSI7czoyMzoiL2VkcHJhY3QvMTAzLzMvMTYzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

3.   Rennie D . CONSORT revised--improving the reporting of randomized trials. JAMA 2001;285:2006–7.[doi:10.1001/jama.285.15.2006](http://dx.doi.org/10.1001/jama.285.15.2006) 
    
    [CrossRef](http://ep.bmj.com/lookup/external-ref?access_num=10.1001/jama.285.15.2006&link_type=DOI) 
    
    [PubMed](http://ep.bmj.com/lookup/external-ref?access_num=11308440&link_type=MED&atom=%2Fedpract%2F103%2F3%2F163.atom) 
    
    [Web of Science](http://ep.bmj.com/lookup/external-ref?access_num=000167995900032&link_type=ISI) 

4.   Willhelm C , Girisch W , Gottschling S , et al . Systematic Cochrane reviews in neonatology: a critical appraisal. Pediatr Neonatol 2013;54:261–6.[doi:10.1016/j.pedneo.2013.03.002](http://dx.doi.org/10.1016/j.pedneo.2013.03.002) 
    
    [CrossRef](http://ep.bmj.com/lookup/external-ref?access_num=10.1016/j.pedneo.2013.03.002&link_type=DOI) 
    
    [PubMed](http://ep.bmj.com/lookup/external-ref?access_num=23602385&link_type=MED&atom=%2Fedpract%2F103%2F3%2F163.atom) 

5.   Williamson PR , Gamble C , Altman DG , et al . Outcome selection bias in meta-analysis. Stat Methods Med Res 2005;14:515–24.[doi:10.1191/0962280205sm415oa](http://dx.doi.org/10.1191/0962280205sm415oa) 
    
    [CrossRef](http://ep.bmj.com/lookup/external-ref?access_num=10.1191/0962280205sm415oa&link_type=DOI) 
    
    [PubMed](http://ep.bmj.com/lookup/external-ref?access_num=16248351&link_type=MED&atom=%2Fedpract%2F103%2F3%2F163.atom) 
    
    [Web of Science](http://ep.bmj.com/lookup/external-ref?access_num=000232555400005&link_type=ISI) 

6.   Kirkham JJ , Dwan KM , Altman DG , et al . The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365.[doi:10.1136/bmj.c365](http://dx.doi.org/10.1136/bmj.c365) 
    
    [Abstract/FREE Full Text](http://ep.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE2OiIzNDAvZmViMTVfMS9jMzY1IjtzOjQ6ImF0b20iO3M6MjM6Ii9lZHByYWN0LzEwMy8zLzE2My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

7.   De Angelis C , Drazen JM , Frizelle FA , et al . International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med 2004;351:1250–1.[doi:10.1056/NEJMe048225](http://dx.doi.org/10.1056/NEJMe048225) 
    
    [CrossRef](http://ep.bmj.com/lookup/external-ref?access_num=10.1056/NEJMe048225&link_type=DOI) 
    
    [PubMed](http://ep.bmj.com/lookup/external-ref?access_num=15356289&link_type=MED&atom=%2Fedpract%2F103%2F3%2F163.atom) 
    
    [Web of Science](http://ep.bmj.com/lookup/external-ref?access_num=000223861300015&link_type=ISI) 

8.   Sinha IP , Williamson PR , Smyth RL . Outcomes in clinical trials of inhaled corticosteroids for children with asthma are narrowly focussed on short term disease activity. PLoS One 2009;4:e6276.[doi:10.1371/journal.pone.0006276](http://dx.doi.org/10.1371/journal.pone.0006276) 
    
    

9.   Sinha I , Jones L , Smyth RL , et al . A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children. PLoS Med 2008;5:e96.[doi:10.1371/journal.pmed.0050096](http://dx.doi.org/10.1371/journal.pmed.0050096) 
    
    [CrossRef](http://ep.bmj.com/lookup/external-ref?access_num=10.1371/journal.pmed.0050096&link_type=DOI) 
    
    [PubMed](http://ep.bmj.com/lookup/external-ref?access_num=18447577&link_type=MED&atom=%2Fedpract%2F103%2F3%2F163.atom) 

10.  Williamson PR , Altman DG , Blazeby JM , et al . Developing core outcome sets for clinical trials: issues to consider. Trials 2012;13:132.[doi:10.1186/1745-6215-13-132](http://dx.doi.org/10.1186/1745-6215-13-132) 
    
    

11.  Karas J , Ashkenazi S , Guarino A , et al . Consensus Group on Outcome Measures Made in Paediatric Enteral Nutrition Clinical Trials (COMMENT). A core outcome set for clinical trials in acute diarrhoea. Arch Dis Child 2015;100:359–63.[doi:10.1136/archdischild-2014-307403](http://dx.doi.org/10.1136/archdischild-2014-307403) 
    
    [Abstract/FREE Full Text](http://ep.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTI6ImFyY2hkaXNjaGlsZCI7czo1OiJyZXNpZCI7czo5OiIxMDAvNC8zNTkiO3M6NDoiYXRvbSI7czoyMzoiL2VkcHJhY3QvMTAzLzMvMTYzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

12. COMET database. [http://www.comet-initiative.org/studies/](http://www.comet-initiative.org/studies/) Accessed 23 Nov 2016.
    
    

13.  Gorst SL , Gargon E , Clarke M , et al . Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and User Survey. PLoS One 2016;11:e0146444.[doi:10.1371/journal.pone.0146444](http://dx.doi.org/10.1371/journal.pone.0146444) 
    
    

14.  Ioannidis JP , Horbar JD , Ovelman CM , et al . Completeness of main outcomes across randomized trials in entire discipline: survey of chronic lung disease outcomes in preterm infants. BMJ 2015;350:h72.[doi:10.1136/bmj.h72](http://dx.doi.org/10.1136/bmj.h72) 
    
    [Abstract/FREE Full Text](http://ep.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE1OiIzNTAvamFuMjZfNy9oNzIiO3M6NDoiYXRvbSI7czoyMzoiL2VkcHJhY3QvMTAzLzMvMTYzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

15.  Gargon E , Gurung B , Medley N , et al . Choosing important health outcomes for comparative effectiveness research: a systematic review. PLoS One 2014;9:e99111.[doi:10.1371/journal.pone.0099111](http://dx.doi.org/10.1371/journal.pone.0099111) 
    
    

16.  Brunton G , Gale C , Webbe J . Qualitative evidence synthesis in core outcome set development: a case of added value. Published in poster form at COMET VI [http://neoepoch.com/presentations/](http://neoepoch.com/presentations/) (Accessed 23 Jan 2017).
    
    

17.  Serrano-Aguilar P , Trujillo-Martín MM , Ramos-Goñi JM , et al . Patient involvement in health research: a contribution to a systematic review on the effectiveness of treatments for degenerative ataxias. Soc Sci Med 2009;69:920–5.[doi:10.1016/j.socscimed.2009.07.005](http://dx.doi.org/10.1016/j.socscimed.2009.07.005) 
    
    [CrossRef](http://ep.bmj.com/lookup/external-ref?access_num=10.1016/j.socscimed.2009.07.005&link_type=DOI) 
    
    [PubMed](http://ep.bmj.com/lookup/external-ref?access_num=19647357&link_type=MED&atom=%2Fedpract%2F103%2F3%2F163.atom) 
    
    [Web of Science](http://ep.bmj.com/lookup/external-ref?access_num=000270642900016&link_type=ISI) 

18.  Goodare H , Smith R . The rights of patients in research. BMJ 1995;310:1277–8.[doi:10.1136/bmj.310.6990.1277](http://dx.doi.org/10.1136/bmj.310.6990.1277) 
    
    [FREE Full Text](http://ep.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMTAvNjk5MC8xMjc3IjtzOjQ6ImF0b20iO3M6MjM6Ii9lZHByYWN0LzEwMy8zLzE2My5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

19.  Dalkey N , Helmer O . An experimental application of the DELPHI Method to the use of experts. Manage Sci 1963;9:458–67.[doi:10.1287/mnsc.9.3.458](http://dx.doi.org/10.1287/mnsc.9.3.458) 
    
    [CrossRef](http://ep.bmj.com/lookup/external-ref?access_num=10.1287/mnsc.9.3.458&link_type=DOI) 
    
    [Web of Science](http://ep.bmj.com/lookup/external-ref?access_num=A1963CBG1200008&link_type=ISI) 

20. NIHR. COMET database launched to encourage use of core outcome sets in health research. [http://www.rds.nihr.ac.uk/latest-news/comet-database-launched-to-encourage-use-of-core-outcome-sets-in-health-research/](http://www.rds.nihr.ac.uk/latest-news/comet-database-launched-to-encourage-use-of-core-outcome-sets-in-health-research/) (Accessed 23 Jan 2017).
    
    

21. CROWN Core outcomes in Women’s Health (CROWN) Initiative. The CROWN Initiative: journal editors invite researchers to develop core outcomes in women’s health. Hum Reprod Update 2014;20:465–6.
    
    [CrossRef](http://ep.bmj.com/lookup/external-ref?access_num=10.1093/humupd/dmu029&link_type=DOI) 
    
    [PubMed](http://ep.bmj.com/lookup/external-ref?access_num=24914173&link_type=MED&atom=%2Fedpract%2F103%2F3%2F163.atom)