Orbit logo

Outcome Reporting Bias in Trials


Background

Selective reporting of information in trials may occur for many aspects of a trial. When the selection process occurs in relation to outcomes, we refer to it as selective outcome reporting [Williamson and Gamble 2005]. However, when outcome selection is driven by the significance or effect size for that outcome, this particularly dangerous form of poor reporting (or non-reporting) is called outcome reporting bias (ORB). Specific harm outcomes may also be subject to ORB if the harm had been measured, but data were presented or suppressed in a way that would mask the harm profile of particular interventions.

ORB has been a major area of research over the last decade which has been led by researcher at the University of Liverpool. This work has been funded mainly through two MRC Methods Research panel grants (ORBIT I and ORBIT II (Outcome Reporting Bias in Trials)) and has been recently described as ‘ground-breaking research’ [Reeves 2014].   Research has included defining ORB [Hutton and Williamson 2000], providing empirical evidence for its existence [Dwan et al 2013], estimating the prevalence and impact of ORB (for both benefit [Kirkham et al 2010] and harm outcomes [Saini et al 2014]), understanding the reasons why the non-reporting of outcomes might lead to bias [Smyth et al 2011] and statistical strategies for adjusting for ORB in a review meta-analysis [Williamson and Gamble 2007Kirkham et al 2012Copas et al 2013]. Selective reporting is not just limited to outcomes. Selective reporting of analyses is another potential form of bias if the inclusion of the analyses in the report is based on the result to those analyses [Dwan et al 2014]. Moreover, selective outcome reporting bias has not only been found in primary research but it has also been found in systematic reviews [Page et al 2013].

Some key findings to our research to date include:

  1. Statistically significant outcomes in a trial are more likely to be fully reported compared with non-significant  outcomes (range of odds ratios: 2.2 to 4.7).
  2. A third of Cochrane reviews contained at least one trial with high risk ORB suspicion for the review primary benefit outcome.
  3. From interviews with trialists, trialists were found to have reported the outcomes in a biased way in about a quarter of trials considered. A general lack of consensus regarding the importance of outcomes in particular clinical settings was evident and impacted on trial design, conduct, analysis and reporting.
  4. ORB in a primary review harm outcome was evident in nearly two thirds of all primary studies included in systematic reviews.
  5. Discrepancies in analyses between publications and other study documentation is common.