This much-anticipated project was initiated in response to multiple reports published from the pharmaceutical industry indicating that more than 70% of published findings could not be reproduced, impacting the ability to build upon these results to develop new therapeutic agents. Despite intense scrutiny around reproducibility in science, this project represents the first practical evaluation of reproducibility rates in the biomedical field with the potential to identify specific methods that result in reproducible studies. This project is completely open and transparent, with all process information and data made publicly available in eLife. In addition, all of the replication studies were conducted by completely independent labs that were part of the Science Exchange network of contract research organizations (CROs), academic labs, and government facilities.

Increasing openness and reproducibility increases the efficiency and quality of knowledge accumulation and application. Increasing access to the content and process of producing research outcomes increases reproducibility of the evidence, and facilitates replication and extension into new domains. False leads can be discovered and discarded more quickly and true leads can be elaborated more efficiently, accelerating the path toward solving humanity’s pressing problems.

Typically, researchers are rewarded with publication for showing the cleanest and most exciting-sounding results, even though typical research results are nuanced and more difficult to interpret. This reward system results in research papers that only show a summary of the work that was actually conducted in the research lab. Transparency of the entire research workflow allows others in the scientific community to examine, build upon, and accurately evaluate all of the content that is usually not preserved. This includes all of the precise methods used in the work, all of the digital and physical materials used to conduct the work, all of the data collected, and the details of the data analysis.

We propose that there is an efficiency crisis not a science crisis. Scientific progress is still being made, but not at the rate at which it could be -- because efficiency is being hampered by an incentive system that does not reward openness and reproducibility, two core values of scholarship. These values underpin the reality that scientific claims do not become credible based on the authority or persuasiveness of their originator. Rather, claims become credible via transparent communication of the supporting evidence and the process of acquiring that evidence.

In the present scholarly culture, openness and reproducibility are values but not standard practice. Incentives driving researchers and service providers do not promote these values. For researchers, the currency of reward is publication. Publishing frequently in the most prestigious outlets possible is the gateway to jobs, promotion, tenure, grants, and awards. Whether the research is open or reproducible is rarely relevant to publication success. Instead, publication depends on achieving novel, positive, clean outcomes. In a competitive marketplace, researchers may make choices--even unwittingly--that increase the likelihood of obtaining publishable outcomes even at the cost of their accuracy. Without transparency or efforts to evaluate reproducibility, the loss of accuracy may go undetected decreasing the credibility of the published literature.

The need to reform the research scientists’ reward system must be better communicated to the public. Funding sources and policy makers are increasingly accountable to voices of non-scientists and those who, in general, may already be biased against the complexity of accurate data reporting.

2013

We determined the 400 most-cited cancer biology papers from Web of Science in 2010, 2011, and 2012 and the top 400 most-cited papers from Scopus in 2010, 2011, and 2012. From these, we excluded clinical trials, papers requiring extremely specialized equipment, and case studies. For the remaining papers, we calculated a normalized impact rating and chose the top 16 or 17 papers from each year. See details of the paper selection method here.

Registered Reports are peer-reviewed research plans. Each Registered Report contains a complete description of the work that will be conducted and a plan for how it will be analyzed. By conducting the peer review before results are known, this process eliminates many of the biases that occur after seeing the results. Furthermore, it focuses expert evaluation of the protocols earlier in the process, where improvements can be added. For example, peer reviewers will expect that various quality control steps are conducted at critical steps in the workflow prior to measuring the final results. Finally, it improves the planned statistical analyses by addressing issues of low sample size and appropriate inference by considering the potential need to accurately interpret null results.

The final Replication Study contains all of the results specified in the Registered Report. The statistical analyses that were specified in the Registered Report are provided, and any unplanned analyses are provided as exploratory results. Unexpected but potentially exciting results that appear when exploring a dataset must be replicated with newly-conducted research before those results can be verified. A fundamental problem in the current process of scientific publishing is the unintentional presentation of exploratory results as more rigorous, confirmatory results.

See our Registered Reports page for more details and complete list of journals using this format.

A preregistration is a research plan created before conducting a study. For the Reproducibility Project: Cancer Biology, each peer-reviewed Registered Report is a preregistration. However, other researchers will often create preregistrations that are not peer-reviewed until the final study is ready to be submitted to a journal for publication, after results are known. This often occurs in clinical sciences using ClinicalTrials.gov, which is a registry where planned research studies are deposited before results are known.

A preregistration can include just details about a study plan, which includes information such as the hypotheses, experimental methods, and the design of the study. A preregistration can also include a complete analysis plan, which will specify exactly how the collected data will be analyzed to address the hypotheses. Preregistrations address two current problems with the process of science. First, they “open the file drawer” by ensuring that all conducted research eventually becomes discoverable through the registry in which the preregistration was created. Second, when they also include an analysis plan, they address questionable research practices that lead to the problems that this project demonstrates: the fact that any dataset can unintentionally show statistical significance even when there is no real effect.

The Center for Open Science is working to broaden the use of preregistrations to the pre-clinical research field to include all empirical research. See the Preregistration Challenge for details.

OSF (Open Science Framework) provides free and open source project management support for researchers across the entire research lifecycle. As a flexible repository, researchers can store and archive their research data, protocols, and materials. As a collaboration tool, researchers can work on projects privately with a limited number of collaborators and make parts of their projects public, or make all the project publicly accessible for broader dissemination. As a workflow system, researchers can connect the many services they use to streamline their process and increase efficiency.

• Structured projects: Researchers can access files, data, code, and protocols in one centralized location and easily build custom organization for each project. Such structure eliminates the need to trawl emails to find files or scramble to recover lost data
• Controlled access: OSF users can control which parts of a project are public or private, making it easy to collaborate and share with the community or just one team
• Enhanced workflow: Researchers can automate version control, get persistent identifiers for projects and materials, preregister research, and connect third party services directly to OSF.

The replication papers are available from eLife: 

All scientific studies are estimates. Does the experimental treatment have an effect compared to the control? How big was that effect and was it statistically different? Each estimate has a margin of uncertainty. The replication tells us whether the original processes and results can be reproduced within that margin. There are many reasons that two studies of the same phenomenon could yield different results, and only one of those reasons is that the original was a false positive. Other causes for different outcomes between original and replication have implications for understanding the phenomenon itself. For example, the conditions necessary to obtain the result may not be yet understood. There may be many reasons for this--poor data quality, incomplete instructions or processes, missing components. The point of replication is to increase the likelihood that these shortcomings will be resolved in advance through better transparency, planning, and communication.