Increasing Precision of Terms Related to Reproducibility and Replicability

Search OpenAlex for reproducibility studies and you will find many papers. Search for replicability studies. Same. Search for robustness studies. Same. A lot of research has been done on these topics during the past decade. 

Reviewing the papers, something else will become obvious too. These terms—reproducibility, robustness, replicability—are not used consistently. If you want to refer to conducting the same analysis on the same data as an original paper, which one are you referencing? Most call that reproducibility, but some call it replicability. Some of the projects that we led at the Center for Open Science (COS) were called “Reproducibility projects” but they involved collecting new data to independently test the original claim. Most call that replicability, not reproducibility. And, in our mission statement we say we aim to increase “openness, integrity, and reproducibility of research”—but reproducibility there isn’t referring specifically to conducting the same analysis on the same data, it is referring to a more general sense of credibility of findings.

It is kind of a mess.

As part of the Systematizing Confidence in Open Research and Evidence (SCORE) program, we adopted and refined emerging standards for these terms to improve the precision and clarity of what we mean. Also, we introduced a new term—repeatability—so that we can avoid dual use of the same words for different purposes.

We summarize the definitions that we have adopted in a new MetaArXiv preprint. We will adopt these definitions in our work and communications to improve our clarity and consistency. Clearer language enables clearer thinking, fairer comparisons of evidence, and better decisions.

Repeatability-Related Terminology

The short preprint provides more detail, but to summarize:

• Reproducibility (reproduction): investigate the same question with the same data and the same analysis.
• Robustness: investigate the same question with the same data and justifiable alternative analyses.
• Replicability (replication): investigate the same question with independent data.
• Repeatability: an umbrella term for the three checks above.
• Credibility or trustworthiness: the confidence warranted by the methods, bias control, precision, and calibration of claims to evidence. Recognizing that most science contains uncertainty, credible and trustworthy are not synonyms of true. Credibility and trustworthiness refer to whether findings, claims, or evidence are worth taking seriously.
  
  

Why This Matters

• Distinct signals, not synonyms. Each assessment provides different information about repeatability. Treating them as interchangeable can mislead.
• A sign of maturation. Research areas often begin with inconsistent language and converge as evidence and practice improve. Agreeing on working definitions is part of that progress.
• More than semantics. To anticipate a finding from the SCORE program, these repeatability measures are also empirically distinct. Successful reproduction does not, by itself, imply successful replication, and that repeatability measures relate modestly and variably to other indicators of credibility.
• Credibility is broader than repeatability, or replicability, or reproducibility. Credibility also depends on many other features of the research process such as managing competing interests, grounding claims in prior knowledge, using validated methods, minimizing bias, planning for sufficient power and precision, and calibrating claims to uncertainty.
  
  

How We Will Adopt These Terms

The initial papers summarizing the SCORE program should be published in early 2026. The SCORE program included reproducibility, robustness, and replicability studies. You will see our improved consistency in using this terminology to share those findings and explore how these assessments of repeatability are related to each other.

Also, our current mission is “to increase the openness, integrity, and reproducibility of research.” In our forthcoming strategic plan, we shift the mission wording from reproducibility to trustworthiness so our aims match current understanding while keeping the same goal, to advance open science by strengthening the credibility of research.

As with everything in science, we anticipate that the terminology will continue to evolve as we learn more about what distinctions matter in understanding the repeatability and credibility of evidence. For now, we hope that this explicitness of definitions is helpful for you understanding what we mean when we talk about these topics. We also hope that these definitions will be useful for your own work and communication.