Introducing the Simulation Studies Preregistration Template: Q&A with Björn S. Siepe, František Bartoš, and Samuel Pawel

In 2023, COS launched a Preregistration Template Evaluation Working Group to help curate, evaluate, and promote high-quality preregistration templates that reflect the needs of different research communities for use on the Open Science Framework (OSF). As part of this ongoing effort, COS invites OSF users to propose new preregistration templates through this submission form. 

These community-developed templates aim to improve transparency, reduce researcher degrees of freedom, and make preregistration—the practice of publicly documenting a research plan before conducting a study—more accessible and adoptable across disciplines.

Introducing the Simulation Studies Preregistration Template

Simulation studies are central to methodological research and are often used to evaluate and improve statistical methods and algorithms.

Although they are widely used as a methodological approach, simulation studies have historically lacked strong norms or guidance around preregistration and reproducibility practices. Strengthening these practices is essential to ensure their credibility and impact across scientific disciplines.

In response to this gap, researchers Björn S. Siepe (Philipps-Universität Marburg, Germany), František Bartoš (University of Amsterdam, Netherlands), and Samuel Pawel (University of Zurich, Switzerland) developed the template for preregistering simulation studies with the help of their co-authors Tim P. Morris (University College London, UK), Anne-Laure Boulesteix (Ludwig-Maximilians-Universität München and Munich Center of Machine Learning, Germany), and Daniel W. Heck (Philipps-Universität Marburg, Germany). Initially submitted through COS’s open call for community-designed templates, the Simulation Studies Template is now part of the expanding collection of preregistration resources available on OSF. It can be accessed directly on OSF Registries or via a Google doc that users can fill out and paste into their registration.

In this Q&A, the developers of the template share their motivation, the unique challenges of preregistering simulation studies, and the potential for this template to elevate research planning, documentation, and reporting practices across disciplines.

Q: What motivated you to develop a preregistration template specifically for simulation studies, and why is this an important step for methodological research?

A: Simulation studies are numerical experiments and the primary research tool for many methodological researchers. They are used to assess the properties of statistical methods and algorithms to understand, for example, how accurate their estimates are, or how well they can quantify uncertainty.

As methodologists, we often advise other researchers on how to conduct their experiments, yet we often fail to apply the same rigour to our own, as previous meta-scientific research on simulation studies has demonstrated (e.g., Siepe et al., 2024, Luijken et al., 2024, Morris et al., 2019). Combined with the flexibility that simulation studies provide, whereby new data can be simulated at any time, this leaves room for suboptimal research practices.

Given the importance of simulation studies in guiding quantitative research across a wide range of scientific fields, it is crucial to ensure their quality, informativeness, and transparency. We believe that simulation studies can be made more rigorous and that preregistration can help with that aim, much as it can help in other empirical research.

Q: The template builds on the ADEMP framework. Can you explain what ADEMP is and why it’s especially well-suited for preregistration in simulation studies?

A: ADEMP stands for the Aims, Data-Generating Mechanisms, Estimands, Methods, and Performance Measures of a simulation. This framework was introduced by Morris et al. (2019) to cover the main aspects of a simulation study and has gained a lot of traction in biostatistics since it was introduced. We believe it to be well-suited because it provides a clear and logical structure to the complex process of designing, conducting, and reporting a simulation study. 

Q: What are some of the most common researcher degrees of freedom in simulation studies, and how does this template help constrain them to improve transparency and reproducibility?

A: As simulation studies are such remarkably flexible tools, researchers have ample degrees of freedom at every step of the study pipeline. This ranges from how researchers generate their data and which methods they use, to how they use these methods and which performance measures they use to compare them. For example, it is rather easy to (inadvertently or not) change a data-generating mechanism until it favors a method of choice. Our template constrains such choices by aiding researchers in reflecting and deciding in advance about each step along the way. Of course, post-hoc changes to the plan can still be made as long as they are reported and well-justified. Preregistration makes such changes more transparent and allows reviewers and readers to put the results into context.

Q: How does the Simulation Studies Template differ from other OSF preregistration templates? What kinds of researchers or disciplines do you think will benefit most from using it?

A: This template is specifically tailored for simulation studies, which differ substantively from other empirical studies and thus from other preregistration templates. Other templates are either not detailed enough to provide enough information about simulation studies or they inquire about many details and procedures that are irrelevant for simulation studies, such as questionnaires used or demographic information. We thus believe that a tailored template is most useful to maximize the efficiency and informativeness of simulation study preregistration. 

We believe that researchers conducting simulation studies across disciplines could benefit from our template. However, it could be argued that some simulation studies, particularly 'early-stage' studies which primarily aim to determine the feasibility of a novel idea, do not require preregistration. We therefore believe that preregistration would be of most benefit to 'later-stage' research, where researchers have a good understanding of the relevant data-generating mechanisms, competitor methods, and evaluation standards. This is particularly important for simulation studies that aim to provide guidance and recommendations to applied researchers.

Beyond researchers wanting to preregister their studies, we also think that students or early-career researchers planning and conducting simulation studies can benefit from the structured outline and guidance through the process.

Q: Looking ahead, what impact do you hope this template will have on the design, reporting, and credibility of simulation studies across disciplines?

First, aside from preregistration, we hope that our template is useful for planning, reporting, and reviewing simulation studies. Simulation studies are often very complex, and it can be difficult to report them clearly and helpfully. Using our template as a guiding structure could be helpful.

Secondly, we hope that, as a preregistration template, it will improve the credibility of studies and the quality of their evidence, in the same way that it can in preregistered experimental studies (e.g., clinical trials or experiments in psychology).

Q: Is there anything else you would like to add?

We thank all our collaborators, and we are very open to receiving (critical) feedback on both our template and the idea of preregistrations for simulation studies in general, as well as other ideas on how to improve the informativeness and quality of methodological research.