Why reproducible manuscripts?
In theory, reproducibility improves accountability to third party reviewers, allows researchers to more easily build on existing work, and can improve internal quality control and organization. In practice, reproducibility in practice tends to stop with the code, at which point authors take the results and manually paste it into the manuscript document. That puts a firewall in the pathway between code/data and manuscript. For researchers, that manual step can be error prone and difficult to manage with complicated and changing data and code. Hidden pathways and manual steps also make it more difficult for third parties to connect the statistics and figures reported with the underlying code, impeding reproducibility.
We ran into this directly while working on SCORE, a large multi-institution project assessing reproducibility in the social and behavioral sciences. We wanted to break down that firewall with a practical solution to a complex analysis and manuscript workflow, ultimately settling on taking standard reproducibility a step further than standard practice with a full reproducible manuscript pipeline.
Reproducible manuscripts are uncommon but by no means new. A push-button reproducible manuscript pipeline should run all analyses from original code and combine it with manuscript text to produce the full manuscript, with minimal or no manual intermediate steps. Unfortunately, existing workflows often trade off ease of collaboration to achieve reproducibility.
For SCORE, we developed a workflow for more collaborative reproducible workflows. The SCORE manuscripts had both a large active author team and a complicated multi-layered data structure, with active quality checking and correction occurring simultaneously with analysis and manuscript writing. The collaborative pushbutton reproducible manuscript workflows we developed were not only desirable from a scientific integrity perspective, but potentially the most practical, time-efficient, and reliable solution.
Workflows for push-button reproducible manuscripts
There are two main workflows to achieve a push-button reproducible manuscript: markup-style documents with embedded code and data, and placeholder-style writing text with placeholders or tags and code that fills them in. Worked examples of each style are available here.
Markup-style approach
The markup-style approach is the most straightforward way to produce a reproducible manuscript, particularly with well established open source tooling like R Markdown. Running the script “knits” everything together to output a populated document. Markup fits well with GitHub-style collaboration and version control, making it a good way for small code-adept teams to work.
If your eyes glazed over reading “R Markdown” and “GitHub,” you have seen the biggest hurdle with collaboration in markup-style approaches. The primary writing and editing platforms are code editors and git, rather than platforms like Google Docs with the conveniences and collaboration of document writing. Collaborative manuscripts do not work when the technical requirements exclude a large portion of the team. For SCORE projects, where we had large teams collaborating on writing, that was a non-starter.
Placeholder style approach
The placeholder style of workflow mirrors the standard way of manually updating the manuscript with calculated values and figures, but where the “update” step is automated. The manuscript is written and edited in document editing software, but with placeholder tags for where calculated objects should be placed. The code now has two tasks: generating the analysis values for those placeholders, and knitting the manuscript text and analysis outputs together by replacing all the placeholder tags with their calculated values.
The major advantage of the placeholder approach is that all the text editing takes place directly in standard dedicated document editors for all collaborators, while the coding can be handled separately. The tradeoff is that it adds inherent complexity, and there are limited workflows that populate a Google Doc with calculated data.
We tried a few approaches for SCORE based on a Google Doc writing platform and R-based analysis and manuscript knitting. Our initial implementation involved outputting calculated values into a Google Sheet and using a Google Apps script to replace all placeholders in the Doc with those values. The main drawbacks to this approach were that they involve at least three manual steps: running the analysis code, running the Google Apps script, and manually updating the figures. These are solvable problems, but ultimately we switched to a different tact altogether.
Our second approach, available as an R package, involved converting the Google Doc to a more easily manipulatable MS Word document, and replacing all placeholder stats and figures in the new .docx file. This enabled a full single step push-button analysis and knit, albeit with some minor formatting issues due to document conversion.
Some lessons learned
Working in a fully reproducible manuscript mode enables a slightly different writing and coding style. Separating study and reporting design from the actual data required a change in writing habits more akin to a preregistered table shell. Over time we learned to break the shortcutting habit of hard-writing tentative results to start with and back filling the “real” code later. We may know with near absolute certainty that there are 62 journals in our dataset, but writing it as a placeholder “{n_journals}” instead of “62” forced us into the mode of reproducibility first, left fewer errors, and even identified a few places where our assumptions were wrong.
Reproducible manuscripts also changed how the analysis code was being written, largely due to it being so closely tied to the manuscript text. This incentivizes more readable, organized, and accurate code, since the connection between statistical results and narrative conclusions becomes more transparent. It also facilitates error correction and major updates, since the costs of changes to the data and code are greatly reduced.
Of course, none of this was frictionless; our internal workflows and collaboration styles needed to adapt to this kind of writing, and building non-standard tooling took quite a bit of time and mistakes. However, given the complexities of the SCORE project, we have little doubt that the effort was worth it.
What is next?
Manuscript reproducibility presents some unusual prospects for more confirmation-oriented work. A prewritten reproducible manuscript might encourage both better planning and interpretation, while reducing the scope for hidden post-hoc analysis along the garden of forking paths. An entire reproducible manuscript and analysis code producing the results of the placeholders can be written, packaged, and registered before data are available. We might even code in different conclusions texts based on different scenarios for what our results end up being, and/or use simulated data for design and testing. Going a step further, the reproducible prewritten manuscript could be peer reviewed for in-principle acceptance, transforming an extremely thorough preregistration to an extremely thorough registered report.
The SCORE papers are a demonstration that push-button reproducible manuscripts work with modern collaborative document editing, alleviating a barrier in the way of their adoption. The biggest feasibility hurdle remaining is developing mature, sustainable, and accessible workflows that more seamlessly integrate modern document editing (e.g. Google Docs) with community preferred coding platforms (e.g. R).
We hope that our proof of concept R package and worked examples can help kickstart that process. This R package for Google Docs or MS Word-based reproducible manuscripts in R is derived directly from the working versions of two SCORE papers as a standalone package. While we are not planning on maintaining or expanding this package, it is open for anyone’s use without restriction, and we hope it is useful for other researchers. There are other comparable tooling with similar ideas (e.g. trackdown R package and eLife’s Executable Research Article) designed around different workflows, and hopefully these will continue to get refined with greater community interest.
Whether reproducible manuscripts can (or should) be adopted en masse is a different question, involving considerations of data complexity and collaboration styles. The SCORE paper focusing on reproducibility, “Investigating the reproducibility of the social and behavioral sciences” demonstrates that we are a long way away from wide adoption of code and data sharing, much less full manuscript reproducibility. That said, if the technical hurdles are minimized, there may only be a small step from code/data reproducibility to push-button reproducible manuscripts.
While reproducible manuscripts are currently rare, our experience suggests that they could become a valuable core practice in our increasingly complex world of science. How do we make push-button reproducible manuscripts and sharing practices rewarding enough for researchers, journals, and funders? Could this practice be more efficient, maybe even time-saving, for researchers in addition to rigor and integrity issues in the long run? When is it unnecessary extra complication, and can we further reduce barriers to entry?
We are excited for more early adopters to lead the way in adapting collaborative reproducible manuscripts into our practices. Please let us know your own experiences and thoughts, we would love to hear from you! Join the conversation on Bluesky (find me at @whaleactually.com or the COS account at @cos.io), or email me at noah@cos.io.