Connecting the Science of Science: Metascience Conference 2021

The Metascience 2021 Conference was a global virtual gathering to connect the study of science across disciplines, methodologies, and regions, and was an initiative of the Center for Open Science (COS), the Association for Interdisciplinary Meta-Research and Open Science (AIMOS), and the Research on Research Institute (RoRI). Over a two week period, hundreds of researchers and stakeholders sharing, debating, and advancing understanding of the latest topics in the scientific process. Presenters and attendees included representation from every part of the research community. The conference provided an opportunity to engage across disciplinary and regional silos to better understand what activities and reforms were occurring in one domain, and how they might or might not translate to application in other domains.  

To get involved and contribute your metascience research visit: Metascience Collection, Metascience Registry, MetaArXiv Preprints

While there are hundreds of hours of presentations, conversations, and debates that you can view here, we asked our COS colleagues to share some of their favorites below. 

Marcy Reedy, Community Manager, Policy

Covering science: Assessing and Representing Uncertainty, Credibility, and Reform

Christie Aschwanden (FiveThirtyEight), Monya Baker (Nature), Richard Harris (National Association of Science Writers ), and Kathleen Hall Jamieson 

Respected journalists discussed lessons learned throughout their career. Specifically, the discussion included the importance of credibility and covering science from an unbiased position. Next the panel discussed uncertainty and the need to convey an evolving understanding of science. Finally, the panel discussed reform and the need for science journalism to adjust to a social media world. In sum, it was emphasized that journalism should “let science be science” and strive to cover all of its twists and turns. 

Nick Fox, Research Scientist, Metascience

Registered Reports

Chris Chambers, Stavroula Kousta, Anne Scheel, Courtney Soderberg, David Mellor

This session was broken into three main parts: concept, action and evaluation. “Concept” provided a complete overview of Registered Reports, or a study proposal is reviewed before the research is undertaken and guaranteed publication. Using Registered Reports mitigates the problem of “results-dependent publishing” since the review occurs before data are collected, and creates an independent process that eliminates publication bias. “Action” described how Nature Human Behavior rolled out of the Registered Report formats, how many submissions they see per year on average, and the pros and cons from a journal perspective of the Registered Report format. Lastly, “Evaluation” included two presentations on the research around Registered Reports. The studies showed a discrepancy in reporting of statistically significant findings between papers that were and were not Registered Reports. Further, research also showed that independent reviewers perceived that papers that had been Registered Reports were higher quality and more rigorous compared to those publications that were not Registered Reports.   

Lesley Markham, Program Manager, Policy

After excellence: Why it’s time to diversify the criteria we use to fund and evaluate research.

Lisette Jong, Cameron Neylon, Zena Sharman, Stephen Pinfield

In this session, the presenters argued that while the academy's notion of 'excellence' underpins research funding, evaluation frameworks, and publishing decisions, it can lead to incentivizing undesirable behaviors such as hyper-competition and exaggerated claims at the expense of rigorous, reproducible research. Rewarding 'excellence' can concentrate funding to people and institutions who are already well-resourced and can uphold historically colonial and Western systems. The perpetualization of such systems can marginalize and undermine contributions from Indigenous and LGBTQ+ researchers and other groups underrepresented in research. The presenters suggested that researchers, institutions, and funders should promote a wider understanding of knowledge generation and contribution to the research ecosystem and cease to rely on excellence as the only measure of success in academia.

Beatrix Arendt, Program Manager, Operations

Structural racism in biomedicine

Aletha Maybank, Carole J. Lee, Kenneth Gibbs

This session focused on structural racism and health inequities in the medical field, and how inequities in education, publishing, and funding have specifically affected communities of color. The speakers sought to call out how certain dominant narratives can actually undermine health narratives. Even when efforts are made to “level the playing field”, racial disparities are only further absorbed into racist structures rather than mitigated. More deliberate efforts to cultivate diversity need to be made by continuing investment at all levels of professional development and within organizations, by creating initiatives and establishing strategic plans that embed racial justice and advance health equity. Organizations and institutions need to advance racial equity by listening to those affected, acknowledge the past racial injustices, and act through resource allocation and implementing policies to effectively bring about change. 

Eric Olson, Institutional Product Owner, Product Team

Preregistration in the Social Sciences Empirical Evidence of its Effectiveness

Sarahanne Field, George Ofosu, Marcel van Assen, and Olmo van den Akker

One of the best parts of working with COS and the metascience communities is the emphasis that nearly everyone puts on listening to different perspectives and facilitating productive discourse. This entire event was a microcosm of this strength, and this session in particular demonstrated a willingness to put our assumptions to the test. Presenting empirical studies that systematically assessed the effectiveness of preregistrations, speakers in this session highlighted where our community efforts have paid off, where they have fallen short, and even where there have been some surprising successes that we can build on going forward.