Charlottesville, VA - The Robert Wood Johnson Foundation (RWJF) has awarded a grant to the Center for Open Science (COS) to enhance the development of automated confidence evaluation of research claims.
COS and its collaborators will extend their work from the Systematizing Confidence in Open Research and Evidence (SCORE) program, administered by the U.S. Defense Advanced Research Projects Agency. The SCORE program demonstrated the potential of using algorithms to evaluate claims on a large scale efficiently. With funding from RWJF, COS enters a research and development phase to revolutionize research claim assessment by providing scalable and valid algorithmic methods.
“The SCORE project is a key part of COS’s mission of improving the rigor, reproducibility, and openness of research by creating scalable tools for assessing research credibility,“ said Tim Errington, COS’s Senior Director of Research. “With the support of the Robert Wood Johnson Foundation, new research, development, and testing will extend the technologies that emerged from SCORE. These will be a benefit to all stakeholders – including research communities, policymakers, practitioners, and the broader public. These technologies have great potential to advance the reproducibility of research and to assess the confidence one should have in the claims generated.”
At scale, confidence assessment technology has the potential to empower research producers in identifying evidence weaknesses, alert research consumers to potential misinformation for closer scrutiny, aid policymakers in allocating additional resources for confident decision-making, and assist practitioners in making informed assessments to guide urgent actions.
The SCORE program supplements existing research evaluation methods, including human judgment, evidence aggregation, and systematic replication. Under the RWJF’s grant, COS, in partnership with researchers at the University of Melbourne and The Pennsylvania State University, will advance the development of these efforts in four ways:
This approach aims to make confidence assessment more scalable and readily deployable. It establishes a robust evidence base to guide its appropriate use and enables automated confidence assessment to complement human processes in evaluating, prioritizing, and applying research findings. At scale, confidence assessment technology has the potential to empower research producers in identifying evidence weaknesses, alert research consumers to potential misinformation for closer scrutiny, aid policymakers in allocating additional resources for confident decision-making, and assist practitioners in making informed assessments to guide urgent actions.
Fiona Fidler, Professor of History and Philosophy of Science and co-director of MetaMelb, at the University of Melbourne, noted, “SCORE is creating new efforts that are an exciting opportunity to extend our research methodology and advancing dataset about confidence assessment that captures the diversity and depth of how humans reason about and judge evidence.”
Sarah Rajtmajer, Assistant Professor of Information Sciences and Technology at The Pennsylvania State University and lead of one of the four AI teams from SCORE noted, “Through the last four years, we have been developing AI capable of scoring the replicability of claims in a published scholarly paper. Our four teams have pushed the state of the art in information retrieval, natural language processing and understanding, knowledge graphs, interpretable machine learning, and artificial markets. But the more we dig in, the more clear it has become that we still have a long way to go before we have an AI that can reliably make these incredibly nuanced assessments. Through this new funding from RWJF, we can continue to develop algorithms to support confidence assessment of scholarly work and expand collaboration and integration with a much broader community of researchers.”
About the Center for Open Science
Founded in 2013, COS is a nonprofit culture change organization with a mission to increase openness, integrity, and reproducibility of scientific research. COS pursues this mission by building communities around open science practices, supporting metascience research, and developing and maintaining free, open source software tools, including the Open Science Framework (OSF). Learn more at cos.io.
Contact: Alexis Rice
email@example.com or (434) 207-2971
Responsible stewards of your support
COS has consistently earned a Guidestar rating of Platinum for its financial transparency, the highest rating available. You can see our profile on Guidestar. COS and the OSF were also awarded SOC 2 accreditation in 2022 after an independent assessment of our security and procedures by the American Institute of CPA’s (AICPA).
We invite all of our sponsors, partners, and members of the community to learn more about how our organization operates, our impact, our financial performance, and our nonprofit status.