--- title: Decision Markets Predict Replicability of Online Experiments description: "A new study published in Nature Human Behaviour suggests that a tool similar to a stock market could help scientists identify research findings that are likely to hold up to scrutiny. The study, conducted by a group of social scientists from around the world, tested the ability of a \"decision market\" to predict which results from previously published social science experiments would be successfully replicated." image: https://www.cos.io/hubfs/blog%20thumb%20sotck.png --- - Our sites: - [OSF](https://osf.io) - [TOP Guidelines](https://www.cos.io/initiatives/top-guidelines) - [* *Bluesky](https://bsky.app/profile/cos.io)[* *Mastodon](https://fosstodon.org/@CenterforOpenScience)[* *Linkedin](https://www.linkedin.com/company/center-for-open-science/)[* *YouTube](https://www.youtube.com/@CenterforOpenScience) - [**  Shop](https://www.redbubble.com/people/OpenScience/shop) - [**  Donate](https://www.cos.io/support-cos) [![Center for Open Science](https://www.cos.io/hs-fs/hubfs/Logos/COS%202024/cos-full-tag-h.png?width=2000&height=293&name=cos-full-tag-h.png "Center for Open Science")](https://www.cos.io) ** ** # Decision Markets Predict Replicability of Online Experiments November 19th, 2024, [Center for Open Science](https://www.cos.io/blog/author/center-for-open-science) Posted in: [Reproducibility](https://www.cos.io/blog/tag/reproducibility), [Research](https://www.cos.io/blog/tag/research) ![](https://www.cos.io/hubfs/blog%20thumb%20sotck.png) A [new study published in *Nature Human Behaviour*](https://www.nature.com/articles/s41562-024-02062-9#Abs1) suggests that a tool similar to a stock market could help scientists identify research findings that are likely to hold up to scrutiny. The study, conducted by a group of social scientists from around the world, tested the ability of a "decision market" to predict which results from previously published social science experiments would be successfully replicated.* * The decision market focused on 41 experiments that used participants recruited through Amazon Mechanical Turk (MTurk), an online platform that connects researchers with people willing to complete tasks for pay. All of the experiments were originally published in the *Proceedings of the National Academy of Sciences* (PNAS) between 2015 and 2018. In the decision market, 162 social scientists bought and sold shares representing the likelihood that a given study's findings would replicate. The market was set up so that the studies whose shares ended up with the 12 highest and 12 lowest prices would be selected for replication. Two additional studies were selected randomly. The results suggest that the decision market was able to successfully distinguish between replicable and non-replicable findings. The percentage of studies that produced statistically significant results in the same direction as the original study was 83% for the studies in the "top 12" price category. In contrast, only 33% of the studies with the lowest decision market prices successfully replicated. "Our study provides a 'proof of concept' that decision markets can be a useful tool for identifying likely true and likely false research findings," says Felix Holzmeister, author on the paper. Overall, replications were successful for 54% of the 26 studies selected for replication. The average effect size observed in the replication studies was 45% of the average original effect size. This overall replication rate is comparable to those found in previous large-scale efforts to replicate findings from social science experiments. The study's findings have important implications for the way that science is conducted. “Decision markets, or similar tools that rely on scientists' collective wisdom, could offer a principled mechanism to prioritize research efforts and resources, ultimately increasing the efficiency and reliability of scientific progress,” says Holzmeister.  ### Recent Posts ![Center for Open Science](https://www.cos.io/hubfs/Logos/COS%202024/cos-full-tag-h-w.png) 6218 Georgia Avenue NW, Suite #1, Unit 3189 Washington, DC 20011 Email: [contact@cos.io](mailto:contact@cos.io) - [* *Bluesky](https://bsky.app/profile/cos.io) - [* *Mastodon](https://fosstodon.org/@CenterforOpenScience) - [* *Linkedin](https://www.linkedin.com/company/center-for-open-science/) - [* *YouTube](https://www.youtube.com/@CenterforOpenScience) **Join our mailing list** [![Creative Commons License](https://i.creativecommons.org/l/by/4.0/88x31.png)](http://creativecommons.org/licenses/by/4.0/) Unless otherwise noted, this site is licensed under a [Creative Commons Attribution 4.0 International (CC BY 4.0) License](https://creativecommons.org/licenses/by/4.0/). [Translation Help](https://www.cos.io/how-to-translate) | [Terms of Use](https://github.com/CenterForOpenScience/centerforopenscience.org/blob/master/TERMS_OF_USE.md) | [Privacy Policy](https://github.com/CenterForOpenScience/centerforopenscience.org/blob/master/PRIVACY_POLICY.md) - [**About**](https://www.cos.io/about) - [Team](https://www.cos.io/about/team) - [Board](https://www.cos.io/about/board) - [Partners](https://www.cos.io/about/partnership-framework) - [News](https://www.cos.io/about/news) - [Impact](https://www.cos.io/impact) - [History](https://www.cos.io/timeline) - [Finances](https://www.cos.io/about/finances) - [Compliance and Standards](https://www.cos.io/compliance-and-standards) - [Diversity, Equity & Inclusion](https://www.cos.io/about/diversity-equity-and-inclusion-at-cos) - [Careers](https://www.cos.io/careers) - [**Engage**](https://www.cos.io/engage) - [Tools](https://www.cos.io/tools) - [Training](https://www.cos.io/services/training) - [Community Building](https://www.cos.io/community-building) - [Policy Reform](https://www.cos.io/policy-reform) - **  [Donate](https://www.cos.io/support-cos) - [**Open Science**](https://www.cos.io/open-science) - [Preregistration](https://www.cos.io/initiatives/prereg) - [Registered Reports](https://www.cos.io/initiatives/registered-reports) - [**Research**](https://www.cos.io/services/research) - [SCORE](https://www.cos.io/score) - [RP: Cancer Biology](https://www.cos.io/rpcb) - [RP: Psychology](https://osf.io/ezcuj/wiki/home/) - [Events](https://www.cos.io/events) - [Blog](https://www.cos.io/blog) - [Contact](https://www.cos.io/contact-us) - [Shop](https://www.redbubble.com/people/OpenScience/shop) - [OSF](https://osf.io) - [TOP Guidelines](https://www.cos.io/initiatives/top-guidelines) **Responsible stewards of your support** [![Charity Navigator](https://www.cos.io/hs-fs/hubfs/Logos/charity-navigator.png?width=100&name=charity-navigator.png)](https://www.charitynavigator.org/ein/461496217)[![Platinum Transparency 2023](https://widgets.guidestar.org/TransparencySeal/9294495)](https://www.guidestar.org/profile/46-1496217)[![SOC 2](https://www.cos.io/hs-fs/hubfs/Logos/badge-soc2.png?width=100&name=badge-soc2.png)](https://www.aicpa-cima.com/resources/landing/system-and-organization-controls-soc-suite-of-services) COS has earned top recognition from **[Charity Navigator](https://www.charitynavigator.org/ein/461496217)** and **[Candid](https://www.guidestar.org/profile/46-1496217)** (formerly GuideStar) for our financial transparency and accountability to our mission. COS and the OSF were also awarded SOC2 accreditation in 2023 after an independent assessment of our security and procedures by the **[American Institute of CPAs (AICPA)](https://www.aicpa-cima.com/resources/landing/system-and-organization-controls-soc-suite-of-services)**. We invite all of our sponsors, partners, and members of the community to learn more about how our organization operates, our impact, our financial performance, and our nonprofit status. **Photo Stream** - - - - - - - - - - - -