OSF Changes: A Note to Users

The Center for Open Science (COS) has announced a significant change for the Open Science Framework (OSF). This post assumes that you are already familiar with the changes coming to OSF over the next several months. If you are not, read this.

This is a big change, but it is important for the effectiveness, resilience, and sustainability of OSF. Simultaneously, we recognize that this will have a significant impact on users, especially early adopters and those who have integrated OSF into their work in routine and creative ways. You put your trust and time into making OSF part of your research workflow, and now functions that are important for you may no longer be supported in OSF itself. I am very sorry that this change will require many of you to redesign established workflows. The COS team has created support materials to aid the transition. Please reach out for help when you need it. We will do everything we can to support you continuing to do your science openly.

OSF users, COS supporters, and contributors to the open science reform movement might be interested to understand more of the history leading up to this change. Part 1 of this post provides the historical rationale for OSF; Part 2 describes the implications and where we are headed.

Part 1

OSF was conceived in a failed 2007 grant proposal. Some reviewers thought it was naive because, they said, researchers don’t want to share. The idea was re-energized in 2011 by Jeff Spies as his dissertation project. We self-funded the project through my lab until receiving our initial grant in 2013 from the Laura and John Arnold Foundation to launch COS. 

Our animating vision was expansive: to build the "connective tissue" for cumulative, transparent science. We wanted to seamlessly integrate services across every stage of the research lifecycle—planning, conducting, reporting, and evaluating—so that any researcher could access all of the contents created and used in research, and play back the history of a project to evaluate or build upon it. We conceived OSF as a collaborative space for teams that naturally documented their work as it happened. And, when the teams were ready to do so, OSF would make it trivial to transition their work, and its documented history, into the open. Working openly and in the cloud was still new and unfamiliar to many; OSF could provide an easy onboarding process.

We opened OSF for public use in 2012, several months before starting COS. Open science concepts were new and different from researchers' existing work routines, so we made two key decisions to recruit innovators to shape how open science could be done with OSF. First, we did not require openness from the start. We perceived that it would be easier for researchers to adopt open science by solving challenges they faced naturally in their work, for example with collaboration and data management. Research labs are dynamic environments with people joining and departing all the time. Data and other records are often being managed inconsistently or barely at all. OSF provided a shared, cloud-based solution so that researchers would not lose their own work for their own use. If a computer exploded or a postdoc moved on, the work would still be accessible to all the other collaborators. While solving that problem on the one hand, we trivialized the act of becoming open on the other. A little button on every page invited users to turn that private content public. Click it, and a “Are you sure?” warning appeared. Click yes, and that research content was open. OSF embodied the idea that making research open should just be a decision one makes during the research lifecycle rather than a whole new set of work appended to the end of the research process. Too much of the open science system, even today, thinks of open science as a retrospective process that begins upon publication. After the researchers are done with the research, they go back and prepare to share everything they did in the years leading up to the paper. Retrospective open science is error-prone and highly burdensome: A common joke-not-joke is that the person that benefits most from preparing and sharing well documented research is yourself six months from now because you will have forgotten everything. We wanted researchers to be preparing for openness from the start with just-in-time actions to document the research. Preparing for openness during the research lifecycle would lower burden and improve rigor, first for oneself, then for the whole community. With a prospective approach to open science, sharing is a trivial act that occurs along the way.

Second, we gave researchers a flexible environment to experiment and develop workflows that suited them. This made it easier for early adopters to try out OSF without disrupting their current work style, and it made OSF an experimental testbed to learn more about open practices. For example, we wanted to promote preregistration, but what should people preregister? How much would that need to vary across methods and disciplines? We didn’t know. OSF allowed researchers to preregister whatever they wanted, and norms emerged bottom-up. My own lab started preregistering studies and iteratively learned what we forgot to document and what documentation was an unnecessary burden. From that, we developed standard operating procedures. Other labs had similar experiences. OSF’s flexibility enabled the early adopters to develop the solutions themselves, grounded in their work and direct experience. We observed a positive feedback loop. Users' experimentation informed how we updated OSF to make the insights available to other users in repeatable, standardized workflows. For example, in addition to a very flexible registration option, we introduced preregistration templates developed by the community. These gave users the option to do research planning with structured guidance.

These two decisions embodied our belief that a successful open science reform movement required bottom-up experimentation, norm setting, and responsive infrastructure. I hoped that OSF would always maintain a dual-role: One part would be a highly-flexible project management space for experimentation and addressing idiosyncratic needs; another part would provide structured, repeatable, scalable, easy-to-use workflows that reflected the extracted lessons from the experimentation. As OSF became integrated with other research tools for planning, data acquisition, data analysis, authoring, and reporting, the space for experimentation would keep expanding. Over time, workflows across services would mature to support seamless, frictionless, accelerated research activity across the entire research lifecycle that is transparent, open to scrutiny, and reusable.

In several ways, OSF succeeded well-beyond my expectations. And, in some key ways, we failed to achieve the ambitious vision. 

On successes, champions and innovators adopted OSF and helped us improve it. Adoption rates grew rapidly and continuously. By now, there are more than 1 million registered users and many many millions more who consume public registrations, papers, data, materials, and code on OSF. There are almost 300,000 registrations, 400,000 public projects, and 150,000 preprints. In these, there are more than 29 million shared files, accounting for hundreds of terabytes of data. That’s a lot of open science.

The successes brought challenges. 

Part 2

As an organization that embraces experimentation, we created several OSF features to see what was useful for users to advance open science. Some were not that useful and were abandoned (remember comments and quickfiles?), some were highly useful and fully integrated, and some were useful to just enough users that we retained them. This made the service increasingly complex and difficult to maintain. OSF became a collaborative management space, a registry of research processes, a repository of research outputs, and a private storage environment. Each part supported distinct use cases, and none of the parts received the full investment needed to excel in features and user experience. I rationalized this situation as manageable, and necessary to achieve the big vision.

Rapid growth brought lots of needs and infrastructure management challenges to support the legitimate uses and to manage the infiltration of the illegitimate uses. I did not anticipate how much resourcing would be needed to prevent, detect, and eliminate spam. OSF’s high SEO (search engine optimization) and low barrier to share content made it an attractive target for spammers, and not just fake science spammers. Spammers set up thousands of accounts and tried to use OSF for sharing pirated movies, advertising health supplements, and anything and everything else.

Beyond spam, not all uses of OSF are helping to advance our mission to increase openness, integrity, and trustworthiness of research. For example, 63% of OSF projects, more than 650,000, have never been made public. We designed OSF to integrate private and public workflows to ease the burden of going public, and there are many legitimate reasons for some content to remain private. However, almost 200,000 users have used OSF only for private storage. Maintaining that is costly and our mission is not served by providing a free, private storage solution that never leads to open science.

With a large, flexible, complex infrastructure, we found ourselves facing an increasing maintenance burden. That meant we were spending more and more resources to maintain OSF’s existing operations and features, and fewer and fewer resources to improve it. OSF had more use cases than we had the financial resources to support effectively. The impact was highest on user experience because resources had to be prioritized for essential functions at the expense of user-facing features to improve usability. This was moving OSF into an unacceptable trajectory of decline. 

Simultaneously, in the 14 years since OSF opened, related services had been growing, maturing, and improving. For example, there are several generalist repositories that are very effective for preserving and sharing data, materials, and code such as Zenodo, Dryad, Dataverse, Vivli, and figshare. They are built and maintained specifically for this purpose, and we partner with these services through the GREI initiative. This partnership offered an opportunity for us to rethink the role of OSF to best serve the mission. If we narrow OSF to the use cases for which it is uniquely suited and impactful, we can lean on our open infrastructure colleagues and partners to fill use cases that OSF no longer supports, lower maintenance burden, and allocate resources to improving the usability of a leaner service. Researchers will have to use more tools, but each of them--including OSF--will be really good at the role it has in supporting open science.
 
This became both a response to financial reality, and a strategic opportunity. Our mission is to advance open science, not the Center for Open Science. Opportunities to increase the sustainability of the infrastructure supporting open science by reducing our own scope and elevating others is in the public interest. This action will reduce redundancy and the aggregate cost of open science infrastructure.

From the outset, we committed to OSF being public good infrastructure that would remain free for researchers to use. That meant relying heavily on philanthropic funding and institutional support. We incorporated COS as a charity and made OSF open source. We worked to develop mission-aligned business models. If what was good for the mission and what was good for the organization were ever misaligned, the former must drive decision-making over the latter.

If we were sticking to our expansive vision for OSF as a comprehensive collaborative management system, we would need to continue raising $4-5 million per year for the next several years. We generate 10 to 20% of that through sustainable yearly revenue, mostly provided by several dozen institutional members. The remainder would have to be solved with grants. That’s a lot of grant writing success, even in the best of times. We are not in the best of times.

Of course we pursue grants continuously, with mixed success. For example, the NSF has a mid-scale infrastructure program to which we applied several times. This would have addressed the entire resource gap for five years. The review process has several rounds, and we were finalists the last two times we applied with very positive external reviews. The final round is a decision internal to NSF. OSF was not prioritized for funding either time.

I am still energized by our original expansive vision, but this change is the right thing for our mission and for the open science movement. OSF will be leaner, more focused, and better connected with a flourishing open system of other tools. This change puts planned features that would benefit the open science movement on hold or outside of OSF’s remit. In the near-term, the connections between tools will be less integrated than the way that OSF has been an all-in-one solution. Our user support resources provide guidance to navigate this. With community investment in open infrastructure those connections could become seamless for users, meeting at least some of the original vision for OSF. I hope that a consortium of infrastructure services will earn collective support to achieve that, with OSF playing its part.
 
Thank you to our early adopters and our global user community. You experimented with us on how to do open science. We evolved and adapted OSF based on your early feedback. We learned an enormous amount from you about the practicalities of transparency and rigor. The flexibility of OSF was costly to maintain, but instrumental for gaining understanding of how researchers could integrate open science into their work across the research lifecycle. Those insights have spread into other services, training materials, reporting guidelines, and policy frameworks.

The system of open infrastructure and tools supporting science has improved dramatically since 2012. Services like Zenodo, Internet Archive, OpenAlex, DataCite, R, and many more, provide essential and unique functions. Nevertheless, the challenges for public goods infrastructure remain significant. It is our responsibility to be good stewards of the resources granted to us by donors and organizations by recognizing our limits and orienting ourselves to be as complementary as possible with other effective services. Our mission to improve the openness, integrity, and trustworthiness of research remains the same. This shift centers OSF and COS on where we are best able to serve that mission and accelerate discovery. 

Even with our reduced scope, we retain the challenge of raising funds to improve the OSF and keep it open and accessible as a public good. Individual donations are incredibly helpful as resources that can be allocated wherever the need is greatest. And, for the sustainability of these public goods resources, you can advocate to your university to become a member. We look forward to continuing this work and earning your support. 

Thank you for your work to improve your science, and science as a system.