Authors: Nicki Lisa Cole, Thanasis Vergoulis, Eleni Adamidi, Joeri Tijdink, Barbara Leitner, Allyson Lister, Susanna-Assunta Sansone
TIER2 aims to better understand the causes, consequences and possible solutions of perceived poor levels of reproductivity of research across research contexts. With a focus on social, life, and computer sciences, as well as research publishers and funders, the project aims to increase awareness, build capacity, and propose innovative solutions sensitive to varied research cultures.
Central to its approach are eight Pilot activities designed to develop, implement, and evaluate new reproducibility-related tools and practices, emphasising stakeholder engagement and collaboration throughout the project's duration until December 2025. Four of the pilots are focused on enhancing reproducibility practices in researchers, two are specifically focused on funders, and two on publishers.
Pilot implementation and assessment plans were developed during 2024 and published in Deliverable 4.2. Now, the TIER2 pilots are well underway, with each engaging appropriate stakeholders to develop and test the tools under development, with expected completion date of September 2025.
Given the diverse nature of the pilots, some are more advanced than others. Here, we showcase the development of three of the most mature pilots.
Pilot 3: Reproducible Workflows
The Reproducible Workflows Pilot is led by Athena Research Center and focuses on enhancing reproducibility in life sciences and computer sciences, by adapting the SCHEMA open-source platform, using technologies like software containerisation, workflow description languages, and experiment packaging specifications to fit specific epistemic needs.
The virtual laboratory, SCHEMA lab was released in September 2024. Computer sciences and life sciences researchers can now monitor computational tasks, submit new ones or cancel running tasks. They can also create experiments by selecting multiple tasks. There is also a first version of workflow execution in place which will be further refined in the coming months.
Throughout this process, the pilot team conducted a stakeholder survey and gathered feedback from webinar participants to inform the work.
An updated version of the SCHEMA lab will be released in July 2025 for further testing.
Pilot 5: Reproducibility Promotion Plans for Funders
The Reproducibility Promotion Plan for Funders (RPP) is led by Amsterdam University Medical Center and serves as inspiration to support research funders to develop internal practices for their funding processes, and to manage external expectations regarding reproducibility practices to inform researchers of funders’ expectations towards them. The RPP consists of a set of recommendations, focusing on different aspects of the funding lifecycle. The recommendations can support funders with varying levels of experience with and knowledge of reproducibility practices. The RPP aims to provide tools for funders to catalyse change within their funding institution and to support researchers (that are funded) to conduct rigorous and reproducible research.
The RPP was co-created by a diverse selection of the TIER2 stakeholder community that consists of funders from international and national funding organizations. The first draft of the RPP was released in November 2024, and piloting started in January 2025, which will continue for six months.
Over the coming months, the RPP will be piloted with multiple funding institutions, data will be collected along with the experiences of funders to refine and enhance the RPP, which is being continuously supplemented and updated with current best practices for reproducibility. Additionally, a survey with a larger group of funders will provide insight and feedback to the further development of the RPP.
Pilot 8: An Editorial Reference Handbook for Reproducibility and FAIRness
The Editorial Reference Handbook, led by University of Oxford, contributes towards a common understanding of what is required by scholarly journals and publishers to assist reproducibility and FAIRness in practice. Co-designed with publishers as part of the TIER2 project, the Handbook is available at https://publishers.fairassist.org.
Co-created by over 20 journals from academic and commercial scholarly publishers, the Handbook targets in-house staff managing the manuscripts primarily, but it will also benefit reviewers, authors and even those providing services to publishers by making the fundamental checks and requirements transparent and understandable.
As well as being a practical product, the Handbook is also a socio-technical pilot to improve the culture of publishing by facilitating the practice, ensuring accessible, verifiable research findings, and leading by example, influencing and informing other publishers and journals. Set to end during summer 2025, the ongoing intervention aims to document what may need to change or improve to successfully implement these checks in terms of in-house capability (e.g., needing more knowledge about how to run them), opportunity (e.g., needing support to apply them), and motivation (e.g., needing to prioritise them). In parallel, the data policies of all participating journals will be ‘audited’ for completeness and clarity, via FAIRsharing, and improved as needed. Some of the journals that have participated in co-creation will act as the positive control group in the intervention; they already have stringent data policies and their own internal guidance and will use the principles of the Handbook to validate and improve their existing methodology.
There are a wide variety of journals and publishers, and intervention participants have joined with different initial levels of reproducibility. Each has begun from their own existing level of reproducibility and has found the best way to implement the checks for their journals. This showcases both the flexibility of the Handbook for a variety of journal types and policies, as well as the interest in the Handbook for journals with varying degrees of reproducibility and FAIRness.
Over the coming months, data from the intervention will be collected and documented along with the experience of publishers and journals. The information will form an article about the co-creation and the intervention phases, to the benefit of other journals and publishing organisations, as well as authors, reviewers and other stakeholders, such as funders who are key agents of change in fostering a culture of responsible data practices.
