TIER2's Decision Aid will provide clarity on the meaning, relevance, and feasibility of ‘reproducibility’ for researchers to aid them in identifying what type of reproducibility is relevant for their research and indicate what they must consider regarding how feasible such ‘reproducibility’ would be for them. The tool will be piloted with two researcher groups (qualitative and machine learning researchers).
Stakeholders: Researchers, publishers, funders
Objectives:
To explore through piloting to what extent the decision aid (tool) is useful and efficient.
The ultimate goal of the tool is to facilitate ‘reproducibility’ where it is relevant, as well as feasible and prevent demands for ‘reproducibility’ where it is irrelevant and/or unfeasible.
The Reproducibility Management Plan (RMP) Pilot aims to create a prototype of key thematic subjects and questions that will serve as the starting point to support reproducibility at the planning stage of research. Work involves defining what an RMP is, integrating it into the ARGOS service, and testing its effectiveness with feedback from the community. The pilot addresses researchers, beneficiaries and funders for its adoption.
Stakeholders: Researchers, research communities, funders, and service providers
Objectives:
To emphasise reproducibility activities within a research output management lifecycle;
To streamline reproducibility practices in publicly funded research projects;
To provide tools and guidance to adopt best reproducibility practices;
To generate case studies to promote a common understanding of reproducibility across various domains.
The Reproducible workflows Pilot focuses on enhancing reproducibility in life sciences and computer sciences, by adapting the SCHEMA open-source platform, using technologies like software containerisation, workflow description languages, and experiment packaging specifications to fit specific epistemic needs.
Stakeholders: Life scientists, computer scientists
Objectives: The main goal is to customise and evaluate tools/practices for reproducible workflows in life and computer sciences, with underlying objectives of extending the SCHEMA open-source platform to support reproducibility in both fields by leveraging software containerisation, workflow description languages (CWL, Snakemake), and experiment packaging specifications (RO-crate), particularly emphasising machine learning in computer science.
In this Pilot, we aim to provide a structure of well-defined checklists and templates that can help review data and code reproducibility for computational social scientists. The checklists and review templates cater for the specific needs of the three research phases, i.e., planning and data collection, process and analysis and finally sharing and archiving the research resources. It results in building trust and authority in the social science research community.
Stakeholders: Computational Social Scientists (Research Producers and consumers)
Objectives: The goal of this Pilot is to enhance reproducibility for code and data through checklists in research lifecycle.
The Pilot will develop a policy template with recommendations for funders to foster reproducible practices both in the research they fund (evaluation and monitoring) and their internal practices.
Stakeholders: Funders
Objectives: This Pilot aims to help funders improve the research quality of the projects and researchers they fund to build and sustain trustworthy outcomes.
The Reproducibility Monitoring Dashboard Pilot aims to develop tools that enable funding agencies to track and monitor the reusability of research artifacts across various projects, programs, topics, and disciplines. This auto-generated dashboard assesses the impacts of policies related to data and code sharing. Furthermore, we are establishing essential requirements to make the dashboard user-friendly for publishers.
Stakeholders: Research Performing Organisations (RPOs), Funders, Publishers and Researchers
Objectives: Develop tools that allow funding agencies to track and monitor the reusability of research artifacts:
Develop, extend, and test a suite of tools for tracking major research artifacts in Computer Sciences such as datasets, software, and methods, with a particular focus on Artificial Intelligence.
Quantify and estimate reusability indicators based on various types of artifacts.
Design and implement a dashboard that allows funding agencies to track and monitor the reusability of research artifacts created in funded projects.
This Pilot is aimed at increasing data sharing in published work. Data sharing is an important building block for increased reproducibility & transparency, but current rates of sharing are low.
Stakeholders: Publishers
Objectives: The pilot will improve our knowledge on data sharing with two activities:
- A randomised controlled trial of an intervention targeting data availability statements with the aim to increase deposition of data in trusted repositories.
- A Delphi-study to gather consensus on the most pressing issues and best paths to improve sharing of research data underlying publications.
This Pilot will co-create and test an Editorial Reference Handbook that contributes towards a common understanding and what is required to assist reproducibility and FAIRness. The Handbook, identified as a priority in a workshop with publishers, will include two components. A structured section will include educational and practical set of checks, defined by reviewing existing material, harmonising and operationalising them. Some journals have internal checks, but the type, richness and stringency vary, and there is little/no consensus among publishers. A narrative component with a general framework will help improve internal processes, defined by describing an ideal process where checks should be applied. There are a variety of internal processes, and how, when and by whom these checks are done vary, and this can also affect the results.
The Pilot includes representatives of Cambridge University Press, Cell Press, EMBO Press, F1000 (Taylor & Francis), GigaScience Press, Lancet, Oxford University Press, PLOS, Springer Nature, Wiley.
Stakeholders: Publishers
Objectives: The Handbook is set to help put the requirements of the journal data policy in action:
The planned intervention will target in-house editorial staff managing the manuscripts, but also benefit reviewers, authors on what compliance to the journal data policy may require, as well as developers to drive their service provisions to publishers.