Establishing Institutional Scores With RTI: Large Scale Analysis of Scientific Reporting Quality - published in Journal of Medical Internet Research

July 27, 2022

JMIR Publications recently published “Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality” in the Journal of Medical Internet Research (JMIR), which reported that improving rigor and transparency measures should lead to improvements in reproducibility across the scientific literature, but assessing measures of transparency tends to be very difficult if performed manually by reviewers.

The overall aim of this study is to establish a scientific reporting quality metric that can be used across institutions and countries, as well as to highlight the need for high-quality reporting to ensure replicability within biomedicine, making use of manuscripts from the Reproducibility Project: Cancer Biology.

The authors address an enhancement of the previously introduced Rigor and Transparency Index (RTI), which attempts to automatically assess the rigor and transparency of journals, institutions, and countries using manuscripts scored on criteria found in reproducibility guidelines (eg, NIH, MDAR, ARRIVE).

Using work by the Reproducibility Project: Cancer Biology, the authors could determine that replication studies scored significantly higher than the original papers which, according to the project, all required additional information from authors to begin replication efforts.

Unfortunately, RTI measures for journals, institutions, and countries all currently score lower than the replication study average. If they take the RTI of these replication studies as a target for future manuscripts, more work will be needed to ensure the average manuscript contains sufficient information for replication attempts.

In an effort to encourage reproducibility, numerous scientific organizations and journals have adopted the Transparency and Openness Promotion guidelines, which focus on establishing best practices at the level of individual journals. 

Along a similar vein, the publisher-driven Materials Design, Analysis, and Reporting framework is a multidisciplinary research framework designed to improve reporting transparency across life science research at the level of individual manuscripts.

This framework provides a consistent, minimum reporting checklist whose criteria were used, in part, to create the first RTI, a journal quality metric focusing on research methodologies and reporting transparency.

Specifically, the authors here introduce the latest version of the RTI, which represents the mean SciScore over a subset of papers, and demonstrate how it can be used to assess reporting transparency within research institutions.

While we cannot simply describe all papers scoring a “2” as not replicable and all papers scoring an “8” as replicable, as numerous fields and their subsequent best practices exist, we can state that higher scores are associated with more methodological detail and as such are likely easier to use to attempt a replication. 

###

DOI - https://doi.org/10.2196/37324

Full-text - https://www.jmir.org/2022/6/e37324

Free Altmetric Report - https://jmir.altmetric.com/details/130343509

JMIR Publications is a leading, born-digital, open access publisher of 30+ academic journals and other innovative scientific communication products that focus on the intersection of health and technology. Its flagship journal, the Journal of Medical Internet Research, is the leading digital health journal globally in content breadth and visibility, and it is the largest journal in the medical informatics field.

To learn more about JMIR Publications, please visit https://www.JMIRPublications.com or connect with us via:

JMIR Publications is a registered trademark of JMIR Publications.