Previously, we have described the efforts of the MDAR group to create a rigor and transparency checklist that will phase in as publisher or journal specific checklists phase out.
Journals, including the British Journal of Pharmacology, Nature, Cell, and many others are doing exceptional work improving rigor reporting using journal or publisher specific checklists. MDAR should be better if publishers adopt the checklist, as planned because everyone will theoretically have the same checklist.
The checklist is currently required by Science (since January of 2020 each publication is accompanied by an MDAR_reproducibility_checklist document, check it out)! Kudos to Science, this is an important step in the right direction.
Why is MDAR frightening to publishers?
Besides hearing the shrieks of authors filling out a 30+ point checklist, in order to be effective in improving manuscripts, someone has to verify the checklist, as journals with checklists in place already know! That is horrifying indeed, as there are no acceptance criteria for what constitutes a well filled out or a badly filled out checklist from the MDAR group.
Due to the high amount of work on the part of authors, Science only requires the MDAR checklist at the final "your manuscript has been accepted" stage. There, it is likely that the checklist is going to be completed without too many audible shrieks from authors. However, this misses an important opportunity to influence the manuscript while authors and reviewers can be alerted to potential problems and the checklist can be part of the conversation about the quality of the science.
...what to do...
On the one hand the checklist is labor intensive and painful to authors, on the other hand the rigor items are important to address during review.
Automation to the Rescue?
The SciScore team thought about this a lot, then stopped thinking and started doing.
SciScore, an automated tool for checking rigor criteria in manuscripts, will get a major upgrade in the coming month. The figure above is a sneak peek at the new automated MDAR checklist (using data from Hansen et al, 2020). The author created MDAR checklist is on the left and the SciScore generated checklist is on the right. These look very similar, the difference is largely in the mechanics. SciScore takes the methods section, and ~1 minute later an MDAR report is created and a completeness score is made available.
The score is a quick way for editors to assess roughly what percentage of criteria are addressed by authors in the methods section making the verification step, a little more tenable for journals.
New Rigor Criteria, now being tested:
Ethics statement: IRB / Consent / IACUC / Field sample permit
Investigator blinding
Randomization of subjects into groups
Power calculation for group size
Statistical tests used
Sex as a biological variable
Subject demographics (age / weight)
Protocols / Clinical Trials IDs verified (including both EU and US clinical trials, and protocol journals / repositories like protocols.io)
Software repository URLs verified (github, bitbucket, google code)
Data repository IDs verified (GEO, ArrayExpress, Uniprot etc)
Inclusion & Exclusion criteria / Attrition
Replication (type and number)
Cell line contamination / Authentication
Presence of or the need for all RRID types (antibodies, cell lines, plasmids, organisms etc)
We still do not know if having a software tool that validates a checklist is going to help authors and reviewers improve manuscripts, but the promise of a quick automated check may entice more journals to look at MDAR in their workflow.
Curious to see a MDAR Sample Report? Click HERE