SCORING FOR RIGOR TABLE (TOTAL 5 POINTS):

The rigor table makes up 5 points of the total score. Those five points are split evenly among the expected rigor criteria. Scores are rounded to the nearest whole number. For each sentence that describes an expected rigor criterion, e.g. blinding, SciScore adds the fractional number of points for that criterion, and if it is unable to find a statement on blinding then this section is labeled "Not Detected" and receives a score of 0. To improve detection, please make sure that your language is clear and written in standard English.

Conditional criteria such as cell line authenticity are only included in the expected list if cell lines are detected in the Key Resources table (Table 2). Likewise, an IACUC statement is expected if an appropriate animal is detected in the Key Resources table. Currently, the field sample permit will be detected but never expected.

When organisms or human participants are detected, it is expected that blinding, group selection criteria such as randomization and inclusion/exclusion, attrition, and demographic information such as sex or gender will be present. Biological variables such as sex should inform subject and group selection.

SciScore attempts to classify papers based on the paper type to reduce the burden of requiring all criteria where it may be irrelevant, however, we tend to err on the side of caution, expecting criteria where SciScore is unsure. We do this because SciScore is primarily a tool that assists peer review by bringing attention to something that may have been omitted.

Protocol, code, and data identifiers refer to persistent identifiers (either a DOI such as DOI:10.17504/protocols.io.9gbh3sn, a URL such as https://github.com/tophat, or an accession number in a repository such as GSE145917). SciScore will then try to authenticate these. For accession numbers, SciScore will check for the identifiers’ existence in their source database. For DOIs and URLs, SciScore will check to see if these resolve. Identifiers that are validated will be displayed in blue, while dead links will be shown in red such as DOI:10.17504/protocols.io.9gbh3snr. This is intended to quickly alert the author or reviewer to potential problems with a website or a typo in an accession number. SciScore does not check the relevance of the cited identifiers, only their existence. In rare cases, a typo may still result in a valid identifier. Consequently, we wish to remind users that SciScore is not a substitute for expert review. Rather, SciScore should be used in concert with reviewers for the best results.

HOW TO GET A BETTER SCORE ON THIS SECTION

Ensure that each criterion that is expected is addressed in your manuscript (refer to what is expected in the rigor items list above). Adding more rigor criteria such as the github URL to the tools developed or a protocol DOI from your favorite protocol repository will increase points for this section.

PRO TIP:

If SciScore expects that a criterion should be filled (e.g., blinding), but you do not believe that this is relevant, address it using a negative statement. Examples:

  • No subjects were excluded from our study.

  • We did not assess whether subjects were male or female because embryos were not genotyped.

  • Experimental subjects were not randomized into groups because this was deemed irrelevant to this study.

  • Experimenters were not blinded to the subject's genotype because knockout mice were visibly different from controls.

  • We did not check for sample sizes using a power analysis because our study does not report statistics on between groups or within group variables.

  • No technical replication was completed because the Sasquatch was visible only once.

Possible Problems: SciScore does not recognize my sentence as fulfilling the criterion. In some cases such as power analysis, there are a surprisingly small number of example sentences in the published literature. This is a serious problem for science, but also for SciScore because text mining analysis depends on seeing lots of syntax patterns. Take a look at the sentences above, these syntax patterns were tested and should be recognized. Writing similar sentences positive or negative should enable SciScore to recognize your sentence.

Tip for reviewers: If you see the word Sasquatch in the manuscript, consider rejecting the paper.

SCORING FOR RESOURCES TABLE (TOTAL 5 POINTS):

The total for the entire Key Resources table is 5 points with scores rounded to the nearest whole number. Each resource that is detected in this section is included in the score. For each valid RRID detected with matching metadata (e.g. catalog number or name), full points are awarded. Because a single resource can often be described in a variety of ways, SciScore utilizes fuzzy matching to correctly link resources with their corresponding RRIDs. In cases where multiple resources and RRIDs are listed out in a single sentence, authors should verify that the resources and RRIDs are correctly matched as SciScore is not perfect. Partial points are awarded if SciScore detects resources where a suggestion can be made, or if an RRID does not resolve properly. Therefore, the way to maximize the points from this section is to add RRIDs and proper citations that include vendor names, catalog numbers, lot and version numbers into the methods section of the manuscript for every key resource used.

How to get a better score on this section Ensure that each antibody, mouse, cell line (etc) has an accompanying RRID. SciScore will point out sentences or parts of a table, where these items are located. Adding more RRIDs for additional antibodies that SciScore did not find does not hurt your score, it improves it. Pro Tip: Use the catalog number and vendor to improve the probability that the antibody name and RRID will be recognized as the same item.

Common Problem: Antibody name shows up in one sentence the next “sentence” contains the RRID so SciScore thinks that my RRID is alone and there is an unidentified antibody. This can happen when a sentence is broken in your document, a return character or another invisible symbol may be at fault. Check your manuscript.

Common Problem: Two antibodies, e.g., anti-mouse-nAChR antibody and nAChR antibody, are the same, but SciScore puts both on the list. To help SciScore, use the same name throughout the methods section. There should be no need to use the RRID twice.

Other Entities Table (not included in scoring)

The other entities table (Table 3) contains:

  • Statistical tests

  • Oligonucleotides

  • Additional problems, if found

Sentences containing entities of interest are shown in the leftmost column, while the specific statistical tests and oligonucleotides detected are displayed in the right column. Again, none of the criteria in the “other entities table” impact the overall score.

General notes on interpretation of text mining results

Incorrect sentences: SciScore is a machine learning, text analysis tool, and it is therefore susceptible to making two types of errors: false positives and false negatives.

FALSE NEGATIVES:

The most common error occurs when the algorithm fails to detect a sentence that contains a rigor criterion or a resource, such as an antibody. False negatives generally occur either because the sentence is complex or in a less common syntax pattern. Generally, simple sentences in clear standard English are simpler to process and result in fewer false negatives. If a truly complex sentence structure is required to describe reagents, a table may help not only SciScore, but also human readers. If an RRID is detected in a sentence, SciScore will be triggered to take a look at the sentence, which may have been skipped otherwise.

FALSE POSITIVES:

This type of error occurs when SciScore falsely detects something including cases where a sentence does not contain an antibody, but the algorithm asserts that this sentence does have an antibody. If many resources are used and all have RRIDs, a single false positive will not reduce the score substantially, if at all. But if only 1-2 resources are used or if the false positive is in the cell line or organism category, it will trigger scoring for cell line authentication and other rigor criteria, which can reduce your SciScore needlessly. False positives are most often seen in the tools portion of table 2, as the algorithm detects company names, where it should not. We try to minimize these false positives using several strategies, however, they still occur in roughly 3-5% of cases. If this impacts your score, please contact our team (http://sciscore.com) and include the sentence where SciScore made the error. While we can't fix the score, SciScore can certainly learn from its mistakes for improved performance next time around.