Five Errors That Survive Peer Review

Peer review is good at some things and poor at others. It reliably catches a missing risk of bias assessment, an absent flow diagram or a search run in one database. It is much less reliable at catching errors that only appear when two parts of a paper are read against each other, or when the paper is read against its own registered protocol. Reviewers are usually given the manuscript, sometimes the PRISMA checklist, and almost never the time to open the registration record and compare it field by field.

Comparing published reviews with their registered protocols is a useful exercise for exactly this reason. It shows what the formal process lets through. Across a run of such comparisons on recently published reviews in clinical and health sciences, most papers were broadly concordant with what they had registered. The errors that did appear were rarely methodological in the sense of a wrong model or a biased estimator. They were errors of consistency and disclosure, and they fell into five recurring types.

None of the examples below identifies an individual paper. The patterns are what matter.

1. The abstract and the results disagree on how many studies were included

This is the most common single error, and the most embarrassing, because it is visible to anyone who reads the first page and the flow diagram. An abstract reports 24 included studies; the results section, the flow diagram and the characteristics table all show 12. Or the abstract gives a pooled sample of 8,412 participants and the forest plots sum to 6,930.

The usual cause is version drift. The abstract is drafted early, often from a provisional analysis, and is not updated when the included set changes at full-text screening or when studies are merged as duplicate reports. Co-authors read the abstract once and the results many times. Reviewers do the reverse: they read the abstract first and form their judgement of care from it.

The error matters beyond appearances. Abstracts are what get indexed, quoted in later reviews and extracted into overviews. An umbrella review that takes the study count from the abstract will propagate the wrong number. PRISMA 2020 has a separate abstract checklist for this reason (Page, McKenzie, et al., 2021), and item 9 of that checklist asks for the number of included studies and participants.

The check. Before submission, list every place the paper states a count: abstract, first paragraph of results, flow diagram, characteristics table, each forest plot, the discussion and the conclusion. Put them in one column. They should all agree, or the disagreement should be explained (for example, a study contributing to some outcomes and not others). This takes ten minutes and catches nearly every instance. Our earlier note on making the flow diagram add up covers the arithmetic inside the diagram itself.

2. Registered outcomes that are never mentioned again

A protocol registers six outcomes. The published review reports four. The remaining two do not appear in the results, are not listed as deviations, and are not mentioned in the limitations. There is no statement that they were sought and not found.

This is the evidence-synthesis version of selective outcome reporting, and it is the reason registration exists. In most cases the cause is innocent: the outcome turned out to be reported by one study, or none, and the team simply dropped it. But a reader cannot tell an outcome that was sought and found empty from an outcome that was analysed and found inconvenient. Outcome reporting bias in primary trials is well documented (Kirkham et al., 2010), and reviewers who know that literature will draw the same inference about a review.

The fix is cheap. Every registered outcome needs a sentence in the results, even when the sentence is "No included study reported time to return to work" or "Two studies reported quality of life using incompatible instruments, so no pooled estimate was calculated; results are described narratively in Supplementary Table S4". An empty outcome is a finding. An absent outcome is a question.

The check. Open the registration record. Copy the primary and secondary outcomes into a list. For each, find the sentence in the results that reports it. Any outcome without a sentence goes either into the results with an explanation or into the deviations statement with a reason.

3. "Data availability: not applicable" on a meta-analysis

A large share of published meta-analyses carry a data availability statement reading "Not applicable", "All data are included in the article", or "Available from the corresponding author on reasonable request", with no supplementary data file.

For a primary study with no new data, "not applicable" can be accurate. For a meta-analysis it almost never is. A meta-analysis generates a dataset: the extracted effect estimates, sample sizes, variances, risk of bias judgements and coding decisions for every included study. That dataset is what another team needs in order to reproduce the pooled estimate, check an extraction or update the review. Forest plots are not a substitute, because they show rounded values and do not show the arm-level data or the conversions that produced them.

PRISMA 2020 item 27 asks authors to report which data, analytic code and other materials are publicly available and where (Page, Moher, et al., 2021). A statement of "not applicable" answers that item incorrectly rather than incompletely.

The check. Deposit the extraction spreadsheet, the analysis script and the risk of bias table in a public repository (OSF, Zenodo, Figshare or the journal's own supplementary system) and cite the DOI in the data availability statement. If some data cannot be shared, for example because it was obtained from trial authors under conditions, say so explicitly. "Extracted study-level data and analysis code are available at [DOI]" is the sentence to aim for.

4. The registration number appears only in the cover letter

The review was registered. The number is on the submission form and in the cover letter. It does not appear anywhere in the published article.

This sounds trivial and is not. PRISMA 2020 item 24a asks for the registration information, including the register name and registration number, to be given in the review itself. Readers, overview authors and AMSTAR 2 assessors look for it in the methods. If it is not there, an assessor applying AMSTAR 2 (Shea et al., 2017) critical domain 2 has to search for the record independently, and may conclude, wrongly, that the review was not registered. The cost to the authors is a lower appraisal rating for a review that met the standard.

A related variant is a registration statement that gives the registry but not the number ("The protocol was registered with PROSPERO"), or gives a number with a typographical error that resolves to a different record. Both are worth checking by following the link exactly as printed.

The check. Put the registry name and full identifier in the first paragraph of the methods, and in the abstract if the journal allows. Paste the identifier from the published paper into the registry search and confirm it opens the right record.

5. Methods that cannot be reproduced from the protocol

The most consequential of the five, and the hardest to spot. The registered protocol describes one method; the published review uses another; and the paper does not say a change was made.

Examples seen in practice: a bibliometric mapping review registered to build its co-occurrence network from author keywords, titles and abstracts, and published with a network built from 30 features the authors coded inductively during extraction. A protocol registering a random-effects model with DerSimonian-Laird estimation, and a paper using a fixed-effect model with no explanation. A protocol specifying RoB 2 and a paper reporting Newcastle-Ottawa scores for the same randomised trials.

In each case the published method may well be better. That is not the issue. The issue is that a reader working from the protocol could not reproduce the review, and a reader working from the paper alone does not know the plan changed. PRISMA 2020 item 24c asks authors to describe and explain any amendments to the information provided at registration. Our note on what counts as a protocol deviation sets out how to write that section. The common failure is not writing it at all.

The check. Read the protocol's methods fields side by side with the published methods section, one paragraph at a time: eligibility, sources, selection, extraction, risk of bias, effect measures, synthesis model, heterogeneity, sensitivity analyses, certainty assessment. Mark each as identical, changed and declared, or changed and undeclared. Every item in the third category needs a sentence.

Why these get through

All five errors share a property: each is invisible unless two documents, or two parts of one document, are read together. Peer reviewers read sequentially and are not usually asked to audit consistency. Editors check that a checklist was submitted, not that every item on it is true. Authors read their own paper so often that they stop seeing it.

There is also an incentive gap. A reviewer who spots an inconsistent study count gets nothing for reporting it except a longer review report. A methodological objection, by contrast, feels like a contribution. So the consistency errors are the ones most likely to be left for someone else, and in practice that someone is the first reader who tries to use the review.

The consequences arrive later. Post-publication, these errors generate correction notices, PubPeer threads and lower AMSTAR 2 ratings in overviews. A correction for an abstract count is not a retraction, but it attaches to the paper permanently and is the first thing a sceptical reader sees.

A forty-minute consistency read

Most of this can be caught with a single structured pass before submission, ideally by a co-author who did not draft the paper.

Ten minutes on counts: every stated number of studies and participants, in one column, reconciled. Ten minutes on outcomes: every registered outcome matched to a results sentence. Five minutes on the data statement: is there a DOI, and does it open a file with the extracted data? Five minutes on registration: is the identifier in the methods, and does it resolve to the right record? Ten minutes on methods: protocol and paper side by side, every change either declared or reverted.

Forty minutes is less time than any revision round, and it removes the class of errors that peer review is least likely to catch and post-publication readers are most likely to find.

References

Kirkham, J. J., Dwan, K. M., Altman, D. G., Gamble, C., Dodd, S., Smyth, R., & Williamson, P. R. (2010). The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ, 340, c365. https://doi.org/10.1136/bmj.c365

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71

Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … McKenzie, J. E. (2021). PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ, 372, n160. https://doi.org/10.1136/bmj.n160

Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008. https://doi.org/10.1136/bmj.j4008