The Protocol That Contradicts Itself

A registered protocol is read at least twice by people who are not its authors. The first time is by a journal editor or peer reviewer who opens the registration record alongside the submitted manuscript to see whether the review answered the question it registered. The second, increasingly, is by anyone running an overview or umbrella review, who uses the registration to decide whether the published review can be trusted on its own terms. AMSTAR 2 makes this explicit: its second critical domain asks whether the review methods were established before the review was conducted, and whether any departures were justified (Shea et al., 2017).

Having read a large number of registered protocols in exercise and health sciences over the past year, mostly on INPLASY and PROSPERO, the striking thing is how rarely the problem is a bad plan. The far more common problem is a record whose parts do not agree with each other. The eligibility criteria describe one design and the comparator field another. Two independent screeners are declared and one author is listed. The dissemination field contains the keyword list. None of this is fraud, and most of it is the product of registering in a hurry. But a reviewer who finds three internal contradictions in the registration will read the manuscript with a different disposition than one who finds none, and that disposition shapes the rest of the review.

What follows are the ten patterns that come up most often, grouped by the kind of problem they create. Each is drawn from records in the public registries; the specifics have been generalised so that no individual registration is identifiable.

Promises the team cannot keep

1. Dual independent screening declared, one author listed. The most frequent single inconsistency. The methods field says that two reviewers will screen titles, abstracts and full texts independently, with a third resolving disagreements. The author list has one name, or two. Reviewers notice this because duplicate screening is the item they are most often asked to check (PRISMA 2020 item 8; Page et al., 2021), and because it is impossible to deliver as declared. If the team is genuinely one person, the honest protocol says so and describes what is being done instead: single screening with a second pass on a random sample, or single screening with the exclusions at full text verified by a collaborator. A reviewer will accept a disclosed limitation. A reviewer will not accept a declared method that the author list makes impossible.

2. More than one appraisal tool named for one study design. Protocols sometimes list RoB 2, the Newcastle-Ottawa Scale, the Jadad score and "the Cochrane risk of bias tool" together, apparently to cover every possibility. This reads as not having decided. The tool follows the design: RoB 2 for randomised trials, ROBINS-I for non-randomised studies of interventions, QUADAS-2 for diagnostic accuracy, and so on. Naming one tool per included design, and saying which outcome the assessment applies to, is all that is needed. Naming four for one design is a signal that the appraisal plan was written before the eligibility criteria were settled.

3. Analysis thresholds the literature will not support. A protocol pre-specifies network meta-analysis for every stratified subgroup, with a feasibility threshold of ten or more trials per network. In a field where the whole evidence base is thirty trials, the stratified networks will never form, and the protocol has committed the team to analyses it will have to abandon. The same pattern appears in pairwise work: eight pre-specified subgroups and a sensitivity analysis on each, in a literature where the main analysis has six studies. The fix is to pre-specify the analyses you expect to be able to run, name the threshold at which each will be attempted, and say what will be reported if the threshold is not met.

Parts that disagree with each other

4. Comparator "None" in a protocol that describes a comparative design. The comparator field is often left as "None" or "Not applicable" out of habit, including in protocols whose eligibility criteria restrict inclusion to case-control studies or controlled trials. A case-control study has a comparator by definition. A reviewer reading the two fields together cannot tell which one the authors meant, and will ask.

5. The title describes one population and the eligibility criteria another. A title promising adults with a condition, with criteria that admit adolescents, or a title about athletes with criteria that do not mention training status. Titles are written first and criteria refined later, and the title is rarely revisited. When the published review's title still does not match the included population, it looks like scope drift rather than an editing lapse.

6. Objectives that name study designs the eligibility criteria exclude. "To synthesise evidence from randomised and observational studies" in the objectives, and "randomised controlled trials only" under study design. The reviewer's question is simple: which one did you do? If the answer is that observational studies were excluded, the objectives have over-promised; if they were included, the criteria have under-reported. Either way the record is not describing the review.

7. Numbered criteria with gaps, and text pasted into the wrong field. Inclusion criteria running 1, 2, 3, 5, 6 tell the reader a criterion was deleted and not renumbered, which is harmless on its own but prompts the question of what was removed and why. More consequential is content in the wrong field: an intervention field that describes the search strategy, a dissemination field that holds the keyword list, a results-style count of records screened appearing in a protocol for a review that has not yet searched. The last of these is the one that matters, because it tells a reviewer the review was already under way when the record was created, whatever the stated review stage says.

8. A search string that stops mid-expression. Registries truncate long strings, and INPLASY in particular does not display full strategies even when authors supply them, so a string that ends abruptly is not in itself a defect and should not be read as one. What is a defect is a record that says the strings "will be improved in the final paper". That is a statement that the search has not been designed, and it undercuts the one thing a prospective registration is meant to establish: that the search was planned before the results were known. If the full strategy cannot fit in the registry, deposit it on OSF or Zenodo and put the link in the record. PRISMA-S expects the full strategy for every database to be reproducible from the published review (Rethlefsen et al., 2021), and the protocol is the first place it should exist.

Commitments that quietly disappear

9. Grey literature declared and not delivered. Trial registries, dissertation databases, conference proceedings and "contact with experts in the field" are declared in the information sources of a large proportion of registered protocols. A much smaller proportion of the resulting reviews report any records from these sources, and most do not mention them at all. Editors do compare the two lists. A protocol should name only the sources the team will actually search, and the review should report every source named, with the number of records retrieved, even when that number is zero. Declaring ClinicalTrials.gov and WHO ICTRP and then not searching them is a deviation; declaring them and reporting zero relevant registrations is a finding.

10. English-only, justified by resources alone. A language restriction is defensible. What reviewers object to is the justification "due to resource and translation constraints" with no acknowledgement of what the restriction might cost. In fields where a substantial share of trials is published in Chinese, Spanish or Portuguese, an English-only review can miss a third of the eligible evidence and will be asked about it. The protocol should say which languages are excluded, estimate what the literature in those languages looks like (a single search in the relevant database with no language filter is enough to say whether the problem is real), and either commit to a sensitivity analysis or explain why none is planned.

Why reviewers weight these so heavily

None of the ten patterns above is a methodological error in the sense of producing a biased estimate. Their weight comes from what they signal. A record that promises two screeners and lists one author, or declares six grey-literature sources that never appear in the review, suggests that the registration was written to satisfy a requirement rather than to describe a plan. Once a reviewer forms that view, every subsequent judgement call in the manuscript is read less charitably, and a reviewer reading uncharitably finds more to object to.

The reverse is also true. A protocol that is modest, internally consistent and specific about its limitations earns a reading in which ambiguities are resolved in the authors' favour. That is worth more than any individual item on the list.

There is also a practical consequence that arrives later. PRISMA 2020 item 24c asks authors to describe and explain any amendments to the registered protocol, and most journals now ask for a registration number at submission. Every inconsistency in the record becomes something the published review must either reconcile or explain. A protocol that contradicts itself guarantees a longer deviations section, written at the point in the project when there is least time to write it.

A one-hour check before registration

PRISMA-P sets out which items a protocol should contain (Shamseer et al., 2015); it cannot tell you whether those items agree with each other. Most of the above can be removed with a single structured read, done by someone other than the person who wrote the record, in the hour before it is submitted. The check does not require methodological expertise; it requires reading the record as a stranger would.

Read the title, objectives and eligibility criteria together and confirm they describe the same population, intervention, comparator and designs. Read the author list against every declared process that needs more than one person, and change the process or add the person. Read each named appraisal tool against the designs the criteria admit, and remove any tool that does not match a design. Read the analysis plan against a realistic count of the studies the field contains, and cut any analysis that cannot plausibly reach its own threshold. Read every field heading against its content. Read the information sources as a list of commitments, and strike any the team will not search. Read the language statement and ask whether it would satisfy a reviewer from a country whose literature it excludes.

Then read the review stage field and the search date field together, and if the search has already run, say so plainly and give the reason. A declared retrospective registration with a reason is routine. An undeclared one, discovered from a record count that should not exist yet, is the one finding on this list that moves a reviewer from sceptical to suspicious.

An hour spent on this before registration saves considerably more than an hour at revision, and it produces a record that a reviewer can read without forming a theory about what the authors were really doing.

References

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71

Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., & PRISMA-S Group. (2021). PRISMA-S: An extension to the PRISMA statement for reporting literature searches in systematic reviews. Systematic Reviews, 10, 39. https://doi.org/10.1186/s13643-020-01542-z

Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & PRISMA-P Group. (2015). Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. BMJ, 349, g7647. https://doi.org/10.1136/bmj.g7647

Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008. https://doi.org/10.1136/bmj.j4008