Is it acceptable to use an LLM as the second screener?

Increasingly it is done, and journals differ on whether they accept it. Disclose it plainly, report the agreement between model and human on a sample, and do not describe the process as dual independent screening without qualification, since the two screeners are not independent in the sense that phrase normally carries.

Do we need to disclose LLM use to the journal?

Assume yes. Disclosure requirements are converging: a joint AI-disclosure reporting standard from STM, COPE, the International Science Council and the Global Young Academy is in consultation, with the second round closing in October 2026, covering what counts as substantive use and how it should be documented. Check the target journal's policy at protocol stage, not at submission.

What if our pilot showed 95% recall?

Record the date, the model version and the dataset, and re-run the pilot if the platform updates before screening begins. A recall figure without a version and date attached is not a reproducible claim.

Before You Screen with an LLM, Prespecify These Five Things

5 September 20265 min

Papers on LLM-assisted title and abstract screening report workload savings between roughly 79% and 91%. Those figures are calculated retrospectively, from a classifier's performance on a fully labelled dataset, and they assume that records the model excluded are never examined again.

Real review teams examine them. Somebody spot-checks the excludes, or a second reviewer samples them, or the clinical lead asks to see everything below a confidence threshold. None of that work appears in the published saving.

Lai and colleagues (2026) proposed adjusting the metric for the proportion of model-excluded records a human still verifies. On identical data, the same model then ranges from about 0% to 91% saving depending purely on that fraction. The saving is not a property of the model. It is a property of the model plus a decision your team probably has not made.

Here are the five decisions to make in the protocol.

1. The verification fraction

What proportion of records the model excludes will a human check?

This is the parameter that determines your timeline. Decide it before screening, write it in the protocol, and calculate what it means in records. On a search returning 14,000 results with a 4% inclusion rate, the excluded pile is around 13,000 records, and a 10% verification fraction is 1,300 abstracts to screen by hand.

That number is the honest input to your project plan.

2. How the verification sample is drawn

Random sampling and borderline-band sampling answer different questions.

A random sample of excluded records estimates the overall miss rate and supports a general statement about recall. A sample drawn from records near the decision threshold is more efficient per record checked, because that is where errors concentrate, but it estimates the miss rate in the band rather than overall.

Both are defensible. Reporting one as though it were the other is not. State which you used and what it licenses you to claim.

3. The model, the version and the date

Which model, which version, accessed when.

This is not bureaucratic detail. Three frontier models given the same bounded judgement task disagree substantially: in a study published in August 2026, Claude Sonnet 4.5, Gemini 3.0 Thinking and ChatGPT 5.1 reviewed the same 50 trials and their verdicts differed significantly, with major-revision rates ranging from permissive to strict across models. If model choice moves the output that much, it is a methodological decision on the same footing as choosing a risk-of-bias tool.

Platform models also change under you. Nested Knowledge upgraded the models behind its entire AI stack on 23–24 August 2026, reporting roughly 50% faster performance with modest recall gains. Any pilot run before that date describes software that no longer exists.

4. The prompt, verbatim

Record the prompt as used, in a supplementary file. If it changed during screening, record each version and the point at which it changed.

Two reasons. Reproducibility, which is the obvious one. And the fact that prompt wording measurably affects performance in ways that are not intuitive: in a five-model ensemble screening study of 4,745 abstracts, the simplified prompt outperformed the elaborate ones. Prompt engineering folklore favours long structured instructions, and at least in that dataset the folklore lost.

5. The disagreement rule

When the model and the human reviewer disagree, what happens?

The usual answer is that the human decides, which is reasonable and worth writing down, because it means your process inherits human error rates rather than model performance. An alternative is that any disagreement escalates to full-text review, which is more conservative and more expensive.

Whichever you choose, the rule should not be invented on the day it first matters.

What the recall figures actually look like

Worth calibrating expectations. A five-model Delphi ensemble over 4,745 abstracts reached 98% recall, with work saved over sampling at 95% recall of only 13% to 42%. High recall, modest workload relief.

A fully automated pipeline benchmarked against two published meta-analyses recovered the included studies at 81.8% and 80.0% sensitivity. Those are creditable engineering results and they are well below any threshold a review would accept at screening.

Task-level benchmarking across four systems on 244 documents found no system leading on every task, with performance falling off most on interpretive analysis and synthesis across sources.

The pattern is consistent. These tools help with the first pass. None of them currently replaces it.

What to write in the methods

"Title and abstract screening was assisted by [model, version], accessed [date], using the prompt reproduced in Supplementary File 2. All records classified as potentially relevant were screened by a human reviewer. A random 15% sample of model-excluded records was independently screened by a human reviewer; [n] records misclassified as irrelevant were identified and reinstated, giving an estimated recall of [x]%. Disagreements were resolved in favour of inclusion for full-text review."

Six sentences, and a reader can evaluate the entire process.

References

Lai, H., Liu, J., Ge, L., & Estill, J. (2026). Reframing workload saving in LLM-assisted screening: A commentary on Sciurti et al. Research Synthesis Methods, 1-5. https://doi.org/10.1017/rsm.2026.10114

Tolend, M., Halabi, R., Ghaouari, K., Lau, Y. C., Alda, M., Hintze, A., ... & Ortiz, A. (2026). Impact of prompt engineering and workflow modifications on the performance of a Delphi-based automated abstract screening system in a psychiatric systematic review. BMC Medical Research Methodology. https://link.springer.com/article/10.1186/s12874-026-02981-3

Common questions

Is it acceptable to use an LLM as the second screener?: Increasingly it is done, and journals differ on whether they accept it. Disclose it plainly, report the agreement between model and human on a sample, and do not describe the process as dual independent screening without qualification, since the two screeners are not independent in the sense that phrase normally carries.
Do we need to disclose LLM use to the journal?: Assume yes. Disclosure requirements are converging: a joint AI-disclosure reporting standard from STM, COPE, the International Science Council and the Global Young Academy is in consultation, with the second round closing in October 2026, covering what counts as substantive use and how it should be documented. Check the target journal's policy at protocol stage, not at submission.
What if our pilot showed 95% recall?: Record the date, the model version and the dataset, and re-run the pilot if the platform updates before screening begins. A recall figure without a version and date attached is not a reproducible claim.