AI peer review: what an AI reviewer can and cannot check in your paper
An AI reviewer reads your manuscript and writes what a peer reviewer would: a recommendation, scores, weaknesses and questions. Used well, it catches the problems reviewers object to most often before anyone else sees the paper. Used carelessly, it produces a confident review that misquotes you or praises references that do not exist. This guide covers what to expect and how to tell the difference.
What an AI reviewer is good at
Most "major revision" decisions come from a short list of problems that a careful reader can spot without being an expert in your exact subfield. An AI reviewer is a fast, tireless careful reader:
- Claims in the abstract or introduction that no result or citation supports.
- Missing baselines, ablations or variance, and comparisons only against weak or old methods.
- Methods that another group could not reproduce from the text: missing hyperparameters, data splits, number of runs.
- Numbers that differ between the abstract, the text and the tables.
- Unclear writing: undefined symbols, figures that cannot be read from their captions, a contribution that is hard to state.
Where AI reviewers fail
- Judging whether a result matters to your field. Significance depends on context a model often lacks; treat its significance score as a prompt to argue your case, not a verdict.
- Knowing the latest work. A model only knows what it was trained on unless it searches the literature; ask whether the tool searched, and for what.
- Checking references from memory. A language model can call a fabricated reference plausible. References should be looked up in Crossref, OpenAlex or arXiv, not judged.
- Quoting you accurately. A review that paraphrases "your" claims may be arguing with a sentence you never wrote.
How to tell a useful AI review from a confident one
- 01Every issue quotes your text
You should be able to find each quoted passage in your PDF. If you cannot, discard the point.
- 02References were looked up, not judged
The review should say which database matched each reference, and flag the ones it could not find.
- 03Reviewers were independent
Two reviews written in separate sessions that agree on a weakness are a stronger signal than one long review. Agreement between them is what to act on first.
- 04Scores come with reasons
A score of 2/5 for soundness is only useful with the sentence explaining it.
- 05It says what it could not do
A useful tool tells you when a PDF had no text layer, when full text was unavailable, or when a search found nothing.
Is it allowed?
Publisher policies on AI are mostly about two things: reviewers must not upload other people’s confidential manuscripts to AI tools, and authors must disclose AI use in writing the paper. Checking your own draft before submission is a different situation, but policies differ, so read your venue’s guidance and keep the final judgement on every change your own.
Using an AI review well
- Run it early enough to act on it: a missing baseline takes longer to fix than a sentence.
- Fix the critical and major issues first, and write down why you reject any point you disagree with; that note often becomes your rebuttal later.
- Re-check after revising, so a fix in one section did not break a number somewhere else.
- Still ask a colleague. An AI review is a first reader, not the last one.
Questions
Can AI replace peer review?
No. It can catch common, checkable problems before submission; judging significance and fit for a field remains a human job.
Is using ChatGPT to review my paper the same thing?
Not quite. A general chat model reviews pasted text in one conversation and can misquote you or invent references. A dedicated AI reviewer should quote your PDF, look references up in scholarly databases and keep reviewers independent.
Will an AI reviewer leak my paper?
Check the tool’s privacy terms: your manuscript should be visible only to you and never used to train models.