AI Reviewers That Score the Wording, Not the Science ── A preprint that defines rhetorical robustness as stability plus discrimination, and proposes SciCore, a reviewer that also judges an extracted science core

1. Same science, different wording, different score

Using large language models to review research papers is no longer a thought experiment. Some researchers run a model over a draft before submission; others use one to draft reviewer comments. That raises a simple question: does an AI reviewer respond to the science, or to the prose?

This paper takes that question head on. The authors point out that two manuscripts reporting the same experiments, results and conclusions can receive different judgments from an AI reviewer once the wording changes. If that happens, polishing the text toward whatever the model prefers pays off more than improving the research. The authors describe this as the risk of rewarding rhetorical optimization over scientific improvement.^[1]

Asking only for insensitivity to wording creates a different trap. A reviewer that gives nearly the same score to every manuscript will not be moved by rewording, but it will also fail to tell a strong paper from a weak one. The paper asks for both properties at once. The primary source is a preprint posted on arXiv and has not been peer reviewed.

2. The core idea: average a full-manuscript judgment with a science-core judgment

The authors first restate what should be measured. They call it Rhetorical Robustness and define it as a joint requirement: judgments should stay stable across rewrites that preserve the content, and they should still discriminate between different papers.

The method they propose, SciCore, can be reduced to one design choice. It produces two judgments, one from reading the full manuscript and one from reading a structured "science core" extracted from it, and averages them. The science core puts the substance, such as the question, the methods and the findings, into a fixed format, so differences in wording have less room to leak in. The full-manuscript branch still looks at the paper as written. Combining the two is meant to keep a manuscript-level assessment while lowering sensitivity to rhetoric.

To test this, the authors also built a benchmark called RobustReview. It works on full manuscripts and contains 1,260 manuscript versions, including content-preserving rewrites, and it is used to compare 30 reviewer configurations.

3. What the abstract reports

Three findings come from the benchmark. First, the authors found what they call false robustness: some reviewers that barely react to rewrites turn out to have collapsed scores across papers, meaning they give similar marks to everything. Second, ranking reviewers by agreement with human reviewers and ranking them by rhetorical robustness produce different orders. Third, a prompting protocol that tells the model to focus on content did not consistently improve robustness across the backbone models tested.

For SciCore, the authors report that in their primary comparison, run with GPT-5.5 as the backbone, it achieved a leading joint stability-discrimination profile among the benchmarked reviewers while keeping competitive agreement with humans. The abstract does not give the actual stability or discrimination values, nor the size of the gap to other configurations.

The conclusion the authors draw is that rhetorical robustness is a separate evaluation target from human alignment. On SciCore itself they use cautious language, saying the results demonstrate the potential of science-core review to improve it.

4. A worked example: letting AI do the first check on internal documents

Consider a team that asks an AI system to do a first-pass check on drafts of study reports or regulatory documents. Two versions carry the same data and the same conclusion. One was polished by an experienced writer; the other is rougher and leans on bullet points.

The second half of that exchange is what the paper calls false robustness. Stability and discrimination have to be checked separately. And within this paper, simply telling the model to focus on content did not reliably fix the problem.

5. What is not new, and what the abstract leaves open

Studies that ask language models to write feedback on papers and compare it with human reviews already exist.^[3] So do systems that chain idea generation, experiments, paper writing and automated reviewing into one pipeline.^[4] Perturbing inputs and watching how outputs shift is also a familiar way to test language models. What this paper adds, on its own account, is a definition that ties stability and discrimination together as one requirement, plus a full-manuscript benchmark built around it.

Several limits remain. First, SciCore's advantage is reported mainly from a comparison on one backbone; whether other models show the same pattern is not clear from the abstract. Second, if extracting the science core also relies on a language model, the extraction step could drop substance or be swayed by wording itself, and the abstract does not say how this was checked. Third, the abstract does not explain how the rewrites were produced or how the authors confirmed that content was preserved. Fourth, the range of research fields is not stated, so it is unknown whether the results carry over to areas such as clinical research.

It is also worth reading "competitive human alignment" carefully. Human reviewers can be swayed by writing quality too, so high agreement with humans does not by itself mean insensitivity to rhetoric. The authors' report that the two rankings diverge is consistent with that point.

6. Why this paper surfaced now, and why it is not part of a cluster

The paper collected 70 upvotes on Hugging Face Daily Papers.^[2] Upvotes are votes from readers who found a paper worth reading; they say nothing about whether its claims are correct or important. In this site's selection, the only signal that fired for this paper was reader votes. No follow-on implementation and no press coverage were found.

No cluster of papers on the same question formed either. Among the papers collected this week, none landed close enough to group with it, so the cluster size was 1. At this point it would overstate things to say that robustness of AI reviewing has become a topic in its own right. It is a single paper that drew attention.

It is still worth covering because the use of AI as an evaluator is spreading beyond research. Grading documents, screening applications and comparing proposals all put a model's judgment in a reviewing role. How to measure a model's sensitivity to wording is a question every one of those settings shares.

7. Where it connects to pharmaceutical and regulatory work

Pharmaceutical companies evaluate documents constantly: promotional material review, quality checks on submission documents, internal assessments of study plans. Many teams are now looking at adding an AI first pass to this work, and the paper makes concrete what should be tested before doing so.

One test is whether judgments stay the same when a document is reworded without changing its content. In promotional review, the strength of a claim's wording can itself be the issue, so the people designing the evaluation need to decide up front what counts as a content-preserving rewrite. The other is not mistaking low variation for good performance. A test set that mixes problematic and acceptable documents should show a clear difference in judgments. Both checks can be written into the validation of any AI-assisted review process.

SciCore's idea of extracting the substance into a structured form before judging resembles how reviewers match claims against source documents. But errors in extraction feed straight into the judgment, so the extracted content would need to be kept in a form a person can inspect. This is a preprint, and none of these operational effects were tested within its scope.