Telling an LLM Not to Think Does Not Make It Stop ── A preprint that measures no-thinking at three levels and reports that settings and instructions alone do not switch reasoning off

1. Thinking modes are everywhere; no-thinking has barely been tested

Many current LLMs can write out a reasoning process before they answer. Some are designed so that a single model switches between a thinking mode and a non-thinking mode.^[4] The intended use is simple: let the model reason on hard questions, and have it answer directly when the question is easy or the output format is fixed.

According to the abstract, the "no-thinking" side of this switch has received far less attention than the thinking side. Prior work has defined no-thinking through proxies, such as a disabled thinking mode or the absence of long traces. The authors argue that both proxies are unreliable. A model with thinking disabled may still emit reasoning, and a long trace may consist of filler rather than actual inference.

The gap matters in practice whenever downstream processing expects only the answer. If a classifier is supposed to return "yes" or "no" and reasoning text appears in front of it, the parser either fails or picks up the wrong value.

2. The core idea: sort the pre-answer text into three levels

The authors first normalize every response into two parts: a pre-answer trace and a final answer. They then evaluate the pre-answer trace at three levels.

The first, Empty-Thinking Rate, is the share of responses whose pre-answer trace is empty, meaning strict compliance with an answer-only instruction. The second, an instruction-aware Question-Pre-answer Relevance score, measures how similar the pre-answer trace is to the question. The third, Explicit Inference Rate, uses another LLM as a judge to decide whether the trace contains visible, explicit inference.

Taken together, the authors say, these metrics separate three kinds of output: answer-only text, text that is relevant but not inferential, and explicit inference. This article treats that separation as the center of the method. Counting whether any text appeared is not enough; the question is whether the text that appeared is reasoning. Once that is measured, the gap between a model's settings and its behavior becomes visible.

3. What the abstract reports

• Scope: six prompting interventions were evaluated on six LLMs across Boolean, multiple-choice and open-ended questions.
• Reasoning that does not stop: explicit no-think controls could not reliably eliminate visible inference. Inference persisted under strict controls and became more common as the answer space opened. The authors call this Thinking Inertia.
• Accuracy: accuracy stayed stable on Boolean and multiple-choice tasks. On open-ended tasks there was a trade-off between answer-only compliance and task accuracy.
• Candidate answers: when the same questions were rewritten across answer formats, supplying candidate answers made answer-only responses easier to produce.

The abstract does not name the six models or the six interventions, and it gives no metric values. Which models were most prone to continued reasoning, and whether models with a built-in thinking mode behaved differently from those without one, cannot be determined from the abstract.

4. A worked example: a screening step that should return only yes or no

Consider an LLM used for first-pass literature screening. Each abstract is passed in with the instruction: "Is this a report of the target clinical trial? Answer only yes or no." The returned string is written directly into one column of a tracking sheet.

For Boolean questions the abstract reports stable accuracy, so the answer itself may well be right. The problem is the format. If the pipeline expects a single word, the second output either fails to parse or spills into the wrong field. A team that checked only the setting, thinking mode off, would find out when the sheet breaks in production.

Now consider an open-ended instruction on the same abstracts, such as "Extract the primary endpoint in one sentence." Here the abstract reports a trade-off: pushing the model toward answer-only output came with lower accuracy. The answer format changes both how well the instruction works and what it costs to enforce it.

5. What is not new, and where the limits are

Telling a model to skip its reasoning is not a new idea. Earlier work prompted reasoning models to bypass the thinking process and reported that, under limited output budgets, this can still perform well.^[5] That line of work mainly measured performance. This paper measures compliance itself: did the model actually stop reasoning when told to? Building metrics that judge the output rather than relying on settings or instructions as proxies is the contribution the abstract supports.

Several limits are visible from the abstract. First, the third metric relies on an LLM judge, so any error in the judge flows directly into the numbers. The abstract does not report agreement between the judge and human raters. Second, the second metric is similarity-based, and it is unclear how well it separates a restatement of the question from genuine reasoning.

Third, only visible reasoning is measured. Whether the model computes internally is outside what these metrics can see. Fourth, because the models and interventions are not listed in the abstract, anyone wanting to apply the result to a specific model has to check the full paper or the released code.^[3]

6. Is this a cluster? A single paper, but the question of controlling thinking continues

The selection system behind this article gives weight to clusters of independent papers on the same question within a short window. This paper did not form one: its cluster size is 1, and only one signal system, reader votes, was triggered.

It received 200 upvotes on Hugging Face Daily Papers, and its public repository had 3 GitHub stars.^[2][3] Upvotes show that researchers noticed the paper. They do not show that the claims are right or that anyone has reproduced them. Without a cluster, this is best read as a single paper.

The background question is still active. Designs that switch between thinking and non-thinking modes inside one model are spreading,^[4] and earlier work has reported cases where skipping the thinking step keeps performance acceptable.^[5] The more switches models offer, the more someone has to check that a switch actually changes behavior. This paper supplies a measuring tool for that check.

7. What connects to pharmaceutical and regulatory practice

When LLMs are built into pharmaceutical workflows, fixed-format output is common: include or exclude decisions in literature screening, a first-pass flag on whether an adverse event report needs a seriousness assessment, or picking a term from a controlled vocabulary. In each case, downstream processing and records assume an answer of a known shape.

The practical lesson is that answer-only output should not be assumed from a model setting or an instruction. During qualification, format compliance deserves its own acceptance criterion, separate from accuracy, measured on questions close to real use. The abstract also shows that how well the instruction works depends on the answer format.

One finding offers a design lead: supplying candidate answers made answer-only output easier. Choosing from a defined list may hold format better than free-text generation. That is an observation within the abstract's scope, though, and whether it holds for a given use case has to be tested on that organization's own questions. Finally, leftover reasoning text can end up in records. Teams should decide in advance whether that text is retained as part of the rationale or discarded, and document the choice.