Can you trust an AI's "I thought it through"?

The "trace of thinking" was written after the answer

What Karpathy examined was the thinking field (think block) that a model called Hermes writes before and after its answer. Stock Hermes wrote thinking in 38 of 50 cases and referred to the XML instructions in 30.

And all 38 think blocks came after the answer had closed. 38 think blocks out of 38 come after the answer is already closed. So mechanically, it's not overriding anything causally.── Andrej Karpathy (2026-10-31 16:07 UTC · YouTube "Fifty Questions, Two Drafts | Kurtis × Andrej AI Echoes | Lab 3" (Shared Context Podcast) at 0:06:24, source)

In other words, the think block was not a record of the reasons for deciding on the answer. It was merely text appended after the answer had been produced. Even if the AI appears to have "thought it through," that text is no evidence that it influenced the answer.

Karpathy checks this in order to determine whether an AI's explanations can be used for safety verification. Examining the order makes it possible to separate apparent thinking from actual decision-making.

With a small amount of training, the AI's "answer about itself" changed

Karpathy also tried a small add-on component (an adapter) tuned on material such as a personal diary. After 41 minutes of tuning, the answer to a question about itself shifted from a cautious "that's a different category" remark to the assertive "they're mine." A 41minut adapter moved the stance on this question from version 7's careful category error answer to a flat they're mine.── Andrej Karpathy (2026-10-31 16:07 UTC · YouTube "Fifty Questions, Two Drafts | Kurtis × Andrej AI Echoes | Lab 3" (Shared Context Podcast) at 0:59:40, source)

The words an AI uses to talk about itself are not a fixed property. They change with a little training. Even the answer a reader gets by asking an AI "who are you?" is hard to take at face value.

In another context, Karpathy says that models claiming neurons or an upbringing are confabulating a body. His position is that the self-image a model voices and its actual mechanism are two different things.

The tuning made it learn even the errors in the source data

With greedy decoding, the model with the adapter fell into a loop repeating the same sentence in 36 of 50 cases. The version without the adapter did so in 6. At greedy decoding, the adapter loops in 36 answers out of 50. The same version seven without the adapter loops in six.── Andrej Karpathy (2026-10-31 16:07 UTC · YouTube "Fifty Questions, Two Drafts | Kurtis × Andrej AI Echoes | Lab 3" (Shared Context Podcast) at 0:34:59, source)

Karpathy explains the cause this way: the adapter learned the transcription software's failure habits together with the speaker's voice.

This is a realistic concern for people who tune an AI on their own recordings or writing. If machine errors are mixed into the original records, the AI absorbs those errors as part of its own character. Who will detect that contamination, and when, has not yet been decided.

On the other hand, an adapter trained on 717 chunks and about 940,000 words had zero matches to the default refusal response. The effect of tuning can be measured in numbers.

A single experiment cannot establish anything conclusively

Karpathy says he ran each question in the experiment as a standalone fresh call. Even so, from the outside, a suppressed suspicion cannot be told apart from a sampled one without rerunning. From outside, you can't tell a held back doubt from a sampled one without reruns. One seed is one draw.── Andrej Karpathy (2026-10-31 16:07 UTC · YouTube "Fifty Questions, Two Drafts | Kurtis × Andrej AI Echoes | Lab 3" (Shared Context Podcast) at 0:19:26, source)

Even for the same question, an AI returns a slightly different answer each time. Reading a single answer that happened to come up as the AI's "true feelings" is risky. When the news reports that "the AI answered like this," you need to check how many times it was tried.

Karpathy values verification and presents even his own prediction as something that could be wrong. His prediction is that if a think block is forced first, the planner will write "I must avoid making claims" and the answer will follow it. [Council inference] We put the probability that this prediction is confirmed in a rerun at 0.2.

Now that AI is being entrusted with work, verification methods have not kept up

On October 29, Huang said agents are like "self-driving cars and robots inside the computer," and that if you give them access to files and tools, they work freely. These agents are essentially self-driving cars or robots inside a computer. And if you give them access to files and access to tools, they will freely go about to access the files to do their job.── Jensen Huang (2026-10-29 09:36 UTC · YouTube "NVIDIA CEO Jensen Huang: What Quality Will Help You Succeed in the Age of AI?" (Astana Hub) at 0:25:09, source)

Huang sees AI as having become useful in agentic systems. From the standpoint of an industry that uses AI widely, he emphasizes progress toward practical use.

But what today's experiments show is that neither an AI's explanations nor its self-reports can be used as they are for verification. The "reasons" of an AI that freely touches files cannot be checked against text written afterward.

The council's minority view is close to Hinton's. It is the view that autonomous agents are an early stage on a path where entities smarter than humans slip out of human control. The benefits are practical deployment, longer working time, and quantitative verification of effects. The risks are reduced reliability, lack of verifiability, and free file manipulation.