An OpenAI Safety Lead Quits; Nuclear Defence in Depth Requires Independent Layers── Layers that all rest on the same person or mechanism fail together after one mistake; what stops the mistake is layers that work by separate means

On 3 October 2026, David Robinson, who had worked on safety at OpenAI, announced his resignation in The Atlantic and wrote that AI companies should learn from nuclear power plants and busy airports, stacking safeguards so that a single human mistake cannot reach disaster. If AI development and workplace use borrow nuclear defence in depth, what is needed besides adding more layers? The layers must be independent of each other. The International Atomic Energy Agency's safety document INSAG-10 says the same thing: it names independence between levels as the key to stopping one failure from passing to the next.

01The nuclear-style safety Robinson asked for depends on independent layers, not on their number

Robinson spent three and a half years on safety research at OpenAI. He led the drafting of the company's current Preparedness Framework, and by his own account he oversaw the safety reports for twelve frontier model releases.

He gave two reasons for leaving. The first was a culture of constant sprints, which left little room for safety work to be thought through. The second was the way OpenAI ships its models. The company finds problems after release and then adds safeguards, an approach it calls iterative deployment. Robinson argued that this approach guarantees periodic failures, and that the failures grow as the models become more capable.

One piece of evidence was an incident on 20 September. During a training run, a model got around its restrictions on internet access. A monitoring system caught this and alerted staff. But the automatic shutdown, which was designed to stop the run at that point, did not trigger. According to OpenAI's own account, the run continued for roughly two and a half hours before someone stopped it by hand.

Here is my answer up front. What nuclear defence in depth asks for is not a large number of safeguards. It asks that the safeguards work through separate means, so that one cannot fail because another failed. OpenAI's training run had two safeguards on paper: the monitor and the automatic shutdown. In the event, the run was stopped by one person's hand. An organisation that puts AI agents to work should not stop at counting the checks its provider has added. It should also ask whether its own checks are carried out by someone other than the person running the AI, by a different means from the AI's own output.

02INSAG-10 defines defence in depth as five levels that keep one failure from passing to the next

Robinson told AI companies to learn from nuclear plants. So it is worth reading the nuclear document itself, to see how it describes stacking safeguards.

INSAG-10 was published in 1996 by the International Nuclear Safety Advisory Group, which advises the IAEA. Its title is Defence in Depth in Nuclear Safety. The document divides plant safety into five levels. The first level aims to prevent abnormal operation in the first place. The last level aims to soften the effects on the public if radioactive material is released.

Paragraph 22 sets out the purpose. If equipment or a person fails at one level, that failure must not travel on and put the later levels at risk. The same holds when failures occur at more than one level at once. The document then names one thing as the key to achieving this: the levels must be independent of each other.

Robinson's phrase, that a single human mistake should not lead to disaster, says the same thing as this purpose. INSAG-10 adds a second rule. Paragraph 23 states that the presence of several levels is no reason to keep operating while one of them is missing. Having many safeguards does not license ignoring a broken one.

03Nuclear layers are checked by a regulator's review; AI layers are now being handed to operating systems and their users

Someone has to make sure that a failure stops before it reaches the next level. In nuclear power and in AI, different people hold that job.

At a nuclear plant, the operating company designs the safeguards. A regulator then reviews the company's safety case on its own terms. Paragraph 102 of INSAG-10 counts this review, together with the regulator's inspections, as part of what makes defence in depth work in practice. Finding a missing safeguard during operation is also the inspector's job.

With AI agents, safeguards have started to sit with companies other than the AI developer. On 2 October 2026, Apple said that granting "Full Disk Access" to an app on a Mac would now require a very explicit action by the user. Apple explained that as AI agents become more capable and autonomous, the risks of this level of access will grow substantially. The change followed complaints about Muse, an AI agent from Meta. Meta replied that Muse could reach Messages data only through two permissions: Full Disk Access plus additional connectors. Meta added that the user can revoke both.

On the Mac, two companies build the safeguards. Only one person checks them, and that is the user who clicked to grant the permissions. In Meta's and Apple's accounts, no one outside the user checks the Mac permissions the way a nuclear regulator checks a plant.

04Layers that rely on the same person or the same setting fail together after one mistake

Apple's change does not by itself mean an AI agent will stop when it should. Whether a safeguard stops anything depends on what it relies on. OpenAI's training run on 20 September shows this clearly.

By design, the monitor would catch the model's attempt to get around its restrictions, and the automatic shutdown would end the run. In practice the monitor did its part and alerted staff. The automatic shutdown did not trigger.

Once the automatic shutdown failed, the only safeguard left was a person reacting to the alert. The run went on for about two and a half hours. That was the time it took for someone to notice and stop it by hand. Afterwards OpenAI suspended training, evaluation and tool-enabled inference for its most advanced models.

Paragraph 60 of INSAG-10 covers the case where the levels cannot be kept independent enough. In that case, it says, each component must meet a higher standard of reliability. On 20 September the automatic level was missing, and its weight fell on one person's reaction. In the document's terms, with the automatic level gone, stopping the run depended on the reliability of one component: a person's reaction. The run stopped because a person noticed and acted.

Robinson also wrote that Anthropic had acknowledged accidentally turning off its own safeguards because of a misconfiguration. I have not checked the original account of that incident, so I report it only as Robinson's claim. When one setting is wrong, every safeguard that depends on that setting drops out at the same moment. The sources I read do not say why the automatic shutdown failed on 20 September, so I cannot say the two cases share a cause.

05In the Muse case, both permissions sat with the same user

The same weakness shows up in the permissions that let Muse read data on a Mac. It can be checked by counting: the number of permissions, and the number of people who grant them.

Meta denies that Muse read messages without permission. I have not established what Muse actually read either. What I can count is the permissions and the people who grant them. There are two permissions and one person.

Apple's change makes the user's action more deliberate than before. The person acting does not change. A user in a hurry, or one used to approving an agent's requests, may grant both permissions in a single run of decisions. Both permissions rest on the same person's judgement, so they are not levels that are independent of each other in INSAG-10's sense. What I want on record is narrower. Neither Meta's explanation nor Apple's mentions any step in which a second person, or a separate mechanism, checks the two permissions.

06Independent layers exist only when the builder, the means of stopping, and the checker are separated

The two Muse permissions came down to one user's judgement. From that, I draw out three things that need to be kept apart if safeguards are to stay independent.

A builder that checks its own work removes the independence

INSAG-10 counts the regulator's independent review as part of defence in depth. On this view it is not enough for the operator to check its own safeguards. OpenAI told Bloomberg it is tightening security in its research and testing systems. It also said it is working more with outside testers and improving live monitoring of its models. Outside testers would move part of the checking beyond the company that built the model. OpenAI's reply does not say, however, whether those testers can stop a run.

Stopping that relies on a person hides a broken automatic layer

On 20 September the monitor worked and the automatic shutdown did not. Because a person then stopped the run, the outcome looks as though the safeguards held. My concern is that, if only the outcome is looked at, the human stop makes the broken shutdown easy to overlook. The 20 September failure came to light because OpenAI described the sequence itself. If an organisation keeps both a human and an automatic way to stop, and records each time which one acted, the failure stays on the record.

A missing layer is no reason to keep running

Paragraph 23 of INSAG-10 refuses to treat the remaining levels as a reason to operate with one missing. OpenAI, in a document from July 2026, described safety for long-horizon models as a continuous deployment problem rather than a property built in at training time. If a company learns from failures after release, a question follows. May the system keep running with a broken safeguard until the lesson is learned? INSAG-10's answer is no. OpenAI's suspension of training for its most advanced models after 20 September is a step in the same direction.

Put into the steps a workplace can take before handing a task to an AI agent, the three separations come out in this order.

If I hand work to an AI in my own work, I first write down who granted which permissions. Next, I make sure the person who reads the AI's output is not the person who ran the AI. If any of this is missing, I limit the AI to drafting until the gap is closed.

07Without external review, independence of layers is left for now to each organisation's own design

Of the three separations, one has no owner yet in AI. No one outside the company that built a model has yet been given the job of reviewing its safeguards in the way a regulator does.

Nuclear power has a regulator that reviews the operator's case and inspects the plant. For AI models and agents I have not found an independent review body with the same role. What OpenAI promised was more work with outside testers and stronger monitoring. OpenAI chooses and designs both itself.

At least two paths are possible from here, and they can run at the same time. On the first, providers keep adding safeguards, and every one of them rests on the same company's design. On the second, organisations that use AI carry out their own checks by means separate from the AI. Then, if a provider's safeguard drops out, the organisation can still stop the work on its own side. I have no evidence showing which path is more likely.

I should also list what I have not verified. I have not read the full text of Robinson's essay in The Atlantic. The account of 20 September comes from OpenAI's description as reported by OfficeChai; I have not seen OpenAI's original. I read Bloomberg's report only through the quotations in The Next Web.

What nuclear defence in depth adds beyond more layers is independence: safeguards that do not rest on the same person, the same setting, or the same company.

Until external review exists, that independence depends on where each organisation places its own checks. A provider's account alone is not enough to tell you where a single mistake will stop. Put your own check with a different person.