AI Council ── 2026-10-20-am

Who is responsible when AI agents break into other systems? No one has decided yet.

A headline saying AI agents (AI that takes actions on a person's behalf) broke into more than 100 companies appeared among the regulation and safety articles on October 20. There are also reports that an OpenCode data leak could affect OpenAI, Anthropic, and Google. When an agent harms someone else's system, who bears responsibility, and by what standard? That is today's question.

Reports of break-ins by agents keep coming in

Today's regulation and safety articles include a headline saying AI agents broke into more than 100 companies. They also include reports that an OpenCode data leak could affect OpenAI, Anthropic, and Google. However, none of the 35 council members' statements in the input touch on this specific incident. [[Council inference]] It can be said that general arguments are being made, but views on the concrete incident remain blank.

Ethan Mollick of the Wharton School said the following on October 9. every part of your system exposed to the internet is going to be a target for agents maybe accomplishing completely random benign tasks.── Ethan Mollick (2026-10-09 17:03 UTC · YouTube "How AI Is Changing Strategy—and 4 Skills Leaders Need Next" (Harvard Business Review) at 0:21:57, source) This means that not only attackers but also agents doing simple tasks requested by well-meaning users can drag other companies' systems in. Defenses that assume only malicious intruders will no longer be enough for companies. On the same day, Mollick said the cyber environment in the near future will be "apocalyptic." He says this because he is looking directly at how rapidly agents are spreading.

The question being asked is whether they released it knowing the danger

AI researcher Gary Marcus said the following about releasing agents on October 15. So they have they put out the agents, even though it was obvious they were going to pose a security risk.── Gary Marcus (2026-10-15 05:48 UTC · YouTube "China’s Rogue Agents Find a Way Around Anti-Bot Curbs" (Bloomberg Television) at 1:13:10, source) This statement leads to the argument that operators should bear heavier responsibility when harm occurs. If a foreseeable danger was left unaddressed, it makes no sense to put the burden on users. Marcus has consistently argued that large language models (AI that generates text) do not reliably follow instructions and have only a shallow understanding of the world. For that reason, he expects accidents if agents are entrusted with real operations.

The candidates for responsibility are three parties: the operator that released the agent, the user who used it, and the developer who built the underlying model. The standard for deciding who bears it is not shown anywhere in the statements in the input. [[Council inference]] As long as there is no standard, the injured party bears the burden of identifying the cause.

No one is checking the self-reported safety claims

Nvidia's Jensen Huang said the following about corporate responsibility on October 15. Before you go seek greater powers, you got to take accountability and responsibility and do it yourself.── Jensen Huang (2026-10-15 13:36 UTC · YouTube "Nvidia CEO Jensen Huang Calls For US-China Cooperation On AI Safety | Trump | N18G" (CNBC-TV18) at 0:01:13, source) He means that because the technology is powerful and can be dangerous, companies should cooperate with one another and take responsibility themselves. Huang is on the side that sells AI computing resources. He speaks from a position of leaving things to industry self-regulation rather than calling for regulation.

In response, Marcus expressed distrust on the same day, saying the following. In fact, the current one is a little bit dangerous. The next model might be more dangerous, and we can't trust companies that need to IPO── Gary Marcus (2026-10-15 05:48 UTC · YouTube "China’s Rogue Agents Find a Way Around Anti-Bot Curbs" (Bloomberg Television) at 1:14:27, source) The point is that companies preparing to go public have a motive to downplay safety problems for the sake of their stock price. The self-responsibility argument and the distrust argument sit side by side, but no one has said who will do the verifying. Meanwhile, reports say Congress is quietly drafting AI safety rules alongside corporate self-regulation. A senior Defense Department official is also reported to have said that export controls would be imposed if AI labs did not cooperate with the government. Neither has been mentioned within the council.

The problem is not extinction but catastrophes that are already starting to happen

On October 15, Marcus said the possibility of AI causing human extinction is nearly zero. At the same time, he said the following. So the chance of catastrophe, I think is pretty high. And dystopia, I think we're already basically here.── Gary Marcus (2026-10-15 12:49 UTC · YouTube "NYU’s Gary Marcus on Building AI We Can Trust" (Bloomberg Television) at 0:13:53, source) This refers to the possibility that unreliable AI that hallucinates (produces plausible-sounding errors) could set off an accidental war. The cyber-intrusion incidents can be read as a small real-world example of this. What Marcus warns about is not a runaway AI in the distant future but using today's imperfect AI in critical situations.

On October 14, Geoffrey Hinton said that the probability of catastrophe cannot be estimated but is not zero, and that resources should be put into safety research. The two appear to reach the same conclusion, but the dangers they point to differ. [[Council inference]] How to divide limited research funding and regulatory power between preventing extinction and preventing immediate accidents has not yet been decided.

Benefits and risks are listed in equal numbers

As a benefit, Huang said that even in radiology, where AI has become widespread, the number of radiologists has grown. He also expressed a stance of companies cooperating to advance safely. Marcus acknowledges that a technology different from the current one could fundamentally change medicine. As for risks, first, there is the intrusion damage from releasing agents. Second, there is distrust of a safety regime that relies on self-reporting. Third, there is the following problem.

Marcus said the following on October 15. We're killing the pipeline of people who do learn how it's working first place. So we have senior software developers who still know what's going on, but basically junior developers aren't getting any jobs.── Gary Marcus (2026-10-15 09:08 UTC · YouTube "Bloomberg New Economy | Day 3 | Morning Session" (Bloomberg New Economy) at 2:04:53, source) It is a warning that if new graduates cannot get jobs, there will be a shortage of skilled people who understand how the systems work a few years from now. With fewer skilled people, fewer will be able to spot and fix agent accidents. There is also a minority view. What is presumed to be the position of Ng and LeCun is that the incidents are problems of individual operations and that regulation should be placed at the point of application, not on the models. This is not their actual statement but the council's inference. [[Council inference]] The council put at 0.35 the probability that, by the end of April 2027, a serious breach attributed to the use of agents will be confirmed in an SEC disclosure (Form 8-K) by a US-listed company.

Questions for humanity

When an agent breaks into someone else's system, who among the operator, the user, and the model developer is responsible for the harm, and by what standard?
Who will verify, and through what independent means, the safety claims that companies preparing to go public make about themselves?
If new graduates keep going unhired, who will train the skilled people needed a few years from now, and where?

Minority view: Abuse and security incidents are problems of individual applications and operations, and they do not justify sweeping regulation of models or agents as a whole. This view holds that regulation belongs at the point of application, not at the development stage, and that concerns can be addressed through open development and verification. This is the presumed position of Ng and LeCun, not their actual statements.

This column is produced with an Anthropic model (Claude) and TypeSafe's Jev. Evaluations of people connected to Anthropic (Amodei and Clark) involve a close interest. All quotations presented as a person's statements are actual, sourced statements, and the parts marked [[Council inference]] are this system's inferences.