01The operator chose none of the three options alone and set an order instead
In July 2026 the operator was partway through some system work with an AI. The AI offered three numbered ways to continue. Option 1 closed off a hazard. Option 2 connected the new part to the live system. Option 3 tried things out at small scale.
The operator did not pick one. The instruction was to run them as 1, then 3, then 2, which moved the small trial ahead of the live connection. The live step also carried a check that existing behavior had not broken.
The next instruction went further into detail. Take the fix from option 1 into the local working copy, but send nothing outside. Then read the system as it currently runs. Only then design the smallest possible trial.
That is as far as the record goes. It does not show whether this order prevented any accident. So the question here is narrower: when AI lists options, is there a sound reason to reorder them instead of following the numbers?
02When every option is needed, the decision is the order, not the pick
Numbered options look like a list from which you take one item. These three were not competing, though. Closing the hazard, testing, and connecting to the live system would all have to happen eventually.
Options like these are stages of one job. Nothing gets discarded, so the only thing left to decide is the sequence. What the sequence changes is how far the first failure travels. The table below reorders the same three tasks using general reasoning. It is not a measurement from the record.
| Order | Hazard during the trial | Where the first failure lands | Work after a failure |
|---|---|---|---|
| 2, 1, 3 | Still open | Live system | Roll back live, and clean up any damage |
| 1, 2, 3 | Closed | Live system | Roll back live |
| 3, 1, 2 | Still open | Trial scope, though the hazard may actually occur | Stop the trial, and clean up any damage |
| 1, 3, 2 | Closed | Trial scope | Stop the trial |
03A failure on the live system is harder to recover from than one in a trial
Once something is live, a failure reaches real work and real users, and undoing it takes more effort. Google's Site Reliability Workbook describes a partial, time-limited deployment of a change to a much smaller subset of production, which is then evaluated. The aim is to find defects quickly with as little impact as possible. If the numbers diverge, you pause and roll back.
The same thinking applies when AI is given wider latitude. Anthropic's guidance on agents notes that autonomy raises both cost and the risk of compounding errors, and recommends extensive testing in sandboxed environments with guardrails. It also says to add complexity only when that demonstrably improves outcomes.
Neither document prescribes 1, then 3, then 2. They do agree on the part that says test first and widen afterward.
The hazard comes before the trial because a trial is exposed to it too. A small trial on real data still does real harm if that data leaks. To run the trial safely, you need the closing step first.
The order has a second effect. A small test produces the evidence for deciding whether to go live at all. Follow the numbers straight to the live connection, and that decision point is gone before any test has run. Sometimes the right result of a trial is not connecting.
04Setting the order includes defining each stage's end and the condition for starting the next
In this article, setting the order means rearranging the AI's options into a sequence of execution and also deciding what must be finished before moving on. Renumbering alone leaves the call about moving on with the AI.
Close
Decide what may be touched, what must stay inside, and what must never happen, and stop those things by setup rather than by request.
Test
Use a small number of items or an environment away from live work, and have a person read the results.
Decide
Compare the results with conditions set in advance. Go on, fix, or stop.
Connect
Connect to the live system, and check that existing behavior still works.
There are cases where this is unnecessary. The first is a true either-or. If you are signing up with one of two services, you have a choice to make, not a sequence.
The second is work that is easy to reverse. Anthropic's Claude Code guidance says its rewind feature makes it low-risk to try the bolder approach first. The same page warns that a file changed only by a shell command is not restored, and rewinding a session does not recall anything already sent outside the company either. So the dividing line is not the size of the work but whether it can be undone.
The third is a hazard you cannot yet name. If you do not know what to close, test first in an environment away from live work, close what you find, and then go on. So 1, 3, 2 is not a fixed recipe. The rule underneath it is to place what is hard to undo toward the end.
05In material review, test AI pre-reading on past materials before it touches live review
Take a pharmaceutical company considering AI as a first reader of promotional materials. This example is not from the record; it applies the pattern. Suppose an AI proposes three options: close the hazards, test small, and connect to live review.
| Stage | What it means in material review | Condition for moving on |
|---|---|---|
| 1 Close | Check internal rules and decide what information may go into the AI. Keep out unpublished trial data and personal information. The AI proposes comments only and does not rewrite the material. | The responsible department has approved the scope, and you have confirmed that excluded information cannot get in. |
| 3 Test | Take a few past materials that have finished review. Compare the comments reviewers made with those the AI makes. | You know which kinds of comment the AI misses and can state what it may be trusted with. |
| 2 Connect | Use it to pre-read materials in live review. Judgment and responsibility stay with the reviewer. | Count misses at regular intervals. If they rise, return to stage 3. |
Past materials work for the test because the human conclusions are already known, so the two can be compared. If the AI is wrong, no material under review is affected.
A good trial still does not settle the matter. NIST's generative AI profile warns that pre-deployment testing can be mismatched to the context of actual use. So choose past materials close to the kinds you will be reviewing next. The same document discusses automation bias, the excessive deference people show to automated systems. After connecting, someone still needs to watch whether reviewers are accepting the AI's comments without question.
Outside material review, the same pattern fits any work that reaches people outside the company, changes source data, feeds into someone else's work, or incurs a cost.
06AI does not know which failures hurt most, so the person sets the order
Anthropic's Claude Code guidance lists things the AI cannot see: a deploy process that lives in someone's head, a file that must never be read, a budget to stay under. How much a failure would hurt belongs on that list. What happens if an error in a material gets out, and whose approval is required, are unknown to the AI unless someone writes them down.
The numbers on a list of options reflect the order of explanation. They are not necessarily an order of execution. Follow them as written and one silently becomes the other.
Simply asking the AI whether your order is good does not solve this. A 2023 paper by Anthropic researchers found that five leading AI assistants tended to tell users what they wanted to hear across several tasks. If you ask "is this order fine?" and get a yes, that yes is no evidence of safety. Ask for what has been overlooked, not for approval.
The closing stage needs one more caution. The Claude Code guidance says a prompt alone cannot guarantee that something will never happen, and it lists separate mechanisms, such as deny rules, that the tool enforces every time. So stage 1 should not end with "please be careful." Make it something you can confirm in settings and permissions.
07From tomorrow: sort options by whether they can be undone, and request only the first stage
When AI lays out options, do these five things.
- Check whether it is either-or or all of them. Ask the AI: "Are these alternatives, or are all of them needed?"
- Sort with two questions. Can a failure be undone? Who does a failure reach? You answer these yourself.
- Reorder. Hazard-closing work first, a trial that can be undone next, hard-to-undo work last.
- Request only the first stage. Write something like: "Do option 1 only, send nothing outside, show me the result, then stop."
- Ask where the order is weak. Request "three hazards this order overlooks," and read the answer before going on.
Write the condition for moving on before the trial begins. Set it afterward and you will be tempted to loosen it to fit the result. Make it something you can check by looking, such as "we can list the kinds of reviewer comment the AI missed."
- If all the options AI lists are needed, they are stages of one job. The decision is their order, and the numbers only reflect the order of explanation.
- Order by whether things can be undone: close the hazard, test where it can be reversed, leave the hard-to-undo work for last, and write the condition for each next stage in advance.
- In material review, decide what information may go in, test on past materials, and only then connect to live review. Judgment and responsibility stay with the reviewer.
AI can line up options quickly, but unless told, it does not know which failure would hurt most. So before comparing options, check whether all of them are needed. If they are, move whatever is hardest to undo to the end. Deciding that sequence stays with the person who hands the work to AI.
- Anthropic. Building effective agents. Anthropic Engineering, 2024. https://www.anthropic.com/engineering/building-effective-agents
- Anthropic. Best practices for Claude Code. Claude Code Docs. https://code.claude.com/docs/en/best-practices
- Google. Canarying Releases. The Site Reliability Workbook. https://sre.google/workbook/canarying-releases/
- National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), 2024. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
- Sharma, M., Tong, M., Korbak, T., et al. Towards Understanding Sycophancy in Language Models. arXiv:2310.13548, 2023. https://arxiv.org/abs/2310.13548