Measuring Where an Agent Got Stuck, Not Just Whether It Finished ── A proposal to score computer-use agents by process rather than by final deliverable

What problem the paper takes on

Open a spreadsheet, tidy the figures, save the file, paste the result into another application. A computer-use agent is a system that takes on that whole sequence. In evaluation, the agent's output file has been checked mechanically against a set of conditions at the end: meet them and the run counts as a success, miss them and it counts as a failure^[2].

What the authors object to is how thin that signal is. Looking only at the final state, after hundreds of actions, leaves no record of what happened along the way. An agent that mistyped on the keyboard and an agent that clicked slightly off target land in the same column of the scoring table, even though the fix for each is entirely different. Without the shape of the failure, there is no handle on what to improve next. This paper is a preprint that has not been through peer review, and everything below should be read as the authors' claim.

What it proposes

The core of the proposal is narrow. Write a task out as a chain of subgoals, and score whether each one was met. The authors implement this as a task set called OSWorld-Pro. By the abstract, it holds over 300 tasks containing over 2800 subgoals, grounded in over 67,000 human annotations.

Whether a subgoal was fulfilled is judged by language-model judges tuned to agree with human raters. Because the subgoals are arranged so that later ones depend on earlier ones, the point at which an agent stopped making progress falls out of the grading as a by-product. The design does not throw away end-state checking; it adds a record of the process in front of it.

What it shows — only what the abstract states

The abstract reports two kinds of result. The first is difficulty. A leading model that reaches 83.4% on the end-state task set scores only 75.7% under the process-based one. The second is a classification of failures: the authors say they isolated process-level failure modes, including actions unrelated to the current subgoal and mistakes in click-based input.

Much is not stated. How the tasks were drawn and from which domains, who wrote the subgoals and by what rule, what happens where the judges and human raters diverge, whether the annotation process carried a bias of its own — none of these conditions appear within the abstract. They should not be filled in by guesswork.

A concrete view

In workplace terms, grading by process moves the emphasis from accepting a deliverable to inspecting a work record.

A handover that cannot name a restarting point invites the same failure again. That blank is what subgoal-level scoring is trying to fill.

What is not new, and where the limits sit

Scoring work in stages is not itself a new idea. Writing out a procedure and marking each step is, if anything, an old habit in human work. The novelty here is better read as engineering: building that habit into an evaluation base for computer-use agents, and grounding it in human annotation rather than in an automatic rewrite of the task.

The limits are visible. First, the way subgoals are written decides the score. Split a task finely and there are more places to lose points; split it coarsely and the process view thins out. No external standard for choosing that granularity appears in the abstract. Second, the judge is itself a language model. Tuned to human agreement or not, the limits of that tuning become the limits of the grade. Third, seeing a failure and being able to remove it are not the same thing. Counting failure types leaves the work of fixing them entirely untouched.

Why process-level evaluation is clustering now

This subject did not appear on its own. Around the same time, work has been published proposing that self-evolving agents be assessed at the level of process rather than endpoint performance^[3]. The objects differ; the question overlaps. Both start from the same complaint: a rising endpoint number that cannot be explained does not tell you what to change next.

That a cluster exists is not evidence that the direction is correct. It says that several groups share a dissatisfaction, and no more than that. The paper discussed here also sits at a stage where community votes and independent reimplementations have not yet appeared. Attention figures measure how much a subject is being talked about, not whether its claims hold.

Where this connects to pharmaceutical and regulatory work

In regulated work, whether an output is correct and whether it was produced correctly are asked separately. A correct deliverable made in a way that leaves no record is hard to inspect. That is the first place an attempt to insert a computer-use agent into promotional material preparation or review runs aground. A system graded only on the correctness of its output cannot answer the audit question: which steps, in what order, cleared by whose judgement.

Subgoal-level scoring moves partway toward that question. A record of how far an agent progressed through a chain resembles the skeleton of an audit trail. But records built for research evaluation and records demanded by regulation serve different ends. The former exists to give researchers a handle on improvement; the latter exists to locate responsibility. A resemblance in shape does not make one usable as the other.

What can be taken into practice is probably the ordering. Before deciding whether to bring an agent into a process, check whether that process can be written out as subgoals at all. Work that cannot be written out that way cannot be inspected once automated, whatever tool is used. The preprint's claims need not be taken on trust for that ordering to be worth testing in-house.