🔍 Intelligence Synthesis JP/EN
Ethics · Regulation · Technology — Pharma Practice Notes

AI Tech Flash ── 2026-11-09

Alibaba's Co-Design Bench Tests Agents on Real Failed Requests

Best agent fully meets 26.4% of tasks, Alibaba researchers report

What changes

According to MLQ.ai, this changes how coding agents are evaluated. The test uses requests that real users struggled to get AI agents to finish.

The paper's authors say a leaderboard listing only the model tells part of the story. They want the model and the harness, the tool that runs the model as an agent, to be reported together.

Why it matters

According to MLQ.ai, the top performer, GPT-6 Astra, satisfied every criterion on only 26.4% of 273 tasks. About three in four outputs would still need a person to finish the job.

Swapping the harness moved results sharply. Qwen3.8-27B's repository score was 0.560 in qwen-code and 0.749 in opencode. How a comparison is run can change how models look against each other.

What it is

Co-Design Bench is a benchmark for AI coding agents. A benchmark is a shared set of tasks used to compare performance.

According to MLQ.ai, it draws on 37,463 deduplicated requests from 26,314 real coding sessions. Experts refined each request and wrote hidden grading rubrics. The rubrics hold 2,331 criteria in all.

There are two tracks. Localized multimodal edits may give the agent images, audio, video or documents. Repository edits need coordinated changes across a codebase. A repository is the place where a program's files are stored.

Who and where

According to MLQ.ai, Alibaba researchers built it. Their paper was posted to arXiv on Oct. 26. arXiv is a public site for research papers.

MLQ.ai's article does not say how the benchmark itself is released or where to get it. This is an account of the Alibaba researchers' paper, as reported by MLQ.ai.

Why it was built

According to MLQ.ai, the work starts from requests that real users struggled to get agents to finish. The authors say older edit benchmarks such as WebCompass are near saturation, with models above 99%. MLQ.ai notes that comparison used a different task pool.

The problem is that older tests make real-world failures hard to see. So the authors ranked candidate requests partly on signs of past failure. Examples include a user rephrasing a request two or more times, or an explicit rollback.

How it works

According to MLQ.ai, grading has two stages. An agentic judge runs each finished app in a browser and checks the source code. Agentic means the AI acts on its own to carry out the work.

A second verifier then challenges every pass or partial verdict. It aims to catch mock-ups and hardcoded results.

The main comparison ran every model through Claude Code as the harness. A harness is the tool that runs a model as an agent. Two models were rerun through other harnesses, and the results reshuffled.

Key points

  1. GPT-6 Astra met 69.9% of criteria on average but fully solved only 26.4% of 273 tasks.
  2. Repository edits were hardest: GPT-6 Astra fully solved 17.9%, against 41.0% of multimodal edits.
  3. Changing the harness moved Qwen3.8-27B from 0.560 to 0.749, so the authors want model and harness reported together.

Conclusion

According to MLQ.ai, Alibaba researchers built Co-Design Bench from real failed requests. Among ten models, the best fully met every criterion on 26.4% of tasks.

The authors urge reporting the model and harness together. The paper also states limits: requests are fixed with no follow-up questions, and AI judges can still make errors.

Quotes and sources

“OpenAI's GPT-6 Astra passed 69.9% of each task's acceptance criteria on average but satisfied every criterion on just 26.4% of the 273 tasks, according to the paper posted to arXiv on Oct. 26”
── MLQ.ai, 11/08 10:21 UTC
“Co-Design Bench draws on 37,463 deduplicated requests from 26,314 real coding sessions”
── MLQ.ai, 11/08 10:21 UTC
“Qwen3.8-27B scored 0.749 on repository edits in opencode and 0.560 in qwen-code”
── MLQ.ai, 11/08 10:21 UTC
“each request is fixed, so agents cannot ask follow-up questions or receive mid-task corrections, and AI judges can still make errors”
── MLQ.ai, 11/08 10:21 UTC

Source articles

Written by AI (Claude) from the cited articles, then checked by Jev (TypeSafe's judgement model), a Gemini fact-check and a verbatim check of every quote.