Turning Scientific Code into Somewhere Agents Can Learn ── A proposal to reuse accumulated research software as verified practice environments

What problem the paper takes on

Scientific software is a store of knowledge distinct from the literature. Analysis procedures, preprocessing conventions, the line at which a number is treated as anomalous — all of it survives as code that runs. The authors point out that this accumulation is not being converted into experience an agent can learn from, and name the gap the scientific experience bottleneck.

Three reasons are given for why the conversion is hard: toolchains are fragmented across fields, domain conventions are implicit rather than written, and correctness criteria are specialised. Running the code can be automated; deciding whether an output is scientifically sound cannot be written by someone outside the field. While that step stays blocked, an agent can attempt as many runs as it likes without learning much. This paper is a preprint that has not been peer reviewed, and what follows is the authors' claim.

What it proposes

The core move is to hand the work of converting a repository into a programmable environment to agents themselves. In the infrastructure, called ScienceIDE, experts first define scientific cases and acceptance criteria. Guided by those, agents transform repositories into executable environments. The resulting environments support task generation, execution, and scientific verification of what came out.

The authors argue that such environments give one shared foundation to three uses: supervised fine-tuning, reinforcement learning, and evaluation. Rather than building a task set, consuming it and discarding it, the repository itself remains as a place that can be used again.

What it shows — only what the abstract states

Using verified interaction trajectories, the authors report training PhAI-IDE-72B, PhAI-IDE-9B and PhAI-IDE-4B. Results are reported in two directions. One is gains on held-out scientific-code repair, meaning tasks not used in training. The other is improvement across selected general-purpose benchmarks in code, reasoning and knowledge, which the authors present as evidence of positive transfer from scientific experience to broader capabilities.

Much is left unstated. Which repositories were used, how the expert acceptance criteria were settled, how the general benchmarks were selected beyond the word "selected", and what the comparison baselines were. None of these conditions appear in the abstract. "Positive transfer" in particular is a phrase with wide interpretive range, and without figures attached it is best received as the authors' reading of their own results.

A concrete view

What the proposal changes is how practice problems come into existence.

Invented exercises carry neither the conventions of a working group nor the feel for what a check is looking for. The aim here is to convert the place work actually happens into the place practice happens.

What is not new, and where the limits sit

Building tasks from real code is not a new idea. Constructing benchmarks out of the record of real software development has been done for some time^[3]. What this paper adds is better read as two things: folding field-specific scientific verification into that pipeline, and delegating the construction of the environment to agents.

The limits are visible. First, the quality of the expert acceptance criteria sets the quality of the environment outright. Loose criteria teach an agent which answers pass easily. Second, because agents build the environments, errors introduced during conversion flow downstream as errors in the training data; who inspects that step is not stated in the abstract. Third, learning skews toward fields that leave code behind. Knowledge that never became code — bench technique, judgement at the point of measurement — cannot be picked up along this route at all.

Why this subject is being picked up now

The paper carries 98 votes in the public venue, and its released implementation has 81 stars — the largest figures among the three papers selected here. Even so, votes and stars measure how much a subject is being discussed, not whether its claims hold. Stars establish that a runnable implementation is public and that people have tried it, and nothing further.

The background worth naming is a shift in what agent training is short of: not methods, but places. Research on how to train is thick; environments where an agent can attempt something and fail are scarce. The idea of converting code that already exists into such a place is being picked up because it touches that shortage directly.

Where this connects to pharmaceutical and regulatory work

Pharmaceutical organisations hold an asset of the same shape. Statistical analysis programs, data-shaping conventions, the thresholds at which a result gets flagged. These usually survive as running code alongside, not inside, the written procedures. Borrowing this paper's view, what is embedded there is not only method but judgement.

Whether that asset can simply be turned into a practice environment is another matter. Analysis code under regulation runs validated. Letting an agent restructure it breaks the premise the validation rested on. Practice, if it happens at all, has to happen on a copy held apart from anything in operation. Leave that boundary vague and something built for training eventually finds its way into production.

The transferable question is probably the acceptance criteria. The heaviest element in this framework is the expert definition of what counts as sound. Ask whether that definition is written down anywhere for your own analyses or material reviews. If it is not, the work of writing it remains open regardless of any decision about automation. That inspection does not have to wait on a preprint being confirmed.