1. What problem does this paper address?

The problem is that evaluations of medical AI reach only part of the work that healthcare actually requires. The authors write that advancing medical AI for inclusive healthcare requires evaluation across diverse healthcare roles, tasks, and care settings, yet benchmark coverage remains uneven[1].

The complaint itself is not new. MedHELM, an earlier effort to evaluate language models across a wide range of medical tasks, argued that near-perfect scores on licensing-exam questions do not reflect the complexity of real clinical practice[3]. Exam questions are easy to build and easy to score. Evaluation effort gathers around them, and work such as explaining a condition to a patient or making a decision with limited resources receives less attention.

The question this paper asks is whether it is possible to show, in the form of a map, which capabilities for which roles are measured and where the gaps are. The work discussed here is a preprint posted to arXiv and has not been peer reviewed; everything below should be read as the authors' claims.

2. What does it propose?

The core idea is to place benchmarks on a grid of healthcare roles and AI capabilities, so that both coverage and gaps become visible. The authors call this grid the InclusionMed Atlas. It has three healthcare roles and three AI capabilities, giving nine cells[1]. The abstract does not say what the three roles and three capabilities are.

Benchmarks enter the grid only after structured audits and independent reproduction. Rather than listing every public benchmark, the authors inspect each candidate and keep only those whose results could be reproduced independently. To these they add InclusionMed-DxFiles, a private diagnostic reasoning benchmark of their own.

Model results are aggregated into the InclusionMed Index. The Index reports not only an overall score but also scores by role and by capability. It is designed less as a single ranking and more as a way to show how results differ across cells. The authors present InclusionMed as an open initiative meant to grow through collaboration and community contributions, and a GitHub repository is available[2].

3. What did it show? (Only what the abstract states)

The results stated in the abstract fall into three parts[1].

The first point deserves the most attention. Of several hundred candidates, only four survived audit and reproduction, and even then only three of nine cells were filled. The numbers themselves show how uneven medical AI evaluation is. The fact that the best overall score stayed at 49.1 also suggests that models do not yet handle well even the tasks that are measured.

From this the authors conclude that evaluation must cover more diverse healthcare needs and that model selection should depend on context. That conclusion is supported by the finding that ranks change across cells. The abstract does not, however, recommend which model to use in which setting.

4. A concrete example

Consider a team that answers information requests from healthcare professionals and is choosing a language model to draft responses. This is a hypothetical scenario for illustration, not an experiment from the paper.

Choosing by overall score

The team looks at a public leaderboard and picks the model with the highest overall score. The rationale is simple and easy to get approved internally. Most requests, however, come from clinicians who are unsure how to manage a particular patient. If the overall score was driven by skill at explaining things to patients, the quality of answers that support clinical judgment may be lower than the overall score suggests. Another model at similar cost might have suited this use better.

Choosing by the relevant cell

The team first decides where its work sits on the grid of roles and capabilities. It compares scores in that cell and lists inference cost alongside. If no benchmark covers that cell, the team records the gap and fills it with its own test questions. The rationale takes more effort to write, but it leaves a record of what was checked and what was not. If an error turns up later, the team can trace where the judgment went wrong.

The difference is not only whether a different model gets chosen. It is whether the record shows that the choice was made with the evaluation gaps in view. The InclusionMed map is useful because it makes those gaps visible.

5. What is not new, and where are the limits?

Evaluating medical language models across many tasks rather than a single exam is not new. MedHELM built a clinician-validated taxonomy of tasks, compared models across many benchmarks, and analysed cost against performance[3]. InclusionMed's contribution lies in filtering candidate benchmarks through audits and independent reproduction, and in laying out a grid of roles and capabilities from the standpoint of inclusive healthcare so that the gaps are explicit.

Several limits are visible even from the abstract. First, six of the nine cells remain empty. The overall Index score reflects only the three covered cells and does not describe medical AI as a whole. Second, one of the five benchmarks is private. A private benchmark is less likely to leak into training data, but outsiders cannot check the quality of its questions. Third, the abstract does not say what "inclusive" means in practice, for example which countries, languages, or resource levels it has in mind.

Fourth, the finding that models score higher on patient communication could reflect what models are good at, or it could reflect differences in how hard the tasks are. The abstract does not separate the two, and an association is not the same as a cause. Fifth, the abstract does not name the models evaluated, describe how the Index is weighted, or explain how inference cost was estimated. Sixth, it is a preprint. Within the scope of this paper, what can be said is that among the benchmarks that passed the authors' audits, medical AI evaluation has large gaps and model rankings change depending on the cell examined.

6. Why is this topic clustering now?

According to the selection record, this paper did not form a cluster. Its cluster size is 1, and no related papers on the same question were recorded. It was selected because two signals from different populations fired. It received 61 upvotes on its Hugging Face paper page and 27 stars on GitHub. Upvotes are votes cast by readers who found it worth reading; stars show interest from people who work with code. Both show that a paper drew attention, not that its evaluation design is sound.

The keywords include diagnostic reasoning benchmark, inference cost, and task-level tags. The interest lies less in producing one ranking and more in showing separately which tasks a model handles, at what cost, and how well. The argument that exam scores alone are not enough has been running for some time in medical AI evaluation[3]. This paper extends it, and what it adds is a map of evaluation coverage itself. With a cluster of one, however, it is too early to call this a topic that has broadly taken off.

7. What connects to pharmaceutical and regulatory practice?

The closest link is the practice of recording, when choosing a language model, which uses it was actually checked for. Pharmaceutical companies try language models for answering requests from healthcare professionals, drafting patient-facing materials, summarising literature, and more. These uses differ in role and in the capability they demand. The overall score on a public leaderboard may match none of them.

In its guidance on large multi-modal models in health, WHO lists safeguarding inclusiveness among its ethical principles and recommends independent third-party audits and impact assessments for models deployed at scale[4]. InclusionMed's use of audits and independent reproduction points in the same direction. Making evaluation gaps explicit also gives material for explaining how far a model has been checked.

The paper does not test this connection. Its evaluation assumes roles on the healthcare provider side and shows no results on pharmaceutical company tasks. A high score on patient communication does not mean a model can be used as is for patient-facing information. Information that pharmaceutical companies provide to patients and healthcare professionals is subject to its own rules on wording and evidence. In practice, a company would need to decide where its use sits on the grid, run its own checks for that cell, and record what it has not checked.