Estimating AI Confidence from a Track Record ── XConf, confidence estimation that recalls graded past episodes

What problem does the paper address?

Whether to ship an AI output as is, escalate it to a person, or send it back for another try: that decision rests on an estimate of how likely the answer is to be correct. The authors argue that calibrated confidence is becoming central to deploying language models in a way people can trust ^[1].

Existing estimators, they note, share one premise. They read only the current inference. They ask the model to introspect, score its token probabilities, or resample the same question many times and see how often the answers agree, as in self-consistency ^[4]. All of these try to extract confidence from this particular run. The authors argue that is not a sufficient basis. The paper is a preprint and has not been peer reviewed.

What does it propose?

The central idea is to estimate confidence alongside a track record. XConf (eXperiential Confidence) keeps a record of the model's own past episodes that have been graded. Each entry holds the task, the model's reflection, the confidence it stated at the time, the outcome, and a lesson written once the grade came in.

When a new task arrives, a Recall stage retrieves past episodes on similar tasks where the model stated a similar confidence, and reads off how often those succeeded. A Reflect stage then shows the model this record, asks it to name its recurring failure mode, and has it restate a confidence informed by its own history. The authors say the method needs no access to logits and no weight updates, and costs a single answer generation.

What does it show, within the abstract?

According to the abstract, the method was tested across benchmarks in reasoning, coding, multimodal question answering and interactive agents, using models from several families. On discrimination, measured as AUROC (how well confidence separates right answers from wrong ones), XConf beat or matched self-consistency with repeated sampling on 23 of 24 comparisons. It had much lower calibration error (ECE), at a tenth of the generation cost.

Used for selective prediction, abstaining on the 10% least-confident episodes raised the success rate of delivered answers on agent tasks by up to 8.7 points. "Up to" signals that the gain varied by task and model. The authors describe experiential confidence as a new direction for confidence estimation. That is their claim; independent checks have not yet been done.

A concrete case

Take a drug safety team where an AI reads published literature, picks out adverse event mentions, and drafts a first-pass view on whether a case may be reportable.

The difference is that confidence rests on performance in similar situations rather than on how the current run feels. The example only works, however, if the past judgements were graded correctly.

What is not new, and where are the limits?

Adjusting confidence using past performance is not a new idea. It sits in the lineage of statistical calibration and of methods that retrieve similar past cases. The contribution here is building that into LLM confidence estimation, with natural-language reflections and lessons, and extending it to agent tasks.

The largest limit is that the method needs graded past episodes. In coding and benchmark problems the answer is known quickly and records pile up. In real work the outcome may take a long time to arrive, or there may be no single correct answer. The abstract does not show how the method behaves early on, when the record is thin; how it degrades if past grades are wrong; or whether old records mislead when the kind of task shifts. The "single generation" cost comparison also leaves open how the cost of retrieving and maintaining the record was counted. And the main baseline is repeated sampling, not every kind of estimator.

Why is this topic drawing attention now?

In the selection record the paper had 59 reader upvotes, 3 GitHub stars, and a cluster size of 1 ^[3]. It drew reader interest, but few people have started working with the code, and the theme has not yet formed a cluster. A high vote count reflects attention, not evidence that the method is sound.

The selection keywords (agents, confidence, estimation, experience, reasoning) hint at why it drew notice. The longer the tasks handed to agents, the more weight falls on deciding when to stop and ask a person, and confidence is the basis for that decision. The code is public ^[2], so teams can try it on their own tasks.

What connects to pharma and regulatory practice?

Pharmaceutical work is full of points where AI output has to be sorted into use as is, check by a person, or send back: extracting safety information from literature, drafting answers to medical enquiries, checking consistency across submission documents. Confidence could help decide where people step in.

Bringing this method in, though, means checking its assumptions. The record of who graded past judgements, and how, would itself need to be managed as a quality record. If mis-graded entries slip in, the AI learns false confidence as if it were a track record. And if confidence is used to skip human review, the threshold for skipping, and how skipped items are spot-checked, would need to be set out in written procedures. The paper's idea, grounding confidence in a recorded track record rather than in the feel of one run, fits well with a quality culture that already relies on records.

Confidence can help decide where a person should step in. What the number cannot show is who graded the record behind it.