IBM Research ships agent consistency tooling that halves the Pass^k gap

An agent that performs well in rehearsal can take a different path during the live demo and fail the same task. IBM Research notes in a Hugging Face blog post that this is embarrassing onstage but a reliability problem in production: a workflow that succeeded once may fail the next time a user makes the same request.

The team quantified the problem on AppWorld. A ReAct agent backed by GPT-4.1 posted a Mean@5 of 77.4% across five runs, but succeeded in all five runs on only 53.0% of tasks, a 24.4-point consistency gap. Nearly a quarter of the benchmark consists of tasks the agent can sometimes solve and sometimes cannot, with nothing about the task changing between runs. For mission-critical work such as reconciling a financial transaction or checking a contract for an obligation, that instability is a showstopper rather than a user-experience detail.

The root of the gap is how agents are measured. Mean@k answers how good an agent is on average; it does not answer whether the same question will work again. That requires Pass^k, the fraction of tasks where the agent succeeds on all k runs. The post stresses that Pass^k is not the familiar Pass@k, which is optimistic and asks whether at least one of k attempts succeeded, and always satisfies Pass^k <= Mean@k <= Pass@k.

Why does temperature 0.0 not fix it? Every agent decision comes out of a probability distribution, and the shape of that distribution decides how much noise it takes to change the outcome. A sharp distribution puts most of its mass on one token and produces the same choice run after run; a flat one spreads comparable mass across near-tied tokens, making the winner close to a coin flip. Because a trajectory chains dozens of decisions, a small per-step chance of flipping compounds, and platform-side effects such as GPU floating-point non-associativity and request batching are enough to reorder a near-tie.

The fix is a two-stage pipeline built on ALTK-Evolve. First, the Consistency Analyzer takes one recorded trajectory and replays each decision step through controlled resampling to measure how much the model's output actually varies at that point. Concretely that is one additional model call per decision step, requesting five completions at once against the already-recorded context, done offline with no new tool calls, no new environment interactions and no second end-to-end rollout. The output is a per-step consistency scorecard that pinpoints which decisions are likely to flip next time.

Second, those signals are written into consistency guidelines, a new guideline type in altk-evolve that is injected back at inference time. The point is that the analyzer targets instability rather than failure, so it catches steps the agent happened to get right this time but could easily get wrong next time.

On AppWorld's test_normal set, with 168 tasks, a GPT-4.1 ReAct agent, guidelines mined from a single baseline trajectory per task and five fresh test runs, aggregate Pass^5 rose from 53.0% to 69.0% while Mean@5 rose from 77.4% to 81.0%, narrowing the gap from 24.4 points to 12.0. Medium-difficulty tasks gained 22.9 points and hard tasks 14.3 points, with easy tasks up 12.2 points, and Mean@5 did not drop at any difficulty level.

The guidelines also transfer. Applied to a related task in the same AppWorld scenario, Pass^5 still improved by 13.0 points, only three points below the same-task figure. On the weaker gpt-oss-120b model, same-task Pass^5 rose from 10.1% to 16.1%, and the similar-task generalization gain of 8.7 points was actually larger, suggesting the guidelines capture reusable failure patterns rather than the specifics of one trajectory.

The tooling is now open source: the ALTK-Evolve repository includes the Consistency Analyzer and consistency-guideline generation used in these experiments, with the full methodology in an arXiv technical report. Consistency is becoming a purchasing criterion alongside accuracy for teams putting agents into production.