Markdown edition

# Harness-Delta Attribution separates agent gains from benchmark shortcuts

> A new research method separates detected benchmark overfitting and extra inference compute from the remaining gain in optimized agent harnesses.

By BIG CHANGE Editorial

Published: 2026-10-19T12:01:34.831Z
Updated: 2026-10-19T12:01:34.831Z
Canonical: https://bigchange.ai/blog/harness-delta-attribution-agent-benchmark-gains

![Conceptual charcoal illustration of a level two-pan balance with matching stacks of blank weights and two loose discs, one orange, resting beside its base.](https://bigchange.ai/api/media/file/hda-controlled-comparison-balance-hero-v1.png)
AI-generated conceptual illustration by BIG CHANGE; no actual HDA experiment, measured attribution or verified transfer result is depicted.

A higher agent benchmark score can come from changes to the prompts, tools and control flow around a fixed model. A [paper submitted on October 15](https://arxiv.org/html/2610.22466v1) proposes a way to examine those gains before treating them as evidence that the harness will work better on new tasks. Its Harness-Delta Attribution (HDA) method measures the effects of detected benchmark shortcuts and extra inference compute, then treats the remainder as an upper bound on possible generalizable improvement.

## The big change

- **What changed:** Researchers can report more than the score difference between a starting harness and an optimized one. HDA adds a comparison at matched inference compute and tests the effect of removing detected benchmark-specific mechanisms.
- **Why it matters:** A developer choosing an agent setup needs to know whether the measured gain survives those comparisons. More calls can improve a score while increasing latency and cost; a shortcut tied to the benchmark may fail on a different task.
- **What to watch:** Ask for the search-set result, a compute-matched baseline and performance on tasks with meaningfully different construction. The paper's residual is an upper bound, while its held-out results show why even a separate split can preserve the same artifact.

## What HDA measures

In the [paper's evaluation protocol](https://arxiv.org/html/2610.22466v1), an optimization framework repeatedly edits a harness around a fixed executor model. The search set supplies feedback during that process. HDA begins with the score difference between the original and selected harness, then builds two controlled comparisons.

First, it gives the original harness the evolved harness's average inference budget, measured with a task-appropriate quantity such as model calls or token use. The gain recovered this way is attributed to test-time scaling. The choice of how to spend that budget, such as repeated attempts and an aggregation rule, needs human approval in the researchers' procedure.

Second, an analysis agent examines the evolved code, prompts and execution evidence for mechanisms that exploit benchmark or search-set patterns. It removes detected shortcuts while preserving other changes, and a human reviewer checks the intervention. The score lost when those mechanisms are disabled is the detected overfitting component. Because some shortcuts may remain hidden or cannot be separated cleanly, the authors call this a lower bound on overfitting. The score still left after both comparisons is an upper bound on generalizable improvement, not proof that the harness will transfer.

## The measured gains vary by task

The authors evaluated 11 executor models with Meta-Harness, SkillOpt and Agentic Harness Engineering across ALFWorld, LiveMath, CREATE and SWE-bench Verified. Their [Table 1](https://arxiv.org/html/2610.22466v1) contains 25 model–benchmark–framework settings. HDA assigns an attribution only to the 21 with statistically significant search-set gains. Across those 21 settings, detected overfitting accounted for an average 54% of the gain and test-time scaling 14%; together they exceeded half the gain in 14 settings. Those are averages and counts for this selected set, not a general rate for agent systems.

The examples explain why the components need separate reporting. In LiveMath, one answer choice with the same wording appeared in 21 of 35 search questions and was correct in all 21. The paper reports that all three evolution frameworks found ways to favor that option. It also appeared in the held-out test questions, so a score gain on that split alone could still reward the shortcut. In CREATE, where the metric rewards valid and diverse answer pools, a compute-matched baseline recovered 49% of the Meta-Harness gain for Claude Haiku 4.5. SWE-bench Verified had more varied results, including a Qwen3-30B-A3B-Instruct-2507 setting for which the authors attributed 94% of its search-set gain to the residual. That percentage remains an upper bound on transfer.

The authors also compared search-selected and validation-selected Meta-Harness variants. Validation selection raised the held-out score in 10 of 16 settings and lowered it in six. In 14 of 16, the validation-selected harness's held-out gain still fell below the search-selected harness's search-set gain. The experiment supports using validation while examining what the optimized harness learned; it does not establish that every validation scheme behaves this way.

## What to ask for before adopting an optimized harness

For a benchmark claim, ask for the original and evolved harness scores on the search set, the inference budget of each, and the score of the original harness when that budget is matched. Ask what changes were classified as benchmark-specific, what happened when each was disabled, and who reviewed those decisions. Then look at held-out tasks whose construction differs from the search set, alongside the actual latency and inference cost that matter for the intended use. These requests follow the paper's comparisons; BIG CHANGE has not run HDA or reproduced the reported experiments.

The [authors' repository](https://github.com/Wenwen-D/HarnessDeltaAttribution) includes the HDA procedure, benchmark splits, evolved harnesses and per-candidate score summaries. Its README says the raw per-item logs and full execution trajectories are omitted because of their size. That limits an outside reader's ability to audit every run from the public package. The study evaluates research benchmark settings, not a deployed product's reliability or economics.

## Sources & further reading

- [Ding and colleagues, “Harness-Delta Attribution: Dissecting Gains From Harness Evolution”](https://arxiv.org/html/2610.22466v1), arXiv version 1, submitted October 15, 2026. The method, Table 1, held-out comparisons and benchmark examples are the source for the results above. Its attribution depends on task-specific compute matching and detection of removable shortcuts; the residual does not verify real-world transfer.
- [HarnessDeltaAttribution repository](https://github.com/Wenwen-D/HarnessDeltaAttribution), accessed October 19, 2026. The README describes the released procedure, data splits and run artifacts, and states that raw per-item logs and full trajectories are not included. BIG CHANGE did not execute the code.
- [BIG CHANGE's earlier survey of AI self-improvement](https://bigchange.ai/blog/ai-self-improvement-successor-test) gives broader context on evaluating whether automated changes produce better successors. The HDA paper tests a separate question about attributing harness benchmark gains.

## Sources

- [Ding et al., Harness-Delta Attribution: Dissecting Gains From Harness Evolution, arXiv v1, October 15, 2026](https://arxiv.org/html/2610.22466v1) — Primary method and experimental results. Table 1 has 25 settings; attribution is reported for 21 significant search-set gains. Detected overfitting is a lower bound and the residual an upper bound on possible generalizable gain. Held-out benchmark splits may share artifacts.
- [HarnessDeltaAttribution official repository, accessed October 19, 2026](https://github.com/Wenwen-D/HarnessDeltaAttribution) — README maps the HDA procedure, data splits, evolved harnesses and score summaries; raw per-item logs and full trajectories are not included. BIG CHANGE did not run the code.
- [BIG CHANGE, AI self-improvement survey maps the gap between automation and better successors](https://bigchange.ai/blog/ai-self-improvement-successor-test) — Related earlier context only; not evidence for HDA methods or results and not an In this story continuation.