# Build an agent feedback loop and check if it improves

> Use Promptfoo, an existing task agent and approved fixtures to compare a baseline with one loaded lesson on development and held-out cases.

By BIG CHANGE Editorial

Published: 2026-10-22T03:27:09.791Z
Updated: 2026-10-22T03:27:09.791Z
Canonical: https://bigchange.ai/blog/agent-feedback-loop-heldout-evaluation-guide

![Conceptual charcoal illustration of a closed steel drawer with an orange key in its lock beside a separate computer on a workbench; the monitor is blank.](https://bigchange.ai/api/media/file/agent-feedback-heldout-cabinet-hero-v1.png)
AI-generated conceptual illustration by BIG CHANGE.

An agent can pass a test after you add a useful instruction. It can also pass because the test disclosed the answer, the grader was too easy, or a tool quietly changed. A feedback loop needs to tell those outcomes apart. This guide takes one existing OpenAI Agents SDK task agent through a baseline, a human-approved lesson, and a comparison on both familiar and untouched cases. A flat score or a regression is a valid result.

The example is an **integration pattern**, based on [Promptfoo's Python Agents SDK guide](https://www.promptfoo.dev/docs/guides/evaluate-openai-agents-python/) and the [SDK's agent](https://openai.github.io/openai-agents-python/agents/) and [runner](https://openai.github.io/openai-agents-python/running_agents/) documentation. BIG CHANGE has not run your agent or measured a gain. You supply a real permissioned task, its existing agent, approved records and the code that checks its actual output. Promptfoo runs and records evaluations; it does not automatically diagnose errors, save lessons or decide to change an agent.

## The big change

- **What changed:** The SDK and Promptfoo can be joined around one existing agent, while an application-owned registry loads an approved lesson into its exact instructions.
- **Why it matters:** Four bound development and held-out result sets let the task owner see individual fixes, regressions and evaluation shortcuts before retaining the change.
- **What to watch:** The integration and real-task oracle require local implementation and testing. Results, traces and billing can contain sensitive data, and no improvement or cost has been measured here.

## 1. Freeze the task before changing the agent

Choose one bounded task with an output that its source owner can check. Put approved cases into `fixtures/development/` and `fixtures/heldout/`. Each case needs an opaque ID, an input file, an expected-result file, the original record's ID/revision/hash, permission to process it, and a case-group ID. Keep paraphrases and answers from one source group in one split. Do not expose held-out inputs to whoever writes the lesson; never expose expected files or the oracle to the agent or its tools. A development case and at least one untouched held-out case are the structural minimum, not evidence that the set is statistically adequate.

Use this private manifest shape, replacing the field values with **actual approved records**. It is a schema example, not a fictional test case:

```json
{"caseId":"<opaque-id>","split":"development|heldOut","caseGroup":"<source-cluster>","sourceId":"<original-record-id>","sourceRevision":"<revision>","sourceSha256":"<64 hex>","sourcePermissionReceipt":"<private-reference>","inputFile":"<private-UTF8-file>","inputSha256":"<64 hex>","expectedFile":"<private-JSON-file>","expectedSha256":"<64 hex>","oracleVersion":"<checked-version>","approvedBy":"<source-owner>","approvedAtUtc":"<timestamp>"}
```

Save one real row per line in `fixtures/manifest.jsonl`. The expected JSON should have a documented schema of required keys, types and order rules. The task owner compares it with the original source, not with an agent answer. Reject missing permissions, duplicate IDs, overlapping case groups, changed file hashes and any task whose tool permissions can read the private oracle or expected files. Use a fixture-controlled or read-only tool environment; do not replay a production write to obtain a score.

Record the existing agent's base instructions, model ID and access, model settings, tools, handoffs, permissions, SDK and Promptfoo versions, Python version, `RunConfig`, max turns, tracing setting, environment and approved spending limit in `agent-config.json`. Pin the installed versions and preserve `pip freeze` and the Promptfoo version output. The [Promptfoo integration example](https://www.promptfoo.dev/docs/guides/evaluate-openai-agents-python/) describes Python 3.10+ and SDK 0.22.x; check its current requirements and your installed packages before adopting an example. Do not assume that any named model is available in your account. Record the resolved model and tool graph, including reachable handoffs. If you cannot identify or freeze them, this comparison cannot attribute a change to one lesson.

## 2. Connect Promptfoo to the existing agent

Freeze the fixture manifest, oracle, agent graph, tool permissions and approved budget before the first call. First create and run an immutable `baseline-development` plan. After source-backed diagnosis, the owner approves one exact lesson; only then can you freeze `candidate-development`. Lock the candidate instructions and precommitted decision rules before preparing the two held-out plans. `baseline-heldOut` and `candidate-heldOut` use the untouched held-out cases under the same manifest, oracle, base agent, tools and budget. Each of the four plans gets a distinct random `runId`, exact allowed case list and split, file/config hashes and private output directory. Baseline binds the original instructions and an empty lesson; both candidate plans bind the same approved lesson bytes and effective agent graph. A plan is immutable once created. No held-out input or result may influence the lesson or acceptance rule.

The following `promptfooconfig.yaml` is the Promptfoo file for one pass. The `make_tests.py` file below generates `tests.json` from only that pass's approved plan, with each row's input as `vars.task_input` and opaque ID as `vars.case_id`. Run it in the plan's private directory and place the provider file there too. Keep the answer and source record outside the config and agent tool sandbox. The separate private oracle below consumes the provider receipt after the run; a JavaScript assertion is optional if your installed Promptfoo version and trusted local assertion environment support it.

```yaml
description: one frozen agent pass
prompts:
  - '{{task_input}}'
providers:
  - id: file://agent_provider.py
tests: file://tests.json
```

```python
"""Generate one pass's Promptfoo tests from its approved private plan."""

import hashlib
import json
import sys
from pathlib import Path


def sha(data):
    return hashlib.sha256(data).hexdigest()


def main(plan_path, plan_sha):
    raw = Path(plan_path).read_bytes()
    if sha(raw) != plan_sha:
        raise ValueError("Plan hash differs")
    plan = json.loads(raw)
    tests = []
    seen = set()
    for row in plan["cases"]:
        case_id = row["caseId"]
        if case_id in seen or row["split"] != plan["split"]:
            raise ValueError("Duplicate case or wrong split")
        seen.add(case_id)
        source = Path(row["inputFile"]).read_bytes()
        if sha(source) != row["inputSha256"]:
            raise ValueError(f"Changed case input: {case_id}")
        tests.append({"vars": {"case_id": case_id,
                                "task_input": source.decode("utf-8")}})
    if not tests:
        raise ValueError("No approved cases in plan")
    target = Path(plan_path).with_name("tests.json")
    with target.open("x", encoding="utf-8") as file:
        json.dump(tests, file, ensure_ascii=False, indent=2)
    print(f"{target} sha256={sha(target.read_bytes())} cases={len(tests)}")


if __name__ == "__main__":
    if len(sys.argv) != 3:
        raise SystemExit("usage: python make_tests.py <private-plan.json> <plan-sha256>")
    main(sys.argv[1], sys.argv[2])
```

Use exactly one test per case, one provider and one prompt, with no repeats. [Promptfoo's Python provider contract](https://www.promptfoo.dev/docs/providers/python/) accepts `async def call_api(prompt, options, context)` and a serializable response with `output` or `error`. Its worker can persist across calls, so the provider must read a fixed plan each time and keep no mutable lesson, session, case counter or result in module globals. The official example uses a shared SQLite session and permissive output conversion; those are unsuitable for independent case identity and strict final-answer grading here.

Copy the following file as `agent_provider.py`. The task owner supplies `task_adapter.py` from the actual application; there is no generic agent or real fixture to import from this article. That adapter exports `build_existing_agent()` (the unchanged `Agent` and its fixture-safe tools), `build_run_config()` (the frozen SDK `RunConfig`), `graph_fingerprint(agent, run_config)` (SHA-256 of a canonical inventory of starting and reachable handoff agents, their effective models/settings/instructions, tool identities/permissions and RunConfig overrides), and `result_evidence(result)` (JSON-safe actual usage, last agent/model, tool/trace IDs or an explicit unavailable marker). Review that inventory and its serialization for the installed SDK version before a paid run. A non-enumerable dynamic handoff or model choice blocks the comparison. The provider imports the real `Runner` and calls it only after its checks.

```python
"""Reader adaptation for one frozen Promptfoo/Agents SDK pass.

Copy to agent_provider.py in a private evaluation directory. Supply task_adapter.py
from the existing application and an owner-approved immutable plan.json.
No model call occurs merely by importing this module.
"""

import hashlib
import json
import os
import re
from pathlib import Path

from agents import Runner
from pydantic import BaseModel
from task_adapter import build_existing_agent, build_run_config, graph_fingerprint, result_evidence


OPAQUE = re.compile(r"^[A-Za-z0-9_-]{1,80}$")


def digest(data):
    return hashlib.sha256(data).hexdigest()


def read_checked(path, expected_hash):
    data = Path(path).read_bytes()
    if digest(data) != expected_hash:
        raise ValueError(f"Hash mismatch: {path}")
    return data


def json_bytes(value):
    return json.dumps(value, sort_keys=True, ensure_ascii=False,
                      separators=(",", ":"), allow_nan=False).encode("utf-8")


def strict_output(value):
    if value is None:
        raise ValueError("Agent ended without final_output")
    if isinstance(value, str):
        return value
    if isinstance(value, BaseModel):
        value = value.model_dump(mode="json")
    if isinstance(value, (dict, list)):
        return json_bytes(value).decode("utf-8")
    raise TypeError(f"Unsupported final_output type: {type(value).__name__}")


def create_once(path, value):
    path = Path(path)
    path.parent.mkdir(parents=True, exist_ok=True)
    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
    try:
        with os.fdopen(fd, "wb") as file:
            file.write(json_bytes(value))
            file.flush()
            os.fsync(file.fileno())
    except BaseException:
        # An incomplete pending record is deliberately left for reconciliation.
        raise
    return digest(path.read_bytes())


def load_plan():
    path = os.environ["LOOP_PLAN_PATH"]
    required_hash = os.environ["LOOP_PLAN_SHA256"]
    plan = json.loads(read_checked(path, required_hash))
    if not OPAQUE.fullmatch(plan["runId"]):
        raise ValueError("Invalid runId")
    if plan["attempt"] != 0 or plan["variant"] not in ("baseline", "candidate"):
        raise ValueError("Unexpected attempt or variant")
    if plan["split"] not in ("development", "heldOut"):
        raise ValueError("Unexpected split")
    return plan, required_hash


async def call_api(prompt: str, options: dict, context: dict) -> dict:
    pending = False
    try:
        plan, plan_hash = load_plan()
        case_id = context["vars"]["case_id"]
        evaluation_id = context["evaluationId"]
        test_id = context["testCaseId"]
        repeat = context.get("repeatIndex")
        if not all(isinstance(v, str) and v for v in (case_id, evaluation_id, test_id)):
            raise ValueError("Missing case/evaluation identity")
        if not OPAQUE.fullmatch(case_id) or repeat not in (None, 0):
            raise ValueError("Invalid case ID or repeated test")
        cases = [row for row in plan["cases"] if row["caseId"] == case_id]
        if len(cases) != 1:
            raise ValueError("Case is absent or duplicated in plan")
        row = cases[0]
        if row["split"] != plan["split"]:
            raise ValueError("Case split differs from plan")
        read_checked(plan["fixtureManifestFile"], plan["fixtureManifestSha256"])
        read_checked(plan["agentConfigFile"], plan["agentConfigSha256"])
        budget = json.loads(read_checked(plan["budgetFile"], plan["budgetSha256"]))
        settings = json.loads(read_checked(plan["toolSettingsFile"],
                                           plan["toolSettingsSha256"]))
        if (budget["status"] != "approved" or not budget["approvedBy"] or
            budget["modelId"] != plan["modelId"] or
            budget["maxTurns"] != plan["maxTurns"] or
            settings["status"] != "approved" or not settings["approvedBy"]):
            raise ValueError("Budget or tool settings lack matching approval")
        task_input = read_checked(row["inputFile"], row["inputSha256"]).decode("utf-8")
        if prompt != task_input:
            raise ValueError("Prompt differs from frozen input")

        private_dir = Path(plan["resultDir"])
        if private_dir.is_symlink():
            raise ValueError("Private result directory is linked")
        outdir = private_dir.resolve(strict=True)
        if not outdir.is_dir():
            raise ValueError("Private result directory is missing or linked")
        binding = outdir / "evaluation-id.json"
        if not binding.exists():
            try:
                create_once(binding, {"runId": plan["runId"],
                                      "evaluationId": evaluation_id})
            except FileExistsError:
                pass
        if json.loads(binding.read_text("utf-8")) != {
            "runId": plan["runId"], "evaluationId": evaluation_id
        }:
            raise ValueError("Evaluation ID changed within the run")
        audit_key = digest(json_bytes([evaluation_id, test_id, 0]))
        create_once(outdir / "audit" / f"{audit_key}.json",
                    {"runId": plan["runId"], "caseId": case_id,
                     "evaluationId": evaluation_id, "testCaseId": test_id,
                     "repeatIndex": 0})
        pending_file = outdir / "cases" / f"{case_id}.pending.json"
        create_once(pending_file, {"runId": plan["runId"], "caseId": case_id,
                                   "planSha256": plan_hash, "attempt": 0})
        pending = True

        base = read_checked(plan["baseInstructionsFile"], plan["baseInstructionsSha256"])
        lesson = read_checked(plan["lessonFile"], plan["lessonSha256"])
        if plan["variant"] == "baseline" and (lesson or plan["separator"]):
            raise ValueError("Baseline lesson and separator must be empty")
        if plan["variant"] == "candidate":
            if not lesson:
                raise ValueError("Candidate lesson is empty")
            approval = json.loads(read_checked(plan["lessonApprovalFile"],
                                              plan["lessonApprovalSha256"]))
            for key in ("lessonId", "lessonRevision", "lessonSha256", "taskAgentId"):
                if approval[key] != plan[key]:
                    raise ValueError(f"Lesson approval differs: {key}")
            if approval["status"] != "approved" or not approval["approvedBy"] or not approval["approvedAtUtc"]:
                raise ValueError("Exact lesson lacks owner approval")
        rendered = base + plan["separator"].encode("utf-8") + lesson
        if digest(rendered) != plan["renderedInstructionsSha256"]:
            raise ValueError("Rendered instruction hash differs")

        original = build_existing_agent()
        run_config = build_run_config()
        if graph_fingerprint(original, run_config) != plan["baseGraphSha256"]:
            raise ValueError("Base agent/tool/handoff/RunConfig graph changed")
        # clone is shallow: neither this file nor the adapter may mutate shared lists.
        agent = original.clone(model=plan["modelId"],
                               instructions=rendered.decode("utf-8"))
        if graph_fingerprint(agent, run_config) != plan["effectiveGraphSha256"]:
            raise ValueError("Effective graph or model override changed")

        result = await Runner.run(agent, task_input, max_turns=plan["maxTurns"],
                                  run_config=run_config)
        output = strict_output(result.final_output)
        evidence = result_evidence(result)  # JSON-safe usage, model, tools, trace ID
        terminal = {"runId": plan["runId"], "caseId": case_id, "attempt": 0,
                    "planSha256": plan_hash, "variant": plan["variant"],
                    "split": plan["split"], "inputSha256": row["inputSha256"],
                    "configSha256": plan["agentConfigSha256"],
                    "lessonSha256": plan["lessonSha256"],
                    "renderedInstructionsSha256": digest(rendered),
                    "output": output, "evidence": evidence}
        receipt_hash = create_once(outdir / "cases" / f"{case_id}.result.json", terminal)
        return {"output": output,
                "metadata": {"runId": plan["runId"], "caseId": case_id,
                             "attempt": 0, "receiptSha256": receipt_hash}}
    except Exception as exc:
        # Preserve the error as an error; never turn it into plausible model text.
        if pending:
            try:
                create_once(outdir / "cases" / f"{case_id}.error.json",
                            {"runId": plan["runId"], "caseId": case_id,
                             "errorType": type(exc).__name__, "error": str(exc)})
            except Exception:
                pass  # A pending record without terminal readback blocks promotion.
        return {"error": f"Case failed: {type(exc).__name__}; inspect private receipt"}
```

This file is **untested reference code** until the reader implements and reviews that application adapter, its file permissions and the real plan. `strict_output` accepts exact text, a JSON-compatible dict/list encoded canonically, or a configured Pydantic result through `model_dump(mode="json")`; it rejects `None`, non-finite values and unknown objects. `Agent.clone` is a [shallow copy](https://openai.github.io/openai-agents-python/agents/#cloningcopying-agents): do not mutate shared tool or handoff lists. A changed tool, handoff, model, permission or budget invalidates the one-lesson comparison.

Each private `plan.json` supplies `runId`, `attempt: 0`, `variant`, `split`, and `cases`. Every case carries `caseId`, `split`, `caseGroup`, `sourceId`, `sourceRevision`, `sourceSha256`, `inputFile`, `inputSha256`, `expectedFile`, and `expectedSha256` from the approved manifest. The plan also carries `fixtureManifestFile/Sha256`, `agentConfigFile/Sha256`, `budgetFile/Sha256`, `toolSettingsFile/Sha256`, `oracleSha256`, `oracleVersion`, `resultDir`, `baseInstructionsFile/Sha256`, `lessonFile/Sha256`, `separator`, `renderedInstructionsSha256`, `modelId`, `baseGraphSha256`, `effectiveGraphSha256`, and `maxTurns`. The owner-approved budget JSON includes `status`, `approvedBy`, `modelId`, `maxTurns`, and its actual call/spend limits; the approved tool-settings JSON includes `status`, `approvedBy`, tool identities and permissions. The adapter's graph fingerprint must reflect those effective limits and settings. All four plans bind the same manifest, oracle, budget, tool settings, base graph, model and max turns; the two variants intentionally differ in rendered instructions, effective graph and lesson hash. The comparison checks case IDs, source IDs and case groups are disjoint across the splits, but the owner still inspects semantic near-duplicates and keeps held-out material away from lesson writing.

Candidate plans also supply `lessonApprovalFile`, `lessonApprovalSha256`, `lessonId`, `lessonRevision`, and `taskAgentId`. The private approval JSON contains `lessonId`, `lessonRevision`, `taskAgentId`, `lessonSha256`, `status: "approved"`, `approvedBy`, and `approvedAtUtc`. The provider accepts only its planned case IDs. Set `oracleSha256` to the exact `grade.py` code hash; the grader verifies it before using private expected files. Baseline uses a zero-byte lesson file, its SHA-256 and an empty separator, so its rendered instructions equal the existing base bytes; candidate uses the reviewed separator and exact approved revision. Compute all hashes from bytes, not normalized text. Set `LOOP_PLAN_PATH` to its fixed private absolute path and `LOOP_PLAN_SHA256` to the hash checked independently before invoking Promptfoo. Never pass a path or a secret through a test variable. Make the result directory private and empty before the run. If plan generation or adapter checks fail, stop before a model call.

Promptfoo may reuse `evaluationId` for many cases. The durable primary key is `(runId, caseId, attempt=0)`, with `(evaluationId, testCaseId, repeatIndex)` as an additional unique audit tuple inside that run. Create a pending receipt exclusively and flush it **before** a model call; write a terminal result or error through exclusive creation, flush it, and read its hash back. The reference file does not use an atomic rename: a crash can leave a partial terminal file. Treat a missing, partial or conflicting terminal file as unresolved, inspect any external effect, and never replay it automatically. Check any external tool effect before considering a new, explicitly approved attempt. Do not let automatic retries silently rerun a case. Call `Runner.run` without a shared session or prior-response ID. If the task requires several turns, give each case a fresh session and bind its ID to the durable key.

For a normal trial, run one pass at a time from its private plan. Check [current CLI options](https://www.promptfoo.dev/docs/usage/command-line/) against your pinned installation, then use its equivalent of:

```sh
python make_tests.py private-plan.json <verified-plan-sha256>
promptfoo eval -c promptfooconfig.yaml --no-cache --no-share -j 1 -o private-results/<runId>/promptfoo.json
python grade.py private-plan.json <verified-plan-sha256>
```

`--no-cache` avoids a cached answer being mistaken for a candidate call; `--no-share` prevents a share URL. Neither controls your application logs, SDK traces or account retention. Save the command, start/end time, exit code, config and test hashes, output hash, and the provider's per-case receipts. Disable sensitive trace capture or restrict private traces before submitting real records. If you assert that a particular tool or handoff was used, configure and verify the integration guide's separate trace bridge; a provider span alone does not prove that internal path.

## 3. Grade against the original, then diagnose

The owner writes a deterministic oracle for this task's expected JSON schema. The source owner first verifies each expected file against the original record and freezes its revision/hash. For every attempted case, the private oracle opens that verified expected file **outside the agent process**, parses the raw final output, and returns `pass: true/false`, `score: 1/0`, a concrete reason, and its oracle version. It fails closed on missing output, malformed JSON, missing required fields, wrong value, changed expected bytes or an oracle exception. Keep errors distinct from a genuine failed answer. Promptfoo [supports custom pass/score/reason assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/), but a private post-run oracle is also valid when the expected files cannot safely enter Promptfoo. Never pass the private answer to the model in a prompt, tool result, or provider metadata.

The configuration above intentionally has no Promptfoo assertion, because the expected files stay private. Its displayed pass rate is **not** the task grade. Use the private `grades.json` as the binary outcome, and match every receipt to its Promptfoo row. If a different task needs to verify an actual tool action, extend the oracle to inspect the authorized trace or external-effect readback before declaring a pass; exact final JSON alone cannot prove that action.

For an exact structured-output task, the following private `grade.py` is a complete deterministic oracle file. Use it only when whole-JSON equality matches the source owner's approved task rule; otherwise implement the real task's rule with the same error handling and source checks. Run it outside the agent/tool environment after each pass, using the plan path and hash. It writes one private `grades.json` and refuses to overwrite a prior grade.

```python
"""Private, deterministic exact-JSON grader for one frozen run.

Use only if exact structured equality matches the real task's approved criterion.
The expected files stay outside the agent/provider environment.
"""

import hashlib
import json
import os
import sys
from pathlib import Path


def sha(data):
    return hashlib.sha256(data).hexdigest()


def checked_json(path, expected_sha):
    data = Path(path).read_bytes()
    if sha(data) != expected_sha:
        raise ValueError(f"Hash mismatch: {path}")
    return json.loads(data)


def canonical(value):
    return json.dumps(value, sort_keys=True, ensure_ascii=False,
                      separators=(",", ":"), allow_nan=False)


def main(plan_path, plan_sha):
    plan = checked_json(plan_path, plan_sha)
    if sha(Path(__file__).read_bytes()) != plan["oracleSha256"]:
        raise ValueError("Oracle code differs from frozen plan")
    directory = Path(plan["resultDir"])
    grades = []
    for row in plan["cases"]:
        case_id = row["caseId"]
        expected = checked_json(row["expectedFile"], row["expectedSha256"])
        result_file = directory / "cases" / f"{case_id}.result.json"
        error_file = directory / "cases" / f"{case_id}.error.json"
        if result_file.exists() and error_file.exists():
            outcome = {"status": "error", "pass": False, "score": 0,
                       "reason": "Conflicting terminal result and error receipts"}
        elif not result_file.exists():
            outcome = {"status": "error", "pass": False, "score": 0,
                       "reason": "No terminal result; inspect pending/error receipt"}
            if error_file.exists():
                outcome["errorReceiptSha256"] = sha(error_file.read_bytes())
        else:
            result_bytes = result_file.read_bytes()
            result = json.loads(result_bytes)
            if (result["planSha256"] != plan_sha or
                result["caseId"] != case_id or
                result["inputSha256"] != row["inputSha256"] or
                result["lessonSha256"] != plan["lessonSha256"]):
                outcome = {"status": "error", "pass": False, "score": 0,
                           "reason": "Receipt binding differs from frozen plan"}
            else:
                try:
                    actual = json.loads(result["output"])
                    passed = canonical(actual) == canonical(expected)
                    outcome = {"status": "pass" if passed else "fail",
                               "pass": passed, "score": 1 if passed else 0,
                               "reason": "Exact approved JSON match" if passed else
                                         "Output differs from approved JSON"}
                except (ValueError, TypeError, KeyError):
                    outcome = {"status": "fail", "pass": False, "score": 0,
                               "reason": "Final output is not valid task JSON"}
            outcome["resultReceiptSha256"] = sha(result_bytes)
        grades.append({"runId": plan["runId"], "caseId": case_id,
                       "split": plan["split"], "variant": plan["variant"], **outcome})
    target = directory / "grades.json"
    with target.open("x", encoding="utf-8") as file:
        json.dump({"planSha256": plan_sha, "oracleVersion": "exact-json-v1",
                   "grades": grades}, file, indent=2)
        file.flush()
        os.fsync(file.fileno())
    print(f"{target} sha256={sha(target.read_bytes())} cases={len(grades)}")


if __name__ == "__main__":
    if len(sys.argv) != 3:
        raise SystemExit("usage: python grade.py <private-plan.json> <plan-sha256>")
    main(sys.argv[1], sys.argv[2])
```

For each baseline development failure, the reviewer checks the raw answer, original record, oracle result and authorized tool evidence. A bad expected file, grader defect, unavailable source or failed tool call is not an agent lesson. Repair the source or grader, freeze the corrected definition, and rerun the baseline before comparing it with any candidate. Only an actual agent mistake gets a diagnosis receipt: case ID, source pointer, observed behavior, suspected cause, alternative explanation, reviewer and proposed narrow instruction. The agent's account of its own mistake is a lead, not evidence.

The owner approves **one** lesson's exact bytes and scope. Save an append-only registry entry with its ID, revision, text hash, source development cases, approving person and time, task/agent scope and rollback pointer. Record rejection too. The provider loads that exact approved revision into the cloned *same agent's* instructions; a saved note or SDK session history is not a loaded lesson. Keep the baseline revision available for a revert.

## 4. Compare all four result sets

Run candidate development with the same case IDs, oracle, model/tool graph and limits. Then run baseline held-out and candidate held-out. The lesson writer sees neither held-out inputs nor results until the candidate instructions are frozen. Each of the four result directories must contain every planned case's input hash, raw output or explicit error, binary grade and reason, run/config/lesson/rendered-instruction hashes, trace ID or unavailable marker, actual calls/tokens/latency where exposed, and actual or `unknown` dollar cost. Check the run receipt against the Promptfoo export; a missing row is not a pass.

The following `compare.py` joins the four private plans and grade files, refuses a changed case set or base configuration, and leaves promotion to the owner. Run it after grading all four passes. Inspect the raw receipts in addition to its case movements.

```python
"""Bind four frozen runs; show within-split case movements for human review."""

import hashlib
import json
import sys
from pathlib import Path


SLOTS = ("baseline-development", "candidate-development",
         "baseline-heldOut", "candidate-heldOut")
COMMON = ("fixtureManifestSha256", "oracleSha256", "agentConfigSha256",
          "baseInstructionsSha256", "baseGraphSha256", "modelId", "maxTurns",
          "budgetSha256", "toolSettingsSha256")
CASE_KEYS = ("caseId", "caseGroup", "sourceId", "sourceRevision",
             "sourceSha256", "inputSha256", "expectedSha256")
BASELINE = ("lessonSha256", "renderedInstructionsSha256", "effectiveGraphSha256")
CANDIDATE = BASELINE + ("lessonId", "lessonRevision", "taskAgentId",
                        "lessonApprovalSha256")


def sha(data):
    return hashlib.sha256(data).hexdigest()


def load_plan(path):
    raw = Path(path).read_bytes()
    return json.loads(raw), sha(raw)


def require_same(plans, keys, label):
    for key in keys:
        values = [plan[key] for plan in plans]
        if not all(value == values[0] for value in values[1:]):
            raise ValueError(f"{label} differs: {key}")


def case_map(plan):
    rows = plan["cases"]
    mapped = {row["caseId"]: row for row in rows}
    if not rows or len(mapped) != len(rows):
        raise ValueError("Empty or duplicate planned cases")
    for row in rows:
        if row["split"] != plan["split"] or not all(row.get(k) for k in CASE_KEYS):
            raise ValueError("Missing case provenance or wrong split")
    return mapped


def validated_grades(plan, plan_hash):
    raw = (Path(plan["resultDir"]) / "grades.json").read_bytes()
    report = json.loads(raw)
    if report["planSha256"] != plan_hash or report["oracleVersion"] != plan["oracleVersion"]:
        raise ValueError("Grades belong to another plan or oracle")
    planned = case_map(plan)
    rows = report["grades"]
    mapped = {row["caseId"]: row for row in rows}
    if len(rows) != len(mapped) or len(rows) != len(planned) or set(mapped) != set(planned):
        raise ValueError("Duplicate, missing or extra case grade")
    for row in rows:
        if (row["runId"] != plan["runId"] or row["split"] != plan["split"] or
            row["variant"] != plan["variant"]):
            raise ValueError("Grade identity differs from plan")
        expected = {"pass": (True, 1), "fail": (False, 0),
                    "error": (False, 0)}
        if row["status"] not in expected:
            raise ValueError("Unknown grade status")
        if (type(row["pass"]) is not bool or type(row["score"]) is not int or
            (row["pass"], row["score"]) != expected[row["status"]] or
            not isinstance(row.get("reason"), str) or not row["reason"]):
            raise ValueError("Inconsistent binary grade")
    return mapped, sha(raw)


def main(paths, target):
    if len(paths) != 4:
        raise ValueError("Four distinct plans required")
    runs = {}
    for path in paths:
        plan, plan_hash = load_plan(path)
        slot = f'{plan["variant"]}-{plan["split"]}'
        if slot not in SLOTS or slot in runs:
            raise ValueError("Duplicate or unexpected pass")
        grade_map, grades_hash = validated_grades(plan, plan_hash)
        runs[slot] = {"plan": plan, "planSha256": plan_hash,
                      "gradesSha256": grades_hash, "grades": grade_map}
    if set(runs) != set(SLOTS):
        raise ValueError("A required pass is missing")
    if len({runs[slot]["plan"]["runId"] for slot in SLOTS}) != 4:
        raise ValueError("Run IDs must be distinct")

    plans = [runs[slot]["plan"] for slot in SLOTS]
    require_same(plans, COMMON + ("oracleVersion",), "Four-pass frozen setup")
    require_same([runs[s]["plan"] for s in SLOTS if s.startswith("baseline")],
                 BASELINE, "Baseline across splits")
    require_same([runs[s]["plan"] for s in SLOTS if s.startswith("candidate")],
                 CANDIDATE, "Candidate across splits")
    if plans[0]["lessonSha256"] == plans[1]["lessonSha256"]:
        raise ValueError("Candidate did not change the lesson")

    split_cases = {}
    for split in ("development", "heldOut"):
        left = case_map(runs[f"baseline-{split}"]["plan"])
        right = case_map(runs[f"candidate-{split}"]["plan"])
        if left != right:
            raise ValueError(f"{split} case input, truth or provenance changed")
        split_cases[split] = left
    development = split_cases["development"]
    heldout = split_cases["heldOut"]
    if set(development) & set(heldout):
        raise ValueError("Case ID appears in development and held-out")
    if ({row["caseGroup"] for row in development.values()} &
        {row["caseGroup"] for row in heldout.values()}):
        raise ValueError("Case group leaks across splits")
    if ({row["sourceId"] for row in development.values()} &
        {row["sourceId"] for row in heldout.values()}):
        raise ValueError("Original source appears in both splits")

    movements = {}
    blocking_errors = []
    for split in ("development", "heldOut"):
        base = runs[f"baseline-{split}"]["grades"]
        candidate = runs[f"candidate-{split}"]["grades"]
        movements[split] = []
        for case_id in sorted(split_cases[split]):
            pair = {"caseId": case_id, "baseline": base[case_id]["status"],
                    "candidate": candidate[case_id]["status"]}
            movements[split].append(pair)
            if "error" in (pair["baseline"], pair["candidate"]):
                blocking_errors.append({"split": split, "caseId": case_id})
    report = {"runBindings": {slot: {"runId": runs[slot]["plan"]["runId"],
                                      "planSha256": runs[slot]["planSha256"],
                                      "gradesSha256": runs[slot]["gradesSha256"]}
                              for slot in SLOTS},
              "movements": movements, "blockingErrors": blocking_errors,
              "promotionEligibleForHumanReview": not blocking_errors,
              "promotion": "blocked_until_human_review",
              "reviewRequired": ["raw answers", "source and grader disputes",
                                 "case regressions", "trace and tool path",
                                 "leakage/gaming", "usage and actual/unknown cost"]}
    with Path(target).open("x", encoding="utf-8") as file:
        json.dump(report, file, indent=2)
    print(f"{target} sha256={sha(Path(target).read_bytes())}")


if __name__ == "__main__":
    if len(sys.argv) != 6:
        raise SystemExit("usage: python compare.py <four-plan-paths> <comparison.json>")
    main(sys.argv[1:5], sys.argv[5])
```

```sh
python compare.py baseline-development/plan.json candidate-development/plan.json baseline-heldOut/plan.json candidate-heldOut/plan.json comparison.json
```

Build `comparison.json` from four bound result sets. Within the development split, join each case’s baseline and candidate grades; do the same separately for the disjoint held-out split. Each case has two outcomes in its own split, never four outcomes across both splits. Show pass→pass, fail→pass, pass→fail and error movements explicitly; aggregate percentages alone hide regressions. Inspect raw answers for memorized IDs, copied grader wording, answer leakage, altered tool paths or behavior that satisfies a narrow assertion while missing the task. Compare calls, tokens, latency and account billing as well as grades. Promptfoo cannot infer a complete bill for a Python provider that may invoke multiple models and tools. Usage is evidence of consumption, not a dollar invoice; report cost as unknown until reconciled with the account and tool charges.

The owner signs a retain-or-revert decision tied to the four immutable result hashes and the precommitted acceptance criteria. An error, missing case, unknown external effect, unexpected configuration, leaked held-out answer, unapproved lesson or unresolved grader dispute prevents promotion. A higher development score alone never proves improvement. If the held-out cases regress, retain neither the new lesson nor a claim of gain; keep the receipts and restore the prior immutable revision in the application. A small or unrepresentative held-out set supports only a narrow local decision.

If you later want recurring runs, make that a separate owner decision after the manual comparison. Specify schedule, data scope, spend cap and alert recipient. The job should stop on missing fixtures, an unknown or failed grade, uncertain tool effect, over-budget run or unapproved lesson, and require human promotion each time. No recurrence is needed to finish this guide.

## Sources & further reading

- [Promptfoo: Evaluate OpenAI Agents (Python SDK)](https://www.promptfoo.dev/docs/guides/evaluate-openai-agents-python/): Documents the Python provider and explicit tracing bridge. Its worked examples are an integration starting point, not a persistent learning system.
- [Promptfoo: Python provider](https://www.promptfoo.dev/docs/providers/python/): Defines the sync/async call shape, context and serializable response; identity and durable receipts are the application owner's work.
- [Promptfoo: Test cases](https://www.promptfoo.dev/docs/configuration/test-cases/): Documents file-backed JSON test rows and variable handling used for one frozen pass.
- [Promptfoo: JavaScript assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/): Shows custom binary grading results and failure reasons; source truth still comes from the task owner.
- [Promptfoo: CLI](https://www.promptfoo.dev/docs/usage/command-line/) and [caching](https://www.promptfoo.dev/docs/configuration/caching/): Document output, serial execution, sharing and cache controls to verify in the installed version.
- [OpenAI Agents SDK: agents](https://openai.github.io/openai-agents-python/agents/) and [running agents](https://openai.github.io/openai-agents-python/running_agents/): Document cloning, dynamic instructions, the runner and run configuration. They do not approve or persist lessons automatically.
- [BIG CHANGE's report on an agent skipping a required skill](https://bigchange.ai/blog/ai-agent-skill-evaluation-no-score-blind-spot) and [report on reward hacking between coding agents](https://bigchange.ai/blog/reward-hacking-spreads-between-coding-agents-study): Context for checking process and gaming separately from a final-answer score; neither establishes a result for your agent.

## Sources

- [Promptfoo: Evaluate OpenAI Agents (Python SDK)](https://www.promptfoo.dev/docs/guides/evaluate-openai-agents-python/) — Documents the Python provider integration and explicit tracing bridge. Its worked example does not supply persistent lesson approval or measured improvement.
- [Promptfoo: Python provider](https://www.promptfoo.dev/docs/providers/python/) — Async call_api(prompt, options, context), context identity fields and serializable output/error; the guide's durable IDs and receipts are application code.
- [Promptfoo: Test cases](https://www.promptfoo.dev/docs/configuration/test-cases/) — File-backed JSON tests and vars support the one-pass test file; they do not provide the real approved cases.
- [Promptfoo: JavaScript assertions](https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/) — Shows boolean or pass/score/reason checks and assertion failures; task-specific original evidence remains the source of truth.
- [Promptfoo: Command line](https://www.promptfoo.dev/docs/usage/command-line/) — Documents -c, -j, -o, --no-share and --no-cache controls, and privacy caveats for exports.
- [Promptfoo: Caching](https://www.promptfoo.dev/docs/configuration/caching/) — Fresh-run cache control; disabling this cache does not prove tools or model providers have no other state.
- [OpenAI Agents SDK: Agents](https://openai.github.io/openai-agents-python/agents/) — Documents Agent.clone, dynamic instructions and shallow-copy implications. Lesson registry and approval are custom application work.
- [OpenAI Agents SDK: Running agents](https://openai.github.io/openai-agents-python/running_agents/) — Documents Runner.run, max turns and RunConfig; exact model, tool and handoff effects require reader readback.