An agent can pass a test after you add a useful instruction. It can also pass because the test disclosed the answer, the grader was too easy, or a tool quietly changed. A feedback loop needs to tell those outcomes apart. This guide takes one existing OpenAI Agents SDK task agent through a baseline, a human-approved lesson, and a comparison on both familiar and untouched cases. A flat score or a regression is a valid result.

The example is an integration pattern, based on Promptfoo's Python Agents SDK guide and the SDK's agent and runner documentation. BIG CHANGE has not run your agent or measured a gain. You supply a real permissioned task, its existing agent, approved records and the code that checks its actual output. Promptfoo runs and records evaluations; it does not automatically diagnose errors, save lessons or decide to change an agent.

The big change

  • What changed: The SDK and Promptfoo can be joined around one existing agent, while an application-owned registry loads an approved lesson into its exact instructions.
  • Why it matters: Four bound development and held-out result sets let the task owner see individual fixes, regressions and evaluation shortcuts before retaining the change.
  • What to watch: The integration and real-task oracle require local implementation and testing. Results, traces and billing can contain sensitive data, and no improvement or cost has been measured here.

1. Freeze the task before changing the agent

Choose one bounded task with an output that its source owner can check. Put approved cases into fixtures/development/ and fixtures/heldout/. Each case needs an opaque ID, an input file, an expected-result file, the original record's ID/revision/hash, permission to process it, and a case-group ID. Keep paraphrases and answers from one source group in one split. Do not expose held-out inputs to whoever writes the lesson; never expose expected files or the oracle to the agent or its tools. A development case and at least one untouched held-out case are the structural minimum, not evidence that the set is statistically adequate.

Use this private manifest shape, replacing the field values with actual approved records. It is a schema example, not a fictional test case:

JSON
{"caseId":"<opaque-id>","split":"development|heldOut","caseGroup":"<source-cluster>","sourceId":"<original-record-id>","sourceRevision":"<revision>","sourceSha256":"<64 hex>","sourcePermissionReceipt":"<private-reference>","inputFile":"<private-UTF8-file>","inputSha256":"<64 hex>","expectedFile":"<private-JSON-file>","expectedSha256":"<64 hex>","oracleVersion":"<checked-version>","approvedBy":"<source-owner>","approvedAtUtc":"<timestamp>"}

Save one real row per line in fixtures/manifest.jsonl. The expected JSON should have a documented schema of required keys, types and order rules. The task owner compares it with the original source, not with an agent answer. Reject missing permissions, duplicate IDs, overlapping case groups, changed file hashes and any task whose tool permissions can read the private oracle or expected files. Use a fixture-controlled or read-only tool environment; do not replay a production write to obtain a score.

Record the existing agent's base instructions, model ID and access, model settings, tools, handoffs, permissions, SDK and Promptfoo versions, Python version, RunConfig, max turns, tracing setting, environment and approved spending limit in agent-config.json. Pin the installed versions and preserve pip freeze and the Promptfoo version output. The Promptfoo integration example describes Python 3.10+ and SDK 0.22.x; check its current requirements and your installed packages before adopting an example. Do not assume that any named model is available in your account. Record the resolved model and tool graph, including reachable handoffs. If you cannot identify or freeze them, this comparison cannot attribute a change to one lesson.

2. Connect Promptfoo to the existing agent

Freeze the fixture manifest, oracle, agent graph, tool permissions and approved budget before the first call. First create and run an immutable baseline-development plan. After source-backed diagnosis, the owner approves one exact lesson; only then can you freeze candidate-development. Lock the candidate instructions and precommitted decision rules before preparing the two held-out plans. baseline-heldOut and candidate-heldOut use the untouched held-out cases under the same manifest, oracle, base agent, tools and budget. Each of the four plans gets a distinct random runId, exact allowed case list and split, file/config hashes and private output directory. Baseline binds the original instructions and an empty lesson; both candidate plans bind the same approved lesson bytes and effective agent graph. A plan is immutable once created. No held-out input or result may influence the lesson or acceptance rule.

The following promptfooconfig.yaml is the Promptfoo file for one pass. The make_tests.py file below generates tests.json from only that pass's approved plan, with each row's input as vars.task_input and opaque ID as vars.case_id. Run it in the plan's private directory and place the provider file there too. Keep the answer and source record outside the config and agent tool sandbox. The separate private oracle below consumes the provider receipt after the run; a JavaScript assertion is optional if your installed Promptfoo version and trusted local assertion environment support it.

YAML
description: one frozen agent pass
prompts:
  - '{{task_input}}'
providers:
  - id: file://agent_provider.py
tests: file://tests.json
Python
"""Generate one pass's Promptfoo tests from its approved private plan."""

import hashlib
import json
import sys
from pathlib import Path


def sha(data):
    return hashlib.sha256(data).hexdigest()


def main(plan_path, plan_sha):
    raw = Path(plan_path).read_bytes()
    if sha(raw) != plan_sha:
        raise ValueError("Plan hash differs")
    plan = json.loads(raw)
    tests = []
    seen = set()
    for row in plan["cases"]:
        case_id = row["caseId"]
        if case_id in seen or row["split"] != plan["split"]:
            raise ValueError("Duplicate case or wrong split")
        seen.add(case_id)
        source = Path(row["inputFile"]).read_bytes()
        if sha(source) != row["inputSha256"]:
            raise ValueError(f"Changed case input: {case_id}")
        tests.append({"vars": {"case_id": case_id,
                                "task_input": source.decode("utf-8")}})
    if not tests:
        raise ValueError("No approved cases in plan")
    target = Path(plan_path).with_name("tests.json")
    with target.open("x", encoding="utf-8") as file:
        json.dump(tests, file, ensure_ascii=False, indent=2)
    print(f"{target} sha256={sha(target.read_bytes())} cases={len(tests)}")


if __name__ == "__main__":
    if len(sys.argv) != 3:
        raise SystemExit("usage: python make_tests.py <private-plan.json> <plan-sha256>")
    main(sys.argv[1], sys.argv[2])

Use exactly one test per case, one provider and one prompt, with no repeats. Promptfoo's Python provider contract accepts async def call_api(prompt, options, context) and a serializable response with output or error. Its worker can persist across calls, so the provider must read a fixed plan each time and keep no mutable lesson, session, case counter or result in module globals. The official example uses a shared SQLite session and permissive output conversion; those are unsuitable for independent case identity and strict final-answer grading here.

Copy the following file as agent_provider.py. The task owner supplies task_adapter.py from the actual application; there is no generic agent or real fixture to import from this article. That adapter exports build_existing_agent() (the unchanged Agent and its fixture-safe tools), build_run_config() (the frozen SDK RunConfig), graph_fingerprint(agent, run_config) (SHA-256 of a canonical inventory of starting and reachable handoff agents, their effective models/settings/instructions, tool identities/permissions and RunConfig overrides), and result_evidence(result) (JSON-safe actual usage, last agent/model, tool/trace IDs or an explicit unavailable marker). Review that inventory and its serialization for the installed SDK version before a paid run. A non-enumerable dynamic handoff or model choice blocks the comparison. The provider imports the real Runner and calls it only after its checks.

Python
"""Reader adaptation for one frozen Promptfoo/Agents SDK pass.

Copy to agent_provider.py in a private evaluation directory. Supply task_adapter.py
from the existing application and an owner-approved immutable plan.json.
No model call occurs merely by importing this module.
"""

import hashlib
import json
import os
import re
from pathlib import Path

from agents import Runner
from pydantic import BaseModel
from task_adapter import build_existing_agent, build_run_config, graph_fingerprint, result_evidence


OPAQUE = re.compile(r"^[A-Za-z0-9_-]{1,80}$")


def digest(data):
    return hashlib.sha256(data).hexdigest()


def read_checked(path, expected_hash):
    data = Path(path).read_bytes()
    if digest(data) != expected_hash:
        raise ValueError(f"Hash mismatch: {path}")
    return data


def json_bytes(value):
    return json.dumps(value, sort_keys=True, ensure_ascii=False,
                      separators=(",", ":"), allow_nan=False).encode("utf-8")


def strict_output(value):
    if value is None:
        raise ValueError("Agent ended without final_output")
    if isinstance(value, str):
        return value
    if isinstance(value, BaseModel):
        value = value.model_dump(mode="json")
    if isinstance(value, (dict, list)):
        return json_bytes(value).decode("utf-8")
    raise TypeError(f"Unsupported final_output type: {type(value).__name__}")


def create_once(path, value):
    path = Path(path)
    path.parent.mkdir(parents=True, exist_ok=True)
    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
    try:
        with os.fdopen(fd, "wb") as file:
            file.write(json_bytes(value))
            file.flush()
            os.fsync(file.fileno())
    except BaseException:
        # An incomplete pending record is deliberately left for reconciliation.
        raise
    return digest(path.read_bytes())


def load_plan():
    path = os.environ["LOOP_PLAN_PATH"]
    required_hash = os.environ["LOOP_PLAN_SHA256"]
    plan = json.loads(read_checked(path, required_hash))
    if not OPAQUE.fullmatch(plan["runId"]):
        raise ValueError("Invalid runId")
    if plan["attempt"] != 0 or plan["variant"] not in ("baseline", "candidate"):
        raise ValueError("Unexpected attempt or variant")
    if plan["split"] not in ("development", "heldOut"):
        raise ValueError("Unexpected split")
    return plan, required_hash


async def call_api(prompt: str, options: dict, context: dict) -> dict:
    pending = False
    try:
        plan, plan_hash = load_plan()
        case_id = context["vars"]["case_id"]
        evaluation_id = context["evaluationId"]
        test_id = context["testCaseId"]
        repeat = context.get("repeatIndex")
        if not all(isinstance(v, str) and v for v in (case_id, evaluation_id, test_id)):
            raise ValueError("Missing case/evaluation identity")
        if not OPAQUE.fullmatch(case_id) or repeat not in (None, 0):
            raise ValueError("Invalid case ID or repeated test")
        cases = [row for row in plan["cases"] if row["caseId"] == case_id]
        if len(cases) != 1:
            raise ValueError("Case is absent or duplicated in plan")
        row = cases[0]
        if row["split"] != plan["split"]:
            raise ValueError("Case split differs from plan")
        read_checked(plan["fixtureManifestFile"], plan["fixtureManifestSha256"])
        read_checked(plan["agentConfigFile"], plan["agentConfigSha256"])
        budget = json.loads(read_checked(plan["budgetFile"], plan["budgetSha256"]))
        settings = json.loads(read_checked(plan["toolSettingsFile"],
                                           plan["toolSettingsSha256"]))
        if (budget["status"] != "approved" or not budget["approvedBy"] or
            budget["modelId"] != plan["modelId"] or
            budget["maxTurns"] != plan["maxTurns"] or
            settings["status"] != "approved" or not settings["approvedBy"]):
            raise ValueError("Budget or tool settings lack matching approval")
        task_input = read_checked(row["inputFile"], row["inputSha256"]).decode("utf-8")
        if prompt != task_input:
            raise ValueError("Prompt differs from frozen input")

        private_dir = Path(plan["resultDir"])
        if private_dir.is_symlink():
            raise ValueError("Private result directory is linked")
        outdir = private_dir.resolve(strict=True)
        if not outdir.is_dir():
            raise ValueError("Private result directory is missing or linked")
        binding = outdir / "evaluation-id.json"
        if not binding.exists():
            try:
                create_once(binding, {"runId": plan["runId"],
                                      "evaluationId": evaluation_id})
            except FileExistsError:
                pass
        if json.loads(binding.read_text("utf-8")) != {
            "runId": plan["runId"], "evaluationId": evaluation_id
        }:
            raise ValueError("Evaluation ID changed within the run")
        audit_key = digest(json_bytes([evaluation_id, test_id, 0]))
        create_once(outdir / "audit" / f"{audit_key}.json",
                    {"runId": plan["runId"], "caseId": case_id,
                     "evaluationId": evaluation_id, "testCaseId": test_id,
                     "repeatIndex": 0})
        pending_file = outdir / "cases" / f"{case_id}.pending.json"
        create_once(pending_file, {"runId": plan["runId"], "caseId": case_id,
                                   "planSha256": plan_hash, "attempt": 0})
        pending = True

        base = read_checked(plan["baseInstructionsFile"], plan["baseInstructionsSha256"])
        lesson = read_checked(plan["lessonFile"], plan["lessonSha256"])
        if plan["variant"] == "baseline" and (lesson or plan["separator"]):
            raise ValueError("Baseline lesson and separator must be empty")
        if plan["variant"] == "candidate":
            if not lesson:
                raise ValueError("Candidate lesson is empty")
            approval = json.loads(read_checked(plan["lessonApprovalFile"],
                                              plan["lessonApprovalSha256"]))
            for key in ("lessonId", "lessonRevision", "lessonSha256", "taskAgentId"):
                if approval[key] != plan[key]:
                    raise ValueError(f"Lesson approval differs: {key}")
            if approval["status"] != "approved" or not approval["approvedBy"] or not approval["approvedAtUtc"]:
                raise ValueError("Exact lesson lacks owner approval")
        rendered = base + plan["separator"].encode("utf-8") + lesson
        if digest(rendered) != plan["renderedInstructionsSha256"]:
            raise ValueError("Rendered instruction hash differs")

        original = build_existing_agent()
        run_config = build_run_config()
        if graph_fingerprint(original, run_config) != plan["baseGraphSha256"]:
            raise ValueError("Base agent/tool/handoff/RunConfig graph changed")
        # clone is shallow: neither this file nor the adapter may mutate shared lists.
        agent = original.clone(model=plan["modelId"],
                               instructions=rendered.decode("utf-8"))
        if graph_fingerprint(agent, run_config) != plan["effectiveGraphSha256"]:
            raise ValueError("Effective graph or model override changed")

        result = await Runner.run(agent, task_input, max_turns=plan["maxTurns"],
                                  run_config=run_config)
        output = strict_output(result.final_output)
        evidence = result_evidence(result)  # JSON-safe usage, model, tools, trace ID
        terminal = {"runId": plan["runId"], "caseId": case_id, "attempt": 0,
                    "planSha256": plan_hash, "variant": plan["variant"],
                    "split": plan["split"], "inputSha256": row["inputSha256"],
                    "configSha256": plan["agentConfigSha256"],
                    "lessonSha256": plan["lessonSha256"],
                    "renderedInstructionsSha256": digest(rendered),
                    "output": output, "evidence": evidence}
        receipt_hash = create_once(outdir / "cases" / f"{case_id}.result.json", terminal)
        return {"output": output,
                "metadata": {"runId": plan["runId"], "caseId": case_id,
                             "attempt": 0, "receiptSha256": receipt_hash}}
    except Exception as exc:
        # Preserve the error as an error; never turn it into plausible model text.
        if pending:
            try:
                create_once(outdir / "cases" / f"{case_id}.error.json",
                            {"runId": plan["runId"], "caseId": case_id,
                             "errorType": type(exc).__name__, "error": str(exc)})
            except Exception:
                pass  # A pending record without terminal readback blocks promotion.
        return {"error": f"Case failed: {type(exc).__name__}; inspect private receipt"}

This file is untested reference code until the reader implements and reviews that application adapter, its file permissions and the real plan. strict_output accepts exact text, a JSON-compatible dict/list encoded canonically, or a configured Pydantic result through model_dump(mode="json"); it rejects None, non-finite values and unknown objects. Agent.clone is a shallow copy: do not mutate shared tool or handoff lists. A changed tool, handoff, model, permission or budget invalidates the one-lesson comparison.

Each private plan.json supplies runId, attempt: 0, variant, split, and cases. Every case carries caseId, split, caseGroup, sourceId, sourceRevision, sourceSha256, inputFile, inputSha256, expectedFile, and expectedSha256 from the approved manifest. The plan also carries fixtureManifestFile/Sha256, agentConfigFile/Sha256, budgetFile/Sha256, toolSettingsFile/Sha256, oracleSha256, oracleVersion, resultDir, baseInstructionsFile/Sha256, lessonFile/Sha256, separator, renderedInstructionsSha256, modelId, baseGraphSha256, effectiveGraphSha256, and maxTurns. The owner-approved budget JSON includes status, approvedBy, modelId, maxTurns, and its actual call/spend limits; the approved tool-settings JSON includes status, approvedBy, tool identities and permissions. The adapter's graph fingerprint must reflect those effective limits and settings. All four plans bind the same manifest, oracle, budget, tool settings, base graph, model and max turns; the two variants intentionally differ in rendered instructions, effective graph and lesson hash. The comparison checks case IDs, source IDs and case groups are disjoint across the splits, but the owner still inspects semantic near-duplicates and keeps held-out material away from lesson writing.

Candidate plans also supply lessonApprovalFile, lessonApprovalSha256, lessonId, lessonRevision, and taskAgentId. The private approval JSON contains lessonId, lessonRevision, taskAgentId, lessonSha256, status: "approved", approvedBy, and approvedAtUtc. The provider accepts only its planned case IDs. Set oracleSha256 to the exact grade.py code hash; the grader verifies it before using private expected files. Baseline uses a zero-byte lesson file, its SHA-256 and an empty separator, so its rendered instructions equal the existing base bytes; candidate uses the reviewed separator and exact approved revision. Compute all hashes from bytes, not normalized text. Set LOOP_PLAN_PATH to its fixed private absolute path and LOOP_PLAN_SHA256 to the hash checked independently before invoking Promptfoo. Never pass a path or a secret through a test variable. Make the result directory private and empty before the run. If plan generation or adapter checks fail, stop before a model call.

Promptfoo may reuse evaluationId for many cases. The durable primary key is (runId, caseId, attempt=0), with (evaluationId, testCaseId, repeatIndex) as an additional unique audit tuple inside that run. Create a pending receipt exclusively and flush it before a model call; write a terminal result or error through exclusive creation, flush it, and read its hash back. The reference file does not use an atomic rename: a crash can leave a partial terminal file. Treat a missing, partial or conflicting terminal file as unresolved, inspect any external effect, and never replay it automatically. Check any external tool effect before considering a new, explicitly approved attempt. Do not let automatic retries silently rerun a case. Call Runner.run without a shared session or prior-response ID. If the task requires several turns, give each case a fresh session and bind its ID to the durable key.

For a normal trial, run one pass at a time from its private plan. Check current CLI options against your pinned installation, then use its equivalent of:

Terminal
python make_tests.py private-plan.json <verified-plan-sha256>
promptfoo eval -c promptfooconfig.yaml --no-cache --no-share -j 1 -o private-results/<runId>/promptfoo.json
python grade.py private-plan.json <verified-plan-sha256>

--no-cache avoids a cached answer being mistaken for a candidate call; --no-share prevents a share URL. Neither controls your application logs, SDK traces or account retention. Save the command, start/end time, exit code, config and test hashes, output hash, and the provider's per-case receipts. Disable sensitive trace capture or restrict private traces before submitting real records. If you assert that a particular tool or handoff was used, configure and verify the integration guide's separate trace bridge; a provider span alone does not prove that internal path.

3. Grade against the original, then diagnose

The owner writes a deterministic oracle for this task's expected JSON schema. The source owner first verifies each expected file against the original record and freezes its revision/hash. For every attempted case, the private oracle opens that verified expected file outside the agent process, parses the raw final output, and returns pass: true/false, score: 1/0, a concrete reason, and its oracle version. It fails closed on missing output, malformed JSON, missing required fields, wrong value, changed expected bytes or an oracle exception. Keep errors distinct from a genuine failed answer. Promptfoo supports custom pass/score/reason assertions, but a private post-run oracle is also valid when the expected files cannot safely enter Promptfoo. Never pass the private answer to the model in a prompt, tool result, or provider metadata.

The configuration above intentionally has no Promptfoo assertion, because the expected files stay private. Its displayed pass rate is not the task grade. Use the private grades.json as the binary outcome, and match every receipt to its Promptfoo row. If a different task needs to verify an actual tool action, extend the oracle to inspect the authorized trace or external-effect readback before declaring a pass; exact final JSON alone cannot prove that action.

For an exact structured-output task, the following private grade.py is a complete deterministic oracle file. Use it only when whole-JSON equality matches the source owner's approved task rule; otherwise implement the real task's rule with the same error handling and source checks. Run it outside the agent/tool environment after each pass, using the plan path and hash. It writes one private grades.json and refuses to overwrite a prior grade.

Python
"""Private, deterministic exact-JSON grader for one frozen run.

Use only if exact structured equality matches the real task's approved criterion.
The expected files stay outside the agent/provider environment.
"""

import hashlib
import json
import os
import sys
from pathlib import Path


def sha(data):
    return hashlib.sha256(data).hexdigest()


def checked_json(path, expected_sha):
    data = Path(path).read_bytes()
    if sha(data) != expected_sha:
        raise ValueError(f"Hash mismatch: {path}")
    return json.loads(data)


def canonical(value):
    return json.dumps(value, sort_keys=True, ensure_ascii=False,
                      separators=(",", ":"), allow_nan=False)


def main(plan_path, plan_sha):
    plan = checked_json(plan_path, plan_sha)
    if sha(Path(__file__).read_bytes()) != plan["oracleSha256"]:
        raise ValueError("Oracle code differs from frozen plan")
    directory = Path(plan["resultDir"])
    grades = []
    for row in plan["cases"]:
        case_id = row["caseId"]
        expected = checked_json(row["expectedFile"], row["expectedSha256"])
        result_file = directory / "cases" / f"{case_id}.result.json"
        error_file = directory / "cases" / f"{case_id}.error.json"
        if result_file.exists() and error_file.exists():
            outcome = {"status": "error", "pass": False, "score": 0,
                       "reason": "Conflicting terminal result and error receipts"}
        elif not result_file.exists():
            outcome = {"status": "error", "pass": False, "score": 0,
                       "reason": "No terminal result; inspect pending/error receipt"}
            if error_file.exists():
                outcome["errorReceiptSha256"] = sha(error_file.read_bytes())
        else:
            result_bytes = result_file.read_bytes()
            result = json.loads(result_bytes)
            if (result["planSha256"] != plan_sha or
                result["caseId"] != case_id or
                result["inputSha256"] != row["inputSha256"] or
                result["lessonSha256"] != plan["lessonSha256"]):
                outcome = {"status": "error", "pass": False, "score": 0,
                           "reason": "Receipt binding differs from frozen plan"}
            else:
                try:
                    actual = json.loads(result["output"])
                    passed = canonical(actual) == canonical(expected)
                    outcome = {"status": "pass" if passed else "fail",
                               "pass": passed, "score": 1 if passed else 0,
                               "reason": "Exact approved JSON match" if passed else
                                         "Output differs from approved JSON"}
                except (ValueError, TypeError, KeyError):
                    outcome = {"status": "fail", "pass": False, "score": 0,
                               "reason": "Final output is not valid task JSON"}
            outcome["resultReceiptSha256"] = sha(result_bytes)
        grades.append({"runId": plan["runId"], "caseId": case_id,
                       "split": plan["split"], "variant": plan["variant"], **outcome})
    target = directory / "grades.json"
    with target.open("x", encoding="utf-8") as file:
        json.dump({"planSha256": plan_sha, "oracleVersion": "exact-json-v1",
                   "grades": grades}, file, indent=2)
        file.flush()
        os.fsync(file.fileno())
    print(f"{target} sha256={sha(target.read_bytes())} cases={len(grades)}")


if __name__ == "__main__":
    if len(sys.argv) != 3:
        raise SystemExit("usage: python grade.py <private-plan.json> <plan-sha256>")
    main(sys.argv[1], sys.argv[2])

For each baseline development failure, the reviewer checks the raw answer, original record, oracle result and authorized tool evidence. A bad expected file, grader defect, unavailable source or failed tool call is not an agent lesson. Repair the source or grader, freeze the corrected definition, and rerun the baseline before comparing it with any candidate. Only an actual agent mistake gets a diagnosis receipt: case ID, source pointer, observed behavior, suspected cause, alternative explanation, reviewer and proposed narrow instruction. The agent's account of its own mistake is a lead, not evidence.

The owner approves one lesson's exact bytes and scope. Save an append-only registry entry with its ID, revision, text hash, source development cases, approving person and time, task/agent scope and rollback pointer. Record rejection too. The provider loads that exact approved revision into the cloned same agent's instructions; a saved note or SDK session history is not a loaded lesson. Keep the baseline revision available for a revert.

4. Compare all four result sets

Run candidate development with the same case IDs, oracle, model/tool graph and limits. Then run baseline held-out and candidate held-out. The lesson writer sees neither held-out inputs nor results until the candidate instructions are frozen. Each of the four result directories must contain every planned case's input hash, raw output or explicit error, binary grade and reason, run/config/lesson/rendered-instruction hashes, trace ID or unavailable marker, actual calls/tokens/latency where exposed, and actual or unknown dollar cost. Check the run receipt against the Promptfoo export; a missing row is not a pass.

The following compare.py joins the four private plans and grade files, refuses a changed case set or base configuration, and leaves promotion to the owner. Run it after grading all four passes. Inspect the raw receipts in addition to its case movements.

Python
"""Bind four frozen runs; show within-split case movements for human review."""

import hashlib
import json
import sys
from pathlib import Path


SLOTS = ("baseline-development", "candidate-development",
         "baseline-heldOut", "candidate-heldOut")
COMMON = ("fixtureManifestSha256", "oracleSha256", "agentConfigSha256",
          "baseInstructionsSha256", "baseGraphSha256", "modelId", "maxTurns",
          "budgetSha256", "toolSettingsSha256")
CASE_KEYS = ("caseId", "caseGroup", "sourceId", "sourceRevision",
             "sourceSha256", "inputSha256", "expectedSha256")
BASELINE = ("lessonSha256", "renderedInstructionsSha256", "effectiveGraphSha256")
CANDIDATE = BASELINE + ("lessonId", "lessonRevision", "taskAgentId",
                        "lessonApprovalSha256")


def sha(data):
    return hashlib.sha256(data).hexdigest()


def load_plan(path):
    raw = Path(path).read_bytes()
    return json.loads(raw), sha(raw)


def require_same(plans, keys, label):
    for key in keys:
        values = [plan[key] for plan in plans]
        if not all(value == values[0] for value in values[1:]):
            raise ValueError(f"{label} differs: {key}")


def case_map(plan):
    rows = plan["cases"]
    mapped = {row["caseId"]: row for row in rows}
    if not rows or len(mapped) != len(rows):
        raise ValueError("Empty or duplicate planned cases")
    for row in rows:
        if row["split"] != plan["split"] or not all(row.get(k) for k in CASE_KEYS):
            raise ValueError("Missing case provenance or wrong split")
    return mapped


def validated_grades(plan, plan_hash):
    raw = (Path(plan["resultDir"]) / "grades.json").read_bytes()
    report = json.loads(raw)
    if report["planSha256"] != plan_hash or report["oracleVersion"] != plan["oracleVersion"]:
        raise ValueError("Grades belong to another plan or oracle")
    planned = case_map(plan)
    rows = report["grades"]
    mapped = {row["caseId"]: row for row in rows}
    if len(rows) != len(mapped) or len(rows) != len(planned) or set(mapped) != set(planned):
        raise ValueError("Duplicate, missing or extra case grade")
    for row in rows:
        if (row["runId"] != plan["runId"] or row["split"] != plan["split"] or
            row["variant"] != plan["variant"]):
            raise ValueError("Grade identity differs from plan")
        expected = {"pass": (True, 1), "fail": (False, 0),
                    "error": (False, 0)}
        if row["status"] not in expected:
            raise ValueError("Unknown grade status")
        if (type(row["pass"]) is not bool or type(row["score"]) is not int or
            (row["pass"], row["score"]) != expected[row["status"]] or
            not isinstance(row.get("reason"), str) or not row["reason"]):
            raise ValueError("Inconsistent binary grade")
    return mapped, sha(raw)


def main(paths, target):
    if len(paths) != 4:
        raise ValueError("Four distinct plans required")
    runs = {}
    for path in paths:
        plan, plan_hash = load_plan(path)
        slot = f'{plan["variant"]}-{plan["split"]}'
        if slot not in SLOTS or slot in runs:
            raise ValueError("Duplicate or unexpected pass")
        grade_map, grades_hash = validated_grades(plan, plan_hash)
        runs[slot] = {"plan": plan, "planSha256": plan_hash,
                      "gradesSha256": grades_hash, "grades": grade_map}
    if set(runs) != set(SLOTS):
        raise ValueError("A required pass is missing")
    if len({runs[slot]["plan"]["runId"] for slot in SLOTS}) != 4:
        raise ValueError("Run IDs must be distinct")

    plans = [runs[slot]["plan"] for slot in SLOTS]
    require_same(plans, COMMON + ("oracleVersion",), "Four-pass frozen setup")
    require_same([runs[s]["plan"] for s in SLOTS if s.startswith("baseline")],
                 BASELINE, "Baseline across splits")
    require_same([runs[s]["plan"] for s in SLOTS if s.startswith("candidate")],
                 CANDIDATE, "Candidate across splits")
    if plans[0]["lessonSha256"] == plans[1]["lessonSha256"]:
        raise ValueError("Candidate did not change the lesson")

    split_cases = {}
    for split in ("development", "heldOut"):
        left = case_map(runs[f"baseline-{split}"]["plan"])
        right = case_map(runs[f"candidate-{split}"]["plan"])
        if left != right:
            raise ValueError(f"{split} case input, truth or provenance changed")
        split_cases[split] = left
    development = split_cases["development"]
    heldout = split_cases["heldOut"]
    if set(development) & set(heldout):
        raise ValueError("Case ID appears in development and held-out")
    if ({row["caseGroup"] for row in development.values()} &
        {row["caseGroup"] for row in heldout.values()}):
        raise ValueError("Case group leaks across splits")
    if ({row["sourceId"] for row in development.values()} &
        {row["sourceId"] for row in heldout.values()}):
        raise ValueError("Original source appears in both splits")

    movements = {}
    blocking_errors = []
    for split in ("development", "heldOut"):
        base = runs[f"baseline-{split}"]["grades"]
        candidate = runs[f"candidate-{split}"]["grades"]
        movements[split] = []
        for case_id in sorted(split_cases[split]):
            pair = {"caseId": case_id, "baseline": base[case_id]["status"],
                    "candidate": candidate[case_id]["status"]}
            movements[split].append(pair)
            if "error" in (pair["baseline"], pair["candidate"]):
                blocking_errors.append({"split": split, "caseId": case_id})
    report = {"runBindings": {slot: {"runId": runs[slot]["plan"]["runId"],
                                      "planSha256": runs[slot]["planSha256"],
                                      "gradesSha256": runs[slot]["gradesSha256"]}
                              for slot in SLOTS},
              "movements": movements, "blockingErrors": blocking_errors,
              "promotionEligibleForHumanReview": not blocking_errors,
              "promotion": "blocked_until_human_review",
              "reviewRequired": ["raw answers", "source and grader disputes",
                                 "case regressions", "trace and tool path",
                                 "leakage/gaming", "usage and actual/unknown cost"]}
    with Path(target).open("x", encoding="utf-8") as file:
        json.dump(report, file, indent=2)
    print(f"{target} sha256={sha(Path(target).read_bytes())}")


if __name__ == "__main__":
    if len(sys.argv) != 6:
        raise SystemExit("usage: python compare.py <four-plan-paths> <comparison.json>")
    main(sys.argv[1:5], sys.argv[5])
Terminal
python compare.py baseline-development/plan.json candidate-development/plan.json baseline-heldOut/plan.json candidate-heldOut/plan.json comparison.json

Build comparison.json from four bound result sets. Within the development split, join each case’s baseline and candidate grades; do the same separately for the disjoint held-out split. Each case has two outcomes in its own split, never four outcomes across both splits. Show pass→pass, fail→pass, pass→fail and error movements explicitly; aggregate percentages alone hide regressions. Inspect raw answers for memorized IDs, copied grader wording, answer leakage, altered tool paths or behavior that satisfies a narrow assertion while missing the task. Compare calls, tokens, latency and account billing as well as grades. Promptfoo cannot infer a complete bill for a Python provider that may invoke multiple models and tools. Usage is evidence of consumption, not a dollar invoice; report cost as unknown until reconciled with the account and tool charges.

The owner signs a retain-or-revert decision tied to the four immutable result hashes and the precommitted acceptance criteria. An error, missing case, unknown external effect, unexpected configuration, leaked held-out answer, unapproved lesson or unresolved grader dispute prevents promotion. A higher development score alone never proves improvement. If the held-out cases regress, retain neither the new lesson nor a claim of gain; keep the receipts and restore the prior immutable revision in the application. A small or unrepresentative held-out set supports only a narrow local decision.

If you later want recurring runs, make that a separate owner decision after the manual comparison. Specify schedule, data scope, spend cap and alert recipient. The job should stop on missing fixtures, an unknown or failed grade, uncertain tool effect, over-budget run or unapproved lesson, and require human promotion each time. No recurrence is needed to finish this guide.

Sources & further reading