# Calibrate a judge for a Claude Code Skill revision
> Calibrate a source-grounded second-model judge for each criterion, then compare one owned text-output Claude Code Skill revision with its baseline on real development and sealed held-out cases.
By BIGCHANGE.AI Editorial
Published: 2026-10-27T06:31:38.333Z
Updated: 2026-10-27T06:31:38.333Z
Canonical: https://bigchange.ai/blog/calibrated-claude-code-skill-judge-guide
AI-generated conceptual illustration by BIG CHANGE.
An owned Claude Code Skill can produce the required words while still making a claim the source does not support. A string check can catch a missing phrase, but it cannot decide whether the answer faithfully used the source. This guide adds a second model to judge that narrower question against owner-approved truth. The owner calibrates the judge on known pass, fail and insufficient-evidence examples for each criterion before it grades any trial.
The method is for one self-contained Skill that returns text. It compares the current Skill with one revision on real, permissioned task records. The development cases expose a failure and test the repair; a separate source-family split stays sealed until the revision is fixed. The final output is a comparison and an exact human decision. No command below installs or activates a Skill.
## The big change
- **What changes:** A source-grounded second model can assess a criterion that literal string checks cannot, with its own calibration receipts and an explicit unknown result when evidence is thin.
- **How it works:** The same frozen input, truth, tests, model controls and one-attempt rule bind both Skill revisions. Every calibration and grading call records its input and output hashes before a score is computed.
- **Why it matters:** A passing text match cannot establish source fidelity. The owner can inspect the judged answer, disagreements and regressions before choosing accept, hold or revert; the decision does not activate the revision.
## 1. Prepare the Skill, evidence and account
You need Python 3.11 or later; an owned Claude Code Skill that fits entirely in one `SKILL.md` and returns text; and approved task inputs with original source records and expected answers. The source owner must permit the selected inputs and truth to be sent to the selected models. Choose normal, edge and stop cases from the work the Skill is supposed to do. Give each original record an ID and revision. Put related records in the same `caseGroup`, and keep each family wholly in development or held-out. Keep the input, expected checks and truth outside the isolated Skill project.
Before any call, verify CLI 2.1.286+, two model IDs, billed access, CLI help and [pricing](https://platform.claude.com/docs/en/about-claude/pricing); set `ANTHROPIC_API_KEY` for bare mode. Budget caps can overshoot. Keep the key out of files and transcripts.
Calls deny tools, MCP and permission prompts. These controls do not prove isolation. A Skill needing tools, supporting files or external writes requires another test; text cannot prove a state change.
Create a private experiment directory containing `reader-kit/eval_loop.py` and the templates printed at the end of this article. Keep the four plan files, evidence files and receipts beside that directory. For each revision, make a separate project with only `.claude/skills/<name>/SKILL.md`; do not put task inputs or truth in it. The runner rejects extra files or links there. Give the Skill simple flat frontmatter with its exact name and a description. The narrow parser in this kit intentionally rejects many otherwise valid YAML forms and privileged frontmatter. Choose another evaluation path if your Skill needs those forms.
```text
experiment/
reader-kit/eval_loop.py
environment.json policy.json calibration-plan.json
calibration-approval.json calibration.json
baseline-development-plan.json candidate-development-plan.json
baseline-heldout-plan.json candidate-heldout-plan.json
baseline-project/.claude/skills/<name>/SKILL.md
candidate-project/.claude/skills/<name>/SKILL.md
private-inputs/ private-truth/ private-expected/
calibration-receipts/ results-baseline-dev/ results-candidate-dev/
results-baseline-heldout/ results-candidate-heldout/
```
The printed `reader-kit/*.template.json` blocks are shapes to copy into the actual paths below, then fill with your evidence. Keep `eval_loop.py` at `reader-kit/eval_loop.py`. The program creates the receipts and scores; it also creates `calibration.json` during finalization, so do not copy `calibration.template.json` over it.
| Printed template | Actual file |
| --- | --- |
| `environment.template.json`, `policy.template.json` | `environment.json`, `policy.json` |
| `calibration-plan.template.json`, `calibration-approval.template.json` | `calibration-plan.json`, `calibration-approval.json` |
| `expected.template.json` | One `private-expected/<case-id>.json` per case |
| `plan.template.json` | `baseline-development-plan.json`, `candidate-development-plan.json`, `baseline-heldout-plan.json`, `candidate-heldout-plan.json` |
| `diagnosis.template.json`, `heldout-unlock.template.json`, `decision.template.json` | `diagnosis.json`, `heldout-unlock.json`, `decision.json` |
| `calibration.template.json` | Reference shape only; `calibration.json` is generated |
Replace every angle-bracket placeholder in the templates with owned evidence. `sha256` means the SHA-256 of the **exact file bytes** named by `file`, relative to the plan's directory. On a shell with Python, `python -c "import hashlib,sys; print(hashlib.sha256(open(sys.argv[1],'rb').read()).hexdigest())" PATH` prints a file hash. The criterion file used in calibration must contain exactly the `criterion` string in the matching expected test. In particular, do not accidentally add a newline to one side. `environment.json` fixes the executable, models, account label, budgets and timeout. `policy.json` fixes the test version, thresholds, one attempt per case and accepted calibration hash. Preserve the approved bytes of both during all four runs.
For each case, `input` is the actual task text, `judgeContext` is bounded source truth the owner approved, and `expected` lists fixed checks. Literal `contains`, `not_contains` and `regex` checks work on output text. A `judge` test holds one binary source-grounded criterion, such as whether every material factual statement in the answer is supported by the supplied record. Write the criterion narrowly enough for a human to label. A model verdict of `unknown` means the supplied evidence cannot decide it. The judge sees the real task input, that truth, the criterion and the Skill's collected output; it has no tools. The output itself is untrusted evidence, so a confident unsupported claim should fail, not become truth through repetition.
## 2. Calibrate each judge criterion first
Before any Skill run, assemble retained owner-reviewed examples for **each exact criterion**. Include at least one known pass, one known fail and one insufficient-evidence example. Each sample has its own real task input, approved truth, known output, criterion file, source-backed `humanEvidence` and reviewer. Keep calibration sources and their families outside the held-out cases. These labels are the human reference for calibration, not generated answers to copy into the test set. Anthropic's [agent-evaluation guidance](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) explains why grader checks and the task's actual outcome both need attention.
Complete `calibration-plan.json` from the template, with one sample object per label for every criterion. Then run:
```sh
python reader-kit/eval_loop.py calibrate-run calibration-plan.json
```
That command makes real judge-model calls. It writes a pending receipt before each call and a terminal receipt afterward. Compare **every** returned verdict and explanation with its owner-approved source. If any verdict differs, the calibration cannot finalize: refine the criterion or evidence, version the plan, and review new examples. Enter the exact plan hash and every terminal receipt hash in `calibration-approval.json`, along with the owner and approval time. Finalize only after that inspection:
```sh
python reader-kit/eval_loop.py calibrate-finalize calibration-plan.json calibration-approval.json
```
The generated `calibration.json` binds the sample verdicts and receipt hashes to the criterion bytes, judge model, prompt, schema and environment. Put its hash in `policy.json`, approve that frozen policy, and use the same policy for all four plans. A missing or mismatched criterion calibration stops grading for that criterion. This is a one-attempt test: a pending receipt without a terminal result signals uncertainty, so inspect the actual effects and write a newly reviewed plan rather than retrying the old call.
## 3. Find the baseline failure and test one revision
Fill `baseline-development-plan.json` from `plan.template.json`. It points to the original Skill file and the development cases. Each case carries the input, expected tests, bounded truth and their hashes. Confirm the policy approval time precedes the run, then execute:
```sh
python reader-kit/eval_loop.py run baseline-development-plan.json
python reader-kit/eval_loop.py judge baseline-development-plan.json CASE_ID TEST_ID
python reader-kit/eval_loop.py score baseline-development-plan.json
```
Run the `judge` line once for **every judge test in every successfully completed case**, changing the IDs. Literal tests need no judge call. Inspect the output and each source record as well as `score.json`. A failed Skill call, missing answer or missing judge result is `unknown`, not a pass. The score is the fraction of all fixed checks marked pass; unknown stays in the denominator. A critical check with any result other than pass blocks eligibility.
Record the observed failure and its source evidence in `diagnosis.json`. Bind the baseline plan, its terminal receipts, its score and the proposed candidate Skill hash. The owner approves this precise revision for testing at a recorded time. Keep the diagnosis private. The approval is only for the candidate trial. Copy `plan.template.json` to `candidate-development-plan.json`; retain the development cases and controls, set `variant` to `candidate`, change `projectDir`, `skillSha256` and `resultDir` to the approved candidate, and add this **top-level** member only after `diagnosis.json` exists:
```json
"diagnosis": {"file": "diagnosis.json", "sha256": "<sha256 of exact diagnosis.json bytes>"}
```
Then run, judge every applicable output, and score:
```sh
python reader-kit/eval_loop.py run candidate-development-plan.json
python reader-kit/eval_loop.py judge candidate-development-plan.json CASE_ID TEST_ID
python reader-kit/eval_loop.py score candidate-development-plan.json
```
The development result can justify opening the holdout. It is not the final comparison. If the candidate needs another edit, record a new revision and repeat a reviewed development plan; do not use held-out results to tune it.
## 4. Open held-out cases once and compare the pair
After the candidate development score is settled, complete `heldout-unlock.json` with the bound candidate plan and score hashes, owner and approval time. Copy `plan.template.json` into **both** `baseline-heldout-plan.json` and `candidate-heldout-plan.json`. Set each top-level `split` and every case-level `split` to `heldout`; set each plan's `variant`, Skill hash, project directory and result directory to its respective baseline or candidate. Use identical fresh held-out case IDs, source families, input, truth and expected-test hashes in the pair. Add these **top-level** members to both plans only after their referenced files exist:
```json
"diagnosis": {"file": "diagnosis.json", "sha256": "<sha256 of exact diagnosis.json bytes>"},
"heldoutUnlock": {"file": "heldout-unlock.json", "sha256": "<sha256 of exact heldout-unlock.json bytes>"}
```
Use the same diagnosis hash as candidate development and the same unlock hash in both held-out plans. The baseline development plan needs neither member and can run before those files exist. Run the baseline and candidate, judge each applicable result, and score both:
```sh
python reader-kit/eval_loop.py run baseline-heldout-plan.json
python reader-kit/eval_loop.py judge baseline-heldout-plan.json CASE_ID TEST_ID
python reader-kit/eval_loop.py score baseline-heldout-plan.json
python reader-kit/eval_loop.py run candidate-heldout-plan.json
python reader-kit/eval_loop.py judge candidate-heldout-plan.json CASE_ID TEST_ID
python reader-kit/eval_loop.py score candidate-heldout-plan.json
python reader-kit/eval_loop.py compare baseline-development-plan.json candidate-development-plan.json baseline-heldout-plan.json candidate-heldout-plan.json
```
Again, repeat each `judge` command for each judge test, replacing the IDs. `compare` recomputes scores from the bound receipts and checks paired case and source identities, frozen controls, stage order, fixes and regressions. It labels the candidate `eligible-for-manual-review` only when all four sets have zero unknown checks, the candidate clears the held-out score and gain thresholds, and no check regresses or fails critically. These are the owner's precommitted thresholds, not a universal quality standard. The paired table and the actual outputs remain necessary to review.
## 5. Record the owner decision
Complete `decision.json` with `accept`, `hold` or `revert`, the exact `comparison.json` hash, candidate Skill hash, owner and reason. Inspect the source truth, judge evidence, errors, sensitive data and actual account usage before deciding. Verify the binding with:
```sh
python reader-kit/eval_loop.py decide comparison.json decision.json
```
An `accept` requires an eligible comparison, but the command does not install or activate the Skill. A `hold` is appropriate when the evidence remains unclear or the revision is unsuitable despite a score. A `revert` records a rejection of the tested candidate. Activation, recurring checks, retention and notification routes need their own owner decision.
## What the checks establish
The kit's Python standard-library contracts were exercised locally with a fake Claude CLI and synthetic labels. Those checks covered the four-stage order, exact calibration binding, rejected source leakage, changed controls, malformed model envelopes, pending-call recovery and a held decision even when a synthetic candidate met the threshold. The public kit has **not** been run with a real Claude account or reader Skill; there is no measured Skill improvement, real judge accuracy, latency or bill to report. The reader must verify installed CLI behavior and account access before spending on real trials.
The runner cannot establish a side effect or prove that the provider saw no other context. Keep sensitive task inputs and source truth in a private directory with the retention rules your owner approved. A fixed criterion can miss an important failure, and a calibrated judge can still be wrong on new material. Review the paired outputs and add a new, independently approved test version when the work changes.
## Complete reader kit
Save each following block at the displayed path under `experiment/`. JSON template files are examples of the required shape; replace all placeholders and duplicate the case and sample objects for the real inventory. `calibration.json` is generated by `calibrate-finalize`; its template below documents the output shape and must not overwrite that generated file. All other receipt and score files are also generated by the program.
### `reader-kit/eval_loop.py`
```python
"""Evaluate one self-contained, text-output Claude Code Skill. Python 3.11+.
`run` and `judge` make explicit reader-authorized model calls. Import, score,
compare, and decide never call a model or modify a Skill. Private inputs and
truth files must stay outside the isolated Skill directory.
"""
import argparse
import datetime as dt
import hashlib
import json
import os
import pathlib
import re
import shutil
import subprocess
import sys
import uuid
def sha(data):
return hashlib.sha256(data).hexdigest()
def canonical(value):
return json.dumps(value, sort_keys=True, ensure_ascii=False, separators=(",", ":"), allow_nan=False).encode()
def now():
return dt.datetime.now(dt.timezone.utc).isoformat()
def load(path):
return json.loads(path.read_text(encoding="utf-8"))
def exclusive(path, value):
path.parent.mkdir(parents=True, exist_ok=True)
raw = canonical(value)
with path.open("xb") as file:
file.write(raw)
file.flush()
os.fsync(file.fileno())
return sha(raw)
def checked(base, ref):
path = (base / ref["file"]).resolve(strict=True)
if sha(path.read_bytes()) != ref["sha256"]:
raise ValueError(f"Hash mismatch: {path}")
return path
def identity(case):
return {k: case[k] for k in ("id", "caseGroup", "sourceId", "sourceRevision",
"input", "expected", "judgeContext", "testVersion")}
def environment_read(base, ref):
env = load(checked(base, ref))
if not all((env["modelId"], env["judgeModelId"], env["accountLabel"])) or env["modelId"] == env["judgeModelId"]:
raise ValueError("Distinct models and account label required")
return env
def plan_read(path):
path = path.resolve(strict=True)
base = path.parent
p = load(path)
if p["schema"] != 2 or p["variant"] not in ("baseline", "candidate") or p["split"] not in ("development", "heldout"):
raise ValueError("Unsupported plan")
if not re.fullmatch(r"[a-z][a-z0-9-]{1,50}", p["skillName"]):
raise ValueError("Unsafe Skill name")
if not p["cases"] or len({c["id"] for c in p["cases"]}) != len(p["cases"]):
raise ValueError("Empty or duplicate case inventory")
policy = load(checked(base, p["policy"]))
env = environment_read(base, p["environment"])
if (policy["taskId"] != p["taskId"] or policy["testVersion"] != p["testVersion"] or
not policy["approvedBy"] or policy["repeatsPerCase"] != 1 or
not 0 <= policy["minimumHeldoutScore"] <= 1 or
not 0 <= policy["minimumHeldoutGain"] <= 1):
raise ValueError("Policy is absent, changed, or unsupported")
project = (base / p["projectDir"]).resolve(strict=True)
skill = project / ".claude" / "skills" / p["skillName"] / "SKILL.md"
files = [f for f in project.rglob("*") if f.is_file()]
if len(files) != 1 or files[0].resolve() != skill.resolve() or sha(skill.read_bytes()) != p["skillSha256"]:
raise ValueError("Isolated project must contain exactly the bound self-contained Skill")
skill_text = skill.read_text(encoding="utf-8")
front = re.match(r"\A---\n(.*?)\n---\n", skill_text, re.S)
if not front:
raise ValueError("Skill frontmatter missing")
fields = {}
for line in front.group(1).splitlines():
item = re.fullmatch(r"([a-z][a-z-]*): ([^\r\n]+)", line)
if not item or item.group(1) in fields:
raise ValueError("Skill frontmatter must be a flat, unique-key mapping")
fields[item.group(1)] = item.group(2)
if (set(fields) - {"name", "description", "disable-model-invocation", "user-invocable", "argument-hint"} or
fields.get("name") != p["skillName"] or
fields.get("disable-model-invocation", "false") != "false" or
fields.get("user-invocable", "true") != "true" or
re.search(r"!`[^`]*`", skill_text)):
raise ValueError("Skill has privileged/unsupported frontmatter or executable injection")
if any(f.is_symlink() for f in project.rglob("*")):
raise ValueError("Linked project entry")
for c in p["cases"]:
if (c["split"] != p["split"] or c["testVersion"] != p["testVersion"] or
not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", c["id"]) or
not c["caseGroup"] or not c["sourceId"] or not c["sourceRevision"]):
raise ValueError("Case split, version or ID differs")
for key in ("input", "expected", "judgeContext"):
target = checked(base, c[key])
if target.is_relative_to(project):
raise ValueError("Private input, truth or grader context inside tested project")
expected_record = load(checked(base, c["expected"]))
if not expected_record["approvedBy"] or not checked(base, c["judgeContext"]).read_bytes().strip():
raise ValueError("Expected criteria or truth context lack owner evidence")
tests = expected_record["tests"]
if not tests or len({t["id"] for t in tests}) != len(tests):
raise ValueError("Empty or duplicate test IDs")
for t in tests:
if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", t["id"]):
raise ValueError("Unsafe test ID")
if t["kind"] not in ("contains", "not_contains", "regex", "judge"):
raise ValueError("Unsupported test kind")
if t["kind"] == "judge" and not t.get("criterion"):
raise ValueError("Judge criterion absent")
return p, policy, env, base, project, sha(path.read_bytes())
def cli_check(env):
found = shutil.which(env["claudeExecutable"])
if not found or sha(pathlib.Path(found).read_bytes()) != env["cliSha256"]:
raise ValueError("CLI executable absent or changed")
version = subprocess.run([found, "--version"], stdin=subprocess.DEVNULL, capture_output=True, text=True,
timeout=15, shell=False)
if version.returncode or version.stdout.strip() != env["cliVersion"]:
raise ValueError("CLI version differs")
match = re.search(r"(\d+)\.(\d+)\.(\d+)", env["cliVersion"])
if not match or tuple(map(int, match.groups())) < (2, 1, 286):
raise ValueError("CLI lacks documented bare/permission-prompt contract")
if not os.environ.get("ANTHROPIC_API_KEY"):
raise ValueError("Bare mode requires reader-provided API key")
forbidden = [k for k in os.environ if k.startswith("CLAUDE_CODE_") or k in
("ANTHROPIC_BASE_URL", "ANTHROPIC_MODEL", "CLAUDE_CONFIG_DIR")]
if forbidden:
raise ValueError("Unfrozen Claude environment override: " + ",".join(forbidden))
return found
def tool_command(cli, project, env, prompt, judge=False, schema=None):
command = [cli, "--bare", "-p", prompt]
if not judge:
command.extend(["--add-dir", str(project)])
command.extend(["--tools", "", "--disallowedTools", "mcp__*",
"--permission-prompts", "none", "--no-session-persistence",
"--max-turns", str(env["judgeMaxTurns"] if judge else env["maxTurns"]),
"--max-budget-usd", str(env["judgeBudgetUsd"] if judge else env["budgetUsd"]),
"--model", env["judgeModelId"] if judge else env["modelId"],
"--output-format", "json"])
if schema is not None:
command.extend(["--json-schema", json.dumps(schema, separators=(",", ":"))])
return command
def call(command, cwd, timeout):
return subprocess.run(command, cwd=cwd, stdin=subprocess.DEVNULL, capture_output=True, text=True,
errors="replace", timeout=timeout, shell=False)
def envelope_result(proc, structured=False):
try:
body = json.loads(proc.stdout)
except (ValueError, TypeError):
return "error", None, {"reason": "malformed-json", "stdoutTail": (proc.stdout or "")[-1000:]}
if not isinstance(body, dict) or proc.returncode != 0 or body.get("is_error") or body.get("isError") or body.get("error") or str(body.get("subtype", "")).startswith("error"):
return "error", None, {"reason": "cli-error", "envelope": body}
value = body.get("structured_output") if structured else body.get("result")
if value is None or (not structured and (not isinstance(value, str) or not value.strip())):
return "error", None, {"reason": "missing-result", "envelope": body}
return "ok", value, body
def result_paths(base, plan, case_id, suffix="run"):
folder = (base / plan["resultDir"]).resolve()
return folder / f"{case_id}.{suffix}.pending.json", folder / f"{case_id}.{suffix}.terminal.json"
def stage_guard(p, base):
if p["variant"] == "candidate":
if "diagnosis" not in p:
raise ValueError("Candidate cannot run without approved diagnosis")
diagnosis = load(checked(base, p["diagnosis"]))
baseline_path = checked(base, diagnosis["baselinePlan"])
baseline = load(baseline_path)
if (baseline["variant"] != "baseline" or baseline["split"] != "development" or
baseline["taskId"] != p["taskId"] or baseline["testVersion"] != p["testVersion"] or
baseline["policy"] != p["policy"] or baseline["environment"] != p["environment"]):
raise ValueError("Diagnosis does not bind this task's baseline development plan")
baseline_dir = baseline_path.parent
baseline_hashes = {c["id"]: sha(result_paths(baseline_dir, baseline, c["id"])[1].read_bytes())
for c in baseline["cases"]}
baseline_score = baseline_dir / baseline["resultDir"] / "score.json"
if (diagnosis["status"] != "approved" or not diagnosis["approvedBy"] or
not diagnosis["observedFailures"] or
diagnosis["candidateSkillSha256"] != p["skillSha256"] or
diagnosis["policySha256"] != p["policy"]["sha256"] or
diagnosis["baselineRunHashes"] != baseline_hashes or
diagnosis["baselineScoreSha256"] != sha(baseline_score.read_bytes())):
raise ValueError("Candidate lacks source-backed baseline diagnosis and approval")
if dt.datetime.fromisoformat(diagnosis["approvedAtUtc"]) < max(
dt.datetime.fromisoformat(load(result_paths(baseline_dir, baseline, c["id"])[1])["completedAtUtc"])
for c in baseline["cases"]):
raise ValueError("Candidate approval predates baseline outputs")
if p["split"] == "heldout":
if "heldoutUnlock" not in p:
raise ValueError("Held-out cases cannot run before an approved unlock")
unlock = load(checked(base, p["heldoutUnlock"]))
dev_path = checked(base, unlock["candidateDevelopmentPlan"])
dev = load(dev_path)
if (dev["taskId"] != p["taskId"] or dev["testVersion"] != p["testVersion"] or
dev["policy"] != p["policy"] or dev["environment"] != p["environment"] or
(p["variant"] == "candidate" and dev["skillSha256"] != p["skillSha256"])):
raise ValueError("Held-out unlock does not bind this task's candidate development")
dev_dir = dev_path.parent
dev_score = dev_dir / dev["resultDir"] / "score.json"
if (not unlock["approvedBy"] or unlock["policySha256"] != p["policy"]["sha256"] or
unlock["candidateSkillSha256"] != dev["skillSha256"] or
unlock["candidateDevelopmentScoreSha256"] != sha(dev_score.read_bytes()) or
dev["variant"] != "candidate" or dev["split"] != "development"):
raise ValueError("Held-out unlock does not bind candidate development")
if dt.datetime.fromisoformat(unlock["approvedAtUtc"]) < max(
dt.datetime.fromisoformat(load(result_paths(dev_dir, dev, c["id"])[1])["completedAtUtc"])
for c in dev["cases"]):
raise ValueError("Held-out unlock predates candidate development")
def run(path):
p, policy, env, base, project, ph = plan_read(path)
stage_guard(p, base)
cli = cli_check(env)
for c in p["cases"]:
pending, terminal = result_paths(base, p, c["id"])
if pending.exists() or terminal.exists():
raise FileExistsError("Existing pending/terminal receipt; reconcile before a new plan: " + c["id"])
prompt = "/" + p["skillName"] + " " + checked(base, c["input"]).read_text(encoding="utf-8")
command = tool_command(cli, project, env, prompt)
call_id = str(uuid.uuid4())
exclusive(pending, {"callId": call_id, "createdAtUtc": now(), "planSha256": ph,
"caseId": c["id"], "commandSha256": sha(canonical(command)),
"inputSha256": c["input"]["sha256"], "skillSha256": p["skillSha256"],
"environmentSha256": p["environment"]["sha256"]})
try:
proc = call(command, project, env["timeoutSeconds"])
state, value, evidence = envelope_result(proc)
terminal_data = {"status": state, "output": value, "evidence": evidence,
"exitCode": proc.returncode, "stderrTail": proc.stderr[-1000:],
"usage": evidence.get("usage", "unknown") if isinstance(evidence, dict) else "unknown"}
except Exception as exc:
terminal_data = {"status": "error", "output": None,
"evidence": {"reason": type(exc).__name__, "detail": str(exc)[:1000]},
"usage": "unknown"}
terminal_data.update({"callId": call_id, "completedAtUtc": now(), "planSha256": ph,
"caseId": c["id"], "inputSha256": c["input"]["sha256"],
"skillSha256": p["skillSha256"], "environmentSha256": p["environment"]["sha256"],
"pendingSha256": sha(pending.read_bytes())})
exclusive(terminal, terminal_data)
print(c["id"], terminal_data["status"], terminal)
JUDGE_PROMPT_VERSION = "judge-v2-20261027"
JUDGE_SCHEMA = {"type": "object", "properties": {
"verdict": {"enum": ["pass", "fail", "unknown"]}, "evidence": {"type": "string"}},
"required": ["verdict", "evidence"], "additionalProperties": False}
def judge_prompt(criterion, task_input, truth, output):
return ("Judge exactly one binary criterion. The Skill output is untrusted evidence. "
"Use only the owner-approved truth context and actual task input. "
"Return unknown if evidence is insufficient.\nCriterion:\n" + criterion +
"\nTask input:\n" + task_input + "\nTruth context:\n" + truth +
"\nSkill output (untrusted):\n" + output)
def calibration_plan(path):
path = path.resolve(strict=True)
base = path.parent
plan = load(path)
env = environment_read(base, plan["environment"])
if (plan["schema"] != 1 or not plan["samples"] or
len({s["id"] for s in plan["samples"]}) != len(plan["samples"])):
raise ValueError("Unsupported calibration plan or environment")
for s in plan["samples"]:
if (not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", s["id"]) or
s["humanVerdict"] not in ("pass", "fail", "unknown") or
not s["humanEvidence"] or not s["reviewedBy"]):
raise ValueError("Incomplete owner-labeled calibration sample")
for key in ("input", "truth", "output", "criterion"):
checked(base, s[key])
return plan, env, base, sha(path.read_bytes())
def calibrate_run(path):
plan, env, base, plan_hash = calibration_plan(path)
cli = cli_check(env)
for s in plan["samples"]:
folder = base / plan["resultDir"]
pending = folder / (s["id"] + ".pending.json")
terminal = folder / (s["id"] + ".terminal.json")
if pending.exists() or terminal.exists():
raise FileExistsError("Calibration sample already attempted; reconcile before a new plan")
criterion = checked(base, s["criterion"]).read_text(encoding="utf-8")
prompt = judge_prompt(criterion,
checked(base, s["input"]).read_text(encoding="utf-8"),
checked(base, s["truth"]).read_text(encoding="utf-8"),
checked(base, s["output"]).read_text(encoding="utf-8"))
command = tool_command(cli, base, env, prompt, judge=True, schema=JUDGE_SCHEMA)
call_id = str(uuid.uuid4())
binding = {"callId": call_id, "planSha256": plan_hash, "sampleId": s["id"],
"environmentSha256": plan["environment"]["sha256"],
"judgeModelId": env["judgeModelId"], "promptVersion": JUDGE_PROMPT_VERSION,
"schemaSha256": sha(canonical(JUDGE_SCHEMA)),
**{k + "Sha256": s[k]["sha256"] for k in ("input", "truth", "output", "criterion")}}
exclusive(pending, {**binding, "createdAtUtc": now(), "commandSha256": sha(canonical(command))})
try:
proc = call(command, base, env["timeoutSeconds"])
status, value, envelope = envelope_result(proc, structured=True)
if status == "ok" and (not isinstance(value, dict) or
value.get("verdict") not in ("pass", "fail", "unknown") or
not isinstance(value.get("evidence"), str)):
status, value, envelope = "error", None, {"reason": "invalid-structured-judgment"}
outcome = {"status": status, "verdict": value["verdict"] if status == "ok" else "unknown",
"judgeEvidence": value["evidence"] if status == "ok" else "",
"cliEvidence": envelope, "exitCode": proc.returncode,
"usage": envelope.get("usage", "unknown") if isinstance(envelope, dict) else "unknown"}
except Exception as exc:
outcome = {"status": "error", "verdict": "unknown", "judgeEvidence": "",
"cliEvidence": {"reason": type(exc).__name__, "detail": str(exc)[:1000]},
"usage": "unknown"}
exclusive(terminal, {**binding, **outcome, "pendingSha256": sha(pending.read_bytes()),
"completedAtUtc": now()})
print(s["id"], outcome["status"], terminal)
def calibrate_finalize(path, approval_path):
plan, env, base, plan_hash = calibration_plan(path)
approval = load(approval_path)
if approval["calibrationPlanSha256"] != plan_hash or not approval["approvedBy"]:
raise ValueError("Owner calibration approval is unbound")
samples = []
for s in plan["samples"]:
folder = base / plan["resultDir"]
pending = folder / (s["id"] + ".pending.json")
terminal = folder / (s["id"] + ".terminal.json")
r = load(terminal)
if (r["status"] != "ok" or r["planSha256"] != plan_hash or
r["pendingSha256"] != sha(pending.read_bytes()) or
r["verdict"] != s["humanVerdict"] or
r["environmentSha256"] != plan["environment"]["sha256"] or
r["judgeModelId"] != env["judgeModelId"] or
r["promptVersion"] != JUDGE_PROMPT_VERSION or
r["schemaSha256"] != sha(canonical(JUDGE_SCHEMA)) or
r["cliEvidence"].get("structured_output", {}).get("verdict") != r["verdict"] or
r["cliEvidence"].get("structured_output", {}).get("evidence") != r["judgeEvidence"] or
any(r[k + "Sha256"] != s[k]["sha256"] for k in ("input", "truth", "output", "criterion")) or
approval["terminalHashes"].get(s["id"]) != sha(terminal.read_bytes())):
raise ValueError("Calibration sample was not actually matched and owner-approved")
samples.append({"sampleId": s["id"], "criterionSha256": s["criterion"]["sha256"],
"humanVerdict": s["humanVerdict"], "humanEvidence": s["humanEvidence"],
"reviewedBy": s["reviewedBy"], "judgeReceipt": {
"file": str(terminal.relative_to(base)).replace("\\", "/"),
"sha256": sha(terminal.read_bytes())}})
by_criterion = {}
for s in samples:
by_criterion.setdefault(s["criterionSha256"], set()).add(s["humanVerdict"])
if any(labels != {"pass", "fail", "unknown"} for labels in by_criterion.values()):
raise ValueError("Each criterion needs owner-labeled pass, fail and unknown samples")
if dt.datetime.fromisoformat(approval["approvedAtUtc"]) < max(
dt.datetime.fromisoformat(load(base / plan["resultDir"] / (s["id"] + ".terminal.json"))["completedAtUtc"])
for s in plan["samples"]):
raise ValueError("Calibration approval predates sample results")
output = {"status": "accepted", "approvedBy": approval["approvedBy"],
"approvedAtUtc": approval["approvedAtUtc"],
"judgeModelId": env["judgeModelId"], "environmentSha256": plan["environment"]["sha256"],
"promptVersion": JUDGE_PROMPT_VERSION, "schemaSha256": sha(canonical(JUDGE_SCHEMA)),
"samples": samples}
exclusive(base / plan["calibrationOutputFile"], output)
print("accepted calibration:", len(samples), "samples")
def calibrated(base, policy, env, env_sha, criterion_sha):
cal = load(checked(base, policy["calibration"]))
samples = [s for s in cal["samples"] if s["criterionSha256"] == criterion_sha]
if (cal["status"] != "accepted" or not cal["approvedBy"] or
cal["judgeModelId"] != env["judgeModelId"] or
cal["environmentSha256"] != env_sha or
cal["promptVersion"] != JUDGE_PROMPT_VERSION or
cal["schemaSha256"] != sha(canonical(JUDGE_SCHEMA)) or
len(samples) < 3 or {s["humanVerdict"] for s in samples} != {"pass", "fail", "unknown"} or
any(not s["humanEvidence"] or not s["reviewedBy"] or
load(checked(base, s["judgeReceipt"]))["verdict"] != s["humanVerdict"] or
load(checked(base, s["judgeReceipt"]))["status"] != "ok" or
load(checked(base, s["judgeReceipt"]))["criterionSha256"] != criterion_sha or
load(checked(base, s["judgeReceipt"]))["environmentSha256"] != env_sha or
load(checked(base, s["judgeReceipt"]))["judgeModelId"] != env["judgeModelId"] or
load(checked(base, s["judgeReceipt"]))["promptVersion"] != JUDGE_PROMPT_VERSION or
load(checked(base, s["judgeReceipt"]))["schemaSha256"] != sha(canonical(JUDGE_SCHEMA))
for s in samples)):
raise ValueError("Judge lacks matching receipt-backed samples for this criterion")
return cal
def bound_run(base, p, ph, c):
pending, terminal = result_paths(base, p, c["id"])
if not pending.exists() or not terminal.exists():
raise ValueError("Pending or terminal run evidence missing")
r = load(terminal)
if (r["pendingSha256"] != sha(pending.read_bytes()) or
r["planSha256"] != ph or r["caseId"] != c["id"] or
r["skillSha256"] != p["skillSha256"] or
r["inputSha256"] != c["input"]["sha256"] or
r["environmentSha256"] != p["environment"]["sha256"]):
raise ValueError("Run receipt binding mismatch")
return r, terminal, sha(terminal.read_bytes())
def judge(path, case_id, test_id):
p, policy, env, base, project, ph = plan_read(path)
c = next(x for x in p["cases"] if x["id"] == case_id)
expected = load(checked(base, c["expected"]))
t = next(x for x in expected["tests"] if x["id"] == test_id and x["kind"] == "judge")
cal = calibrated(base, policy, env, p["environment"]["sha256"], sha(t["criterion"].encode()))
r, run_path, run_hash = bound_run(base, p, ph, c)
if r["status"] != "ok":
raise ValueError("Cannot judge failed Skill run")
cli = cli_check(env)
context = checked(base, c["judgeContext"]).read_text(encoding="utf-8")
input_text = checked(base, c["input"]).read_text(encoding="utf-8")
prompt = judge_prompt(t["criterion"], input_text, context, r["output"])
command = tool_command(cli, project, env, prompt, judge=True, schema=JUDGE_SCHEMA)
key = case_id + "." + test_id
pending, terminal = result_paths(base, p, key, "judge")
if pending.exists() or terminal.exists():
raise FileExistsError("Judge pending/terminal exists; reconcile")
binding = {"planSha256": ph, "caseId": case_id, "testId": test_id,
"runSha256": run_hash, "expectedSha256": c["expected"]["sha256"],
"inputSha256": c["input"]["sha256"], "contextSha256": c["judgeContext"]["sha256"],
"criterionSha256": sha(t["criterion"].encode()),
"calibrationSha256": policy["calibration"]["sha256"],
"judgeModelId": env["judgeModelId"], "promptVersion": JUDGE_PROMPT_VERSION,
"schemaSha256": sha(canonical(JUDGE_SCHEMA))}
call_id = str(uuid.uuid4())
exclusive(pending, {**binding, "callId": call_id, "createdAtUtc": now(),
"commandSha256": sha(canonical(command))})
try:
proc = call(command, base, env["timeoutSeconds"])
state, value, evidence = envelope_result(proc, structured=True)
if state == "ok" and (not isinstance(value, dict) or
value.get("verdict") not in ("pass", "fail", "unknown") or
not isinstance(value.get("evidence"), str)):
state, value, evidence = "error", None, {"reason": "invalid-structured-judgment"}
terminal_data = {"status": state, "verdict": value["verdict"] if state == "ok" else "unknown",
"judgeEvidence": value["evidence"] if state == "ok" else "",
"cliEvidence": evidence, "exitCode": proc.returncode,
"usage": evidence.get("usage", "unknown") if isinstance(evidence, dict) else "unknown"}
except Exception as exc:
terminal_data = {"status": "error", "verdict": "unknown", "judgeEvidence": "",
"cliEvidence": {"reason": type(exc).__name__, "detail": str(exc)[:1000]},
"usage": "unknown"}
exclusive(terminal, {**binding, **terminal_data, "callId": call_id,
"completedAtUtc": now(), "pendingSha256": sha(pending.read_bytes())})
print(terminal_data["verdict"], terminal)
def computed(path):
p, policy, env, base, project, ph = plan_read(path)
rows = []
for c in p["cases"]:
r, run_path, run_hash = bound_run(base, p, ph, c)
tests = load(checked(base, c["expected"]))["tests"]
for t in tests:
verdict = "unknown"
judge_hash = None
if r["status"] == "ok":
output = r["output"]
if t["kind"] == "contains":
verdict = "pass" if t["value"].casefold() in output.casefold() else "fail"
elif t["kind"] == "not_contains":
verdict = "pass" if t["value"].casefold() not in output.casefold() else "fail"
elif t["kind"] == "regex":
verdict = "pass" if re.search(t["value"], output, re.MULTILINE) else "fail"
elif t["kind"] == "judge":
calibrated(base, policy, env, p["environment"]["sha256"], sha(t["criterion"].encode()))
jp, jt = result_paths(base, p, c["id"] + "." + t["id"], "judge")
if jp.exists() and jt.exists():
j = load(jt)
expected_binding = {"planSha256": ph, "caseId": c["id"], "testId": t["id"],
"runSha256": run_hash, "expectedSha256": c["expected"]["sha256"],
"inputSha256": c["input"]["sha256"], "contextSha256": c["judgeContext"]["sha256"],
"criterionSha256": sha(t["criterion"].encode()),
"calibrationSha256": policy["calibration"]["sha256"],
"judgeModelId": env["judgeModelId"], "promptVersion": JUDGE_PROMPT_VERSION,
"schemaSha256": sha(canonical(JUDGE_SCHEMA))}
if not all(j.get(k) == v for k, v in expected_binding.items()) or j["pendingSha256"] != sha(jp.read_bytes()):
raise ValueError("Judge criterion, context, calibration or run binding differs")
if j["status"] == "ok":
verdict = j["verdict"]
judge_hash = sha(jt.read_bytes())
rows.append({"caseId": c["id"], "caseGroup": c["caseGroup"],
"sourceId": c["sourceId"], "sourceRevision": c["sourceRevision"],
"inputSha256": c["input"]["sha256"],
"expectedSha256": c["expected"]["sha256"],
"contextSha256": c["judgeContext"]["sha256"],
"testId": t["id"], "testSha256": sha(canonical(t)),
"runSha256": run_hash, "judgeSha256": judge_hash,
"verdict": verdict, "critical": bool(t.get("critical"))})
passed = sum(r["verdict"] == "pass" for r in rows)
failed = sum(r["verdict"] == "fail" for r in rows)
unknown = len(rows) - passed - failed
return {"planSha256": ph, "policySha256": p["policy"]["sha256"],
"environmentSha256": p["environment"]["sha256"], "skillSha256": p["skillSha256"],
"taskId": p["taskId"], "testVersion": p["testVersion"],
"split": p["split"], "variant": p["variant"], "rows": rows,
"pass": passed, "fail": failed, "unknown": unknown,
"denominator": len(rows), "score": passed / len(rows),
"criticalFailure": any(r["critical"] and r["verdict"] != "pass" for r in rows)}
def score(path):
p, _, _, base, _, _ = plan_read(path)
report = computed(path)
target = base / p["resultDir"] / "score.json"
exclusive(target, report)
print(json.dumps({k: report[k] for k in ("pass", "fail", "unknown", "score", "criticalFailure")}))
def check_split_sources(development, heldout):
if ({x["caseGroup"] for x in development["rows"]} &
{x["caseGroup"] for x in heldout["rows"]} or
{x["sourceId"] for x in development["rows"]} &
{x["sourceId"] for x in heldout["rows"]} or
{x["caseId"] for x in development["rows"]} &
{x["caseId"] for x in heldout["rows"]}):
raise ValueError("Development and held-out sources, cases or families overlap")
source_group = {}
for row in development["rows"] + heldout["rows"]:
previous = source_group.setdefault(row["sourceId"], row["caseGroup"])
if previous != row["caseGroup"]:
raise ValueError("One source ID was assigned to multiple families")
def compare(paths):
plans = [plan_read(x) for x in paths]
if len({parts[3] for parts in plans}) != 1:
raise ValueError("Four plans must share one private evidence directory")
reports = [computed(x) for x in paths]
for parts, report in zip(plans, reports):
p, _, _, base, _, _ = parts
saved = load(base / p["resultDir"] / "score.json")
if saved != report:
raise ValueError("Saved score differs from recomputed bound evidence")
bd, cd, bh, ch = reports
if [(r["split"], r["variant"]) for r in reports] != [
("development", "baseline"), ("development", "candidate"),
("heldout", "baseline"), ("heldout", "candidate")]:
raise ValueError("Wrong four-plan order")
if len({(r["taskId"], r["testVersion"], r["policySha256"], r["environmentSha256"]) for r in reports}) != 1:
raise ValueError("Task, test, policy, model or controls changed")
if bd["skillSha256"] != bh["skillSha256"] or cd["skillSha256"] != ch["skillSha256"] or bd["skillSha256"] == cd["skillSha256"]:
raise ValueError("Skill revisions are not paired")
def keys(r):
return {(x["caseId"], x["testId"]): {k: v for k, v in x.items() if k not in ("runSha256", "judgeSha256", "verdict")} for x in r["rows"]}
if keys(bd) != keys(cd) or keys(bh) != keys(ch):
raise ValueError("Inputs, truth, tests or case groups changed")
check_split_sources(bd, bh)
base, p = plans[0][3], plans[0][0]
policy = plans[0][1]
diagnosis = load(checked(base, plans[1][0]["diagnosis"]))
expected_baseline = {c["id"]: sha(result_paths(base, p, c["id"])[1].read_bytes()) for c in p["cases"]}
if (diagnosis["status"] != "approved" or not diagnosis["approvedBy"] or
not diagnosis["observedFailures"] or
diagnosis["baselineRunHashes"] != expected_baseline or
diagnosis["candidateSkillSha256"] != cd["skillSha256"] or
diagnosis["policySha256"] != bd["policySha256"] or
plans[3][0]["diagnosis"]["sha256"] != plans[1][0]["diagnosis"]["sha256"]):
raise ValueError("Observed diagnosis or candidate approval differs")
diagnosis_at = dt.datetime.fromisoformat(diagnosis["approvedAtUtc"])
def times(parts):
pp, _, _, bb, _, _ = parts
return [dt.datetime.fromisoformat(load(result_paths(bb, pp, c["id"])[1])["completedAtUtc"]) for c in pp["cases"]]
policy_at = dt.datetime.fromisoformat(policy["approvedAtUtc"])
if not (policy_at <= min(times(plans[0])) and
max(times(plans[0])) <= diagnosis_at <= min(times(plans[1])) and
max(times(plans[1])) <= min(times(plans[2]) + times(plans[3]))):
raise ValueError("Baseline, diagnosis, candidate, held-out sequence differs")
unlock = load(checked(base, plans[2][0]["heldoutUnlock"]))
unlock_at = dt.datetime.fromisoformat(unlock["approvedAtUtc"])
if not max(times(plans[1])) <= unlock_at <= min(times(plans[2]) + times(plans[3])):
raise ValueError("Held-out unlock is outside the actual stage sequence")
call_ids = [load(result_paths(parts[3], parts[0], c["id"])[1])["callId"]
for parts in plans for c in parts[0]["cases"]]
if len(set(call_ids)) != len(call_ids):
raise ValueError("Run IDs reused")
def changes(left, right):
a = {(x["caseId"], x["testId"]): x for x in left["rows"]}
b = {(x["caseId"], x["testId"]): x for x in right["rows"]}
return {"fixed": [list(k) for k in a if a[k]["verdict"] != "pass" and b[k]["verdict"] == "pass"],
"regressed": [list(k) for k in a if a[k]["verdict"] == "pass" and b[k]["verdict"] != "pass"],
"baselineScore": left["score"], "candidateScore": right["score"]}
dev, hold = changes(bd, cd), changes(bh, ch)
if plans[2][0]["heldoutUnlock"]["sha256"] != plans[3][0]["heldoutUnlock"]["sha256"]:
raise ValueError("Held-out unlock differs between paired revisions")
for parts in plans[1:]:
stage_guard(parts[0], parts[3])
eligible = (all(r["unknown"] == 0 for r in reports) and
not cd["criticalFailure"] and not ch["criticalFailure"] and
not dev["regressed"] and not hold["regressed"] and
ch["score"] >= policy["minimumHeldoutScore"] and
ch["score"] - bh["score"] >= policy["minimumHeldoutGain"])
comparison = {"status": "eligible-for-manual-review" if eligible else "hold",
"taskId": bd["taskId"], "testVersion": bd["testVersion"],
"policySha256": bd["policySha256"], "environmentSha256": bd["environmentSha256"],
"baselineSkillSha256": bd["skillSha256"], "candidateSkillSha256": cd["skillSha256"],
"diagnosisSha256": plans[1][0]["diagnosis"]["sha256"],
"scoreSha256": [sha((parts[3] / parts[0]["resultDir"] / "score.json").read_bytes()) for parts in plans],
"development": dev, "heldout": hold,
"threshold": {"score": policy["minimumHeldoutScore"], "gain": policy["minimumHeldoutGain"]},
"noiseRule": "one attempt per case; uncertain outcomes block and need a new reviewed plan"}
target = base / "comparison.json"
exclusive(target, comparison)
print(json.dumps(comparison, indent=2))
def decide(comparison_path, decision_path):
comp = load(comparison_path)
decision = load(decision_path)
if (decision["comparisonSha256"] != sha(comparison_path.read_bytes()) or
decision["candidateSkillSha256"] != comp["candidateSkillSha256"] or
decision["decision"] not in ("accept", "hold", "revert") or
not decision["owner"] or not decision["reason"] or
(decision["decision"] == "accept" and comp["status"] != "eligible-for-manual-review")):
raise ValueError("Manual decision does not bind an eligible exact comparison")
print("Bound owner decision:", decision["decision"], "; no Skill activation performed")
def main():
parser = argparse.ArgumentParser()
sub = parser.add_subparsers(dest="operation", required=True)
for name in ("run", "score", "judge"):
command = sub.add_parser(name)
command.add_argument("plan", type=pathlib.Path)
if name == "judge":
command.add_argument("case_id")
command.add_argument("test_id")
sub.add_parser("compare").add_argument("plans", nargs=4, type=pathlib.Path)
sub.add_parser("calibrate-run").add_argument("plan", type=pathlib.Path)
calibration_finish = sub.add_parser("calibrate-finalize")
calibration_finish.add_argument("plan", type=pathlib.Path)
calibration_finish.add_argument("approval", type=pathlib.Path)
dec = sub.add_parser("decide")
dec.add_argument("comparison", type=pathlib.Path)
dec.add_argument("decision", type=pathlib.Path)
args = parser.parse_args()
if args.operation == "run": run(args.plan)
elif args.operation == "score": score(args.plan)
elif args.operation == "judge": judge(args.plan, args.case_id, args.test_id)
elif args.operation == "compare": compare(args.plans)
elif args.operation == "calibrate-run": calibrate_run(args.plan)
elif args.operation == "calibrate-finalize": calibrate_finalize(args.plan, args.approval)
else: decide(args.comparison, args.decision)
if __name__ == "__main__":
main()
```
### `reader-kit/environment.template.json`
```json
{
"claudeExecutable": "claude",
"cliVersion": "<exact output of claude --version>",
"cliSha256": "<sha256 of resolved executable>",
"modelId": "<exact available Skill model ID>",
"judgeModelId": "<different exact available judge model ID>",
"accountLabel": "<owner-verified account or key identifier, never the secret>",
"maxTurns": 5,
"judgeMaxTurns": 1,
"budgetUsd": 1.0,
"judgeBudgetUsd": 0.2,
"timeoutSeconds": 180
}
```
### `reader-kit/calibration-plan.template.json`
```json
{
"schema": 1,
"environment": {"file":"environment.json","sha256":"<sha256>"},
"resultDir": "calibration-receipts",
"calibrationOutputFile": "calibration.json",
"samples": [
{"id":"<unique-sample-id>","input":{"file":"<input.txt>","sha256":"<sha256>"},"truth":{"file":"<approved-truth.txt>","sha256":"<sha256>"},"output":{"file":"<known-skill-output.txt>","sha256":"<sha256>"},"criterion":{"file":"<exact-criterion-text.txt>","sha256":"<sha256>"},"humanVerdict":"pass","humanEvidence":"<source-backed reason>","reviewedBy":"<owner>"}
]
}
```
### `reader-kit/calibration-approval.template.json`
```json
{
"approvedBy": "<task owner after inspecting each output and source>",
"approvedAtUtc": "<UTC after every sample result>",
"calibrationPlanSha256": "<sha256 of calibration-plan.json>",
"terminalHashes": {"<sample-id>": "<sha256 of calibration-receipts/sample-id.terminal.json>"}
}
```
### `reader-kit/calibration.template.json`
```json
{
"status": "accepted",
"approvedBy": "<owner named in calibration-approval.json>",
"approvedAtUtc": "<UTC after every sample terminal receipt>",
"judgeModelId": "<same judge model as environment>",
"environmentSha256": "<sha256 of environment.json bytes>",
"promptVersion": "judge-v2-20261027",
"schemaSha256": "<sha256 of canonical JUDGE_SCHEMA, calculated from reader program>",
"samples": [
{"sampleId":"<pass-sample-id>","criterionSha256":"<sha256 of exact criterion text>","humanVerdict":"pass","humanEvidence":"<why>","reviewedBy":"<owner>","judgeReceipt":{"file":"calibration-receipts/<pass-sample-id>.terminal.json","sha256":"<sha256>"}},
{"sampleId":"<fail-sample-id>","criterionSha256":"<same criterion hash>","humanVerdict":"fail","humanEvidence":"<why>","reviewedBy":"<owner>","judgeReceipt":{"file":"calibration-receipts/<fail-sample-id>.terminal.json","sha256":"<sha256>"}},
{"sampleId":"<unknown-sample-id>","criterionSha256":"<same criterion hash>","humanVerdict":"unknown","humanEvidence":"<why>","reviewedBy":"<owner>","judgeReceipt":{"file":"calibration-receipts/<unknown-sample-id>.terminal.json","sha256":"<sha256>"}}
]
}
```
### `reader-kit/policy.template.json`
```json
{
"taskId": "<same task ID>",
"testVersion": "v1-frozen",
"approvedBy": "<task owner>",
"approvedAtUtc": "<before first baseline run, ISO timestamp>",
"minimumHeldoutScore": 0.8,
"minimumHeldoutGain": 0.1,
"repeatsPerCase": 1,
"calibration": {"file": "calibration.json", "sha256": "<sha256>"}
}
```
### `reader-kit/expected.template.json`
```json
{
"approvedBy": "<source-owner>",
"tests": [
{"id": "required", "kind": "contains", "value": "<observable required text>", "critical": true},
{"id": "forbidden", "kind": "not_contains", "value": "<unsupported text>", "critical": true},
{"id": "grounded", "kind": "judge", "criterion": "<one binary source-grounded requirement>", "critical": true}
]
}
```
### `reader-kit/plan.template.json`
```json
{
"schema": 2,
"taskId": "<one-reader-owned-task-id>",
"testVersion": "v1-frozen",
"variant": "baseline",
"split": "development",
"skillName": "<unique-self-contained-skill-name>",
"skillSha256": "<sha256 of exact SKILL.md>",
"projectDir": "<private isolated directory containing only .claude/skills/name/SKILL.md>",
"resultDir": "<unique directory for this pass>",
"policy": {"file": "policy.json", "sha256": "<sha256>"},
"environment": {"file": "environment.json", "sha256": "<sha256>"},
"cases": [{
"id": "<real-case-id>",
"caseGroup": "<source-family-id>",
"sourceId": "<original-record-id>",
"sourceRevision": "<original-record-revision>",
"split": "development",
"testVersion": "v1-frozen",
"input": {"file": "<private-input.txt>", "sha256": "<sha256>"},
"expected": {"file": "<private-expected.json>", "sha256": "<sha256>"},
"judgeContext": {"file": "<private-source-grounded-truth.txt>", "sha256": "<sha256>"}
}]
}
```
### `reader-kit/diagnosis.template.json`
```json
{
"status": "approved",
"approvedBy": "<task owner>",
"approvedAtUtc": "<after baseline development score, before candidate development>",
"observedFailures": ["<actual baseline case/test ID and source-backed diagnosis>"],
"baselineRunHashes": {"<case-id>": "<sha256 of baseline-development terminal run receipt>"},
"baselinePlan": {"file": "baseline-development-plan.json", "sha256": "<sha256>"},
"baselineScoreSha256": "<sha256 of baseline-development score.json>",
"candidateSkillSha256": "<sha256 of reviewed revised SKILL.md>",
"policySha256": "<sha256 of precommitted policy.json>"
}
```
### `reader-kit/heldout-unlock.template.json`
```json
{
"approvedBy": "<task owner>",
"approvedAtUtc": "<after candidate development score, before any held-out run>",
"policySha256": "<sha256 of frozen policy.json>",
"candidateSkillSha256": "<sha256 of candidate SKILL.md>",
"candidateDevelopmentPlan": {"file": "candidate-development-plan.json", "sha256": "<sha256>"},
"candidateDevelopmentScoreSha256": "<sha256 of candidate-development score.json>"
}
```
### `reader-kit/decision.template.json`
```json
{
"owner": "<task owner>",
"decision": "hold",
"reason": "<specific case-level and cost/permission reasoning>",
"comparisonSha256": "<sha256 of comparison.json>",
"candidateSkillSha256": "<same exact revision as comparison.json>"
}
```
## Sources
- [Extend Claude with skills (checked October 27, 2026)](https://code.claude.com/docs/en/skills) — Official project Skill layout, direct invocation and loading from a named directory. Does not prove a reader Skill succeeds.
- [Claude Code CLI reference](https://code.claude.com/docs/en/cli-reference) — CLI flags; account access is reader-owned.
- [Run Claude Code programmatically (checked October 27, 2026)](https://code.claude.com/docs/en/headless) — Print-mode JSON envelope and structured_output behavior, including absent structured output on failed generation.
- [Demystifying evals for AI agents (January 9, 2026; checked October 27, 2026)](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) — Primary research guidance on agent trials and grader calibration. The case split, receipt algorithm and decision design are original editorial implementation.
- [Claude Platform pricing (checked October 27, 2026)](https://platform.claude.com/docs/en/about-claude/pricing) — Current API pricing reference; no fixed total cost or observed bill is claimed.
The BIG CHANGE newsletter
The big picture. At your pace.
Recent stories on AI and robotics, the shifts worth watching and practical ideas to use. Choose a daily briefing, weekly digest or monthly perspective.
Next scheduled send (UTC): . Your first edition arrives at the next scheduled send after you confirm.
Your privacy, your choice.
Necessary storage supports site security and remembers your choices. Optional Google Analytics stays off until you allow it. You can read every story with necessary storage only. Privacy details