# Schedule an AI skill experiment with human promotion
> Build a scheduled experiment that generates one classification-skill variant, checks it against verified cases, records uncertain calls and leaves promotion to an owner.
By BIGCHANGE.AI Editorial
Published: 2026-10-27T05:34:48.866Z
Updated: 2026-10-27T05:34:48.866Z
Canonical: https://bigchange.ai/blog/scheduled-ai-skill-experiment-github-actions-guide
AI-generated conceptual illustration by BIG CHANGE.
A scheduled AI skill experiment can show whether a changed classification instruction holds up on fresh records. This guide gives the owner of a private, exact-label classification skill a repeatable procedure: generate one instruction candidate from approved development examples, compare it with the current instruction, and retain the evidence for a human decision. You supply the actual task, records, verified labels, model access and spending authority.
## The big change
- **What:** One run generates a new instruction and checks it against the existing instruction on development and fresh held-out cases.
- **How:** A committed pending marker, locally flushed call receipts and a bound result expose interrupted paid work for reconciliation.
- **Why it matters:** The candidate remains a proposal. An owner can inspect individual errors and the bill before changing the skill that people use.
The kit below is for an exact-label classifier whose answer must be one of your approved labels. It does not measure email opens, sales, website conversions or other live business outcomes. Preparation used synthetic cases and an isolated fake response transport. There has been no real provider call or GitHub Actions run for this guide, so treat the first manual run as a live test of your own account and repository.
## 1. Choose the task, access and budget
Use a private repository that may lawfully retain the inputs, source references, expected labels, raw responses and run history. The operator needs Python 3.11, permission to run and write GitHub Actions, an account with access to the selected model, and authority for its API and Actions charges. Review who can read the repository, run logs and artifacts. Use records and labels checked against their original sources; the examples below are schemas to fill, not sample observations.
The repository layout after setup is:
```text
experiment.py
baseline-skill.txt
config.json
cases.jsonl
truth/<case-id>.txt
.github/workflows/autoresearch.yml
results/ # created by the first prepared run
```
Put the current approved instruction in `baseline-skill.txt`. It must tell the model to classify the supplied input using exactly the labels you configure. Do not put the answer key in that instruction. This file shape shows the required owner-supplied parts; replace them with your real approved skill before running anything:
```text
Classify the supplied input for <owner-defined exact-label task>.
Allowed labels: <the same labels as config.json>.
Decision rules and tie breaks: <owner-approved rules from the real task>.
```
Save the following as `config.json`, replacing every bracketed value. `model` names the requested model; `resolvedModel` must match the returned model. Verify both and the current account price before approving them.
```json
{
"model": "<owner-verified-supported-model-id>",
"resolvedModel": "<owner-verified-exact-response-model-id>",
"labels": ["<real-label-a>", "<real-label-b>"],
"repeats": 3,
"maxCalls": 200,
"maxOutputTokens": 256,
"maxInputChars": 6000,
"minDevelopment": 12,
"minHeldout": 12,
"approvedBy": "<task-and-account-owner>",
"approvedAtUtc": "<UTC timestamp>",
"dataPermission": "<private permission receipt>"
}
```
The example demands at least 12 development and 12 held-out cases, with three calls per instruction and case. At that minimum, the planned ceiling is `1 + 2 × 3 × 24 = 145` requests: one generated variant, then both instructions on every case three times. `maxCalls` prevents a run above its configured request count. `maxOutputTokens` restricts output tokens; it does not cap input charges, GitHub charges or the final dollar bill. Choose an approved spending envelope and review account usage after each real run, especially when a request times out.
Write one JSON object per line in `cases.jsonl` using this schema. Fill every field with a real permissioned input and a source-checked answer. Keep case IDs opaque. `group` identifies the source family, `sourceId` its stable original record, and `sourceRevision` the exact version that supplied the answer. Put the same checked label, with a newline allowed, in `truth/<case-id>.txt`:
```text
<one configured label>
```
Compute each SHA-256 from the exact UTF-8 input string and truth-file bytes. The program verifies both hashes and equality with `expected` before it can prepare a run.
```json
{"id":"<opaque-id>","split":"development","group":"<source-family-id>","sourceId":"<stable-original-record-id>","sourceRevision":"<exact-revision>","input":"<approved-input>","inputSha256":"<sha256-of-exact-UTF8-input>","truthFile":"truth/<opaque-id>.txt","truthSha256":"<sha256-of-truth-file>","expected":"<configured-label>","sourcePermissionReceipt":"<private-approval-reference>"}
```
Change `split` to `heldout` for the other cases. Development and held-out groups, source IDs and exact input hashes must be disjoint. The generator receives development inputs, source IDs, verified labels and baseline observations, including wrong answers. It never receives the held-out records or answers as generation context. The evaluation calls themselves send each case input, so use only data approved for this provider. Reserve fresh held-out sources for every subsequent run. Renaming or reordering an earlier case will not make it new: the state remembers its source family, source ID and exact input hash.
## 2. Install the runner and the parked workflow
Save this complete standard-library program as `experiment.py`. It makes no provider request when imported or when `check` runs. `prepare` and `execute` require a GitHub run identity; the workflow commits the pending state before invoking `execute`.
```python
"""Reader-copyable, bounded prompt-skill experiment. No API call on import/check."""
import hashlib
import json
import os
import pathlib
import re
import sys
import urllib.error
import urllib.request
ROOT = pathlib.Path(__file__).resolve().parent
STATE = ROOT / "results" / "state.json"
PENDING = ROOT / "results" / "pending.json"
HEX = set("0123456789abcdef")
GRADER_VERSION = "exact-label-v2"
SCHEMA_VERSION = "strict-skill-and-label-v2"
def canonical(value):
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
def digest(data):
return hashlib.sha256(data).hexdigest()
def read(path):
return pathlib.Path(path).read_bytes()
def read_json(path):
return json.loads(read(path))
def write_once(path, value):
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("x", encoding="utf-8") as handle:
handle.write(canonical(value) + "\n")
handle.flush()
os.fsync(handle.fileno())
def write_replace(path, value):
temporary = path.with_suffix(".tmp")
with temporary.open("x", encoding="utf-8") as handle:
handle.write(canonical(value) + "\n")
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, path)
def execution_identity():
names = ("GITHUB_RUN_ID", "GITHUB_RUN_ATTEMPT", "GITHUB_REPOSITORY",
"GITHUB_REF", "GITHUB_SHA")
identity = {name: os.environ.get(name) for name in names}
if (any(not value for value in identity.values()) or
not identity["GITHUB_RUN_ID"].isdigit() or
not identity["GITHUB_RUN_ATTEMPT"].isdigit()):
raise ValueError("Complete current GitHub execution identity required")
return identity
def load():
config_file = ROOT / "config.json"
config = read_json(config_file)
required = ("model", "resolvedModel", "labels", "repeats", "maxCalls", "maxOutputTokens",
"maxInputChars", "minDevelopment", "minHeldout", "approvedBy",
"approvedAtUtc", "dataPermission")
if any(not config.get(k) for k in required):
raise ValueError("Missing approved config field")
for field in ("model", "resolvedModel", "approvedBy", "approvedAtUtc", "dataPermission"):
if not isinstance(config[field], str) or "<" in config[field]:
raise ValueError(f"Replace config placeholder: {field}")
if not (2 <= len(config["labels"]) <= 12 and
len(set(config["labels"])) == len(config["labels"]) and
all(isinstance(x, str) and x and "<" not in x
for x in config["labels"])):
raise ValueError("Invalid labels")
if not (2 <= config["repeats"] <= 5 and
1 <= config["maxOutputTokens"] <= 2048 and
1 <= config["maxCalls"] <= 1000 and
1 <= config["maxInputChars"] <= 20000 and
1 <= config["minDevelopment"] <= 500 and
1 <= config["minHeldout"] <= 500):
raise ValueError("Invalid case, call, repeat, input or token cap")
cases_file = ROOT / "cases.jsonl"
cases = [json.loads(line) for line in cases_file.read_text("utf-8").splitlines() if line.strip()]
ids, groups, source_ids, input_hashes = set(), {"development": set(), "heldout": set()}, {
"development": set(), "heldout": set()}, {"development": set(), "heldout": set()}
for row in cases:
if (row.get("split") not in groups or not isinstance(row.get("id"), str) or
not re.fullmatch(r"[A-Za-z0-9_-]{1,80}", row["id"]) or
row["id"] in ids or not isinstance(row.get("group"), str) or
not row["group"] or not isinstance(row.get("input"), str) or
not row["input"] or len(row["input"]) > config["maxInputChars"] or
row.get("expected") not in config["labels"] or
not row.get("sourcePermissionReceipt") or
"<" in row["sourcePermissionReceipt"] or
"<" in row["expected"] or
not isinstance(row.get("sourceId"), str) or
not row["sourceId"] or "<" in row["sourceId"] or
not isinstance(row.get("sourceRevision"), str) or
not row["sourceRevision"] or "<" in row["sourceRevision"] or
not isinstance(row.get("inputSha256"), str) or
row["inputSha256"] != digest(row["input"].encode()) or
not isinstance(row.get("truthSha256"), str) or
len(row["truthSha256"]) != 64 or
any(ch not in HEX for ch in row["truthSha256"]) or
row.get("truthFile") != f'truth/{row["id"]}.txt'):
raise ValueError("Invalid case identity, source/truth provenance, input or permission")
truth_path = ROOT / row["truthFile"]
if (truth_path.is_symlink() or not truth_path.is_file() or
digest(read(truth_path)) != row["truthSha256"] or
read(truth_path).decode("utf-8").strip() != row["expected"]):
raise ValueError("Verified truth file differs from approved label/hash")
ids.add(row["id"])
groups[row["split"]].add(row["group"])
source_ids[row["split"]].add(row["sourceId"])
input_hashes[row["split"]].add(row["inputSha256"])
if (groups["development"] & groups["heldout"] or
source_ids["development"] & source_ids["heldout"] or
input_hashes["development"] & input_hashes["heldout"]):
raise ValueError("Source group, identity or exact input appears in both splits")
dev = [x for x in cases if x["split"] == "development"]
hold = [x for x in cases if x["split"] == "heldout"]
if len(dev) < config["minDevelopment"] or len(hold) < config["minHeldout"]:
raise ValueError("Insufficient real approved cases")
calls = 1 + 2 * config["repeats"] * len(cases)
if calls > config["maxCalls"]:
raise ValueError("Worst-case API calls exceed approved cap")
baseline = read(ROOT / "baseline-skill.txt").decode("utf-8")
if not baseline.strip() or len(baseline) > config["maxInputChars"]:
raise ValueError("Missing or oversized owned baseline")
return config, cases, baseline, {
"configSha256": digest(read(config_file)),
"casesSha256": digest(read(cases_file)),
"heldoutSha256": digest(canonical(hold).encode()),
"heldoutSourceIds": sorted(source_ids["heldout"]),
"heldoutGroupIds": sorted(groups["heldout"]),
"heldoutInputHashes": sorted(input_hashes["heldout"]),
"allSourceIds": sorted(source_ids["development"] | source_ids["heldout"]),
"allGroupIds": sorted(groups["development"] | groups["heldout"]),
"allInputHashes": sorted(input_hashes["development"] | input_hashes["heldout"]),
"allProvenance": [
{"caseId": x["id"], "split": x["split"], "sourceId": x["sourceId"],
"group": x["group"], "sourceRevision": x["sourceRevision"],
"inputSha256": x["inputSha256"], "truthSha256": x["truthSha256"]}
for x in cases],
"heldoutProvenance": [
{"caseId": x["id"], "sourceId": x["sourceId"], "group": x["group"],
"sourceRevision": x["sourceRevision"], "inputSha256": x["inputSha256"],
"truthSha256": x["truthSha256"]} for x in hold],
"baselineSha256": digest(baseline.encode()),
"runnerSha256": digest(read(__file__)),
"workflowSha256": digest(read(ROOT / ".github" / "workflows" / "autoresearch.yml")),
"schemaVersion": SCHEMA_VERSION,
"graderVersion": GRADER_VERSION,
"configuredModel": config["model"],
"expectedResolvedModel": config["resolvedModel"],
"toolsEnabled": False,
"worstCaseCalls": calls,
}
def response_text(payload):
if payload.get("status") != "completed":
raise RuntimeError("Model response did not complete")
texts = [part["text"] for item in payload.get("output", [])
if item.get("type") == "message" and item.get("status") == "completed"
for part in item.get("content", []) if part.get("type") == "output_text"]
if len(texts) != 1:
raise RuntimeError("Expected one completed text output")
return texts[0]
def call_model(config, instructions, user_input, schema, counter, stage, case_id=None,
repetition=None):
counter["calls"] += 1
if counter["calls"] > config["maxCalls"]:
raise RuntimeError("Approved call cap exceeded")
key = os.environ.get("OPENAI_API_KEY")
if not key:
raise RuntimeError("OPENAI_API_KEY absent")
body = {
"model": config["model"],
"store": False,
"tools": [],
"tool_choice": "none",
"max_output_tokens": config["maxOutputTokens"],
"input": [{"role": "developer", "content": instructions},
{"role": "user", "content": user_input}],
"text": {"format": {"type": "json_schema", "name": "experiment_result",
"strict": True, "schema": schema}},
}
request = urllib.request.Request(
"https://api.openai.com/v1/responses",
data=canonical(body).encode(),
headers={"Authorization": "Bearer " + key, "Content-Type": "application/json"},
method="POST")
call_dir = ROOT / "results" / "calls" / counter["runId"]
call_dir.mkdir(parents=True, exist_ok=True)
stem = f'{counter["calls"]:04d}'
attempt_file = call_dir / (stem + ".attempt.json")
response_file = call_dir / (stem + ".response.json")
success_file = call_dir / (stem + ".success.json")
error_file = call_dir / (stem + ".error.json")
write_once(attempt_file, {
"runId": counter["runId"], "identity": counter["identity"],
"ordinal": counter["calls"], "stage": stage, "caseId": case_id,
"repetition": repetition, "requestBody": body,
"requestSha256": digest(canonical(body).encode()),
"runnerSha256": counter["runnerSha256"],
})
try:
with urllib.request.urlopen(request, timeout=60) as answer:
payload = json.load(answer)
# Raw provider output is durably saved before interpretation or another call.
write_once(response_file, {"runId": counter["runId"], "ordinal": counter["calls"],
"payload": payload})
if payload.get("model") != config["resolvedModel"]:
raise RuntimeError("Resolved model differs from approved model")
parsed = json.loads(response_text(payload))
if (not isinstance(parsed, dict) or set(parsed) != set(schema["properties"]) or
("label" in parsed and parsed["label"] not in config["labels"]) or
("skill" in parsed and not isinstance(parsed["skill"], str))):
raise RuntimeError("Response failed local output schema check")
meta = {"responseId": payload.get("id"), "model": payload.get("model"),
"usage": payload.get("usage"), "status": payload.get("status")}
if not meta["responseId"]:
raise RuntimeError("Completed response lacks provider ID")
write_once(success_file, {"runId": counter["runId"],
"ordinal": counter["calls"],
"responseSha256": digest(read(response_file)),
"responseId": meta["responseId"],
"model": meta["model"],
"usage": meta["usage"]})
counter["records"].append({
"ordinal": counter["calls"], "stage": stage, "caseId": case_id,
"repetition": repetition, "attemptSha256": digest(read(attempt_file)),
"responseSha256": digest(read(response_file)),
"successSha256": digest(read(success_file))})
usage = meta["usage"]
tokens = usage.get("total_tokens") if isinstance(usage, dict) else None
if not isinstance(tokens, int):
counter["tokensKnown"] = False
else:
counter["tokens"] += tokens
return parsed, meta
except Exception as exc:
reason = f"Provider HTTP {exc.code}" if isinstance(exc, urllib.error.HTTPError) else type(exc).__name__
write_once(error_file, {"runId": counter["runId"], "ordinal": counter["calls"],
"stage": stage, "reason": reason,
"outcome": "unknown; inspect response, run and account before retry",
"responseSaved": response_file.exists()})
raise RuntimeError(f"Call {counter['calls']} blocked: {reason}; inspect saved receipts") from None
def generate(config, baseline, development, counter):
schema = {"type": "object", "properties": {"skill": {"type": "string"}},
"required": ["skill"], "additionalProperties": False}
instruction = ("Propose one revised instruction for the same classification task. "
"Return JSON with skill. Do not claim measured improvement. "
"Use only these development observations. Keep the label set and task unchanged.")
summary = [{"id": row["id"], "input": row["input"],
"inputSha256": row["inputSha256"], "sourceId": row["sourceId"],
"sourceRevision": row["sourceRevision"],
"expected": row["expected"], "observed": row["observed"],
"repetition": row["repetition"]}
for row in development]
user_input = canonical({"baseline": baseline, "labels": config["labels"],
"developmentObservations": summary})
if len(user_input) > config["maxInputChars"] * 4:
raise ValueError("Generation context exceeds cap")
write_once(ROOT / "results" / "runs" / (counter["runId"] + ".generation-context.json"),
{"runId": counter["runId"], "input": user_input,
"inputSha256": digest(user_input.encode()),
"source": "approved development cases only"})
result, meta = call_model(config, instruction, user_input, schema, counter, "generate")
skill = result["skill"]
if not isinstance(skill, str) or not skill.strip() or len(skill) > config["maxInputChars"]:
raise ValueError("Invalid generated skill")
if skill == baseline:
raise ValueError("Generated skill equals baseline")
return skill, meta
def evaluate(config, skill, cases, counter):
schema = {"type": "object",
"properties": {"label": {"type": "string", "enum": config["labels"]}},
"required": ["label"], "additionalProperties": False}
outcomes = []
for row in cases:
for repetition in range(config["repeats"]):
stage = counter["stage"]
output, meta = call_model(config, skill, row["input"], schema, counter,
stage, row["id"], repetition)
label = output.get("label")
outcomes.append({"id": row["id"], "split": row["split"],
"repetition": repetition, "expected": row["expected"],
"observed": label, "pass": label == row["expected"],
"provider": meta})
return outcomes
def compare(baseline, candidate, cases, repeats):
lookup = {(x["id"], x["repetition"]): x for x in candidate}
movements = []
for left in baseline:
right = lookup[(left["id"], left["repetition"])]
movements.append({"id": left["id"], "split": left["split"],
"repetition": left["repetition"],
"baselinePass": left["pass"], "candidatePass": right["pass"]})
parts = {split: [x for x in movements if x["split"] == split]
for split in ("development", "heldout")}
counts = {split: {
"baselinePass": sum(x["baselinePass"] for x in rows),
"candidatePass": sum(x["candidatePass"] for x in rows),
"regressions": sum(x["baselinePass"] and not x["candidatePass"] for x in rows),
"improvements": sum(not x["baselinePass"] and x["candidatePass"] for x in rows)}
for split, rows in parts.items()}
# Editorially conservative screen, not a significance claim or automatic release.
eligible = all(
v["regressions"] == 0 and len({
x["id"] for x in parts[split]
if not x["baselinePass"] and x["candidatePass"]
}) >= 2 for split, v in counts.items())
return {"counts": counts, "movements": movements,
"screen": "eligible_for_owner_review" if eligible else "hold",
"promotion": "requires_owner_review"}
def prepare():
config, cases, baseline, binding = load()
if PENDING.exists():
raise RuntimeError("Unresolved pending run; inspect and reconcile before retry")
if not STATE.exists() and STATE.parent.exists() and any(STATE.parent.iterdir()):
raise ValueError("Exposure state missing beside prior results; restore and reconcile history before paid calls")
state = read_json(STATE) if STATE.exists() else {
"stateVersion": 3,
"baselineSha256": binding["baselineSha256"],
"usedAllSourceIds": [], "usedAllGroupIds": [], "usedAllInputHashes": [],
"usedAllProvenance": [], "usedHeldoutSourceIds": [],
"usedHeldoutGroupIds": [], "usedHeldoutInputHashes": [],
"historySha256": digest(canonical([]).encode()), "cycle": 0}
required_history = ("usedAllSourceIds", "usedAllGroupIds", "usedAllInputHashes",
"usedAllProvenance", "usedHeldoutSourceIds",
"usedHeldoutGroupIds", "usedHeldoutInputHashes", "historySha256")
if (state.get("stateVersion") != 3 or any(key not in state for key in required_history) or
state["historySha256"] != digest(canonical(state["usedAllProvenance"]).encode()) or
set(state["usedAllSourceIds"]) != {x["sourceId"] for x in state["usedAllProvenance"]} or
set(state["usedAllGroupIds"]) != {x["group"] for x in state["usedAllProvenance"]} or
set(state["usedAllInputHashes"]) != {x["inputSha256"] for x in state["usedAllProvenance"]} or
set(state["usedHeldoutSourceIds"]) != {x["sourceId"] for x in state["usedAllProvenance"] if x["split"] == "heldout"} or
set(state["usedHeldoutGroupIds"]) != {x["group"] for x in state["usedAllProvenance"] if x["split"] == "heldout"} or
set(state["usedHeldoutInputHashes"]) != {x["inputSha256"] for x in state["usedAllProvenance"] if x["split"] == "heldout"}):
raise ValueError("Missing, stale or inconsistent v3 exposure history; reconcile before paid calls")
if state["baselineSha256"] != binding["baselineSha256"]:
raise ValueError("Baseline differs from durable state")
if (set(binding["heldoutSourceIds"]) & set(state["usedAllSourceIds"]) or
set(binding["heldoutGroupIds"]) & set(state["usedAllGroupIds"]) or
set(binding["heldoutInputHashes"]) & set(state["usedAllInputHashes"])):
raise ValueError("Held-out source, group or input already exposed; seal fresh cases")
development = [x for x in binding["allProvenance"] if x["split"] == "development"]
if any(x["sourceId"] in state["usedHeldoutSourceIds"] or
x["group"] in state["usedHeldoutGroupIds"] or
x["inputSha256"] in state["usedHeldoutInputHashes"] for x in development):
raise ValueError("Prior held-out source entered development; seal fresh cases")
identity = execution_identity()
run_id = identity["GITHUB_RUN_ID"]
state["usedAllSourceIds"] = sorted(set(state["usedAllSourceIds"]) | set(binding["allSourceIds"]))
state["usedAllGroupIds"] = sorted(set(state["usedAllGroupIds"]) | set(binding["allGroupIds"]))
state["usedAllInputHashes"] = sorted(set(state["usedAllInputHashes"]) | set(binding["allInputHashes"]))
state["usedAllProvenance"] += binding["allProvenance"]
state["usedHeldoutSourceIds"] = sorted(set(state["usedHeldoutSourceIds"]) | set(binding["heldoutSourceIds"]))
state["usedHeldoutGroupIds"] = sorted(set(state["usedHeldoutGroupIds"]) | set(binding["heldoutGroupIds"]))
state["usedHeldoutInputHashes"] = sorted(set(state["usedHeldoutInputHashes"]) | set(binding["heldoutInputHashes"]))
state["historySha256"] = digest(canonical(state["usedAllProvenance"]).encode())
STATE.parent.mkdir(exist_ok=True)
if not STATE.exists():
write_once(STATE, state)
else:
write_replace(STATE, state)
write_once(PENDING, {"runId": run_id, "binding": binding,
"identity": identity,
"stateSha256": digest(read(STATE)),
"approvedBy": config["approvedBy"]})
print("Prepared bounded run " + run_id)
def execute():
config, cases, baseline, binding = load()
pending = read_json(PENDING)
if pending["binding"] != binding or pending["stateSha256"] != digest(read(STATE)):
raise ValueError("Pending run inputs or state changed")
if pending["identity"] != execution_identity():
raise ValueError("Current GitHub run, attempt, repository, ref or SHA differs")
run_id = pending["runId"]
run_file = ROOT / "results" / "runs" / (run_id + ".json")
if run_file.exists():
raise RuntimeError("Run receipt already exists")
counter = {"calls": 0, "tokens": 0, "tokensKnown": True, "records": [],
"runId": run_id, "identity": pending["identity"],
"runnerSha256": binding["runnerSha256"]}
dev = [x for x in cases if x["split"] == "development"]
# Development baseline observations alone enter the generator context.
counter["stage"] = "baseline-development"
base_dev = evaluate(config, baseline, dev, counter)
by_id = {x["id"]: x for x in dev}
feedback = [{**by_id[x["id"]], "observed": x["observed"],
"repetition": x["repetition"]}
for x in base_dev]
candidate_skill, generation = generate(config, baseline, feedback, counter)
write_once(ROOT / "results" / "runs" / (run_id + ".candidate.json"),
{"runId": run_id, "skill": candidate_skill,
"skillSha256": digest(candidate_skill.encode()),
"generationResponseId": generation["responseId"]})
counter["stage"] = "candidate-development"
candidate_dev = evaluate(config, candidate_skill, dev, counter)
hold = [x for x in cases if x["split"] == "heldout"]
counter["stage"] = "baseline-heldout"
base_hold = evaluate(config, baseline, hold, counter)
counter["stage"] = "candidate-heldout"
candidate_hold = evaluate(config, candidate_skill, hold, counter)
if len(counter["records"]) != binding["worstCaseCalls"]:
raise RuntimeError("Per-call success inventory incomplete; no comparison")
for record in counter["records"]:
stem = f'{record["ordinal"]:04d}'
call_dir = ROOT / "results" / "calls" / run_id
for suffix, field in (("attempt", "attemptSha256"),
("response", "responseSha256"),
("success", "successSha256")):
if digest(read(call_dir / (stem + "." + suffix + ".json"))) != record[field]:
raise RuntimeError("Per-call receipt changed; no comparison")
if (call_dir / (stem + ".error.json")).exists():
raise RuntimeError("Call has conflicting error receipt; no comparison")
comparison = compare(base_dev + base_hold, candidate_dev + candidate_hold,
cases, config["repeats"])
receipt = {"runId": run_id, "binding": binding, "candidateSkill": candidate_skill,
"candidateSkillSha256": digest(candidate_skill.encode()),
"generation": generation, "baseline": base_dev + base_hold,
"candidate": candidate_dev + candidate_hold,
"comparison": comparison, "actualCalls": counter["calls"],
"callReceipts": counter["records"],
"totalTokensReported": counter["tokens"] if counter["tokensKnown"] else "unknown",
"dollarCost": "unknown; reconcile with account billing"}
write_once(run_file, receipt)
state = read_json(STATE)
state["cycle"] += 1
state["latestProposalRunId"] = run_id
write_replace(STATE, state)
PENDING.unlink()
print(canonical({"runId": run_id, "screen": comparison["screen"],
"calls": counter["calls"], "dollarCost": "unknown"}))
def promote(approval_path):
"""Separate owner decision after reading raw result and account bill."""
if PENDING.exists():
raise RuntimeError("Unresolved run pending")
approval = read_json(approval_path)
run_id = approval.get("runId")
if not isinstance(run_id, str) or not run_id.isdigit():
raise ValueError("Invalid approved run ID")
run_file = ROOT / "results" / "runs" / (run_id + ".json")
run_bytes = read(run_file)
result = json.loads(run_bytes)
state = read_json(STATE)
if (result["comparison"]["screen"] != "eligible_for_owner_review" or
state["latestProposalRunId"] != run_id or
approval.get("resultSha256") != digest(run_bytes) or
approval.get("candidateSkillSha256") != result["candidateSkillSha256"] or
approval.get("decision") != "promote" or
not approval.get("reviewedBy") or not approval.get("reviewedAtUtc") or
not approval.get("billingChecked") or not approval.get("rawResultsChecked")):
raise ValueError("Exact eligible result lacks complete owner approval")
baseline_path = ROOT / "baseline-skill.txt"
if digest(read(baseline_path)) != state["baselineSha256"]:
raise ValueError("Current baseline differs from state")
if result["binding"]["baselineSha256"] != state["baselineSha256"]:
raise ValueError("Proposal based on another baseline")
receipt_path = ROOT / "results" / "promotions" / (run_id + ".json")
write_once(receipt_path, {"runId": run_id,
"approvalSha256": digest(read(approval_path)),
"previousBaselineSha256": state["baselineSha256"],
"newBaselineSha256": result["candidateSkillSha256"]})
baseline_path.write_bytes(result["candidateSkill"].encode("utf-8"))
state["baselineSha256"] = digest(read(baseline_path))
state["latestPromotionRunId"] = run_id
write_replace(STATE, state)
print("Promoted exact owner-approved candidate " + run_id)
if __name__ == "__main__":
try:
if len(sys.argv) < 2 or sys.argv[1] not in ("check", "prepare", "execute", "promote"):
raise ValueError("usage: python experiment.py check|prepare|execute|promote APPROVAL.json")
if sys.argv[1] == "check":
_, cases, _, binding = load()
print(canonical({"caseCount": len(cases), **binding, "modelCalls": 0}))
elif sys.argv[1] == "prepare":
prepare()
elif sys.argv[1] == "execute":
execute()
elif len(sys.argv) == 3:
promote(sys.argv[2])
else:
raise ValueError("promote requires APPROVAL.json")
except Exception as error:
print(f"BLOCKED: {error}", file=sys.stderr)
raise SystemExit(1)
```
Save the following as `.github/workflows/autoresearch.yml`. Leave its schedule commented out. The single concurrency group serializes this workflow's runs; it does not prevent a person or another workflow from writing the same branch. Treat any conflicting write as a reconciliation problem.
```yaml
# Copy only after the owner has approved a private repository, input scope and spend.
# Keep the schedule commented out in .github/workflows until the reviewed manual run.
name: Bounded skill experiment
on:
workflow_dispatch:
# After an owner-reviewed real manual run, add under on:
# schedule:
# - cron: '17 9 * * 1'
concurrency:
group: bounded-skill-experiment-${{ github.ref }}
cancel-in-progress: false
queue: max
permissions:
contents: write
jobs:
experiment:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Check approved inputs without model calls
run: python experiment.py check
- name: Commit pending marker before any paid call
run: |
python experiment.py prepare
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add results/state.json results/pending.json
git commit -m "Skill experiment: mark run ${{ github.run_id }} pending"
git push
- name: Generate, evaluate and save proposal
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: python experiment.py execute
- name: Commit completed proposal
run: |
git add results/
git commit -m "Skill experiment: save run ${{ github.run_id }}"
git push
- name: Retain local evidence even when a step fails
if: always()
uses: actions/upload-artifact@v4
with:
name: skill-experiment-${{ github.run_id }}
path: results/
retention-days: 14
- name: Show recovery location
if: always()
run: |
echo "### Bounded skill experiment" >> "$GITHUB_STEP_SUMMARY"
echo "Run ${{ github.run_id }}; inspect results/pending.json and the run artifact before any retry." >> "$GITHUB_STEP_SUMMARY"
```
Run `python experiment.py check` locally after installing all these files, including the workflow. It checks case provenance, truth, counts, request cap, baseline and the installed workflow hash without calling a model. A reported binding includes the runner, workflow, input and schema hashes, grader version, tool prohibition and configured/resolved model IDs. Fix a failed check before starting a paid run. Keep the runner and installed workflow synchronized; changing either after preparation blocks execution.
## 3. Authorize and inspect one manual run
The owner first approves the exact repository, private data scope, model, secret access, request cap, job duration and spending limit. Add `OPENAI_API_KEY` as a GitHub Actions secret. Commit the files on the default branch and start one `workflow_dispatch` run. This is a real paid operation when you do it; this article did not start one.
The first workflow push commits `results/state.json` and `results/pending.json` before any model request. Each call starts with an attempt receipt; a received reply gets a response receipt, and a validated reply gets a success receipt in `results/calls/RUN`. It binds each attempt to the GitHub run ID, attempt, repository, ref and checkout SHA, as well as the request, runner, workflow, grader, schema and model. An HTTP failure, timeout, unexpected model or invalid answer leaves the pending marker for investigation. A missing or inconsistent history beside prior results also blocks a new run; restore and reconcile the real history instead of bootstrapping a blank one.
Inspect the Actions run and remote branch. Match the run ID and attempt to the pending marker's first commit, each call's attempt/response/success receipts, the generated candidate, all case movements, the final commit and the artifact. Check the actual provider bill. The result's dollar cost remains `unknown` until you reconcile it with the account. A valid completed run records both instructions' answers for every development and held-out case and every repetition. The screening rule requires no observed regressions and improvements on at least two distinct case IDs in each split. This rule is only a conservative prompt for review; repeated answers do not prove a statistically reliable gain. Examine the raw cases and task relevance yourself.
If the first marker push fails, this workflow has made no provider call because `execute` has not started. Inspect the remote branch, run log and account usage before trying again. If a later call is uncertain, earlier received responses remain in local receipts, but only a successful commit makes them durable in the repository. The artifact step may retain local evidence; cancellation or runner loss can prevent its upload. If the final push fails, the remote pending marker blocks recurrence. Recover the exact saved bytes and reconcile the bill before clearing it. Do not replay uncertain paid calls or turn missing answers into failures scored as zero.
## 4. Enable recurrence only after the manual result is understood
After inspecting a completed real manual run, separately approve a cadence. Add this under `on:` in the workflow and commit it on the default branch:
```yaml
schedule:
- cron: '17 9 * * 1'
```
This example requests Mondays at 09:17 UTC. Scheduled workflows run from the default branch and may be delayed or dropped. Compare the due times with Actions history; do not infer a successful experiment from a quiet inbox. The account that creates or later changes the schedule controls its scheduled-run notifications under GitHub's rules, subject to that account's settings. Confirm the intended recipient and one actual notification path. This workflow sends no separate message.
On the first actual due run, inspect its committed pending marker, complete receipt inventory, proposal, cost and final state exactly as you did for the manual run. A prior manual result does not validate a later run's fresh cases or model behavior. If a due run is absent, check the schedule and Actions history. If it is blocked, reconcile the existing marker or state instead of starting another job.
## 5. Decide whether the proposed instruction becomes the baseline
The workflow never promotes its candidate. A `hold` screen or an eligible screen can both end with a human hold. For a candidate you accept after checking every raw result and the account bill, create private `approval.json` with the exact completed run ID, hash of `results/runs/RUN.json`, candidate hash from that result, reviewer and review time:
```json
{"runId":"<run-id>","resultSha256":"<sha256-of-results/runs/RUN.json>","candidateSkillSha256":"<candidate-hash-in-run>","decision":"promote","reviewedBy":"<owner>","reviewedAtUtc":"<UTC timestamp>","billingChecked":true,"rawResultsChecked":true}
```
Run `python experiment.py promote approval.json`, inspect the updated `baseline-skill.txt`, state and promotion receipt, then commit those exact bytes. Promotion accepts only the latest eligible proposal against the unchanged old baseline. To hold, leave the baseline alone and record the owner's decision with the run evidence; do not manufacture a promotion approval. Before the next due experiment, prepare newly permissioned held-out families and truth files. If you cannot supply fresh cases or reconcile a pending call, let the run block and resolve that evidence first.
## Sources
- [GitHub Actions: events that trigger workflows](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#schedule) — Documents default-branch schedule and delayed or dropped due runs; does not guarantee that a cron interval executes.
- [GitHub Actions: workflow concurrency](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency) — Supports the workflow group and queue policy; external branch writers remain outside it.
- [GitHub Actions: secrets](https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-secrets) — Supports the injected key; it gives no permission for reader data or guarantee about log privacy.
- [GitHub Actions: artifacts](https://docs.github.com/en/actions/tutorials/store-and-share-data) — Supports a finite recovery artifact; a failed upload or lost runner can leave only the remotely committed pending marker.
- [GitHub Actions: notifications](https://docs.github.com/en/subscriptions-and-notifications/how-tos/managing-github-actions-notifications) — Explains owner-configured Actions notifications; this guide configures no recipient.
- [GitHub Actions: notifications for workflow runs](https://docs.github.com/en/actions/concepts/workflows-and-actions/notifications-for-workflow-runs) — Documents recipient effects of creating, changing and re-enabling a schedule.
- [OpenAI Responses API reference](https://developers.openai.com/api/reference/resources/responses/methods/create) — Supports the request, response model/status/output and usage fields; no live account or model response was tested.
- [OpenAI Structured Outputs](https://developers.openai.com/api/docs/guides/structured-outputs?api-mode=responses) — Supports JSON schema output shape; the reader's source-checked truth remains separate.
- [OpenAI evaluation best practices](https://developers.openai.com/api/docs/guides/evaluation-best-practices) — Supports task-specific repeated evaluation as guidance; the kit's threshold is original and has no statistical or live outcome claim.
The BIG CHANGE newsletter
The big picture. At your pace.
Recent stories on AI and robotics, the shifts worth watching and practical ideas to use. Choose a daily briefing, weekly digest or monthly perspective.
Next scheduled send (UTC): . Your first edition arrives at the next scheduled send after you confirm.
Your privacy, your choice.
Necessary storage supports site security and remembers your choices. Optional Google Analytics stays off until you allow it. You can read every story with necessary storage only. Privacy details