Markdown edition

# Pilot Claude's Opus adviser with Sonnet or Haiku and check quality and cost

> Run an approved real task with the same Claude executor alone and with an Opus adviser, verify both answers, and compare complete usage and billed cost.

By BIGCHANGE.AI Editorial

Published: 2026-10-23T14:20:37.349Z
Updated: 2026-10-23T14:20:37.349Z
Canonical: https://bigchange.ai/blog/claude-advisor-quality-cost-pilot-guide

![AI-generated conceptual charcoal illustration of two equally open drawers in a steel file cabinet, each holding a stack of indistinct private records.](https://bigchange.ai/api/media/file/claude-advisor-paired-receipts-hero-v1.png)
AI-generated conceptual illustration by BIG CHANGE. No paid Advisor request, quality outcome, complete bill or measured benefit is depicted.

Anthropic's [Advisor tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/advisor-tool) lets a Claude Sonnet or Haiku executor ask Claude Opus for guidance during a Messages API response. The executor chooses when to consult it. To see whether that helps your work, run the same approved task through the same executor alone, then with the adviser available, and check both results and the complete bill.

## The big change

- **What changed:** The beta API can run an Opus adviser sub-inference inside a Sonnet or Haiku executor turn, including a paused turn that needs a continuation request.
- **Why it matters:** One controlled task pair can show whether the advised result passes your own quality checks and what the whole attempt costs.
- **What to watch:** An adviser call costs extra, can fail or stop at its token cap, and may return encrypted advice. A tool-use attempt is not proof of successful advice.

## 1. Pick a task and approve its boundaries

Use one to three real tasks with an output a person can judge. A multi-step coding or research task gives the executor a reason to seek a plan or course correction; a short fact lookup is unlikely to tell you much about this feature. Freeze each prompt, input revision, starting state and acceptance criteria before running either route. For code, name the repository revision and tests in your private case record, then review any proposed diff and run the checks yourself. This harness sends text and receives text. It cannot edit a repository or execute tests.

You need Python 3, an Anthropic API account with beta access, an approved budget and an approved destination for the task data. Check the account's available models, current rates, spending controls, provider and region. The [current compatibility table](https://platform.claude.com/docs/en/agents-and-tools/tool-use/advisor-tool#model-compatibility), checked 23 October 2026, lists `claude-sonnet-5` and `claude-haiku-5-5` as executors that can pair with `claude-opus-5-5`. This sample calls the Claude API endpoint; the tool's beta availability differs across cloud providers. Keep secrets out of case prompts and local source control. Obtain the task owner's permission before sending private material to the API.

The sample sets 2,048 output tokens for each top-level request and 2,048 for each adviser call. These limits do not cap the entire conversation's cost. The script may make an initial request and up to three `pause_turn` continuations for each route. Set a real account spending control and a stopping point before authorizing paid requests. No saving or quality improvement is assumed.

## 2. Save and validate the pilot

Copy the complete Python block below into `advisor-pilot.py`. Copy the complete JSON block into `cases-template.json`, then make a private copy called `cases.json`. Replace every bracketed value with a task you are allowed to submit and criteria you can independently check. Keep the case file and response receipts private. The script uses the Python standard library and makes no request unless you add `--run-paid`.

Run this first:

```bash
python advisor-pilot.py cases.json receipts-dry-run.jsonl --executor claude-sonnet-5
```

Expected result: it reports the number of valid cases and makes no API call or receipt file. If you choose `claude-haiku-5-5`, use that executor for **both** routes. A placeholder, duplicate case ID or missing criterion stops validation.

After account, budget and data approval, set `ANTHROPIC_API_KEY` using your organization's approved secret mechanism. Run the same command with `--run-paid` and a **new private output filename**. The script creates the output exclusively, starts the baseline before the advised route for each case, and stops on a held or uncertain request. Do not put the key in the JSON or command arguments.

```bash
python advisor-pilot.py cases.json receipts-paid-01.jsonl --executor claude-sonnet-5 --run-paid
```

### Complete Python pilot

In the script docstring, `workflow.md` refers to the procedure given in this article; no separate workflow file is required.

```python
"""Reader-run, paid Claude API pilot. Stdlib only; never run by editorial preparation.

Input: a private JSON file with {"cases": [{"id": ..., "prompt": ..., "criteria": [...]}]}.
Output: a NEW private JSONL file, one durable raw receipt per completed API request,
plus route summaries or uncertainty rows.
Run only after reviewing workflow.md and passing --run-paid.
"""

import argparse
import datetime as dt
import json
import os
from pathlib import Path
import sys
import urllib.error
import urllib.request


API = "https://api.anthropic.com/v1/messages"
EXECUTORS = ("claude-sonnet-5", "claude-haiku-5-5")
ADVISOR = "claude-opus-5-5"
MAX_OUTPUT = 2048
MAX_PAUSE_CONTINUATIONS = 3


def utc_now():
    return dt.datetime.now(dt.timezone.utc).isoformat(timespec="seconds")


def check_cases(path):
    data = json.loads(path.read_text(encoding="utf-8"))
    cases = data.get("cases")
    if not isinstance(cases, list) or not 1 <= len(cases) <= 3:
        raise ValueError("Supply one to three approved cases")
    seen = set()
    for case in cases:
        if not isinstance(case, dict) or set(case) != {"id", "prompt", "criteria"}:
            raise ValueError("Each case needs exactly id, prompt and criteria")
        key, prompt, criteria = case["id"], case["prompt"], case["criteria"]
        if not isinstance(key, str) or not key or key in seen or "[" in key:
            raise ValueError("Case ids must be unique and filled")
        if not isinstance(prompt, str) or not 20 <= len(prompt) <= 10000 or "[" in prompt:
            raise ValueError("Use a filled, bounded, authorized prompt")
        if not isinstance(criteria, list) or not criteria or any(not isinstance(x, str) or not x.strip() or "[" in x for x in criteria):
            raise ValueError("Supply real acceptance criteria")
        seen.add(key)
    return cases


def post(payload, api_key, advised):
    raw = json.dumps(payload).encode("utf-8")
    headers = {
        "x-api-key": api_key,
        "anthropic-version": "2023-06-01",
        "content-type": "application/json",
    }
    if advised:
        headers["anthropic-beta"] = "advisor-tool-2026-03-01"
    request = urllib.request.Request(API, data=raw, headers=headers, method="POST")
    with urllib.request.urlopen(request, timeout=120) as response:
        if response.status != 200:
            raise RuntimeError(f"Unexpected HTTP {response.status}")
        return json.load(response)


def append_receipt(out, row):
    out.write(json.dumps(row, ensure_ascii=False) + "\n")
    out.flush()
    os.fsync(out.fileno())


def advisor_consultations(responses):
    attempts, results = {}, {}
    ambiguous_ids = 0
    for response in responses:
        for block in response.get("content", []):
            if block.get("type") == "server_tool_use" and block.get("name") == "advisor":
                if not block.get("id") or block["id"] in attempts:
                    ambiguous_ids += 1
                attempts[block.get("id")] = block
            elif block.get("type") in ("advisor_tool_result", "advisor_tool_result_error"):
                if not block.get("tool_use_id") or block["tool_use_id"] in results:
                    ambiguous_ids += 1
                results[block.get("tool_use_id")] = block
    counts = {"attempted": len(attempts), "successful": 0, "failed": 0, "pending": 0, "truncated": 0}
    for tool_id in attempts:
        result = results.get(tool_id)
        if not result:
            counts["pending"] += 1
            continue
        detail = result.get("content")
        if result.get("type") == "advisor_tool_result_error" or result.get("error_code") or (isinstance(detail, dict) and (detail.get("type") == "advisor_tool_result_error" or detail.get("error_code"))):
            counts["failed"] += 1
        elif isinstance(detail, dict) and detail.get("stop_reason") == "max_tokens":
            counts["truncated"] += 1
        elif isinstance(detail, dict) and detail.get("type") in ("advisor_result", "advisor_redacted_result") and detail.get("stop_reason") == "end_turn":
            counts["successful"] += 1
        else:
            counts["pending"] += 1
    # A result without a matching use, or a missing/duplicate id, is not proof of advice.
    counts["unmatchedResults"] = sum(1 for tool_id in results if tool_id not in attempts)
    counts["ambiguousIds"] = ambiguous_ids
    return counts


def run_case(case, route, api_key, executor, out):
    advised = route == "advised"
    messages = [{"role": "user", "content": case["prompt"]}]
    receipts = []
    for request_index in range(1, MAX_PAUSE_CONTINUATIONS + 2):
        payload = {
            "model": executor,
            "max_tokens": MAX_OUTPUT,
            "messages": messages,
        }
        if advised:
            payload["tools"] = [{"type": "advisor_20260301", "name": "advisor", "model": ADVISOR, "max_tokens": 2048}]
        try:
            response = post(payload, api_key, advised)
        except Exception as exc:
            append_receipt(out, {"atUtc": utc_now(), "caseId": case["id"], "route": route, "executor": executor, "requestIndex": request_index, "status": "request-uncertain", "errorType": type(exc).__name__, "priorResponseIds": [r.get("id") for r in receipts]})
            raise RuntimeError("Request failed or outcome uncertain; inspect saved responses and provider usage before another attempt") from exc
        append_receipt(out, {"atUtc": utc_now(), "caseId": case["id"], "route": route, "executor": executor, "advisor": ADVISOR if advised else None, "requestIndex": request_index, "requestKind": "initial" if request_index == 1 else "pause-continuation", "responseId": response.get("id"), "status": "api-response", "response": response})
        receipts.append(response)
        reason = response.get("stop_reason")
        if reason == "pause_turn":
            # Anthropic documents round-tripping the complete assistant content,
            # including encrypted advisor result blocks, for a paused turn.
            messages.append({"role": "assistant", "content": response["content"]})
            continue
        consultations = advisor_consultations(receipts) if advised else None
        if reason != "end_turn":
            return {"status": "hold", "reason": reason, "advisorConsultations": consultations}
        if advised and (consultations["successful"] == 0 or consultations["failed"] or consultations["pending"] or consultations["truncated"] or consultations["unmatchedResults"] or consultations["ambiguousIds"]):
            return {"status": "hold", "reason": "adviser consultation not fully successful", "advisorConsultations": consultations}
        return {"status": "complete", "advisorConsultations": consultations}
    return {"status": "hold", "reason": "pause continuation cap", "advisorConsultations": advisor_consultations(receipts) if advised else None}


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("cases", type=Path, help="Private filled case JSON")
    parser.add_argument("output", type=Path, help="New private JSONL receipt path")
    parser.add_argument("--executor", choices=EXECUTORS, default="claude-sonnet-5", help="Choose one supported executor for both routes")
    parser.add_argument("--run-paid", action="store_true", help="Explicitly authorize up to six paid route/case starts plus pause continuations")
    args = parser.parse_args()
    cases = check_cases(args.cases)
    if not args.run_paid:
        print(f"Validated {len(cases)} case(s). No API call. Add --run-paid only after checking account budget, privacy and access.")
        return 0
    key = os.environ.get("ANTHROPIC_API_KEY")
    if not key:
        raise ValueError("ANTHROPIC_API_KEY is required; do not put it in the case file")
    # Exclusive creation prevents silently mixing or overwriting another paid run.
    with args.output.open("x", encoding="utf-8") as out:
        for case in cases:
            for route in ("baseline", "advised"):
                print(f"Starting paid {case['id']} {route}; inspect each receipt before any rerun.", file=sys.stderr)
                result = run_case(case, route, key, args.executor, out)
                append_receipt(out, {"atUtc": utc_now(), "rowType": "route-summary", "caseId": case["id"], "criteria": case["criteria"], "route": route, "executor": args.executor, "advisor": ADVISOR if route == "advised" else None, **result})
                if result["status"] != "complete":
                    raise RuntimeError("Held incomplete response; inspect receipt before any further paid request")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())
```

### Complete cases template

```json
{
  "cases": [
    {
      "id": "[your-case-id]",
      "prompt": "[One approved real task with enough context and permitted input; no secrets]",
      "criteria": [
        "[One independently checkable criterion for the completed result]"
      ]
    }
  ]
}
```

## 3. Check the results before comparing cost

Each completed API request gets its own `api-response` JSONL row with the raw response, case, route, request index, response ID and usage. The script flushes and fsyncs that row **before** sending a `pause_turn` continuation, preserving the earlier billable ID and encrypted result blocks if a later request fails. A subsequent failure adds `request-uncertain` with the attempted index and prior response IDs, then stops without retrying. Inspect those rows and the provider's usage or billing record before considering another approved attempt. The failed request itself may have incurred a charge.

For an advised route, read `advisorConsultations` in the `route-summary` row and inspect the raw response. Match each `server_tool_use` with `name: "advisor"` to an `advisor_tool_result` by ID. Check the nested `content.type`, `error_code` and `stop_reason`. An `advisor_redacted_result` is the expected encrypted form for Opus 5.5; retain the block unchanged in a private receipt. It can count as successful when its stop reason is `end_turn`. A failed, unmatched, pending or `max_tokens` result, or zero successful advice, holds the adviser-benefit comparison. Top-level `max_tokens`, unexpected `tool_use` and the continuation cap also hold the route.

Read each final executor answer. Apply the **same** acceptance criteria to both outputs and record the human time needed to inspect or repair each. For a code task, inspect the proposed change in the approved repository and run the actual tests independently. A complete API response alone is not a correct artifact.

## 4. Record the whole decision once

Use this table as your private decision record. Include every continuation and any separately authorized retry. The API's top-level `usage` counts executor tokens; `usage.iterations[]` separates executor `message` entries from billed `advisor_message` entries and cache categories. Reconcile the complete routes to current provider rates and the account's authoritative bill. Compare money only when the same inputs, executor, starting state and acceptance standard held, and both outputs passed. If the advised route costs more, record that result. An incomplete route or unequal quality supports an observation, not a saving claim.

| Field | Your actual observation |
| --- | --- |
| Case ID, owner-approved task and input revision |  |
| Approved API destination, region, data rights and account beta access |  |
| Executor, adviser, current rate schedule, spending control and stop point |  |
| Same prompt, starting state, executor and acceptance criteria for both routes |  |
| Baseline final response ID, result and independent pass/fail evidence |  |
| Advised final response ID, result and independent pass/fail evidence |  |
| Adviser attempted/successful/failed/pending/truncated counts, matched IDs, nested errors and stop reasons |  |
| Executor input/output/cache units by route and request |  |
| Adviser `advisor_message` input/output/cache units and call count |  |
| Every response ID, pause, continuation, partial receipt, uncertain request and authorized retry |  |
| Human review and rework time for each route |  |
| Provider bill or attributable full-route cost for each route |  |
| Decision: viable here, more expensive, quality failure or inconclusive |  |

Only after a comparable pass and positive baseline cost should you compute `(baseline cost - advised cost) / baseline cost`. Treat the answer as evidence for these cases, not a general saving rate. For production use, plan routing, permissions, fallback, retention and monitoring separately.

## If the run stops

An HTTP 400 `invalid_request_error` calls for a check of the beta header, exact model pair and account access. Do not silently switch the model in one route. An adviser result with nested `stop_reason: max_tokens` needs a newly approved test design and a comparable baseline if you raise the cap. If a result fails the task criteria, preserve it and revise the task or design before authorizing another paired run. Keep private case files and raw receipts, including encrypted blocks, under your normal access and retention rules.

This is a documentation-backed pilot. BIG CHANGE did not make a paid Advisor API request, run reader cases, observe a quality result or measure a saving.

## Sources & further reading

- [Anthropic Advisor tool manual](https://platform.claude.com/docs/en/agents-and-tools/tool-use/advisor-tool): beta request shape, compatibility, result variants, pause continuation, usage and platform availability; current undated manual checked 23 October 2026.
- [Claude Opus 5.5 release coverage](https://bigchange.ai/blog/claude-opus-5-5-cost-performance-access): model pricing and access context. Check current account rates before this pilot.
- [Claude Code Opus-to-Haiku subagent guide](https://bigchange.ai/blog/claude-code-opus-haiku-subagent-coding-guide): a separate delegation method for a coding task.

## Sources

- [Anthropic: Advisor tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/advisor-tool) — Current undated beta manual checked October 23, 2026. Quick start, parameters, result variants, errors, pause continuation, usage and billing, compatibility and platform availability support the documented procedure. No paid API run, account access or actual saving was observed.
- [BIG CHANGE: Claude Opus 5.5 release coverage](https://bigchange.ai/blog/claude-opus-5-5-cost-performance-access) — Related published article #24 provides model release context, not evidence of this pilot's outcome or a dated story continuation. Current reader account rates must be checked separately.
- [BIG CHANGE: Claude Code Opus-to-Haiku subagent guide](https://bigchange.ai/blog/claude-code-opus-haiku-subagent-coding-guide) — Published article #451 covers a distinct custom subagent coding task, not the Messages API Advisor tool or this paired quality and full-cost test.