THE WORLD IS NOT STANDING STILL.RSS
BIG CHANGE.
BIG CHANGE, one tap away
BIG CHANGE, one tap away

Make room for the big picture.

Add BIG CHANGE to your home screen or desktop. It opens in its own window, with the latest stories when you’re online.

iPhone & iPad
Open bigchange.ai in Safari. Tap Share, then Add to Home Screen. Turn on Open as Web App if shown, then tap Add.
Android
Open bigchange.ai in Chrome. In the browser menu, choose Install app or Add to Home screen.
Desktop
In Chrome or Edge, use the install icon in the address bar or the browser’s app menu. In Safari on Mac, choose File → Add to Dock.

Markdown edition

# Evaluate an AI agent product across three decision axes

> Map one agent product’s runtime and data paths, orchestration and oversight, and interface contract against your own mandatory requirements with a copyable decision worksheet.

By BIGCHANGE.AI Editorial

Published: 2026-10-25T12:16:24.953Z
Updated: 2026-10-25T12:16:24.953Z
Canonical: https://bigchange.ai/blog/evaluate-ai-agent-product-three-axis-decision-guide

![Conceptual charcoal illustration of one reader comparing a blank three-part worksheet on a stand with a generic document displayed on a monitor.](https://bigchange.ai/api/media/file/agent-product-three-axis-evidence-review-hero-v1.png)
AI-generated conceptual illustration by BIG CHANGE.

Choosing an AI agent product starts with the work it must do. Write down the real input, required output, permitted actions and conditions under which a person must intervene. Then inspect the product on three axes: where it runs and sends data, who controls each step, and the contract by which your application calls it and receives a result. A polished demo cannot answer those questions for your workflow.

This guide gives you a blank decision worksheet and a versioned documentation example using the [OpenAI Agents SDK for Python 0.24.0](https://github.com/openai/openai-agents-python/releases/tag/v0.24.0), released on 15 October 2026. The example maps documented capabilities. BIG CHANGE did not install the SDK, run a model, test an account or verify a deployment. Your own requirements and authorized trials determine the decision.

## The big change

- **What changed:** Agent products offer several ways to run code, route work and connect tools. Their feature lists do not specify where a particular application runs or which permissions it grants. A three-axis map turns those choices into questions an owner can verify.
- **Why it matters:** A documented handoff or approval feature has little value to a team whose deployed workflow does not use it. Connecting product evidence to the team's mandatory requirements exposes that gap before real data or consequential tool access is added.
- **What to watch:** The decision changes only when a missing requirement is supplied, an unknown is resolved with evidence, or an authorized normal and failure trial verifies the configured workflow. A newly released version needs its own check.

## Before you compare products

Choose **one named product, release and access tier** for each copy of the worksheet. Gather its current primary manual, API or SDK reference, release notes, pricing and privacy terms. Record when you checked each source. A feature available in a different tier, variant or version does not fill the row for the product under review.

The decision owner supplies the actual task, input and output, users, data categories, allowed side effects, deployment and retention rules, deadline and spending limit. Mark each criterion **mandatory**, **preferred** or **not required**, with the real threshold where one exists. If these inputs are missing, the decision is **requirements pending**. The worksheet can still map product facts, but it cannot endorse a fit.

Use `documented` for a claim read in the product's versioned manual, `observed` or `measured` only for a real authorized check, and `inferred` for a conclusion drawn from documented behavior. Give each row a result of `supported`, `unsupported`, `unknown` or `not applicable`. Record why `not applicable` is legitimate; it cannot remove a mandatory need.

## 1. Locate the runner and every data path

Find out whether the runner executes on a reader-operated device or server, on a vendor host, or in a mixed arrangement. Draw the path from input to model, tool, storage and final output. Name the operator at each stop. Check platform and language requirements, network dependency, available regions and service access against the requirements you wrote down. When the manual describes a software component but not your hosting location or residency, mark the deployment row **unknown** until your architecture and the provider's terms establish it.

Also record who maintains the runtime and who can see logs or traces. “The code runs in our process” does not settle whether prompts, tool arguments or telemetry travel to a provider.

## 2. Map routing, effects and oversight

Trace who selects the next step: the model, fixed application code or a person. Identify the owner of any manager agent or handoff, the tools reachable after each transition and the effects those tools can have. Look for a real approval point before an effect, a turn limit, the failure stop, a way to resume or recover, and a trace that the operator is permitted to inspect.

An SDK may offer all these controls while an individual application enables only some of them. Record both the documented capability and the actual configuration. Treat an absent application setting or inaccessible run record as **unknown** for that application. A model API key authenticates model requests; it does not authorize users of your application or grant every downstream tool action. Those boundaries need their own permissions and checks.

## 3. Write down the complete interface contract

Specify the input type and schema, size or format limits, and what happens to missing or unsupported input. Do the same for final output, partial output, interruption and error. Name the trigger: an SDK call, HTTP endpoint, UI action, event or schedule. State who can invoke it and how that caller is authenticated. Record the package or API version, dependency compatibility, migration notes and the owner of any adapter your team must build.

Finish with access tier, model and tool charges, usage controls, privacy settings, retention, rights to inputs and outputs, licensing and support. A license for client code does not set the price or rights for model calls. Check each vendor term that matters to your data and publication plan; leave the row **unknown** when the applicable contract is unavailable.

## Copy the decision worksheet

Save this blank table for **one product and release**. In each requirement cell write `mandatory`, `preferred` or `not required` plus the actual threshold. In each evidence cell write the exact URL or authorized record ID, section, version, check date and stage (`documented`, `observed`, `measured` or `inferred`). The result cell takes `supported`, `unsupported`, `unknown` or reasoned `not applicable`. The last cell names the owner and next check. Keep private credentials and personal data out of the sheet.

Product/release/access tier: **\_**\_  Decision owner and date: **\_**\_

Real task, input, output and allowed effects: **\_**\_  Users and data category: **\_**\_

Permitted data locations and retention: **\_**\_  Trial environment and allowed effects: **\_**\_

Deadline and spend ceiling: **\_**\_  Primary documentation and checked dates: **\_**\_

| Axis and criterion | Requirement and threshold | Evidence, version, date and stage | Result | Owner, check or recovery |
| --- | --- | --- | --- | --- |
| 1. Runner location and operator | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| 1. Model, tool, data and trace paths; residency | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| 1. Platform, runtime, network and availability | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| 2. Next-step selection: model, code or person | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| 2. Manager, handoff and permitted tool effects | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| 2. Approval, turn limit, failure stop, trace and recovery | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| 3. Input type, schema, limits and invalid input | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| 3. Final/partial output, schema, interruption and errors | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| 3. Trigger and authorized caller | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| 3. Model authentication and separate app/tool permissions | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| 3. Version, dependencies and migration | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| Across axes: access, charges and usage controls | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |
| Across axes: privacy, retention, rights and support | **\_**\_ | **\_**\_ | **\_**\_ | **\_**\_ |

Decision: **\_**\_  Mandatory failed/unknown rows: **\_**\_  Preferred gaps: **\_**\_

Authorized normal trial and result: **\_**\_  Denied or missing-input trial and result: **\_**\_

Actual output, tool effects, failure behavior and usage record: **\_**\_  Tested scope/date: **\_**\_

## A versioned documentation example

The [tagged quickstart](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/quickstart.md) installs `openai-agents` in a Python project, sets `OPENAI_API_KEY` for the default OpenAI model path and starts an agent with `Runner.run`. [Package metadata](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/pyproject.toml) identifies version 0.24.0, Python 3.10 or later, an `openai` dependency in the 3.x range and an MIT license. The standard `Agent` plus `Runner` path therefore points to a runner in the application's Python process, with model requests sent through the configured provider; [the Agents manual](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/agents.md) names the Responses API as the OpenAI model default. The process location is an inference from the published code path, not a verified hosting or residency arrangement. The operator must document the chosen host, network route and region.

For routing, the [orchestration manual](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/multi_agent.md) describes model-selected and code-controlled flow, including a manager agent calling specialists as tools and handoffs that change the active agent. The [runner manual](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/running_agents.md) describes the loop and `max_turns`. [Function tools](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/tools.md) can wrap Python functions; the [human approval manual](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/human_in_the_loop.md) describes an approval interruption and resume. These are SDK options. Which route, tool permissions, approval rule, turn limit and recovery procedure a particular application uses remains to be inspected.

The runner accepts a string, Responses-format input items or saved run state on resume, according to the [runner manual](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/running_agents.md). The [results manual](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/results.md) documents a final output, run items, last agent and interruptions; a streaming result is incomplete until processing finishes. The quickstart's trigger is a Python call. It does not establish a public endpoint, timer, application input validator or the caller permissions for one. The reader must supply those parts of the contract and test its own incomplete and error handling.

For cost and privacy, the [usage manual](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/usage.md) records requests and tokens per run, while [API pricing](https://developers.openai.com/api/docs/pricing) depends on the chosen model and usage. The [tracing manual](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/tracing.md) says tracing is on by default and sensitive model/tool inputs and outputs can be included by default; it documents controls to disable or limit collection. Review the configured tracing path, retention and applicable terms before entering real data. No account entitlement, fixed task cost or operational result follows from these manuals.

**Decision for this example: requirements pending.** The reader's task, thresholds and deployed configuration are blank. This documentation map establishes capabilities and questions to check. It makes no product-fit claim.

## Apply the decision and verify the result

1. **Fill every mandatory threshold.** If the task or a mandatory threshold is missing, write `requirements pending` and name the missing input. Do not assign a score.
2. **Check the evidence row by row.** Any mandatory `unsupported` result means `no documented fit` for the stated scope. Cite the row. Any mandatory `unknown` means `hold for evidence`; name the exact manual, account setting, contract or bounded trial needed. Resolve an unsupported requirement before treating any other rows as an endorsement.
3. **Separate documentation from operation.** If all mandatory rows are documented as `supported`, record `documented fit for the stated scope; operational validation pending`. Preferred gaps and limits stay visible. The existence of a product option does not prove that the configured application uses it.
4. **Run authorized checks before adoption.** With a real permitted input and an agreed spend and effects boundary, run a representative normal case and a denied or missing-input case. Inspect actual inputs, outputs, tool effects, errors, approval behavior and usage. Record `validated fit for tested scope` only if the required normal and failure behavior passes. State exactly what environment and version were tested. This article ran no such trial.

If a document has moved, the chosen version differs or access is missing, preserve the old citation, mark affected rows **unknown**, and refresh the exact official clause or arrange an authorized check. A failed trial can make a row **unsupported** for that configuration; record the observed result before revising the decision. Never turn an empty source response into evidence that a feature is absent, and never send secrets or personal data merely to complete the worksheet.

For a narrower check of one OpenAI SDK function tool's permissions, see BIG CHANGE's [tool-boundary guide](https://bigchange.ai/blog/openai-agents-sdk-function-tool-boundary-guide). The three-axis decision here covers the surrounding product and application contract.

## Sources

- [OpenAI Agents SDK Python release 0.24.0](https://github.com/openai/openai-agents-python/releases/tag/v0.24.0) — Release identity and date for the versioned example; checked 25 October 2026. A release record does not validate the reader’s deployment.
- [Agents SDK 0.24.0 quickstart](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/quickstart.md) — Documented Python setup, API key and Runner call; checked 25 October 2026. The example does not define a production host or external trigger.
- [Agents SDK 0.24.0 package metadata](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/pyproject.toml) — Version, Python and dependency bounds, and MIT code license; checked 25 October 2026. Account and provider compatibility still require a reader check.
- [Agents SDK 0.24.0 agents](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/agents.md) — Plain Agent configuration and default Responses model path; checked 25 October 2026. It does not specify the application host.
- [Agents SDK 0.24.0 orchestration](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/multi_agent.md) — Model-selected and code-controlled patterns, managers and handoffs; checked 25 October 2026. An application must choose and configure its route.
- [Agents SDK 0.24.0 running agents](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/running_agents.md) — Runner input shapes, loop and turn limit; checked 25 October 2026. It does not provide the reader’s application schema.
- [Agents SDK 0.24.0 tools](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/tools.md) — Tool categories and function-tool boundary; checked 25 October 2026. The app owner must establish each tool’s authority and effects.
- [Agents SDK 0.24.0 human approval](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/human_in_the_loop.md) — Approval interruptions and resume; checked 25 October 2026. Availability of this option does not prove it is configured.
- [Agents SDK 0.24.0 results](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/results.md) — Final, incomplete and streaming result surfaces; checked 25 October 2026. The application must define its own external output and error contract.
- [Agents SDK 0.24.0 tracing](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/tracing.md) — Default tracing and sensitive-data capture controls; checked 25 October 2026. Actual retention and application data policy need separate review.
- [Agents SDK 0.24.0 usage](https://raw.githubusercontent.com/openai/openai-agents-python/v0.24.0/docs/usage.md) — Per-run request and token accounting; checked 25 October 2026. It supplies no fixed task cost.
- [OpenAI API pricing](https://developers.openai.com/api/docs/pricing) — Model and usage-dependent API charges; checked 25 October 2026. Readers must calculate their own task cost and access terms.