An AI agent that acts in connected accounts can change data or trigger business processes. Before choosing a vendor, a business owner and security reviewer can compare the controls for one proposed workflow with the organization's written requirements. The result is a dated evidence record. A missing answer stays open; a product claim alone does not show that an action will be stopped. The method draws on the NIST AI Risk Management Framework Playbook and OWASP's AI Agent Security Cheat Sheet; neither has assessed the candidate vendor.
The big change
- What changed: An agent purchase can include delegated actions in business systems. The buyer needs to know who can make the agent act, on what, and under whose approval.
- Why it matters: A buyer can tie each proposed action to a named control and the evidence for it. That gives the business owner and security reviewer a concrete basis for limiting access or holding the purchase.
- What to watch: The decision turns on evidence for the exact product version and workflow. A control shown in documentation may still need a supervised demonstration before the reviewer accepts it for the intended use.
Gather four inputs
Use the existing written risk profile approved by your organization. It should identify the business owner, intended outcome, permitted data and destinations, allowed actions, approval authority, unacceptable consequences and stop conditions. If it does not exist, ask the responsible people to define it before assessing the supplier. This guide cannot set the organization's tolerance for them.
NIST's MANAGE 3.1 guidance calls for applying organizational risk tolerance to third-party AI systems and documenting their controls. Its Playbook offers voluntary suggestions, not a universal procurement checklist. The action-control questions below also draw on OWASP's agent-specific security guidance. These frameworks identify useful questions; the supplier's own evidence must answer them.
Name one candidate vendor and the exact product and deployment version under consideration. Obtain current product documentation and any evidence the supplier is authorized to share. Give the security reviewer and procurement lead access to the comparison record. Keep sensitive architecture or evidence files in an approved private location; the worksheet can link to them rather than copy secrets into it.
1. Turn the workflow into requirements
List every proposed agent action, its target system and the data it must read or change. Separate reading from writing and externally visible or irreversible actions. For each, copy the relevant requirement from the written risk profile: permitted identity, tool and target scope, approval, logging, data handling, interruption and recovery. Record the profile's owner and date.
Create one worksheet row per requirement. This makes an unknown visible instead of burying it in an overall vendor score. A requirement that the agent never accesses a particular account needs a row just as much as an allowed action does.
2. Trace who acts and what can execute
Ask which human, service account or agent identity initiates each action, where credentials are held, and whether one agent can pass a request to another. Identify the component that checks permission at execution. Compare the vendor's actual tool permissions with the profile's allowed tools, operations, accounts and targets. Request configuration or architecture evidence that applies to this product version; a statement in a prompt is weak evidence of an enforced boundary.
For actions requiring approval, ask what the reviewer sees before execution and how approval is bound to the actor, exact tool call, target and parameters. Find out what happens when approval expires, is denied, is used twice or the requested action changes. Record how access can be revoked and who can stop an active run. An approval screen and an execution check answer different questions; request evidence for both when the profile requires both.
3. Follow the data and the record of action
Draw the proposed path from input to model, tools, external recipients, stored copies and logs. Ask what data each component receives, who can retrieve it, and how retention and deletion work. Include any subprocessors or cross-system transfers that matter to your profile. Mark an unanswered destination or retention question unknown; do not infer privacy protection from a broad policy statement.
Request an execution record for the relevant version. Can the reviewer connect an actor and policy version to the proposed tool call, approval or denial, exact target and outcome? Check whether the supplied record shows a decision only, an actual execution, or both. A screenshot, a configuration export and a complete event trail establish different things. Note any redaction that prevents the comparison, while keeping credentials and personal data out of the worksheet.
4. Ask for failure evidence
Request version-bound evidence for a permitted action and the failure paths the profile requires: denied or expired approval, out-of-scope tool or target, interrupted execution, and a logging or policy-check failure where relevant. Record what was tested, when, under which policy and tool settings, the expected result and the observed result. A test plan is a request for evidence, not an observed result.
The accountable security reviewer decides whether a supervised demonstration is needed and authorizes its scope. Use permitted non-production inputs and accounts. Do not connect live systems merely to complete a procurement worksheet. If the supplier cannot provide evidence, record the gap and the next request rather than imagining the outcome of a test.
5. Record the decision and the open work
For each row, mark fit only when the supplied evidence addresses the written requirement for this version and workflow. Mark gap when it conflicts, or unknown when evidence is missing or too weak. Note conflicting records and who must resolve them. The business owner and accountable security reviewer then record their actual decision: request more evidence, seek approval for a constrained pilot, defer or reject. A fit entry is an evidence judgment, not a guarantee that the system is safe or a substitute for the organization's purchase process.
Keep the record dated and revisit affected rows when the workflow, product version, policy, connected accounts or data destinations change. A later pilot needs its own authorization, limits and review.
Requirement from written profile | Vendor evidence, version and date | Evidence type and what it shows | Fit, gap or unknown | Reviewer and next action |
|---|---|---|---|---|
[One action or control requirement] | [Private link or document reference; version; date] | [Claim, configuration, event record or observed demonstration; limitation] | [Choose one and explain] | [Name, evidence request or decision] |
Record header: workflow and owner; risk-profile version and approval date; candidate product and deployment version; review date; security reviewer; procurement lead; location of restricted evidence. Decision record: decision, unresolved rows, accepted limits or compensating controls, accountable approver and next review trigger. Leave those fields blank until the responsible people actually review them.
The public guidance used to shape these questions is freely available. Preparing the comparison takes staff and reviewer time. Supplier evidence, confidential access and any authorized demonstration may add cost or limit what can be verified; obtain actual terms from the candidate. This documentation exercise neither tests a vendor nor establishes legal compliance or a universal purchase threshold.
Sources & further reading
- NIST AI RMF Playbook, Manage supports applying an organization's own risk tolerance to third-party AI resources, documenting controls and evaluating them. The online page says AI RMF 1.0 is being updated and that the Playbook will follow; it does not state a page revision date. It does not evaluate this vendor.
- NIST AI RMF Playbook introduction describes the suggestions as voluntary and says the Playbook is not a checklist to follow in full. The worksheet here is BIG CHANGE's synthesis for the reader's stated task.
- OWASP AI Agent Security Cheat Sheet covers scoped tools, execution-side authorization, approvals tied to actions, logging, data protection and version-bound validation evidence. The current web page states no version or release date. Its recommendations are evidence categories, not proof that a supplier implements them.
