A fluent answer does not show whether an AI agent used the required procedure. AWS's September 22 guide explains how Strands Evals and Amazon Bedrock AgentCore evaluate skill selection and instruction following from recorded runs. The two systems handle an absent evaluation differently, with consequences for anyone reading their pass flags.
The big change
- What changed: These tools distinguish whether a procedure loaded, whether it suited the task and whether its steps were followed.
- Why it matters: The inspected Strands code can mark an unjudged case as passing. A release decision based on that flag alone can overlook a required skill that never loaded.
- What to watch: Check evaluation coverage alongside scores: which expected loads were observed, and which invocations had enough recorded evidence to judge.
Three checks answer different questions
The Agent Skills specification defines a skill around a SKILL.md file. Its name and description help an agent decide when to use it; the instruction body loads after activation. That separation makes successful loading a distinct question from appropriate selection or execution.
Check | Evidence and result | Limit |
|---|---|---|
Strands | Recorded successful load of a named skill; deterministic pass or fail | Does not assess suitability or execution |
AgentCore selection accuracy | Invoked skill, request, prior context and catalog when available; model judgment of Yes or No | Does not judge a decision to invoke nothing |
AgentCore instruction following | Loaded instructions and full session; five levels from fully followed to not followed | Assesses recorded step-following, not the underlying work's correctness |
The pinned SkillInvoked implementation requires a matching load event with status loaded. Without a successful load, absent, failed or unresolved attempts fail this assertion. No judge model is called. This provides a direct check when a test case requires a particular skill.
AgentCore can produce zero results
According to AgentCore's documentation, both judges skip a tool call when neither invocation nor skill-body signals are detected. A session without detected skill invocations yields zero results.
A missing catalog does not stop selection judging: the evaluator uses the request and conversation context. Instruction following requires both the invoked skill and its loaded body, with the full session supplying evidence of subsequent steps.
Strands can return a passing N/A row
Source review reveals a different representation in Strands Evals commit 9ea55c9, inspected September 23. This describes that implementation, not every installed SDK version.
With a trajectory present but no invocation, the selection evaluator returns a not-applicable row: placeholder score 0.0, test_pass=true. Whether declining to use a skill was correct is outside that judge's scope.
The instruction-following evaluator also returns passing N/A rows for no invocation, a refused load or an unavailable body. A missing trajectory instead returns N/A with a false pass flag in both evaluators.
Their aggregate function excludes N/A rows from the mean. When every row is N/A, it preserves a placeholder zero and combines the rows' pass flags. The number and flag therefore need their label and reason to be interpreted correctly.
For required routing, pair the judges with a named-load assertion. For adherence, inspect whether instructions and execution evidence were captured before treating a passing flag as a judgment. BIG CHANGE reviewed documentation and source code; it did not execute these evaluators.
