# An AI agent can skip a required skill and leave no score behind

> New skill evaluators separate loading, selection and instruction following. Their missing-result behavior shows why teams must treat evaluation coverage as evidence of its own.

By BIG CHANGE Editorial

Published: 2026-09-23T04:51:32.802Z
Updated: 2026-09-23T09:28:46.768Z
Canonical: https://bigchange.ai/blog/ai-agent-skill-evaluation-no-score-blind-spot

![Precision tools hang on a steel workshop board around one empty orange-rimmed shaped holder.](https://bigchange.ai/api/media/file/agent-skill-toolboard-evidence-hero-v4-1.png)
AI-generated conceptual illustration by BIG CHANGE. Conceptual illustration of a workshop tool board with an empty holder.

A fluent answer does not show whether an AI agent used the required procedure. [AWS's September 22 guide](https://aws.amazon.com/blogs/machine-learning/evaluate-skill-equipped-agents-with-strands-evals-and-amazon-bedrock-agentcore/) explains how Strands Evals and Amazon Bedrock AgentCore evaluate skill selection and instruction following from recorded runs. The two systems handle an absent evaluation differently, with consequences for anyone reading their pass flags.

## The big change

- **What changed:** These tools distinguish whether a procedure loaded, whether it suited the task and whether its steps were followed.
- **Why it matters:** The inspected Strands code can mark an unjudged case as passing. A release decision based on that flag alone can overlook a required skill that never loaded.
- **What to watch:** Check evaluation coverage alongside scores: which expected loads were observed, and which invocations had enough recorded evidence to judge.

## Three checks answer different questions

The [Agent Skills specification](https://agentskills.io/specification) defines a skill around a `SKILL.md` file. Its name and description help an agent decide when to use it; the instruction body loads after activation. That separation makes successful loading a distinct question from appropriate selection or execution.

| Check | Evidence and result | Limit |
| --- | --- | --- |
| Strands `SkillInvoked` | Recorded successful load of a named skill; deterministic pass or fail | Does not assess suitability or execution |
| AgentCore selection accuracy | Invoked skill, request, prior context and catalog when available; model judgment of Yes or No | Does not judge a decision to invoke nothing |
| AgentCore instruction following | Loaded instructions and full session; five levels from fully followed to not followed | Assesses recorded step-following, not the underlying work's correctness |

The pinned [`SkillInvoked` implementation](https://github.com/strands-agents/evals/blob/9ea55c96d461d70b0d742f547a95389e4c7fe18d/src/strands_evals/evaluators/deterministic/skill_invoked.py) requires a matching load event with status `loaded`. Without a successful load, absent, failed or unresolved attempts fail this assertion. No judge model is called. This provides a direct check when a test case requires a particular skill.

## AgentCore can produce zero results

According to [AgentCore's documentation](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/skill-evaluators.html), both judges skip a tool call when neither invocation nor skill-body signals are detected. A session without detected skill invocations yields zero results.

A missing catalog does not stop selection judging: the evaluator uses the request and conversation context. Instruction following requires both the invoked skill and its loaded body, with the full session supplying evidence of subsequent steps.

## Strands can return a passing N/A row

Source review reveals a different representation in Strands Evals commit `9ea55c9`, inspected September 23. This describes that implementation, not every installed SDK version.

With a trajectory present but no invocation, the [selection evaluator](https://github.com/strands-agents/evals/blob/9ea55c96d461d70b0d742f547a95389e4c7fe18d/src/strands_evals/evaluators/skill_selection_accuracy_evaluator.py) returns a not-applicable row: placeholder score `0.0`, `test_pass=true`. Whether declining to use a skill was correct is outside that judge's scope.

The [instruction-following evaluator](https://github.com/strands-agents/evals/blob/9ea55c96d461d70b0d742f547a95389e4c7fe18d/src/strands_evals/evaluators/skill_instruction_following_evaluator.py) also returns passing N/A rows for no invocation, a refused load or an unavailable body. A missing trajectory instead returns N/A with a false pass flag in both evaluators.

Their [aggregate function](https://github.com/strands-agents/evals/blob/9ea55c96d461d70b0d742f547a95389e4c7fe18d/src/strands_evals/evaluators/evaluator.py) excludes N/A rows from the mean. When every row is N/A, it preserves a placeholder zero and combines the rows' pass flags. The number and flag therefore need their label and reason to be interpreted correctly.

For required routing, pair the judges with a named-load assertion. For adherence, inspect whether instructions and execution evidence were captured before treating a passing flag as a judgment. BIG CHANGE reviewed documentation and source code; it did not execute these evaluators.

## Sources

- [AWS: Evaluate skill-equipped agents with Strands Evals and Amazon Bedrock AgentCore](https://aws.amazon.com/blogs/machine-learning/evaluate-skill-equipped-agents-with-strands-evals-and-amazon-bedrock-agentcore/) — September 22 guide introducing the evaluator split and recorded-run workflow. Its constructed tutorial examples are not reported as incidents.
- [Amazon Bedrock AgentCore: Skill evaluators](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/skill-evaluators.html) — Documents AgentCore signal requirements, catalog fallback, full-session instruction evidence and zero-result behavior.
- [Strands Evals: Skill selection accuracy evaluator at inspected commit](https://github.com/strands-agents/evals/blob/9ea55c96d461d70b0d742f547a95389e4c7fe18d/src/strands_evals/evaluators/skill_selection_accuracy_evaluator.py) — Pinned code: no invocation produces a passing N/A placeholder, not a judged selection. Applies to this inspected commit.
- [Strands Evals: Skill instruction following evaluator at inspected commit](https://github.com/strands-agents/evals/blob/9ea55c96d461d70b0d742f547a95389e4c7fe18d/src/strands_evals/evaluators/skill_instruction_following_evaluator.py) — Pinned code: no invocation, failed loads and absent bodies yield passing N/A rows; missing trajectories fail. These flags do not establish adherence.
- [Strands Evals: Deterministic SkillInvoked evaluator at inspected commit](https://github.com/strands-agents/evals/blob/9ea55c96d461d70b0d742f547a95389e4c7fe18d/src/strands_evals/evaluators/deterministic/skill_invoked.py) — Pinned source code showing that the named-skill assertion requires a recorded successful load and distinguishes absent, failed and unresolved attempts without a model judge.
- [Agent Skills specification](https://agentskills.io/specification) — Open specification for the SKILL.md structure and progressive disclosure: metadata is available for routing, while the instruction body loads after activation.
- [Strands Evals: N/A aggregation at inspected commit](https://github.com/strands-agents/evals/blob/9ea55c96d461d70b0d742f547a95389e4c7fe18d/src/strands_evals/evaluators/evaluator.py) — Aggregate implementation excludes N/A from scored means but retains a placeholder zero and combined pass flags when all rows are N/A.