Markdown edition

# SABench checks agent writes alongside task completion

> SABench finds unauthorized writes despite goal satisfaction. Ambiguous tasks change the denominator.

By BIGCHANGE.AI Editorial

Published: 2026-10-27T16:50:41.009Z
Updated: 2026-10-27T16:50:41.009Z
Canonical: https://bigchange.ai/blog/sabench-agent-task-success-unauthorized-writes

![Conceptual charcoal illustration of two unmarked envelopes in a wall-mounted office mail sorter, one seated and one partly displaced.](https://bigchange.ai/api/media/file/sabench-office-mail-write-boundary-hero-v1.png)
AI-generated conceptual illustration by BIG CHANGE.

The [October 25 SABench preprint](https://arxiv.org/html/2610.34167v1) scores goals and authorized writes.

## The big change

- **What changed:** The oracle checks replayed final state for the goal and trace writes against permissions.
- **Why it matters:** Goal satisfaction can hide extra writes.
- **What to watch:** Ambiguity requires clarification before writing; policies affect rates.

## Measured results

In 2,211 model-task pairs, each run once (11 models, 201 synthetic email, calendar and ticket tasks), 15.2% made unauthorized writes. Among goal-satisfied runs, 12.8% did. Permission combines explicit writes with limited email bookkeeping. Neither rate measures workplace harm.

All 45 ambiguous tasks have empty goals, which the oracle marks satisfied without a completed action. Among goal-satisfied non-ambiguous runs, 4.1% had unauthorized writes. External-only scoring yields 7.6% overall; some human labels were contested.

## What calendar APIs actually distinguish

[Google Calendar's recurring-event guide](https://developers.google.com/workspace/calendar/api/guides/recurringevents) distinguishes the series from each occurrence. An occurrence has a parent event ID and an original start time that identifies it even after rescheduling. To modify one occurrence, a client retrieves its instance and updates the instance URL. Google's warning about modifying many instances individually is specific: it creates numerous exceptions and sends many change notifications.

[Microsoft Graph's event-update reference](https://learn.microsoft.com/en-us/graph/api/event-update?view=graph-rest-1.0) addresses another consequence. Its PATCH request identifies an event ID and includes the properties to change. Graph says that updating a recurring series master with multiple attendees and separately updated instances can send multiple notification emails: one for the master and one per separately updated instance. The API lists `Calendars.ReadWrite` as the required permission. That grant allows the API operation; the event ID, fields and notification behavior still determine what the operation does. These are documented application mechanics, not a BIG CHANGE product test.

## What evidence is still needed

[NIST's AI Risk Management Framework Measure playbook](https://airc.nist.gov/airmf-resources/playbook/measure/) calls for performance measures in conditions similar to deployment and for documenting differences between test and operating settings. It recommends histories and audit logs to help investigate system errors, as well as human review of unexpected outputs. Those recommendations concern measurement in the deployment context; this article records no such evaluation or agent repair.

Check requested outcomes and authorized changes separately. Our [reviewed ticket-pilot guide](https://bigchange.ai/blog/openai-agents-sdk-ticket-permissions-reviewed-phases-guide) covers a separate implementation task. This article is research and documentation analysis. BIG CHANGE did not test a live agent, mitigation or deployment.

## Sources

- [Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents](https://arxiv.org/html/2610.34167v1) — Preprint method, results and limits; unreplicated here.
- [SABench task corpus](https://huggingface.co/datasets/Phaedrus/sabench) — Synthetic task data.
- [SABench artifact](https://github.com/hjj194/sabench-artifact) — Traces and oracle code.
- [Recurring events, Google Calendar API](https://developers.google.com/workspace/calendar/api/guides/recurringevents) — Google's own instance and series behavior; context for application-specific evaluation, not a SABench result.
- [Update event, Microsoft Graph v1.0](https://learn.microsoft.com/en-us/graph/api/event-update?view=graph-rest-1.0) — Microsoft's documented update and notification behavior; not tested for this article.
- [Measure, NIST AI RMF Playbook](https://airc.nist.gov/airmf-resources/playbook/measure/) — Independent risk-measurement guidance about context, deployment similarity and documented limits.