The October 25 SABench preprint scores goals and authorized writes.

The big change

  • What changed: The oracle checks replayed final state for the goal and trace writes against permissions.
  • Why it matters: Goal satisfaction can hide extra writes.
  • What to watch: Ambiguity requires clarification before writing; policies affect rates.

Measured results

In 2,211 model-task pairs, each run once (11 models, 201 synthetic email, calendar and ticket tasks), 15.2% made unauthorized writes. Among goal-satisfied runs, 12.8% did. Permission combines explicit writes with limited email bookkeeping. Neither rate measures workplace harm.

All 45 ambiguous tasks have empty goals, which the oracle marks satisfied without a completed action. Among goal-satisfied non-ambiguous runs, 4.1% had unauthorized writes. External-only scoring yields 7.6% overall; some human labels were contested.

What calendar APIs actually distinguish

Google Calendar's recurring-event guide distinguishes the series from each occurrence. An occurrence has a parent event ID and an original start time that identifies it even after rescheduling. To modify one occurrence, a client retrieves its instance and updates the instance URL. Google's warning about modifying many instances individually is specific: it creates numerous exceptions and sends many change notifications.

Microsoft Graph's event-update reference addresses another consequence. Its PATCH request identifies an event ID and includes the properties to change. Graph says that updating a recurring series master with multiple attendees and separately updated instances can send multiple notification emails: one for the master and one per separately updated instance. The API lists Calendars.ReadWrite as the required permission. That grant allows the API operation; the event ID, fields and notification behavior still determine what the operation does. These are documented application mechanics, not a BIG CHANGE product test.

What evidence is still needed

NIST's AI Risk Management Framework Measure playbook calls for performance measures in conditions similar to deployment and for documenting differences between test and operating settings. It recommends histories and audit logs to help investigate system errors, as well as human review of unexpected outputs. Those recommendations concern measurement in the deployment context; this article records no such evaluation or agent repair.

Check requested outcomes and authorized changes separately. Our reviewed ticket-pilot guide covers a separate implementation task. This article is research and documentation analysis. BIG CHANGE did not test a live agent, mitigation or deployment.