An August 2026 USENIX study, highlighted by NYU on October 20, examined chatbot advice about technology-facilitated abuse.
The big change
- Answers can omit safety context.
- Perception differs from expert assessment.
- Older results cannot establish current performance.
What researchers found
Experts assessed single-turn answers to public-source questions; GPT and Claude received safety-centered prompts. Of 268 answers, 86% were imperfect in at least one of accuracy, completeness or safety—not necessarily unsafe. The sample included 139 GPT-4o (gpt-4o-2024-08-06) answers and 43 each from Claude 3.7 Sonnet (claude-3-7-sonnet-20250219), Ruth and Aimee. Missing warnings mattered: a change could alert an abuser.
Separately, 114 self-reported survivors rated perceived confidence and difficulty. Some feared escalation; their sample cannot represent all survivors. BIG CHANGE did not test the systems.
Sources & further reading
USENIX paper: methods; NYU: same study.
