In two randomized fraction studies, people given a chat assistant solved more problems while it was available, then solved fewer of the final problems on their own than people who had worked without it. The latest paper by Grace Liu and colleagues also reports a reading-comprehension experiment with a similar immediate result. Its measure of persistence was whether participants skipped a problem. It did not track their learning or behavior after the session.

The big change

  • What changed: Random assignment let the researchers compare brief AI assistance with unaided practice on the same final problems. The AI group’s accuracy advantage during assisted work reversed when the tool was removed.
  • Why it matters: A correct answer obtained with help and the ability to answer the next question alone are different outcomes. The studies give educators and AI designers a reason to measure both.
  • What remains unknown: The experiments ended after an immediate unaided test. They cannot establish lasting loss of skill, a general decline in focus, or the effects of routine AI use over months.

What the two fraction experiments measured

The researchers recruited U.S. participants through the online platform Prolific. In the first experiment, 354 people were randomly assigned to solve fraction problems with either a GPT-5 chat sidebar for the first 12 questions or no AI throughout. Everyone then faced the same three final questions without AI. The assistant had been given each problem and its solution, so a participant could ask directly for an answer. Participants could also skip a question, and an incorrect answer carried no payment penalty. Both groups saw the correct solution after submitting a wrong answer.

After attention and performance exclusions, the analysis included 307 people. On the three unaided questions, the earlier AI group solved an average of 57% correctly, compared with 73% for the control group. It skipped 20% of questions, compared with 11%. The reported standardized effects were −0.42 for solving, with a 95% confidence interval from −0.66 to −0.19, and 0.25 for skipping, with an interval from 0.02 to 0.48.

The first exclusion rule created a problem for interpreting that comparison. It removed people who solved too few early questions, but someone using AI could pass that threshold by submitting AI answers. The final groups were also uneven: 185 in the AI group and 122 in the control group.

The second experiment recruited 667 people and tested basic fraction ability before anyone received assistance. During the main phase, the AI group had the chat sidebar for 11 questions. The control group had a sidebar showing solutions already revealed in the pretest. Both sidebars disappeared before the same three unaided final questions. This design addressed the first study’s skill-selection concern and made the interface change more comparable.

After exclusions based on the pretest and an attention check, 585 people remained: 308 in the AI group and 277 controls. The earlier AI group solved 71% of final questions, versus 77% for controls. The standardized difference was −0.19, with a 95% confidence interval from −0.36 to −0.03. Its 10% skip rate exceeded the control group’s 7%, but that overall difference was statistically uncertain: the reported interval for the standardized effect ran from −0.07 to 0.26. This replication supports a smaller immediate accuracy difference; it does not establish an overall persistence difference on its own.

The researchers also compared how people within the AI group said they used the assistant. Of 308 analyzed AI participants, 189 reported primarily getting direct answers, 82 reported seeking hints or clarification, and 37 reported no AI use. The direct-answer group had lower final accuracy and a higher skip rate than controls and the hint group. Those usage categories were chosen by participants, not randomly assigned. Differences between them cannot show that asking for answers itself caused the gap.

The reading result and the limits of the claim

A third experiment recruited 201 people for SAT-style reading questions. The AI group could consult a chat sidebar for five questions; controls had a sidebar with general reading tips. Both groups then answered three questions without the sidebar. The final analysis included 168 participants. The earlier AI group answered 76% of the final questions correctly, compared with 89% for controls, and skipped 8% versus 1%. The paper reports P=0.007 for accuracy and P=0.008 for skipping. For this experiment, an answer submitted in under five seconds also counted as a skip.

The randomized assignments support conclusions about these short tasks and the immediate phase after assistance ended. They do not identify a lasting change in ability or motivation. The authors say they cannot tell whether the effects persist for hours or days, accumulate with daily use, or fade. Their assistant was set up to supply complete answers on request; the experiments did not test whether a different tutoring design would produce the same result.

The October 9 Berkeley News account describes the work as evidence that brief AI use impairs focus and persistence. The paper measured correct answers and skips, not a separate focus outcome. Its second fraction experiment found an uncertain overall skip-rate difference. Berkeley also quotes a researcher drawing implications for expertise across education and work. Those are concerns for future study, beyond what these sessions measured.

Sources & further reading