A new study of Jev follows a question left open by BIG CHANGE’s coverage of its September launch: whether faster, cheaper action choices improve the whole job. Across 4,500 executions, the researchers found cheaper action selection in every tested agent. In two mobile-agent integrations, task completion fell sharply enough to change the economics of a delivered result.

The October 9 preprint is an author-reported experiment, not an independent replication or a BIG CHANGE test. Its five agent families worked in two text-observed environments: ALFWorld’s household tasks and AndroidWorld’s mobile app tasks. The researchers ran each configuration three times on fixed task instances. They replaced the action-selection step with Jev while retaining each agent’s surrounding workflow, including other model calls and environment interaction.

Where completion changed

On ALFWorld, Jev variants stayed close to their original configurations. ReAct completed 96.02% of tasks with Jev, against 97.51% in its baseline; Reflexion completed 98.01%, against 99.75%. The researchers also tested a control that received the same candidate actions as Jev but used a general LLM to choose among them. This matters because giving an agent a candidate list changes the interface as well as the decision model.

The mobile results were less even. T3A’s completion rate fell from 57.47% to 44.25% with Jev, a 13.22 percentage-point gap. FastAgent fell from 50.00% to 31.90%, an 18.10-point gap. The paper’s task-cluster bootstrap intervals for both differences were entirely below zero. AutoDroid declined from 18.68% to 16.38%, but its interval included zero; both versions completed fewer than one in five tasks. These are results for the tested integrations, not estimates for every mobile agent.

A cheaper attempt can cost more per success

Jev reduced mean action-selection cost by 66.1% to 78.4% and decision time by 73.9% to 81.5% across the five families. Total task cost also fell in each family’s observed mean, though by less. Extra reasoning, more actions and time spent interacting with the environment absorbed some of the local saving.

T3A is the clearest example of the difference between paying for an attempt and paying for an outcome. Its mean inference cost per task attempt fell 17.1%. But when the researchers divided the cost of all attempts, including failures, by the number of completed tasks, the mean rose from 68.70 to 73.97 thousandths of a dollar per success. That 7.7% increase has an uncertainty interval spanning zero, so the study does not establish a precise underlying increase. It does show that lower spending per attempt did not guarantee a lower observed delivery cost in this setup.

FastAgent’s average task time was nearly unchanged despite much faster decisions. With fewer completed tasks, time per delivered success rose from 239.4 to 377.8 seconds. The Jev version made 44.1% more decisions, and unlike its baseline, executed one tool per step. That design difference is part of the tested integration and limits how much of the gap can be assigned to Jev alone.

The authors examined selected traces to understand what happened after a bad choice. In one mobile task, the Jev version chose an ambiguously labelled Wi-Fi switch and declared completion before meeting the goal. In Reflexion’s ALFWorld runs, Jev succeeded on more first trials than the baseline but recovered fewer initial failures: 11 of 19, against 25 of 26. Those fractions concern different sets of failed attempts. They illustrate why an agent’s ability to notice and repair an error matters beyond the price of its first decision.

The big change

Jev’s reported decision savings survived at the task level most clearly where action selection dominated the work and completion stayed high. ReAct was cheaper and faster per delivered success. In the mobile integrations, other calls, environment time and lower completion narrowed or reversed parts of that benefit. For builders, the paper makes a concrete measurement case: compare completed tasks and resources across all attempts, then inspect recovery and stopping decisions. A per-call price or latency figure cannot answer whether a workflow delivered more useful work.

The study used text observations, bounded action choices, fixed task instances and three repetitions with a shared seed. Its cost figures use stated model prices and its own accounting rules. Visually grounded agents, longer workflows and new task distributions remain outside this evidence. The paper evaluates Jev in agent workflows; it does not test or validate SubQ Snap.

Sources & further reading