Experts and experienced forecasters have repeatedly set AI benchmark targets below the results that followed, according to a September 22 review by the Forecasting Research Institute. The institute's comparisons also show why that pattern cannot settle questions about what AI will do in workplaces or society. A laboratory trial found less help from language models than forecasters expected, and most forecasts of economic and social effects have yet to reach their deadlines.
The big change
- What changed: Several measurable AI capabilities advanced faster than the median forecasts collected before the results. The review includes both completed questions and interim checks; it does not establish a single error rate for all AI forecasts.
- Why it matters: A researcher or policymaker using a benchmark date to plan evaluations can see where earlier expectations proved too slow. Decisions about practical use need separate evidence: one controlled biology task produced the opposite forecasting error, and broad impact predictions remain largely unresolved.
- What to watch: As later questions reach their deadlines, the record can reveal overestimates that an early progress check cannot. Results from tasks and adoption measures will show whether benchmark surprises carry over into actual use.
The measured misses
The Forecasting Research Institute (FRI) drew on forecasting projects run from mid-2022 through August 2026. Its September review compares mostly the median predictions of domain experts and superforecasters with resolved results, observed values before a deadline, and some model-generated projections. The expert pool changes between projects; the first Longitudinal Expert AI Panel report, for example, summarizes responses from 330 experts and 59 superforecasters collected from late June to mid-August 2025. These are selected questions across different studies, rather than a controlled contest with one fixed group.
In that panel, the median expert forecast that the best model would reach 31% accuracy on FrontierMath's first three tiers by the end of 2025; the median superforecaster said 30%. FRI records a 40.7% result at the January 1, 2026 resolution point. There is a complication: benchmark operator Epoch subsequently reported errors in about 42% of FrontierMath problems and adjusted its scoring. FRI uses the score that matched the rules forecasters saw at the time, while acknowledging that the best resolution value is debatable.
The newer LiveCodeBench Pro forecast is an even wider interim gap. The median expert predicted 14% on its hard coding problems by the end of 2026, and the median superforecaster predicted 12%. FRI says the leading score had reached 53.8% by May. The question asks for the highest score ever achieved by the deadline, so an already recorded score crosses both medians even though the end-of-year question remains formally open. FRI also notes that some respondents may not have understood that models could test on questions from earlier quarters. That wording issue limits how confidently the gap can be attributed to an underlying belief about AI progress.
A task result ran the other way
FRI's February account of an Active Site randomized trial provides a different kind of check. Before the result, 30 superforecasters, 30 biosecurity experts and 17 virology experts predicted how often novices would complete three laboratory tasks. The eight-week trial enrolled 153 participants. In the group with internet and access to mid-2025 language models, 5.2% completed all three tasks, against a median prediction of 27% across the forecasters. The internet-only group completed them at 6.6%, against a 12% median prediction. The difference between the two trial groups was not statistically significant.
That result concerns a specific set of wet-lab tasks and the models available in summer 2025. It shows the danger of transferring a benchmark trend directly to an assisted human task. It does not measure every biological use of AI or establish what newer models would do in the same experiment.
Revenue is a reported pace, not a final result
FRI also compares forecasts with AI company revenue reports. In its economic-effects study, median forecasts for the largest AI company's annual recurring revenue at the end of 2026 were $20 billion from AI experts, $16 billion from economists and $25 billion from superforecasters. FRI says September reporting put Anthropic's annualized revenue pace above $100 billion. Its separate summer 2026 panel forecasts for the combined OpenAI and Anthropic year-end run rate were $70 billion from experts and $90 billion from superforecasters, below a reported current sum of $140 billion.
Those comparisons are signals, not resolved forecast scores. The revenue figures are reported annualized paces rather than independently audited full-year revenue, and the year-end questions have not closed. FRI further says its survey design may have led some respondents to overlook a revenue figure that was already public when they forecast. The article therefore cannot tell how much of that miss reflects an unexpected business change.
Why the wider verdict must wait
FRI selected illustrative questions from a larger set and emphasized those with results or usable interim data. An early check reveals a forecast that is too low as soon as a benchmark passes it; a forecast that is too high generally cannot be confirmed until its deadline. The institute says this gives its current review a built-in tilt toward underestimation. Its GPT-5.5 projections for some unresolved questions were also produced in August 2026, months after the human forecasts, with newer information available.
The distinction matters most for claims about consequences. FRI says too few forecasts about employment, growth, catastrophic risks and benefits have resolved to judge accuracy. Some adoption measures are mixed or uncertain. The evidence supports revisiting timelines for particular benchmark evaluations. It does not yet show that faster scores will bring faster economic change, or that experts as a group are poor forecasters of AI's social effects.
