When a new AI model is released, a claim that it was "evaluated" tells users little about what the tests found. In an October 8 analysis, SemiAnalysis counted 857 releases by nine China-based developers from 2021 through September 15, 2026. It found a published safety result for 31 releases (3.6%). For just nine (1.1%), a result was available at or before the model's release.

The figures describe what the researchers could find in public developer materials. They do not show how many models were tested privately or whether any particular model is safe.

The big change

  • What changed: SemiAnalysis has put a number on a documentation gap across a defined set of releases: nine of 857 had a model-specific safety result people could read by launch day.
  • Why it matters: Someone assessing a model at release needs results for that version. A generic assurance of testing, or a result published after release, cannot answer the same question at that moment.
  • What to watch: For subsequent releases, the useful signal is a dated result tied to the exact model and available when users first get access. The audit's disclosure rate can change as documents appear, but later publication does not change what was available on launch day.

What the audit counted

SemiAnalysis says its census covered 741 product-model releases and 116 research-model releases from Alibaba, ByteDance, Tencent, Baidu, DeepSeek, Moonshot, Zhipu (Z.ai), MiniMax and StepFun. It gave each release a first-public date, then checked developer model cards, release notes and technical reports for a safety result it could match to that release.

A qualifying result had to give a quantitative or substantive finding about a named model. The subjects included harmful output, resistance to jailbreaks, toxicity, privacy, refusal behavior and dangerous capabilities. A statement that a model was safety-trained or evaluated, without a finding, did not qualify. Nor did the researchers extend a result for one flagship model to other sizes or later snapshots.

That strict matching rule makes the denominator important. The 857 entries are counted releases, including variants, rather than 857 equally capable frontier systems. SemiAnalysis notes that developers name variants at different levels of detail. Its count is therefore a way to examine documentation across this set of releases, not a sound basis for ranking individual companies.

When results appeared

Of the 31 releases with a qualifying published result, nine had one at or before release. Sixteen were documented later; SemiAnalysis reports a median delay of 42 days for that group. For the other six, the researchers could not establish the timing or the match to a specific model. Those six belong in the 31 with some published result, but they cannot be counted as launch-day disclosures.

The remaining cases also need careful wording. SemiAnalysis found 10 releases with a claim of evaluation but no figures, and three known only through press or investor accounts for which it could not retrieve developer documentation. It found no safety disclosure for 813 releases in the materials it checked. Together, those groups and the 31 with results make up the 857 releases.

The Reuters report on the analysis also stresses that companies may have performed private tests. "No disclosure found" is a statement about the public record examined here. It cannot establish that testing did not happen, and a published test does not by itself establish the model's underlying safety. SemiAnalysis did not provide a comparable release-level rate for US developers in this audit.

For a reader choosing or assessing a model, the practical distinction is simple: identify the exact version, look for a dated, model-specific result, and check whether that result was available when the model was released. That evidence says what the developer made visible. It is only one part of evaluating a model for a particular use.

Sources & further reading