QwenLM's BlenderWorld benchmark asks AI agents to deliver editable 3D scenes in Blender. Its 223 tasks start with text briefs, image references or video references. The submission centers on a native final.blend project, with any requested images, videos and texture files. That lets an evaluator inspect the scene as well as the pictures made from it.

Released on October 14 and updated with a paper revision on October 16, the public repository provides task data, submission rules and an evaluation runner. For artists and teams trying an agent on Blender work, the useful advance is a way to ask whether its output satisfies a brief inside the project file. BIG CHANGE reviewed the documentation and project page; we did not run the benchmark or inspect the submitted scenes ourselves.

The big change

  • What changed: BlenderWorld packages 223 creation requests and 2,651 task-specific binary criteria with a public evaluator for native Blender submissions.
  • Why it matters: A rendered image can hide a missing object, incorrect dimensions or an unusable project. The benchmark checks requirements with scene inspection and visual evidence, alongside a separate visual-quality score.
  • What to watch: Independent runs using the published tasks, code and model settings would show how reproducible the authors' scores are. A team's own review of saved projects remains necessary before using them in production.

From brief to editable file

The dataset documentation defines 78 text tasks, 130 image-plus-text tasks and 15 video-plus-text tasks. Fifty-nine also have specification attachments that can set dimensions, materials or camera requirements. Each task has an ID, the request and any references; its rubric is stored separately. The generation agent is not shown the evaluation rubric while making the scene.

The repository quickstart documents bw prepare for staging a task's brief, references and attachments. An agent then works in Blender and saves final.blend, together with the exports and dependencies the brief calls for. bw pack-submission creates one ZIP per task. The evaluator opens the Blender file as the primary project; it does not treat a polished final render as a substitute for that file.

This is a useful distinction when assessing an AI 3D assistant. A still image may communicate the intended look, while the editable scene carries geometry, materials, layout and animation that another artist might need to inspect or revise. BlenderWorld's format makes both kinds of evidence available, though a valid file alone does not establish that every part of a professional workflow is ready to use.

How the scoring works

According to the evaluation protocol, a rendering agent selects eight views from the submitted scene. Scene inspection handles criteria that need access to the project; a vision judge handles visual criteria using renders and other visual evidence. Each task has binary pass conditions weighted by importance. Eight separate ratings cover scene fidelity, physical plausibility, style and color, realism, spatial layout, proportion and scale, lighting and materials, and completeness.

The final task score gives half its weight to the binary rubric and half to the mean visual-dimension rating. Missing or failed submissions receive zero in the aggregate, rather than disappearing from its denominator. The project site reports scores for 13 models, but those are the authors' results under their chosen tasks, generation setups and automatic judges. The site also presents 181 scene previews from seven models across 26 tasks as curated examples, explicitly warning that they are not a random quality sample.

The benchmark therefore offers more diagnostic detail than a single image-preference score: a result can fail a structural requirement, a visual requirement or a broader quality dimension. It cannot by itself establish that a model's 3D output meets a particular studio's editability standards, export pipeline or reliability needs. The published human-preference comparison evaluates agreement with its automatic judge on selected pairs; it is not a certification of downstream usability.

What a reproducible run requires

The public instructions specify Linux with Python 3.10 or later and Blender 4.2.10 for preparing new submissions. Evaluation adds Node.js 22.17.0, ffmpeg and ffprobe, Claude Code 2.1.220 for rendering and scene inspection, and a configured vision-capable model endpoint. The repository provides bw verify to check dataset resources, bw evaluate --dry-run to inspect a planned run, and a resumable evaluator that writes per-task evidence and summary files. These are documented commands, not steps BIG CHANGE tested.

The code is MIT-licensed. Running the evaluation still needs the listed software, suitable compute and credentials for the configured model services; the documentation does not establish a fixed per-run price. A reader comparing agents should keep the task set, Blender version, evaluator configuration and model aliases fixed, then examine both the aggregate and the failed tasks. A plausible picture and a high mean score answer narrower questions than whether a team can revise and ship the actual scene.

Sources & further reading