# How BlenderWorld evaluates editable 3D scenes made by AI agents

> QwenLM's BlenderWorld benchmark asks agents to submit native Blender projects. Its task rubrics and visual judges test different aspects of a 3D scene, with limits for real production use.

By BIG CHANGE Editorial

Published: 2026-10-17T10:34:13.532Z
Updated: 2026-10-17T10:34:13.532Z
Canonical: https://bigchange.ai/blog/blenderworld-editable-3d-scenes-ai-evaluation

![Conceptual charcoal illustration of a cutaway miniature room containing a table and orange chair, with the detached roof resting beside it.](https://bigchange.ai/api/media/file/blenderworld-cutaway-room-concept-v1.png)
AI-generated conceptual illustration by BIG CHANGE; physical model analogy, not a BlenderWorld submission or benchmark result.

QwenLM's [BlenderWorld benchmark](https://github.com/QwenLM/BlenderWorld) asks AI agents to deliver editable 3D scenes in Blender. Its 223 tasks start with text briefs, image references or video references. The submission centers on a native `final.blend` project, with any requested images, videos and texture files. That lets an evaluator inspect the scene as well as the pictures made from it.

Released on October 14 and updated with a paper revision on October 16, the public repository provides task data, submission rules and an evaluation runner. For artists and teams trying an agent on Blender work, the useful advance is a way to ask whether its output satisfies a brief inside the project file. BIG CHANGE reviewed the documentation and project page; we did not run the benchmark or inspect the submitted scenes ourselves.

## The big change

- **What changed:** BlenderWorld packages 223 creation requests and 2,651 task-specific binary criteria with a public evaluator for native Blender submissions.
- **Why it matters:** A rendered image can hide a missing object, incorrect dimensions or an unusable project. The benchmark checks requirements with scene inspection and visual evidence, alongside a separate visual-quality score.
- **What to watch:** Independent runs using the published tasks, code and model settings would show how reproducible the authors' scores are. A team's own review of saved projects remains necessary before using them in production.

## From brief to editable file

The [dataset documentation](https://github.com/QwenLM/BlenderWorld/blob/9c911d848a30ff5e169a4af7dbe239ceef0299cc/docs/dataset.md) defines 78 text tasks, 130 image-plus-text tasks and 15 video-plus-text tasks. Fifty-nine also have specification attachments that can set dimensions, materials or camera requirements. Each task has an ID, the request and any references; its rubric is stored separately. The generation agent is not shown the evaluation rubric while making the scene.

The [repository quickstart](https://github.com/QwenLM/BlenderWorld/tree/9c911d848a30ff5e169a4af7dbe239ceef0299cc) documents `bw prepare` for staging a task's brief, references and attachments. An agent then works in Blender and saves `final.blend`, together with the exports and dependencies the brief calls for. `bw pack-submission` creates one ZIP per task. The evaluator opens the Blender file as the primary project; it does not treat a polished final render as a substitute for that file.

This is a useful distinction when assessing an AI 3D assistant. A still image may communicate the intended look, while the editable scene carries geometry, materials, layout and animation that another artist might need to inspect or revise. BlenderWorld's format makes both kinds of evidence available, though a valid file alone does not establish that every part of a professional workflow is ready to use.

## How the scoring works

According to the [evaluation protocol](https://github.com/QwenLM/BlenderWorld/blob/9c911d848a30ff5e169a4af7dbe239ceef0299cc/docs/evaluation.md), a rendering agent selects eight views from the submitted scene. Scene inspection handles criteria that need access to the project; a vision judge handles visual criteria using renders and other visual evidence. Each task has binary pass conditions weighted by importance. Eight separate ratings cover scene fidelity, physical plausibility, style and color, realism, spatial layout, proportion and scale, lighting and materials, and completeness.

The final task score gives half its weight to the binary rubric and half to the mean visual-dimension rating. Missing or failed submissions receive zero in the aggregate, rather than disappearing from its denominator. The [project site](https://blenderworld.github.io/) reports scores for 13 models, but those are the authors' results under their chosen tasks, generation setups and automatic judges. The site also presents 181 scene previews from seven models across 26 tasks as curated examples, explicitly warning that they are not a random quality sample.

The benchmark therefore offers more diagnostic detail than a single image-preference score: a result can fail a structural requirement, a visual requirement or a broader quality dimension. It cannot by itself establish that a model's 3D output meets a particular studio's editability standards, export pipeline or reliability needs. The published human-preference comparison evaluates agreement with its automatic judge on selected pairs; it is not a certification of downstream usability.

## What a reproducible run requires

The public instructions specify Linux with Python 3.10 or later and Blender 4.2.10 for preparing new submissions. Evaluation adds Node.js 22.17.0, `ffmpeg` and `ffprobe`, Claude Code 2.1.220 for rendering and scene inspection, and a configured vision-capable model endpoint. The repository provides `bw verify` to check dataset resources, `bw evaluate --dry-run` to inspect a planned run, and a resumable evaluator that writes per-task evidence and summary files. These are documented commands, not steps BIG CHANGE tested.

The code is MIT-licensed. Running the evaluation still needs the listed software, suitable compute and credentials for the configured model services; the documentation does not establish a fixed per-run price. A reader comparing agents should keep the task set, Blender version, evaluator configuration and model aliases fixed, then examine both the aggregate and the failed tasks. A plausible picture and a high mean score answer narrower questions than whether a team can revise and ship the actual scene.

## Sources & further reading

- [BlenderWorld repository and quickstart](https://github.com/QwenLM/BlenderWorld/tree/9c911d848a30ff5e169a4af7dbe239ceef0299cc). QwenLM, pinned commit `9c911d848a30ff5e169a4af7dbe239ceef0299cc`, October 16, 2026. Documents task counts, submission files, commands and the score formula.
- [BlenderWorld dataset format](https://github.com/QwenLM/BlenderWorld/blob/9c911d848a30ff5e169a4af7dbe239ceef0299cc/docs/dataset.md). Same pinned revision. Defines task records, references, attachments and rubric fields.
- [Evaluation protocol and runtime](https://github.com/QwenLM/BlenderWorld/blob/9c911d848a30ff5e169a4af7dbe239ceef0299cc/docs/evaluation.md). Same pinned revision. Specifies environment, evidence routing, failure handling and scoring.
- [BlenderWorld project and results site](https://blenderworld.github.io/). Authors' live project page, checked October 17, 2026. Explains the benchmark and labels its displayed scenes as curated examples; its scores are not an independent replication.

## Sources

- [BlenderWorld repository and quickstart, pinned October 16 revision](https://github.com/QwenLM/BlenderWorld/tree/9c911d848a30ff5e169a4af7dbe239ceef0299cc) — The authors' pinned repository describes task counts, required Blender deliverables, preparation, packaging and evaluation commands. BIG CHANGE reviewed the documentation but did not run the benchmark.
- [BlenderWorld dataset format, pinned October 16 revision](https://github.com/QwenLM/BlenderWorld/blob/9c911d848a30ff5e169a4af7dbe239ceef0299cc/docs/dataset.md) — Defines task records, references, attachments and rubric criteria in the authors' dataset. It does not independently validate scene quality.
- [BlenderWorld evaluation protocol and runtime, pinned October 16 revision](https://github.com/QwenLM/BlenderWorld/blob/9c911d848a30ff5e169a4af7dbe239ceef0299cc/docs/evaluation.md) — Specifies evaluator software, render and scene evidence, judge routing, failure handling and scoring; no BIG CHANGE run or independent replication.
- [BlenderWorld project and results site](https://blenderworld.github.io/) — Authors' live project page reports model scores and shows curated scene previews. The display is not a random sample or independent replication.