BrickBench: Evaluating Agentic Brick Design

Peter Kulits^1,2 Yiqing Xu¹ R. Kenny Jones¹ Cordelia Schmid³ Jiajun Wu¹

¹Stanford University ²Max Planck Institute for Intelligent Systems ³Inria

arXiv

Code

PyPI

Gallery

We evaluate coding agents on their ability to design LEGO assemblies from a text prompt across three settings with different build constraints: Model (at most 400 parts), Set (400–4000 parts), and Alt-Build (the part inventory of set 10698). Each of the assemblies shown below is composed of actual LEGO parts and can be physically built. Click on a build to view it in the gallery and rotate it.

GPT-6 Astra's build: Pelican riding a bicycle

Claude Opus 5.5's build: Pelican riding a bicycle

Task-specific ApproachesA general coding agent effectively saturates prior evaluations

BrickNet and BrickGPT finetune language models to autoregressively assemble brick structures. We evaluate GPT-5.6 Luna, a general coding agent given our environment, on the BrickNet validation set.

A small robot: the held-out assembly and three systems' outputs

A minifigure: the held-out assembly and three systems' outputs

Method	PE ↑	SigLIP 2 ↑	VQAScore ↑
BrickGPT	0.157	0.052	0.050
BrickNet-0.6B	0.279	0.603	0.557
BrickNet-1.7B	0.282	0.631	0.593
BrickNet-4B	0.283	0.639	0.615
BrickNet-8B	0.284	0.647	0.608
BrickNet-14B	0.283	0.625	0.602
GT	0.315	0.826	0.748
GPT-5.6 Luna	0.313	0.818	0.732

Sample inputs and outputs from BrickNet’s validation set.

Caption alignment following BrickNet’s evaluation protocol.

Without task-specific training, Luna outperforms both, within 0.02 of the held-out assemblies on each metric. BrickNet evaluates small objects of at most 100 parts, far from the complexity of a full LEGO set. This motivates our design of a new benchmark.

The benchmarkBrickBench: Evaluating Agentic Brick Design

Given a text prompt, an agent must produce an assembly that satisfies semantic and design criteria and that can be physically built. It adds two challenges prior evaluations leave unexplored, physical complexity and design quality. It consists of 300 prompts over three constraint settings:

How we score it

Valid: meets the part requirement, collision-free, and stable.
VQA: the fraction of the prompt’s requirements satisfied, split into Yes/No questions and evaluated by a VLM.
ELO: Bradley–Terry ratings from pairwise VLM judgments, of semantic alignment with the prompt (Align ELO) and of design agnostic of it (Design ELO).
Cost: the list price of the tokens consumed per assembly in USD.

Question graph

The question graph for the calm dog burning prompt above. Questions are colored by type: entity, attribute, or relation. An arrow from one question to another means the second is satisfied only if the judge answers yes to the first. HoverTap a question to trace what it depends on and what depends on it.

The environmentBrickAgent: Agentic LEGO Design

BrickAgent is the environment of BrickBench. It gives agents tools to programmatically construct, inspect, and validate LEGO assemblies, placing parts through the connector system of BrickNet.

ResultsPrompt adherence saturates; headroom lies in design quality

We evaluate eleven frontier agents and two data-driven baselines. Five agents deliver a valid assembly for every prompt. VQA separates them gently and Design ELO much more sharply.

System	Valid	VQA	ELO	Align ELO	Design ELO	Cost ($)

Metrics per setting and in the aggregate (Overall). Higher is better, except for Cost; ELO intervals are 95%. The two data-driven baselines apply only to the Model setting.Tap a column to sort.

(a) ELO against mean assembly cost.

(b) Design ELO against human judgments of design quality, for the nine reference agents.

(a) Rank does not follow price: GPT-6.1 Sol matches Astra on ELO at a fifth of its cost. (b) Design ELO agrees with human raters on 34 of 36 agent pairs (Kendall τ = 0.89).

Environment ablationRemoving the environment sharply reduces physical validity

GPT-6 Astrawith BrickAgent1.00 without0.40

GPT-5.6 Lunawith BrickAgent1.00 without<0.01

Share of valid assemblies over the 300 prompts, with BrickAgent or with only the LDraw library and a primer on format.

System	Valid ↑	Collisions ↓	Components	VQA ↑	ELO ↑
GPT-6 Astra	1.00	0.00	2.21	0.954	1297 ±22
without BrickAgent	0.40	7.19	6.12	0.940	1309 ±24
GPT-5.6 Luna	1.00	0.00	4.60	0.792	898 ±16
without BrickAgent	<0.01	155.51	83.00	0.813	960 ±20

Pooled over the three settings (300 prompts). Components are connected components, each per assembly.

Astra remains valid on 40% of prompts and Luna on one in 300. However, alignment and design do not fall: without the restriction of buildability, Luna makes assemblies that depict the prompt as well and look better, but that cannot be built.

DesignEqually valid, yet differing dramatically in design

The agents vary in their interpretation of the prompts. ClickTap any build to open it in the gallery.

ModelA green dumpster on four small black wheels has its lid pushed open by a tall cluster of red, orange, and yellow flames. A crumpled white sheet protrudes from one corner, and a tipped-over trash can lies beside it.

ModelA cramped mailroom corner has a wall covered in white notices, envelopes, and photographs joined by tangled red strings. A wild-eyed, brown-haired man in a white shirt and loosened striped tie gestures toward the board. Cardboard boxes and overflowing stacks of mail crowd the floor beneath it.

ModelA blue dolphin leaps above a brilliant turquoise sea, its pale belly turned away from the rainbow. A bright rainbow arches behind its back between soft white clouds. The dolphin’s tail curves back toward a little burst of white spray, while gentle blue waves fill the space beneath the rainbow.

no buildMuse Spark 1.3

GLM 5.3 Flash

SetA masked swordsman dressed entirely in black sits opposite a small balding man at a rough stone table on a rocky hillside. Two silver goblets stand between them. Behind the balding man, a blindfolded blonde woman in a red dress sits with her hands bound. The swordsman rests one gloved hand near his goblet.

SetA gigantic blue toy locomotive with a round gray smiling face bursts through the side of a two-story suburban house. The locomotive is nearly as tall as the house, and its red buffer beam projects over the front lawn. Broken wall panels lie below it. Two people beside a small parked car look up at the engine.

SetA blonde woman in a black off-the-shoulder top leans across a restaurant table, shouting and pointing directly at a white cat seated opposite her. A dark-haired woman stands behind her, gripping her shoulders and trying to pull her back. The cat sits upright on a dining chair with its ears splayed and its mouth slightly open, looking back at the pointing woman. A plate of green salad sits directly in front of the cat. Drinking glasses, cutlery, and folded napkins cover the table, with upholstered chairs and a low restaurant partition surrounding the confrontation.

SetA huge, roughly spherical ball is covered in household objects pointing in every direction: a chair, a television, an umbrella, a frying pan, a banana, and several books. A tiny green figure with a long cylindrical head pushes against its base. Ahead of the ball, a traffic cone and a teapot remain on the floor.

Alt-BuildA green sea serpent rises from the water beside a small pirate ship. Its neck bends over the bow, while its tail emerges from the water behind the stern.

Alt-BuildA tiny blue-armored knight braces behind a shield and points a lance at a snail twice his height. The snail has a large spiral shell and two raised eye stalks. They face each other across a grassy patch.

Alt-BuildA steam locomotive crossing a bridge over a river. The bridge has a single arch directly beneath the engine.

The ceilingA floor a simulator can verify, an open ceiling on design quality

Physical validity

Connected? Collision-free? Stable under gravity?

→ pass or fail

Design quality

Clever part usage, shapes, proportions, and details.

→ no definitive pass/fail test

We asked people familiar with LEGO design to tell agent builds from human-designed models of a similar part count, as a sort of "LEGO Turing test."

How often raters took each system’s build for the human-designed one; 0.5 is chance.

Human-designed models from the BrickNet dataset.

Raters consistently found the human build. No agent was taken for human more than one time in five.

What remainsAgents learn to build what works, but not yet what makes a great design

Design quality

Part usage, proportion, and surface detail.

Headroom lies here

Physical validity and prompt adherence

Connections, collisions, stability, and the prompt.

Largely satisfied with BrickAgent

Paper

Abstract

We propose BrickBench, a benchmark for agentic text-conditioned LEGO-set design. Given a prompt, an agent is tasked with producing an assembly that not only satisfies semantic and design criteria, but that can also be physically built. To do so, it must select parts from a discrete library and reason jointly about local and global constraints. We score validity, alignment, and design across three settings that vary in scale and part availability. We provide BrickAgent, an environment for coding agents to construct, inspect, and validate their designs. We find that leading agents largely satisfy verifiable physical and semantic requirements, but fall short of human designs. We release our benchmark and environment at http://www.brickben.ch.

BibTeX

@misc{kulits2026brickbenchevaluatingagenticbrick,
      title={BrickBench: Evaluating Agentic Brick Design},
      author={Peter Kulits and Yiqing Xu and R. Kenny Jones and Cordelia Schmid and Jiajun Wu},
      year={2026},
      eprint={2610.12452},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2610.12452},
}