Imagine a child sketching the Eiffel Tower and asking a robot to bring it to life. Today’s robot manipulation systems can’t act on such sketches directly—they require precise 3D block poses as goals, which in turn demand structural analysis and expert tools like CAD. We present StackItUp, a system that enables non-experts to specify complex 3D structures using only 2D front-view hand-drawn sketches. StackItUp introduces an abstract relation graph to bridge the gap between rough sketches and accurate 3D block arrangements, capturing the symbolic geometric relations (e.g., left-of) and stability patterns (e.g., two-pillar-bridge) while discarding noisy metric details from sketches. It then grounds this graph to 3D poses using compositional diffusion models and iteratively updates it by predicting hidden internal and rear supports—critical for stability but absent from the sketch. Evaluated on sketches of iconic landmarks and modern house designs, StackItUp consistently produces stable, multilevel 3D structures and outperforms all baselines in both stability and visual resemblance.
StackItUp uses an abstract relation graph 𝒪 as an intermediate between a rough 2D sketch (left) and the generated 3D block arrangement (right). The graph (middle) encodes high-level geometric and stability relations while abstracting away exact poses. StackItUp first extracts 𝒪0 from visible blocks in the sketch (blue nodes), then iteratively grounds it to 3D poses using compositional diffusion models. If instability is detected, the graph is updated with predicted hidden supports (green nodes), and re-grounded to 3D poses.
In the forward step (left), given a graph 𝒪t, compositional diffusion models generate 3D block poses. In the backward step (center-left), 𝒪t is decomposed into subgraphs based on stability patterns, and each is checked via physics simulation. If unstable, new support blocks (green nodes) and relations are added. These updates are aggregated into an updated graph 𝒪t+1 for re-grounding. The right column shows the 3D structure before and after the graph update, with an added support ensuring stability.
Top row: sketches where hidden supports are predicted for stability. Bottom row: sketches that require no hidden supports.
StackItUp consistently produces structures that are both visually faithful and physically stable.
StackItUp adapts the 3D poses to different block geometries. Given a sketch, it adjusts poses to accommodate geometric variations under the same 𝒪 (a). If the structure is stable through pose adjustment alone, StackItUp extends 𝒪 with hidden objects and re-grounds it to new poses (b).
@inproceedings{xu2025stackup3dstable,
title={"Stack It Up!": 3D Stable Structure Generation from 2D Hand-drawn Sketch},
author={Yiqing Xu and Linfeng Li and Cunjun Yu and David Hsu},
year={2025},
booktitle={Conference on Robot Learning (CoRL)},
}