Shape from Semantics 3D Shape Generation from Multi-View Semantics

Liangchen Li1     Caoliwen Wang1     Yuqi Zhou1     Bailin Deng2     Juyong Zhang1
1University of Science and Technology of China 2Cardiff University

We propose and address Shape from Semantics, a novel generative problem.

Given a set of semantics and corresponding views as input, our method can produce high-quality shapes that exhibit geometry and appearance consistent with the semantics from each view and are feasible for real-world fabrication.

Abstract

We propose "Shape from Semantics", which is able to create 3D models whose geometry and appearance match given semantics when observed from different views. Traditional "Shape from X" tasks usually use visual input (e.g., RGB images or depth maps) to reconstruct geometry, imposing strict constraints that limit creative explorations. As applications, works like Shadow Art and Wire Art often struggle to grasp the embedded semantics of their design through direct observation and rely heavily on specific setups for proper display. To address these limitations, our framework uses semantics as input, greatly expanding the design space to create objects that integrate multiple semantic elements and are easily discernible by observers. Considering that this task requires a rich imagination, we adopt various generative models and structure-to-detail pipelines. Specifically, we adopt multi-semantics Score Distillation Sampling (SDS) to distill 3D geometry and appearance from 2D diffusion models, ensuring that the initial shape is consistent with the semantic input. We then use image restoration and video generation models to add more details as supervision. Finally, we introduce neural signed distance field (SDF) representation to achieve detailed shape reconstruction. Our framework generates meshes with complex details, well-structured geometry, coherent textures, and smooth transitions, resulting in visually appealing and eye-catching designs.

🎯Novel Problem Definition

Methods

For given semantics, we first adopt a 2D generative model to supervise and design a rough geometric shape, and then use diffusion models for image restoration and video generation to generate more high-quality images. Finally, we use SDF representation to reconstruct the detailed shape under the supervision of generated images and Gaussian masks. Our shape is fabricable and can be produced by 3D printing.

Shape Design based on Multi-Semantics SDS

In the first stage, we obtain the target structure based on multi-semantics SDS and use 3D Gaussian Splatting as the 3D representation.

Generation of Detailed and Multi-View Supervision

In the second stage, we use various generative models to provide supervision for the subsequent neural SDF representation based on neural radiance supervision.

Reconstruction of Refined Shapes with Neural SDF

In the final stage, we use neural SDF as the representation and train based on neural radiance using the masks and RGB supervision obtained from the first and second stages. This results in a geometry with exceptionally high geometric quality and exquisite textures.

Results

Comparison Results

We compare the outcomes of Shape from Semantics with those of Fabricable Wire Art.

More Results

We demonstrate more generated cases.

Fabrication Results

We demonstrate various fabrication results of 3D printing.

BibTeX

If you find StructuredField useful for your work please cite: