We propose "Shape from Semantics", which is able to create 3D models whose geometry and appearance match given semantics when observed from different views. Traditional "Shape from X" tasks usually use visual input (e.g., RGB images or depth maps) to reconstruct geometry, imposing strict constraints that limit creative explorations. As applications, works like Shadow Art and Wire Art often struggle to grasp the embedded semantics of their design through direct observation and rely heavily on specific setups for proper display. To address these limitations, our framework uses semantics as input, greatly expanding the design space to create objects that integrate multiple semantic elements and are easily discernible by observers. Considering that this task requires a rich imagination, we adopt various generative models and structure-to-detail pipelines. Specifically, we adopt multi-semantics Score Distillation Sampling (SDS) to distill 3D geometry and appearance from 2D diffusion models, ensuring that the initial shape is consistent with the semantic input. We then use image restoration and video generation models to add more details as supervision. Finally, we introduce neural signed distance field (SDF) representation to achieve detailed shape reconstruction. Our framework generates meshes with complex details, well-structured geometry, coherent textures, and smooth transitions, resulting in visually appealing and eye-catching designs.
In the first stage, we obtain the target structure based on multi-semantics SDS and use 3D Gaussian Splatting as the 3D representation.
In the second stage, we use various generative models to provide supervision for the subsequent neural SDF representation based on neural radiance supervision.
In the final stage, we use neural SDF as the representation and train based on neural radiance using the masks and RGB supervision obtained from the first and second stages. This results in a geometry with exceptionally high geometric quality and exquisite textures.
We compare the outcomes of Shape from Semantics with those of Fabricable Wire Art.
We demonstrate more generated cases.
We demonstrate various fabrication results of 3D printing.
If you find StructuredField useful for your work please cite: