We present a novel approach to the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space, which can then be decoded into a volumetric representation for rendering view-consistent appearance and geometry. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and de-normalization operations to learn a 3D diffusion from 2D images or monocular videos of rigid or articulated objects. Our approach is flexible enough to use either existing camera supervision or no camera information at all -- instead efficiently learning it during training. Our evaluations demonstrate that our generation results outperform state-of-the-art alternatives on various benchmark datasets and metrics, including multi-view image datasets of synthetic objects, real in-the-wild videos of moving people, and a large-scale, real video dataset of static objects.
Our proposed two-stage framework: Stage 1 trains an autodecoder with two generative components, G1 and G2. It learns to assign each training set object a 1D embedding that is processed by G1 into a latent volumetric space. G2 decodes these volumes into larger radiance volumes suitable for rendering. Note that we are using only 2D supervision to train the autodecoder. In Stage 2, the autodecoder parameters are frozen. Latent volumes generated by G1 are then used to train the 3D denoising diffusion process. At inference time, G1 is not used, as the generated volume is randomly sampled, denoised, and then decoded by G2 for rendering.
We train an unconditional 3D Diffusion model on the Latent Features of a 3D AutoDecoder trained on Objaverse. After 256 diffusion steps, we upsample the generated latent volume to a 64x64x64 RGB-D grid. We produce and show renders at 128x128 from multiple views.
We train an text-conditioned 3D Diffusion model on the Latent Features of a 3D AutoDecoder trained on Objaverse. Captions were extracted using MiniGPT4. After 256 diffusion steps, we upsample the generated latent volume to a 64x64x64 RGB-D grid. During diffusion we apply classifier-free guidance with weight 3. We produce and show renders from multiple views.
We train an unconditional 3D Diffusion model on the Latent Features of a 3D AutoDecoder trained on MVImgNet. After 256 diffusion steps, we upsample the generated latent volume to a 64x64x64 RGB-D grid. We produce and show renders from multiple views.
We train an text-conditioned 3D Diffusion model on the Latent Features of a 3D AutoDecoder trained on MVImgNet. Captions were extracted using MiniGPT4. After 256 diffusion steps, we upsample the generated latent volume to a 64x64x64 RGB-D grid. During diffusion we apply classifier-free guidance with weight 3. We produce and show renders from multiple views.
We visualize results novel views at -10, 0, and 10 degrees in the left, middle, and right part respectively . We use a real video to drive the articulated motion of the generated faces. No Camera information is provided to the network; it is inferred during training. We use 256 diffusion steps and classifier-free guidance with weight 3.
@misc{ntavelis2023autodecoding,
title={AutoDecoding Latent 3D Diffusion Models},
author={Evangelos Ntavelis and Aliaksandr Siarohin and
Kyle Olszewski and Chaoyang Wang and Luc Van Gool and Sergey Tulyakov},
year={2023},
eprint={2307.05445},
archivePrefix={arXiv},
primaryClass={cs.CV}
}