Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach

Yunuo Chen1,2*, Junli Cao1,2, Vidit Goel2, Sergei Korolev2, Chenfanfu Jiang1, Jian Ren2 Sergey Tulyakov2, Anil Kag2,
1University of California, Los Angeles, 2Snap Inc.
*Work done during internship at Snap
arXiv

Abstract

We present a novel video generation framework that integrates 3-dimensional geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D point trajectories and align them in pixel space. The resulting 3D-aware video dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling it to track 2D objects with 3D Cartesian coordinates. Building on this, we regularize the shape and motion of objects in the video to eliminate undesired artifacts, e.g., non-physical deformation. Consequently, we enhance the quality of generated RGB videos and alleviate common issues like object morphing, which are prevalent in current video models due to a lack of shape awareness. With our 3D augmentation and regularization, our model is capable of handling contact-rich scenarios such as task-oriented videos, where 3D information is essential for perceiving shape and motion of interacting solids. Our method can be seamlessly integrated into existing video diffusion models to improve their visual plausibility.

Pipeline

Method Overview. During training, we sample video-point pairs, concatenate them along the channel dimensions, and use the augmented data to train a latent diffusion model. We introduce cross-attention between video and point data in corresponding channels to enhance alignment between the two modalities. The model predicts both RGB video and 3D points, leveraging the 3D information to further regularize video generation by applying a misalignment penalty during the diffusion process. During inference, the model generates both video and points from random noise, conditioned on a text-image prompt.

BibTeX

@article{chen2025towards,
      title={Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach},
      author={Yunuo Chen and Junli Cao and Vidit Goel and Sergei Korolev and Chenfanfu Jiang and Jian Ren and Sergey Tulyakov and Anil Kag},
      journal={arXiv preprint arXiv:2502.03639},
      year={2025},
}