VideoGPT: Video Generation using VQ-VAE and Transformers

Citations Over Time

Abstract

We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models. Samples and code are available at https://wilson1yan.github.io/videogpt/index.html

Related Papers

• → Initial sampling using multi-fidelity information in simulation optimization of manufacturing systems(2018)1 cited
• → Towards Understanding the Interplay of Generative Artificial Intelligence and the Internet(2023)9 cited
• → Generative Model for Person Re-Identification: A Review(2020)
• → Falsification of Learning-Based Controllers through Multi-Fidelity Bayesian Optimization(2022)
• → TC-VAE: Uncovering Out-of-Distribution Data Generative Factors(2023)