Photo by Andrea Piacquadio on Pexels

The world of AI-generated video is rapidly evolving, blurring the lines between reality and synthetic creation. Companies like OpenAI (Sora), Google DeepMind (Veo 3), and Runway (Gen-4) are spearheading this revolution, producing video clips that are increasingly difficult to distinguish from real-world footage or high-end CGI. Even entertainment giants like Netflix are embracing AI, using AI-driven visual effects in productions such as ‘The Eternaut.’

As access expands with models like Sora and Veo 3 now available in ChatGPT and Gemini for subscribers, more creators can harness the power of AI video. This accessibility, however, introduces complexities. Creators face new competition, and the internet is increasingly susceptible to the spread of misinformation through AI-generated fake news footage. Another significant hurdle is the substantial energy consumption associated with video generation compared to text or image creation.

At the heart of this technology are ‘latent diffusion transformers.’ Diffusion models, trained to reverse the pixelation process, form the foundation. These are combined with large language models (LLMs) and trained on massive text and image datasets. For video, diffusion models clean up sequences of images, or video frames. Latent diffusion enhances the process by compressing video into a mathematical code, focusing on key features. Transformers maintain consistency across frames, ensuring continuity of objects and lighting. Sora uses a diffusion model paired with a transformer to process videos in spatial and temporal chunks.

Google DeepMind’s Veo 3 stands out with its ability to generate synchronized audio and video by compressing both into a single data point within the diffusion model.

While diffusion models are predominantly used for media like images, video, and audio, and transformers dominate text generation in LLMs, these lines are blurring. Google DeepMind is exploring diffusion models for text, suggesting a potential path toward more energy-efficient LLMs in the future, given the efficiency of diffusion models.