Back to AI News
arxiv

AURA streams audio-driven talking-head video with a rolling KV cache

AURA is an audio-driven video diffusion model that generates video chunk by chunk using a rolling KV cache, so it only needs the current chunk's audio instead of multi-second look-ahead. Two sizes are reported: AURA 1.3B streams 576p at 20.7 fps on a single GPU for a 16 Hz playback rate, while AURA 5B streams 720p at 9.7 fps. A two-stage distillation pipeline aligns a causal student with a bidirectional teacher, then applies Distribution Matching Distillation with backward simulation. The authors report AURA 1.3B runs 23x faster than its teacher and AURA 5B runs 17x faster while matching its TalkVerse teacher at 720p, and they release both models' weights plus 30K synthetic videos and precomputed teacher trajectories.

AI Brief

It offers a concrete low-latency streaming approach for interactive avatars and releases weights and data for reproducibility.

arXiv:2611.26540v1 Announce Type: cross Abstract: Current audio-driven video diffusion models can synthesize realistic talking-head videos but generally require dozens of diffusion denoising steps over multi-second video chunks, making them unsuitable for streaming applications such as voice assistants and virtual avatars, where low-latency and online causal generation are required properties. AURA generates video chunk by chunk with a rolling KV cache, requiring only the audio of the current chunk to start generating, rather than a multi-second audio look-ahead like bidirectional models. It comes in two sizes: AURA 1.3B streams 576p video in real time on a single GPU (20.7 fps for a 16 Hz playback rate), making it suited to interactive avatar applications, and AURA 5B streams 720p video at 9.7 fps. Our two-stage distillation pipeline first aligns the causal student with the teacher, by training it to match the teacher's trajectories with fewer diffusion steps and a chunk-wise sliding window causal attention mask, then applies Distribution Matching Distillation (DMD) with backward simulation to close the remaining quality gap. On standard talking-head benchmarks, AURA 1.3B runs 23x faster than its own bidirectional teacher and reaches a quality comparable to TalkVerse at a lower resolution, and AURA 5B matches its TalkVerse teacher at 720p while running 17x faster. To support reproducibility and further research on streaming avatar generation, we release the weights of both models along with a dataset of 30K synthetic videos and their precomputed teacher trajectories.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!