Back to AI News
arxiv

AURA 2.0 Adds Model-Initiated Conversations to Streaming Video LLMs

AURA 2.0 is a streaming video large language model that introduces Model-Initiated Conversation, allowing it to identify discussion-worthy topics in an evolving video stream and start interactions without a user query. It also adds Stream-Aware Tool Calling, Interactive Visual Grounding for user-marked objects and frame-aligned bounding boxes, and Long-Horizon Interaction Memory for contextual continuity. The model was trained with a dedicated streaming interaction dataset and a three-stage training scheme. The authors report leading performance among open-source streaming models on public benchmarks and open-source the model weights and inference code.

AI Brief

It documents a concrete open-source research advance toward proactive, tool-using video assistants rather than purely prompt-driven interaction.

arXiv:2611.17048v1 Announce Type: cross Abstract: Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) enables the model to invoke external tools using the current visual and conversational context and incorporate returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities. We open-source the model weights and inference code.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!