arXiv:2611.24431v1 Announce Type: cross
Abstract: Vision-language models (VLMs) can answer questions about long interleaved inputs, such as documents of many pages and collections of hundreds of images, but they are slow to start: on inputs of about 100K tokens, the prefill takes most of the wait for the first output token. A common approach prunes the input by scoring its tokens with the target model itself or with a small model of the same family that first writes an answer. However, this scoring takes much of the time that pruning saves. We find that the evidence for a question lies in a few whole units (images or paragraphs), and that a small hybrid model finds them with one attention head of an early layer, even at half image resolution. Based on these findings, we propose SkimPrefill, which lets the small hybrid model read the whole input first, up to that early layer, with images at half resolution. Most layers of the small hybrid model (Qwen3.5-0.8B) run in linear time, so this pass is short. SkimPrefill then ranks every unit with that head and gives the target the top units as an ordinary shorter prompt. One untrained selector serves targets of different model families. Extensive experiments on MMLongBench at 128K tokens, with 19 datasets of five task types, show the effectiveness of SkimPrefill. With 10% of the input tokens kept, SkimPrefill speeds up the prefill of Qwen3.5-9B, Gemma-4-12B and Qwen3-VL-32B by 3.7 times, 11.6 time and 8.2 times while maintaining their average accuracy.
arxiv
SkimPrefill Uses a Small Model to Speed Up Long Multimodal Prefill
Researchers propose SkimPrefill, a method that lets a small hybrid model (Qwen3.5-0.8B) read a long interleaved image-text input first, with images at half resolution, and rank whole units using an early-layer attention head. The target model then receives only the top-ranked units as a shorter prompt. With 10% of input tokens kept, the method reports prefill speedups of 3.7x, 11.6x and 8.2x for Qwen3.5-9B, Gemma-4-12B and Qwen3-VL-32B respectively while maintaining average accuracy on MMLongBench at 128K tokens.
It offers a concrete inference-efficiency technique for reducing the long-input prefill bottleneck in vision-language models without retraining the target model.
Comments & Questions (0)
No comments yet
Be the first to comment!