arXiv:2610.41420v1 Announce Type: cross
Abstract: Gated Delta Networks (GDNs) offer linear-time sequence modeling, yet long-context prefill remains costly due to dense intra-chunk computation, recurrent-state updates, and intermediate state storage. We introduce SparseGDN, a hardware-aware framework that combines structured intra-chunk approximation with input-dependent recurrent-event selection while producing outputs for every token. A block-adjacent solver reduces local Delta-solve computation, while a dynamic selector allocates recurrent-update budgets across head pairs. The GPU implementation materializes updates only for selected events and reconstructs token outputs using compact state snapshots and group-local interactions, reducing computation and memory traffic. We evaluate SparseGDN on Qwen3.5-4B and Qwen3.5-9B across RULER tasks and context lengths from 4K to 128K. Retaining approximately 29% of recurrent events on average, SparseGDN achieves 1.50-1.94x GDN-core prefill speedups over native FLA kernels, including online selection overhead.
arxiv
SparseGDN: Chunk Sparse in Linear Attention
SparseGDN introduces a hardware-aware framework for Gated Delta Networks that reduces computation and memory traffic by combining structured intra-chunk approximation with input-dependent recurrent-event selection. Evaluated on Qwen3.5 models across context lengths up to 128K, it retains about 29% of recurrent events while achieving 1.50-1.94x prefill speedups over native FLA kernels.
Provides a practical method to accelerate long-context sequence modeling with linear attention, offering significant speedups without sacrificing output quality.
Comments & Questions (0)
No comments yet
Be the first to comment!