Live intelligence

Latest AI updates

5 updates
Research Editor selected

The Value of Thinking: Learning When to Reason in Hybrid Decision Models

The paper proposes a zero-or-T thinking paradigm in which a single model either answers from implicit computation in one forward pass without generating tokens or performs full autoregressive reasoning when the expected benefit justifies the token cost. It implements this as a hybrid decision model with a frozen backbone and a lightweight self-routing block that estimates the value of thinking against a per-request price. For settings without per-decision labels, such as episodic agentic tasks, the authors introduce Calibrated Decision Policy Optimisation to train the routing block from episode-level returns. Evaluations on MMLU-Pro, AGIEval, GPQA-Diamond, JevBench, BANKING77 and MiniWoB report accuracy and token-cost improvements, including a 72.5% median latency reduction on JevBench and an 80.1% token-cost cut on MiniWoB with HDM-27B.

Why it matters: It presents a concrete routing and reinforcement-learning method for reducing reasoning token costs while preserving accuracy across QA and agentic tasks.

adaptive reasoningLLM inference efficiencyreinforcement learning
Research

Study compares weight quantization against option-order averaging under a fixed inference budget

The paper asks whether lower-bit weight storage degrades the benefit of averaging answer-option reorderings, and which use of a fixed budget is cheaper. Using simulated round-to-nearest quantization on Qwen3-0.6B and Qwen3-1.7B, evaluated on ARC-Easy, ARC-Challenge, MMLU and HellaSwag with 300 items each, the authors report that 6-bit weights lost no accuracy against fp16 while 4-bit raised answer-slot skew. Averaging four cyclic orders recovered +12.6 points at 4 bits for the 0.6B model. In cost-matched pairs measured as weight bytes read times passes, extra orders beat extra bits only at 6 or more bits, leading to a suggested rule of 6-bit weights plus up to four order passes.

quantizationLLM inference efficiencysmall language models
Research

Phrase-Level Speculative Decoding Drafts Multi-Word Units on a Frozen Target

The paper proposes changing the drafting unit for speculative decoding from single target tokens to one- to three-word phrases stored in a separate draft vocabulary. A small MLP head reads the frozen target model's hidden state, predicts a phrase entry, and a greedy longest-match expansion maps it back into the target tokenizer's tokens; candidates are verified in one forward pass behind a prefix-tree mask. Because the target and tokenizer stay frozen and candidates are accepted only when they match the target's argmax, the authors state the served output is identical to greedy decoding up to floating-point tie-breaking. On a single A5000 at batch 1 against HuggingFace generate(), a 21.94M-parameter head reports 2.391x on GSM8K, 2.444x on MATH-500 and 1.544x on MT-Bench, with a smaller 8,000-entry head described as 64x smaller than EAGLE-3's drafter.

speculative decodingLLM inference efficiencyQwen3
Research

Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure

The paper proposes Head-Adaptive Sparse Attention (HASA), a training-free sparse attention framework for LLMs that adapts selection resolution along feature and sequence dimensions. It uses each pretrained head's query-key operator to determine an effective rank for a compact response space, then merges neighboring regions based on directional coherence while keeping heterogeneous regions separate. Exact attention is computed on selected regions using the original full-dimensional KV states. The authors report capturing 88.1% of dense-attention mass, accuracy improvements over UNIQUE of up to 3.47 and 2.57 points on RULER and LongBench-Pro, and up to 14.07x speedup over dense attention.

sparse attentionLLM inference efficiencylong-context LLMs
Research

Berkeley AI Research Survey Maps Adaptive Parallel Reasoning for Efficient LLM Inference

A Berkeley AI Research blog post surveys adaptive parallel reasoning (APR), a paradigm where reasoning models decide when to decompose tasks, spawn concurrent threads, and coordinate them. It contrasts APR with fixed methods like self-consistency and tree search, and details inference system designs including Multiverse's KV-cache stitching and ThreadWeaver's engine-agnostic client orchestration. The post covers training approaches and highlights trade-offs in memory management and throughput.

adaptive parallel reasoningLLM inference efficiencyinference systems