Live intelligence

Latest AI updates

8 updates
Research

DForkTree Speeds Speculative Decoding With Fork-Aware Draft Trees

DForkTree is a training-free speculative decoding method that re-anchors block-diffusion draft-tree branches on their own tokens. It adds n-gram pointwise mutual information from the target model's outputs to drafter scores, weights those scores more after a fork, and builds the optimal tree on the GPU. On Qwen3.5-4B and 9B, with shared drafter, tree pipeline, and verifier, it was the fastest tree method on every benchmark, up to 41.5% faster than DFlash and 15.1% faster than DDTree, with average speedups of up to 4.75x over autoregressive decoding.

Speculative decodingBlock diffusionLLM inference acceleration
Research

Phrase-Level Speculative Decoding Drafts Multi-Word Units on a Frozen Target

The paper proposes changing the drafting unit for speculative decoding from single target tokens to one- to three-word phrases stored in a separate draft vocabulary. A small MLP head reads the frozen target model's hidden state, predicts a phrase entry, and a greedy longest-match expansion maps it back into the target tokenizer's tokens; candidates are verified in one forward pass behind a prefix-tree mask. Because the target and tokenizer stay frozen and candidates are accepted only when they match the target's argmax, the authors state the served output is identical to greedy decoding up to floating-point tie-breaking. On a single A5000 at batch 1 against HuggingFace generate(), a 21.94M-parameter head reports 2.391x on GSM8K, 2.444x on MATH-500 and 1.544x on MT-Bench, with a smaller 8,000-entry head described as 64x smaller than EAGLE-3's drafter.

speculative decodingLLM inference efficiencyQwen3
Research Editor selected

D3D: Decoupled Diffusion Drafting for Asynchronous Speculative Decoding

D3D is a standalone diffusion drafter that removes inter-round dependency in speculative decoding, allowing drafting of successor blocks during verification in a packed pass. It combines blockwise denoising with auxiliary autoregressive supervision and prunes layers to reduce cost. Across seven benchmarks, D3D achieves 5.00x and 5.71x speedups with Qwen3-8B and Qwen3-32B targets, outperforming DFlash.

Why it matters: This paper presents a concrete, recent AI development with measurable speedup improvements and open-source code, offering practical value for efficient LLM inference.

Speculative DecodingDiffusion ModelsEfficient Inference
Research

Adversarial Speculative Reasoning Improves Long Horizon Agents

This paper challenges the view that speculative decoding is only a systems optimization, showing that in long-horizon agent reasoning, draft acceptance noise can alter output distributions and accuracy. The authors train the draft model to be rejected at critical decision points, forcing verification restarts so that key reasoning tokens are sampled from the target model under less noisy conditions. On complex agentic coding benchmarks, this approach improves end-to-end success rates without changing the target model, using only 0.06% to 0.28% of hosted parameters.

speculative decodingagent reasoningmodel optimization
Research Editor selected

SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding

SpecOPD introduces a branch-parallel framework for on-policy distillation in speculative decoding, enabling draft models to be trained without sequential draft-and-verify rollouts. By packing independent branches into proposal trees and using tree masks, sparse attention, and indexed state replay, the method achieves up to 41.45% higher mean acceptance length and 16.09× supervised-token throughput over sequential-rollout OPD. Experiments across 8B to 125B parameter targets and multiple architectures show up to 3.56× generation speedup over autoregressive decoding.

Why it matters: Offers a concrete, high-impact technique to accelerate LLM inference, directly relevant to practitioners working on model deployment and efficiency.

speculative decodinglarge language model inferenceon-policy distillation
Research

Only Accepted Tokens Count: Energy Accounting for Speculative Decoding Under Latency Targets

The paper measures the energy cost of speculative decoding under production serving conditions on H100 and H200 GPUs using SGLang with CUDA graphs, at batch sizes up to 64 across five corpora. It finds that on dense models verification, not drafting, dominates energy, and that the saving shrinks as batch size grows because batching has already shared the weight read that speculation would divide among accepted tokens. At large batch sizes block drafters push verification past the arithmetic-memory balance point into a compute-bound regime where the saving becomes a loss, and low acceptance rates raise energy per token. Meeting a tighter latency target with speculation therefore costs additional energy, while larger batches lower energy per token without speculation.

Speculative DecodingLLM Inference ServingEnergy Efficiency
Research

TextSeal2: Distortion-Free LLM Watermarking That Survives Speculative Decoding

TextSeal2 is a distortion-free watermark for large language models that retains its detection signal under speculative decoding: tokens replacing rejected drafts stay watermarked with the same keys, and the acceptance rate is unchanged. Instead of TextSeal's coin-flip routing between two keys, it merges multiple keys into a single virtual key coupled to fresh noise via a Gaussian copula, so detection is not diluted even with publicly shared keys. Under speculative decoding it verifies keyed drafts with an additional exponential race over the leftovers of the draft races, which by memorylessness reproduces block verification and keeps output distribution exact. Across three LLMs it detects 8-22 percentage points more 100-token answers than TextSeal at a 10^-3 false-positive rate, and 18 and 22 points more 200-token answers than TextSeal and textGrain respectively under speculative decoding.

LLM watermarkingspeculative decodingAI content provenance
Research Editor selected

FlashMap: Flow Map Drafter Achieves 6.14x Faster LLM Decoding

Researchers from Polymathic AI, Princeton, and partners released FlashMap, a speculative decoding drafter using continuous flow maps to draft 16-token blocks in one pass. On Qwen3-8B, it reports up to 6.14x lossless speedup on GSM8K, outperforming retrained DFlash on all tested benchmarks. The 0.5B-parameter drafter is available on Hugging Face with vLLM and SGLang plugins.

Why it matters: Offers a concrete, novel AI development with significant speedup claims and open-source availability, valuable for practitioners optimizing LLM inference.

speculative decodingLLM accelerationflow-based models