Live intelligence

Latest AI updates

2 updates
Research

BubbleTea Reuses MoE Expert-Parallel Idle Time for Fine-Tuning

The paper introduces BubbleTea, a colocation system that fills idle GPU bubbles during Mixture-of-Experts LLM inference with short chunks of fine-tuning work. It predicts each GPU's per-layer bubble from the router's token assignment and uses compiler passes to derive fine-tuning chunks from the served model without porting architectures to a separate training engine. Evaluated on three MoE models across four A100 GPUs, BubbleTea reports 1.9–6.2× faster fine-tuning throughput than LLMStation while keeping p90 inference latency within 10% of serving alone on 50 of 63 workloads; on bursty traces it trains 3.3× faster than LLMStation and 14× faster than a stock trainer colocated under CUDA MPS.

Mixture-of-ExpertsGPU colocationLLM fine-tuning
Research Editor selected

LoopVQ: Sub-1-bit KV Cache Quantization for Looped Transformers

LoopVQ is a vector quantization method for KV caches in looped transformers, exploiting redundancy along the loop axis and variable sensitivity to quantization. It achieves 0.5 bits per element, outperforming existing KV quantization methods on reasoning and retrieval tasks, and combined with TaSQ retains near-BF16 accuracy at 0.25 bits. Implemented in SGLang, it nearly halves decode latency at 32K context and boosts serving throughput up to 6.1x.

Why it matters: This paper presents a concrete, novel optimization for an emerging architecture class, with measurable efficiency gains and practical deployment in SGLang.

Looped transformersKV cache compressionModel quantization