arXiv:2611.16502v1 Announce Type: new
Abstract: While quantization compresses LLMs and accelerates inference, outlier activations impede efficient low-bit representation. In particular, massive outliers cause substantial performance degradation. Although rotation-based methods that rely on randomized Hadamard transforms for online inference aim to mitigate this, they can produce fragmented distributions with empty bins when a massive outlier dominates a token's activation. This fragmentation severely undermines uniform quantization performance. Moreover, massive outliers concentrate in the down-projection input, where the preceding nonlinearity requires the rotation to be applied at inference time. This constraint prevents training-based approaches such as SpinQuant from learning the online rotation at this position, as a learned dense matrix cannot use the Fast Walsh-Hadamard Transform and would require a full matrix multiplication during inference. To address this, we propose SignRot (Sign-adjusted Rotation), which identifies massive outlier channels and constructs a sign adjustment vector that unifies the signs of the corresponding Hadamard row, collapsing the bimodal distribution into a unimodal profile amenable to quantization. Since the aligned outlier shifts the activation mean away from zero, we pair SignRot with asymmetric activation quantization, whose zero-point absorbs this shift. As a lightweight, drop-in improvement compatible with any Hadamard-based quantization method, SignRot requires only a single calibration forward pass and adds only element-wise sign flips at inference, without an additional matrix multiplication. Applying SignRot to QuaRot achieves a WikiText2 perplexity of 5.88 and an average zero-shot reasoning accuracy of 72.73, surpassing SpinQuant on the LLaMA3-70B model. Code is available at https://github.com/HYU-ARK-Lab/SignRot.
arxiv
SignRot: Outlier-Aware Sign-Adjusted Rotation for LLM Quantization
SignRot is a quantization method that addresses massive outlier activations in LLMs by adjusting the signs of Hadamard rows, collapsing bimodal activation distributions into a unimodal profile that is easier to quantize. It is a lightweight, drop-in improvement for Hadamard-based quantization methods, requiring only a single calibration forward pass and adding only element-wise sign flips at inference. Applied to QuaRot on LLaMA3-70B, SignRot achieves a WikiText2 perplexity of 5.88 and an average zero-shot reasoning accuracy of 72.73, surpassing SpinQuant. Code is available on GitHub.
Offers a practical, low-overhead enhancement to LLM quantization with strong reported results on a 70B model, useful for efficient inference work.
Comments & Questions (0)
No comments yet
Be the first to comment!