Back to AI News
arxiv

Sirius-Tiamat: When Compression Flips a 421M Decision Encoder's Answers

The Sirius-Tiamat study compresses Laya, a 421M-parameter one-pass decision encoder that returns option probabilities in a single forward pass, and measures how often compressed variants disagree with the full-precision fp32 model while tracking expected calibration error. On 1,221 in-family synthetic items, bf16 changes 0.2% of decisions but 2.2% of 800 out-of-family ARC, MMLU and HellaSwag questions; 6-bit grouped rounding changes 1.3% and 5.9%, and deleting six layers changes 0.9% and 9.1% while ECE rises from 5.9% to 39.2%. A single fitted temperature requiring no labels restores ECE to 3.6%, and at equal stored size rounding flips fewer decisions than layer deletion. CPU dynamic int8 changes 30% of decisions almost entirely through one layer, and distilling a 16-layer student cuts in-family flips from 44% to 9%, though the authors caution the data are synthetic, LLM-labelled and cover one architecture.

AI Brief

Gives deployment teams concrete per-method flip and calibration numbers showing that quantization and pruning degrade out-of-family decisions far more than in-family ones.

arXiv:2611.26339v1 Announce Type: new Abstract: A one-pass decision encoder reads a text, a question and several named options, and returns a probability for each option in a single forward pass. We ask how far Laya, a 421M-parameter encoder of this kind, can be compressed before its decisions change. We count decisions that differ from Laya's own full-precision (fp32) answer, and track calibration (how well confidence matches accuracy, measured by expected calibration error, ECE), on 1,221 synthetic items with the training schemas, 273 with an unseen schema, and 800 out-of-family questions from ARC, MMLU and HellaSwag. Layer pruning of a related model family is known; we add flips, calibration, held-out data and retraining. bf16 changes 0.2% of in-family decisions but 2.2% out of family; 6-bit grouped rounding 1.3% and 5.9%; deleting 6 layers 0.9% and 9.1%, while ECE rises from 5.9% to 39.2%. One fitted temperature, needing no labels, restores it to 3.6%. One scale per tensor fails below six bits, and CPU dynamic int8 changes 30% of decisions, nearly all through one layer. Total variation tracks flips for rounding but not for deletion, and an early exit saves layers over a static cut only if its confidence gate is first recalibrated. Distilling a 16-layer student cuts its in-family flips from 44% to 9% (45% out of family). At equal stored size, rounding flips fewer decisions than deletion. Data are synthetic and LLM-labelled, for one architecture.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!