Back to AI News
arxiv

Protein Language Models Fail Amino-Acid Relabeling Invariance Test

A new arXiv paper introduces a diagnostic framework that globally relabels the amino-acid alphabet while preserving alignment-level information, column entropies, and pairwise edit distances. Classical pseudolikelihood Potts models and inverse-covariance estimators remain invariant under this transformation, but protein language models do not. Across ESM-2, MSA Transformer, and AMPLIFY, and scales from 8M to 3B parameters, pLM contact maps collapse under arbitrary relabeling. The authors attribute this to reliance on canonical residue identities, or "alphabet anchoring," rather than pure coevolutionary statistics.

AI Brief

It exposes a concrete representational bias in protein language models and cautions against assuming they learn the same invariant coevolutionary signals as classical methods.

arXiv:2611.16881v1 Announce Type: cross Abstract: Protein language models (pLMs) produce highly accurate contact maps from single-sequence inputs. People often take this accuracy as proof that these models implicitly learn invariant coevolutionary statistics, extracting the same signals as classical alignment-based methods like Potts. In this work, we introduce a diagnostic framework around a global bijective relabeling of the amino-acid alphabet. This transformation permutes residue letters while leaving all alignment-level information, column entropies, and pairwise edit distances unchanged. Pseudolikelihood Potts models and inverse-covariance estimators remain invariant under this transformation, but pLMs fail the test. Across multiple architectures (ESM-2, MSA Transformer, AMPLIFY) and model scales (8M to 3B parameters), pLM contact maps collapse under arbitrary relabeling. Using Hamming distance controls and biochemical grouping interventions, we attempt to isolate what likely causes this degradation and find that these models rely on explicit, canonical residue identities (``alphabet anchoring'') rather than pure coevolutionary statistics. Increasing scale makes models more responsive to biochemical classes, but their performance still degrades under relabeling. Our findings show a gap between how pLMs and classical coevolutionary models compute contacts. More broadly, our work shows how diagnostic perturbations that leave classical models provably unchanged can expose hidden representation biases and prevent over-reliance on neural black boxes in biological modeling.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!