Back to AI News
arxiv

Balanced Accuracy Can Hide Silent Class Loss in Neural-Signal Test-Time Adaptation

A new arXiv paper evaluates full test-time adaptation across 11 dataset-encoder cells on Siena/EEG, PTB-XL/ECG and Sleep-EDF/EEG under fixed splits and update rules. It shows that in the binary case balanced accuracy can score a decoder that stopped predicting a minority class the same as one predicting both classes, with a class covering about a tenth of a subject's samples going unpredicted in 26-47 of 112 subject-cells versus 7-9 for controls. The study also finds the within-subject class contrast follows the encoder rather than the aggregate metric, with aggregate and contrast naming different encoders in 10 of 11 cells, and recommends reporting class-level and subject-level change alongside aggregate scores.

AI Brief

Researchers and practitioners deploying test-time adaptation for biosignal models get concrete evidence that aggregate metrics like balanced accuracy can mask per-class and per-subject failures, plus a protocol for more rigorous reporting.

arXiv:2611.17925v1 Announce Type: new Abstract: Adaptation for neural-signal classification is usually transductive: each batch is scored after updating on it, and the gain depends on the adapted model [4]. This paper asks what an aggregate change fails to characterise inside the cohort. Full test-time adaptation (full_ttt) was evaluated in 11 dataset-encoder cells (four each on Siena/EEG and PTB-XL/ECG, three on Sleep-EDF/EEG), with splits and source-model evaluation fixed; the subject was the unit, exact sign tests were Holm-corrected and conditional on the shared stream, and bootstrap intervals accompanied a dependence audit. Two results of different kinds follow. In the binary case balanced accuracy gives a one-class decoder the same value as one reporting both classes at 50%, so the aggregate cannot identify a class that stopped being reported; a class covering a tenth of a subject's samples went unpredicted in 26-47 of 112 subject-cells, against 7-9 for the controls, which is a descriptive count. With cohort, split and update rule fixed, the within-subject class contrast follows the encoder, tested directly rather than inferred from tests against zero: two of six Siena encoder pairs survive the dependence readings at their nominal reference and one at a fold-level reference, while two of ten Sleep-EDF stage pairs pass the run-clustered gate at its nominal reference; aggregate and contrast named different encoders in 10 of 11. The protocol bounds both results, since a stream-interleaving effect reverses sign with state sharing (+3.5 to -2.4 pp). An aggregate score can hide individual harm, as [1] showed, but it cannot distinguish a uniform loss from a deleted class; class-level change should therefore be reported alongside the aggregate and subject-level change alongside it, with both results conditional on the cell, the rule and the stream construction.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!