Back to AI News
arxiv

Frozen ASL Pose Encoder Transfers to Low-Resource Sign Language Retrieval

Researchers evaluate a pose encoder trained with metric learning on American Sign Language, kept frozen, on six low-resource sign language dictionary datasets. With one enrolled clip per sign, the encoder ranks the correct sign first in 36.4–80.7% of queries, compared with 16.0–69.6% for angle-based dynamic time warping and a six-target mean of 39.3% for an ASL-trained OpenHands graph network. An ablation finds that a per-pair InfoNCE objective and blocking attention between body and hands improve accuracy, with the best configuration reaching 68.0% one-shot accuracy versus 59.0% for the reference encoder. On the 4,765-word CISLR Indian Sign Language dictionary, top-1 accuracy is 17.3–19.0% versus 16.8% for the RGB baseline, while embedding geometry has only a minor effect.

AI Brief

It quantifies zero-shot cross-lingual transfer for sign language dictionary retrieval and identifies training and architecture choices that improve low-resource performance.

arXiv:2611.16621v1 Announce Type: new Abstract: Most sign languages have no annotated corpus, only small dictionaries with one or a few example videos per sign. This study examines how well a pose encoder trained on a single sign language serves such languages without any target-language training. An encoder trained with metric learning on American Sign Language (ASL-Citizen) is evaluated frozen on six datasets of Turkish, Russian, Indian, Argentinian, Pakistani and Sri Lankan sign languages, with one or four clips per sign enrolled and, where signer identities are available, the query's signer excluded from the gallery. With one enrolled clip, the encoder retrieves the correct sign first in 36.4-80.7 % of queries, against 16.0-69.6 % for the best learning-free method (angle-based dynamic time warping) and a six-target mean of 39.3 % for an ASL-trained OpenHands graph network. In a crossed ablation of the encoder (embedding geometry, aggregation of body and hand embeddings, training objective, and attention between articulators), two choices matter: a per-pair InfoNCE objective outperforms supervised contrastive and cross-entropy training, and blocking attention between body, left hand and right hand inside the encoder raises accuracy by up to 5.1 points. The best configuration reaches 68.0 % one-shot accuracy (59.0 % for the reference encoder). On CISLR, an Indian Sign Language dictionary of 4,765 words with one video per word, the encoders find the correct word first for 17.3-19.0 % of queries, above the 16.8 % of the original RGB baseline. Over eight seeds with the joint encoder, spherical, Euclidean and hyperbolic embedding spaces differ by at most 1.6 points, except the hyperbolic token-attention head trained with InfoNCE, which is 1.4-2.0 points worse, so the geometry is a minor factor.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!