CardioLM: A multilingual suite of small language models for the cardiology domain
Researchers introduce CardioCCC, a corpus of 508 cardiology case reports originally in Spanish and extended to Czech, Dutch, English, Italian, Romanian and Swedish via machine translation, annotation projection and clinician review, annotated for diseases, symptoms, procedures and medications. The accompanying CardioLM suite comprises CardioBERTa encoders built by continued pretraining on cardiology and biomedical corpora, plus CardioNER named-entity-recognition models fine-tuned on CardioCCC. Cardiology-specific pretraining cut pseudo-perplexity by up to a factor of eight but shifted strict span F1 by at most 0.04, while jointly trained multilingual XLM-RoBERTa-large models gave the best strict F1 in nearly every language and entity type, beating the strongest monolingual system by up to 0.13 F1. Clinicians rated CardioNER well above a rule- and dictionary-based baseline, with mean ratings rising from 26% to 73%, though many residual errors were boundary-related, especially for medications.