Audit Finds LLM Health Advice Varies by User Persona, Factual Floor Helps but Not Uniformly
A new arXiv study audits six LLMs from four providers on smoking-cessation advice, finding that responses vary significantly based on user persona. Models mention FDA-approved cessation medications less often to teen personas (by 32–67 percentage points) and often give fewer facts to users claiming healthcare expertise. A system prompt enforcing a factual floor reduced persona differences in all models, but its effect on overall fact coverage varied across models, indicating limited transferability. The authors release all 2,880 responses and analysis code.
Why it matters: This study provides concrete, recent evidence of persona-dependent variation in LLM health advice and tests a practical intervention, offering actionable insights for AI deployment in high-stakes domains.