Back to AI News
arxiv

FondueQA Benchmark Probes How LLMs Fill Gaps in Incomplete Retrieval Evidence

Researchers introduce FondueQA, a multi-hop QA benchmark in which every required fact is uniquely localized to a designated evidence document, enabling controlled document-level evidence ablations. The design addresses benchmark contamination concerns that make it hard to tell whether correct answers under missing evidence reflect general parametric knowledge or memorization of benchmark instances. Experiments show parametric compensation depends on entity popularity and the structural role of the missing evidence, with bridging evidence sometimes more useful than the answer-bearing document. Models can recall a missing fact in isolation yet fail to use it under partial evidence, especially when plausible competing entities appear in context, reducing Exact Match by up to 15 percentage points.

AI Brief

It offers a contamination-resistant way to measure whether models can fall back on internal knowledge when retrieval is incomplete, a practical concern for RAG system builders.

arXiv:2611.09274v1 Announce Type: cross Abstract: Large Language Models (LLMs) show remarkable performance based on parametric knowledge, i.e., their ability to recall facts and associations internalized in a model's weights. However, studying this capability on established multi-hop benchmarks may be complicated by benchmark contamination, as many of these benchmarks have been publicly available long enough to plausibly appear in modern pretraining corpora. As a result, correct answers under missing evidence may reflect either general parametric knowledge or memorization of benchmark-specific instances. To study how models use internal knowledge when retrieval evidence is incomplete, we introduce FondueQA, in which every required multi-hop fact is uniquely localized to a designated evidence document. This enables controlled document-level evidence ablations to study parametric compensation under incomplete retrieval. Our empirical evaluation reveals that parametric compensation strongly depends on both entity popularity and the structural role of missing evidence. Generally, models compensate more successfully for facts involving popular entities. On FondueQA, we show that bridging evidence can be more useful than the answer-bearing document itself. Moreover, models may recall a missing fact in isolation but fail to use it under partial evidence, particularly when the context contains plausible competing entities, which can reduce Exact Match by up to 15 percentage points.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!