LeCat: Measuring LLM Category Learning from Pretraining Data
LeCat is a two-part framework for analyzing how LLMs learn category knowledge from pretraining data, distinguishing memorization, generalization, and flexible use. Experiments on open-source models and a controlled 68M model show that verbatim repetition reduces flexible learning, while diverse knowledge expressions improve robustness. Contextual flexibility remains a persistent challenge.