Research
GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR
This paper introduces GIST, a two-level data selection method for offline reinforcement learning with verifiable rewards (RLVR). It derives a factorization of the KL divergence between the current policy and the Gibbs-optimal target, identifying which responses and tokens are most informative. GIST selects correct responses on-policy and incorrect responses near the policy's mode, and supervises only the tokens where the mismatch lies. Experiments show accuracy gains across GSM8K, MATH-500, and MBPP for three models, with 66.6% fewer supervised tokens in a controlled comparison.
Reinforcement LearningData SelectionToken Efficiency