Back to AI News
arxiv

An Augmented Lagrangian Approach to Lifted Bellman Linear Programming

This paper introduces AL-LBLP, an augmented Lagrangian method for solving Lifted Bellman Linear Programming in offline reinforcement learning. Instead of fixed hinge-penalty coefficients, it learns multipliers via dual ascent with a single penalty parameter and shows that in the convex case fixed points are KKT pairs of LBLP. The critic can drive a Gaussian actor, a flow actor, or best-of-N sampling, and for the Gaussian actor the authors propose an advantage-normalized DDPG+BC objective. Using one Q and V and no target network, AL-LBLP with best-of-N reports 60.1% average success across 50 state-based OGBench tasks, 81.5% on humanoidmaze-giant, 79.8% on antmaze-giant, and no collapse up to batch size 8192.

AI Brief

It offers a concrete offline-RL algorithm with a cleaner optimization formulation and strong reported benchmark results, useful for researchers working on value-based reinforcement learning.

arXiv:2611.14183v1 Announce Type: new Abstract: Temporal-difference critics regress onto a bootstrapped target and usually need a target network. Lifted Bellman Linear Programming (LBLP) instead expresses the in-sample optimal values of offline RL as the solution to a linear program over the dataset. Its neural solver ALBUM relaxes the constraints into hinge penalties with fixed coefficients, and its guarantees are conditions on coefficients chosen in advance. We solve the same program with the method of multipliers: the coefficients become multipliers learned by dual ascent, with one penalty parameter $\rho$. In the convex case the fixed points are the KKT pairs of LBLP, and at every stationary point the applied constraint weights balance the objective exactly. The critic can serve a Gaussian actor, a flow actor, and best-of-N sampling, and for the Gaussian actor we propose an advantage-normalized DDPG+BC objective. With a single $Q$ and $V$ and no target network, our AL-LBLP with best-of-N reaches 60.1\% average success on the 50 state-based OGBench tasks. On long-horizon tasks, it reaches 81.5% on humanoidmaze-giant and 79.8% on antmaze-giant. It does not collapse up to batch size 8192.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!