Back to AI News
arxiv

Simple Generate-and-Select Baseline Matches Costlier Prompt Optimizers, Audit Finds

Researchers introduce Demonstration-Conditioned Prompt Search (DCPS), a deliberately simple control that freezes a generator, conditions it on randomly sampled demonstrations, scores candidates on validation data, and keeps the best candidate without learned updates, reflection, or evolution. Across two optimizer families, DCPS reportedly matches StablePrompt's PPO-based classification accuracy within 0.5 percentage points at roughly one-third the measured cost, and matches GEPA's macro accuracy on GPT-4.1-mini in DSPy compound systems using 55% fewer rollouts at 32% lower measured API cost on average. The audit finds gains are task-dependent: proposal-and-selection recovers much of heavier optimizers' performance on structured and instruction-following tasks, while extra mechanisms still help in some open-ended and reasoning-heavy settings. The authors recommend benchmarking prompt optimizers against generate-and-select controls and reporting rollout counts, token usage, and dollar costs alongside accuracy.

AI Brief

It gives practitioners a cheap, reproducible baseline for judging whether elaborate prompt-optimization pipelines actually earn their extra compute and API spend.

arXiv:2611.22605v1 Announce Type: new Abstract: Automatic prompt optimization is growing more elaborate. Recent methods stack policy-gradient updates, trace reflection, and evolutionary search. These methods share a generate-and-select step; without controlling for it, reported gains may conflate the contributions of optimizer mechanisms and larger sampling budgets. We isolate this step as Demonstration-Conditioned Prompt Search (DCPS), repurposing the Automatic Prompt Engineer (APE) recipe as a mechanism-attribution control: it freezes the generator, conditions it on randomly sampled demonstrations, scores candidates on validation data, and keeps the best---no learned update, reflection, or evolution. Across two optimizer families, DCPS replaces Proximal Policy Optimization (PPO) in StablePrompt, matching classification accuracy within 0.5 pp at roughly one-third the measured cost; in DSPy-based compound AI systems, under the reported budgets, it matches GEPA's macro accuracy on GPT-4.1-mini while using 55% fewer rollouts at 32% lower measured API cost on average. The audit reveals a task-dependent pattern: proposal and selection recover much of heavier optimizers' performance on structured and instruction-following tasks, while additional mechanisms retain advantages in some open-ended and reasoning-heavy settings. We recommend benchmarking prompt optimizers against generate-and-select controls and reporting rollout counts, token usage, and dollar costs alongside accuracy. Code and artifacts: https://github.com/bhc6/DCPS.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!