METR expands AI time-horizon evaluation to judgment-heavy tasks
METR updated its widely cited time-horizon plot with more open-ended tasks that require judgment, moving beyond easily verifiable coding and math benchmarks. The report says AI capabilities continue improving exponentially on verifiable tasks, but models still fall short of humans on tasks such as designing useful AI evaluations and scoring philosophical arguments. Some newest models show “borderline above the bar” performance on evaluation design, though none produced evaluations METR considered useful. Experts welcome the expansion but caution that scoring hard-to-verify tasks remains difficult and extrapolation is uncertain.
Why it matters: It offers a clear, expert-informed update on how AI evaluation is expanding beyond verifiable tasks, a key question for automating AI R&D.