Lessons from Building an LLM Agent for Automated Experimentation
The paper reports practical lessons from building and operating an LLM-based agent for automating experimentation workflows. The central finding is that nondeterminism should be bounded by having the model select and sequence procedures while deterministic code computes all reported quantities. Surprising outcomes include improved answers when information is withheld, high demand for mundane retrieval tasks, and risks to statistical validity from the agent's eagerness to help.