Research
Contain, Conform, Compare (Extended Version): A Red-Teamed Evaluation Kit for Small Coding Agents and the Full Record of Its Experiments
The paper presents drydock-eval, a local evaluator for small coding agents that runs jobs in fresh network-isolated containers and checks local scoring behavior against a hosted scorer. It reports throughput of about 250 runs per hour for Gemma 4 E4B on one RTX 5090 and releases the full experimental record, including 9,660 agent runs, 2,494 grading controls and 72 serving measurements. Experiments found no detected E4B solve-rate change from four prompt divergences and no confirmation of three agent changes that had led their initial screens. The unchanged baseline also moved by 1.17 points when rerun, underscoring run-to-run variability in agent evaluation.
drydock-evalcoding agentsLLM evaluation