EngineAgent: Agentizing On-Device LLM Inference Engines with Inference Stack Hierarchy Awareness and Hardware Abilities
EngineAgent is an agentic system that automates enabling new LLM models in on-device inference engines by separating reusable engine and hardware knowledge from model-specific implementation. It builds a hierarchy-aware engine index, pre-validates backend workflows, and uses a builder-auditor loop to coordinate cross-layer changes and device-side debugging. Across 12 tasks covering three mobile LLMs and four engine-backend pairs on CPUs, GPUs, and NPUs, it achieved 75% end-to-end success versus 33.3% for the strongest coding-agent baseline, while reducing average per-task time and model-API cost.
It presents a concrete agentic approach with measured gains for the costly engineering problem of bringing new LLMs onto heterogeneous mobile inference stacks.
Comments & Questions (0)
No comments yet
Be the first to comment!