LEDGE: A Simulator for Distributed LLM Serving on Heterogeneous Edge Devices
The paper introduces LEDGE, a lightweight simulator for evaluating distributed LLM serving on heterogeneous edge devices without executing tensor operations. It uses event graphs supporting non-uniform tensor and pipeline parallelism, calibrated reusable performance models, and shared-link contention simulation. On a 10-device Jetson and Raspberry Pi testbed covering 17 model-precision combinations across four LLM families, LEDGE reports request-level MAPE below 3.6% on Qwen deployments and a 135.3x aggregate speedup over physical inference. A model calibrated on one LLM family achieved 7.44% zero-shot end-to-end latency MAPE across three other families, and the source code is publicly available.
It offers researchers and engineers a fast, open-source way to explore distributed LLM deployment strategies on heterogeneous edge hardware without repeated physical experiments.
Comments & Questions (0)
No comments yet
Be the first to comment!