Back to AI News
arxiv

LEDGE: A Simulator for Distributed LLM Serving on Heterogeneous Edge Devices

The paper introduces LEDGE, a lightweight simulator for evaluating distributed LLM serving on heterogeneous edge devices without executing tensor operations. It uses event graphs supporting non-uniform tensor and pipeline parallelism, calibrated reusable performance models, and shared-link contention simulation. On a 10-device Jetson and Raspberry Pi testbed covering 17 model-precision combinations across four LLM families, LEDGE reports request-level MAPE below 3.6% on Qwen deployments and a 135.3x aggregate speedup over physical inference. A model calibrated on one LLM family achieved 7.44% zero-shot end-to-end latency MAPE across three other families, and the source code is publicly available.

AI Brief

It offers researchers and engineers a fast, open-source way to explore distributed LLM deployment strategies on heterogeneous edge hardware without repeated physical experiments.

arXiv:2611.24085v1 Announce Type: cross Abstract: Evaluating distributed large language model (LLM) serving on heterogeneous edge devices requires exploring diverse model partitions, device placements, and network conditions, making repeated physical experiments costly. An efficient simulator for edge LLM serving must support diverse, fine-grained placement configurations, capture runtime behavior, and accurately predict execution times while keeping simulation overhead low. We present LEDGE, a lightweight simulator for fast, fine-grained evaluation of distributed LLM serving on heterogeneous edge devices without executing tensor operations. LEDGE constructs event graphs supporting non-uniform tensor and pipeline parallelism, estimates execution costs using reusable performance models calibrated from end-to-end measurements, and simulates shared-link contention. We evaluate LEDGE on a heterogeneous testbed of 10 Jetson and Raspberry Pi devices, covering 17 model--precision combinations across 4 LLM families. On Qwen deployments, LEDGE achieves request-level mean absolute percentage errors below 3.6% and a 135.3 times aggregate speedup over physical inference. Performance model calibrated on one LLM family yield 7.44% zero-shot end-to-end latency MAPE across three other families. Further simulations show how shared-link contention affects placement decisions and network topology constrains scaling. Across four edge environments, LEDGE completes 26,773 simulation runs in 83.8 minutes, enabling efficient exploration of deployment strategies under diverse resource conditions and serving objectives. The source code is publicly available at https://github.com/gdfwj/LEDGE.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!