Back to AI News
arxiv

LM-SimBench: A Unified Framework, Dataset, and Benchmark for Large Language Model Training Simulation

LM-SimBench is a unified framework, profiling dataset, and benchmark for evaluating LLM training simulators. It decomposes simulation into functional stages from model compilation to schedule-level aggregation and connects execution representations and cost estimators to measurable evaluation targets. The dataset profiles 23 model settings from 10 dense and Mixture-of-Experts LLM families on an Ascend NPU cluster, yielding 348 feasible configurations across five parallelism dimensions and activation recomputation. Evaluations show the benchmark can support simulator accuracy comparisons, configuration-selection assessment, and prediction-error diagnosis, with matched GPU-NPU experiments validating cross-platform applicability.

AI Brief

It gives researchers and infrastructure engineers a structured way to compare LLM training simulators and parallelism configurations without expensive distributed training runs.

arXiv:2611.14285v1 Announce Type: new Abstract: Evaluating candidate parallelization strategies through distributed training runs makes strategy selection prohibitively expensive at scale. Performance simulation offers a lower cost alternative, yet controlled comparisons among existing simulators remain difficult because they differ in the training stages they model and the workloads on which they are evaluated. We introduce LM-SimBench, a unified framework, profiling dataset, and benchmark for evaluating LLM training simulators. The framework decomposes simulation into functional stages from model compilation to schedule level aggregation, connecting execution representations and cost estimators to measurable evaluation targets. Following this framework, we profile 23 model settings from 10 dense and Mixture-of-Experts (MoE) LLM families on an Ascend Neural Processing Unit (NPU) cluster. The resulting dataset contains 348 feasible configurations that jointly vary five parallelism dimensions and activation recomputation, with six views spanning configuration, rank, operator, event, and kernel granularity. Evaluations of representative training simulators demonstrate how LM-SimBench supports comparisons of prediction accuracy, assessment of configuration selection, and diagnosis of prediction errors. Matched GPU-NPU experiments validate the cross-platform applicability of the collection and evaluation protocol and demonstrate NPU-guided GPU configuration selection.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!