Back to AI News
arxiv

RecoGEM: A Serving System for Production-Scale Generative Recommendation

RecoGEM is a production serving system for generative recommendation, where item identifiers are generated autoregressively instead of scoring candidate items. It uses a runtime-engine decoupling architecture to handle GR-specific responsibilities such as candidate-state transitions and live artifact updates, while the execution engine optimizes self-attention, candidate selection, and execution planning for short decoding sequences and large candidate populations. The system has been deployed for over nine months across production lines serving hundreds of millions of users and more than ten GR model families. On OneModel workloads, it reports over 1,000 QPS per serving instance, P99 latency below 35 ms, and a 2.88x throughput improvement over representative baselines.

AI Brief

It offers concrete architecture and production performance details for serving generative recommendation models at industrial scale.

arXiv:2611.12296v1 Announce Type: cross Abstract: Generative recommendation (GR) is moving industrial recommender systems (RecSys) from scoring candidate items to autoregressively generating item identifiers. This shift promises higher recommendation quality, but makes serving substantially harder: each request executes a short sequence of dependent decoding steps that explore a large item space and select promising candidates on the latency-critical path. GR not only creates structural mismatches with conventional RecSys pipelines, but also challenges the architectural and optimization assumptions of LLM serving stacks, making direct adoption of existing systems difficult. We present RecoGEM, our production GR serving system. RecoGEM uses a runtime--engine decoupling architecture to move GR-specific serving responsibilities, such as candidate-state transitions and live artifact updates, to a dedicated runtime. Its execution engine exploits GR's short token sequences, large candidate populations, and predictable step shapes to optimize self-attention, candidate selection, and execution planning. RecoGEM has been deployed for over nine months across our production lines, serving applications with hundreds of millions of users and supporting more than ten GR model families. On our OneModel production workloads, RecoGEM reaches over 1,000 QPS per serving instance with P99 latency below 35 ms and a 2.88$\times$ throughput improvement over representative baselines.
Be the first to rate this article.

Comments & Questions (0)

Captcha
Please be respectful — let's keep the conversation friendly.

No comments yet

Be the first to comment!