Research
RESYNC: Residual Latent-State Synchronization for Deadline-Bound Streaming Inference
RESYNC addresses high-rate streaming inference where a large model may miss per-observation deadlines. The large model runs asynchronously while a fast path updates its stale latent state using newer observations and reuses the unchanged prediction head, learning from frozen-model representations without task labels. Across one real and two synthetic streams and five architecture families, RESYNC outperformed compact or distilled alternatives in all twelve dataset-architecture pairs at anchor ages of 4–12 observations, and in eleven pairs at older ages up to 32. In 1 ms arrival replay, it served at least 99.4% of arrivals within the interval with 0.35–0.62 ms median latency on one CPU thread.
RESYNCstreaming inferencelatency optimization