https://nalar.dev/tags/serving-systems/ Recent content in Serving Systems on Nalar Hugo en-us Thu, 24 Sep 2026 00:00:00 +0000 https://nalar.dev/continuous-batching-replaces-static-request-groups-with-iteration-level-scheduling/ Thu, 24 Sep 2026 00:00:00 +0000 https://nalar.dev/continuous-batching-replaces-static-request-groups-with-iteration-level-scheduling/ <p>Autoregressive generation does not finish every request at the same iteration. One sequence may emit a stop token after a few decode steps while another remains active for hundreds more. A static batch keeps those requests coupled until the batch boundary. Continuous batching breaks that coupling by letting the active set change between model iterations.</p> <p>The key change is scheduling granularity. A request is no longer the indivisible scheduling unit for the entire generation. The serving system can construct an execution batch for one iteration, update request state after that iteration, remove finished sequences, and admit queued work before the next execution step.</p>