Adding an inference replica creates capacity only after the new instance can serve requests. Between starting a process and generating a response, an LLM server may need to load weights, initialize GPU resources, compile kernels, capture execution graphs, and warm up the engine. Restarts, model switches, and scale-out can repeat this work.
The community is addressing these costs at several levels. vLLM and SGLang are developing ways to reuse weights and initialized state. Serving platforms are building workflows around warm instances and snapshots. Kubernetes, container runtimes, and the Open Container Initiative (OCI) are working on related interfaces and artifact questions.
These efforts have different boundaries and are progressing independently. Understanding what each approach preserves helps explain both its benefits and its current scope.
Why starting an inference server takes time
A typical startup includes several kinds of work. The server launches processes and imports libraries. It reads model files, constructs tensors, and performs any required weight transformations. It initializes the execution runtime and communication between workers, allocates memory, and prepares the engine to serve.
Some work happens specifically to improve subsequent inference performance. Compilation produces executable kernels; autotuning selects implementations for a configuration. CUDA Graph capture records GPU operations for later replay with lower launch overhead. The vLLM optimization documentation describes the tradeoff between startup work and steady-state execution performance.
The balance depends on the model, software, hardware, and storage. Once model files are cached locally, engine initialization may become more visible. Once weights can be reused in GPU memory, imports, tuning, and graph capture may account for more of the remaining delay.
What can be reused instead of rebuilt
Startup optimizations can be grouped by the work they preserve:
| What is reused | Approach | Representative projects or features | Work that can remain |
| Model files | Pre-staging and efficient loaders | llm-d model caches, fastsafetensors | Weight processing and initialization |
| Compiled artifacts and tuning results | Persistent caches | vLLM compilation and autotune caches | Runtime setup and graph capture |
| Processed weights | Daemons and weight transfer | vLLM preload, SGLang weight cache, ModelExpress | Construction of the rest of the engine |
| A live engine | Sleep/wake and warm instances | vLLM sleep, SGLang memory saver, llm-d FMA | Restoring released resources |
| Initialized execution state | Engine or container snapshots | vLLM snapshots, SGLang snapshot proposal, Snapshot, llm-d/GKE | Validation, restoration, and recovery work |
These approaches can appear together in a system, with integration conditions specific to each implementation. For example, an engine might reuse compiled artifacts while loading weights through a specialized loader. A snapshot workflow might discard an empty key/value (KV) cache before capture and recreate it after restoration. This cache holds attention state computed while processing tokens.
Efficient loading remains a separate area of work. SGLang documents loaders such as pre-sharded state, fastsafetensors, and streaming loaders. llm-d documents model-file caches alongside other startup paths. These mechanisms affect how weights reach an engine; they preserve different states from a snapshot of an initialized process. SGLang model loading, llm-d startup overview.
What vLLM and SGLang are doing
vLLM
vLLM’s Fast Start work centers on a persistent weight daemon. It loads and processes weights, holds the resulting tensors in GPU memory, and exposes them to compatible engines through CUDA inter-process communication. The weight-daemon implementation and the vllm preload command have been merged. Current preload documentation also describes preparing FlashInfer autotune results before an engine starts. Fast Start implementation, preload documentation.
This separates the lifetime of cached weights from an engine restart. In the zero-copy mode, an engine maps the daemon’s existing allocations. Configuration checks determine whether reuse is appropriate, and the documented path can fall back to ordinary loading. The daemon must remain alive while engines use its memory. CUDA IPC handles are local to a node; documented multi-node deployments provide the relevant shards locally on each node.
vLLM also has sleep/wake support, which retains a live server while releasing much of its GPU memory. That provides a different lifecycle from starting a new engine against cached weights. The preload documentation identifies restrictions on combining its zero-copy mode with weight offloading through sleep mode. Sleep mode.
A separate, merged implementation adds experimental initialized-engine snapshots. The vllm snapshot workflow uses CRIU to capture and restore Linux process state, with CUDA checkpoint support handling GPU state. Its documented scope requires the same machine and compatible environment, with one NVIDIA GPU and tensor parallelism of one. It releases weights and KV state before capture and reloads them during restoration, retaining a dependency on the model files. Restoration also checks a recorded generation result. Initialized engine snapshots.
SGLang
SGLang has merged a weight-cache daemon that holds processed weights and shares them with new engine instances through CUDA IPC. Its Fast Engine Recovery roadmap discusses extending this foundation, including broader model and quantization coverage, memory transport, daemon lifecycle, and remaining initialization costs. The roadmap describes directions beyond the merged initial implementation. Weight-cache implementation, Fast Engine Recovery roadmap.
SGLang also exposes memory release/resume and generation pause/continue operations used by external snapshot integrations. Whether weights remain available after release depends on the configured memory policy, including CPU backup. These operations let integrations prepare an engine for capture and resume it afterward; the surrounding workflow still needs to coordinate them.
Initialized-engine snapshots are another direction. The open RFC and implementation PR propose capturing SGLang after initialization and before its HTTP service binds. Their initial scope is one NVIDIA GPU, tensor parallelism of one, and the same host and software environment. Model weights remain external and are reloaded after restore. Initialization snapshot RFC, implementation PR.
Both frameworks also have open proposals about cooperating with an external checkpoint orchestrator. The engine would prepare its own state and resume serving; the external system would capture, store, and restore the process. These proposals address a practical boundary: a node agent can operate checkpoint tools, while the inference engine knows when its requests, caches, and communication resources are safe to capture. vLLM proposal, SGLang proposal.
How platforms turn mechanisms into startup workflows
Snapshot and Dynamo
The independent Snapshot project in the ai-dynamo organization provides checkpoint and restore primitives for GPU workloads on Kubernetes. Its operator and node agent coordinate CRIU and CUDA checkpoint operations, with resources such as PodSnapshot and SnapshotJob. Dynamo integrates these primitives into its serving workflows. Snapshot project.
Snapshot’s documented storage path uses a shared persistent volume accessible by node agents. Agent-side components manage artifact movement and restore access. Its current storage documentation supports storage through a Kubernetes PersistentVolumeClaim (PVC); object-storage backends are reserved for future use. Hardware, driver, runtime, and workload compatibility still constrain restoration. Storage model.
The framework examples also show the application work involved. The SGLang example mounts an entrypoint that warms the engine, pauses generation, releases memory, and signals that capture is safe. After restoration, it resumes memory and generation, validates a generation, and starts its API. The vLLM guide validates its own entrypoint on v0.27.1 and explicitly says that example does not work on v0.28. This illustrates a compatibility boundary for that integration. SGLang guide, vLLM guide.
Dynamo separately develops GPU Memory Service, which moves GPU-memory ownership outside the engine process. Its integration and snapshot support need to be read by version: Dynamo’s v1.4.0 Snapshot documentation explicitly describes GMS plus Snapshot as disabled and unsupported for production. GPU Memory Service, versioned Snapshot documentation.
llm-d
llm-d documents several startup approaches. Its Pod Snapshots guide currently covers single-GPU vLLM and SGLang deployments using GKE Pod Snapshots, GKE Sandbox based on gVisor, and Google Cloud Storage. The initial instance loads and warms the model, prepares its memory, and triggers a snapshot. Compatible later Pods restore that state and resume the framework. The guide describes a platform-independent CRIU backend as work in progress. Pod Snapshots guide.
This backend differs from Snapshot’s CRIU path. gVisor saves sandbox state through its own checkpoint mechanism; GKE documents GPU-state capture through NVIDIA checkpoint support. Similar high-level workflows therefore do not imply identical checkpoint formats or APIs. GKE Pod snapshots.
Fast Model Actuation takes another approach. It uses controllers and a requester/launcher Pod arrangement to manage vLLM instances. Depending on available state and placement, it can wake a sleeping instance or start one through a launcher that has already loaded Python modules. Its hot path depends on matching the retained instance to the assigned GPU. FMA guide.
llm-d also documents ModelExpress for transferring weights from a seed replica to peers using NIXL and remote direct memory access (RDMA) over a compatible network. This changes weight distribution while leaving other engine-startup work to the receiving instance. These three paths illustrate why a platform’s startup work cannot be represented by a single checkpoint feature. ModelExpress integration.
Where Kubernetes checkpoint and restore stands
The Kubernetes Checkpoint Restore Working Group provides a place to coordinate use cases, API proposals, and integration with SIGs and runtimes. Its charter also identifies checkpoint archive efficiency as an area to revisit. The group’s scope concerns platform orchestration; inference frameworks continue to develop their internal startup mechanisms independently. Working-group charter.
Kubernetes already has container checkpointing through the work associated with Kubernetes Enhancement Proposal (KEP) 2008, which reached beta in v1.30. KEP-5823 addresses Pod-level checkpoint and restore. It is marked implementable with a v1.38 alpha target; that milestone is a delivery target. Its initial boundaries exclude device state and cross-node restore. GPU support has a separate provisional proposal, KEP-6274, whose enhancement PR remains open and discusses device allocation and compatibility. Container checkpoint API, Pod-level proposal, GPU proposal.
There is concrete progress at the Container Runtime Interface, or CRI, boundary. The PR adding CheckpointPod and RestorePod merged on July 22, 2026. The current interface describes consistent capture, resumption after checkpointing, and cleanup on restore failure. Restore prepares containers before their execution starts. The Kubernetes orchestration and containerd implementation PRs remain open; CRI-O’s implementation PR is a draft. CRI API change, Kubernetes implementation, containerd implementation, CRI-O implementation.
CRI leaves checkpoint files and their layout to the runtime. OCI addresses a different concern: packaging and distribution. Its image specification supports generic artifacts, and its distribution specification provides registry transfer and referrers mechanisms. These facilities can carry checkpoint payloads without defining their execution-state semantics. OCI artifact guidance, OCI distribution specification.
The checkpoint-specific OCI image proposal, issue #962, remains open. Podman can create checkpoint images, checkpointctl can build OCI-compatible images from checkpoint archives, and containerd has its own checkpoint media types. These are existing packaging practices alongside an unresolved proposal for a common definition. Registry storage alone does not establish restore compatibility across runtimes or GPU environments. Checkpoint image proposal, checkpointctl, containerd media types.
Directions being explored
The roadmaps cover several remaining costs. vLLM’s Fast Start work includes weight-cache validation, tuning, process startup, profiling, and graph-capture questions. SGLang’s Fast Engine Recovery roadmap includes broader model coverage, memory transport, daemon lifecycle, and initialization beyond weight loading. Their task lists describe directions; individual implementations carry their own status and supported configurations.
Snapshot lifecycle proposals also explore how applications report that capture is safe and how they learn that restoration has completed. The Snapshot project’s application-hooks design discusses files, HTTP, and Unix signals for these transitions. The framework proposals approach the same boundary from the engine side. Their existence does not establish a shared protocol already adopted across projects. Application lifecycle hooks proposal.
At the platform and runtime layers, ongoing work concerns Pod and GPU contracts, checkpoint artifacts, and compatibility. For readers following this area, the useful questions remain concrete: which work is preserved, which component owns it, what restoration still requires, and which environments the implementation supports.
Further reading
- vLLM Fast Start roadmap: the direction beyond weight loading, including tuning, initialization, and validation.
- SGLang Fast Engine Recovery roadmap: weight-cache evolution and other startup costs under discussion.
- vLLM initialized-engine snapshot design: capture boundaries, compatibility checks, and restoration validation.
- External checkpoint orchestration RFC: proposed responsibilities of the inference engine and the external orchestrator.
- OCI checkpoint image definition proposal: artifact layout, metadata, and interoperability questions.
- Kubernetes Checkpoint Restore Working Group: community scope, meeting notes, and entry points for following platform work.