Keep a model warm

The first request to a model that isn't already in memory is slow: the worker has to load the model's weights into the GPU before it can answer, which can take anywhere from a second to half a minute depending on the model's size. This is a cold start. Keep warm loads the model ahead of time and keeps it resident, so that first request is as fast as the rest.

What it does

Installing a model only downloads it to the worker's disk — it isn't loaded into memory until a request actually needs it. On a worker that serves a few different models, an idle model can also be unloaded again to make room for others, so the next request to it cold-starts once more.

Turn Keep warm on for a model and the worker loads it into memory right away, without waiting for a request, and then keeps it there — it won't be unloaded to make room for another model. It's also reloaded automatically if the worker restarts, so it stays ready. The result is that requests to that model never pay the cold-start cost.

It works for every kind of model you can run on a worker — chat, embeddings, image generation, transcription, and speech — so whichever model your app calls first, that first call is fast.

One model per worker

A worker keeps one model warm at a time. While a model is kept warm, the control is unavailable for the other models on that worker — to keep a different one warm, switch the current one off first. If you want several models always-resident, give them their own workers.

The trade-off

A kept-warm model holds its GPU memory the whole time, even while it's idle. That's the deal: no cold starts in exchange for memory that stays reserved. On a worker with room to spare it's a clear win for a model you use regularly — so keep warm the one model that matters most on each worker, and let the rest load on demand.

Because a kept-warm model never gives up its memory, it leaves less room for other models on the same worker. If a request asks for a different model that won't fit in the GPU memory left around the kept-warm one, the worker turns that request down right away with a clear "doesn't fit" error (HTTP 507) instead of trying and failing — the kept-warm model keeps serving uninterrupted. If you need both models always available, give them their own workers.

The opposite: unload a model when it goes idle

Sometimes you want the reverse of keep warm: a model you use in bursts, but that shouldn't tie up GPU memory the rest of the time. Turn on Idle timeout for a model and the worker unloads it from the GPU once it hasn't received a request for the length of time you choose — freeing its memory for other models until the next request loads it again (paying one cold start at that point).

It's opt-in and set per model. Type the number of minutes and turn it on — anything from 10 minutes to 24 hours (it defaults to 60). A model with an idle timeout still loads on demand exactly as before — the only change is that a quiet spell now releases its memory instead of holding it indefinitely. It works for every model type: chat, embedding, image, transcription, and speech.

Idle timeout and Keep warm are opposites, so a model can have one or the other but not both — keep warm means "always loaded", idle timeout means "unload when quiet". The console shows only the one that's off as available; turn the active one off first to switch. Find Idle timeout in the same place as Keep warm: expand the model's row under Models & context — every model type uses the same panel.

Spotting a cold start

On the Usage page, any request that had to cold-load its model is flagged with a small warning marker, and hovering its duration shows a Model load time alongside the other timings — the part of the response time that went into loading the model rather than generating the answer. If you see cold starts on a model you call often, that's the signal to keep it warm.

How to enable it

Open the worker's page in the console and find the model under Models & context. Expand the model's row with the chevron on the right, and you'll find the Keep warm switch alongside the model's other settings — every model type uses the same panel. It's per model, and owners and operators can change it; one model per worker can be kept warm at a time. Switch it back off at any time to unload the model and free its memory again — it stays installed on disk and will simply cold-start on its next request.

Turning it off is safe to do while the worker is busy: any requests already running on that model are allowed to finish first, and only then is the memory released. Requests that arrive during those few seconds get a retryable error rather than a partial answer, so a client that retries (the official OpenAI and Anthropic SDKs do so automatically) won't notice.

A model that isn't currently being served on the worker can't be kept warm — there'd be nothing to answer. If the control is unavailable and no other model is warm, check whether serving is turned off for it.