Keep a model warm

The first request to a model that isn't already in memory is slow: the worker has to load the model's weights into the GPU before it can answer, which can take anywhere from a second to half a minute depending on the model's size. This is a cold start. Keep warm loads the model ahead of time and keeps it resident, so that first request is as fast as the rest.

What it does

Installing a model only downloads it to the worker's disk — it isn't loaded into memory until a request actually needs it. On a worker that serves a few different models, an idle model can also be unloaded again to make room for others, so the next request to it cold-starts once more.

Turn Keep warm on for a model and the worker loads it into memory right away, without waiting for a request, and then keeps it there — it won't be unloaded to make room for another model. It's also reloaded automatically if the worker restarts, so it stays ready. The result is that requests to that model never pay the cold-start cost.

One model per worker

A worker keeps one model warm at a time. While a model is kept warm, the control is unavailable for the other models on that worker — to keep a different one warm, switch the current one off first. If you want several models always-resident, give them their own workers.

The trade-off

A kept-warm model holds its GPU memory the whole time, even while it's idle. That's the deal: no cold starts in exchange for memory that stays reserved. On a worker with room to spare it's a clear win for a model you use regularly — so keep warm the one model that matters most on each worker, and let the rest load on demand.

Because a kept-warm model never gives up its memory, it leaves less room for other models on the same worker. If a request asks for a different model that won't fit in the GPU memory left around the kept-warm one, the worker turns that request down right away with a clear "doesn't fit" error (HTTP 507) instead of trying and failing — the kept-warm model keeps serving uninterrupted. If you need both models always available, give them their own workers.

Spotting a cold start

On the Usage page, any request that had to cold-load its model is flagged with a small warning marker, and hovering its duration shows a Model load time alongside the other timings — the part of the response time that went into loading the model rather than generating the answer. If you see cold starts on a model you call often, that's the signal to keep it warm.

How to enable it

Open the worker's page in the console, expand the model under Models & context, and switch Keep warm on. It's per model and only the org owner can change it; one model per worker can be kept warm at a time. Switch it back off at any time to unload the model and free its memory again — it stays installed on disk and will simply cold-start on its next request.