KV prefix cache

The KV prefix cache is an opt-in worker setting that makes long, back-and-forth conversations reply faster. It's off by default; you turn it on per worker from that worker's page in the console.

What it does

Each time you send a chat request, the worker has to read through the whole conversation so far before it can start replying. In a long chat — a coding assistant, a support thread, anything with many turns — that work grows with every message, so each reply takes a little longer to start than the last.

With the KV prefix cache on, the worker keeps the conversation it just processed loaded in memory. On your next message it reuses everything the two requests have in common and only reads the new part, so it starts replying sooner. The reply itself is identical — this only changes how quickly it begins, not what the model says.

If you run more than one worker, Pendra automatically sends the follow-up messages of a conversation back to the worker that handled the earlier ones, so the warm copy is there to reuse. You don't have to pin anything or change how you call the API — keep sending the full conversation each turn as usual.

The trade-off

Keeping a conversation warm holds extra GPU memory on the worker, even while it's sitting idle between requests. That's the deal: faster multi-turn replies in exchange for higher GPU memory use. On a worker with memory to spare it's a clear win; on one that's already tight on VRAM, that memory may be better left free for a larger context window or more parallel requests. The worker's Models & context card in the console shows this directly: each model's memory bar breaks out a Context segment, so you can see how much the warm cache is holding and whether the worker still has room to spare.

When to turn it on

• Turn it on for chat or coding-assistant workloads — lots of multi-turn conversations — on a worker that has GPU memory headroom.
• Leave it off for mostly one-shot requests (single prompts, embeddings, batch jobs), where there's no prior conversation to reuse, or on a worker that's already memory-constrained.

How to enable it

Open the worker's page in the console, find Settings, and switch KV prefix cache on. It's a per-worker choice that only the org owner can change, and it takes effect on the worker's next chat request — no restart needed. Switch it back off at any time to free the memory again.