KV cache

The KV cache is an opt-in worker setting that makes long, back-and-forth conversations reply faster. It's off by default; you turn it on per worker from that worker's page in the console, choosing between Memory and Memory + NVMe.

What it does

Each time you send a chat request, the worker has to read through the whole conversation so far before it can start replying. In a long chat — a coding assistant, a support thread, anything with many turns — that work grows with every message, so each reply takes a little longer to start than the last.

With the KV cache on, the worker keeps the conversation it just processed loaded in memory. On your next message it reuses everything the two requests have in common and only reads the new part, so it starts replying sooner. The reply itself is identical — this only changes how quickly it begins, not what the model says.

If you run more than one worker, Pendra automatically sends the follow-up messages of a conversation back to the worker that handled the earlier ones, so the warm copy is there to reuse. You don't have to pin anything or change how you call the API — keep sending the full conversation each turn as usual.

Keeping it across restarts

The warm copy lives in GPU memory, so it disappears whenever that memory is reclaimed — when the worker loads a different model, when a model is unloaded after sitting idle, or when the worker restarts. The next message then pays the full cost of reading the conversation again, which on a long prompt can mean waiting seconds before the reply starts.

Setting the KV cache to Memory + NVMe also writes the warm copy to the worker's own disk. When a later message starts with the same text — the same system prompt, the same document, the same opening turns of a chat — the worker loads it back from disk in a fraction of the time it would take to read it again, even if the model was swapped out or the worker was restarted in between.

This is worth turning on when your prompts share a long, stable opening: a large system prompt, a style guide, a contract or codebase you ask repeated questions about. It does the most good on a fast SSD, which is where the speed-up comes from. The worker keeps the cache under a size cap (250 GB by default, which you can change per worker in the console) and drops the least recently used entries first, so it won't grow without bound.

Two things it deliberately doesn't do. It only helps when a later prompt starts with text the worker has seen before — changing something near the beginning means there's nothing to reuse. And it's skipped automatically for a few model families whose memory can't be saved and restored safely, so those models keep working exactly as they do today, just without the disk speed-up.

The trade-off

Keeping a conversation warm holds extra GPU memory on the worker, even while it's sitting idle between requests. That's the deal: faster multi-turn replies in exchange for higher GPU memory use. On a worker with memory to spare it's a clear win; on one that's already tight on VRAM, that memory may be better left free for a larger context window or more parallel requests. The worker's Models & context card in the console shows this directly: each model's memory bar breaks out a Context segment, so you can see how much the warm cache is holding and whether the worker still has room to spare.

Memory + NVMe costs disk space rather than GPU memory — the saved copies can run to several gigabytes each on a large context. The console shows how much the cache is currently using, and the worker keeps the total under the size cap — 250 GB by default, which you can raise or lower per worker in the console — so it can't fill the disk.

Switching away from Memory + NVMe stops saving and loading, but deliberately keeps what's already saved, so switching back doesn't mean re-reading every prompt from scratch. The console keeps showing how much that is; delete the worker's cache directory if you want the space back.

When to turn it on

• Memory for chat or coding-assistant workloads — lots of multi-turn conversations — on a worker that has GPU memory headroom.
• Memory + NVMe when your requests share a long, stable opening (a big system prompt, a document you ask repeated questions about), when the worker serves several models and swaps between them, or when you want long prompts to stay fast after a restart. Best on a fast SSD with room to spare.
• Off for mostly one-shot requests (single prompts, embeddings, batch jobs), where there's no prior conversation to reuse, or on a worker that's already memory-constrained.

How to enable it

Open the worker's page in the console, find Settings, and set KV cache to Memory or Memory + NVMe. It's a per-worker choice that owners and operators can change, and it takes effect on the worker's next chat request — no restart needed. Set it back to Off at any time to free the memory again.

By default the worker stores the disk copies under its own configuration directory (~/.pendra/kvcache), capped at 250 GB. To put them on a different drive — a fast scratch NVMe, say — start the worker with PENDRA_KV_CACHE_DIR pointing at it (see configuration). pendra config on the worker prints the directory and cap in effect.