KV cache

The KV cache is a worker setting that makes long, back-and-forth conversations reply faster. It's on by default on every worker, set to Memory + NVMe — kept both in GPU memory and on disk. You can change it, or turn it off, per worker from that worker's page in the console.

What it does

Each time you send a chat request, the worker has to read through the whole conversation so far before it can start replying. In a long chat — a coding assistant, a support thread, anything with many turns — that work grows with every message, so each reply takes a little longer to start than the last.

With the KV cache on, the worker keeps the conversation it just processed loaded in memory. On your next message it reuses everything the two requests have in common and only reads the new part, so it starts replying sooner. The reply itself is identical — this only changes how quickly it begins, not what the model says.

If you run more than one worker, Pendra automatically sends the follow-up messages of a conversation back to the worker that handled the earlier ones, so the warm copy is there to reuse. You don't have to pin anything or change how you call the API — keep sending the full conversation each turn as usual.

Keeping it across restarts

The warm copy lives in GPU memory, so it disappears whenever that memory is reclaimed — when the worker loads a different model, when a model is unloaded after sitting idle, or when the worker restarts. The next message then pays the full cost of reading the conversation again, which on a long prompt can mean waiting seconds before the reply starts.

That's why every worker ships on Memory + NVMe, which also writes the warm copy to the worker's own disk. When a later message starts with the same text — the same system prompt, the same document, the same opening turns of a chat — the worker loads it back from disk in a fraction of the time it would take to read it again, even if the model was swapped out or the worker was restarted in between.

It pays off most when your prompts share a long, stable opening: a large system prompt, a style guide, a contract or codebase you ask repeated questions about. It does the most good on a fast SSD, which is where the speed-up comes from. The worker keeps the cache under a size cap and drops the least recently used entries first, so it won't grow without bound. You don't have to pick that cap: each worker sizes one to its own disk — a fifth of the drive, up to 250 GB — and the console shows the number it chose. Set your own per worker if you'd rather, and enter 0 to hand it back to the worker and have it sized to the disk again. On a small drive — one where that fifth would come to less than 8 GB — the worker doesn't run the disk tier at all, because entries that size would be swept away as fast as they were written. The console says so when that happens. On the rare host where the worker can’t read the drive’s size at all it falls back to 250 GB, and the console labels that number a default rather than one sized to your drive — so you know to set your own.

Two things it deliberately doesn't do. It only helps when a later prompt starts with text the worker has seen before — changing something near the beginning means there's nothing to reuse. And it's skipped automatically for a few model families whose memory can't be saved and restored safely, so those models keep working exactly as they do today, just without the disk speed-up.

Keep changing values out of the cached part of your prompt

The cache matches your prompt against what it already holds from the beginning, and stops at the first difference. Everything after that point is processed again from scratch — even if it is identical to last time.

That makes one mistake unusually expensive: putting something that changes on every request near the start of your prompt. A current timestamp, a session or request id, or a per-turn random value in your system prompt will end the match within the first few tokens, so a 100,000-token conversation re-processes all of it, every turn. Nothing looks broken — the cache is on, and it reports almost no hits.

The fix is free: put anything that varies after the part of your prompt that stays the same. A system prompt, tool definitions and the conversation so far are stable and belong first; the current time belongs in the latest user message, not above it.

Pendra spots this for you. When a prompt would have matched the cache again just past a short changing section, the worker logs prefix cache: reuse lost to an early divergence along with how many tokens it had to reprocess — so you can tell "my prompts can't be cached" apart from "my prompts can be cached and something small is preventing it".

One setting to watch: if you also have Run requests together switched on for the worker, saved conversations don't last as long. They're still written to disk, and they still survive a busy period where one conversation loses its place on the GPU to another — but they're cleared when the model is unloaded or the worker restarts. So on a batching worker the disk tier helps while a model stays loaded, and a restart starts from scratch. The console shows a warning on the KV cache control when the worker is in this state, and pendra doctor marks the row on the machine itself. If surviving restarts matters more to you than running several requests in the same pass, turn Run requests together off.

The trade-off

Keeping a conversation warm holds extra GPU memory on the worker, even while it's sitting idle between requests. That's the deal: faster multi-turn replies in exchange for higher GPU memory use. On a worker with memory to spare it's a clear win; on one that's already tight on VRAM, that memory may be better left free for a larger context window or more parallel requests. The worker's Memory & storage card in the console shows this directly: each memory bar breaks out a KV cache segment, so you can see how much the warm cache is holding and whether the worker still has room to spare.

Some of that cache lives in the worker's system memory rather than on the GPU — saved conversations the worker keeps to one side so that picking one back up costs a copy instead of re-reading the whole prompt. The System memory bar breaks that out too, so you can tell the cache's share of the machine's RAM from everything else running on it. If the bar shows usage not reported by this worker, that worker is on an older release — update it and the figures appear.

Memory + NVMe costs disk space rather than GPU memory — the saved copies can run to several gigabytes each on a large context. The console shows how much the cache is currently using, and the worker keeps the total under the size cap it sized to your disk (which you can raise or lower per worker in the console), so it can't fill the drive.

Switching away from Memory + NVMe stops saving and loading, but deliberately keeps what's already saved, so switching back doesn't mean re-reading every prompt from scratch. The console keeps showing how much that is; delete the worker's cache directory if you want the space back.

Conversation memory density

A separate control on the same card changes how densely the worker stores that conversation memory, rather than whether it keeps it:

• Standard — full precision. The default, and what every worker did before this setting existed.
• Compact — about half the GPU memory per word, so roughly twice as much conversation fits on the same card. The room can go to a longer context window, to more parallel requests, or to a larger model.

It applies to every request, not just cached ones, so it's worth considering even with the KV cache Off — a long prompt costs GPU memory while it's being answered either way. The trade is a small amount of extra work per word to pack and unpack the memory; on most workers that is not noticeable next to the extra context it buys, but it's worth measuring against your own prompts before leaving it on.

Two things to expect. A few model families keep using Standard automatically — their memory use is measured rather than calculated, and storing it more densely would make that measurement wrong — so switching to Compact may leave some of your models unchanged. And switching either way makes anything already saved by the NVMe tier unreadable, so the next request after a switch re-reads its prompt once before the cache is useful again.

Which setting to use

Every worker starts on Memory + NVMe, which suits most workloads. Here is what each of the three settings is best for:

• Memory for chat or coding-assistant workloads — lots of multi-turn conversations — on a worker that has GPU memory headroom.
• Memory + NVMe when your requests share a long, stable opening (a big system prompt, a document you ask repeated questions about), when the worker serves several models and swaps between them, or when you want long prompts to stay fast after a restart. Best on a fast SSD with room to spare.
• Off for mostly one-shot requests (single prompts, embeddings, batch jobs), where there's no prior conversation to reuse, or on a worker that's already memory-constrained.

Private inference and the disk tier

Requests sent with private inference never touch the disk tier. Their conversation memory is kept in GPU memory for the life of the request and is never written to the worker's drive, so nothing from an encrypted conversation is left at rest. Those requests still get the Memory tier's speed-up; only the on-disk copies are skipped. You don't need to change any setting for this — it applies whatever the worker's KV cache is set to.

How to change it

Open the worker's page in the console, find Settings, and set KV cache to Off, Memory or Memory + NVMe. It starts on Memory + NVMe. It's a per-worker choice that owners and operators can change, and it takes effect on the worker's next chat request — no restart needed. Set it to Off at any time to free the memory again.

By default the worker stores the disk copies under its own configuration directory (~/.pendra/kvcache). To put them on a different drive — a fast scratch NVMe, say — start the worker with PENDRA_KV_CACHE_DIR pointing at it (see configuration). pendra config on the worker prints the directory and cap in effect.

When the cache is on its own drive, the console shows it that way. The Memory & storage card on the worker page draws the drive your model files are on; a cache living somewhere else is reported beside it on its own line, naming the directory and that drive’s size, instead of being counted against the model-files drive. Freeing space on one drive doesn’t give the other more room, and the console no longer implies it does.

Running the worker in Docker? Give it a volume. A container with no volume writes the cache — and your model files — inside the container itself, where they still take up space on the host disk but are thrown away the moment you recreate the container. Mount one at the worker's data directory (/root/.pendra) and both survive an upgrade, as the examples in Install the worker do.