KV cache

The KV cache is a worker setting that makes long, back-and-forth conversations reply faster. On a worker you run it starts on, set to Memory + NVMe — kept both in GPU memory and on disk. You can change it, or turn it off, per worker from that worker's page in the console.

Workers that Pendra operates start on Memory instead, so no prompt content is written to a disk Pendra runs unless somebody turns the on-disk tier on for that worker. To make up for it they keep more conversations in system memory (80% of it rather than half — see Saved conversations in system memory). GPUs you rent through Pendra are yours alone and start on Memory + NVMe like any other worker you run.

What it does

Each time you send a chat request, the worker has to read through the whole conversation so far before it can start replying. In a long chat — a coding assistant, a support thread, anything with many turns — that work grows with every message, so each reply takes a little longer to start than the last.

With the KV cache on, the worker keeps the conversation it just processed loaded in memory. On your next message it reuses everything the two requests have in common and only reads the new part, so it starts replying sooner. The reply itself is identical — this only changes how quickly it begins, not what the model says.

If you run more than one worker, Pendra automatically sends the follow-up messages of a conversation back to the worker that handled the earlier ones, so the warm copy is there to reuse. You don't have to pin anything or change how you call the API — keep sending the full conversation each turn as usual.

Saved conversations in system memory

The GPU holds one conversation per model at a time (more with Run requests together). The moment a different conversation, or a different model, needs that room, the worker copies the conversation it was holding into the machine's system memory before letting it go. When that conversation comes back, the worker copies it back instead of reading the whole thing again — a fraction of a second instead of many seconds on a long chat.

This is what makes a worker fast when it serves many people at once: several users of the same app share a system prompt, take turns, and each gets their own conversation back rather than just the shared opening. The saved copies also survive the model being swapped out and unloaded after sitting idle, so the first message after a quiet spell is fast too.

You don't size it. By default the worker may use up to half of the memory it can use for these copies — the machine's RAM, or the container's memory limit if you run it in Docker with --memory — and it keeps a safety margin free for everything else on the machine. When another program needs memory, the worker gives the oldest copies back on its own within a couple of seconds. On a Mac, where models and saved copies share the same memory, the worker also gives copies back before it would unload a model to make room. Workers that Pendra operates use up to 80%, because they don't keep copies on disk. To change it for a worker, set RAM cache in the worker's settings to a different share, back to Automatic, or Off.

On a machine that is already short of memory — a 16 GB Mac with a model loaded, for example — there may be nothing to spare above that safety margin. The worker then saves nothing to system memory until memory frees up, and its settings say Not enough free memory for the RAM cache right now. With Memory + NVMe, shorter conversations may still be saved to disk while there is room to copy them.

Copies in system memory are lost when the worker restarts, and a copy nobody has used for 24 hours is dropped. With Memory + NVMe the same copies are also saved to disk (that is what lets them survive a restart); with Memory the worker never writes them to disk, but on a machine with swap enabled the operating system may page memory out like any other program's — if nothing from a conversation may ever touch a disk, use Memory and run the worker with swap off.

Keeping it across restarts

The warm copy lives in GPU memory and the saved copies in system memory, so a restart loses both. The next message then pays the full cost of reading the conversation again, which on a long prompt can mean waiting seconds before the reply starts.

That's why every worker ships on Memory + NVMe, which also writes the warm copy to the worker's own disk. When a later message starts with the same text — the same system prompt, the same document, the same opening turns of a chat — the worker loads it back from disk in a fraction of the time it would take to read it again, even if the model was swapped out or the worker was restarted in between.

It pays off most when your prompts share a long, stable opening: a large system prompt, a style guide, a contract or codebase you ask repeated questions about. It does the most good on a fast SSD, which is where the speed-up comes from. The worker keeps the cache under a size cap and drops the least recently used entries first, so it won't grow without bound. You don't have to pick that cap: each worker sizes one to its own disk — a fifth of the drive, up to 250 GB — and the console shows the number it chose. (In Docker, that's the drive the worker's volume lives on — see how the container uses disk.) Set your own per worker if you'd rather, and enter 0 to hand it back to the worker and have it sized to the disk again. On a small drive — one where that fifth would come to less than 8 GB — the worker doesn't run the disk tier at all, because entries that size would be swept away as fast as they were written. The console says so when that happens. On the rare host where the worker can’t read the drive’s size at all it falls back to 250 GB, and the console labels that number a default rather than one sized to your drive — so you know to set your own.

It only helps when a later prompt starts with text the worker has seen before — changing something near the beginning means there's nothing to reuse. It works with the small chat models too, so after a restart a long conversation on one of them resumes from its last saved point rather than from scratch.

Keep changing values out of the cached part of your prompt

The cache matches your prompt against what it already holds from the beginning, and stops at the first difference. Everything after that point is processed again from scratch — even if it is identical to last time.

That makes one mistake unusually expensive: putting something that changes on every request near the start of your prompt. A current timestamp, a session or request id, or a per-turn random value in your system prompt will end the match within the first few tokens, so a 100,000-token conversation re-processes all of it, every turn. Nothing looks broken — the cache is on, and it reports almost no hits.

The fix is free: put anything that varies after the part of your prompt that stays the same. A system prompt, tool definitions and the conversation so far are stable and belong first; the current time belongs in the latest user message, not above it.

Pendra spots this for you. When a prompt would have matched the cache again just past a short changing section, the worker logs prefix cache: reuse lost to an early divergence along with how many tokens it had to reprocess — so you can tell "my prompts can't be cached" apart from "my prompts can be cached and something small is preventing it".

With Run requests together switched on, saved conversations are kept the same way: in system memory first, and on disk when system memory needs the room, so they survive a model swap and a restart like any other. (Workers on releases older than this one cleared them when the model was unloaded; the console shows a warning on those.)

The trade-off

Keeping a conversation warm holds extra GPU memory on the worker, even while it's sitting idle between requests. That's the deal: faster multi-turn replies in exchange for higher GPU memory use. On a worker with memory to spare it's a clear win; on one that's already tight on VRAM, that memory may be better left free for a larger context window or more parallel requests. The worker's Memory & storage card in the console shows this directly: each memory bar breaks out a KV cache segment, so you can see how much the warm cache is holding and whether the worker still has room to spare.

Some of that cache lives in the worker's system memory rather than on the GPU — saved conversations the worker keeps to one side so that picking one back up costs a copy instead of re-reading the whole prompt. The System memory bar breaks that out too, so you can tell the cache's share of the machine's RAM from everything else running on it. If the bar shows usage not reported by this worker, that worker is on an older release — update it and the figures appear.

Memory + NVMe costs disk space rather than GPU memory — the saved copies can run to several gigabytes each on a large context. The console shows how much the cache is currently using, and the worker keeps the total under the size cap it sized to your disk (which you can raise or lower per worker in the console), so it can't fill the drive.

Switching away from Memory + NVMe stops saving and loading, and deletes what the worker has already saved. Switching back means re-reading each prompt once before the cache starts paying off again. If you only want to pause the saving for a moment, that is the cost to weigh.

Moving a worker back to an earlier release? Switch it away from Memory + NVMe first and wait until the console shows the on-disk cache as empty, then install the earlier release. An earlier release may not recognise copies saved by a later one, and would leave them on disk instead of reclaiming the space.

Saved copies are removed when they have not been read from disk for 24 hours, when the cache reaches its size limit (least recently used first), and when you switch the on-disk tier off. You do not have to clear the cache directory by hand to reclaim the space. A conversation the worker has been serving from memory the whole time doesn't count as a read, so its saved copy can expire while the conversation is still live — it is re-saved the next time the worker does real work on that prompt.

Seeing what the cache is saving you

Next to Memory & storage on the worker's page there is a Cache breakdown tab. It splits the cache by where it is kept — the live conversation, saved copies in system memory, and the on-disk tier — and shows two things for each: how much it is holding right now, and how many tokens it has reused instead of working out again.

Read the reused figure first. It is the one that tells you whether the tier is earning its space: one long prompt served from the cache is worth many short ones, so the token count prices the cache against the work it saved far better than a percentage does. The NVMe row also shows how many of its lookups found something, as a second line.

Three things to know when reading it. The figures count from the last time the worker started, so a number that drops means the worker restarted — it is not a fault. They are measured differently from the Cache percentage in the strip at the top of the page, which covers only the requests listed on that page, so the two will not add up. And on a worker on an older release with Run requests together switched on, the on-disk row is what the engine is keeping to one side while it runs rather than a saved copy that outlives it — the row says so.

Every row reports what it reused. The System RAM row also shows how much it may hold right now next to what it is holding — that limit drops when other programs on the machine need memory — and, like the on-disk row, how many of its lookups found something. A worker on an older release shows a dash where it doesn't measure a figure — update it and the figure appears. A dash always means “not measured”, never zero. On a Mac, or any machine where the graphics chip shares memory with the rest of the system, you'll see two rows instead of three — there is only one pool of memory to report.

Conversation memory density

A separate control on the same card changes how densely the worker stores that conversation memory, rather than whether it keeps it:

• Standard — full precision. The default, and what every worker did before this setting existed.
• Compact — about half the GPU memory per word, so roughly twice as much conversation fits on the same card. The room can go to a longer context window, to more parallel requests, or to a larger model.

It applies to every request, not just cached ones, so it's worth considering even with the KV cache Off — a long prompt costs GPU memory while it's being answered either way. The trade is a small amount of extra work per word to pack and unpack the memory; on most workers that is not noticeable next to the extra context it buys, but it's worth measuring against your own prompts before leaving it on.

Two things to expect. A few model families keep using Standard automatically — their memory use is measured rather than calculated, and storing it more densely would make that measurement wrong — so switching to Compact may leave some of your models unchanged. And switching either way makes anything already saved by the NVMe tier unreadable, so the next request after a switch re-reads its prompt once before the cache is useful again.

Which setting to use

Every worker starts on Memory + NVMe, which suits most workloads. Here is what each of the three settings is best for:

• Memory for chat or coding-assistant workloads — lots of multi-turn conversations — on a worker that has GPU memory headroom.
• Memory + NVMe when your requests share a long, stable opening (a big system prompt, a document you ask repeated questions about), when the worker serves several models and swaps between them, or when you want long prompts to stay fast after a restart. Best on a fast SSD with room to spare.
• Off for mostly one-shot requests (single prompts, embeddings, batch jobs), where there's no prior conversation to reuse, or on a worker that's already memory-constrained.

Private inference and the disk tier

Requests sent with private inference never touch the disk tier. Their conversation memory is kept in the worker's GPU and system memory only and is never written to the worker's drive, so nothing from an encrypted conversation is left at rest. Those requests still get the in-memory speed-up; only the on-disk copies are skipped. You don't need to change any setting for this — it applies whatever the worker's KV cache is set to.

How to change it

Open the worker's page in the console, find Settings, and set KV cache to Off, Memory or Memory + NVMe. It starts on Memory + NVMe. It's a per-worker choice that owners and operators can change, and it takes effect on the worker's next chat request — no restart needed. Set it to Off at any time to free the memory again: switching the cache off also empties the saved copies in system memory.

RAM cache, in the same settings, sets how much of the worker's system memory the saved copies may use (see above). It takes effect immediately. pendra config and pendra doctor on the worker print the share in effect.

By default the worker stores the disk copies under its own configuration directory (~/.pendra/kvcache). To put them on a different drive — a fast scratch NVMe, say — start the worker with PENDRA_KV_CACHE_DIR pointing at it (see configuration). pendra config on the worker prints the directory and cap in effect.

When the cache is on its own drive, the console shows it that way. The Memory & storage card on the worker page draws the drive your model files are on; a cache living somewhere else is reported beside it on its own line, naming the directory and that drive’s size, instead of being counted against the model-files drive. Freeing space on one drive doesn’t give the other more room, and the console no longer implies it does.

Running the worker in Docker? Give it a volume. A container with no volume writes the cache — and your model files — inside the container itself, where they still take up space on the host disk but are thrown away the moment you recreate the container. Mount one at the worker's data directory (/root/.pendra) and both survive an upgrade, as the examples in Install the worker do.