Choosing a context size
The context size (sometimes called n_ctx or
the context window) is how many tokens of prompt + reply a model can hold
at once. Bigger windows fit longer documents and conversations but use
more memory; smaller windows free that memory for faster replies and more
parallel requests. This page covers how Pendra picks a window for you, and
how to change it.
Auto-tuning is the default
By default Pendra sizes each chat model's window automatically to fit that specific model in the hardware it detects, then clamps it to the model's trained context (asking for more than a model was trained on only wastes memory). Because it accounts for how much memory the model itself needs, a compact model gets a larger window than a heavy one on the same card. You don't have to set anything — auto-tuning picks a safe, useful window, and the console shows the value it settled on for each model running on your worker. The worker page also draws a memory map of your worker's memory — how much each running model is using, versus what's free — so you can see at a glance how much is left for context or for loading another model. It appears for every worker, whatever it serves (chat, image, or audio) and whatever its hardware uses: a discrete GPU's dedicated VRAM, the shared memory on an Apple Silicon or integrated GPU, or system RAM on a CPU-only worker. A model that runs on the CPU of a worker that has a GPU, as a speech model does when the GPU has too little free memory for it, is shown by name on the System memory bar with the memory it is using. It takes none of the GPU's memory, so it does not appear on the GPU's bar.
Auto-tuning sizes each model to the largest window your hardware can run reliably — it uses the memory the system can actually dedicate to the window, not just the total installed, so you get as much context as the hardware really supports rather than a cautious default.
It also learns as it goes, so a model keeps serving when a window doesn't work out. If memory turns out to be tighter than planned, the worker serves the request at the largest window it can set aside and tries the full size again about half an hour later. If a window proves too large for a model to run reliably, auto-tuning eases it down to a smaller, safe value and tries larger again later — the console shows the window it eased off from and when it'll try again. Either way a one-off wobble can't leave a model stuck small, and you don't have to babysit it.
For reference, the console also notes when your recent requests are consistently small — the smaller window that would cover them, handy if you'd rather free memory for more parallel requests. It's only a note: Auto keeps the larger window unless you pin the smaller one yourself.
Auto-tuning works out each model's real memory needs and sizes the window to what your hardware can actually hold. As a rough guide — the fallback when a model's exact needs can't be worked out ahead of time — the window grows with the memory it finds:
| Detected memory | Typical window |
|---|---|
| under 6 GB | ~4,096 |
| 6–12 GB | ~8,192 |
| 12–24 GB | ~16,384 |
| 24–48 GB | ~32,768 |
| 48 GB and up | up to 65,536 |
The exact window also depends on the model you're running, so treat these
as ballpark figures — the console always shows the real value. Some models
use memory far more efficiently than the table suggests, so on a big card
they can run windows well above the row you'd expect (for example
qwen3.6:27b on a 48 GB card runs far more than ~32,768, and
gemma4:12b gets the same treatment and runs well above its row
too). On NVIDIA
hosts auto-tuning scales each chat model's window up as the available GPU
memory allows — all the way to the full window the model was trained for
when the card has room. For example, qwen3.8:27b runs its full
262,144 tokens on a 96 GB card. If your worker serves more than one
model, a model only uses that extra room when the next model can still load
alongside it with its usual window; otherwise it keeps its usual window of
up to 65,536 tokens. Pinning a window isn't affected by this: pin one if you
want a model to have more. Very long prompts take a while to read before the first token
appears — minutes, at the top of a large window. On Apple Silicon
auto-tuning sizes the window to the memory the
system can safely dedicate to it — which grows well past a few thousand
tokens on a typical Mac, while staying within what Metal can run reliably.
CPU-only hosts are sized from system memory in the same way; a large
window there is slow rather than unsafe, so pin a smaller one if speed
matters more. Embedding models size their own window automatically and aren't
affected by this setting.
When to raise it
- Long documents — summarising or doing RAG over big files, where the whole input has to fit in one request.
- Agents and tools — multi-step runs that accumulate a long history of tool calls and results.
- Large system prompts — detailed instructions or few-shot examples that you prepend to every request.
Check it before you send. The Playground shows the window next to the worker picker, for example Context 8,192 tok, so you can tell whether a large paste (a CSV export, a long document) will fit before you run it. Your prompt and the reply have to fit in it together. A ~ in front of the number means the model isn't loaded right now, so the figure is an estimate and can come in lower if other models are using the memory; the exact window appears once it has served a request. With the picker on Auto, it shows the smallest window among the online workers serving that model, because your request can go to any of them. Owners and operators get a Change link that opens the worker's page, where you can pin a larger window for the model.
When to lower it
- Save memory — a smaller window frees memory for the model weights themselves, letting you run a larger model on the same card.
- More parallel requests — each in-flight request reserves part of the window, so a smaller window lets a worker serve more at once.
- Short prompts — chat or classification that never needs more than a few thousand tokens gains nothing from a big window.
Pinning a value — per model
Each model has its own window, so you can pin a large window on a big model and leave the rest on Auto. In the console, open Workers → your worker → Models & context (owner or operator), expand the model to open its settings, switch it to Custom, and choose any window up to 1,048,576 tokens. A pin is clamped down to that model's trained context and to what your worker's memory can actually hold — so a pin larger than the hardware can serve runs at the largest size that fits rather than failing to load. When that happens the model shows a VRAM-limited note with the window actually being served, so you can lower the pin or move the model to a worker with more memory. The change applies on the model's next request; an idle model reloads to pick it up. Switch the model back to Auto to return it to auto-tuning.
You can also set a pin from your own code, for example in a deployment script, with an API key that has the Manage workers permission:
curl -X PATCH https://api.pendra.ai/api/v1/workers/wrk-3f9b8a2d9e104c1a/models/qwen3.6:27b/config \
-H "Authorization: Bearer pdr_sk_..." \
-H "Content-Type: application/json" \
-d '{"context_size": 32768}'
Send 0 to switch the model back to Auto. See
Managing workers for how to
confirm the worker has applied it.
Pinning also overrides a window auto-tuning has eased down after an instability. Saving a pin serves the pinned window from the next request instead of the eased one — and if a pinned model is being served below its pin after an instability, its settings show Retry to try the full pin again straight away. The safety net stays on: if the pinned window proves too large, the worker eases it down again and tries the full pin later. Left alone, an eased model grows back to its pin (or its Auto size) by itself once it runs reliably, without a reload.
There's no worker-wide context setting and no config-file or environment variable — context is tuned per model, from the console or the API, which is the right granularity for a worker serving a mix of large and small models.
See worker configuration for the full settings reference and system requirements for the hardware behind the windows above.