Worker
Choosing a context size
The context size (sometimes called n_ctx or
the context window) is how many tokens of prompt + reply a model can hold
at once. Bigger windows fit longer documents and conversations but use
more memory; smaller windows free that memory for faster replies and more
parallel requests. This page covers how Pendra picks a window for you, and
how to change it.
Auto-tuning is the default
By default Pendra sizes each chat model's window automatically to fit that specific model in the hardware it detects, then clamps it to the model's trained context (asking for more than a model was trained on only wastes memory). Because it accounts for how much memory the model itself needs, a compact model gets a larger window than a heavy one on the same card. You don't have to set anything — auto-tuning picks a safe, useful window, and the console shows the value it settled on for each model running on your worker. The worker page also draws a memory map of your worker's memory — how much each running model is using, versus what's free — so you can see at a glance how much is left for context or for loading another model. It appears for every worker, whatever it serves (chat, image, or audio) and whatever its hardware uses: a discrete GPU's dedicated VRAM, the shared memory on an Apple Silicon or integrated GPU, or system RAM on a CPU-only worker.
Auto-tuning sizes each model to the largest window your hardware can run reliably — it uses the memory the system can actually dedicate to the window, not just the total installed, so you get as much context as the hardware really supports rather than a cautious default.
It also learns as it goes. If a window turns out to be too large for a model to run reliably, auto-tuning eases it down to a smaller, safe value to keep serving, then tries a larger window again later — so a one-off wobble can't leave a model stuck small, and you don't have to babysit it. When this happens the console shows the window it eased off from and when it'll try larger again.
For reference, the console also notes when your recent requests are consistently small — the smaller window that would cover them, handy if you'd rather free memory for more parallel requests. It's only a note: Auto keeps the larger window unless you pin the smaller one yourself.
As a rough guide, the window auto-tuning picks grows with the memory it finds:
| Detected memory | Typical window |
|---|---|
| under 6 GB | ~4,096 |
| 6–12 GB | ~8,192 |
| 12–24 GB | ~16,384 |
| 24–48 GB | ~32,768 |
| 48 GB and up | up to 65,536 |
The exact window also depends on the model you're running, so treat these as ballpark figures — the console always shows the real value. On NVIDIA hosts auto-tuning scales each chat model's window up as the available GPU memory allows. On Apple Silicon auto-tuning sizes the window to the memory the system can safely dedicate to it — which grows well past a few thousand tokens on a typical Mac, while staying within what Metal can run reliably. CPU-only hosts stay modest, since a large window there is slow rather than unsafe. Embedding models size their own window automatically and aren't affected by this setting.
When to raise it
- Long documents — summarising or doing RAG over big files, where the whole input has to fit in one request.
- Agents and tools — multi-step runs that accumulate a long history of tool calls and results.
- Large system prompts — detailed instructions or few-shot examples that you prepend to every request.
When to lower it
- Save memory — a smaller window frees memory for the model weights themselves, letting you run a larger model on the same card.
- More parallel requests — each in-flight request reserves part of the window, so a smaller window lets a worker serve more at once.
- Short prompts — chat or classification that never needs more than a few thousand tokens gains nothing from a big window.
Pinning a value — per model
Each model has its own window, so you can pin a large window on a big model and leave the rest on Auto. Pins are set from the console: open Workers → your worker → Models & context (owner only), expand the model to open its settings, switch it to Custom, and choose any window up to 1,048,576 tokens. A pin is clamped down to that model's trained context and to what your worker's memory can actually hold — so a pin larger than the hardware can serve runs at the largest size that fits rather than failing to load. When that happens the model shows a VRAM-limited note with the window actually being served, so you can lower the pin or move the model to a worker with more memory. The change applies on the model's next request; an idle model reloads to pick it up. Switch the model back to Auto to return it to auto-tuning.
There's no worker-wide context setting and no config-file or environment variable — context is tuned per model from the console, which is the right granularity for a worker serving a mix of large and small models.
See worker configuration for the full settings reference and system requirements for the hardware behind the windows above.