Speculative decoding

Speculative decoding is an opt-in setting that makes a supported model generate faster. It's off by default; you turn it on per model from that worker's page in the console. It doesn't change how the model answers — you just get the answer sooner.

What it does

Normally a model writes its answer one token at a time, checking back with itself after every single word-piece. With speculative decoding on, the model is paired with a small, fast draft that runs ahead and proposes several tokens at once. The full model then verifies that batch in a single step and keeps every token it would have written itself, discarding the rest. You always get the full model's answer, never the draft's — the draft only decides how many steps it takes to get there.

Is the answer exactly the same?

Almost always, and it is always the model's own answer rather than the draft's. It is not, however, guaranteed to be word-for-word identical to the same request with the setting off. Checking a batch of tokens in one pass and checking them one at a time are the same calculation, but a GPU doesn't produce bit-identical numbers for the two — so at a point where the model rates two possible next words as near-equal, the two modes can settle on different ones, and the wording carries on differently from there.

That's a difference in phrasing, not in quality — both are answers the model stands behind. But if you need a response you can reproduce byte-for-byte against an earlier run, leave the toggle off for that model.

When it applies

Speculative decoding only kicks in for deterministic requests — those you send with temperature set to 0 (the model always picks its single most-likely next token). That's the common setting for coding, extraction, classification, and other tasks where you want a repeatable answer — and it's where the speed-up costs you nothing in answer quality.

For requests that use sampling (a temperature above 0), the worker quietly falls back to normal generation, so leaving the toggle on never changes a creative or higher-temperature response.

It also doesn't apply on a worker with continuous batching switched on — that's the worker setting that serves several in-flight requests together in one pass, so they don't queue behind each other. Those shared passes are served without a draft step, so speculative decoding has nothing to do. Pick one: continuous batching for high-volume parallel traffic, speculative decoding for the fastest single deterministic response. The console greys the Speculative toggle out and tells you why while continuous batching is on.

Which models support it

Speculative decoding is available on models that ship a matching draft — currently the Gemma 4 family (all sizes). When you install one of these on a worker, Pendra downloads its draft alongside the model and a Speculative toggle becomes available in that model's settings. Models without a draft simply don't show the toggle.

The trade-off

The draft is small, but it does use a little extra GPU memory while the model is loaded. On a worker with memory to spare this is a clear win for deterministic workloads; on one that's already tight on VRAM, that memory may be better left free for a larger context window or more parallel requests. How much speed-up you see depends on your hardware and the model.

How to enable it

Open the worker's page in the console and find the Models & context list. On a supported model, expand its row to open the settings and switch Speculative on. It's a per-model choice that owners and operators can change, and it takes effect on that model's next deterministic request — no restart needed. Switch it back off at any time.