Speculative decoding

Speculative decoding is an opt-in setting that makes a supported model generate faster. It's off by default; you turn it on per model from that worker's page in the console. The reply you get is exactly the same — only the speed changes.

What it does

Normally a model writes its answer one token at a time, checking back with itself after every single word-piece. With speculative decoding on, the model is paired with a small, fast draft that runs ahead and proposes several tokens at once. The full model then verifies that batch in a single step and keeps every token it would have written itself, discarding the rest. Because the model only ever emits tokens it agrees with, the output is identical to running without it — you just get there in fewer steps.

When it applies

Speculative decoding only kicks in for deterministic requests — those you send with temperature set to 0 (the model always picks its single most-likely next token). That's the common setting for coding, extraction, classification, and other tasks where you want a repeatable answer, and it's exactly where the speed-up is free: the result is byte-for-byte the same as with the setting off.

For requests that use sampling (a temperature above 0), the worker quietly falls back to normal generation, so leaving the toggle on never changes a creative or higher-temperature response.

Which models support it

Speculative decoding is available on models that ship a matching draft — currently the Gemma 4 family (all sizes). When you install one of these on a worker, Pendra downloads its draft alongside the model and a Speculative toggle becomes available in that model's settings. Models without a draft simply don't show the toggle.

The trade-off

The draft is small, but it does use a little extra GPU memory while the model is loaded. On a worker with memory to spare this is a clear win for deterministic workloads; on one that's already tight on VRAM, that memory may be better left free for a larger context window or more parallel requests. How much speed-up you see depends on your hardware and the model.

How to enable it

Open the worker's page in the console and find the Models & context list. On a supported model, expand its row to open the settings and switch Speculative on. It's a per-model choice that only the org owner can change, and it takes effect on that model's next deterministic request — no restart needed. Switch it back off at any time.