Speculative decoding
Speculative decoding makes a supported model generate faster without changing how it answers. There is nothing to switch on: your worker measures, for each model, whether it really is faster on that machine, and uses it only where it is. Each model's row in the console shows what the worker decided, and one setting per worker turns it off if you'd rather not use it.
What it does
Normally a model writes its answer one token at a time, checking back with itself after every single word-piece. With speculative decoding, the model is paired with a small, fast draft that runs ahead and proposes the next token or two. The full model then checks that proposal in a single step and keeps every token it would have written itself, discarding the rest. You always get the full model's answer, never the draft's — the draft only decides how many steps it takes to get there.
The worker decides for you
Whether a draft helps depends on the model and the hardware. On a large model the draft is tiny by comparison, so a correct guess saves a lot. On a small model that is already fast, running the draft can cost more than it saves. In our own testing on a single 24 GB GPU, Gemma 4 12B generated about 35–45% faster with it and the much smaller Gemma 4 E2B about 10–15% faster, while on an Apple laptop Gemma 4 E2B came out about level. Set up less favourably — the small model with a much longer draft — it came out 30% slower, which is the case the worker protects you from.
So the worker doesn't ask you to choose. It times each model both ways on the requests you are already sending, and then:
- uses speculative decoding for a model it measured as faster;
- leaves it off for a model it measured as no faster, so a small model is never slowed down;
- keeps checking — a request that turns out not to benefit finishes the ordinary way, and a model that is off is re-tried from time to time in case your traffic has changed.
The measurement starts again whenever the model is loaded, so a new GPU, a worker update or a different model size is picked up without you doing anything. The first few requests after a model loads are the ones being measured.
What the console shows
Open the worker's page in the console, find the Models & context list and expand a supported model's row. The Speculative decoding line is a status, not a setting (the one setting is per worker — see Turning it off):
| Status | Meaning |
|---|---|
| On, +24% | The worker measured this model as faster with speculative decoding and is using it. The figure is the measured speed-up in generation. |
| Off, −18% | The worker measured this model as no faster on this machine and isn't using it. The figure is what it would have cost. |
| Measuring | The worker is timing this model on its first requests. |
| Automatic | The model hasn't served a request since it loaded, so there is nothing measured yet. |
| Paused | The worker has Run requests together on. Requests served together don't use a draft. |
| Off (worker's Settings) | Someone in your organisation has switched it off for this worker — see Turning it off. |
| Off (by Pendra) | Pendra has switched it off for this worker, usually while looking into a problem with it. Your requests are served as normal, without the draft. Contact us if you need it back. |
| Starting | It begins within moments, or the next time the worker reconnects. |
| Needs a worker update | This worker's version doesn't decide automatically yet. Update the worker. |
Speculative decoding is being rolled out gradually. If a supported model shows no Speculative decoding line at all, it isn't available to your organisation yet.
Is the answer exactly the same?
It is always the model's own answer rather than the draft's, and it is
written with exactly the settings you sent — your temperature,
top_p, top_k, penalties and seed all
apply as usual. The draft only ever proposes; every token that reaches you is
one the full model chose for itself under your settings, so it cannot make a
model more repetitive, more random, or less careful.
It is not, however, guaranteed to be word-for-word identical to the same request served without it. Checking a couple of tokens in one pass and checking them one at a time are the same calculation, but a GPU doesn't produce bit-identical numbers for the two — so at a point where the model rates two possible next words as near-equal, the two can settle on different ones, and the wording carries on differently from there.
That's a difference in phrasing, not in quality — both are answers the model stands behind. But if you need a response you can reproduce byte-for-byte against an earlier run, turn it off for that worker.
When it applies
Speculative decoding applies to your ordinary chat traffic. It works at any
temperature, with top_p/top_k/
min_p, with frequency, presence and repetition penalties, with
logit_bias and a fixed seed, and on requests that
call tools or ask for JSON with
response_format — you don't have to send
temperature: 0 to benefit from it.
It works alongside the KV prefix cache — the worker setting, on by default, that lets a follow-up turn re-use the prompt it already holds in memory. The two do different jobs and you get both on every turn: the cache means a follow-up starts replying sooner, because the conversation so far doesn't have to be read again, and speculative decoding means the reply itself is written faster. That holds for the first turn of a conversation and for every turn after it.
How much it speeds a given request up depends on how predictable that request's text is. Low-temperature and code-like answers are the easiest for the draft to guess ahead on and speed up the most; a high-temperature creative answer is less predictable, so more of the draft is discarded and the gain is smaller. Either way the answer itself is unaffected — a draft that guesses badly costs a little speed, never quality, and the worker stops using it for a request where it isn't paying.
Continuous batching takes priority over it — that's the worker setting (Run requests together) that serves several in-flight requests together in one pass, so they don't queue behind each other. Those shared passes are served without a draft step, so the status reads Paused while it is on.
Which models support it
Speculative decoding is available on models that ship a matching draft — currently the Gemma 4 family (all sizes). When you install one of these on a worker, Pendra downloads its draft alongside the model. Models without a draft don't show a Speculative decoding line.
Seeing how well the draft is doing
Every reply that used speculative decoding reports the share of draft tokens
the model kept, as draft_acceptance_rate (0 to 1), next to
tokens_per_second:
- in the reply itself, under
pendra.timing; - on each row of the usage log.
A request that didn't use it has no draft_acceptance_rate (it is
null in the usage log). A higher rate means the draft guessed
more of the reply, but it is not the whole story: a small model can keep
over half of its draft and still be slower, which is why the worker decides
on measured speed rather than on this number.
The trade-off in memory
The draft is small, but it does use a little extra GPU memory while the model is loaded. On a worker with memory to spare that's cheap; on one that's already tight on VRAM, that memory may be better left free for a larger context window or more parallel requests.
Turning it off
You shouldn't need to: a model that doesn't benefit is left alone automatically. If you want it off regardless — for byte-for-byte reproducible replies, say — open the worker's page in the console, go to the Settings tab and set Speculative decoding to Off. It applies to every model on that worker from its next request, with no restart, and stays off until you set it back to Automatic. Owners and operators can change it.
The setting tells you when choosing wouldn't change anything:
- while Run requests together is on, Automatic is not in use, because requests served together don't use a draft;
- if Pendra has switched it off for the worker, it stays off whichever you choose. Your choice is kept and applies once Pendra switches it back;
- a worker on an older version asks you to update it first.
From the API
The same setting, with an API key that has the Manage workers permission:
curl -X PATCH https://api.pendra.ai/api/v1/workers/$WORKER_ID/speculative-decoding -H "Authorization: Bearer $PENDRA_API_KEY" -H "Content-Type: application/json" -d '{"mode": "off"}'
mode is "off" or "auto". A
202 means the change was sent to the worker. You get
409 if the worker needs an update before it takes the setting,
503 if it isn't connected, and 404 if speculative
decoding isn't available to your organisation yet. Each worker in
GET /api/v1/workers
reports your choice as speculative_pinned_off
(true for Off, null for a worker that needs an
update first).
In the worker's settings file
You can also add this to the worker's policies.json file and
restart the worker, which works on a worker too old to take the console
setting as long as it shows a speculative status. The file is in the
same folder as config.yaml
(Configuration says where that is
on your system), and pendra config prints the folder:
{
"speculative_decoding_disabled": true
}Keep any other settings already in the file. It is the same switch as the console's, so the console then shows Off for the worker. Remove the line and restart to go back to automatic, or set it back from the console once the worker is on a version that takes the setting.