Chat completions

Where a model's authors publish recommended sampling settings, Pendra uses them for any parameter you don't set. Anything you send overrides them. Each model's settings are listed as sampling_defaults in the models list.

Other optional fields

• top_p, top_k, min_p, max_tokens, stop — standard OpenAI sampling controls. max_completion_tokens, OpenAI's newer name for max_tokens, is accepted too and wins if you send both. If you send neither, the reply stops at 32,768 tokens, thinking included, with the truncated_at_default_max_tokens notice below.
• frequency_penalty, presence_penalty, repeat_penalty, logit_bias — repetition and token-bias controls. repeat_penalty is a classic repetition penalty (1.0 = off; e.g. 1.1–1.3 discourages repeating recent tokens), and repeat_last_n sets how many recent tokens it looks back over (default 64).
• logprobs, top_logprobs — request token log-probabilities (model permitting).

Any other standard OpenAI Chat Completions field is forwarded to the serving worker as-is. Whether a given field takes effect depends on the model serving the request.

Response

Non-streaming responses come back as a single OpenAI-shaped chat.completion object (see example). usage is always populated; finish_reason is "stop" on a natural finish or "length" when capped by max_tokens.

For a reasoning model, the chain-of-thought comes back on message.reasoning_content (mirrored on message.reasoning), separate from the answer in message.content — streamed as delta.reasoning_content. See Thinking.

When you send follow-up turns of the same conversation, Pendra reuses the part of the prompt it already processed instead of reprocessing it from scratch. usage.prompt_tokens_details.cached_tokens reports how many of that request's prompt_tokens were served from this cache — the same field OpenAI returns, so a cache hit rate of cached_tokens / prompt_tokens is comparable. It's present only when some tokens were reused (a first, cold request omits it). A higher rate means faster time-to-first-token on that request.

Streaming

With stream: true, Pendra returns Server-Sent Events matching OpenAI's format: each event is a data: { ... } line containing a delta chunk, terminated by data: [DONE]. Pendra flushes each chunk immediately, so tokens arrive as they're generated.

curl https://api.pendra.ai/api/v1/chat/completions \
  -H "Authorization: Bearer pdr_sk_..." \
  -N \
  -d '{
    "model": "qwen3.6:27b",
    "stream": true,
    "messages": [{"role": "user", "content": "Hello"}]
  }'

stream = client.chat.completions.create(
    model="qwen3.6:27b",
    stream=True,
    messages=[{"role": "user", "content": "Hello"}],
)
for event in stream:
    if not event.choices:  # usage / timing chunk — no text
        continue
    print(event.choices[0].delta.content or "", end="")

const stream = await client.chat.completions.create({
  model: 'qwen3.6:27b',
  stream: true,
  messages: [{ role: 'user', content: 'Hello' }],
});
for await (const event of stream) {
  process.stdout.write(event.choices[0]?.delta?.content ?? '');
}

Each chunk is an OpenAI-shaped chat.completion.chunk. The last chunks of a stream report on the request rather than continuing the answer: one carries usage (Pendra always sets stream_options.include_usage = true server-side) and one carries pendra.timing. Both have an empty choices array — so read the text with something that tolerates that, e.g. chunk.choices[0]?.delta?.content in JavaScript or a if not chunk.choices: continue in Python, rather than indexing choices[0] unconditionally.

data: {"id":"chatcmpl-9f2b","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-9f2b","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-9f2b","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":8,"completion_tokens":2,"total_tokens":10}}

data: {"id":"chatcmpl-9f2b","object":"chat.completion.chunk","choices":[],"pendra":{"timing":{"ttft_ms":115,"tokens_per_second":58.34,"queue_wait_ms":16600,"worker_queue_wait_ms":14100}}}

data: [DONE]

Keep-alive while the model starts

A long prompt takes a moment to read before the first token appears, and on a busy worker that can run to several minutes. So that nothing between you and Pendra drops a connection that looks idle, Pendra sends an SSE comment every 15 seconds until the first real chunk arrives, and again during any long gap:

: keep-alive

: keep-alive

data: {"id":"chatcmpl-9f2b","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

A comment line carries no event and no data, so the OpenAI SDKs — and any other Server-Sent Events client — drop it for you and you'll never see it. You only need to know it's there if you parse the stream by hand: skip lines starting with : rather than treating them as a malformed chunk.

Errors on a stream

If Pendra can tell up front that it will never be able to answer, you get a normal HTTP error before the stream starts — the same status and detail message a non-streaming call would give you. That covers a model id that doesn't exist, a model no connected worker has installed, a model that doesn't do chat (a reranking model, say), a worker you pinned with X-Pendra-Worker-Id that can't serve the request, and an encrypted request sealed to a key none of your workers holds any more (a 503 with X-Pendra-Error-Code: no_private_inference_worker — fetch your organisation's key again and retry. The Pendra SDKs re-fetch it for you on the next call, unless you pinned the key, in which case they report the error so you can decide whether to trust the new one). Handle these exactly as you handle any other error: check the status code first. If none of your workers are online at all you get a 503 with a Retry-After header the same way — starting a worker is the thing to fix first.

One failure can't be known up front: every worker for your model is busy (a 429). Pendra waits a few moments for a slot to free up, and by then the stream has already started with a 200 — so it arrives as a normal data: frame carrying an error object (with a message, a type, and the HTTP code it would have been), followed by data: [DONE]. A 503 arrives the same way in the rare case that the busy worker goes offline while your request is waiting for it. The error can't use response headers at that point, so on a capacity 429 the honest back-off rides in the frame itself: retry_after (seconds to wait, sized to how backed up the model is) and queue_depth (how many requests were already waiting). Back off for retry_after with a little jitter, then retry.

data: {"error":{"message":"All workers serving 'qwen3.5:2b' are at capacity. Retry shortly, or add additional workers to handle more concurrent requests.","type":"rate_limit_exceeded","code":429,"retry_after":7,"queue_depth":9}}

data: [DONE]

A non-streaming request that runs long enough for Pendra to start keeping the connection alive (about 80 seconds) is in the same position — its status line is already sent — so a late failure there carries the identical error object in its JSON body. See Errors.

Web tool steps

When the worker serving your request has Web tools enabled, the model can fetch pages and run web searches while it answers. Each of those steps is reported back on the response under a Pendra-specific pendra field, so you can show what the model looked at. It sits alongside choices and is safe to ignore — the OpenAI SDKs skip unknown fields, so your existing code is unaffected.

Non-streaming responses carry the full list as pendra.web_tool_steps; streaming responses emit one pendra.web_tool_step event per step (a "call" when the model asks, then a "result" once it runs) on a chunk whose choices delta is empty. Each step has: name (web_fetch or web_search), arguments, ok, a truncated result preview, result_chars (the full length), and error when a step failed.

{
  "choices": [{ "index": 0, "message": { "role": "assistant", "content": "…" }, "finish_reason": "stop" }],
  "usage": { "prompt_tokens": 812, "completion_tokens": 43, "total_tokens": 855 },
  "pendra": {
    "web_tool_steps": [
      { "index": 0, "name": "web_fetch", "arguments": "{\"url\":\"https://example.com\"}",
        "ok": true, "result": "Example Domain…", "result_chars": 129 }
    ]
  }
}

Notices

The same pendra field also carries the occasional advisory about a response, under pendra.notice — a stable code worth switching on, and a human-readable message that already says what to do. A response carries at most one:

• truncated_during_reasoning — a reasoning model spent its whole max_tokens budget thinking and returned no answer (content empty, finish_reason: "length"). Retry with a larger budget or with thinking off (see Thinking).
• reasoning_budget_reached — the model's thinking used up its thinking budget, so it was ended there and the model went on to answer. The reply is complete and finish_reason is whatever the answer ended with; raise the budget if you want the model to think for longer.
• truncated_at_reasoning_budget — the thinking used up its budget on a model that can't be moved on to its answer part-way through, so the reply stopped there with no answer (finish_reason: "length"). Raise the budget, or turn thinking off.
• truncated_at_default_max_tokens — you set no max_tokens and the reply reached Pendra's default output limit, so it stopped there (finish_reason: "length"). The message gives the limit. Set max_tokens to allow a longer reply.
• truncated_during_structured_output — you asked for JSON with response_format and generation stopped before the JSON was complete, so the reply won't parse. Retry with a larger max_tokens, or a shorter prompt if the context window ran out (see Structured outputs).
• context_window_exhausted — the reply stopped because the prompt and the reply together filled the model's context window on that worker, not because max_tokens ran out (both end with finish_reason: "length", and only this notice tells them apart). The message gives the prompt and reply token counts and the window. Raise the model's context window on the worker (see Context size), or shorten the prompt — a larger max_tokens won't help.
• structured_output_not_json — you asked for JSON with response_format, the model finished normally, and the reply still isn't JSON. On a reasoning model the constraint only takes effect once the model finishes thinking, so a reply that never got there comes back unconstrained — retry, or lower reasoning_effort (see Structured outputs). On a request that includes an image, response_format is not applied at all (see Requests with images). And on an ordinary text request it can mean the schema couldn't be enforced on that reply at all, where retrying won't help — ask for the JSON in the prompt too, or simplify the schema (see When the schema wasn't enforced). The notice message says which applies.
• strict_schema_not_enforced — you set response_format.json_schema.strict but part of the schema couldn't be held to, so validate the reply before relying on it (see Structured outputs).

Streaming responses emit the notice on a chunk whose choices delta is empty, just before the final chunk. Like everything under pendra, it's safe to ignore — OpenAI SDKs skip unknown fields.

Timing

Replies report their latency and throughput under pendra.timing, so you can log them per request without a second call. This is the reliable place to read them: the Server-Timing header carries the same figures, but only on non-streaming replies that finish before Pendra starts keeping the connection alive, which it does after about 80 seconds.

A non-streaming reply carries it in the body, next to choices:

{"id":"chatcmpl-9f2b","object":"chat.completion","choices":[...],
 "pendra":{"timing":{"ttft_ms":115,"tokens_per_second":58.34,
                     "queue_wait_ms":16600,"worker_queue_wait_ms":14100}}}

A streaming reply carries it on the last chunk before [DONE]. That chunk has an empty choices array and no text:

data: {"id":"chatcmpl-9f2b","object":"chat.completion.chunk","choices":[],
       "pendra":{"timing":{"ttft_ms":115,"tokens_per_second":58.34,
                           "queue_wait_ms":16600,"worker_queue_wait_ms":14100}}}

Every field is present only when it was measured, and timing is left out when nothing was (an older worker reports no timings). With private inference the SDKs read timing for you on streaming replies only; on a non-streaming reply it's in the raw response body, next to the encrypted reply. On a stream the last chunk is still sent, because it also says which worker served the request. Like everything under pendra, it's safe to ignore.

Which worker served the request

Every reply says which of your workers ran it, under pendra.worker, together with the request's pendra.request_id. Use it to confirm where a request was placed (on the machine or site you expected, for example) or to trace a slow reply back to the worker that produced it.

A non-streaming reply carries it in the body, next to choices:

{"id":"chatcmpl-9f2b","object":"chat.completion","choices":[...],
 "pendra":{"worker":{"id":"wrk-1a2b3c4d","name":"london-gpu-01","version":"3.115.1"},
           "request_id":"3f1c8a2e-5b7d-4e19-9c0a-6d2f8b1e4a73"}}

A streaming reply carries it on the last chunk before [DONE], alongside pendra.timing. To learn the worker as the stream opens, read the response headers:

data: {"id":"chatcmpl-9f2b","object":"chat.completion.chunk","choices":[],
       "pendra":{"timing":{"ttft_ms":115,"tokens_per_second":58.34},
                 "worker":{"id":"wrk-1a2b3c4d","name":"london-gpu-01","version":"3.115.1"},
                 "request_id":"3f1c8a2e-5b7d-4e19-9c0a-6d2f8b1e4a73"}}

Non-streaming replies also name the worker in the response headers, but the body is the reliable place to read it: a request slow enough that Pendra keeps the connection alive for it has its headers sent before the answer exists. They name the worker that has the request at that moment, and none if it is still waiting for one. With private inference, the worker is reported alongside the encrypted reply rather than inside it; the Python and Node SDKs (0.17 and later) read it for you. If a worker fails the request, the error names it instead, as error.worker_id.

Response headers

Non-streaming chat responses carry these headers. A streaming response carries X-Request-Id, and X-Worker-Id and X-Worker-Name when a worker is free straight away, so you know where the stream is running before its first token. If the request has to wait for a busy worker (more than about a second), the stream starts without the two worker headers rather than keep you waiting. They name the worker the request was first sent to: if that worker drops out before the first token, Pendra moves the stream to another one, so the final chunk is the definitive record. It always names the worker, alongside the request id as pendra.request_id. A streaming response has no Server-Timing. A non-streaming reply slow enough that Pendra keeps the connection alive for it (after about 80 seconds) has its headers sent before the answer exists. It has no Server-Timing. Its X-Worker-Id, X-Worker-Name and X-Request-Id name the worker that has the request at that moment; if the request is still waiting for a busy worker, the two worker headers are left out and X-Request-Id doesn't identify the request. If that worker drops out afterwards and Pendra moves the request, the headers still name the first one. Every value here is also in the body under pendra, so read it from there if you need it on every request.

A non-streaming response includes a Server-Timing header so you can read the per-request latency and throughput without any extra call — handy when benchmarking or sizing concurrency. When you do, read pendra.timing from the body instead (or as well): the requests that queue longest are the ones most likely to pass the 80-second mark and arrive without the header, so a benchmark built on the header alone quietly leaves out the slowest ones.

Server-Timing: ttft;dur=115, tps;dur=0;desc="58.3", queue;dur=16600, worker_queue;dur=14100

ttft measures prompt processing only. If your request had to wait its turn, that wait is reported separately in queue rather than inflating ttft — so a slow first token and a busy queue stay distinguishable.

One exception: when the model uses web tools, it works in several steps and you see nothing until the first of them produces something. ttft then measures that whole wait — the model deciding to search, the search or page fetch itself, and re-reading the results — so it matches what you actually waited for rather than the first step in isolation.

The two queue figures tell you what to do about it. worker_queue is time spent waiting behind other requests on the worker that served you; the remainder (queue minus worker_queue) is time spent waiting for any worker with a free slot at all. In the example above, 14.1s was spent queued on the worker and 2.5s waiting for one to become available — the first points at that worker's concurrency settings, the second at needing more workers.

A streaming response can't carry this header — headers are sent before the first token — so streaming requests get the same figures on a pendra.timing chunk instead. Either way they're also recorded against each request in the console's Usage view.

Timeouts

A single non-streaming chat request can run up to ~30 minutes. While a slow or large model works, Pendra automatically keeps the connection alive, so you no longer need to switch to streaming just to avoid a timeout on a long generation. Such a reply arrives without the response headers, but the body still says which worker served it and how long it took. stream: true is still the best choice for interactive UIs — it shows partial tokens as they're generated rather than waiting for the whole reply.

OpenAI SDK compatibility

Point the OpenAI SDK at Pendra by setting OPENAI_BASE_URL=https://api.pendra.ai/api/v1 and OPENAI_API_KEY=pdr_sk_…. No other code changes needed.

The OpenAI convention https://api.pendra.ai/v1 works too, so if you already have a base URL ending in /v1 you can leave it as-is — chat/completions, embeddings, and models all resolve at both /api/v1 and /v1.