Choosing a model size

A model's size is its parameter count — the 4B, 9B, or 70B figure on each catalogue model (B = billions of parameters). It's the single biggest lever on how capable a model is: bigger models reason better and follow instructions more reliably, but they need more memory and run slower. When you install a catalogue model, picking the size is the first choice — the quantization just fine-tunes the size you land on.

The trade-off

Bigger model	Smaller model
Smarter — better reasoning, coding, instruction-following	Lighter — runs on modest GPUs and CPUs
More memory (VRAM / RAM) and disk	Far less memory and disk
Slower responses, fewer parallel requests	Faster, more concurrent requests

How to pick

• Pick the largest size that fits your worker comfortably. Capability scales with size, so the biggest model your hardware runs well is usually the best answer. Leave headroom — the weights aren't the only thing using memory; the context window needs some too.
• Match the size to the job. Small models (1–4B) are great for classification, extraction, autocomplete, and high-volume simple tasks. Mid-size (7–14B) handle general chat and most assistant work. Large (30B+) are worth it for hard reasoning, coding, and agents.
• Size beats precision. If you're memory-constrained, a larger model at a smaller quantization almost always beats a smaller model at a higher one. Choose the size first, then trim with the quant.
• Check the hardware. See system requirements for what each GPU tier can serve. The console shows each variant's download size before you install, which is a good proxy for the memory it needs.
• Let the console recommend one. On a model's page, pick one of your workers and Pendra marks the size and quantization it recommends for that worker's memory. You can still install any option you like; it's a guide, not a gate.

What happens if it doesn't fit

Installing a model bigger than the worker's GPU memory isn't an error. The worker runs it partially offloaded — it keeps as many layers (and, for mixture-of-experts models, as many experts) on the GPU as fit and runs the rest on the CPU. It still serves every request; it's just slower than a model that fits entirely in GPU memory. The worker's page in the console marks it "Partial offload" so you can spot it. For full speed, move the model to a worker with more GPU memory or drop to a smaller quantization.

Three special cases:

• Image-generation models load on one GPU and can't be spread across cards either. On a multi-GPU worker they use the card with the most free memory, so what matters is that card — not the machine's total. One that doesn't fit stays installed and keeps taking requests, but those requests come back as 507. Free memory on that worker by turning off other models, pick a smaller quantization, or run it on a worker with a larger card.
• Vision models on a multi-GPU worker: a vision model too big for one card is spread across the cards and keeps full image input, chat, coding, and tool use — using all your GPUs' memory together. As long as it fits that combined memory it runs at full speed; one right at the limit runs partially offloaded (slower, still serving every request), just like any oversized model. (On a single-GPU worker a vision model that doesn't fit is marked "Won't fit here" and isn't sent requests — there's no second card to spread onto.)
• Apple Silicon and other unified-memory hosts, where the GPU and CPU share one pool of memory, mark a model that doesn't fit as "Won't fit here" and send it no requests. Moving layers to the CPU frees nothing, so rather than risk the machine's stability the worker declines the model. This applies to every model on the host, vision models included. "Won't fit here" is about the machine's total memory, so it doesn't come and go while you work — pick a smaller size or quantization, or a host with more memory. A model that does fit stays listed even when the machine is temporarily short of memory; if there isn't room at that moment the request comes back as a 507 saying so, and it serves again once you quit some applications or turn off another model.

What the API says about a model that won't run

When a model is marked "Won't fit here" or "Won't run here" on every worker that has it, requests for it come back as a 404 that names the reason and tells you it's already installed — for example:

Model 'gemma4:31b' is installed on a connected worker, but its weights
don't fit that worker's GPU memory, so requests aren't routed to it.
Install it on a worker with more VRAM, or use a smaller quantisation —
re-installing it won't change this.

Installing the model again won't help: it's already there. Fix the mismatch instead — a memory shortfall wants a worker with more GPU memory or a smaller quantization, while a "won't run here" wants a different kind of GPU (see known issues below). On the worker itself, pendra models shows the same verdict per model in its STATUS column, and pendra doctor explains it.

Known issues on your hardware

Size isn't the only thing that can go wrong. Some models don't get along with some hardware — they crash on load, return garbled output, or quietly fall back to the CPU on one particular GPU or platform. When we confirm one of those combinations, we record it, and every worker running it flags the affected models for you.

A flagged model is marked "Known issue" on the worker's page in the console — amber when it still works but is degraded, red when it's outright broken on that hardware. Open Details on the badge to see what goes wrong, why it happens, and what to do about it.

It's a heads-up, not a block: the model stays installed and still receives requests, so you can judge whether the problem matters for what you're doing. The same model usually runs fine elsewhere — install it on a worker with a different GPU or platform and the flag won't follow it there. Some are fixed in a newer worker instead: update the worker and the flag clears on its own.

Not every model ships every size — the catalogue offers the sizes a model was actually released in. You can install several sizes on the same worker and switch between them per request. Some small models are also distilled from a much larger one, which lets them punch above their size — worth knowing when you weigh a size against its quality.