Worker
Choosing a model size
A model's size is its parameter count — the
4B, 9B, or 70B figure on each
catalogue model (B = billions of parameters). It's the single biggest
lever on how capable a model is: bigger models reason better and follow
instructions more reliably, but they need more memory and run slower.
When you install a catalogue model, picking the size is the first
choice — the quantization just
fine-tunes the size you land on.
The trade-off
| Bigger model | Smaller model |
|---|---|
| Smarter — better reasoning, coding, instruction-following | Lighter — runs on modest GPUs and CPUs |
| More memory (VRAM / RAM) and disk | Far less memory and disk |
| Slower responses, fewer parallel requests | Faster, more concurrent requests |
How to pick
- Pick the largest size that fits your worker comfortably. Capability scales with size, so the biggest model your hardware runs well is usually the best answer. Leave headroom — the weights aren't the only thing using memory; the context window needs some too.
- Match the size to the job. Small models (1–4B) are great for classification, extraction, autocomplete, and high-volume simple tasks. Mid-size (7–14B) handle general chat and most assistant work. Large (30B+) are worth it for hard reasoning, coding, and agents.
- Size beats precision. If you're memory-constrained, a larger model at a smaller quantization almost always beats a smaller model at a higher one. Choose the size first, then trim with the quant.
- Check the hardware. See system requirements for what each GPU tier can serve. The console shows each variant's download size before you install, which is a good proxy for the memory it needs.
- Let the console recommend one. On a model's page, pick one of your workers and Pendra marks the size and quantization it recommends for that worker's memory. You can still install any option you like; it's a guide, not a gate.
What happens if it doesn't fit
Installing a model bigger than the worker's GPU memory isn't an error. The worker runs it partially offloaded — it keeps as many layers (and, for mixture-of-experts models, as many experts) on the GPU as fit and runs the rest on the CPU. It still serves every request; it's just slower than a model that fits entirely in GPU memory. The worker's page in the console marks it "Partial offload" so you can spot it. For full speed, move the model to a worker with more GPU memory or drop to a smaller quantization.
Two exceptions, where a model that doesn't fit is marked "Won't fit here" and isn't sent any requests:
- Image and vision models can't be split across cards, so they have to fit on a single GPU — even on a machine with several.
- Apple Silicon and other unified-memory hosts, where the GPU and CPU share one pool of memory. Moving layers to the CPU frees nothing, so rather than risk the machine's stability the worker declines the model. Pick a smaller size or quantization, or a host with more memory.
Known issues on your hardware
Size isn't the only thing that can go wrong. Some models don't get along with some hardware — they crash on load, return garbled output, or quietly fall back to the CPU on one particular GPU or platform. When we confirm one of those combinations, we record it, and every worker running it flags the affected models for you.
A flagged model is marked "Known issue" on the worker's page in the console — amber when it still works but is degraded, red when it's outright broken on that hardware. Open Details on the badge to see what goes wrong, why it happens, and what to do about it.
It's a heads-up, not a block: the model stays installed and still receives requests, so you can judge whether the problem matters for what you're doing. The same model usually runs fine elsewhere — install it on a worker with a different GPU or platform and the flag won't follow it there.
Not every model ships every size — the catalogue offers the sizes a model was actually released in. You can install several sizes on the same worker and switch between them per request. Some small models are also distilled from a much larger one, which lets them punch above their size — worth knowing when you weigh a size against its quality.