Worker
Choosing a quantization
Quantization is how much each model weight is compressed when it's packed into a GGUF file. Lower precision means a smaller download and less memory to run, at a small cost to answer quality. It's the same model either way — only the numerical precision of its weights changes. When you install a catalogue model, most sizes let you choose a quantization; this page helps you pick.
The choices
Catalogue models offer up to three quantizations per size. They're listed smallest-to-largest in the console, and the console shows each option's exact download size before you install:
| Quantization | Quality | Size & memory | Use it when… |
|---|---|---|---|
Q4_K_M (balanced default) | Excellent — the standard choice for quantised open models | Smallest | You're not sure — start here. Best size-to-quality ratio. |
Q6_K | Near-lossless | ~40–50% larger than Q4_K_M | You have memory to spare and want a little more fidelity. |
Q8_0 | Effectively lossless vs. the full-precision model | Roughly double Q4_K_M | Quality matters most and the worker has plenty of VRAM/disk. |
How to pick
- Start with
Q4_K_M. For almost every use case the quality difference from a higher quant is hard to notice, and it leaves the most memory free for a longer context window or more parallel requests. - Step up to
Q6_KorQ8_0if the worker has spare VRAM and you want the last few percent of fidelity — e.g. for code, math, or careful instruction-following where small errors compound. - Prefer a bigger model over a higher quant. If you have the memory, a larger model at
Q4_K_Malmost always beats a smaller model atQ8_0. Use the quant choice to fine-tune within a size you've already picked.
Whichever you choose, the worker downloads and checksum-verifies the GGUF into its models directory the same way — see Install a worker for the install flow. Not every size ships every quantization; the console only offers the ones that exist for that model.