How much GPU do you need to run an open model?

Self-hosting an open model comes down to one blunt question: does it fit in memory? Get that right and everything else (speed, context length, cost) tends to follow. Get it wrong and the model either won't load or crawls along spilling onto the CPU. This is a practical map from real hardware tiers to the models in the Pendra catalogue that actually run on them, so you can choose a model before you commit to silicon.

Every model below is one you can run on Pendra today, whether you bring your own GPU or use managed Pendra infrastructure. We're using the mid-2026 GPU market as the hardware backdrop and the Pendra catalogue as the menu.

The one rule that decides everything

A model has to fit in memory to run fast. That means GPU VRAM on a discrete card, or unified memory on Apple Silicon and the new ARM boxes. If the weights don't fit, the model either fails to load or offloads layers to system RAM, where throughput falls off a cliff.

A couple of rules of thumb get you most of the way:

  • Roughly half a gigabyte per billion parameters at 4-bit. A 4-bit (Q4) quantised model needs about 0.5–0.6 GB of memory per billion dense parameters. A 30B model lands around 18 GB; a 70B around 42 GB.
  • Full precision is roughly four times heavier. The same 7B model is about 14 GB in 16-bit precision but under 5 GB at 4-bit. Quantisation is the single biggest lever you have.
  • Always leave headroom. A model whose weights exactly fill the card won't run at a useful context length, because the KV cache for your prompt and generation needs memory too. Budget the weights to comfortably under your total, not right up to it.

One wrinkle worth knowing about mixture-of-experts (MoE) models such as gpt-oss, qwen3.6, qwen3-coder and glm-4.7-flash: they still need their full weight set resident in memory, but only activate a fraction of it per token. So a MoE model runs noticeably faster than a dense model of the same on-disk size. It buys you speed, not a smaller memory footprint.

Quantisation, in one paragraph

Quantisation shrinks the weights by storing them at lower precision. 4-bit (Q4) is the practical default: it cuts memory roughly four-fold versus full precision while keeping quality close enough that most people can't tell the difference in normal use. Pendra also offers higher-fidelity Q6 and Q8 variants when you have memory to spare. The on-disk size of the default Q4 build is a good first proxy for the VRAM you'll need, and that's the number we use in the table below. (For the full picture, see quantisation and choosing a model size in the docs.)

The hardware tiers, cheapest to most capable

Memory capacity is what separates the tiers, and it tracks price fairly closely. From the value end upward:

  • Used 24GB cards. A second-hand RTX 3090 (24GB) is the value entry point and still the best memory-per-pound in the budget bracket. Two of them over NVLink gives you ~48GB for 70B-class models.
  • AMD RX 7900 XTX (24GB). A strong-value new card on Linux with ROCm, if you'd rather buy new at the 24GB tier.
  • RTX 4090 (24GB) / RTX 5090 (32GB). The 5090 is the best single new consumer card; the extra 32GB buys real context headroom at the 30–35B tier. A used 4090 remains competitive.
  • RTX 6000 Ada (48GB) / RTX PRO 6000 Blackwell (96GB). Workstation cards for 70B-class dense models and the larger MoEs.
  • Unified-memory boxes. Apple Silicon (M3 Ultra, up to 512GB at ~819 GB/s) and the ARM boxes (DGX Spark, Jetson Thor, 128GB at ~273 GB/s) trade raw bandwidth for enormous capacity. They run very large models, just more slowly, so they are great for capacity but not for high concurrency.
  • Datacentre GPUs. A100, H100, H200 and AMD MI300X sit at the top, with the bandwidth and capacity for the largest frontier MoEs.

The map: memory tier → recommended models

Sizes shown are the default Q4 build from the catalogue, in gigabytes. "Fits" assumes you leave some headroom for context, so aim a little under your card's total.

Memory Hardware example Comfortable picks (Q4 size in GB) Notes
~8 GB 8GB laptop / desktop GPU qwen3.5:4b (2.7), gemma4:e4b (5.0), qwen3-vl:8b (5.0), ministral-3:8b (5.2), lfm2-5:8b (5.3), glm-4.6v-flash (6.2), nemotron-3-nano:4b (2.9), phi-4-mini-reasoning (2.5) Entry tier. Vision is available here via gemma4:e4b, qwen3-vl:8b or glm-4.6v-flash. Every embedding and rerank model fits comfortably.
~12 GB RTX 3060 12GB gemma4:12b (7.1), phi-4-reasoning (9.1), ministral-3:14b (8.2), gpt-oss:20b (12.1, tight) gpt-oss:20b is the headline near-the-edge pick: as a MoE it runs faster than its size suggests.
~16 GB RTX 4060 Ti 16GB qwen3.5:27b (16.7), qwen3.6:27b (16.8), gemma4:26b (16.9), devstral-small-2:24b (14.3), magistral (14.3) The first tier that runs a serious 24–27B model. devstral-small-2 adds coding and vision.
~24 GB Used RTX 3090 / RX 7900 XTX / RTX 4090 qwen3.5:35b (22), qwen3.6:35b (22.1), glm-4.7-flash (18.3), gemma4:31b (18.3), qwen3-coder:30b (18.6), qwen3-vl:32b (19.8) The value sweet spot. Runs a genuinely capable 30–35B model with room left for context.
~32 GB RTX 5090 Everything in the 24GB row, plus more context headroom; nemotron-3-nano:30b (24.6) sits comfortably Best single new card. Spend the extra memory on longer context and a bigger KV cache at the 30–35B tier.
~48 GB RTX 6000 Ada / dual RTX 3090 (NVLink) llama3.3:70b (42.5), qwen3.5:122b (46.6, MoE), qwen3-coder-next (48.5, tight) Where 70B-class dense models and the smaller large-MoE models become practical.
~96 GB RTX PRO 6000 Blackwell gpt-oss:120b (63.4) Workstation maximum. Runs the big MoEs that fit under ~90GB; the very largest frontier MoEs still need more.
Unified memory Apple M3 Ultra (≤512GB); DGX Spark / Jetson Thor (128GB) 128GB: gpt-oss:120b (63.4) easily. 512GB: qwen3.5:397b (244), qwen3-coder:480b (290), deepseek-v4-flash (171.9), mimo-v2.5 (190.8) Huge capacity, lower bandwidth (Apple ~819 GB/s; Spark/Thor ~273 GB/s). Ideal for running very large models slowly, not for heavy concurrency.
Datacentre A100 / H100 / H200 / MI300X The largest models in the catalogue: qwen3-coder:480b (290), qwen3.5:397b (244), mimo-v2.5-pro (629.6), mimo-v2.5 (190.8), deepseek-v4-flash (171.9) The frontier-MoE tier. mimo-v2.5-pro is the only catalogue model that exceeds even 512GB of unified memory.

Capability, not just size

Pick for what you need to do, then fit it to your tier. A few shortcuts:

  • Vision (image understanding): gemma4, qwen3.5, qwen3-vl and glm-4.6v-flash span everything from 8GB cards up.
  • Coding: qwen3-coder and devstral-small-2 are purpose-built for it; devstral-small-2 fits comfortably at the 16GB tier.
  • Reasoning: phi-4-reasoning, magistral, glm-4.7-flash, nemotron-3-nano and ministral-3-reasoning think step-by-step before answering.
  • Very long context: qwen3.5 and qwen3-vl handle 262K tokens; deepseek-v4-flash and mimo-v2.5 reach 1M. Long context eats memory, so leave extra headroom.
  • Search and retrieval: the embedding and rerank models (qwen3-embedding, bge-m3, embeddinggemma, nomic-embed-text, bge-reranker-v2-m3) are all small enough to run anywhere.

Our recommendation

If you're starting out and want one answer: buy a used 24GB card (an RTX 3090, an RX 7900 XTX, or a used 4090) and run a 30–35B model. qwen3.5:35b, qwen3-coder:30b and glm-4.7-flash all fit with room for a healthy context window, and they're genuinely capable models, not toys. That tier is the clear value sweet spot today: the cheapest hardware that runs a model you'd actually want to put in front of users. Step up to a 32GB or 48GB card only once a real workload proves it needs the extra room.

The nice part: Pendra runs the same catalogue whether the model is on your own GPU or on managed Pendra infrastructure. So you can develop against qwen3-coder:30b on a card under your desk, then scale the exact same model up to a bigger box, or hand it to us to run, without changing a line of your code.

Browse the full list on the models page, or read choosing a model size to go deeper.