We use cookies for analytics and personalized ads. We use cookies for basic analytics and, where we show ads, to personalize them. Privacy PolicyRead our Privacy Policy.
We use cookies for analytics and personalized ads. We use cookies for basic analytics and, where we show ads, to personalize them. Privacy PolicyRead our Privacy Policy.
Estimate VRAM for running a model locally, or for fine-tuning one with full fine-tuning, LoRA, or QLoRA — then see which GPU tier fits and roughly what it costs.
Quick answer: VRAM (GB) ≈ parameters in billions × bytes per parameter × 1.2. A 7B model needs about 16.8 GB at FP16, 8.4 GB at INT8, and 4.2 GB at INT4, before KV cache. A 70B model needs about 42.0 GB even at INT4. QLoRA fine-tuning a 7B model takes roughly 5.2 GB.Last updated October 1, 2026
Two very different questions both come down to the same GPU-memory math: "will this model fit to run locally?" and "will this model fit while I fine-tune it?" This calculator covers both. Switch modes below — inference mode covers weights at your chosen precision plus a real KV cache calculation; fine-tuning mode adds gradients, optimizer state, and activation memory for full fine-tuning, LoRA, or QLoRA. The sections below go deeper on quantization, PagedAttention, fine-tuning method trade-offs, and how Ollama, llama.cpp, and vLLM differ on the memory they leave available.
Count Tokens LocallyFor Mixture-of-Experts models, enter total parameters, not active parameters: every expert must fit in memory.
The rule of thumb: VRAM ≈ parameters (billions) × bytes per parameter × 1.2. The 1.2 covers about 20% runtime overhead. KV cache is extra and depends on context length and concurrency, so use the calculator above for your exact setup.
| Model size | FP16 / BF16 | INT8 / FP8 | INT4 |
|---|---|---|---|
| 3B | 7.2 GB 12 GB GPU | 3.6 GB 12 GB GPU | 1.8 GB 12 GB GPU |
| 7B | 16.8 GB 24 GB GPU | 8.4 GB 12 GB GPU | 4.2 GB 12 GB GPU |
| 13B | 31.2 GB 48 GB GPU | 15.6 GB 24 GB GPU | 7.8 GB 12 GB GPU |
| 32B | 76.8 GB 80 GB GPU | 38.4 GB 48 GB GPU | 19.2 GB 24 GB GPU |
| 70B | 168 GB 3× 80 GB GPUs | 84.0 GB 141 GB GPU | 42.0 GB 48 GB GPU |
| 405B | 972 GB 13× 80 GB GPUs | 486 GB 7× 80 GB GPUs | 243 GB 4× 80 GB GPUs |
Weights plus 20% overhead, no KV cache. The second line is the smallest single GPU tier that fits, or the number of 80 GB cards needed.
| Model size | Full fine-tuning | LoRA (FP16 base) | QLoRA (4-bit base) |
|---|---|---|---|
| 7B | 126 GB 141 GB GPU | 17.0 GB 24 GB GPU | 5.2 GB 12 GB GPU |
| 13B | 234 GB 3× 80 GB GPUs | 31.2 GB 48 GB GPU | 9.3 GB 12 GB GPU |
| 34B | 611 GB 8× 80 GB GPUs | 80.1 GB 141 GB GPU | 23.0 GB 24 GB GPU |
| 70B | 1257 GB 16× 80 GB GPUs | 163 GB 3× 80 GB GPUs | 45.8 GB 48 GB GPU |
Estimated with this calculator's fine-tuning formulas: sequence length 2,048, batch size 1, LoRA rank 16, 32-bit Adam, gradient checkpointing on. Longer sequences, larger batches, or turning checkpointing off raise activation memory.
A self-hosted LLM trades a per-token bill for a hardware bill you pay once. That trade only makes sense for specific situations — it isn't automatically cheaper or better than calling an API, just different.
If self-hosting fits, the calculator above and the sections below cover the two numbers that actually decide hardware: how much memory the model's weights take at your chosen precision, and how much extra the KV cache adds once you're serving real context length and concurrent requests — not just holding the model in memory doing nothing.
Quantization is a form of model compression — it stores each parameter in fewer bits, which shrinks memory use almost linearly. It's the single biggest lever in the calculator above, and usually the first thing worth trying before reaching for more GPUs.
The common default for serving. Full quality, 2 bytes per parameter. A 7B model needs roughly 14 GB just for weights.
Half the memory of FP16 at 1 byte per parameter, with a small, usually acceptable quality drop for most chat and coding tasks.
A quarter of FP16's memory at 0.5 bytes per parameter — the difference between needing a 24 GB card and a 12 GB one on a 7B model. Quality loss is more noticeable on harder reasoning tasks.
The quantized file formats llama.cpp and Ollama load directly, typically offered in several bit-widths (Q4, Q5, Q8) so you can pick the same input/output trade-off from the table on the left.
Quantization only compresses the weights — it does nothing to the KV cache, which is calculated separately below from context length and concurrency. A heavily quantized model can still run out of memory at long context if the KV cache isn't sized alongside it.
Quantization shrinks a model you're only running. Fine-tuning it adds a second, much bigger cost: gradients and optimizer state for every parameter you actually train. Which method you pick changes that cost by an order of magnitude or more.
Every parameter is trainable. With a standard Adam optimizer, that's roughly 16 bytes per parameter across weights, gradients, and optimizer state — a 13B model needs well over 200 GB before activations, ruling out anything short of a multi-GPU cluster.
The base model is frozen and loaded at your chosen precision; only small adapter matrices train. Gradient and optimizer memory apply to a tiny fraction of total parameters, so most of the cost is just holding the frozen base in memory.
LoRA with the frozen base quantized to 4-bit (NF4). Base-weight memory drops to about a quarter of FP16 — this is what lets a 70B model fine-tune on a single 48 GB card at a 2,048-token sequence (about 46 GB by this calculator), with an 80 GB card leaving room for longer sequences or larger batches, instead of a multi-GPU cluster.
One common mix-up: QLoRA and GGUF 4-bit are not the same thing. QLoRA is a training-time technique (NF4 base + trainable adapters). GGUF is an inference-time file format used by llama.cpp and Ollama. You typically fine-tune with QLoRA, merge or keep the adapter, and only convert to GGUF afterward for cheap local serving — switch the mode toggle above to "Running locally" to size that step.
The KV cache number above is a straightforward calculation — but naive serving frameworks waste a large share of that memory in practice. PagedAttention, introduced by the vLLM project, is the fix: instead of reserving one large contiguous memory block per request sized for its worst-case length, it splits the KV cache into fixed-size blocks and allocates them on demand, the same way an operating system pages virtual memory.
Each request reserves KV cache memory for its maximum possible length upfront, even if the actual response is much shorter. Most of that reserved memory sits unused for the entire request — reported waste in early serving stacks ran as high as 60–80%.
Memory is allocated in small blocks as generation actually progresses, and freed immediately when a request finishes. That near-zero waste is what lets vLLM push meaningfully higher batch sizes and throughput out of the same GPU compared to a framework without it.
The "Total Serving VRAM Needed" figure above is the theoretical KV cache size — what PagedAttention lets you actually approach in practice, rather than the much larger figure a naive allocator would reserve for the same context length and batch size.
The "Framework Overhead" slider in the calculator above exists because the inference engine you choose to actually run the model changes how much memory is left over for weights and KV cache. These are the three most common options for self-hosting.
A wrapper around llama.cpp with a simple CLI, a model library, and near-zero setup — the usual starting point for a personal LLM on a laptop or desktop.
Trade-off: the convenience layer adds modest overhead on top of llama.cpp, and its batching is limited — fine for one user, less so for serving several concurrent requests efficiently.
The underlying C/C++ inference engine Ollama builds on, and the reference implementation for the GGUF quantized format. Runs on CPU, Apple Silicon, or GPU with fine-grained control.
Trade-off: lowest overhead and the most flexible hardware support, but a rougher setup experience than Ollama, with more manual configuration.
A production-serving engine built around PagedAttention and continuous batching, aimed at maximizing throughput across many concurrent requests rather than single-user convenience.
Trade-off: the best throughput-per-GPU of the three at real concurrency, but a heavier install and typically a datacenter-class GPU to get the full benefit.
As a rule of thumb: reach for Ollama first for a personal or offline LLM you're running alone, llama.cpp when you need tighter control over quantization and hardware, and vLLM once you're actually serving multiple concurrent users and need the memory efficiency PagedAttention provides.
12 GB VRAM per GPU
24 GB VRAM per GPU
48 GB VRAM per GPU
80 GB VRAM per GPU
141 GB VRAM per GPU
The cheapest fitting tier is a consumer GPU without standard cloud rental pricing — bump up model size or context to compare against a data-center tier.
Illustrative on-demand ballparks, not live quotes — check your provider's current pricing before budgeting.
At FP16, a 70B-parameter model needs roughly 140 GB just for weights (70B × 2 bytes) — already past a single 80 GB datacenter GPU before adding any overhead or KV cache. Quantizing to INT4 (0.5 bytes per parameter) drops that to about 35 GB, plus ~20% framework overhead (~7 GB), landing around 42 GB for weights alone — comfortably inside a single 48 GB workstation card.
Serving that with a 8K-token context and a handful of concurrent requests adds a further few GB of KV cache on top — worth running through the KV cache calculator above with your own layer count and hidden dimension rather than assuming a round number, since KV cache size varies a lot by model architecture, not just parameter count.
Full fine-tuning a 13B model with a standard Adam optimizer needs roughly 13 × 16 = 208 GB just for weights, gradients, and optimizer state — before any activation memory — which rules out anything short of a multi-GPU A100/H100 setup.
Switching to QLoRA loads the same 13B base in 4-bit (≈6.5 GB) and trains a rank-16 adapter (about 62 million trainable parameters), whose gradients and optimizer state add about 1 GB. Add activation memory for your sequence length and batch size and the total lands around 9 GB at a 2,048-token sequence — comfortably inside a single 24 GB consumer GPU at moderate context lengths, which is exactly what switching the mode toggle above to "Fine-tuning" and picking QLoRA will show.