GPU Memory (VRAM) Calculator for LLMs
Enter the model size, precision and context to estimate how much VRAM you need.
Your numbers
Your results
GPU Memory (VRAM) Calculator for LLMs by TheCalcTools
How the GPU Memory (VRAM) Calculator for LLMs works
Model weights take parameters × bytes per parameter: 2 bytes at FP16, 1 byte at INT8 and about half a byte with 4-bit quantisation. An 8-billion-parameter model needs about 16 GB at FP16 or 4.5 GB at 4 bits.
The KV cache stores keys and values for every token in context: 2 × layers × hidden size × (KV heads ÷ heads) × context × batch × bytes. It grows with long contexts and many concurrent users.
Add 10 to 20% for activations, the CUDA context and fragmentation. Results are estimates; serving frameworks differ.
The formula
Worked example
Using the default values shown in the calculator:
| Parameters (billions) | 8 |
|---|---|
| Weight precision | FP16 / BF16 (16-bit) |
| Layers | 32 |
| Hidden size | 4096 |
| KV heads ÷ attention heads (1 = no GQA) | 0.25 |
| Context length (tokens) | 8192 |
| Batch size (concurrent sequences) | 1 |
| KV cache precision | 16-bit |
| Runtime overhead | 10 % |
- 8B parameters at 16-bit: 8 × 2 = 16.0 GB of weights.
- KV cache for 8,192 tokens: 1.07 GB. With 10% overhead: 18.8 GB, so it fits on One 24 GB GPU.
Result:
| Total VRAM needed (GB) | 18.8 |
|---|---|
| Model weights (GB) | 16.0 |
| KV cache (GB) | 1.07 |
| Fits on | One 24 GB GPU |
Frequently asked questions
Can I run a 70B model on a 24 GB GPU?
Not fully on the GPU: 70B at 4 bits is about 40 GB of weights. You would need two 24 GB GPUs, a 48 GB GPU, or CPU offloading, which is much slower.
Where do I find layers and hidden size?
In the model’s config.json on its model card (num_hidden_layers, hidden_size, num_attention_heads, num_key_value_heads).
Sources and further reading
- Kaplan et al. (2020) Scaling Laws for Neural Language Models
- Wikipedia: Large language model
- Hugging Face: Model memory anatomy
Related calculators
More in AI & Computing · All calculators
This calculator gives estimates for information only. Check important results independently before you rely on them. See our disclaimer.