LLM Training Compute Calculator (FLOPs, GPU Hours, Cost)
Enter model size, training tokens and your GPUs to estimate training time and cost.
Your numbers
Your results
LLM Training Compute Calculator (FLOPs, GPU Hours, Cost) by TheCalcTools
How the LLM Training Compute Calculator (FLOPs, GPU Hours, Cost) works
Training a dense transformer takes about 6 floating-point operations per parameter per token: 2 for the forward pass and 4 for the backward pass.
Real training runs achieve only part of a GPU’s peak throughput (model FLOPs utilisation, often 30 to 50%). GPU-hours = FLOPs ÷ (peak × MFU) ÷ 3,600; divide by the number of GPUs for wall-clock time.
The formula
Worked example
Using the default values shown in the calculator:
| Parameters (billions) | 7 |
|---|---|
| Training tokens (billions) | 2000 |
| Peak GPU throughput (TFLOPS, BF16 dense) | 989 |
| Model FLOPs utilisation (MFU) | 40 % |
| Number of GPUs | 512 |
| Price per GPU-hour | $2.5 |
- 6 × 7 × 10⁹ × 2 × 10¹² = 8.4e+22 FLOPs.
- At 40% of 989 TFLOPS that is 58,982 GPU-hours: 4.8 days on 512 GPUs, about $147,455.
Result:
| Training time (days) | 4.8 |
|---|---|
| GPU-hours | 58,982 |
| Compute cost | $147,455 |
| Total compute (FLOPs) | 8.4e+22 |
| Tokens per parameter | 285.7 |
Frequently asked questions
How many tokens should I train on?
The Chinchilla study found compute-optimal training uses about 20 tokens per parameter; many current models train far longer to make inference cheaper.
Sources and further reading
- Kaplan et al. (2020) Scaling Laws for Neural Language Models
- Hoffmann et al. (2022) Training Compute-Optimal Large Language Models (Chinchilla)
Related calculators
More in AI & Computing · All calculators
This calculator gives estimates for information only. Check important results independently before you rely on them. See our disclaimer.