Transformer Parameter Count Calculator
Enter the architecture to estimate how many parameters a transformer has.
Your numbers
Your results
Transformer Parameter Count Calculator by TheCalcTools
How the Transformer Parameter Count Calculator works
Each block has attention projections (query and output d², plus keys and values scaled by the KV-head ratio) and a feed-forward network. Classic MLPs have 2 matrices of d × 4d; SwiGLU networks have 3 matrices of d × (multiplier × d).
Embeddings add vocabulary × width, twice if the output layer is not tied. Layer-norm and bias parameters are tiny and left out.
The formula
Worked example
Using the default values shown in the calculator:
| Layers | 32 |
|---|---|
| Model width (d_model) | 4096 |
| Vocabulary size | 128000 |
| Feed-forward multiplier | 3.5 |
| KV heads ÷ heads | 0.25 |
| Input and output embeddings | Separate |
- 32 blocks of 218 million plus 1,049 million in embeddings ≈ 8.028 billion.
Result:
| Total parameters | 8.028 billion |
|---|---|
| Per transformer block (millions) | 218.1 |
| Embeddings (millions) | 1,048.6 |
| Share in embeddings | 13.1% |
Frequently asked questions
Why does my result differ slightly from a published count?
Published counts include norms and biases and use exact intermediate sizes rounded to hardware-friendly numbers.
Sources and further reading
- Kaplan et al. (2020) Scaling Laws for Neural Language Models
- Wikipedia: Transformer (deep learning architecture)
Related calculators
More in AI & Computing · All calculators
This calculator gives estimates for information only. Check important results independently before you rely on them. See our disclaimer.