Free tool · browser-only calculation
Transformer parameter count and memory calculator
Calculate Transformer parameters by component and memory, with BERT-base and GPT-2 small presets and an Adam estimate. With guided examples and shareable.
How the calculation works
Formula as plain text
N_{params}=N_{embed}+L\left(4d_{model}^2+2d_{model}d_{ff}\right)+N_{head}+N_{norm}Transformer parameter count decomposes into embeddings, attention, feed-forward layers, normalisation and the output head. Each block projects Q, K, V and output and applies two linear layers with optional bias. Weight tying between embeddings and the head avoids counting the same matrix twice.
d_model, layer count and feed-forward width determine scale; vocabulary and maximum length affect learned embeddings and positional encoding. Head count changes attention partitioning, not standard projection totals. The tool itemises components so the total can be audited.
Weight memory varies with fp32, bf16, int8 and int4. The Adam mixed-precision estimate uses roughly 16 bytes per parameter and excludes activations, dynamic gradients, KV cache and allocator overhead. Public presets provide a reference before entering a custom model.