Single vs Multi-Head Attention

Attention: 9 of 11

Library › Attention

1. QKV Projection

2. Attention Computation

3. Self Attention

4. Cross Attention

5. Self Attention vs Cross Attention

6. Self Attention (Shared KV)

7. Multi-Head Attention

8. Fused QKV (Multi-Head)

9. Single vs Multi-Head Attention

10. Multi-Query Attention

11. Grouped-Query Attention

The most important fact about multi-head attention is that it has the same parameter count as single-head attention. The difference is purely structural: the same total Wqkv weights, partitioned into smaller q–k–v triples.

Paid members: the interactive diagram and the full breakdown continue below ↓