• Vaswani et al. (2017), Attention Is All You Need, §3.2
• Dao et al. (2022), FlashAttention
• PyTorch: scaled_dot_product_attention
• PyTorch: MultiheadAttention

This is a simulation with fixed, untrained example embeddings and projections, not a trained language model. Scores, weights, outputs and storage sizes are computed. Positions are numbered from 1. Dropout, biases, batching and positional encodings are omitted from the teaching head. Independent lesson, not affiliated with PyTorch.