Reading Transformer: Why Attention Could Replace Recurrence

A close reading of scaled dot-product attention, multi-head attention, positional encoding, and the engineering conditions that made Transformer consequential.

阅读中文版 →
On this page

Paper: Attention Is All You Need, Vaswani et al., NeurIPS 2017.

The paper is often summarized as “attention without recurrence.” The deeper change was computational: any two sequence positions could communicate in one attention step, while training became a collection of large matrix operations that parallel hardware could execute efficiently.

The problem it reframed

An RNN carries information through a sequence of hidden-state updates. That is a useful inductive bias, but long-range information must cross many steps and training remains sequential. CNNs parallelize well, yet need additional layers to enlarge their receptive field.

Transformer instead allows each position to inspect the sequence and assign context-dependent weights to other positions. It shortens the information path and maps naturally to GPU computation.

Scaled dot-product attention

Each token is projected into a Query, Key, and Value:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

The scale factor matters. Dot products grow in magnitude with dimensionality, which can push softmax into regions with very small gradients. Dividing by √dₖ stabilizes optimization.

Attention is more than a search for a related word. At every layer, it computes a dynamic routing graph for information under the current context.

Why multiple heads

A single representation space may struggle to capture several relationships at once. Multi-head attention projects inputs into multiple subspaces, runs attention independently, and combines the results. Different heads can learn positional, syntactic, referential, or task-specific patterns.

Heads are not guaranteed to have clean human interpretations. From an engineering perspective, head count, head dimension, and total compute must be considered together; more heads do not automatically mean a better model.

Recovering order without recurrence

Self-attention alone is insensitive to input order, so the paper adds sinusoidal positional encodings. Later systems use relative positions or rotary embeddings, but the requirement is unchanged: a model needs an explicit signal for order and distance.

Residual connections, Layer Normalization, and position-wise feed-forward networks are equally important. Attention moves information between tokens; the feed-forward block transforms each position nonlinearly.

The lasting engineering result

The paper reported 28.4 BLEU on WMT 2014 English-to-German and 41.8 BLEU on English-to-French. The lasting result was the combination of quality, parallel training, and short paths between distant tokens.

Standard self-attention still builds a quadratic attention matrix. As context grows, memory bandwidth, capacity, and inference latency become constraints. FlashAttention, sparse attention, and linear variants all revisit the same systems question: how can we preserve flexible information routing at lower cost?

My takeaway

Transformer is most useful as a design lesson, not a frozen diagram. Reconsider the path information must travel, then choose a computation that fits the hardware. A model architecture breakthrough is often a systems breakthrough at the same time.

RELATED WRITING

DISCUSSION

Discussion