Attention Is All You Need

第0章起步Chapter 0: Getting Started

论文 Abstract & Introduction（第1-2页）

这一章不谈公式。先把「这篇论文到底干了什么、为什么重要」这件事说清楚，让你在进入技术细节前有一个全景认知。No formulas this chapter. First, let's get clear on what this paper did and why it matters, so you have a bird's-eye view before diving into technical details.

学完这一章你应该能做到After this chapter you should be able to

用一句话向不看 AI 的人解释 Transformer 做了什么Explain what the Transformer does in one sentence to someone outside AI
说出循环网络在训练时的根本瓶颈是什么Name the fundamental bottleneck of recurrent networks during training
说出 Transformer 相对于 RNN 的两个核心优势Name two core advantages of the Transformer over RNNs

无需前置知识。这里从零开始。No prerequisites. We start from scratch.

0.1 一句话说清楚0.1 In One Sentence

2017 年，Google 的一个八人团队做了一件事：他们设计了一种处理语言的新方法，完全不用「按顺序一个一个读」的循环网络，也不用滑窗式的卷积，而是让每个词同时看到句子里的所有其他词——这个方法叫 Transformer。结果是什么？翻译质量更好，训练速度快了好几倍。后来的 GPT、BERT、ChatGPT，全都是在这个地基上盖起来的。In 2017, a team of eight at Google did one thing: they designed a new way to process language that completely abandoned the "read one-by-one" recurrent networks and sliding-window convolutions. Instead, every word sees all other words simultaneously. This is the Transformer. The result? Better translation quality, several times faster training. GPT, BERT, ChatGPT—everything that followed was built on this foundation.

打个比方Analogy

想象你在读一本书。循环网络的做法是：从第一页开始，一页一页往后翻。读到第 100 页的时候，它对第 1 页的记忆已经模糊了——而且你必须等它读完第 99 页才能开始读第 100 页，没法同时读。Imagine reading a book. RNNs do it page by page: start at page 1, flip forward one at a time. By page 100, the memory of page 1 has faded—and you can't start page 100 until page 99 is done. No parallelism.

Transformer 的做法是：把所有页面同时摊开在桌上，每一页都可以直接看到其他所有页。这样不仅没有「遗忘」问题，而且所有页面可以同时处理，训练速度大幅提升。The Transformer spreads all pages on the table at once. Each page can directly see all others. No forgetting problem, and all pages can be processed simultaneously—training is much faster.

这个比方在哪里就不灵了：看书时你确实需要顺序（不然你不知道情节发展），但 Transformer 不是在「读故事」，它是在计算每个词与所有词的关联强度——这种关联计算不需要顺序。真正的顺序信息是靠后面要讲的「位置编码」额外注入的。Where the analogy breaks: reading a book does require order (otherwise you can't follow the plot). But the Transformer isn't "reading a story"—it's computing association strength between every word pair, which doesn't need order. Actual order information is injected separately via "positional encoding."

0.2 为什么要扔掉循环网络0.2 Why Throw Away Recurrence

在 Transformer 之前，处理序列数据（也就是一句话、一段文字这类有先后顺序的数据）的主力是循环神经网络（Recurrent Neural Network，简称 RNN），以及它的升级版 LSTM 和 GRU。这些网络的工作方式是：拿到一个词，算出一个「隐藏状态」，然后传给下一个词；下一个词拿到前一个词的隐藏状态，连同自己一起算出新的隐藏状态，再传给下一个——如此循环。Before the Transformer, the main tool for sequential data (sentences, text—things with order) was the Recurrent Neural Network (RNN), and its upgrades LSTM and GRU. These networks work like this: take a word, compute a "hidden state," pass it to the next word. The next word takes the previous hidden state, combines it with itself to compute a new hidden state, and passes it along—repeating this cycle.

这种「一个接一个」的设计有一个根本问题：无法并行。你必须等第 t 步算完才能算第 t+1 步。这意味着 GPU 的并行计算能力完全浪费了——你有 8 块 P100，但它们只能干等着。This "one-after-another" design has a fundamental problem: no parallelism. You must finish step t before starting step t+1. This means GPU parallel computing power is completely wasted—you have 8 P100s sitting idle.

为什么并行这么重要Why Parallelization Matters

一句话有 30 个词，循环网络需要串行跑 30 步。Transformer 可以把这 30 个词同时扔给 GPU，一步搞定。在百万级句子组成的训练集上，这个差距是「训练 3.5 天」和「训练几周」的区别。论文里说，base 模型训练 100,000 步只需 12 小时；在那之前，同级别的模型训练动辄需要数周。A sentence has 30 words. RNNs must run 30 sequential steps. The Transformer throws all 30 to the GPU at once—one step. On a million-sentence training set, the difference is "3.5 days" vs. "weeks." The paper says the base model trains 100,000 steps in 12 hours; before this, comparable models took weeks.

除了无法并行，循环网络还有另一个问题：「远距离依赖」。当句子很长时，隐藏状态从第 1 个词传到第 30 个词，中间经过 29 次变换，信息严重衰减。虽然 LSTM 用门控机制缓解了这个问题，但没根治。论文引用了 Hochreiter 等人 2001 年的工作——梯度在长序列中会消失或爆炸，这是循环网络的基因缺陷。Beyond no parallelism, RNNs have another problem: "long-range dependencies." In a long sentence, the hidden state passes through 29 transformations from word 1 to word 30, with information severely degraded. LSTM's gating alleviates but doesn't cure this. The paper cites Hochreiter et al. 2001: gradients vanish or explode in long sequences—this is an RNN genetic defect.

0.3 论文的主要成果0.3 Key Results

这篇论文的成果可以浓缩成三个数字：The results distill into three numbers:

三个关键数字：
1. 28.4 BLEU——英译德翻译，比之前最好的模型（含集成模型）还高 2 个 BLEU
2. 41.8 BLEU——英译法翻译，单模型新纪录
3. 3.5 天——8 块 P100 GPU 上完成训练，远低于竞争对手的训练成本 Three Key Numbers:
1. 28.4 BLEU—English-to-German, beating the best prior model (including ensembles) by over 2 BLEU
2. 41.8 BLEU—English-to-French, new single-model record
3. 3.5 days—training on 8 P100 GPUs, far less than competitors' training costs

你可能会问：BLEU 是什么？BLEU（Bilingual Evaluation Understudy）是机器翻译的自动评估指标，分数越高翻译越好。为了给你一个尺度感：BLEU 从 26 到 28 在翻译领域是一个显著的提升，通常意味着人类读者能感知到翻译质量的变化。2 个 BLEU 的提升在那个年代是碾压级的。You might ask: what's BLEU? BLEU (Bilingual Evaluation Understudy) is an automatic metric for machine translation—higher is better. For scale: going from 26 to 28 BLEU is a noticeable jump, typically visible to human readers. A 2-BLEU improvement in that era was crushing.

论文还做了英译法（WMT 2014 English-to-French），成绩是 41.8 BLEU——同样是单模型的新纪录，而且训练成本只有之前最好的模型的四分之一不到。The paper also did English-to-French (WMT 2014), scoring 41.8 BLEU—also a single-model record, at less than 1/4 the training cost of the previous best.

更厉害的是，Transformer 还泛化到了非翻译任务：英文句法分析（English constituency parsing）。在这个任务上，只用 4 万句训练数据的 Transformer 就打败了当时最好的判别式模型。这说明 Transformer 不是只能做翻译——它是一个通用的序列处理架构。后来的 BERT、GPT 证明了这一点。Even more impressively, the Transformer generalized to non-translation tasks: English constituency parsing. With only 40K training sentences, it beat the best discriminative model of the time. This showed the Transformer isn't just for translation—it's a universal sequence processing architecture. BERT and GPT later proved this.

关于作者About the Authors

论文署名了八位作者，脚注里说贡献是均等的，排序是随机的。但脚注也透露了一个有趣的细节：Jakob 最先提出用自注意力替代 RNN 的想法，Ashish 和 Illia 设计并实现了最早的 Transformer 模型，Noam 提出了缩放点积注意力、多头注意力和无参数位置表示。这意味着这篇论文不是一个人的功劳，而是一群人各自贡献了关键拼图。Eight authors are listed, with a footnote saying contributions are equal and order is random. But it also reveals: Jakob first proposed replacing RNNs with self-attention, Ashish and Illia designed and implemented the first Transformer, Noam proposed scaled dot-product attention, multi-head attention, and the parameter-free position representation. This paper wasn't one person's achievement—everyone contributed key puzzle pieces.

0.4 这篇论文的结构0.4 Paper Structure

论文的正文结构如下，你可以在脑海中有个地图：The paper's structure is as follows—keep this map in your head:

论文章节结构Paper Structure
章节	内容	本精读对应章
Abstract & 1	动机：为什么要扔掉循环	第0、1章
2 Background	前人工作：卷积方法、自注意力历史	第1章
3 Model Architecture	Transformer 架构全貌	第2-11章
3.1 Encoder/Decoder	编码器/解码器堆叠	第9、10章
3.2 Attention	注意力机制（核心）	第2-5章
3.3 FFN	位置前馈网络	第6章
3.4 Embeddings	嵌入与 Softmax	第7章
3.5 Positional Encoding	位置编码	第8章
4 Why Self-Attention	自注意力的理论优势分析	第12章
5 Training	训练设置	第13章
6 Results	翻译结果 + 消融实验	第14、15章
7 Conclusion	结论与展望	第16章

我能用一句话向不看 AI 的人解释 Transformer 做了什么I can explain what the Transformer does in one sentence to a non-AI person

我能说出循环网络在训练时的两个根本问题I can name two fundamental problems of RNNs during training

我能说出 Transformer 的三个关键成果数字I can state the Transformer's three key result numbers

答辩：如果我是审稿人Debate: If I Were a Reviewer

你说 Transformer 更快——但注意力的计算量是 O(n^2 * d)，序列长了不也一样爆炸？论文怎么解决长序列问题的？You say the Transformer is faster—but attention complexity is O(n^2 * d). Doesn't it also blow up for long sequences? How does the paper address this?

参考防守（先自己组织语言再看）Reference defense (try your own words first)

论文确实承认了这个问题。在论文第7页，作者指出当序列长度 n 小于表示维度 d 时（这在机器翻译中通常成立，因为句子不会比 512 维长太多），自注意力比循环网络更快。对于超长序列，论文在论文 4 章末尾提到了「受限自注意力」（restricted self-attention）——只看大小为 r 的邻域——作为未来工作。后来的 Longformer、BigBird 等正是沿着这个方向做的。The paper does acknowledge this. On page 7, the authors note that when sequence length n is smaller than representation dimension d (usually true in machine translation, where sentences rarely exceed 512 tokens), self-attention is faster than recurrence. For very long sequences, the paper mentions "restricted self-attention"—looking only at a neighborhood of size r—as future work. Longformer, BigBird, and others later followed this direction.

一句话记住：Transformer 扔掉了按顺序处理的循环网络，让每个词同时看到所有词——结果更好、更快，成了后来一切大模型的基础。In one sentence:The Transformer throws away sequential recurrence, letting every word see all others simultaneously. The result is better, faster, and became the foundation for everything that followed.

L1 · 基础认识论文核心主张Core thesis

用一句话总结这篇论文的核心主张。Transformer 与之前的序列模型（RNN、CNN）最本质的区别是什么？Summarize the paper's core thesis in one sentence. What is the most fundamental difference between the Transformer and prior sequence models (RNNs, CNNs)?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果抌「attention」换成「recurrence」，标题会变成什么？这说明了什么？

本章自测Chapter Quiz

以下题目由系统自动判分，答题记录接入间隔重复算法。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

Transformer 的核心思想很简单：扔掉循环和卷积，只用注意力。这个看似简单的决定解决了一个根本问题——序列训练的串行瓶颈。结果是翻译质量提高、训练时间缩短。从此，深度学习进入了一个新的时代。The core idea is simple: throw away recurrence and convolution, use only attention. This seemingly simple decision solves a fundamental problem—the sequential bottleneck of sequence training. The result: better translation quality, shorter training time. Deep learning entered a new era.

第1章序列建模的旧世界Chapter 1: The Old World of Sequence Modeling

论文 Introduction & Background（第2页）

在理解 Transformer 之前，你需要知道它是从什么手里接过接力棒的。这一章讲清楚循环网络、卷积方法和早期注意力机制各自的问题。Before understanding the Transformer, you need to know what it replaced. This chapter explains the problems of recurrent networks, convolutional approaches, and early attention mechanisms.

学完这一章你应该能做到After this chapter you should be able to

解释 RNN 的串行计算为什么是根本瓶颈Explain why RNN's sequential computation is a fundamental bottleneck
说出卷积方法（ConvS2S、ByteNet）的两个问题Name two problems with convolutional approaches (ConvS2S, ByteNet)
解释早期注意力机制与 RNN 的耦合方式Explain how early attention mechanisms coupled with RNNs

需要先读第0章起步。Requires Chapter 0 first.

1.1 循环网络的统治时代1.1 The Reign of Recurrence

2017 年之前，自然语言处理的序列问题几乎被循环神经网络垄断。LSTM（Long Short-Term Memory，长短期记忆网络）和 GRU（Gated Recurrent Unit，门控循环单元）是两大主力。Before 2017, sequence problems in NLP were almost monopolized by RNNs. LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) were the two main forces.

隐藏状态（Hidden State）：循环网络中，每一步计算出的中间表示。它像是网络的「记忆」，从前一步继承，传递给后一步。Hidden State: The intermediate representation computed at each step of an RNN. It's the network's "memory," inherited from the previous step and passed to the next.

循环网络的核心公式是：h_t = f(h_{t-1}, x_t)。这个看似简单的递推公式暗藏杀机：The core RNN formula: h_t = f(h_{t-1}, x_t). This simple recurrence hides a deadly trap:

串行瓶颈The Sequential Bottleneck

要算 h_t，你必须先算完 h_{t-1}。这意味着 30 个词的句子，GPU 必须「等」30 步。论文原文说得很精确：「This inherently sequential nature precludes parallelization within training examples」——这种固有的串行性质阻碍了训练样例内部的并行化。To compute h_t, you must first finish h_{t-1}. This means for a 30-word sentence, the GPU must "wait" 30 steps. The paper states it precisely: "This inherently sequential nature precludes parallelization within training examples."

论文还提到了两个缓解方案——factorization tricks（分解技巧，Kuchaiev & Ginsburg 2017）和 conditional computation（条件计算，Shazeer et al. 2017）。前者通过分解 LSTM 矩阵来加速，后者通过稀疏激活来提升效率。但论文紧接着说：「The fundamental constraint of sequential computation, however, remains」——串行计算的根本约束依然存在。这些技巧像是在堵漏，而 Transformer 直接把屋顶掀了重建。The paper mentions two mitigations: factorization tricks (Kuchaiev & Ginsburg 2017) and conditional computation (Shazeer et al. 2017). The former speeds up by factoring LSTM matrices; the latter improves efficiency through sparse activation. But the paper immediately adds: "The fundamental constraint of sequential computation, however, remains." These tricks patch leaks; the Transformer rebuilds the roof.

1.2 卷积的尝试1.2 The Convolution Attempt

为了绕开串行瓶颈，有些研究开始用卷积来处理序列。论文提到了三个代表：Extended Neural GPU（Kaiser & Bengio 2016）、ByteNet（Kalchbrenner et al. 2017）和 ConvS2S（Gehring et al. 2017）。这些模型用卷积层替代循环层，确实实现了并行——所有位置可以同时计算。但卷积有一个新问题。To bypass the sequential bottleneck, some researchers turned to convolutions for sequences. The paper mentions three representatives: Extended Neural GPU (Kaiser & Bengio 2016), ByteNet (Kalchbrenner et al. 2017), and ConvS2S (Gehring et al. 2017). These use convolutional layers instead of recurrent ones, achieving parallelism. But convolution introduces a new problem.

卷积的远距离问题Convolution's Long-Range Problem

卷积层的感受野受限于 kernel 大小。想知道第 1 个词和第 30 个词的关系，需要堆叠很多层卷积——ConvS2S 需要 O(n/k) 层（k 是 kernel 宽度），ByteNet 需要 O(log_k n) 层（用了膨胀卷积）。也就是说，两个位置之间的「路径长度」随距离增长。A convolutional layer's receptive field is limited by kernel size. To connect word 1 and word 30, you need to stack many layers—ConvS2S needs O(n/k) layers (k = kernel width), ByteNet needs O(log_k n) layers (using dilated convolution). The "path length" between two positions grows with distance.

这为什么是问题？因为路径越长，前向信号和反向梯度要经过的变换越多，学习就越困难。论文引用了 Hochreiter et al. 2001 的经典工作来支撑这个论点。Why is this a problem? Because the longer the path, the more transformations forward signals and backward gradients must traverse, making learning harder. The paper cites the classic Hochreiter et al. 2001 to support this argument.

对比一下三种方法的「最大路径长度」（即信号从最远的两个位置之间需要经过多少步计算）：Compare "maximum path length" (how many computational steps between the farthest two positions) across three methods:

论文 Table 1 节选：不同层类型的最大路径长度Paper Table 1 excerpt: Maximum path lengths for different layer types
层类型	每层复杂度	串行操作数	最大路径长度
自注意力	O(n^2 * d)	O(1)	O(1)
循环	O(n * d^2)	O(n)	O(n)
卷积	O(k * n * d^2)	O(1)	O(log_k n)
受限自注意力	O(r * n * d)	O(1)	O(n/r)

关键对比：自注意力的最大路径长度是 O(1)——不管两个词隔多远，它们之间的计算路径都是一步。循环网络是 O(n)，卷积是 O(log_k n)。这是 Transformer 的核心优势之一：远距离依赖的学习变得极其自然。Key takeaway: self-attention's max path length is O(1)—no matter how far apart two words are, the computational path between them is one step. RNN is O(n), convolution is O(log_k n). This is one of the Transformer's core advantages: learning long-range dependencies becomes natural.

1.3 早期注意力机制1.3 Early Attention

注意力机制不是 Transformer 发明的。在它之前，Bahdanau et al. 2014 就在机器翻译中引入了注意力——让解码器的每一步都能「看」到编码器的所有隐藏状态。但关键是：当时的注意力是与 RNN 耦合使用的，不是单独使用。Attention wasn't invented by the Transformer. Before it, Bahdanau et al. 2014 introduced attention in machine translation—letting the decoder "look at" all encoder hidden states at each step. The key: attention was used in conjunction with RNNs, not alone.

自注意力（Self-Attention，也叫 Intra-Attention）：一种让序列中的每个位置直接计算与其他位置关联的注意力机制。不是编码器到解码器的注意力，而是同一段序列内部的注意力。Self-Attention (also Intra-Attention): An attention mechanism where each position in a sequence directly computes its relationship with all other positions—not encoder-to-decoder attention, but within a single sequence.

论文提到，自注意力已经在阅读理解、摘要生成、文本蕴含等任务中被成功使用（Cheng et al. 2016, Parikh et al. 2016, Paulus et al. 2017, Lin et al. 2017）。End-to-end memory networks（Sukhbaatar et al. 2015）也用了类似的机制。但论文强调：「To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.」The paper notes self-attention was already used successfully in reading comprehension, summarization, textual entailment (Cheng et al. 2016, Parikh et al. 2016, Paulus et al. 2017, Lin et al. 2017). End-to-end memory networks (Sukhbaatar et al. 2015) used similar mechanisms. But the paper emphasizes: "To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution."

关键区别：Transformer 不是发明了注意力——它是第一个完全依赖注意力、彻底抛弃 RNN 和卷积的序列转换模型。Key distinction:The Transformer didn't invent attention—it was the first to rely entirely on it, completely abandoning RNNs and convolutions for sequence transduction.

我能解释为什么 RNN 的串行计算是根本瓶颈I can explain why RNN's sequential computation is a fundamental bottleneck

我能说出卷积方法在远距离依赖上的两个问题I can name two problems of convolutional approaches for long-range dependencies

我能解释 Transformer 与之前注意力机制的关键区别I can explain the key difference between the Transformer and prior attention mechanisms

L2 · 变形迁移 RNN 并行化瓶颈RNN parallelization bottleneck

为什么 RNN 无法在序列内并行化计算？从数学上解释 h_t = f(h_{t-1}, x_t) 这个递归关系为什么是根本瓶颈。Why can't RNNs parallelize computation within a sequence? Explain mathematically why the recursive relation h_t = f(h_{t-1}, x_t) is a fundamental bottleneck.

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果序列长度只有1，RNN 的并行化问题还存在吗？

本章小结Chapter Summary

Transformer 之前有两条路：RNN 能建模长距离依赖但无法并行；卷积能并行但远距离路径太长。早期注意力机制有用，但都和 RNN 耦合。Transformer 的突破在于：只用注意力，彻底抛弃循环和卷积——既并行又不受距离限制。Before the Transformer there were two paths: RNNs model long-range dependencies but can't parallelize; convolutions parallelize but have long paths. Early attention existed but was coupled with RNNs. The Transformer's breakthrough: attention only, abandoning recurrence and convolution—both parallel and distance-agnostic.

第2章自注意力的核心直觉Chapter 2: The Core Intuition of Self-Attention

论文 3 Model Architecture（第3页）

注意力不是一个新概念，但这篇论文把它放到了舞台中心。这一章先建立直觉：注意力到底在干什么？Q、K、V 是什么？为什么要这么设计？Attention isn't a new concept, but this paper puts it center stage. This chapter builds intuition: what does attention actually do? What are Q, K, V? Why this design?

学完这一章你应该能做到After this chapter you should be able to

用自己的话解释 Query、Key、Value 在注意力中各自的角色Explain the roles of Query, Key, Value in attention in your own words
解释 softmax 在注意力中起什么作用Explain what softmax does in attention
说出自注意力和编码器-解码器注意力的区别Distinguish self-attention from encoder-decoder attention

需要先读第0章和第1章。Requires Chapter 0 and Chapter 1.

2.1 编码器-解码器框架2.1 The Encoder-Decoder Framework

在深入注意力之前，先理解 Transformer 所处的宏观框架。论文说「Most competitive neural sequence transduction models have an encoder-decoder structure」——大多数有竞争力的序列转换模型都是编码器-解码器结构。Before diving into attention, understand the macro framework. The paper says competitive sequence transduction models have an encoder-decoder structure.

编码器做的事：把输入序列 (x_1, ..., x_n) 变成一个连续表示 z = (z_1, ..., z_n)。比如「The cat sat on the mat」这 6 个词，编码后变成 6 个 512 维向量。The encoder: maps input sequence (x_1, ..., x_n) to continuous representations z = (z_1, ..., z_n). For example, 6 words become 6 vectors of 512 dimensions.

解码器做的事：拿到 z，一步一步生成输出序列 (y_1, ..., y_m)。关键点是「自回归」（auto-regressive）——每生成一个词，就把这个词当作下一步的输入。就像写作文时，你写的上一个词会影响下一个词的选择。The decoder: given z, generates output sequence (y_1, ..., y_m) one element at a time. The key point is "auto-regressive"—each generated word becomes input for the next step. Like writing an essay: the last word you wrote influences the next.

自回归（Auto-regressive）：模型在生成第 i 个词时，把之前生成的第 1 到 i-1 个词都当作输入。这意味着生成过程是串行的——这是 Transformer 在推理阶段仍需串行的原因，尽管训练时可以并行。Auto-regressive: The model uses previously generated words 1 through i-1 as input when generating word i. This makes generation sequential—a reason why the Transformer is still sequential at inference, though training can be parallelized.

2.2 注意力到底是什么2.2 What Attention Actually Is

论文对注意力下了一个精确的定义：「An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.」The paper gives a precise definition: "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors."

翻译成人话：想象你在一个图书馆查资料。你有一个问题（Query），图书馆里每本书有一个书名/标签（Key）和内容（Value）。注意力做的事就是：拿你的问题去和每本书的标签做比对，算出一个匹配度分数；然后对这些分数做 softmax（变成一组加起来等于 1 的权重）；最后把每本书的内容按权重加权求和——匹配度高的书贡献大，匹配度低的贡献小。你得到的就是一个「根据问题定制的信息摘要」。In plain language: imagine looking up information in a library. You have a question (Query). Each book has a title/label (Key) and content (Value). Attention does this: compare your question to each book's label, computing a match score. Softmax these scores into weights summing to 1. Then weight-sum all books' contents—highly matched books contribute more. You get a "question-customized information summary."

图书馆的比方The Library Analogy

Query = 你的检索问题（「我想了解注意力机制」）
Key = 每本书的书名和标签（「深度学习」「菜谱」「注意力机制原理」...）
Value = 每本书的实际内容

注意力做的事：拿 Query 和每个 Key 算相似度，相似度高的书权重高；然后按权重把所有 Value 混在一起，得到一个「针对你的问题的综合答案」。Query = your search question ("I want to understand attention mechanisms")
Key = each book's title and label ("Deep Learning," "Cookbook," "Attention Mechanism Principles"...)
Value = each book's actual content

Attention: compute similarity between Query and each Key. High similarity = high weight. Weight-sum all Values to get a "question-tailored answer."

这个比方在哪里不灵：图书馆里你只借几本书，但注意力是把所有书的内容按权重混合——即使权重很小，每本书都参与了。而且 Key 和 Value 在 Transformer 中不一定对应同一种东西，它们是从同一个输入学出来的不同投影。Where it breaks: in a library you borrow just a few books, but attention mixes all books weighted—every book participates, even with tiny weight. Also, Key and Value in the Transformer aren't necessarily the same thing; they're different learned projections of the same input.

2.3 输出是怎么算出来的2.3 How Output Is Computed

论文说得很简洁：「The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.」The paper says it concisely: "The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key."

分两步：Two steps:

1. 算权重：用 compatibility function（兼容函数）算 Query 和每个 Key 的匹配度。在 Transformer 里这个函数就是点积，再除以 sqrt(d_k)，再过 softmax。1. Compute weights: Use a compatibility function to score the match between Query and each Key. In the Transformer, this is dot product, divided by sqrt(d_k), then softmax.

2. 加权求和：用这些权重对 Value 做加权平均。2. Weighted sum: Weight-average the Values using these weights.

为什么用 softmax？因为 softmax 把任意大小的分数变成一组 0 到 1 之间、总和为 1 的概率。这正好符合「权重」的要求——每个位置的权重是正的，而且所有权重加起来等于 1。这意味着输出是 Value 的凸组合。Why softmax? It transforms arbitrary scores into weights between 0 and 1 that sum to 1—exactly the requirements for weights. This means the output is a convex combination of Values.

2.4 自注意力 vs 编码器-解码器注意力2.4 Self-Attention vs Encoder-Decoder Attention

在理解了 Q、K、V 之后，最重要的区分是：它们从哪里来？After understanding Q, K, V, the most important distinction is: where do they come from?

自注意力（Self-Attention）：Q、K、V 都来自同一个地方——上一层的输出。比如编码器中的自注意力，每个位置看看自己所在的序列里其他所有位置，从自己的角度决定「我应该关注谁」。Self-Attention: Q, K, V all come from the same place—the previous layer's output. In the encoder's self-attention, each position looks at all other positions in its own sequence, deciding "who should I attend to."

编码器-解码器注意力（Encoder-Decoder Attention）：Q 来自解码器上一层，K 和 V 来自编码器最终输出。这让解码器的每个位置都能看到输入序列的所有位置——就像翻译时，每翻译一个词都要回头看原文。Encoder-Decoder Attention: Q comes from the decoder's previous layer; K and V come from the encoder's final output. This lets each decoder position see all input positions—like looking back at the source text while translating each word.

一句话记住：注意力 = 拿 Query 和每个 Key 算匹配度，用 softmax 变成权重，然后对 Value 做加权求和。自注意力是 Q/K/V 同源的注意力。In one sentence:Attention = compute Query-Key match, softmax to weights, weighted-sum Values. Self-attention is when Q/K/V share the same source.

我能用自己的话解释 Q、K、V 各自的角色I can explain the roles of Q, K, V in my own words

我能解释 softmax 在注意力里为什么是必要的I can explain why softmax is necessary in attention

我能说出自注意力和编码器-解码器注意力的 Q/K/V 来源区别I can distinguish Q/K/V sources between self-attention and encoder-decoder attention

L2 · 变形迁移 Q/K/V 概念Q/K/V concept

用数据库查询的类比解释 Query、Key、Value 三者的关系，并写出注意力权重的计算步骤（点积→缩放→softmax→加权求和）。Using a database query analogy, explain the relationship between Query, Key, and Value. Write out the attention weight computation steps (dot product -> scale -> softmax -> weighted sum).

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果 Q 和 K 完全相同，注意力权重会是什么样的？

本章小结Chapter Summary

注意力的本质是：用一个 Query 去和一组 Key 匹配，根据匹配度对 Value 做加权求和。Transformer 把这个机制放到核心位置，让序列内的每个位置都能直接和其他所有位置交互——这就是自注意力。下一篇章将进入论文最核心的公式：缩放点积注意力。The essence of attention: use a Query to match against a set of Keys, weight-sum Values by match scores. The Transformer puts this mechanism at the core, letting each position interact directly with all others—this is self-attention. The next chapter enters the paper's most central formula: scaled dot-product attention.

第3章缩放点积注意力Chapter 3: Scaled Dot-Product Attention

论文 3.2.1 Scaled Dot-Product Attention（第4页），图 2 左

这是整篇论文最核心的公式——Attention(Q,K,V) = softmax(QK^T/sqrt(d_k))V。一行公式，但每一个细节都有来历。这一章把它拆到分子级别。This is the paper's most central formula—Attention(Q,K,V) = softmax(QK^T/sqrt(d_k))V. One line, but every detail has a backstory. This chapter takes it apart to the molecular level.

学完这一章你应该能做到After this chapter you should be able to

默写出缩放点积注意力的完整公式并解释每个符号Write the scaled dot-product attention formula from memory and explain every symbol
解释为什么要除以 sqrt(d_k) 而不是 d_k 或 d_k^2Explain why divide by sqrt(d_k) and not d_k or d_k^2
说出点积注意力和加性注意力的区别和各自优劣Distinguish dot-product from additive attention and their tradeoffs
在实验室里亲手算出一组注意力输出Compute attention output by hand in the lab

需要先读第2章自注意力的核心直觉。Requires Chapter 2.

3.1 公式全貌3.1 The Full Formula

论文给出的公式（公式 1）：The paper's formula (Equation 1):

Attention(Q, K, V) = softmax(QK^T / √d_k)V

(1)

别被这行公式吓到，它其实就是三步操作的紧凑写法：Don't be intimidated—it's three operations written compactly:

推导：从直觉到公式Derivation: From Intuition to Formula

第一步：算匹配度。Q 乘以 K 的转置。Q 是 n 行 d_k 列的矩阵（n 个 query，每个 d_k 维），K 也是 n 行 d_k 列。QK^T 得到一个 n x n 的矩阵——第 i 行第 j 列就是第 i 个 query 和第 j 个 key 的点积，也就是它们的匹配度。Step 1: Compute compatibility. Q times K transpose. Q is n x d_k (n queries, each d_k dimensional), K is also n x d_k. QK^T gives an n x n matrix—entry (i,j) is the dot product of query i and key j, i.e., their compatibility.
第二步：缩放。把上一步的结果除以 sqrt(d_k)。这一步看起来不起眼，但它是整篇论文里最容易被忽视的关键设计——后面会详细解释为什么。Step 2: Scale. Divide by sqrt(d_k). This seemingly trivial step is the most easily overlooked critical design in the paper—we'll explain why shortly.
第三步：softmax + 加权。对每一行做 softmax（让每个位置的权重变成 0-1 之间、加起来等于 1），然后把得到的权重矩阵乘以 V——就是对 Value 做加权求和。Step 3: Softmax + weight. Softmax each row (weights become 0-1, summing to 1), then multiply by V—weighted sum of Values.

符号	含义	维度
Q	Query 矩阵，每个查询向量	n x d_k
K	Key 矩阵，每个键向量	n x d_k
V	Value 矩阵，每个值向量	n x d_v
d_k	Key/Query 的维度（base 模型中 = 64）	标量
QK^T	Query 和 Key 的点积矩阵（匹配度	n x n
softmax(...)	逐行归一化，变成权重	n x n
输出	注意力的最终输出	n x d_v

3.2 为什么要缩放：sqrt(d_k) 的来历3.2 Why Scale: The Story of sqrt(d_k)

这是整篇论文最精妙的设计之一。论文用了一个脚注（脚注 4）来解释，这个脚注的信息量比很多论文的整节还大。This is one of the paper's most elegant designs. The explanation comes in a footnote (#4) that contains more information than many papers' entire sections.

问题：点积为什么会爆炸Problem: Why Dot Products Explode

假设 q 和 k 的每个分量都是独立的随机变量，均值为 0，方差为 1。那么它们的点积 q . k = sum_{i=1}^{d_k} q_i * k_i 的均值为 0，但方差是 d_k。这意味着当 d_k 很大时（比如 64），点积的值会在很大范围内波动——标准差是 sqrt(d_k) = 8。Assume q and k components are independent random variables with mean 0, variance 1. Their dot product q . k = sum_{i=1}^{d_k} q_i * k_i has mean 0 but variance d_k. When d_k is large (e.g., 64), the dot product fluctuates wildly—standard deviation sqrt(d_k) = 8.

这为什么是个问题？因为点积的值太大会把 softmax 推到「饱和区」。softmax 的公式是 exp(x_i) / sum_j exp(x_j)。当输入值非常大时，最大值对应的 exp 项会远大于其他项，导致 softmax 输出几乎变成 one-hot（一个位置接近 1，其他接近 0）。在这种饱和区域，softmax 的梯度趋近于零——学不动了。Why is this a problem? Large dot products push softmax into the "saturation zone." Softmax: exp(x_i) / sum_j exp(x_j). When inputs are very large, the max term dominates, making output almost one-hot. In saturation, softmax gradients approach zero—learning stalls.

解决方案：除以 sqrt(d_k)。这个操作把点积的方差从 d_k 降回 1（标准差从 sqrt(d_k) 降回 1），让 softmax 工作在一个梯度健康的区间。Solution: divide by sqrt(d_k). This reduces dot product variance from d_k back to 1 (std from sqrt(d_k) to 1), keeping softmax in a healthy-gradient regime.

存疑点：为什么不除以 d_kOpen Question: Why Not d_k?

论文没有解释为什么选 sqrt(d_k) 而不是 d_k。从方差分析来看，除以 sqrt(d_k) 恰好把方差标准化为 1——这在统计上是最合理的缩放。如果除以 d_k，会把方差压到 1/d_k，可能过度压缩。但论文没有做对比实验验证这个选择的敏感性。这是一个「看起来合理但未经验证」的设计决策。The paper doesn't explain why sqrt(d_k) instead of d_k. From variance analysis, dividing by sqrt(d_k) normalizes variance to 1—statistically the most reasonable. Dividing by d_k would compress variance to 1/d_k, possibly over-compressing. But no sensitivity experiments validate this choice. It's a "reasonable but unverified" design decision.

实验室 3-1：缩放点积注意力Lab 3-1: Scaled Dot-Product Attention可运行

序列长度 nSeq length n 4

维度 d_kDimension d_k 64

随机种子Random seed 42

点击运行查看结果

说明：Q、K、V 用指定种子随机生成（均值 0，方差 1）。实验室计算 QK^T，除以 sqrt(d_k)（或不除），做 softmax，乘以 V。输出显示：(1) 原始点积矩阵的范围（最大值/最小值），(2) softmax 后的最大权重（接近 1 表示饱和），(3) 输出向量第一行的方差。数字来自论文 3.2.1 节的方差分析。这个实验能证明缩放如何防止 softmax 饱和，但不能证明缩放是全局最优选择。 Note: Q, K, V generated randomly (mean 0, var 1). Lab computes QK^T, divides by sqrt(d_k) (or not), softmaxes, multiplies by V. Output shows: (1) raw dot product range, (2) max softmax weight (near 1 = saturation), (3) output row 1 variance. Numbers from paper Section 3.2.1 variance analysis. This lab demonstrates how scaling prevents softmax saturation, but can't prove scaling is globally optimal.

3.3 点积注意力 vs 加性注意力3.3 Dot-Product vs Additive Attention

论文对比了两种主流注意力：点积注意力（dot-product attention）和加性注意力（additive attention）。The paper compares two mainstream attention types: dot-product and additive attention.

加性注意力（Additive Attention，Bahdanau et al. 2014）：用一个前馈网络（单隐藏层）来计算 Query 和 Key 的兼容函数。公式大致是 v^T * tanh(W_1 * q + W_2 * k)。Additive Attention (Bahdanau et al. 2014): uses a feed-forward network (single hidden layer) to compute Query-Key compatibility: v^T * tanh(W_1 * q + W_2 * k).

论文指出的关键区别：Key differences the paper identifies:

方面	点积注意力	加性注意力
理论复杂度	相似	相似
实际速度	快得多	较慢
空间效率	高（矩阵乘法优化好）	较低
小 d_k 时性能	与加性相当	与点积相当
大 d_k 时性能	不缩放时比加性差	更好

为什么点积更快Why Dot-Product Is Faster

点积注意力 QK^T 本质上就是矩阵乘法，这是 GPU 最擅长的操作——高度优化的 BLAS 库（cuBLAS）可以让矩阵乘法跑得飞快。加性注意力需要逐元素计算 tanh 和额外的矩阵乘法，无法直接利用这些优化。论文说得很直接：「it can be implemented using highly optimized matrix multiplication code」。Dot-product attention QK^T is essentially matrix multiplication—what GPUs do best. Highly optimized BLAS (cuBLAS) makes this extremely fast. Additive attention requires element-wise tanh and extra matrix multiplies, unable to directly leverage these optimizations. The paper says: "it can be implemented using highly optimized matrix multiplication code."

论文引用了 Britz et al. 2017 的发现：在不加缩放的情况下，当 d_k 较大时，加性注意力优于点积注意力。而 Transformer 的解决方案是：保持点积的效率优势，通过缩放来弥补它在高维时的性能劣势——两全其美。The paper cites Britz et al. 2017: without scaling, additive attention outperforms dot-product for large d_k. The Transformer's solution: keep dot-product's efficiency, fix its high-dimensional weakness with scaling—best of both worlds.

一句话记住：缩放点积注意力 = 点积（快） + 除以 sqrt(d_k)（防饱和） + softmax（归一化） + 乘 V（加权）。除以 sqrt(d_k) 是点积注意力在大维度下不输加性注意力的关键。In one sentence:Scaled dot-product attention = dot product (fast) + divide by sqrt(d_k) (prevent saturation) + softmax (normalize) + multiply V (weight). Dividing by sqrt(d_k) is what makes dot-product not lose to additive attention at high dimensions.

我能默写缩放点积注意力公式并解释每个符号I can write the scaled dot-product attention formula from memory and explain every symbol

我能解释 sqrt(d_k) 缩放的统计学理由I can explain the statistical reasoning behind sqrt(d_k) scaling

我能在实验室里看到去掉缩放后 softmax 怎么饱和I can see softmax saturation without scaling in the lab

我能说出点积注意力和加性注意力各自的优势I can state the advantages of dot-product vs additive attention

L2 · 变形迁移缩放因子Scaling factor

假设 d_k = 100，Q 和 K 的分量均值为 0、方差为 1。不缩放时，QK^T 中单个元素的方差是多少？缩放后呢？Given d_k = 100, Q and K components have mean 0, variance 1. Without scaling, what's the variance of a single QK^T element? After scaling?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果 d_k = 1，还需要缩放吗？为什么？

L3 · 构造反例 softmax 饱和softmax saturation

假设 softmax 的输入是 [10, 0, -5]。请计算 softmax 输出，并解释为什么这种状态下梯度会消失。如果输入变成 [0.5, 0, -0.25] 呢？Given softmax input [10, 0, -5], compute the output and explain why gradients vanish. What if input becomes [0.5, 0, -0.25]?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果 softmax 输入全为同一个值 c，输出是什么？梯度呢？

L4 · 多概念综合设计选型Design choice

为什么 Transformer 选择了点积注意力而不是加性注意力？如果让你在 d_k = 512 的场景下设计注意力，你会用哪种方案？给出你的理由。Why did the Transformer choose dot-product over additive attention? If you were designing attention for d_k = 512, which would you choose and why?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果 d_k 极小（比如 4），缩放还有必要吗？从方差角度分析。

答辩：如果我是审稿人Debate: If I Were a Reviewer

你说除以 sqrt(d_k) 是为了控制方差——但这个分析基于 Q 和 K 的分量是均值 0、方差 1 的独立随机变量的假设。训练后的模型里 Q 和 K 真的还是这个分布吗？如果不满足，缩放还有意义吗？You say divide by sqrt(d_k) to control variance—but this analysis assumes Q and K components are independent with mean 0, variance 1. In trained models, do Q and K still follow this distribution? If not, does scaling still make sense?

参考防守（先自己组织语言再看）Reference defense (try your own words first)

这是一个合理的质疑。训练后的 Q 和 K 不一定是严格的均值 0、方差 1。但缩放的意义在于提供一个「数量级修正」——即使分布不完全满足假设，除以 sqrt(d_k) 也能把点积从可能非常大的值（如几百）降到合理的量级（如几十）。更重要的是，层归一化（LayerNorm）在 Transformer 中的大量使用会帮助控制输入到注意力的向量的尺度，使得方差分析作为一阶近似仍然有指导意义。事实上，后来的研究发现即使去掉缩放，配合更好的初始化和归一化策略也能训练，但缩放作为「安全网」仍然是一个好设计。This is a fair objection. Trained Q and K won't have exact mean 0, variance 1. But scaling provides an "order-of-magnitude correction"—even if the distribution doesn't perfectly satisfy the assumption, dividing by sqrt(d_k) brings potentially huge dot products (hundreds) to reasonable magnitude (tens). More importantly, LayerNorm in the Transformer helps control input vector scale, making the variance analysis a useful first-order approximation. Later research shows training without scaling is possible with better initialization and normalization, but scaling as a "safety net" remains a good design.

一句话记住：Attention(Q,K,V) = softmax(QK^T/sqrt(d_k))V。除以 sqrt(d_k) 是因为点积方差是 d_k，不缩放会让 softmax 饱和、梯度消失。点积注意力比加性注意力快，缩放弥补了它在大维度下的劣势。In one sentence:Attention(Q,K,V) = softmax(QK^T/sqrt(d_k))V. Divide by sqrt(d_k) because dot product variance is d_k; without scaling, softmax saturates and gradients vanish. Dot-product is faster than additive; scaling fixes its weakness at high dimensions.

本章自测Chapter Quiz

以下题目由系统自动判分，答题记录接入间隔重复算法。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

缩放点积注意力是 Transformer 的心脏。三步操作——点积、缩放、softmax+加权——每一步都有明确的设计理由。点积提供高效性，缩放保证训练稳定性，softmax 让权重归一化。下一章将看到，单个注意力不够——多头注意力让模型同时从不同角度关注序列。Scaled dot-product attention is the Transformer's heart. Three steps—dot product, scaling, softmax+weighting—each with clear rationale. Dot product provides efficiency, scaling ensures training stability, softmax normalizes weights. The next chapter shows that single attention isn't enough—multi-head attention lets the model attend from different perspectives simultaneously.

第4章多头注意力Chapter 4: Multi-Head Attention

上一章我们学会了「缩放点积注意力」——它让序列中的每个位置都能「看到」所有其他位置。但如果只有一只「眼睛」，它只能学到一个视角的注意力模式。这就像你只有一只眼睛：能看到东西，但没有深度感知。

Transformer 的解决方案是：给模型安上多只眼睛，每只眼睛从不同的子空间看同一句话，然后把这些视角拼接起来。这就是「多头注意力」（Multi-Head Attention）。

In the previous chapter we learned scaled dot-product attention — it lets every position in a sequence "see" all other positions. But with only one "eye," the model can only learn a single attention pattern. It's like having one eye: you can see, but you lack depth perception.

The Transformer's solution: give the model multiple eyes, each looking at the same sentence from a different subspace, then concatenate these views. This is Multi-Head Attention.

4.1 为什么要多头？4.1 Why Multiple Heads?

先想一个具体场景。英文句子 "The animal didn't cross the street because it was too tired." 这里的 "it" 指什么？是 animal 还是 street？

人类的阅读过程中会同时做几件事：（1）语法层面，"it" 是主语，需要找动词；（2）语义层面，"tired" 通常修饰动物而非街道；（3）指代消解，回头看最近的可能指代对象。这些是不同层面的注意力模式。

如果只有一个注意力头，它必须用同一组 Q/K/V 投影来同时捕获所有这些模式。这就像让一个人同时当语法学家、语义学家和指代消解专家——能做，但每个方面都做得不够好。

Consider the sentence "The animal didn't cross the street because it was too tired." What does "it" refer to — the animal or the street?

Human reading simultaneously performs several tasks: (1) syntactically, "it" is a subject that needs a verb; (2) semantically, "tired" typically modifies animals, not streets; (3) coreference resolution, looking back at the nearest plausible referent. These are different types of attention patterns.

With a single attention head, the model must use the same Q/K/V projections to capture all these patterns simultaneously. It's like asking one person to be a syntactician, semanticist, and coreference expert all at once — doable, but nothing done well.

论文原文说得很清楚：

"Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this."

The paper states clearly: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this."

关键词是 "representation subspaces"（表示子空间）。每个头有自己独立的 W_Q、W_K、W_V 投影矩阵，把 512 维的输入投影到 64 维的子空间。8 个头就是在 8 个不同的 64 维子空间里分别做注意力，然后把结果拼回来。

The key word is "representation subspaces." Each head has its own independent W_Q, W_K, W_V projection matrices, projecting the 512-dimensional input into a 64-dimensional subspace. 8 heads perform attention in 8 different 64-dimensional subspaces, then concatenate the results back.

4.2 数学公式4.2 Mathematical Formulation

多头注意力的公式分两步。第一步，每个头独立做注意力：

The multi-head attention formula has two steps. First, each head independently performs attention:

head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)

其中 W_i^Q 是一个 512×64 的矩阵（把 d_model=512 维投影到 d_k=64 维），同理 W_i^K 和 W_i^V。每个头用不同的投影矩阵，所以它们看同一个输入的方式不同。

Where W_i^Q is a 512×64 matrix (projecting from d_model=512 to d_k=64 dimensions), similarly for W_i^K and W_i^V. Each head uses different projection matrices, so they "look at" the same input differently.

第二步，把 8 个头的输出拼接起来，再做一次线性变换：

Second step, concatenate the outputs of 8 heads, then apply another linear transformation:

MultiHead(Q, K, V) = Concat(head₁, ..., head_h) W^O

每个 head 的输出是 d_v=64 维，8 个头拼起来就是 8×64=512 维，正好和 d_model 一致。然后乘以 W^O（512×512 矩阵）做一次混合，得到最终输出。

Each head's output is d_v=64 dimensional; 8 heads concatenated give 8×64=512 dimensions, matching d_model. Then multiply by W^O (512×512 matrix) for a final mixing, producing the output.

维度变化全程跟踪（以 base 模型为例）：

1. 输入 Q, K, V：各为 (seq_len, 512)

2. 第 i 个头投影：Q×W_i^Q → (seq_len, 64)，同理 K, V

3. 第 i 个头注意力：softmax(QK^T/√64)V → (seq_len, 64)

4. 拼接 8 个头：(seq_len, 512)

5. 乘以 W^O：(seq_len, 512) × (512, 512) → (seq_len, 512)

输入和输出维度一致！这意味着多头注意力可以堆叠多层。

Full dimension tracking (base model):

1. Input Q, K, V: each (seq_len, 512)

2. Head i projection: Q×W_i^Q → (seq_len, 64), same for K, V

3. Head i attention: softmax(QK^T/√64)V → (seq_len, 64)

4. Concatenate 8 heads: (seq_len, 512)

5. Multiply by W^O: (seq_len, 512) × (512, 512) → (seq_len, 512)

Input and output dimensions match! This means multi-head attention can be stacked in multiple layers.

4.3 计算成本分析：为什么「多头」不亏？4.3 Computational Cost: Why Multi-Head Is "Free"

你可能担心：8 个头是不是让计算量翻 8 倍？答案是没有。论文做了一个精妙的设计：每个头的维度从 512 降到了 64，8×64=512，总维度不变。

You might worry: do 8 heads make computation 8× more expensive? The answer is no. The paper makes a clever design: each head's dimension drops from 512 to 64, and 8×64=512, keeping the total dimension constant.

具体来说，单头全维度注意力的投影矩阵是 512×512，计算量是 O(n×512×512)。多头注意力每个头的投影矩阵是 512×64，8 个头的总计算量是 8×O(n×512×64) = O(n×512×512)。完全一样！

Specifically, single-head full-dimension attention has projection matrices of 512×512, with cost O(n×512×512). Multi-head attention has each head's projection at 512×64, total cost 8×O(n×512×64) = O(n×512×512). Exactly the same!

论文原话："Due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality." 这是整个 Transformer 设计哲学的一个缩影：不增加成本，但增加表达能力。

The paper says: "Due to the reduced dimension of each head, the total computational cost is similar to that of single-head attention with full dimensionality." This embodies the Transformer's design philosophy: no extra cost, but more expressive power.

4.4 多头到底做了什么？实验证据4.4 What Do Heads Actually Do? Experimental Evidence

论文的附录展示了注意力可视化（Figure 3-5），揭示了不同头确实学到了不同的功能：

The paper's appendix shows attention visualizations (Figures 3-5), revealing that different heads indeed learn different functions:

长距离依赖：Figure 3 展示了第 5 层的注意力头如何将 "making" 和相隔很远的 "more difficult" 关联起来。多个注意力头（不同颜色）都关注到了这个长距离依赖。
指代消解：Figure 4 展示了第 5 层的两个注意力头（head 5 和 head 6）在做指代消解。对于句子 "The Law will never be perfect, but its application should be just..."，"its" 的注意力非常尖锐地指向 "Law"。
不同的结构模式：Figure 5 展示了两个不同的头在做完全不同的事——一个头关注相邻词（学习局部语法），另一个头关注句末的标点（学习句子结构）。
Long-distance dependencies: Figure 3 shows how attention heads in layer 5 connect "making" with the distant "more difficult." Multiple heads (different colors) attend to this long-range dependency.
Anaphora resolution: Figure 4 shows two heads (head 5 and head 6) in layer 5 performing coreference resolution. For "The Law will never be perfect, but its application should be just...", "its" attends sharply to "Law."
Different structural patterns: Figure 5 shows two different heads doing completely different things — one attends to adjacent words (learning local syntax), another attends to sentence-final punctuation (learning sentence structure).

这些可视化是论文的「彩蛋」——它们不仅证明了多头机制有效，还让 Transformer 比当时的 RNN/CNN 模型更可解释。论文第 4 节最后提到："As side benefit, self-attention could yield more interpretable models."

These visualizations are a "bonus" of the paper — they not only prove that multi-head mechanism works, but also make the Transformer more interpretable than contemporary RNN/CNN models. Section 4 concludes: "As side benefit, self-attention could yield more interpretable models."

实验室：多头注意力模拟器Lab: Multi-Head Attention Simulator

输入句子，观察 8 个注意力头的注意力分布。用不同的颜色区分不同头，看看它们是否关注了不同的词。

Input a sentence and observe the attention distribution across 8 heads. Different colors distinguish heads — see if they attend to different words.

句子：Sentence:
观察词位置：Observe word position: 头数：Number of heads:

L2 · 变形迁移多头维度计算Multi-head dimension

d_model = 512，h = 8。每个头的维度 d_k 是多少？8 个头拼接后的总维度是多少？为什么说多头注意力不增加计算成本？Given d_model = 512, h = 8. What is each head's dimension d_k? What is the total dimension after concatenating 8 heads? Why does multi-head attention not increase computational cost?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果 h=512（每头1维），会发生什么？

检查点：你能用自己的话解释多头注意力为什么不需要额外计算成本吗？关键在于维度分配。

Checkpoint: Can you explain in your own words why multi-head attention doesn't require extra computational cost? The key is dimension allocation.

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

多头注意力是 Transformer 的精妙设计：将 d_model 维空间切成 h 个子空间，每个头在一个子空间里独立做注意力，最后拼接回来。由于每个头的维度按比例缩小（d_k = d_model/h），总计算量与单头全维度注意力相同。实验证明不同头确实学到了不同的功能——语法、语义、指代、结构——让模型更强大也更可解释。

Multi-head attention is the Transformer's elegant design: split the d_model-dimensional space into h subspaces, each head performs attention independently in one subspace, then concatenate. Since each head's dimension scales proportionally (d_k = d_model/h), total computation equals single-head full-dimension attention. Experiments prove different heads learn different functions — syntax, semantics, coreference, structure — making the model both more powerful and more interpretable.

第5章注意力在 Transformer 中的三种用法Chapter 5: Three Applications of Attention in the Transformer

到目前为止，我们学了「注意力」这个工具。但工具要放在哪里、怎么用，决定了它能做什么。Transformer 把多头注意力用在了三个不同的地方，每个地方 Q/K/V 的来源不同，功能也不同。

So far we've learned the "attention" tool. But where and how to place it determines what it can do. The Transformer uses multi-head attention in three different places, each with different sources for Q/K/V and different functions.

5.1 编码器自注意力（Encoder Self-Attention）5.1 Encoder Self-Attention

在编码器的每一层中，Q、K、V 全部来自同一个来源——上一层的输出。这意味着源语言句子的每个词都可以直接关注到句子里所有其他词。

In each encoder layer, Q, K, V all come from the same source — the output of the previous layer. This means every word in the source sentence can directly attend to all other words in the sentence.

举个翻译的例子：英文 "The bank of the river" → 德语翻译。在编码器自注意力中，"bank" 可以同时关注到 "river"（确定 bank 是河岸而非银行）、"the"（语法角色）、"of"（介词关系）。这种全局视野让编码器能生成包含全局上下文的表示。

Translation example: English "The bank of the river" → German. In encoder self-attention, "bank" can simultaneously attend to "river" (determining bank = riverbank, not financial institution), "the" (grammatical role), "of" (prepositional relationship). This global view lets the encoder produce context-rich representations.

关键特征：Q=K=V 来源相同（都是上一层输出），无掩码，双向注意力。

Key features: Q=K=V share the same source (previous layer output), no masking, bidirectional attention.

5.2 编码器-解码器注意力（Encoder-Decoder Attention）5.2 Encoder-Decoder Attention

这是连接编码器和解码器的「桥梁」。在这里：

This is the "bridge" connecting encoder and decoder. Here:

Q 来自解码器上一层的输出
K 和 V 来自编码器最后一层的输出
Q comes from the previous decoder layer's output
K and V come from the encoder's final layer output

这意味着什么？解码器在生成每个目标词时，可以「回看」整个源语言句子，决定该关注源句的哪些部分。这正是传统序列到序列模型中注意力机制的功能。

What does this mean? When generating each target word, the decoder can "look back" at the entire source sentence and decide which parts to focus on. This is exactly the function of attention in traditional sequence-to-sequence models.

直觉类比：编码器像是一个翻译官读完整篇英文原文，在脑子里形成了理解。解码器像是另一个翻译官，一边写德语翻译，一边不断回头看第一个翻译官的笔记（编码器输出），决定当前该翻译原文的哪个部分。

类比的失效处：这个类比中两个翻译官是分开的，但实际上编码器和解码器是同一个神经网络的两个部分，通过梯度反向传播共同训练。

Intuitive analogy: The encoder is like a translator who reads the entire English text and forms an understanding. The decoder is another translator who writes the German translation while constantly glancing back at the first translator's notes (encoder output), deciding which part of the original to translate now.

Where the analogy breaks: In this analogy the two translators are separate, but the encoder and decoder are actually two parts of the same neural network, jointly trained through backpropagation.

5.3 解码器自注意力（Decoder Self-Attention, with Masking）5.3 Decoder Self-Attention (with Masking)

解码器也有自注意力层，但有一个关键限制：掩码（masking）。在生成第 i 个词时，解码器只能看到前 i-1 个词，不能看到第 i 个词及之后的词。这保证了自回归性质——预测不能「偷看答案」。

The decoder also has self-attention layers, but with a crucial restriction: masking. When generating the i-th word, the decoder can only see the first i-1 words, not the i-th word or later. This preserves the auto-regressive property — predictions cannot "peek at answers."

具体实现：在 softmax 之前，把对应「未来位置」的 QK^T 值设为负无穷（-∞）。softmax 后这些位置的权重就变成 0，相当于这些连接不存在。

Implementation: before softmax, set the QK^T values corresponding to "future positions" to negative infinity (-∞). After softmax, these positions get zero weight, effectively making these connections non-existent.

Mask = [[0, -∞, -∞, -∞],
         [0,   0,   -∞, -∞],
         [0,   0,     0,   -∞],
         [0,   0,     0,     0]]

上面的矩阵是一个 4×4 的下三角掩码。位置 1 只能看位置 1，位置 2 能看 1-2，以此类推。这个掩码加上 QK^T/√d_k 后再过 softmax，就保证了因果性。

The above is a 4×4 lower-triangular mask. Position 1 can only see position 1, position 2 can see 1-2, and so on. This mask is added to QK^T/√d_k before softmax, ensuring causality.

5.4 三种用法对比表5.4 Comparison Table

用法Usage	Q 来源Q Source	K/V 来源K/V Source	掩码Mask	功能Function
编码器自注意力Encoder Self-Attn	上一层输出Prev layer	上一层输出Prev layer	无None	源句全局理解Source context
编码器-解码器Enc-Dec Attn	解码器上层Decoder layer	编码器最终输出Encoder output	无None	桥接源与目标Bridge source/target
解码器自注意力Decoder Self-Attn	上一层输出Prev layer	上一层输出Prev layer	因果掩码Causal mask	目标序列自回归Auto-regressive

L3 · 构造反例三种注意力用法Three attention applications

Transformer 中有三种注意力用法。分别列出每种用法的 Q、K、V 来源，并说明是否需要掩码（masking）。There are three attention applications in the Transformer. For each, identify the source of Q, K, V and state whether masking is required.

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果去掉 enc-dec 注意力，解码器还能做翻译吗？

检查点：画一个 Transformer 的简化图，标出三种注意力各自的位置、Q/K/V 来源和掩码情况。

Checkpoint: Draw a simplified Transformer diagram, marking the three attention types' positions, Q/K/V sources, and masking.

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

Transformer 在三个位置使用多头注意力，每种用法的 Q/K/V 来源不同：编码器自注意力让源句双向关注全局上下文；编码器-解码器注意力让解码器回看源句；解码器自注意力用因果掩码保证自回归性质。三种用法配合，形成了完整的编码-解码翻译管道。

The Transformer uses multi-head attention in three positions, each with different Q/K/V sources: encoder self-attention enables bidirectional global context in the source; encoder-decoder attention lets the decoder look back at the source; decoder self-attention uses causal masking to preserve auto-regression. Together, they form a complete encode-decode translation pipeline.

第6章逐位置前馈网络Chapter 6: Position-wise Feed-Forward Network

注意力层让序列中不同位置的词相互交流。但交流完之后，每个位置还需要独立地「消化」这些信息——把接收到的上下文做一次非线性变换。这个任务由逐位置前馈网络（Position-wise Feed-Forward Network, FFN）完成。

The attention layer lets words at different positions communicate. But after communication, each position needs to independently "digest" the information — applying a nonlinear transformation to the received context. This task is handled by the Position-wise Feed-Forward Network (FFN).

6.1 FFN 的结构6.1 FFN Structure

FFN 是一个两层全连接网络，中间夹一个 ReLU 激活函数：

FFN is a two-layer fully connected network with a ReLU activation in between:

FFN(x) = max(0, xW₁ + b₁)W₂ + b₂

维度变化：输入是 d_model=512 维，第一层把它扩展到 d_ff=2048 维（4 倍扩展），ReLU 激活后，第二层再压回 512 维。

Dimension flow: input is d_model=512, first layer expands to d_ff=2048 (4× expansion), after ReLU, second layer compresses back to 512.

为什么先扩展再压缩？ 这是一种常见的神经网络设计模式。扩展到更高维度的空间，让 ReLU 有足够的空间做非线性分离（在高维空间中线形不可分的模式可能变得可分），然后再映射回原始维度。可以类比支持向量机（SVM）的核技巧——把数据映射到高维空间使其更易分离。

Why expand then compress? This is a common neural network design pattern. Expanding to a higher-dimensional space gives ReLU room for nonlinear separation (patterns linearly inseparable in low dimensions may become separable in high dimensions), then mapping back. This is analogous to SVM kernel tricks — mapping data to higher dimensions for easier separation.

6.2 「逐位置」是什么意思？6.2 What Does "Position-wise" Mean?

"逐位置"（position-wise）意味着 FFN 对序列中的每个位置独立地、相同地施加。也就是说，位置 1 的 FFN 和位置 2 的 FFN 使用相同的权重矩阵，但处理的是不同位置的输入向量。

"Position-wise" means the FFN is applied independently and identically to each position in the sequence. That is, the FFN for position 1 and position 2 use the same weight matrices, but process different position input vectors.

论文说："Another way of describing this is as two convolutions with kernel size 1." 核大小为 1 的卷积就是对每个位置独立施加的线性变换——数学上等价。

The paper says: "Another way of describing this is as two convolutions with kernel size 1." A kernel-1 convolution is a linear transformation applied independently to each position — mathematically equivalent.

注意：虽然同一层内所有位置共享权重，但不同层的 FFN 有不同的参数。第 1 层的 W₁, W₂ 和第 2 层的 W₁, W₂ 是不同的。

Note: while all positions within a layer share weights, different layers have different FFN parameters. Layer 1's W₁, W₂ differ from Layer 2's.

6.3 FFN 的角色：注意力之后的「思考」6.3 FFN's Role: "Thinking" After Attention

如果把注意力层比作「开会讨论」——各个位置交换信息，那么 FFN 就是会后每个人独立「消化整理」的环节。注意力层负责信息聚合，FFN 负责信息变换。

If the attention layer is like a "meeting" — positions exchanging information — then FFN is the post-meeting "digestion" where each person independently processes what they heard. Attention aggregates information; FFN transforms it.

从函数近似的角度看，注意力是一个相对平滑的操作（加权平均），而 FFN 通过 ReLU 引入了非线性。如果没有 FFN，Transformer 的每一层只是做加权平均的嵌套，表达能力会大大受限。

From a function approximation perspective, attention is a relatively smooth operation (weighted average), while FFN introduces nonlinearity through ReLU. Without FFN, each Transformer layer would just be nested weighted averages, severely limiting expressiveness.

后续发展：后来的研究发现 FFN 的设计可以改进。例如 T5 用 GELU 代替 ReLU，GPT 系列用扩展比更大的 FFN（如 d_ff = 4×d_model），LLaMA 使用 SwiGLU 激活函数。这些变化都是在保持「逐位置变换」核心思想的基础上优化非线性能力。

Later developments: Subsequent research found FFN design can be improved. T5 uses GELU instead of ReLU, GPT series use larger expansion ratios, LLaMA uses SwiGLU. These changes optimize nonlinear capacity while keeping the core "position-wise transformation" idea.

L3 · 构造反例 FFN 非线性作用FFN nonlinearity

如果把 FFN 中的 ReLU 换成恒等函数 f(x) = x，两个线性变换会怎样？用矩阵运算证明你的结论。If you replace ReLU in the FFN with identity f(x) = x, what happens to the two linear transformations? Prove your conclusion using matrix operations.

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果把 ReLU 换成恒等函数 f(x)=x，等价于什么？

检查点：解释为什么 FFN 需要非线性激活函数。如果去掉 ReLU，FFN 会退化成什么？

Checkpoint: Explain why FFN needs a nonlinear activation. Without ReLU, what would FFN degenerate into?

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

逐位置前馈网络由两个线性变换和一个 ReLU 组成，对序列中每个位置独立施加。它将 512 维扩展到 2048 维再压回 512 维，引入非线性能力。FFN 与注意力层互补：注意力负责信息聚合，FFN 负责信息变换。同一层内各位置共享权重，不同层的 FFN 参数不同。

The position-wise FFN consists of two linear transformations with a ReLU, applied independently to each position. It expands 512 to 2048 then back to 512, introducing nonlinearity. FFN complements attention: attention aggregates information, FFN transforms it. Positions within a layer share weights; different layers have different FFN parameters.

第7章嵌入与 SoftmaxChapter 7: Embeddings and Softmax

神经网络不认识文字，只认识数字。把文字变成数字的第一步就是「嵌入」（Embedding）。这一章我们讨论 Transformer 如何做嵌入，以及一个看似奇怪但重要的设计——权重共享。

Neural networks don't understand text, only numbers. The first step in converting text to numbers is "embedding." This chapter discusses how the Transformer handles embeddings and a seemingly strange but important design — weight sharing.

7.1 嵌入层的作用7.1 The Role of Embedding Layers

嵌入层本质上是一个查找表。假设词表有 37000 个词（EN-DE 任务的 BPE 词表大小），每个词对应一个 512 维的向量。嵌入矩阵的大小是 37000×512。给定一个词的 ID，嵌入层就返回对应的行向量。

An embedding layer is essentially a lookup table. With a vocabulary of 37000 tokens (EN-DE BPE vocabulary size), each token corresponds to a 512-dimensional vector. The embedding matrix is 37000×512. Given a token ID, the embedding layer returns the corresponding row vector.

Transformer 需要两个嵌入层：一个在编码器端（源语言），一个在解码器端（目标语言）。此外，解码器最后还有一个线性变换 + Softmax，把 512 维向量映射到词表大小的概率分布。

The Transformer needs two embedding layers: one at the encoder (source language), one at the decoder (target language). Additionally, the decoder has a final linear transformation + Softmax, mapping the 512-dimensional vector to a vocabulary-sized probability distribution.

7.2 权重共享：三个部件，一套权重7.2 Weight Sharing: Three Components, One Weight Matrix

论文做了一个大胆的设计：三个部件共享同一套权重矩阵——

The paper makes a bold design: three components share the same weight matrix —

编码器输入嵌入矩阵
解码器输入嵌入矩阵
解码器输出端 pre-softmax 的线性变换矩阵
Encoder input embedding matrix
Decoder input embedding matrix
Decoder output pre-softmax linear transformation matrix

为什么可以共享？在机器翻译中，源语言和目标语言虽然不同（如英语和德语），但它们使用相同的 BPE 子词单元，共享同一个词表。因此编码器和解码器的嵌入矩阵维度相同，可以共享。

Why can they share? In machine translation, while source and target languages differ (e.g., English and German), they use the same BPE subword units, sharing one vocabulary. So encoder and decoder embedding matrices have the same dimensions and can be shared.

至于 pre-softmax 线性变换——它的作用是「把 512 维的隐状态映射回词表维度」。如果嵌入层是「词表维度 → 512 维」，那么 pre-softmax 就是「512 维 → 词表维度」，数学上正好是嵌入矩阵的转置。所以共享权重在数学上是自然的。

As for the pre-softmax linear transformation — it maps "512-dim hidden state → vocabulary-dim." If embedding is "vocabulary-dim → 512-dim," then pre-softmax is "512-dim → vocabulary-dim," mathematically the transpose of the embedding matrix. So sharing weights is mathematically natural.

权重共享的好处：

1. 参数量减少：从 3 个 37000×512 矩阵变成 1 个，节省约 2×37000×512 ≈ 3800 万参数。

2. 正则化效果：共享权重约束了嵌入和输出投射的对齐，相当于一种正则化，减少过拟合。

3. 语义一致性：同一个词在输入端和输出端有相同的向量表示，强制了语义的一致性。

Benefits of weight sharing:

1. Fewer parameters: From 3 matrices of 37000×512 to 1, saving ~38M parameters.

2. Regularization: Sharing constrains embedding-output alignment, acting as regularization.

3. Semantic consistency: The same token has the same vector representation at input and output, enforcing semantic consistency.

7.3 为什么要乘 √d_model？7.3 Why Multiply by √d_model?

论文提到："In the embedding layers, we multiply those weights by √d_model." 这是一个容易被忽略但很重要的细节。原因如下：

The paper mentions: "In the embedding layers, we multiply those weights by √d_model." This is an easily overlooked but important detail. The reason:

嵌入层的权重通常用初始化方法（如 Xavier 初始化）生成，其方差约为 1/d_model。这意味着嵌入向量的每个分量方差很小，整体 L2 范数约为 1。但位置编码是用 sin/cos 函数生成的，其值域是 [-1, 1]，L2 范数约为 √(d_model/2)。

Embedding weights are typically initialized (e.g., Xavier) with variance ~1/d_model. This means each component of the embedding vector has small variance, with overall L2 norm ~1. But positional encodings use sin/cos functions with values in [-1, 1], L2 norm ~√(d_model/2).

当嵌入向量和位置编码相加时，如果嵌入向量的范数远小于位置编码的范数，位置编码就会「淹没」嵌入的语义信息。乘以 √d_model 把嵌入向量的范数提升到和位置编码同一数量级，确保两者都有足够的「声音」。

When embedding and positional encoding are added, if the embedding's norm is much smaller than the positional encoding's norm, the positional encoding would "drown out" the semantic information. Multiplying by √d_model scales the embedding's norm to the same order as positional encoding, ensuring both have sufficient "voice."

scaled_embedding = embedding_vector × √d_model = embedding_vector × √512 ≈ embedding_vector × 22.6

L3 · 构造反例权重共享Weight sharing

Transformer 中哪三个部件共享权重？共享能节省多少参数（词表 37000，d_model = 512）？共享还有什么附带好处？Which three components share weights in the Transformer? How many parameters does sharing save (vocab 37000, d_model = 512)? What are the additional benefits of sharing?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果源语言和目标语言词表不同，还能共享权重吗？

检查点：解释权重共享减少多少参数，以及为什么嵌入向量要乘 √d_model。

Checkpoint: Explain how many parameters weight sharing saves, and why embeddings are multiplied by √d_model.

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

Transformer 的编码器嵌入、解码器嵌入和 pre-softmax 线性变换共享同一套权重矩阵，减少参数量并提供正则化效果。嵌入向量乘以 √d_model 以确保与位置编码相加时两者数量级匹配。权重共享在数学上是自然的——嵌入和 pre-softmax 互为转置操作。

The Transformer's encoder embedding, decoder embedding, and pre-softmax linear transformation share one weight matrix, reducing parameters and providing regularization. Embeddings are multiplied by √d_model to match positional encoding magnitude when summed. Weight sharing is mathematically natural — embedding and pre-softmax are transpose operations.

第8章位置编码Chapter 8: Positional Encoding

Transformer 去掉了 RNN 和 CNN，换成了纯注意力。但注意力机制有一个「缺陷」：它是置换不变的（permutation invariant）。也就是说，把句子顺序打乱，注意力给出的结果完全一样（只是行的排列不同）。这对语言来说显然不对——"猫追狗"和"狗追猫"意思完全相反。

解决方案：人为注入位置信息。这就是位置编码（Positional Encoding, PE）的任务。

The Transformer removes RNNs and CNNs, replacing them with pure attention. But attention has a "flaw": it's permutation invariant. Shuffle the sentence order, and attention gives the same result (just with rows permuted). This is clearly wrong for language — "cat chases dog" and "dog chases cat" have opposite meanings.

The solution: inject positional information manually. This is the task of Positional Encoding (PE).

8.1 为什么需要位置编码？8.1 Why Positional Encoding?

论文说："Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence."

The paper says: "Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence."

在 RNN 中，位置信息是隐式的：隐藏状态 h_t 依赖于 h_{t-1}，信息按时间顺序逐步传递，自然就包含了位置信息。CNN 通过卷积核的局部感受野，不同位置有不同的连接模式，也隐式包含了位置。但注意力对所有位置一视同仁，必须显式地添加位置信息。

In RNNs, positional information is implicit: hidden state h_t depends on h_{t-1}, information flows sequentially, naturally encoding position. CNNs encode position through local receptive fields — different positions have different connection patterns. But attention treats all positions equally, requiring explicit positional information.

8.2 正弦余弦位置编码8.2 Sinusoidal Positional Encoding

Transformer 使用 sin 和 cos 函数来生成位置编码：

The Transformer uses sin and cos functions to generate positional encodings:

PE(pos, 2i) = sin(pos / 10000^2i/d_model)

PE(pos, 2i+1) = cos(pos / 10000^2i/d_model)

其中 pos 是位置（0, 1, 2, ...），i 是维度索引（0 到 d_model/2-1）。也就是说，512 维的位置编码由 256 个 sin 和 256 个 cos 组成，每个对应不同的频率。

Where pos is the position (0, 1, 2, ...), i is the dimension index (0 to d_model/2-1). The 512-dimensional positional encoding consists of 256 sin and 256 cos components, each at a different frequency.

频率如何变化？对于第 i 个维度对，频率是 1/10000^2i/d_model。当 i=0 时，频率最高（1/1=1），周期最短（2π）；当 i=255 时，频率最低（1/10000），周期最长（10000×2π≈62832）。论文说："The wavelengths form a geometric progression from 2π to 10000 × 2π."

How do frequencies vary? For dimension pair i, the frequency is 1/10000^2i/d_model. When i=0, frequency is highest (1/1=1), period shortest (2π); when i=255, frequency is lowest (1/10000), period longest (10000×2π≈62832). The paper says: "The wavelengths form a geometric progression from 2π to 10000 × 2π."

直觉理解：想象一排不同频率的时钟。高频时钟（小 i）每走一步就大幅变化，能精确定位较近的位置；低频时钟（大 i）走很多步才变化，能区分较远的位置。512 个维度就是 256 对「时钟」，从高频到低频，像一把不同精度的尺子，共同编码位置。

Intuitive understanding: Imagine a row of clocks at different frequencies. High-frequency clocks (small i) change significantly each step, precisely encoding nearby positions; low-frequency clocks (large i) change slowly, distinguishing distant positions. 512 dimensions = 256 "clock pairs" from high to low frequency, like rulers of different precision, jointly encoding position.

8.3 为什么选 sin/cos 而不是学习？8.3 Why sin/cos Instead of Learned?

论文也实验了「可学习的位置嵌入」（learned positional embeddings），结果几乎一样（Table 3 row E：train PPL 4.92 vs 4.92，BLEU 25.7 vs 25.8）。那为什么最终选了 sin/cos？

The paper also experimented with "learned positional embeddings," with nearly identical results (Table 3 row E: train PPL 4.92 vs 4.92, BLEU 25.7 vs 25.8). So why choose sin/cos in the end?

论文给出了一个关键理由："We chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training."

The paper gives a key reason: "We chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training."

可学习的位置嵌入需要为每个位置分配一个可训练向量。如果训练时最长序列是 100 个词，推理时来了 150 个词的句子，第 101-150 个位置没有学过的嵌入，就不知道怎么处理。而 sin/cos 函数可以无限延伸，理论上可以处理任意长度的序列。

Learned positional embeddings need a trainable vector for each position. If training sequences are max 100 tokens, a 150-token sentence at inference has no learned embeddings for positions 101-150. Sin/cos functions extend infinitely, theoretically handling any length.

8.4 相对位置的可表示性8.4 Representability of Relative Positions

论文还给出了一个更深层的理由："We hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PE_pos+k can be represented as a linear function of PE_pos."

The paper also gives a deeper reason: "We hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PE_pos+k can be represented as a linear function of PE_pos."

这是 sin/cos 的一个数学特性：对于固定的偏移量 k，PE(pos+k) 可以表示为 PE(pos) 的线性变换。具体来说，利用三角恒等式：

This is a mathematical property of sin/cos: for fixed offset k, PE(pos+k) can be expressed as a linear transformation of PE(pos). Specifically, using trigonometric identities:

sin(pos+k) = sin(pos)cos(k) + cos(pos)sin(k)

cos(pos+k) = cos(pos)cos(k) - sin(pos)sin(k)

也就是说，PE(pos+k) = M(k) × PE(pos)，其中 M(k) 是一个只依赖于 k（不依赖 pos）的旋转矩阵。这意味着模型可以通过学习一个线性投影来捕获相对位置关系——"这个词在我前面 3 个位置"这样的模式可以被很容易地学到。

That is, PE(pos+k) = M(k) × PE(pos), where M(k) is a rotation matrix depending only on k (not pos). This means the model can capture relative positional relationships by learning a linear projection — patterns like "this word is 3 positions before me" can be easily learned.

后续发展：虽然 sin/cos 位置编码在原始 Transformer 中表现良好，后续模型大多改用了可学习的位置编码或更复杂的方案。例如 GPT 系列使用可学习嵌入，BERT 也使用可学习嵌入。而像 ALiBi、RoPE（旋转位置编码）等方案则进一步改进了位置信息的注入方式，特别是在外推能力上。这说明原始 sin/cos 方案虽然理论上可外推，但实际效果后续被其他方案超越。

Later developments: While sin/cos PE works well in the original Transformer, subsequent models mostly adopted learned PE or more complex schemes. GPT series use learned embeddings, BERT also uses learned embeddings. ALiBi, RoPE (Rotary Position Embedding) etc. further improved positional information injection, especially in extrapolation. This shows that while sin/cos is theoretically extrapolatable, subsequent methods surpassed it in practice.

实验室：位置编码可视化Lab: Positional Encoding Visualization

调整序列长度和维度数，观察正弦余弦位置编码的波形。高频维度变化快（精确近距），低频维度变化慢（区分远距）。

Adjust sequence length and dimension count, observe sin/cos PE waveforms. High-frequency dimensions change fast (precise nearby), low-frequency dimensions change slowly (distinguish far).

序列长度：Seq length: 显示维度对：Show dim pairs:

L3 · 构造反例位置编码必要性Positional encoding necessity

为什么需要位置编码？如果没有位置编码，注意力会有什么问题？说明 sin/cos 位置编码的两个优势。Why is positional encoding necessary? What problem would attention have without it? Explain two advantages of sin/cos positional encoding.

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果把位置编码从加法改为乘法（embedding * PE），效果会怎样？

检查点：解释为什么 sin/cos 位置编码理论上可以处理比训练时更长的序列，以及它如何表达相对位置。

Checkpoint: Explain why sin/cos PE can theoretically handle sequences longer than training, and how it expresses relative positions.

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

Transformer 使用正弦余弦函数生成位置编码，注入到嵌入向量中。不同维度对应不同频率的 sin/cos，波长从 2π 到 10000×2π 几何递增。选择 sin/cos 而非可学习嵌入的理由是外推能力和相对位置的线性可表示性。实验证明两种方案效果几乎相同，后续研究发展出了 RoPE、ALiBi 等更先进的方案。

The Transformer uses sin/cos functions to generate positional encodings, added to embeddings. Different dimensions correspond to different sin/cos frequencies, wavelengths geometrically from 2π to 10000×2π. Sin/cos is chosen over learned embeddings for extrapolation capability and linear representability of relative positions. Experiments show nearly identical results; subsequent research developed RoPE, ALiBi etc.

第9章编码器结构Chapter 9: Encoder Structure

前面几章我们拆解了 Transformer 的各个零件——注意力、FFN、嵌入、位置编码。现在该把它们组装起来了。这一章看编码器的完整结构。

Previous chapters disassembled the Transformer's components — attention, FFN, embeddings, positional encoding. Now let's assemble them. This chapter examines the encoder's complete structure.

9.1 N=6 层堆叠9.1 N=6 Layer Stack

编码器由 N=6 个相同的层堆叠而成。每一层有两个子层：（1）多头自注意力；（2）逐位置前馈网络。每个子层外面都包着残差连接和层归一化。

The encoder consists of N=6 identical layers stacked. Each layer has two sub-layers: (1) multi-head self-attention; (2) position-wise feed-forward network. Each sub-layer is wrapped with a residual connection and layer normalization.

output = LayerNorm(x + Sublayer(x))

这是每个子层的标准形式：输入 x 先经过子层处理得到 Sublayer(x)，然后与 x 做残差相加，再做层归一化。这种设计有两个好处：残差连接帮助梯度流动，层归一化稳定训练。

This is the standard form for each sub-layer: input x passes through the sub-layer to get Sublayer(x), then adds to x (residual), then layer normalizes. This design has two benefits: residual connections help gradient flow, layer normalization stabilizes training.

9.2 单层编码器的数据流9.2 Single Encoder Layer Data Flow

让我们追踪一个句子通过单层编码器的完整数据流：

Let's trace the complete data flow of a sentence through a single encoder layer:

输入：x (seq_len, 512) —— 来自上一层的输出（或嵌入层+位置编码）
子层1：多头自注意力：x → MultiHead(x, x, x) → attn_output (seq_len, 512)
残差+LN：LayerNorm(x + attn_output) → norm1 (seq_len, 512)
子层2：FFN：norm1 → FFN(norm1) → ffn_output (seq_len, 512)
残差+LN：LayerNorm(norm1 + ffn_output) → output (seq_len, 512)
Input: x (seq_len, 512) — from previous layer (or embedding+PE)
Sub-layer 1: Multi-head self-attention: x → MultiHead(x, x, x) → attn_output (seq_len, 512)
Residual+LN: LayerNorm(x + attn_output) → norm1 (seq_len, 512)
Sub-layer 2: FFN: norm1 → FFN(norm1) → ffn_output (seq_len, 512)
Residual+LN: LayerNorm(norm1 + ffn_output) → output (seq_len, 512)

注意：输入和输出维度始终是 512。这让 6 层可以无缝堆叠——第 i 层的输出直接作为第 i+1 层的输入。

Note: input and output dimensions are always 512. This lets 6 layers stack seamlessly — layer i's output feeds directly into layer i+1.

9.3 维度一致性原则9.3 Dimension Consistency Principle

论文强调："To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension d_model = 512." 这是 Transformer 设计的一条铁律：所有子层的输出维度必须等于 d_model。如果没有这条件，残差相加 x + Sublayer(x) 就维度不匹配了。

The paper emphasizes: "To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension d_model = 512." This is an iron rule of Transformer design: all sub-layer outputs must be d_model. Otherwise, residual addition x + Sublayer(x) would have dimension mismatch.

注意：原始 Transformer 使用的是 Post-LN 结构——先做残差相加，再做 LayerNorm。后来的研究发现 Pre-LN（先做 LayerNorm，再做子层，再残差相加）训练更稳定，尤其是在深层模型中。GPT-2 之后的模型大多采用 Pre-LN。这是一个原始论文没有预见到的改进方向。

Note: The original Transformer uses Post-LN — residual addition first, then LayerNorm. Later research found Pre-LN (LayerNorm first, then sub-layer, then residual) is more stable to train, especially in deep models. Models after GPT-2 mostly use Pre-LN. This is an improvement direction not foreseen in the original paper.

L2 · 变形迁移编码器子层维度Encoder sublayer dimensions

编码器每层有哪两个子层？每个子层的输出维度是多少？为什么必须保持 d_model = 512？What are the two sublayers in each encoder layer? What is each sublayer's output dimension? Why must it remain d_model = 512?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果某个子层的输出维度不是512，怎么解决？

检查点：画出单层编码器的数据流图，标出每一步的维度变化。解释为什么所有子层输出维度必须是 d_model。

Checkpoint: Draw the single encoder layer data flow diagram, marking dimensions at each step. Explain why all sub-layer outputs must be d_model.

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

编码器由 6 个相同层堆叠而成，每层包含多头自注意力和 FFN 两个子层，每个子层外包残差连接和层归一化。所有子层输出维度为 d_model=512，保证残差相加和层间堆叠。原始设计为 Post-LN 结构，后续发展为 Pre-LN 以获得更好的训练稳定性。

The encoder consists of 6 identical stacked layers, each with multi-head self-attention and FFN sub-layers, each wrapped with residual connections and layer normalization. All sub-layer outputs are d_model=512, ensuring residual addition and inter-layer stacking. The original design is Post-LN; later development moved to Pre-LN for better training stability.

第10章解码器结构Chapter 10: Decoder Structure

解码器和编码器类似，但有两个关键区别：多了一个子层（编码器-解码器注意力），以及自注意力层加了因果掩码。让我们看看完整的解码器结构。

The decoder is similar to the encoder but with two key differences: an additional sub-layer (encoder-decoder attention) and causal masking in the self-attention layer. Let's examine the complete decoder structure.

10.1 三个子层10.1 Three Sub-layers

每层解码器有三个子层（比编码器多一个）：

Each decoder layer has three sub-layers (one more than the encoder):

掩码多头自注意力（Masked Multi-Head Self-Attention）：Q=K=V 来自解码器上一层输出，但用因果掩码防止看未来。
编码器-解码器注意力（Encoder-Decoder Attention）：Q 来自子层1输出，K=V 来自编码器最终输出。
逐位置 FFN：与编码器的 FFN 完全相同。
Masked Multi-Head Self-Attention: Q=K=V from previous decoder layer, with causal mask preventing future access.
Encoder-Decoder Attention: Q from sub-layer 1 output, K=V from encoder final output.
Position-wise FFN: identical to the encoder's FFN.

每个子层同样外包残差连接和层归一化。所以每层解码器执行 3 次 LayerNorm(x + Sublayer(x))。

Each sub-layer is also wrapped with residual connections and layer normalization. So each decoder layer performs 3 × LayerNorm(x + Sublayer(x)).

10.2 自回归生成10.2 Auto-regressive Generation

解码器以「自回归」方式工作：生成第 t 个词时，只能看到前 t-1 个已生成的词。论文说："This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i."

The decoder works in an "auto-regressive" manner: when generating the t-th token, it can only see the first t-1 generated tokens. The paper says: "This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i."

"offset by one position" 是什么意思？训练时，解码器的输入是完整的目标序列右移一位（shifted right）。例如目标序列是 [I, am, fine, <EOS>]，输入就是 [<BOS>, I, am, fine]。这样位置 1 看到 <BOS> 预测 I，位置 2 看到 <BOS>, I 预测 am，以此类推。结合掩码，每个位置只能看到自己之前的词。

What does "offset by one position" mean? During training, the decoder input is the complete target sequence shifted right by one. E.g., if target is [I, am, fine, <EOS>], input is [<BOS>, I, am, fine]. So position 1 sees <BOS> to predict I, position 2 sees <BOS>, I to predict am, etc. Combined with masking, each position only sees words before itself.

10.3 训练 vs 推理：一个重要区别10.3 Training vs Inference: A Critical Difference

训练时：目标序列已知，一次前向传播处理整个序列。掩码保证了每个位置只看历史。这叫「teacher forcing」——用真实标签作为输入，而不是模型自己的预测。训练可以高度并行化。

推理时：目标序列未知，必须逐词生成。每次生成一个词后，把它加入输入序列，再生成下一个词。这是序列化的，无法并行。这就是为什么 GPT 生成文本时是一个 token 一个 token 吐出来的。

这个区别是 Transformer 推理速度的主要瓶颈。后续研究（如 KV-cache、推测解码 speculative decoding）都是在尝试优化这个问题。

During training: the target sequence is known, processed in one forward pass. Masking ensures each position only sees history. This is "teacher forcing" — using ground truth as input rather than model's own predictions. Training is highly parallelizable.

During inference: the target sequence is unknown; tokens must be generated one by one. After generating each token, it's added to the input to generate the next. This is sequential, cannot be parallelized. This is why GPT generates text token by token.

This difference is the main bottleneck for Transformer inference speed. Subsequent research (KV-cache, speculative decoding) optimizes this.

L3 · 构造反例训练vs推理并行性Training vs inference parallelism

为什么 Transformer 训练时可以并行但推理时不能？从 teacher forcing 和自回归生成的角度解释。Why can the Transformer parallelize during training but not during inference? Explain from the perspective of teacher forcing vs autoregressive generation.

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：有什么方法可以部分加速推理？

检查点：解释训练和推理时解码器的工作方式有何不同，以及为什么推理无法并行化。

Checkpoint: Explain how the decoder works differently during training vs inference, and why inference cannot be parallelized.

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

解码器每层有 3 个子层：掩码自注意力、编码器-解码器注意力、FFN。掩码保证自回归性质，编码器-解码器注意力连接源与目标。训练时用 teacher forcing 高度并行化，推理时必须逐词生成无法并行——这是 Transformer 推理速度的主要瓶颈。

Each decoder layer has 3 sub-layers: masked self-attention, encoder-decoder attention, FFN. Masking ensures auto-regression; encoder-decoder attention bridges source and target. Training uses teacher forcing for parallelism; inference generates token by token — the main bottleneck for Transformer inference speed.

第11章残差连接与层归一化Chapter 11: Residual Connections and Layer Normalization

残差连接和层归一化是深度学习中的两项基础技术，不是 Transformer 发明的。但 Transformer 高度依赖它们来完成 6 层（以及后续更深层）的训练。这一章我们深入理解这两项技术为什么重要。

Residual connections and layer normalization are foundational deep learning techniques, not invented by the Transformer. But the Transformer heavily relies on them to train 6 layers (and beyond). This chapter deeply understands why these matter.

11.1 残差连接（Residual Connection）11.1 Residual Connection

残差连接来自 He et al. (2016) 的 ResNet。核心思想简单：不直接学习子层的变换 H(x)，而是学习残差 F(x) = H(x) - x，即 H(x) = x + F(x)。如果最优变换是恒等映射，网络只需把 F(x) 学到 0 即可，比从头学恒等映射容易得多。

Residual connections come from He et al. (2016)'s ResNet. The core idea is simple: instead of directly learning the sub-layer transformation H(x), learn the residual F(x) = H(x) - x, so H(x) = x + F(x). If the optimal transformation is identity, the network only needs to learn F(x) = 0, much easier than learning identity from scratch.

在 Transformer 中，每个子层的输出是 LayerNorm(x + Sublayer(x))。如果没有残差连接，深层模型的梯度会vanish（消失）或explode（爆炸）。残差连接提供了一条「高速公路」，让梯度可以直接流回浅层，不受中间层变换的影响。

In the Transformer, each sub-layer outputs LayerNorm(x + Sublayer(x)). Without residual connections, gradients in deep models would vanish or explode. Residual connections provide a "highway" letting gradients flow directly back to shallow layers, unaffected by intermediate transformations.

11.2 层归一化（Layer Normalization）11.2 Layer Normalization

层归一化来自 Ba et al. (2016)。它对单个样本的所有特征维度做归一化——计算当前样本在 d_model 个维度上的均值和方差，然后用它们做标准化。

Layer Normalization comes from Ba et al. (2016). It normalizes all feature dimensions of a single sample — computing mean and variance across d_model dimensions, then standardizing.

LN(x) = (x - mean(x)) / sqrt(var(x) + epsilon) × gamma + beta

其中 mean 和 var 是沿特征维度计算的，gamma 和 beta 是可学习的缩放和偏移参数。

Where mean and var are computed along the feature dimension, gamma and beta are learnable scale and shift parameters.

LayerNorm vs BatchNorm：BatchNorm 在 batch 维度归一化（一个特征在所有样本上的统计量），LayerNorm 在特征维度归一化（一个样本在所有特征上的统计量）。BatchNorm 依赖 batch 统计量，在 batch 小或序列长度变化时不稳定；LayerNorm 不依赖 batch，更适合序列模型。这就是为什么 NLP 模型普遍用 LayerNorm。

LayerNorm vs BatchNorm: BatchNorm normalizes along the batch dimension (statistics of one feature across all samples); LayerNorm normalizes along the feature dimension (statistics of one sample across all features). BatchNorm depends on batch statistics, unstable with small batches or variable sequence lengths; LayerNorm is batch-independent, better for sequence models. This is why NLP models universally use LayerNorm.

11.3 Post-LN vs Pre-LN11.3 Post-LN vs Pre-LN

原始 Transformer 使用 Post-LN：output = LayerNorm(x + Sublayer(x))。但研究发现 Post-LN 在深层时训练不稳定，需要仔细的学习率预热（warmup）。

The original Transformer uses Post-LN: output = LayerNorm(x + Sublayer(x)). But research found Post-LN unstable in deep models, requiring careful learning rate warmup.

Pre-LN 改变了顺序：output = x + Sublayer(LayerNorm(x))。先归一化再做子层变换。Pre-LN 的梯度流更直接（残差路径上没有 LayerNorm），训练更稳定。GPT-2、GPT-3 及后续大模型大多使用 Pre-LN。

Pre-LN changes the order: output = x + Sublayer(LayerNorm(x)). Normalize first, then sub-layer transformation. Pre-LN has more direct gradient flow (no LayerNorm on the residual path), more stable training. GPT-2, GPT-3, and most subsequent large models use Pre-LN.

L3 · 构造反例残差连接作用Residual connection

残差连接和层归一化各解决什么问题？为什么 6 层 Transformer 需要残差连接？层归一化为什么适合变长序列？What problems do residual connections and layer normalization solve? Why does a 6-layer Transformer need residual connections? Why is layer normalization suitable for variable-length sequences?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果没有残差连接，6层 Transformer 能训练成功吗？

检查点：解释残差连接如何帮助深层网络训练。比较 LayerNorm 和 BatchNorm 的区别，以及为什么 NLP 用 LayerNorm。

Checkpoint: Explain how residual connections help deep network training. Compare LayerNorm and BatchNorm, and why NLP uses LayerNorm.

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

残差连接让梯度通过「高速公路」直接流回浅层，解决深层网络梯度消失问题。层归一化沿特征维度归一化，不依赖 batch 统计，适合变长序列模型。原始 Transformer 使用 Post-LN（先残差后归一化），后续大模型转向 Pre-LN（先归一化后残差）以获得更好的训练稳定性。

Residual connections let gradients flow directly back to shallow layers via a "highway," solving vanishing gradients in deep networks. Layer normalization normalizes along feature dimensions, independent of batch statistics, suitable for variable-length sequence models. The original Transformer uses Post-LN; subsequent large models switched to Pre-LN for better training stability.

第12章为什么是自注意力Chapter 12: Why Self-Attention

论文第 4 节专门论证为什么选择自注意力而不是 RNN 或 CNN。这一节是整篇论文的理论核心——它不仅展示了 Transformer 有效性，还解释了「为什么有效」。

Section 4 of the paper specifically argues why self-attention was chosen over RNNs or CNNs. This section is the theoretical core — it shows not just that the Transformer works, but "why it works."

12.1 三个评价标准12.1 Three Evaluation Criteria

论文提出三个标准来比较自注意力、RNN 和 CNN：

The paper proposes three criteria to compare self-attention, RNN, and CNN:

每层计算复杂度（Computational complexity per layer）
可并行化的计算量（Amount of parallelizable computation, measured by minimum sequential operations）
长距离依赖的路径长度（Path length between long-range dependencies）
Computational complexity per layer
Parallelizable computation (measured by minimum sequential operations)
Path length between long-range dependencies

12.2 Table 1：三种层类型对比12.2 Table 1: Comparison of Three Layer Types

层类型Layer Type	每层复杂度Complexity/Layer	顺序操作数Seq Ops	最大路径长度Max Path
Self-Attention	O(n²·d)	O(1)	O(1)
Recurrent	O(n·d²)	O(n)	O(n)
Convolutional	O(k·n·d²)	O(1)	O(log_k(n))
受限自注意力Self-Attn (restricted)	O(r·n·d)	O(1)	O(n/r)

其中 n 是序列长度，d 是表示维度，k 是卷积核大小，r 是受限注意力的邻域大小。

Where n is sequence length, d is representation dimension, k is convolution kernel size, r is restricted attention neighborhood size.

12.3 逐项分析12.3 Item-by-Item Analysis

复杂度Complexity

自注意力是 O(n²·d)，RNN 是 O(n·d²)。当 n < d 时（大多数 NLP 任务中序列长度 < 表示维度，如 n=128, d=512），自注意力更快。当 n >> d 时（如长文档 n=4096），RNN 的 O(n·d²) 更有优势。这也是为什么后续研究提出了「受限自注意力」（如 Longformer、BigBird）来处理长序列。

Self-attention is O(n²·d), RNN is O(n·d²). When n < d (most NLP tasks, e.g., n=128, d=512), self-attention is faster. When n >> d (e.g., long documents n=4096), RNN's O(n·d²) wins. This is why subsequent research proposed "restricted self-attention" (Longformer, BigBird) for long sequences.

并行化Parallelization

自注意力和 CNN 都是 O(1) 顺序操作——可以完全并行。RNN 是 O(n) 顺序操作——必须逐步计算，无法并行。这是 Transformer 训练速度远超 RNN 的唯一原因。

Self-attention and CNN both have O(1) sequential operations — fully parallelizable. RNN has O(n) sequential operations — must compute step by step, not parallelizable. This is the sole reason Transformer training is far faster than RNN.

路径长度Path Length

路径长度指信号从一个位置到另一个位置需要经过多少层。自注意力是 O(1)——任意两个位置直接相连。RNN 是 O(n)——首尾两个位置需要经过 n 层。CNN 是 O(log_k(n))——需要堆叠 O(n/k) 层的卷积。路径越短，学习长距离依赖越容易。

Path length = how many layers a signal must traverse from one position to another. Self-attention is O(1) — any two positions directly connected. RNN is O(n) — head and tail need n layers. CNN is O(log_k(n)) — needs O(n/k) stacked convolutions. Shorter paths make long-range dependencies easier to learn.

关键洞察：论文引用 Hochreiter et al. (2001) 指出，影响学习长距离依赖的关键因素是前向和反向信号在网络中需要穿越的路径长度。路径越短，越容易学。自注意力让任意两个位置之间只有 O(1) 的路径，这是它能在翻译任务上超越 RNN 的根本原因。

Key insight: The paper cites Hochreiter et al. (2001) that the key factor for learning long-range dependencies is the path length forward and backward signals must traverse. Shorter paths = easier learning. Self-attention gives O(1) path between any two positions — the fundamental reason it outperforms RNN on translation.

12.4 可解释性：额外的礼物12.4 Interpretability: A Bonus

论文最后提到一个意外的收获："As side benefit, self-attention could yield more interpretable models." 注意力权重天然是一种可视化工具——你可以直接看到模型在关注哪些词。RNN 的隐藏状态是一个不透明的向量，无法直接解读。论文附录的 Figure 3-5 展示了不同注意力头学到的语法、语义和结构模式，这在 RNN 时代是很难做到的。

The paper mentions an unexpected bonus: "As side benefit, self-attention could yield more interpretable models." Attention weights are a natural visualization tool — you can directly see which words the model focuses on. RNN hidden states are opaque vectors. Figures 3-5 show different heads learning syntax, semantics, and structure patterns — difficult in the RNN era.

L4 · 多概念综合自注意力 vs RNN 复杂度Self-attention vs RNN complexity

对比自注意力和 RNN 在复杂度、并行性、最大路径长度三个维度的差异。什么条件下自注意力更优？什么条件下 RNN 更优？Compare self-attention and RNN on three dimensions: complexity, parallelism, and maximum path length. Under what conditions is self-attention better? When is RNN better?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果处理一本书（n=100000），自注意力还合适吗？

检查点：用 Table 1 的数据解释为什么自注意力在大多数 NLP 场景中优于 RNN 和 CNN。什么情况下自注意力会变得不划算？

Checkpoint: Use Table 1 data to explain why self-attention outperforms RNN and CNN in most NLP scenarios. When does self-attention become disadvantageous?

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

论文从复杂度、并行化、路径长度三个维度论证了自注意力的优势：当 n < d 时复杂度低于 RNN，O(1) 顺序操作支持完全并行化，O(1) 路径长度让长距离依赖更容易学习。可解释性是额外收获。当 n >> d 时自注意力复杂度变高，后续研究用受限注意力解决。

The paper argues self-attention's advantages from complexity, parallelization, and path length: lower complexity than RNN when n < d, O(1) sequential operations for full parallelization, O(1) path length for easier long-range dependency learning. Interpretability is a bonus. When n >> d, complexity grows, addressed by restricted attention in later work.

第13章训练策略Chapter 13: Training Strategy

一篇好论文不只展示「做什么」，还详细说「怎么做」。论文第 5 节描述了训练 Transformer 的完整配方——数据、硬件、优化器、学习率调度、正则化。这些细节对复现和后续研究至关重要。

A good paper shows not just "what" but "how." Section 5 describes the complete recipe for training the Transformer — data, hardware, optimizer, learning rate schedule, regularization. These details are crucial for reproducibility and subsequent research.

13.1 训练数据13.1 Training Data

任务Task	数据集Dataset	句对数Sentence Pairs	词表Vocabulary
EN-DE	WMT 2014	~4.5M	~37K (BPE, shared)
EN-FR	WMT 2014	~36M	~32K (word-piece)

一个训练 batch 包含约 25000 个源语言 token 和 25000 个目标语言 token。batch 按「近似序列长度」分组——长度相近的句子放在一起，减少 padding 浪费。

A training batch contains ~25000 source tokens and ~25000 target tokens. Batches are grouped by "approximate sequence length" — similar-length sentences together, reducing padding waste.

13.2 硬件与训练时间13.2 Hardware and Training Time

模型Model	GPUGPU	步数Steps	每步时间Step Time	总时间Total Time
Base	8× P100	100K	~0.4s	~12 hours
Big	8× P100	300K	~1.0s	~3.5 days

作为对比，当时最强的 RNN 模型之一 GNMT+RL 训练成本约 2.3×10^19 FLOPs（EN-DE），而 Transformer base 只需 3.3×10^18 FLOPs——少了约 7 倍，但 BLEU 更高。

For comparison, the strongest RNN model GNMT+RL costs ~2.3×10^19 FLOPs (EN-DE), while Transformer base needs only 3.3×10^18 FLOPs — about 7× less, but higher BLEU.

13.3 优化器与学习率调度13.3 Optimizer and Learning Rate Schedule

使用 Adam 优化器（beta1=0.9, beta2=0.98, epsilon=10^-9）。学习率调度是论文最有特色的训练细节：

Using Adam optimizer (beta1=0.9, beta2=0.98, epsilon=10^-9). The learning rate schedule is the paper's most distinctive training detail:

lrate = d_model^(-0.5) × min(step^(-0.5), step × warmup^(-1.5))

其中 warmup_steps = 4000。这个公式做了两件事：

Where warmup_steps = 4000. This formula does two things:

预热阶段（step < 4000）：学习率线性增长。min 取 step×warmup^(-1.5)，这是一个关于 step 的线性函数。从 0 开始逐步增大学习率，避免训练初期因随机初始化的大梯度而导致不稳定。
衰减阶段（step > 4000）：学习率按 step^(-0.5) 衰减。min 取 step^(-0.5)，学习率随步数平方根反比下降。
Warmup phase (step < 4000): learning rate increases linearly. min takes step×warmup^(-1.5), a linear function of step. Gradually increasing LR from 0 avoids instability from large gradients with random initialization.
Decay phase (step > 4000): learning rate decays as step^(-0.5). min takes step^(-0.5), LR inversely proportional to sqrt of steps.

d_model^(-0.5) 是一个全局缩放因子，确保不同模型大小（base vs big）的学习率量级合适。大模型 d_model 更大，学习率更小。

d_model^(-0.5) is a global scaling factor, ensuring appropriate LR magnitude for different model sizes (base vs big). Larger d_model means smaller LR.

为什么需要 warmup？ Transformer 训练初期，所有参数都是随机初始化的。注意力权重接近均匀分布，梯度可能很大。如果一开始就用大学习率，参数会剧烈震荡甚至发散。warmup 让模型先在低学习率下「找到大致方向」，然后再加速。这个 warmup 策略后来成为 Transformer 训练的标准做法，几乎所有后续模型都保留了某种形式的 warmup。

Why warmup? At the start of Transformer training, all parameters are randomly initialized. Attention weights are near-uniform, gradients can be large. Starting with high LR causes violent oscillation or divergence. Warmup lets the model "find direction" at low LR first, then accelerate. This warmup strategy became standard for Transformer training — nearly all subsequent models retain some form of warmup.

13.4 正则化13.4 Regularization

论文使用了三种正则化：

The paper uses three types of regularization:

残差 Dropout（P_drop=0.1 for base, 0.3 for big EN-DE）：在每个子层输出加上残差连接之前应用 dropout。还在嵌入和位置编码之和上应用 dropout。
Label Smoothing（epsilon_ls=0.1）：训练时，目标分布不是 one-hot，而是把 0.1 的概率均分给所有其他类别。论文说："This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score."
Checkpoint Averaging：base 模型取最后 5 个 checkpoint（每 10 分钟保存一次）的平均，big 模型取最后 20 个。这是一种隐式正则化，相当于权重空间中的模型集成。
Residual Dropout (P_drop=0.1 base, 0.3 big EN-DE): applied to each sub-layer output before residual addition. Also applied to sum of embeddings and positional encodings.
Label Smoothing (epsilon_ls=0.1): target distribution isn't one-hot but spreads 0.1 probability across all other classes. The paper says: "This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score."
Checkpoint Averaging: base model averages last 5 checkpoints (saved every 10 min), big model averages last 20. Implicit regularization, equivalent to weight-space ensembling.

Label Smoothing 的反转：这是论文中一个有趣的发现——label smoothing 让 perplexity（困惑度）变差但 BLEU 变好。困惑度衡量概率分布与真实分布的匹配，BLEU 衡量翻译质量。Label smoothing 鼓励模型不要过度自信，虽然概率分布的「紧密程度」下降（PPL 变差），但泛化能力提升（BLEU 变好）。这提醒我们：优化指标和评估指标不一定一致。

The Label Smoothing paradox: An interesting finding — label smoothing worsens perplexity but improves BLEU. PPL measures probability distribution match; BLEU measures translation quality. Label smoothing discourages overconfidence; while distribution "tightness" drops (worse PPL), generalization improves (better BLEU). Lesson: optimization metrics and evaluation metrics don't always align.

L3 · 构造反例 warmup + label smoothingWarmup and label smoothing

Warmup 学习率策略的前 4000 步做了什么？为什么需要 warmup？Label smoothing 让 PPL 变差但 BLEU 变好，这说明了什么？What does the warmup learning rate schedule do in the first 4000 steps? Why is warmup needed? Label smoothing worsens PPL but improves BLEU — what does this tell us?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果不用 warmup 直接用最终学习率，会发生什么？

检查点：解释 warmup 学习率调度的工作原理和必要性。为什么 label smoothing 让 PPL 变差但 BLEU 变好？

Checkpoint: Explain how warmup LR schedule works and why it's necessary. Why does label smoothing worsen PPL but improve BLEU?

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

Transformer 在 WMT 2014 EN-DE (4.5M 句对) 和 EN-FR (36M 句对) 上训练。使用 Adam 优化器配合 warmup=4000 的学习率调度——线性预热后按平方根反比衰减。三种正则化：dropout（0.1/0.3）、label smoothing（0.1）、checkpoint averaging。Label smoothing 的 PPL-BLEU 悖论揭示了优化指标与评估指标的不一致性。Base 模型训练 12 小时，Big 模型 3.5 天，远低于竞争对手。

The Transformer trains on WMT 2014 EN-DE (4.5M pairs) and EN-FR (36M pairs). Uses Adam with warmup=4000 LR schedule — linear warmup then inverse-sqrt decay. Three regularizations: dropout (0.1/0.3), label smoothing (0.1), checkpoint averaging. The PPL-BLEU paradox of label smoothing reveals optimization-evaluation metric misalignment. Base model trains in 12 hours, Big in 3.5 days — far less than competitors.

第14章翻译结果Chapter 14: Machine Translation Results

理论说得再好，最终要看结果。论文第 6.1 节展示了 Transformer 在两个翻译任务上的成绩，结果令人震惊——不仅超越所有已发表模型，训练成本还低得多。

Theory means nothing without results. Section 6.1 shows the Transformer's performance on two translation tasks — results that shocked the field: not just surpassing all published models, but at far lower training cost.

14.1 EN-DE 翻译14.1 English-to-German Translation

在 WMT 2014 英德翻译任务上，Transformer (big) 达到了 28.4 BLEU，比之前最好的结果（包括集成模型）高出 2.0 BLEU 以上。这是一个巨大的提升——在机器翻译领域，0.5 BLEU 的提升就已经值得发论文了。

On WMT 2014 EN-DE, Transformer (big) achieves 28.4 BLEU, surpassing the best previous result (including ensembles) by over 2.0 BLEU. This is a massive jump — in machine translation, 0.5 BLEU improvement is already paper-worthy.

更惊人的是，即使是 base 模型（27.3 BLEU），也已经超越了所有已发表的模型和集成模型，而且训练成本只有竞争对手的几分之一。

Even more惊人的是, even the base model (27.3 BLEU) surpasses all published models and ensembles, at a fraction of competitors' training cost.

14.2 EN-FR 翻译14.2 English-to-French Translation

在 WMT 2014 英法翻译任务上，Transformer (big) 达到 41.8 BLEU，创造了新的单模型 SOTA。训练只用了 3.5 天（8×P100），不到之前最强模型训练成本的 1/4。

On WMT 2014 EN-FR, Transformer (big) achieves 41.8 BLEU, setting a new single-model SOTA. Training took only 3.5 days (8×P100), less than 1/4 the training cost of the previous best model.

注意：EN-FR 的 big 模型使用了 dropout=0.1（而非 EN-DE 的 0.3）。论文说这是因为 EN-FR 数据集更大（36M vs 4.5M 句对），大数据集本身就有正则化效果，不需要那么强的 dropout。

Note: the EN-FR big model uses dropout=0.1 (vs 0.3 for EN-DE). The paper explains this is because EN-FR dataset is larger (36M vs 4.5M pairs) — larger data inherently regularizes, requiring less dropout.

14.3 Table 2：与竞争对手对比14.3 Table 2: Comparison with Competitors

Model	EN-DE BLEU	EN-FR BLEU	EN-DE FLOPs	EN-FR FLOPs
ByteNet	23.75	-	-	-
Deep-Att + PosUnk	-	39.2	-	1.0×10^20
GNMT + RL	24.6	39.92	2.3×10^19	1.4×10^20
ConvS2S	25.16	40.46	9.6×10^18	1.5×10^20
MoE	26.03	40.56	2.0×10^19	1.2×10^20
GNMT+RL Ensemble	26.30	41.16	1.8×10^20	1.1×10^21
ConvS2S Ensemble	26.36	41.29	7.7×10^19	1.2×10^21
Transformer (base)	27.3	38.1	3.3×10^18	-
Transformer (big)	28.4	41.8	2.3×10^19	2.3×10^19

看这张表的关键洞察：

1. Transformer (base) 以 3.3×10^18 FLOPs 的训练成本就达到了 27.3 BLEU，比花费 1.8×10^20 FLOPs 的 GNMT+RL 集成还高 1.0 BLEU。成本只有 1/55。

2. Transformer (big) 的 EN-DE 成本（2.3×10^19）和 GNMT+RL 单模型一样，但 BLEU 高了 3.8。

3. 即使是最强的集成模型（7.7×10^19 FLOPs 的 ConvS2S Ensemble, 26.36 BLEU），也被只用 3.3×10^18 FLOPs 的 base 模型（27.3 BLEU）超越。

Key insights from this table:

1. Transformer (base) achieves 27.3 BLEU at 3.3×10^18 FLOPs — higher than GNMT+RL Ensemble (1.8×10^20 FLOPs, 26.30 BLEU) at 1/55 the cost.

2. Transformer (big) EN-DE costs the same as GNMT+RL single model (2.3×10^19) but 3.8 BLEU higher.

3. Even the strongest ensemble (ConvS2S Ensemble: 7.7×10^19 FLOPs, 26.36 BLEU) is beaten by the base model (3.3×10^18 FLOPs, 27.3 BLEU).

14.4 推理策略14.4 Inference Strategy

推理使用 beam search，beam size=4，长度惩罚 alpha=0.6。最大输出长度设为输入长度+50，但可以提前终止。这些超参数在开发集上调优。Checkpoint averaging 在推理时也使用——不是用最后一个 checkpoint，而是最后几个 checkpoint 的权重平均。

Inference uses beam search with beam size=4, length penalty alpha=0.6. Max output length is input+50, with early termination. These hyperparameters were tuned on the dev set. Checkpoint averaging is also used at inference — averaging the last few checkpoints' weights rather than using the final one.

14.5 英语句法分析：泛化能力验证14.5 English Constituency Parsing: Generalization Test

论文还测试了 Transformer 在非翻译任务上的表现——英语成分句法分析（constituency parsing）。这个任务的输出有强结构约束，且比输入长得多。使用 4 层 Transformer（d_model=1024），在 WSJ 数据集上达到 91.3 F1（仅 WSJ 训练）和 92.7 F1（半监督设置）。

The paper also tests the Transformer on a non-translation task — English constituency parsing. This task has strong structural constraints on output, which is much longer than input. Using a 4-layer Transformer (d_model=1024), it achieves 91.3 F1 (WSJ only) and 92.7 F1 (semi-supervised).

论文总结道："despite the lack of task-specific tuning our model performs surprisingly well." 这证明 Transformer 不是一个只能做翻译的专用模型，而是一个通用架构——这是后续 BERT、GPT 等通用语言模型能够出现的基础。

The paper concludes: "despite the lack of task-specific tuning our model performs surprisingly well." This proves the Transformer isn't a translation-specific model but a general architecture — the foundation for subsequent general language models like BERT and GPT.

L3 · 构造反例成本对比 + 泛化能力Cost comparison and generalization

Base 模型与之前的 SOTA 模型集成相比，成本差多少？句法分析实验说明了 Transformer 的什么特质？为什么这很重要？Compared to prior SOTA model ensembles, how much cheaper is the base model? What does the parsing experiment reveal about the Transformer's nature? Why is this important?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果 Transformer 只在翻译上有效但其他任务不行，影响力会怎样？

检查点：从 Table 2 中总结 Transformer 相比竞争对手的优势。为什么句法分析实验对 Transformer 意义重大？

Checkpoint: Summarize the Transformer's advantages over competitors from Table 2. Why is the parsing experiment significant for the Transformer?

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

Transformer 在 WMT 2014 EN-DE 达到 28.4 BLEU（超 SOTA 2.0+），EN-FR 达到 41.8 BLEU（新单模型 SOTA），训练成本远低于竞争对手。Base 模型以 1/55 的成本超越最强集成。在英语句法分析上也表现优异，证明其通用性。使用 beam search 和 checkpoint averaging 进行推理。

Transformer achieves 28.4 BLEU on WMT 2014 EN-DE (+2.0 over SOTA) and 41.8 BLEU on EN-FR (new single-model SOTA), at far lower training cost. Base model surpasses the strongest ensemble at 1/55 the cost. Also performs well on English parsing, proving generality. Uses beam search and checkpoint averaging for inference.

第15章消融实验Chapter 15: Ablation Study

消融实验（Ablation Study）是科学论文的试金石：逐个去掉模型的各个组件，看哪个最重要。论文 Table 3 系统地变化了 Transformer 的多个超参数，让我们能精确理解每个设计选择的贡献。

Ablation study is the litmus test of scientific papers: remove components one by one, see which matters most. Table 3 systematically varies multiple Transformer hyperparameters, letting us precisely understand each design choice's contribution.

15.1 头数实验（行 A）15.1 Attention Heads (Row A)

保持总计算量不变，变化头数 h 和每头维度 d_k：

Keeping total computation constant, varying head count h and per-head dimension d_k:

h	d_k	d_v	train PPL	BLEU (dev)	params (M)
1	512	512	5.29	24.9	80
4	128	128	5.00	25.5	80
8 (base)	64	64	4.92	25.8	65
16	32	32	5.01	25.4	60
32	16	16	5.16	25.1	58

关键发现：1 个头比 8 个头差 0.9 BLEU——多头确实有用。但太多头也不好：16 头比 8 头差 0.4 BLEU，32 头差 0.7 BLEU。论文说："While single-head attention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads."

Key finding: 1 head is 0.9 BLEU worse than 8 — multi-head is indeed useful. But too many heads also hurts: 16 heads is 0.4 BLEU worse, 32 heads 0.7 worse. The paper says: "While single-head attention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads."

为什么太多头会变差？ 当头数增加时，每个头的维度 d_k 减小。h=32 时 d_k=16，每个头只有 16 维的表示空间。维度太低，每个头的注意力计算 QK^T/√d_k 的分辨能力下降——16 维的点积不够区分细微的语义差异。这就像给每个人分太少的工作空间，虽然人多但每人什么都做不好。h=8（d_k=64）是「金发姑娘」甜点。

Why do too many heads hurt? As head count increases, per-head dimension d_k shrinks. At h=32, d_k=16 — only 16 dimensions per head. Low dimension reduces attention's discriminating power: 16-dim dot products can't distinguish fine semantic differences. Like giving each person too little workspace — many people but each can do nothing well. h=8 (d_k=64) is the Goldilocks sweet spot.

15.2 关键维度实验（行 B, C）15.2 Key Dimension Experiments (Rows B, C)

行 B 减小 d_k（不改变 h=8）：d_k=32 掉 0.4 BLEU，d_k=16 掉 0.4 BLEU。论文说："This suggests that determining compatibility is not easy and that a more sophisticated compatibility function than dot product may be beneficial."

Row B reduces d_k (keeping h=8): d_k=32 loses 0.4 BLEU, d_k=16 loses 0.4. The paper says: "This suggests that determining compatibility is not easy and that a more sophisticated compatibility function than dot product may be beneficial."

行 C 变化模型大小（N, d_model, d_ff）：更大的模型更好。d_model=1024, d_ff=4096, N=4 达到 26.0 BLEU（比 base 高 0.2），但参数从 65M 增到 168M。

Row C varies model size (N, d_model, d_ff): bigger is better. d_model=1024, d_ff=4096, N=4 achieves 26.0 BLEU (+0.2 over base), but parameters grow from 65M to 168M.

15.3 正则化实验（行 D）15.3 Regularization Experiments (Row D)

P_drop	eps_ls	train PPL	BLEU (dev)
0.1	0.1	4.92	25.8
0.0	0.1	5.77	24.6
0.2	0.1	4.95	25.5
0.1	0.0	4.67	25.3
0.1	0.2	5.47	25.7

关键发现：dropout=0（去掉 dropout）掉 1.2 BLEU——dropout 非常有效。label smoothing=0（去掉 label smoothing）掉 0.5 BLEU。两者都是重要的正则化手段。

Key findings: dropout=0 (removing dropout) loses 1.2 BLEU — dropout is very effective. Label smoothing=0 loses 0.5 BLEU. Both are important regularizers.

有趣的是，去掉 label smoothing 后 train PPL 反而更好（4.67 vs 4.92）——模型在训练集上拟合更好了——但 dev BLEU 更差。这再次验证了 label smoothing 的作用：牺牲训练拟合换泛化能力。

Interestingly, removing label smoothing improves train PPL (4.67 vs 4.92) — better training fit — but worse dev BLEU. This again validates label smoothing: sacrifice training fit for generalization.

15.4 位置编码实验（行 E）15.4 Positional Encoding (Row E)

用可学习的位置嵌入替代正弦余弦编码：train PPL 4.92（相同），BLEU 25.7（几乎相同，差 0.1）。论文说："nearly identical results."

Replacing sinusoidal encoding with learned positional embeddings: train PPL 4.92 (same), BLEU 25.7 (nearly same, -0.1). The paper says: "nearly identical results."

这个结果有点出乎意料——一个复杂的数学函数和一个简单的可学习嵌入效果一样。但论文选择 sin/cos 的理由不是性能，而是外推能力（可以处理比训练更长的序列）。

This result is somewhat surprising — a complex mathematical function performs the same as a simple learned embedding. But the paper chose sin/cos not for performance but for extrapolation ability (handling longer sequences than training).

15.5 Big 模型配置15.5 Big Model Configuration

参数Param	Base	Big
层数 NLayers N	6	6
d_model	512	1024
d_ff	2048	4096
h	8	16
d_k = d_v	64	64
P_drop	0.1	0.3
训练步数Steps	100K	300K
参数量Params	65M	213M
dev PPL	dev PPL	4.92	4.33
dev BLEU	dev BLEU	25.8	26.4

Big 模型主要在三个维度上扩大：d_model 翻倍（512→1024），d_ff 翻倍（2048→4096），头数翻倍（8→16），更大的 dropout（0.1→0.3）和更多训练步数（100K→300K）。参数量从 65M 增至 213M，dev BLEU 提升 0.6。

The big model scales in three dimensions: d_model doubled (512→1024), d_ff doubled (2048→4096), heads doubled (8→16), with larger dropout (0.1→0.3) and more steps (100K→300K). Parameters grow from 65M to 213M, dev BLEU improves by 0.6.

L4 · 多概念综合消融实验结论Ablation findings

消融实验的三个主要发现是什么？哪个组件去掉影响最大？多头数为什么 h=8 最优？What are the three main findings of the ablation experiments? Which component removal has the biggest impact? Why is h=8 the optimal head count?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果消融发现去掉某个组件完全不影响 BLEU，说明什么？

检查点：总结消融实验的三个最重要发现。为什么头数太少和太多都不行？

Checkpoint: Summarize the three most important findings from the ablation study. Why are both too few and too many heads bad?

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

消融实验揭示了：（1）多头注意力有效——1 头差 0.9 BLEU，但太多头（32）也差 0.7，h=8 是最佳；（2）dropout 非常重要——去掉掉 1.2 BLEU；（3）label smoothing 牺牲 PPL 换 BLEU；（4）sin/cos 与可学习位置编码效果几乎相同。Big 模型通过扩大 d_model、d_ff、头数和增加正则化获得 0.6 BLEU 提升。

The ablation study reveals: (1) multi-head attention works — 1 head loses 0.9 BLEU, but 32 heads also lose 0.7, h=8 is optimal; (2) dropout is crucial — removing loses 1.2 BLEU; (3) label smoothing trades PPL for BLEU; (4) sin/cos and learned PE are nearly identical. The big model gains 0.6 BLEU by scaling d_model, d_ff, heads, and increasing regularization.

第16章贡献与影响Chapter 16: Contributions and Impact

2017 年发表的这篇论文，标题只有五个词："Attention Is All You Need." 当时没人能预见它将彻底改变人工智能的轨迹。让我们回顾论文的核心贡献，以及它在发表后引发的连锁反应。

Published in 2017, this paper's title is just five words: "Attention Is All You Need." No one could have predicted it would reshape AI's trajectory. Let's review the core contributions and the chain reaction that followed.

16.1 论文的直接贡献16.1 Direct Contributions

论文做了四件事：

The paper did four things:

提出了 Transformer 架构——第一个完全基于注意力、不使用 RNN 或 CNN 的序列转换模型。这不是一个小改进，而是范式转变。
证明了纯注意力可行——在此之前，注意力是 RNN 的辅助组件。论文证明注意力可以独立承担序列建模，效果更好。
创造了新的 SOTA——在两个翻译任务上大幅超越现有最佳结果，同时训练成本远低于竞争对手。
展示了泛化能力——在句法分析任务上表现优异，证明 Transformer 不是翻译专用，而是通用序列建模工具。
Proposed the Transformer architecture — the first sequence transduction model based entirely on attention, without RNNs or CNNs. Not an incremental improvement, but a paradigm shift.
Proved pure attention works — before this, attention was an RNN auxiliary. The paper proved attention can independently handle sequence modeling, with better results.
Created new SOTA — significantly surpassed best results on two translation tasks, at far lower training cost.
Demonstrated generalization — excellent performance on parsing, proving the Transformer is not translation-specific but a general sequence modeling tool.

16.2 后续影响：Transformer 生态16.2 Subsequent Impact: The Transformer Ecosystem

论文发表后，Transformer 的影响远超机器翻译领域。以下是关键的时间线：

After publication, the Transformer's impact extended far beyond machine translation. Key timeline:

年份Year	模型Model	创新点Innovation
2017	Transformer	原始架构，编码器+解码器Original architecture, encoder+decoder
2018	BERT	只用编码器，预训练+微调范式Encoder-only, pretrain+finetune
2018	GPT	只用解码器，自回归语言模型Decoder-only, autoregressive LM
2019	GPT-2	更大规模，Pre-LN，零样本能力Larger scale, Pre-LN, zero-shot
2019	T5	编码器+解码器，统一文本到文本框架Enc+Dec, unified text-to-text
2020	GPT-3	175B 参数，少样本学习175B params, few-shot learning
2021	Vision Transformer	Transformer 用于图像Transformer for images
2022	ChatGPT / InstructGPT	RLHF 对齐，对话能力RLHF alignment, conversational
2023+	GPT-4, LLaMA, Claude...	多模态，MoE，更长上下文Multimodal, MoE, longer context

16.3 论文遗留的开放问题16.3 Open Problems from the Paper

论文在结论中提到了几个未来方向，有些已经实现，有些仍在探索：

The paper's conclusion mentions several future directions — some realized, some still open:

多模态："extend the Transformer to problems involving input and output modalities other than text" ——已实现（ViT、DALL-E、GPT-4V 等）。
受限注意力："investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video" ——已实现（Longformer, BigBird, Flash Attention 等）。
减少序列化生成："Making generation less sequential is another research goal" ——仍在探索（speculative decoding, parallel decoding 等，但自回归生成仍是主流）。
Multimodal: "extend the Transformer to problems involving input and output modalities other than text" — realized (ViT, DALL-E, GPT-4V, etc.).
Restricted attention: "investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video" — realized (Longformer, BigBird, Flash Attention, etc.).
Less sequential generation: "Making generation less sequential is another research goal" — still exploring (speculative decoding, parallel decoding, but autoregressive generation remains dominant).

16.4 为什么是这篇论文？16.4 Why This Paper?

自注意力机制不是这篇论文发明的——论文第 2 节引用了多篇前人工作。多头机制的概念也有前人探索。那为什么是这篇论文成为了分水岭？

Self-attention wasn't invented by this paper — Section 2 cites several prior works. Multi-head concepts were also explored before. So why did this paper become the watershed?

答案在于「组合」与「证明」。论文不是提出某个小改进，而是完整地将自注意力、多头、残差、层归一化、位置编码、缩放点积等组件组装成一个端到端可训练的架构，并给出了令人信服的实验结果和价值。它证明了一个大胆的假设：注意力足够强大，不需要循环，不需要卷积。这个证明打开了新世界的大门。

The answer lies in "combination" and "proof." The paper didn't propose a small improvement but completely assembled self-attention, multi-head, residual, layer norm, positional encoding, scaled dot-product into an end-to-end trainable architecture, with convincing experimental results. It proved a bold hypothesis: attention is powerful enough — no recurrence, no convolution needed. This proof opened the door to a new world.

论文标题的深意："Attention Is All You Need" 不仅仅是说「我们用了注意力」。它在说：你之前信赖的那些东西——RNN、LSTM、卷积——你不需要它们。注意力就够了。这个命题如此大胆，以至于 2017 年很多人持怀疑态度。但后续的发展——BERT、GPT、ChatGPT——一次次证明了这句话的正确性。

The title's deeper meaning: "Attention Is All You Need" isn't just saying "we used attention." It's saying: the things you relied on — RNNs, LSTMs, convolutions — you don't need them. Attention is enough. This proposition was so bold that many were skeptical in 2017. But subsequent developments — BERT, GPT, ChatGPT — repeatedly proved this statement correct.

L4 · 多概念综合论文贡献与影响Contributions and impact

总结 Transformer 论文的四大直接贡献。为什么说这篇论文是 AI 领域的分水岭？如果没有开源 tensor2tensor，影响力会不同吗？Summarize the Transformer paper's four direct contributions. Why is this paper considered a watershed moment in AI? Would the impact have been different without open-sourcing tensor2tensor?

粘贴到外部大模型获得评分后填入：

评分标准：0-2分为不通过，3-5分为通过。分数越高，下次复习间隔越长。

变式：如果这篇论文没有代码开源（tensor2tensor），影响力会不同吗？

检查点：总结 Transformer 论文的四大直接贡献。解释为什么这篇论文成为了 AI 领域的分水岭。

Checkpoint: Summarize the Transformer paper's four direct contributions. Explain why this paper became a watershed moment in AI.

机判测验Chapter Quiz

以下题目系统自动判分，你的答题情况会记录到间隔重复算法中。Machine-graded questions. Your answers feed into the spaced repetition algorithm.

本章小结Chapter Summary

Transformer 论文的四大贡献：提出纯注意力架构、证明注意力可独立工作、创造翻译 SOTA、展示泛化能力。后续影响催生了 BERT、GPT、ChatGPT 等改变世界的产品。论文遗留的多模态和受限注意力方向已实现，非序列化生成仍在探索。论文的伟大在于大胆假设加上令人信服的证明——「注意力就是你所需要的一切」。

The Transformer paper's four contributions: proposing pure attention architecture, proving attention works independently, creating translation SOTA, demonstrating generalization. Subsequent impact spawned BERT, GPT, ChatGPT. The paper's open directions on multimodal and restricted attention are realized; less sequential generation is still explored. The paper's greatness lies in a bold hypothesis with convincing proof — "Attention Is All You Need."

名词索引Glossary

自注意力（Self-Attention）：一种注意力机制，Q/K/V 来源于同一序列，让序列内部各位置相互关注。Self-Attention: An attention mechanism where Q/K/V come from the same sequence, letting positions attend to each other.

缩放点积注意力（Scaled Dot-Product Attention）：Attention(Q,K,V) = softmax(QK^T/sqrt(d_k))V，通过除以 sqrt(d_k) 防止 softmax 饱和。Scaled Dot-Product Attention: Attention(Q,K,V) = softmax(QK^T/sqrt(d_k))V, dividing by sqrt(d_k) to prevent softmax saturation.

多头注意力（Multi-Head Attention）：将注意力投影到多个子空间并行计算，再拼接回来，增加表达能力。Multi-Head Attention: Projecting attention into multiple subspaces in parallel, then concatenating, for more expressiveness.

前馈网络（FFN）：两层全连接网络（512→2048→512），带 ReLU，逐位置独立施加。Feed-Forward Network (FFN): Two-layer FC (512→2048→512) with ReLU, applied position-wise.

位置编码（Positional Encoding）：用 sin/cos 函数生成的位置信号，注入到嵌入中，弥补注意力缺乏位置感知的缺陷。Positional Encoding: Position signals generated with sin/cos, injected into embeddings to compensate for attention's position blindness.

残差连接（Residual Connection）：output = x + Sublayer(x)，帮助梯度流动。Residual Connection: output = x + Sublayer(x), helps gradient flow.

层归一化（Layer Normalization）：沿特征维度归一化，不依赖 batch，适合变长序列。Layer Normalization: Normalizes along feature dimensions, batch-independent, suitable for variable-length sequences.

因果掩码（Causal Mask）：解码器中防止位置看到未来信息的下三角掩码。Causal Mask: Lower-triangular mask in decoder preventing positions from seeing future information.

自回归（Auto-regressive）：生成第 t 个词只能依赖前 t-1 个已生成词。Auto-regressive: Generating token t depends only on the first t-1 generated tokens.

Teacher Forcing：训练时用真实标签而非模型预测作为解码器输入。Teacher Forcing: Using ground truth rather than model predictions as decoder input during training.

Label Smoothing：把目标分布从 one-hot 变为平滑分布，牺牲 PPL 换 BLEU。Label Smoothing: Changing target distribution from one-hot to smooth, trading PPL for BLEU.

Warmup：学习率线性预热阶段（前 4000 步），防止训练初期不稳定。Warmup: Linear LR ramp-up phase (first 4000 steps), preventing early training instability.

权重共享（Weight Sharing）：编码器嵌入、解码器嵌入和 pre-softmax 线性变换共用权重。Weight Sharing: Encoder embedding, decoder embedding, and pre-softmax linear transform share weights.

BPE（Byte-Pair Encoding）：子词分词方法，通过合并频率最高的字符对来构建词表。BPE (Byte-Pair Encoding): Subword tokenization, building vocabulary by merging most frequent character pairs.

BLEU：机器翻译评估指标，衡量生成翻译与参考翻译的 n-gram 重叠度。BLEU: Machine translation evaluation metric, measuring n-gram overlap between generated and reference translations.

参考来源Sources

Vaswani, A. et al. (2017). "Attention Is All You Need." arXiv:1706.03762v7. NIPS 2017.
arXiv 全文：https://arxiv.org/abs/1706.03762
tensor2tensor 代码库：https://github.com/tensorflow/tensor2tensor
Vaswani, A. et al. (2017). "Attention Is All You Need." arXiv:1706.03762v7. NIPS 2017.
arXiv full text: https://arxiv.org/abs/1706.03762
tensor2tensor repository: https://github.com/tensorflow/tensor2tensor

本精读页面基于论文原文撰写，所有公式、数据、表格均来自原论文。解释和类比为原创内容，旨在帮助理解。This reading guide is based on the original paper. All formulas, data, and tables are from the paper. Explanations and analogies are original content for understanding.

Attention Is All You Need Attention Is All You Need

为什么不只是再写一篇解读Why Not Just Another Summary

动手验证Hands-on Verification

图解架构Visual Architecture

测出真懂Test What You Know

随时提问Ask Anytime

六个值得停下来把玩的地方Six Things Worth Pausing For

注意力公式实验室Attention Formula Lab

多头注意力拆解Multi-Head Split

位置编码波形Positional Encoding Waves

编码器堆叠图解Encoder Stack Diagram

翻译结果对比Translation Results

消融实验探索Ablation Lab

怎么用这个页面How to Use This Page

第0章 起步Chapter 0: Getting Started

学完这一章你应该能做到After this chapter you should be able to

0.1 一句话说清楚0.1 In One Sentence

打个比方Analogy

0.2 为什么要扔掉循环网络0.2 Why Throw Away Recurrence

为什么并行这么重要Why Parallelization Matters

0.3 论文的主要成果0.3 Key Results

关于作者About the Authors

0.4 这篇论文的结构0.4 Paper Structure

答辩：如果我是审稿人Debate: If I Were a Reviewer

本章自测Chapter Quiz

本章小结Chapter Summary

第1章 序列建模的旧世界Chapter 1: The Old World of Sequence Modeling

学完这一章你应该能做到After this chapter you should be able to

1.1 循环网络的统治时代1.1 The Reign of Recurrence

串行瓶颈The Sequential Bottleneck

1.2 卷积的尝试1.2 The Convolution Attempt

卷积的远距离问题Convolution's Long-Range Problem

1.3 早期注意力机制1.3 Early Attention

本章小结Chapter Summary

第2章 自注意力的核心直觉Chapter 2: The Core Intuition of Self-Attention

学完这一章你应该能做到After this chapter you should be able to

2.1 编码器-解码器框架2.1 The Encoder-Decoder Framework

2.2 注意力到底是什么2.2 What Attention Actually Is

图书馆的比方The Library Analogy

2.3 输出是怎么算出来的2.3 How Output Is Computed

2.4 自注意力 vs 编码器-解码器注意力2.4 Self-Attention vs Encoder-Decoder Attention

本章小结Chapter Summary

第3章 缩放点积注意力Chapter 3: Scaled Dot-Product Attention

学完这一章你应该能做到After this chapter you should be able to

3.1 公式全貌3.1 The Full Formula

推导：从直觉到公式Derivation: From Intuition to Formula

3.2 为什么要缩放：sqrt(d_k) 的来历3.2 Why Scale: The Story of sqrt(d_k)

问题：点积为什么会爆炸Problem: Why Dot Products Explode

存疑点：为什么不除以 d_kOpen Question: Why Not d_k?

3.3 点积注意力 vs 加性注意力3.3 Dot-Product vs Additive Attention

为什么点积更快Why Dot-Product Is Faster

答辩：如果我是审稿人Debate: If I Were a Reviewer

本章自测Chapter Quiz

本章小结Chapter Summary

第4章 多头注意力Chapter 4: Multi-Head Attention

4.1 为什么要多头？4.1 Why Multiple Heads?

4.2 数学公式4.2 Mathematical Formulation

4.3 计算成本分析：为什么「多头」不亏？4.3 Computational Cost: Why Multi-Head Is "Free"

4.4 多头到底做了什么？实验证据4.4 What Do Heads Actually Do? Experimental Evidence

实验室：多头注意力模拟器Lab: Multi-Head Attention Simulator

机判测验Chapter Quiz

本章小结Chapter Summary

第5章 注意力在 Transformer 中的三种用法Chapter 5: Three Applications of Attention in the Transformer

5.1 编码器自注意力（Encoder Self-Attention）5.1 Encoder Self-Attention

5.2 编码器-解码器注意力（Encoder-Decoder Attention）5.2 Encoder-Decoder Attention

5.3 解码器自注意力（Decoder Self-Attention, with Masking）5.3 Decoder Self-Attention (with Masking)

5.4 三种用法对比表5.4 Comparison Table

机判测验Chapter Quiz

本章小结Chapter Summary

第6章 逐位置前馈网络Chapter 6: Position-wise Feed-Forward Network

6.1 FFN 的结构6.1 FFN Structure

6.2 「逐位置」是什么意思？6.2 What Does "Position-wise" Mean?

6.3 FFN 的角色：注意力之后的「思考」6.3 FFN's Role: "Thinking" After Attention

机判测验Chapter Quiz

本章小结Chapter Summary

第7章 嵌入与 SoftmaxChapter 7: Embeddings and Softmax

7.1 嵌入层的作用7.1 The Role of Embedding Layers

7.2 权重共享：三个部件，一套权重7.2 Weight Sharing: Three Components, One Weight Matrix

7.3 为什么要乘 √d_model？7.3 Why Multiply by √d_model?

机判测验Chapter Quiz

第0章起步Chapter 0: Getting Started

第1章序列建模的旧世界Chapter 1: The Old World of Sequence Modeling

第2章自注意力的核心直觉Chapter 2: The Core Intuition of Self-Attention

第3章缩放点积注意力Chapter 3: Scaled Dot-Product Attention

第4章多头注意力Chapter 4: Multi-Head Attention

第5章注意力在 Transformer 中的三种用法Chapter 5: Three Applications of Attention in the Transformer

第6章逐位置前馈网络Chapter 6: Position-wise Feed-Forward Network

第7章嵌入与 SoftmaxChapter 7: Embeddings and Softmax

第8章位置编码Chapter 8: Positional Encoding

第9章编码器结构Chapter 9: Encoder Structure

第10章解码器结构Chapter 10: Decoder Structure

第11章残差连接与层归一化Chapter 11: Residual Connections and Layer Normalization

第12章为什么是自注意力Chapter 12: Why Self-Attention

第13章训练策略Chapter 13: Training Strategy

第14章翻译结果Chapter 14: Machine Translation Results

第15章消融实验Chapter 15: Ablation Study

第16章贡献与影响Chapter 16: Contributions and Impact