学透 Kimi K3 技术报告

第0章 起步：读这篇论文之前，你需要的全部基础

这一章不属于论文。它是一块地基。读完它，你会拥有读懂后面十五章所需的每一个概念——不是「听说过」，而是「能用自己的话讲清楚，并且算得出来」。如果你已经知道什么是注意力机制和 KV 缓存，可以跳到第1章；但我建议你至少做一遍本章末尾的自测，看看是不是真的知道。

0.1 先给你一个方向感：这篇论文在讲什么

Kimi K3 是月之暗面（Moonshot AI）在 2026 年 7 月发布的一个大语言模型，同时公开了模型权重。这份 47 页的技术报告讲的是：他们怎么把它造出来的。

造一个这种规模的模型，大致要回答四类问题，而这份报告的结构正好对应这四类：

1. 模型长什么样（第 2 节，本站第 3–6 章）——里面有哪些零件，为什么这么摆。
2. 怎么把知识灌进去（第 3 节，本站第 7 章）——用什么数据、训多久、怎么定超参数。
3. 怎么让它学会做事（第 4 节，本站第 8–9 章）——从「会说话」到「会干活」，靠的是后训练和强化学习。
4. 怎么让这一切在真实机器上跑得动（第 5 节，本站第 10–11 章）——2.8 万亿个参数装不进任何一张显卡，这本身就是个大工程。

剩下的是第 6 节的评测（本站第 12 章）、第 7 节的案例（第 13 章）和附录里的几个证明（第 14 章）。

0.2 语言模型到底在做什么：猜下一个词

这件事说出来会让人失望，但它是真的：大语言模型做的唯一一件事，是根据前面的文字，猜下一个字最可能是什么。

比如给它「中华人民共和」，它会给出一个概率分布：

然后按这个分布抽一个字出来，接在后面，变成「中华人民共和国」，再拿这个新的句子去猜下下一个字。如此反复，就「写」出了一整段话。

下一 token 预测（Next-Token Prediction, NTP）：上面这个「看前文、猜下一个」的任务。整篇论文里所有的预训练，做的都是这一件事。

0.3 token：模型眼里的「字」

上一节我说「猜下一个字」，那是简化。模型实际处理的单位叫 token，中文常译作「词元」。

token 不是字，也不是词，而是一种介于两者之间的、由统计方法切出来的片段。比如英文 unhappiness 可能被切成 un + happi + ness 三个 token；中文里常见字通常一个字就是一个 token，生僻字可能被拆成两三个。

词表（vocabulary）：所有可能 token 的清单。Kimi K3 的词表大小是 160K（16 万个），这个数字你在表 1 里会看到。

所以更准确的说法是：模型看到的是一串编号（比如 [3612, 8829, 174, ...]），它要预测的是下一个编号，从 16 万个候选里挑。

0.4 向量：把 token 变成一串数

计算机不能直接对「编号 3612」做数学。所以第一步是把每个 token 变成一串数字，叫做向量。

向量（vector）：一串有顺序的数，比如 (0.3, −1.2, 0.8)。这串数有多长，就叫它的维度。

Kimi K3 用 7168 维的向量表示一个 token。也就是说，「猫」这个字在模型内部是 7168 个小数组成的一串。

0.5 矩阵乘法：神经网络里唯一真正重的运算

这一节是全章最重要的一节。如果你只学一个数学工具，学这个。

矩阵（matrix）：把数排成一个长方形的表格。一个 3 × 2 的矩阵有 3 行 2 列。

矩阵乘以向量是这样算的：把向量竖着放在矩阵右边，矩阵的每一行与这个向量逐个相乘再求和，得到结果向量的一个分量。

注意形状：一个 3×2 的矩阵乘一个 2 维向量，得到一个 3 维向量。矩阵把向量从一个维度搬到了另一个维度，同时把信息重新混合了一遍。

0.6 一层神经网络 = 矩阵乘法 + 掰弯

如果只有矩阵乘法，会出问题：连续做很多次矩阵乘法，效果等同于只做一次（两个矩阵可以先乘起来变成一个矩阵）。那么叠 93 层就毫无意义。

所以每做完一次矩阵乘法，要插进一个非线性函数——一个不能被矩阵乘法模拟的、把直线掰弯的操作。

激活函数（activation function）：那个负责「掰弯」的函数。常见的有：

第5章的主角 SiTU-GLU 就是在 Swish 的基础上改造出来的，你到那时会需要这张表。

0.7 softmax：把任意一串数变成一组概率

模型算出 16 万个候选 token 的「分数」之后，怎么变成概率？分数可能是负的，加起来也不等于 1。

softmax 解决这个问题，两步：

1. 对每个数取指数 e^x（结果一定是正数，而且大的数被放大得更厉害）
2. 除以所有指数的总和（这样加起来正好是 1）

动手算一遍：三个候选的分数是 (2, 1, 0)。

0.8 注意力：让每个 token 自己去「查资料」

现在到了最关键的机制。注意力（attention）回答的问题是：处理「它」这个词时，模型怎么知道「它」指的是前面哪个东西？

答案是：让「它」主动去前面查一遍，看谁跟自己最相关，就多读谁的信息。具体分三步。

查询、键、值（Query, Key, Value，简称 Q、K、V）：每个 token 用三个不同的矩阵，从自己的向量里造出三个新向量。Q 是「我要找什么」，K 是「我是什么，供别人搜」，V 是「如果你选中我，我给你什么内容」。

0.9 注意力的代价：为什么长文本这么贵

注意力的代价来自一个简单的事实：序列里每个 token 都要跟前面所有 token 打一次分。

如果序列有 n 个 token，打分次数大约是 n²/2。这叫平方复杂度。

从 1000 涨到 100 万，长度涨 1000 倍，计算量涨一百万倍。这就是为什么「100 万 token 上下文」在工程上是件大事。

还有第二个代价，而且往往更要命。

KV 缓存（KV cache）：生成文字时，模型是一个 token 一个 token 往外吐的。每吐一个新 token，它都要回头查之前所有 token 的 K 和 V。为了不重复计算，就把算过的 K 和 V 全都存在显存里——这就是 KV 缓存。

0.10 残差连接：给信息修一条高速公路

网络很深（K3 有 93 层）时会出现一个问题：信息在层层传递中被反复揉搓，原始内容丢失，而且训练时的梯度信号传不回前面的层。

残差连接（residual connection）的做法极其简单：让每一层的输出等于「这一层算出来的东西」加上「这一层的输入」。

0.11 归一化：不让数字失控

深层网络里数值容易越滚越大或越滚越小。归一化（normalization）就是每隔一段把一组数拉回到一个标准尺度。

RMSNorm（Root Mean Square Normalization）：先算这组数的「均方根」（每个数平方、求平均、开根号），再让每个数都除以它。结果是这组数的整体大小被拉到 1 附近，但相互之间的比例关系不变。

论文里 RMSNorm 出现过至少五次，每次都是同一个动机：某个地方的数值尺度可能失控，先归一化再往下走。你在第4章（防止某一层主导注意力权重）、第5章（防止专家聚合后的尺度乱跳）都会见到它。

0.12 MoE：一个巨大的专家团，每次只请几个

常规神经网络里，每个 token 都要经过全部参数。参数越多，算得越慢。这让「模型变大」和「推理变贵」死死绑在一起。

混合专家（Mixture-of-Experts, MoE）解开了这个绑定：

1. 把一层里的前馈网络复制成很多份，每份叫一个专家（expert）。K3 有 896 个。
2. 加一个小小的路由器（router），它看一眼当前 token，给 896 个专家各打一个分。
3. 只把分数最高的几个专家叫来干活。K3 每个 token 只激活 16 个。

0.13 三个必须分清的量：总参数、激活参数、FLOPs

0.14 训练与推理：两种完全不同的活

训练（training）：给模型看海量文本，让它猜下一个 token，猜错了就微调所有参数让它下次猜得准一点。这个「根据错误反过来调参数」的过程叫反向传播（backpropagation），它需要把前向过程中的中间结果（叫激活值，activations）都保存下来——这是训练时显存爆炸的主因，第10章会详谈。

推理（inference）：模型训练好了，参数冻结，只用来生成文字。推理又分两个阶段：

• 预填充（prefill）：把你输入的整段提示一口气读进去。这一步所有 token 可以并行处理，算力吃满，是「计算受限」的。
• 解码（decode）：一个一个往外吐字。这一步必须串行（要先吐出第 5 个字才能吐第 6 个），每吐一个字都要把整个模型的权重过一遍，所以是「带宽受限」的。

0.15 三段人生：预训练、后训练、强化学习

强化学习（Reinforcement Learning, RL）：模型自己尝试完成一个任务（比如「修好这个 bug」），产生一整条行动轨迹；然后有个奖励函数给这条轨迹打分；模型据此调整参数，让高分行为更容易发生。

智能体（agent）：能自主使用工具、多步执行任务的模型。工具调用（tool call）指模型输出一段结构化的请求（比如「运行这段 Python」），外部系统执行后把结果送回来作为新的输入。一次任务里模型可能这样来回几百次，这叫长时程（long-horizon）。

0.16 GPU：为什么显存和带宽经常比算力更要命

训练和运行大模型全靠 GPU（图形处理器）。你需要知道三个数，以及它们之间的紧张关系。

还有一组必须知道的名词——当一张卡装不下时，就要把模型切开放到多张卡上，这叫并行：

0.17 100 万 token 的上下文，到底是什么概念

上下文窗口（context window）：模型一次能「看见」的最大 token 数。超出的部分它就完全不知道了。

K3 是 100 万 token。给你一点尺度感：

而 K3 的前一代 K2 只有 128K（12.8 万）。表 1 里那个「8×」就是这么来的。

0.18 符号速查表

下面这些符号会在后面十五章反复出现。现在不需要背，但读到卡壳时回来查。

Chapter 0 · Getting started: everything you need before you read this paper

This chapter is not part of the paper. It is a foundation. Once you have read it you will hold every concept the following fifteen chapters need — not “I have heard of it”, but “I can explain it in my own words, and I can compute it”. If you already know what attention and the KV cache are, skip to Chapter 1; but I suggest you at least work through the self-test at the end of this chapter to see whether you really do know.

0.1 A sense of direction first: what this paper is about

Kimi K3 is a large language model released by Moonshot AI in July 2026, with the model weights made public at the same time. This 47-page technical report is about how they built it.

Building a model at this scale means answering roughly four kinds of question, and the structure of the report maps exactly onto those four:

1. What the model looks like (Section 2; Chapters 3–6 of this site) — which parts are in it, and why they are arranged this way.
2. How the knowledge gets poured in (Section 3; Chapter 7 of this site) — what data, how long the training runs, how the hyperparameters are set.
3. How it learns to do things (Section 4; Chapters 8–9 of this site) — going from “can talk” to “can work” rests on post-training and reinforcement learning.
4. How to make all of this run on real machines (Section 5; Chapters 10–11 of this site) — 2.8 trillion parameters do not fit on any single GPU, and that alone is a large engineering job.

What is left is the evaluation in Section 6 (Chapter 12 of this site), the case studies in Section 7 (Chapter 13), and a few proofs in the appendix (Chapter 14).

0.2 What a language model is actually doing: guessing the next word

Saying it out loud is a let-down, but it is true: the only thing a large language model does is guess, from the text in front of it, what the next character is most likely to be.

Give it “the People's Republic of Chin”, for example, and it hands you a probability distribution:

Then it draws one character from that distribution, sticks it on the end to give “the People's Republic of China”, and takes the new sentence to guess the character after that. Repeat, and it has “written” a whole passage.

Next-token prediction: the task above — look at what came before, guess what comes next. All the pre-training in the entire paper is this one thing.

0.3 token: the “character” as the model sees it

In the last section I said “guess the next character”. That was a simplification. The unit the model actually works on is called a token; Chinese usually renders it as “cíyuán”.

A token is not a character and not a word, but a fragment somewhere between the two, cut out by statistics. English unhappiness, for instance, may be cut into the three tokens un + happi + ness; in Chinese a common character is usually a token on its own, while a rare character may be split into two or three.

Vocabulary: the list of every possible token. Kimi K3's vocabulary size is 160K (160,000 of them), a number you will meet in Table 1.

So the more accurate statement is: the model sees a string of ID numbers (say [3612, 8829, 174, ...]), and what it has to predict is the next ID number, picked out of 160,000 candidates.

0.4 Vectors: turning a token into a string of numbers

A computer cannot do mathematics directly on “ID number 3612”. So the first step is to turn every token into a string of numbers, called a vector.

Vector: an ordered string of numbers, for example (0.3, −1.2, 0.8). How long that string is is called its dimension.

Kimi K3 uses a 7168-dimensional vector to represent one token. That is: inside the model, the word “cat” is a string of 7168 decimal numbers.

0.5 Matrix multiplication: the only genuinely heavy operation in a neural network

This is the most important section in the chapter. If you learn one piece of mathematics, learn this one.

Matrix: numbers laid out in a rectangular table. A 3 × 2 matrix has 3 rows and 2 columns.

A matrix times a vector is computed like this: stand the vector up on the right of the matrix, take each row of the matrix and multiply it against that vector element by element, then sum, giving one component of the result vector.

Watch the shapes: a 3×2 matrix times a 2-dimensional vector gives a 3-dimensional vector. A matrix carries a vector from one dimension into another, and remixes the information on the way.

0.6 One layer of a neural network = matrix multiplication + a bend

With matrix multiplication alone there is a problem: doing many matrix multiplications in a row has the same effect as doing one (two matrices can be multiplied together into a single matrix first). Which would make stacking 93 layers pointless.

So after every matrix multiplication you insert a non-linear function — an operation no matrix multiplication can imitate, one that bends the straight line.

Activation function: the function in charge of the bending. Common ones:

SiTU-GLU, the lead character of Chapter 5, is built on top of Swish, and you will need this table when you get there.

0.7 softmax: turning any string of numbers into a set of probabilities

Once the model has computed a “score” for each of 160,000 candidate tokens, how do those become probabilities? A score can be negative, and they do not add up to 1 either.

softmax solves this, in two steps:

1. Take the exponential e^x of each number (the result is always positive, and larger numbers are blown up harder)
2. Divide by the sum of all the exponentials (so they add up to exactly 1)

Work it through by hand: three candidates with scores (2, 1, 0).

0.8 Attention: letting every token “look things up” for itself

Now for the most important mechanism of all. Attention answers this question: when it processes the word “it”, how does the model know which earlier thing “it” refers to?

The answer: let “it” go back and run a search over what came before, see who is most related to it, and read more from whoever that is. Three steps, concretely.

Query, key, value: every token uses three different matrices to build three new vectors out of its own vector. Q is “what I am looking for”, K is “what I am, for others to search against”, V is “what I hand you if you pick me”.

0.9 The price of attention: why long text is so expensive

The price of attention comes from one simple fact: every token in the sequence has to score itself against every token before it.

If the sequence has n tokens, the number of scorings is about n²/2. This is called quadratic complexity.

Going from 1000 to 1 million, the length goes up 1000-fold and the compute goes up a millionfold. This is why “a 1 million token context” is a serious matter in engineering.

There is a second price, and it is often the deadlier one.

KV cache: when it generates text, the model spits out one token at a time. For each new token it spits out, it has to go back and look at the K and V of every earlier token. To avoid computing them again, every K and V already computed is kept in GPU memory — that is the KV cache.

0.10 Residual connections: building a motorway for information

When the network is deep (K3 has 93 layers) a problem shows up: information is kneaded over and over as it passes from layer to layer, the original content is lost, and during training the gradient signal cannot get back to the earlier layers.

What a residual connection does is extremely simple: make each layer's output equal to “what this layer computed” plus “what came into this layer”.

0.11 Normalisation: keeping the numbers from running away

In a deep network values easily snowball larger and larger, or smaller and smaller. Normalisation is pulling a group of numbers back to a standard scale every so often.

RMSNorm: first compute the “root mean square” of the group (square each number, take the average, take the square root), then divide every number by it. The result is that the overall size of the group is pulled to somewhere near 1, but the ratios between them do not change.

RMSNorm appears at least five times in the paper, with the same motivation every time: the numeric scale somewhere may run away, so normalise before going on. You will meet it in Chapter 4 (stopping one layer from dominating the attention weights) and in Chapter 5 (stopping the scale from jumping around after the experts are aggregated).

0.12 MoE: a huge panel of experts, only a few called in each time

In an ordinary neural network every token has to pass through all the parameters. More parameters, slower compute. That binds “make the model bigger” and “make inference more expensive” tightly together.

Mixture-of-Experts (MoE) unties the binding:

1. Copy the feed-forward network in a layer many times over; each copy is called an expert. K3 has 896 of them.
2. Add a tiny router, which glances at the current token and gives each of the 896 experts a score.
3. Call in only the few experts with the highest scores to do the work. K3 activates only 16 per token.

0.13 Three quantities you must keep apart: total parameters, activated parameters, FLOPs

0.14 Training and inference: two completely different jobs

Training: show the model an enormous amount of text, have it guess the next token, and when it guesses wrong adjust every parameter a little so that it guesses more accurately next time. That process of “adjusting the parameters backwards from the error” is called backpropagation, and it requires keeping all the intermediate results of the forward pass (the activations) — the main reason GPU memory explodes during training, which Chapter 10 goes into in detail.

Inference: the model is trained, the parameters are frozen, and it is only used to generate text. Inference splits again into two stages:

• Prefill: read the whole prompt you typed in one go. Every token in this step can be processed in parallel, the compute units run flat out, and it is “compute-bound”.
• Decode: spit characters out one at a time. This step has to be serial (character 5 must come out before character 6 can), and every character spat out means running through the whole model's weights, so it is “bandwidth-bound”.

0.15 Three lives: pre-training, post-training, reinforcement learning

Reinforcement learning: the model tries to complete a task on its own (say “fix this bug”), producing a whole trajectory of actions; then a reward function scores that trajectory; the model adjusts its parameters accordingly, so that high-scoring behaviour becomes more likely.

Agent: a model that can use tools on its own and carry out a task over many steps. A tool call (tool call) means the model emits a structured request (say “run this Python”); an external system executes it and sends the result back as new input. In one task the model may go back and forth like this several hundred times, which is called long-horizon (long-horizon).

0.16 GPU: why memory and bandwidth are so often deadlier than compute

Training and running a large model rests entirely on GPUs (graphics processing units). You need to know three numbers, and the tension between them.

One more set of terms you have to know — when one card cannot hold the model, you cut the model up across several cards, and that is called parallelism:

0.17 A context of 1 million tokens: what that actually means

Context window: the largest number of tokens the model can “see” at once. Anything beyond it, it knows nothing about at all.

K3 is 1 million tokens. To give you a sense of the scale:

K2, the generation before K3, had only 128K (128,000). That “8×” in Table 1 is where it comes from.

0.18 Symbol quick reference

The symbols below come back again and again over the next fifteen chapters. You do not need to memorise them now, but come back and look them up when you get stuck.

第1章 全景：这篇论文在主张什么，以及知识地图

在钻进任何细节之前，你需要一张地图。这一章做三件事：说清这篇论文想解决什么问题、把它的十六个知识点排成一张依赖图、给你一条推荐的行进路线。读完这一章，你后面每读到一个新概念，都能知道它挂在哪根枝上。

1.1 论文开篇的那个论断：缩放有两条轴，开源界只走了一条

论文的第一段做了一件很聪明的事：它先给整个领域画了一个坐标系，然后指出坐标系里有一块空地，而 K3 就是要去占这块空地。

第一条轴：部署前的算力。 在大语言模型发展的大部分时间里，「缩放」意味着在部署之前投入更多算力——训更大的模型、喂更多的数据。这是 2020 年那批缩放定律论文奠定的路线。

第二条轴：测试时的算力。 推理模型的兴起确立了第二条轴：让模型在回答问题的时候多花算力。论文点名了几条支线：OpenAI 的 o 系列用强化学习扩展测试时推理；Anthropic 的扩展思考让模型分配自适应的思考预算并把思考与工具调用交错；DeepSeek-R1 和 Kimi K1.5 表明大规模强化学习能从强预训练模型中引出复杂的推理行为；Kimi K2.5 Agent Swarm 则把测试时缩放从「顺序推理」扩展到「并行的智能体协同」。

1.2 摘要拆解：每一个短语对应哪一节

下面这张表是本站最实用的一张。摘要里的每个短语，都能在论文里找到一整节来展开。你也可以把它当作全站的导航。

1.3 知识地图：谁依赖谁

下面这张图是整篇论文的骨架。箭头 A → B 读作「要理解 B，先得理解 A」。

1.4 架构总纲：把信息流往三个方向撑开

论文第 2 节开头给了一个非常清晰的组织原则，值得单独拎出来记住。K3 的架构设计沿三个互补的维度扩展信息流：

1.5 论文的四条贡献，以及它们的证据强度

论文在引言末尾自己列了四条贡献。这里逐条列出，同时标注证据强度——这是训练「批判性阅读」的第一次练习。

1.6 推荐的行进路线

十六章不必按顺序读完。下面是三条路线，按你的目标挑一条。

Chapter 1 · The big picture: what this paper claims, and the knowledge map

Before you dig into any detail, you need a map. This chapter does three things: it states what problem the paper is trying to solve, arranges its sixteen ideas into a dependency graph, and gives you a recommended route through them. Once you have read it, every new concept you meet later will have a branch to hang on.

1.1 The claim the paper opens with: scaling has two axes, and open source has walked only one

The paper's first paragraph does something clever: it draws a coordinate system for the whole field, then points out an empty patch inside it — and K3 is what goes there.

Axis one: compute before deployment. For most of the history of large language models, “scaling” meant putting more compute in before deployment — training a bigger model, feeding it more data. This is the route laid down by the scaling-law papers of 2020.

Axis two: compute at test time. The rise of reasoning models established a second axis: let the model spend more compute while it is answering the question. The paper names several strands: OpenAI's o series scales test-time reasoning with reinforcement learning; Anthropic's extended thinking lets the model allocate an adaptive thinking budget and interleave thinking with tool calls; DeepSeek-R1 and Kimi K1.5 show that large-scale reinforcement learning can draw complex reasoning behaviour out of a strong pretrained model; and Kimi K2.5 Agent Swarm extends test-time scaling from “sequential reasoning” to “parallel agent collaboration”.

1.2 The abstract taken apart: which section each phrase belongs to

The table below is the most useful one on this site. Every phrase in the abstract has a whole section of the paper behind it. You can also use it as navigation for the whole site.

1.3 The knowledge map: what depends on what

The figure below is the skeleton of the whole paper. Read an arrow A → B as “to understand B, you first have to understand A”.

1.4 The architecture in outline: pushing information flow out in three directions

The opening of §2 gives a very clear organising principle, worth pulling out and remembering on its own. K3's architecture scales information flow along three complementary dimensions:

1.5 The paper's four contributions, and how strong the evidence is

At the end of the introduction the paper lists four contributions of its own. Here they are one by one, with the strength of the evidence marked alongside — this is the first exercise in reading critically.

1.6 The recommended route

You do not have to read all sixteen chapters in order. Here are three routes; pick one according to your goal.

第2章 Kimi Delta Attention：把无限增长的记忆压成一块固定大小的黑板

这是全站最核心也最难的一章。KDA 是 K3 架构的地基——93 层里有 69 层是它，第10章的内核与并行、第11章的前缀缓存，全都是在为它擦屁股。这一章我们会从「记忆该怎么建模」这个最朴素的问题出发，一步步走到论文的式 1 到式 6。请慢一点读，每个小节末尾的题都做。

2.1 出发点：注意力的诅咒，和一个被放弃的老方案

第0章讲过注意力的两个代价：计算量随长度平方增长，KV 缓存随长度线性增长且永不缩小。在 100 万 token 的尺度上，第二个代价尤其致命——你算过那道题，光是一份缓存就能吃掉一张顶级显卡三成的显存。

但在注意力发明之前，序列建模用的是另一套方案：循环神经网络（RNN）。它的做法是从左到右读，把读过的一切压缩进一个固定大小的状态里。

RNN 的固定状态有一个无可替代的优点：不管序列多长，内存都不变。但它当年被抛弃，是因为两个致命伤：

• 记忆被冲淡：状态就那么大，读到第 1000 个词时，第 1 个词的信息早被后来的内容挤掉了。
• 不能并行：必须先算完第 999 步才能算第 1000 步。而 GPU 的全部威力都建立在「同时算很多东西」上。

线性注意力（linear attention）：一类把 softmax 去掉、从而可以改写成「固定大小状态 + 递推更新」的注意力变体。KDA 属于这一类，它是 Kimi Linear 那篇工作的延续。第0章的 softmax 那一节你算过：softmax 里的指数让每个 query 必须亲自和每个 key 打分，无法提前汇总。去掉 softmax，汇总就成为可能——这是整个线性注意力家族的数学起点。

2.2 状态是一块黑板：怎么「写」和「读」

先把 KDA 的状态 S 想象成一块黑板。它是一个矩阵，形状是 d_k × d_v——第0章讲过，矩阵就是一个数字表格。

怎么往黑板上写一个「键 → 值」的配对？ 答案是外积：

第0章提过，一个竖向量乘一个横向量得到一个矩阵。kv^⊤ 就是这么一个矩阵——它可以理解为「在 k 指向的那个方向上，刻下 v 这个内容」。

怎么从黑板上读？ 用查询向量去乘它：

为什么这样能读出来？做一次代数就明白了。假设黑板上只写过一对 (k, v)，那么：

注意 k^⊤q 是一个数（两个向量的点积，也就是它们的相似度）。所以读出来的是「v 乘以一个相似度」。

2.3 delta 规则：写之前先擦

朴素写入 S ← S + kv^⊤ 有个明显的毛病：如果同一个键被写了两次，两个值会叠在一起。

假设「首都」这个键先被写成「北京」，后来上下文变了要改成「东京」。朴素写入的结果是黑板上同时有「北京」和「东京」，读出来是两者的混合——一团糨糊。这正是 RNN 「记忆被冲淡」的数学本质。

delta 规则的想法很直接：写新的之前，先把这个键上原有的内容擦掉。

关键在那个 (I − β_tk_tk_t^⊤)。它长得吓人，但意思很简单。让我们看看它作用在旧状态上会发生什么。

2.4 加上遗忘：式 1 的完整形态

delta 规则解决了「同一个键被反复覆盖」的问题。但还有一个问题它解决不了：那些再也用不到的旧信息，怎么慢慢淡出？

比如你读一篇长文档，第一章的细节到第十章基本就不需要了。delta 规则只会在「同一个键被重写」时才擦，从没被重写的旧内容会永远占着黑板。

解法是加一个遗忘门：每走一步，就让整块黑板按一定比例淡化一点。这就得到了论文的式 1：

2.5 式 2：这些 q、k、v、α、β 是从哪来的

式 1 里的每个量都要从 token 向量 x_t 算出来。论文式 2 给了具体做法（h 是头的编号）：

2.6 分块并行形式：让串行的东西跑在 GPU 上

现在面对 RNN 的第二个致命伤：式 1 是严格串行的。S_t 依赖 S_t−1，一百万个 token 就要一百万步，GPU 完全用不上。

解法叫分块并行形式（chunkwise parallel form），思路是一个漂亮的折中：

要写出这个形式，先需要一个记号。论文式 3 定义累积衰减：

有了它，再加上一个叫 UT 变换的技巧产出的 U_[t] 和 W_[t]，定义伪值项 Ṽ_[t] := U_[t] − W_[t]S_[t]，块内所有输出就可以并行算出来：

这条式子不需要你能自己推出来（论文自己也把完整推导推给了 Kimi Linear 那篇），但你需要读懂它在说什么：

2.7 下界衰减：一个溢出问题如何改写了数学

这一节是全章最精彩的地方，因为它展示了一件在论文里很少被讲得这么透的事：硬件的数值精度限制，可以反过来决定模型的数学形式。

问题：那个除法会爆炸

回看式 4 里的 K_[t] / Γ_[t]^1→C——键要除以累积衰减。

而 Γ 是一串 (0,1) 之间的数连乘。连乘会发生什么？

BF16：训练大模型最常用的数字格式，每个数占 2 字节。它能表示的最大值约为 3.4 × 10³⁸。超过就变成「无穷大」，计算全毁。

论文原文的说法是：「因为 Γ 是一串 (0,1) 中保留因子的乘积，这个倒数可以无界增长，在有限精度下溢出。」

前代的解法，以及它留下的疙瘩

Kimi Linear（KDA 的前身）用两招控制这个数值范围：

1. 在对数空间计算相对衰减——连乘变成连加，动态范围大幅压缩。
2. 把每个块再划分成 16 个 token 的次级小片（tile）——只在小片内部算相对衰减，跨小片的部分另外处理。

这两招之后，非对角的小片可以直接用稠密矩阵乘法在 Tensor Core 上算。但对角线上的小片不行——论文原话是：对角小片「仍然需要显式的位置对计算，这仍是块内的主要瓶颈」。

Tensor Core：现代 GPU 里专门做矩阵乘法的硬件单元，比通用计算单元快好几倍。但它只吃「标准的稠密矩阵乘法」这一种形式。任何需要「逐个位置对单独处理」的计算都用不上它，只能退回慢得多的通用路径。

K3 的解法：换掉衰减函数本身

这里是关键的思维转折。前人的思路是「数会爆炸，那我想办法在计算上绕开」。K3 的思路是「数会爆炸，那我从一开始就不让它爆炸」。

回忆 2.5 节留下的悬念：从衰减 logit z 到保留因子 α 的映射被推迟了。现在揭晓这个映射：

对比一下前代的做法。Kimi Linear（以及 GDN、Mamba-2）用的是负 softplus 映射：

差别只有一个字：值域。前代是 (−∞, 0)，K3 是 (−5, 0)。前代无下界，K3 有下界。这就是「下界衰减」（lower-bounded decay）这个名字的来源。

为什么 −5 这个数字刚刚好

论文给出的推理链条非常干净，值得完整走一遍：

2.8 满秩输出门：最后一道闸

式 1 算出的 ō_t 还不是这一层的最终输出。KDA 在它后面加了一道门：

Chapter 2 · Kimi Delta Attention: compressing a memory that grows without bound onto one fixed-size blackboard

This is the most central and the hardest chapter on this site. KDA is the foundation of the K3 architecture — 69 of the 93 layers are KDA, and the kernels and parallelism of Chapter 10 and the prefix cache of Chapter 11 all exist to clean up after it. This chapter sets out from the plainest question there is, “how should memory be modelled”, and walks step by step to Equations 1 through 6 of the paper. Read slowly, and do the questions at the end of every section.

2.1 Starting point: the curse of attention, and an old approach that was given up

Chapter 0 covered the two costs of attention: compute grows with the square of the length, and the KV cache grows linearly with the length and never shrinks. At the scale of 1M tokens the second cost is especially deadly — you did that exercise, and one copy of the cache alone can eat thirty percent of the memory on a top-end graphics card.

But before attention was invented, sequence modelling used a different approach: the recurrent neural network (RNN). It reads left to right and compresses everything it has read into a state of fixed size.

The RNN's fixed state has one irreplaceable advantage: however long the sequence, the memory does not change. It was abandoned back then because of two fatal weaknesses:

• The memory gets diluted: the state is only so big, and by the time you reach word 1000, the information from word 1 has long been squeezed out by everything that came after.
• It cannot be parallelised: you have to finish step 999 before you can compute step 1000. And the entire power of a GPU rests on “computing many things at once”.

Linear attention: a family of attention variants that drop the softmax and can therefore be rewritten as “a fixed-size state + a recurrent update”. KDA belongs to this family; it continues the work of the Kimi Linear paper. You worked this out in the softmax section of Chapter 0: the exponential inside softmax forces every query to score against every key in person, so nothing can be aggregated in advance. Drop the softmax and aggregation becomes possible — that is the mathematical starting point of the whole linear attention family.

2.2 The state is a blackboard: how to “write” and “read”

Start by picturing KDA's state S as a blackboard. It is a matrix of shape d_k × d_v — Chapter 0 covered this: a matrix is just a table of numbers.

How do you write a “key → value” pair onto the blackboard? The answer is the outer product:

Chapter 0 mentioned that a column vector times a row vector gives a matrix. kv^⊤ is exactly such a matrix — you can read it as “along the direction that k points in, carve in the content v”.

How do you read from the blackboard? Multiply it by the query vector:

Why does that read anything out? One pass of algebra makes it clear. Suppose only one pair (k, v) has ever been written on the blackboard. Then:

Note that k^⊤q is a number (the dot product of two vectors, that is, their similarity). So what comes out is “v times a similarity”.

2.3 The delta rule: erase before you write

The naive write S ← S + kv^⊤ has an obvious flaw: if the same key is written twice, the two values pile on top of each other.

Suppose the key “capital” is first written with “Beijing”, and then the context changes and it has to become “Tokyo”. With the naive write, the blackboard now holds both “Beijing” and “Tokyo”, and what you read out is a blend of the two — mush. This is exactly the mathematical content of the RNN's “memory gets diluted”.

The idea of the delta rule is direct: before writing the new content, erase what is already on that key.

Everything hinges on that (I − β_tk_tk_t^⊤). It looks frightening, but the meaning is simple. Look at what happens when it acts on the old state.

2.4 Adding forgetting: the full form of Equation 1

The delta rule solves the problem of “the same key being overwritten again and again”. But there is one problem it cannot solve: how does old information that will never be used again fade out?

Say you are reading a long document: by chapter ten you basically no longer need the details from chapter one. The delta rule only erases when “the same key is rewritten”, so old content that is never rewritten occupies the blackboard forever.

The fix is to add a forget gate: at every step, fade the whole blackboard by some proportion. That gives Equation 1 of the paper:

2.5 Equation 2: where these q, k, v, α, β come from

Every quantity in Equation 1 has to be computed from the token vector x_t. Equation 2 of the paper gives the concrete recipe (h is the head index):

2.6 The chunkwise parallel form: getting something serial to run on a GPU

Now for the RNN's second fatal weakness: Equation 1 is strictly serial. S_t depends on S_t−1, so a million tokens means a million steps, and the GPU is of no use at all.

The fix is called the chunkwise parallel form, and the idea is an elegant compromise:

Writing this form down needs a piece of notation first. Equation 3 of the paper defines the cumulative decay:

With that, plus a trick called the UT transform, which produces U_[t] and W_[t], define the pseudo-value term Ṽ_[t] := U_[t] − W_[t]S_[t], and every output inside the chunk can be computed in parallel:

You are not required to be able to derive this equation yourself (the paper itself hands the full derivation off to the Kimi Linear paper), but you do need to read what it says:

2.7 Lower-bounded decay: how an overflow problem rewrote the maths

This section is the best part of the chapter, because it shows something the paper is rarely this clear about: a limit on hardware numerical precision can turn around and decide the mathematical form of the model.

The problem: that division blows up

Look back at K_[t] / Γ_[t]^1→C in equation 4 — the keys have to be divided by the cumulative decay.

And Γ is a chain of numbers in (0,1) multiplied together. What happens when you keep multiplying?

BF16: the number format used most often for training large models; each number takes 2 bytes. The largest value it can represent is about 3.4 × 10³⁸. Go past that and the number becomes “infinity”, and the computation is wrecked.

The paper's own wording: “because Γ is a product of retention factors in (0,1), this reciprocal can grow without bound and overflow at finite precision.”

How the previous generation solved it, and the knot it left behind

Kimi Linear (KDA's predecessor) used two tricks to keep this numerical range under control:

1. Compute the relative decay in log space — a running product becomes a running sum, which shrinks the dynamic range enormously.
2. Split each chunk again into sub-chunks of 16 tokens (tiles) — compute the relative decay only inside a sub-chunk, and handle the part that crosses sub-chunks separately.

After these two tricks, the off-diagonal sub-chunks can be computed directly as dense matrix multiplications on Tensor Cores. The sub-chunks on the diagonal cannot — in the paper's own words, diagonal sub-chunks “still require explicit position-pair computation, which remains the main bottleneck inside a chunk”.

Tensor Core: the hardware unit inside a modern GPU built specifically for matrix multiplication, several times faster than the general-purpose units. But it eats exactly one shape of work: “a standard dense matrix multiplication”. Any computation that needs “each position pair handled on its own” cannot use it and falls back to the much slower general path.

K3's solution: replace the decay function itself

Here is the key turn of thought. The earlier line of thinking was “the numbers blow up, so I will find a way to route around it in the computation”. K3's line of thinking is “the numbers blow up, so I will not let them blow up in the first place”.

Recall the loose end left in section 2.5: the map from the decay logit z to the retention factor α was postponed. Here is that map:

Compare the previous generation. Kimi Linear (and GDN, Mamba-2) used a negative softplus map:

The difference is one thing only: the range. The previous generation had (−∞, 0), K3 has (−5, 0). The previous generation had no lower bound, K3 has one. That is where the name “lower-bounded decay” comes from.

Why −5 is exactly the right number

The chain of reasoning the paper gives is very clean, and worth walking through end to end:

2.8 The full-rank output gate: the last valve

The ō_t that equation 1 computes is not yet the final output of this layer. KDA puts one more gate behind it:

第3章 Gated MLA 与混合注意力：两种记忆的分工

一句话导语：K3 不把「注意力」当成一种东西，而是让两种机制分工——KDA 负责顺着句子往下记、天然带先后顺序，Gated MLA 负责回头在全篇里随便翻、不受位置约束。这一章讲清它们按什么比例混、各自补什么、以及为什么 MLA 层可以干脆不要位置编码。

3.1 先看地图：K3 把信息流往三个方向撑开

先把最基础的事情说清楚。一个语言模型拿到一段话，会先把它切成一个个 token(token，可以粗略理解成「词或词的一部分」)，每个 token 变成一串数字。K3 里这串数字有 7,168 个（论文表 1 的 Hidden Dimension）。之后模型做的事情，翻来覆去就一件：让这些数字互相看一看、混一混。所谓「设计架构」，说白了就是决定谁跟谁混、怎么混、混几次。

论文 §2 开篇给的框架是：混合可以发生在三个互不相同的方向上，K3 沿这三个方向分别做了扩展。

第一个方向是序列长度，即 token 之间怎么混。K3 的做法叫混合注意力(Hybrid Attention)：每个 block 里放 3 个 KDA 层加 1 个 Gated MLA 层，论文说这样「为长上下文的 token 混合提供了高效机制，同时保留了有选择性的高容量注意力」。第二个方向是网络深度，即层与层之间怎么混：注意力残差(Attention Residuals, AttnRes) 让每个模块能从词嵌入、当前块、以及之前的块里「挑着取」，而不是只能沿一条残差链均匀累加——这是第4章的内容。第三个方向是模型宽度，即同一个 token 内部 7,168 个通道之间怎么混：每个注意力层后面跟一个 Stable LatentMoE 层做稀疏的通道混合，每个 token 只激活 896 个路由专家里的 16 个。

3.2 三个 KDA 配一个 Gated MLA：比例，和末尾补的那一层

论文 §2.1 给的分工写得很明确：KDA 层提供位置敏感、近因感知(position-sensitive and recency-aware)的混合，MLA 层提供不受限的全局内容交互(unrestricted global content interaction)。翻成人话：KDA 管「最近发生了什么，按顺序记着」；MLA 管「整篇文章里跟我现在这个词最相关的是哪几个，不管它在多远」。

混合注意力（Hybrid Attention）：K3 的做法是逐层混合(layerwise hybrid)——不是在同一层里把两种机制拼起来，而是整层整层地交替。每个 block 里放 3 个 KDA 层，后面跟 1 个 Gated MLA 层，比例 3:1，这个模式在整个骨干里重复。此外，骨干末尾额外放一个 Gated MLA 层，确保最后一层总是执行全局注意力。

3.3 MLA：把一个 token 的键值压成一个小向量

多头潜注意力（Multi-head Latent Attention, MLA）：由 DeepSeek-V2 提出。它的想法是：既然缓存太大，那就别缓存 K 和 V 本身，改成缓存一个能重建出它们的小东西。具体做法是把每个 token 的键值表示压进一个低维潜向量(latent vector)：

推理时只缓存 c_t。真正要算注意力的时候，再用学到的上投影(learned up-projections)把 c_t 重建成各个头的内容键和值。论文的原话是：这个分解「在保留全局 token 到 token 注意力的同时，减少了 KV 缓存的足迹」。Kimi K2 和 K2.5 都用了 MLA，K3 把它保留在周期性出现的全局注意力层里。

代价出在哪？至少三处。第一是计算换存储：每次算注意力都要多做一次上投影，把小向量重新撑回大矩阵，省下的显存是用额外乘法换的，论文没讨论这部分开销有多大。第二是表达力受限：所有头的键和值都必须从同一个低维 c_t 里长出来，压得越狠，不同头拿到的信息越接近；论文没有量化这一损失。第三是复杂度没变：注意力「每个 token 看所有前面 token」的平方级计算量，MLA 一点没减少。

3.4 NoPE：位置这件事，交给谁管

无位置编码（No Position Encoding, NoPE）：和 Kimi K2、K2.5 不同，K3 沿用 Kimi Linear 的混合设计，对所有 MLA 层采用 NoPE——它们的查询和键上不施加任何显式位置编码。

敢这么干的理由，论文写得很直白：夹在中间的 KDA 层本身就是位置敏感、近因感知的（它按顺序一步步更新状态，把顺序换掉结果就变了），而 MLA 层负责提供不受限的全局内容交互。也就是说，位置信息在进入 MLA 层之前，已经被 KDA 层写进隐藏向量里了；MLA 层不需要再被单独告知一遍，它可以专心做「按内容找相关」这件事。

还有第二个好处，纯工程但很实在：论文指出，这种分工避免了在扩展上下文长度时去改位置编码的参数，比如重调 RoPE 的频率基，或者套用 YaRN 这类插值方法。K3 的训练上下文从 K2 的 128K 一路推到 1M（表 1，8 倍），如果位置编码是显式的，每一次扩长都要重新调参、重新验证；NoPE 直接把这个环节从流程里删掉了。

3.5 输出门：让每个 token 自己决定听多少（式 7）

K3 给 MLA 加了一个输入相关、逐通道、满秩的输出门。记 ō_t 为位置 t 上未经门控的 MLA 输出，门控后的输出是：

唯一需要额外解释的词是满秩。W_g 把 7,168 维的 x_t 映射成一个同样长的门向量。如果这个映射是低秩的——比如先压到 32 维再撑回去——那么无论输入是什么，所有可能的门向量都被困在一个 32 维子空间里，很多通道只能被绑在一起开关。满秩则意味着门向量原则上可以指向 7,168 维空间里的任何方向，每条通道可以独立开关。论文的说法是：这个门让每个 token 去调制它从全局注意力读到的通道；W_g 满秩这一点与 K3 里 KDA 的新参数化保持一致。

3.6 把注意力输出留在 FP32，以及被逼出来的内核重设计

K3 的处理是：训练时把注意力输出保持在 FP32，采用文献 [98] 的方法纠正这个有偏舍入误差。这听起来只是「多用点精度」，但它在 GPU 上引发了一连串连锁反应，论文把这条链完整写了出来，值得跟着走一遍。

tile（分块）与片上占用：GPU 算大矩阵时不会一次搬进整个矩阵，而是切成小块（tile），一块块搬进共享内存(shared memory)里算。共享内存是紧挨着计算单元的一小块存储，快得多也小得多，寸土寸金。

把输出从 16 位改成 32 位，意味着输出 tile 的片上占用直接翻倍。共享内存就那么大，有东西变胖就得有东西让位。论文的解法是重新设计训练内核：让变胖的输出 tile 去和 KV 暂存缓冲区(KV staging buffers，从显存往片上搬 K、V 时的中转区)重叠，而不是像原来那样和 query tile 重叠。腾出来的共享内存用于更深的 KV 流水线——同时在途的 K、V 块更多，搬运和计算叠得更好，计算单元等待更少。论文的结论是：这样反而换来更高的训练吞吐。

3.7 对账时间：69 + 24 = 93，可 69 ÷ 24 = 2.875

现在把论文表 1 里跟本章相关的几行拉出来，做一次真正的核对。这一节的目的不是记住数字，而是练一件事：拿到一张表，能不能从里面把架构反推出来。

Chapter 3 · Gated MLA and hybrid attention: a division of labour between two kinds of memory

In one sentence: K3 does not treat “attention” as one thing. It splits the work between two mechanisms — KDA remembers as it goes down the sentence and carries order naturally, Gated MLA looks back and flips through the whole text at will, free of any positional constraint. This chapter makes clear in what ratio they are mixed, what each of them supplies, and why the MLA layers can drop position encoding altogether.

3.1 The map first: K3 pushes information flow out in three directions

Start with the most basic thing. A language model takes a passage of text and first chops it into tokens(token, roughly “a word or a piece of a word”), and each token becomes a string of numbers. In K3 that string has 7,168 numbers in it (Hidden Dimension in Table 1 of the paper). After that, everything the model does is one thing over and over: let those numbers look at each other and mix. “Designing an architecture”, plainly put, is deciding who mixes with whom, how they mix, and how many times.

The framework the paper gives at the opening of §2 is this: mixing can happen along three distinct directions, and K3 extends along each of the three.

The first direction is sequence length, that is, how tokens mix with each other. K3’s approach is called hybrid attention(Hybrid Attention): each block holds 3 KDA layers plus 1 Gated MLA layer, which the paper says “provides an efficient mechanism for long-context token mixing while preserving selective, high-capacity attention”. The second direction is network depth, that is, how layers mix with layers: attention residuals(Attention Residuals, AttnRes) let every module “pick and choose” from the word embeddings, the current block and earlier blocks, instead of only accumulating uniformly along a single residual chain — that is the subject of Chapter 4. The third direction is model width, that is, how the 7,168 channels inside a single token mix: every attention layer is followed by a Stable LatentMoE layer doing sparse channel mixing, and each token activates only 16 of the 896 routed experts.

3.2 Three KDA to one Gated MLA: the ratio, and the extra layer at the end

The division of labour given in §2.1 of the paper is stated plainly: KDA layers provide position-sensitive, recency-aware(position-sensitive and recency-aware) mixing, and MLA layers provide unrestricted global content interaction(unrestricted global content interaction). In plain words: KDA handles “what happened recently, remembered in order”; MLA handles “which few things in the whole document are most relevant to the word I am on now, however far away they are”.

Hybrid attention: K3’s approach is a layerwise hybrid(layerwise hybrid) — not stitching the two mechanisms together inside one layer, but alternating whole layers at a time. Each block holds 3 KDA layers followed by 1 Gated MLA layer, a ratio of 3:1, and this pattern repeats throughout the backbone. On top of that, one extra Gated MLA layer is placed at the end of the backbone, making sure the last layer always performs global attention.

3.3 MLA: squeezing one token’s keys and values into a small vector

Multi-head latent attention: introduced by DeepSeek-V2. The idea is that since the cache is too large, do not cache K and V themselves — cache instead a small thing that can reconstruct them. Concretely, the key-value representation of each token is compressed into a low-dimensional latent vector(latent vector):

At inference time only c_t is cached. When the attention actually has to be computed, learned up-projections(learned up-projections) rebuild c_t into the content keys and values of each head. The paper’s own words: this factorisation “reduces the KV cache footprint while preserving global token-to-token attention”. Kimi K2 and K2.5 both used MLA, and K3 keeps it in the periodically occurring global attention layers.

Where does the cost fall? In at least three places. First, compute traded for storage: every attention computation has to do one more up-projection, pushing the small vector back out into a large matrix, so the memory saved is bought with extra multiplications, and the paper does not discuss how large that overhead is. Second, limited expressiveness: the keys and values of all heads have to grow out of the same low-dimensional c_t, so the harder the compression, the more alike the information different heads receive; the paper does not quantify this loss. Third, the complexity is unchanged: MLA does not reduce one bit of the quadratic compute of attention’s “every token looks at every token before it”.

3.4 NoPE: who is put in charge of position

No position encoding: unlike Kimi K2 and K2.5, K3 follows the hybrid design of Kimi Linear and uses NoPE for all MLA layers — no explicit position encoding of any kind is applied to their queries and keys.

The paper states the reason for daring to do this quite bluntly: the KDA layers sandwiched in between are themselves position-sensitive and recency-aware (they update the state step by step in order, so change the order and the result changes), while the MLA layers are responsible for supplying unrestricted global content interaction. In other words, the position information has already been written into the hidden vectors by the KDA layers before it reaches an MLA layer; the MLA layer does not need to be told again separately, and can concentrate on “finding what is relevant by content”.

There is a second benefit, purely engineering but very real: the paper points out that this division of labour avoids having to change position-encoding parameters when the context length is extended — retuning RoPE’s frequency base, say, or applying an interpolation method like YaRN. K3’s training context is pushed all the way from K2’s 128K to 1M (Table 1, 8×), and if the position encoding were explicit, every extension would mean retuning and revalidating; NoPE simply deletes that step from the pipeline.

3.5 The output gate: letting each token decide for itself how much to listen (Equation 7)

K3 adds an input-dependent, per-channel, full-rank output gate to MLA. Write ō_t for the MLA output at position t before gating; the gated output is:

The only word that needs extra explanation is full-rank. W_g maps the 7,168-dimensional x_t into a gate vector of the same length. If that map were low-rank — squeezed down to 32 dimensions first and then pushed back out, say — then whatever the input, every possible gate vector would be trapped in a 32-dimensional subspace, and many channels could only be switched on and off tied together. Full-rank means the gate vector can in principle point in any direction of the 7,168-dimensional space, so every channel can be switched independently. The paper puts it this way: this gate lets each token modulate the channels it reads from global attention; the fact that W_g is full-rank is consistent with the new parameterisation of KDA in K3.

3.6 Keeping the attention output in FP32, and the kernel redesign it forced

K3’s handling: during training it keeps the attention output in FP32, using the method of reference [98] to correct this biased rounding error. This sounds like nothing more than “spend a bit more precision”, but on a GPU it sets off a chain of consequences, and the paper writes the whole chain out — it is worth walking through.

tile and on-chip occupancy: when a GPU computes a large matrix it does not move the whole matrix in at once, it cuts it into small blocks (tiles) and moves them in one at a time to compute in shared memory(shared memory). Shared memory is a small piece of storage right next to the compute units, far faster and far smaller, and every inch of it is precious.

Going from 16 bits to 32 bits means the on-chip occupancy of the output tile doubles outright. Shared memory is only so big; if something gets fatter, something else has to give way. The paper’s solution is to redesign the training kernel: it is now with the KV staging buffers(KV staging buffers, the transit area used when moving K and V from GPU memory onto the chip) that the fattened output tile is made to overlap, instead of with the query tile as before. The shared memory freed up goes to a deeper KV pipeline — more K and V blocks in flight at once, transfer and compute overlapping better, compute units waiting less. The paper’s conclusion: this buys higher training throughput instead.

3.7 Reconciliation time: 69 + 24 = 93, but 69 ÷ 24 = 2.875

Now pull out the rows of Table 1 in the paper that bear on this chapter and do a real check. The point of this section is not to memorise numbers, it is to practise one thing: given a table, can you reconstruct the architecture from it.

第4章 Attention Residuals：让「深度」也用上注意力

一句话导语：Transformer 当年用注意力取代了 RNN 沿时间的顺序传递，AttnRes 把同一套方法论搬到深度上——让第 40 层不必顺着残差链一路继承，而是像挑选 token 一样，从所有先前层里挑着取。这一章讲清这个类比为什么成立、在哪失效，以及为什么它最后必须被「块化」才养得起。

4.1 灵魂所在：把「时间上的老问题」搬到「深度」上

这个类比是本章的灵魂。RNN(Recurrent Neural Network，循环神经网络)处理句子是从左到右一个词一个词读，每读一个就更新一次隐藏状态；到第 100 个词时，前面 99 个词的信息全都挤在那一个状态里，想调用第 3 个词的细节只能希望它还没被冲掉。序列建模领域怎么解决的？Transformer 用注意力取代了循环：每个位置不再只能看「上一步传下来的状态」，而是用数据相关的权重直接访问所有先前位置，需要谁就给谁高权重。

把这段话里的「位置」全部换成「层」，就是 AttnRes 的全部想法。论文写得非常直接：AttnRes 把同样的方法论用到深度上——每一层从所有先前层里有选择地检索表示，而不是均匀地累加它们。

4.2 全注意力残差：伪查询、键值和 softmax 核（式 8、式 9）

要把注意力搬到深度上，得先回答三个问题：谁来当查询？谁来当键和值？权重怎么算？论文逐个给了答案。

伪查询（pseudo-query）：对每一层 l，定义一个该层专属的、可学习的查询向量 q_l = w_l ∈ ℝ^d。请特别注意「pseudo（伪）」这个字的含义：在普通注意力里，查询是由当前输入算出来的（q = W_qx）；而这里的 q_l 不依赖任何输入，它就是这一层自己的一组参数，训练完之后固定不变。

键和值则来自各层的输出，两者取同一个东西：

i = 0 这一项要单独拿出来说：它把词嵌入也放进了可检索的源里。这意味着无论网络多深，任何一层都有一条直达最原始输入的通道，永远不必担心原始信息在中途被磨没了。这个细节在下一节的块化里会被专门保留。

接下来是权重。论文用的是 softmax 核 φ(q, k) = exp(q^⊤ RMSNorm(k))，其中 RMSNorm 的作用是防止那些输出幅值大的层主导权重。然后归一化：