Self-Attention: Queries, Keys and Values

The last chapter gave the intuition: each token asks a question, compares it against what the others offer, and pulls back a blend of their contents. Now we make "question", "offer" and "contents" concrete. They become three vectors per token — the query, the key, and the value — each produced by its own learned matrix.

First, why not just compare the embeddings?

Before introducing three matrices, it is worth seeing why the obvious cheap idea fails — because that failure is the whole reason the matrices exist.

The simplest way to score how related two tokens are is to take the dot product of their embeddings: large when the vectors point the same way, small when they do not. So to decide how much a token should attend to another, just dot their embedding vectors and be done. Why not?

Consider the sentence "the dog chased the ball but could not catch it." The final word, it, refers to the ball, not the dog — so when the model processes it, it should attend mostly to ball. But the embedding of it is a fixed vector, and so are the embeddings of dog and ball. Dotting it with dog and it with ball gives two numbers that reflect only the generic, context-free similarity of those words — and there is no reason that would single out ball. The plain dot product has no way to express "in this sentence, as a thing that gets caught, the ball is the relevant one." It measures similarity, but the relationship we need is subtler than similarity, and it changes with context.

When we cannot write down the rule for a relationship, deep learning offers a standard move: don't hand-design it — add trainable weights and let training discover it. That is precisely what the query, key and value matrices are. Rather than comparing the raw embeddings, we first pass each token through learned projections that are free to pull out just the aspects that matter for attending, and compare those instead. The matrices start random and are shaped by gradient descent until the comparison lands where it should — on ball rather than dog.

Three projections of the same vector

Start from a token's vector $h$ (its current entry in the residual stream). We compute three new vectors from it, using three learned weight matrices $W_Q$, $W_K$, $W_V$:

• The query $q$ encodes what this token is looking for.
• The key $k$ encodes what this token offers to others — how it advertises itself.
• The value $v$ encodes the content this token will hand over if attended to.

"Self"-attention means all three come from the same sequence: every token produces a query, a key, and a value, and the tokens attend to each other. (Later, in cross-attention, queries come from one sequence and keys/values from another — but the machinery is identical.)

Step 1 — Score every pair with a dot product

How well does token $i$'s query match token $j$'s key? Use the dot product, which is large when two vectors point the same way:

For a sequence of $n$ tokens this gives an $n \times n$ table of scores: row $i$ holds how much token $i$'s query matches every token's key. This is the "compare my question to every offer" step, done for all tokens at once.

Step 2 — Turn scores into weights with softmax

The scores are raw numbers. To blend, we need weights that are positive and sum to 1, so we apply softmax across each row:

Now $\alpha_{ij}$ is the attention weight: the fraction of its attention that token $i$ pays to token $j$. Each row sums to 1 — token $i$ distributes a total of 100% of its attention across all tokens, most of it on the keys its query matched best.

Step 3 — Blend the values

Finally, token $i$'s output is the weighted sum of values, using those weights:

A token it attends to strongly contributes most of its value; a token it ignores contributes almost nothing. This blended vector — token $i$, enriched with exactly the context its query sought — is added back into the residual stream.

The whole thing in one line

Stacking the queries, keys and values for all tokens into matrices $Q$, $K$, $V$ (one row per token), the three steps collapse to the compact formula at the heart of the transformer:

Read it right to left in words: score every query against every key ($QK^{\top}$), normalize each row into weights (softmax), blend the values (multiply by $V$). (We will add one small scaling factor inside the softmax in the next chapter — it matters for stability but not for the idea.)

Why three separate roles

A natural question: why not just use the token's vector directly for all three? Because the same token plays different parts depending on the direction of the interaction. What a token looks for (query) is generally different from what it advertises (key), which is different again from the content it contributes (value). Giving each role its own learned matrix lets the model tune them independently — and, crucially, $W_Q$, $W_K$, $W_V$ are trained by the ordinary gradient descent and backpropagation you already know; attention adds no new training machinery, only a new wiring of matrix multiplies and a softmax.

The next chapter explains the one missing detail — why we scale the scores before the softmax — and the chapter after adds the rule that a token may only look at the tokens before it.