The idea that dissolved every limitation of recurrence at once — letting each word look directly at every other word and decide, on the fly, which ones matter. The finale of Chapter 3, and the doorway to the Transformer.
In 3.6 we laid out exactly what recurrence couldn't do: process in parallel, connect distant tokens directly, or avoid squeezing everything through one fixed-size memory. We even sketched, in negative space, what a fix would have to look like. This lesson introduces that fix. It's a single mechanism — attention — and understanding it is the bridge from everything you've learned so far into Chapter 4, where the Transformer is built entirely out of it.
The motivating failure was translation. A sequence-to-sequence RNN read an entire source sentence, crushed it into one fixed vector, and asked the decoder to generate the translation from that single summary. For short sentences it worked; for long ones, the context bottleneck lost detail badly — the model would forget the beginning of a sentence by the time it reached the end. In 2014, Bahdanau and colleagues asked: instead of forcing everything through one vector, what if the decoder could look back at every source word, and focus on the relevant ones for each word it generates? That "looking back and focusing" is attention.
A plain RNN translator is like a student who reads a paragraph once, closes the book, and must answer every question from memory alone. Everything rides on what they managed to retain in a single mental summary — and for a long paragraph, the early details are gone. That's the fixed-context bottleneck.
Attention turns it into an open-book exam. For each question, the student can glance back at the whole text and focus on the relevant sentences — heavily weighting the parts that matter, ignoring the rest. They don't need to have memorized everything, because they can look. And crucially, every question gets its own custom focus: translating "bank" looks back at "river" if it's there, or "money" if it isn't. Attention is the network deciding, for each word it produces, where to look and how hard.
Strip attention to its essence and it's three steps. First, score: for the word you're focused on, compute a relevance score against every other word — how much should this one pay attention to that one? Second, normalize: pass those scores through a softmax (recall Chapter 2 — softmax turns a vector of numbers into weights that sum to 1) so they become a clean distribution of "focus." Third, blend: take a weighted sum of all the words' values using those weights, producing a new representation that mixes in the most relevant context.
The profound part is what this buys you. Every word connects to every other word directly — a path length of one, no walking the chain (limitation 2, gone). All words are scored simultaneously — fully parallel, no waiting on the previous step (limitation 1, gone). And nothing is forced through a single fixed vector — the model attends to all positions (limitation 3, gone). One mechanism, all three walls of 3.6 dissolved at once. That's why it didn't improve recurrence — it replaced it.
Click any word in the sentence. Lines fan out to every other word, with thickness and darkness showing how much attention that word pays to each one. The bars below give the exact weights — and they always sum to 1, because softmax makes them a distribution of focus.
The classic example: click "it" and watch attention flow to "animal" — the word it refers to. The network learns to resolve what "it" means by attending to the right earlier word, across any distance, in one direct hop.
This is the whole idea made visible. "it" reaches back to "animal" directly — no chain of hidden states to walk, no decay over distance. Change the sentence and "bank" leans toward "river" or "money" depending on context. The weights are learned, but the mechanism is exactly this: score every word, soften into focus, blend. Chapter 4 will show these weights come from learned Query, Key, and Value projections — previewed just below.
One word attends to three others. We score, softmax into weights, then blend their values.
Focus word relevance scores against 3 words:
score(river) = 4.0
score(the) = 1.0
score(money) = 0.5
STEP 1 — softmax the scores into focus weights:
e^4.0=54.6, e^1.0=2.72, e^0.5=1.65 (sum=58.9)
w(river) = 54.6 / 58.9 = 0.927
w(the) = 2.72 / 58.9 = 0.046
w(money) = 1.65 / 58.9 = 0.028
(weights sum to 1.0 — a distribution of focus)
STEP 2 — blend the words' value vectors by weight:
output = 0.927·V(river) + 0.046·V(the) + 0.028·V(money)
≈ mostly "river", a touch of the rest
Result: the focus word's new representation is dominated
by "river" — it attended where it mattered.
One question remains: where do the relevance scores come from? This is the part Chapter 4 builds in full, but here's the intuition so the transition is smooth. Each word is projected into three learned vectors:
Each word becomes three learned vectors:
Query (Q) — "what am I looking for?"
Key (K) — "what do I offer to others?"
Value (V) — "what do I actually contribute?"
Score = how well a Query matches each Key (a dot product).
Then soften and blend the Values:
Attention(Q,K,V) = softmax( Q·Kᵀ / √d ) · V
└── the focus weights ──┘ └ blend ┘
Attention first arrived as an add-on to RNNs in 2014, relieving the translation bottleneck while recurrence still did the heavy lifting. Then in 2017, the paper "Attention Is All You Need" took the radical step its title promised: remove the recurrence entirely and build the whole network from attention. With recurrence gone, every position could finally be processed in parallel, and the Transformer was born — the architecture behind essentially every large language model since.
That is the doorway out of Chapter 3 and into Chapter 4.
Recurrence asked the network to remember the past. Attention let it simply look at the past — all of it, directly, and all at once. That shift from remembering to looking is the single most consequential idea in modern AI. On the move that defined the era
Step back and see the whole journey. The MLP (3.1) gave us depth and non-linearity, but was blind to structure. The CNN (3.2) added a sense of space for images. The RNN (3.3) added memory for sequences, and the LSTM (3.4) and GRU (3.5) protected that memory across long distances. Then 3.6 revealed the ceiling no amount of gating could lift — sequential computation and chain-length path — and attention (3.7) dissolved it by letting every position connect to every other directly and in parallel.
You now hold the complete intuition for attention: score, soften, blend, with Query/Key/Value as the source of the scores. Chapter 4 builds the full Transformer on top of it — multi-head attention, positional encoding, the encoder–decoder stack, and how these assemble into the models powering modern AI. Everything from here forward stands on the foundation you just finished building.
| Property | Value | Notes |
|---|---|---|
| Core operation | Score → softmax → blend | Weighted focus over all positions |
| Path length | O(1) | Every token reaches every other directly |
| Parallelism | Full | All positions scored at once — no sequential wait |
| Context | All positions | No fixed-size bottleneck — attends to everything |
| Source of scores | Query · Key | Learned projections; softmax(QKᵀ/√d)·V |
| First use | RNN add-on (2014) | Relieved the seq2seq translation bottleneck |
| The leap | Transformer (2017) | "Attention Is All You Need" — recurrence removed |
| Legacy | Powers modern LLMs | GPT, Claude, Gemini, Llama — all attention-based |
Attention is the idea that let a network look at everything and weigh what matters, instead of remembering a compressed summary. It scores every word against every other, softens those scores into a distribution of focus, and blends the values accordingly — giving direct connections, full parallelism, and no fixed bottleneck, all at once. It dissolved every limitation of recurrence that 3.6 identified, first as an add-on to RNNs and then, in 2017, as a complete replacement for them. With the Query/Key/Value intuition in hand, you've reached the end of Chapter 3 and the threshold of the Transformer — the architecture Chapter 4 builds, and the engine of every modern large language model.