Gating fixed memory. It could not fix speed. Why every recurrent network — RNN, LSTM, GRU alike — is capped by the one thing they all share: they must process a sequence one step at a time.
We've spent four lessons building recurrence up — adding memory in 3.3, protecting it with gates in 3.4, streamlining it in 3.5. This lesson is the turn. It steps back and asks: for all their cleverness, what is it that the entire recurrent family cannot do, no matter how good the gating gets? The answer set the research agenda that produced the Transformer, and it's worth understanding precisely, because it explains why attention — the subject of 3.7 — wasn't just an improvement but a replacement.
There are two big limitations, and they're different in kind. One is about memory (the long-range decay we've been chipping away at). The other is about speed — and it's the one gating can't touch at all. That second limitation is the real reason recurrence fell.
Picture an old fire-fighting bucket brigade: a line of people passing one bucket hand to hand from the well to the fire. Person 5 cannot do anything until person 4 hands them the bucket. It doesn't matter if you have a thousand volunteers standing by — the water still moves at the pace of a single chain, one handoff at a time. Add more people to the line and it gets longer, not faster.
That's a recurrent network. Each hidden state must wait for the previous one, so the computation is a chain of handoffs that can't be sped up by throwing more hardware at it. Now picture the alternative: a crowd where everyone can act at once, each person free to look at whoever they need to. If you have a thousand hands available, a thousand things happen simultaneously. That crowd is what attention enables — and the gap between the brigade and the crowd is the entire motivation for what comes next.
Each hidden state depends on the one before it: h_t needs h_(t-1). So the steps must be computed in order, one after another. Modern hardware — GPUs and TPUs — is built to do thousands of operations simultaneously, but a recurrent network can't use that power across the time dimension. A 1,000-token sequence takes 1,000 sequential steps, full stop. This caps both training speed and inference speed, and it's the limitation gating does nothing to fix. This is the decisive one.
LSTMs and GRUs dramatically reduced the vanishing-memory problem, but they didn't eliminate it. Information from a token still has to travel step-by-step through every intermediate state to reach a distant one. Across hundreds or thousands of steps, even a gated cell's signal can weaken, and the path length between two related tokens grows with the distance between them. The further apart two words are, the harder it is to relate them.
An RNN compresses everything it has read into a single fixed-size hidden state vector. For a short sentence that's fine; for a long document, the whole past must be squeezed into the same small vector, and detail is inevitably lost. In sequence-to-sequence tasks like translation, forcing an entire source sentence through one fixed vector — the context bottleneck — became a notorious failure point, and the direct trigger for the first attention mechanisms.
Both lanes process the same sequence. The top lane is a recurrent network: each cell can only light up after the one before it, so a clock ticks step by step. The bottom lane represents the parallel approach attention enables: every cell processes at once. Run them and watch the recurrent lane's time grow with the sequence length while the parallel lane finishes in essentially one tick. Then try a longer sequence and watch the gap widen.
The recurrent lane always takes as many time-steps as there are tokens — double the sequence, double the wait. The parallel lane finishes in a single step regardless of length, because nothing waits on anything else. On a real 1,000-token document that difference is the gap between feasible and painfully slow, and it's why, once a parallel alternative existed, the field switched almost overnight.
The crux is a property called path length — how many computational steps a signal must travel to get from one position to another. It determines both how fast the model runs and how easily distant tokens can influence each other.
For a sequence of length N:
RECURRENT (RNN / LSTM / GRU)
Sequential operations ......... O(N) ← N steps in order
Max path between two tokens ... O(N) ← signal walks the chain
Parallelizable across time? ... No
ATTENTION (the 3.7 alternative)
Sequential operations ......... O(1) ← all positions at once
Max path between two tokens ... O(1) ← every token sees every
other directly
Parallelizable across time? ... Yes
Put the three limitations together and they describe, in negative space, exactly what a better architecture would need. It would have to let any token interact with any other directly — no walking the chain (fixing the path-length and long-range problems). It would have to process all positions in parallel — no waiting on the previous step (fixing the speed problem). And it would have to avoid cramming everything through one fixed vector — letting the model look back at all previous states, not just a compressed summary (fixing the bottleneck).
There is a single mechanism that does all three: attention. Instead of passing memory hand-to-hand down a chain, attention lets every position look directly at every other position and decide, on the fly, which ones matter — computed for all positions simultaneously. Direct connections, full parallelism, no fixed bottleneck.
Attention first appeared as an add-on to RNNs (to relieve the translation bottleneck of limitation 3). Then in 2017 a paper titled "Attention Is All You Need" made the radical claim in its title literal: drop the recurrence entirely and keep only the attention. That architecture is the Transformer, and it's where Chapter 3 has been heading all along.
Recurrence solved how to remember. It never solved how to do so quickly, or how to let distant things connect directly. Attention answered both at once — and that is why it didn't refine the RNN, it retired it. On why the field changed course
| Limitation | Cause | Consequence | Fixed by attention? |
|---|---|---|---|
| No parallelism | h_t depends on h_(t-1) | Slow training & inference; can't use parallel hardware | Yes — O(1) sequential |
| Long-range decay | Signal walks the chain | Distant tokens hard to relate; path length O(N) | Yes — O(1) path |
| Context bottleneck | Fixed-size hidden vector | Long inputs lose detail; seq2seq failures | Yes — attend to all states |
| Memory (residual) | Even gates aren't perfect | Very long sequences still degrade | Yes — direct connections |
Recurrence has a ceiling that gating cannot raise. Because each step depends on the last, RNNs, LSTMs, and GRUs all process sequences one step at a time — unable to exploit parallel hardware, forcing signals to walk a long chain between distant tokens, and squeezing everything through a single fixed-size memory. Gating eased the memory problem but left speed and path length untouched. The fix had to come from a different direction entirely: a mechanism letting every token connect to every other directly and in parallel. That mechanism is attention, and the next lesson builds the intuition for it — the idea that replaced recurrence and launched the era of the Transformer.