The architecture that reads in order. A hidden state loops back on itself, carrying the past forward so the network can finally understand sequences — language, audio, time.
In 3.2 the CNN gained a sense of space. But it ended on a limitation: the world is full of sequences — sentences, speech, stock prices, sensor streams — where meaning depends on order and on what came before. A CNN's fixed window has no memory of "earlier," and no natural way to handle inputs that vary in length. Read "dog bites man" and "man bites dog" through a feed-forward net and it can't tell that order flipped the meaning.
The Recurrent Neural Network (RNN) solves this with one elegant move: a loop. It processes a sequence one element at a time, and at each step it keeps a hidden state — a running summary of everything seen so far. That hidden state is fed back into the network alongside the next input, so the past literally flows forward into the present. The same small set of weights is reused at every time step, which is why an RNN can read a sentence of any length.
Picture yourself reading a mystery novel. You don't re-read the whole book at every sentence — you carry a running mental summary: who the suspects are, what's happened, what feels important. Each new sentence updates that summary. By the final page, your understanding of "the butler did it" rests entirely on a memory built up across hundreds of pages, even though you're only looking at one sentence right now.
That running mental summary is the hidden state. Each new word is the input at this time step. The act of updating your understanding as you read is the recurrence. And crucially, you use the same reading brain for every sentence — you don't grow a new brain per word. That's weight sharing across time. The catch, which we'll meet at the end: details from chapter one start to blur by chapter twenty. Memory fades.
At each time step t, the RNN takes two things: the current input x_t (this word) and the previous hidden state h_(t-1) (the summary so far). It combines them, squashes the result through an activation — almost always tanh, because it keeps the state bounded and centered (recall from Chapter 2 why that matters for stability) — and produces the new hidden state h_t. That new state then becomes the input memory for the next step. Optionally, an output y_t is read off the hidden state at any step.
The defining feature is that h_t depends on h_(t-1), which depended on h_(t-2), all the way back to the start. So the hidden state at the end is, in principle, a function of every input that came before it. One loop, unrolled across time, gives the network memory.
At each time step t: h_t = tanh( W_xh · x_t + W_hh · h_(t-1) + b ) y_t = W_hy · h_t (optional output) Three weight matrices, reused at EVERY step: W_xh : input → hidden (reads the new element) W_hh : hidden → hidden (carries the memory forward) W_hy : hidden → output (reads off a prediction) h_0 usually starts as all zeros (a blank memory).
A tiny RNN with a single hidden unit reads a 2-element sequence. Watch how step 2's hidden state depends on step 1's.
Weights: W_xh = 0.8ic W_hh = 0.5ic b = 0
Sequence: x = [1.0, 0.5] h_0 = 0
STEP 1 (x_1 = 1.0)
h_1 = tanh(0.8×1.0 + 0.5×0 + 0)
= tanh(0.8) = 0.664
STEP 2 (x_2 = 0.5)
h_2 = tanh(0.8×0.5 + 0.5×0.664 + 0)
= tanh(0.4 + 0.332)
= tanh(0.732) = 0.625
↑
this term carries step 1's memory forward
0.5×0.664 term injects the memory of step 1. If x_1 had been different, h_1 would differ, and h_2 would shift even though x_2 stayed the same. That dependency on the past is memory. Run a longer sequence live below.
Pick a sequence, then step through it one time step at a time. Each cell is the network at one moment: the input x_t enters, combines with the hidden state arriving from the left, and produces a new hidden state that passes to the right. The hidden state's color shows its value. Watch how the influence of the first input gradually dilutes as it passes through step after step — the seed of the problem 3.4 will solve.
The gold tint tracks how much of the very first input still survives in the hidden state. On the short sequence it stays visible; on the long one it fades toward nothing well before the end. The network technically "remembers" everything, but in practice the early signal gets overwritten step after step. That gap between in principle and in practice is the whole story of the next two lessons.
An RNN is often drawn as a single cell with an arrow looping back on itself. To actually compute or train it, we unroll it: lay out one copy of the cell per time step, in a row, with the hidden state flowing left to right. That's the picture in the demo. It's still one network with one set of weights — unrolling is just the loop stretched out in time so we can see (and backpropagate through) every step. Training this way has a name: backpropagation through time (BPTT).
Because inputs and outputs can each be single or sequential, RNNs flex to many task shapes: many-to-one (read a whole review, output one sentiment), one-to-many (one image in, a caption sequence out), many-to-many aligned (tag every word with its part of speech), and sequence-to-sequence (read a full English sentence, then emit a French one — the encoder-decoder setup we'll return to in 3.7). This flexibility is why RNNs dominated language and speech for years.
Here's the crack. To carry memory forward, the RNN multiplies by the same hidden-to-hidden matrix W_hh at every step. Reach back twenty steps and that influence has been multiplied through twenty times. If those multiplications shrink the signal (values below 1), the early information decays toward zero — the vanishing gradient problem, now stretched across time. If they grow it (values above 1), it explodes instead. Either way, learning long-range dependencies becomes very hard.
Consider the sentence: "I grew up in France... [fifty words later] ...so I speak fluent French." To predict "French," the network must recall "France" from fifty steps back. A plain RNN's memory of that word has usually faded to nothing by then. It handles short-range context well and long-range context poorly — a serious problem, because language is full of long-range dependencies.
The fix isn't to abandon recurrence — it's to give the cell a controlled memory: gates that decide what to keep, what to forget, and what to add at each step, so important information can flow across many steps untouched. That's the LSTM (3.4), and its leaner cousin the GRU (3.5).
A recurrent network can, in theory, remember everything it has read. In practice, it remembers the recent past vividly and the distant past barely at all. Closing that gap is what the next two architectures are for. On the promise and the limit of recurrence
| Property | Value | Notes |
|---|---|---|
| Core idea | Hidden state loop | A running summary fed forward through time |
| Processing | One step at a time | Sequential — each step depends on the previous |
| Weights | Shared across time | Same W_xh, W_hh, W_hy at every step → any-length input |
| Activation | tanh (typically) | Bounded, zero-centered — keeps the state stable |
| Training | BPTT | Backpropagation through the unrolled sequence |
| Strength | Short-range sequence | Order, variable length, recent context |
| Blind spot | Long-range memory | Vanishing/exploding gradients over time — motivates LSTM/GRU |
| Bonus weakness | Can't parallelize | Sequential by nature — slow on long sequences (key for 3.6) |
The RNN gives a network memory by looping a hidden state forward through time, reusing one set of weights at every step so it can read sequences of any length. It unlocked language, speech, and time-series modeling, and flexes to many task shapes from sentiment to translation. But its memory fades: multiplying by the same matrix at every step makes long-range dependencies vanish or explode, and its step-by-step nature can't be parallelized. The next lessons fix the memory with gated cells — LSTM and GRU — before 3.6 and 3.7 reveal why even those weren't enough, and what replaced them.