A recurrent cell with gates — small valves that decide what to forget, what to store, and what to reveal — and a protected cell state that lets information travel across hundreds of steps untouched.
In 3.3 we watched a plain RNN's memory fade. Because it multiplies by the same matrix at every step, the trace of an early input decays toward zero long before a long sequence ends. You saw it literally on the demo: the gold tint of input #1 dissolving by step twelve. The network could carry memory in principle, but not in practice.
The Long Short-Term Memory (LSTM) network, introduced by Hochreiter and Schmidhuber in 1997, fixes this with two ideas. First, a separate cell state — a memory conveyor belt that runs straight through every step with only minor, controlled edits, so information can flow across long distances without being repeatedly squashed. Second, a set of gates: tiny learned valves that decide, at each step, what to erase from memory, what to write into it, and what to expose to the rest of the network. Memory stops being a side effect of the math and becomes something the network actively manages.
Imagine you're tracking a long meeting on a whiteboard. You don't rewrite the whole board every minute. Instead you do three small things as new points come up: you erase what's no longer relevant (the topic that just got resolved), you add the genuinely new decisions worth keeping, and when someone asks a question, you read off just the part of the board that's relevant to answer them. The board itself persists across the whole meeting — most of what you wrote an hour ago is still there, untouched.
That whiteboard is the cell state — long-lived memory that survives across many steps. The eraser is the forget gate. Adding new notes is the input gate. Reading off the relevant part to answer is the output gate. A plain RNN had to smear its entire memory through a tanh at every step, like rewriting the whole board every minute until the early notes turned to mush. The LSTM just makes small, deliberate edits — so "France" from fifty words ago is still legible when you finally need to say "French."
Every gate is itself a tiny neural layer that outputs numbers between 0 and 1 — produced by a sigmoid (recall from Chapter 2: sigmoid squashes to 0…1, perfect for a valve). A gate value near 0 means "block this," near 1 means "let it through." Multiplying the memory by these gate values is how the LSTM controls flow.
Looks at the new input and the previous output, and decides what fraction of each item in the cell state to keep. 0 = erase it, 1 = keep it fully.
Decides what new information to write into the cell state. A sigmoid picks how much to write; a tanh proposes the candidate values to add.
Decides what part of the (now-updated) cell state to reveal as this step's hidden output — the value passed to the next step and any prediction.
The cell state (written C_t) is the conveyor belt these gates act on. Its update is beautifully simple: forget some of the old, add some of the new. That's it. Because the dominant operation on the cell state is addition rather than repeated multiplication-and-squashing, gradients can flow back across many steps without vanishing. This is the whole reason the LSTM remembers.
Gates (each a sigmoid over [previous output, new input]):
f_t = σ( W_f · [h_(t-1), x_t] + b_f ) forget
i_t = σ( W_i · [h_(t-1), x_t] + b_i ) input
o_t = σ( W_o · [h_(t-1), x_t] + b_o ) output
Candidate new memory:
C̃_t = tanh( W_c · [h_(t-1), x_t] + b_c )
Update the cell state (the conveyor belt):
C_t = f_t * C_(t-1) + i_t * C̃_t
└─ keep old ─┘ └─ add new ─┘
Produce the hidden output:
h_t = o_t * tanh( C_t )
One cell unit holds a memory value. We want it to preserve a stored fact through a step where nothing relevant happens, then update when something does.
Stored memory coming in: C_(t-1) = 0.90 ("France" is remembered)
STEP A — an irrelevant word arrives ("and")
Gates decide nothing important changed:
f = 0.95 (keep almost all of memory)
i = 0.05 (write almost nothing)
C̃ = 0.40 (some candidate, but barely used)
C_t = f×C_(t-1) + i×C̃
= 0.95×0.90 + 0.05×0.40
= 0.855 + 0.020 = 0.875 ← memory nearly intact
STEP B — the relevant cue arrives ("speak")
Now the gates open to update:
f = 0.60 (release some old)
i = 0.90 (write strongly)
C̃ = 0.95
C_t = 0.60×0.875 + 0.90×0.95
= 0.525 + 0.855 = 1.000 (clipped) ← memory refreshed
The thick belt running along the top is the cell state — long-term memory. Below it, each step shows the three gates as valves that fill up (open) or empty (closed). Step through a sequence and watch: on "boring" steps the forget gate stays open and the input gate stays shut, so the cell state glides through almost unchanged. On the highlighted keep-this step, the memory written there persists all the way to the end — the exact thing the plain RNN failed to do in 3.3.
Compare this directly to the 3.3 demo. There, input #1's gold trace faded to nothing. Here, the memory written on the keep-this step rides the cell-state belt to the end at near-full strength, because the forget gate chooses to keep it. That choice — held memory versus forced decay — is the entire advance of the LSTM.
In a plain RNN, the path from an early input to a late one runs through a tanh at every step, and the gradient gets multiplied down each time — it vanishes. In an LSTM, the cell state's main path is additive (C_t = f·C_(t-1) + i·C̃). When the forget gate stays near 1, the gradient flows backward along that path almost unchanged — a near-constant-error highway across time. That's the mechanism behind "long-range memory" in one sentence.
For roughly two decades, LSTMs were the workhorse of sequence modeling: machine translation, speech recognition, handwriting, time-series forecasting. If it was a sequence, an LSTM was likely under the hood.
Two costs remain. First, complexity: four neural layers (three gates plus the candidate) per cell means more parameters and slower training than a plain RNN. The GRU (3.5) is a direct response — a leaner gated cell that merges some of these for similar performance with less machinery.
Second, and far more important: an LSTM is still sequential. It must process step 1 before step 2 before step 3 — there's no way to compute them in parallel. On modern hardware built for massive parallelism, this is a severe bottleneck, and it grows with sequence length. That single limitation — not memory, but speed and parallelism — is what eventually motivated attention and the Transformer. We meet it head-on in 3.6.
The LSTM didn't make the network remember more by trying harder. It made memory a thing the network could choose to protect — and that one shift in design held up the entire field of sequence modeling for twenty years. On why gating mattered
| Property | Value | Notes |
|---|---|---|
| Core addition | Cell state + gates | A protected memory belt managed by learned valves |
| Forget gate | Sigmoid (0–1) | How much old memory to keep |
| Input gate | Sigmoid + tanh | How much new info to write, and what |
| Output gate | Sigmoid (0–1) | How much of the cell state to reveal |
| Key mechanism | Additive cell update | Gradient highway — solves vanishing over time |
| Strength | Long-range memory | Hundreds of steps; translation, speech, time series |
| Cost | 4× the parameters | Heavier than RNN — motivates the GRU |
| Hard limit | Still sequential | Can't parallelize across time — motivates attention |
The LSTM gives a recurrent network a memory it can protect. A separate cell state carries information across long distances along a mostly-additive path, while three gates — forget, input, output — decide at each step what to erase, what to store, and what to reveal. This solved the vanishing-memory problem that crippled plain RNNs, and made LSTMs the backbone of sequence modeling for two decades. But they pay for it with four times the parameters, which the GRU trims next, and they remain fundamentally sequential — the one limitation gating cannot fix, and the door through which attention and the Transformer eventually arrive.