A leaner gated cell that merges two of the LSTM's gates into one and drops the separate cell state — near-identical performance, fewer parameters, faster to train.
In 3.4 the LSTM solved long-range memory beautifully — but at a cost: four neural layers per cell (three gates plus a candidate), a separate cell state and hidden state, and a lot of parameters to train. A natural question followed: is all that machinery necessary, or can we get the same protected-memory behavior more cheaply?
The Gated Recurrent Unit (GRU), introduced by Cho and colleagues in 2014, is the answer. It keeps the core insight of gating — letting the network choose what to keep and what to update — but streamlines it. Two LSTM gates (forget and input) are merged into a single update gate, a new reset gate controls how much past to use when forming new candidate memory, and the separate cell state is dropped entirely: the GRU carries everything in just the hidden state. The result is roughly 25% fewer parameters, faster training, and — on most tasks — performance statistically indistinguishable from the LSTM.
Back at the whiteboard from 3.4. The LSTM gave you two separate controls: one switch for "how much to erase" and another for "how much to write." But think about it — those two decisions are really two sides of one choice. If you're keeping the old note, you don't need the new one; if you're writing the new one, you're replacing the old. So why two switches?
The GRU replaces them with a single dial: the update gate. Turn it toward "keep" and the old memory stays; turn it toward "update" and new information takes its place — but it's one motion, a smooth blend between old and new, not two independent valves. There's also a small second control, the reset gate, which decides how much of the old memory to even glance at when drafting the new note. Fewer controls, nearly the same result — like a photographer who gets the same shot with two dials that a rival needs four to manage.
The GRU has two gates instead of three, and no separate cell state. The reset gate (r) decides how much of the previous hidden state to use when computing a candidate new memory — turn it down and the cell "forgets" the past when drafting its proposal, useful for detecting the start of a new pattern. The update gate (z) then performs the key blend: it interpolates between the old hidden state and the candidate, deciding the mix of remembered-versus-refreshed.
That interpolation is the elegant part. Where the LSTM wrote C_t = f·C_(t-1) + i·C̃ with two independent gates, the GRU writes h_t = (1-z)·h_(t-1) + z·h̃ — one gate z controlling both sides at once. When z is near 0, the old memory passes through almost untouched (the same additive, gradient-friendly highway the LSTM relied on). When z is near 1, the candidate takes over. One dial, both jobs.
Two gates (each a sigmoid over [previous hidden, new input]):
z_t = σ( W_z · [h_(t-1), x_t] ) update gate
r_t = σ( W_r · [h_(t-1), x_t] ) reset gate
Candidate hidden state (reset gate filters the past):
h̃_t = tanh( W · [ r_t * h_(t-1) , x_t ] )
Blend old and new with ONE gate:
h_t = (1 - z_t) * h_(t-1) + z_t * h̃_t
└── keep old ──┘ └── take new ──┘
No separate cell state. h_t carries everything.
One GRU unit holds a memory. Watch how a single update gate z decides the balance between keeping the old value and adopting a new candidate.
Previous hidden state: h_(t-1) = 0.80 (a stored fact)
CASE 1 — nothing important happens (update gate stays low)
z = 0.10 (mostly keep)
h̃ = 0.30 (some candidate, but z gates it out)
h_t = (1 - 0.10)×0.80 + 0.10×0.30
= 0.90×0.80 + 0.10×0.30
= 0.720 + 0.030 = 0.750 ← memory nearly preserved
CASE 2 — a strong new cue arrives (update gate opens)
z = 0.85 (mostly update)
h̃ = 0.95 (strong candidate)
h_t = (1 - 0.85)×0.80 + 0.85×0.95
= 0.15×0.80 + 0.85×0.95
= 0.120 + 0.8075 = 0.928 ← memory refreshed
One gate z did the whole keep-vs-update decision —
no second gate needed.
z keeps the past (0.75, close to the original 0.80); a high z swaps in the new value (0.93). The LSTM needed a forget gate and an input gate to express this. The GRU does it with one. See it run beside an LSTM live below.
The same sequence runs through both architectures at once. The top track is an LSTM's cell-state memory; the bottom is a GRU's hidden state. A fact is stored at the ★ step, then must survive a run of irrelevant inputs. Step through and watch both lines hold the memory across the gap — the GRU keeping pace with the LSTM despite having one fewer gate and no separate cell state. The readout tracks how close they stay.
The two lines track each other closely the whole way — both hold the stored fact across the gap and both refresh when the cue returns. That's the headline result: the GRU matches the LSTM's memory behavior with about 25% less machinery. On a small model that's a minor saving; across a deep stack trained on huge data, it's faster training and less memory for essentially the same accuracy.
There is no universal winner. Across years of benchmarks, the two perform comparably on most sequence tasks, and which one wins is usually dataset-dependent and small. The practical rule of thumb: start with a GRU when data or compute is limited and you want faster iteration, and reach for an LSTM when you have a large dataset and very long dependencies where its extra control might help. Often, you simply try both and keep whichever validates better.
Both were enormously successful. Together they powered the golden age of sequence learning — neural machine translation, speech-to-text, and more — from roughly 2014 to 2018.
The GRU asked a sharp question: how much of the LSTM's complexity actually earns its keep? The answer — "less than you'd think" — gave the field a lighter tool that, for most jobs, works just as well. On simplification as a design virtue
For all their cleverness about memory, the GRU and the LSTM share the RNN family's defining constraint: they are sequential. Each step must finish before the next begins, because every hidden state depends on the one before it. Memory was solved; speed and parallelism were not.
That is the wall the entire recurrent family hits — and 3.6 Limitations of RNNs examines it directly: why sequential computation can't exploit modern parallel hardware, why long sequences stay slow no matter how clever the gating, and why these twin problems set the stage for attention to replace recurrence altogether.
| Property | Value | Notes |
|---|---|---|
| Gates | 2 (update, reset) | One fewer than the LSTM's three |
| Update gate (z) | Blends old/new | (1−z)·old + z·candidate — merges forget & input |
| Reset gate (r) | Filters the past | How much history to use when drafting the candidate |
| State | Hidden only | No separate cell state — simpler than LSTM |
| Parameters | ~25% fewer | Faster training, lighter footprint |
| Performance | ≈ LSTM | Comparable on most tasks; winner is dataset-dependent |
| Best use | Limited data/compute | Good default when you want speed and simplicity |
| Shared limit | Still sequential | Can't parallelize across time — motivates attention |
The GRU is the LSTM trimmed down: two gates instead of three, one state instead of two, about a quarter fewer parameters — and, on most tasks, the same performance. Its update gate merges the LSTM's forget and input gates into a single keep-versus-refresh dial, while a reset gate controls how much past to use when drafting new memory. Pick a GRU for speed and smaller data, an LSTM for very long, complex sequences, or just try both. But both belong to the recurrent family, and both inherit its fatal constraint: they process one step at a time and cannot be parallelized. That limitation — not memory, but speed — is what the next lesson dissects, and what ultimately ended the reign of recurrence.