No background needed. We'll build a working neural network one small piece at a time — starting from a single decision, ending with something you can play with.
A gentle walkthrough · every term explained the moment it appears
Let's start with something you already do without thinking.
Suppose you're deciding whether to take an umbrella. You weigh a few things: How dark are the clouds? What did the forecast say? Did it rain yesterday? Some of these matter more than others. The forecast probably matters a lot. Yesterday's rain, not so much. You mentally give each one some importance, add them up, and if the total crosses some line in your head — you grab the umbrella.
That's it. That's a neuron. A neural network is just a lot of those little weigh-things-up-and-decide steps, wired together so the answer of one becomes a question for the next. Everything in this lesson is built from that single idea, so let's make it concrete.
By the end of this page you'll understand a feed forward network — the simplest kind of neural network, and the foundation every other one in this chapter is built on. "Feed forward" just means information flows one way: in one side, out the other, no going back. We'll get there in three steps: one neuron, then a layer of them, then a full little network you can poke at.
A neuron takes in a few numbers, decides how much each one matters, adds them up, and produces one number out. Let's use the umbrella example with real numbers.
Say we describe today with three numbers, each from 0 to 1:
These input numbers are called inputs, and we'll label them x₁, x₂, x₃ — just names, like "the first input, the second input." The little number is just which one we mean.
Now, how much should each matter? We assign each input an importance number called a weight. A big weight means "this matters a lot"; a weight near zero means "mostly ignore this"; a negative weight means "this actually argues against." Let's say forecast matters most:
A number that says how important an input is. The network learns these — adjusting them until its answers get good. Learning a neural network is finding good weights. That's the whole game.
To combine them, we multiply each input by its weight and add everything up. That sum has a name — the weighted sum — but it's just "importance × value, all added together":
(cloud 0.8 × weight 0.5) = 0.40
(forecast 0.9 × weight 1.0) = 0.90
(yesterday 0.3 × weight 0.2)= 0.06
add → 1.36
One more small piece. Sometimes a neuron should lean "yes" or "no" before it even looks at the inputs — a baseline mood. We add a single number called the bias to nudge the total up or down. If our umbrella-neuron is naturally cautious, we might subtract a bias of 0.5 (raising the bar for grabbing the umbrella):
weighted sum = 1.36
bias = -0.5
total → 0.86
In shorthand people write this as z = (w · x) + b. Don't let the letters scare you: z is the total we just computed, w is the weights, x is the inputs, b is the bias. The dot just means "multiply each pair and add." You already did exactly this with the umbrella.
There's one final twist that turns this sum into a real decision — but it's important enough to deserve its own moment, and it'll land better once you've seen a few neurons working together. Hold that thought; we'll come back to it at "the twist." For now: a neuron = weigh inputs, add them, add a bias.
One neuron makes one small judgment. Real problems need many. So we do two things.
First, we put several neurons side by side, all looking at the same inputs but with different weights — so each learns to spot something different. One umbrella-neuron might focus on "is a storm coming," another on "will it just drizzle." A row of neurons like this is called a layer.
Second, we stack layers in a line. The outputs of one layer become the inputs to the next. So the first layer looks at raw facts, the second layer looks at the first layer's judgments, and so on — each layer working with something a little more refined than the last.
Think of a loan being approved at a bank. Junior staff look at the raw facts — income, credit score, amount. Each forms an opinion and passes a summary up to a manager. The manager weighs those opinions and passes a judgment to the director, who makes the final call. Information only flows upward — nobody sends the file back down. Each level works with a more refined version of the one below.
That's a feed forward network exactly. Raw facts = the input layer. Junior staff and managers = the hidden layers (called "hidden" simply because they sit in the middle — you don't see their answers directly). The director = the output layer, giving the final answer.
Any layer between the input and the output. It's where the network builds up useful in-between ideas. "Hidden" isn't mysterious — it just means you don't read its numbers directly; they're stepping stones to the final answer.
And that's a whole network: inputs on the left, one or more hidden layers in the middle, an output on the right, with every number flowing strictly left to right. Now let's stop describing it and let you run one.
Here's a real, working network: the umbrella-decider. Three inputs feed two hidden neurons, which feed one output that gives the chance you should take an umbrella. Drag the sliders and watch the numbers flow through, left to right, live. Everything you just read is happening on screen — the panel on the right narrates each step as it computes.
3 inputs → 2 hidden neurons → 1 output. Drag anything. Watch it flow.
Back to that held thought. After a neuron computes its weighted sum plus bias, it does one last thing: it passes that number through a small shaping function called an activation function. In the playground, that's the ReLU / Sigmoid / Tanh buttons you just saw.
Why bother? Here's the surprising part. Without this step, stacking layers would be pointless. If every neuron only ever did "multiply and add," then chaining a hundred layers together would mathematically collapse into a single "multiply and add." A hundred layers would do no more than one. The network could only ever draw a straight line between yes and no — useless for anything curvy or complex, which is basically everything real.
The activation function bends the straight line. By reshaping each neuron's output a little, it lets the network learn curves, corners, and complicated boundaries. That bending is what makes deep networks powerful — without it, depth buys you nothing.
You met these shaping functions in Chapter 2. A quick reminder of the three in the playground:
| Activation | What it does, plainly | You'll see… |
|---|---|---|
| ReLU | If the number is negative, output 0. Otherwise pass it through unchanged. | Neurons can switch fully "off" (go dark) when their sum drops below zero. |
| Sigmoid | Squashes any number into a 0-to-1 range — like a probability. | Outputs hug a smooth S-curve; nothing ever goes negative. |
| Tanh | Squashes into a −1 to +1 range, centered at zero. | Gentler, balanced swings around the middle. |
1. Multiply each input by its weight. 2. Add them up. 3. Add the bias. 4. Pass the result through an activation function.
The output of that becomes an input to the next layer. Repeat, layer by layer, until the answer pops out the right side. That repeat-until-the-end is the entire "forward pass" — and the whole reason it's called a feed forward network.
You might wonder: why use several hidden layers instead of one? Because layers build on each other. An early layer learns rough, simple patterns. The next layer combines those into richer ones. The next, richer still.
The classic example is recognizing a face in a photo. The first hidden layer might only detect tiny edges — light meeting dark. The next layer combines edges into small shapes: a curve, a corner. The next combines shapes into parts: an eye, a nose. The final layers combine parts into "this is a face." Nobody told the network about eyes — it built that idea up, layer by layer, from raw pixels. This stacking-up of simpler ideas into complex ones is why we call it deep learning. The "deep" is literally the number of layers.
One layer can only learn simple things. Many layers can learn simple things, then combine them, then combine those — which is a far more efficient way to capture something complicated. Depth lets a network build understanding in stages, like learning arithmetic before algebra before calculus.
This humble network is powerful, but it has two real blind spots — and naming them tells you exactly why the rest of this chapter exists.
A feed forward network treats its inputs as a plain, unordered list. To it, a pixel in the corner of a photo has no relationship to the pixel right next to it — they're just two items on a list. But in an image, neighboring pixels are deeply related. There's also a size problem: a small 200×200 photo has 40,000 pixels, meaning each neuron in the first layer would need 40,000 weights. Wasteful, and blind to the structure of the picture.
Feed it a sentence and it can't tell "dog bites man" from "man bites dog" — same words, and it processes each input independently with no sense of sequence or what came before. For anything where order matters — language, music, time — this network is lost.
These two gaps are the whole story of what comes next. Convolutional networks (3.2) fix the first by giving the network a sense of space. Recurrent networks (3.3) fix the second by giving it memory of order. But both are built on exactly the neuron-and-layer idea you just learned. You now own the foundation; everything ahead is this same machine, upgraded.
A feed forward network is just neurons — weigh, add, bend — stacked in layers, with information flowing one way. Simple to the point of feeling like it shouldn't work. And yet it's the seed of every model in this chapter, including the ones behind modern AI.
| Idea | In plain words |
|---|---|
| Neuron | Weigh each input, add them up, add a bias, then bend the result. |
| Weight | How much an input matters. The network learns these. |
| Bias | A baseline nudge toward "yes" or "no." |
| Activation | The bend that lets layers actually add power. Without it, depth is pointless. |
| Layer | Neurons side by side. Stack layers and ideas get richer each step. |
| Feed forward | Information flows one way, input to output. No loops, no memory. |
A feed forward network is the simplest neural network: layers of neurons that weigh inputs, add them up, and bend the result, with everything flowing strictly forward. Stack enough layers and it can learn remarkably complex things by building simple ideas into rich ones. Its limits — no sense of space, no memory of order — are exactly what the next architectures fix. Master this, because every network ahead is this same idea given a new sense.