What is an MLP? The multilayer perceptron inside ChatGPT, explained

MLP stands for multilayer perceptron, the oldest and simplest kind of neural network. It’s layers of small units called neurons. Each neuron multiplies each of its inputs by a weight, adds them up, adds one more number called a bias, and then, in the hidden layers, bends the result. Stack enough of these, train the weights on examples, and the network can learn surprisingly complicated patterns.

MLPs also make up a big share of the large language models behind ChatGPT, Claude and Gemini. Every layer of those models has an attention step followed by an MLP, and in GPT-3 the MLPs hold about two thirds of all the model’s numbers. Our guide to how LLMs work mentions them in passing. This one builds one from a single neuron, with a tiny network you can train yourself further down.

One neuron: weigh, add, bend

A neuron takes some numbers in and gives one number out. Each input arrives along a line with its own weight. A big weight means that input counts for a lot, a small one means it barely counts, and a negative weight pulls the total down. The neuron multiplies each input by its weight, adds the results, then adds its bias, which shifts the total up or down whatever the inputs are.

Take a neuron with two inputs, both 1, both weights 1 and a bias of −1:

1 × 1 + 1 × 1 − 1 = 1

That total then goes through an activation function, the bend. The most common one today is ReLU (rectified linear unit): it passes anything above zero unchanged and turns anything below zero into zero. So 1 stays 1, and if the inputs were both 0 the total would be −1, which ReLU turns into 0. Language models such as GPT-2 use GELU, a smoother version of the same bend that lets small negative numbers through a little.

The weights and the bias are the neuron’s parameters. When people say a model has 175 billion parameters, they mean 175 billion of these numbers.

Layers of neurons

An MLP puts neurons side by side in layers and connects every neuron in one layer to every neuron in the next. The inputs come in on the left, pass through one or more hidden layers, and come out of the output layer on the right. They’re called hidden because you never see their values directly: you set the inputs and read the outputs.

“Multilayer” means it has at least one hidden layer. Output neurons usually skip the bend, so they can give any number. Another name you’ll see is feed-forward network, because the numbers only ever flow one way, from inputs to outputs.

Where the perceptron came from

The psychologist Frank Rosenblatt described the perceptron at Cornell Aeronautical Laboratory, first in a January 1957 report and then in a 1958 paper in Psychological Review. His perceptron was one layer of adjustable weights that learned from its mistakes.

On 7 July 1958 the US Navy, which funded the work, showed it off in Washington, running as a program on the Weather Bureau’s IBM 704 computer. The next day’s New York Times carried a UPI report that began:

“The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.”

What it had actually learned was to tell cards with a square marked on the left from cards with a square marked on the right, after about 50 tries according to Cornell’s account. A follow-up in the Times on 13 July said it got 97 out of 100 right.

Rosenblatt’s team then built the perceptron as a machine, the Mark I Perceptron. Its operators’ manual is dated February 1960. Its “eye” was 400 light sensors in a 20 by 20 grid, and its weights were potentiometers, a kind of dial, turned by small electric motors as it learned. It’s now in the Smithsonian.

XOR and what one layer can’t do

In 1969 Marvin Minsky and Seymour Papert, both at MIT, published a book called Perceptrons. It worked out mathematically what perceptrons with a single layer of adjustable weights can and can’t do. Its main results are about harder problems such as telling whether a shape is connected, but the example everyone remembers is XOR, short for exclusive or: on when one input is on, but not both.

x₁	x₂	XOR
0	0	0 (off)
0	1	1 (on)
1	0	1 (on)
1	1	0 (off)

Draw the four input pairs as the corners of a square. A single neuron can only split that square with one straight line: on one side it says on, on the other it says off. XOR wants the top left and bottom right corners on and the other two off, and no straight line puts them on different sides. Try it with a ruler: one corner always ends up on the wrong side.

The book is often blamed for a long slump in neural network funding through the 1970s. How much it caused that is still argued about. Its maths only covers one layer of adjustable weights, but training anything deeper wasn’t practical yet.

A hidden layer and a bend solve it

Add a hidden layer with two neurons. Give them the same weights, 1 and 1, but different biases, 0 and −1. Then the output takes the first, minus twice the second:

• h₁ = ReLU(x₁ + x₂)
• h₂ = ReLU(x₁ + x₂ − 1)
• output = h₁ − 2 × h₂

x₁	x₂	h₁	h₂	output
0	0	0	0 (was −1)	0
0	1	1	0	1
1	0	1	0	1
1	1	2	1	0

That’s XOR. The bend did the work: at (0, 0) the second neuron’s total is −1 and ReLU folds it flat to 0. Across the whole square the output is a ridge, high along the line where x₁ + x₂ = 1 and down at the two off corners.

Now switch the bend off. Without ReLU, h₂ stays −1 at (0, 0), and the whole network works out as one formula:

output = 2 − x₁ − x₂

That gives 2, 1, 1, 0: a flat, tilted slope that can’t be right at both off corners. This isn’t bad luck with the weights. A weighted sum of weighted sums is still just a weighted sum, so a network without bends, however many layers it has, can do no more than a single layer. The bends are what let an MLP model curves, folds and ridges. With enough hidden neurons, even a single hidden layer can get as close as you like to any continuous pattern over a limited range of inputs, a result first proved in 1989 by George Cybenko and, separately, by Kurt Hornik and colleagues.

How a network learns its weights

Nobody sets the weights of a real network by hand. Training starts from small random weights, runs the examples through, and measures how wrong the answers are with a loss. Here it’s the mean squared error: the average of each answer’s miss, squared. Then it nudges every weight and bias a little in whichever direction makes the loss smaller, and repeats. That’s gradient descent, and the size of each nudge is the learning rate.

The direction for each weight comes from backpropagation. It starts at the output, works out how much each output contributed to the error, then works backwards layer by layer to share that blame out to every weight, using the chain rule from calculus. David Rumelhart, Geoffrey Hinton and Ronald Williams popularised it for neural networks in a 1986 paper in Nature, “Learning representations by back-propagating errors”, which showed hidden layers learning useful features of their own. The maths is older: Seppo Linnainmaa described the general method in 1970, and Paul Werbos suggested it for training neural networks in his 1974 PhD thesis.

The run in the video uses the two hidden neurons above, starting from random weights, with a learning rate of 0.1 and all four XOR examples at every step. Its error starts at 0.375. By step 81 all four answers are within 0.1 of the right one, and by step 200 the error is under 0.00001. It didn’t find the hand-set answer: it settled on its own, something closer to “how far apart are x₁ and x₂”. The playground above starts from exactly the same weights, so pressing Train once repeats it.

That run was picked because it works. With only two hidden neurons, most random starts get stuck with one or two corners wrong, often because a hidden neuron ends up below zero for every example, so it passes nothing back and stops learning. In our checks of 200 random starts, about 1 in 4 solved XOR. With four hidden neurons, about 2 in 3 did. Real networks have far more neurons than they strictly need, partly for this reason.

The MLP inside ChatGPT and other LLMs

A large language model reads text as tokens and turns each token into a long list of numbers. Those lists then climb a stack of layers. Each layer has two parts: attention, where each token looks back at the tokens before it and takes in what it needs, then an MLP, which works on each token’s list on its own. Each part adds its result onto the token’s running list rather than replacing it.

In GPT-2 small, OpenAI’s open model from 2019, each token carries 768 numbers. Each layer’s MLP spreads them across 3,072 hidden neurons, bends them with GELU, then sums them back down to 768. It has 12 layers, so 12 of these MLPs.

GPT-3 is the same design at a bigger size: 96 layers, each with an MLP that goes from 12,288 numbers to 49,152 neurons and back. Working through its published sizes, the MLPs hold about 116 billion of its roughly 175 billion parameters, about two thirds, and attention holds about 58 billion. Small models are different: in GPT-2 small the MLPs are about 45% of its 124 million, because the table of token lists takes up so much of a small model.

Newer models change the details. Meta’s Llama models use a gated version called SwiGLU: the MLP makes two sets of hidden values, bends one, multiplies the two together, then sums back down. Mixture-of-experts models such as Mixtral and DeepSeek V3 give their layers many separate MLPs, the experts, and send each token through only a few of them.

What MLPs seem to store

If attention moves information between tokens, what are the MLPs doing? Two papers gave the best-known answers.

In 2021 Mor Geva, Roei Schuster, Jonathan Berant and Omer Levy described the MLP as a set of key-value memories. Each hidden neuron’s incoming weights act like a key that responds to a pattern in the text: surface patterns such as a particular word in the lower layers, patterns about topic or meaning higher up. Its outgoing weights act like a value that pushes up the words likely to come next after that pattern, most clearly in the upper layers. They studied a 16-layer model trained on Wikipedia articles, not ChatGPT.

In 2022 Kevin Meng, David Bau, Alex Andonian and Yonatan Belinkov traced where GPT-2 XL recalls facts such as which city a landmark is in. Across 1,000 facts, MLPs in the middle layers mattered most while the model was reading the last word of the subject. Their method, ROME (rank-one model editing), then changed one MLP layer’s weights to rewrite a single fact. In one example they made the model say the Eiffel Tower is in Rome. It then described the tower as being across from St Peter’s Basilica. The real tower stayed where it was.

Later work suggests the picture is messier. The same team’s 2023 follow-up, MEMIT, spread its edits across several MLP layers. Peter Hase and colleagues found that where tracing points to a fact doesn’t predict which layer is best to edit. Geva and colleagues found recall happens in steps, with attention pulling the answer out of what earlier MLPs added. And a 2023 study at Google DeepMind found sports facts about athletes spread across several MLP layers, with no single neuron holding them. “Facts live in the MLPs” is a useful rough picture, not a map.

Words to know

• Neuron: a unit that multiplies its inputs by weights, adds them and a bias, and usually bends the total.
• Weight: how much one input counts towards a neuron’s total. Negative weights pull the total down.
• Bias: a number added to the total whatever the inputs, shifting where the neuron switches on.
• Activation function: the bend after the sum. ReLU turns negatives into zero; GELU and SwiGLU are smoother relatives.
• Hidden layer: a layer between the inputs and the output.
• MLP (multilayer perceptron) or feed-forward network: layers of neurons, each connected to every neuron in the next layer.
• Parameters: all the weights and biases. Training sets them.
• Loss: one number measuring how wrong the answers are.
• Gradient descent: nudging every parameter a little downhill on the loss, over and over.
• Backpropagation: working out those nudges from the output backwards.

Sources

• Rosenblatt, 1957: The Perceptron: A Perceiving and Recognizing Automaton (Cornell Aeronautical Laboratory report 85-460-1); Rosenblatt, 1958: The perceptron: a probabilistic model for information storage and organization in the brain, Psychological Review 65(6)
• The New York Times, 8 July 1958: New Navy Device Learns by Doing; 13 July 1958: Electronic ‘Brain’ Teaches Itself
• Cornell Chronicle, 2019: Professor’s perceptron paved the way for AI, 60 years too soon
• Hay, Lynch and Smith, 1960: Mark I Perceptron Operators’ Manual; Smithsonian National Museum of American History: Mark I Perceptron
• Minsky and Papert, 1969: Perceptrons: An Introduction to Computational Geometry, MIT Press
• Rumelhart, Hinton and Williams, 1986: Learning representations by back-propagating errors, Nature 323; Jürgen Schmidhuber: Who invented backpropagation?
• Cybenko, 1989: Approximation by superpositions of a sigmoidal function; Hornik, Stinchcombe and White, 1989: Multilayer feedforward networks are universal approximators
• OpenAI: GPT-2 model code; Brown and others, 2020: Language Models are Few-Shot Learners (GPT-3)
• Touvron and others, 2023: LLaMA: Open and Efficient Foundation Language Models; Shazeer, 2020: GLU Variants Improve Transformer
• Geva, Schuster, Berant and Levy, 2021: Transformer Feed-Forward Layers Are Key-Value Memories
• Meng, Bau, Andonian and Belinkov, 2022: Locating and Editing Factual Associations in GPT (project page)
• Meng and others, 2023: Mass-Editing Memory in a Transformer (MEMIT); Hase and others, 2023: Does Localization Inform Editing?; Geva and others, 2023: Dissecting Recall of Factual Associations in Auto-Regressive Language Models; Nanda and others, 2023: Fact Finding: Attempting to Reverse-Engineer Factual Recall
• The film’s network and training run: our own, with the same maths as the playground above.