How LLMs like ChatGPT work, and how they got this good

ChatGPT, Claude, Gemini and the other large language models, or LLMs, write their answers one small piece at a time. Each piece is the model’s guess at what comes next, given everything before it. This guide follows one guess from your text going in to the next token coming out, then covers how training sets the model’s numbers and the changes since 2017 that took these models from finishing sentences to reasoning and running as agents. It’s up to date to 29 September 2026.

The examples use GPT-2 small, the open model OpenAI released in 2019, because every number inside it can be checked. The playground runs its real tokeniser and shows its real next-token chances.

Text becomes tokens

A model doesn’t read letters or whole words. A tokeniser chops your text into tokens: common words, pieces of words, punctuation and spaces, each from a fixed list called the vocabulary. GPT-2’s vocabulary has 50,257 tokens. OpenAI’s rule of thumb is that a token in ChatGPT averages about three quarters of an English word.

Each token is really a number. Most words carry their space at the front, so “ kettle”, with the space, is one token, number 40,231. “Kettle” at the start of a sentence is two: K and ettle. Names that aren’t common enough get split, so Hana becomes H and ana.

GPT-2’s tokeniser uses byte pair encoding. It starts from the raw bytes of your text and works through a list of 50,000 merges, learned from its training text, each joining a common pair into a bigger piece. Because it starts from bytes it can take any text at all. An emoji or a word in Korean just comes out as several tokens, some holding only part of a character.

That’s one reason chatbots used to miscount the r’s in strawberry. In “How many r’s are in strawberry?”, GPT-2 sees “ strawberry” as a single token, number 41,236, with the letters hidden inside it. The tokenisers for GPT-4 and GPT-4o do the same. As Andrej Karpathy puts it in his Deep Dive into LLMs like ChatGPT: “Number one, the models don’t see characters, they see tokens. And number two, they are not very good at counting.”

Each token becomes a list of numbers

Each token’s number picks out a row in a big table, called the embedding table. Every row is a list of numbers: 768 of them in GPT-2 small, and 12,288 in GPT-3. Training pulls tokens that are used in similar ways close together, so the rows nearest “ tea” in GPT-2 include coffee and caffeine.

The model also needs to know where each token sits in the text. GPT-2 adds a second learned list for each position. Most newer models use rotary position embeddings instead, which work inside attention.

Attention

The lists then climb a stack of layers: 12 in GPT-2 small, 96 in GPT-3. Each layer starts with attention, where every token looks at itself and the tokens before it, never ahead.

Each token turns its current list into three new ones:

• a query, for what it’s looking for
• a key, for what it offers
• a value, for what it passes on

Its query is compared with its own key and every earlier one. The scores become weights that add up to 100%, and the token takes in that mix of values.

In “The cat sat by the fire because it was cold”, one of GPT-2 small’s attention heads, in its fifth layer, puts 81% of its weight from “it” on “cat”, so “it” takes in mostly cat’s value. That’s the most cat-focused of its 144 heads, and heads don’t usually resolve words this cleanly. Each layer runs 12 heads side by side, each free to pick up on something different.

The MLP and the residual stream

After attention, each layer has an MLP, or multilayer perceptron, which works on each token separately. It stretches the token’s list to four times the length (3,072 numbers in GPT-2 small), then squeezes it back to 768. Research suggests much of what a model knows about the world sits in these blocks, though that’s still debated.

Each layer adds its result onto the token’s running list, called the residual stream, rather than replacing it. You can read that stream partway up by putting it through the model’s final step early. For the kettle sentence, after six of GPT-2 small’s 12 layers, tea is already the top guess.

Picking the next token

At the top, the last token’s list becomes a score for every token in the vocabulary, all 50,257, and the scores are turned into probabilities that add up to 100%. After “Hana put the kettle on and made a cup of”, GPT-2 small gives tea 35% and coffee 16%, and the 50,249 tokens outside its top eight share 37% between them.

Chatbots usually pick at random, weighted by those chances, and many cut off the long tail of unlikely tokens first. Temperature changes the weighting: the scores are divided by it before they become probabilities.

• At 1 the chances are the model’s own.
• At 0.5 tea gets 81%. Lower temperatures favour the top guess.
• At 1.5 tea gets just 6%, and the tokens outside the top eight rise to 85% between them. That’s where odd words slip in.

The chosen token joins the text and the whole stack runs again for the next one, which is why answers appear a piece at a time.

The KV cache

Running the whole text through again for every new token would repeat a lot of work, so models keep a KV cache: the keys and values of every earlier token, saved as they go. Each new token only works out its own.

But the cache grows with the text. For Meta’s Llama 3.1 70B, a full 128,000-token context needs over 40 GB of cache at 16-bit precision, on top of the model itself. That’s already with Llama sharing keys and values between heads, which makes the cache eight times smaller. DeepSeek compresses them instead.

Training

All of those tables and layers are numbers, called parameters or weights. GPT-2 small has 124 million of them. They start out random, so at first every guess is rubbish.

Training has the model read a huge amount of text, predicting each next token. The loss measures how wrong each prediction was. Backpropagation works out which way to nudge every parameter to make the right token a bit more likely, and gradient descent nudges them, batch after batch.

For the video we trained a tiny model, about 3.2 million parameters, on Pride and Prejudice, one character at a time, for under 10 minutes on a laptop. At the start it wrote noise. After 100 steps it had spaces between words. After 3,000 it wrote “She considered it most important for herself”, which isn’t a line from the book.

The big models do the same at a vastly larger scale. Llama 3.1’s largest version has 405 billion parameters, trained on 15.6 trillion tokens using up to 16,000 H100 GPUs.

Does predicting words take understanding?

Geoffrey Hinton, who shared the 2024 Nobel Prize in Physics for work that made machine learning with neural networks possible, told 60 Minutes in 2023: “It’s true they’re just trying to predict the next word. But if you think about it, to predict the next word, you have to understand the sentences.”

Plenty of researchers disagree. Emily Bender and Alexander Koller argued in 2020 that “a system trained only on form has a priori no way to learn meaning”. Richard Sutton, on the Dwarkesh Podcast in 2025, said: “To mimic what people say is not really to build a model of the world at all.”

How we got here

Neural networks were predicting next words by 2003, but most of what’s above dates from 2017 or later.

• 2003: Yoshua Bengio’s team published a neural network that predicted the next word, learning a list of numbers for each word along the way.
• 2010: Recurrent neural networks, which read text one word at a time and carry a memory along, became strong language models.
• June 2017: a Google team published “Attention Is All You Need”. Their Transformer dropped the one-word-at-a-time reading, so a whole passage could be processed at once in training, on GPUs. Co-author Noam Shazeer, at NVIDIA’s GTC in 2024: “RNNs were like the steam engine and Transformer is like internal combustion.”
• June 2018: OpenAI’s GPT-1 trained a Transformer on over 7,000 unpublished books, then fine-tuned it for each task.
• October 2018: Google’s BERT learned by filling in hidden words. It was good at reading text, not writing it.
• February 2019: GPT-2. OpenAI held back the full 1.5 billion parameter version until November, over worries about misuse. Ask the small one how many r’s are in strawberry and its most likely reply is, “R’s are the most common type of strawberry.”
• January 2020: OpenAI’s scaling laws showed the loss falling smoothly as models, data and computing power grew.
• May 2020: GPT-3, with 175 billion parameters, could pick up a new task from a few examples in the prompt, without any retraining.
• January 2022: InstructGPT was fine-tuned on answers people wrote, then tuned with reinforcement learning from human feedback, or RLHF: people ranked its replies, a reward model learned from those rankings, and the model was trained to score well against it. Labellers preferred the 1.3 billion parameter InstructGPT’s answers to those of the 175 billion parameter GPT-3.
• March 2022: DeepMind’s Chinchilla showed that, for the same computing budget, a smaller model trained on more data does better. Its 70 billion parameters beat the 280 billion parameter Gopher.
• 30 November 2022: OpenAI launched ChatGPT, trained in much the same way as InstructGPT.
• 2022 onwards: quantisation stores each parameter in fewer bits. At 4 bits instead of 16, a 70 billion parameter model drops from about 140 GB to about 40 GB.
• February 2023: Meta released LLaMA’s weights to researchers, then Llama 2 to anyone in July 2023. Alibaba’s Qwen followed in August, Mistral in September, DeepSeek in November and, in August 2025, OpenAI’s gpt-oss, its first open weights since GPT-2.
• March 2023: GPT-4 took in images as well as text.
• June 2023: OpenAI’s function calling let models ask for tools to be used, with the developer’s code running them.
• December 2023: Mistral’s Mixtral used a mixture of experts, sending each token through only 2 of 8 expert blocks in each layer. DeepSeek-V3, a year later, has 671 billion parameters but uses 37 billion for each token.
• February 2024: Google’s Gemini 1.5 could read up to a million tokens at once, in a preview for developers, against GPT-3’s 2,048. Rotary position embeddings and FlashAttention, a faster way to compute attention, both helped windows grow.
• September 2024: OpenAI’s o1 was trained with reinforcement learning to work through problems before answering, and did better the longer it thought. Noam Brown, who worked on o1, saw the same effect years earlier with a poker bot: “Spending 20 seconds thinking in a hand of poker got the same boost in performance as scaling up the size of the model and the training by 100,000x.”
• November 2024: Anthropic’s Model Context Protocol, or MCP, gave tools a common plug, so a tool written once works with any AI app that supports it.
• January 2025: DeepSeek’s R1 paper showed a recipe for reasoning: reward the model when its final answer checks out, like a correct maths result or code that passes its tests, and longer, more careful working emerges.
• 2025: models increasingly run as agents, in a loop: act, check the result, try again. The research group METR measures how long a software task, in skilled human time, AI can finish half the time. By its estimate, that has doubled about every seven months since 2019, and faster since 2024.

Where things stand

On 29 September 2026, the newest models from OpenAI (GPT-6.1 Sol, released that day), Anthropic (Claude Opus 5.5 and Sonnet 5.5) and Google (Gemini 3.8 Flash) reason when they need to, use tools and read about a million tokens at once. None of the three publishes the full design of its flagship models.

The published designs still run the same loop: tokens in, attention and MLPs up the stack, one token out, again and again. Some open models mix attention with other kinds of layer, and model names change almost every month.

Sources

• GPT-2 small (124M) run with Hugging Face’s gpt2 model and tokeniser, 29 and 30 September 2026: every GPT-2 token, weight and chance in this guide and the playground (the strawberry reply is greedy decoding, with a straight apostrophe)
• OpenAI, 2019: Language Models are Unsupervised Multitask Learners (GPT-2) and GPT-2: 1.5B release
• OpenAI Help Center: Understanding and counting tokens
• Bengio and others, 2003: A Neural Probabilistic Language Model
• Mikolov and others, 2010: Recurrent neural network based language model
• Vaswani and others, June 2017: Attention Is All You Need
• Radford and others, June 2018: Improving Language Understanding by Generative Pre-Training (GPT-1)
• Devlin and others, October 2018: BERT
• Geva and others, 2020: Transformer Feed-Forward Layers Are Key-Value Memories
• Kaplan and others, January 2020: Scaling Laws for Neural Language Models
• Brown and others, May 2020: Language Models are Few-Shot Learners (GPT-3)
• OpenAI, 27 January 2022: Aligning language models to follow instructions; Ouyang and others: Training language models to follow instructions with human feedback (InstructGPT)
• Hoffmann and others, March 2022: Training Compute-Optimal Large Language Models (Chinchilla)
• OpenAI, 30 November 2022: Introducing ChatGPT
• OpenAI, March 2023: GPT-4 Technical Report
• OpenAI, 13 June 2023: Function calling and other API updates
• Meta, 24 February 2023: Introducing LLaMA; Meta, 18 July 2023: Llama 2; Alibaba Qwen-7B, 3 August 2023 (TechNode); Mistral, 27 September 2023: Mistral 7B; DeepSeek LLM; OpenAI, 5 August 2025: Introducing gpt-oss
• Bartowski, Llama 3.1 70B Instruct quantised files: Hugging Face
• Mistral, 11 December 2023: Mixtral of experts; DeepSeek, December 2024: DeepSeek-V3 Technical Report
• Google, 15 February 2024: Our next-generation model: Gemini 1.5
• Su and others: RoFormer (rotary position embeddings); Dao and others: FlashAttention
• Meta, July 2024: The Llama 3 Herd of Models (Llama 3.1)
• OpenAI, 12 September 2024: Learning to reason with LLMs (o1)
• Anthropic, 25 November 2024: Introducing the Model Context Protocol
• DeepSeek, January 2025: DeepSeek-R1
• METR: Measuring AI Ability to Complete Long Software Tasks and Time Horizon 1.1
• The Nobel Prize, 8 October 2024: Nobel Prize in Physics 2024 press release
• Bender and Koller, 2020: Climbing towards NLU; Richard Sutton on the Dwarkesh Podcast, 2025
• Model pages, 29 September 2026: OpenAI models, Anthropic models overview, Gemini 3.8 Flash model card
• Clips in the video: Andrej Karpathy, Deep Dive into LLMs like ChatGPT; Geoffrey Hinton on 60 Minutes (CBS transcript); Noam Shazeer at the NVIDIA GTC 2024 Transforming AI panel; Noam Brown, AI Won’t Plateau, if We Give It Time To Think (TED)

This guide is based on what had been published by 29 September 2026. Model names and figures in this field change fast, and we’ll update it when they do.