Tiny LLM
Paste a paragraph. Train a small transformer on it and watch it learn to write, one character at a time. The whole model, training and all, runs in your browser. Nothing is downloaded and nothing is sent.
1. Give it something to read
The model learns only from what is in this box. A paragraph gives you a model that can quote the paragraph. A few pages give you one that starts to sound like the author. More text means a longer wait.
2. Shape the model
3. Train it
Ready. Training starts the moment you press the button and runs on this computer only.
What it writes right now
Nothing yet.
4. Make it write
Train a model, or load a weights file, and the output appears here.
Low temperature plays it safe and repeats what it knows. High temperature takes chances and makes more mistakes. The weights file holds the whole model: every number it learned, plus the settings used to train it.
What you are looking at
Loss
How surprised the model is by the real next character, averaged over the text. A fresh model is as surprised as a fair die with one side per character. As it learns, the number falls.
Why gibberish at first
It starts with random numbers. The first thing it picks up is which characters are common. Then pairs. Then the shape of words. Real words climb out after a few hundred passes.
Why it quotes the paragraph
With one paragraph there is nothing else to learn. Tick the held-out box and you will see the held-out loss stop falling while the training loss keeps going. That is memorizing, not understanding. More text is the only fix.
Blow it up on purpose
Untick clipping, set the learning rate to 3, and train. Within a few seconds the loss turns into "not a number" and the weights are ruined. Every lab deals with this. Clipping is the standard cure, and it has limits: at that rate it stops the crash but the model learns nothing.
The same code, in Python
This page is a port of feel-smart-llm, one file of numpy with the backward pass written by hand and a gradient check you can run. Read it in one sitting.
What is missing
Real models stack dozens of these blocks, use several attention heads, normalize between layers, work on word pieces instead of characters and read terabytes. The math in each block is the same as here.