Build your own small language model from any text and follow how it picks the next word.
The steps are a suggested order. Feel free to jump between them.
Training data
Context length
The number of previous words the model takes into account when choosing the next one.
How a word is chosen
1.0
Write at least one word.
Train the model and it will start writing.
The model has not been trained yet.
context chosen worddead end
The Language Model Lab
Build, analyse and evaluate your own language model, step by step.
1
How does the model choose the next word? Follow how the model continues a text and how it decides which word comes next.
2
What does the model remember? Explore how many previous words the model takes into account, and what that does to the text.
3
How is the model's choice affected? Try different ways of picking the next word, and see what temperature does to the result.
4
How does the training data shape the model? Switch between texts and see how the language changes. Then combine freely, and finally compare the model with ChatGPT.