Next word prediction from the ground up
See how predicting one word at a time produces whole answers, and what temperature changes
A taste of a lesson
I asked the same question twice and got two different answers. Is it broken?
Not broken, it is working as designed. At each step, the model produces probabilities for possible next words, and most chat apps pick among the likely ones with a bit of randomness, called sampling. A different early word sends the answer down a different path. This variety is useful for creative tasks. For factual questions, differing answers are a hint to check: if two runs disagree on a fact, at least one is wrong. Try asking a factual question three times. Do the facts stay the same?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Describe the loop a model uses to generate text token by token
- Explain how probabilities, sampling and temperature shape an answer
- Explain why the same question can give different answers
- Evaluate the claims 'just autocomplete' and 'thinks like us' with balanced reasons
Lesson plan
- 1 The prediction game Experience next word prediction yourself and connect it to models. Start
- 2 The generation loop Understand how one token at a time becomes a full answer. Start
- 3 Choosing words: greedy, sampling, temperature See how selection settings change style and reliability. Start
- 4 Why it seems so capable Understand how a simple objective leads to rich behaviour. Start
- 5 Autocomplete or mind? Weigh two popular descriptions of language models fairly. Start
Try asking
About this tutor
For beginners who have heard that language models 'just predict the next word' and want to know what that really means and why it is both true and misleading. You play prediction games yourself, see how a model turns text into probabilities for the next token, and watch how sampling settings such as temperature change the result. You learn why the same question gives different answers, why models sometimes repeat themselves, and how such a simple objective can produce surprisingly capable behaviour. You finish able to explain the generation loop clearly and to judge claims that models are either 'just autocomplete' or 'thinking like us'.
Reviews
4.5
4 ratingsSample
- Ewa P.Sample
Simple and honest. I teach secondary school and borrowed the prediction game for my own class.
- Clara E.Sample
The guessing game at the start made everything click. Now I understand why my answers vary and why disagreement between runs is a warning sign.
- Hassan M.Sample
Clear explanation of temperature. I liked the balanced take on 'just autocomplete'. Could use one more example of top p.
- Jorge L.Sample
Good beginner tutor. The bit about early words derailing an answer explained some odd replies I had seen.
About the teacher
I explain how language models really work, from tokens to attention, without hand waving
9 tutors 525 lessons taught Sample
I like taking the mystery out of language models. I teach what happens between typing a question and reading an answer: tokens, context windows, embeddings, attention, training and fine tuning. I start every topic with a picture or a small worked example and only add maths when it earns its place. My work background is in software and teaching workshops...
See Bastian's profile and tutorsMore like this
Other tutors on the same or nearby topics.