Fine tune, prompt or retrieve?
Choose between prompting, retrieval and fine tuning for your problem, and know why
A taste of a lesson
Our team wants to fine tune a model on our HR policy documents so it can answer staff questions. Good idea?
Probably not as the first step. Answering policy questions is mainly a knowledge problem: the model needs the exact, current wording of your policies. Fine tuning stores facts unreliably, cannot easily cite the source, and goes stale the moment a policy changes. Retrieval fits better: search the policy documents, give the model the relevant passages and ask it to answer from them with references. Fine tuning could later help with tone or format if prompting falls short. Start by writing 40 real staff questions with correct answers. Quick check: what happens to a fine tuned model when the leave policy changes next month?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Diagnose whether a failure is an instruction, knowledge or behaviour problem
- Match each problem type to prompting, retrieval, fine tuning or a combination
- Compare the cost, data needs and upkeep of each approach
- Build a small evaluation set to guide the decision
- Spot warning signs that fine tuning is premature
Lesson plan
- 1 Diagnose before you choose Classify a model's failures as instruction, knowledge or behaviour problems. Start
- 2 Prompting first Get the most from clear instructions, context and examples before anything else. Start
- 3 Retrieval for knowledge Understand when retrieval is the right fix and what it costs to run well. Start
- 4 Fine tuning for behaviour Recognise problems fine tuning solves well and the ones it does not. Start
- 5 Costs, upkeep and combinations Compare total cost of ownership and design combined approaches. Start
- 6 Deciding with an evaluation set Use a small test set to compare approaches and justify the choice. Start
Try asking
About this tutor
For anyone deciding how to make a language model work for a specific job, from managers to engineers. Instead of a technical deep dive, this tutor teaches a decision framework: first diagnose whether your problem is about instructions, missing knowledge, or behaviour and style, then match it to prompting, retrieval or fine tuning, or a combination. You will compare the costs, data needs, upkeep and freshness of each approach, work through realistic cases such as a policy assistant, a ticket classifier and a brand voice writer, and learn the warning signs of fine tuning too early. Every lesson comes back to one habit: build a small evaluation set before choosing anything.
Reviews
4.7
3 ratingsSample
- Patricia L.Sample
As a non technical manager I finally have a way to question proposals. Our team had planned a fine tune for what turned out to be a retrieval problem.
- Zainab H.Sample
Writing 40 test questions first sounded boring and was the most useful thing we did. Free and better than paid courses I have tried.
- Andrei V.Sample
The instruction, knowledge, behaviour split is simple and it works. I would have liked one more worked case with a combined approach.
About the teacher
Fine tuning with judgment: when to do it, how to do it well, and how to know it worked
9 tutors 428 lessons taught Sample
I teach fine tuning and post training: choosing between prompting, retrieval and tuning, building datasets, parameter efficient methods, instruction and preference tuning, and evaluating the result. My background is in applied machine learning projects where the expensive mistake was usually tuning a model before anyone had defined what better meant. That is why I start every topic with the evaluation...
See Neha's profile and tutorsMore like this
Other tutors on the same or nearby topics.