Running a language model on your own computer
Work out what your machine can run, set up a local model, and know what to expect from it
A taste of a lesson
I have a laptop with 16 GB of memory and no separate graphics card. What can I run?
Let's do the arithmetic. At 4 bit, weights take about half a byte per parameter, so a 7 or 8 billion parameter model needs roughly 4 to 5 GB, plus a GB or two for the runtime and context. That fits comfortably. If your laptop has unified memory, the built in GPU can use it and speed is reasonable; on CPU alone expect slower replies. A 14 billion model at 4 bit, around 8 to 9 GB, may fit but leaves less room. Exercise: estimate a 3 billion model at 8 bit.
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Estimate the memory a model needs from parameter count, precision and context length
- Explain quantisation and its effect on size, speed and quality
- Match a model size to your hardware and choose a suitable kind of local tool
- Set realistic expectations and handle licences and safe downloads
Lesson plan
- 1 The memory arithmetic Estimate how much memory a model needs. Start
- 2 Quantisation Understand how lower precision shrinks models and what it costs. Start
- 3 Hardware and speed See how GPUs, unified memory and CPUs affect what runs and how fast. Start
- 4 Kinds of local tools Choose between desktop apps, command line runners and local servers. Start
- 5 Realistic expectations and responsibilities Know what local models do well and what you must manage yourself. Start
Try asking
About this tutor
For curious users, privacy minded professionals and developers who want to run a language model locally instead of through a cloud service. You learn how to estimate the memory a model needs from its parameter count and precision, what quantisation does to size and quality, how GPUs, unified memory and CPUs differ, and why speed is measured in tokens per second. You compare the general kinds of local tools, from simple desktop apps to command line runners and local servers, without depending on any one product. You also learn realistic expectations: smaller local models can be very useful but are usually less capable than the largest hosted ones, and you remain responsible for updates and licences.
Reviews
4.3
4 ratingsSample
- Megan F.Sample
Useful, but my old laptop could only run tiny models and results were weak. The tutor did warn me, to be fair.
- Tariq Y.Sample
Finally understood why my model slowed down with long documents: the KV cache. Excellent explanation of quantisation trade offs.
- Stefan J.Sample
The memory arithmetic made everything make sense. I now run a small model locally for client notes that cannot leave my machine.
- Priya D.Sample
Practical and honest about limits. I wanted specific app recommendations, but the categories were enough to find one.
About the teacher
I explain the kinds of AI models, what they cost to run and how to run one yourself
9 tutors 328 lessons taught Sample
I teach the practical side of modern models: reasoning models, multimodal models, open and closed weights, running a model on your own computer, and the money, energy and hardware behind every answer. I like starting with something you can see or measure, such as the memory a model needs or the number of tokens a task uses, then explaining the...
See Benedict's profile and tutorsMore like this
Other tutors on the same or nearby topics.