GPUs and AI hardware for beginners
Understand what GPUs do for AI and estimate whether a model will fit on your hardware
A taste of a lesson
My laptop GPU has 8 GB of memory. Can I run a 13 billion parameter model?
Let us estimate. At 16 bit, each parameter takes 2 bytes, so 13 billion parameters need about 26 GB: far too much. At 8 bit it is about 13 GB, still too much. At 4 bit it is about 6.5 GB, and once you add overhead for the runtime and the context cache you are right at the edge of 8 GB, so it may only work with short conversations, or with some layers offloaded to slower system memory. A 7 billion parameter model at 4 bit would fit comfortably. Your turn: how much memory would a 3 billion parameter model need at 8 bit, before overhead?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Explain why GPUs suit neural networks better than CPUs
- Estimate the memory needed to run a model at different precisions
- Explain why training needs many times more memory than inference
- Distinguish memory capacity from memory bandwidth and why both matter
- Read a hardware spec sheet critically and ask the right questions
Lesson plan
- 1 CPUs, GPUs and the arithmetic of AI Understand why neural network maths runs so well on many simple parallel cores. Start
- 2 Will it fit? Estimating model memory Calculate the memory a model needs from its parameter count and precision. Start
- 3 Capacity versus bandwidth See why reading weights quickly often limits text generation speed. Start
- 4 Why training needs so much more Estimate training memory and understand the options when it does not fit. Start
- 5 The wider hardware landscape Recognise other accelerators and how machines with several GPUs work together. Start
- 6 Cloud or local, and reading spec sheets Weigh renting against owning and read headline numbers with scepticism. Start
Try asking
About this tutor
For beginners who keep hearing that AI needs GPUs and want to understand why, and what to look for. You will learn how a GPU differs from a CPU, why neural networks are mostly the kind of arithmetic GPUs are built for, and why memory capacity and memory bandwidth often matter more than raw speed. You practise estimating how much memory a model needs to run and to train, see why training needs many times more than running, and compare cloud and local options in general terms. You finish able to read a hardware spec sheet with healthy scepticism. No maths beyond multiplication.
Reviews
4.7
3 ratingsSample
- Adaeze I.Sample
I finally understand why my model generates slowly even though the GPU looks barely busy. The bandwidth lesson was the one.
- Ethan K.Sample
Clear and refreshingly free of product hype. I would have liked a little more on multi GPU setups, but it is a beginner course after all.
- Carmen V.Sample
The parameters times bytes rule answered every question I came with. I stopped downloading models that could never fit on my machine.
About the teacher
Making models fast, small and affordable: hardware, quantisation, serving and edge
9 tutors 355 lessons taught Sample
I teach the engineering side of running models: what GPUs actually do, how memory limits shape every decision, how to quantise or distil a model, and how to serve it efficiently on a server or a small device. I have spent my working life close to hardware, first on embedded systems and later on inference infrastructure, so I tend to...
See Magnus's profile and tutorsMore like this
Other tutors on the same or nearby topics.