Gradient Boosting for Tabular Data
Understand, tune and debug boosted trees, the workhorse of structured data problems
A taste of a lesson
My boosted model hits its best validation score at tree 80 with learning rate 0.3. Should I just ship that?
Probably not yet. Stopping that early with a high learning rate often means each tree takes big, noisy steps, so the model is sensitive to which rows landed in validation. Try a lower rate, say 0.05, with early stopping and a generous patience; expect several hundred trees. Compare validation loss and its spread across folds, not just one split. Also confirm early stopping used validation data, never the test set. What depth are your trees, and are you subsampling rows or columns?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Explain boosting as fitting trees to the negative gradient of a loss
- Tune learning rate, tree count and depth with early stopping
- Use subsampling and leaf penalties to control overfitting
- Handle categorical features and missing values without leakage
- Decide when boosting is the wrong tool for a dataset
Lesson plan
- 1 Boosting as fitting residuals Build intuition by boosting a regression problem by hand on squared error. Start
- 2 From residuals to gradients Generalise boosting to any differentiable loss. Start
- 3 Learning rate, trees and early stopping Choose tree count with early stopping and set the learning rate deliberately. Start
- 4 Tree shape and randomness Control complexity with depth, leaves, subsampling and penalties. Start
- 5 Categories, missing values and constraints Prepare features correctly and add domain constraints. Start
- 6 Diagnosing and choosing Debug a boosted model and judge when another model family is better. Start
Try asking
About this tutor
An advanced tutor for practitioners who already use tree models and want a deep, practical grasp of gradient boosting. You will see boosting as gradient descent in function space, then work through learning rate and tree count trade offs, early stopping, tree complexity, subsampling and regularisation. Lessons cover categorical features, missing values, monotonic constraints, custom losses and how boosting overfits differently from forests. You will also learn when boosting is not the best choice. The tutor stays library neutral and explains concepts that carry across the popular implementations.
Reviews
4.7
3 ratingsSample
- Yuki T.Sample
Caught a leak in my target encoding during the categorical lesson. Worth it for that alone.
- Rafael D.Sample
Exactly the level I needed. The residuals to gradients walkthrough finally explained why custom losses need a second derivative in some libraries.
- Amara B.Sample
Dense but excellent. I would have liked a worked example with a quantile loss, which was only briefly covered.
About the teacher
Classical machine learning models, worked through on paper before any code
9 tutors 379 lessons taught Sample
I teach the classical machine learning models: regression, trees, ensembles, nearest neighbours, support vector machines and naive Bayes. My background is in applied analytics, where I spent a lot of time explaining to colleagues why a model made a particular prediction, and that shaped how I teach. I like to start with a tiny dataset you can hold in your...
See Kavya's profile and tutorsMore like this
Other tutors on the same or nearby topics.