Fairness Checks for ML Models
Measure how a model treats different groups and understand the trade offs between fairness metrics
A taste of a lesson
Our loan model doesn't use ethnicity at all. Doesn't that mean it can't be biased?
Unfortunately not. Other features, such as postcode, employer or shopping patterns, can be strongly linked to ethnicity and carry the same signal. Historical labels can also reflect past unfair decisions. And without the attribute, you cannot even measure disparities. A better approach is to keep ethnicity available for auditing, with proper governance, and compare approval rates and error rates across groups. Your organisation's legal team should advise on what is permitted where you operate. Which group metrics could you compute with the data you have now?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Define groups, outcomes and error costs for a fairness review
- Compute selection rates and error rates by group
- Explain the main fairness criteria and why they can conflict
- Identify proxies, label bias and small group uncertainty
- Document findings and mitigation options for reviewers
Lesson plan
- 1 Framing a fairness review Define the decision, affected people, groups and error costs. Start
- 2 Group metrics Compute selection, error and calibration metrics per group. Start
- 3 Fairness criteria Understand demographic parity, equal opportunity, equalised odds and calibration. Start
- 4 Proxies and biased labels Find hidden routes for bias in features and labels. Start
- 5 Mitigation options Compare interventions before, during and after training. Start
- 6 Documenting and monitoring Record findings and plan ongoing checks. Start
Try asking
About this tutor
An intermediate tutor for data scientists, analysts and reviewers who need to check machine learning models for unfair outcomes. You will define groups and outcomes, compute selection rates and error rates by group, and learn the main fairness criteria, why they can conflict, and what that means for real decisions. Lessons cover proxies for protected attributes, small group uncertainty, intersectional groups, mitigation options before, during and after training, and documentation. Legal requirements differ by country and sector, so the tutor teaches methods and defers legal questions to qualified advisers.
Reviews
4.7
3 ratingsSample
- Peter J.Sample
Practical and honest about trade offs. I would have liked a longer section on intersectional groups.
- Akosua D.Sample
Helped me prepare a fairness review for our risk committee. Clear about deferring legal points, which I appreciated.
- Yasmin E.Sample
The impossibility discussion was explained with a simple example instead of maths. It changed how our team talks about fairness metrics.
About the teacher
Experiments, causal questions and responsible models, explained for decision makers
9 tutors 437 lessons taught Sample
I help analysts and product people answer the question behind most data work: did this change cause that result? I teach A/B testing, experiment design and the basics of causal inference, plus the responsible side of modelling: fairness checks and explaining predictions. My background is in product analytics and experimentation, where I learned that a clear sentence to a decision...
See Kojo's profile and tutorsMore like this
Other tutors on the same or nearby topics.