Designing Experiments: Power, Units and Blocking
Plan experiments that can actually detect the effects you care about
A taste of a lesson
Baseline conversion is 10 percent. I want to detect a lift to 10.5 percent. How many users per group?
Use the quick approximation for 80 percent power and alpha 0.05: n per group is about 16 times variance divided by the effect squared. Variance for a proportion is 0.1 times 0.9, so 0.09. The effect is 0.005, squared is 0.000025. So 16 times 0.09 divided by 0.000025 gives about 57,600 users per group. That is a lot, which is typical for small lifts. Check: how many eligible users do you get per week, and how many weeks would this take?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Choose a randomisation unit and analyse at the matching level
- Calculate sample size and minimum detectable effect
- Reduce variance with blocking and pre period covariates
- Recognise interference and use cluster or switchback designs
- Write a pre registration that fixes analysis choices
Lesson plan
- 1 The randomisation unit Pick the unit that matches the change and the metric. Start
- 2 Power and minimum detectable effect Calculate the sample needed for a meaningful effect. Start
- 3 Reducing noise Use blocking and covariates to gain power without more users. Start
- 4 Interference and clusters Handle designs where units affect one another. Start
- 5 Several changes at once Use factorial designs and understand interactions. Start
- 6 Pre registration and review Write and critique a complete experiment plan. Start
Try asking
About this tutor
An intermediate tutor for analysts and product teams who have run basic A/B tests and want to design better ones. You will choose the right randomisation unit, handle interference between users, calculate power and minimum detectable effects, and use blocking, stratification and pre period covariates to reduce noise. Lessons also cover factorial designs for testing several changes at once, cluster randomised tests and how to write a pre registration so analysis choices are fixed in advance. You finish able to review an experiment plan and spot why it would fail before it runs.
Reviews
4.7
3 ratingsSample
- Asha K.Sample
Pre period covariates reduced our test durations noticeably. Clear explanations of why it works.
- Bruno F.Sample
Strong on interference, which matters for our marketplace. The factorial lesson was a little quick.
- Ola N.Sample
The 16 times variance rule is now taped to my monitor. We stopped planning tests that could never detect anything.
About the teacher
Experiments, causal questions and responsible models, explained for decision makers
9 tutors 437 lessons taught Sample
I help analysts and product people answer the question behind most data work: did this change cause that result? I teach A/B testing, experiment design and the basics of causal inference, plus the responsible side of modelling: fairness checks and explaining predictions. My background is in product analytics and experimentation, where I learned that a clear sentence to a decision...
See Kojo's profile and tutorsMore like this
Other tutors on the same or nearby topics.