A/B Testing for Beginners
Run your first A/B test properly, from hypothesis to a decision you can defend
A taste of a lesson
After two days, version B has 12 percent more signups and my tool says 'significant'. Can I ship it?
I would wait. Early results swing a lot, and if you check every day and stop the first time it says significant, your real false positive rate is much higher than 5 percent. Two days also misses weekly patterns, since weekday and weekend users often behave differently. Stick to the sample size and duration you planned, ideally at least one full week. Meanwhile, check the split is close to 50/50. Did you set a planned sample size before starting, and if so, what was it?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Write a testable hypothesis with one primary metric and guardrails
- Estimate whether you have enough traffic for the effect you care about
- Randomise users correctly and run tests for full weeks
- Read results with a confidence interval rather than a single number
- Avoid early stopping, novelty effects and sample ratio mismatch
Lesson plan
- 1 Hypothesis and decision Write a clear hypothesis tied to a decision. Start
- 2 Metrics and guardrails Pick a primary metric and guardrails that protect against harm. Start
- 3 How many users you need Estimate sample size from the effect you want to detect. Start
- 4 Running the test Randomise properly and run for full weekly cycles. Start
- 5 Reading the result Interpret the estimated difference and its interval. Start
- 6 The decision note Record and share the outcome so the team learns. Start
Try asking
About this tutor
A beginner tutor for marketers, product people, founders and new analysts who want to test changes instead of guessing. You will write a clear hypothesis, choose one primary metric and a few guardrails, decide sample size before starting, randomise correctly and run the test for full weeks. Lessons cover reading results with confidence intervals, common traps such as stopping early, novelty effects and broken randomisation, and how to write a short decision note. Examples include email subject lines, pricing pages and onboarding screens. No advanced maths needed.
Reviews
4.7
3 ratingsSample
- Ana P.Sample
Exactly the right level for a marketer. No heavy maths, but nothing important skipped.
- Chloe D.Sample
I run a small online shop and this helped me realise my traffic was too low for tiny tweaks. Now I test bigger changes and write decision notes.
- Ibrahim S.Sample
Clear and friendly. The sample ratio mismatch check caught a tracking bug on our site.
About the teacher
Experiments, causal questions and responsible models, explained for decision makers
9 tutors 437 lessons taught Sample
I help analysts and product people answer the question behind most data work: did this change cause that result? I teach A/B testing, experiment design and the basics of causal inference, plus the responsible side of modelling: fairness checks and explaining predictions. My background is in product analytics and experimentation, where I learned that a clear sentence to a decision...
See Kojo's profile and tutorsMore like this
Other tutors on the same or nearby topics.