Sequential Testing and the Peeking Problem
Monitor experiments early and often without inflating your false positive rate
A taste of a lesson
We use a Bayesian tool and stop when 'probability to beat control' passes 95 percent. That avoids peeking issues, right?
Not automatically. The posterior is a valid summary of belief at any moment, but a rule like 'stop the first time it passes 95 percent' can still declare winners far more often than you expect when there is no real difference. How often depends on your prior, threshold and how frequently you check. The honest test is to simulate A/A experiments with your exact stopping rule and count false wins. Also expect early stops to overstate the effect. How many times per test does your team typically check?
Written by the teacher as an example. In your lesson the tutor answers your own questions, and like any AI it can be wrong.
What you will be able to do
- Explain why repeated peeking inflates false positives
- Design a group sequential test with alpha spending boundaries
- Use always valid intervals and know their trade offs
- Assess Bayesian monitoring rules by simulation
- Write a team policy for interim looks, harm and futility
Lesson plan
- 1 The peeking problem Quantify how repeated looks inflate false positives. Start
- 2 Group sequential designs Plan interim analyses with adjusted boundaries. Start
- 3 Alpha spending Use spending functions to allow flexible look timing. Start
- 4 Always valid inference Use confidence sequences valid at any stopping time. Start
- 5 Bayesian monitoring Evaluate Bayesian stopping rules honestly. Start
- 6 A team policy Turn methods into rules people can follow. Start
Try asking
About this tutor
An advanced tutor for experimentation specialists and analysts who need to look at test results before the planned end. You will see, through simulation reasoning, exactly how repeated peeking inflates false positives, then learn the main ways to monitor safely: group sequential designs with alpha spending, always valid inference and Bayesian monitoring with honest error properties. Lessons cover stopping for harm, stopping for futility, practical policies for teams and how to explain all of this to stakeholders who want answers by Friday.
Reviews
Students can review a tutor after a paid lesson. Nobody has yet.
About the teacher
Experiments, causal questions and responsible models, explained for decision makers
9 tutors 437 lessons taught Sample
I help analysts and product people answer the question behind most data work: did this change cause that result? I teach A/B testing, experiment design and the basics of causal inference, plus the responsible side of modelling: fairness checks and explaining predictions. My background is in product analytics and experimentation, where I learned that a clear sentence to a decision...
See Kojo's profile and tutorsMore like this
Other tutors on the same or nearby topics.