HackMD
Introduction to Sequential Testing
Advantage of Sequential Testing
Sequential testing is one of the methods that "can reduce the samples of AB testing". Also, it can peek the testing result during the test process. The advantage is clearly two mainly points:
- Reducing the number of samples and speeding up the process of experiments
- Peeking the testing results during testing process.
Suitable Scenarios for Sequential Testing
However, after I tried sequential testing once, I found that there are two scenarios suitable for sequential testing:
- You are certianly sure that treatment group is better (worse) than control group, but you still need to do the experiments to make sure that thing.
- Continuing Monitoring the control in order to avoid deviating from "normal".
I recommend to use sequential testing when you face one of the two scenarios. Besides these two scenarios, I will recommend methods other than sequantial testing.
Different Types of Sequential Testing
Actually, there are lots of testing methods under the group of sequential testing. I can simply put the methods into two types.
- You need to decide how many times to want to peek the testing results before the number of samples reaches the threshold before testing. e.g. Group Sequential Designs.
- keep peeking during the experiment. e.g. Sequential Probability Ratio Test
For the first type, you decide the number of peeking beforehand. Lots of way to choose when to peek, but the most simple way is to set peeking time points evenly during experiment process. Although first type of sequential testing cannot always peek the results like second type, it has simpler setting and larger power.
For the second type, you do not need to decide any number before experiments, and you can peek anytime you want. Therefore, it usually needs continuing data or steaming data to support this method. Also, the setting is more complcated than first ones.
Sequential Testing Methods
Simple Sequential Testing - Evan Miller
Evan Millar developed a simple sequential testing from random walk. The proof and the method is truly simple, so I recommend people who never used sequential testing before can try it.
resource: https://www.evanmiller.org/sequential-ab-testing.html
SPRT - Sequential Probability Ratio Test
Developed by Wald. Actually, this is just like sequential likelihood ratio test. The more description in Wiki.
resource: https://en.wikipedia.org/wiki/Sequential_probability_ratio_test
mSPRT - Mixture Sequential Probability Ratio Test
Since SPRT can only test simple hypothesis, that is, whether a series larger (smaller) than constant, mSPRT is developed to enable multiple tests without inflating the false positive rate (FPR).
more detailed: https://docs.statsig.com/experiments-plus/sequential-testing
originally, I want to show my project with sequential testing. However, since the data is more complex than I thought, I need more time to clean it up. The article written about example of sequential testing will or will not show up in the future, all depends on my lazziness.
