Lukas Vermeer's manual sample ratio mismatch (SRM) cheker

To manually check for a sample ratio mismatch (SRM) head over to Lukas Vermeer's sample ratio mismatch checker at LukasVermeer.nl. Here's how to use it:

1. Plug in the ratio you wanted your sample split into. By default the checker is pre-populated with .5 (aka 50%) for the "Expected ratio #1" and "Expected ratio 2" form fields. This means the intended split was 50/50. This is the most common split ratio, but it's also possible to choose any other ratio that makes sense in your situation.
2. Go to your A/B testing tool and get the sample size for both the control and the treatment groups. You can usually find these numbers by looking for something labeled "visitors," "users," or "participants."
3. Plug in the sample size numbers you observe in your A/B testing tool. These numbers go in the fields labeled "Observed sample #1" for your control and "Observed sample #2" for your treatment.
4. Look at what the colored banner below the fields says. The checker works on-the-fly, so there's no need to push any buttons or do anything else.

How to read the manual SRM check result

If the banner is green and says, "No indication of sample ratio mismatch with p = ..." then there is no sample ratio mismatch in your data. The figure below illustrates what a result without an SRM looks like.

If the banner is yellow and says, "Warning. Possible sample ratio mismatch detected with p < 0.0001. The observed sample size in treatment does not match the expected proportion of the total sample size. This is an indicator for randomisation failure or missing data issues. Comparative statistics may be invalid as a result."

This simply means you're like to have an SRM in your data, and you should not trust the result.