Compare Two Versions: Is the Difference Real or Luck?
Give each new player one of the two versions at random, decide how many you need before you start, and look only at the end. Then ask how often luck alone would give a gap as big as yours, because a number that moved is not yet a number that changed.
How do you tell a real gap from luck?
Pudding Rally is an invented racing game, and these numbers are made up for this example. Its maker thought the first boss race was too hard, so he made a gentler track. Version A is the old one, version B the new one. Each new session got one at random. Some call this an A/B test.
| Won the race | Share | |
|---|---|---|
| Version A | 88 of 200 | 44% |
| Version B | 104 of 210 | 50% |
B is ahead by 5.5 points, and that looks like a win. So ask the one useful question: if A and B were really the same, how often would luck alone give a gap this big? For this table the answer is about 28% of the time, more than one try in four. So this could easily be luck. The real gap is very likely between minus 4 and plus 15 points, which includes "made it slightly worse".
The usual bar is 1 in 20, or 5%. The maths here is Fisher's exact test, which also works for small groups.
Note what the result is not. It does not say the two tracks are the same. It says this much data cannot tell. To see a gap of this size, he would need about 1,280 players per version.
So he kept going, to that planned number. With 1,100 players each, he got 484 wins against 561: 44% and 51%. Now luck would give a gap this big less than 1% of the time, and the real gap is very likely between 3 and 11 points. That is a real improvement, and a small one.
How many players does a change need?
The size of the change decides, and small gaps are expensive.
| From | To | Players per version |
|---|---|---|
| 50% | 80% | 39 |
| 50% | 70% | 93 |
| 50% | 60% | 388 |
| 50% | 55% | 1,565 |
| 50% | 52% | 9,806 |
So in a small game, only test big, bold changes.
Which rules protect you?
- Random, not before and after. If last week's players got A and this week's got B, you are comparing the weeks as well.
- Fix the number first, then do not peek. Evan Miller works through what happens if you check a test after each visitor and stop at the first good-looking result. In his worst case, the false alarm rate reached 26.1% and not the 5% you think you have.
- Change one thing. With two changes, you learn nothing about either.
- Pick one number. Check twenty at once and about one will look real by luck. If you must check several, divide the 5% by how many.
- Check the split. 300 players on A and 180 on B is not chance. Something is broken, so fix that first.
One honest limit: with anonymous data, one person who plays twice can land in both groups. For a small game that usually matters little, but say so if a result is close. The comparison recipe runs all of this. For the very first look at a change, five people is often enough.
Paste this to your AI
Help me find out if a change to my game really helped. The change: [what]. The number that should move: [for example: share who win the boss].
Before any data:
1. Draw both versions as a Mermaid state diagram, from a new session getting a version at random to the result. Mark the one thing that differs.
2. Write down what should happen, and how big a gap would be worth it. Use get_recipe with compare-two-versions if Spielerando is connected, and tell me how many players that needs per version. If that is more than I can get, say so now.
3. Add a random pick of version for each new session, and write the version into the event log. Keep the log anonymous.
Then wait until that number is reached. Do not show me results before that, even if I ask.
At the end, show both counts, the split check, the gap, its 95% range and how often luck alone would give it. If it could be luck, say so, and do not call that a win or a proof that nothing changed.
Or skip this and connect the MCP: your AI then finds all of this by itself.Connect the MCP
How to check it worked
- Check the split first. Both groups should be about the same size.
- Redo the counts by hand from the log, for one version.
- Look for a second change that slipped in with the same build.