How Many Players Do You Need for Game Data?
Five people are enough to find what is broken, about 30 give you a rough number, and about 100 pin a share down to ten points either way. Every time you want to be twice as exact, you need four times the players.
Why is 6 out of 10 not the same as 60%?
Because a small group can land at 60% by luck. Pocket Comet is an invented arcade game, and these numbers are made up for this example. In every row below, 60% of players beat the first boss. Only the number of players changes.
| Beat the boss | Share | The true share is very likely between |
|---|---|---|
| 3 of 5 | 60% | 23% and 88% |
| 6 of 10 | 60% | 31% and 83% |
| 18 of 30 | 60% | 42% and 75% |
| 60 of 100 | 60% | 50% and 69% |
| 240 of 400 | 60% | 55% and 65% |
| 600 of 1,000 | 60% | 57% and 63% |
The last column is what we call a sure range: the true share sits inside it about 19 times out of 20. The method is the Wilson score interval, which the NIST statistics handbook presents as a recommended way to do this. You never have to do this sum yourself.
Look at what each row lets you decide. With 10 players, the boss could be brutal or a walkover. With 30, you know it is not a walkover. With 100, you can say it sits around the middle, which is enough to act on in a small game. Going from 100 to 400 costs four times the players and only halves the range.
How many do you need for what?
| What you want to know | Players |
|---|---|
| What is badly broken or confusing | 5, and you watch them |
| Roughly how a level does | About 30 who reach it |
| A share to about 10 points either way | About 100 |
| A share to about 5 points either way | About 400 |
| Whether 60% became 70% after a change | About 360 per version |
Remember who you are counting. If 100 start the game and 20 reach level 6, level 6 has 20 players, not 100. So the end of a game always has the thinnest numbers.
Rare events need a special word of caution. If nobody out of 30 found the secret door, that does not mean nobody ever will. A simple rule called the rule of three says the true rate could still be up to about 3 divided by the number of tries: here 3 out of 30, so about 1 in 10. Our range above gives 11% for the same case, so the two agree closely.
What if you only have 12 players?
Then be honest and use them for the thing small numbers are good at. Watch them play and collect problems. Do not write "58% finish" in your notes, because the next 12 could easily say 40% or 75%. A big gap can still show up. If 11 of 12 fail the same jump, that is very likely real, because the range for 1 of 12 only reaches about 35%.
Paste this to your AI
Before we read any numbers from my game, tell me how far they can be trusted.
1. Draw my game's path from start to finish as a Mermaid state diagram, with a count of real sessions at each point.
2. For each point, give the share that carried on and its 95% range. Use the Wilson score interval, and show hits and total next to it, such as "41 of 100".
3. Mark each point: enough to act on, rough, or too thin to say anything. Under 30 is too thin.
4. For the question I care about, [for example: is level 4 harder than level 3], tell me how many players I would need, and how many I have.
Never give me a bare percentage without the two numbers behind it. If the data cannot answer my question yet, say so plainly and suggest what I could learn from watching a few people instead.
Or skip this and connect the MCP: your AI then finds all of this by itself.Connect the MCP
How to check it worked
- Look for the counts. Every percentage your AI gives you should come with "x of y".
- Check one range by hand. Paste 41 and 100 into any online Wilson calculator. You should get about 32% to 51%.
- Split your players in two. Do both halves say about the same? If not, you do not have enough yet.
- Add a week of data. A trustworthy number moves little. A number that jumps was never settled.