Playtesting
Learning paths through this topic
Did testers solve the mystery from the clues, or by guessing the culprit?
A correct accusation looks like success, but a lucky one teaches you nothing about your clues. This entry separates solved-by-reasoning from solved-by-luck, finds the clue that people miss most, and spots the testers who had the clues and still failed.
My new feature looked great in week one, then the effect faded away
People play more at first with anything new, so early numbers flatter a change. This entry shows how to line players up by how long they have known the change instead of by calendar date, and how to tell a lasting gain from a short thrill.
Match a watcher's scribbled notes to what happened in the game
Someone sits next to the tester with a stopwatch and writes "gasped at 2:05". Later you want to know which scene that was. This entry fixes the clocks that never start together and decides when a note belongs to an event and when it belongs to nothing.
Events sent after a phone was offline arrive in the wrong order
A phone that played on a train sends its log all at once later, so the order of arrival tells you nothing about the order of play. This entry shows what to put on every event, how to line them up again and how to spot holes.
Coming back after a night gives a different result than playing that night live
A game that skips time with a quick formula often disagrees with what the same hours would have produced tick by tick, usually in the favour of nobody. This entry shows how to make the skip use the very same rules as live play and still stay fast.
After an update some players' saved progress looks wrong: find out how many
Every change to what you store leaves old saves behind, and some end up with values the game never expected. This entry shows a short list of rules every save must follow, and a count that tells which old version of the save is the one producing broken players.
After my update one level of thirty looks worse. Is it real or just chance?
Compare enough levels and one of them will always look suspicious, even when nothing changed. This entry shows how likely that false alarm is, how many you should expect, and a step-by-step rule that keeps the flags honest without hiding a real problem.
One stuck player tries a level forty times and my fail rate shoots up
If every attempt counts the same, a single stubborn player can outweigh everybody else on a level. This entry shows how to limit what one person can contribute, and which extra number tells you whether the level is hard for many or deadly for a few.
One tap on a phone sometimes counts as two moves in my puzzle
Screens and fingers produce a quick second touch, and in a move-counting puzzle that costs players their star. This entry shows how to read the right waiting time out of your own tap logs instead of guessing it, and how to apply it so a run of touches is not wiped out.
Testers rate the whole story by its best moment and its last one: check it
People do not average their feelings; they remember a few moments. If you can find out which moment drives the overall verdict of your testers, you know which scene to polish first. This entry compares peak, last scene and plain average.
How many players do I need to see whether my level tweak helped?
A small improvement needs far more players to show up than most people expect, and the players who never reach the changed level do not count at all. This entry gives a way to work out the number before you start and shows why half the effect costs four times the crowd.
Offline rewards pay out too much because some players change their phone clock
A game that pays for time away trusts the phone's clock, and a clock can be pushed forward or back by hand. This entry shows how to compare two clocks to tell a real absence from a trick, what to pay in each case and why a restart blinds the check.