Playtesting
Learning paths through this topic
Did testers notice that their choice mattered, or only say so to be nice?
Ask "did your choice change something?" and almost everybody says yes. This entry adds one trick to the question so the answer means something, and then tells you for each choice whether the consequence is invisible, missing, or really felt.
Which choices are agonising and which ones have an obvious answer?
A choice that everybody answers the same way, in a second, is not a choice. A choice that takes forever might be a great dilemma or just badly worded. This entry tells them apart and removes the time people spent simply reading the options.
"It wasn't scary" is bad news, "it wasn't confusing" is good: sorting tester comments
Counting words in tester comments goes wrong at "not". The same little word turns a compliment into a complaint depending on what you wanted the player to feel. This entry sorts comments into problems and praise by looking at the feeling you aimed for.
My daily puzzle is solved faster later in the day: is the answer leaking?
When everyone gets the same puzzle, the answer travels. This entry shows how to look at solve times by how long the puzzle has been out, how to tell a leak from simply different players arriving later, and when to treat the day's numbers as spoiled.
Which scenes are so dark that testers reach for the brightness setting?
Dark is scary until the player cannot see the door. When testers fight the brightness slider, they are telling you a scene went too far. This entry counts the people, not the clicks, and ignores those who turned it up and straight back down.
Who really came back the next day: late-night installs fool the count
Counting a player as returned the next day sounds simple, until someone installs at ten to midnight and plays again twenty minutes later. This entry shows two honest ways to define a return day, when each one fits, and why time zones matter for one of them.
Day-seven return of my new version looks terrible, but half the players are too new
Comparing how many players came back after a week is unfair to a version that is only a few days old, because its players have not had the chance yet. This entry shows which players to leave out of the count and how to read the two versions side by side.
Which lines of dialogue did testers click away without reading?
Plot hints hidden in text nobody reads are wasted. This entry tells you which lines were skimmed by most testers, and how to avoid blaming a line for a tester who simply went to make coffee or for short lines that cannot be judged at all.
Median clear time looks shorter in version B, but the times are very uneven
Times to finish a level are lopsided, with a few very long ones, so the usual tests for averages fit badly. This entry shows how to ask the question about the typical time directly, by reshuffling your own numbers, and how to read the answer as a range.
Version B finished 8 points better than A: ship it, or wait for more players?
Comparing two versions of a level or a screen looks simple until you ask how many players you needed and whether looking early fooled you. This entry gives a straight answer on whether the difference is real, and how many players to wait for.
Testers keep finding the same endings: how likely is a new one next time?
After many sessions you know which endings were seen, but not whether more are lurking out of reach. This entry estimates the chance that the next tester sees something nobody has seen before, and lists the endings that still lack a single witness.
One tester rates every scene a five and another a two: whose scores count?
People use the rating scale in very different ways, so averaging their numbers mostly measures their habits. This entry rescales each tester against themselves before it ranks the scenes, and shows what to do with the tester who never changes their answer.