What to measure
Learning paths through this topic
Does the skill rating really predict who wins, or is it guessing?
If a rating says one side should win nine times out of ten, the logs should show close to that. This entry explains how to compare promise and outcome in groups without counting every match twice from both sides.
Why do match-found screens fail: players who decline or players who are away?
When a found match falls apart you need to know whether people said no or simply were not there, because the cures are opposite. This entry shows how to tell the two apart in the logs, including players who changed their mind.
How many players who drop out of a match manage to get back in time?
A shaky connection can drop and return several times in a minute, which makes a naive count look wonderful or terrible. This entry shows how to count whole interruptions and what counts as being back too late.
One angry player reported a teammate twelve times: how many people really complained?
A pile of reports against one player can come from a single person clicking again and again. This entry shows how to count people and matches instead of clicks, so the list of troublemakers shows who really bothers others.
After how much play time do players first run out of bag space?
Bag space shapes how players treat loot, but players who left early never had the chance to fill it. This entry shows how to report the share of players who hit the limit by a given point without counting those who were not around that long.
Do potions before a boss really help, or do only the stronger players carry them?
Players who arrive well stocked tend to win more, but they are also the ones who played longer and grew stronger. This entry shows how to compare like with like before you decide that supplies are the cause.
Is every chapter about as long for everyone, or does one drag on for some players?
An average chapter length hides that half the players finish in twenty minutes and a few need three hours. This entry shows how to measure the spread so the chapters that trap some players stand out.
Is one starting class really harder to finish, or is it just a small group?
A class with fewer finishers might be unbalanced, or might just have too few players to say. This entry shows how to compare finishing rates with an honest margin and without counting newcomers who have not had time.
Which crafting recipes do players ignore even when they could make them?
A recipe nobody makes might be pointless, or it might need materials that players rarely hold. This entry shows how to tell the two apart by looking at the moments when the player really could craft it.
Which dropped weapons and armour do players find but never put on?
Loot that nobody wears is wasted effort, but an item can be ignored simply because it was worse than what the player already had. This entry shows how to count only the cases where the item was really an upgrade.
Which dialogue answers does nobody pick, and which were just rarely seen?
An answer you wrote with care but nobody picks may be dead weight, or it may sit behind a door few players open. This entry shows how to count so the two cases are not mixed up.
Do players pick the top dialogue choice just because it is on top?
If one answer is picked far more than the others, is it the better answer or just the one in the best place? This entry shows what to log and when the data simply cannot tell you.