Read Playtest Feedback With AI, Without Fooling Yourself
Have your AI sort playtest comments by theme, count people and not comments, and quote each one word for word so nothing can be invented. Then check what people said against what your log shows they did, because the two often differ.
What goes wrong when an AI summarises feedback?
It tells a smooth story. Paste 46 comments and ask "what do players think?", and you will get a tidy paragraph that sounds true and that you cannot check. So the prompt below makes it show its work.
Goblin Post Office is an invented game, and these comments and numbers are made up for this example. 19 playtesters left 46 comments. The first pass sorted them like this.
| Theme | Comments | Different people |
|---|---|---|
| Too hard | 14 | 6 |
| Love the goblins | 12 | 9 |
| Too long | 7 | 5 |
| Wants multiplayer | 5 | 1 |
| Other | 8 | 7 |
Watch what happens when you read across. "Wants multiplayer" has five comments and looks like a demand. It is one enthusiastic person. Meanwhile "love the goblins" comes from 9 of the 19, the widest agreement in the list. Count people, or the loudest voice wins.
Now open the biggest theme. "Too hard" sounds like a reason to make the game easier. But with each comment quoted in full, 9 of the 14 turn out to be about one rope bridge in level 3: "the bridge wobbles and I can't see my feet". That is not a difficulty problem. It is a camera problem on one screen, and a much smaller job.
Why compare comments with the log?
Because what people say and what they do are different things. Usability researcher Jakob Nielsen put it in three rules: watch what people actually do, do not believe what they say they do, and definitely do not believe what they predict they may do later.
Two checks on this game, with its anonymous event log:
| They said | The log says | Reading |
|---|---|---|
| "Level 4 goes on forever" (5 people) | Middle time 11 minutes, plan was 4 | True, and measured |
| "I always used the map" (6 people) | 2 of 19 sessions opened it | Not what happened |
The first is confirmed, so it goes on the list to fix. The second tells you people believe they use a map they barely open. If the map matters, it has a problem, and praise will not reveal it.
A log cannot tell you how anything felt, though. "Love the goblins" appears nowhere in it, and it may be the best thing you learn all week. Keep the praise: it tells you what not to break.
With 19 people, "6 people said it's too hard" is 32%, but the real share could be 15% to 54%. So use themes to find what to look at, and do not quote them as percentages.
Before you paste anything into an AI, remove names, e-mail addresses and handles. The comments are about the game, not about who wrote them.
What should you do with the result?
- Fix what several people hit, and what the log confirms.
- Treat requests as symptoms. "Add multiplayer" may mean "I got lonely in level 2".
- Go and watch. Comments lead to the next five-person test.
Paste this to your AI
I am pasting playtest comments, with a made-up label for each person such as P1. Names are already removed.
[paste comments]
My event log is in [file or folder].
1. Before sorting anything, draw what I intended players to do as a Mermaid state diagram, from reading the game.
2. Give every comment a number. Sort them into themes, and let a comment sit in more than one.
3. Show a table of themes, with comments and different people. Sort by people.
4. Under each theme, quote every comment word for word, with its number. Do not reword them, and never add a quote that is not in my text.
5. Inside each theme, look for a smaller, exact cause, such as one place in the game.
6. For each claim the log could settle, compare it with the log and with the diagram: what they said, what happened, the counts behind it.
Mark each finding: confirmed by the log, contradicted by the log, or opinion only. Do not write percentages for fewer than 30 people. Tell me what you cannot tell from this data, and do not summarise beyond what the quotes show.
Or skip this and connect the MCP: your AI then finds all of this by itself.Connect the MCP
How to check it worked
- Pick three quotes at random and find them in your file.
- Add up the table. Do the comment numbers cover everything you pasted?
- Check one log claim yourself. Open the file and count.
- Read the "other" pile. Good surprises hide there.