Counting themes without fooling yourself

Don't trust a count that the AI tool typed into a summary. Tools write numbers the same way they write sentences, and a number can be wrong while sounding exact.

Count from the comments yourself. A spreadsheet does it better than a chatbot.

Why the tool's numbers can be off

A chatbot is built to write text, not to run a count over your rows. Ask it "how many people mentioned parking?" and you may get a tidy figure that doesn't match the file.

That is the same problem as any invented answer. A hallucination is a false or misleading reply, given as if it were fact.¹

A quick test: ask the same question twice, or ask in a new chat. If the number changes, don't use either one.

How to count properly

1. Fix the theme list first. Don't change the themes halfway through counting.
2. Give every comment its own row in a spreadsheet, with a number.
3. Add a column for each theme. Mark it yes or blank.
4. Let the tool suggest the marks, then spot-check a sample against the comment text.
5. Count with the spreadsheet's own filter or count function, not with the tool.
6. Report the total number of comments next to each count.

That last point stops a common slip. "Twelve said the wait was long" means something different out of twenty comments than out of five hundred.

A comment can belong in two themes

A comment like "great food, awful parking" counts under both food and parking. Then your theme counts add up to more than the number of comments. That's fine, but say so.

Coding guides stress clear rules for categories. One says each code should apply to just one category, and the categories should cover everything.² The same guide describes simultaneous coding, where two or more codes apply to the same part of the data.²

Decide which approach you're using, and write it at the top of the sheet.

Small numbers jump around

Few comments means shaky counts. A statistics source gives the example of averaging only two or three people. The result would vary wildly from one try to the next.³

It adds that a larger sample can generally shrink the likely error.³

For you, that means a small pile of comments should not be treated like a poll. If six people mention the queue this month and four next month, don't say the problem is shrinking. You don't have enough to say.

The people who answered aren't everyone

This one catches almost everyone. Your comments come from people who chose to write. That is not all of your customers.

A source on participation bias explains that results can become non-representative when the people taking part share traits that affect the outcome.⁴ It also names self-selection bias, where people choose to join a group themselves.⁴

Think of a restaurant's comment box. People with strong feelings, good or bad, fill it in. The quiet majority who had an ordinary night don't. So the counts describe the people who wrote, not all diners.

Words to use in your report

Pick careful wording:

• "Of the comments we received, some mention..." rather than "customers say...".
• "Mentioned in a few comments" rather than "a growing problem".
• The count and the total together, every time.
• A note on how the comments were gathered.

And never write "the AI found". Write what you found, with the tool's help, and that you checked it.

Before you share a number

Go back through this short list:

• Did I count in the spreadsheet?
• Is the total beside the count?
• Did I read a sample of the comments in that theme?
• Have I said who answered, and who may not have?

To check the written summary itself, use the checking page. The sorting page explains how themes get made.