How to check an AI summary of customer feedback

A summary reads smoothly whether it's right or wrong. So don't judge it by how it sounds. Judge it by whether you can find its claims in the raw comments.

Why a check is needed

An Australian regulator's guide to commercial AI products warns that AI systems can produce inaccurate or false results.¹ The guide is about privacy. The warning still fits any summary.

The usual name for a confident false output is a hallucination.² Chatbots built on large language models can mix falsehoods into otherwise sensible text.²

In feedback work, that shows up as an invented complaint, a quote nobody wrote, or a theme that sounds right but has no comments behind it.

Check one: find the claim in the comments

Take each claim in the summary, one at a time. Say it reads "customers find the website confusing". Search your raw file for words such as "website", "site", "online", "app" or "login". Read what comes up.

Do the comments say what the summary says? Or something nearby? "Couldn't find the opening hours" is not the same as "the website is confusing", though a tool might blur them.

Check two: ask for the evidence, then verify it

Ask the tool to list the comment numbers behind each claim. Then open those numbers yourself.

• If the numbers don't match the claim, the claim isn't supported.
• If the tool gives quotes, search for each quote in your file word for word.
• If a quote isn't there, delete it from the summary.

Never paste a "customer quote" into a report until you've found it in the original.

Check three: read the other side

Tools tend to find what's easy. They can skip a small theme with strong feelings, or flatten a split view into "mixed".

Pick a random handful of comments that the summary never mentions. Read them. If several say something the summary leaves out, the summary is missing a theme.

A quick way to pick: choose every tenth row of your spreadsheet. Don't pick the ones that catch your eye, since those tend to be the loudest.

Check four: a second reader

Research teams often use more than one person to sort the same data. One guide notes that some studies use several coders who work separately on the same data.³

You can borrow the idea cheaply. Ask a colleague to sort twenty comments without seeing the tool's answer. Then compare. If you disagree about many of them, the themes need clearer definitions.

Check five: watch the wording

Look for words that promise more than the comments do:

• "Most customers" when only a handful said it.
• "Always" and "never" where comments say "sometimes".
• "Customers are happy" built from a few polite remarks.
• A cause given as fact, such as "sales fell because of the queue", when the comments only mention a queue.

A comment tells you what one person said. It rarely proves why something happened.

Keep a person in the loop

The same guide tells businesses to consider how human oversight can be built into their processes.¹ For feedback, that means a named person signs off the summary before it goes to anyone else.

Write the sign-off down: who checked, on what date, which claims, and what changed. Keep the original comments next to the summary so anyone can go back to them.

Numbers need their own care. A tool may count wrongly even when the themes are right. The counting page deals with that. And if the themes themselves are shaky, go back to the sorting page.