Sorting comments into themes with an AI tool

Start with a copy. Never let a tool work on your only copy. Ever.

Then sort in rounds, not in one big request. Keep each round small. Check it before you move on to the next.

Where the idea comes from

Researchers who study written answers use a method called coding: you tag each comment with a short label. One description calls coding "the primary process for developing themes".¹

An AI tool does a fast, rough version of this. Your job is to act like the careful researcher who looks over its work.

The method is used on small and large sets alike. Data can range from a short reply to one open-ended survey question up to hundreds of pages.¹

Round one: get draft themes

Put the comments in one plain list, one comment per line. Give each a number so you can find it later. Then ask the tool for a draft list of themes, each with a one-line meaning.

Tell it what you want, in simple words:

• Give each theme a short name.
• Say what the theme means in one sentence.
• Say which comment numbers fit it.
• Say "none of these" when a comment doesn't fit anywhere.

That last instruction matters. Without it, a tool tends to squeeze every comment into some pile.

Round two: read what's under each theme

Open each theme and read the comments the tool put there. You needn't read every one. Read enough to judge whether the label is fair to the people who wrote them.

Look for three problems:

1. A theme that mixes two ideas, such as "price" and "value for money".
2. Two themes that mean the same thing.
3. Comments sitting in the wrong theme.

A quick example. A hairdresser's comments might produce "wait times" and "booking problems". Open both and you may find the same complaint: people couldn't get a slot. Merge them.

Rules that keep the list usable

A good set of themes follows a few plain rules. One guide to coding says "one code should apply to only one category and categories should be comprehensive".² In everyday terms, each comment has a clear home, and nothing is left without one.

Some comments do say two things. Let those sit in two themes on purpose, and note that you did.

The same guide says the work usually means "reducing the themes to a manageable number".² A list of forty themes is a sorting job you haven't finished. Aim for a list you could say out loud to your staff in a minute.

Round three: check and fix

Ask the tool to show the exact words behind each theme. Then find those words in your original file. If a quoted line isn't there, the tool invented it. An invented answer presented as fact is what researchers call a hallucination.³

The simplest hand method still works. One guide describes "highlighting different concepts with different colours".² If you have a spreadsheet, colour a sample of rows yourself and see whether you agree with the tool.

Keep a record

Keep a short note. It should hold these.

• The date and the name of the tool.
• The instructions you gave it.
• The final themes and what each one means.
• Any comments you moved by hand.

Next month you'll have forgotten why "delivery" and "packaging" were kept apart, and the note will remind you in a few seconds. Simple.

Once the list is ready, you'll want to know how many people said each thing. That's a separate skill, and the counting page explains it.