Open-text survey questions and how AI reads them

The question decides how useful the answers are. A tool can only sort what people wrote, and people write what you ask for.

Open and closed, in one minute

A closed question wants a short answer, such as yes or no. An open-ended question can't be answered that way. It is phrased so that it "requires a longer answer".¹

"Did you enjoy your visit?" is closed. "What could we have done better today?" is open.

Closed answers are easy to count. Open answers carry more detail, but somebody has to read and group them. That's the job an AI tool is used for.

Questions that sort well

Good open questions are narrow. They ask about one thing. Compare these.

The last one is close to a question the open-ended question page lists as an example: "Why did you choose that answer?"¹

A "why" after a rating is useful. You keep the score and the reason together, so you can look at the reasons behind low scores on their own.

Questions that sort badly

Watch for these:

• Two questions in one. "How was the food and the service?" gets answers about both, tangled together.
• Leading wording. "What did you love about us?" pushes people toward praise.
• Very vague prompts. "Anything else?" collects everything, including things that have nothing to do with your business.
• Questions people can't answer from memory. Ask close to the visit or the purchase.

A vague question doesn't break the tool. It just hands you a messier pile to check.

Where the tool struggles with the answers

Short answers lack context. "Fine." could be happy or cross. Sarcasm is worse. A source on sentiment analysis says computer systems have trouble with negations, jokes and sarcasm.²

So "Not bad at all" may be read as negative, because of the word "not". A person gets it at once.

Answers with several topics are another snag. "Loved the staff, hated the parking" belongs in two themes. Decide before you start whether a comment can sit in more than one.

Planning the sort before you ask

Survey researchers talk about when answers get turned into codes. Questionnaire data can be pre-coded, meaning codes are set up before people answer. It can also be post-coded, which is "coding of open questions on completed questionnaires".³

Open text is the post-coded kind. You find the groups after the answers arrive. That's why the tool's draft themes need your review: nobody agreed the groups in advance.

One thing you can do ahead of time is write down the themes you expect, such as price, speed and friendliness. Then compare them with what the tool finds. Surprises are worth a close look.

A short checklist for a new question

1. Does it ask about one thing only?
2. Could a stranger answer it without extra context?
3. Does it avoid hinting at the "right" answer?
4. Will you ask it soon after the experience?
5. Do you know where the answers will be saved, and who can see them?

The last point links to privacy. Free-text boxes invite people to type names, phone numbers and health details you never asked for. Our privacy lesson covers what to do.

Once you have answers, the sorting page walks through the steps.