A few weeks ago our AI-graded chat reports had a problem. Every morning, the report flagged a handful of “missed sales opportunities” in our live chats. And every morning, our sales team looked at them and said: no, that wasn’t an opportunity. That customer already had a partner. That question was purely technical.
The AI wasn’t stupid. It was confidently wrong in the same way every day, because nothing told it otherwise.
That’s the moment I understood the most important design rule for AI reports in marketing: an analysis nobody can contradict repeats its mistakes every day.
Thumbs down, with a reason
The fix was almost embarrassingly simple. Every AI grade in our reports now has a thumbs up and a thumbs down. The thumbs down has one condition: you have to write a reason. One sentence is enough. “Customer already works with a partner.” “This was a support question, not a sales lead.”
The next day’s run reads those reasons before it grades anything. The mistakes don’t disappear overnight, but they stop repeating. And, just as important, the sales team stopped ignoring the report, because they now had a way to shape it.
Two things I learned about this:
- The reason matters more than the vote. A thumbs down on its own tells the AI that it was wrong, but not why. Without the why, it just gets more cautious everywhere.
- Feedback is also a trust signal. People trust a report they can argue with far more than a polished one they can’t.
Don’t let the AI invent categories
The second lesson came from topic labels. We categorise every chat and ticket by topic, so we can see what customers ask about most. For one run, I gave the AI sub-agents a list of example topics to help them along.
They took it as inspiration. We got back fifteen new, invented labels, all slightly different, that I then had to map back to our real categories by hand. A trend chart built on labels that change from run to run is worthless.
The rule now: the labels are a closed list. If a conversation doesn’t fit, the sub-agent answers “other” and adds a suggestion. At the end of the run, the main session looks at all suggestions at once and decides whether a new label is worth adding. Of the first eight suggestions, three became real labels. The other five were variations of existing ones.
If you want to compare numbers over time, the AI may fill in categories, but it must not define them.
Decline instead of guess
We also built a documentation bot that answers product questions from our knowledge base. Before letting it near customers, we tested it on 200 real questions from support tickets and had another model judge the answers.
The result was sobering and useful. Only a small share of answers were fully correct. Most were partially right: helpful for pointing someone to the right article, but incomplete. A few percent were wrong, and one was a proper hallucination: a command that doesn’t exist.
We didn’t give up on the bot. We changed its job. It’s now a documentation navigator, not a support replacement. And we added a confidence gate: if the retrieved documentation doesn’t clearly cover the question, the bot says it doesn’t know and hands over. An honest “I don’t know” is worth more than a fluent wrong answer.
The same principle applies to our analysis tools. One of our rules for the documentation gap report: never claim something is “not documented” without searching for it with at least two different phrasings. Customers and documentation writers rarely use the same words.
Stop, don’t invent
Many of our reports run automatically every morning, with nobody watching. That makes one rule non-negotiable: if data is missing or a connection fails, the run stops and says so. It does not fill in the gaps with plausible numbers.
This sounds obvious. It isn’t. A language model’s instinct is to be helpful, and “helpful” with missing data means making something up. You have to tell it, explicitly and in writing, that an aborted report is better than an invented one.
Keep the evidence next to the verdict
The last pattern is the simplest. Every grade, every flagged ticket, every content idea in our tools links back to the source: the full chat transcript, the actual ticket, the real Instagram comments an idea was based on. Anyone can click through and check.
This does two things. It makes mistakes visible fast. And it keeps humans in the habit of looking at the real conversations, which is where the actual insight is anyway.
A checklist for your own AI reports
If you’re building AI-assisted reports in marketing, here’s what I’d put in from day one:
- A thumbs down that requires a reason, and a next run that reads it.
- Closed lists for anything you want to count over time. “Other plus a suggestion” instead of invented labels.
- A confidence gate. Decline instead of guess.
- Abort on missing data. Written into the instructions, not assumed.
- Evidence next to every verdict. A link to the source, always.
None of this is about better models. It’s about making the AI’s work something people can check, argue with and improve. That’s what turns an impressive demo into a tool a team actually relies on.