Schap

Tag: Content

  • Measure the answer, not the question

    Measure the answer, not the question

    Every company sits on a huge pile of unread market research. It’s called the support inbox. Every live chat, every ticket, every email is a customer telling you in their own words what they don’t understand, what they want, and where your product or your content lets them down.

    Marketing rarely reads it. Not because nobody cares, but because nobody has time to read thousands of conversations. With AI, that excuse is gone. An agent can read every conversation from yesterday before you’ve had your coffee.

    We’ve been doing this for a few weeks now. Here’s what we learned.

    The trick: look at what support had to explain

    Our first idea was obvious: take the customer questions, check whether our knowledge base answers them, and list the gaps. It didn’t work well. Customers ask vague questions in their own words. “It doesn’t work anymore” doesn’t map to any article.

    The breakthrough was to flip it around. Don’t measure the question. Measure the answer. Look at what the support agent had to write to solve the case. If support had to explain something in detail, in writing, that explanation is exactly what’s missing from the documentation.

    With that change, the gap analysis became useful overnight. For every closed ticket, the tool compares the support agent’s answer with our knowledge base. It uses plain keyword search on a local copy of all articles, which is fast and needs no AI at all. Where there’s a real gap, it drafts the text to add. The documentation team marks each finding as open, adopted or dismissed.

    One rule keeps it honest: a topic only counts as “not documented” if the search came up empty with at least two different phrasings.

    Documentation gaps found in support answers, with suggested text
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    Finding 1: speed was never the problem

    When we started grading our live chats, I expected slow response times. Everybody complains about waiting in chat.

    Wrong. When someone picked up a chat, they picked it up within seconds. The problem was that too many chats weren’t picked up at all. It was about coverage, not speed.

    That’s a completely different problem with a completely different fix: shift planning, not training. Once it was visible every morning, the share of chats that got answered went up noticeably within a few weeks. Nobody had to be told to hurry up. The number just had to be on the table.

    Finding 2: the silence after the first reply

    We saw the same pattern in support tickets. We started by reading a random sample of a hundred closed tickets, to see the reality before building anything. First responses were fast. But a meaningful share of tickets were closed without a real answer to the customer, and some sat silent for almost two weeks in the middle of the conversation.

    The speed was right. The silence afterwards wasn’t.

    So we built a small watch list: open tickets where the last message is from the customer and nobody has replied in days. It’s not an analysis, it’s a to-do list. It’s also one of the most-used pages we have.

    Ticket watch: open tickets where the customer spoke last
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    Finding 3: “answered” doesn’t mean answered

    In our social media analysis we wanted to know how well we answer comments and direct messages. The analytics tool said: almost all of them.

    When we looked closer, most of those “answers” were automatic replies sent within a minute. A human had never looked at them. We now treat any brand reply within sixty seconds as an auto-reply and list the people who are still waiting for a real one.

    The lesson generalises: whenever a tool gives you a suspiciously good number, check how it was counted.

    Why this is marketing’s job

    You could argue this is all customer service. It isn’t only that. What customers ask in support is what they will search for before they buy. What support has to explain is what your website, your product pages and your content don’t explain. Where customers get stuck is where your messaging makes a promise the product experience doesn’t keep.

    It’s also the best content briefing you’ll ever get. Every repeated support explanation is a blog post, a video or a help article waiting to be written, and it comes with the customer’s exact wording.

    How to start

    1. Read a sample yourself first. Pick a hundred random conversations and read them. You’ll know what to measure afterwards, and you’ll recognise when the AI gets it wrong.
    2. Keep the full conversation next to every AI verdict. People need to be able to check.
    3. Measure the answer, not the question, if you’re looking for content gaps.
    4. Turn findings into lists, not charts. “These twelve customers are waiting” is more useful than a trend line.
    5. Be suspicious of good numbers. Check how they’re counted.

    Your customers are already telling you what to fix and what to write. The only new thing is that you can finally afford to listen to all of them.

  • Why chatbots rot

    Why chatbots rot

    A few years ago we had a support chatbot. It was expensive, it was built by a specialist vendor, and on launch day it worked well. A year later, people avoided it.

    Nothing had broken, technically. The bot still answered every question, quickly and politely. The problem was that more and more of the answers were wrong. Products had changed, processes had changed, the documentation had moved on. The bot hadn’t.

    That experience sets the bar for every chatbot we’ve built since. And it taught me the most important thing I know about them: a chatbot is not a technology project. It’s a content maintenance problem dressed up as one.

    How bots rot

    Classic chatbots are built from scripted answers. Someone writes a list of questions and the matching responses, the vendor trains a model to recognise the questions, and off it goes.

    From that day on, every change in your business creates a small gap. A new product version. A new return process. A price change. A feature that got renamed. Each gap is tiny. Nobody owns closing them, because the bot was a project and the project is finished.

    Six months later, the bot is confidently telling customers things that were true last spring. The customers notice before you do.

    What’s different with LLMs, and what isn’t

    Modern language models change one thing fundamentally: you don’t have to script answers anymore. You can point the bot at your documentation, and it answers from there. When the documentation changes, the answers change with it.

    That sounds like the rot problem is solved. It’s only moved.

    The bot is now exactly as good as the documentation it reads. If the docs are outdated, incomplete or written in words your customers don’t use, the bot will be too, just more fluently. Freshness of your knowledge base becomes the real KPI of your chatbot.

    What we do differently this time

    When we replaced the old vendor bot with a language-model-based assistant this year, we set a few rules.

    Human first. If someone from the team is online, the customer gets a person. The bot only answers when nobody is available, for example at night or on weekends. It’s a safety net, not a gatekeeper. We even deliberately launched one support channel without any AI at all, because the people using it needed a human more than an instant answer.

    One bot, one job. A bot that helps people learn the product and a bot that answers pre-sales questions need different sources, different tone and different boundaries. We keep them separate and give each its own name, so customers and the team know which one they’re talking to.

    Decline instead of guess. Before going live, we tested the documentation bot against 200 real support questions. Most answers were partially right, a few were wrong, and one was a proper hallucination. So the bot now has a confidence gate: if the documentation doesn’t clearly cover a question, it says so and hands over. I wrote more about this in Make your AI contradictable.

    The documentation bot answers with a source, and hands over when it isn't sure
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    Speak the customer’s language. Customers describe symptoms; documentation describes features. When we added the documentation’s own vocabulary to each search, the number of customer phrasings the bot could answer roughly doubled in our tests. That’s not a model improvement. It’s a content improvement.

    Every answer can be rated. Thumbs up, thumbs down, with a reason. The ratings don’t just improve the bot, they point at the articles that need work.

    The failure nobody expects

    One more story, because it’s the kind of thing you only learn by running a bot in production.

    On one of its first days, our bot suddenly made hundreds of calls to the chat system within seconds. The cause was a single configuration detail: the bot’s “I can’t help, let me forward you” answer was placed in a spot where the chat system treated it as a new question. The bot answered its own forwarding message, which triggered another forwarding message, and so on.

    Nothing bad happened to customers, and the fix was one line. But it’s a good reminder: a bot is a system that talks to other systems, and those systems have their own logic. Watch it closely in the first weeks.

    Who owns the bot?

    This is the question that decides whether a bot rots.

    The wrong answer is “the person who built it”. That person will move on to the next project, and the bot becomes an orphan.

    The right answer is “the team whose knowledge it serves”. In our case that’s the people who write and maintain the documentation, together with support. They see the thumbs down. They see which questions the bot declined. They fix the articles, and the bot gets better without anyone touching the bot itself.

    A checklist before you launch a bot

    1. Who keeps the knowledge current? Name a team, not a person.
    2. Human first or bot first? Decide deliberately, per channel.
    3. One job per bot. Separate bots for separate purposes.
    4. A confidence gate. Decline instead of guess.
    5. Ratings with reasons, routed to the people who own the content.
    6. Watch the first weeks closely. Bots talking to systems do surprising things.

    A chatbot doesn’t rot because the technology gets worse. It rots because nobody feels responsible for what it knows. Solve that, and the technology is the easy part.