Schap

Tag: AI Agents

  • The Lab: why marketing should build its own tools

    The Lab: why marketing should build its own tools

    For most of my career, getting a new tool in marketing looked like this: write a requirements document, wait for an IT slot, wait for a budget, get something six months later that almost does what you needed. Or buy a SaaS product that does 80% of it and makes the remaining 20% impossible.

    A month ago we tried something else. We started what we call the Lab: an internal portal where the marketing and sales teams build their own tools with AI coding agents. None of us are developers. I know basic HTML and CSS; my colleagues know about as much.

    Four weeks in, the Lab has a daily chat quality report, a documentation gap finder, a market metrics page, a watch list for customer tickets that went quiet, a social media analyser, and a coaching tool for our seminar venue. Most of these took a day or two to get to a first useful version.

    This post is about how that works, and why the most important ingredient is not the AI model.

    The agent is the easy part

    Modern coding agents are very good at writing code. That’s not the bottleneck anymore. The bottleneck is that the agent knows nothing about your business. It doesn’t know what a “qualified lead” means in your company, which CRM field holds the partner tier, why the Instagram numbers from last quarter look strange, or that your support team closes tickets in a particular way.

    Give an agent a vague task and no context, and you get a generic dashboard that looks great and answers the wrong question. Give it the right context, and it builds something that feels like it was made by someone who has worked with you for years.

    So the real work in the Lab is not prompting. It’s building context.

    Three kinds of context

    1. What we know: plain Markdown files. Everything the agents need to understand lives as text files. I keep my own knowledge in an Obsidian vault: who we are, how we talk about our products, brand guidelines written for AI agents instead of designers, definitions of our metrics. Meeting summaries flow into the vault automatically every evening.

    Every project repository has one entry file, AGENTS.md, that every agent reads first, whether it’s Claude Code, Codex or something else. Next to it is a docs/ folder with decisions and lessons, each with a date and a reason. When an agent or a human learns something non-obvious, it gets written there. Not in a chat. Not in one tool’s memory. In the repo, where the next agent will find it.

    One rule turned out to matter a lot: every definition lives in exactly one place. When I cleaned up my vault in August, I found our brand colours defined in six places and one process described in nine. Agents will happily pick whichever version they find first.

    2. What’s happening: live connections to our systems. Context also means data. Through MCP servers (the “USB ports” that let AI agents talk to other software) and APIs, the agents can read our CRM, our help desk, our live chat, our social media analytics and our product database. A tool that shows “active partners per market” doesn’t use an export from last month. It queries the CRM when it runs.

    That’s also how one of our most useful pages came about. The same metric was circulating in slides with three different definitions. Now there’s one definition, written down once, and the tool calculates it live. The discussion moved from “whose number is right” to “what do we do about it”.

    3. What went wrong: lessons written into the tools. Every skill and every tool carries its own list of traps. For example: “never let an AI sub-agent pull tickets from the help desk, the filters get lost and you get a million results instead of a few hundred.” These notes look boring. They’re the reason the second run of a tool is better than the first.

    What we’ve actually built

    A few examples, to make this concrete:

    • Chat reports. Every live chat from the day before is read and graded by an AI: was it answered, how long did the customer wait, was a sales opportunity missed, was the customer sent away too early. The full transcript sits next to the grade, so anyone can check.
    • Documentation gaps. Closed support tickets are compared with our public knowledge base. Whatever support had to explain in writing is probably missing from the docs. The tool proposes the text to add.
    • Social analyser. KPIs across our brand accounts, sentiment for every comment and DM, a list of people still waiting for an answer, and content ideas that link back to the real posts and comments they came from. A marketing colleague built most of it, not me.
    • Ticket watch. Open tickets where a customer is waiting and nobody has replied in days.
    • Venue coaching. Enquiries for our seminar venue that didn’t turn into bookings, why, and what they would have been worth.

    None of these are sophisticated software. They are small, specific, and built by the people who use them. That’s the point.

    Chat report: AI-graded chats with the transcript next to the grade
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    How teams stay independent (from IT, and from each other)

    Each team gets its own zone in the portal: its own small app, its own path, its own deploy. If marketing breaks something, it breaks marketing’s zone, not the whole portal. A new colleague gets an invitation and a “copy prompt” button. The prompt tells their own AI agent to clone the project, read AGENTS.md and work through the onboarding checklist. The knowledge stays in the repo; the prompt only points there.

    Everything runs on Cloudflare with a login in front, so nothing is public and nobody had to build a login system. Viewers just need a company email address.

    We looked at the alternatives. A BI tool with per-user pricing and no way to enter data. A platform product for multi-tenant apps that cost more than it was worth for three teams. Separate repositories per team. We ended up with the simplest option that works: one repository, several zones, agreement instead of enforcement.

    Is this a threat to IT?

    No. IT still runs the systems of record, security and everything that has to be rock-solid. What changes is the long tail: the hundreds of small reports, checks and helpers that were never important enough for an IT project but that make a real difference to a marketing team’s week.

    For those, the old answer was “live without it” or “build it in Excel”. The new answer is: build it yourself, and give your agent the context it needs.

    Where to start

    If you want to try this in your team:

    1. Write down what you know. Start with one Markdown file: who you are, what your metrics mean, what your tone of voice is. Keep every definition in one place.
    2. Connect one system. Pick the one you look at most (CRM, help desk, social analytics) and connect it through an MCP server or API.
    3. Build one small tool for one real question. Not a dashboard. A question, like “which customers are waiting for an answer right now?”
    4. Write down what went wrong, with a date. That file will become your most valuable asset.

    The tools are the visible part. The context is what you’re really building.

    Social analyser: people still waiting for a real answer, with a drafted reply
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.
  • LLMs read your comment section

    LLMs read your comment section

    Earlier this year I had a conversation with people from a large software company that stuck with me. Their observation: when you ask an AI assistant about a brand, the answer leans surprisingly heavily on public sentiment. Forum threads, review sites, the comments under social posts. Not the carefully written product pages. Not the press releases.

    Think about what that means for a moment. For years, a critical comment under an Instagram post was a small customer service issue. Someone is annoyed, maybe you reply, maybe you don’t, and a week later nobody remembers it.

    Now that comment is training material for how machines describe you.

    The public picture is skewed

    Here’s the uncomfortable part. Most brands with a loyal community have a lopsided public footprint:

    • The happy customers talk in closed places. Partner portals, private groups, internal forums, direct conversations with their contact person. Lots of goodwill, almost none of it visible to a crawler.
    • The unhappy customers talk in public. A review site, a comment under your latest video, a thread in an open forum. That’s where people go when they feel they aren’t being heard anywhere else.

    So the public picture is often worse than reality. And the public picture is the one that AI assistants see.

    You can’t fix that with more ads or a better About page. You fix it where it happens: in the comments.

    Social listening becomes narrative work

    Social listening used to be a reporting job. Count mentions, measure sentiment, put a chart in the monthly deck. Useful, but passive.

    In the AI age, it becomes narrative work. The question isn’t just “how do people feel about us?” but “what story does the public record tell about us, and are we part of that story?”

    That changes what a good reply looks like.

    The canned reply makes it worse. “We’re sorry to hear that, please contact our support team.” Everyone has seen this reply a hundred times. It tells the reader, and any machine reading along, that the problem is still unsolved and the brand didn’t engage with it.

    The specific reply changes the record. Answer the actual problem, in public, in a few sentences. If it’s a known issue, say what the fix is. If it needs a conversation, say who will call and when. The next person with the same problem finds the answer right there, and so does the AI.

    Then close the loop. When someone’s problem has really been solved, it’s fair to ask whether they’d update their review or add a comment. Many will, because they were never angry at the brand, they were angry at being ignored.

    Where AI agents help

    This is a lot of work if you do it by hand across several accounts and platforms. It’s exactly the kind of work where agents shine, as long as a human stays in charge.

    Here’s the setup I’ve been building up over the last few months:

    1. A weekly sentiment run. An agent pulls comments and messages from all brand accounts, scores the sentiment of each one and groups them by topic.
    2. A “still waiting” list. Everyone who asked a question or complained and hasn’t had a real answer yet. Not a chart. A list of people. One lesson here: most “answered” messages in our analytics turned out to be automatic replies within seconds. We now treat those as unanswered.
    3. Drafted replies. For each open item, the agent drafts a reply that follows our own guidelines: tone of voice, what we say about known issues, when to hand over to support. The drafts are starting points, not autopilot.
    4. A human checks and posts. Always. The agent doesn’t have the context to know whether this customer already talked to someone yesterday, and it shouldn’t speak for the brand on its own.
    5. Actively collect good reviews. A few new reviews every week from customers who are clearly happy, so the public picture isn’t defined only by the loudest few.

    None of this is sophisticated technology. It’s a routine, and the agent makes the routine cheap enough to actually keep up.

    Community view: who is still waiting, and a drafted reply for a human to check
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    What not to do

    A few things I’d avoid:

    • Don’t let the agent post on its own. One badly judged automatic reply to an angry customer, in public, undoes a lot of careful work.
    • Don’t argue in public. If a review is unfair, state the facts once, calmly, and offer to talk. The reader decides who looks reasonable.
    • Don’t fake it. Paid or invented reviews are not a shortcut. They’re a liability, for people and machines alike.
    • Don’t confuse volume with coverage. Ten fast replies to easy questions don’t make up for one unanswered complaint that sits there for a month.

    The short version

    AI assistants learn about your brand from what’s public. What’s public is often skewed towards the unhappy few, because the happy many talk elsewhere.

    So treat every public comment as part of your brand’s story. Answer the real problem, in public, like a human. Use agents to find everything that’s waiting and to draft the replies. Keep a person on the send button.

    Every comment is a customer. And, these days, every comment is also a source.

  • Why chatbots rot

    Why chatbots rot

    A few years ago we had a support chatbot. It was expensive, it was built by a specialist vendor, and on launch day it worked well. A year later, people avoided it.

    Nothing had broken, technically. The bot still answered every question, quickly and politely. The problem was that more and more of the answers were wrong. Products had changed, processes had changed, the documentation had moved on. The bot hadn’t.

    That experience sets the bar for every chatbot we’ve built since. And it taught me the most important thing I know about them: a chatbot is not a technology project. It’s a content maintenance problem dressed up as one.

    How bots rot

    Classic chatbots are built from scripted answers. Someone writes a list of questions and the matching responses, the vendor trains a model to recognise the questions, and off it goes.

    From that day on, every change in your business creates a small gap. A new product version. A new return process. A price change. A feature that got renamed. Each gap is tiny. Nobody owns closing them, because the bot was a project and the project is finished.

    Six months later, the bot is confidently telling customers things that were true last spring. The customers notice before you do.

    What’s different with LLMs, and what isn’t

    Modern language models change one thing fundamentally: you don’t have to script answers anymore. You can point the bot at your documentation, and it answers from there. When the documentation changes, the answers change with it.

    That sounds like the rot problem is solved. It’s only moved.

    The bot is now exactly as good as the documentation it reads. If the docs are outdated, incomplete or written in words your customers don’t use, the bot will be too, just more fluently. Freshness of your knowledge base becomes the real KPI of your chatbot.

    What we do differently this time

    When we replaced the old vendor bot with a language-model-based assistant this year, we set a few rules.

    Human first. If someone from the team is online, the customer gets a person. The bot only answers when nobody is available, for example at night or on weekends. It’s a safety net, not a gatekeeper. We even deliberately launched one support channel without any AI at all, because the people using it needed a human more than an instant answer.

    One bot, one job. A bot that helps people learn the product and a bot that answers pre-sales questions need different sources, different tone and different boundaries. We keep them separate and give each its own name, so customers and the team know which one they’re talking to.

    Decline instead of guess. Before going live, we tested the documentation bot against 200 real support questions. Most answers were partially right, a few were wrong, and one was a proper hallucination. So the bot now has a confidence gate: if the documentation doesn’t clearly cover a question, it says so and hands over. I wrote more about this in Make your AI contradictable.

    The documentation bot answers with a source, and hands over when it isn't sure
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    Speak the customer’s language. Customers describe symptoms; documentation describes features. When we added the documentation’s own vocabulary to each search, the number of customer phrasings the bot could answer roughly doubled in our tests. That’s not a model improvement. It’s a content improvement.

    Every answer can be rated. Thumbs up, thumbs down, with a reason. The ratings don’t just improve the bot, they point at the articles that need work.

    The failure nobody expects

    One more story, because it’s the kind of thing you only learn by running a bot in production.

    On one of its first days, our bot suddenly made hundreds of calls to the chat system within seconds. The cause was a single configuration detail: the bot’s “I can’t help, let me forward you” answer was placed in a spot where the chat system treated it as a new question. The bot answered its own forwarding message, which triggered another forwarding message, and so on.

    Nothing bad happened to customers, and the fix was one line. But it’s a good reminder: a bot is a system that talks to other systems, and those systems have their own logic. Watch it closely in the first weeks.

    Who owns the bot?

    This is the question that decides whether a bot rots.

    The wrong answer is “the person who built it”. That person will move on to the next project, and the bot becomes an orphan.

    The right answer is “the team whose knowledge it serves”. In our case that’s the people who write and maintain the documentation, together with support. They see the thumbs down. They see which questions the bot declined. They fix the articles, and the bot gets better without anyone touching the bot itself.

    A checklist before you launch a bot

    1. Who keeps the knowledge current? Name a team, not a person.
    2. Human first or bot first? Decide deliberately, per channel.
    3. One job per bot. Separate bots for separate purposes.
    4. A confidence gate. Decline instead of guess.
    5. Ratings with reasons, routed to the people who own the content.
    6. Watch the first weeks closely. Bots talking to systems do surprising things.

    A chatbot doesn’t rot because the technology gets worse. It rots because nobody feels responsible for what it knows. Solve that, and the technology is the easy part.

  • Make your AI contradictable

    Make your AI contradictable

    A few weeks ago our AI-graded chat reports had a problem. Every morning, the report flagged a handful of “missed sales opportunities” in our live chats. And every morning, our sales team looked at them and said: no, that wasn’t an opportunity. That customer already had a partner. That question was purely technical.

    The AI wasn’t stupid. It was confidently wrong in the same way every day, because nothing told it otherwise.

    That’s the moment I understood the most important design rule for AI reports in marketing: an analysis nobody can contradict repeats its mistakes every day.

    Thumbs down, with a reason

    The fix was almost embarrassingly simple. Every AI grade in our reports now has a thumbs up and a thumbs down. The thumbs down has one condition: you have to write a reason. One sentence is enough. “Customer already works with a partner.” “This was a support question, not a sales lead.”

    The next day’s run reads those reasons before it grades anything. The mistakes don’t disappear overnight, but they stop repeating. And, just as important, the sales team stopped ignoring the report, because they now had a way to shape it.

    Two things I learned about this:

    A graded chat with a thumbs down and a written reason
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.
    • The reason matters more than the vote. A thumbs down on its own tells the AI that it was wrong, but not why. Without the why, it just gets more cautious everywhere.
    • Feedback is also a trust signal. People trust a report they can argue with far more than a polished one they can’t.

    Don’t let the AI invent categories

    The second lesson came from topic labels. We categorise every chat and ticket by topic, so we can see what customers ask about most. For one run, I gave the AI sub-agents a list of example topics to help them along.

    They took it as inspiration. We got back fifteen new, invented labels, all slightly different, that I then had to map back to our real categories by hand. A trend chart built on labels that change from run to run is worthless.

    The rule now: the labels are a closed list. If a conversation doesn’t fit, the sub-agent answers “other” and adds a suggestion. At the end of the run, the main session looks at all suggestions at once and decides whether a new label is worth adding. Of the first eight suggestions, three became real labels. The other five were variations of existing ones.

    If you want to compare numbers over time, the AI may fill in categories, but it must not define them.

    Decline instead of guess

    We also built a documentation bot that answers product questions from our knowledge base. Before letting it near customers, we tested it on 200 real questions from support tickets and had another model judge the answers.

    The result was sobering and useful. Only a small share of answers were fully correct. Most were partially right: helpful for pointing someone to the right article, but incomplete. A few percent were wrong, and one was a proper hallucination: a command that doesn’t exist.

    We didn’t give up on the bot. We changed its job. It’s now a documentation navigator, not a support replacement. And we added a confidence gate: if the retrieved documentation doesn’t clearly cover the question, the bot says it doesn’t know and hands over. An honest “I don’t know” is worth more than a fluent wrong answer.

    A documentation bot that declines and hands over instead of guessing
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    The same principle applies to our analysis tools. One of our rules for the documentation gap report: never claim something is “not documented” without searching for it with at least two different phrasings. Customers and documentation writers rarely use the same words.

    Stop, don’t invent

    Many of our reports run automatically every morning, with nobody watching. That makes one rule non-negotiable: if data is missing or a connection fails, the run stops and says so. It does not fill in the gaps with plausible numbers.

    This sounds obvious. It isn’t. A language model’s instinct is to be helpful, and “helpful” with missing data means making something up. You have to tell it, explicitly and in writing, that an aborted report is better than an invented one.

    Keep the evidence next to the verdict

    The last pattern is the simplest. Every grade, every flagged ticket, every content idea in our tools links back to the source: the full chat transcript, the actual ticket, the real Instagram comments an idea was based on. Anyone can click through and check.

    This does two things. It makes mistakes visible fast. And it keeps humans in the habit of looking at the real conversations, which is where the actual insight is anyway.

    A checklist for your own AI reports

    If you’re building AI-assisted reports in marketing, here’s what I’d put in from day one:

    1. A thumbs down that requires a reason, and a next run that reads it.
    2. Closed lists for anything you want to count over time. “Other plus a suggestion” instead of invented labels.
    3. A confidence gate. Decline instead of guess.
    4. Abort on missing data. Written into the instructions, not assumed.
    5. Evidence next to every verdict. A link to the source, always.

    None of this is about better models. It’s about making the AI’s work something people can check, argue with and improve. That’s what turns an impressive demo into a tool a team actually relies on.

  • Your best-performing post is lying to you

    Your best-performing post is lying to you

    “Which of our posts work best, and what should we do more of?”

    It’s the most natural question to ask an AI about your social media. It’s also a question where the AI can be completely, confidently wrong, and give you recommendations that point in exactly the wrong direction.

    It happened to us. Here’s how, and the rules we now follow.

    The quarter that wasn’t

    When I first had an agent analyse our YouTube and Instagram performance, the results looked great. One quarter stood out as our strongest by far, with average views per video many times higher than any other quarter. The AI’s recommendation: do more of what we did then.

    The problem: that quarter included two big campaign films with serious media budget behind them. They weren’t “performing”. They were paid to be seen. Once we took them out, the average for that quarter dropped by roughly a factor of ten, and it turned out to be our weakest quarter, not our best.

    Every recommendation built on the first version would have been wrong. And it would have looked perfectly plausible, with charts and all.

    Rule 1: separate paid from organic before you rank anything. Not as a footnote, not as a filter you can optionally apply. As the first step, every time. We now call it our iron rule, and it’s written into the analysis instructions so no agent can skip it.

    One outlier makes everything else look bad

    Even after removing campaigns, we hit a second problem. One promoted video had so many views that, compared with it, every normal video looked tiny. In a ranking based on averages, a regular good video looked more than a hundred times worse than the leader. That’s not insight, that’s noise.

    We switched to medians. The median tells you what a typical post does. It doesn’t care about the one post that went viral or got a boost. Suddenly the differences between formats and topics became visible again: which kinds of posts reliably do a bit better, which reliably do a bit worse.

    Rule 2: use the median for “typical”, and look at outliers separately. Outliers are interesting. They just shouldn’t define your baseline.

    Not all “engagement” is the same

    Different platforms count different things. On one network, the analytics tool’s “engagement” included link clicks; on another, it didn’t. Put them side by side and one platform looks far more engaging than the others, for purely technical reasons.

    Rule 3: compare within a platform, not across platforms, unless you’ve checked that the numbers mean the same thing.

    One metric, three definitions

    The same problem exists inside companies. For one of our core business metrics, we found three different definitions in circulation, in three different slide decks. Each one was defensible. Together they meant that every meeting started with a debate about whose number was right.

    We fixed it by writing the definition down once, in a single place, and building a small tool that calculates it live from the CRM. Nobody has to agree with the definition forever. But everyone uses the same one until it’s changed, in that one place.

    Rule 4: every metric has exactly one written definition. Especially when an AI is doing the calculating. It will pick whichever definition it finds first.

    Watch the small print

    A few more traps we’ve run into:

    • Currencies. Comparing partner revenue across countries against a single threshold made partners in countries with a different currency look many times bigger than they were. Always convert before you compare.
    • Auto-replies. “Replied to 95% of messages” can mean “sent an automatic message to 95% of people”. Look at how fast the replies came.
    • Privacy in the data. Some analytics exports include profile image links with access tokens in them. Strip those before anything gets stored or shown.

    Let the AI do the counting, not the thinking

    None of this means AI is bad at analytics. It’s excellent at the tedious parts: pulling data from APIs, cleaning it, scoring the sentiment of thousands of comments, drafting content ideas that link back to the real posts and comments they came from.

    But it has no idea that a post was paid for, that a platform counts differently, or that your business uses a metric in a particular way. It will produce a beautiful, confident report either way.

    So the checklist is short:

    1. Paid out first. Always.
    2. Medians for the baseline, outliers on their own.
    3. Compare like with like.
    4. One definition per metric, written down once.
    5. Read a few of the top posts yourself before you believe the ranking.

    Your best-performing post might really be your best. Just make sure it earned it.