Schap

Category: AI Workflows

  • The Lab: why marketing should build its own tools

    The Lab: why marketing should build its own tools

    For most of my career, getting a new tool in marketing looked like this: write a requirements document, wait for an IT slot, wait for a budget, get something six months later that almost does what you needed. Or buy a SaaS product that does 80% of it and makes the remaining 20% impossible.

    A month ago we tried something else. We started what we call the Lab: an internal portal where the marketing and sales teams build their own tools with AI coding agents. None of us are developers. I know basic HTML and CSS; my colleagues know about as much.

    Four weeks in, the Lab has a daily chat quality report, a documentation gap finder, a market metrics page, a watch list for customer tickets that went quiet, a social media analyser, and a coaching tool for our seminar venue. Most of these took a day or two to get to a first useful version.

    This post is about how that works, and why the most important ingredient is not the AI model.

    The agent is the easy part

    Modern coding agents are very good at writing code. That’s not the bottleneck anymore. The bottleneck is that the agent knows nothing about your business. It doesn’t know what a “qualified lead” means in your company, which CRM field holds the partner tier, why the Instagram numbers from last quarter look strange, or that your support team closes tickets in a particular way.

    Give an agent a vague task and no context, and you get a generic dashboard that looks great and answers the wrong question. Give it the right context, and it builds something that feels like it was made by someone who has worked with you for years.

    So the real work in the Lab is not prompting. It’s building context.

    Three kinds of context

    1. What we know: plain Markdown files. Everything the agents need to understand lives as text files. I keep my own knowledge in an Obsidian vault: who we are, how we talk about our products, brand guidelines written for AI agents instead of designers, definitions of our metrics. Meeting summaries flow into the vault automatically every evening.

    Every project repository has one entry file, AGENTS.md, that every agent reads first, whether it’s Claude Code, Codex or something else. Next to it is a docs/ folder with decisions and lessons, each with a date and a reason. When an agent or a human learns something non-obvious, it gets written there. Not in a chat. Not in one tool’s memory. In the repo, where the next agent will find it.

    One rule turned out to matter a lot: every definition lives in exactly one place. When I cleaned up my vault in August, I found our brand colours defined in six places and one process described in nine. Agents will happily pick whichever version they find first.

    2. What’s happening: live connections to our systems. Context also means data. Through MCP servers (the “USB ports” that let AI agents talk to other software) and APIs, the agents can read our CRM, our help desk, our live chat, our social media analytics and our product database. A tool that shows “active partners per market” doesn’t use an export from last month. It queries the CRM when it runs.

    That’s also how one of our most useful pages came about. The same metric was circulating in slides with three different definitions. Now there’s one definition, written down once, and the tool calculates it live. The discussion moved from “whose number is right” to “what do we do about it”.

    3. What went wrong: lessons written into the tools. Every skill and every tool carries its own list of traps. For example: “never let an AI sub-agent pull tickets from the help desk, the filters get lost and you get a million results instead of a few hundred.” These notes look boring. They’re the reason the second run of a tool is better than the first.

    What we’ve actually built

    A few examples, to make this concrete:

    • Chat reports. Every live chat from the day before is read and graded by an AI: was it answered, how long did the customer wait, was a sales opportunity missed, was the customer sent away too early. The full transcript sits next to the grade, so anyone can check.
    • Documentation gaps. Closed support tickets are compared with our public knowledge base. Whatever support had to explain in writing is probably missing from the docs. The tool proposes the text to add.
    • Social analyser. KPIs across our brand accounts, sentiment for every comment and DM, a list of people still waiting for an answer, and content ideas that link back to the real posts and comments they came from. A marketing colleague built most of it, not me.
    • Ticket watch. Open tickets where a customer is waiting and nobody has replied in days.
    • Venue coaching. Enquiries for our seminar venue that didn’t turn into bookings, why, and what they would have been worth.

    None of these are sophisticated software. They are small, specific, and built by the people who use them. That’s the point.

    Chat report: AI-graded chats with the transcript next to the grade
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    How teams stay independent (from IT, and from each other)

    Each team gets its own zone in the portal: its own small app, its own path, its own deploy. If marketing breaks something, it breaks marketing’s zone, not the whole portal. A new colleague gets an invitation and a “copy prompt” button. The prompt tells their own AI agent to clone the project, read AGENTS.md and work through the onboarding checklist. The knowledge stays in the repo; the prompt only points there.

    Everything runs on Cloudflare with a login in front, so nothing is public and nobody had to build a login system. Viewers just need a company email address.

    We looked at the alternatives. A BI tool with per-user pricing and no way to enter data. A platform product for multi-tenant apps that cost more than it was worth for three teams. Separate repositories per team. We ended up with the simplest option that works: one repository, several zones, agreement instead of enforcement.

    Is this a threat to IT?

    No. IT still runs the systems of record, security and everything that has to be rock-solid. What changes is the long tail: the hundreds of small reports, checks and helpers that were never important enough for an IT project but that make a real difference to a marketing team’s week.

    For those, the old answer was “live without it” or “build it in Excel”. The new answer is: build it yourself, and give your agent the context it needs.

    Where to start

    If you want to try this in your team:

    1. Write down what you know. Start with one Markdown file: who you are, what your metrics mean, what your tone of voice is. Keep every definition in one place.
    2. Connect one system. Pick the one you look at most (CRM, help desk, social analytics) and connect it through an MCP server or API.
    3. Build one small tool for one real question. Not a dashboard. A question, like “which customers are waiting for an answer right now?”
    4. Write down what went wrong, with a date. That file will become your most valuable asset.

    The tools are the visible part. The context is what you’re really building.

    Social analyser: people still waiting for a real answer, with a drafted reply
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.
  • Why chatbots rot

    Why chatbots rot

    A few years ago we had a support chatbot. It was expensive, it was built by a specialist vendor, and on launch day it worked well. A year later, people avoided it.

    Nothing had broken, technically. The bot still answered every question, quickly and politely. The problem was that more and more of the answers were wrong. Products had changed, processes had changed, the documentation had moved on. The bot hadn’t.

    That experience sets the bar for every chatbot we’ve built since. And it taught me the most important thing I know about them: a chatbot is not a technology project. It’s a content maintenance problem dressed up as one.

    How bots rot

    Classic chatbots are built from scripted answers. Someone writes a list of questions and the matching responses, the vendor trains a model to recognise the questions, and off it goes.

    From that day on, every change in your business creates a small gap. A new product version. A new return process. A price change. A feature that got renamed. Each gap is tiny. Nobody owns closing them, because the bot was a project and the project is finished.

    Six months later, the bot is confidently telling customers things that were true last spring. The customers notice before you do.

    What’s different with LLMs, and what isn’t

    Modern language models change one thing fundamentally: you don’t have to script answers anymore. You can point the bot at your documentation, and it answers from there. When the documentation changes, the answers change with it.

    That sounds like the rot problem is solved. It’s only moved.

    The bot is now exactly as good as the documentation it reads. If the docs are outdated, incomplete or written in words your customers don’t use, the bot will be too, just more fluently. Freshness of your knowledge base becomes the real KPI of your chatbot.

    What we do differently this time

    When we replaced the old vendor bot with a language-model-based assistant this year, we set a few rules.

    Human first. If someone from the team is online, the customer gets a person. The bot only answers when nobody is available, for example at night or on weekends. It’s a safety net, not a gatekeeper. We even deliberately launched one support channel without any AI at all, because the people using it needed a human more than an instant answer.

    One bot, one job. A bot that helps people learn the product and a bot that answers pre-sales questions need different sources, different tone and different boundaries. We keep them separate and give each its own name, so customers and the team know which one they’re talking to.

    Decline instead of guess. Before going live, we tested the documentation bot against 200 real support questions. Most answers were partially right, a few were wrong, and one was a proper hallucination. So the bot now has a confidence gate: if the documentation doesn’t clearly cover a question, it says so and hands over. I wrote more about this in Make your AI contradictable.

    The documentation bot answers with a source, and hands over when it isn't sure
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    Speak the customer’s language. Customers describe symptoms; documentation describes features. When we added the documentation’s own vocabulary to each search, the number of customer phrasings the bot could answer roughly doubled in our tests. That’s not a model improvement. It’s a content improvement.

    Every answer can be rated. Thumbs up, thumbs down, with a reason. The ratings don’t just improve the bot, they point at the articles that need work.

    The failure nobody expects

    One more story, because it’s the kind of thing you only learn by running a bot in production.

    On one of its first days, our bot suddenly made hundreds of calls to the chat system within seconds. The cause was a single configuration detail: the bot’s “I can’t help, let me forward you” answer was placed in a spot where the chat system treated it as a new question. The bot answered its own forwarding message, which triggered another forwarding message, and so on.

    Nothing bad happened to customers, and the fix was one line. But it’s a good reminder: a bot is a system that talks to other systems, and those systems have their own logic. Watch it closely in the first weeks.

    Who owns the bot?

    This is the question that decides whether a bot rots.

    The wrong answer is “the person who built it”. That person will move on to the next project, and the bot becomes an orphan.

    The right answer is “the team whose knowledge it serves”. In our case that’s the people who write and maintain the documentation, together with support. They see the thumbs down. They see which questions the bot declined. They fix the articles, and the bot gets better without anyone touching the bot itself.

    A checklist before you launch a bot

    1. Who keeps the knowledge current? Name a team, not a person.
    2. Human first or bot first? Decide deliberately, per channel.
    3. One job per bot. Separate bots for separate purposes.
    4. A confidence gate. Decline instead of guess.
    5. Ratings with reasons, routed to the people who own the content.
    6. Watch the first weeks closely. Bots talking to systems do surprising things.

    A chatbot doesn’t rot because the technology gets worse. It rots because nobody feels responsible for what it knows. Solve that, and the technology is the easy part.

  • Make your AI contradictable

    Make your AI contradictable

    A few weeks ago our AI-graded chat reports had a problem. Every morning, the report flagged a handful of “missed sales opportunities” in our live chats. And every morning, our sales team looked at them and said: no, that wasn’t an opportunity. That customer already had a partner. That question was purely technical.

    The AI wasn’t stupid. It was confidently wrong in the same way every day, because nothing told it otherwise.

    That’s the moment I understood the most important design rule for AI reports in marketing: an analysis nobody can contradict repeats its mistakes every day.

    Thumbs down, with a reason

    The fix was almost embarrassingly simple. Every AI grade in our reports now has a thumbs up and a thumbs down. The thumbs down has one condition: you have to write a reason. One sentence is enough. “Customer already works with a partner.” “This was a support question, not a sales lead.”

    The next day’s run reads those reasons before it grades anything. The mistakes don’t disappear overnight, but they stop repeating. And, just as important, the sales team stopped ignoring the report, because they now had a way to shape it.

    Two things I learned about this:

    A graded chat with a thumbs down and a written reason
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.
    • The reason matters more than the vote. A thumbs down on its own tells the AI that it was wrong, but not why. Without the why, it just gets more cautious everywhere.
    • Feedback is also a trust signal. People trust a report they can argue with far more than a polished one they can’t.

    Don’t let the AI invent categories

    The second lesson came from topic labels. We categorise every chat and ticket by topic, so we can see what customers ask about most. For one run, I gave the AI sub-agents a list of example topics to help them along.

    They took it as inspiration. We got back fifteen new, invented labels, all slightly different, that I then had to map back to our real categories by hand. A trend chart built on labels that change from run to run is worthless.

    The rule now: the labels are a closed list. If a conversation doesn’t fit, the sub-agent answers “other” and adds a suggestion. At the end of the run, the main session looks at all suggestions at once and decides whether a new label is worth adding. Of the first eight suggestions, three became real labels. The other five were variations of existing ones.

    If you want to compare numbers over time, the AI may fill in categories, but it must not define them.

    Decline instead of guess

    We also built a documentation bot that answers product questions from our knowledge base. Before letting it near customers, we tested it on 200 real questions from support tickets and had another model judge the answers.

    The result was sobering and useful. Only a small share of answers were fully correct. Most were partially right: helpful for pointing someone to the right article, but incomplete. A few percent were wrong, and one was a proper hallucination: a command that doesn’t exist.

    We didn’t give up on the bot. We changed its job. It’s now a documentation navigator, not a support replacement. And we added a confidence gate: if the retrieved documentation doesn’t clearly cover the question, the bot says it doesn’t know and hands over. An honest “I don’t know” is worth more than a fluent wrong answer.

    A documentation bot that declines and hands over instead of guessing
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    The same principle applies to our analysis tools. One of our rules for the documentation gap report: never claim something is “not documented” without searching for it with at least two different phrasings. Customers and documentation writers rarely use the same words.

    Stop, don’t invent

    Many of our reports run automatically every morning, with nobody watching. That makes one rule non-negotiable: if data is missing or a connection fails, the run stops and says so. It does not fill in the gaps with plausible numbers.

    This sounds obvious. It isn’t. A language model’s instinct is to be helpful, and “helpful” with missing data means making something up. You have to tell it, explicitly and in writing, that an aborted report is better than an invented one.

    Keep the evidence next to the verdict

    The last pattern is the simplest. Every grade, every flagged ticket, every content idea in our tools links back to the source: the full chat transcript, the actual ticket, the real Instagram comments an idea was based on. Anyone can click through and check.

    This does two things. It makes mistakes visible fast. And it keeps humans in the habit of looking at the real conversations, which is where the actual insight is anyway.

    A checklist for your own AI reports

    If you’re building AI-assisted reports in marketing, here’s what I’d put in from day one:

    1. A thumbs down that requires a reason, and a next run that reads it.
    2. Closed lists for anything you want to count over time. “Other plus a suggestion” instead of invented labels.
    3. A confidence gate. Decline instead of guess.
    4. Abort on missing data. Written into the instructions, not assumed.
    5. Evidence next to every verdict. A link to the source, always.

    None of this is about better models. It’s about making the AI’s work something people can check, argue with and improve. That’s what turns an impressive demo into a tool a team actually relies on.