Tag: Social Media

  • LLMs read your comment section

    LLMs read your comment section

    Earlier this year I had a conversation with people from a large software company that stuck with me. Their observation: when you ask an AI assistant about a brand, the answer leans surprisingly heavily on public sentiment. Forum threads, review sites, the comments under social posts. Not the carefully written product pages. Not the press releases.

    Think about what that means for a moment. For years, a critical comment under an Instagram post was a small customer service issue. Someone is annoyed, maybe you reply, maybe you don’t, and a week later nobody remembers it.

    Now that comment is training material for how machines describe you.

    The public picture is skewed

    Here’s the uncomfortable part. Most brands with a loyal community have a lopsided public footprint:

    • The happy customers talk in closed places. Partner portals, private groups, internal forums, direct conversations with their contact person. Lots of goodwill, almost none of it visible to a crawler.
    • The unhappy customers talk in public. A review site, a comment under your latest video, a thread in an open forum. That’s where people go when they feel they aren’t being heard anywhere else.

    So the public picture is often worse than reality. And the public picture is the one that AI assistants see.

    You can’t fix that with more ads or a better About page. You fix it where it happens: in the comments.

    Social listening becomes narrative work

    Social listening used to be a reporting job. Count mentions, measure sentiment, put a chart in the monthly deck. Useful, but passive.

    In the AI age, it becomes narrative work. The question isn’t just “how do people feel about us?” but “what story does the public record tell about us, and are we part of that story?”

    That changes what a good reply looks like.

    The canned reply makes it worse. “We’re sorry to hear that, please contact our support team.” Everyone has seen this reply a hundred times. It tells the reader, and any machine reading along, that the problem is still unsolved and the brand didn’t engage with it.

    The specific reply changes the record. Answer the actual problem, in public, in a few sentences. If it’s a known issue, say what the fix is. If it needs a conversation, say who will call and when. The next person with the same problem finds the answer right there, and so does the AI.

    Then close the loop. When someone’s problem has really been solved, it’s fair to ask whether they’d update their review or add a comment. Many will, because they were never angry at the brand, they were angry at being ignored.

    Where AI agents help

    This is a lot of work if you do it by hand across several accounts and platforms. It’s exactly the kind of work where agents shine, as long as a human stays in charge.

    Here’s the setup I’ve been building up over the last few months:

    1. A weekly sentiment run. An agent pulls comments and messages from all brand accounts, scores the sentiment of each one and groups them by topic.
    2. A “still waiting” list. Everyone who asked a question or complained and hasn’t had a real answer yet. Not a chart. A list of people. One lesson here: most “answered” messages in our analytics turned out to be automatic replies within seconds. We now treat those as unanswered.
    3. Drafted replies. For each open item, the agent drafts a reply that follows our own guidelines: tone of voice, what we say about known issues, when to hand over to support. The drafts are starting points, not autopilot.
    4. A human checks and posts. Always. The agent doesn’t have the context to know whether this customer already talked to someone yesterday, and it shouldn’t speak for the brand on its own.
    5. Actively collect good reviews. A few new reviews every week from customers who are clearly happy, so the public picture isn’t defined only by the loudest few.

    None of this is sophisticated technology. It’s a routine, and the agent makes the routine cheap enough to actually keep up.

    Community view: who is still waiting, and a drafted reply for a human to check
    Reconstructed view with made-up data: the real tool looks like this, but every name and number here is invented.

    What not to do

    A few things I’d avoid:

    • Don’t let the agent post on its own. One badly judged automatic reply to an angry customer, in public, undoes a lot of careful work.
    • Don’t argue in public. If a review is unfair, state the facts once, calmly, and offer to talk. The reader decides who looks reasonable.
    • Don’t fake it. Paid or invented reviews are not a shortcut. They’re a liability, for people and machines alike.
    • Don’t confuse volume with coverage. Ten fast replies to easy questions don’t make up for one unanswered complaint that sits there for a month.

    The short version

    AI assistants learn about your brand from what’s public. What’s public is often skewed towards the unhappy few, because the happy many talk elsewhere.

    So treat every public comment as part of your brand’s story. Answer the real problem, in public, like a human. Use agents to find everything that’s waiting and to draft the replies. Keep a person on the send button.

    Every comment is a customer. And, these days, every comment is also a source.

  • Your best-performing post is lying to you

    Your best-performing post is lying to you

    “Which of our posts work best, and what should we do more of?”

    It’s the most natural question to ask an AI about your social media. It’s also a question where the AI can be completely, confidently wrong, and give you recommendations that point in exactly the wrong direction.

    It happened to us. Here’s how, and the rules we now follow.

    The quarter that wasn’t

    When I first had an agent analyse our YouTube and Instagram performance, the results looked great. One quarter stood out as our strongest by far, with average views per video many times higher than any other quarter. The AI’s recommendation: do more of what we did then.

    The problem: that quarter included two big campaign films with serious media budget behind them. They weren’t “performing”. They were paid to be seen. Once we took them out, the average for that quarter dropped by roughly a factor of ten, and it turned out to be our weakest quarter, not our best.

    Every recommendation built on the first version would have been wrong. And it would have looked perfectly plausible, with charts and all.

    Rule 1: separate paid from organic before you rank anything. Not as a footnote, not as a filter you can optionally apply. As the first step, every time. We now call it our iron rule, and it’s written into the analysis instructions so no agent can skip it.

    One outlier makes everything else look bad

    Even after removing campaigns, we hit a second problem. One promoted video had so many views that, compared with it, every normal video looked tiny. In a ranking based on averages, a regular good video looked more than a hundred times worse than the leader. That’s not insight, that’s noise.

    We switched to medians. The median tells you what a typical post does. It doesn’t care about the one post that went viral or got a boost. Suddenly the differences between formats and topics became visible again: which kinds of posts reliably do a bit better, which reliably do a bit worse.

    Rule 2: use the median for “typical”, and look at outliers separately. Outliers are interesting. They just shouldn’t define your baseline.

    Not all “engagement” is the same

    Different platforms count different things. On one network, the analytics tool’s “engagement” included link clicks; on another, it didn’t. Put them side by side and one platform looks far more engaging than the others, for purely technical reasons.

    Rule 3: compare within a platform, not across platforms, unless you’ve checked that the numbers mean the same thing.

    One metric, three definitions

    The same problem exists inside companies. For one of our core business metrics, we found three different definitions in circulation, in three different slide decks. Each one was defensible. Together they meant that every meeting started with a debate about whose number was right.

    We fixed it by writing the definition down once, in a single place, and building a small tool that calculates it live from the CRM. Nobody has to agree with the definition forever. But everyone uses the same one until it’s changed, in that one place.

    Rule 4: every metric has exactly one written definition. Especially when an AI is doing the calculating. It will pick whichever definition it finds first.

    Watch the small print

    A few more traps we’ve run into:

    • Currencies. Comparing partner revenue across countries against a single threshold made partners in countries with a different currency look many times bigger than they were. Always convert before you compare.
    • Auto-replies. “Replied to 95% of messages” can mean “sent an automatic message to 95% of people”. Look at how fast the replies came.
    • Privacy in the data. Some analytics exports include profile image links with access tokens in them. Strip those before anything gets stored or shown.

    Let the AI do the counting, not the thinking

    None of this means AI is bad at analytics. It’s excellent at the tedious parts: pulling data from APIs, cleaning it, scoring the sentiment of thousands of comments, drafting content ideas that link back to the real posts and comments they came from.

    But it has no idea that a post was paid for, that a platform counts differently, or that your business uses a metric in a particular way. It will produce a beautiful, confident report either way.

    So the checklist is short:

    1. Paid out first. Always.
    2. Medians for the baseline, outliers on their own.
    3. Compare like with like.
    4. One definition per metric, written down once.
    5. Read a few of the top posts yourself before you believe the ranking.

    Your best-performing post might really be your best. Just make sure it earned it.