The Scanner is the Symptom

A writer on Substack ran his own essay through Pangram, the AI-detection tool the platform quietly bolted onto every post starting July 21. Clean scan: 100% human. He fixed a typo. The score flipped to AI-assisted. He changed a single word. It flipped back to human. He edited one more sentence. AI-assisted again.

He posted the sequence as a Note. It picked up over a thousand hearts in a day, most of them from people treating it as slapstick. Nobody in the replies seemed to notice they were watching a live demo of a much bigger problem: this is what a trust-and-safety feature looks like in week one, when the company shipping it hasn't reckoned with what it actually built.

I don't think this is a story about AI. I think it's a story about what happens when you ship a classifier to solve a problem your own product created, and don't touch the thing that created it. Every PM reading this has some version of that story already in their backlog, or has lived something similar.

What got announced vs. what got shipped

Substack's founder Chris Best framed the launch carefully. The problem, in his telling, isn't AI use — plenty of writers use AI for research, editing, or drafting without deceiving anyone. The problem is what he called "Claudefishing": a mismatch between what a reader believes they're getting and what they're actually getting, specifically when a reader invests attention or trust in something with no real person behind it.

That's a coherent, narrow, defensible problem statement. It's a trust problem, and trust problems are usually solved with disclosure norms, identity signals, and reputation — slow, structural, unglamorous tools.

I should add that had Substack not implemented their own Social Media layer – Notes – and remained true to their original mission, this wouldn't ever have been a problem.

What actually shipped is a per-post AI-probability score, opt-outable by the writer, built on a detector with known architectural weak points. That's not a trust feature. That's a classifier wearing a trust feature's press release. The gap between the announcement and the mechanism is where this whole story lives, and it's worth sitting with, because it's a gap most of us have shipped at least once: we scope a hard, structural problem, then ship the cheapest thing that produces a number, and hope the number reads as having solved the problem it was named after.

The incentive history nobody put in the launch post

Here's the part that turns this from an unfortunate rollout into a genuinely instructive case study. Substack didn't stumble into an AI-slop problem from nowhere. For the past two years, the company has actively coached writers that posting frequency on Notes — its Twitter-shaped, algorithmically-ranked discovery feed — drives growth. Publicly shared internal data put a number on it: writers who posted three or more Notes in their launch week saw roughly 50% more subscriber growth than those who didn't. Notes itself now reportedly accounts for something like half of all new subscriptions on the platform, free and paid combined.

Read the Claudefishing announcement again with that in mind. It doesn't mention Notes. It doesn't mention the feed. It doesn't mention ranking. The one place an engagement-optimized discovery mechanism appears is a passing note that creator-level signals and reader preferences are "under consideration" for later.

You don't accidentally leave the thing driving half your subscriber growth out of a launch post about protecting trust and quality. You leave it out because naming it means admitting the volume problem you're now policing is one you spent two years manufacturing on the growth side. This is the part I'd bet didn't get resolved in a single meeting — my guess, and it's only a guess, is a growth team defending a number and a trust & safety function shipping the only lever available to it, on two different timelines, reporting to two different people, neither of whom owned the whole loop.

If you've ever watched your company ship an abuse-prevention feature for a behavior your onboarding flow was actively rewarding six months earlier, you already know how this meeting went without me telling you.

The instrument makes it worse, not just late

Even setting the incentive problem aside, the tool itself has a design flaw worth understanding, because it's not specific to Pangram — it's generic to any classifier deployed at "mostly accurate."

Pangram's developers claim roughly 1 error in 10,000 for false positives — human text wrongly flagged as AI. That figure holds up reasonably well in controlled testing. But it's a sterile-conditions number: it degrades meaningfully on edited, mixed-provenance, or heavily revised text, which is to say, on most real writing. Two structural reasons why: the detector scores text in chunks of roughly 150 to 350 words, and a chunk with even a partial AI contribution can flip the whole segment's classification — which is exactly the mechanism behind that flip-flopping note. And its human-text reference corpus is drawn from 2021 and earlier, before any of today's writing-assist tools existed, which means clean, well-organized prose written in 2026 is being measured against a baseline that predates the tools that helped produce it.

Here's the part that should worry a PM more than the raw error rate: aggregate accuracy and individual risk are not the same claim, and conflating them is the actual failure mode. At a false-positive rate of even 0.01%, scanning a million posts wrongly flags around a hundred — statistical noise at the population level. But point the same tool at one specific writer, and the math inverts completely: for that person, the flag is either right or catastrophically wrong, full stop, and the population-level number offers them nothing. A tool that's "almost always right" doesn't feel almost-always-right to the person it's wrong about, and it doesn't feel almost-always-right to their readers either, who have no way to know they're looking at the rare miss.

There's a sharper version of this worth internalizing as a design principle, not just a critique of Substack: a detector that's obviously unreliable gets treated with appropriate skepticism. A detector that's mostly reliable gets trusted uncritically, right up until it produces the one high-visibility wrong call that costs someone their reputation — and a system people stop double-checking is more dangerous than one they never trusted in the first place.

What this means if you're the one shipping the scanner

A few things I'd take into any review of an enforcement or trust-and-safety feature after watching this unfold:

• Fix the incentive before, or instead of, the classifier. If your enforcement layer is targeting a behavior pattern your growth mechanics reward, the classifier will read as punitive no matter how well-intentioned the launch copy is, because you're asking users to absorb the cost of a problem your own product design created.
• Know which claim you're actually making. "99.99% accurate" and "safe to flag individuals with" are different claims, and only one of them is usually true. If your feature makes individual-level accusations, you need individual-level guarantees, not population-level ones — and if you don't have those, say so, publicly, before someone finds out the hard way.
• Put the cost of the false positive on the team that owns the number the enforcement is meant to offset. If growth doesn't feel the pain of the trust feature's false positives, growth will keep incentivizing the exact behavior the trust feature is trying to suppress, and you'll be stuck relitigating this every quarter.
• Watch for the "we did something" feature as its own failure mode. A visible, symbolic response to a structural problem can be worse than no response, because it burns credibility on a fix that was never going to work, and makes the real fix — the unglamorous, structural one — harder to sell internally once the flashy version has already visibly failed.

Substack will probably end up fine here, one way or another. Either the incentive gets fixed and the scanner becomes a quiet background signal nobody argues about, or the opt-out rate climbs high enough that the feature discredits itself and gets quietly reworked. Either way, it's a live case study running in public, and it's a better one than most of what shows up in a product-strategy textbook, because you get to watch the org chart in real time by reading what the launch post left out.