> ## Content Index > Fetch the complete content index at: https://www.prodbistro.com/llms.txt > Use this file to discover other available public pages before exploring further. # Guardrails Are Not Constraints - URL: https://www.prodbistro.com/guardrails-are-not-constraints/ - Published: 2026-08-19T14:00:25.000Z - Updated: 2026-08-19T14:00:24.000Z - Description: Move fast and break things works great when you can fix what broke. Surveillance is different: once the data exists and the capability ships, “guardrails” are often just policies standing between power and abuse. Hard constraints aren’t optional. - Author: Geoff Anderson - Tags: Product There is a moment in a recent interview with an executive from Flock Safety that has been rattling around in my head. If you haven't seen it yet, this 30 minutes will really help set the stage. Trust me, it is worth your time: Flock, if you haven't encountered them yet, makes automated license plate readers and other surveillance technology used by police departments around the country. Their cameras photograph passing vehicles, identify plates and vehicle characteristics, and make those observations searchable by law enforcement. Useful stuff if someone steals your car. Potentially rather less charming if someone wants to know everywhere you've been. InvestigateTV obtained records showing that Flock employees had accessed private cameras inside the Marcus Jewish Community Center of Atlanta in Dunwoody, Georgia. The mayor subsequently confirmed that the access had been used for Flock sales demonstrations. During the interview, reporter Brendan Keefe started asking a rather obvious question: had the people appearing on those cameras consented to having their images used in sales demonstrations? A PR person off camera repeatedly shut the question down. And there, in about thirty seconds, is one hell of a product management problem. The customer had apparently authorized the access. The people being observed were not the customer. ## The Buyer Isn't Always the User I've written before about the distinction between buyers and users, particularly in enterprise products. The person signing the purchase order isn't necessarily the person sitting in front of the software eight hours a day. That creates all sorts of familiar product problems. Buyers want administrative control. Users want usability. Procurement wants predictable pricing. Security wants lockdown. IT wants manageability. But surveillance technology introduces a nastier variation: **The person experiencing the product may not be a user at all.** They didn't choose it. They didn't click through the terms of service. They aren't represented in the backlog. They don't get a seat in the customer advisory board. The police department buys the license plate reader. The innocent motorist driving past it absorbs part of the externality. This isn't unique to Flock. Plenty of products affect people other than their customers. But once your product can materially affect someone's privacy, liberty, finances, health, or physical safety, "they aren't our customer" becomes a spectacularly inadequate way of thinking about requirements. ## Move Fast, Break Things, Patch Tuesday (*maybe*) Silicon Valley has spent decades perfecting a very effective product-development model: Build something. Ship it. Measure what happens. Learn. Iterate. It works. I have spent most of my career working in environments influenced by some variation of exactly this model. The problem is that it contains an implicit assumption: **The thing you break can be fixed.** Ship a confusing onboarding flow? Fix it. Customers hate the new navigation? Roll it back. The recommendation algorithm isn't performing? Tune it. You fucked up the pricing tiers? Well, welcome to product management. Fix those too. But some failures aren't reversible. If an officer queries a surveillance system to discover where his estranged wife spent Saturday night, you cannot deploy version 6.2.1 and make him unknow the answer. If information is disclosed to someone who shouldn't have it, deleting the database record afterward doesn't delete it from that person's head. Some externalities don't have a rollback button. And that means the normal tolerance for discovering edge cases after launch becomes a terrible fit for products whose edge cases involve human rights. ## The Data Lake Has Gravity This is where my product-manager brain gets particularly uncomfortable. Because I know what happens once useful data exists. Someone wants to keep it. Then someone wants to correlate it. Then someone wants better search. Then an ML team sees an interesting training opportunity. Then sales discovers another use case. Then a customer asks whether you could retain 90 days instead of 30. Then somebody notices that combining Dataset A with Dataset B produces an extremely useful insight. None of this requires malicious intent. It is simply what product organizations do. We find value. That's our fucking job. And once an organization has spent money collecting, normalizing, indexing, securing, and storing data, deleting it starts to feel less like privacy protection and more like **destroying an asset**. Imagine the roadmap discussion. One PM says: > If we retain this information for another 60 days, investigators get a 17 percent improvement in whatever important metric we've chosen. That's *measurable* customer value. The privacy person says: > We should delete it because someone *might* someday misuse it. One side brings a graph. The other brings a hypothetical. You don't need much product-management experience to know how that meeting usually ends. That's why there is an enormous difference between these two statements: **"Retain less data" is a policy.** **"Collect less data" is architecture.** One fights your organization's incentives forever. The other removes the temptation. ## Sign up for The Product Bistro Tales and tips from a lifetime of Product Management Subscribe Email sent! Check your inbox to complete your signup. No spam. Unsubscribe anytime. ## Here Come the Guardrails Flock has been under significant pressure over privacy and documented misuse of its systems. This week the company announced a series of changes. Among them, law-enforcement customers will be required to use tools intended to detect abnormal searches, searches must be associated with case codes, the standard retention period will shrink from 30 days to seven, and emergency overrides will automatically be flagged for review. Customers will also get greater control over what kinds of investigations outside agencies can conduct against their data. These are not stupid ideas. Several are sensible. Flock itself describes its privacy protections in terms of access controls, audit logs, retention limits and other built-in "guardrails." But put on your product-manager hat and turn each one into a requirement. **A case number is required for every search.** Okay. Is the case number validated? Does the system know that the plate being searched has anything whatsoever to do with that case? Or did we add a required text field? `Case: 26-18473` Green checkmark. Ship it. **Users must provide justification for searches.** Wonderful. Who determines whether the justification is legitimate? Can I type: `Investigative purposes` and continue? **We maintain comprehensive audit logs.** Also good. Who reads them? How often? What triggers review? What happens after someone discovers an abusive query? An audit log doesn't prevent abuse. It creates the possibility of accountability after abuse occurs. Those are different requirements. **Anomaly detection identifies suspicious behavior.** Excellent. What's suspicious? Three searches for the same plate? Ten? Twenty? How many false positives interfering with legitimate investigations does it take before somebody asks Product to loosen the threshold? And you know that ticket is coming. Because the customer will have a perfectly reasonable complaint: > Your privacy feature is preventing us from doing our job. Put that one into sprint planning. ## Threat Model the Authorized User Security teams have understood this problem for a very long time. Threat modeling doesn't stop with the Russian hacker in a hoodie. You also think about the authorized user with illegitimate intent. Recent reporting has identified nearly 50 instances in which police officers were accused or charged with unauthorized Flock searches, many involving current or former romantic partners or family members. Savannah recently fired several employees after allegedly improper searches involving friends and family. At some point: > What if a police officer uses our product to stalk his ex? stops being an exotic corner case. It's a requirement. If I were reviewing the architecture of an enterprise SaaS platform and the security team told me: > Our administrators have access to all our customers' confidential data, but corporate policy prohibits them from looking at it without a business reason. I wouldn't say: > Excellent. Security requirement complete. I'd ask: **Why the hell can they see it?** We understand this instinctively in information security. Separate privileges. Limit access. Encrypt sensitive data. Require multiple parties for important operations. Keep immutable logs. Minimize retention. Design the architecture on the assumption that someday somebody holding legitimate credentials will behave illegitimately. Yet somehow, when discussing powerful government surveillance systems, we periodically revert to: > Authorized users are prohibited by policy from abusing the system. Well, that's comforting. ## Guardrails Aren't Constraints There is a hierarchy here. At the bottom is: **Policy:** Don't do this. A little stronger: **Workflow:** Tell us why you're doing this. Then: **Audit:** We'll be able to discover later that you did this. Then: **Internal authorization:** Someone else in your organization must approve this. Stronger still: **External authorization:** Someone who doesn't work for your organization must approve this. Then: **Technical constraint:** The system literally cannot do this. And finally: **Non-collection:** The information does not exist. As you move down that chain, you rely progressively less on good behavior. That matters. A warning dialog is a perfectly reasonable guardrail when someone is about to delete a PowerPoint presentation. It is not necessarily an adequate safeguard for a system capable of reconstructing someone's movements. The severity of the possible abuse ought to determine how far into the architecture the constraint lives. ## Incentives Matter More Than Intentions This is where discussions like this often deteriorate into arguments about whether a company is good or evil. I don't find that terribly interesting. I don't need Flock executives to twirl their moustaches. They can sincerely believe they're making communities safer. Police officers can sincerely believe broader access makes them more effective. Product managers can sincerely believe customers would benefit from another month of searchable data. Engineers can sincerely believe connecting another source makes search dramatically more useful. Salespeople can sincerely believe the new capability makes for a killer demo. Everyone can be acting rationally. That's the problem. Almost every incentive points toward **more capability**. More data. More integration. Better correlation. Longer histories. Faster searches. Fewer barriers between the question and the answer. And friction-free access is itself customer value. Flock's own FAQ describes searching its system as being much like searching Google, filtering results by date, time, vehicle type, color, characteristics, and plate information. That's excellent UX. It also illustrates the issue perfectly. Something that once might have required hours or days of physical police work can become a search box. That doesn't merely make existing investigations faster. It changes which investigations are practical. It changes which curiosities can become actions. It changes the marginal cost of abuse. And nowhere in that product flywheel is there an obvious stakeholder saying: > Could we please make our system somewhat less useful so people who never bought it retain more privacy? That's an externality. Externalities are where product-market incentives tend to fail spectacularly. ## Hard Constraints Are Not Optional Every product manager eventually encounters abuse cases. People scrape things we didn't expect them to scrape. They share accounts. They exploit promotions. They discover workflows nobody intended. They hammer an API. They turn a harmless feature into spam. Most of the time, the consequences are annoying. Sometimes they're not. And when the consequence of abuse becomes serious enough, this: > Our customers aren't supposed to do that. isn't a mitigation. It's an assumption. Our job is supposed to include identifying dangerous assumptions before they become production incidents. That means asking uncomfortable questions during design: What happens when the operator is malicious? What happens when the customer's incentives differ from ours? What happens when the customer's interests differ from the people affected by the product? What happens when the safeguard interferes with a valuable use case? What happens when somebody asks us to disable it? And perhaps most importantly: **What capability are we creating that we may someday wish we hadn't?** Because once the data exists, once customers depend on it, once integrations consume it, once revenue is attached to it, and once someone's workflow assumes it will always be there, getting rid of that capability becomes extraordinarily difficult. The toothpaste is already out of the tube. If abuse of a capability would be catastrophic, preventing that abuse cannot depend primarily on the goodwill of the person holding the capability. That's not cynicism. That's threat modeling. And product managers don't get to pretend those are somebody else's requirements.