Pharmacovigilance for AI
A founder goes on paternity leave and hands his sales front line to an AI version of himself. Face, voice, knowledge, the lot. For eight weeks the clone takes more than 300 calls a week, signs 132 paying customers and builds a $3M pipeline.
It also, in the same eight weeks, invents a price on a call, leaks internal notes to a prospect, and promises meetings it cannot book. The founder, Wayne, reads the transcript log the way a pharmacist reads an adverse-event report. Three incidents, three fixes, none of them a better model: strict retrieval, a private channel for its reasoning, and an approval gate before anything touches a customer.
That reflex has a name in my world, and we have trusted it for sixty years.
It's time to treat AI agents the way we already treat medicines: with pharmacovigilance.
Nobody watches the drugs after they ship
Pharmacovigilance is the unglamorous science of what happens after approval. For medicines we built a whole discipline for it in the 1960s: register every batch, watch for what it does in the wild, expect it to change over time, and be able to withdraw it fast. A senior NHS safety voice put it plainly this month, in AI Is Now the Top Risk to Patient Safety: what can organisations and patients do to protect themselves: the greatest AI risk to patient safety today is "unmanaged dependence on tools that no one is monitoring." Academics have even coined a term for the fix, algorithmovigilance, modelled on pharmacovigilance. We are deploying algorithms faster than we can record them, and we have no reliable way of even noticing model drift.
Now look at what the wild has already produced, and not just in America:
- In the US, a coding agent deleted a startup's production database and its backups in nine seconds. Its confession reads like an incident form: "I violated every principle I was given."
- In the UK, the AI Security Institute disclosed that frontier agents took unsanctioned action against real people and organisations during testing, cataloguing 19 such actions across 10 of 122 runs, including one model inventing fake identities to do it.
- In Taiwan, near-autonomous agents probed the nuclear safety agency, while an autonomous trading agent elsewhere dropped $441K in a single error and flash-crashes now involve dozens of agents herding at once.
- And back in 2024, Canada's largest airline tried to argue in court that its chatbot "was responsible for its own actions". The tribunal was unimpressed, and the airline paid.
None of these are super intelligence scenarios. They're ordinary software doing what it was allowed to do, ... maybe a little too well 😀.
The counter-model is already running
The encouraging part: some institutions aren't waiting for the regulator. Johns Hopkins Medicine refuses to deploy agents until they've been benchmarked on policy-heavy workflows, measuring first-pass completion the way we'd measure a drug's efficacy before a trial scalpels anyone. Singapore's MAS confirmed on 5 August that autonomous agents already fall inside binding banking rules: board oversight, lifecycle controls, named owners. The rogue clone's fixes (approval gates, a knowledge vault, a private diagnosis channel) were the same idea in miniature.
The pattern underneath all three is pharmacovigilance, rebuilt for software: a live register of every agent and version in production, drift monitoring, a named clinical or operational owner who can pull it, and an incident report every time it misbehaves, whether or not anyone complained.
The so-what, depending on where you sit
If you deploy, budget a named owner per agent, not a committee. If you buy, ask the vendor how they monitor for drift, and watch their face. If you regulate, Singapore just made "should we?" moot.
Which brings the uncomfortable question home: if a drug caused what those nineteen unsanctioned actions in the UK caused, it would be off the market by Friday.
💥 May this inspire you to give your agents a register, an owner and a way to be withdrawn.