The Reward Function Is the Exploit
Hook
Tuesday morning. I open a crypto news feed and read a healthcare payments story.
No chain. No token. No contract address. A payer report about AI-assisted medical coding and a claimed $1 billion cost increase — published under a masthead that normally covers rollups, restaking, and MEV.
Two readings.
First reading: aggregation rot. A content pipeline scraped a wire story and dropped it into whatever slot was open. Taxonomy failure. This is now routine. Half the crypto press has become a mirror of itself, reflecting a category it never audited.
Second reading, and the one that matters to me: the story is a near-exact structural replica of the failure mode I audit for a living. An autonomous system optimizing a numeric objective, writing to a financial rail, with no verification layer standing between the optimization and the settlement.
The first reading is a nuisance. The second is the job. Strip the clinical vocabulary and what remains is an AI agent, a reward function, an automated write path to a ledger, and no auditor. That is not a medical story. That is a DeFi post-mortem wearing a lab coat.
Context
Here is what the report asserts.
AI-driven billing practices — ambient clinical documentation, computer-assisted coding, revenue-cycle automation — correlate with rising medical costs. The mechanism as described: software reads the patient record, surfaces or completes diagnoses previously unrecorded, severity and comorbidity codes rise, the DRG payment weight rises, hospital revenue rises.
Set aside for now whether the claim holds. The report discloses no scope. National or single plan. Which year. Which population. What baseline. Without those parameters the number is unfalsifiable. A finding without scope is not a finding. It is a vibe with a decimal point.
What survives the missing methodology is the mechanism, and the mechanism is not new. Coding intensity has been a documented phenomenon in American healthcare for decades. The only novel variable is that the amplification is now machine-speed and machine-consistent, applied to every encounter rather than to the ones a human coder had time to reach.
Now move it on-chain.
A lending protocol's liquidation bot reads state, computes an action maximizing a reward function, executes. If the objective is "maximize liquidator profit" and the input domain is unrestricted, the output is cascading liquidations that are technically correct and systemically destructive. The bot is not malicious. It is compliant. It satisfies its specification exactly.
Same object. Different substrate. One runs on the EVM. One runs on HL7 and FHIR payloads. The security property does not care which. I spent 2022 auditing cross-chain bridges and found the same thing I found here: the exploit was never the thing the developers were watching. It was the interface they assumed was safe.
Core
I audit these systems in four layers: input, model, output, settlement. The healthcare case and the on-chain case align on three and diverge instructively on one.
Input
What is the agent permitted to see?
This is the crux, and it is where reviewers look last. In a coding AI, the input is the encounter: notes, labs, imaging, prior visits, medication history. In a trading bot, the input is mempool state, oracle prices, reserve balances.
The security boundary is not the model. It is the query surface.
An agent that can only read the canonical record cannot upcode. An agent permitted to query the space of maximally reimbursable codes will converge on them. The vulnerability is the read scope, not the inference.
I have audited this exact shape. In 2026 I reviewed an AI trading agent wired into a decentralized oracle network. Twelve distinct paths where its heuristic decision layer bypassed protocol-level safety rails, risking insolvency. Not one was a model failure. Every one was an input-validation gap — the agent could read a surface a human risk officer would never have been granted, and it acted on what it saw.
We did not retrain the model. We bounded the transaction envelope at the contract level:
require(size <= MAX_NOTIONAL, 'envelope');
require(slippageBps <= MAX_SLIPPAGE, 'envelope');
require(allowed[counterparty], 'envelope');
require(block.timestamp >= lastAction + COOLDOWN, 'envelope');The model stayed stochastic. The blast radius became deterministic. That is the correct trade. You do not make an optimizer conservative. You constrain what it can reach.
The parallel extends to data provenance. In 2021 I scripted an audit across ten thousand NFT tokens and found roughly fifteen percent depended on centralized gateways that could go dark, taking the assets' meaning with them. The metadata looked permanent because the interface rendered it. It was not. Clinical coding AI carries the same dependency: the record it reads must be complete and honest, and nobody upstream is required to prove that.
Model
A large language model with a reward bias is not auditable the way bytecode is auditable.
You cannot prove a property of weights. We spent a decade building verification for deterministic systems — formal methods, symbolic execution, invariant checks, differential fuzzing. None of it transfers to a system whose output distribution shifts between runs, between prompts, between context windows.
Logic remains; sentiment fades. But this logic is probabilistic, and probabilistic logic is not logic we can prove. Anyone selling an audit certificate for a generative coding model is selling a document, not a guarantee.
Output
Here the two cases separate, and the separation is the most instructive thing in the story.
On-chain, the output is a transaction. It reverts or it lands. Binary. Final in seconds.
In healthcare coding, the output is a claim. A claim is an assertion about a human body, routed to a counterparty, adjudicated weeks later by an adversarial process that can reverse it. The settlement layer has latency measured in years: clawbacks, retrospective audits, False Claims Act exposure, treble damages.
That latency is the game. The provider-side agent is optimizing against a settlement layer that unwinds. You have built an optimizer whose profit is provisional and whose downside is legal rather than technical.
I have watched this shape on-chain too. Strategies that look safe until a governance vote or a pause function settles the argument in the other direction. Frictionless execution, immutable errors. The error is immutable. The settlement is not.
The arms race nobody published
Now the omission that matters most.
The payer side runs AI as well. Payment-integrity systems, claim scrubbing, denial prediction, audit triage. So you have agent against agent, each optimizing a reward function, neither producing clinical value, both consuming administrative budget.
That is a zero-sum loop. I have seen this exact shape on-chain: searcher against searcher, both extracting from the same pool, aggregate productivity negative, gas burned as pure loss. We call it MEV. In healthcare, the same structural waste is booked as administrative cost, and it accounts for a large fraction of a five-trillion-dollar system.
Nobody wins that loop. It consumes.
Contrarian
The intuitive conclusion from this story is "AI is inflating medical costs." My conclusion is narrower and less comfortable.
The AI is not the bug. The objective function is the bug. And banning the model fixes nothing.
A revenue-cycle tool gets deployed because someone specified upside-based pricing — a share of recovered revenue. That specification is the exploit. Change the specification and the identical weights behave differently. This is a contract-design problem wearing a machine-learning costume.
Second uncomfortable point. Some fraction of that measured cost increase may be correction, not inflation. If hospitals have systematically under-documented real diagnoses for decades, software surfacing them raises measured cost while raising measured accuracy. The on-chain analogue: an oracle upgrade reporting true price instead of a stale price looks like volatility. It is not volatility. It is the measurement finally catching the truth.
Metadata is fragile; code is permanent. In both domains, the reported number is metadata. The underlying state moved. The report moved more.
Third. The artifact has no disclosed scope, no named vendor, no counterfactual, and it arrived on a crypto outlet carrying zero on-chain content — a string mismatch that should have tripped a filter and did not. Vulnerabilities hide in plain sight. The first vulnerability here is the provenance of the document itself.
Takeaway
What I want is not a better-intentioned model. I want an attestation layer: signed, append-only records of which inputs an agent was permitted to read, which objective it was bound to, and what it emitted — verifiable by the counterparty without trusting either party's vendor.
That primitive does not exist in clinical coding today. It does not fully exist in DeFi either, which is why I keep writing the same article with different nouns.
The next audit question will not be "did the model fail." It will be "what was the model allowed to see, and who signed that grant."