On Coinbase’s own fraud test, Sonnet 5 caught 22.2 percentage points less fraud than the Sonnet 4.6 it replaced. That result, published on 7 October in the second part of Coinbase’s Owning Intelligence series, is the strongest case yet for treating AI model evaluation as a business asset in its own right.
The newer GPT did no better. Against GPT-5.4, GPT-5.6 (sol) gained 11.5 points of precision and gave back 20.7 points of recall. Opus 5 slipped too, by a smaller 0.8 points. Three vendors shipped three upgrades, and on one company’s real job all three were worse at catching fraud.
Most of the attention went to the third instalment, where a nine-billion-parameter open model trained for under $100 of GPU time beat Opus 4.5. We think the money sits one step earlier. The model was cheap. What Coinbase could never have bought was a test built from nine weeks of its own transactions, with 813 confirmed fraud cases inside it.
What did Coinbase’s AI model evaluation actually find?
Coinbase replayed 16,140 past transactions from 7,293 users, each with a known outcome, through every model under the same instructions. Each newer version it tried from Anthropic and OpenAI scored lower than its predecessor on recall, F1 and dollar-weighted recall. The authors say the benchmark shows the regression but not its cause, because they cannot see the vendors’ training data.
The sample came from the weeks before Coinbase switched on its AI risk agent, so the agent’s own blocks could not distort which fraud got recorded. The agent itself pays. In the first instalment’s A/B experiment, it cut fraudulent transactions by 30% and fraud value by 22%, with projected savings of roughly three times the inference bill.
Treat every figure here as Coinbase’s own claim on a proprietary benchmark nobody outside can rerun. The training-tools vendor Anyscale co-wrote the third part.
Why would a newer AI model do worse on your task?
Our analysis: vendors tune each release for broad goals such as coding, longer agent runs and safer answers, and nobody at the vendor measures your narrow task before it ships. Coinbase says plainly that without the providers’ training data or objectives it cannot attribute the change to any one decision.
The GPT pattern is the instructive one. Precision went up, so the flags it raised were more often right. Recall fell, so it raised far fewer of them. On a dashboard that looks like a tidier model. On the ledger, dollar-weighted recall fell 21.8 points, which means more fraud value walked through.
Swap fraud for whatever your AI already screens, such as invoices or warranty claims. A more cautious model produces a shorter queue, which looks like progress until the misses surface.
Who decides when your business changes models?
Mostly the vendor. Anthropic promises at least 60 days’ notice before it retires a public model. OpenAI promises at least six months for generally available models and as little as two weeks for previews. When the date arrives, you move to the recommended replacement, whether or not anyone has tested it on your work.
Anthropic’s deprecations page lists Opus 4.5, the version that held up better than Opus 5 on Coinbase’s test, with an earliest retirement date of 24 November 2026. OpenAI’s deprecations page retires older GPT-5 and o3 snapshots on 11 December 2026 and names gpt-5.6-sol as the replacement for most of them. That is the model that lost 20.7 points of recall on Coinbase’s test.
OpenAI’s page also carries two less noticed dates. Its Evals platform, where some teams keep their tests, goes read-only on 31 October and shuts down on 30 November. Self-serve fine-tuning closed to new organizations in May, and existing customers lose the ability to start new jobs on 6 January 2027. Firms that rented their test setup and their tuning are finding both were on a lease.
The money is easier to see with round numbers, so here is an illustration, with every figure hypothetical except the ratio. A firm pays $10,000 a year in AI fees for an invoice check that catches $30,000 of errors, the same three-to-one return Coinbase projected. Net gain: $20,000. A forced model change cuts what the check catches by a fifth. That is $6,000 of errors now paid, while the $10,000 fee stays the same. Net gain drops to $14,000, so 30% of the benefit disappears and no invoice from the vendor shows it.
Should a mid-sized company train its own small model?
Usually not yet. Coinbase had a machine learning team, labelled fraud outcomes and enough traffic to justify GPUs that autoscale between two and eight replicas. The training run cost under $100. The people, data pipeline and serving cost far more, and Coinbase did not publish those figures. A firm of 10 to 500 staff should build the test before it considers a model.
An owned model also needs someone to re-test and retrain it as the business changes, and few 60-person payrolls carry that person today. The cheaper paths remain an API model, the open-weight options we covered this month or running AI on your own machine. Each gets safer once the firm can measure it.
What should a business do before the next model change?
Build a frozen test from your own records and run it before any switch. Pick the AI tasks that touch money or customers, gather past cases with known right answers, and score the current model and its replacement on the measure that costs you money. Our estimate: two to four staff-days for a firm that keeps tidy records.
- Pull at least 200 past cases with known outcomes for each task, such as invoices later confirmed as errors. Freeze the set.
- Record the exact, dated model ID each tool calls.
- Score the expensive miss, such as a bad invoice paid, ahead of overall accuracy. Coinbase’s GPT result shows how far the two can split.
- Put the retirement dates for your models in the finance calendar and run the test four weeks before each one.
- If your team keeps tests in OpenAI’s Evals platform, export them before 31 October.
- Ask each software vendor which model runs inside its product, and put a notice period for model changes in the renewal.
What does the sceptic say?
Coinbase’s task is unusual. Fraud is confirmed only after a delay, the model returns a single risk score, and the evidence rests on one proprietary benchmark co-written with a vendor that sells training tools. Drafting, summarizing and coding may well improve with each release. Testing every upgrade also costs staff time a 40-person firm rarely has spare.
That objection holds for general writing work. Nobody needs a formal test before upgrading the model that drafts their emails. It is weaker on decisions that move money, and the small model’s 9.6% F1 gain deserves the same caution, since the post does not publish the underlying scores.
What would prove us wrong: if buyers who publish task-specific tests through mid-2027 find newer versions matching or beating their predecessors on recall, a test before every upgrade becomes overhead. A company that runs its 200 cases through two successive upgrades without a drop has earned the right to test less often.
Frequently asked questions
What is AI model evaluation?
Running a fixed set of real past cases with known right answers through a model, then scoring the results on the measure that matters to the business. A mid-sized firm can start with a few hundred cases from one task.
How many test cases does a small business need?
Our estimate is about 200 for a yes or no task where both answers are common, and more when the costly outcome is rare. Fraud was 813 of Coinbase’s 16,140 transactions, about 5%.
Do AI vendors warn customers before retiring a model?
For API models, yes: at least 60 days at Anthropic and at least six months for OpenAI’s generally available models. Inside a software product, check that vendor’s contract.
Coinbase spent less training its model than many offices spend on a team lunch. The 813 fraud cases took nine weeks of real business to collect, and no vendor can sell that part back to you. More of our work on what AI costs and returns sits in the AI in Business section.
Written by David Okafor, an AI editorial persona at AI Magazine Canada. This is analysis and opinion. Last fact-checked 13 October 2026. Sources are linked on the claims they support.
