How much premium should a firm pay for a specialist AI tool? A Spanish energy company answered that question with a test, not a demo. Its legal team scored three tools without knowing which tool wrote which answer. The held detail is how long the test ran after the blind round ended. Eight weeks of live work decided the contract.
The direct answer: Repsol’s legal team ran a blind evaluation of Harvey, Microsoft Copilot and its own internal prototype. Harvey scored highest on quality. An eight-week pilot with 50 lawyers across 12 jurisdictions followed. Harvey won on accuracy and time savings. The firm then signed a full rollout.
- The test, step by step
- What the numbers actually prove
- The cost line most firms skip
- How a Canadian firm runs the same test
The test, step by step
The blind round came first. Lawyers judged output without brand names attached. That order matters. A famous logo flatters weak work, and a home-built tool gets judged by its builders. Blind scoring removes both biases in one move.
The pilot came second. Fifty lawyers used the tools on real matters for eight weeks. Twelve jurisdictions meant different laws, different templates and different habits. A tool that only works for one team fails that design. Harvey reported the results after the rollout. Lawyers saved four to six hours per week. Adoption reached 96% across the legal department.
Notice what the test never asked. It never asked which tool felt modern. It never asked which demo impressed the partners. It asked which output a lawyer could trust, and how much time came back each week.
What the numbers actually prove
Four to six hours a week is a payroll number. Across a 50-lawyer pilot, that is 200 to 300 hours returned every week. A firm can price that against any licence fee in one line of arithmetic. The 96% adoption figure matters as much as the hours. A tool that staff avoid returns nothing, however strong its benchmark scores look.
Benchmarks alone mislead buyers. The best AI in the world scored 12% on real professional work shows how far test scores sit from office work. The Repsol design closes that gap. It measures the firm’s own work, in the firm’s own files, under the firm’s own deadlines.
The timeline also deserves a line. The blind test ran first, the pilot opened in January 2024, and the full rollout was signed in April 2024. The whole sequence took months, not years. A firm does not need a transformation program to run it. It needs ten past files, eight weeks of live work and the discipline to score before it shops.
There is a second lesson in the losing tools. Copilot was already in the building, inside the suite the firm pays for. The internal prototype cost staff time instead of licence fees. The premium tool still won, because the test priced the whole job. Cheap per seat is not cheap per finished contract.
The cost line most firms skip
The full cost of one finished task includes more than the subscription. It includes model calls, retries and failed attempts. It includes the human review before the work can leave the building. Count only the invoice and the free tool wins every time. Count the review hours and the ranking can flip.
Token counts will not save you here. Stop counting tokens. Even the man who sells them calls the metric fake explains why usage totals hide the real bill. Price one completed unit instead. One contract reviewed. One file closed. One report a partner signed.
That discipline also explains a national pattern. Canadian AI ROI, Only 2% of Businesses Report a Return found almost no firms reporting a return on AI spend. Firms that never define a finished unit cannot report one. The test forces the definition before the money moves.
How a Canadian firm runs the same test
Pick one high-volume task. Contract review, intake summaries or proposal drafts all work. Take ten real past files for that task. Strip the names from the outputs. Have the person who signs the work score each output blind.
Then run the best two tools live for four weeks. Track three numbers only. Accuracy against the signed standard. Hours saved per week, per person. Share of staff still using the tool in week four. AI is handing the second founder the first founder’s bench shows how fast small teams gain leverage when a tool survives real work. The same test tells a small firm whether it needs the bench at all.
Buy only after the pilot, and price the contract per finished unit where the vendor allows it. If the vendor refuses blind scoring, that refusal is data. A tool that cannot survive your own files will not survive your clients’ files either. The next premium renewal on your desk deserves this test before it deserves a signature.
