A manager reviewing a paper checklist beside a laptop on an office desk

Your first always-on agent looks great on day 9. Judge it on day 90

An always-on agent demos well because a demo is short. The agent answers fast, the room nods, and the pilot gets a wider job by Friday. The problems with a persistent agent do not arrive in week 1. They arrive quietly, in week 6 or week 10, when nobody is watching the demo anymore and the agent has been running real work on real access for 2 months. Drift is the failure mode. The agent keeps working. It just works a little further from the job you gave it, every week, in ways nobody wrote down.

The direct answer: Judge a persistent agent at day 90 with a written watch list. A day 9 demo shows launch behaviour only. Check 5 things on a set date: the access it still holds, the errors it made that nobody caught, the instructions it still follows, the records it kept, and the bill it ran up. An agent that passes all 5 earns a wider job. An agent that fails 1 gets a smaller job. The failure gets fixed and rechecked before the wider job returns. This holds in any Canadian office that runs an agent on client files, invoices or bookings.

  • Why day 9 tells you almost nothing
  • The 5 checks on the day 90 list
  • What to do with the result

Why day 9 tells you almost nothing

A new agent works from fresh instructions, a clean memory and a short task list. Its access was granted last week, by a person who still remembers granting it. The team still reads its output with care, because the tool is new and nobody trusts it yet. Every condition that makes the demo look good is temporary.

Advertisement

By day 90, each of those conditions has moved. Staff have added tasks in side conversations. Someone widened a permission to get a job done faster. The instructions written in September describe a process the office changed in October. And the team reads the output faster, because 3 months of mostly fine work teaches people to skim. The agent did not get worse in a dramatic way. The ground under it shifted, and no review date forced anyone to look.

This is the gap Give your AI agent an 8-line job description before access was written to close at the start. A job description sets the agent’s work on day 1. The watch list is the same discipline at day 90: the job, checked against the record, on a date somebody owns.

The 5 checks on the day 90 list

Check 1: access. List every system, file and account the agent can reach today. Compare the list with the access it was granted at launch. Every addition needs a name beside it: who granted it, when, and for what task. Access with no name comes off the same day. An always-on agent with quiet extra reach is a standing risk, and it stays standing on weekends.

Check 2: quiet errors. Pull a sample of the agent’s finished work from the last 2 weeks and read it against the source records. Count the errors nobody caught before the work went out. A loud error teaches the team something. A quiet error teaches a worse lesson. Repeated week after week, it trains everyone to accept the wrong output as the right shape. The sample does not need to be large. It needs to be read by a person who knows the work.

Check 3: stale instructions. Read the agent’s standing instructions beside the current process. Offices change suppliers, prices, approval steps and wording. An agent follows the old version with full confidence until a person updates it. Your agent’s memory needs an editor, not a bigger limit made this point for memory: a named editor owns what the agent keeps. The same editor owns what the agent still believes about how your firm works.

Check 4: the record. Pick 3 tasks the agent marked complete and ask for the record each one should have left behind. Pick one project marked ‘done’ and ask for the record it should have left behind set that test for human projects, and it works on agents with no changes. A task with no record counts as a claim. It does not count as a result. If the agent cannot show its work, its work does not count in the review.

Check 5: the bill. Total what the agent cost over the 90 days, in CAD: usage charges, the seats or plan tier it needs, and the staff time spent checking and fixing its output. A US plan price converted at a September rate understates a November bill, so use the rate on the invoice. Claude Projects runs threads in parallel. Set the thread budget first showed how fast parallel work multiplies a usage bill. An always-on agent has no natural stopping point, so the budget review is the stopping point a person sets. Compare the total with the cost of the work it replaced, honestly, with the checking time inside the number.

A normal office, at day 90

Picture a Canadian firm that gave an agent 3 jobs in September. It drafts the weekly client update from the project board. It chases missing timesheets on Fridays. It files supplier invoices into the right folders. On day 9 all 3 jobs looked clean. At the day 90 review, the access check found the agent could also open the payroll folder, added in October so it could answer a question about a missing pay stub, and never removed. The quiet-error sample found 2 client updates that carried a project name the firm had retired in November, because the standing instructions still held the old name. The record check passed: every invoice the agent filed carried its source email and its approval line. The bill came in modest. In this example the checking time was the larger cost, at about 1 hour a week across 2 staff.

That review took 1 meeting. The fixes took less: payroll access removed the same day, the retired project name replaced in the instructions, and the timesheet job widened to a second team because its record was the cleanest of the 3. Without a set date, the payroll access and the old project name would still be in place at day 180, and both would have surfaced later in a worse setting: a client call, or an audit question nobody could answer quickly.

What to do with the result

Score the review in plain words, in a short note the team can find later. Pass: the agent keeps its job and the next review date goes in the calendar before the meeting ends. Fix: the agent keeps a smaller job, the named failure gets a named owner, and the recheck runs in 30 days. Stop: the agent loses access the same day, and its task list returns to a person until a new launch review starts from the job description again.

Resist the middle outcome where the review raises concerns and changes nothing. A watch list that ends in a shrug trains the team to treat drift as normal. The agent’s whole value is that it works while nobody watches. The review is the price of that value: 1 hour, every 90 days, with the record open on the table.

The recap is simple. Day 9 shows you an agent on its best behaviour, with fresh instructions and a careful audience. Day 90 shows you the agent your firm actually has: its real access, its real error rate, its real cost.

Book the day 90 review on the day the agent launches, while the launch date is still in front of you. A review booked at launch happens. A review left for later competes with every busy week between now and then, and it loses.

Total
0
Shares
Prev
California says OpenAI lost control of its agents 80 times. Canada has no such count
Stone columns and steps at the front of a courthouse

California says OpenAI lost control of its agents 80 times. Canada has no such count

California filed suit against OpenAI in San Francisco Superior Court on November

You May Also Like