24 of the 31 large Canadian websites we checked on October 31 name no AI crawler in robots.txt, and none carries the new one-line AI training opt-out yet.
The AI training opt-out that OpenAI, Google, Microsoft and Anthropic backed on October 30, 2026 is a single robots.txt group, and the catch sits in the same announcement: only Anthropic says it has started reading it. For a Canadian business, that makes it a decision to take on purpose this fall.
We read the robots.txt files of 31 large Canadian websites the morning after the announcement. Banks, telecoms, retailers, an airline, two universities and the City of Calgary say nothing to AI crawlers at all. News publishers say a lot, and some of what they say has already gone out of date.
__usage_ai-training. It states a preference about training and does not stop a crawler from fetching pages.One sign on the gate, four companies reading it
The AI training opt-out works through a “policy token”, a name placed on the User-Agent line of robots.txt that stands for a purpose instead of a crawler. A site writes User-Agent: __usage_ai-training, then Disallow and Allow lines for the paths it wants out of, or in, AI model training.
Think of it as one sign on the gate that every participating company agrees to read, in place of a separate sign for each company’s truck. The IETF draft that defines it, written by an Anthropic standards lead and dated October 30, adds two rules a site owner should know before copying anything. A catch-all User-Agent: * group never counts as a training preference. If your file has no token group, you have told the participating companies nothing either way.
The commitments differ in timing. OpenAI’s announcement promises the signal “in the coming months”, with no date, and keeps GPTBot and its other crawler controls in force. Microsoft’s Bing post gives February 1, 2027. Google’s Search Central post names 2027 as the target, and a note added to its crawler documentation narrows that to early 2027.
What 31 Canadian robots.txt files say today
Most say nothing to AI trainers. Of the 31 large Canadian sites whose robots.txt we could read on October 31, 24 name none of the 11 AI crawlers and tokens we searched for, and 0 carry the new token. The six news sites are the exception: every one blocks GPTBot.
Two of the news files show why a shared token is overdue. CBC and CTV News both block GPTBot and an Anthropic label, “anthropic-ai”. Neither names ClaudeBot, the robot that Anthropic’s crawler page says collects content that could contribute to training. Read by a standard parser, both files leave ClaudeBot free to fetch the whole site. Lists of crawler names go stale, and one shared token is the fix the four companies are offering.
Five of the six news sites also block OAI-SearchBot. OpenAI’s crawler documentation says opting out of that bot keeps a site out of ChatGPT search answers, apart from navigational links. That is a fair choice for a paywalled newsroom. For a dental clinic in Lethbridge it would mean disappearing from an answer engine on purpose.
Should a Canadian business add the AI training opt-out?
Publishers, course creators and anyone selling written expertise should add it now, beside their existing per-crawler rules, because Anthropic has started and the others have dates. A business whose website is mostly marketing should wait and decide training and AI answers as two separate questions, because one blanket opt-out can reach further than training.
The reason is Google’s reading. Google says a token opt-out in a file that never mentions Google-Extended will be read as a Google-Extended block. Google-Extended governs Gemini training and also decides whether a site can appear in cited Gemini app answers. A Calgary renovation firm that wants Gemini to cite its price guide should add an explicit Allow for Google-Extended before adding the blanket line. Microsoft is splitting the same question into a separate grounding setting at page level.
Shopify merchants have one more wrinkle. Shopify’s help page on robots.txt says product data sent to ChatGPT or Copilot storefronts through Shopify Catalog goes out “independently of /robots.txt”. Blocking crawlers changes open-web discovery only. If you have weighed AI shopping agents for your Shopify store, that setting lives in a different screen.
The part the token cannot do
It cannot enforce anything. The draft says plainly that the token “is not a directive for crawling behavior”, so a company that ignores it can still fetch your pages, and the preference that counts is the one in the file when a page was retrieved. Pages fetched before you add the line were fetched under your old file. Microsoft adds that its new settings “do not establish licensing or payment arrangements”.
We ran a read-only check of 31 public files. We did not test whether any crawler obeys the token, because Anthropic’s own standards lead wrote on October 30 that it would “take us some time to add support across our services”, and the other three have not started. A user agent is also only a label, which is why spoofed crawlers can wave it around, as we found in our look at fake AI crawlers asking for password files. If OpenAI and Google still do not read the token by March 1, 2027, the date Bing stops treating its old NoArchive and NoCache tags as AI preferences, per-crawler lines remain the only control that works.
Your robots.txt in 20 minutes on Monday
Open yourdomain.ca/robots.txt in a browser and write down three answers. Does any group name GPTBot, ClaudeBot or Google-Extended? Does the file carry a __usage_ai-training group? Does it say what you meant about AI search answers?
User-Agent: __usage_ai-training Disallow: / User-Agent: Google-Extended Allow: / User-Agent: GPTBot User-Agent: ClaudeBot Disallow: /
The pass mark is a file where every AI line matches a decision someone in the company signed off, recorded in one sentence for training and one for AI answers. On WordPress with Yoast, check whether your plugin writes the file before you edit it. If you are still weighing whether AI answers send you any traffic, our view on paying for AI search visibility is the place to start. More tool checks for Canadian teams sit in our AI tools for Canadian businesses hub.
FAQ
What is the AI training opt-out in robots.txt?
It is a robots.txt group whose User-Agent line reads __usage_ai-training, followed by Disallow and Allow lines. It tells participating AI companies which paths you do not want used for AI model training. It is defined in an IETF draft dated October 30, 2026.
When will OpenAI, Google and Microsoft honour the AI training opt-out?
Microsoft says from February 1, 2027. Google says early 2027 in its crawler documentation. OpenAI says it aims to implement it in the coming months, with no date. Anthropic says it has started implementing support, which will take time across its services.
Does the AI training opt-out stop AI crawlers from visiting my site?
No. The draft says the token is not a directive for crawling. To limit fetching, keep groups for each crawler, such as GPTBot or ClaudeBot. Robots.txt rules are also advisory, so a crawler that ignores them can still fetch pages.
Will blocking AI training hurt my Google search ranking?
Google says Google-Extended does not affect inclusion or ranking in Google Search. It does decide whether Gemini can use your pages for grounding in Gemini app answers, and Google will read a blanket opt-out as a Google-Extended block unless you name Google-Extended yourself.
Does robots.txt control what Shopify sends to ChatGPT shopping?
No. Shopify’s help centre says product data reaches agentic storefronts through Shopify Catalog independently of robots.txt. Those choices sit in your agentic storefront settings.
One line is a real improvement over a list of names nobody updates, and the CBC and CTV files show why. Add it if your content is the product. If your website is the shop window, decide the Google-Extended line first, then add the token.
Facts in this column were checked on October 31, 2026 against the OpenAI, Google, Microsoft and Shopify pages linked above, the IETF draft and Anthropic’s October 30 working group message, and the robots.txt files we fetched that morning. No crawler’s behaviour was tested for this column.
