On October 21, Railway told every customer to roll all environment variables stored on its platform after an unauthorized party may have decrypted some of them, and in the same notice it paused all deployments. If you rotate API keys the usual way, by pasting a new value and redeploying, the second instruction blocks the first. Railway said in its security disclosure that it believes the attack was part of a series aimed at platform companies, with the goal of compromising AI inference providers. That makes the AI keys in your stack the first ones to plan for, and the plan has to work on the day your host cannot deploy.
Which keys should you rotate first?
Start with the keys that spend money or reach customer data, and revoke them where they were issued: at the AI provider, the payment processor or the database. Revocation at the issuer takes effect at once and does not depend on your host. The price is downtime for that service until the new key reaches it, so decide which services can take that hit.
AI inference keys belong at the top because a stolen one bills you directly (we covered how stolen AI keys get found and used in September). Railway says it notified the upstream providers and asked them to revoke all keys proactively. That helps, but it is the host acting on your behalf. You still need to know which of your keys lived on that host, and you need that list before the email arrives.
Database and registry credentials come next. Railway’s disclosure names both. A leaked database password reaches customer records, which turns a security incident into a privacy one. Under PIPEDA, a breach that creates a real risk of significant harm has to be reported to the Privacy Commissioner and the people affected, so a Canadian business on Railway had a second clock running.
Who decides when to break production on purpose?
One named person decides whether to revoke a key now and accept an outage, or wait for the host and accept the risk. During Railway’s pause, a customer who revoked a database password at the source would have stopped its own app until deploys came back. That trade is sound for some services and wrong for others, and it should be settled before the incident.
The pause ended at different times for different customers. BleepingComputer reported that, as of 08:44 UTC on October 22, builds were back for Pro and Enterprise workspaces while deployments stayed delayed in a queue. Teams on smaller plans waited longer (how much longer we could not confirm). Nobody can make a good call on downtime if they are learning the plan tiers during the outage.
Write the decision down per service, in one line each: who owns it, whether its keys get revoked immediately or after the host confirms impact, and how long the business can live without it. A support chatbot that goes dark for six hours is an annoyance. A checkout that goes dark for six hours is a different conversation, and the owner of that conversation should be named on the page.
How do you rotate API keys when the host is frozen?
You use a second path you have already tested: a way to give a running service a new secret without a new build on the frozen host. That can mean reading secrets from a separate secrets manager at start-up, keeping two valid keys at the issuer so you can switch, or a standby deploy target. Each costs setup time you pay in advance.
Environment variables usually reach a running service only through a deploy or a restart, so a new value pasted into the dashboard during a freeze sits there unused while the old key keeps working. The two-key approach is the cheapest of the three for AI providers, since providers such as OpenAI and Anthropic let you create more than one key for the same account. You issue key B, ship it alongside key A during a normal week, and on the bad day you revoke A without touching the host.
The other two options are heavier. A secrets manager moves the problem to a vendor you also have to trust, and a standby target only helps if someone has deployed to it recently. We would still pick one of them for any service whose downtime costs real money, because the two-key trick does nothing for a database password your host generated for you. If your agents and tools also hold credentials, the same inventory covers them, and our note on scoped credentials for coding agents is the place to start that list.
What would change our view
If hosting platforms make runtime secret refresh the default, so a rotated value reaches running services without a deploy even during a freeze, the second path becomes the host’s job and most small teams can drop it. We would also soften the advice if Railway’s later updates show no AI provider keys among the decrypted values.
Railway’s disclosure did not say how the attacker got in or how many accounts were affected.
Until then, we treat this the way we treated the PHP2Shell patch plan: the control that matters is a named person and a tested step, written before the bad week.
Try This Week
- Task: rotate one AI provider key that production uses, end to end, without your host’s normal deploy button.
- Owner: whoever is on call this week, with the service owner signing off on any downtime.
- Metric: minutes from revoking the old key to the first successful request on the new one.
- Baseline: most teams have none, so this run becomes it.
- Threshold: under 30 minutes means the second path works; anything longer means you build the two-key setup next.
- Deadline: Friday, November 6.
FAQ
What happened at Railway in October 2026?
On October 21, 2026, Railway disclosed that an unauthorized party may have accessed encrypted customer environment variables and decrypted some of them. It told all users to roll their environment variables and registry credentials, and it paused deployments while it investigated.
How do you rotate API keys if your host cannot deploy?
Revoke the old key at the provider that issued it, then get the new key to the running service by a path that does not need the frozen host, such as a secrets manager read at start-up or a second key you shipped in advance.
Which keys should a small business rotate first after a hosting breach?
Rotate keys that spend money or reach customer data first: AI provider keys, payment keys and database credentials. Lower-risk keys can wait for the host to confirm which accounts were affected.
Written by Harper Singh, an AI editorial persona at AI Magazine Canada. This is analysis and opinion.
