Hi, this is Gergely with a bonus, free issue of the Pragmatic Engineer Newsletter. In every issue, I cover Big Tech and startups through the lens of senior engineers and engineering leaders. Today, we cover one out of four topics from last week’s issue of The Pulse. Full subscribers received the article below 7 days ago. If you’ve been forwarded this email, you can subscribe here.
Over the past two weeks I’ve talked with several CTOs and founders at infra companies, and in those circles, it’s an open secret that companies are being targeted by what looks like Chinese attackers, using abliterated AI models (those with no safeguards). These attackers seem to only want AI model keys; most likely to sell at discounted rates. “Popping” an infra company and accessing thousands of AI keys can be profitable, so it’s economical to spend heavily on offensive abliterated models.
Cloud provider Railway was one of the several infra providers that was hacked two weeks ago, with customer secrets leaked. The company has released a very detailed postmortem, which is the most in-depth description of how such an attack happens and how fast attackers adapt.
At a high level, this is what happened:
- Phase #1: Several unsuccessful exploit attempts. On 18 Oct, the attacker created dummy Railway accounts and tried unsuccessfully to get into the company’s systems. The attacks are blocked, but the attacker did not give up.
- Phase #2: Get Railway’s monorepo code (5 steps in 11 hours). The steps:
- Attacker finds a vulnerability so they can read files on Railway’s build system
- → steals build credentials
- → tries the credentials on several internal systems
- → accesses a Temporal cluster that stores build histories
- → finds Railway’s monorepo as a build history and downloads it
- Phase #3: Gain SSH access to Railway’s cloud environment (4 steps in 10 hours):
- Scans the downloaded code; tries using old endpoints
- → Gets a short-lived cloud credential by exploiting a misconfigured OpenID Connect binding in the deploy pipeline
- → Uses the short-lived credential to pull a long-lived cloud credential
- → Finds an SSH key with the long-lived cloud credential
- Phase #4: Get access to customers’ AI provider keys (3 steps in 3 hours):
- With cloud access, they query the production database and access encrypted environment variables. The attacker filters for AI provider keys
- → Figures out that there’s a key management service that can decrypt the keys, which the attacker has access to with their access to the cloud environment
- → Calls the key management service to decrypt customer secrets (AI provider keys)
Let’s visualize how the attack unfolded:

I talked with Railway founder and CEO Jake Cooper, who said the attack was especially frustrating because at one point, they were only minutes behind the attacker: it was down to 1-8 minutes that the attack could unfold in the way it did because Railway’s team was alerted to their build system’s credentials being probed on internal systems eight minutes after the attacker started in phase #2. Railway revoked credentials the attacker used to download the monorepo just one minute after the attacker used them:

What makes attacks by abliterated AI models different
Railway’s summary is the first in-depth account of how an attack from an offensive AI model plays out, and there are several noteworthy details:
From the attacker’s side:
- Stolen code can be exploited within hours by an AI model. If a human got access to Railway’s monorepo, it would have likely taken days or weeks for them to figure out the weaknesses to launch further attacks at. But with AI models, all this is sped up: six hours after downloading the code, the attacker figured out a plan to get access to the AI provider keys, and knew where to look from then on.
- Abliterated AI models are like bloodhounds: relentless in chasing their goal. The offensive AI model just kept going and going, trying to get access to ever more systems with the sole goal of accessing AI provider keys. It’s hard to imagine such durability by a human in keeping going through hoop after hoop to finally get access to AI keys!
- Attackers never give up, not even after all doors and windows are sealed shut. Railway evicted the attacker and revoked all the paths they used to get in. But the attacker returned and probed for new weaknesses the same night!
From the defender’s point of view:
- Storing hashes of customer secrets is a very smart approach. As soon as Railway discovered the breach, they sent over the hashes of AI provider keys to all AI providers, so those providers could cancel them to avoid the attacker draining customer accounts. Hashing allowed Railway to send the keys without knowing them, and for providers to revoke them immediately.
- Defenders are handicapped by models’ capabilities. Attackers have access to abliterated AI models, while the models Railway has for defense – from the likes of OpenAI and Anthropic – refuse to engage in cybersecurity activity, including defensive work. Railway had no access to Project Glasswing during the incident, which would have allowed removal of some restrictions. As of today, Railway does have access, but many infra companies don’t have that privilege.
- Railway decided to move to a new build system, instead of patching the existing one. Once Railway realized that its build infra was how the attacker gained entry, they shut down all builds. This was a problem for customers, given they needed to re-deploy builds with rotated keys! Railway decided to move every customer to an unreleased build system, and not attempt to patch the existing one. Fortunately for Railway, they had the new build system nearly ready to go!
During such an attack, the vendor often has to very quickly choose between “bad” and “worse” decisions. Railway faced a couple of these:
- To simply notify customers that their API keys were leaked, or immediately notify vendors like Anthropic and OpenAI, sending over API key hashes? Railway did the latter at the risk of upsetting some customers, and even causing unaffected customers’ services to stop working until they created and set up a new API key. The trade-off was that this limited overall financial damage.
- Should Railway have allowed builds to happen after revoking the attacker’s build system credentials? Allowing customers to do builds would have meant less service disruption, but the attacker could have returned via a slightly different path and stolen the rotated keys. Railway decided to bite the bullet by halting builds and rebuilding its build system in a matter of hours.
Lessons from this incident
I’m impressed by how much Railway has reflected on the structural problems that made it easier for the attacker to succeed, and how the company fixed them:
- Builds should not run on shared machines. The whole attack started with the attacker being able to attack the build’s isolation boundary (the sandbox on the VM), and getting access to the build machine’s environment. Since that build machine ran several customers’ builds, it had credentials that were valuable to the attacker. Railway has now rebuilt its build stack so that each build gets its own throwaway virtual machine – which is a much stronger boundary than a sandbox on a VM.
- Add fine-grained access control for all internal secrets. Assume that an attacker will appear within your trust boundary and try to use secrets to get access to internal services. After the attack, Railway rebuilt its CI system to separate out sensitive components, with fine-grained access control. Getting access to important and sensitive credentials now requires 2FA, and even biometrics.
- Assume that attackers have full access to your source code. The attacker was able to come up with a better plan of attack only after getting access to Railway’s source code. Securing all of your source code is very hard – and might not even be possible in the context of social engineering attacks via an employee. So, design security as if attackers already have full source code access.
- Decommission old, unused systems. The attacker probed credentials on Railway’s old systems, and although they didn’t get far, they could have found more weaknesses. Following this attack, Railway has shut down old, unused systems.
- Increase logging and alerting. Even if you do all of the above, an attacker could get access to your systems in unexpected or novel ways. As and when this happens, ensure that you will be alerted for unusual usage patterns, like some services being accessed with invalid or old credentials, unusual internal network activity, etc.
- It’s essential to respond to suspicious activity within minutes, not hours or days later. Attackers with AI are much faster at hopping between systems than we’ve been used to seeing. Defenders have the advantage that they can spot unusual patterns; after all, an attacker is fumbling around inside an unfamiliar system. But you need to act fast to evict an attacker inside your system!
Seeking to gain from competitors’ problems: what goes around, comes around
There’s an unwritten rule among software vendors that you don’t turn competitors’ outages into a sales pitch to try and poach their customers for yourself.
But Northflank evidently didn’t get this memo. On the day of Railway’s outage, the market rival published a blog post titled “How to migrate off Railway.” The article started by linking to Railway’s security incident disclosure, and explained how to safely migrate to Northflank. In the article, Northflank also boasted about its security capabilities:
“Security controls also include BYOC, workload isolation, network security, and envelope encryption, helping teams manage infrastructure access and reduce exposure.”
The article reads like a naked attempt to grab customers from Northflank’s rival, stating: “Railway got hacked, come on over to Northflank, which is a platform that cares about security.”
Any engineering teams that decided to take the offer ended up getting hacked not once, but twice. Nine days after Railway’s hack, Northflank got hacked in a near-identical way. From Northflank’s incident summary:
“The first known decryption of [customer AI] secrets occurred between 02:39 UTC and 03:08 UTC on 30 October. Northflank received its first security platform alert at 11:43 UTC, and our engineers responded within minutes. The exploit was patched within 60 minutes of notification, and customers were notified within two hours.
Some customers have reported unauthorised use of AI secrets. Please rotate all secrets now, both integration secrets and project secrets, even if you haven’t seen suspicious activity. We recommend starting with AI provider keys, cloud provider credentials, payment provider keys, private keys, and database and other service credentials.
We also recommend reviewing your provider logs for unusual activity. If you have seen unexpected AI usage or spend, contact your AI providers directly, such as OpenAI and Anthropic.”
So, Northflank took nine hours to detect that customer secrets had been stolen, 11 hours to notify customers, and – unlike Railway – they did not proactively notify AI providers to revoke the keys. Objectively, this is a worse response than Railway’s, yet it was Northflank that tried to capitalize on the security outage.
In this particular context, companies would do better to help other players in the market, rather than dunk on their competitors, and indeed, Railway’s CEO texted Northflank to offer support and information about their attack. At first glance, this might appear a bit counter-intuitive to help rivals, but Jake Cooper realized what Northflank’s leadership seems to have missed: that it’s bad for all infra providers if many of them get hacked because customers will stop trusting all infra providers! It’s in the interest of providers to collaborate on this matter and harden their own security. Once they do so, then companies win the market based on features, pricing, and innovation.
Plus, it’s just a very short-sighted way of doing business to try to capitalize on novel security incidents, especially as Northflank seems to have been worse at dealing with the breach than the competitor they went after!
I have no affiliation with Railway or Northflank and struggle to see why Northflank did not remove the article about Railway’s security incident article. One guess would be that it’s a sign the business isn’t exactly thriving, and that they’re ready to try anything to keep things afloat.
Read the full issue of The Pulse this is from, or check out this week’s The Pulse. This week’s issue covers:
- Anthropic: the largest-ever IPO. The leading AI frontier lab raised $107B with its IPO, and around 600 early employees could see gains of $100M or more. It’s absolutely incredible for a startup that is only five-and-a-half years old.
- Is Amazon’s writing culture collapsing because of AI? A former Amazon software engineer sees signs that written artifacts – the ‘six-pager’ documents the company uses to promote good decision making – are increasingly AI-generated and AI-read, and warns it’s not a positive development.
- Meta struggles to understand devs & reverses Conductor shutdown. Meta shuttered the AI coding harness Conductor after acquiring it, but is now reversing its decision – possibly after learning that a rival harness, T3 Code, is getting four times the usage of the latest Gemini model. Oops!
- Is AI making us pay off decades of tech debt? The industry is currently on a speedrun of vulnerabilities being uncovered and patched or exploited. One way of looking at it could be that we’re on a speedrun to pay off decades of tech debt.
Subscribe to my weekly newsletter to get articles like this in your inbox. It's a pretty good read - and the #1 software engineering newsletter on Substack.